From Karafka Ractors to Yoichi Whisky: My RubyKaigi 2026 Experience

Introduction

I just returned from RubyKaigi 2026, held from April 22nd to 24th in Hakodate, Hokkaido. For those unfamiliar with it, RubyKaigi is the biggest Ruby conference in the world, drawing speakers, committers, and Rubyists from across the globe. As always, it managed to combine deep technical talks with a uniquely Japanese atmosphere that no other conference comes close to.

This year's edition had a distinctly Hokkaido feel: the cold winds of the northernmost island, the smell of the sea, kaiseki dinners, sashimi at conference parties, and onsens - a lot of onsens. The conference itself was in the south of Hokkaido (Hakodate). Still, for me, the trip extended significantly further north and east thanks to my Japanese friend Hasumi Hitoshi, who once again made my Japan adventure something far beyond just a conference visit.

What I love most about RubyKaigi is how it bridges the gap between the Japanese and Western Ruby worlds. Despite Ruby coming from Japan, these two communities still feel oddly separate in day-to-day work - both reinventing the wheel from time to time, both not drawing as much from each other as we should. RubyKaigi is the rare place where these worlds collide. You meet the people whose code you've used for years, you hand them a beer, and suddenly Ruby's global ecosystem feels a bit more whole.

Pre-Conference: Tokyo Stopover and Heading North

My journey started in Tokyo, where I landed early in the morning. Passport control took 40 minutes, unusually long, but otherwise the trip was smooth. With a couple of hours to kill before my Shinkansen to Hakodate, I wandered around Tokyo Station and walked through the park near the Imperial Palace. There's something about Japanese parks that always gets me - even ten minutes from a major train station, the city noise disappears.

The Shinkansen ride was uneventful, but the local train from Shin-Hakodate to Hakodate brought an unexpected moment: an earthquake alarm went off mid-ride. Shortly after we pulled into Hakodate Station, tsunami sirens started wailing across the platform. But not a single Japanese person around me was panicking or evacuating, so I figured everything would be fine. It was. Welcome to Japan.

Day 0 - Hakodate Settling In

I arrived in Hakodate, settled into my hotel (which had its own quirks: a glitchy speaker playing a melody on loop and a wheezing AC unit), and did what any jet-lagged traveler does on day one: I slept. About 15 to 16 hours, with one break for breakfast.

Day 0 also brought the ANDPAD Welcome Drinkup, the unofficial kickoff for many of us. Roughly a hundred people crammed into Jimotoya, a local izakaya with great food.

The Conference Experience

Day 1 - Tagomori, Hasumi, and the Wind

The conference opened with Satoshi Tagomori's keynote on Box, his ongoing work on namespacing in Ruby. As I wrote about last year regarding namespaces, I have strong reservations about the feature. My core concern remains: until the surrounding ecosystem (RubyGems, Bundler) properly supports it, releasing Box risks fragmenting the Ruby ecosystem in ways we'll spend years cleaning up.

That said, Tagomoris absolutely deserved this keynote slot. The scope, the thinking, the engineering rigor, and the fact that he's willing to engage with critics (including me) about hard tradeoffs all make him one of the people I most respect in this community. I love this guy. I don't love this particular feature, at least not yet.

Later that day, Hasumi Hitoshi delivered his PicoRuby talk. Each year, I'm more impressed with how fast PicoRuby is evolving. Running Ruby on IoT devices used to feel like a curiosity; now it has serious, real-world use cases. There were noticeably more PicoRuby-related talks at this year's RubyKaigi, which suggests where the community sees this going. Hasumi-san's work on this, alongside PRK Firmware, IRB, and Reline, continues to be foundational for a whole subset of the Ruby world that doesn't always get the spotlight at major conferences.

The day wrapped with the RubyKaigi 2026 Official Party. The Japanese style of holding the official party in a hotel always slightly surprises me at first - back home, we'd probably do this kind of event in a pub or bar that had been reserved entirely for us - but late April in Hokkaido makes the indoor decision obvious. Big plus for the beer selection, which was excellent.

I stayed out until 1 am, wandering through Hakodate with a great group. The city, which on first impression I'd written off as a slightly run-down outpost, was already starting to grow on me. The wind, on the other hand, was absolutely brutal: without my winter hat, my head would have fallen off.

Day 2 - Nutter, Zhu, Tenderlove, and My Talk

Day 2 was, for me, the strongest technical day of the conference.

It opened with Charles Nutter's keynote on JRuby's history. This was well executed, presented as a journey rather than a lecture, packed with anecdotes, and giving real credit to all the people who built JRuby over the years. It's genuinely a bummer that Charles still has to spend time explaining what JRuby is to newcomers; given its capabilities, JRuby should be far more mainstream in many use cases than it actually is. The work he and the JRuby team do is impressive, and this talk did a great job of contextualizing it within the broader Ruby story.

Next came Peter Zhu with his talk on the next-generation GC for Ruby. Two things I always say about Peter: first, he delivers from both a presentation and a content standpoint; his talks are dense without being overwhelming. Second, his work on CRuby's GC is genuinely among the most important infrastructure work in our community right now. He laid out a clear roadmap and walked through the technical motivation. Easily one of my favorite talks of the conference.

Then Aaron Patterson took the stage to talk about faster FFI. I've been spending significant time recently rewiring large parts of Karafka and rdkafka-ruby to reduce the number of crossings between Ruby and C via FFI. Every transition has overhead, and at Karafka's scale, it adds up fast. Watching Aaron work on the language side of the same problem was encouraging. I plan to set up a stable benchmarking machine for the Karafka ecosystem, and I hope Aaron's work will eventually appear in those benchmarks.

A side note worth making explicit, every year: the RubyKaigi translation team is incredible. Simultaneous interpretation between Japanese and English is one of the things that make this conference uniquely accessible, and the people doing that work rarely get the credit they deserve. Thank you.

My Talk - Ractors in Karafka, in Production

Like in 2025, I want to give my own talk its own short section.

I spoke about using Ractors in production in Karafka. The hardest part of giving this talk was that while I was talking about my actual implementation of Ractor support in Karafka, I was also discussing Ractors themselves, which I obviously didn't implement and which still have well-known limitations.

The feedback was good, including from members of the Ruby Core Team, which mattered a lot to me. The core message I wanted to land was: Ractors, even with their current limitations, have a real, workable niche today, and they can be used in production. This is the chicken-and-egg problem we have to break. If no one uses Ractors because they're limited, no one will provide the production data the core team needs to improve them. Someone has to go first.

The slides are available below.

After the talks, the evening continued with the Treasure Data Hakodate Night Drinkup on the Tram. Big shout-out to Treasure Data for one of the most creative event ideas I've seen at any conference: a tram filled with sake and beer, wandering through Hakodate. I have a soft spot for this kind of slightly-crazy execution. Met some great people, had excellent sake, and capped it off with one more drink at a hotel bar with a group of Japanese Rubyists.

The night, however, kept going. I ended up at one of those tiny, deeply local izakayas: six chairs, paper menus in Japanese only, no prices, and one older man behind the counter, grilling yakitori on a tiny, smoking grill. ¥100 a stick. I had six. They were excellent. I will never understand the economics of these places, how a tiny restaurant with capacity for one Japanese regular and one bewildered tourist stays in business. Still, I am eternally grateful that they do. My jacket smelled like a campfire for the next 48 hours, and it was worth every minute.

Day 3 - John Hawthorn and Hunting GC Bugs

Day 3 brought John Hawthorn's talk on Write Barriers, which hit close to home. I ran into a brutal write-barrier-related issue in 2025, which I documented in detail in When Your Hash Becomes a String: Hunting Ruby's Million-to-One Memory Bug. If you've never had to debug a GC issue in production where a Ruby Hash somehow turns into a String, count yourself lucky.

It's encouraging to know that people are working to improve GC-related tooling, because anyone who has hunted for these bugs understands the unique pain of debugging something that, by definition, only manifests when the runtime is doing something invisible to the user. John's talk was a great roadmap, and I felt every second of the GC-debugging journey he described.

The Hallway Track

Before moving on, I want to call out something that doesn't appear on any official schedule but is the single most important part of every RubyKaigi: the hallway track. The conversations between sessions, in the lobby, over coffee, at the booths, in the food line - that's where actual collaboration happens.

If you ever make it to RubyKaigi, please don't try to attend every talk. The talks are recorded. The hallway track isn't.

The Evening Events - Thanks to the Sponsors

On Day 3 evening, before the post-conference adventures began, I attended a few more drink-ups. There are so many sponsored events at RubyKaigi that you genuinely can't go to all of them, and one of the highlights of every Kaigi is choosing which ones to drop into.

I want to thank the sponsors who made these events possible. Across the conference, I either attended or benefited indirectly from events organized by:

  • ANDPAD Inc. - Welcome Drinkup (Day 0) and Code Party (Day 2)
  • Treasure Data - the sake-and-beer tram ride (Day 2)
  • The RubyKaigi 2026 organizing team - the Official Party (Day 1)
  • All the smaller drinkups across Days 1-3 from STORES, Hello Inc., IVRy, Leaner Technologies, ESM, CodeCast, note inc., Findy, Studist, OPTiM, giftee, Coincheck, GMO Internet Group, hacomono, Link and Motivation, mov, freee, pixiv, Net Protections, and many more I'm probably forgetting.

These events are what turn RubyKaigi from a conference into a community.

Post-Conference Adventures with Hasumi-san

Now for the part that, as in 2025, ends up being almost as important as the conference itself: the trip after.

This year, I had the enormous privilege of being invited by Hasumi-san for a road trip through Hokkaido. He had planned the route, organized the rental car, arranged accommodations, and generously shared his time over multiple days to show me parts of Hokkaido I would never have seen on my own. None of this would have happened without him. Thank you, Hasumi-san. 🙏

Hokuto, Futamata Radium Spa, Soba in Makkari, and Lake Hangetsu

We rented a car after the conference and drove first to Hokuto for the cherry blossoms. From there to Futamata Radium Spa (二股ラヂウム温泉旅館), an old onsen tucked deep in the forest. The water is famously radon-rich and forms a 25-meter limestone dome, one of only two formations of its kind in the world (the other being at Yellowstone), now a registered natural monument of Hokkaido.

For lunch, we stopped at Ishimame Soba in the village of Makkari, on the southern slopes of Mount Yōtei - a small, family-run place ranked among the top 75 soba restaurants in Japan and accessible only by car. Then on to Lake Hangetsu (半月湖), a crater lake at the foot of Mount Yōtei (often called "Ezo Fuji" for its resemblance to the real thing), formed roughly 3,000 years ago by a side eruption.


Otaru - Ceramics, a Polish Connection, and Finally the Nigori Sake

From Lake Hangetsu, we headed to Otaru. This beautifully preserved port town boomed during the Meiji and Taishō eras, then quietly fell out of fashion when the herring industry collapsed, which is exactly why all the old buildings are still standing.

Otaru also gave me the moment I'd been chasing since the start of the trip: finally finding nigori sake. I'd been turned away from some izakayas in Hakodate, and it took until a quiet evening at a small Otaru bar to finally get a glass of it.

Yoichi - Whisky and History

From Otaru, we took a short train ride to the Nikka Whisky Yoichi Distillery. The short version of the story: Masataka Taketsuru, son of a Hiroshima sake-brewing family, traveled alone to Scotland in 1918 to learn whisky-making. He studied chemistry at Glasgow, worked at three Scottish distilleries, married a Scottish woman named Rita Cowan in 1920 against the wishes of both their families, and brought her back to Japan. After working at what would become Suntory, he founded his own distillery in Yoichi in 1934 - chosen specifically for how closely it resembled Scotland: cold winters, peat, sea air, mountain water.

Walking through Yoichi today, you can feel that story in every building. The pagoda-roofed kiln towers, the pot stills heated directly by coal fire (one of the few major distilleries in the world still doing this), and ten buildings now designated Important Cultural Properties of Japan.

A Detour to Sapporo

Our route did not include Sapporo, but circumstances led us to make a quick detour into the city. I had about 45 minutes there - just enough to grab some Japanese green tea and a Pokémon toy for my son. Sapporo looked beautiful and was worth a proper visit some other time. It's only 40 minutes from Otaru, much closer than I'd realized.

Lake Shikotsu

For the final leg, we drove on to Lake Shikotsu (支笏湖) and the historic Marukoma Onsen Ryokan (丸駒温泉旅館), founded in 1915, where we stayed for the night.

Marukoma has one of the most unusual onsens in Japan: among its several pools, there is one outdoor bath whose water level rises and falls with the lake, as it is naturally connected to it. Unfortunately, the lake level was too low during our stay, so this particular pool was out of use - but the other pools were just as relaxing. We went in twice, once in the late afternoon and again at 4:30 am the next morning, my last day in Japan, to catch the sunrise over the lake and the surrounding mountains. Worth every minute of lost sleep.

Lake Shikotsu itself is the second-deepest lake in Japan (363 m), one of the clearest in the world (visibility around 20 meters), and so deep and warm that it never freezes, even in Hokkaido winters.

A Final Day in Narita

After Hasumi-san dropped me off at New Chitose Airport, I flew down to Narita in the morning, where I had about eight hours before my Tokyo-Warsaw flight. I had originally planned to treat Narita as an airport with a hotel and was bracing myself for a boring last day. I could not have been more wrong.

The town itself is a 1080-year-old pilgrimage center built around Naritasan Shinshōji, one of the top three pilgrimage temples in the country (about 5.5 million visitors a year), with a beautifully preserved Edo-period shopping street leading up to it. You can eat unagi from restaurants that have been preparing it the same way for over 300 years.

I had unagi-don for lunch, then bought local sake at Choumeisen (長命泉), a small brewery on Omotesando. I'd hoped to bring back a bottle of nigori. Still, the owner gently warned me that her unpasteurized nigori would ferment in my luggage and possibly explode mid-flight - a wonderful, very Japanese moment of someone protecting me from myself. I bought a pasteurized junmai instead.

I also attended the Goma Fire Ritual at Daihondō, the temple's main hall. Five times a day, every day, for over 1,000 years without interruption, monks light a sacred fire while chanting Shingon Buddhist mantras, accompanied by drums and bells. The flames rise meters into the air. Worshippers can hand over personal items to be blessed in the smoke.

If you ever have an eight-hour layover at Narita, do yourself a favor and skip the airport lounge. Go to Naritasan instead.

Why RubyKaigi Matters

Looking back at RubyKaigi 2026, I again notice that the talks, while excellent, are not what make this conference unique. The talks will be on YouTube. What you can't replicate is the connections, the conversations, the post-conference trips.

A few things I keep thinking about:

The Japanese and Western Ruby scenes are still too separated. Work and products evolve in parallel, with both sides reinventing each other's wheels because we don't talk enough. RubyKaigi pushes against this every year, but three days isn't enough. Honestly, this should be five.

The hospitality of Japanese Rubyists is on another level. Hasumi-san planned a multi-day trip to show me parts of Hokkaido I'd never have found on my own; Japanese attendees patiently practicing English with foreigners at every drink-up; sponsors going out of their way to organize creative events. I don't know any other tech community that operates like this.

The post-conference time matters. People treat RubyKaigi as part vacation, part professional development, and that combination of sightseeing and serious technical conversation is what makes it different from any other conference I've attended. The "official" three days are a fraction of the real experience.

Summary and Final Thoughts

Reflecting on RubyKaigi 2026: the technical content was great, the social events were unmatched, and the post-conference trip across Hokkaido turned this into something far beyond a conference. None of that last part would have happened without Hasumi-san - thank you, once again, for being an extraordinary friend and host.

RubyKaigi 2027 will be in Miyazaki. I'm already looking forward to it. If you've never been to a RubyKaigi, start planning now.

🥃🛕🍶

One Thread to Poll Them All: How a Single Pipe Made WaterDrop 50% Faster

This is Part 2 of the "Karafka to Async Journey" series. Part 1 covered WaterDrop's integration with Ruby's async ecosystem and how fibers can yield during Kafka dispatches. This article covers another improvement in this area: migration of the producer polling engine to file descriptor-based polling.

When I released WaterDrop's async/fiber support in September 2025, the results were promising - fibers significantly outperformed multiple producer instances while consuming less memory. But something kept nagging me.

Every WaterDrop producer spawns a dedicated background thread for polling librdkafka's event queue. For one or two producers, nobody cares. But Karafka runs in hundreds of thousands of production processes. Some deployments use transactional producers, where each worker thread needs its own producer instance. Ten worker threads means ten producers and ten background polling threads - each competing for Ruby's GVL, each consuming memory, each doing the same repetitive work. Things will get even more intense once Karafka consumer becomes async-friendly, as it is under development.

The Thread Problem

Every time you create a WaterDrop producer, rdkafka-ruby spins up a background thread (rdkafka.native_kafka#<n>) that calls rd_kafka_poll(timeout) in a loop. Its job is to check whether librdkafka has delivery reports ready and to invoke the appropriate callbacks.

With one producer, you get one extra thread. With 25, you get 25. Each consumes roughly 1MB of stack space. Each competes with your application threads for the GVL. And most of the time, they're doing nothing - sleeping inside poll(timeout), waiting for events that may arrive once every few milliseconds.

I wanted one thread that could monitor all producers simultaneously, reacting only when there's actual work to do.

How librdkafka Polling Works (and Why It's Wasteful)

librdkafka is inherently asynchronous. When you produce a message, it gets buffered internally and dispatched by librdkafka's own I/O threads. When the broker acknowledges delivery, librdkafka places a delivery report on an internal event queue. rd_kafka_poll() drains that queue and invokes your callbacks.

The problem is how rd_kafka_poll(timeout) waits. Calling rd_kafka_poll(250) blocks for up to 250 milliseconds. From Ruby's perspective, this is a blocking C function call. The rdkafka-ruby FFI binding releases the GVL during this call so other threads can run, but the calling thread is stuck until either an event arrives or the timeout expires.

Every rd_kafka_poll(timeout) call must release the GVL before entering C and reacquire it afterward. This cycle happens continuously, even when the queue is empty. With 25 producers, that's 25 threads constantly cycling through GVL release/reacquire. And there's no way to say "watch these 25 queues and wake me when any of them has events."

The File Descriptor Alternative

Luckily for me, librdkafka has a lesser-known API that solves both problems: rd_kafka_queue_io_event_enable().

You can create an OS pipe and hand the write end to librdkafka:

int pipefd[2];
pipe(pipefd);
rd_kafka_queue_io_event_enable(queue, pipefd[1], "1", 1);

Whenever the queue transitions from empty to non-empty, librdkafka writes a single byte to the pipe. The actual events are still on librdkafka's internal queue - the pipe is purely a wake-up signal. This is edge-triggered: it only fires on the empty-to-non-empty transition, not per-event.

The read end of the pipe is a regular file descriptor that works with Ruby's IO.select. The Poller thread spends most of its time in IO.select, which handles GVL release natively. When a pipe signals readiness, we call poll_nb(0) - a non-blocking variant that skips GVL release entirely:

100,000 iterations:
  rd_kafka_poll:    ~19ms (5.1M calls/s) - releases GVL
  rd_kafka_poll_nb: ~12ms (8.1M calls/s) - keeps GVL
  poll_nb is ~1.6x faster

Instead of 25 threads each paying the GVL tax on every iteration, one thread pays it once in IO.select and then drains events across all producers without GVL overhead.

One Thread to Poll Them All

By default, a singleton Poller manages all FD-mode producers in a single thread:

When a producer is created with config.polling.mode = :fd, it registers with the global Poller instead of spawning its own thread. The Poller creates a pipe for each producer and tells librdkafka to signal through it.

The polling loop calls IO.select on all registered pipes. When any pipe becomes readable, the Poller drains it and runs a tight loop that processes events until the queue is empty or a configurable time limit is hit:

def poll_drain_nb(max_time_ms)
  deadline = monotonic_now + max_time_ms
  loop do
    events = rd_kafka_poll_nb(0)
    return true if events.zero?       # fully drained
    return false if monotonic_now >= deadline  # hit time limit
  end
end

When IO.select times out (~1 second by default), the Poller does a periodic poll on all producers regardless of pipe activity - a safety net for edge cases like OAuth token refresh that may not trigger a queue write. Regular events, including statistics.emitted callbacks, do write to the pipe and wake the Poller immediately.

The Numbers

Benchmarked on Ruby 4.0.1 with a local Kafka broker, 1,000 messages per producer, 100-byte payloads:

Producers Thread Mode FD Mode Improvement
1 27,300 msg/s 41,900 msg/s +54%
2 29,260 msg/s 40,740 msg/s +39%
5 27,850 msg/s 40,080 msg/s +44%
10 26,170 msg/s 39,590 msg/s +51%
25 24,140 msg/s 36,110 msg/s +50%

39-54% faster across the board. The improvement comes from three things: immediate event notification via the pipe, the 1.6x faster poll_nb that skips GVL overhead, and consolidating all producers into a single polling thread that eliminates GVL contention.

The Trade-offs

Callbacks execute on the Poller thread. In thread mode, each producer's callbacks ran on its own polling thread. In FD mode with the default singleton Poller, all callbacks share the single Poller thread. Don't perform expensive or blocking operations inside message.acknowledged or statistics.emitted. This was never recommended in thread mode either, but FD mode makes it worse - if your callback takes 500ms, it delays polling for all producers on that Poller, not just one.

Don't close a producer from within its own callback when using FD mode. Callbacks execute on the Poller thread, and closing from within would cause synchronization issues. Close producers from your application threads.

How to Use It

producer = WaterDrop::Producer.new do |config|
  config.kafka = { 'bootstrap.servers': 'localhost:9092' }
  config.polling.mode = :fd
end

Pipe creation, Poller registration, lifecycle management - all handled internally.

You can differentiate priorities between producers:

high = WaterDrop::Producer.new do |config|
  config.polling.mode = :fd
  config.polling.fd.max_time = 200  # more polling time
end

low = WaterDrop::Producer.new do |config|
  config.polling.mode = :fd
  config.polling.fd.max_time = 50   # less polling time
end

max_time controls how long the Poller spends draining events for each producer per cycle. Higher values mean more events processed per wake-up but less fair scheduling across producers.

Dedicated Pollers for Callback Isolation

By default, all FD-mode producers share a single global Poller. If a slow callback in one producer risks starving others, you can assign a dedicated Poller via config.polling.poller:

dedicated_poller = WaterDrop::Polling::Poller.new

producer = WaterDrop::Producer.new do |config|
  config.kafka = { 'bootstrap.servers': 'localhost:9092' }
  config.polling.mode = :fd
  config.polling.poller = dedicated_poller
end

Each dedicated Poller runs its own thread (waterdrop.poller#0, waterdrop.poller#1, etc.). You can also share a dedicated Poller between a subset of producers to group them - for example, giving critical producers their own shared Poller while background producers use the global singleton. The dedicated Poller shuts down automatically when its last producer closes.

When config.polling.poller is nil (the default), the global singleton is used. Setting a custom Poller is only valid with config.polling.mode = :fd.

The Rollout Plan

I'm being deliberately cautious. Karafka runs in too many production environments to rush this.

Phase 1 (WaterDrop 2.8, now): FD mode is opt-in. Thread mode stays the default.

Phase 2 (WaterDrop 2.9): FD mode becomes the default. Thread mode remains available with a deprecation warning.

Phase 3 (WaterDrop 2.10): Thread mode is removed. Every producer uses FD-based polling.

A full major version cycle to test before it becomes mandatory.

What's Next: The Consumer Side

The producer was the easier target - simpler event loop, more straightforward queue management. I'm working on similar improvements for Karafka's consumer, where the gains could be even more significant. Consumer polling has additional complexity around max.poll.interval.ms and consumer group membership, but the core idea is the same: replace per-thread blocking polls with file descriptor notifications and efficient multiplexing.


Find WaterDrop on GitHub and check PR #780 for the full implementation details.

Copyright © 2026 Closer to Code

Theme by Anders NorenUp ↑