This gem had a huge impact in May of this year on one of Airbnb’s Ruby services. Last year when I was still at Airbnb I did the research and foundational work to make Airbnb’s HTTP client gem compatible with the Async gem and Falcon server. I’m no longer there but this year, one of my friends who’s still at Airbnb put that work to use on Airbnb’s Ruby service that talks to a Genesys API.
Most Genesys API calls have reasonable response times but there are these occasional, weird, unpredictable 30-second response times. The slow responses can come in bursts, where you can a bunch of them at once. This makes it extremely hard to predictably scale the Ruby service that calls it, because in normal synchronous mode, every thread that calls Genesys is blocked and on longer available to take more client requests. You could scale up the service to assume these 30-second response times will happen all the time, but then you’ve got huge server costs just for the 1% case.
But using Async and Falcon (an Async-compatible HTTP server), Airbnb was able to safely scale down the service by around 85% and turn on autoscaling, and it’s working beautifully.
I would love to learn a little more here. So for 30 second tail latency cases, doesn't Ruby HTTP client support a timeout with cancellation? The only useful case where I believe Async Ruby can help is the parallel requests cases.
Disclaimer: I am not a full time Ruby person, so if it sounds naive please feel free to point it out.
Yes, Ruby’s HTTP client supports timeout. As I mentioned, we weren’t using Ruby’s HTTP client, we were using Airbnb’s HTTP client with some additional capabilities. But it also supports timeout. The issue here was that I don’t think it would help to time out with cancellation and then try again (if that’s what you’re suggesting). Genesys API calls weren’t failing, they were just returning very slowly. A retry wasn’t necessarily likely to return any faster since the slow responses tended to happen in bursts, presumably based on something weird happening on their side. Meanwhile, phone calls continued coming in to Airbnb’s support center (this gem handles incoming calls) and we needed to continue routing those to Genesys as well.
You mentioned “The only useful case where I believe Async Ruby can help is the parallel requests cases.” That’s what was happening here: because of the tail latency, we sometimes needed to have a lot more parallel requests.
Ruby’s built in timeout library creates another thread with a sleep call. If something is blocking on a socket in the C code, the timeout doesn’t work.
> any blocking operation (a method where Ruby interpreter waits) is compatible with Async and will work asynchronously within Async code block with Ruby 3.0 and later.
That's pretty magical. Does this mean it would be possible to implement structured concurrency [0], but without the function coloring problem?
Regardless, I think I prefer how this looks from a code read/writeability perspective compared to Zig's or Swift's (and potentially future Rust's [2]) approaches.
It's obvious from the examples provided in the post. For example, you can use the method `URI.open` both synchronously and asynchronously.
It's something I didn't want to mention in the article because it's a relatively advanced async concept, but yea - Async Ruby is already colorless and it's great.
I find it deeply irritating that this is how we choose to describe things these days. Rather than talking about modality, or even using a few more words to describe the actual differences in terms of the underlying programing language models.
Telling me that Ruby's async is colorless implies that I know about the description of programming models as "colors" of code, which was just some random analogy someone came up with which caught on. Nothing about this has anything to do with color. Even syntax highlighting doesn't really fit. This is akin to particle physicists getting cheeky and calling one of the quantum states "color" when it has nothing to do with wavelength (correct me if I'm wrong).
This is an appeal to stop trying to be so cute. Thank you.
As someone who operates mainly in Python, I am so jealous. As far as I'm aware [1], you have to re-write your tooling to take advantage of async in Python. Does anyone have any insight into why Python async doesn't work the same way? Does it come down to fundamental language differences?
> Does it come down to fundamental language differences?
No, I don't think there's anything fundamentally different.
Ruby 3.0 implements a "fiber scheduler" feature that enables "colorless Async". Fiber scheduler is an obscure Ruby feature, but the end result is brilliant. It was also a huge amount of work.
Side note: Fiber scheduler was implemented by the same guy who created Async Ruby - Samuel Williams. This guy is the mastermind (and master-coder) behind this project.
> Does anyone have any insight into why Python async doesn't work the same way?
Ruby async seems to be implemented using stackful coroutines. IIRC Guido has been opposed to adding these to core Python, preferring stack-less coroutines because they require every yield-point to be explicit (i.e. marked with `await`).
There are libraries for a python that support stackful coroutines, such as gevent.
I'm optimistic that this will be less painful as more of the Python ecosystem becomes async-friendly. We have `aiohttp` as a suitable replacement for `requests`, and major libraries like Django (not ORM yet), Flask, FastAPI, SQLAlchemy now have async support as well.
I am actually not a 100% how far the compatibility shows. When I test Async gem with httparty and open-uri, there is no speed-up when comparing with threads:
I stopped using python long ago, out of the frustration with the breakage caused in the ecosystem first due to the 2/3 transition, and then by splitting again the ecosystem with all the async stuff they introduced, when having instead gone the way of gevent would have ended up in something like this.
The company I was at the time tried really hard to move to python 3. It was a huge codebase with thousands of engineers. After more than a year of work they cancelled the project and said they'll never upgrade to python 3, and instead started writing smaller "microservices" around the core product just to calm engineers anxiety to use python3. Now Django is suffering kind of the same problem with all the async stuff being bolted on, adding a lot of gotchas and side effects and incompatibilities around every corner. It is a mess.
I left that company and I will never use python again.
I love how Ruby seems to care about the ecosystem, and both ruby and rails keep innovating and providing awesome full stack solutions.
Whether or not a software project succeeds is substantially influenced by politics.
When I read your post a strong anti python 3 feeling is conveyed.
It may well be that it’s impossible to upgrade such a company/code base when the engineers have such a negative attitude.
The barrier was perhaps not technical but about the attitude of the people.
There were lots of python 2 developers who were rabidly anti python 3. Some of the most well known python developers were aggressively very publicly anti python 3. Imagine a python 3 upgrade project with that attitude prevailing. No chance of success.
Today python 3 is more popular than ever before and is arguably the most popular language in the world after javascript. Lots of people love python 3.
I think its kind of disingenuous to make such statements. If you see how google is still unable to move from python2 to python3 we will know its a technical problem [1]. And remember chromium is a project with huge number of employee paid with very high amount of renumeration. So the barrier is mostly technical. I think such attitude comes from frustration which gets complemented with time.
I personally think python3, ruby etc. are way better than JS in many case but I think world is a bit unfair due to browser language monopoly here.
Yes. The 2->3 change was not engineered to make migration easy, and as a maintainer of a large and project that people have built on top of this is something that both current and potential users will judge you on.
Not speaking from a place of scorn, I and my team have made this same mistake, we lost both users and momentum. Big learning experience.
There may be some huge piece I'm missing, but how exactly is the "starting multiple async tasks wrapped in blocks and waiting for them to finish at the end of the main Async block" different from "starting multiple threads wrapped in blocks and manually collecting them at some point"? I thought Ruby did release the GIL when a thread is blocked (waiting for IO etc).
The difference is just that fiber overhead is lower so you can run more fibers than threads on a given system. Even though fibers have been around a while, people rarely used them because they are cooperative rather than preemptive, so you had to manually write the scheduling logic. Much easier to just use threads.
I think the big breakthrough for Ruby Async is that fiber scheduler in Ruby 3.0 now makes it possible for the runtime to manage fibers in a less manual way, so you now get the lightweight option more easily. The Async gem seems to be wrapping all that up in a very nice interface so you can write simple code and get good concurrency without much effort.
- Async Ruby is much more performant than threads. There are less context switches, enabled by the event reactor. The performance benefits are visible in simple scenarios like making a thousand HTTP requests.
- Async is more scalable, can handle millions concurrent tasks (like HTTP connections). There can only be a couple thousand threads at the same time.
- Threads are really hard to work with - race conditions everywhere. Async doesn't have this.
> - Threads are really hard to work with - race conditions everywhere. Async doesn't have this.
Having worked a lot with various flavours of async (I was one of the many people in the loop for the design of DOM Promise and JavaScript async), I regret that, while many developers believe it, *this is generally false*.
In languages with a GIL or run-to-completion semantics, of course, you get some degree of atomicity, which is a nice property. However, regardless of the language, once you have async, you have race conditions and reentrancy issues, often without the benefit of standard tools (e.g. Mutex, RwLock) to solve them [1].
Ruby's async syntax and semantics look neat, and I'm happy to see this feature, but as far as I can tell from the examples in the OP, they're going to have these exact same issues.
[1] Rust is kind of an exception, as its type system already forces you to either &mut/Mutex/RwLock/... anything that could suffer from data race conditions (or mark stuff as unsafe), even in async code. But even that is because of the race conditions and reentrancy issues mentioned above.
> There can only be a couple thousand threads at the same time.
You can easily have tens of thousands of threads on Linux. Beyond 50 000 or so you may need to adjust some settings using `sysctl`, but after that you should be able to push things much further.
Task counts themselves are also a pretty useless metric. Sure, you can have may fibers doing nothing. But once they start doing something, that may no longer be the case (this of course depends on the workload).
> Threads are really hard to work with - race conditions everywhere. Async doesn't have this.
You can still have race conditions in async code, as race conditions aren't limited to just parallel operations.
> race conditions everywhere. Async doesn't have this
What does Async (Fibers underneath) do differently than normal Threads? Using threads to handle concurrent work doesn't immediately bring race conditions, unless the programmer explicitly creates them (accessing the same stuff from different threads).
Fibers themselves AFAIK don't stop you from accessing the same stuff, aside from the obvious "the fiber code runs 'atomically' until it hits the next yield (which non-blocking Fibers take away anyway).
Please, someone, correct me if I've misunderstood.
The big difference appears to be that async Ruby does not merely give you an easy sugar to perform the sync-over-async antipattern you have described. The real innovation is that, as far as the user is concerned, Ruby is magically turning blocking methods into non-blocking ones.
That's basically how I'm thinking of things as well. To illustrate a bit further, consider the following:
Given a blocking method call `foo(x)`, I can make it non-blocking by wrapping it in a "thunk" as `λx.foo(x)`.
Where things start to get interesting is when I add another method call `foo(x) + bar(x)`. Now to keep things "async" I need to transform the abstraction into something more like `λx.foo(x) + λx.bar(x)`, and have the `+` call dispatch both fibers and wait for them before performing its operation.
Doing this automatically seems pretty cool, I'll have to think about this a bit more sometime.
The difference only shows itself in the real world, when you do a bit more per thread/coroutine and end up mutating shared state. This is where threads can lead to race conditions, whereas coroutines will not (unless you're basically ask for it)
Threads use quite much more memory than coroutines. Spawning eg. 3 threads for each request, x 1000 requests per second would probably eat a ton of memory.
Stackful coroutines (often used to implement colorless async) is the same except you can specify stack size. You can specify thread stack size yourselves as well, though no one does that.
OTOH, growable stack is useful, as Go demonstrated.
That seems mostly like the question of "Does this HTTP framework support HTTP pipelining". While I don't know the answer, it doesn't seem highly relevant. Most clients went away from using pipelining, since follow-up requests on the same connection are subject to unknown latency (stuck behind the first request) and a connection failure can impact all of those requests.
The better approach is to use either more connections, or proper request multiplexing via HTTP/2 or /3. In the latter case a server framework would just see multiple request invocations in parallel.
I just tried this out. Falcon with count=1 & sinatra. Worked perfectly. It seems every request is processed in an async block so literally no special code is required. A request waiting on network will allow others to go through.
It does, and that’s specifically why I’m really looking forward to benchmarks for rack frameworks switching to falcon - while it may not take Rails to Phoenix’s performance and latency, I bet it would close the gap considerably.
Just tried it out. Async blocks evaluate to Task objects which have a wait method and a result attribute which evaluates to the value of the block.
require 'async'
res = Async do |task|
name_task = task.async do
sleep 2
"Jenny"
end
task.async do
sleep 5
9
end
"Hello #{name_task.wait}"
end
puts res # => "Hello Jenny" after 5 seconds.
After using Python's and JS's async implementations this seems beautiful by comparison. Here's the a rough Python equivalent.
Which means that you can just write a whole normal Ruby program, that just uses Task::Async.current.async(...) wherever it likes to schedule subtasks (sort of like calling spawn/3 in Erlang), and then treat them as regular futures, even returning the future out of the current lexical scope without thunking it; and then have exactly one Async block at the toplevel that kicks off your main driver logic and then #wait s on it. All without having to pass the current task down the call stack everywhere.
(And if you want to schedule a bunch of stuff to happen in parallel and then wait for it all to be done, but you're below the toplevel Async block, you'd do that by scheduling the subtasks against an Async::Barrier: https://socketry.github.io/async/guides/getting-started/inde...)
It's trivially easy to get the results, here's a quick example:
require "async"
require "open-uri"
results = []
Async do |task|
task.async do
results << URI.open("https://httpbin.org/delay/1.6")
end
task.async do
results << URI.open("https://httpbin.org/delay/1.6")
end
end
I'm not sure if this is the most realistic example since you're implicitly relying on this being at the top level and there being a global await at the end of the block. Surely any real program will have all the work done inside a single top-level event loop.
require 'async'
Async do
results = []
Async do
sleep 1
results << "Hello"
end
puts results # => []
end
No function colors! Is this like Go, where each fiber maintains a separate stack at runtime, or is this like Rust, where each task is effectively transformed into a state machine?
I think this deserves more attention, especially it is coming from the original author of sequel.
>Polyphony is a library for writing highly concurrent Ruby apps. Polyphony harnesses Ruby fibers and a powerful io_uring-based I/O runtime to provide a solid foundation for building high-performance concurrent Ruby apps.
Readit News
Most Genesys API calls have reasonable response times but there are these occasional, weird, unpredictable 30-second response times. The slow responses can come in bursts, where you can a bunch of them at once. This makes it extremely hard to predictably scale the Ruby service that calls it, because in normal synchronous mode, every thread that calls Genesys is blocked and on longer available to take more client requests. You could scale up the service to assume these 30-second response times will happen all the time, but then you’ve got huge server costs just for the 1% case.
But using Async and Falcon (an Async-compatible HTTP server), Airbnb was able to safely scale down the service by around 85% and turn on autoscaling, and it’s working beautifully.
Disclaimer: I am not a full time Ruby person, so if it sounds naive please feel free to point it out.
You mentioned “The only useful case where I believe Async Ruby can help is the parallel requests cases.” That’s what was happening here: because of the tail latency, we sometimes needed to have a lot more parallel requests.
Does that clear it up?
That's pretty magical. Does this mean it would be possible to implement structured concurrency [0], but without the function coloring problem?
Regardless, I think I prefer how this looks from a code read/writeability perspective compared to Zig's or Swift's (and potentially future Rust's [2]) approaches.
[0] https://vorpus.org/blog/notes-on-structured-concurrency-or-g...
[1] https://kristoff.it/blog/zig-colorblind-async-await/
[2] https://blog.yoshuawuyts.com/async-overloading/
It's obvious from the examples provided in the post. For example, you can use the method `URI.open` both synchronously and asynchronously.
It's something I didn't want to mention in the article because it's a relatively advanced async concept, but yea - Async Ruby is already colorless and it's great.
> Async Ruby is colorless!
I find it deeply irritating that this is how we choose to describe things these days. Rather than talking about modality, or even using a few more words to describe the actual differences in terms of the underlying programing language models.
Telling me that Ruby's async is colorless implies that I know about the description of programming models as "colors" of code, which was just some random analogy someone came up with which caught on. Nothing about this has anything to do with color. Even syntax highlighting doesn't really fit. This is akin to particle physicists getting cheeky and calling one of the quantum states "color" when it has nothing to do with wavelength (correct me if I'm wrong).
This is an appeal to stop trying to be so cute. Thank you.
1. https://stackoverflow.com/a/63179518/4728007
No, I don't think there's anything fundamentally different.
Ruby 3.0 implements a "fiber scheduler" feature that enables "colorless Async". Fiber scheduler is an obscure Ruby feature, but the end result is brilliant. It was also a huge amount of work.
Side note: Fiber scheduler was implemented by the same guy who created Async Ruby - Samuel Williams. This guy is the mastermind (and master-coder) behind this project.
Ruby async seems to be implemented using stackful coroutines. IIRC Guido has been opposed to adding these to core Python, preferring stack-less coroutines because they require every yield-point to be explicit (i.e. marked with `await`).
There are libraries for a python that support stackful coroutines, such as gevent.
[0] https://cr.openjdk.java.net/~rpressler/loom/loom/sol1_part2....
https://gist.github.com/coezbek/07061fb7de2b036bf5c69f8a8b8c...
I might be using it wrong...
The company I was at the time tried really hard to move to python 3. It was a huge codebase with thousands of engineers. After more than a year of work they cancelled the project and said they'll never upgrade to python 3, and instead started writing smaller "microservices" around the core product just to calm engineers anxiety to use python3. Now Django is suffering kind of the same problem with all the async stuff being bolted on, adding a lot of gotchas and side effects and incompatibilities around every corner. It is a mess.
I left that company and I will never use python again.
I love how Ruby seems to care about the ecosystem, and both ruby and rails keep innovating and providing awesome full stack solutions.
When I read your post a strong anti python 3 feeling is conveyed.
It may well be that it’s impossible to upgrade such a company/code base when the engineers have such a negative attitude.
The barrier was perhaps not technical but about the attitude of the people.
There were lots of python 2 developers who were rabidly anti python 3. Some of the most well known python developers were aggressively very publicly anti python 3. Imagine a python 3 upgrade project with that attitude prevailing. No chance of success.
Today python 3 is more popular than ever before and is arguably the most popular language in the world after javascript. Lots of people love python 3.
I personally think python3, ruby etc. are way better than JS in many case but I think world is a bit unfair due to browser language monopoly here.
1. https://bugs.chromium.org/p/chromium/issues/detail?id=942720
Transitioning big, heavily used, daily changed by thousands of engineers codebases is not easy.
Do not make it look like just people is incompetent. It is a big financial, management and engineering effort.
Maybe you're not understanding the scale of these things.
Not speaking from a place of scorn, I and my team have made this same mistake, we lost both users and momentum. Big learning experience.
And this company has a HUGE codebase. It has its managements problems too, but the situation I'm talking about didn't make it any easier.
I think the big breakthrough for Ruby Async is that fiber scheduler in Ruby 3.0 now makes it possible for the runtime to manage fibers in a less manual way, so you now get the lightweight option more easily. The Async gem seems to be wrapping all that up in a very nice interface so you can write simple code and get good concurrency without much effort.
- Async is more scalable, can handle millions concurrent tasks (like HTTP connections). There can only be a couple thousand threads at the same time.
- Threads are really hard to work with - race conditions everywhere. Async doesn't have this.
Having worked a lot with various flavours of async (I was one of the many people in the loop for the design of DOM Promise and JavaScript async), I regret that, while many developers believe it, *this is generally false*.
In languages with a GIL or run-to-completion semantics, of course, you get some degree of atomicity, which is a nice property. However, regardless of the language, once you have async, you have race conditions and reentrancy issues, often without the benefit of standard tools (e.g. Mutex, RwLock) to solve them [1].
Ruby's async syntax and semantics look neat, and I'm happy to see this feature, but as far as I can tell from the examples in the OP, they're going to have these exact same issues.
[1] Rust is kind of an exception, as its type system already forces you to either &mut/Mutex/RwLock/... anything that could suffer from data race conditions (or mark stuff as unsafe), even in async code. But even that is because of the race conditions and reentrancy issues mentioned above.
You can easily have tens of thousands of threads on Linux. Beyond 50 000 or so you may need to adjust some settings using `sysctl`, but after that you should be able to push things much further.
Task counts themselves are also a pretty useless metric. Sure, you can have may fibers doing nothing. But once they start doing something, that may no longer be the case (this of course depends on the workload).
> Threads are really hard to work with - race conditions everywhere. Async doesn't have this.
You can still have race conditions in async code, as race conditions aren't limited to just parallel operations.
What does Async (Fibers underneath) do differently than normal Threads? Using threads to handle concurrent work doesn't immediately bring race conditions, unless the programmer explicitly creates them (accessing the same stuff from different threads).
Fibers themselves AFAIK don't stop you from accessing the same stuff, aside from the obvious "the fiber code runs 'atomically' until it hits the next yield (which non-blocking Fibers take away anyway).
The big difference appears to be that async Ruby does not merely give you an easy sugar to perform the sync-over-async antipattern you have described. The real innovation is that, as far as the user is concerned, Ruby is magically turning blocking methods into non-blocking ones.
Given a blocking method call `foo(x)`, I can make it non-blocking by wrapping it in a "thunk" as `λx.foo(x)`.
Where things start to get interesting is when I add another method call `foo(x) + bar(x)`. Now to keep things "async" I need to transform the abstraction into something more like `λx.foo(x) + λx.bar(x)`, and have the `+` call dispatch both fibers and wait for them before performing its operation.
Doing this automatically seems pretty cool, I'll have to think about this a bit more sometime.
From my limited understanding, the programmer has to be explicit about starting and resuming the work within a Fiber as opposed to the Ruby VM.
Async gem is starting fibers automatically. Fiber pausing and resuming is handled by event reactor (also called "event loop") from 'nio4r' gem.
io_uring support with asynchronous File IO is also on the way.
OTOH, growable stack is useful, as Go demonstrated.
For example:
* HTTP request 1 requires information for which we can't respond immediately.. so we hold the connection open and respond in a few seconds.
* Can HTTP request 2 come in at the same time?
EventMachine enabled the above with what they called "streaming responses"
The better approach is to use either more connections, or proper request multiplexing via HTTP/2 or /3. In the latter case a server framework would just see multiple request invocations in parallel.
(This kind of architecture might also be useful for http pipelining, but that's not the question in this scenario)
Also known as "head of line blocking" https://en.wikipedia.org/wiki/Head-of-line_blocking
In fact this is a perfect use case for Falcon + Async.
This is awesome! :)
Which means that you can just write a whole normal Ruby program, that just uses Task::Async.current.async(...) wherever it likes to schedule subtasks (sort of like calling spawn/3 in Erlang), and then treat them as regular futures, even returning the future out of the current lexical scope without thunking it; and then have exactly one Async block at the toplevel that kicks off your main driver logic and then #wait s on it. All without having to pass the current task down the call stack everywhere.
(And if you want to schedule a bunch of stuff to happen in parallel and then wait for it all to be done, but you're below the toplevel Async block, you'd do that by scheduling the subtasks against an Async::Barrier: https://socketry.github.io/async/guides/getting-started/inde...)
It's trivially easy to get the results, here's a quick example:
Deleted Comment
>Polyphony is a library for writing highly concurrent Ruby apps. Polyphony harnesses Ruby fibers and a powerful io_uring-based I/O runtime to provide a solid foundation for building high-performance concurrent Ruby apps.
https://noteflakes.com
Source link: https://digital-fabric.github.io/polyphony/faq/#why-does-pol...
> Polyphony “patches” some Ruby core and stdlib APIs, providing behavioraly compatible fiber-aware implementations