I wrote Sidekiq, which Oban is based on. Congratulations to Shannon and Parker on shipping this!
I had to make this same decision years ago: do I focus on Ruby or do I bring Sidekiq to other languages? What I realized is that I couldn't be an expert in every language, Sidekiq.js, Sidekiq.py, etc. I decided to go a different direction and built Faktory[0] instead, which flips the architecture and provides a central server which knows how to implement the queue lifecycle internally. The language-specific clients become much simpler and can be maintained by the open source community for each language, e.g. faktory-rs[1]. The drawback is that Faktory is not focused on any one community and it's hard for me to provide idiomatic examples in a given language.
It's a different direction but by focusing on a single community, you may have better outcomes, time will tell!
Thanks Mike! You are an inspiration. Parker and I have different strengths both in life and language. We're committed to what this interop brings to both Python and Elixir.
Faktory was a big influence/inpiration for Ocypod[0], a job queuing system I wrote a few years back (similarly language agnostic). Much appreciated for making it all open source.
By “based on” I don’t mean a shared codebase or features but rather Parker and I exchanged emails a decade ago to discuss business models and open source funding. He initially copied my Sidekiq OSS + Sidekiq Pro business model, with my blessing.
Maybe you didn’t intend it this way, but your comment comes across as an attempt to co-opt the discussion to pitch your own thing. This is generally looked down upon here.
It was an off-the-cuff comment and probably not worded ideally but the intent was to discuss how Oban is branching off into a new direction for their business based on language-specific products while I went a different direction with Faktory. Since I came to the exact same fork in the road in 2017, I thought it was relevant and an interesting topic on evolving software products.
Knowing Mike and his work over the years, that is not the case. He is a man of integrity who owns a cornerstone product in the Ruby world. He is specifically the type of person I want to hear from when folks release new software having to do with background jobs, since he has 15 years of experience building this exact thing.
> Oban allows you to insert and process jobs using only your database. You can insert the job to send a confirmation email in the same database transaction where you create the user. If one thing fails, everything is rolled back.
This is such a key feature. Lots of people will tell you that you shouldn't use a relational database as a worker queue, but they inevitably miss out on how important transactions are for this - it's really useful to be able to say "queue this work if the transaction commits, don't queue it if it fails".
Brandur Leach wrote a fantastic piece on this a few years ago: https://brandur.org/job-drain - describing how, even if you have a separate queue system, you should still feed it by logging queue tasks to a temporary database table that can be updated as part of those transactions.
This resonates so much. I spent years in an org watching "domain events" vanish into the ether because of the Dual Write Problem. We had these high-performance, sharded, distributed monsters that were "fast" on paper, but they couldn't guarantee a simple message would actually send after a record was saved.
Moving back to a rock-solid SQL-backed approach solved it overnight. But since there are no more "1% glitches," people have forgotten there was ever a fire. It’s a thankless win. The organization now thinks the system is "easy" and the "async purists" still lobby for a separate broker just to avoid "polluting" the DB. They’d rather trust complex, custom-built async logic than the most reliable part of their stack. (The transactional outbox pattern is essential, I just prefer mine backed by the same ACID guarantees as my data).
It’s tricky stuff. I'm an application dev, not a DB internalist, but I've realized that a week spent actually learning isolation levels and commit-ordering saves you a year of "distributed system" debugging. Even when teams layer an ORM like Entity Framework on top to "hide" the complexity, that SQL reality is still there. It’s not magic; it’s just ACID, and it’s been there the whole time.
We're building an AI powered app builder. We use elixir, phoenix and of course OBAN.
It feels like such a super power. What you're describing is particularly important in the era of long running AI processes. Something as simple as running a deploy creates pressure on your agent orchestration. But if everything is powered by OBAN you have wonderful ways to build durability.
By the way, it's all "free" and built-in.
In other language ecosystems, people literally pay for durability that has this shape (like temporal)
The Oban folks have done amazing, well-engineered work for years now - it's really the only option for Elixir. That said, I'm very confused at locking the process pool behind a pro subscription - this is basic functionality given CPython's architecture, not a nice-to-have.
For $135/month on Oban Pro, they advertise:
All Open Source Features
Multi-Process Execution
Workflows
Global and Rate Limiting
Unique Jobs
Bulk Operations
Encrypted Source (30/90-day refresh)
1 Application
Dedicated Support
I'm going to toot my own horn here, because it's what I know, but take my 100% free Chancy for example - https://github.com/tktech/chancy. Out of the box the same workers can mix-and-match asyncio, processes, threads, and sub-interpreters. It supports workflows, rate limiting, unique jobs, bulk operations, transactional enqueuing, etc. Why not move these things to the OSS version to be competitive with existing options, and focus on dedicated support and more traditional "enterprise" features, which absolutely are worth $135/month (the Oban devs provide world-class support for issues). There are many more options available in the Python ecosystem than Elixir, so you're competing against Temporal, Trigger, Prefect, Dagster, Airflow, etc etc.
> It supports workflows, rate limiting, unique jobs, bulk operations, transactional enqueuing, etc. Why not move these things to the OSS version to be competitive with existing options, and focus on dedicated support and more traditional "enterprise" features, which absolutely are worth $135/month (the Oban devs provide world-class support for issues).
We may well move some of those things to the OSS version, depending on interest, usage, etc. It's much easier to make things free than the other way around. Some Pro only features in Elixir have moved to OSS previously, and as a result of this project some additional functionality will also be moved.
Support only options aren't going to cut it in our experience; but maybe that'll be different with Python.
> There are many more options available in the Python ecosystem than Elixir, so you're competing against Temporal, Trigger, Prefect, Dagster, Airflow, etc etc.
There's a lot more of everything available in the Python ecosystem =)
> Support only options aren't going to cut it in our experience; but maybe that'll be different with Python.
That's totally fair, and I can only speak from the sidelines. I haven't had a chance to review the architecture - would it possibly make sense to swap from async as a free feature to the process pool, and make async a pro feature? This would help with adoption from other OSS projects, if that's a goal, as the transition from Celery would then be moving from a process pool to a process pool (for most users). The vast, vast majority of Python libraries are not async-friendly and most still rely on the GIL. On the other hand, Celery has absolutely no asyncio support at all, which sets the pro feature apart.
On the other hand, already released and as you said it's much harder to take a free feature and make it paid.
Thanks again for Oban - I used it for a project in Elixir and it was painless. Missing Oban was why I made Chancy in the first place.
Thanks. My two cents would be to not lock technical features behind a paywall. Lock "enterprise" features like encryption, FIPS, compliance reports, etc which make sense for a big corp. This would be far more palatable for someone small like my one-two person teams to adopt it and pay if we ever become big enough to care about enterprisey features.
I'm sure you are aware but sharing anyway. Django 6.0 shipped an API called Django Tasks for background jobs so all Django code can implement portable, backend agnostic background jobs and swap backends on the fly but there are zero actual backends out there right now one can use in production. If you could add inbuilt Django Tasks support to Chancy or create a `django-chancy` package that bridged it, I think you'd see a lot of early adoption by Django projects.
OSS Oban has a few limitations, which are automatically lifted in the Pro version:
Single-threaded asyncio execution - concurrent but not truly parallel, so CPU-bound jobs block the event loop.
This makes it not even worth trying. Celery's interface kind of sucks, but I'm used to it already, and I can get infinitely parallel expanding vertically and horizontally for as long as I can afford the resources.
I also don't particularly like ayncio, and if I'm using a job queue wouldn't expect to need it.
Edit: I looked into it a bit more, and it seems we can launch multiple worker nodes, which doesn't seem as bad as what I originally thought
Ooof. I don't mind the OSS/pro feature gate for the most part, but I really don't love that "Pro version uses smarter heartbeats to track producer liveness."
There's a difference between QoL features and reliability functions; to me, at least, that means that I can't justify trying to adopt it in my OSS projects. It's too bad, too, because this looks otherwise fantastic.
This is what get's me about Oban too. It's very cool but it feels bad to have 'same thing but better' gated behind payment.
I want the base thing to be the best version possible and extra features be paid (so that when I need them I might start paying). Feels like sometimes boundaries between Pro and Free are made in weird place almost as if someone went out of their way to make things behave differently (maybe it's just my inexperience with Oban though).
With a typical Redis or RabbitMQ backed durable queue you’re not guaranteed to get the job back at all after an unexpected shutdown. That quote is also a little incorrect—producer liveness is tracked the same way, it’s purely how “orphaned” jobs are rescued that is different.
"jobs that are long-running might get rescued even if the producer is still alive" indicates otherwise. It suggests that jobs that are in progress may be double-scheduled. That's a feature that I think shouldn't be gated behind a monthly pro subscription; my unpaid OSS projects don't justify it.
While this is a Cool Thing To See, I do wish things would go the other way—and have all the BI/ML/DS pipelines and workflows folks are building in Python and have them come to Elixir (and, as would follow, Elixir). I get where the momentum is, but having something functional, fault-tolerant, and concurrent underpinning work that’s naturally highly concurrent and error-prone feels like a _much_ more natural fit.
Very, very different tools, though they cover similar areas.
Temporal - if you have strict workflow requirements, want _guarantees_ that things complete, and are willing to take on extra complexity to achieve that. If you're a bank or something, probably a great choice.
Oban - DB-backed worker queue, which processes tasks off-thread. It does not give you the guarantees that Temporal can because it has not abstracted every push/pull into a first-class citizen. While it offers some similar features with workflows, to multiple 9's of reliability you will be hardening that yourself (based on my experience with Celery+Sidekiq)
Based on my heavy experience with both, I'd be happy to have both available to me in a system I'm working on. At my current job we are forced to use Temporal for all background processing, which for small tasks is just a lot of boilerplate.
I’ll say that, I think this varies by language/SDK - at least with the Temporal TypeScript SDK, a simple single idempotent step background task is however many lines of code to do the actual work in an activity, and then the only boilerplate is like 3 lines of code for a simple workflow function to call the activity.
We migrated from Celery to Prefect a couple of years back and have been very happy. But ours is a small op which handles tasks in 1000s and not millions. It’s been night and day in terms of visibility and tracking. I would definitely recommend it.
It’s a heavy weight that covers a lot of use cases. But we just run simple ProcessWorkers for our regular needs and ECS worker for heavier ML tasks.
I'm just coming back to web/API development Python after 7-8 years working on distributed systems in Go. I just built a Django+Celery MVP given what I knew from 2017 but I see a lot of "hate" towards Celery online these days. What issues have you run into with Celery? Has it gotten less reliable? harder to work with?
Celery + RabbitMQ is hard to beat in the Python ecosystem for scaling. But the vast, vast majority of projects don't need anywhere that kind of scale and instead just want basic features out of the box - unique tasks, rate limiting, asyncio, future scheduling that doesn't cause massive problems (they're scheduled in-memory on workers), etc. These things are incredibly annoying to implement over top of Celery.
Most Django projects just need a basic way to execute timed and background tasks. Celery requires seperate containers or nodes, which complicates things unnecessarily. Django 6.0 luckily has tasks framework -- which is backported to earlier django versions has well, which can use the database. https://docs.djangoproject.com/en/6.0/topics/tasks/
I like celery but I started to try other things when I had projects doing work from languages in addition to python. Also I prefer the code work without having to think about queues as much as possible. In my case that was Argo workflows (not to be confused with Argo CD)
Oban is cool but I really like the idea of pgflow.dev, which is based on pgmq (rust) Postgres plugin doing the heavy lifting as it makes it language agnostic (all the important parts live in Postgres). I've started an Elixir adapter which really is just a DSL and poller, could do the same in Python, etc.
Readit News
I had to make this same decision years ago: do I focus on Ruby or do I bring Sidekiq to other languages? What I realized is that I couldn't be an expert in every language, Sidekiq.js, Sidekiq.py, etc. I decided to go a different direction and built Faktory[0] instead, which flips the architecture and provides a central server which knows how to implement the queue lifecycle internally. The language-specific clients become much simpler and can be maintained by the open source community for each language, e.g. faktory-rs[1]. The drawback is that Faktory is not focused on any one community and it's hard for me to provide idiomatic examples in a given language.
It's a different direction but by focusing on a single community, you may have better outcomes, time will tell!
[0]: https://github.com/contribsys/faktory [1]: https://github.com/jonhoo/faktory-rs
https://github.com/sidekiq/sidekiq/blob/ba8b8fc8d81ac8f57a55...
Additionally, delayed_job came before resque.
[0]: https://github.com/davechallis/ocypod
Sidekiq is pretty bare bones compared to what Oban supports with workflows, crons, partitioning, dependent jobs, failure handling, and so forth.
This is such a key feature. Lots of people will tell you that you shouldn't use a relational database as a worker queue, but they inevitably miss out on how important transactions are for this - it's really useful to be able to say "queue this work if the transaction commits, don't queue it if it fails".
Brandur Leach wrote a fantastic piece on this a few years ago: https://brandur.org/job-drain - describing how, even if you have a separate queue system, you should still feed it by logging queue tasks to a temporary database table that can be updated as part of those transactions.
Moving back to a rock-solid SQL-backed approach solved it overnight. But since there are no more "1% glitches," people have forgotten there was ever a fire. It’s a thankless win. The organization now thinks the system is "easy" and the "async purists" still lobby for a separate broker just to avoid "polluting" the DB. They’d rather trust complex, custom-built async logic than the most reliable part of their stack. (The transactional outbox pattern is essential, I just prefer mine backed by the same ACID guarantees as my data).
It’s tricky stuff. I'm an application dev, not a DB internalist, but I've realized that a week spent actually learning isolation levels and commit-ordering saves you a year of "distributed system" debugging. Even when teams layer an ORM like Entity Framework on top to "hide" the complexity, that SQL reality is still there. It’s not magic; it’s just ACID, and it’s been there the whole time.
This looks like a good definition too: https://www.milanjovanovic.tech/blog/outbox-pattern-for-reli...
It feels like such a super power. What you're describing is particularly important in the era of long running AI processes. Something as simple as running a deploy creates pressure on your agent orchestration. But if everything is powered by OBAN you have wonderful ways to build durability.
By the way, it's all "free" and built-in.
In other language ecosystems, people literally pay for durability that has this shape (like temporal)
For $135/month on Oban Pro, they advertise:
I'm going to toot my own horn here, because it's what I know, but take my 100% free Chancy for example - https://github.com/tktech/chancy. Out of the box the same workers can mix-and-match asyncio, processes, threads, and sub-interpreters. It supports workflows, rate limiting, unique jobs, bulk operations, transactional enqueuing, etc. Why not move these things to the OSS version to be competitive with existing options, and focus on dedicated support and more traditional "enterprise" features, which absolutely are worth $135/month (the Oban devs provide world-class support for issues). There are many more options available in the Python ecosystem than Elixir, so you're competing against Temporal, Trigger, Prefect, Dagster, Airflow, etc etc.We may well move some of those things to the OSS version, depending on interest, usage, etc. It's much easier to make things free than the other way around. Some Pro only features in Elixir have moved to OSS previously, and as a result of this project some additional functionality will also be moved.
Support only options aren't going to cut it in our experience; but maybe that'll be different with Python.
> There are many more options available in the Python ecosystem than Elixir, so you're competing against Temporal, Trigger, Prefect, Dagster, Airflow, etc etc.
There's a lot more of everything available in the Python ecosystem =)
That's totally fair, and I can only speak from the sidelines. I haven't had a chance to review the architecture - would it possibly make sense to swap from async as a free feature to the process pool, and make async a pro feature? This would help with adoption from other OSS projects, if that's a goal, as the transition from Celery would then be moving from a process pool to a process pool (for most users). The vast, vast majority of Python libraries are not async-friendly and most still rely on the GIL. On the other hand, Celery has absolutely no asyncio support at all, which sets the pro feature apart.
On the other hand, already released and as you said it's much harder to take a free feature and make it paid.
Thanks again for Oban - I used it for a project in Elixir and it was painless. Missing Oban was why I made Chancy in the first place.
Single-threaded asyncio execution - concurrent but not truly parallel, so CPU-bound jobs block the event loop.
This makes it not even worth trying. Celery's interface kind of sucks, but I'm used to it already, and I can get infinitely parallel expanding vertically and horizontally for as long as I can afford the resources.
I also don't particularly like ayncio, and if I'm using a job queue wouldn't expect to need it.
Edit: I looked into it a bit more, and it seems we can launch multiple worker nodes, which doesn't seem as bad as what I originally thought
There's a difference between QoL features and reliability functions; to me, at least, that means that I can't justify trying to adopt it in my OSS projects. It's too bad, too, because this looks otherwise fantastic.
https://youtu.be/iV1EcfZSdCM?si=KAJW26GVaBqZjR3M
This helps with keeping it on track writing idiomatic elixir and using good patterns: https://skills.sh/agoodway/.claude/elixir-genius
How much context is eaten up by skills that rehash what a SOTA model should already know?
Maybe token-wise, it's a wash: Elixir/OTP does a lot without third-party libs, which would require massive npm dependencies to achieve the same thing.
I've never heard of Oban until now and the one we've considered was Temporal but that feels so much more than what we need. I like how light Oban is.
Does anyone have experience with both and is able to give a quick comparison?
Thanks!
Temporal - if you have strict workflow requirements, want _guarantees_ that things complete, and are willing to take on extra complexity to achieve that. If you're a bank or something, probably a great choice.
Oban - DB-backed worker queue, which processes tasks off-thread. It does not give you the guarantees that Temporal can because it has not abstracted every push/pull into a first-class citizen. While it offers some similar features with workflows, to multiple 9's of reliability you will be hardening that yourself (based on my experience with Celery+Sidekiq)
Based on my heavy experience with both, I'd be happy to have both available to me in a system I'm working on. At my current job we are forced to use Temporal for all background processing, which for small tasks is just a lot of boilerplate.
It’s a heavy weight that covers a lot of use cases. But we just run simple ProcessWorkers for our regular needs and ECS worker for heavier ML tasks.
https://github.com/agoodway/pgflow