Show HN: Sidequest.js – Background jobs for Node.js using your database

I'm not sure I follow with the LGPL requirement.. how do you envision downstream users complying with the relinking requirement, particularly in cases where Sidequest is orchestrating jobs across tightly-coupled backend infrastructure?

shrikrishna · a month ago

AFAIK, the relinking requirement only applies if you distribute the software to someone.

If you’re running Sidequest entirely on your own infrastructure to orchestrate jobs across your backend, you’re not distributing the software at all, you’re providing a service. The tight coupling does not itself trigger extra obligations. What matters legally is distribution, not architecture.

Edgecase is if you give your software to a customer to run on their own servers (self‑hosted deployment/docker image shipped to customer). In those cases, you would need to allow them to replace Sidequest.js (ie, not obfuscating it away).

Someone more knowledgeable can correct me, if I'm wrong

yonixw · a month ago

Layman here also, but I think you are correct about GPL and LGPL (this case), but not for AGPL which adds a requirement that: "... If your software can interact with users remotely through a computer network, you should also make sure that it provides a way for users to get its source"

https://www.gnu.org/licenses/gpl-faq.html#AGPLv3ServerAsUser

yonixw · a month ago

My guess is the 2-level separation. You WILL need to make some part in your system LGPL by using "Sidequest.js" tightly, but then expose very simple APIs (like /start /status etc..) that will stop the LGPL from "infecting" other parts, as they can replace the "linking" with anything that will support those simple APIs.

Which is a good way to get improvements other are doing that relate to your source code through LGPL's source exposing, while not forcing it everywhere (GPL case). Especially, for backend libs.

And since AGPL will essentially make it non viable for SaaS (as network separation won't do), LGPL is what left.

Really like your approach of using existing Postgres/MySQL instead of dragging in Redis. It feels genuinely drop-in, but still Sidekiq-class. I know it's a bit early to ask about production patterns, but I was curious: if the worker thread flood hits the same Postgres that serves the web API, how do the job-fetch queries avoid contending with OLTP traffic? Does Sidequest follow Oban's advisory-lock approach or use a different throttling strategy?

giovaniguizzo · a month ago

Hello! One of the creators of Sidequest here.

Great question!

Sidequest uses transaction-level row locks (`SELECT ... FOR UPDATE SKIP LOCKED`) when fetching jobs. This means each worker thread only locks the specific job rows it’s about to process, minimizing lock contention and avoiding blocking other queries. This pattern is inspired by Oban’s advisory-lock approach, but instead of using explicit advisory locks, Sidequest relies on the database’s built-in row locking mechanism.

The only drawback is that Sidequest will require one or two connections to your DB. If you enqueue jobs from within other jobs, then each job that requests an enqueue will also connect to your DB (lazily connected upon request - if your job does not enqueue, no connection is created). However, you can configure the number of concurrent workers per queue and globally, so you control how much load Sidequest puts on your database.

I hope that answers your question :)

sorentwo · a month ago

Oban doesn't use advisory locks for fetching jobs (unless there is uniqueness involved)—it uses `FOR UPDATE SKIP LOCKED` as well to pull jobs.

MutedEstate45 · a month ago

Thanks for the clarification. That's a clean approach. I just stared your repo. Looking forward to seeing where sidequest.js goes :)

kristianc · a month ago

spiffytech · a month ago

This is great! I sometimes have projects that could use SQLite, but they're just big enough to need a persistent job queue. Great to have an option for that.

I also like that there's a dashboard. That's really important when a project gets serious, and a surprising number of job queue libraries don't have admin tools.

sleiben · a month ago

Looks really interesting, especially with the dashboards.

Just to validate an idea here: I’m using k8s (20+ services) and trying to stick to the 12factor design pattern by having all the baking/companion services, also like cron jobs deployed from within the service directory. Right now I’m using k8s cron services for cron jobs and log the steps and observe with DataDog. Using k8s cron services feels right somehow but the observability with DataDog not. So, would it make sense for me, to deploy a baking k8s web service running sidequest instead?

Well, Sidequest comes with a comprehensive built-in dashboard that provides real-time monitoring, job management, and performance analytics, so there's that.

You can probably deploy Sidequest a job processor for all your 20+ services. Each service can enqueue the jobs and you can run as many machines as you want with Sidequest just to run the jobs. You maintain the 12-factor principle - your services remain stateless and delegate scheduled work to Sidequest.

The only requirement here is that the scripts to be executed must be in the same path in all machines. As long as you follow that, you can deploy it and run your jobs :)

seymon · a month ago

How does Sidequest compare to Graphile Worker https://worker.graphile.org/ ?

Interesting lib!

Graphile Worker: PostgreSQL only

Sidequest: Multiple backends (PostgreSQL, MySQL, SQLite, MongoDB)

Graphile Worker: No built-in dashboard - you need external monitoring

Sidequest: Comprehensive built-in web dashboard for job monitoring

Graphile Worker: Single queue with job prioritization

Sidequest: Multiple queues with individual: i) Concurrency limits; ii) Priority levels; iii) State management (active/paused); iv) Isolated workloads.

Graphile Worker: Direct PostgreSQL integration, very lightweight

Sidequest: Worker threads for non-blocking processing, more comprehensive job lifecycle management

Graphile Worker: Optimized for PostgreSQL performance (3ms latency)

Sidequest: Balanced performance with rich feature set

I hope that helps answering your question.

anonzzzies · a month ago

Ah!Nice, was looking for that as all are either redis or postgres (and we don't use postgres and prefer not to use redis). So thanks for that.

drewrbaker · a month ago

This looks like exactly what I need, but our DB is Postgres on Supabase and I don’t think their version supports transaction locking…

Well, you don't need to worry. Sidequest only locks the rows it's gonna use, i.e., the job rows it manages. Any other data in your DB is not touched by Sidequest, so you should be safe :)

drewbitt · a month ago

https://github.com/timgit/pg-boss https://github.com/hatchet-dev/hatchet are both on top of Postgres, but I like the SQLite here for a project I have in mind.

Nice to know SQLite will be useful. Just be mindful that SQLite does not work well in highly concurrent systems. So, if you have a single job runner process, it will work wonderfully. However, if you have many workers, then you might see a few issues.