rcombatwombat (u/rcombatwombat)

rcombatwombat commented on Why was Apache Kafka created? bigdata.2minutestreaming.... · Posted by u/enether

selkin · 8 days ago

KIP-932[0] adds queue semantics for Kafka. You may still want to use another queue though: as always, no one size fits all.

[0] https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A...

rcombatwombat · 2 days ago

Still not what I would call 'proper' queuing (IMHO!) because ultimately it's still just Kafka and just append only: you can't 'atomically acknowledge+delete' individual messages anywhere in the stream, you only dealing with offsets and that's a fundamental thing that has many consequences (e.g. you don't get ordering between batches) and also means that you will get re-deliveries in some scenarios that you would not have with a system that implements proper queuing.

rcombatwombat commented on Why was Apache Kafka created? bigdata.2minutestreaming.... · Posted by u/enether

mrweasel · 8 days ago

I previously helped clients setup and run Kafka clusters. Why they'd need Kafka was always our first question, never got a good answer from a single one of them. That's not to say that Kafka isn't useful, it is, in the right setting, but that settings is never "I need a queue". If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. It's not that Kafka can't do it, but you are making things harder than they needed to be.

rcombatwombat · 2 days ago

Let's not forget NATS.io which does queuing (and a whole lot of other things from at least once low latency pub/sub to request/reply, streaming and all the way to KV and Object store) very well and is very light weight to run and much simpler to run than Kafka.

rcombatwombat commented on What If We Could Rebuild Kafka from Scratch? morling.dev/blog/what-if-... · Posted by u/mpweiher

nchmy · 4 months ago

NATS is also in the process of a open source license rugpull...

https://news.ycombinator.com/item?id=43783452

rcombatwombat · 4 months ago

No rug pull: https://www.cncf.io/announcements/2025/05/01/cncf-and-synadi...

rcombatwombat commented on Protecting NATS and the integrity of open source cncf.io/blog/2025/04/24/p... · Posted by u/tomncooper

d4mr · 4 months ago

I just finished rearchitecting a core service at work to use NATS for distributed state. Really unsure about the viability of NATS as an open source product if Synadia stops contributing. It's making me reconsider our architecture change, maybe we should stick with a more stable (albeit less aligned) alternative.

summarising some context for folks;

From Synadia's Cease and Demand Letter:

> As should be clear, the NATS.io project has failed to thrive as a CNCF project, with essentially all growth of the project to date arising from Synadia’s efforts and at Synadia’s expense. It is for this reason that Synadia requests to end its relationship with the Foundation and receive full control of the nats.io domain name and Github repository within two weeks.

From Synadia's exit proposal:

> We propose that NATS.io exit from the CNCF Foundation effective immediately... Upon leaving CNCF, Synadia will adopt the Business Source License (BSL) for the NATS.io server... specific use cases (such as offering NATS as a managed service or integrating it into specific commercial offerings) will require a commercial license.

From CNCF's response:

> Let's be clear: this is not a typical license change or fork. It's an attempt to "take back" a mature, community-driven open source project and convert it into a proprietary product—after years of growth and collaboration under open governance and CNCF's stewardship.

Primary sources:

- Synadia's Cease and Demand Letter: https://github.com/cncf/foundation/blob/main/documents/nats/...

- Synadia's Exit Proposal: https://github.com/cncf/foundation/blob/main/documents/nats/...

- CNCF's Response: https://www.cncf.io/blog/2025/04/24/protecting-nats-and-the-...

rcombatwombat · 4 months ago

Two points:

From the exit proposal: "Over 97% of contributions to the NATS.io server were made by employees of Synadia and its predecessor company".

Also, when they applied for graduation in 2018, they were told no because most of the contributors work for Synadia (https://github.com/cncf/toc/pull/168). As of now 7 years later, it's still not graduated by CNCF. At this point it likely never will.

Putting yourself in their shoes, are your surprised they want to take it back from CNCF?

rcombatwombat commented on Kafka at the low end: how bad can it get? broot.ca/kafka-at-the-low... · Posted by u/alexwebr

relistan · 6 months ago

Agree. RabbitMQ is a Swiss Army knife that has a lot of available patterns, scales fairly well, and is very robust. If you don’t know what to choose, start with Rabbit. It will let you figure out which patterns you need and you probably won’t scale out of it. Pay someone to host it.

On the other hand, if you know what you need to do and it’s supported by it, NATS is IME the way to go (particularly JetStream).

rcombatwombat · 6 months ago

IMHO NATS is an even better Swiss army knife than Rabbit

rcombatwombat commented on Kafka at the low end: how bad can it get? broot.ca/kafka-at-the-low... · Posted by u/alexwebr

MuffinFlavored · 6 months ago

NATS/WebSockets are good for 1 publisher -> many consumer (pubsub)

RabbitMQ is good for 1 producer -> 1 consumer with ack/nack

Right?

rcombatwombat · 6 months ago

NATS does much more than pub/sub (with any fan out), it also does queueing in Core NATS (no persistence) and streaming including queuing over streams (with ack/nack/term). Don't be fooled into thinking NATS doesn't do queueing because it doesn't have a "Q" in the name :).

rcombatwombat commented on How to use Postgres for everything github.com/Olshansk/postg... · Posted by u/Olshansky

pdimitar · 9 months ago

I mean the following (let me simplify it). You pull stuff from Kafka in batches of 4 and immediately send each message to a parallel worker, first parsing them in-thread and sequentially and these are the results of the parsing:

1. ok

2. error

3. ok

4. ok

I cannot not-ACK message#2 because that means message#1 is not ACK-ed as well.

Does NATS solve this? F.ex. can I get a reference to each message in my parallel workers for them to also say "I am not ACK-ing this because I failed processing it, let the next batch include it again"?

rcombatwombat · 9 months ago

Yes, one of the many differences (advantages) of NATS JetStream over Kafka: with NATS you can ack explicitly each message individually, and even better if you set your stream to be in 'work-queue' mode it will also automatically (and atomically) delete the ack'd message from the stream (i.e. like a 'proper' queue) another difference with Kafka where you can't delete individual messages in the middle of a stream (only trim the tail end).

You can also 'negative ack' messages, specify a back-off period before the message is re-delivered (because NATS automatically re-delivers un-acked (or nacked) messages) when you can't temporarily process it, or 'term' a message (don't try to re-deliver it, e.g. because the payload is bad), or even 'ask for more time before needing to ack the message (if you are temporarily too slow at processing the message).

rcombatwombat commented on RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1) eranstiller.com/rabbitmq-... · Posted by u/gslin

inkyoto · 2 years ago

Event-driven architecture is an architectural principle, and Kafka, RabbitMQ/ActiveMQ, Pulsar, NATS and so forth are implementations that support the event-driven architectural principle. Yet, all of them range in a variety, complexity and extent of features they provide which may or may not be a good fit for a particular use case.

Traditional message brokers (RabbitMQ and similar) do support the event-driven architecture, yet the data they handle is ephemeral. Once a message has been processed, it is gone forever. Connecting a new raw data source is not an established practice and requires a technical «adapter» of sorts to be built. High concurrency levels is problematic for scenarios where the strict message processing ordering is required: the traditional message brokers do not handle it well in highly parallel scenarios out of the box.

Kafka and similar also support event-driven architectures, yet they allow the data to be processed multiple times – by existing (i.e. a data replay) and, most importantly, new or unknown at the time consumers (note: this is distinct from the data replay!). This is allows to plug existing data source(s) into a data streaming platform (Kafka) and incrementally add new data consumers and processors over the time with the datasets being available intact. This is an important distinction. Kafka and similar also improve on the strict processing order guarantee by allowing a message source (a Kafka topic) to be explicitly partitioned out and guaranteeing that the message order will be retained and enforced for a consumer group receiving messages from that partition.

To recap, traditional message brokers are a good fit for handling the ephemeral data, and data streaming platforms are a good fit for connecting data sources and allowing the data to be ingested multiple times. Both implement and support event-driven architectures in a variety of scenarios.

rcombatwombat · 2 years ago

NATS with JetStream provides _both_ queuing like a traditional message broker and multiple data replay from offset (plus KV, and request/reply)

rcombatwombat commented on RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1) eranstiller.com/rabbitmq-... · Posted by u/gslin

kumarvvr · 2 years ago

Could you expand on this a bit more? I am curious.

rcombatwombat · 2 years ago

Also see https://www.youtube.com/watch?v=C4BnJ5QLeTY

rcombatwombat commented on RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1) eranstiller.com/rabbitmq-... · Posted by u/gslin

zerbinxx · 2 years ago

Jetstream does indeed have message persistence: I can issue queries like “get messages on topic since 5 minutes ago” - I do this a ton. However, that seems to be the extent of the storage/query API that it exposes for historical messages. I’m quite a big fan, and would recommend it with the caveat that Jetstream is considerably more complex than simple nats and I get the feeling I’m barely scratching the surface with it.

rcombatwombat · 2 years ago

NATS JetStream also implements subject-based addressing at the stream level (unlike Kafka where 1 stream = 1 topic, and you can only use the message's key for distribution, not for addressing).

So you can for example ask for [the first/the last/all] of the messages on a particular subject, or on a hierarchy of subjects by using wildcards. All the filtering is done at the server level.