Back in 2014, I was rebuilding the core of an event processing engine. At the time, the decision was between Apache Kafka and rolling our own. After investigating Zookeeper, we decided to roll our own and chose ZeroMQ as the messaging layer as our on-prem customers probably didn't want to own and manage Zookeeper.
ZeroMQ was absolutely solid and stable; incredibly trouble free and the only issues we ran into were when IT teams didn't open the ports we documented in the configuration procedure. (The resultant architecture actually looks a lot like Flink)
In any case, ZeroMQ is a fantastic piece of technology that I feel like I don't see out in the wild quite enough. It's dead simple and incredibly stable from my experience.
The problem is that it started out as any but stable and reliable. It asserted on received data, which in a network application is a super newbie mistake. When I looked at it, the pub/sub socket would hang forever if the other end interrupted the connection. So the zeromq guide which said "look how easy it is" was only true if you ignored errors. If you are writing network code and ignore errors, well, good luck. That was a long time ago (~10yrs) so if it is better now, good for them. Also, both founders have left the project. One passed from cancer, the other didn't like what he built and started over in C. Not that they can't be replaced, but transitions can be hard and take time.
In a company I work for they decided to do the same, around the same time. I believe it was a wrong call. Over time requirements have grown and we ended up bolting all kinds of kafka features on top of the zeromq thing, but of course much crappier. And in the meantime kafka doesn't require zookeeper anymore and is the de-facto standard
Of course, ZeroMQ and Kafka are two very different tools that serve different purposes and one needs to understand the tradeoffs.
For us, delivering an on-prem commercial off the shelf solution, it was untenable to expect the customer IT team to operate a separate, relatively huge piece of tech (remember, this is 2014). Maybe the heuristics would be different today with K8s and advancement of Kafka. But ZeroMQ as an in-process, distributed messaging layer is dead simple. If your use case requires anything else on top of that, it's on the team to design the right solution like resiliency, statefulness, etc.
For a high throughput, distributed compute focused use case, I think ZMQ is still a great choice for coordinating execution. But Kafka and other options on the market now are great choices for higher order abstractions.
A company I worked for had the same problem. Messages were being dropped, and either no one on backend knew how or wanted to investigate. I was on the data team and we just had to deal with it.
I remember we had a similar use-case. This was for collecting live statistics from a sizable Varnish Cache cluster. We wrote a in memory database to store the data. It's been chugging away for about 10 years now and last I heard the zmq traffic alone was about 3Gbps with zero issues.
Aside from some scaling issues, it still is a great solution for having real-time insight into the performance of Varnish and the cache.
We even went as far as writing a Golang version of the VCS server to better handle some of the scaling challenges. I don’t remember the exact library that we are using to call into ZeroMQ, but the CGO overhead was minimal.
IIRC it does all I/O on separate background threads which means every operation actually goes through an inter-thread queue. Which can be good and bad.
The problem with ZeroMQ is that it is highly opinionated about threading and concurrency in a way that doesn't necessarily mesh well with other components of your application stack that have their own opinions
There's nothing wrong with the opinions presented there -- but it really wants to own your whole threading / serving stack, and isn't really compatible with e.g. a Rust async tokio binary where threads can "move around" etc.
Well, the original non-thread-safe sockets (e.g., ZMQ_SUB, etc.) were that way because they support multi-part messages. And yes, the fact that they are not thread-safe is a bit of a stumbling-block for ZeroMQ newbies (including me, back in the day).
In any case, newer socket types (e.g., ZMQ_CLIENT) have since been defined that are thread-safe, but necessarily don't support multi-part messages. (They tend to be different in other ways as well -- e.g., ZMQ_RADIO/ZMQ_DISH are "sort of" replacements for the original ZMQ_PUB/ZMQ_SUB sockets, but have other constraints as well).
If you have played around with GNU Radio, ZeroMQ is baked-in for communications with outside applications. I've played with it a bit, and found it to be very much fire-and-forget once you have it set up properly.
That's cool, I've played with GNU Radio some, wasn't aware it used ZeroMQ, actually, I've used several *MQ's but not ZeroMQ, guess I should look into it more.
I wonder if this is one of those problems Erlang and the Beam VM solve out of the box. Native clustering, message passing from one node to the other, no need for a separate dependency... That's the same design, isn't it?
Disclaimer: I don't work with any of that tech right now, but I'm looking forward to working with Elixir, which is in the same ecosystem.
Found Pieter and his books a couple years ago and it make a big impact. The Collective Code Construction Contract is a brilliant piece of community engineering and should be adopted/adapted by every FOSS project on the planet. We can accomplish so much more together if we have C4 and similar methods of coordination. Next time you hear of a burnt-out maintainer, have them read https://rfc.zeromq.org/spec/42/
They legally have to ask every contributor. The exception/loophole used by big companies (but also FSF for example) is that every contributor had to sign a CLA where they legally reassign ownership of their code to the project owner.
CLA’s don’t fully transfer ownership (hence the L) and don’t unilaterally allow changing the software license. Though that does of course depend on the nature of each CLA since there’s no singular contract associated.
A CTA (transfer vs license) does allow unilateral license changes after the fact.
The other "loophole" is not to relicense, but instead for a corporation to make their own future contributions with a different license. This doesn't work too well from GPL-like licenses, but is fine for file-based copyleft.
People contacted were people who made major modifications, not every single contributor. And yeah, if the contributor don't agree to the new license, you can either don't do the license change, or remove the code they wrote as they don't agree to the new license.
Sometimes companies who do "FOSS" make you sign some sort of agreement that they own the code you produce and you won't have any say about re-licensing, so maybe the projects you're thinking about have done that?
IANAL, but if the license isn’t materially compatible (which perhaps is the case for LGPL and MPL, I don’t know MPL enough to say), I would imagine this is the only legal way to do it. When I contribute to a project, I release my code on the agreed upon license. You can’t change that without my permission.
Yes very common. I’ve helped some projects relicense in the past (because the original developer didn’t actually understand the license they chose) and it’s arduous having to contact each individual (and sometimes companies) to relicense.
You can do license changes without that however as long as:
1. You license per file in the repository. This can be quite arduous but many projects will move the old licensed stuff into a sub project to make that easier to grok.
2. Your new license is compatible with the old license.
Yeah. This is how Squeak changed the license from SqueakL to Apache and MIT. That code has a lot of history, so it was a pretty big effort, but worth it in the end.
Fun ZeroMQ fact: bitcoind has used zeromq since 2015. This is used by client software including lightning node daemons to get notification of incoming transactions and blocks in a highly reliable and rapid way.
O wow! It's been a while since the last release, in the back of my mind I had the idea that the project was slowly dying because of Pieter Hintjes' untimely demise... good to see it's still kicking. Very solid software, an absolute joy to work with.
The main difference is the LGPL works at the “library” level, and requires the source for the library is open source. It requires in particular you can replace the library, which means you can’t really static link to it from a closed source program (or open source but LGPL incompatible license).
MPL works on the “source file” level. You have to release any changes you make to MPL licensed files, but you can link those files into a closed source program any way you like.
So I could compile a program, dynamically link an LGPL library and that's fine, but the moment I statically link the same program I'd be violating the LGPL. I never considered that, don't that make the LGPL a pretty poor license for pretty much anything?
If you statically link against an LGPLed library, you must also provide your application in an object (not necessarily source) format, so that a user has the opportunity to modify the library and relink the application.
Improved interop with other open source programs because they used to have a 'one off' license. That makes using ZeroMQ harder than it should be because there is always the question of whether or not their license is more or less permissive than the one that you've picked for your project and risks future unintended side effects. This move should lay that all to rest.
Lawyers have mostly looked at the common open source licenses long enough to know what they say: what is compatible with each other, what is compatible with their [corporate/clients] other needs, and otherwise how it works. While mostly they have not been tested in court, there general consensus they will hold up somehow (nobody knows for sure and there are a couple hundred countries that could each decide their own thing).
When you do your own license though lawyers need to figure everything out for just your project.
Readit News
ZeroMQ was absolutely solid and stable; incredibly trouble free and the only issues we ran into were when IT teams didn't open the ports we documented in the configuration procedure. (The resultant architecture actually looks a lot like Flink)
In any case, ZeroMQ is a fantastic piece of technology that I feel like I don't see out in the wild quite enough. It's dead simple and incredibly stable from my experience.
You just gave me openstack+zeromq flashbacks
What do mature network applications do instead?
For us, delivering an on-prem commercial off the shelf solution, it was untenable to expect the customer IT team to operate a separate, relatively huge piece of tech (remember, this is 2014). Maybe the heuristics would be different today with K8s and advancement of Kafka. But ZeroMQ as an in-process, distributed messaging layer is dead simple. If your use case requires anything else on top of that, it's on the team to design the right solution like resiliency, statefulness, etc.
For a high throughput, distributed compute focused use case, I think ZMQ is still a great choice for coordinating execution. But Kafka and other options on the market now are great choices for higher order abstractions.
I remember we had a similar use-case. This was for collecting live statistics from a sizable Varnish Cache cluster. We wrote a in memory database to store the data. It's been chugging away for about 10 years now and last I heard the zmq traffic alone was about 3Gbps with zero issues.
We even went as far as writing a Golang version of the VCS server to better handle some of the scaling challenges. I don’t remember the exact library that we are using to call into ZeroMQ, but the CGO overhead was minimal.
IIRC it does all I/O on separate background threads which means every operation actually goes through an inter-thread queue. Which can be good and bad.
http://wiki.zeromq.org/blog:multithreading-magic
There's nothing wrong with the opinions presented there -- but it really wants to own your whole threading / serving stack, and isn't really compatible with e.g. a Rust async tokio binary where threads can "move around" etc.
However, it's not that hard to work around their non-thread-safe nature with a single dispatch thread (e.g., see <https://github.com/nyfix/OZ/blob/master/src/transport.c#L457>), and that approach has other potential benefits.
In any case, newer socket types (e.g., ZMQ_CLIENT) have since been defined that are thread-safe, but necessarily don't support multi-part messages. (They tend to be different in other ways as well -- e.g., ZMQ_RADIO/ZMQ_DISH are "sort of" replacements for the original ZMQ_PUB/ZMQ_SUB sockets, but have other constraints as well).
But the truth is that ZeroMQ has a steep learning curve, and some rather sharp edges that can leave nasty cuts if one is not careful.
If you're interested in becoming familiar with ZeroMQ, you might want to check out our implementation: <https://github.com/nyfix/OZ/>.
Disclaimer: I don't work with any of that tech right now, but I'm looking forward to working with Elixir, which is in the same ecosystem.
Deleted Comment
I feel like I've heard of many larger companies doing relicenses on their open source without this kind of effort.
Linux doesn’t have a CLA, and it’s the most popular operating system in the world.
A CTA (transfer vs license) does allow unilateral license changes after the fact.
[0] https://heathermeeker.com/2021/06/01/fsf-drops-assignment-re...
There is more background here: https://github.com/zeromq/libzmq/issues/2376
Sometimes companies who do "FOSS" make you sign some sort of agreement that they own the code you produce and you won't have any say about re-licensing, so maybe the projects you're thinking about have done that?
You can do license changes without that however as long as:
1. You license per file in the repository. This can be quite arduous but many projects will move the old licensed stuff into a sub project to make that easier to grok.
2. Your new license is compatible with the old license.
Deleted Comment
https://github.com/bitcoin/bitcoin/blob/master/doc/zmq.md
https://github.com/zeromq/libzmq/issues/4315 and https://github.com/zeromq/libzmq/issues/4455
I wonder what made the maintainer change his mind.
[1]: https://zguide.zeromq.org
MPL works on the “source file” level. You have to release any changes you make to MPL licensed files, but you can link those files into a closed source program any way you like.
Incorrect.
https://www.gnu.org/licenses/gpl-faq.html#LGPLStaticVsDynami...
This is false. You can static link as long as you follow https://www.gnu.org/licenses/gpl-faq.html#LGPLStaticVsDynami...
Additionally, LGPL3 can be tweaked to add an exception to allow unrestricted static linking. This is like... exactly what ZeroMQ did.
When you do your own license though lawyers need to figure everything out for just your project.
TLDR MPL v2: https://www.tldrlegal.com/license/mozilla-public-license-2-0...