When you feel sufficiently amazed by this demo, I also recommend the Elixir in Action book, by this same author, to get started with this incredible ecosystem and paradigm.
---
It's hard to imagine a better platform than Erlang's for writing distributed and networked systems of any kind. I fell in love with Elixir in 2016 and have been using it full time since, first adopting as CTO in my previous company, now powering my new solo business. It's such a good language, with great community and stewardship, running on a rock solid platform that's so advanced compared to most popular languages today.
Erlang was created 38 years ago, and it's good to see the ideas of Armstrong, Virding and Williams to be vindicated today. They simply were too far ahead of their time.
I read Elixir in Action which was a great intro, but it was Concurrent Data Processing in Elixir [1] that really made the penny drop for me.
I have only finished one Elixir app. It was a lot fun and the app has proven rock solid. For me it felt a bit like using microservices but with lots of the footguns and ops overhead removed. I would love to use it for more than a hobby project.
Soul of erlang got me hooked to Elixir recently, trying to get my hands dirty as well.
Other than distributed/concurrent system use-cases, could you share what kind of products are best when built with elixir/erlang compared to easier to write languages like Go, for example.
Anything that spends most of its time waiting on IO, and doesn't do a great deal of number crunching: i.e. any kind of server talking over a network socket.
Go, and any other language, lowers the barrier to running concurrent code, but there is much more to concurrent servers than concurrency: fault tolerance, isolation, shared state management, instrumentation, introspection, clustering, process migration. The BEAM and its ecosystem gives you all of that out of the box.
Also, the BEAM offers an immutable, functional environment. Data races are impossible, which are the biggest pain and source of heisenbugs in any kind of system with > 1 concurrent thread. This is huge. You can model your entire system as concurrent processes without ever having to deal with concurrency issues. You only ever have to think in "single-threaded" mode.
Not sure how the 'best products' constraint is supposed to be interpreted, but Elixir is a more flexible tool for developers than Golang. It has a decent REPL, can be used for scripting, and so on.
I haven't used Golang for things like binary protocols so I can't really compare, but Elixir or Erlang would be a good fit since they're very good for expressing grammars and fundamentally treat strings as byte sequences.
If pattern matching helps you express your problem domain succinctly they're also a good fit. Same goes for macros. My impression is that Golang commonly requires quite verbose or complex code compared to Elixir.
I expect raw number crunching performance to be better in Golang, but BEAM processes are very lightweight so it might win on either performance or developer ergonomics if the task can be solved in parallel.
Thanks for the talk link, it completely kept me engrossed and was a pleasure to watch! Addendum question: what is the state of database drivers for Elixir? Does it have mature libraries for Postgres for eg? I imagine if it's targeting web development, it will need those.
I’m an elixir noob, but I’m actually surprised that the distributed story is not _better_. For example, a big thing about OTP is that it allows running many processes that communicate by message passing: but all these processes are _colocated_ to a given node, i would have expected that OTP (or genserver or whatever) handles clustering of machines, scheduling of workloads and routing of messages for me so that I can start processes and call them without caring about where they run.
What about “distributed systems” is elixir actually shining at? From my reading, it seems rather like a good option to _avoid_ distributing (thanks to the light processes and concurrency handling)
As others have commented it can do all this. What it doesn't do is automatically balance/place GenServers and processes across your cluster. This is up to you. The ways in which you distribute work across a cluster is full of trade-offs.
Need raft concensus? Paxos? Something else? Want to place work with consistent hashing? Want to hydrate/dehydrate with persistence vs memory only?
Those are all important decisions and the languages do not force you to one option.
The structure of the system is such that you can call any function on any node in the cluster.
In order to do it well, you need to understand your own abstractions. If we have a cluster are we better off passing every function around the cluster or running some locally? Is the network overhead worth it in every case or only in some?
It's up to your code to make those determinations about your application, but everything you describe is fairly simple to implement on the BEAM.
First, if you write process-concurrent code in Elixir/Erlang, it will run on all the cores in your machine. It's automatically locally scalable. The number of processes should range from some small multiple of the no. of cores (say, 10x cores), up to 100k - 1m on a big memory multicore machine.
Second, as the other comments say, Elixir/Erlang have very transparent built-in ways to address other nodes, and the programs will scale as expected: more cpus v. network data distribution cost. So good for small tasks that require lots of compute. The native clustering will run to a few dozen nodes over a trusted LAN.
But soon, you will want some layer managing the cluster, try Swarm:
Erlang/Elixir nodes can actually be rather trivially clustered, you can spawn processes on other nodes and transparently pass messages to processes on other nodes, including things like closures. It does not magically schedule workloads across the cluster, but there are libraries that can do so.
Elixir is amazing. Had a stab at it back in 2018, then shifted to work with other traditional langs/stacks, but Elixir has always stayed in the back of my head teasing me. It's an elegant language, that you can see that the authors have put effort in keeping it lean, powerful, expressive. Phoenix (which is the goto web framework) may seem magical in the beginning, but once you dive in you realize that there is no magic, it's all there explicitly in modules (well maybe for macros which do feel magical). What I was missing back in 2018 is IDE support which was iffy. I wonder how that changed recently, if anyone who knows cares to share?
VSCode support for the language is better than ever, outside of VSCode too! NextLS and ElixirLS provide the editor integration you’re probably looking for :)
I like a lot of things about Elixir but I struggled with what felt like clumsy IDE capabilities. Stuff like 'go to definition' that I use all the time in other languages didn't work right. Lack of strong types makes autocomplete problematic. I've started to mess around with Gleam hoping that it spans this gap because I want to like this ecosystem.
I see so much effusive praise for Elixir that I feel like something must have gone really wrong with our codebase, because build times from scratch are horrible (10 minutes or so), and I frequently have problems with the incremental build when switching between branches which force me to do a clean build.
Coming from a Node and PHP background makes me really appreciate the lack of compile step.
I'm fascinated by Elixir, but more so for using to do neural network experiments with more unique topologies, training methods, and activation functions. Being able to spin every neuron out into a process just seems too damn elegant for doing work with spiking neural networks. Will it be as fast as the python stacks? Probably not, but not being constrained to the implicit assumptions in a lot of those libraries will be worth a lot. Plus, there's already good neural network and genetic algorithm libraries for Elixir. I really think this langauge is gonna go places.
Not a lot of technology I have work with every day “sparks joy”.
But the Elixir language is one that’s usually an exception!
I really encourage people to take a look at it even if you don’t end up using it right away- the OTP actor pattern is elegant yet powerful and extremely fun to prototype with.
I recently used Elixir for Advent of Code 2023. It is elegant in many places, at least from what I have explored during puzzle solving. However, to me it does not reach the elegance of Scheme or Lisp and I switched back to Guile while solving more puzzles.
Due to its not as simple syntax, naming choices are more limited than in many Lispy languages. Also the way anonymous functions are written did not appeal to me especially that much and how they must be written when being passed to other function and how to reference functions in the same module sometimes requiring to still write the name of the module and all that is not that intuitive.
I don't know exactly where it happened, but here [1] is a typical AoC code using Elixir. At some point I got so annoyed by Elixir telling me it does not find the functions of the very same module, that I simply defaulted to always writing the module name.
“Why not Elixir” is a more interesting question and I suggest you go ask it to engineering managers of polyglot organizations. They will usually bring you the super low nps from not-elixir-only devs and the resignation letters from elixir “talents” that are asked to do non elixir stuff.
We were an all Elixir (and legacy Ruby) shop with some wildly smart engineers.
Got acquired and jammed into another larger org. Got pressured to switch to Java and Go without considering that maybe our system was designed around the way OTP apps work.
Fast forward a few years and lots of us left, and found successful Elixir jobs where we were much happier elsewhere, making much more interesting technology.
You've successfully been dealing with a sand problem with shovels. Then the new boss tells you about these amazing forks that they've used in the past to address any and all issues and wants you to start using them too.
Quitting is not you being difficult. It's just opting out of a bad developer experience. Contrary to popular belief, you're entitled to also be happy as a professional programmer.
> resignation letters from elixir “talents” that are asked to do non elixir stuff.
I mean, I would prefer to be working in Erlang, but I took a Rusty job recently. OTOH, if I was working in Erlang for you, and you made me switch to something else, I would most likely not be happy and if my job is changing, I may as well change jobs, or at least consider it. Leaving behind simple concurrency and hot loading means it takes a lot longer for me to get things done, and it's one thing to work at a job where things take forever, but it's even worse to go from being able to get things done quickly to slowly.
about the second part of your answer; my (probably very rare) opinion is that our job is not to "work in erlang" or "work in rust", is "solve problems/automate stuff". If I ask you to work in Foo instead of Erlang, it's the same job. I highly doubt that your job is slow because Go and fast because Rust, it's slow because process/idiots in other teams/idiots in your team/idiots as your "agile coach" etc.
I understand wanting to have a good career, but language is never the obstacle to a successful career. Also, this implicit bias that people who know exotic languages are better is completely false.
This often happens regardless of tech. The old school Unix guy that does everything in C... The C# person who hates Java... The Ruby guy that won't do Python... The backend developer that despises Javascript... on and on. Some people compromise, some don't.
Readit News
"The Soul of Erlang and Elixir • Saša Jurić • GOTO 2019" - https://www.youtube.com/watch?v=JvBT4XBdoUE
When you feel sufficiently amazed by this demo, I also recommend the Elixir in Action book, by this same author, to get started with this incredible ecosystem and paradigm.
---
It's hard to imagine a better platform than Erlang's for writing distributed and networked systems of any kind. I fell in love with Elixir in 2016 and have been using it full time since, first adopting as CTO in my previous company, now powering my new solo business. It's such a good language, with great community and stewardship, running on a rock solid platform that's so advanced compared to most popular languages today.
Erlang was created 38 years ago, and it's good to see the ideas of Armstrong, Virding and Williams to be vindicated today. They simply were too far ahead of their time.
I have only finished one Elixir app. It was a lot fun and the app has proven rock solid. For me it felt a bit like using microservices but with lots of the footguns and ops overhead removed. I would love to use it for more than a hobby project.
[1] https://pragprog.com/titles/sgdpelixir/concurrent-data-proce...
Other than distributed/concurrent system use-cases, could you share what kind of products are best when built with elixir/erlang compared to easier to write languages like Go, for example.
Go, and any other language, lowers the barrier to running concurrent code, but there is much more to concurrent servers than concurrency: fault tolerance, isolation, shared state management, instrumentation, introspection, clustering, process migration. The BEAM and its ecosystem gives you all of that out of the box.
Also, the BEAM offers an immutable, functional environment. Data races are impossible, which are the biggest pain and source of heisenbugs in any kind of system with > 1 concurrent thread. This is huge. You can model your entire system as concurrent processes without ever having to deal with concurrency issues. You only ever have to think in "single-threaded" mode.
I haven't used Golang for things like binary protocols so I can't really compare, but Elixir or Erlang would be a good fit since they're very good for expressing grammars and fundamentally treat strings as byte sequences.
If pattern matching helps you express your problem domain succinctly they're also a good fit. Same goes for macros. My impression is that Golang commonly requires quite verbose or complex code compared to Elixir.
I expect raw number crunching performance to be better in Golang, but BEAM processes are very lightweight so it might win on either performance or developer ergonomics if the task can be solved in parallel.
https://hexdocs.pm/ecto/Ecto.html
What about “distributed systems” is elixir actually shining at? From my reading, it seems rather like a good option to _avoid_ distributing (thanks to the light processes and concurrency handling)
A process can be anywhere in the cluster, and you can interact with it as if it was on your local node.
I use all of these facilities in production on my product with ease. I’m happy to answer your specific questions.
Need raft concensus? Paxos? Something else? Want to place work with consistent hashing? Want to hydrate/dehydrate with persistence vs memory only?
Those are all important decisions and the languages do not force you to one option.
The structure of the system is such that you can call any function on any node in the cluster.
In order to do it well, you need to understand your own abstractions. If we have a cluster are we better off passing every function around the cluster or running some locally? Is the network overhead worth it in every case or only in some?
It's up to your code to make those determinations about your application, but everything you describe is fairly simple to implement on the BEAM.
Erlang provides "process groups" that allow you to structure groups of processes across a cluster.
It doesn't try to be magic. It's not a workload scheduler or "automatic distributed machine".
You structure things the way you want, on top of it's primitives. It's just that all of the primitives you need are built in and work really well.
Second, as the other comments say, Elixir/Erlang have very transparent built-in ways to address other nodes, and the programs will scale as expected: more cpus v. network data distribution cost. So good for small tasks that require lots of compute. The native clustering will run to a few dozen nodes over a trusted LAN.
But soon, you will want some layer managing the cluster, try Swarm:
https://github.com/bitwalker/swarm
If you're a Jetbrains addict, there's this IDE extension which has worked well for me: https://github.com/KronicDeth/intellij-elixir
Coming from a Node and PHP background makes me really appreciate the lack of compile step.
But the Elixir language is one that’s usually an exception!
I really encourage people to take a look at it even if you don’t end up using it right away- the OTP actor pattern is elegant yet powerful and extremely fun to prototype with.
Not sure what you mean here. If the function is in the same module then you should be able to refer to it without including the module name.
Can you give some example code to show what you mean?
[1]: https://codeberg.org/ZelphirKaltstahl/advent-of-code-2023/sr...
Edit: Ah yes, I remember it happening in |> pipes a lot.
We were an all Elixir (and legacy Ruby) shop with some wildly smart engineers.
Got acquired and jammed into another larger org. Got pressured to switch to Java and Go without considering that maybe our system was designed around the way OTP apps work.
Fast forward a few years and lots of us left, and found successful Elixir jobs where we were much happier elsewhere, making much more interesting technology.
Sounds more like an org problem to me.
Quitting is not you being difficult. It's just opting out of a bad developer experience. Contrary to popular belief, you're entitled to also be happy as a professional programmer.
(A similar story has happened at Walmart when their architects did not understand F# and instituted a Java rewrite of an acquired company)
Net Promoter Score?
> resignation letters from elixir “talents” that are asked to do non elixir stuff.
I mean, I would prefer to be working in Erlang, but I took a Rusty job recently. OTOH, if I was working in Erlang for you, and you made me switch to something else, I would most likely not be happy and if my job is changing, I may as well change jobs, or at least consider it. Leaving behind simple concurrency and hot loading means it takes a lot longer for me to get things done, and it's one thing to work at a job where things take forever, but it's even worse to go from being able to get things done quickly to slowly.
about the second part of your answer; my (probably very rare) opinion is that our job is not to "work in erlang" or "work in rust", is "solve problems/automate stuff". If I ask you to work in Foo instead of Erlang, it's the same job. I highly doubt that your job is slow because Go and fast because Rust, it's slow because process/idiots in other teams/idiots in your team/idiots as your "agile coach" etc.
I understand wanting to have a good career, but language is never the obstacle to a successful career. Also, this implicit bias that people who know exotic languages are better is completely false.