We should have developer benchmarks for programming languages such as "time to first successful compile", "time to first running copy", "time to find the root cause of a certain bug". Similar to that Sun vs NeXT competition back in the day: https://www.youtube.com/watch?v=UGhfB-NICzg
Maybe platforms like Leetcode and HackerRank can publish such statistics.
I'd find such benchmarks way more interesting than comparisons of milliseconds for solutions to some arbitrary algorithms that we'll never use in real life. I'm not saying they aren't useful, but they aren't interesting.
Live competition. Benchmarks would just be metrics recorded during the live broadcast, not goals. You can later aggregate those metrics to come up with accurate benchmarks about many scenarios about software development. Think of NBA but for programming.
Just posting numbers without commentary doesn't seem that useful. Per the GitHub "Note that implementations might be using different optimizations, e.g. with or w/o multithreading, please do read the source code to check if it's a fair comparision or not."
I think this shows that Zig isn't a slow language despite its relative youth, but it'd be much more useful if someone did the work to look through the code and provide commentary on comparable the entries are.
The optimizations both looked correct. Both told the compiler to target broadwell. The fastest nbody was rust, but it was non-portably using x86 intrinsics. Zig has explicit simd vectors in the stdlib and so did better than the portable explicit simd of the third place rust entry. However, zig is using optimized float mode equivalent to gcc ffastmath so it is almost certainly getting the wrong answers simce it didn't use the iterative sqrt trick.
https://github.com/hanabi1224/Programming-Language-Benchmark...
Is zig's optimized float mode also extremely error-prone like gcc ffastmath?
Reminder: With gcc/clang, -ffastmath makes it undefined behavior to run a calculation that results in an infinity or NaN. Due to the way UB works, the compiler can end up miscompiling not just the floating-point calculation, but also other code nearby (e.g. delete array bounds checks).
This is why Rust does not have any fastmath-equivalent: it would allow violating memory safety in safe code.
> I think this shows that Zig isn't a slow language despite its relative youth
Which shouldn't surprise anyone since the heavy lifting is done by LLVM and zig doesn't attempt to add too much semantics on top of it. (For the same reason I don't think benchmarking unsafe Rust makes much sense either)
It makes sense to benchmark unsafe Rust against safe Rust, both for the same algorithm without changes as well as alternative architectures to get around the constraints of the borrow checker and type system. I would go as far as saying that if you want to use unsafe this should be a precondition. It happens too often that someone reaches for unsafe first "for performance wins" that then aren't manifested in practice.
The compiler might make unfortunate decisions which no optimization can fix. E.g. the amount of pointer chasing may affect things a lot. Mutability and aliasing influence drastically which optimizations LLVM would be allowed to make, also stack vs heap allocation for short-lived objects, etc.
I think a more interesting comparison is Zig vs Nim. Where Nim beats or at the very least matches performance against Zig in multiple tests, while having an infinitely nicer, friendlier syntax.
I think Nim looks good, but I'm more likely to learn Rust or Zig personally. I've tried to put my finger on why that is, and I think it's because Nim looks like another good language that is just good all around, it makes good trade-offs and finds a local optimum being good at everything. A lot of language have tried this though, I've seen it all before. Rust did something new, they accepted that "yeah, our language might be really painful to write your seat-of-the-pants business logic in" and they did the borrow checker stuff. Zig is similar, keeping many of the pains of C, but making some important improvements.
I feel the same way. I wish Nim was more popular, but I'm not letting that stop me from using it. It's already extremely useful enough. I spent the last few weeks writing Nim code (specifically an OpenCV binding that wraps just the C++ libraries I need). Nim let's me write fast, compiled, "Pythonic" code. And the INim interactive shell let's me program "1 line at a time" the same way I would with the Python REPL or JavaScript console.
That pythonic syntax is something that prevents Nim to gain more popularity. I think it would be a great move from Nim side to introduce dual syntax support: python like and c/d/java/JavaScript/typescript/go/rust/zig ... like
Afaik, nim was influenced by and originally developed in D-lang. Both are wonderful languages and a real joy to develop on. My ultimate dream is a language that's low level like zig but has the flexibility of nim.
I imagine leetcode has a lot of data on benchmarks of highly optimized solutions implemented in different languages. I wish they would do a blog post or something.
Seems kind of pointless, since the implementations for different languages are independently written and thus the results depend mostly on the skill of the programmers who wrote them (at least for languages that are efficient), and on the quality of the specific standard library features being used if any.
Not entirely pointless since this is open source and programmers of some caliber may be able to contribute better or more efficient code. All benchmarks should be taken with a huge grain of salt since this is rarely what these languages are used for by the majority. As well you shouldn’t be choosing a language solely based on speed and not on the individual application of the language. Memory usage though is something useful and impressive in these examples.
It's not a good idea to use this site (or the Computer Language Benchmarks Games https://benchmarksgame-team.pages.debian.net/benchmarksgame/ that it is partially based off) as an indicator of how may rewrites are necessary in order to generate good programs. The skill levels of the contributors and the size/popularity of the various programming language communities can vary a lot. The benchmark rules have been changing over time and contributors have been figuring out better algorithms over time so these both result in the contributed programs getting updated over time as well. Older programming languages have been around longer than newer programming languages and will have had more contributed programs consequently. Some programming languages are under more active development so require more revamps of existing programs. Etc...
Tldr; they’re about even. Which is what I’d expect given they’re both performance oriented languages which compile via llvm.
As I see it, performance wise Rust has one advantage and one disadvantage compared to zig. Rust’s advantage is that it can add the equivalent of C’s noalias all over the place because of the rules imposed by the borrow checker. This can help the optimizer. And the drawback of rust is that all array accesses are bounds checked. (Well, at least in safe rust). But thanks to prediction intrinsics, the slow down from this is much less than I always expect. Bounds checks do bloat the binary size though.
So rust and zig trading blows benchmark to benchmark is about what I would expect to happen. And that’s exactly what I’m seeing here.
> it can add the equivalent of C’s noalias all over the place
There's an open proposal to do this in Zig as well, with the ability to opt out at the individual parameter level (and with safety checks in debug builds).
Either way we can definitely thank Rust for blazing the trail. noalias in LLVM had never been stress-tested to that degree, and they were finding and fixing noalias-related optimizer bugs for years
People keep being surprised that bounds-checking doesn’t really seem to incur that much cost but frankly it seems pretty straightforward to me.
In the years of Rust code I’ve written, I don’t think I’ve ever actually indexed into an array manually. If I have it’s been an incredibly small number of cases. I’m almost always iterating, which makes bounds checks essentially unnecessary.
As usual, I recall the remarks of C.A.R Hoare on his 1980's Turing Award speech, regarding bounds checking on Algol compilers and customers point of view on how it should be unlawful to do otherwise.
Never, ever, since 1986, have bounds checking been the major source of performance issues on applications I have written.
I suspect it's more that bounds checking actually helps performance in many circumstances in that it can improve branch prediction. Not always, but sometimes.
Sadly, not yet. Even when (Safe) Rust's type system guarantees the noalias annotations will be inserted correctly, the Rust compiler team has been very cautious of turning this on because of numerous LLVM miscompilation issues. It was turned on two years ago in https://github.com/rust-lang/rust/pull/82834 but was subsequently rolled back in https://github.com/rust-lang/rust/pull/86036 because of a miscompliation bug.
i see bound checks as advantage. I don't want random stuff from program tbh. And, if someone really wants to disable bound checks for certain stuff, they can always do that.
I'm divided. The majority of the time i know how big the array is with information the compiler doesn't have. However I have to admit 10% of the time i'm off by one.
Bounds checks can sometimes be optimized out, such as when using iterators or a for loop over a fixed range. LLVM is pretty decent at optimizing out redundant checks too.
You can noalias in C as well, but it is used so infrequently in production C code that llvm didn’t even compile it correctly until a couple years ago. Figuring out when parameters in a C program can safely be declared noalias is very tricky, and I’ve almost never seen anyone bother with it. I assume the same is true in zig.
Nbody looked about the same, zig had really long lines for stuff that rustfmt split. But that aside, I agree the zif metaprogramming really shome here.
I think that this kind of benchmarks should always be taken with a grain of salt, as anyone with some experience with benchmarks knows that there are many ways to distort the results.
Having said that, performance is a key feature for both languages and from what I can see the methodology seems legit.
I'd prefer to also have a C or C++ to use as a performance baseline and to get a clearer overall picture of where the current optimizers stands relative to each others.
Also, competition is good, I believe that we'll all benefit from it in the end, even if the heated debates and quasi-religious stances can be annoying, in the long run this is mostly noise.
Readit News
Maybe platforms like Leetcode and HackerRank can publish such statistics.
I'd find such benchmarks way more interesting than comparisons of milliseconds for solutions to some arbitrary algorithms that we'll never use in real life. I'm not saying they aren't useful, but they aren't interesting.
I think this shows that Zig isn't a slow language despite its relative youth, but it'd be much more useful if someone did the work to look through the code and provide commentary on comparable the entries are.
Reminder: With gcc/clang, -ffastmath makes it undefined behavior to run a calculation that results in an infinity or NaN. Due to the way UB works, the compiler can end up miscompiling not just the floating-point calculation, but also other code nearby (e.g. delete array bounds checks).
This is why Rust does not have any fastmath-equivalent: it would allow violating memory safety in safe code.
Not checking the correctness of the output sounds like a pretty bad oversight for a benchmark
Which shouldn't surprise anyone since the heavy lifting is done by LLVM and zig doesn't attempt to add too much semantics on top of it. (For the same reason I don't think benchmarking unsafe Rust makes much sense either)
https://programming-language-benchmarks.vercel.app/zig-vs-ni...
I love Nim so much, I wish it became more popular somehow. Unfortunately it has stagnated in popularity.
Deleted Comment
That isn't something the website does show.
Language A programs can be implemented in language B without being "independently written".
As I see it, performance wise Rust has one advantage and one disadvantage compared to zig. Rust’s advantage is that it can add the equivalent of C’s noalias all over the place because of the rules imposed by the borrow checker. This can help the optimizer. And the drawback of rust is that all array accesses are bounds checked. (Well, at least in safe rust). But thanks to prediction intrinsics, the slow down from this is much less than I always expect. Bounds checks do bloat the binary size though.
So rust and zig trading blows benchmark to benchmark is about what I would expect to happen. And that’s exactly what I’m seeing here.
There's an open proposal to do this in Zig as well, with the ability to opt out at the individual parameter level (and with safety checks in debug builds).
https://github.com/ziglang/zig/issues/1108
Either way we can definitely thank Rust for blazing the trail. noalias in LLVM had never been stress-tested to that degree, and they were finding and fixing noalias-related optimizer bugs for years
In the years of Rust code I’ve written, I don’t think I’ve ever actually indexed into an array manually. If I have it’s been an incredibly small number of cases. I’m almost always iterating, which makes bounds checks essentially unnecessary.
Never, ever, since 1986, have bounds checking been the major source of performance issues on applications I have written.
Rather ill chosen algorithms or data structures.
You can see it in the function arguments in llvmir: https://rust.godbolt.org/z/a3c366nG6
Having said that, performance is a key feature for both languages and from what I can see the methodology seems legit.
I'd prefer to also have a C or C++ to use as a performance baseline and to get a clearer overall picture of where the current optimizers stands relative to each others.
Also, competition is good, I believe that we'll all benefit from it in the end, even if the heated debates and quasi-religious stances can be annoying, in the long run this is mostly noise.
(I'm assuming the Rust code isn't using "unsafe", if it is ten the "safe" zig numbers are be uninteresting)