While all the points are valid what scares me most about Julia is what is covered under the section "The core language is unstable".
Recently I tried the latest beta version and came across an issue - I just could not add packages. (`] add JSON` or
). No matter what package I tried to add, it failed with a stacktrace pointing to libuv.
I tried downgrading to lower versions of Julia but no luck.
Turns out it is a regression introduced by a commit dated Sep 23 2020. In libuv.
As I tried to fix it, I realized Julia uses a version of libuv that is significantly diverged from libuv main branch ("123 commits ahead, 144 commits behind libuv:v1.x"
as I write this comment) and the bug was in the code. Using the code from the corresponding function from libuv upstream fixed the problem.
To summarize
1. There's a bug introduced close to a year ago that breaks basic functionality on a mainstream operating system.
2. Julia has its own version of, of all things, libuv, that is significantly diverged from the original.
3. The bug is in one of the changes introduced.
While each of the above is defensible on its own. Taken together, they do scare me away from considering Julia for production use. I am hoping am wrong somewhere.
I think it is a lovely language.
Do you have a link to an open issue tracking this problem?
Several Julia core devs are also libuv maintainers or contributors. The main reason for the divergence is that Julia’s libuv fork has a significantly more flexible event loop model that allows using libuv from multiple threads efficiently. The main libuv project has been reluctant to accept that change since it’s a quite advanced capability that Node doesn’t need.
> The main reason for the divergence is that Julia’s libuv fork has a significantly more flexible event loop model that allows using libuv from multiple threads efficiently.
I don't buy the argument that not fastidiously following libuv upstream is an example of Julia not being stable. It's the opposite: that's an indicator of some measure of stability.
The situation could easily be opposite: suppose libuv does stupid things and breaks, and so if Julia tracks the daily build of libuv, it breaks.
"unstable" is not a word which means "staying on top of the development of every dependency". That could literally be a clause in a working definition of "unstable"!
If you lag behind in updating dependencies then there will be situations where you don't get some bugfix for a while
Maintaining your own fork is a good idea, because sometimes fixes are security issues from advisories. You don't necessarily want to jump to the latest and greatest libxy.so, picking up 75 other changes to fix one security item: those items are risky, because they can contain undiscovered bugs. You can carry the security patch for now and then drop it when you update.
Indeed. And in this case, the #3 contributor (and #1 in recent times, really) to Julia is also the current primary maintainer of libuv so it’s very unlikely that there would be any critical fixes that would be missed.
This particular example turns out to be a case where a feature was developed in Julia first and there’s some corner case on this poster’s Windows 10 setup that triggers a bug in the new feature.
I mean, this just looks like your run-of-the-mill bug, or? If you filter on the bug label on GitHub you will find many more. It would be interesting to try figure out what is special about your setup since no one has reported this bug since it got introduced (and it doesn't show up on any of the CI).
I was curious about that too. Many other people use Windows 10 and this is the first time this issue has come up, so there must be something else unusual going on here. If no one could install packages on Windows 10, we'd have heard about it.
I mean, that's bad and scary, but you did mention that you were using a beta version of julia. Finding this sort of thing is exactly why the beta versions exist.
If the beta was stable and ready for production, it wouldn't be called beta...
Oh, the problem is present in 1.6.2(current stable release) as well as v1.5.4(released March '21 available under old releases). I could not find any other 1.5+ version to try.
OP, thank you for this post. I use Julia daily, and it's likewise my favorite language. So I went into this post prepared to be...defensive. But several times in the post I found myself thinking "no! Wait, well, maybe, hmm, urgh, yeah that's not great."
I'd like to read an epilogue or follow-up "so what" post. If Julia is your favorite language but many of these problems seem like some combination of a) unsolvable given the design priorities of the language; b) unsolvable until a hypothetical future 2.0; c) solvable but only with huge resource increases, how do you manage these problems day to day? What do you see as the moderate term future?
That sounds like a good idea. The problem is that I just don't know enough about compilers, Julia's internals, and how the core dev team works to really forecast the future and know what's possible. How hard would it be to implement caching of native code? A fast interpreter that runs side by side with the compiler? No idea.
Maybe it would still be useful to make a post explaining how I cope with the limitations of Julia. I'll think about it.
Great article, thanks for writing this, I’ve had a lot of frustration with the Julia community being unwilling to see faults in the language.
I really want to like Julia and I think I’m theory it has a lot to offer but due to much of what you listed I find it hard to develop in.
One thing I would add is that package management is also odd to me. It seems to have a lot of overhead and behave in unexpected ways.
In general I think Julia could learn a lot from Go. When I develop in Go most everything is simple. The type system is easy and just works, packaging is simple and usable, it compiles quickly. I fend writing the language seamless and productive.
Unfortunately Go also isn’t good for data science due to the memory model so Julia has a lot of room to step in there.
>I’ve had a lot of frustration with the Julia community being unwilling to see faults in the language.
What? It's basically a meme at this point for major contributors to write a list of what they think is bad. The language's creator did a nice Youtube video "What's bad about Julia?" at JuliaCon 2019 (https://www.youtube.com/watch?v=TPuJsgyu87U), one of my most popular blog posts of all time is "7 Julia Gotchas and How To Handle Them" (https://www.stochasticlifestyle.com/7-julia-gotchas-handle/), etc. What the community pushes back about is things that are factually incorrect or just silly and repetitive, like the n+1th discussion about 1-based indexing. It's stuff below like, "I want to do an interactive workflow but not use the REPL, so it's not working well"... well the REPL is literally the interactive piece that caches the compilation so just use it if that's what you're trying to do? Yes, nobody has time for that nonsense, but everyone chimes in for real substantive discussions like changing the Base iterator API, programming patterns that would reduce recompilation, etc.
I don't think that's fair. Your blog post is not about what you think is bad - at least, it doesn't read like it. It reads like a list of stumbling blocks and how to get around them. This is not the same as listing the major problems with the language.
Jeff's talk is indeed great, but... it's not mostly about what he thinks is bad, either. That part is quickly swept aside with "just so you know we know". The talk is really about some very specific problems in the type system that are so obscure that I don't even consider them worth mentioning in a blogpost about Julia's weaknesses.
As someone who has used Julia for production use, I basically agree with everything you've said.
IMO:
1) Julia rushed to v1.0, but its understandable - it is sort of a catch-22 issue. More developers won't join unless it is stable (v1.0). But it won't get stable unless a lot of developers use it and feedback is heard.
2) I personally found that Julia community is slightly hostile to feedback and negativity. May be it is just me but it has way too much hype-driven-positivity that leads to delusion.
3) A lot happened between v0.2 to v1.0 which IMO should have been carefully and slowly done.
4) Developer experience should have been one of the major focuses. Stack traces should be beautiful and absolutely transparent. Debuggers are clunky - I used the Atom (Juno) debugger and wanted to toss the laptop off the window - just shows how frustrating it was. Compile times and smoothness of dev experience pays dividends and it was largely ignored.
5) I really like official libraries, not SomeBasicFunctionality.jl dependency. I am spoiled by Python.
6) Marketing around speed is misplaced IMO. It lures people like me into fanboys of Julia. There is so much more to a programming language than speed.
I don't want to be overly negative. The language and the original paper is beautiful - and this is all hard stuff. Kudos for the progress made so far.
> 2) I personally found that Julia community is slightly hostile to feedback and negativity. May be it is just me but it has way too much hype-driven-positivity that leads to delusion.
I want to push back on this a bit, which I acknowledge is very ironic. In the past few years people consistently post on Discourse asking for fundamental changes to the language to make it more resemble python, C++, or whatever their preferred language is.
People often say Go is great because there is "only one way of doing things", yet people are very resistant to being told "the way" to do something in Julia. This has happened enough that it's prompted a pinned PSA on discourse: https://discourse.julialang.org/t/psa-julia-is-not-at-that-s...
It gets tiring! And i'm not sure how the community should handle these requests, but I don't think it's fair to blame all of the negativity on the Julia community when these somewhat misinformed, or even bad-faith posts are so frequent.
Completely agree with 4 and 5. I use Julia regularly and while it is great for scripts and proof of concepts of mathematical stuff it might be too complicated to implement in Python or in C++ once one starts to grow an application it is tremendously frustrating.
I have clear and satisfying workflows in C (vim + make + gdb + valgrind) and Python (vim/vs code, pdb) but I cannot say that debugging and tracing errors is a comfortable experience in Julia. Essential development tools are relegated to third-party libraries (e.g. Revise.jl and Debugger.jl) and are often clunky or responsiveness is limited (e.g. Revise.jl won't work when you change a struct)
I believe the problem is that, because of Julia is marketed towards scientific computing and HPC, it draws mostly from an academic user base which is not that interested in software development.
> I personally found that Julia community is slightly hostile to feedback and negativity. May be it is just me but it has way too much hype-driven-positivity that leads to delusion.
Part of the problem, at least from what I observed here and in other dev communities, was that Julia had a large number of bad faith critics (probably larger than the number of active users, at the time) from about 0.2 to 0.5 or so. I'm talking about everyone who compared the runtime of hello world in the REPL, with start up time, to that of a compiled executable. There were piles and piles of fake benchmarks that did little more than compare BLAS bindings as though the differences were intrinsic to the language.
Sure, the Julia community could have reacted better to such (and a few people did: BenchmarkTools.jl was an early win in this area), but few people do react that way in practice.
> 5) I really like official libraries, not SomeBasicFunctionality.jl dependency. I am spoiled by Python.
Python's standard library is full of bad code that nobody has time to improve [1]. Over time, Julia packages will surpass the standard library too but we'll still be hauling it around and spending core-dev time on it for for backward-compatibility reasons. We should have a much much smaller standard library.
Others have commented on stacktraces, so I wanted to mention that the debugging experience in VS Code (the replacement for Atom-based Juno) is much improved as of the past couple of months. Worth a revisit if you're using it already.
Regarding stdlib support, I feel like it's a toss-up. For numeric and data processing code, Julia has a much richer stdlib whereas I need at least Numpy in Python (and even then, working with anything not array-shaped is a pain). For more systems-y stuff though, Python has a much fuller stdlib (e.g. reading/writing a bunch of archive formats). The end result for me is that scripting with Julia feels high friction, and doing anything which requires any kind of throughput with Python feels like pulling teeth.
> I’ve had a lot of frustration with the Julia community being unwilling to see faults in the language.
A related thing that really turned me off in the early days was one of the main developers constantly criticizing other languages. That wouldn't be so bad by itself, but much of it was uninformed, and it told me at least one of the core devs was not going to learn from other languages. There was always an attitude that "I know what to do, unlike everyone else".
The reason I never moved heavily into Julia was that it was too easy to write slow code (back then, things may have changed) relative to compiled languages like Fortran. Maybe I'd have stuck with it longer if I found the community to be pleasant, but an unstable and often slow language plus an unpleasant core developer was more than I was willing to put up with.
Opposite experience here, it was crazy to me how fast I could prototype my number crunching on price series, how trivial it was to distribute it across multiple machines in my house and finally how easy it was to flip to gpu computation. On every step I was surprised how fast it was.
Community is great, met only kind people, exceptionally knowledgable bunch.
A couple of very vocal individuals have an attitude, which reflects badly. But the community is getting bigger, so it's becoming less of an issue as most interactions don't include those people.
It's still easy to write slow code. But writing fast code is often easier than it would be in C++ or whatever.
It’s easy to write slow code in any language. I wrote some slow Fortran before I learned how not to. Generally you need to know a good amount about not only the language, but about the machine you’re targeting, to write really efficient code.
I don't think Julia is trying to be anything like Go and it shouldn't be.
Probably Julia's best feature is the powerful metaprogramming support that allows you to, e.g., create array packages that automatically run code on GPU. You can't have such features with a simple language.
That doesn’t mean it can’t learn from it. Yeah they are solving somewhat different problems but in sheer usability of a language Julia could take a lot from Go.
> powerful metaprogramming support ... can't have such features with a simple language
Various Scheme implementations are among the simplest languages I've ever used yet possesses some of the most powerful metaprogramming I know of. Perhaps I misunderstood you somehow?
All languages have problems. Don't take my post to mean "don't use Julia" - the very first words of the post is that it's my favorite language. Also do note that the webpage is built with Julia. Not a coincidence.
But even good systems need critique, otherwise there can be no improvement or growth.
Things can have faults and be worth investing in, it doesn't need to be an all or nothing false dilemma. Nothing would get fixed if we all just self selected in this way.
Julia is incredibly fascinating for me as a language to observe, although I haven't got the time/patience to try it out yet. Its language design seems incredibly progressive and yet pragmatic: they've repopularized dynamic dispatch (which makes perfect sense in a mathematics-oriented language), and has a unique mix of dynamic and static typing that seems perfect in paper.
In practice, Julia seems to suffer from the initial architectural decisions that the creators has made when the language was young - the biggest which is its heavy dependence on LLVM. This is a problem since LLVM is mainly used for static compilation of code for languages like C++/Rust/Zig and not with dynamic execution - its included JIT is famous for being unreasonably slow. Even the dev seems to know this but they don't seem to have that much choice - since too much of their codebase is dependent on it. (For example: https://discourse.julialang.org/t/jeff-bezanson-remarks-on-l... - LLVM codegen keeps getting slower, which in turn cancels out all the little optimizations they can do for the language) It's a language that seems so perfect on paper but suffers from its real-world implementation - and therefore captivates me in a special way. Some major architectural overhaul might be a much better move (such as a fallback bytecode interpreter independent from LLVM) - but it's definitely going to be a major effort.
Using LLVM is not exactly an indelible architectural decision — switching to a bespoke JIT would be entirely possible, it’s just not worth it yet. Moreover, if you want to generate world-class high speed code, you really need something like LLVM. So yes, it’s a bit of a drag but it’s also a massive enabler of critical performance. That’s the real reason the project continues to use it.
There are a number of approaches to reducing latency further, which are being pursued. The most promising current avenue is better tooling around system images (compiled executables with additional functionality beyond the core language pre-loaded): generating them faster, generating them incrementally (load one, add a few things and produce another), and generating them automatically for the set of packages in the current active project (not feasible until the first two are done). That will give an experience comparable to compiled languages.
How much communication is there between the Julia compiler team and the LLVM team? I don't get the impression LLVM has been very sensitive to Julia's needs so far.
You may be interested in the current initiative around compiler plugins [1]. One of the explicit design goals is offload to alternative backends like GPUs or Wasm, but you could see how that would extend to something like Cranelift as well.
That said, LLVM codegen performance is very much a tail wagging the dog thing, and I wish upstream put a significantly larger focus on it [2]
In my opinion the most important positive thing Julia has is the Compsci community it attracted.
There has never been so much enthusiasm in oftware for doing physical simulations and correct and stable arithmetics. It made a sizeable dent in the Fortran/C++/Python/Matlab environment physicists were in.
That said even after 10 years Julia has still so many rough edges that I wouldn't employ it for anything outside the aforementioned use case.
Just getting it to Plot something in a headless environment was a pain. Essential utilities like debuggers (!) are external dependencies. Matrix/Vector/Array confusion permeates the air. Ditto for a syntax annoyingly in the middle between Matlab, Python and Fortran.
Given it is already not that young I predict it will never really attract the masses beyond its niche, but it can still kill the remaining Fortran codebases.
It's a little hard to say "after 10 years" when Julia has been at 1.0 for 3 years only. And remember, that's not with Rust's niche of low-level developers, but with a niche of people who are generally not programmers, and therefore much more conservative about their languages!
It's surprisingly hard to get good data about how quickly Julia grows, but it certainly doesn't look like it's plateauing yet. It's just too early to know how far it'll go and where it will end up in 10 years.
If anything, Julia becoming popular with academics makes me extremely skeptical about its missing guardrails. Obviously anecdotal, but I have many years of experience working with academics and statisticians and I know to be wary when they suddenly like a new technology. Usually it's because it makes it easier to swallow errors somehow, or because their friend wrote it and sent them an email about it.
I like Julia because of the trivial 10x performance improvements I get over writing similar code in Python/numpy.
What I don't like is how Julia attempts to force you into a REPL workflow. Everything is optimized for REPL work. I just want to write a script in a text editor and run it from the command line. Please make this easy for me.
> In programming I tend to think the whole computer and its OS and so on as the programming tool. My impression is that for Julia folks Julia is something totally separate from anything else in the computer. But I don't understand why. Is it that its felt that changing syntax on the fly is too cognitively demanding or something? This is something that one gets used to very fast.
This comment really does not resonate with me. It sounds like a criticism of Smalltalk, not Julia.
Not sure what it means to change "syntax on the fly", but that does sound like the sort of design that is maximally flexible, at a considerable cost to readability. Macros already allow you to embed DSLs (as long as they parse), and almost everyone agree that they should be used sparingly.
> And also tends to ease the IMHO weird hung-ups on syntax. In fact, I think Julia has made some (IMHO misguided) syntactic decisions because they just want to do something differently from Python just for the sake of being different. I find this totally senseless.
I am not aware of any decision that is done just for the sake of being different. If there are any, then yes that's senseless.
> Not sure what it means to change "syntax on the fly"
GP meant that Julia enthusiasts seem to dislike the cognitive overhead of switching between languages (e.g. Bash scripts, SQL, C code, Python scripts, and as GP wants, Julia scripts as well) while doing multi-language development using the whole Unix computer as a programming tool.
It's really hard to do, because having a command-line centered development experience with Julia is a complete nonstarter until compile time latency is near zero. And that's just really, really hard, and unlikely to happen, at least any time soon.
That comment you linked is amazing - it steps back a little bit and gets our head of out the dirt to see the horizon. As a bunch of programming ostriches try to muck about this or that - the author instructively dictates that general purpose programming languages are part of the toolbox to get the job done...on an operating system of choice.
We need to listen to people that have gone through the experience of putting complex systems together and their learnings are immeasurably valuable. This is the eternal balance of theoriticians and practioners.
It also further cements something that I always felt but can be offputting to some people - Julia is an academic language used by academicians. From a software engineering perspective (not computer science, mind you), there is much to be desired.
I really like Julia, and also get the sense that the community is a little young and unsettled in some ways. The biggest way I've seen this manifest is in terms of a total lack of decent documentation for reputable, well known projects. For example, there's this trend in the Julia community of putting little clever puns in the github package description. Okay I'm not trying to rain on anyone's parade and I appreciate humor but when I'm looking for packages that little github description area is critical. Maybe save the puns for the readme doc? Beyond that, it's frustrating to find core libraries with absolutely zero API documentation or with totally outdated/incomplete/misleading docs.
I'm not all that mad about it or anything so I won't point fingers. I just find it to be out of touch and mostly just a buzzkill. In the past when considering investing my time and energy in building a project in Julia, I've found it frustrating to have to do all this guesswork to get a sense for where critical packages are at and what their limitations are.
So from an optimistic angle, I think this probably reflects the fact that there are quite a few researchers writing Julia packages, which is really cool. There are some awesome cutting edge techniques implemented as Julia packages. I don't think researchers should be off the hook for maintaining crappy packages, but I do think it takes some higher level thinking to cultivate good tools and collective habits around this stuff. For whatever reason, this is evidently challenging for the Julia community.
I hope that some of the culture and tooling around documentation and package maintenance gets more mature over the next few years. If anyone knows about efforts to improve documentation in the Julia ecosystem, I'd love to hear about them.
Not to call anyone out but examples? Every time I look at packages I’m blown away by how much effort has gone into their docs. But people keep saying this kind of thing so there must be some packages they’re hitting that are under-documented.
I find that "online" docs are good, but "in-repl" docs are usually poor to inexistent (in packages, that is).
Base Julia has improved on this a lot, though there's still room for improvement (especially in terms of useful examples or linking to related functions).
Edit: tldr, Julia could use something with the ease of use and functionality of docs.rs, but built especially with Julia's typesystem in mind.
So after thinking about this, I think there are two things going on here. 1) it's actually the lack of browsable autogenerated API docs I'm frustrated by... I'm spoiled by the most excellent Rust docs.rs API docs, which give a great, quick, readable, comprehensive overview of what's in a package even if the maintainer hasn't made any docstrings and (2) rust has had this emphasis on usable documentation as a seamless part of the development experience for a while, but Julia is definitely catching up.
So I went back and checked and was happy to find that the JuliaGPU packages that I previously couldn't find docs for definitely have some docs now! In particular, GPUArrays.jl. There were also some astronomy packages I looked at had been rewritten with docs left hanging for like a year.
That said, in the autogenerated API docs for GPUArrays.jl, if there's a function with no docstrings, on JuliaHub it just shows a big yellow warning to the developer. I'd prefer if it showed some useful information about the types the function is defined over and its return types. I'd also love if there was some quick way to see a list of included types and functions, along with their type signature and even a way to view the code.
Really I think I'm just spoiled by the Rust community's amazing auto-generated API docs on docs.rs, which seamlessly integrate with examples and readme style docs. If there's a rust package I wanna use, docs.rs will give me a nice consistent, browsable overview of the code and I can usually figure out what's in there just from that, even if the package maintainer hasn't actually written any example docs or docstrings, just using info from the typesystem. It's so nice to be able to go to one place and see what's in a package, the traits, structs and function signatures, all alongside docs generated from docstrings and handwritten docs. Did I mention that this information is always in the same place on docs.rs? These aren't just "filler" docs, they're super usable.
Most Julia package docs are more freeform and I have to click around to find the API docs, and honestly I'm not sure if every package even has these. Whereas on docs.rs they're right there immediately with no cognitive overhead. Freeform docs are awesome, and I'm always excited when a package has lots of well thought out documentation, but it's no substitute for up to date, informative API docs that give you a solid window into the fundamentals and interfaces.
Julia has such a cool typesystem, I could imagine there are some interesting opportunities to use that to make the autogenerated Julia docs much more usable and informative.
As far as the comprehensive approach is concerned, I honestly don't think Julia is that far off, it's really a shift in emphasis and a streamlining that I'm wishing for. I can see that a lot of progress has been made.
Maybe it'd be worthwhile to consider hiring some of those ex-Mozilla people to help the Julia community get to the next level on this?
The compile time latency is the biggest the biggest show stopper for me.
Yes, you can cope with it by staying in the REPL and they are actively working on it but it is very unfortunate that the "slower" competitors like Python and R feel so much more responsive while developing.
Other than that, Julia is a absolutely beautiful language. Sure, it is not offering the same static safety as Rust or Haskell but I would not hold that against it. There is always some trade offs to be made regarding the dynamic vs static spectrum and the dynamic aspects serve Julia well for the typical use cases.
The compile time latency is really bad for some libraries. I was recently testing out computing class groups using https://github.com/oscar-system/Oscar.jl and was surprised to hit *several minute* latency due to LLVM compilation. For me, the most striking things are that (1) there is no indication in the REPL that calling a function is slow due to LLVM compilation, rather than the function actually running, and (2) it is difficult to avoid paying the compilation penalty every single time you restart Julia, since caching the JIT in general is a technically very, very difficult problem. I wonder if there is an easy fix for (1) that I don't know about: some way of showing a spinner or something saying "I'm compiling code right now".
Readit News
Turns out it is a regression introduced by a commit dated Sep 23 2020. In libuv.
As I tried to fix it, I realized Julia uses a version of libuv that is significantly diverged from libuv main branch ("123 commits ahead, 144 commits behind libuv:v1.x" as I write this comment) and the bug was in the code. Using the code from the corresponding function from libuv upstream fixed the problem.
To summarize
1. There's a bug introduced close to a year ago that breaks basic functionality on a mainstream operating system. 2. Julia has its own version of, of all things, libuv, that is significantly diverged from the original. 3. The bug is in one of the changes introduced.
While each of the above is defensible on its own. Taken together, they do scare me away from considering Julia for production use. I am hoping am wrong somewhere. I think it is a lovely language.
Several Julia core devs are also libuv maintainers or contributors. The main reason for the divergence is that Julia’s libuv fork has a significantly more flexible event loop model that allows using libuv from multiple threads efficiently. The main libuv project has been reluctant to accept that change since it’s a quite advanced capability that Node doesn’t need.
> Do you have a link to an open issue tracking this problem?
Yes [#41642](https://github.com/JuliaLang/julia/issues/41642)
> The main reason for the divergence is that Julia’s libuv fork has a significantly more flexible event loop model that allows using libuv from multiple threads efficiently.
Glad to hear this. That's impressive.
The situation could easily be opposite: suppose libuv does stupid things and breaks, and so if Julia tracks the daily build of libuv, it breaks.
"unstable" is not a word which means "staying on top of the development of every dependency". That could literally be a clause in a working definition of "unstable"!
If you lag behind in updating dependencies then there will be situations where you don't get some bugfix for a while
Maintaining your own fork is a good idea, because sometimes fixes are security issues from advisories. You don't necessarily want to jump to the latest and greatest libxy.so, picking up 75 other changes to fix one security item: those items are risky, because they can contain undiscovered bugs. You can carry the security patch for now and then drop it when you update.
This particular example turns out to be a case where a feature was developed in Julia first and there’s some corner case on this poster’s Windows 10 setup that triggers a bug in the new feature.
If the beta was stable and ready for production, it wouldn't be called beta...
I'd like to read an epilogue or follow-up "so what" post. If Julia is your favorite language but many of these problems seem like some combination of a) unsolvable given the design priorities of the language; b) unsolvable until a hypothetical future 2.0; c) solvable but only with huge resource increases, how do you manage these problems day to day? What do you see as the moderate term future?
That sounds like a good idea. The problem is that I just don't know enough about compilers, Julia's internals, and how the core dev team works to really forecast the future and know what's possible. How hard would it be to implement caching of native code? A fast interpreter that runs side by side with the compiler? No idea.
Maybe it would still be useful to make a post explaining how I cope with the limitations of Julia. I'll think about it.
I really want to like Julia and I think I’m theory it has a lot to offer but due to much of what you listed I find it hard to develop in.
One thing I would add is that package management is also odd to me. It seems to have a lot of overhead and behave in unexpected ways.
In general I think Julia could learn a lot from Go. When I develop in Go most everything is simple. The type system is easy and just works, packaging is simple and usable, it compiles quickly. I fend writing the language seamless and productive.
Unfortunately Go also isn’t good for data science due to the memory model so Julia has a lot of room to step in there.
What? It's basically a meme at this point for major contributors to write a list of what they think is bad. The language's creator did a nice Youtube video "What's bad about Julia?" at JuliaCon 2019 (https://www.youtube.com/watch?v=TPuJsgyu87U), one of my most popular blog posts of all time is "7 Julia Gotchas and How To Handle Them" (https://www.stochasticlifestyle.com/7-julia-gotchas-handle/), etc. What the community pushes back about is things that are factually incorrect or just silly and repetitive, like the n+1th discussion about 1-based indexing. It's stuff below like, "I want to do an interactive workflow but not use the REPL, so it's not working well"... well the REPL is literally the interactive piece that caches the compilation so just use it if that's what you're trying to do? Yes, nobody has time for that nonsense, but everyone chimes in for real substantive discussions like changing the Base iterator API, programming patterns that would reduce recompilation, etc.
Jeff's talk is indeed great, but... it's not mostly about what he thinks is bad, either. That part is quickly swept aside with "just so you know we know". The talk is really about some very specific problems in the type system that are so obscure that I don't even consider them worth mentioning in a blogpost about Julia's weaknesses.
> well the REPL is literally the interactive piece that caches the compilation so just use it if that's what you're trying to do?
This answer may be exactly the kind of hostility that the GP is talking about.
People may have legitimate reasons for not using the REPL. Instead of adressing these reasons you dismiss them. A textbook case of "holding it wrong".
Deleted Comment
IMO:
1) Julia rushed to v1.0, but its understandable - it is sort of a catch-22 issue. More developers won't join unless it is stable (v1.0). But it won't get stable unless a lot of developers use it and feedback is heard.
2) I personally found that Julia community is slightly hostile to feedback and negativity. May be it is just me but it has way too much hype-driven-positivity that leads to delusion.
3) A lot happened between v0.2 to v1.0 which IMO should have been carefully and slowly done.
4) Developer experience should have been one of the major focuses. Stack traces should be beautiful and absolutely transparent. Debuggers are clunky - I used the Atom (Juno) debugger and wanted to toss the laptop off the window - just shows how frustrating it was. Compile times and smoothness of dev experience pays dividends and it was largely ignored.
5) I really like official libraries, not SomeBasicFunctionality.jl dependency. I am spoiled by Python.
6) Marketing around speed is misplaced IMO. It lures people like me into fanboys of Julia. There is so much more to a programming language than speed.
I don't want to be overly negative. The language and the original paper is beautiful - and this is all hard stuff. Kudos for the progress made so far.
I want to push back on this a bit, which I acknowledge is very ironic. In the past few years people consistently post on Discourse asking for fundamental changes to the language to make it more resemble python, C++, or whatever their preferred language is.
People often say Go is great because there is "only one way of doing things", yet people are very resistant to being told "the way" to do something in Julia. This has happened enough that it's prompted a pinned PSA on discourse: https://discourse.julialang.org/t/psa-julia-is-not-at-that-s...
It gets tiring! And i'm not sure how the community should handle these requests, but I don't think it's fair to blame all of the negativity on the Julia community when these somewhat misinformed, or even bad-faith posts are so frequent.
I have clear and satisfying workflows in C (vim + make + gdb + valgrind) and Python (vim/vs code, pdb) but I cannot say that debugging and tracing errors is a comfortable experience in Julia. Essential development tools are relegated to third-party libraries (e.g. Revise.jl and Debugger.jl) and are often clunky or responsiveness is limited (e.g. Revise.jl won't work when you change a struct)
I believe the problem is that, because of Julia is marketed towards scientific computing and HPC, it draws mostly from an academic user base which is not that interested in software development.
Part of the problem, at least from what I observed here and in other dev communities, was that Julia had a large number of bad faith critics (probably larger than the number of active users, at the time) from about 0.2 to 0.5 or so. I'm talking about everyone who compared the runtime of hello world in the REPL, with start up time, to that of a compiled executable. There were piles and piles of fake benchmarks that did little more than compare BLAS bindings as though the differences were intrinsic to the language.
Sure, the Julia community could have reacted better to such (and a few people did: BenchmarkTools.jl was an early win in this area), but few people do react that way in practice.
> 5) I really like official libraries, not SomeBasicFunctionality.jl dependency. I am spoiled by Python.
Python's standard library is full of bad code that nobody has time to improve [1]. Over time, Julia packages will surpass the standard library too but we'll still be hauling it around and spending core-dev time on it for for backward-compatibility reasons. We should have a much much smaller standard library.
[1] https://pyfound.blogspot.com/2019/05/amber-brown-batteries-i...
Regarding stdlib support, I feel like it's a toss-up. For numeric and data processing code, Julia has a much richer stdlib whereas I need at least Numpy in Python (and even then, working with anything not array-shaped is a pain). For more systems-y stuff though, Python has a much fuller stdlib (e.g. reading/writing a bunch of archive formats). The end result for me is that scripting with Julia feels high friction, and doing anything which requires any kind of throughput with Python feels like pulling teeth.
Then, proper debugging and introspection tools will be easier to build.
A related thing that really turned me off in the early days was one of the main developers constantly criticizing other languages. That wouldn't be so bad by itself, but much of it was uninformed, and it told me at least one of the core devs was not going to learn from other languages. There was always an attitude that "I know what to do, unlike everyone else".
The reason I never moved heavily into Julia was that it was too easy to write slow code (back then, things may have changed) relative to compiled languages like Fortran. Maybe I'd have stuck with it longer if I found the community to be pleasant, but an unstable and often slow language plus an unpleasant core developer was more than I was willing to put up with.
Community is great, met only kind people, exceptionally knowledgable bunch.
It's still easy to write slow code. But writing fast code is often easier than it would be in C++ or whatever.
Come try again if you like.
https://julialang.org/blog/2012/02/why-we-created-julia/
Probably Julia's best feature is the powerful metaprogramming support that allows you to, e.g., create array packages that automatically run code on GPU. You can't have such features with a simple language.
Various Scheme implementations are among the simplest languages I've ever used yet possesses some of the most powerful metaprogramming I know of. Perhaps I misunderstood you somehow?
How about just moving on? It's not like there is a lack of languages and tools out there.
But even good systems need critique, otherwise there can be no improvement or growth.
Things can have faults and be worth investing in, it doesn't need to be an all or nothing false dilemma. Nothing would get fixed if we all just self selected in this way.
Nowadays I’m doing my ML in rust which has been pretty nice, but the borrow checker does bite you a bit and there’s not much of a community.
Julia seems really promising, the community just gets their head in the sand a bit
In practice, Julia seems to suffer from the initial architectural decisions that the creators has made when the language was young - the biggest which is its heavy dependence on LLVM. This is a problem since LLVM is mainly used for static compilation of code for languages like C++/Rust/Zig and not with dynamic execution - its included JIT is famous for being unreasonably slow. Even the dev seems to know this but they don't seem to have that much choice - since too much of their codebase is dependent on it. (For example: https://discourse.julialang.org/t/jeff-bezanson-remarks-on-l... - LLVM codegen keeps getting slower, which in turn cancels out all the little optimizations they can do for the language) It's a language that seems so perfect on paper but suffers from its real-world implementation - and therefore captivates me in a special way. Some major architectural overhaul might be a much better move (such as a fallback bytecode interpreter independent from LLVM) - but it's definitely going to be a major effort.
There are a number of approaches to reducing latency further, which are being pursued. The most promising current avenue is better tooling around system images (compiled executables with additional functionality beyond the core language pre-loaded): generating them faster, generating them incrementally (load one, add a few things and produce another), and generating them automatically for the set of packages in the current active project (not feasible until the first two are done). That will give an experience comparable to compiled languages.
Dead Comment
That said, LLVM codegen performance is very much a tail wagging the dog thing, and I wish upstream put a significantly larger focus on it [2]
[1] https://hackmd.io/bVhb97Q4QTWeBQw8Rq4IFw?view [2] https://www.npopov.com/2020/05/10/Make-LLVM-fast-again.html
There has never been so much enthusiasm in oftware for doing physical simulations and correct and stable arithmetics. It made a sizeable dent in the Fortran/C++/Python/Matlab environment physicists were in.
That said even after 10 years Julia has still so many rough edges that I wouldn't employ it for anything outside the aforementioned use case.
Just getting it to Plot something in a headless environment was a pain. Essential utilities like debuggers (!) are external dependencies. Matrix/Vector/Array confusion permeates the air. Ditto for a syntax annoyingly in the middle between Matlab, Python and Fortran.
Given it is already not that young I predict it will never really attract the masses beyond its niche, but it can still kill the remaining Fortran codebases.
It's surprisingly hard to get good data about how quickly Julia grows, but it certainly doesn't look like it's plateauing yet. It's just too early to know how far it'll go and where it will end up in 10 years.
What I don't like is how Julia attempts to force you into a REPL workflow. Everything is optimized for REPL work. I just want to write a script in a text editor and run it from the command line. Please make this easy for me.
(Other HN comments have said this in more detail, for example: https://news.ycombinator.com/item?id=27310747.)
This comment really does not resonate with me. It sounds like a criticism of Smalltalk, not Julia. Not sure what it means to change "syntax on the fly", but that does sound like the sort of design that is maximally flexible, at a considerable cost to readability. Macros already allow you to embed DSLs (as long as they parse), and almost everyone agree that they should be used sparingly.
> And also tends to ease the IMHO weird hung-ups on syntax. In fact, I think Julia has made some (IMHO misguided) syntactic decisions because they just want to do something differently from Python just for the sake of being different. I find this totally senseless.
I am not aware of any decision that is done just for the sake of being different. If there are any, then yes that's senseless.
GP meant that Julia enthusiasts seem to dislike the cognitive overhead of switching between languages (e.g. Bash scripts, SQL, C code, Python scripts, and as GP wants, Julia scripts as well) while doing multi-language development using the whole Unix computer as a programming tool.
As opposed to doing everything in a Julia REPL.
We need to listen to people that have gone through the experience of putting complex systems together and their learnings are immeasurably valuable. This is the eternal balance of theoriticians and practioners.
It also further cements something that I always felt but can be offputting to some people - Julia is an academic language used by academicians. From a software engineering perspective (not computer science, mind you), there is much to be desired.
I'm not all that mad about it or anything so I won't point fingers. I just find it to be out of touch and mostly just a buzzkill. In the past when considering investing my time and energy in building a project in Julia, I've found it frustrating to have to do all this guesswork to get a sense for where critical packages are at and what their limitations are.
So from an optimistic angle, I think this probably reflects the fact that there are quite a few researchers writing Julia packages, which is really cool. There are some awesome cutting edge techniques implemented as Julia packages. I don't think researchers should be off the hook for maintaining crappy packages, but I do think it takes some higher level thinking to cultivate good tools and collective habits around this stuff. For whatever reason, this is evidently challenging for the Julia community.
I hope that some of the culture and tooling around documentation and package maintenance gets more mature over the next few years. If anyone knows about efforts to improve documentation in the Julia ecosystem, I'd love to hear about them.
Base Julia has improved on this a lot, though there's still room for improvement (especially in terms of useful examples or linking to related functions).
So after thinking about this, I think there are two things going on here. 1) it's actually the lack of browsable autogenerated API docs I'm frustrated by... I'm spoiled by the most excellent Rust docs.rs API docs, which give a great, quick, readable, comprehensive overview of what's in a package even if the maintainer hasn't made any docstrings and (2) rust has had this emphasis on usable documentation as a seamless part of the development experience for a while, but Julia is definitely catching up.
So I went back and checked and was happy to find that the JuliaGPU packages that I previously couldn't find docs for definitely have some docs now! In particular, GPUArrays.jl. There were also some astronomy packages I looked at had been rewritten with docs left hanging for like a year.
That said, in the autogenerated API docs for GPUArrays.jl, if there's a function with no docstrings, on JuliaHub it just shows a big yellow warning to the developer. I'd prefer if it showed some useful information about the types the function is defined over and its return types. I'd also love if there was some quick way to see a list of included types and functions, along with their type signature and even a way to view the code.
Really I think I'm just spoiled by the Rust community's amazing auto-generated API docs on docs.rs, which seamlessly integrate with examples and readme style docs. If there's a rust package I wanna use, docs.rs will give me a nice consistent, browsable overview of the code and I can usually figure out what's in there just from that, even if the package maintainer hasn't actually written any example docs or docstrings, just using info from the typesystem. It's so nice to be able to go to one place and see what's in a package, the traits, structs and function signatures, all alongside docs generated from docstrings and handwritten docs. Did I mention that this information is always in the same place on docs.rs? These aren't just "filler" docs, they're super usable.
Most Julia package docs are more freeform and I have to click around to find the API docs, and honestly I'm not sure if every package even has these. Whereas on docs.rs they're right there immediately with no cognitive overhead. Freeform docs are awesome, and I'm always excited when a package has lots of well thought out documentation, but it's no substitute for up to date, informative API docs that give you a solid window into the fundamentals and interfaces.
Julia has such a cool typesystem, I could imagine there are some interesting opportunities to use that to make the autogenerated Julia docs much more usable and informative.
As far as the comprehensive approach is concerned, I honestly don't think Julia is that far off, it's really a shift in emphasis and a streamlining that I'm wishing for. I can see that a lot of progress has been made.
Maybe it'd be worthwhile to consider hiring some of those ex-Mozilla people to help the Julia community get to the next level on this?
Yes, you can cope with it by staying in the REPL and they are actively working on it but it is very unfortunate that the "slower" competitors like Python and R feel so much more responsive while developing.
Other than that, Julia is a absolutely beautiful language. Sure, it is not offering the same static safety as Rust or Haskell but I would not hold that against it. There is always some trade offs to be made regarding the dynamic vs static spectrum and the dynamic aspects serve Julia well for the typical use cases.