I think the top-level take away here is not that Julia is a great language (although it is) and that they should use it for all the things (although that's not the worst idea), but that its design has hit on something that has made a major step forwards in terms of our ability to achieve code reuse. It is actually the case in Julia that you can take generic algorithms that were written by one person and custom types that were written by other people and just use them together efficiently and effectively. This majorly raises the table stakes for code reuse in programming languages. Language designers should not copy all the features of Julia, but they should at the very least understand why this works so well, and be able to accomplish this level of code reuse in future designs.
Interesting. Can you give an example of generic algorithms plus custom types, in practice? Off the top of my head I thought that any dynamic language or static language with good genetics would have this property, but maybe there's something that Julia does differently.
As an additional example, I really like the combination of unitful and diffeq [1]. But you're right that the core feature that allows this stuff is duck typing, but by itself it's not enough. Your notduck had to not only quack, but other animals have to look at it and act like it's a duck sometimes and like a notduck when you want it to do more than a duck. Multiple Dispatch (plus parametric subtyping) allows you to trivially define both notduck + A (notduck.+(A) in OOP languages) and A + notduck (extending A whatever A is) and it's really fast. That allows for the core Julia concept of specialization, easily customizing the particular behavior of any agent at any point to get both the common behavior right and the extended behavior.
For static languages you can implement part of it with, for example, interfaces (you'll face the same restrictions if the language is single dispatch), but even if you can extend the interface freely for already existing objects, there must be an agreement between the multiple packages to comply to the same interfaces (and you might either end with tons of interfaces since there are tons of possible behaviors for each entity and purpose or giant interfaces to fit all). In Julia you can use specialization to surgically smooth the integration between two packages that had no knowledge of each other and didn't even decide to comply to any (informal) interface (which do exist in Julia, like the Julia Array and Tables.jl interfaces).
DiffEq on ForwardDiff Dual numbers to calculate sensitivity via forward mode AD.
DiffEq on Tracker's TrackedArray to calculate sensitivity via reverse mode AD.
Measurements.jl's numbers that track measurement error input any algorithm to compute the transformed measurement error after the algorithm is applied.
NamedDims.jl + Flux.jl to give everything that PyTorch's awesome Named Tensors feature gives.
I really like Julia a lot and actually used it in a work project a few years back.
However, there's the debugger issue. There are several debugger alternatives. It's tough to figure out which debugger is canonical (or is any of them the canonical debugger?). The one that seems to being used most at this point is Debugger.jl. However, it's exceedingly slow if you're debugging sizeable operations (matrix multiplies, for example) - I'm talking hitting 'n' to step to the next line and then waiting several minutes for it to get there. There's also Rebugger.jl, MagneticReadHead.jl (IIRC) and Infiltrator.jl among others. I finally found that Infiltrator.jl was a lot faster for that machine learning program I was trying to debug, but it's rather limited in features (the only way to set breakpoints it seems is by editing your source, for example).
And this isn't the only case where there are multiple packages for achieving some task and you're not quite sure which one is the one that's the most usable. I think what the Julia community needs to do is maybe add some kind of rating system for packages so you can see which packages have the highest rating.
I wrote 2 of them.
MagneticReadHead.jl
and MixedModeDebugger.jl
MagneticReadhead is interesting as it is a purely compiled debugger via extensive source code transfroms.
Same magic that powers Zygote.jl for AutoDiff.
Also same general concept as is behind Jax.
It has huge compile time overhead, so is not practically usable.
In that way it is the opposite of Debugger.jl, rather than being slow at runtime it is very slow at JIT compiling.
MixedModeDebugger.jl is a proof of concept.
A small Source Code Transform to allow the debugger to run entirely compiled until it is going to do something, then it swaps to interpreted just like Debugger.jl
Early benchmarks are extremely promising.
But it's not really hardened enough for use.
1. Debugger.jl is A LOT smoother if you run it in compiled mode, which is a checkbox in the Juno interface. I've found that stepping to next line is instant in compiled mode, but takes forever without it.
2. Infiltrator.jl is great at what it's designed for, which is to dump you in a REPL deep within a call stack and let you see what's going on. But, Debugger in compiled mode also does this well.
> 1. Debugger.jl is A LOT smoother if you run it in compiled mode, which is a checkbox in the Juno interface. I've found that stepping to next line is instant in compiled mode, but takes forever without it.
Is there a way to do this if I'm not running Juno? I'd guess there must be some parameter that can be passed to @enter or @run?
There are parts of Julia I really like but it has some problems.
* Multiple dispatch is an odd design pattern that seems to over complicate things. I know there are people that love it and claim it’s better, but after working with it for some time I just want a struct with methods. It’s much easier to reason about.
* The packaging is cumbersome. I understand their reasoning behind it but in practice it’s just more of a pain than denoting functions as public one way or another.
* The tooling is poor. I work with Go for my day job and it’s toolchain is an absolute pleasure. The Julia toolchain isn’t in the same arena.
* The JIT is slowwww to boot. I was amazed the first time running a Julia program how slow it was. You could practically compile it faster.
* Editor support has never been great.
* As others have mentioned type checks don’t go deep enough
I think it has some neat ideas and in certain scientific arenas it will be useful, but IMO they need to focus a bit more on making it a better general purpose language.
While I don't agree with many of these points, I do agree that some of these can be substantially improved. We continue to work hard at it. Some are research problems, while others need elbow grease.
Just to present the other side, here's a recent thread on Julia discourse about why people love Julia. Many chiming in there are recent users of Julia and I think it is insightful.
Yea I really wanted to like Julia overall and many of the parts I like about it are on this thread. I think it's apparent we need a better numeric language than python, I just wish Julia would focus a bit more on utility.
I've been using Julia along with python and Pytorch, not yet for machine learning until flux is more mature but for NLP scripts, and I have to say that I'm starting to like it. Multiple dispatch, linear algebra and numpy built in, dynamic language but with optional types, user defined types, etc.
I hope Julia will be more popular in bioinformatics. Personally, I have a high hopes for BioJulia[1][2][3] and the amazing AI framework FluxML[4][5] + Turing.jl[6][7]. Apart from the speed, they offer some interesting concepts too - I recommend to check them out.
I will have to disagree. FluxML is indeed great, but it changes often and it does not support many of the advanced features of TensorFlow (neither is there a package that seamlessly works with Flux in order to support these features). It is getting there, but Tensorflow v2 is pretty great itself, and frequently faster. But FluxML might soon be as good or better.
Also, to be fair, FluxML is backed by a couple of people while Tensorflow is backed by megacorps, so it is already impressive how much they have done.
How much of the BioJulia stuff would you say currently works? It looks like a lot of repos have been created, and the scope is pretty impressive (looks like there are repos for everything from structural bioinformatics to population genetics), but a lot of them look to be basically empty(https://github.com/BioJulia/PopGen.jl), or have really scary looking issues:(e.g https://github.com/BioJulia/GeneticVariation.jl/issues/25).
% of the repos in the org on github? That number is lower than I'd like. % of the repos that are actively maintained? Much higher.
One of the great things about julia is that it's really easy to throw together a package and register it. One of the bad things about julia is how easy it is for those one-off projects or idea dumps to pollute the space. We could definitely do a better job labeling the repos that are no longer being maintained or that aren't actually ready for prime time. There's a tradition in julia of a lot of really functional libraries to stay < v1.0, because we all take semver seriously, and if the interface is still in a bit of flux, making the switch to 1.0 is a big deal (DataFrames.jl, looking at you). But it does make it hard for new users to distinguish between a super robust package and someone's weekend hobby.
Not my field, but at least some of it appears to be worked on seriously. This was an interesting recent blog post about making DNA-sequence processing go fast:
I've always found Julia a little lacking for bioinformatics, but I'm not doing ML. I have very high hopes for Nim, which I think has better performance potential than Julia across domains and can produce binaries.
Nim made me really sad when they gave up on multiple dispatch.
I has such potential as a language but I just can't imagine using one without multiple dispatch anymore
Julia is great. It’s significantly simpler than Python while also being much more expressive. It’s too bad the typing is only for dispatch, but hopefully someone will write a typechecker someday. I’ve found it surprisingly refreshing to not have to think about classes and just add functions on objects wherever I want. Some languages solve this with monkey patching (which is bad), others like Scala with an extension class (reasonable, but you still don’t get access to private properties), but the Julia approach is cleaner.
I wouldn’t use Julia for a non-scientific computing app as I don’t think it’s suitable, but for anything data science related, it’s great! And with the Python interop, I don’t really think there’s any reason not to use Julia for your next data science project. I suspect that over the next 5 years Python will no longer be used for these applications at all.
Python will still be used 20 years from now. The clear advantage of Python is the enormous ecosystem that is available, the millions of questions on SO giving solutions to every problem you can run into, the books and learning materials etc, programmers and corporations having invested loads of time and effort in building, maintaining and battle-testing libraries.
Don't get me wrong, i think Julia is an amazing language, but being an amazing language is neither necessary or sufficient to succeed. R shows how you can succeed just fine with a kinda weird language.
I remember when Python was a new and exciting language and people said the exact same thing about Perl. "You'll never see Python replace Perl for string processing, Perl has such a huge ecosystem!". At the time Perl was the de facto interpreted/"scripting" language.
Sure Perl is around still, but it has become a rather niche language used in a small number of specific communities.
I use Python everyday, and still haven't had enough time to properly learn Julia, and I would certainly not be shocked to see Julia take over the lion's share of numeric computing from Python in 20 years.
Python's ecosystem is great - but Julia's is growing incredibly fast, and in some cases Julia has already surpassed what is available in other languages (for example, take a look at the whole differential equations ecosystem: https://github.com/JuliaDiffEq).
Also, Python's ecosystem is only a 'pyimport(name)' away (using the PyCall.jl package). Same thing is true for R and a number of other languages (RCall.jl, JavaCall.jl, etc.)
I've been using SymPy, QisKit, matplotlib and other Python packages with no problem in Julia.
I would be careful treating this too much like a popularity contest. Python will still be used 20 years from now, but that doesn't necessarily mean that it's going to be dominant 20 years from now.
Source: I used to do a lot of programming in various dialects of Basic, which, a bit over 20 years ago, was popular largely because of its own ubiquity and popularity. And, while I still maintain some Basic code, I was surprised how quickly it died. One day everything was being written in it. The next day, we were were writing new things in a fancy new language that everyone agreed was more productive, and talking to the existing stuff through an FFI. And, a day later, we were replacing modules in order to get them off of the "legacy" platform.
>The clear advantage of Python is the enormous ecosystem that is available, the millions of questions on SO giving solutions to every problem you can run into
All of this is also available in Julia via its near-seamless Python interop. Its package manager even does a better job of managing Python packages than Python does.
> The clear advantage of Python is the enormous ecosystem that is available
Could have said that about Perl back in the day too, or a great number of languages.
The key problem Python faces competing against Julia in the long run is that Julia runs faster with less resources. By that I mean that because packages in Julia combine easier and are written all in Julia, it is simply much faster to develop equivalent functionality in Julia compared to Python.
Thus as Julia grows the speed of advancement will just keep growing. There is also a sort of asymptotic curve for most software. As you reach certain complexity advancing gets slower. Python due to its age has acquired a lot of cruft which will slow development down.
I know exactly how this feels from having worked on very similar software products of different age. The older software really had problems keeping up speed. The younger software moved ahead faster due to cleaner design. We needed far less people to add more features than the competition.
Python will struggle with old design decisions it can no longer undo. Look at e.g. the enormous amount of man hours required to get JIT compilation working in Python. It still does not work well. Meanwhile Julia has require less manpower making a whole language with better JIT compilation.
Language design matters over time. I don't claim Julia will overtake Python any time soon. But over a long time frame I think it is inevitable, because legacy goes against Python in too many areas.
> R shows how you can succeed just fine with a kinda weird language.
One thing that most people don't appreciate about R is the subtle influence from lisp world. This makes it really feel like the language is optimized for data science down to the most basic syntax level.
> the millions of questions on SO giving solutions to every problem you can run into
I definitely know what you mean. I too have the experience of typing in basically natural language queries into Google and have stack overflow turn it into copy-paste-able code for me.
But what's also true is that in 20 years the problems being solved will likely be bigger, and more complicated, and in many cases we'll be solving them on platforms that require good abstractions over multiple cores. So none of that is really in Python's wheelhouse. (Complexity is an issue because abstrations in Python almost always come at a cost.)
I've been using Julia for non-scientific computing programs for almost 5 years now, and (especially now that it is stable since the 1.0 release) have found it well suited for general programming as well.
Having a language that is easy to write (like Python), runs fast (like C/C++), and incredibly flexible & expressive (like Lisp) makes programming fun again!
> It’s too bad the typing is only for dispatch, but hopefully someone will write a typechecker someday
I was under the impression that if you write a function that takes very specific sub-types as its parameters, and then try to call it with invalid parameters (that is, values that cannot be converted to the right type via the promotion rules) that Julia will complain about this?
What part of Julia isn't lispy enough that you don't consider it a lisp? I'm not saying you're wrong, just because there a scheme in there, just curious.
Trashing is a little harsh, but Julia devs probably were mostly Python devs back in the day and are intimately familiar with the inadequacies of the language: the work required when you had to drop into C, the bad syntax for math, the constant conversions between ndarray, array, and lists, etc etc.
At my old bioinformatics lab, Python literally wasted thousands and thousands of collective man-hours which would have been saved by Julia if it had existed at the time. Since a lot of these researchers couldn’t really program that well, they would write code that would literally take weeks to run. And then they would call me and I would rewrite it in C and it would then take an hour or two. Julia solves this problem.
The amount of unnecessary supercomputer time (and electricity) that our lab (and others) were wasting with Python was, honestly, disgusting.
I haven't seen that at all - many Julia programmers are (or were) also Python programmers.
I think there is a lot of respect in the Julia community for Python & the Python ecosystem.
There have even been a number of Julia talks at various PyCons over the past few years.
Hrm ... I recall Pythonistas trashing Perl as line noise, and other languages as well, for many many years. I'm not saying its right, just that karma, sometimes, comes back into focus.
Julia is a language I really wanted to like, and to a certain extent I do. However after spending some time working with it and hanging out on the Discourse channels (they certainly have a very friendly and open community, which I think is a big plus), I've come to the tentative conclusion that its application domains are going to be more limited than I would have hoped.
This article hits on some of the issues that the community tends to see as advantages but that I think will prove limiting in the long run.
> Missing features like:
> Weak conventions about namespace pollution
> Never got around to making it easy to use local modules, outside of packages
> A type system that can’t be used to check correctness
These are some of my biggest gripes about Julia, especially the last two. To these I would add:
* Lack of support for formally defined interfaces.
* Lack of support for implementation inheritance.
Together with Julia's many strengths I think these design choices and community philosophy lead to a language that is very good for small scale and experimental work but will have major issues scaling to very complex systems development projects and will be ill-suited to mission critical applications.
In short I think Julia may be a great language for prototyping an object detection algorithm, but I wouldn't want to use it to develop the control system for a self-driving car.
Unfortunately this means that Julia probably isn't really going to solve the "2 language problem" because in most cases you're still going to need to rewrite your prototypes in a different language just like you would previously in going from, for example, a Matlab prototype to a C++ system in production.
You touch upon some interesting pain points. I really like Julia and working with it is a pleasure.
Except the Module system, which feels unnecessarily arcane. I'm happy to be educated on why, but it seems to successfully combine the awkwardness of C-style #include with the mess of a free-form module system. The end result is a Frankenstein monster where technically everything is possible, everything could be included anywhere, there are no boundaries or even conventions. It makes for a frustrating experience for a newbie.
Say you have a package, and inside is a file called `xyz.jl`. You open the file and it defines a module called Xyz. But this tells you absolutely nothing about where in the package Xyz will appear. It could be included somewhere deep in the hierarchy, or it could be a submodule. It could be included multiple times in multiple places! That's bad design for sure, but the language places no boundaries on you. You open another file `abc.jl`, and see no modules at all, just a bunch of functions, which in turn call other functions that are defined God knows where. A julia file does not have to contain any information about where the symbols it's using come from, since it will be just pasted in verbatim to some location somewhere.
The whole module system feels like one big spaghetti of spooky action at a distance.
It's a shame too, because the rest of the language is very neat. Once one gets over the hurdle of the modules, it is possible to establish conventions to bring some sanity in there, but it's a hurdle that many people will probably not want to deal with.
It seems great to me that paths & source files are mostly irrelevant, you're free to re-organise without changing anything. And that `using Xyz` is always talking to the package manager. You can make sub-modules and say `using .Xyz`, but there's very little need to do so, and few packages do.
You can shoot yourself in the foot by including source twice, as you can by generating it with a macro, or simply copy-pasting wrong.
Author of post here:
there were a major gripe for me starting out too.
It took me a fair while to conclude that they allowed to useful things in the bigger picture.
and it certainly is not a pure win.
I do miss static typing.
I would question the claim it doesn't scale to production.
I know people who have build hugely complex production systems in perl that are still running today 20 years later.
Further, I myself work on what we believe to be the largest closed source julia code base, in terms of number of contributors, number of packages and total size. (Its also pretty large in general, though i have yet to work out how it stacks up against DiffEq-verse).
And I have seen thing go from research prototype into running in production.
It works.
I am not going to deny though there are advantages to other languages.
There are many trade-offs in the world
People write large-scale systems in dynamically-typed languages all the time. Multiple dispatch and macros make clean scaling easier than it would be in most other dynamic languages. Its competitors in numerical performance are C/C++ and Fortran, which are both minefields (C much more so). Julia is definitely safer in practice than these kind of languages with weak, unsafe type systems.
I'm not saying static types don't have benefits as well, but it would also be very against the design goals as a Matlab/R competitor.
Inheritance would also directly clash and overlap with multiple dispatch, which is strictly more powerful.
> I'm not saying static types don't have benefits as well,
It's funny; formerly I was a die-hard fan of static typing, but, lately, my opinions have become more nuanced. Something more along the lines of, "Yeah, I'd never want to just remove the type annotations from my Java code and then try to maintain the result, but some dynamic languages allow me to have a fraction as much code to maintain in the first place."
I'm also beginning to wonder if my feelings about dynamic languages have been unduly influenced by some particularly popular, and also particularly undisciplined, dynamic languages. JavaScript and PHP, for example.
> multiple dispatch, which is strictly more powerful.
In a language that supports classes I can have class B inherit from class A and automatically provide all of class A's functionality without adding a single extra line of code. I can extend class B's functionality by adding only code that is specific to it.
I don't see how to do that with multiple dispatch, at least the way it's implemented in Julia.
I don't understand type systems very well. When you say "check correctness" do you mean something beyond static linting with type-hints, like in Python? Or do you mean something deeper, like in functional languages like Elm and F#?
Also, is there always a trade-off between types and flexible meta-programming? Like, OCaml has meta-programming capabilities, but they make type-checking way harder, according to my PhD friend who's written extensively in Scheme and OCaml.
Yep this has been my conclusion as well, I really wanted to like Julia and there are parts of it I do, but I think it misses the mark in some big ways.
Look under Static Julia Compiler here [1]. I've successfully done it in Linux, don't know about Windows. It doesn't work when you have binary dependencies like GTK, etc.
Readit News
https://twitter.com/_cormullion/status/1224640188518932483
For static languages you can implement part of it with, for example, interfaces (you'll face the same restrictions if the language is single dispatch), but even if you can extend the interface freely for already existing objects, there must be an agreement between the multiple packages to comply to the same interfaces (and you might either end with tons of interfaces since there are tons of possible behaviors for each entity and purpose or giant interfaces to fit all). In Julia you can use specialization to surgically smooth the integration between two packages that had no knowledge of each other and didn't even decide to comply to any (informal) interface (which do exist in Julia, like the Julia Array and Tables.jl interfaces).
[1] https://tutorials.juliadiffeq.org/html/type_handling/03-unit...
Measurements.jl's numbers that track measurement error input any algorithm to compute the transformed measurement error after the algorithm is applied.
NamedDims.jl + Flux.jl to give everything that PyTorch's awesome Named Tensors feature gives.
However, there's the debugger issue. There are several debugger alternatives. It's tough to figure out which debugger is canonical (or is any of them the canonical debugger?). The one that seems to being used most at this point is Debugger.jl. However, it's exceedingly slow if you're debugging sizeable operations (matrix multiplies, for example) - I'm talking hitting 'n' to step to the next line and then waiting several minutes for it to get there. There's also Rebugger.jl, MagneticReadHead.jl (IIRC) and Infiltrator.jl among others. I finally found that Infiltrator.jl was a lot faster for that machine learning program I was trying to debug, but it's rather limited in features (the only way to set breakpoints it seems is by editing your source, for example).
And this isn't the only case where there are multiple packages for achieving some task and you're not quite sure which one is the one that's the most usable. I think what the Julia community needs to do is maybe add some kind of rating system for packages so you can see which packages have the highest rating.
MagneticReadhead is interesting as it is a purely compiled debugger via extensive source code transfroms. Same magic that powers Zygote.jl for AutoDiff. Also same general concept as is behind Jax.
It has huge compile time overhead, so is not practically usable. In that way it is the opposite of Debugger.jl, rather than being slow at runtime it is very slow at JIT compiling.
MixedModeDebugger.jl is a proof of concept. A small Source Code Transform to allow the debugger to run entirely compiled until it is going to do something, then it swaps to interpreted just like Debugger.jl Early benchmarks are extremely promising. But it's not really hardened enough for use.
1. Debugger.jl is A LOT smoother if you run it in compiled mode, which is a checkbox in the Juno interface. I've found that stepping to next line is instant in compiled mode, but takes forever without it.
2. Infiltrator.jl is great at what it's designed for, which is to dump you in a REPL deep within a call stack and let you see what's going on. But, Debugger in compiled mode also does this well.
Is there a way to do this if I'm not running Juno? I'd guess there must be some parameter that can be passed to @enter or @run?
* Multiple dispatch is an odd design pattern that seems to over complicate things. I know there are people that love it and claim it’s better, but after working with it for some time I just want a struct with methods. It’s much easier to reason about.
* The packaging is cumbersome. I understand their reasoning behind it but in practice it’s just more of a pain than denoting functions as public one way or another.
* The tooling is poor. I work with Go for my day job and it’s toolchain is an absolute pleasure. The Julia toolchain isn’t in the same arena.
* The JIT is slowwww to boot. I was amazed the first time running a Julia program how slow it was. You could practically compile it faster.
* Editor support has never been great.
* As others have mentioned type checks don’t go deep enough
I think it has some neat ideas and in certain scientific arenas it will be useful, but IMO they need to focus a bit more on making it a better general purpose language.
Just to present the other side, here's a recent thread on Julia discourse about why people love Julia. Many chiming in there are recent users of Julia and I think it is insightful.
https://discourse.julialang.org/t/in-as-few-lines-as-possibl...
One that I particularly enjoyed reading about:
https://discourse.julialang.org/t/in-as-few-lines-as-possibl...
[1] https://biojulia.net/
[2] https://github.com/BioJulia
[3] https://github.com/BioJulia
[4] https://fluxml.ai/
[5] https://github.com/FluxML/
[6] https://turing.ml/dev/
[7] https://github.com/TuringLang
Also, to be fair, FluxML is backed by a couple of people while Tensorflow is backed by megacorps, so it is already impressive how much they have done.
One of the great things about julia is that it's really easy to throw together a package and register it. One of the bad things about julia is how easy it is for those one-off projects or idea dumps to pollute the space. We could definitely do a better job labeling the repos that are no longer being maintained or that aren't actually ready for prime time. There's a tradition in julia of a lot of really functional libraries to stay < v1.0, because we all take semver seriously, and if the interface is still in a bit of flux, making the switch to 1.0 is a big deal (DataFrames.jl, looking at you). But it does make it hard for new users to distinguish between a super robust package and someone's weekend hobby.
https://biojulia.net/post/seq-lang/
Deleted Comment
I wouldn’t use Julia for a non-scientific computing app as I don’t think it’s suitable, but for anything data science related, it’s great! And with the Python interop, I don’t really think there’s any reason not to use Julia for your next data science project. I suspect that over the next 5 years Python will no longer be used for these applications at all.
Don't get me wrong, i think Julia is an amazing language, but being an amazing language is neither necessary or sufficient to succeed. R shows how you can succeed just fine with a kinda weird language.
Sure Perl is around still, but it has become a rather niche language used in a small number of specific communities.
I use Python everyday, and still haven't had enough time to properly learn Julia, and I would certainly not be shocked to see Julia take over the lion's share of numeric computing from Python in 20 years.
Source: I used to do a lot of programming in various dialects of Basic, which, a bit over 20 years ago, was popular largely because of its own ubiquity and popularity. And, while I still maintain some Basic code, I was surprised how quickly it died. One day everything was being written in it. The next day, we were were writing new things in a fancy new language that everyone agreed was more productive, and talking to the existing stuff through an FFI. And, a day later, we were replacing modules in order to get them off of the "legacy" platform.
All of this is also available in Julia via its near-seamless Python interop. Its package manager even does a better job of managing Python packages than Python does.
Could have said that about Perl back in the day too, or a great number of languages.
The key problem Python faces competing against Julia in the long run is that Julia runs faster with less resources. By that I mean that because packages in Julia combine easier and are written all in Julia, it is simply much faster to develop equivalent functionality in Julia compared to Python.
Thus as Julia grows the speed of advancement will just keep growing. There is also a sort of asymptotic curve for most software. As you reach certain complexity advancing gets slower. Python due to its age has acquired a lot of cruft which will slow development down.
I know exactly how this feels from having worked on very similar software products of different age. The older software really had problems keeping up speed. The younger software moved ahead faster due to cleaner design. We needed far less people to add more features than the competition.
Python will struggle with old design decisions it can no longer undo. Look at e.g. the enormous amount of man hours required to get JIT compilation working in Python. It still does not work well. Meanwhile Julia has require less manpower making a whole language with better JIT compilation.
Language design matters over time. I don't claim Julia will overtake Python any time soon. But over a long time frame I think it is inevitable, because legacy goes against Python in too many areas.
One thing that most people don't appreciate about R is the subtle influence from lisp world. This makes it really feel like the language is optimized for data science down to the most basic syntax level.
So much technology has come and gone since 2000. If innovation continues to accelerate I'd be hesitant to predict what 2040 will look like.
Programming might become so simple and automated that we won't need StackOverflow to the extent we do today.
It's really hard to predict the next year or two let alone the next 20.
I definitely know what you mean. I too have the experience of typing in basically natural language queries into Google and have stack overflow turn it into copy-paste-able code for me.
But what's also true is that in 20 years the problems being solved will likely be bigger, and more complicated, and in many cases we'll be solving them on platforms that require good abstractions over multiple cores. So none of that is really in Python's wheelhouse. (Complexity is an issue because abstrations in Python almost always come at a cost.)
I was under the impression that if you write a function that takes very specific sub-types as its parameters, and then try to call it with invalid parameters (that is, values that cannot be converted to the right type via the promotion rules) that Julia will complain about this?
Deleted Comment
But in general I agree. Fir me, Lisp languages, Haskell, and Python are more general purpose.
You mean Julia as language is not enough good for general purpose or it lacks ecosystem (frameworks, libraries) for general purpose use?
It's getting really tiring
At my old bioinformatics lab, Python literally wasted thousands and thousands of collective man-hours which would have been saved by Julia if it had existed at the time. Since a lot of these researchers couldn’t really program that well, they would write code that would literally take weeks to run. And then they would call me and I would rewrite it in C and it would then take an hour or two. Julia solves this problem.
The amount of unnecessary supercomputer time (and electricity) that our lab (and others) were wasting with Python was, honestly, disgusting.
This article hits on some of the issues that the community tends to see as advantages but that I think will prove limiting in the long run.
> Missing features like:
> Weak conventions about namespace pollution
> Never got around to making it easy to use local modules, outside of packages
> A type system that can’t be used to check correctness
These are some of my biggest gripes about Julia, especially the last two. To these I would add:
* Lack of support for formally defined interfaces.
* Lack of support for implementation inheritance.
Together with Julia's many strengths I think these design choices and community philosophy lead to a language that is very good for small scale and experimental work but will have major issues scaling to very complex systems development projects and will be ill-suited to mission critical applications.
In short I think Julia may be a great language for prototyping an object detection algorithm, but I wouldn't want to use it to develop the control system for a self-driving car.
Unfortunately this means that Julia probably isn't really going to solve the "2 language problem" because in most cases you're still going to need to rewrite your prototypes in a different language just like you would previously in going from, for example, a Matlab prototype to a C++ system in production.
Except the Module system, which feels unnecessarily arcane. I'm happy to be educated on why, but it seems to successfully combine the awkwardness of C-style #include with the mess of a free-form module system. The end result is a Frankenstein monster where technically everything is possible, everything could be included anywhere, there are no boundaries or even conventions. It makes for a frustrating experience for a newbie.
Say you have a package, and inside is a file called `xyz.jl`. You open the file and it defines a module called Xyz. But this tells you absolutely nothing about where in the package Xyz will appear. It could be included somewhere deep in the hierarchy, or it could be a submodule. It could be included multiple times in multiple places! That's bad design for sure, but the language places no boundaries on you. You open another file `abc.jl`, and see no modules at all, just a bunch of functions, which in turn call other functions that are defined God knows where. A julia file does not have to contain any information about where the symbols it's using come from, since it will be just pasted in verbatim to some location somewhere.
The whole module system feels like one big spaghetti of spooky action at a distance.
It's a shame too, because the rest of the language is very neat. Once one gets over the hurdle of the modules, it is possible to establish conventions to bring some sanity in there, but it's a hurdle that many people will probably not want to deal with.
It seems great to me that paths & source files are mostly irrelevant, you're free to re-organise without changing anything. And that `using Xyz` is always talking to the package manager. You can make sub-modules and say `using .Xyz`, but there's very little need to do so, and few packages do.
You can shoot yourself in the foot by including source twice, as you can by generating it with a macro, or simply copy-pasting wrong.
Deleted Comment
I do miss static typing.
I would question the claim it doesn't scale to production. I know people who have build hugely complex production systems in perl that are still running today 20 years later.
Further, I myself work on what we believe to be the largest closed source julia code base, in terms of number of contributors, number of packages and total size. (Its also pretty large in general, though i have yet to work out how it stacks up against DiffEq-verse). And I have seen thing go from research prototype into running in production. It works.
I am not going to deny though there are advantages to other languages. There are many trade-offs in the world
Sure, but there aren't many programmers who would want to maintain such a system.
I'm not saying static types don't have benefits as well, but it would also be very against the design goals as a Matlab/R competitor.
Inheritance would also directly clash and overlap with multiple dispatch, which is strictly more powerful.
It's funny; formerly I was a die-hard fan of static typing, but, lately, my opinions have become more nuanced. Something more along the lines of, "Yeah, I'd never want to just remove the type annotations from my Java code and then try to maintain the result, but some dynamic languages allow me to have a fraction as much code to maintain in the first place."
I'm also beginning to wonder if my feelings about dynamic languages have been unduly influenced by some particularly popular, and also particularly undisciplined, dynamic languages. JavaScript and PHP, for example.
In a language that supports classes I can have class B inherit from class A and automatically provide all of class A's functionality without adding a single extra line of code. I can extend class B's functionality by adding only code that is specific to it.
I don't see how to do that with multiple dispatch, at least the way it's implemented in Julia.
Also, is there always a trade-off between types and flexible meta-programming? Like, OCaml has meta-programming capabilities, but they make type-checking way harder, according to my PhD friend who's written extensively in Scheme and OCaml.
PackageCompilerX replaced PackageCompiler (and there's a PR open that will pull all the X work in soon).
The binaries produced bundle the whole Julia sysimage by default and they're quite big, but they are quite fast!
A more traditional static compilation approach is being tried with StaticCompiler.jl by tshort, but it's in early development.
The moment this ships, there will no longer be any reason to use any other garbage-collected language.
[1] https://github.com/JuliaLang/PackageCompiler.jl
https://github.com/JuliaLang/PackageCompiler.jl
Deleted Comment