I've been working on something like this on and off for the past 4 years or so, although with something more like generators than streams.
I think it's a very, very promising idea (I admit to heavily being biased towards anything APL-influenced) although surprisingly difficult to get right. Gilad Bracha is obviously way smarter than me so I'm definitely curious where he goes with this.
One additional idea that I keep trying to make work is integrating variations of (constraint) logic programming and treating solutions to a predicate as a generator or stream that operations can be lifted to rank-polymorphically. As a simple example a range function could be defined and used like (imaginary illustrative syntax)
range(n,m,x) :- n <= x, x <= m
primesUpto(n) = range(2,n,r), # create a generator containing all solutions wrt r
mask(not(contains(outerProduct(*, r, r), r)), r) # as in the video
I've never really gotten this to work nicely and it always feels like there's a sort of separation between the logic world and the array world. However this feels incredibly powerful, especially as part of some sort of database, so I keep returning to it even though I'm not really sure it goes anywhere super useful in the end.
As a long time user of q/kdb, I can confirm that working with array-based languages definitely changes how you view data and code. It makes you a better coder in other languages as well.
Random thought: cool triptych behind the main speaker. Anyone know what it is?
Can I ask how you got into using q/kdb? I've been using it in my current work, partly because it's well-suited to the table-oriented work I'm doing, and partly just because I wanted to learn it. But have no idea where to find opportunities using it.
I was working at a large bank back in 2010 - as a quant trader. The firm acquired an enterprise license so that came with training and general openness to using this.
More often than not, kdb is chosen as a basic time-series storage and retrieval. Which is a bit sad because thats really a small part of what it does well. Other firms use KDB in a more holistic manner, for building a distributed services system for example. (look up Torq from AquaQ)
Where to find opportunities? Probably going to a job board and searching for KDB is a good place to start - most opportunities will be in trading or trading-tech related roles.
Quick summary: Gilad Bracha introduces a new programming language called ShapeRank. It seems to be based on APL but introduces the concept of streams which are [vectors | tensors | arrays] of unbounded length.
The ideas of APL and its successors, the array programming languages, were two generations ahead of their time. These languages are based on the notion that everything is a tensor, and all operations are rank-polymorphic: they extend automatically to tensors of any rank. These ideas are perfectly suited to an era of machine learning, large scale data, GPUs and other accelerators. Building on recent academic research, we are building ShapeRank, a new statically typed, purely functional language for industrial use, that extends rank-polymorphism to streams. We’ll introduce the key ideas and show how they are realized in ShapeRank.
Easier means - less effort in learning coming from someone who knows mainstream languages like Java.
The idea is not only limited to APL. I don't like crafting for loops or maintain indexes. Fortran has something similar. With Matlab many operators operate in an intuitive way on vectors and matrices. It breaks down quite quickly if you try to do something more complex though. This somewhat extends to Julia. In Ruby also you can have .map or .each.
Personally I think array programming languages are the future and one of the most popular programming languages in science and engineering is Matlab, and it is to some extent an array based programming language [1].
I am surprised that the author (and reviewers of the paper) has missed to perform proper literature review, for example it missed other recent and promising works on functional array programming languages namely Single Assignment C (SAC) and Futhark [2],[3].
ShapeRank also seems to take vector algebra "tensor" concept to the extreme and to be honest it's better to based on "versor" since geometric algebra is probably the future of computer algebra [4].
Last but not least and probably the most controversial is that why create another standalone array language from scratch? It will be better to make a seamless DSL based on general purpose language like D language and you do not have to re-invent most of the libraries (and C library support in D is second to none). Arguably the most successful recent effort on array based scientific programming language is Julia and it is still very much dependent on some Fortran based libraries for speed. While with D you can go "turtle all the way down" and still meet the speed requirements that are needed in scientific computing [5].
I'd love to drop Python and go all in with a true array language (i.e. not Matlab), but you only really have 4 options:
Dyalog APL, J, Kdb+, Shakti.
All of those are closed source and expensive (Dyalog is fairly affordable, but still a paid product) with the exception of J. J is a cool language, but isn't quite my cup of tea.
So if one of the new projects ever picked up steam and got a decent sized community with hooks into all the same numeric libraries as Numpy, and some decent charting libraries...then we would have something nice.
> Arguably the most successful recent effort on array based scientific programming language is Julia and it is still very much dependent on some Fortran based libraries for speed.
That is because some libraries (e.g. BLAS implementations) have many developer-years of careful optimizations. These are not trivial algorithms to implement. Implementing them in Julia, nonetheless, has been a long-standing goal of the Julia community.
Array programming is the future and always will be (since 1958). Same as LISP.
APL and LISP both had the problem that they went straight to the (extreme) logical conclusions of a paradigm. Which is means you have to put an effort to actually derive the benefits be can’t just “wing it till you understand it” like you can in most other languages. Notably, C++ has left that club with C++14 (maybe even 11), but popular languages at the time of their popularity have always been there, and likely always will be.
> ShapeRank also seems to take vector algebra "tensor" concept to the extreme and to be honest it's better to based on "versor" since geometric algebra is probably the future of computer algebra [4].
I'm curious why you say that? Quantum computing or something? I was very interested in geometric algebra several years ago, but have never found need for it. Seems like it helps simplify problems that are already simple.
In my undergrad at UNSW I did a "baby's first interpreter" project on a language with a similar concept - function evaluation is defined in terms of unbounded-length arrays, being able to cache function evaluation for performance etc.
Looking back it was pretty cool and helped form a more abstracted view of programming beyond the low-level
Fortran has had operations on arrays and array sections since the 1990 standard. I wonder why it is hardly mentioned in this discussion of array programming languages.
I don't know if we've solved parallelizable computation in Rx, but it doesn't seem like it should be too much of an abstraction on top. I didn't get the sense that the speaker was aware of Reactive-Streams, but hopefully they're aware of the existing effort!
I think array languages could be a great alternative to java/c++ for a lot of operations. For anyone wanting to try another array based language, I've been working on jq an open source q language implementation:
https://github.com/timestored/jq
It's implemented in java, which will allow loading in a wide range of libraries. An older version can be tried online here (1 min load time):
http://www.timestored.com/jq/online
I was aware of the json parser and it did put me off the naming. Unfortunately jq is the most obvious name. For example there is jruby/jython. Longer term when a stable version is reached I would like to change to it's own name as eventually the goal is to extend the current q language. jq++, jq# :)
jq is also a jQuery abbreviation that is recognized as such by Google and other search engines, e.g. "jq on" will have jQuery.on() spec as the first hit.
What is a tensor? I remember from physics course that a tensor is some multidimensional matrix that maps vectors/matrices to other vectors/matrices, and vary from point to point in space. Looks like all arrays are called tensors now? Is it for coolness reasons?
It is my (probably incorrect) understanding that tensors are the generalized form of scalars, vectors, matrices, etc. A scalar being a 0 dimensional tensor, a vector being a 1 dimensional tensor, a matrix being a 2 dimensional tensor, and so on.
Readit News
I think it's a very, very promising idea (I admit to heavily being biased towards anything APL-influenced) although surprisingly difficult to get right. Gilad Bracha is obviously way smarter than me so I'm definitely curious where he goes with this.
One additional idea that I keep trying to make work is integrating variations of (constraint) logic programming and treating solutions to a predicate as a generator or stream that operations can be lifted to rank-polymorphically. As a simple example a range function could be defined and used like (imaginary illustrative syntax)
I've never really gotten this to work nicely and it always feels like there's a sort of separation between the logic world and the array world. However this feels incredibly powerful, especially as part of some sort of database, so I keep returning to it even though I'm not really sure it goes anywhere super useful in the end.It's a bit special-cased (afaict), but it sounds like it makes at least a solid step in this sort of direction.
Random thought: cool triptych behind the main speaker. Anyone know what it is?
EDIT: A search for "famous triptychs" revealed: https://en.wikipedia.org/wiki/The_Garden_of_Earthly_Delights
More often than not, kdb is chosen as a basic time-series storage and retrieval. Which is a bit sad because thats really a small part of what it does well. Other firms use KDB in a more holistic manner, for building a distributed services system for example. (look up Torq from AquaQ)
Where to find opportunities? Probably going to a job board and searching for KDB is a good place to start - most opportunities will be in trading or trading-tech related roles.
The ideas of APL and its successors, the array programming languages, were two generations ahead of their time. These languages are based on the notion that everything is a tensor, and all operations are rank-polymorphic: they extend automatically to tensors of any rank. These ideas are perfectly suited to an era of machine learning, large scale data, GPUs and other accelerators. Building on recent academic research, we are building ShapeRank, a new statically typed, purely functional language for industrial use, that extends rank-polymorphism to streams. We’ll introduce the key ideas and show how they are realized in ShapeRank.
https://2020.splashcon.org/details/splash-2020-rebase/26/A-R...
Easier means - less effort in learning coming from someone who knows mainstream languages like Java.
The idea is not only limited to APL. I don't like crafting for loops or maintain indexes. Fortran has something similar. With Matlab many operators operate in an intuitive way on vectors and matrices. It breaks down quite quickly if you try to do something more complex though. This somewhat extends to Julia. In Ruby also you can have .map or .each.
Julia:
I am surprised that the author (and reviewers of the paper) has missed to perform proper literature review, for example it missed other recent and promising works on functional array programming languages namely Single Assignment C (SAC) and Futhark [2],[3].
ShapeRank also seems to take vector algebra "tensor" concept to the extreme and to be honest it's better to based on "versor" since geometric algebra is probably the future of computer algebra [4].
Last but not least and probably the most controversial is that why create another standalone array language from scratch? It will be better to make a seamless DSL based on general purpose language like D language and you do not have to re-invent most of the libraries (and C library support in D is second to none). Arguably the most successful recent effort on array based scientific programming language is Julia and it is still very much dependent on some Fortran based libraries for speed. While with D you can go "turtle all the way down" and still meet the speed requirements that are needed in scientific computing [5].
[1]https://en.m.wikipedia.org/wiki/Array_programming
[2]http://www.sac-home.org/doku.php
[3]https://futhark-lang.org/
[4]https://en.m.wikipedia.org/wiki/Comparison_of_vector_algebra...
[5]http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/...
Dyalog APL, J, Kdb+, Shakti.
All of those are closed source and expensive (Dyalog is fairly affordable, but still a paid product) with the exception of J. J is a cool language, but isn't quite my cup of tea.
So if one of the new projects ever picked up steam and got a decent sized community with hooks into all the same numeric libraries as Numpy, and some decent charting libraries...then we would have something nice.
That is because some libraries (e.g. BLAS implementations) have many developer-years of careful optimizations. These are not trivial algorithms to implement. Implementing them in Julia, nonetheless, has been a long-standing goal of the Julia community.
APL and LISP both had the problem that they went straight to the (extreme) logical conclusions of a paradigm. Which is means you have to put an effort to actually derive the benefits be can’t just “wing it till you understand it” like you can in most other languages. Notably, C++ has left that club with C++14 (maybe even 11), but popular languages at the time of their popularity have always been there, and likely always will be.
I'm curious why you say that? Quantum computing or something? I was very interested in geometric algebra several years ago, but have never found need for it. Seems like it helps simplify problems that are already simple.
Looking back it was pretty cool and helped form a more abstracted view of programming beyond the low-level
edit:found some papers https://cartesianprogramming.files.wordpress.com/2020/07/sem...http://www.cse.unsw.edu.au/~plaice/archive/JAP/U-CSE-201306....
Another interesting language in this vein is Dex. The authors are creators of Jax and Pytorch, and they have a lot of interesting ideas.
I don't know if we've solved parallelizable computation in Rx, but it doesn't seem like it should be too much of an abstraction on top. I didn't get the sense that the speaker was aware of Reactive-Streams, but hopefully they're aware of the existing effort!
It's implemented in java, which will allow loading in a wide range of libraries. An older version can be tried online here (1 min load time): http://www.timestored.com/jq/online
https://stedolan.github.io/jq/
Any suggestions?