I want a good parallel language [video]

Munksgaard · 3 months ago

Interesting talk. He mentions Futhark a few times, but fails to point out that his ideal way of programming is almost 1:1 how it would be done in Futhark.

His example is:

  sequence
    .map(|x: T0| ...: T1)
    .scan(|a: T1, b: T1| ...: T1)
    .filter(|x: T1| ...: bool)
    .flat_map(|x: T1| ...: sequence<T2>)
    .collect()

It would be written in Futhark something like this:

  sequence
    |> map (\x -> ...)
    |> scan (\x y -> ...)
    |> filter (\x -> ...)
    |> map (\x -> ...) 
    |> flatten

Munksgaard · 3 months ago

Also, while not exactly the algorithm Raph is looking for, here is a bracket matching function (from Pareas, which he also mentions in the talk) in Futhark: https://github.com/Snektron/pareas/blob/master/src/compiler/...

I haven't studied it in depth, but it's pretty readable.

pythomancer · 3 months ago

(author here) check_brackets_bt is actually exactly the algorithm that Raph mentions

snthpy · 3 months ago

Very cool. I have seen Futhark mentioned before but I don't know anything about it.

The example you showed is very much how I think about PRQL pipelines. Syntax is slightly different but semantics are very similar.

At first I thought that PRQL doesn't have scan but actually loop fulfills the same function. I'm going to look more into comparing those.

fifilura · 3 months ago

SQL.

It is a joke, but an SQL engine can be massively parallel. You just don't know it, it just gives you what you want. And in many ways the operations resembles what you do for example in CUDA.

CUDA backend for DuckDB or Trino would be one of my go-to projects if i was laid off.

drivebyhooting · 3 months ago

My issue with SQL is lack of composability and difficulty of debugging intermediate results.

mamcx · 3 months ago

Yes, SQL is poor.

What could be good is relational + array model. I have some ideas on https://tablam.org, and building not just the language but the optimizer in tandem I think will be very nice.

zozbot234 · 3 months ago

You can use SQL CTE's and/or VIEW's as a composable abstraction over queries and inspect intermediate results. The language features are there.

kragen · 3 months ago

The standard things that someone should always say when someone brings up this problem is:

• Datalog is much, much better on these axes.

• Tutorial D is also better than SQL.

Too · 3 months ago

Check out https://prql-lang.org/

It solves all the warts of sql while still being true to its declarative execution. Trailing commas, from statement first and reads as a a composable pipeline, temporary variables for expressions, intuitive grouping.

asadm · 3 months ago

is it a language problem though? it's just lack of tooling.

gnulinux · 3 months ago

Even in this thread people underestimate how good e.g. DuckDB can be if you swallow its quirks. Yeah SQL has many problems, but with a slightly extended language with QoL features and seamless parallelism DuckDB is extremely productive if you want to crunch bunch of numbers in the order of minutes, hours etc (not real time).

Sometimes I have a problem, I just generate bunch of "possible solutions" with a constraint solver (e.g. Minizinc) which generates GBs of CSVs describing bunch of solutions, then let DuckDB analyze which ones are suitable, DuckDB is amazing.

taeric · 3 months ago

More generally, the key here is that the more magic you want in the execution of your code, the more declarative you want the code to be. And SQL is pretty much the poster child declarative language out there.

Term rewriting languages probably work better at this than I would expect? It is kind of sad how little experience with that sort of thing that I have built up. And I think I'm above a large percentage of developers out there.

dvrp · 3 months ago

If you want to work in data engineering for massive datasets (many petabytes) pls hit me up!

fifilura · 3 months ago

Sorry, wrong continent :)

ChadNauseam · 3 months ago

Raph and I also talked about this subject here: https://www.popovit.ch/interviews/raph-levien-simd The discussion covers things at a relatively basic level as we wanted it to be accessible to a wide audience. So we explain SIMD vs SIMT, predication, multiversioning, and some more.

Raph is a super nice guy and a pleasure to talk to. I'm glad we have people like him around!

v9v · 3 months ago

There were a few languages designed specifically for parallel computing spurred by DARPA's High Productivity Computing Systems project. While Fortress is dead, Chapel is still being developed.

jandrewrogers · 3 months ago

Those languages were not effective in practice. The kind of loop parallelism that most people focus on is the least interesting and effective kind outside of niche domains. The value was low.

Hardware architectures like Tera MTA were much more capable but almost no one could write effective code for them even though the language was vanilla C++ with a couple extra features. Then we learned how to write similar software architecture on standard CPUs. The same problem of people being bad at it remained.

The common thread in all of this is people. Humans as a group are terrible at reasoning about non-trivial parallelism. The tools almost don't matter. Reasoning effectively about parallelism involves manipulating a space that is quite evidently beyond most human cognitive abilities to reason about.

Parallelism was never about the language. Most people can't build the necessary mental model in any language.

kevindamm · 3 months ago

This was, I think, the greatest strength of MapReduce. If you could write a basic program you could understand the map, combine, shuffle and reduce operations. MR and Hadoop etc. would take care of recovering from operational failures like disk or network outages by idempotencies in the workings behind the scenes, and programmers could focus on how data was being transformed, joined, serialized, etc.

To your point, we also didn't need a new language to adopt this paradigm. A library and a running system were enough (though, semantically, it did offer unique language-like characteristics).

Sure, it's a bit antiquated now that we have more sophisticated iterations for the subdomains it was most commonly used for, but it hit a kind of sweet spot between parallelism utility and complexity of knowledge or reasoning required of its users.

jimbokun · 3 months ago

That's why programming languages are important for solving this problem.

The syntax and semantics should constrain the kinds of programs that are easy to write in the language to ones that the compiler can figure out how to run in parallel correctly and efficiently.

That's how you end up with something like Erlang or Elixir.

kragen · 3 months ago

Maybe we can find better abstractions. Software transactional memory seems like a promising candidate, for example. Sawzall/Dremel and SQL seem to also be capable of expressing some interesting things. And, as RoboToaster mentions, in VHDL and Verilog, people have successfully described parallel computations containing billions of concurrent processes, and even gotten them to work properly.

zokier · 3 months ago

iirc those were oriented more towards large HPC clusters rather than computation on single node?

Jtsummers · 3 months ago

Chapel, at least, aims for both. You can write loops that it will try to compile to use SIMD instructions, or even for the GPU: https://chapel-lang.org/docs/technotes/gpu.html

convolvatron · 3 months ago

the distinction matters less and less. Inside the GPU there is already plenty of locality to exploit (catches, schedulers, warps). nvlink is a switch memory access network, so that already gets you some fairly large machines with multiple kinds of locality.

throwing infiniband or IP on top is really structurally more of the same.

Chapel definitely can target a single GPU.

TJSomething · 3 months ago

It seems like there are two sides to this problem, both of which are hard and go hand in hand. There is the HCI problem of having abstractions are rich enough to handle problems like parsing and scheduling on the GPU. Then you need a sufficiently smart compiler problem of lowering these problems to the GPU. But of course, there's a limit to how smart a compiler can be, which loops back to your abstraction design.

Overall, it seems to be a really interesting problem!

dandanua · 3 months ago

I think a good parallel language will be the one that takes your code written with tasks and channels, understands its logic, rewrites and compiles it in the most efficient way. I don't feel that I have to write something harder than that as a pity human.

convolvatron · 3 months ago

mapping from channels to SIMD seems kind of intractable, its a kind of lifting that involves looking across the producers and the consumers.

going the other direction, making channel runtimes run SIMD, is trivial

abejfehr · 3 months ago

Bend comes to mind as an attempt at this: https://github.com/HigherOrderCO/Bend

Disclaimer: I did not watch the video yet

awaymazdacx5 · 3 months ago

Lower-level programming language, which is either object-oriented like python or after compilation a real-time system transposition would assemble the microarchitecture to an x86 chip.