The inventor of the shell pipeline, Douglas McIlroy, always understood the equivalency between pipelines and coroutines; it was deliberate. See https://www.cs.dartmouth.edu/~doug/sieve/sieve.pdf It goes even deeper than it appears, too. The way pipes were originally implemented in the Unix kernel was when the pipe buffer was filled[1] by the writer the kernel continued execution directly in the blocked reader process without bouncing through the scheduler. Effectively, arguably literally, coroutines; one process call the write function and execution continues with a read call returning the data.

Interestingly, Solaris Doors operate the same way by design--no bouncing through the scheduler--unlike pipes today where long ago I think most Unix kernels moved away from direct execution switching to better support multiple readers, etc.

[1] Or even on the first write? I'd have to double-check the source again.

Bit annoying, but serviceable. Though there's nothing wrong with your approach either.

Allow me, too, to disagree. I think the right term is "function composition".

Instead of writing

  h(g(f(x)))

We can use whatever syntax we want for that, but I like Elm syntax which would look like:

  x |> f >> g >> h

You make an interesting point about debugging which is something I have also encountered in practice. There is an interesting tension here which I am unsure about how to best resolve.

In PRQL we use the pipelining approach by using the output of the last step as the implicit last argument of the next step. In M Lang (MS Power BI/Power Query), which is quite similar in many ways, they use second approach in that each step has to be named. This is very useful for debugging as you point out but also a lot more verbose and can be tedious. I like both but prefer the ergonomics of PRQL for interactive work.

Update: Actually, PRQL has a decent answer to this. Say you have a query like:

    from invoices
    filter total > 1_000
    derive invoice_age = @2025-04-23 - invoice_date
    filter invoice_age > 3months

    from invoices
    filter total > 1_000
    into tmp
    
    from tmp
    derive invoice_age = @2025-04-23 - invoice_date
    filter invoice_age > 3months

I believe the correct definition for this concept is the Thrush combinator[0]. In some ML-based languages[1], such as F#, the |> operator is defined[2] for same:

  [1..10] |> List.map (fun i -> i + 1)

0 - https://leanpub.com/combinators/read#leanpub-auto-the-thrush

1 - https://en.wikipedia.org/wiki/ML_(programming_language)

2 - https://fsharpforfunandprofit.com/posts/defining-functions/

3 - https://github.com/typelevel/mouse?tab=readme-ov-file

    data = File.readlines("haystack.txt")
    data = data.map(&:strip)
    data = data.grep(/needle/)
    data = data.map { |i| i.gsub('foo', 'bar') }
    data = data.map { |i| File.readlines(i).count }

Processes run in parallel, but they process the data in a strict sequential order: «grep» must produce a chunk of data before «sed» can proceed, and «sed» must produce another chunk of data before «xargs» can do its part. «xargs» in no way can ever pick up the output of «grep» and bypass the «sed» step. If the preceding step is busy crunching the data and is not producing the data, the subsequent step will be blocked (the process will fall asleep). So it is both, a pipeline and a chain.

It is actually a directed data flow graph.

Also, if you replace «haystack.txt» with a /dev/haystack, i.e.

  grep needle < /dev/haystack | sed 's/foo/bar/g' | xargs wc -l

Given File.readlines("haystack.txt"), the entire file must be resident in memory before .grep(/needle/) is performed, which may cause unnecessary utilization. Iirc, in frameworks like Polars, the collect() chain ending method tells the compiler that the previous methods will be performed as a stream and thus not require pulling the entirety into memory in order to perform an operation on a subset of the corpus.

I've only ever heard the term 'pipelining' in reference to GPUs, or as an abstract umbrella term for moving data around.

Other than that I think both styles are fine.

Java streams are the closest equivalent, both by the concurrent execution model, and syntactically. And yes, the Java debugger can show you the state of the intermediate streams.

the chaining really only works if your language is strongly typed and you are somewhat guaranteed that variables will be of expected type.

I do it in a similar way you mentioned

But with actually checked in code, the tradeoff in readability is pretty substantial

https://github.com/tc39/proposal-pipeline-operator

I'm very excited for it.

``` params.get("user") |> create_user |> notify_admin ```

Even more concise and it doesn't even require a special language feature, it's just regular syntax of the language ( |> is a method like .get(...) so you could even write `params.get("user").|>(create_user) if you wanted to)

This is usually the Thrush combinator[0], exists in other languages as well, and can be informally defined as:

  f(g(x)) = g(x) |> f

The problem is that method-chaining is common in several OO languages, including Ruby. This means the functions on an object return an object, which can then call other functions on itself. In contrast, the pipe operator calls a function, passing in what's on the left side of it as the first argument. In order to work properly, this means you'll need functions that take the data as the first argument and return the same shape to return, whether that's a list, a map, a string or a struct, etc.

When you add a pipe operator to an OO language where method-chaining is common, you'll start getting two different types of APIs and it ends up messier than if you'd just stuck with chaining method calls. I much prefer passing immutable data into a pipeline of functions as Elixir does it, but I'd pick method chaining over a mix of method chaining and pipelines.

Note also that it works well in Elixir because it was created at the same time as most of the standard library. That means that the standard library takes the relevant argument in the first position all the time. Very rarely do you need to pipe into the second argument (and you need a lambda or convenience function to make that work).

Is there any language with a single feature that gives the best of both worlds?

[1] https://tour.gleam.run/functions/pipelines/

(No disagreements with your post, just want to give credit where it's due. I'm also a big fan of the syntax)

Instead of:

```

fetch_data()

|> (fn

  {:ok, val, _meta} -> val

  :error -> "default value"

|> String.upcase()

```

Something like this:

```

fetch_data()

|>? {:ok, val, _meta} -> val

|>? :error -> "default value"

|> String.upcase()

```

That's sort of an argument for the existence of macros as a whole, you can't really do this as neatly in something like python (although I've tried) - I can see the downside of working in a codebase with hundreds of these kind of custom language features though.

Recently I started using Nushell, which feels very similar.

And for exceptions, why not solve it in the data model, and reify failures? Push it further downstream, let your pipeline's nodes handle "monadic" result values.

Point being, it's always a tradeoff, but you can usually lessen the pain more than you think.

And that's without mentioning that a lot of "pipelining" is pure sugar over the same code we're already writing.

Exception handing is only a problem in languages that use exceptions. Fortunately there are many modern alternatives in wide use that don't use exceptions.

I've encountered and used this pattern in Python, Ruby, Haskell, Rust, C#, and maybe some other languages. It often feels nice to write, but reading can easily become difficult -- especially in Haskell where obscure operators can contain a lot of magic.

Debugging them interactively can be equally problematic, depending on the tooling. I'd argue, it's commonly harder to debug a pipeline than the equivalent imperative code and, that in the best case it's equally hard.

Programming should be focused on the happy path. Much of the syntax in primitive languages concerning exceptions and other early returns is pure noise.

The old adage of not writing code so smart you can’t debug it applies here.

Pipelining runs contrary enough to standard imperative patterns. You don’t just need a new mindset to write code this way. You need to think differently about how you structure your code overall and you need different tools.

That’s not to say that doing things a different way isn’t great, but it does come with baggage that you need to be in a position to carry.

If you need to handle an unhappy path in a way that isn’t optimal for nested function calls then you shouldn’t be nesting your function calls. Pipelining doesn’t magically make things easier nor harder in that regard.

But if a particular sequence of function calls do suit nesting, then pipelining makes the code much more readable because you’re not mixing right-to-left syntax (function nests) with left-to-right syntax (ie you’re typical language syntax).

Rust chains everything because of this. It's often unpleasant (see: all the Rust GUI toolkits).

Yeah, I agree that this can be problem when you lean heavily into monadic handling (i.e. you have fallible operations and then pipe the error or null all the way through, losing the information of where it came from).

But that doesn't have much to do with the article: You have the same problem with non-pipelined functional code. (And in either case, I think that it's not that big of a problem in practice.)

> The author uses examples that function very differently.

Yeah, this is addressed in one of the later sections. Imo, having a unified word for such a convenience feature (no matter how it's implemented) is better than thinking of these features as completely separate.

Just before that statement, he says that it is an article/hot take about syntax. He acknowledges your point.

So I think when he says "semantics beat syntax every day of the week", that's him acknowledging that while he prefers certain syntax, it may not be the best for a given situation.

>Let me make it very clear: This is [not an] article it's a hot take about syntax. In practice, semantics beat syntax every day of the week. In other words, don’t take it too seriously.