func OrderedLoop[A, B any](in <-chan A, done chan<- B, n int, f func(a A))
Instead of: func OrderedLoop[A, B any](in <-chan A, done chan<- B, n int, f func(a A, canWrite <-chan struct{}))
Or is there a reason that "f" can't be wrapped in an anonymous function to handle the blocking logic?Typically function "f" does two things. 1. Performs calculations (this can be parallelized). 2. Writes results somewhere (this must happen sequentially and in the correct order).
Here's the typical OrderedLoop usage example from the article:
OrderedLoop(in, out, n, func(a A, canWrite <-chan struct{}) {
// [Do processing here]
// Everything above this line is executed concurrently,
// everything below it is executed sequentially and in order
<-canWrite
// [Write results somewhere]
})
Without the "canWrite" the entire body of the "f" will be executed either concurrently or sequentially. With it - we can have this dual behavior, similar to critical sections protected by mutexes.It's worth mentioning that OrderedLoop is a low-level primitive. User-level functions like OrderedMap, OrderedFilter and others do not need the "canWrite" channel to be exposed.
PS. I realize you present even better solution; still, first version seems like a thing nice enough to have in a toolbox
I've just made a small but important clarification to the article. While in many cases it's easier and even preferred to calculate all results, accumulate them somewhere, then sort; this article focuses on memory bound algorithms that support infinite streams and backpressure.
Create a bunch of sequentially numbered jobs that then update their output into postgres database. Then have N number of workers process the jobs. Something like GCP's CloudTasks is perfect for this because the "workers" are just GCP Cloud Functions, so you can have a near infinite number of them (limited by concurrent DB connections).
This approach also buys you durability of the queue for free (ie: what happens when you need to stop your golang process mid queue?).
Then it is just a query:
select * from finished_jobs order by job_num;
Don’t know what’s the reason: my bad prompting or these models being tuned to work with photos/illustrations only