Hey HN! Roughly 4 years ago I started building a lazy functional iteration library for JS/TS. I had a few goals for the library:
- Supporting sync, sequential async, and concurrent async iteration
- Limiting it to a small number of orthogonal concepts that compose beautifully to solve problems
- Making it fully tree-shakeable
I built it for myself and have (mostly) been its only user as I refined it. I've used it in lots of personal projects and really enjoyed it.
I recently decided it would be nice to spread that enjoyment so I created a documentation website complete with a playground where you can try out the library.
I hope you enjoy using it as much as I do! Looking forward to hearing your thoughts :)
Split the array of elements to process. Set up a promise.all against a function that handles sync/async function calls against each chunk and accumulates an array of results. This isn't the end of the world for any of the practical use cases I can think of. The efficiency loss and memory consumption is miniscule even across hundreds of thousands of records. This is as concurrent as anything else - the event loop is still a mandatory part of the processing loop. You can even defer every 1000 records to the event loop. Javascript is syncronous so there is no concurrency without threading. I can't think of an IO or non-IO function that would create the issues described of not yielding. We used to have chunks of 1000000+ records that would freeze the event loop. We solved this with structured yields to the event loop to allow other processes to continue without dropping TCP packets etc.
If the results are objects, you're not even bloating memory too much. 800KB for 100,000 records seems fine for any practical use case.
I like the idea of an async generator to prevent over-pre-calculating the result set before it's needed but I couldn't justify pivoting to a set of new functions called pipe, map, etc that require a few too many logical leaps for your typical ECMAScript developers to understand.
What's the win here? It must be for some abuse of Javascript that should be ported to WASM. Or some other use-case that I've somehow not seen yet.
It's a combination for writing the iteration code in a functional style, which some people like/prefer, while retaining certain performance characteristics that you'd get from writing more imperative code.
For example, the "naive" approach to functional programming + concurrency results in some unintentional bottlenecks (see the concurrent iteration example on the home page).
And the third examples are much easier to maintain with a simple function I drafted up for sake of argument `processArrayWithStages(array: T[], limit: number, stages: Array<(item: T) => Promise<T>>)`
In my experience, unless you're doing math in promises which I would recommend shifting to WASM, you're not going to feel the overhead of these intermediary arrays at scales that you shouldn't be in WASM already (meaning over several millions of entries).
The amazing people and teams working on V8 have done well to optimize these flows in the past four years because they are extremely common in popular applications. Strings aren't even copied in the sense you might think, you have a pointer to a chunk that can be extended and overwritten which incurs a penalty but saves tons of RAM.
processArrayWithStages: https://gist.github.com/DevBrent/0ee8d6bbd0517223ac1f95d952b...
> should be ported to WASM
valid.
One function I've written that I frequently use is a generic iterate <T> function which (in JS/TS land) allows you to loop over a T[], Array<T>, ArrayLike<T>, Iterable<T>, AsyncIterable<T>, including generators and async generators.
It is just easier to always be able to write: "for await (const item of iterate(iterable))" in most places, without worrying about the type of the item I am looping over.
I like the reducers and collectors in your library! Going to try it out.
Also, something new I discovered is the Array.fromAsync() method in JS, which is like Array.from() but for async values. I don't think it is available in all browsers/runtimes yet though.
I actually have a TODO for what you're describing haha
https://github.com/TomerAberbach/lfi/blob/69cdca0b2ee2bd078f...
I like Lfi better, great documentation!
Here is the example I use to compare: https://stackblitz.com/edit/stackblitz-starters-ia9ujg6m?fil...
Still, I love seeing innovation in this field as much as anyone! And it's true that I've been dragging my feet on building concurrency support for iter-tools...
That said, iter-tools still has some killer APIs like peekerate that I would find it quite hard to live without, and for iter-tools 8 I have some major tricks planned like making the sync/async divide go away completely.
Looks good (I understand all the word and even the whole sentence).
But what is it good for? ("What's in it for me?")
What are the use cases? How is it difference / how does it improve on / when should I use it instead of / ... lodash, ramda, functional.js, RxJs, etc.?
I think the examples on the home page and in the "getting started" answer these questions to be honest. Do you feel it's unclear or something is missing?
> How is it difference / how does it improve on / when should I use it instead of / ... lodash, ramda, functional.js, RxJs, etc.?
I think some of that info is implicitly there in the docs, but I think this is good feedback. I should probably have a page comparing lfi vs other libraries.
To answer here though, for most of the libraries you mentioned, they don't provide support for _concurrent_ async iteration (and some of them don't support even sequential async iteration).
RxJS is the exception there I think. Although, I think lfi has a bit of a simpler API (that also matches the sync versions of the APIs) that's primarily designed for iteration rather than generalized observables. Plus, I don't _think_ RxJS has the same performance characteristics around tree-shakeability.
- Some people like/prefer writing iteration in a functional style, which this library enables
- It's pretty easy to create unintentional async/concurrency bottlenecks with the simple `Promise.all` approach (see the home page example)
Maybe an in memory cache hit---that you explicitly code---causes a sync resolve to happen, woohoo!
But if it goes to IO and you are waiting for the event loop to schedule you back in you could be in for a shock even if the IO op is tiny. Especially doing it alot.
Only matters probably on a high CPU usage server which hopefully you avoid by scaling up on cpu and using all the cores.
On the other hand... you are being cooperative :)
But tight loop IO is an antipattern ... as is doing a lot---millions---of IO ops from node (per request or job) that you need to worry about this (use a different language for your DB server!).
I think it just has completely different goals. Read through the home page examples and getting started and you'll see it's pretty different.
Any thoughts on what Lfi does better/different than IxJS?
[0] https://github.com/ReactiveX/IxJS
As far as I can tell, it has some similar ideas (and seems pretty nice!), but I noticed the following differences:
- I don't _think_ it supports "concurrent" iteration [1]
- It doesn't have a "reducer" concept with composition and what not [2]
- It doesn't have a concept of "optionals" [3]
[1] https://lfi.dev/docs/concepts/concurrent-iterable
[2] https://lfi.dev/docs/concepts/reducer
[3] https://lfi.dev/docs/concepts/optional
I believe what you are calling a "concurrent" iteration is one of the use cases for Observables, so would be on the RxJS side (the "parent" project).
> It doesn't have a "reducer" concept with composition and what not
Interesting. I'll read further on what you are doing there. Seems possibly similar to Lenses/Prisms in FP, with different rules.
> It doesn't have a concept of "optionals"
Good point. So far I've not found a JS approach to optionals I'm entirely happy with. I've had some suggest I should give Effect [0] a deeper look, but given I'm mostly happy with RxJS the bulk of Effect doesn't appeal to me. The dual of an Option being an iterator with 0 or 1 values is something I've seen before and something to keep in mind.
[0] https://effect.website/docs/data-types/option/