/s
/s
Not only is code not the best documentation, it's not documentation at all, outside of special cases where the entire code describes e.g. an algorithm and is encapsulated into one function.
It is a very important aspect of the readability, otherwise you will just sink in details when reading. But it is not easy to formally validate with tools, so naturally it is better to have both code and a picture and have discipline to keep them in sync.
Write the specification as code - make sure it's something that can actually be checked. Then you'll know whether it's actually any good.
The task at hand wasn’t easy at all, it required both:
1. Making numerous network requests (some endpoints support batching, so do that as much as possible) 2. The entire main loop needs to be as fast as reasonably possible
The only real way to satisfy both, given that serially executing the network activity would waste tons of time, was some fairly tricky concurrent logic. I ended up using a few queue structures to collect day before making batch network calls, and stitching together all of the results at the end (block on all work threads, collect, report, then start next iteration)
Every step along the way made sense, every addition of complexity felt justified. And the first person to come to me asking how the fuck it got so complicated is frustrated, so my knee jerk reaction is to get frustrated back saying well how else could we satisfy complex requirements with a complex solution.
We should always be challenging ourselves and others to keep solutions simple, but remember that the simplest solution may itself still be complex.
Design-wise:
The IO multiplexing — event loop(s), the completion callbacks go to the "Processing".
Processing — usually beneficial to model as a processing graph (tree/pipeline being specific examples of a graph without feedback loops). It's a clean, low overhead abstraction, easy to visualise (and thus discuss with others), easy to change structurally or in the "aspect oriented" way.
Examples in JVM world: Netty pipelines, AKKA actors, streams, and graphs, in C++ world ASIO & its executors sub-framework.
Queues/Threads is still a sufficiently low level to comfortably operate on.
And then maybe you could just customize and optimize your own mode for local use. Almost like mixing and matching different modules. It would be nice to have a model that only knows and does what you need it to