Worth trying Cagra (Raft)/CuVS and Lucene-CuVS for the vector search. (https://github.com/SearchScale/lucene-cuvs)

Given that all of this tech is already being used inside Twitter, I feel it is unethical to discuss any non-public info (like discussions with cloud vendors on cost etc.) in public. The engineers have been kicked out, the tech still belongs to Twitter. This is not "decimation" of the engineering, just of engineers' egos. Someone else will do the job, the show will go on.

This is misleading. Likely, those 50 engineers were standing on the shoulders of the giants, i.e. those open source maintainers who build wonderful value for everyone out there.

I thought this is about multi-threaded processing of batch jobs.

If this project were to be governed by Apache Software Foundation, I'd have associated more credibility to this effort.