The architecture is sound - typically called ELT these days. Dump contents of upstream straight into a database and apply stateless and deterministic operations to achieve the final result tables.

SQL server is where this breaks though. You'll get yelled by DBAs for bad db practices like storing wide text fields without casting them to varchar(32) or varchar(12), primary keys on strings or no indexes at all, and most importantly taking majority of storage on the db host for tbese raw dumps. SQL Server and any traditional database scales by adding machines, so you end up paying compute costs for your storage.

If you use a shared disk system with decoupled compute scaling from storage, then your system is the way to go. Ideally these days dump your files into a file storage like s3 and slap a table abstraction over it with some catalog and now you have 100x less storage costs and about 5-10x increased compute power with things like duckdb. Happy data engineering!

In post Czechoslovakia if you moved out of your hometown, you lost valuable networks. Healthcare and childcare were always connection based - you needed to know the local doctor, teacher, homebuilder - to get connected with specialists, skip the queues, get preferred treatment, faster processing of documents, etc. Once you part with your network, you start to build new one from scratch which takes many years. Unlike in US which is more market based - as long as you have the money you can recreate similar lifestyle elsewhere in the country.

Second, there was never really a need to rush to move to another location offering better opportunities. As consequence of the 1990s policies the local capital vanished into the hands of western entities and with it the opportunities worth moving for. The post 2000 capital which moved to the region just found spots with cheap labor to build new factories or logistic centers to keep the German powerhouse running. With an unfair advantage of cheap eastern energy, cheap eastern workers across the border and cheap euro currency as a result of sharing it with the unproductive european south.

What do you need non-columnar layout for? Do you expect thousands of concurrent single row writes at a time?

If you use embedded duckdb on the client, unless the person goes crazy clicking their mouse at 60 clicks/s, duckdb should handle it fine.

If you run it on the backend and expect concurrent writes, you can buffer the writes in concatenated arrow tables, one per minibatch, and merge to duckdb every say 10 seconds. You'd just need to query both the historical duckdb and realtime arrow tables separately and combine results later.

I agree that having a native support for this so called Lambda architecture would be cool to have natively in duckdb. Especially when drinking fast moving data from a firehose.

Two people already form a 'company' and the hard part of convincing somebody else of your idea has already been accomplished. You can argue growing from 1 person to 2 is the most critical part of company growth. With 2+ cofounders this critical part is already figured out making it much less risky proposition than a single founder who hasn't convinced anybody else to join them yet.

How does one output properly csv quoted rows? It seems thr csv flag works only for parsing inputs.