Tour de France teams don’t care about the cost savings from cheaper sugar, they do care about the time and weight effects of drinking more water to absorb the glucose.

There are multiple kinds of carbohydrate and the differences can matter.

Gatorade’s ingredients list shows sucrose (glucose+fructose) and dextrose (glucose).

Maltodextrin is isotonic in a 6x stronger solution than fructose and glucose. This means you need to drink meaningfully less water during a race to digest maltodextrin. So do we want all maltodextrin? Nope! Fructose can be absorbed by a separate pathway, which is less efficient, but increases your total intake of carbohydrates into the bloodstream.

So you’re probably not going to see Tour de France teams giving their riders Gatorade. You want more maltodextrin and less glucose.

I very, very nearly migrated to a full Duckdb solution for customer-facing historical stock data. It would have been magical, and ridiculously, absurdly, ungodly fast. But the cloud costs ended up being close to a managed analytics solution, with significantly more moving parts (on our end). But I think thats just our use case, going forward I'd look at duckdb as an option for any large-scale datasets.

Using ECS/EKS containers reading from a segmented dataset in EFS is a really solid solution, you can get sub second performance over 6 billion rows / 10000 columns with proper management and reasonably restrictive queries.

Another option is to just deploy a couple huge EC2 instances that can fully fit the dataset. Costs here were about the same, but with a little more pain in server management. But the speed man, its just unbelievable.

Do any data scientists here use duckdb daily? Keen to hear your experiences and comparisons to other tools you used before it.

I love tools that make life simpler. I've been toying with the idea of storing 1TB of data in S3 and querying it using duckdb on an EC2. That's really old/boring infrastructure, but is hugely appealing to me, since it's so much simpler than than what I currently use.

Would love to hear of others' experiences with duckdb.