Great idea, poor naming. If you’re aiming for a standard of sorts, tying it to a specific software by reusing its name feels counter productive.
“Ducklake DuckDB extension” really rolls off the tongue /s.
Readit Newsiampims commented on DuckLake is an integrated data lake and catalog format ducklake.select/... · Posted by u/kermatt
iampims commented on OpenAI: Scaling PostgreSQL to the Next Level pixelstech.net/article/17... · Posted by u/thunderbong
They are one of the most well capitalized company/startup/foundation/non-profit in the planet and just spent 6,5 billion to hire a designer.
They should be using the best technical and cheapest solution, and they owe it to their investors. At their scale they will never be able to use anything else than a cloud solution.
They could solve these issues at the number of users they report, for a monthly bill below 25 million dollars.
"6,311 database instances running the PostgreSQL-compatible and MySQL-compatible editions of Amazon Aurora processed more than 376 billion transactions, stored 2,978 terabytes of data, and transferred 913 terabytes of data" - https://aws.amazon.com/blogs/aws/how-aws-powered-prime-day-2...
but that'd be real money, not the Monopoly money they used to buy Ive/Windsurf...
iampims commented on OpenAI: Scaling PostgreSQL to the Next Level pixelstech.net/article/17... · Posted by u/thunderbong
Argh. Shard the damn database already.
Why are they not sharing by user/org yet? It is so simple and would fix the primary issue they are running into.
All these work arounds they go through to avoid a straight forward fix.
Not sure I would qualify sharding a DB that get 1M qps as straight forward.
I agree with you that it seems that an org would be a natural sharding key, but we know that at this scale, nothing really is ever straight forward, especially when it's your first rodeo.
iampims commented on Strobelight: A profiling service built on open source technology engineering.fb.com/2025/0... · Posted by u/birdculture
I just wish Meta would open source Scuba.
Sampling is lossy though
lossy and simpler.
IME, I've found sampling simpler to reason about, and with the sampling rate part of the message, deriving metrics from logs works pretty well.
The example in the article is a little contrived. Healthchecks often originate from multiple hosts and/or logs contain the remote address+port, leading to each log message being effectively unique. So sure, one could parse the remote address into remote_address=192.168.12.23 remote_port=64780 and then decide to drop the port in the aggregation, but is it worth the squeeze?
Or sampling :)
iampims commented on Infinite Git repos on Cloudflare workers gitlip.com/blog/infinite-... · Posted by u/plesiv
Some serious engineering here. Kudos!
iampims commented on Alert Evaluations: Incremental Merges in ClickHouse highlight.io/blog/alert-e... · Posted by u/vadman97
At a certain scale, exact computations (p50 for instance) become impractical. I’ve had great luck switching to approximate calculations with guaranteed error bounds.
An approachable paper on the topic is "Effective Computation of Biased Quantiles over Data Streams" http://dimacs.rutgers.edu/%7Egraham/pubs/papers/bquant-icde....
I have been meaning to ask the observability experts this question:
Why not dump all metrics , events and logs into Clickhouse ? and purge data as necessary? For small to medium sized businesses/solution ecosystem, will this be be enough ?
It'll work. Clickhouse has even experimental support for storing prometheus metrics natively. A big missing piece is alerting.
If ICANN-approved root.zone and ICANN-approved registries are the only options.
As an experiment I created own registry, not shared with anyone. For many years I have run own root server, i.e., I serve own custom root.zone to all computers I own. I have a search experiment that uses a custom TLD that embeds a well-known classification system. The TLD portion of the domainname can catgorise any product or service on Earth.
ICANN TLDs are vague, ambiguous, sometimes even deceptive.