Readit NewsI’d like to see a 1 trillion row challenge.
One exists https://blog.coiled.io/blog/1trc.html
Cloud network egress costs is one of the reasons why increasingly infrastructure vendors likely to incur high data transfer costs (database / data warehouse services, message buses, machine learning, etc..) are moving to introduce BYOC or "Bring Your Own Cloud" deployment model (Data Plane runs in the customer VPC - original Databricks deployment model), instead of only pure SaaS (Data Plane runs in the vendor VPC - e.g. Snowflake) in their cloud offerings.
Databricks now has fully hosted too, but Snowflake still did not go the other way to introduce BYOC. Newer entrants, however - RedPanda, AnyScale, ClickHouse... - are going the way of providing both BYOC and pure SaaS options in their cloud. There are many other reasons to prefer BYOC, legal, security, data privacy, but network egress costs is one of them.
>The recent efforts to move the JSON type to production-ready status will be highly applicable to our logging use case. This feature is currently being rearchitected, with the development of the Variant type providing the foundation for a more robust implementation. When ready, we expect this to replace our map with more strongly typed (i.e. not uniformly typed) metadata structures that are also possibly hierarchical.
Very happy to see ClickHouse dogfooding itself for storing logs - hope this will help to hasten the work on improving the the json type more suitable to dynamic documents.
Variant datatype is already available as experimental in 24.1, Dynamic datatype is WIP (PR almost ready), and JSON datatype is next up. Check out the latest comment on that issue with how the Dynamic datatype will work: https://github.com/ClickHouse/ClickHouse/issues/54864#issuec...