Readit News logoReadit News
hubertzhang commented on EloqKV, a distributed database with Redis compatible API (GPLv2 and AGPLv3)   github.com/eloqdata/eloqk... · Posted by u/cloudsql
hodr_super · 5 days ago
I built a SaaS app with per-tenant caches. Initially I used Redis but ran into scale-up issues, so I tried DragonflyDB. It works well in general, but my Lua script use case isn’t supported by default.

The use case is straightforward: each tenant has cached objects like: `cache:{tenant_id}:{object_id} → cached JSON/doc`

I also maintain a tag index to find all object IDs with a given tag: `tag:{tenant_id}:{tag} → set of object_ids (tag example: “pricing”, “profile”)`

When a tag changes (say “pricing”), I use a single Lua script to look up all object IDs in the tag set and then delete their cache entries in one atomic operation.

hubertzhang · 5 days ago
That use case aligns perfectly with EloqKV’s capabilities. In pure cache mode, batching multi-key deletions within Lua scripts can significantly reduce latency by minimizing client–server round trips.
hubertzhang commented on EloqKV, a distributed database with Redis compatible API (GPLv2 and AGPLv3)   github.com/eloqdata/eloqk... · Posted by u/cloudsql
hodr_super · 5 days ago
Distributed Lua is cool. Is your implementation similar to DragonflyDB, which doesn’t allow handling undeclared keys in Lua? For example, if I want to generate a new key dynamically inside a script like: `local queue_key = "queue:user:" .. uid` How does your system handle such cases?
hubertzhang · 5 days ago
Yes, Lua in EloqKV has no such limitations. You can freely read data, generate new keys, and even query those keys within Lua scripts. Underneath, EloqKV’s transaction layer is powered by our data substrate, which provides full ACID guarantees. FYI https://www.eloqdata.com/blog/2025/07/14/technology

Could you share a bit more about your specific use case? That will help me explain how EloqKV can best support it.

hubertzhang commented on EloqKV, a distributed database with Redis compatible API (GPLv2 and AGPLv3)   github.com/eloqdata/eloqk... · Posted by u/cloudsql
hodr_super · 5 days ago
Isn’t Redis already a distributed database with Redis Cluster?
hubertzhang · 5 days ago
I’m Hubert, CTO of EloqData. That’s a great question.

Redis Cluster is often thought of as a distributed database, but in reality it’s not truly distributed. It relies on a smart client to route queries to the correct shard—similar to how mongos works in MongoDB. This design means Redis Cluster cannot perform distributed transactions, and developers often need to use hashtags to manually place related data on the same shard.

EloqKV takes a different approach. It’s a natively distributed database with direct interconnects between nodes. You can connect to any node with a standard Redis client, and still read or write data that physically resides on other nodes. This architecture enables true distributed transactions across shards, fully supporting MULTI/EXEC and Lua scripts without special client logic or manual sharding workarounds.

hubertzhang commented on EloqKV, a distributed database with Redis compatible API (GPLv2 and AGPLv3)   github.com/eloqdata/eloqk... · Posted by u/cloudsql
esafak · 6 days ago
A person who wants transactions and a relational layer isn't going to use KVRocks, but sure, if you only care about the key-value part. I gave tikv/tidb as an example; there are others.

Writing your own Redis-like interface is trivial, so tidis et al don't matter to me. Even with Redis you should write an interface so you can swap it out.

hubertzhang · 5 days ago
Some Redis variants focus on persistence—KVrocks, for example, or MemoryDB, which emphasizes durability through redo logs to minimize data loss. However, they are not truly transactional, since they lack fundamental rollback semantics and distributed transaction.

EloqKV, by contrast, is fully transactional. It supports distributed Lua, MULTI/EXEC, and even SQL-style BEGIN/COMMIT/ROLLBACK syntax. This means you get the transactional guarantees of a database with Redis-level read performance. Writes are slightly slower since EloqKV ensures durability, but in return you gain full ACID safety. Most importantly, you no longer need to worry about cache coherence issues between a Redis cache and a separate SQL database—EloqKV unifies them into a single, reliable system.

hubertzhang commented on Neki – Sharded Postgres by the team behind Vitess   planetscale.com/blog/anno... · Posted by u/thdxr
hubertzhang · 13 days ago
Does Neki still need sharding key in query, just like Citus?
hubertzhang commented on Ursa: A leaderless, object storage–based alternative to Kafka   streamnative.io/products/... · Posted by u/netpaladinx
hubertzhang · 24 days ago
I’m the co-founder of EloqData. I recently gave a talk on EloqDoc and Pulsar integration at Data Streaming Summit 2025. It was a great event, and Ursa was definitely one of the hottest topics discussed.

I believe object storage is shaping the future architecture of cloud databases. The first big shift happened in the data warehouse space, where we saw the move from Teradata and Greenplum to Snowflake accelerate around 2016. Snowflake’s adoption of object storage as its primary storage layer not only reduced costs but also unlocked true elasticity.

Now, we’re seeing a similar trend in the streaming world. If I recall correctly, Ursa was the first to GA an object-storage–based streaming service, with Kafka(WarpStream) and AutoMQ following afterward.

I also believe the next generation of OLTP databases will use object storage as their main storage layer. This blog post shares some insights into this trend and the unique challenges of implementing object storage correctly for OLTP workloads, which are much more latency-sensitive.

https://www.eloqdata.com/blog/2025/07/16/data-substrate-bene...

hubertzhang commented on Everyone is asking the wrong questions about TikTok   sfconservancy.org/blog/20... · Posted by u/pabs3
hubertzhang · 7 months ago
Is Meta/X/Google open source? Is Gamil/Outlook open source? All of them gather user's data. They are all "free" software

Dead Comment

hubertzhang commented on Show HN: EloqKV – Scalable distributed ACID key-value database with Redis API   eloqdata.com/blog/2024/08... · Posted by u/hubertzhang
AtlasBarfed · a year ago
Yet another distributed database announcement with nary a mention of CAP tradeoffs. Then again, no sales pitch starts with the limitations. Also, one man's ACID isn't another person's. At least they aren't advertising SQL joins with handwaves to the distributed complexities.

It superficially sounds like a series of server processes fronting actual database servers, which sounds like another layer of partition vulnerability and points of failure. But I also had similar high level concerns about the complexity of FoundationDB, people seem satisfied as to the validity of that architecture.

I fail to see how one would scale underlying resources if the persistence is done by various storage systems, and you'd be subject to limitations of those persistence engines. That sounds like a suspect claim, like the "Data Substrate" is a big distributed cache that scales in front of delayed persistence to an actual database. Again, sounds like oodles of opportunities for failure.

"Data substrate draws inspirations from the canonical design of single-node relational database management systems (RDBMS)." Look, I don't think a good distributed database starts from a CA system and bolts on partition tolerance. I get you can get far with big nodes, fast networks, and sharding but ... I mean, they talk about B+ trees but a foundational aspect of Cassandra and other distributed databases is that B+ trees don't scale past a certain point, because there exist trees so deep/tall that even modern hardware chokes on it, and updating the B+ tree with new data gets harder as it gets bigger.

As others have said, I'll leave it to Aphyr to possibly explain.

hubertzhang · a year ago
> It superficially sounds like a series of server processes fronting actual database servers, which sounds like another layer of partition vulnerability and points of failure

I agree that introducing a middleware layer on top of databases adds more points of failure. EloqKV avoids this by integrating Compute, Logging, and Storage into a single system. In this setup, storage can be a database like Cassandra, but users will not access it directly; all requests go through EloqKV, which manages ACID transactions. EloqKV is responsible for handling crashes of the TxServer, LogServer, and Storage. You can think of the distributed Storage Engine just as a Disk in traditional DBMS. Obviously its failure will affect the system. But no more than hard disk failure. In fact, in a cloud, all disks (i.e. EBS) are actually distributed storage systems.

This situation is a rethinking of the Redis and MySQL combination, which suffers from similar issues. Both systems can fail independently, resulting in only eventual consistency. EloqKV aims to resolve this problem.

hubertzhang commented on Show HN: EloqKV – Scalable distributed ACID key-value database with Redis API   eloqdata.com/blog/2024/08... · Posted by u/hubertzhang
gregwebs · a year ago
Is the WAL 1:1 with a TxMap instance? Is the TxMap 1:1 with a node? For a distributed transaction with distributed storage and multiple nodes how does a transaction get coordinated?
hubertzhang · a year ago
Good question! WAL and TxService(TxMap) are fully decoupled, allowing you to deploy them on the same node or across different machines. If your workload is write-intensive, you might consider a large WAL cluster, which leverages multiple disks to perform fsync operations in parallel. This is particularly important in cloud environments, where local SSDs are ephemeral and cloud-native databases often use EBS for WAL. However, high-performance EBS options like io2 can be costly. EloqKV's decoupled architecture allows you to scale write throughput up to 600K Write OPS on a single TxServer as you increase the number of cheap gp3 disks. Since WAL logs can be truncated after a checkpoint, the required disk size is typically quite small, making gp3 a cost-effective choice.

For more details, please check our scaling disks benchmark report.

https://www.eloqdata.com/blog/2024/08/25/benchmark-txlog#exp...

> For a distributed transaction with distributed storage and multiple nodes how does a transaction get coordinated?

EloqKV is a multi-writer system, similar to many distributed databases (FoundationDB, TiDB, CockroachDB), but we have a set of new innovations on transaction protocols, for example, the entire system operates without a single synchronization point, not even a global sequencer. We drew many inspirations from the Hekaton and the TicToc paper.

u/hubertzhang

KarmaCake day16June 29, 2024
About
hubertzhang@eloqdata.com
View Original