Tangential, but what I appreciate about Elixir/Erlang is that you have ETS out of the box that is basically a key-value store on a lightweight Erlang process (without socket access or network calls) with flexible 'types', and then DETS brings persistence, mnesia adds schemas, queries, transactions, replication and other DBMS-like features... all this and you can use Erlang's actor model to cluster and horizontally scale, or just use it like a simple L1 cache, as described in the article
Most people use "order of magnitude" to mean "by a factor 10". I've always thought this was inappropriate. The origin of this expression is that the scientific use of magnitude implies a logarithmic scale. But "magnitude" was first used in astronomy with a scale that had a 2.5 base, not a 10 base.
Caching follows Pareto principle - users will get 80% of the benefit just by using either tool effectively, doesn't matter which.
That makes me more interested in comparing the developer experience.
From a practical perspective, Redis is probably a better cache choice for a typical web developer who just wants to get things done and doesn't have major scaling/performance concerns.
It has two main advantages:
(1) Much better tooling. Memcached doesn't have a lot of support to figure out what's going on, measure hit ratio, run functions on cache data, etc. Redis has better built-in diagnostics and third-party tools and libraries.
(2) It's ideal for smaller teams to reduce infrastructure dependencies. Since Redis has many capabilities beyond in-memory caching, there's a good chance you'll be able to take advantage of it for more than just caching in the future (while it's still not really any more complicated to install/maintain than memcached).
1) memcached has excellent support for checking cache hit ratio. Just telnet to it and execute "stats". Most client libraries have support for the stats command, and there is a Prometheus exporter for memcached that works really well.
2) A problem I've seen with Redis is that it is _stateful_. While, "it's still not really any more complicated to install/maintain than memcached" is true for running it locally, I keep seeing over and over again how engineers
i) don't think about backup/restore at all.
ii) expect it to behave like a disk-backed database and don't consider it's limited to the amount of memory you have.
iii) don't consider that it (usually) is a secondary data store that can be out of sync with primary datastore (distributed commits are hard).
iv) don't consider that a proper Redis setup likely needs support for point-in-time recovery and the complexities around that and redundancy/failover.
While Redis is easy to setup as a cache and likely doesn't need the above, I keep seeing them organically grow into being used for other use cases where the above things are not in place, yielding very high operational risks sometimes not acknowledged anywhere.
Thanks. On (2) I'm thinking only of Redis as a memory cache, in that case I think it's fairly simple, as with Memcached.
Once you start doing persistence, then yes, it becomes a bit more complicated with a lot of other things to worry about, especially that you need RAM >= disk space. While it's another advantage of Redis that persistent caching is possible (and can be bolted on at a later date), taking advantage of it definitely adds some complexity and room for error.
2) III - Distributed commits aren't just hard they don't exist. Redis and your primary dB can never hold the same data - at most you can guarantee only for IMMUTABLE data that what is in DB is also in redis.
I've seen others and myself included try to sync them when solving some problems. Just because it is so natural to believe it possible to do. Then you find out you are running circles.
Honestly for the use cases I've met Redis has only made sense to be used as a cache for performance or message bus. Your primary data store is gonna give you the other guarantees of backups, consistency etc...
Unless somebody usew it as their primary store, but everybody chooses their own religion I guess.
There is still the misconception that if you just want to cache, you don't need data types. One of the biggest Redis use cases, Twitter (I didn't check if they still use it in the last two years, I guess so, and anyway used it for many years), beg to differ. But in general almost all the serious caching scenarios using Redis represent part of the stored data, in memory, with an organization that allows updating the cache and selectively retrieving from the cache like if it was the primary (oh, not just the primary, specified queries you could do against the primary), so everything is incremental and there is no need to cache multiple "views" of the same thing.
I switched from Memcached to Redis 10 years ago because I had a situation where I needed to cache lookups of (x, y) but needed to invalidate (x, *), something that is trivial with Redis hashes, but extremely difficult in memcached.
Redis is probably my favorite software of the past decade. I abuse it liberally and it always takes it with resounding uptime and performance. In the stack that I utilize it's always among the things I have to worry the least about.
Agreed, Redis is truly wonderful, I've used it extensively as both a cache lookalike and also as primary datasource (as long as your data size/growth is known and manageable on RAM, which is pretty common for many projects and current hosts) and it's always super fast, reliable and easy to work with.
Not tested recently, but 3 years ago it was said that Redis was slightly faster than memcached for PHP sessions -for example- and I benchmarked it to decide to change. Add persistence to that formula and Redis was a winner for me (put your app into maintenance mode, reboot server(s), enable app again and your users can just resume operations...)
Looks like this article got one bit of updated information but missed everything else... I'll address some things point by point:
data structures: - yes, fewer datastructures, if any. The point isn't that they have the features or not, but that memcached is a distributed system _first_, so any feature has to make sense in that context.
"Redis is better supported, updated more often. (or maybe memcached is "finished" or has a narrower scope?)" - I've been cutting monthly releases for like 5 years now (mind the pandemic gap). Sigh.
Memory organization: This is mostly accurate but missing some major points. The sizes of the slab classes doesn't change, but slabs pages can and do get re-assigned automatically. If you assign all memory to the 1MB page class, then empty that class, memory will go back to a global pool to get re-assigned. There are edge cases but it isn't static and hasn't been for ten years.
Item size limit: The max slab size has actually been 512k internally for a long time now, despite the item limit being 1mb. Why? Because "large" items are stitched together from smaller slab chunks. Setting a 2mb or 10mb limit is fine in most use cases, but again there are edge cases, especially for very small memory limits. Usually large items aren't combined with small memory limits.
You can also _reduce the slab class overhead_ (which doesn't typically exceed 5-10%), by lowering the "slab_chunk_max" option, which puts the slab classes closer together at the expense of stitching items larger than this class. IE; if all of your objects are 16kb or less, you can freely set this limit to 16kb and reduce your slab class overhead. I'd love to make this automatic or at least reduce the defaults.
LRU: looks like the author did notice the blog post (https://memcached.org/blog/modern-lru/) - I'll add that the LRU bumping (mutex contention) is completely removed from the _access path_. This is why it scales to 48 threads. The LRU crawler is not necessary to expire items, there is also a specific thread that does the LRU balancing.
The LRU crawler is used to proactively expire items. It is highly efficient since it independently scans slab classes; the more memory an object uses the fewer neighbors it has, and it schedules when to run on each slab class, so it can "Focus" on areas with higest return.
Most of the thread scalability is pretty old; not just since 2020.
Also worth noting memcached has an efficient flash backed storage system: https://memcached.org/blog/nvm-caching/ - requires RAM to keep track of keys, but can put value data on disk. With this tradeoff we can use flash devices without burning them out, as non-get/non-set operations do not touch the SSD (ie; delete removes from memory, but doesn't cause a write). Many very huge installations of this exist.
I've also been working on an internal proxy which is nearing production-readiness for an early featureset: https://github.com/memcached/memcached/issues/827 - scriptable in lua, will have lots of useful features.
For people who don't know and didn't realize this from the comment: dormando is the principal maintainer of memcached and has been for years (e.g., bradfitz was much less involved after he joined Google).
I’m sorry for asking a rookie question, but you seem to know memcached really well and I couldn’t find an answer online.
Is there a way to obtain/monitor the time stamp of LTU evictions?
I want to get a sense of how memory constrained my memcached server is, and it seems intuitive to me to monitor the “last used” date of recent evictions. Like, if The server is evicting values that haven’t been accessed in 3 months; great. But if the server is evicting values that were last used < 24 hours ago; I have concerns.
There are stats in "stats items" / "stats slabs". Last access time for most recent eviction per slab class, etc. (see doc/protocol.txt from the tarball).
"watch evictions" command will also show a stream of details for items being evicted.
Not sure if there's a better place to ask? But ill just try here. Curious about a design decision in the extstore. It seems to include a lot of extra stuff around managing writes and whats in memory and whats on disk. Why do you think this is better then just mmap-ing and letting the OS decide whats in memory using the fs cache and what pages are still disk?
That's an excellent question; it turns out there are a _lot_ of semantics that the OS is covering up for you when using mmap. For instance (this may be fixed by now), but any process doing certain mmap syscalls locked access to any open mmap's in an OS. So some random cronjob firing could clock your mmap'ed app pretty solidly.
There are also wild bugs; if you google my threads on the LKML you'll find me trying to hunt down a few in the past.
Mainly what I'm doing with extstore is maintaining a clear line between what I want the OS doing and what I want the app doing: a hard rule that the memcached worker threads _cannot_ be blocked for any reason. When they submit work to extstore, they submit to background threads then return to dequeueing network traffic. If the flash disk hiccups for any reason it means some queue's can bloat but other ops may still succeed.
Further, by controlling when we defrag or drop pages we can be more careful with where writes to flash happen.
TLDR: for predictable performance. Extstore is also a lot simpler than it may sound; it's a handful of short functions built on a lot of design decisions instead of a lot of code building up an algorithm.
It was an algorithmic/lock scaling limit. Originally it was single threaded, then when it was first multi-threaded it scaled up to 4 threads. Then I split up some locks and it scaled to 8 threads (depending). Then I rewrote the LRU and now reads mostly scale linearly and writes don't. If there's enough interest we'll make writes scale better.
Partly this is because the software is so old that the thread scalability tends to track how many CPU's people actually have.
Readit News
I see this mistake all the time, even in print! It's millis -> micros -> nanos, so...that's a lot more than a single order of magnitude.
that phrase (for countless years) feels like an attempt to sound precise, but is imprecise. great example there.
> Memcached slabs once assigned never change their size.
This is incorrect. Modern versions of Memcached has support for this. See for example [1].
[1] https://www.loginradius.com/blog/async/memcach-memory-manage...
I wrote about Pelikan, a unified cache framework, and its relationship to Memcached and Redis in this blog from 2019: https://twitter.github.io/pelikan/2019/why-pelikan.html
Specifically talked about what we would like to change about each.
I've fixed the post as well, thanks for finding this bug.
That makes me more interested in comparing the developer experience.
From a practical perspective, Redis is probably a better cache choice for a typical web developer who just wants to get things done and doesn't have major scaling/performance concerns.
It has two main advantages:
(1) Much better tooling. Memcached doesn't have a lot of support to figure out what's going on, measure hit ratio, run functions on cache data, etc. Redis has better built-in diagnostics and third-party tools and libraries.
(2) It's ideal for smaller teams to reduce infrastructure dependencies. Since Redis has many capabilities beyond in-memory caching, there's a good chance you'll be able to take advantage of it for more than just caching in the future (while it's still not really any more complicated to install/maintain than memcached).
1) memcached has excellent support for checking cache hit ratio. Just telnet to it and execute "stats". Most client libraries have support for the stats command, and there is a Prometheus exporter for memcached that works really well.
2) A problem I've seen with Redis is that it is _stateful_. While, "it's still not really any more complicated to install/maintain than memcached" is true for running it locally, I keep seeing over and over again how engineers
i) don't think about backup/restore at all.
ii) expect it to behave like a disk-backed database and don't consider it's limited to the amount of memory you have.
iii) don't consider that it (usually) is a secondary data store that can be out of sync with primary datastore (distributed commits are hard).
iv) don't consider that a proper Redis setup likely needs support for point-in-time recovery and the complexities around that and redundancy/failover.
While Redis is easy to setup as a cache and likely doesn't need the above, I keep seeing them organically grow into being used for other use cases where the above things are not in place, yielding very high operational risks sometimes not acknowledged anywhere.
Once you start doing persistence, then yes, it becomes a bit more complicated with a lot of other things to worry about, especially that you need RAM >= disk space. While it's another advantage of Redis that persistent caching is possible (and can be bolted on at a later date), taking advantage of it definitely adds some complexity and room for error.
I've seen others and myself included try to sync them when solving some problems. Just because it is so natural to believe it possible to do. Then you find out you are running circles.
Honestly for the use cases I've met Redis has only made sense to be used as a cache for performance or message bus. Your primary data store is gonna give you the other guarantees of backups, consistency etc...
Unless somebody usew it as their primary store, but everybody chooses their own religion I guess.
Lua in both places. So (ab)useful!
data structures: - yes, fewer datastructures, if any. The point isn't that they have the features or not, but that memcached is a distributed system _first_, so any feature has to make sense in that context.
"Redis is better supported, updated more often. (or maybe memcached is "finished" or has a narrower scope?)" - I've been cutting monthly releases for like 5 years now (mind the pandemic gap). Sigh.
Memory organization: This is mostly accurate but missing some major points. The sizes of the slab classes doesn't change, but slabs pages can and do get re-assigned automatically. If you assign all memory to the 1MB page class, then empty that class, memory will go back to a global pool to get re-assigned. There are edge cases but it isn't static and hasn't been for ten years.
Item size limit: The max slab size has actually been 512k internally for a long time now, despite the item limit being 1mb. Why? Because "large" items are stitched together from smaller slab chunks. Setting a 2mb or 10mb limit is fine in most use cases, but again there are edge cases, especially for very small memory limits. Usually large items aren't combined with small memory limits.
You can also _reduce the slab class overhead_ (which doesn't typically exceed 5-10%), by lowering the "slab_chunk_max" option, which puts the slab classes closer together at the expense of stitching items larger than this class. IE; if all of your objects are 16kb or less, you can freely set this limit to 16kb and reduce your slab class overhead. I'd love to make this automatic or at least reduce the defaults.
LRU: looks like the author did notice the blog post (https://memcached.org/blog/modern-lru/) - I'll add that the LRU bumping (mutex contention) is completely removed from the _access path_. This is why it scales to 48 threads. The LRU crawler is not necessary to expire items, there is also a specific thread that does the LRU balancing.
The LRU crawler is used to proactively expire items. It is highly efficient since it independently scans slab classes; the more memory an object uses the fewer neighbors it has, and it schedules when to run on each slab class, so it can "Focus" on areas with higest return.
Most of the thread scalability is pretty old; not just since 2020.
Also worth noting memcached has an efficient flash backed storage system: https://memcached.org/blog/nvm-caching/ - requires RAM to keep track of keys, but can put value data on disk. With this tradeoff we can use flash devices without burning them out, as non-get/non-set operations do not touch the SSD (ie; delete removes from memory, but doesn't cause a write). Many very huge installations of this exist.
I've also been working on an internal proxy which is nearing production-readiness for an early featureset: https://github.com/memcached/memcached/issues/827 - scriptable in lua, will have lots of useful features.
Is there a way to obtain/monitor the time stamp of LTU evictions?
I want to get a sense of how memory constrained my memcached server is, and it seems intuitive to me to monitor the “last used” date of recent evictions. Like, if The server is evicting values that haven’t been accessed in 3 months; great. But if the server is evicting values that were last used < 24 hours ago; I have concerns.
"watch evictions" command will also show a stream of details for items being evicted.
There are also wild bugs; if you google my threads on the LKML you'll find me trying to hunt down a few in the past.
Mainly what I'm doing with extstore is maintaining a clear line between what I want the OS doing and what I want the app doing: a hard rule that the memcached worker threads _cannot_ be blocked for any reason. When they submit work to extstore, they submit to background threads then return to dequeueing network traffic. If the flash disk hiccups for any reason it means some queue's can bloat but other ops may still succeed.
Further, by controlling when we defrag or drop pages we can be more careful with where writes to flash happen.
TLDR: for predictable performance. Extstore is also a lot simpler than it may sound; it's a handful of short functions built on a lot of design decisions instead of a lot of code building up an algorithm.
Partly this is because the software is so old that the thread scalability tends to track how many CPU's people actually have.