It is based on sharded sqlite files, and it's surprisingly capable, even in multiprocessing based env like a django site.
Sqlite is really showing up in all sort of use cases these days, like huey, a task queue that doesn't need any broker to be installed to be useful: https://pypi.org/project/diskcache/
Now things can get fun...
With supervisord (supervisord.org) you can manage multiple processes, and with fastapi + starlette, you can serve your WSGI site without a proxy pass, including static sites: https://fastapi.tiangolo.com/advanced/wsgi/
Then with shiv you can bunddle all that in a pyz file: shiv.readthedocs.io/
Meaning you can now just upload a pyz file to your server, run python on it, and that's it, you have a full stack deployed, with a web framework, db, caching, task queues, static file serving, which can handle around 100k visitor each day without a sweat.
No more infra needed. Just Python code, all bundle in one file.
Last year I tried Huey for a project where usage was very lightweight. Performance was more than enough with SQLite. Unfortunately Huey with SQLite was not great at handling unexpected crashes like server and docker restarts.
Totally agree on hoping there is at least some movement forward on async--particularly async ORM. I love Django and DRF. It is my go-to for a lot of things, but the lack of async support is increasingly a problem for more organizations. Honestly, it would not surprise me that if a solid, kitchen sink web framework emerges in golang, Django will be abandoned rather quickly.
I've been writing web applications using python and django for close to 15 years now. Smooth, concise and stable. Doing the same using a verbose and boilerplate heavy language like golang doesn't sound attractive to me, at all.
Not sure I understand the fuss about upgrades either, at my current job I've gone from python 2 to 3 and from Django 1.8 all the way to 3.2 LTS without too many headaches. The release notes provide detailed instructions on what is deprecated and what you should do to fix it.
Yes, maybe in 10 years when this Go framework catches up with all the features and becomes as battle proven, well documented, maintained and funded as Django and somehow we know is not yet another thing that will be abandoned next year.
The Go hype is as bad as the frontend hype in my opinion.
If you're replacing Django with Go then either you never needed Django, or you have no idea what you're getting into.
At a previous job backend team decided to replace rails "because monolith slow ruby not fun anymore" with Go. Three years later they were still reimplementing things rails gives you since day 1. I've seen abominations as doing migrations as SQL scripts ran by cobbled together bash scripts because "as we don't have orms then orms are bad".
No need to say now they're about 4 years in and the one still running the business and making all the money is the monolith, not all the Go microservices around it.
Go is a replacement for c/c++, maybe java and for building infrastructure probably.
If you're using it for doing CRUD forms, you're using it wrong.
Inconsistent behavior around constraints checking in unit tests, can lead to inexplicable behavior and difficulty testing when a constraint is violated: https://code.djangoproject.com/ticket/11665
In general, the ORM functionality and the inconsistency, bugs being introduced in patch releases has me looking elsewhere for all new development and recommending to other engineers the same.
We have a running joke in our engineering team that every bug we hit in Django will have an open or "closed wontfix" issue 5-10 years old.
I really wish they'd prioritize multi-column primary keys. I don't have a ticket handy, but it's been out there for many, many years as well. I feel like it would be huge for projects built on legacy database, where you have to rely on existing schema.
Even if it isn’t a legacy project, sometimes you just so happen to have a natural composite key on a table and don’t want to assign a separate single column primary key just to satisfy your ORM.
> Now they are shipping a built in Redis cache back-end?
no. actual link as you surmised states:
This PR aims at adding support for Redis to be used as a caching
backend with Django. As redis is the most popular caching backend,
adding it to django.core.cache module would be a great addition for
developers who previously had to rely on the use of third party
packages.
I was somewhat surprised to see that the django.core.cache module didn't support an mmap based cache. The default cache is per-process, so it's somewhat limited in effectiveness. Something like Redis solves that, but adds another running process to manage.
In the PHP world, for example, acpu is a popular user-facing cache that uses mmap so that it works cross-process, then you can move to redis if you want to scale across more than one host.
This is a welcome addition but isn't such a big deal... There are great third-party packages (i.e https://github.com/jazzband/django-redis) for whoever needs a redis cache backend in a project.
Generally the approach is fragment caching... so you're caching some block of HTML that would otherwise hit the DB (Think something like a block of configurable sidebar links, or a "latest posts" block.
HTML, not assets.
Some such setups (like one I wrote for a busy Drupal newspaper site way back when) can support clearing relevant cache entries on DB write, so you're essentially never serving stale data, but can save _substantial_ DB load - and possibly webserver load too, depending on how computationally intensive the HTML generation is.
What's annoying with Python though is how hard it is to just cache stuff in some dict or other data structure, since the GIL basically forces each request to have it's separate process (instead of separate thread). So non-trivial to share some state application wide without having to store it outside the application.
Of course a shared cache is sensible when there are multiple nodes running the application. But not everything needs a global cache either, or maybe the point of the cache was exactly to avoid a round trip to something else.
However, it seems this will enable the community to establish using redis as a best practice. So it will be a bigger deal than adding just another cache backend.
Django also needs to pin a default Postgres container image, and default NGINX image. The default stack should be extensively end-to end tested and optimized, including an optional patched kernel, cgroup2 settings, and untrusted nsjail worker process for handling media files.
Readit News
I assume the async story is going to be more fun.
By the way, if you are looking for a good key/value store with expiration without the need to setup redis, have a look at the excellent diskcache:
https://pypi.org/project/diskcache/
It is based on sharded sqlite files, and it's surprisingly capable, even in multiprocessing based env like a django site.
Sqlite is really showing up in all sort of use cases these days, like huey, a task queue that doesn't need any broker to be installed to be useful: https://pypi.org/project/diskcache/
Now things can get fun...
With supervisord (supervisord.org) you can manage multiple processes, and with fastapi + starlette, you can serve your WSGI site without a proxy pass, including static sites: https://fastapi.tiangolo.com/advanced/wsgi/
Then with shiv you can bunddle all that in a pyz file: shiv.readthedocs.io/
Meaning you can now just upload a pyz file to your server, run python on it, and that's it, you have a full stack deployed, with a web framework, db, caching, task queues, static file serving, which can handle around 100k visitor each day without a sweat.
No more infra needed. Just Python code, all bundle in one file.
That's really neat.
https://unpoly.com/
https://mitchel.me/slippers/
Switching to Redis solved all of these issues.
Maybe it's better now.
What about HTTPS?
2 cents.
Not sure I understand the fuss about upgrades either, at my current job I've gone from python 2 to 3 and from Django 1.8 all the way to 3.2 LTS without too many headaches. The release notes provide detailed instructions on what is deprecated and what you should do to fix it.
If you're replacing Django with Go then either you never needed Django, or you have no idea what you're getting into.
At a previous job backend team decided to replace rails "because monolith slow ruby not fun anymore" with Go. Three years later they were still reimplementing things rails gives you since day 1. I've seen abominations as doing migrations as SQL scripts ran by cobbled together bash scripts because "as we don't have orms then orms are bad".
No need to say now they're about 4 years in and the one still running the business and making all the money is the monolith, not all the Go microservices around it.
Go is a replacement for c/c++, maybe java and for building infrastructure probably.
If you're using it for doing CRUD forms, you're using it wrong.
.Net already offers a kitchensink with async, it’s growing a lot and is much more attractive than a decade ago.
I think you put the wrong link for Huey, btw.
However, as a user of Django I sincerely wish they would focus more on stability:
Changing jsonb deserialization in a patch release, 3.1.1: https://code.djangoproject.com/ticket/31956#comment:16
Queryset .get() can cause OOM, ticket has gone back and forth, problem existed as recently as Django 2.X: https://code.djangoproject.com/ticket/6785
Django's cascade options differ from the database's, are manually implemented: https://code.djangoproject.com/ticket/21961
Can't update keys via ORM, related to the above: https://code.djangoproject.com/ticket/21265
Inconsistent behavior around constraints checking in unit tests, can lead to inexplicable behavior and difficulty testing when a constraint is violated: https://code.djangoproject.com/ticket/11665
In general, the ORM functionality and the inconsistency, bugs being introduced in patch releases has me looking elsewhere for all new development and recommending to other engineers the same.
We have a running joke in our engineering team that every bug we hit in Django will have an open or "closed wontfix" issue 5-10 years old.
It claims you can either just run it or hook it into your CI. It will automatically take care of some of the upgrade chores for you.
Now they are shipping a built in Redis cache back-end?
That would be so totally crazy that I am having trouble believing this headline.
How about "Django 4.0 will include built-in Redis cache support"
no. actual link as you surmised states:
so just a bad link titleIn the PHP world, for example, acpu is a popular user-facing cache that uses mmap so that it works cross-process, then you can move to redis if you want to scale across more than one host.
Myself, I just use good ol' memcached for all my caching needs (https://docs.djangoproject.com/en/3.2/topics/cache/#memcache...). It is rock solid and has never failed me till now!
I don't understand what there is to cache in a web framework.
Is it to cache rendered pages without dynamic content?
If so, I think it is better to let a CDN handle that. Instead of the server that runs the web application.
What am I missing?
But you can cache other things that are not the entire page itself:
- database queries
- user sessions
- api calls
- page fragment
- function memoization
What's more, cache backends have the ability to expire things, which can be helpful for all sort of stuff.
E.g. results from external services, calculations, anything that an application might want to cache.
Am working on a somewhat large Django system, and there is a ton of different things we cache in Redis.
Also HTML template results as another comment mentioned, but mostly something else entirely, related to the application logic.
HTML, not assets.
Some such setups (like one I wrote for a busy Drupal newspaper site way back when) can support clearing relevant cache entries on DB write, so you're essentially never serving stale data, but can save _substantial_ DB load - and possibly webserver load too, depending on how computationally intensive the HTML generation is.
What's annoying with Python though is how hard it is to just cache stuff in some dict or other data structure, since the GIL basically forces each request to have it's separate process (instead of separate thread). So non-trivial to share some state application wide without having to store it outside the application.
Of course a shared cache is sensible when there are multiple nodes running the application. But not everything needs a global cache either, or maybe the point of the cache was exactly to avoid a round trip to something else.
However, it seems this will enable the community to establish using redis as a best practice. So it will be a bigger deal than adding just another cache backend.
This is good because the default is local-memory caching, and redis can easily beat that. https://docs.djangoproject.com/en/3.2/topics/cache/#local-me...