If you are tired of using Apache or Nginx to proxy your app and implement validation or limiting you may be interested in this hobbyist project. Thanks for all feedback.
Similar but not quite the same is "forward auth" which is supported in Caddy, Nginx and Traefik Proxy and others as well.
With "forward auth", when the server receives a request it first sends the key information about the request to a separate HTTP server - if the response is a 200 then the proxy server continues processing the request, any other response rejects the request.
You can do all sorts of authorization and authentication and some validation (the request body is not typically sent to the forward auth server).
I wrote Checkpoint401 which is a forward auth server desgined specifically for doing the job of handling forward auth requests from proxy servers. Its written in TypeScript/Deno - all you do is write short authorization functions. https://github.com/crowdwave/checkpoint401
One of the good things about forward auth is that it is not a proxy so it is not sitting as a barrier in between client and server shoveling data back and forth - forward auth focuses only on examining the inbound request - this makes things more simple.
Forward auth does not have all the validation and filtering which Dracan seems to have but they solve some similar problems.
just to make them easier to instrument for my experiments. The first one I made because after trying all tutorials on haproxy/nginx to make proxying work without the target domain being resolvable (so no dns entry until it's up), I got annoyed (nothing worked while everything only and gpt said it should work) and just did it like this. Also it makes it very easy to throw in my own, complex as I want rules and logging and telemetry (to whatever I want) at any level.
The second I needed to test an existing, not able to change (they have an old version running they cannot update at the moment, don't ask) software against a redis sentinel setup.
The main thing is that I am more and going towards having the program language as config instead of greenspun's tenth rule where everyone builds some annoying and limited 'almost (usually esoteric) programming language' as config language over time. I want that hard lifting to be done in my native language (Lisp, Go, Rust) and have the config be for ip addresses and ports only.
I would expect my web application framework to handle all of these tasks, except perhaps header filtering. If it didn’t I’d rather fix that problem in the web application itself instead of adding a complicating layer of infrastructure that I now need to include in integration tests and release process.
I see some merit to moving the size limits etc out of the application to reduce CPU waste there on overly large requests, but either way I’m still burning some CPUs on it.
Is the use-case for this mostly about sticking some validation in front of a system who’s code you can’t or don’t want to modify for some reason, like in front of Wordpress?
Yes, I agree that applications themselves should handle requests that are not correct for them. I in my experience (maybe not so great :)) have encountered several times products even of large corporations that goofed up with problematic requests and/or payloads in that request. But yes I agree with you,it is adding an extra layer and complexity to the deployment, but sometimes it is a very convenient tool - for example as I mentioned before if you don't want to use off-the-shelf products like apache or nginx. Which should do the job and sift out the bad from the good.
I wasn't trying to be dismissive, I really am curious to hear more about those use-cases, it's interesting because its not something I've experienced or know much about
Python is fast and it's a good choice as a career language for sure.
Now that the GIL will be removed and adding a JIT is an inevitable step as well, we're looking into replacing everything written in Java with Python in the perspective of the next 10-20 years, depending on how soon people retire in a specific geographic region around the world.
The generation of people who, between 1995-2010, rewrote everything from C++ and COBOL into Java is now in their late 40s and 50s, so it's safe to assume there will be plenty of work for Python people until the next generation begins to mature around 2035-2040.
Now, whether it makes sense today to rewrite in Python something like a proxy, which is not a very complex type of software in itself?
If, starting today, you'd like to build within a year a proxy for something like StackOverflow, it's better to leave it for lower-level languages, like Go and Rust. These are replacements for C and C++, rather than Java, so they would likely be a better choice.
That said, my real message is, don't stick to writing such simple software for too long anyway.
If it's for educational purposes, to learn how all the various protocols work, or how to design server-side software, or to learn how to build an online community, that's a different story.
But you have this high level language in Python that lets you easily accomplish things that the lower-level languages just aren't best suited for, so once you wrote your first proxy and it can handle a few hundred or thousand requests/s, pick a high-level goal and work towards that instead! :-)
I mostly agree with you, I learned java at university but dropped it because python (mostly scripts) are very useful for DevOps / SysAdmin work. I definitely agree that writing proxies in Python is not the best idea, but it is the language I know best. And yes, it's not a high-level goal, but as I replied to someone earlier in this comments section. I don't want to make a career out of this, I just wrote a tool that was needed in my organisation and brought it here because I thought it might be useful to someone. Thank you for the extended comment.
P.S: Yes I am looking for some high-level project to participate in or just help with the knowledge I have.
Compared to compiled languages and JIT languages CPython is not fast, and removing the GIL does not improve the single thread performance of the language, right now it causes a 1-2% performance regression of single thread. Something like request validation seems like it wouldn’t benefit from more threads much; or if you need N threads to validate a request in 1ms in Python, it’s likely you could validate the same request in 1 thread in 1ms with another language.
But is it performant? People use and do many things that are awful in production, so that's not really an objective plus. Also; not that many systems need that much optimised performance, so it might be ok to be slow if there are many nice features that are more important.
Ok, HAProxy is a potential solution, but according to the description I posted, the idea behind this project is to reduce the writing of configuration files. Instead of writing an elaborate conf, just provide two JSONs to achieve the same thing. And yes I am aware of the fact that solutions like Apache, HaProxy and NginX are more popular and preferred for large solutions. I do not want to create yet another miracle product in well researched matter.
But I am grateful for the feedback :)
Readit News
With "forward auth", when the server receives a request it first sends the key information about the request to a separate HTTP server - if the response is a 200 then the proxy server continues processing the request, any other response rejects the request.
You can do all sorts of authorization and authentication and some validation (the request body is not typically sent to the forward auth server).
I wrote Checkpoint401 which is a forward auth server desgined specifically for doing the job of handling forward auth requests from proxy servers. Its written in TypeScript/Deno - all you do is write short authorization functions. https://github.com/crowdwave/checkpoint401
One of the good things about forward auth is that it is not a proxy so it is not sitting as a barrier in between client and server shoveling data back and forth - forward auth focuses only on examining the inbound request - this makes things more simple.
Forward auth does not have all the validation and filtering which Dracan seems to have but they solve some similar problems.
https://github.com/tluyben/go-proxy
https://github.com/tluyben/redis-sentinel-proxy
just to make them easier to instrument for my experiments. The first one I made because after trying all tutorials on haproxy/nginx to make proxying work without the target domain being resolvable (so no dns entry until it's up), I got annoyed (nothing worked while everything only and gpt said it should work) and just did it like this. Also it makes it very easy to throw in my own, complex as I want rules and logging and telemetry (to whatever I want) at any level.
The second I needed to test an existing, not able to change (they have an old version running they cannot update at the moment, don't ask) software against a redis sentinel setup.
The main thing is that I am more and going towards having the program language as config instead of greenspun's tenth rule where everyone builds some annoying and limited 'almost (usually esoteric) programming language' as config language over time. I want that hard lifting to be done in my native language (Lisp, Go, Rust) and have the config be for ip addresses and ports only.
I see some merit to moving the size limits etc out of the application to reduce CPU waste there on overly large requests, but either way I’m still burning some CPUs on it.
Is the use-case for this mostly about sticking some validation in front of a system who’s code you can’t or don’t want to modify for some reason, like in front of Wordpress?
Now that the GIL will be removed and adding a JIT is an inevitable step as well, we're looking into replacing everything written in Java with Python in the perspective of the next 10-20 years, depending on how soon people retire in a specific geographic region around the world.
The generation of people who, between 1995-2010, rewrote everything from C++ and COBOL into Java is now in their late 40s and 50s, so it's safe to assume there will be plenty of work for Python people until the next generation begins to mature around 2035-2040.
Now, whether it makes sense today to rewrite in Python something like a proxy, which is not a very complex type of software in itself?
If, starting today, you'd like to build within a year a proxy for something like StackOverflow, it's better to leave it for lower-level languages, like Go and Rust. These are replacements for C and C++, rather than Java, so they would likely be a better choice.
That said, my real message is, don't stick to writing such simple software for too long anyway.
If it's for educational purposes, to learn how all the various protocols work, or how to design server-side software, or to learn how to build an online community, that's a different story.
But you have this high level language in Python that lets you easily accomplish things that the lower-level languages just aren't best suited for, so once you wrote your first proxy and it can handle a few hundred or thousand requests/s, pick a high-level goal and work towards that instead! :-)
P.S: Yes I am looking for some high-level project to participate in or just help with the knowledge I have.
Compared to compiled languages and JIT languages CPython is not fast, and removing the GIL does not improve the single thread performance of the language, right now it causes a 1-2% performance regression of single thread. Something like request validation seems like it wouldn’t benefit from more threads much; or if you need N threads to validate a request in 1ms in Python, it’s likely you could validate the same request in 1 thread in 1ms with another language.