Readit Newsk4k4 commented on Show HN: Dracan – Open-source, 1:1 proxy with simple filtering/validation config github.com/Veinar/dracan... · Posted by u/k4k4
Big enterprise solution and custom not well tested solution, it smells like possibility of failure. But fingers crossed ;)
That's why i can focus on making it even better :)
Ah yes, how silly of me. Writing two config files in fucking JSON of all things is absolutely better than writing one config file.
Firstly, you don't have to be rude. I won't fall for this bait.
Secondly, if you are tied to writing one configuration file, I understand, after all you have a free choice. For me, the separation of logic is a positive aspect, because I don't have to think and search for entries about the specific server I'm proxying to. JSONs used here are overcomplicated I see. And the difficulty of mounting configuration files from ConfigMap with one file or two is practically the same - yet with this software you store two files in ConfigMap. I understood from your post that you are a HAProxy fanatic and I won't bother you in any way.
I wasn't trying to be dismissive, I really am curious to hear more about those use-cases, it's interesting because its not something I've experienced or know much about
We had a problem with a solution from a large international supplier. While doing an external audit (pentesting), we came across vulnerabilities and the possibility of ‘breaking’ the system as a whole (apart from the fact that natively the system did not support service-mesh, which was required by the client...). So, to mitigate the open vulnerabilities and add service-mesh (sort of), we used Apache (httpd), but writing the configuration and figuring out how to patch it well was driving me crazy. That's why this solution was created, if I'm honest it took me a week to write it, which is about the time I had to solve the problem described earlier. I got the green light from my superiors for something of my own and that's how this project came about. Plus the fact that I wanted to see how rusty my Python was :)
a quick perusal of the listed features makes me think HAProxy would be a better solution for this.
Ok, HAProxy is a potential solution, but according to the description I posted, the idea behind this project is to reduce the writing of configuration files. Instead of writing an elaborate conf, just provide two JSONs to achieve the same thing. And yes I am aware of the fact that solutions like Apache, HaProxy and NginX are more popular and preferred for large solutions. I do not want to create yet another miracle product in well researched matter.
But I am grateful for the feedback :)
Python is fast and it's a good choice as a career language for sure.
Now that the GIL will be removed and adding a JIT is an inevitable step as well, we're looking into replacing everything written in Java with Python in the perspective of the next 10-20 years, depending on how soon people retire in a specific geographic region around the world.
The generation of people who, between 1995-2010, rewrote everything from C++ and COBOL into Java is now in their late 40s and 50s, so it's safe to assume there will be plenty of work for Python people until the next generation begins to mature around 2035-2040.
Now, whether it makes sense today to rewrite in Python something like a proxy, which is not a very complex type of software in itself?
If, starting today, you'd like to build within a year a proxy for something like StackOverflow, it's better to leave it for lower-level languages, like Go and Rust. These are replacements for C and C++, rather than Java, so they would likely be a better choice.
That said, my real message is, don't stick to writing such simple software for too long anyway.
If it's for educational purposes, to learn how all the various protocols work, or how to design server-side software, or to learn how to build an online community, that's a different story.
But you have this high level language in Python that lets you easily accomplish things that the lower-level languages just aren't best suited for, so once you wrote your first proxy and it can handle a few hundred or thousand requests/s, pick a high-level goal and work towards that instead! :-)
I mostly agree with you, I learned java at university but dropped it because python (mostly scripts) are very useful for DevOps / SysAdmin work. I definitely agree that writing proxies in Python is not the best idea, but it is the language I know best. And yes, it's not a high-level goal, but as I replied to someone earlier in this comments section. I don't want to make a career out of this, I just wrote a tool that was needed in my organisation and brought it here because I thought it might be useful to someone. Thank you for the extended comment.
P.S: Yes I am looking for some high-level project to participate in or just help with the knowledge I have.
I agree with you, and also about the Postgres part. And that was my point: if it's performant enough or not depends on the use case; it's just here on HN everyone thinks they 'will make it' (something something facebook/google etc scale) while they won't. So then anything works fine, because you have got no traffic or data at any significant volume.
To be honest, I don't want to make a career out of it and I didn't plan it for this project, I just needed a tool like this, so I wrote it because I was tired of configuring Apache :) I thought it might be useful to someone else, so I published it here. Thanks, I just had to say it
Only asked if it was performant based on what the parent said about Python. But cool.
Results using Powershell with setup:
Windows 11 / I7-13700F / 16GB RAM / Z790 chipset
With usage of this script:
https://gist.github.com/Veinar/bd8abc12ed3ce3367980da5a335f7...
Used rules that are example within repository.
Results: https://pastebin.com/61Fyy2Pe ( too long to past it here... sorry )
Request Time: The average request time in all tests is about the same, ranging from 0.006 to 0.007 seconds. Max request time does increase with more requests; it peaks for the most substantial test of 100,000 requests at 0.136 seconds, which does show that some requests take much longer.
Requests per Second: The number of requests per second is highest in the smaller tests, around 143 RPS for the 10 requests, whereas for 100,000 requests it goes down to about 122 RPS. A probable conclusion in this case could be that while increasing the number of requests, some little slowdown starts to develop in the system.
Percentiles: The median, which usually stands at approximately 0.0035 seconds, essentially means half of the requests are done in under that time. The far higher values of the 90th and 99th percentiles just prove that while most of the requests may be fast, the others take considerably longer.
In general, it performs quite well under a reasonable load but biffs a bit if the number of requests is increased.
I can test OKD/k8s on Thursday at the earliest.
I would expect my web application framework to handle all of these tasks, except perhaps header filtering. If it didn’t I’d rather fix that problem in the web application itself instead of adding a complicating layer of infrastructure that I now need to include in integration tests and release process.
I see some merit to moving the size limits etc out of the application to reduce CPU waste there on overly large requests, but either way I’m still burning some CPUs on it.
Is the use-case for this mostly about sticking some validation in front of a system who’s code you can’t or don’t want to modify for some reason, like in front of Wordpress?
Yes, I agree that applications themselves should handle requests that are not correct for them. I in my experience (maybe not so great :)) have encountered several times products even of large corporations that goofed up with problematic requests and/or payloads in that request. But yes I agree with you,it is adding an extra layer and complexity to the deployment, but sometimes it is a very convenient tool - for example as I mentioned before if you don't want to use off-the-shelf products like apache or nginx. Which should do the job and sift out the bad from the good.
But is it performant? People use and do many things that are awful in production, so that's not really an objective plus. Also; not that many systems need that much optimised performance, so it might be ok to be slow if there are many nice features that are more important.
I'll write some JMeter test (or even simpler way) to check how efficient it is. And return with this information.