An interesting thing that I've noticed is that some of the attackers watch the Certificate Transparency logs for newly issued certificates to get their targets.
I've had several instances of a new server being up on a new IP address for over a week, with only a few random probing hits in access logs, but then, maybe an hour after I got a certificate from Let's Encrypt, it suddenly started getting hundreds of hits just like those listed in the article. After a few hours, it always dies down somewhat.
The take-away is, secure your new stuff as early as possible, ideally even before the service is exposed to the Internet.
> The take-away is, secure your new stuff as early as possible, ideally even before the service is exposed to the Internet.
Honestly it feels like you'll need at least something like basicauth in front of your stuff from the first minutes it's publicly exposed. Well, either that, or run on your own CA and use self-signed certs (with mTLS) before switching over.
For example, when some software still has initial install/setup screens where you create the admin user, connect to the DB and so on, as opposed to specifying everything initially in the environment variables, config files, or other more specialized secret management solutions.
I wish. I use basicauth to protect all my personal servers, the problem is Safari doesn't appear to store the password! I always have to re-authenticate when I open the page. Sometimes even three seconds later.
Was looking into Certificate Transparency logs recently. Are there any convenient tools/methods for querying CT logs? i.e. search for domains within a timeframe
Cloudflare’s Merkle Town[0] is useful for getting overviews, but I haven’t found an easy way to query CT logs. ct-woodpecker[1] seems promising, too
A compromised wildcard certificate has a much higher potential for abuse. The strong preference in IT security is a single-host or UCC (SAN) certificate.
Renewing a wildcard is also unfun when you have services which require a manual import.
I host so many services, but I gave up totally on exposing them to the internet. Modern VPNs are just too good. It lets me sleep at night. Some of my stuff is, for example, photo hosting and backup. Just nope all the way.
If you're the only one accessing those services, then why use a VPN instead of port mapping those services to localhost of the server, and then forwarding that localhost port to your client machine's localhost port via SSH?
I work as a security engineer and, yes, the CT logs are extremely useful not only for identifying new targets the moment you get a certificate but also for identifying patterns in naming your infra (e.g., dev-* etc.).
A good starting point for hardening your servers is CIS Hardening Guides and the relevant scripts.
Fun anecdote - I wrote a new load balancer for our services to direct traffic to an ECS cluster. The services are exposed by domain name (e.g. api-tools.mycompany.com), and the load balancer was designed to produce certificates via letsencrypt for any host that came in.
I had planned to make the move over the next day, but I moved a single service over to make sure everything was working. Next day as I'm testing moving traffic over, I find that I've been rate limited by Lets Encrypt for a week. I check the database and I had provisioned dozens of certificates for vpn.api-tools.mycompany.com, phpmyadmin.api-tools.mycompany.com, down the list of anything you can think of.
There was no security issue, but it was very annoying that I had to delay the rollout by a week and add a whitelist feature.
On censys.io you can search by domain for example. Some internet facing appliances generate certificate automatically with letsencrypt but use a central DNS server, meaning every one of these appliances is on the same domain, using random Subdomains.
Once you figured out what the domain is, you can easily build a list of IPs out of the cert transparency log and if there is ever an exploit for this specific type of appliances, attackers now have a bespoke list of IPs to hack, a dream come true.
I don't see a solution for this particular use case, I would argue self signed certs would be more secure in this case.
Same! As soon as a new cert is registered for a new subdomain, I get a small burst of traffic. It threw me off at first assuming I had some tool running that was scanning it.
> The take-away is, secure your new stuff as early as possible, ideally even before the service is exposed to the Internet.
What? Ideally..before? Seriously? It is 2024.. and this was true even decades ago, absolutely mandatory.
(Still remembering that dev that discovered file sharing in his exposed mongo instance (yes, that!! :D) without password only hours after putting it up.. "but how could they know the host it is secret!!" :D ).
Back when I started managing self-hosted sites, I would look through access logs as well. We even had an IDS for while that would aggregate the data and flag incoming attack attempts for us.
Eventually I stopped proactively reviewing logs and stopped paying for the IDS. It was a waste of time and a distraction.
It's not hard to find really useful content that summarizes common vulnerabilities and attacks, and just use that to guide your server management. There are a ton of best practices guides for any common web server technology. Just executing these best practices to 100% will put you way ahead of almost all attackers.
And then the next best use of your time and resources is to prioritize the fastest possible patching cadence, since the vast majority of attacks target disclosed vulnerabilities.
Where logs are super helpful is in diagnosing problems after they happen. We used log analysis software to store and search logs and this was helpful 2-3 times to help find (and therefore address) the root cause of attacks that succeeded. (In every case it turned out to be a known vulnerability that we had been too slow to patch.)
> In every case it turned out to be a known vulnerability that we had been too slow to patch.
Yes. This is why relying on "patching" is a bound to fail at some point. Maybe it's a 0-day, or maybe the attackers are just quicker.
The solution to this is defence in depth, and it's very easy for most services, especially when self-hosting personal things. Few tips most people can do is.
Put up a firewall in front or put it behind VPN/tailscale.
Hide it in a subfolder. The automated attacks will go for /phpmyadmin/ , putting it in /mawer/phpmyadmin/ means 99.9% of the attackers won't find it. (This is sometimes called security by obscurity and people correctly say you should not rely on it, but as a additional layer it's very useful).
Sandbox the app, and isolate the server. If the attackers get in, make it hard or impossible for them to get anywhere else.
Keep logs, they allow you to check if and how you got attacked, if the attack succeeded and so on.
Depending on the service, pick one or more of these. Add more as necessary.
The key thing is that you should not rely on any ONE defence, be it keeping it patched or firewalled, because they will all fail at some point.
> This is sometimes called security by obscurity and people correctly say you should not rely on it, but as an additional layer it's very useful.
Anyone who thinks “security by obscurity” is useless should try reverse engineering some properly obfuscated executables (or even code). Obscurity is absolutely useful; definitely not a complete solution by itself, but a very useful component to a security solution.
I detest this phrase right up there with "fake it till you make it". All security is by definition obscurity. Just a meaningless platitude that rhymes.
I suppose un-formalized un-proofed security practices will eventually be broken or counting on hackers not to do any investigation of your system will get you hacked, doesn't roll off the tongue though.
"and if at all possible keep them off the public internet", this is the way. I would recommend going beyond a VPN to implement zero trust networking which does outbound-only connections so that its impossible to be subject to external network attacks. Tailscale does part of that, other exist, such as the open source project I work on - https://github.com/openziti
> And then the next best use of your time and resources is to prioritize the fastest possible patching cadence, since the vast majority of attacks target disclosed vulnerabilities.
Just curious, do you leverage any tools to decide when to patch or is it time-interval based? We currently attempt[0] to update our packages quarterly but it would be nice to have a tool alert us of known vulnerabilities so we can take action on them immediately.
[0] "Attempt" meaning we can't always upgrade immediately if the latest version contains difficult-to-implement breaking changes or if it's a X.0.0 release that we don't yet trust
So then you set up something like a cron job to scan everything for you and email the results once a week or whatever if you don't want to monitor things actively.
The simplest is to only use packages from a distribution like Debian and run unattended-upgrades or equivalent for the security-updates repository. They usually fix vulnerabilities in less than a day.
The author says they aren’t a security person, so to correct a minor thing: the first examples are credential and configuration discovery, not directory traversal. The latter, to the best of my knowledge, is reserved for techniques where the attacker “escapes” the webroot or otherwise convinces the server to serve things outside of its normal directories.
The elephant in the room is that–at least in my experience–a lot of these attacks come from hostile nation states. This is going to be controversial, but one may find it useful to block entire IP ranges of problematic states that you cannot do business with. I was able to block 100% of probes to one of my new services by doing this.
American websites don't typically have customers dialing in from North Korea. Just saying that IP blocking is something more businesses should consider. Traffic can also be routed to different subdomains for businesses that need to provide a subset of services to a region.
15+ years ago I was reviewing server logs for small, local businesses we hosted and came to the conclusion we should just block all APNIC IP addresses.
I work in application/product security and have managed WAFs for multi-billion dollar companies for many many years.
Move DNS to Cloudflare and put a few WAF rules on your site (managed challenge if bot score less than 2 / attack score == x). I doubt you'll even pay anything, and it will resolve a lot of your problems. Just test it before moving it to production please (maybe setup a test domain). Remember, a WAF is not an end-all be all, it's more of a band-aid. If you app isn't hardened to handle attacks, no amount of advanced WAF/bot protection will save it.
Selfhoster here. I use mutual TLS rules with CloudFlare's WAF to filter out everyone but my known-good callers. Works great. Since the only folks with access are my family, it was pretty easy to setup as well (everyone gets a unique cert that I can revoke if need be).
Usually I only manage internal facing applications these days, which makes the attack surface greatly reduced compare to public ones.
But since you seem to have a lot of knowledge in this area. Have you manage solutions which also includes infrastructure in Azure combined with Cloudflare?
And if so, any suggestions on things people usually miss? except for the usual stuff of OWASP and what not
Can you please not post in the flamewar style? It's not what this site is for, and destroys what it is for.
You're welcome to make your substantive points thoughtfully but it needs to be within the rules. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
We should be suggesting self hosted and decentralized solutions to website hosting and file hosting.
On that note, does anyone have any secure methods of providing serving a file from your computer to anyone with a phone/computer that doesn't require them downloading/installing something new? Just a password or something? Magic-wormhole almost seems great, but it requires the client to install wormhole (on a computer, not phone), and then type specific commands along with the password.
> Once Cloudflare starts using attestation to block anyone not on Chrome/iOS Safari it'll be too late to do anything about it.
That's just plain bs...
Eg
1) they have customers and their customers want protection, with minimal downsides.
2) Cloudflare is the only one with support for Tor. I'm 100% sure you didn't knew that.
What "examples" do you have to blame them for something they aren't doing? Based on what?
I'm getting tired of people blaming Cloudflare for providing a service that no one else can provide for free to small website owners => DDOS protection.
> Hacker news is so funny, they complain about the amount of power we've allowed Google, Amazon, and Microsoft to have, and then go right around and recommend putting everything behind Cloudflare.
It’s almost as if those saying contradictory things are actually different people despite being on the same website. But it can’t be that, surely? Truly a perplexing phenomenon that I hope someone can one day explain.
Thank you! I've been self-hosting for about a year running a 400-line http/s server of my own design, and it's remarkable all the attacker traffic my 3 open ports (22, 80, 443) get, although I've never taken the time to analyze what the attackers are actually trying to do! This post fills in a LOT of blanks.
Would be cool to do the same thing for the weird stuff I see in /var/log/auth.log!
It's crazy that attackers would bother with me since the code is entirely open source and there is no server-side state. The best outcome for an attacker would be root access on a $5/mo VPS, and perhaps some (temporary) defacement of the domain. A domain no-one visits!
These are all automated bots. No one is “bothering”. You open the 3 most well known ports you’re going to get connections. They don’t know what you’re running nor do they care.
By "bothering with me" I mean "add my IP to the long list of IPs they are scanning".
By the way, I find it annoying that my logs get filled with this kind of trash. It has the perverse effect of making me long for something like Google Analytics since they rarely if ever bother running a javascript runtime.
Access to your VPN is a great way to launch attacks on other machines, and to add another layer to covering his tracks. Not to mention hosting malware to be downloaded to other places, and even a crypto miner.
I set up a honeypot once and logged the passwords of created accounts. I then used 'last' to find the incoming ip.
I then used ssh to try and connect to the originator (from an external box). I went back 5 jumps until I got to a windows server box on a well known hosting service that I could not get into.
Lots of what looked like print servers and what looked like linux machines connected to devices. Maybe just the exploit at the time.
Consider blocking 22 except whitelist your own IP. My ISP changes my IP rarely in practice and when they do I can log into the hosting web admin panel and update the rule.
I make a point of running fail2ban on my servers and will even add custom jails to catch attacks specific to the types of functionality I may be exposing on the site(s) hosted on them. But it’s been a long time since I checked whether fail2ban’s other defaults are still comprehensive enough to block the most common attacks. I guess I’ll bookmark this link for when I get around to doing that.
I also check the access logs collected by my self-hosted services, and I think there's a detail that's conspicuously absent from this analysis: the bulk of these malicious requests are made by regular people running plain old security scanners that are readily available from sites such as GitHub. These are largely unsofisticated attacks that consist of instances of one of these projects just hammering a server without paying any attention to responses or even if they have been throttled or not.
Some attacks don't even target the IP and instead monitor domains and its subdomains, and periodically run the same scans from the exact same range of IP addresses.
For example, on a previous job we had a recurring scan made over and over again from a single static IP address located in Turkey that our team started to refer to it as "the Turkish guy", and our incident response started featuring a preliminary step to identify weird request patterns that was basically checking if it was the Turkish guy toying with our services.
Readit News
I've had several instances of a new server being up on a new IP address for over a week, with only a few random probing hits in access logs, but then, maybe an hour after I got a certificate from Let's Encrypt, it suddenly started getting hundreds of hits just like those listed in the article. After a few hours, it always dies down somewhat.
The take-away is, secure your new stuff as early as possible, ideally even before the service is exposed to the Internet.
Honestly it feels like you'll need at least something like basicauth in front of your stuff from the first minutes it's publicly exposed. Well, either that, or run on your own CA and use self-signed certs (with mTLS) before switching over.
For example, when some software still has initial install/setup screens where you create the admin user, connect to the DB and so on, as opposed to specifying everything initially in the environment variables, config files, or other more specialized secret management solutions.
Just SSH scanning can be a big issue.
I wish. I use basicauth to protect all my personal servers, the problem is Safari doesn't appear to store the password! I always have to re-authenticate when I open the page. Sometimes even three seconds later.
Cloudflare’s Merkle Town[0] is useful for getting overviews, but I haven’t found an easy way to query CT logs. ct-woodpecker[1] seems promising, too
[0] https://ct.cloudflare.com/
[1] https://github.com/letsencrypt/ct-woodpecker
It uses an API provided by https://crt.sh/
I use LE wildcard certs and they're great, you can use them internally.
Renewing a wildcard is also unfun when you have services which require a manual import.
A good starting point for hardening your servers is CIS Hardening Guides and the relevant scripts.
I had planned to make the move over the next day, but I moved a single service over to make sure everything was working. Next day as I'm testing moving traffic over, I find that I've been rate limited by Lets Encrypt for a week. I check the database and I had provisioned dozens of certificates for vpn.api-tools.mycompany.com, phpmyadmin.api-tools.mycompany.com, down the list of anything you can think of.
There was no security issue, but it was very annoying that I had to delay the rollout by a week and add a whitelist feature.
Once you figured out what the domain is, you can easily build a list of IPs out of the cert transparency log and if there is ever an exploit for this specific type of appliances, attackers now have a bespoke list of IPs to hack, a dream come true.
I don't see a solution for this particular use case, I would argue self signed certs would be more secure in this case.
What? Ideally..before? Seriously? It is 2024.. and this was true even decades ago, absolutely mandatory.
(Still remembering that dev that discovered file sharing in his exposed mongo instance (yes, that!! :D) without password only hours after putting it up.. "but how could they know the host it is secret!!" :D ).
Eventually I stopped proactively reviewing logs and stopped paying for the IDS. It was a waste of time and a distraction.
It's not hard to find really useful content that summarizes common vulnerabilities and attacks, and just use that to guide your server management. There are a ton of best practices guides for any common web server technology. Just executing these best practices to 100% will put you way ahead of almost all attackers.
And then the next best use of your time and resources is to prioritize the fastest possible patching cadence, since the vast majority of attacks target disclosed vulnerabilities.
Where logs are super helpful is in diagnosing problems after they happen. We used log analysis software to store and search logs and this was helpful 2-3 times to help find (and therefore address) the root cause of attacks that succeeded. (In every case it turned out to be a known vulnerability that we had been too slow to patch.)
Yes. This is why relying on "patching" is a bound to fail at some point. Maybe it's a 0-day, or maybe the attackers are just quicker.
The solution to this is defence in depth, and it's very easy for most services, especially when self-hosting personal things. Few tips most people can do is.
Put up a firewall in front or put it behind VPN/tailscale.
Hide it in a subfolder. The automated attacks will go for /phpmyadmin/ , putting it in /mawer/phpmyadmin/ means 99.9% of the attackers won't find it. (This is sometimes called security by obscurity and people correctly say you should not rely on it, but as a additional layer it's very useful).
Sandbox the app, and isolate the server. If the attackers get in, make it hard or impossible for them to get anywhere else.
Keep logs, they allow you to check if and how you got attacked, if the attack succeeded and so on.
Depending on the service, pick one or more of these. Add more as necessary.
The key thing is that you should not rely on any ONE defence, be it keeping it patched or firewalled, because they will all fail at some point.
Anyone who thinks “security by obscurity” is useless should try reverse engineering some properly obfuscated executables (or even code). Obscurity is absolutely useful; definitely not a complete solution by itself, but a very useful component to a security solution.
I detest this phrase right up there with "fake it till you make it". All security is by definition obscurity. Just a meaningless platitude that rhymes.
I suppose un-formalized un-proofed security practices will eventually be broken or counting on hackers not to do any investigation of your system will get you hacked, doesn't roll off the tongue though.
Just curious, do you leverage any tools to decide when to patch or is it time-interval based? We currently attempt[0] to update our packages quarterly but it would be nice to have a tool alert us of known vulnerabilities so we can take action on them immediately.
[0] "Attempt" meaning we can't always upgrade immediately if the latest version contains difficult-to-implement breaking changes or if it's a X.0.0 release that we don't yet trust
https://github.com/quay/clair
https://github.com/anchore/grype/
https://github.com/eliasgranderubio/dagda/
https://github.com/aquasecurity/trivy
So then you set up something like a cron job to scan everything for you and email the results once a week or whatever if you don't want to monitor things actively.
I imagine blocking North Korea, Iran, etc will probably impact 0.5% of traffic and 0.001% of revenue for most sites.
> and push determined adversaries to use proxies?
That might mean you're left with 10% of the previous attackers so it could be worth it.
Most of probing to my server comes from USA, and a far second place is Netherlands. Probably most of it comes from AWS and other data centers.
Move DNS to Cloudflare and put a few WAF rules on your site (managed challenge if bot score less than 2 / attack score == x). I doubt you'll even pay anything, and it will resolve a lot of your problems. Just test it before moving it to production please (maybe setup a test domain). Remember, a WAF is not an end-all be all, it's more of a band-aid. If you app isn't hardened to handle attacks, no amount of advanced WAF/bot protection will save it.
Message/email me if you need help.
The Free Managed Ruleset appears to be deployed by default, and Cloudflare keeps a changelog here: https://developers.cloudflare.com/waf/change-log
But since you seem to have a lot of knowledge in this area. Have you manage solutions which also includes infrastructure in Azure combined with Cloudflare?
And if so, any suggestions on things people usually miss? except for the usual stuff of OWASP and what not
I can imagine that might be needed if some company for some reason has to run some not really up to date stuff but yeah it is just a bandaid.
You're welcome to make your substantive points thoughtfully but it needs to be within the rules. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
We should be suggesting self hosted and decentralized solutions to website hosting and file hosting.
On that note, does anyone have any secure methods of providing serving a file from your computer to anyone with a phone/computer that doesn't require them downloading/installing something new? Just a password or something? Magic-wormhole almost seems great, but it requires the client to install wormhole (on a computer, not phone), and then type specific commands along with the password.
Is there a simple `iroh serve myfile.file` from server and then client goes to https://some.domain.iroh/a086c07f862bbe839c928fce8749 and types in a password/ticket you give them?
That would be wonderful.
That's just plain bs...
Eg
1) they have customers and their customers want protection, with minimal downsides.
2) Cloudflare is the only one with support for Tor. I'm 100% sure you didn't knew that.
What "examples" do you have to blame them for something they aren't doing? Based on what?
I'm getting tired of people blaming Cloudflare for providing a service that no one else can provide for free to small website owners => DDOS protection.
It’s almost as if those saying contradictory things are actually different people despite being on the same website. But it can’t be that, surely? Truly a perplexing phenomenon that I hope someone can one day explain.
Would be cool to do the same thing for the weird stuff I see in /var/log/auth.log!
It's crazy that attackers would bother with me since the code is entirely open source and there is no server-side state. The best outcome for an attacker would be root access on a $5/mo VPS, and perhaps some (temporary) defacement of the domain. A domain no-one visits!
By the way, I find it annoying that my logs get filled with this kind of trash. It has the perverse effect of making me long for something like Google Analytics since they rarely if ever bother running a javascript runtime.
I then used ssh to try and connect to the originator (from an external box). I went back 5 jumps until I got to a windows server box on a well known hosting service that I could not get into.
Lots of what looked like print servers and what looked like linux machines connected to devices. Maybe just the exploit at the time.
I also limit concurrent connections, which significantly reduces data usage during aggressive attacks
Some attacks don't even target the IP and instead monitor domains and its subdomains, and periodically run the same scans from the exact same range of IP addresses.
For example, on a previous job we had a recurring scan made over and over again from a single static IP address located in Turkey that our team started to refer to it as "the Turkish guy", and our incident response started featuring a preliminary step to identify weird request patterns that was basically checking if it was the Turkish guy toying with our services.