This type of issue can be incredibly annoying to deal with, because the legitimate answer to the abuse report ("someone is spoofing my IP, it isn't me, and the machine is not compromised") is the exact same excuse that a malicious actor would provide.
Then, as noted in the article, you're trying to prove a negative to someone who doesn't really care at all, which is borderline impossible.
Hertzner says in the email that no response is necessary.
Automated abuse reports of things that are easily spoofed don't justify a report, but might justify a quick check to make sure your box is still operating correctly and hasn't been taken over.
>but we do expect you to check it and to resolve any potential issues.
That's the important part.
If they receive another one (or two, or a few) more abuse reports, they assume it is not fixed, and will expect a response then. Which ends up being annoying.
> the legitimate answer to the abuse report ("someone is spoofing my IP, it isn't me, and the machine is not compromised") is the exact same excuse that a malicious actor would provide.
The legitimate answer would include some sort of real-world attestation about you from a trusted third party. Probably the very least, some evidence of your identity and jurisdiction. Maybe including a video call or something. Not just you anonymously claiming you're a good guy over the internet and expecting to be believed.
If there is technology and established protocols to prevent spoofing, but some ISPs refuse to follow these protocols, why should it be your burden to prove it wasn’t you?
Is it reasonable to allow people to get credit cards with your SSN, when it’s physically possible to confirm their identity when they present your SSN, but the bank is too lazy to do it, and we put it on you to show up and cancel the credit cards? And of course present 3rd party attestation that it wasn’t you who did this. Maybe even bring an alibi?
>The legitimate answer would include some sort of real-world attestation about you from a trusted third party.
It's annoying to find someone (or some service) that is willing to attest on your behalf and have that person (or service) be trusted by your provider more than whoever filed the abuse complaint.
>Maybe including a video call or something.
It's annoying to find someone at your provider who will take the time to do this. It's annoying to take my time to have to do this.
My point, overall, was that this is just a really annoying problem.
> The internet was broken 25 years ago and is still broken 25 years later. Spoofed source IP addresses should not still be a problem in 2024, but the larger internet community seems completely unwilling to enforce any kind of rules or baseline security that would make the internet safer for everyone.
Same with spoofed MAC addresses, email addresses, ARP messages, Neighbor Discovery, MitM TLS certificates ... It's amazing anything works anymore :D
The thing is, obviously, that the Internet isn't broken, it has incredible utility and reliability. If it was designed and operated to be perfect, then it would likely be massively broken quite often. It is the tolerance for mild brokenness that has contributed significantly to its robustness and utility.
That isn't an argument for not improving things though, just a warning against perfection, if you chase it then you're liable to make really big mistakes that ruin everything.
Retaining functionality even in the face of mild-to-moderate borkedness is sorta the inciting goal for even making it in the first place, way back in the cold war days. Building on top of "How do we make a communications network that can handle a bunch of nukes" sets you up for a very resilient baseline :)
Spoofed and Randomized are not the same thing. Spoofing implies you are deliberately copying another machine’s MAC address in order to appear as that machine to the network.
Not especially, but most websites are protected by TLS, so the problem that DNS is insecure is less of a problem. It's mainly a coordination problem, you have up get a lot of people on board to design a new DNS-SECure, and then everyone would also have to adopt it. Which they did (create DNSSEC, that is), but it has not seen the desired adoption. The other one is DoH, DNS over https. It's not without issue either though. So there are efforts, it's just a hairy coordination problem.
It’s quite sad the only mail server out there which checks if you are allowed to use a email address is exchange. With all others you can set the from: header however you like.
Who cares whether it's the MTA that does it or a collection of daemons invoked by the MTA? Just get things configured correctly, and you should be gold.
Now as far as every other mail operator setting up their stuff right such that From spoofing is no longer feasible, well... Can't help ya there. I don't run my email to make money, so the incentive to adopt pathological configs for the sake of maximizing the number of users/Domains who can send from one IP ain't there.
Back in the day I would scan for DrDoS reflectors in a similar way, no hosting provider wants to get reports for port scanning so the source address of the scan would belong to an innocent cloud provider with a reputable IP that reflectors would happily send UDP replies to. The cloud provider would of course get a massive influx of complaints but you would just say that you aren't doing any scanning from your server (which they would verify) and they wouldn't shut your service off. The server sending out the spoofed scan packets is undetectable so you're able to scan the entire internet repeatedly without the typical abuse issues that come with it.
I'm not sure how often this happens in practice but tracing the source of a spoofed packet is possible if you can coordinate with transit providers to follow the hops back to the source. One time JPMorgan worked with Cogent to tell us to stop sending packets with their IP addresses (Cogent is one of the most spoofer friendly tier 1's on the internet btw).
This is the first time I've heard of this being used to target TOR specifically which seems counterintuitive, you would think people sending out spoofed packets would be advocates of TOR. Probably just a troll, luckily providers that host TOR won't care about this type of thing.
This is nothing new. A few years back, I implemented a very basic firewall rule: if I received a TCP packet with SYN=1 and ACK=0 to destination port 22, the source IP would get blacklisted for a day. But then I started getting complaints about certain sites and services not working. It turned out that every few days, I'd receive such packets from IPs like 8.8.8.8 or 1.1.1.1, as well as from Steam, Roblox, Microsoft, and all kinds of popular servers—Facebook, Instagram, and various chat services. Of course, these were all spoofed packets, which eventually led me to adjust my firewall rules to require a bit more validation.
So, I can assure you this is quite common. As a personal note, I know I’m a bit of an exception for operating multiple IP addresses, but I need the flexibility to send packets with any of my source addresses through any of my ISPs. That’s critical for me, and if an ISP filters based on source, it’s a deal-breaker—I’ll switch to a different ISP.
> As a personal note, I know I’m a bit of an exception for operating multiple IP addresses, but I need the flexibility to send packets with any of my source addresses through any of my ISPs. That’s critical for me, and if an ISP filters based on source, it’s a deal-breaker—I’ll switch to a different ISP.
If you actually have your own IP addresses this is normal and expected, but if you're able to use ISP A's IP addresses through ISP B or vice versa that has always been a bug that you are wrong to use.
If you are doing the latter this is firmly in the "reenable spacebar heating" category and I hope your ISPs fix their broken networks.
Okay, looks like I will reply to a few of the comments to clarify things.
I’ll give a concrete, real example.
I worked at a company that hosted some web assets on-prem in one of their branches. They had a 1Gbps connection there. However, at HQ, we had multiple 10G connections and a pretty good data center. So, we moved the web VM to HQ but kept the assigned IP address (a public static from ISP-A). We routed it through a VPN to HQ. The server used our default GW and sent responses with source IP (ISP-A) via ISP-B (10G).
That way, we utilized 10G outbound, even though the inbound was limited to 1G. It was only for GET requests anyway. I know this wasn’t the most optimal setup, and we eventually changed the IP, but it seems like a valid use case.
Scenario 2: We had two connections from two different ISPs (our own ASN, our own /23 addresses). We wanted to load balance some traffic and sent half of our IPs through ISP-A and the other half through ISP-B. It worked fine, but when we tried to mix the balance a bit, we found an interesting glitch. We announced the first /24 to ISP-A and the second /24 to ISP-B, but ISP-A had RP filtering. So, we had to announce all the IPs to them.
The way the RP filter works, as you may guess, means we cannot prepend or anything. All traffic must come through them. If they see a better route for that prefix, they will filter it. For a few months, they refused to fix this, citing security. There’s no shame in security best practices, so I might as well name the ISP—Virgin Media.
Note that the internet with rp_filter is not $20/month. It was more like 5K+/month!! And we did not change it due to lack of alternatives there. But otherwise guess who loses the contract :)
For your second scenario you should announce the /23 to both and each /24 to one of them. Usually you can also prepend your own AS, ISPs I've worked could also prepend for you with select communities.
I don't think your cases are good enough to allow anyone to spoof by default.
In your first scenario, any connections established through the ISP-A's IP address would be routed back through the VPN connection that they came in on. If that server were to establish it's own connections to external resources, it would feasibly be able to use the 10g connection from ISP-B. It would not be able to dictate what source address was used with connections coming from ISP-B.
>As a personal note, I know I’m a bit of an exception ...That’s critical for me, and if an ISP filters based on source, it’s a deal-breaker—I’ll switch to a different ISP.
"...and obviously, Pennywise, I must spoof ingress and egress..."
If it's your real IP, it's not spoofing, even if you send the packet through a different ISP than the one which gave you the IP. If you think about it: if you got an IP directly from ARIN you wouldn't have to send your packets through ARIN to make them legitimate.
Not really. Early IPv6 documentation kind of assumed that the vast address space would lead towards hierarchical addressing and that a multi-homed user would use addresses assigned by all of their ISPs, but at least in my experience, that doesn't really pan out --- if you have router advertisements from two different ISP prefixes, automatic configuration on common OSes (windows, linux, freebsd) will lead towards often sending traffic with ISP A through the router from ISP B, which doesn't really work well, especially if either or both ISPs run prefix filters. There's probably ways to make that style of multihoming work, but it's not fun.
Turns out, most multiphomed IPv6 users need provider indepdent addresses, just like with IPv4. And then you need to make sure your all your ISPs allow you to use all your prefixes. On the plus side, it's much more likely to get an IPv6 allocation that's contiguous and that you won't outgrow; so probably you only need one v6 prefix, and you may not need to change it as often as with v4.
The “someone hates Tor relays” theory doesn’t sound worth the effort. This could be an entity running malicious relays, while also trying to unethically take down legitimate relays to increase the percentage of the network that they control.
This is almost certainly it. There’s a lot of head-sand-burying around here about just how easily an attacker with access to logs of a not-even-that-large segment of the nodes can gain visibility into individuals’ service access patterns.
Yeah. If you hate the tor network an easier thing to do is just to overwhelm it with traffic and degrade the service. Running some bittorrent downloads might be enough.
> Which means, if you just find one transit provider which doesn’t do BCP38 filtering… you can send IP packets tagged with any source IP you want! And unfortunately, even though the origins of BCP38 date back to 1998… there are still network providers 25 years later that don’t implement it.
What would it take to get enough network providers to start rejecting traffic from all ASes that don't implement this, so that spoofing was no longer possible?
Cloudflare is probably enough. They already control enough ingress that their "checking the security of your connection" could actually mean something.
You'd have to find some way to make network providers care. Especially 'tier 1' transit providers and other networks of unusual size.
It's much easier to work on reducing reflection multipliers though, because you can scan (ipv4 anyway) for reflection vectors and yell at people that will respond with 10x the input bytes.
It seems like systems shouldn't report abuse (at least automatically) for single packet, no round trip, requests unless its reaching denial of service levels of traffic (and maybe these are). Like in particular for SSH there's no way thats even a valid connection attempt until some sort of handshake has occurred.
But since anyone can submit an abuse complaint, maybe server providers should actually check the abuse reports before triggering the "respond in 2 days or we suspend your server" or similar measure of their ToS.
I've had my main server thrown offline by a bogus abuse report claiming that they received an over 1Gbps DoS attack from my IP even though my server only has a 400 Mbps cap. Had a human actually read the report, they would've seen it was impossible and wouldn't have had to spend 2 days arguing with phone support on my holiday.
Readit News
Then, as noted in the article, you're trying to prove a negative to someone who doesn't really care at all, which is borderline impossible.
Automated abuse reports of things that are easily spoofed don't justify a report, but might justify a quick check to make sure your box is still operating correctly and hasn't been taken over.
That's the important part.
If they receive another one (or two, or a few) more abuse reports, they assume it is not fixed, and will expect a response then. Which ends up being annoying.
The legitimate answer would include some sort of real-world attestation about you from a trusted third party. Probably the very least, some evidence of your identity and jurisdiction. Maybe including a video call or something. Not just you anonymously claiming you're a good guy over the internet and expecting to be believed.
If there is technology and established protocols to prevent spoofing, but some ISPs refuse to follow these protocols, why should it be your burden to prove it wasn’t you?
Is it reasonable to allow people to get credit cards with your SSN, when it’s physically possible to confirm their identity when they present your SSN, but the bank is too lazy to do it, and we put it on you to show up and cancel the credit cards? And of course present 3rd party attestation that it wasn’t you who did this. Maybe even bring an alibi?
I hope I misunderstood your comment.
It's annoying to find someone (or some service) that is willing to attest on your behalf and have that person (or service) be trusted by your provider more than whoever filed the abuse complaint.
>Maybe including a video call or something.
It's annoying to find someone at your provider who will take the time to do this. It's annoying to take my time to have to do this.
My point, overall, was that this is just a really annoying problem.
Deleted Comment
Nice try Winnie Poo
Same with spoofed MAC addresses, email addresses, ARP messages, Neighbor Discovery, MitM TLS certificates ... It's amazing anything works anymore :D
That isn't an argument for not improving things though, just a warning against perfection, if you chase it then you're liable to make really big mistakes that ruin everything.
You're not copying the MAC of someone else on the network.
Now as far as every other mail operator setting up their stuff right such that From spoofing is no longer feasible, well... Can't help ya there. I don't run my email to make money, so the incentive to adopt pathological configs for the sake of maximizing the number of users/Domains who can send from one IP ain't there.
I'm not sure how often this happens in practice but tracing the source of a spoofed packet is possible if you can coordinate with transit providers to follow the hops back to the source. One time JPMorgan worked with Cogent to tell us to stop sending packets with their IP addresses (Cogent is one of the most spoofer friendly tier 1's on the internet btw).
This is the first time I've heard of this being used to target TOR specifically which seems counterintuitive, you would think people sending out spoofed packets would be advocates of TOR. Probably just a troll, luckily providers that host TOR won't care about this type of thing.
> Probably just a troll
Or someone wanting TOR to be treated like nuclear waste, because it offends their surveillance ops.
So, I can assure you this is quite common. As a personal note, I know I’m a bit of an exception for operating multiple IP addresses, but I need the flexibility to send packets with any of my source addresses through any of my ISPs. That’s critical for me, and if an ISP filters based on source, it’s a deal-breaker—I’ll switch to a different ISP.
If you actually have your own IP addresses this is normal and expected, but if you're able to use ISP A's IP addresses through ISP B or vice versa that has always been a bug that you are wrong to use.
If you are doing the latter this is firmly in the "reenable spacebar heating" category and I hope your ISPs fix their broken networks.
for those that need more context regarding the "reenable spacebar heating" comment
I worked at a company that hosted some web assets on-prem in one of their branches. They had a 1Gbps connection there. However, at HQ, we had multiple 10G connections and a pretty good data center. So, we moved the web VM to HQ but kept the assigned IP address (a public static from ISP-A). We routed it through a VPN to HQ. The server used our default GW and sent responses with source IP (ISP-A) via ISP-B (10G).
That way, we utilized 10G outbound, even though the inbound was limited to 1G. It was only for GET requests anyway. I know this wasn’t the most optimal setup, and we eventually changed the IP, but it seems like a valid use case.
Scenario 2: We had two connections from two different ISPs (our own ASN, our own /23 addresses). We wanted to load balance some traffic and sent half of our IPs through ISP-A and the other half through ISP-B. It worked fine, but when we tried to mix the balance a bit, we found an interesting glitch. We announced the first /24 to ISP-A and the second /24 to ISP-B, but ISP-A had RP filtering. So, we had to announce all the IPs to them.
The way the RP filter works, as you may guess, means we cannot prepend or anything. All traffic must come through them. If they see a better route for that prefix, they will filter it. For a few months, they refused to fix this, citing security. There’s no shame in security best practices, so I might as well name the ISP—Virgin Media.
Note that the internet with rp_filter is not $20/month. It was more like 5K+/month!! And we did not change it due to lack of alternatives there. But otherwise guess who loses the contract :)
I don't think your cases are good enough to allow anyone to spoof by default.
"Of course, Agent Bond."
As someone who always enables rp_filter everywhere... I'm very curious why?
I mean, technically those ISPs would be in violation too. You need your own ASN.
don't we want source based filtering tho? sounds like the problem is a LACK of source based filtering.
Turns out, most multiphomed IPv6 users need provider indepdent addresses, just like with IPv4. And then you need to make sure your all your ISPs allow you to use all your prefixes. On the plus side, it's much more likely to get an IPv6 allocation that's contiguous and that you won't outgrow; so probably you only need one v6 prefix, and you may not need to change it as often as with v4.
What would it take to get enough network providers to start rejecting traffic from all ASes that don't implement this, so that spoofing was no longer possible?
It's much easier to work on reducing reflection multipliers though, because you can scan (ipv4 anyway) for reflection vectors and yell at people that will respond with 10x the input bytes.
I suppose a difference is that they use unaffiliated parties to send the complaint, instead of contacting the authority directly.
I've had my main server thrown offline by a bogus abuse report claiming that they received an over 1Gbps DoS attack from my IP even though my server only has a 400 Mbps cap. Had a human actually read the report, they would've seen it was impossible and wouldn't have had to spend 2 days arguing with phone support on my holiday.