Some of the people I've talked with over the years study things like nuclear weapons arms control or cyberwarfare. The most paranoid of the bunch have resorted to having Virtual Private Servers screen shot websites with headless browsers once it loads and pipe it back to their research machine. I can't remember if it's a table of PNGs or just one big one, but either way it's sent back over a SSH tunnel and when you click the server knows what you're trying to click on and preforms the action for you, and will randomly forward the click to a new VPS.
It's not perfect because the IP blocks make it obvious that it comes from DigitalOcean, AWS, etc, but it's sure better than loading untrusted PDFs or JS locally. Still vulnerable to a network attack, though.
>I generally do not connect to web sites from my own machine, aside from a few sites I have some special relationship with. I usually fetch web pages from other sites by sending mail to a program (see https://git.savannah.gnu.org/git/womb/hacks.git) that fetches them, much like wget, and then mails them back to me. Then I look at them using a web browser, unless it is easy to see the text in the HTML page directly. I usually try lynx first, then a graphical browser if the page needs it
open multiple browser sessions for the user, and randomly choose one of them as the 'result' (but still click on all of them, even if the resultant page isn't viewed).
Isn't that worse, a big brother in the middle watching everything and even doing TLS termination? Unless it's running on a Tor-like distributed system?
This type of tracking seems to assume the user is not bothering to send a fake Referer, e.g. she can just use the URL she is requesting, or just omit the header. One could argue such users are "low-hanging fruit".
Very few websites will vary the response if there is no Referer. Sending it really offers little benefit to the user.
Setting up a "headless" browser also seems like overkill. Firefox 57 and later has a -screenshot command line option which saves a PNG. No need to launch X11 for this to work.
I can't speak to why it was originally defined, but since the Referer [sic] header has existed for decades, many sites depend on it to function. The Smart Referer extension whitelist[1] and bug tracker[2] have several examples.
Believe it or not, there actually exist websites that rely on the Referer header for navigation. The last time I bumped into this was a few years ago, but a local government site refused to work unless my browser sent that header.
Granted, this is probably rare enough that it's safe to disable the header for the vast majority of websites, but it's something to keep in mind.
Beyond what other people mentioned, some sites and frameworks also rely on the Referer header as part of CSRF protection. It's not truly necessary to check, but it's an OWASP recommendation so it seems like a decent number of places implemented it by default.
I recently got the Pyramid Python framework to make it possible to disable Referer-checking for the built-in CSRF protection, but they're still going to keep requiring the header by default: https://github.com/Pylons/pyramid/issues/3508
More discussion about it in these pull requests too:
The new version with it being optional hasn't been released yet, so as of right now almost everyone using Pyramid will still require users to send a Referer header to get past any CSRF checks.
I had an old website hosted under www. When it was decided to build a new website, to preserve the old content, the new site was built without a leading subdomain.
The problem was that chrome cached www as the default for anyone who'd visited the old site, and had started hiding www from the address bar.
I used Caddy to redirect all requests to the subdomain free site unless the request came with a referrer from that site, fixing the caching and allowing for free navigation between and within both the old and new site.
Conversely, if a big browser makes a new default that ends up being the wrong decision, that default might spread to other browsers and things will definitely be broken.
The css value `100vh` meant the height of the viewport of the browser, until it didn't.
I hope there is a light at the end of the tunnel for all of this. It seems like there will always be a cat and mouse effort to be just one step ahead of the other. Like how many websites have those popups now where they ask you to turn off ad-blocking. Intrusive ads and website tracking should both be a problem by default. I guess not all ads can be a problem, but I am unsure if the same could be said about tracking...
We're willing to play the cat and mouse game indefinitely, if that's what it takes. Widely deployed trackers are limited in how fast they can try new tricks. And in practice, we know that ITP is working pretty well to block cross-site tracking: https://daringfireball.net/linked/2019/12/09/the-information...
> Widely deployed trackers are limited in how fast they can try new tricks.
How so? Tracking scripts are often included by a script tag that points at a website. Can’t the code be updated, “deployed” to websites immediately, and take advantage of the relatively slower release cycle of Safari?
This is so prevalent already. Brands disguised as users posting "content" that is mostly just an advert for their brand.
It has got to the point where any time someone posts something that seems to too clearly show a brand name or speaks too highly of a product I suspect its the PR people at work and I downvote it.
I've said before, and I'll say it again, much of the content I want to consume is basically advertising, but the way today's internet works, is that I have to view ads for stuff I don't want in order to see the ads I want to see. And for some reason, people call avoiding this "stealing".
>> Like how many websites have those popups now where they ask you to turn off ad-blocking
Handle them the same way as websites with a cookie or gdpr warning that blocks everything else, vote with your valet by leaving the site and find another site instead
Its funny how our brains have a kind of built in adblocker named banner blindness. There have been a few times I was unable to understand a UI because the important part was rectangle and too prominent so I ignore it entirely without realizing it.
Why can't a browser solve this (except for IP) by simply having an option to not leak any data? Make audio and GL calls constant time, and don't persist anything past the tab / window / site? No fonts or cache reuse beyond the host? No referrers etc.
What's the hard problem here that prevents major browsers from having an option like this?
Firefox has an about:config preference called "privacy.resistFingerprinting" to enable some of Tor's mitigations against fingerprinting. Tor is based on Firefox code and Mozilla merges some of Tor's code changes into Firefox to make updating easier for the Tor team.
The difficult bit is that your browser is programmable and browsers are different across vendors, devices and releases. This means whatever a bad guy can think of as a test can be sent back over the wire, and you can't realistically block sites from sending data back to servers. So long as there are different browsers etc, there will be tests that can differentiate between them.
Currently canvas fingerprinting [1] is a popular option, but there's quite a lot of options for that next thing you can use. Even generic code execution time could be used to an extent. Realistically there is hope at the end of the tunnel, but it's a very long way to go given just how complex of a corner we've painted ourselves into with modern web standards.
While it wouldn't stop purely malicious actors, I personally think it might be easier to address the whole situation on the legislation side rather than with technology. Imagine GDPR, except tracking would be illegal altogether: there will always be actors who will work to bypass it, but the majority would do their best to conform, lest they want to go bust with fines.
Readit News
It's not perfect because the IP blocks make it obvious that it comes from DigitalOcean, AWS, etc, but it's sure better than loading untrusted PDFs or JS locally. Still vulnerable to a network attack, though.
>I generally do not connect to web sites from my own machine, aside from a few sites I have some special relationship with. I usually fetch web pages from other sites by sending mail to a program (see https://git.savannah.gnu.org/git/womb/hacks.git) that fetches them, much like wget, and then mails them back to me. Then I look at them using a web browser, unless it is easy to see the text in the HTML page directly. I usually try lynx first, then a graphical browser if the page needs it
https://stallman.org/stallman-computing.html
Or, just don't use the website if they do this.
I also anticipate someone will do smart diffing on target websites to better auto nuke ads, trackers, etc.
Very few websites will vary the response if there is no Referer. Sending it really offers little benefit to the user.
Setting up a "headless" browser also seems like overkill. Firefox 57 and later has a -screenshot command line option which saves a PNG. No need to launch X11 for this to work.
1. https://gitlab.com/smart-referer/smart-referer/blob/gh-pages...
2. https://gitlab.com/smart-referer/smart-referer/issues?scope=...
Granted, this is probably rare enough that it's safe to disable the header for the vast majority of websites, but it's something to keep in mind.
I recently got the Pyramid Python framework to make it possible to disable Referer-checking for the built-in CSRF protection, but they're still going to keep requiring the header by default: https://github.com/Pylons/pyramid/issues/3508
More discussion about it in these pull requests too:
https://github.com/Pylons/pyramid/pull/3512
https://github.com/Pylons/pyramid/pull/3518
The new version with it being optional hasn't been released yet, so as of right now almost everyone using Pyramid will still require users to send a Referer header to get past any CSRF checks.
The problem was that chrome cached www as the default for anyone who'd visited the old site, and had started hiding www from the address bar.
I used Caddy to redirect all requests to the subdomain free site unless the request came with a referrer from that site, fixing the caching and allowing for free navigation between and within both the old and new site.
This is going to break a lot of things. Things that probably should be broken, but it will cause headaches nonetheless.
The css value `100vh` meant the height of the viewport of the browser, until it didn't.
How so? Tracking scripts are often included by a script tag that points at a website. Can’t the code be updated, “deployed” to websites immediately, and take advantage of the relatively slower release cycle of Safari?
It has got to the point where any time someone posts something that seems to too clearly show a brand name or speaks too highly of a product I suspect its the PR people at work and I downvote it.
Handle them the same way as websites with a cookie or gdpr warning that blocks everything else, vote with your valet by leaving the site and find another site instead
What is meant by cross-site here? Does it mean a different eTLD+1, or a different origin (as used by CORS)?
Specifically, if I make a request from https://www.example.com/path?query to https://api.example.com will the referer header contain the "/path?query"? or will that get blocked as well?
https://www.youtube.com/watch?v=Iw3G80bplTg
Honestly, this shit gets confusing, can someone please ML us out of it? Or maybe we just design a sane and understandable First-Party only policy?
What's the hard problem here that prevents major browsers from having an option like this?
More details in this ghacks article:
https://www.ghacks.net/2018/03/01/a-history-of-fingerprintin...
Currently canvas fingerprinting [1] is a popular option, but there's quite a lot of options for that next thing you can use. Even generic code execution time could be used to an extent. Realistically there is hope at the end of the tunnel, but it's a very long way to go given just how complex of a corner we've painted ourselves into with modern web standards.
While it wouldn't stop purely malicious actors, I personally think it might be easier to address the whole situation on the legislation side rather than with technology. Imagine GDPR, except tracking would be illegal altogether: there will always be actors who will work to bypass it, but the majority would do their best to conform, lest they want to go bust with fines.
[1] https://en.wikipedia.org/wiki/Canvas_fingerprinting