This is a nice idea, but the one thing I haven't been able to divorce myself from is YouTube. I really hate how Google has allowed such a wealth of constant information that completely dwarfs alternative video hosting sites. As censorious as Google can be(now "up next" is always some video from CNN or Fox), blocking YouTube from my network would mean cutting myself off from a large portion of the world.
It’s fascinating how differently the YouTube algorithm treats people. I've not once seen a Fox or CNN video recommended, I didn't even know they had a presence on YouTube
Personalization makes it incredibly hard to "watch the watchers," because everyone is getting a slightly different view of what Google is doing. I would like to see a program where users submitted data about their recommendations to researchers so that we could uncover Google's opinions. It would have a lot of financial value to YouTubers and would make it harder for Google to abuse their role as censor.
I could imagine shadow-burning YouTubers without banning them by shrinking their recommendation audience.
Further, it would be good for Google. Every little shift in the weather is going to get blamed on them whether they deserve it or not, now that it's common knowledge that they weild this power in more than zero cases. Google is about to discover why judges write opinions. Administering justice from secret meetings leads to popular dissent more than it leads to justice.
There is also this project which is similar but much better, more polished, and with more features than what I have currently, but I haven't tried it to see if it supports a similar kind of selective proxying:
https://github.com/omarroth/invidious
Then there's Freetube, which supports proxying but I'm not sure of the details either. It doesn't scrape Youtube itself as far as I know; instead it consults with the main Invidous instance at invidio.us which provides an api:
https://github.com/FreeTubeApp/FreeTube
My experience is if you watch videos on a given topic, they try to show you more of the same topic. So they probably decided you like American cable news.
It can get frustrating when it only recommends a single topic. I might go through a phase where I want to see videos about something specific. The recommendation algorithm will re-enforce that and prevent me from moving on to something else. I found that if I make some effort to watch a lot of videos about other topics, they appear. You can also manually edit your viewing history.
The trouble is that they often take viewing a video as a sign that you’re obsessed with that topic. You click one Flat Earth video to see what the crazy sounds like, and for the next three months half of your recommendations are “Scientists don’t want you to know this!”
I would rather that it would just play the next video by the current channel in reverse chron. Maybe if a channel made multiple videos in the last 24 hours, play that and then play other stuff. Instead, it immediately moves me to cable news if I am watching anything political, even though I never watch cable news voluntarily.
Try invidio.us - it hooks directly to the video feed of youtube, which means no ads, no tracking, reddit comments, your own subscriptions with rss which don't require "hitting the bell" and I just tested it works even when youtube hostname is redirected to localhost in /etc/hosts
YouTube and Maps are one of the few Google services left which are still available over Tor. You can proxy youtube-dl and retain some of your privacy this way.
I've been thinking of setting up a super-tiny (about $15/year) VPS as a youtube-dl proxy for a while now. It's the only Google service that still remains valuable to me; I enjoy channels like Bad Obsession Motorsports and various indie musician channels, and Vimeo just doesn't have enough of that type of content, sadly. I know proxying through a VPS that I pay for doesn't 100% divorce me from Google's watchful eye, but it's enough abstraction that hopefully they don't get enough info to build a profile of the real me.
I've got Little Snitch configured to block Youtube (and most Google services) when my browsers request them, but if there's a YouTube video that's interesting enough to warrant the extra effort, I just switch to my terminal and use youtube-dl to grab it and play it back locally.
What I do, and all the sites I visit (techie) are surprisingly not broken (aside from recaptcha spam):
1) use Firefox with multi-account containers, and disable 3rd party cookies.
2) put youtube in it's own "youtube" container. do not login to that container
3) put all other google stuff in it's own "google" container
If you do that, and don't login to google except in the "google" container it makes it more difficult for google to know who you are on youtube or other non-google sites.
But to make it so they REALLY don't know who you are, you need to do the above plus use a VPN. In my own usage I've discovered that youtube will recommend you videos based on your IP address's recent views if your not logged in.
Video-hosting website alternatives as YouTube are indeed, pretty difficult to use in hope to replace entirely that service.
I’ll say that peertube is going into a great way, but if it actually continues to gain success, it will surely take too many longs.
You can however, if its mainly for telemetry purposes, use something else like invidio.us which i’ve been using alongside it, since i’ve deleted my Google account.
I set up a daily script to download new videos from channels I like using youtube-dl. It works really well, I rarely visit the actual YouTube site anymore.
pleroma (and i think maybe mastodon) provide media proxy that i think work for youtube. so when one person shares a video, one instance serves it to all the other users
The problem is that JS Fonts and other CDNed stuff won't load and websites will hang or work weird - particularly Stackoverflow. Bc it's all over https you can't MITM it and inject your own with OpenWRT/piholes. Decentraleyes (a Firefox browser extension) fixes some of this, but not all. If anyone has any additional suggestions, please let me know (it makes life bearable in China without a VPN)
It is great that you could local cache the top X fonts in Google Fonts and never have to redownload them from Google's CDN. It's just too bad that having fonts locally installed or not can be a signal to trackers or otherwise it would be a lot easier to recommend to everyone to just install larger font banks.
You can create a self-signed certificate for Google domains and trust it on your machines. Then you can MITM. This won't work well if you want to do it at a scale, with a number of 3rd party users, but if the only user is you or your family, it should do the trick.
I use a very similar setup (based on unbound): for Stackoverflow to properly work you need to whitelist ajax.googleapis.com.
> it makes life bearable in China without a VPN
If you're already a firefox user, you might try the "FoxyProxy Standard" extension to selectively bypass the GFW for the domains you need. Friends in China are reporting a varying degree of success with setting up forwarding on Apache (TLS1.3 with padding). Obvs, don't forget to set authentication. Once you're there you can add your own DoH to the mix.
Just this morning I setup a greasemonkey script to rewrite those URLs to a local webserver (things like ajax.googleapis.com serving things like jquery). Pages load faster now too. Very limited, but works in many cases:
// ==UserScript==
// @name localize ajax googleapis
// @version 1
// @grant none
// @run-at document-start
var scripts = document.getElementsByTagName("script");
for (i=0; i<scripts.length; i++) {
var parent = scripts[i].parentElement;
var url = new URL(scripts[i].src);
if (url.host === "ajax.googleapis.com") {
url.host = "ajax.googleapis.com.local";
var newscript = document.createElement("script");
newscript.type="text/javascript";
newscript.src = url;
parent.insertBefore(newscript, scripts[i]);
parent.removeChild(scripts[i]);
console.log("Rewrote url as " + url);
}
}
// ==/UserScript==
EDIT: I just read the other comments and installed decentraleyes. I'm sure it's way better than this grease I just posted.
I’ve added your suggestion to our list, and I will try to see if I can make, a separate list mainly for those dedicated web services. For myself, I always block all JS fonts, and CDN domains, and I think, only really use Decentraleyes for that (Or LocalCDN as an alternative), and most of the time, It's usable, but not on the few cases when no content at all is being pulled from those domains.
I find Google Container to be an excellent plugin to segregate my Google account from the rest of my browsing. It's not an official plugin from Mozilla, but it is forked from the Facebook container plugin.
Same here - been using Google Container for 6+ months now and very happy with it. Highly recommended - you can do this yourself with just normal containers in Firefox, but this comes preconfigured with all the non-obvious domains you might not know about. No connection - just a satisfied user.
Only problem with it is now reCAPTCHA sites are a huge pain to use since you have to answer about 15 challenges before you can get (since you look totally unknown to Google outside of the container). It is often better to just ignore these sites now, but it is not always possible.
I just use Firefox containers. It puts every site into its own container. Then I made a "Google" container so that at least my login will hold across the various Google services.
Check into Containerise. You can setup wildcards and get the same effect with a lot more bad actors. Facebook properties are the worst in my opinion, with Google being a close second.
Another approach is whitelisting. Like a default firewall rule of "block all" and a set of specific exceptions, I find this approach can be easier to manage. Probably not going to work for everyone but works for me.
Figure out what domains I need to access for the content I am after[1] and just allow those. "Block" everything else. For example, I might need something like .googlevideo.com once in a while but I will never need something like googletagmanager.net.
1. To do this, I just go through the logs of a local authoritative nameserver that I run solely for this purpose, i.e. collecting lists of needed domains. Then I add the necessary DNS data to /etc/hosts or another local authoritative server, e.g., tinydns. I believe unbound or pdns_recursor can serve static data as well.
Does the author mention avoiding using Google as a third party DNS service. In the beginning, PiHole, i.e., preconfigured dnsmasq, was pointed at some third party DNS service, maybe Google. Not sure what the default configuration is today. If it was Google, then is there any irony in that a project designed to blocks ads is by default having its users send their IP and ISP location to an advertising company probably hundreds if not thousands of times over in a single day of web use.
>Another approach is whitelisting. Like a default firewall rule of "block all" and a set of specific exceptions, I find this approach can be easier to manage.
I tried the whitelisting approach but quickly found out this breaks many websites with shopping cart and credit-card checkouts because they use payments api gateways. Because the url for the card processing gateway is a different company from the ecommerce site you're visiting, it has a totally different spelling so you can't predict what to put in a whitelist beforehand. In turn, if you do whitelist the payment gateway url, you might then find out it makes another api call to a fraud detection url which is another totally different url that you didn't know you had to whitelist.
Whitelisting DNS entries is workable for use inside of a single virtual machine that deliberately restricts a web browser to access a few websites like youtube.
However, I don't see how it's possible to use the whitelisting strategy on a PiHole that globally filters the entire family accessing it with multiple desktops and smartphones. It's not easy to tell if a spinning hourglass or beachball is happening because the a website is slow or whether the whitelist is missing some url entries. The family members would constantly be visiting new and legitimate urls so it seems very cumbersome to try and keep up with adding new whitelist entries for everybody.
For commercial web use, I use a DNS cache just like the website creator would expect; I use a popular browser in these instances, too. Nothing out of the ordinary. For exactly the reason you mention. If something goes wrong I want to be able to say I am the "typical user", not an enlightened one.
However, I rarely use the web for commercial purposes. Almost all use is non-commercial.
I do not use a Pi-Hole. I do like dnsmasq. I prefer djbdns. I use older hardware running Net/OpenBSD as routers and newer hardware running OpenWRT.
I also do not use popular graphical browsers much. I probably would not use whitelisting if I was doing all web use via a popular graphical browser. I reasonably consistent speed across all websites by using text-only browsers and tcp/http clients.
Cannot really speak for other users. Everyone is different. For me, whitelisting works well.
I have been running a pi-hole server at my home for almost a year. We have, at times, around thirty devices on our network, (thermostat (non-nest), several Google Home devices, numerous phones, 4 desktops, 4 laptops, 3 ipads, 1 TV, a chromecast/roku/firestick, a few smart receptacles, and a Xfinity modem) and sometimes the traffic is pretty neat to examine. Its interesting to see which devices phone home.
Whenever a necessary site is blocked it only takes a few seconds to whitelist it. I can also easily blacklist sites. The GUI is very easy to access and use. We have never had an issue with YouTube (YT premium) or anything else really, but occasionally a link will be blocked because of Google or other ad traffic. This has never happened with YT or any other streaming services.
One thing to remember is VPN traffic ignores the Pi-Hole server. Even when the router/computer/device DNS is set to use it. This has never been an issue for us, as only a handful of devices here are using VPN, but I suppose it could be under the right circumstances, but easily fixable.
Yeah, It's due to myself being french, so I proposed including this in the readme.md as GAFAM, as other more ¨international¨ depiction of these group were less prominent. If you think that FAANG is a better acronym, i’ll change it.
It's not a shortcut or anything, those are just Google's SPF records (which as noted above are drastically incomplete, they are only the IPs from which Google sends email).
Has anyone actually used this? Does the web become completely unusable? I suspect blocking their fonts and their CDN for jquery would be enough to make most of the web unusable.
I can, but getting my whole house to use it including the iPhones may be a bit tough (this is a Pi-hole add on so it needs to work without device changes)
Well, myself actually. What i habitually do in those case, when this is really not usable, I just usually temporary whitelist them only for this specific domain i’m trying to access. This break obviously the purpose of the filter list partially tho.
Readit News
I could imagine shadow-burning YouTubers without banning them by shrinking their recommendation audience.
Further, it would be good for Google. Every little shift in the weather is going to get blamed on them whether they deserve it or not, now that it's common knowledge that they weild this power in more than zero cases. Google is about to discover why judges write opinions. Administering justice from secret meetings leads to popular dissent more than it leads to justice.
It means it is the most representative of what would be a common YouTube experience.
There is also this project which is similar but much better, more polished, and with more features than what I have currently, but I haven't tried it to see if it supports a similar kind of selective proxying: https://github.com/omarroth/invidious
Then there's Freetube, which supports proxying but I'm not sure of the details either. It doesn't scrape Youtube itself as far as I know; instead it consults with the main Invidous instance at invidio.us which provides an api: https://github.com/FreeTubeApp/FreeTube
It can get frustrating when it only recommends a single topic. I might go through a phase where I want to see videos about something specific. The recommendation algorithm will re-enforce that and prevent me from moving on to something else. I found that if I make some effort to watch a lot of videos about other topics, they appear. You can also manually edit your viewing history.
1) use Firefox with multi-account containers, and disable 3rd party cookies.
2) put youtube in it's own "youtube" container. do not login to that container
3) put all other google stuff in it's own "google" container
If you do that, and don't login to google except in the "google" container it makes it more difficult for google to know who you are on youtube or other non-google sites.
But to make it so they REALLY don't know who you are, you need to do the above plus use a VPN. In my own usage I've discovered that youtube will recommend you videos based on your IP address's recent views if your not logged in.
Very rarely have any news outlet.
Dead Comment
Would automatically remap to
https://cdnjs.cloudflare.com/ajax/libs/jquery/3.4.1/jquery.m...
Deleted Comment
Can you point to or write up a blog post with a proof of concept?
> it makes life bearable in China without a VPN
If you're already a firefox user, you might try the "FoxyProxy Standard" extension to selectively bypass the GFW for the domains you need. Friends in China are reporting a varying degree of success with setting up forwarding on Apache (TLS1.3 with padding). Obvs, don't forget to set authentication. Once you're there you can add your own DoH to the mix.
Deleted Comment
https://addons.mozilla.org/en-US/firefox/addon/google-contai...
Only problem with it is now reCAPTCHA sites are a huge pain to use since you have to answer about 15 challenges before you can get (since you look totally unknown to Google outside of the container). It is often better to just ignore these sites now, but it is not always possible.
Figure out what domains I need to access for the content I am after[1] and just allow those. "Block" everything else. For example, I might need something like .googlevideo.com once in a while but I will never need something like googletagmanager.net.
1. To do this, I just go through the logs of a local authoritative nameserver that I run solely for this purpose, i.e. collecting lists of needed domains. Then I add the necessary DNS data to /etc/hosts or another local authoritative server, e.g., tinydns. I believe unbound or pdns_recursor can serve static data as well.
Does the author mention avoiding using Google as a third party DNS service. In the beginning, PiHole, i.e., preconfigured dnsmasq, was pointed at some third party DNS service, maybe Google. Not sure what the default configuration is today. If it was Google, then is there any irony in that a project designed to blocks ads is by default having its users send their IP and ISP location to an advertising company probably hundreds if not thousands of times over in a single day of web use.
I tried the whitelisting approach but quickly found out this breaks many websites with shopping cart and credit-card checkouts because they use payments api gateways. Because the url for the card processing gateway is a different company from the ecommerce site you're visiting, it has a totally different spelling so you can't predict what to put in a whitelist beforehand. In turn, if you do whitelist the payment gateway url, you might then find out it makes another api call to a fraud detection url which is another totally different url that you didn't know you had to whitelist.
Whitelisting DNS entries is workable for use inside of a single virtual machine that deliberately restricts a web browser to access a few websites like youtube.
However, I don't see how it's possible to use the whitelisting strategy on a PiHole that globally filters the entire family accessing it with multiple desktops and smartphones. It's not easy to tell if a spinning hourglass or beachball is happening because the a website is slow or whether the whitelist is missing some url entries. The family members would constantly be visiting new and legitimate urls so it seems very cumbersome to try and keep up with adding new whitelist entries for everybody.
However, I rarely use the web for commercial purposes. Almost all use is non-commercial.
I do not use a Pi-Hole. I do like dnsmasq. I prefer djbdns. I use older hardware running Net/OpenBSD as routers and newer hardware running OpenWRT.
I also do not use popular graphical browsers much. I probably would not use whitelisting if I was doing all web use via a popular graphical browser. I reasonably consistent speed across all websites by using text-only browsers and tcp/http clients.
Cannot really speak for other users. Everyone is different. For me, whitelisting works well.
Whenever a necessary site is blocked it only takes a few seconds to whitelist it. I can also easily blacklist sites. The GUI is very easy to access and use. We have never had an issue with YouTube (YT premium) or anything else really, but occasionally a link will be blocked because of Google or other ad traffic. This has never happened with YT or any other streaming services.
One thing to remember is VPN traffic ignores the Pi-Hole server. Even when the router/computer/device DNS is set to use it. This has never been an issue for us, as only a handful of devices here are using VPN, but I suppose it could be under the right circumstances, but easily fixable.
Never seen it listed out like that, I thought it was FAANG. Or is FAANG only used in reference to top salaries in the Bay Area?
> dig TXT +short _netblocks{,2,3}.google.com | tr ' ' '\n' | egrep "(ip4:|ip6:)"
Gives you a full list of all of Google's IP blocks. You can just blackhole those.
Another method is using GeoIP's ASN database, but they also run many ASNs so it would require a little effort to ensure you have them all
Deleted Comment
I tried .apple.com, .yahoo.com, etc. and got nothing.