Even for malware, hiding from Activity Monitor would be quite a feat. Short of an exploit, you couldn't hide your process without a kernel rootkit, but macOS has required user approval to load kernel extensions for several versions now. I suppose you could go the low-tech way and just name your process "WindowServer" to confuse the user, but you'd still end up with two WindowServers.
The idea that Chrome's auto-updater is doing this is ludicrous.
For those of you who are unaware, Comex is one of the most respected security researchers of all time and has done extensive research into Chrome and won Pwn2Own multiple times. He is completely correct, keystone is just Chrome's auto updater. The technical content of this article is super thin. This shouldn't be on the top of hacker news.
I remember quite well the Google update process being notorious for evading Little Snitch firewall rules. It popped up every intervall as if it were a new (unseen) process. There's definitely something going on with the app signature. (Haven't used Chrome for some time by now.)
I am curious why that happens. From some Googling it seems like the updater copies itself to a random directory each time it runs [1], and I guess Little Snitch classifies programs by their full path, so it sees it as a new program each time.
Copying itself is odd and probably suboptimal behavior, but it is explainable without assuming malice: my guess is that it’s related to some kind of “before we update let’s update the updater” bootstrapping logic. I could be wrong, but IIRC the updater code is open source, so it should be easy to find out.
[Edit: Oops, it's actually not open source. Their Windows updater (Omaha) is open source, but their Mac one (Keystone) is not. Of course, one can still open the binaries in a decompiler.]
Regardless, moving a program around on disk, or even deleting the program while it’s running (which is possible on macOS), would not prevent it from showing up in Activity Monitor.
While it's indeed unlikely that there is deliberately hiding, there is a chance that the google updater triggers something in some (OS-level) components via some (implicit) IPC mechanism that causes load spikes in those components. You'd hardly see any load for the real culprit itself but those other connected components may run hot.
This user says their WindowServer runs hot. Some Chrome-related software may have entered a state where it accidentally DoS'es the WindowServer either due to a bug in google software or a bug in WindowServer that google software triggers, at least in the system configurations particular to this user.
True, that much is possible. But fairly unlikely, especially since Chrome's updater normally does not even pop up a GUI. We'd need stronger evidence than "I deleted some Google stuff and rebooted and now feels faster". I suppose the author did claim to have reproduced it on two computers, but only once each, and with no objective performance measurements. Sadly, there are a lot of things that can make macOS 'randomly' seem slower or faster, especially after rebooting, and with subjective measurements, confirmation bias is a huge factor.
I’ve been having WindowServer problems for month taking almost 100% of cpu out of the blue. I never could find any solutions and believe me it’s really painful when working when my IDE becomes so slow if takes seconds for every keystroke. So far it seems to fix my problem and I have the same computer setups. I switched to Brave to get back the Dev Tools.
I don’t have a day to take off to analyze the my system deeply without knowing where to start so in my case it’s a welcome fix
Afaik you don't need to hide from the monitor to cause load by WindowServer while staying silent yourself. You just need to do graphics-intensive stuff. WindowServer then shows up as the manifestation of graphics being processed.
In fact, Un-googled Chromium (without the autoupdater) easily causes load that's displayed as WindowServer's, for me—when I open a Youtube video on my oldish Macbook with a shitty integrated video card. However there's little chance that I would think this to be caused by something else.
I may have got it wrong, but I interpreted the page to say only that Chrome and/or its updater made Window Server work hard, not that they're impersonating it.
Chrome does have the capacity for screen sharing (e.g. Chrome Remote Desktop), but I assume it wouldn't inject any extensions into the window server unless it was actively invoked.
The author went so far as to create a domain for this rant, and yet didn't even do any performance metrics or basic forensic analysis at all. I'm not saying the author is wrong, but there's a lot of questions here. How does Keystone hide itself from the Activity Monitor? Is there a Keystone process if you run `ps aux`? Did you run dtruss/strace? What is it doing when Chrome isn't running that causes it to degrade performance?
Because this sounded outlandish, I decided to have a look on my own system. Since the author didn't give any technical details, I grepped `~/Library` and `/Library` for files containing `keystone` in either name or contents. This yielded the following interesting candidates, along with similar entries for daemons etc:
They all reference the same binary, `~/Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle/Contents/Resources/GoogleSoftwareUpdateAgent.app/Contents/MacOS/GoogleSoftwareUpdateAgent`, and frankly are wholly uninteresting configuration files to integrate with launchd and other stuff.
Looking through a full system trace in Instruments didn't yield anything interesting either (outside of discovering that I forgot to shut down one of my virtual machines).
All in all, this sounds like hot air, and I can't help but wonder what the motivation behind making this page was.
Thank you for digging in to it on your machine. That level of forensics is exactly what we need in this issue.
Because there is a rich history of issues that only affect certain configurations, and those configurations are typically only revealed in hindsight: your results have to be seen as a single datapoint and not a nail in the coffin.
To others who are curious about this: please contribute more datapoints
Another thing you can do: profile the process to find out what exactly it’s spending all that CPU time on. There are a few different ways to do this on macOS, including the GUI Instruments.app which comes with Xcode, and dtrace. But if you’re the sort who always has a terminal open, the quickest way is to just run
sudo spindump
Also useful when a process is hung, even if it’s not taking up CPU time.
Sometimes the output isn’t useful due to a binary having been stripped of symbol names, but all macOS system libraries have symbols, so you can usually figure out something.
Speaking of the GUI Instruments.app. Is there a way to start a process from there, instead of attaching to an already running process? I used it the other day and could only figure out how to attach to an already running process with it.
> The author went so far as to create a domain for this rant
It's not a 'far' thing to purchase a domain anymore. Domains are cheap as chips, and have gotten cheaper now since the advent of novel generic TLDs where usually the first year is sold cheap but to roll it over into the next few years the price is hiked (which is why so many people now purchase a domain for a year only).
It is common now and expected to see single-serving-sites[0] that are just thinly veiled blogposts.
Yeah. After reading your comment I tried out a couple of similar domains on the form x is bad and x is good and they were both parked waiting for someone to pay whatever ridiculous price the owner is wanting to demand to sell it. I wish it was illegal to do that.
Hating Google is very fashionable on HN these days, I'm not surprised to see this so highly voted. I also wouldn't be surprised if most of the upvotes are based on the title/domain rather than the actual "content".
I'm not low-level enough to confirm or deny the specific claims in the article.
I will say that my experience with regards to CPU performance "leaks" and Chrome seeming to always be running in the background even when I didn't want it to, as well as Chrome starting up on system startup when there didn't seem to be any references to it anywhere I'd expect to cause that, are consistent with the article.
I saw a lot of "low-level integration" with my system going on, a lot more than I understood, ever wanted or asked for, and there was no way I knew of to turn it off. It was like IE on Windows all over again.
I was also able to solve the issues by removing Chrome from my systems.
I sometimes install it temporarily to do Web testing and remove it shortly afterwards, but I think I'll do that in a VM from now on.
Chrome can/does definitely run in the background if you have chrome Remote Desktop enabled. It wouldn’t surprise me if they had a daemon always running in case you want to enable Remote Desktop.
I have a standard install of chrome on big sur, as well as brave, and a bunch of electron apps. When I start up my mac none of these are running, there is no process with chrome or chromium in the name, and nothing in the process list that I can link back to chrome. When I launch some of the electron apps, chrome_crashpad_handler launches, which I assume is some kind of default electron behavior. When I close those apps, all chrome-related processes disappear from my process list. There is nothing untoward about chrome I can tell, nor is my system seeing any kind of recognizable slowdown, with or without chrome running. I also have chrome on very low-end macs, a 2009 mini with 8 GB ram and a 2014 air with 4 GB RAM, both running catalina. It runs fine on both, without seeming to cause particular performance issues. I just went through a reinstall of the 2009, and there was no performance difference before and after installing chrome.
I'm not saying yours or other people's bad experiences with chrome are not real, but I do wonder how it's possible that there are such very different experiences out there with what is ostensibly the same product. Maybe it's not just one cause: bad extensions, broken profiles, old installs with broken updaters (I had a broken microsoft updater causing havoc for a while), badly configured enterprise deployments, etc...
On mac OS, Activity Monitor itself causes WindowServer CPU usage to spike. This is the "observer effect".
To test, just open a terminal and compare a few samples of `ps aux | grep WindowServer` with Activity Monitor open and with it closed.
Mine averages 7-11% with Activity Monitor closed and 20-40% with Activity Monitor open. It's even more noticeable if your refresh rate is set to "Very Often". Closing Activity Monitor brings the WindowServer CPU back to normal.
As other have noted, this site provides no evidence. The keystone daemon is not running in the background constantly. When it does run, it's plainly visible and not "hiding itself". According to the launchd job config located at `/Library/LaunchAgents/com.google.keystone.agent.plist` my keystone process is set to run itself every 3623 seconds. Probably randomized to avoid the thundering herd problem. When it does run it checks for updates, often downloading new ones, and then quits.
This is because since macOS Catalina Activity Monitor did something to itself to make itself spend half a core just updating its table view and literally nobody who works on it (which I suspect is <1 person) understands that a monitoring tool that uses more CPU than the thing it is meant to monitor is absurd. I have seriously considered rewriting it multiple times and if the app stays like it is one day I probably will.
Please do! I like Mac, but I yearn a quality of 1st and 3rd party Windows tools. Task manager is nicely readable and lightweight, and process explorer is a godsend.
Meanwhile Activity Monitor updates once per 30 seconds when the system is under load.
This should not be the #1 story on HN right now. :(
It's a single anecdote that deleting Chrome on two computers sped them up. It provides zero evidence (even anecdotal) that it has anything to do with "Keystone" specifically. It provides zero evidence for the idea that "Keystone" is able to "nefariously hide itself from Activity Monitor". And it also completely contradicts the normal user experience of Chrome, which is that most people's computers don't slow down after a Chrome install. (WindowServer on my MBP usually uses <10% CPU and I've used Chrome for many, many years.)
Why is this nonsense being upvoted? I get people dislike Google and Chrome, but wouldn't it be better to stick to things backed by real evidence?
Indignation attracts upvotes, then flags. When an indignant post is relatively empty or misleading, flags will eventually win, but it's like the immune system—it takes some time to marshall enoguh white blood cells.
This is a nice case because moderators only saw it after the process was done.
Hundreds of people commented and how many tens of thousands read it first though? Reminds me of the moderation problem big tech companies face. Humans are prone to confirmation bias which means that when they read inaccurate information that confirms a bias it even further twist people's brains. For example, this article likely twisted thousands of people just a little further against Google even with though there was no good reason for it.
My grandpa twenty years ago had some legitimate gripes with the other side of the political aisle. However, he has seen so many biased and false news stories that his confirmation bias has accepted as truth without critical thinking that he now believes the other side is literally made up of evil criminals intent on destroying the country.
The only solution I can think of is something like a libel law that makes anyone who publishes false information liable to anyone who reads that information... Basically, cut out the enormous amount of garbage produced by media and people chasing pennies. People should only publish things they firmly believe based on evidence to be true.
My WindowServer usage was over 90% before but it went back down to under 10% after I did what he said. I also only needed it to test one site I was developing. I had the exact same things he did: MBP slow even when nothing is running, WindowServer always running high. So I think I can vouch for it being true. It seems to have worked for me also.
EDIT: Maybe it's just placebo. I checked the CPU usage on WindowServer before and after deleting & restarting.
Maybe you are right. A lot of comments here are complaining about the post in terms of scant evidence. Someone could pretty easily myth-bust this, but I have not seen that in the comments here yet.
At the same time even "hard" evidence would likely get dismissed as anecdotal, and there's certainly enough of it now (and even plenty in the past) to point a clear finger at Chrome/Keystone.
This certainly beat filing it in the black hole that is Chromium's bug reporter where it would have been ignored / works-on-my-machine'd / or dismissed as anecdotal there.
Whatever it is doing is sketchy and causing WindowServer to thrash. And this is not the first sketchy thing it has done.
I will say that it seemed like this was a pretty big story to be not be covered in the tech press. I do wonder if there really is much evidence.
On the other hand, I generally feel that Chrome is garbage that slows computers down, so I wouldn't be surprised. But I don't see enough evidence here.
To make an argument of this scale you have the burden of proof. There is no proof here past casual correlation.
An analogy I see here is the scene from an old film where a woman walks behind a wooden screen and a goose walks out thus the woman is a shape shifting witch.
Oh this is normal. People upvote the title, then comment, then read it if the comments aren’t agreeing with their preconception. Humans are a mess of a species.
Many of us believe, with some justice, the causalization of computers has been to the diminishment of power users.
Therefore chrome bad gets hundreds of upvotes. I guarantee you not everyone read the entire article, and then ran their own diagnostic CPU benchmarking to verify the results, and also compared it against chrome competitors.
It's more like the common folk like chrome, and chrome ruin PCS for us
This is why dates on articles is so important. I can’t tell if this website is newer or older than that response so I can’t tell if this is a problem today or was a problem that was resolved
I've had noticeable issues with "WindowServer" in the past (taking 80% CPU usage and causing the fans to ramp up even when the system seemed like it should be idling even after a reboot) so I decided to give this a shot.
This is anecdotal but there is a noticeable difference for me with responsiveness after following these steps. Currently the WindowServer process is hovering around 14%.
Most likely the issue was resolved by the reboot, not by uninstalling Chrome. From my experience Mac OS tends to slow down after a few days. Animations and scrolling gets choppy and there's nothing I can do to fix it, short of rebooting.
I can assure you that it wasn't related just to rebooting as this has been something that's been bugging me for months. I had previously rebooted earlier in the day because of this exact same issue (a runaway WindowServer process), before I saw this article. I also did a fresh reboot before following these instructions too as a sort of "control".
The change only happened for me after removing Chrome as well as the launch agent for keystone.
I use macOS daily for months at a time without rebooting and have not seen this problem on any of the 5 Macs I've been using for the past several years, for what that's worth. It does seem like it's something specific to your machine.
(I have recently started using Apple tools for developping with Swift/Xcode on iPad. I am very unimpressed by the quality of the tools, and quite impressed by the range and depth of features....)
Readit News
Even for malware, hiding from Activity Monitor would be quite a feat. Short of an exploit, you couldn't hide your process without a kernel rootkit, but macOS has required user approval to load kernel extensions for several versions now. I suppose you could go the low-tech way and just name your process "WindowServer" to confuse the user, but you'd still end up with two WindowServers.
The idea that Chrome's auto-updater is doing this is ludicrous.
Copying itself is odd and probably suboptimal behavior, but it is explainable without assuming malice: my guess is that it’s related to some kind of “before we update let’s update the updater” bootstrapping logic. I could be wrong, but IIRC the updater code is open source, so it should be easy to find out.
[Edit: Oops, it's actually not open source. Their Windows updater (Omaha) is open source, but their Mac one (Keystone) is not. Of course, one can still open the binaries in a decompiler.]
Regardless, moving a program around on disk, or even deleting the program while it’s running (which is possible on macOS), would not prevent it from showing up in Activity Monitor.
[1] http://www.reecefowell.com/2012/11/16/ksfetch-annoyance-on-m...
This user says their WindowServer runs hot. Some Chrome-related software may have entered a state where it accidentally DoS'es the WindowServer either due to a bug in google software or a bug in WindowServer that google software triggers, at least in the system configurations particular to this user.
In fact, Un-googled Chromium (without the autoupdater) easily causes load that's displayed as WindowServer's, for me—when I open a Youtube video on my oldish Macbook with a shitty integrated video card. However there's little chance that I would think this to be caused by something else.
Looking through a full system trace in Instruments didn't yield anything interesting either (outside of discovering that I forgot to shut down one of my virtual machines).
All in all, this sounds like hot air, and I can't help but wonder what the motivation behind making this page was.
Because there is a rich history of issues that only affect certain configurations, and those configurations are typically only revealed in hindsight: your results have to be seen as a single datapoint and not a nail in the coffin.
To others who are curious about this: please contribute more datapoints
Why would you immediately jump to assuming bad faith?
No grand conspiracy required... the author believes that Chrome is slowing down his Mac and decided to share it. The end.
Sometimes the output isn’t useful due to a binary having been stripped of symbol names, but all macOS system libraries have symbols, so you can usually figure out something.
Many years back I couldn't sleep and noticed my HDD's was working hard. I checked and it was google updater that was scanning everything...
I will never know why, they might be stealing IP (i was working on OCR at the time) or they just wanted to create a profile to serve better ads...
It's not a 'far' thing to purchase a domain anymore. Domains are cheap as chips, and have gotten cheaper now since the advent of novel generic TLDs where usually the first year is sold cheap but to roll it over into the next few years the price is hiked (which is why so many people now purchase a domain for a year only).
It is common now and expected to see single-serving-sites[0] that are just thinly veiled blogposts.
[0] https://en.wikipedia.org/wiki/Single-serving_site
If you already have payment integrate, adding a domain to a blog is a few clicks - if even.
I, too, was hoping for something, well, technical. Considering #1 on hacker news...
It's nearly an exact match of the meme like utterance often seen on reddit: "chrome bad".
Dead Comment
I'm not low-level enough to confirm or deny the specific claims in the article.
I will say that my experience with regards to CPU performance "leaks" and Chrome seeming to always be running in the background even when I didn't want it to, as well as Chrome starting up on system startup when there didn't seem to be any references to it anywhere I'd expect to cause that, are consistent with the article.
I saw a lot of "low-level integration" with my system going on, a lot more than I understood, ever wanted or asked for, and there was no way I knew of to turn it off. It was like IE on Windows all over again.
I was also able to solve the issues by removing Chrome from my systems.
I sometimes install it temporarily to do Web testing and remove it shortly afterwards, but I think I'll do that in a VM from now on.
Chrome itself would run on system startup.
I no longer use that system, and don't remember the details well enough.
I have a standard install of chrome on big sur, as well as brave, and a bunch of electron apps. When I start up my mac none of these are running, there is no process with chrome or chromium in the name, and nothing in the process list that I can link back to chrome. When I launch some of the electron apps, chrome_crashpad_handler launches, which I assume is some kind of default electron behavior. When I close those apps, all chrome-related processes disappear from my process list. There is nothing untoward about chrome I can tell, nor is my system seeing any kind of recognizable slowdown, with or without chrome running. I also have chrome on very low-end macs, a 2009 mini with 8 GB ram and a 2014 air with 4 GB RAM, both running catalina. It runs fine on both, without seeming to cause particular performance issues. I just went through a reinstall of the 2009, and there was no performance difference before and after installing chrome.
I'm not saying yours or other people's bad experiences with chrome are not real, but I do wonder how it's possible that there are such very different experiences out there with what is ostensibly the same product. Maybe it's not just one cause: bad extensions, broken profiles, old installs with broken updaters (I had a broken microsoft updater causing havoc for a while), badly configured enterprise deployments, etc...
To test, just open a terminal and compare a few samples of `ps aux | grep WindowServer` with Activity Monitor open and with it closed.
Mine averages 7-11% with Activity Monitor closed and 20-40% with Activity Monitor open. It's even more noticeable if your refresh rate is set to "Very Often". Closing Activity Monitor brings the WindowServer CPU back to normal.
As other have noted, this site provides no evidence. The keystone daemon is not running in the background constantly. When it does run, it's plainly visible and not "hiding itself". According to the launchd job config located at `/Library/LaunchAgents/com.google.keystone.agent.plist` my keystone process is set to run itself every 3623 seconds. Probably randomized to avoid the thundering herd problem. When it does run it checks for updates, often downloading new ones, and then quits.
Meanwhile Activity Monitor updates once per 30 seconds when the system is under load.
It's a single anecdote that deleting Chrome on two computers sped them up. It provides zero evidence (even anecdotal) that it has anything to do with "Keystone" specifically. It provides zero evidence for the idea that "Keystone" is able to "nefariously hide itself from Activity Monitor". And it also completely contradicts the normal user experience of Chrome, which is that most people's computers don't slow down after a Chrome install. (WindowServer on my MBP usually uses <10% CPU and I've used Chrome for many, many years.)
Why is this nonsense being upvoted? I get people dislike Google and Chrome, but wouldn't it be better to stick to things backed by real evidence?
This is a nice case because moderators only saw it after the process was done.
My grandpa twenty years ago had some legitimate gripes with the other side of the political aisle. However, he has seen so many biased and false news stories that his confirmation bias has accepted as truth without critical thinking that he now believes the other side is literally made up of evil criminals intent on destroying the country.
The only solution I can think of is something like a libel law that makes anyone who publishes false information liable to anyone who reads that information... Basically, cut out the enormous amount of garbage produced by media and people chasing pennies. People should only publish things they firmly believe based on evidence to be true.
EDIT: Maybe it's just placebo. I checked the CPU usage on WindowServer before and after deleting & restarting.
(High CPU use isn’t necessarily a real performance issue and might be caused by just having Activity Monitor open.)
Dead Comment
At the same time even "hard" evidence would likely get dismissed as anecdotal, and there's certainly enough of it now (and even plenty in the past) to point a clear finger at Chrome/Keystone.
This certainly beat filing it in the black hole that is Chromium's bug reporter where it would have been ignored / works-on-my-machine'd / or dismissed as anecdotal there.
Whatever it is doing is sketchy and causing WindowServer to thrash. And this is not the first sketchy thing it has done.
But this post alleges it is really bad when not running, which seems like huge news if true.
I don't see enough evidence here, but I'd be open to the idea that this is true if someone looked into it deeper.
'facebook advertises based on IRL conversation' is another example of something that feels true to people but nobody has made stick as far as I know
upvotes don't always mean 'yes, guilty, firing squad'. I think they can mean 'yes tell me more, let's bookmark and start investigating'.
On the other hand, I generally feel that Chrome is garbage that slows computers down, so I wouldn't be surprised. But I don't see enough evidence here.
Google = Bad
Comment. (comment.com)
An analogy I see here is the scene from an old film where a woman walks behind a wooden screen and a goose walks out thus the woman is a shape shifting witch.
And yet, as of this writing, it has 200+ upvotes and is at the top of the front page.
HN audience really needs to do better. If you are reading this and are one of the people who upvoted the story, please really ask yourself why.
Many of us believe, with some justice, the causalization of computers has been to the diminishment of power users.
Therefore chrome bad gets hundreds of upvotes. I guarantee you not everyone read the entire article, and then ran their own diagnostic CPU benchmarking to verify the results, and also compared it against chrome competitors.
It's more like the common folk like chrome, and chrome ruin PCS for us
https://news.ycombinator.com/item?id=21065504
This is anecdotal but there is a noticeable difference for me with responsiveness after following these steps. Currently the WindowServer process is hovering around 14%.
The change only happened for me after removing Chrome as well as the launch agent for keystone.
Just for the sake of the argument, do you have installed Chrome?
Very sad.
(I have recently started using Apple tools for developping with Swift/Xcode on iPad. I am very unimpressed by the quality of the tools, and quite impressed by the range and depth of features....)