Correction: The first official distro internally was "grhat." It was born out of need. We used LDAP+kerberos for auth and our homedirs mounted on first login. This had all kinds of fun problems (looking at you, nscd!) but worked well enough most of the time. Goobuntu came a couple years later. In the in between, lots of people ran their own installed version and we worked together to get things working (even Slackware).
>> LDAP+kerberos for auth and our homedirs mounted on first login
My first "real" job as a sysadmin had this kind of setup, albeit with Sun servers and workstations running Solaris 8. I was impressed by how well it actually worked most of the time given all the quirkiness of kerberos/nfs/nsc/etc..
My first job at University was dealing with this type of setup (Solaris workstations) except we weren't dealing with LDAP, Kerberos, and NFS, but rather AD and Microsoft DFS.
It did NOT go that well, but was a good learning experience.
I've used kerberos + some auth (actually AD in addition to LDAP) and automounted home (and data dirs) a number of times (various clusters, departmental servers, corps). I'd say it was fairly mature (the OS stack supported it, it didn't crash all the time, etc) but still had (and has) sharp edges when it comes to high throughput computing.
Same, though much newer! UW CSE did the same when I worked there as a help desk tech, student directories mounted on login to the machines. IIRC, it was just a samba share linked into kerberos auth somehow.
Doing a rolling distro sounds like a great idea for Google, at their scale, and they also have the expertise and resources to handle any downsides of that.
For early startups, I've been doing "Debian Stable" as the default, partly so that we don't have to spend any time on any surprise distro changes when we're focused on MVP, etc.
And in 2 years, the startup will probably have a lot more resources, and we can look at whether that still makes sense, though we might just end up doing an in-place APT `dist-upgrade`, or coast on `oldstable` awhile if the timing is not right yet.
I also try to use the same distro on workstations and in production, to permit lightweight efficiencies, like experimenting and debugging outside of containers on workstations, without special tooling, while still having a close match to production. So, Debian Stable everywhere.
A recent Ubuntu LTS would also work for this (and sometimes be easier for things like Nvidia SDK), though Debian has been arguably a bit better for security and stability lately.
For things not in the Debian Stable we're using, such as if we need a bleeding-edge version of some key thing, and we have big security/reliability requirements... I manage non-Debian-packaged third-party dependencies in our own Git repo, and track and vet updates. This also means trying to minimize these dependencies, more than we would if we were pulling in 100 packages casually from a language-specific package manager, since each package is additional work.
Debian Stable (same as Ubuntu LTS) doesn't have newer package versions, it might sound OK for Windows people, that are accustomed having release once every few years, but in case of mode hardcore Linux users it wouldn't be the best user experience.
Even just the git has many releases during Debian Stable/Ubuntu LTS, which enhance user experience, that holding back users is not the best idea.
And in most cases security fixes land first in the newest versions and are backported, so it is most likely to have a fix in HEAD than in some random older release.
> in case of mode hardcore Linux users it wouldn't be the best user experience.
The "best" user experience you're referring to is getting surprises like your OpenSSH 9 suddenly stopping to work with your key agent [1] out of the blue because they decided to change the protocol (no big deal right?) without any hint of this whatsoever during normal usage, and you just casually upgraded your packages because, well, when did Linux package upgrades ever hurt anybody?
(Just the latest incredibly frustrating surprise your fellow "Windows person" literally had to spend hours on tracking down. And that I actually remember. Most definitely not the only one.)
Debian has a backports repository, providing newer versions of software for the existing stable release. This "Debian Stable has old software" canard is way out of date.
> Even just the git has many releases during Debian Stable...
It just so happens the software you mentioned, git, is in stable backports, so it is updated frequently. Regardless, I've been on Debian stable since 2007, and running "bleeding edge" stuff is trivial, depending on your desired workflow. (For example, I just install Debian testing packages in a schroot.) I used to bounce from distribution to distribution in the old days (Red Hat, Mandrake, Slackware, Gentoo, etc.) but once I found Debian Stable I could concentrate on getting work done.
I use Debian stable for my workstation, but get everything I care about from somewhere else. Go, Node, Emacs, Postgres... all installed from some other package repo. (I actually like Homebrew a lot on Linux.)
So I might be two years out of date on ls, but it probably hasn't changed in the last 2 years anyway.
I work for a smaller company running most of our stuff on older Ubuntu LTS versions. While there are things we want to update faster than the distribution does, most of them are leaf packages near the stuff we’re directly working on and are easy enough to either package ourselves or find a more recent package for from a trust worthy source.
It’s really nice to have the rest of the distribution more or less set it and forget it.
For a general desktop OS though that does not sound like a huge problem at all - long term stable always is behind. Thats part of the selling point in a way.
Debian is not a good distro for development. They have an intrusive package management philosophy that expects to be able to manage everything on the system, installing up to date things from a language package manager is harder on Debian than elsewhere.
(And, as another reply mentioned, you'll miss out on a lot of usability improvements to development tools by running stuff that's 2 years old or more)
> For things not in the Debian Stable we're using, such as if we need a bleeding-edge version of some key thing, and we have big security/reliability requirements... I manage non-Debian-packaged third-party dependencies in our own Git repo, and track and vet updates. This also means trying to minimize these dependencies, more than we would if we were pulling in 100 packages casually from a language-specific package manager, since each package is additional work.
Making it harder to have up-to-date dependencies is costing you far more than you're saving by not upgrading frequently.
> Making it harder to have up-to-date dependencies is costing you far more than you're saving by not upgrading frequently.
Not if you're separating where you live from where you work - in a manner of speaking. Applying global changes to your system (e.g. using package manager to install a specific versions of packages required by your production app) is a terrible, terrible idea that will cost you more than isolation (standardized docker, dev VMs or similar, which ought to reflect your running environment). Debian stable provides a good workstation environment - the actual work should not be done against Debian-blessed packages. The opposite is true - your workstation packages should not be dictated by the needs of your prod software.
> installing up to date things from a language package manager is harder on Debian than elsewhere
For better or for worse, I think this is partly where my preference to use rbenv, pyenv, asdf, etc comes from.
The other part was from hosing my system installed pip packages for the Nth time because long ago I didn't fully understand pip (still don't, but I didn't then either) and I used a mix of "pip install" and "sudo pip install" when the former didn't work. I could only work on one thing at a time with this approach, and switching was a huge task because I had to figure out all the places where packages had landed and either remove them individually or blow the whole thing away.
This is a great point! Quite insightful and detailed on how Ubuntu could have been the greatest program the world could have benefitted from. I think it was too shortlived and misunderstood for its time.
> Besides, the "effort to upgrade our Goobuntu fleet usually took the better part of a year. With a two-year support window, there was only one year left until we had to go through the same process all over again for the next LTS. This entire process was a huge stress factor for our team, as we got hundreds of bugs with requests for help for corner cases."
Ummm.. No? Ubuntu LTS has a 5-year support window. Not 2.
Or maybe it didn't all the way back then? I don't recall.
A rolling distro does make sense though. Especially in a large org where you have a relatively homogenous usepatterns. All the edgecases you can find centrally and pre-empt them before rolling out the updates.
Desktop was 3 years from 6.06 onwards, only server had 5 years. Might have changed recently. This meant you had a year to upgrade the old LTS before it went end of life (from 2008 releases moved back to April, 6.06 was the first LTS and I assume they took a little longer getting it out. I think the version before was 5.10)
For all the talk I've heard over the years about their in house Linux distro. Their desktop application support, and sometimes web application support, for Linux is non existent.
Most of their server estate will be Linux based, including what's powering their desktop applications. But almost none of their efforts give back and enrich the desktop ecosystem. That said they do contribute a lot towards frameworks, libraries and the kernel. Certainly they are among the top alongside Microsoft in recent years and Facebook. So I do recognize my entitled call for more... with good cause
As a Fedora user writing this from my brand new Google Pixel phone, who uses Gmail and YouTube, and pretty much nothing else from Google in 2022, what is missing?
A lot of the stuff that annoys me and gets in my way while using Linux is still the same basic quality-of-life stuff that has been broken forever. Bluetooth pairing fails occasionally. Plugging in a new display doesn't immediately work. A USB microphone fails to register as an input sound device.
I cannot imagine that Google engineers would live with such deeply broken systems as daily drivers. So whatever that secret is, whether it consists of drivers, known hardware configurations, or even just config files they use to ensure optimal operation – that's what I'd like them to Open Source. I'm happy to buy whatever laptop spec Google uses for gLinux for myself to go along with their OS.
Google used to actually have a handful of native applications on linux. I remember uploading to google play music with a linux client (not web) and stuff like google drive were "promised" or something like that.
It was all really short lived though. I remember that uploader broke fairly quickly as, of course, some abi change in some library dep and of course there was never another release.
It's not really what's missing... it's just google's 'here today, gone tomorrow'. They really shouldn't be called engineers.
Well their contributions and 'desktop support' only go towards... ChromeOS.
The fact that this effort is going towards this OS tells us they don't care about open-source and especially the Linux Desktop either. It is only to the Kernel which is fine, but it clear that the wider Linux Desktop ecosystem has had the same 20 year old issues still present today.
Having never worked at a place the size of Google, I'm surprised that their IT team directly manages 100,000 machines. In my (apparently naive) mind, if you're smart enough to work at Google, you're smart enough to manage your own OS.
Is it a security thing? Are Googlers not allowed to directly control their own computers?
(recent Xoogler here) There is quite a lot of freedom, though if you do custom stuff it's just not supported. I usually replaced the desktop, installed a custom terminal app, etc. There is also a lot of freedom in IDE choice - you can use literally whatever you want, it's just that the support is crappy if you don't use a supported one. And Vim support was IIRC completely volunteer based - there was no vim team, just people working on plugins and building releases as 20% etc. I contributed one plugin which sadly not a lot of people used, but Googlers out there - try BlazeDebugCurrentTest (it's part of the Blaze plugin) or something like that!
Actually one of my favorite Google experiences starting out was that I wanted to work with an Apple Magic Keyboard, and to get proper support for it I needed to compile some kernel driver I found on github. I asked IT support if I can do that, and the answer was "you're a SWE, review the code, make sure it looks legit and doesn't contain any security holes, and then it's your responsibility if you mess anything up". Which I did, and it worked just fine. It was in 2018 so it's not like I'm referring to some early days thing.
> Are Googlers not allowed to directly control their own computers?
First, if the computer is provided by the company, for company work, they're not "their own computers." They're the company's computers. And I don't know about Google, but yes it is VERY common in most mid- to large-size organizations that employees are not allowed to change things on the machines they use for work.
The IT department is usually required by either contracts or regulation to follow certain security standards and many of those prohibit end-user modification of machines. And further, these are often enforced through annual, semi-annual, or quarterly audits. Failure to follow these standards can result in the loss of a sales contract, a critical certification, or fines.
So, just like a manufacturing worker is not allowed to modify a machine on the factory floor (even to fix it, or improve it, regardless of their skill to do so), employees are not allowed to modify the organization's computing equipment.
And, it's important to note, the IT department generally has NO say in these rules since they are a business or legal requirement.
Edited to add: I've also worked in companies where the developers who write the in-house software might be masters of the business problem domain and their programming language, but have exactly ZERO knowledge of hardware or the most basic system administration principles. It doesn't cross their mind that RAM and disk space are actually finite things, for example, and that it makes for a Bad Day when one of their programs starts to consume ALL of it. These are the people who should not really be managing their own operating systems at work.
Which business and legal requirements, do you think, bind most small and medium enterprises to lock down workstations while engineers across Silicon Valley have root on their MacBooks from huge important companies?
I gave a demo in Mountain View of some excellent viz, on their big screen in some room. I wanted to make some tiny change to (plug something in) edit no not a new device, being polite, I suggested to move a device of theirs already on that network, closer to the overhead projector .. the alarm, consternation and definite NO that rippled through the three quickly-relaying employees there, was literally like an electric current. I was amazed that they couldn't change even the smallest thing, ever.
of course I was trying to get my brilliant colleague without a degree, noticed with that code; zero response, like stonewall. Later I read, that year Google got over 1million job applications.
There probably a lot of brilliant minds at Google who have their very first taste of Linux there. Designers, management types, even developers, who suddenly need a Linux install to run a specific tool or dev build. Having a consistent Distro for them to just turn on and go is very useful.
If something doesn't work, they can communicate back to the (well-bearded) developer who sent them that weird build. The developer surely has some crazy esoteric distro on their bring-your-own device, but they also have desktops running the consistent distro available to them all around the office to reproduce the issue on. It's a bridge between worlds.
I work in academia and we manage our own Ubuntu spin for this reason, despite everyone we ship it to being very conventionally smart.
> if you're smart enough to work at Google, you're smart enough to manage your own OS.
I also like to think of myself as "smart" for self-managing most OS and cloud things. But the role of smarts is probably minor compared to the decade+ of experience I have doing it. When it's something I'm good at, I think it's because of intelligence. When it's something I can't do, I assume the people who can have lots of practice.
> The developer surely has some crazy esoteric distro on their bring-your-own device
Nope. If you want access to more than just the basic corp resources, you’re doing it from a fully managed and approved OS on a company owned machine or VM.
> if you're smart enough to work at Google, you're smart enough to manage your own OS.
No disrespect to you but this is a really naive take on how IT management works.
For one thing, not everyone at Google is an OS expert. There are people working for Google in marketing, sales, support, data science, graphic design, hardware design, and other fields where you can't depend on every person to "do the right thing" when it comes to "managing their own OS."
And even if they can, is that where you want them spending their time? Manually managing their OS? Don't you want them to do the job they were hired to do instead? Their computer is a work tool, not tinkertown.
Basically, the opposite of what you're saying is true: because 100,000 people work at Google, each person represents another way to potentially mess something up and cause problems for the whole company.
Assign each employee with probabilities:
- Probability of falling for a phishing scam
- Probability of installing malware
- Probability of an employee missing an announcement regarding OS/security policy
- Probability of an employee opening a support ticket with IT
These probabilities are very low for people who are computer-savvy, educated, very highly qualified people. The thing is, all you have to do is multiply those probabilities by 100,000 and now you've got a big problem.
It's much easier to run with zero IT automation if you've only got 10 people in your company and they all know each other.
Most of the people you listed don't run Linux desktops at Google. Of those that do, most are able to manage their own OS.
For your probabilities, you describe many issues that aren't big issues at Google. Phishing credentials is mitigated by mandatory FIDO tokens, binary whitelisting heavily reduces installs of malware, security policy isn't needed on these desktops (really just needed for mobile devices incl laptops).
I am a google engineer, and a fairly successful one.
I would absolutely despise it if I had to manage my own OS, just like I despise thinking about my keyboard layout or text editor.
Hacker News overrepresents the hacker type which loves the feeling of full control but I'd estimate that at least 33% of engineers, including many very talented ones, don't want that, and only want to focus on the concrete problems they are trying to solve.
It’s for security and for homogeneity, a lot of google is setup around the principle that “works on my machine” is terrible, and also removing needless cleverness. You have root and can run anything you want*, but you have to go out of your way to configure anything differently than others, and the result is (hopefully) it just works the same everywhere. The monorepo also runs only natively on these Linux machines through a magical fuse interface so most development is either using the web ide or ssh-ing into the Linux box if you aren’t sitting in front of it. There were big economies of scale running this way and at least this setup, definitely felt pretty great and efficient I gotta say.
* By default, a tool called “Santa” keeps a naughty and nice list of runnable programs but all it took to get a program added to the nice list is any other googler vouching for it in an automated web tool.
I work at google, and I’m also someone who has used and messed around with Gentoo for like 12 years. When it comes to my day job I rather not spend time messing around with my own OS (my workflow aside) and leave it to other people. Multiply that by 100k or so engineers and the productivity boost is insane.
We’re root on both desktops and laptops though, I just rather use my time for work and I’m sure my employer would rather I do that too.
> I rather not spend time messing around with my own OS (my workflow aside) and leave it to other people.
I don't really understand what "messing around mean" in your context. Exotic things like slackware or alpine aside, all major popular distros now offer unattended updates, easy upgrade path between major releases.
Your vscode, neovim, jetbrains whatever or emacs do not work differently from one distro to another and since nowadays many external stuff is either distributed as static binaries, containers or flatpacks or appimages everything is quite straightforward and not distro specific.
I never worked at Google, but based on my experience elsewhere it is partly security and partly standardization of guaranteed minimum functionality.
For example, security aside, as an engineer working on X that depends on internal tools Y, Z and W, I do not particularly care what variant of Linux is under the hood as long as I can run my favorite WM, editor and user apps. But I do want support on Y, Z and W if they misbehave. If I run a non-standard OS or distro I will likely get a lukewarm support because those teams will put debugging problems in "non-standard configurations" as a low priority.
Personally, I learned to live with most Linux distros. When I did roll out a non-standard setup I had a standby system in a standard configuration that I can show failures on before asking for help. My 2c.
> Having never worked at a place the size of Google, I'm surprised that their IT team directly manages 100,000 machines. In my (apparently naive) mind, if you're smart enough to work at Google, you're smart enough to manage your own OS.
Smart isn't really a key point here. You can be smart and make choices that solve your immediate needs but may not be good for the company in the long run. The distro provides for some standardization and baseline in among other variance.
> Is it a security thing?
Yes, one of the forms of standardization are audit solutions that keep track of all kinds of data about the machines. In the time I worked there, I once received outreach because a service I had started on the machine had retained an outgoing port to a third party service provider for an extended period of time. I explained what it was, and I did not get in trouble, but I also shut it down. There are many other forms of audit systems.
> Are Googlers not allowed to directly control their own computers?
It depends on your role, but many roles have root on their machines. You can of course do just about anything from there, however some things may get undone by periodic scripts if they're defensive (e.g. removal of certain software), or in other cases trigger some kind of outreach as described above. For certain other policies, you may need to file a bug with a business justification to request specific policies - you can also do this for entire teams if it applies to their work.
We have root on our own (personal) boxes, so we can essentially do what we want (within reason).
There's a tremendous value in having a somewhat homogenous environment. We have our own package repo and .deb packages to access internal resources. Even little stuff like editor plugins customized to our internal stuff we have packages for.
Plus, we hire a great deal of folks that aren't Linux experts. Have a sane set of defaults gets people going faster.
It makes a lot of sense to have our own distro when we're talking this kind of scale.
I've worked at large tech companies where very capable software engineers struggled to install or upgrade their OS, let alone be familiar with the intricacies of Linux. I was surprised at first too, but I do think there's a large class of engineers who think about code who don't think at all about the other stuff, and engineering is broad enough that that's totally fine.
Not to mention all the non-engineers, even in tech roles like PM, design, etc.
The way I see it, the engineers aren't being paid to install and repair hardware and operating system stuff. Even if they are perfectly capable, it makes sense to have IT handle that and let the engineers get on with their engineering.
I think it depends on what the priorites are. I care a lot about my own systems but when I'm working I really don't want to have to care about anything but the activity I'm getting paid for.
Software Engineers are worth more to the company when they're writing code, not when they're debugging issues with the OS. It's much more worthwhile to standardize the machines people have and let people focus on doing their actual work more effortlessly.
Googler opinion, it feels like the right balance between letting me be in control while automating away the boring stuff that I don't want to deal with.
> if you're smart enough to work at Google, you're smart enough to manage your own OS
Regardless of the scale, that is a recipe for having your highly capable, highly paid engineers losing hours, days, even worse - to annoying configuration issues etc. Sometimes it's the right thing to do (e.g. very early stage, no IT infra) but if you can have an IT person who is actually good/experienced at this sort of thing sort out the core problems, it's way more efficient.
It's astonishing how much collective time can be wasted on "well, it works on my machine" issues. This makes it worth coming up with some sort of systematic way to avoid (mostly) it.
Why would I want to manage my own Linux distro? I use it to code for work, that's it.
> Is it a security thing? Are Googlers not allowed to directly control their own computers?
This and simplicity I think. There's definitely some controls over what kind of things you install on the MacBooks as well, though it's actually not that restrictive.
> if you're smart enough to work at Google, you're smart enough to manage your own OS.
Oh boy, have I got some stories to tell you!
Although to be fair the question of who "manages" a workstation is not always as simple as just "who has the technical ability to manage it". It's the intersection of who is legally responsible for the device and it's data (this is BY FAR the most important part), who has the time to maintain it (that's often NOT the user), and who technically can maintain it according to whatever standard it has to be maintained to (this gets complicated when you factor in reporting, patching etc).
In my experience, a common approach on non-Windows systems is to give the users privileged access to do what they need to do while also monitoring that closely, managing patching for them, and if you have some legal requirement to do so; running some kind of AV for reporting purposes. This tends to strike a happy medium between giving IT/Security peace of mind to some extent while also not totally ruining the developer experience.
It's also a matter of efficiency. I can spend more time on projects that matter instead of on system administration. The people doing the system administration are more efficient since they don't need to spend extra time figuring out what to do. It's not a matter of being smart enough to do it, but instead it's a matter of time efficiency.
Not everyone is interested in playing with their own customized OS configurations. Perhaps 90% of the employees would be okay if everything "just works". This is especially true due to lots of internal tooling and figuring out the right configurations for most of them would be a very painful time-consuming process for newcomers.
I wish people never used analogies. It adds nothing to conversation and you are always inclined to pick the most ridiculous one. Neurosurgeon disassembling and reassembling a whole car, sounds indeed crazy.
Does anyone know what they do for malware and virus detection / protection? There are a lot of commercial products but they are all quite opaque in how they work, vendors don't discuss the cpu/latency impacts of their solutions (or perhaps they don't know what these impacts even are...). Would be great to know if there was something or what approaches Google takes to protect themselves and their employees.
Readit News
"can someone telnet to 10200 on my box to reset nscd!?"
My first "real" job as a sysadmin had this kind of setup, albeit with Sun servers and workstations running Solaris 8. I was impressed by how well it actually worked most of the time given all the quirkiness of kerberos/nfs/nsc/etc..
It did NOT go that well, but was a good learning experience.
For early startups, I've been doing "Debian Stable" as the default, partly so that we don't have to spend any time on any surprise distro changes when we're focused on MVP, etc.
And in 2 years, the startup will probably have a lot more resources, and we can look at whether that still makes sense, though we might just end up doing an in-place APT `dist-upgrade`, or coast on `oldstable` awhile if the timing is not right yet.
I also try to use the same distro on workstations and in production, to permit lightweight efficiencies, like experimenting and debugging outside of containers on workstations, without special tooling, while still having a close match to production. So, Debian Stable everywhere.
A recent Ubuntu LTS would also work for this (and sometimes be easier for things like Nvidia SDK), though Debian has been arguably a bit better for security and stability lately.
For things not in the Debian Stable we're using, such as if we need a bleeding-edge version of some key thing, and we have big security/reliability requirements... I manage non-Debian-packaged third-party dependencies in our own Git repo, and track and vet updates. This also means trying to minimize these dependencies, more than we would if we were pulling in 100 packages casually from a language-specific package manager, since each package is additional work.
Even just the git has many releases during Debian Stable/Ubuntu LTS, which enhance user experience, that holding back users is not the best idea. And in most cases security fixes land first in the newest versions and are backported, so it is most likely to have a fix in HEAD than in some random older release.
The "best" user experience you're referring to is getting surprises like your OpenSSH 9 suddenly stopping to work with your key agent [1] out of the blue because they decided to change the protocol (no big deal right?) without any hint of this whatsoever during normal usage, and you just casually upgraded your packages because, well, when did Linux package upgrades ever hurt anybody?
(Just the latest incredibly frustrating surprise your fellow "Windows person" literally had to spend hours on tracking down. And that I actually remember. Most definitely not the only one.)
[1] https://www.openssh.com/agent-restrict.html
It just so happens the software you mentioned, git, is in stable backports, so it is updated frequently. Regardless, I've been on Debian stable since 2007, and running "bleeding edge" stuff is trivial, depending on your desired workflow. (For example, I just install Debian testing packages in a schroot.) I used to bounce from distribution to distribution in the old days (Red Hat, Mandrake, Slackware, Gentoo, etc.) but once I found Debian Stable I could concentrate on getting work done.
So I might be two years out of date on ls, but it probably hasn't changed in the last 2 years anyway.
It’s really nice to have the rest of the distribution more or less set it and forget it.
Windows users might be accustomed to having infrequent OS version releases, but they're used to their _apps_ being current.
(And, as another reply mentioned, you'll miss out on a lot of usability improvements to development tools by running stuff that's 2 years old or more)
> For things not in the Debian Stable we're using, such as if we need a bleeding-edge version of some key thing, and we have big security/reliability requirements... I manage non-Debian-packaged third-party dependencies in our own Git repo, and track and vet updates. This also means trying to minimize these dependencies, more than we would if we were pulling in 100 packages casually from a language-specific package manager, since each package is additional work.
Making it harder to have up-to-date dependencies is costing you far more than you're saving by not upgrading frequently.
Not if you're separating where you live from where you work - in a manner of speaking. Applying global changes to your system (e.g. using package manager to install a specific versions of packages required by your production app) is a terrible, terrible idea that will cost you more than isolation (standardized docker, dev VMs or similar, which ought to reflect your running environment). Debian stable provides a good workstation environment - the actual work should not be done against Debian-blessed packages. The opposite is true - your workstation packages should not be dictated by the needs of your prod software.
For my development needs I prefer Debian Stable and install the dev tools locally in my home directory rather than using the packaged versions.
Since I don't share those boxes, it works well for me. I keep up with the frequent Android Studio / VSCode updates that way.
For better or for worse, I think this is partly where my preference to use rbenv, pyenv, asdf, etc comes from.
The other part was from hosing my system installed pip packages for the Nth time because long ago I didn't fully understand pip (still don't, but I didn't then either) and I used a mix of "pip install" and "sudo pip install" when the former didn't work. I could only work on one thing at a time with this approach, and switching was a huge task because I had to figure out all the places where packages had landed and either remove them individually or blow the whole thing away.
Ummm.. No? Ubuntu LTS has a 5-year support window. Not 2.
Or maybe it didn't all the way back then? I don't recall.
A rolling distro does make sense though. Especially in a large org where you have a relatively homogenous usepatterns. All the edgecases you can find centrally and pre-empt them before rolling out the updates.
https://en.wikipedia.org/wiki/Ubuntu_version_history#Version...
Internal users demanded the upgrade every two years anyway.
Most of their server estate will be Linux based, including what's powering their desktop applications. But almost none of their efforts give back and enrich the desktop ecosystem. That said they do contribute a lot towards frameworks, libraries and the kernel. Certainly they are among the top alongside Microsoft in recent years and Facebook. So I do recognize my entitled call for more... with good cause
I cannot imagine that Google engineers would live with such deeply broken systems as daily drivers. So whatever that secret is, whether it consists of drivers, known hardware configurations, or even just config files they use to ensure optimal operation – that's what I'd like them to Open Source. I'm happy to buy whatever laptop spec Google uses for gLinux for myself to go along with their OS.
It was all really short lived though. I remember that uploader broke fairly quickly as, of course, some abi change in some library dep and of course there was never another release.
It's not really what's missing... it's just google's 'here today, gone tomorrow'. They really shouldn't be called engineers.
Deleted Comment
The fact that this effort is going towards this OS tells us they don't care about open-source and especially the Linux Desktop either. It is only to the Kernel which is fine, but it clear that the wider Linux Desktop ecosystem has had the same 20 year old issues still present today.
Is it a security thing? Are Googlers not allowed to directly control their own computers?
Actually one of my favorite Google experiences starting out was that I wanted to work with an Apple Magic Keyboard, and to get proper support for it I needed to compile some kernel driver I found on github. I asked IT support if I can do that, and the answer was "you're a SWE, review the code, make sure it looks legit and doesn't contain any security holes, and then it's your responsibility if you mess anything up". Which I did, and it worked just fine. It was in 2018 so it's not like I'm referring to some early days thing.
First, if the computer is provided by the company, for company work, they're not "their own computers." They're the company's computers. And I don't know about Google, but yes it is VERY common in most mid- to large-size organizations that employees are not allowed to change things on the machines they use for work.
The IT department is usually required by either contracts or regulation to follow certain security standards and many of those prohibit end-user modification of machines. And further, these are often enforced through annual, semi-annual, or quarterly audits. Failure to follow these standards can result in the loss of a sales contract, a critical certification, or fines.
So, just like a manufacturing worker is not allowed to modify a machine on the factory floor (even to fix it, or improve it, regardless of their skill to do so), employees are not allowed to modify the organization's computing equipment.
And, it's important to note, the IT department generally has NO say in these rules since they are a business or legal requirement.
Edited to add: I've also worked in companies where the developers who write the in-house software might be masters of the business problem domain and their programming language, but have exactly ZERO knowledge of hardware or the most basic system administration principles. It doesn't cross their mind that RAM and disk space are actually finite things, for example, and that it makes for a Bad Day when one of their programs starts to consume ALL of it. These are the people who should not really be managing their own operating systems at work.
Deleted Comment
of course I was trying to get my brilliant colleague without a degree, noticed with that code; zero response, like stonewall. Later I read, that year Google got over 1million job applications.
weird place
If something doesn't work, they can communicate back to the (well-bearded) developer who sent them that weird build. The developer surely has some crazy esoteric distro on their bring-your-own device, but they also have desktops running the consistent distro available to them all around the office to reproduce the issue on. It's a bridge between worlds.
I work in academia and we manage our own Ubuntu spin for this reason, despite everyone we ship it to being very conventionally smart.
> if you're smart enough to work at Google, you're smart enough to manage your own OS.
I also like to think of myself as "smart" for self-managing most OS and cloud things. But the role of smarts is probably minor compared to the decade+ of experience I have doing it. When it's something I'm good at, I think it's because of intelligence. When it's something I can't do, I assume the people who can have lots of practice.
Nope. If you want access to more than just the basic corp resources, you’re doing it from a fully managed and approved OS on a company owned machine or VM.
No disrespect to you but this is a really naive take on how IT management works.
For one thing, not everyone at Google is an OS expert. There are people working for Google in marketing, sales, support, data science, graphic design, hardware design, and other fields where you can't depend on every person to "do the right thing" when it comes to "managing their own OS."
And even if they can, is that where you want them spending their time? Manually managing their OS? Don't you want them to do the job they were hired to do instead? Their computer is a work tool, not tinkertown.
Basically, the opposite of what you're saying is true: because 100,000 people work at Google, each person represents another way to potentially mess something up and cause problems for the whole company.
Assign each employee with probabilities:
- Probability of falling for a phishing scam
- Probability of installing malware
- Probability of an employee missing an announcement regarding OS/security policy
- Probability of an employee opening a support ticket with IT
These probabilities are very low for people who are computer-savvy, educated, very highly qualified people. The thing is, all you have to do is multiply those probabilities by 100,000 and now you've got a big problem.
It's much easier to run with zero IT automation if you've only got 10 people in your company and they all know each other.
For your probabilities, you describe many issues that aren't big issues at Google. Phishing credentials is mitigated by mandatory FIDO tokens, binary whitelisting heavily reduces installs of malware, security policy isn't needed on these desktops (really just needed for mobile devices incl laptops).
I would absolutely despise it if I had to manage my own OS, just like I despise thinking about my keyboard layout or text editor.
Hacker News overrepresents the hacker type which loves the feeling of full control but I'd estimate that at least 33% of engineers, including many very talented ones, don't want that, and only want to focus on the concrete problems they are trying to solve.
* By default, a tool called “Santa” keeps a naughty and nice list of runnable programs but all it took to get a program added to the nice list is any other googler vouching for it in an automated web tool.
We’re root on both desktops and laptops though, I just rather use my time for work and I’m sure my employer would rather I do that too.
I don't really understand what "messing around mean" in your context. Exotic things like slackware or alpine aside, all major popular distros now offer unattended updates, easy upgrade path between major releases.
Your vscode, neovim, jetbrains whatever or emacs do not work differently from one distro to another and since nowadays many external stuff is either distributed as static binaries, containers or flatpacks or appimages everything is quite straightforward and not distro specific.
For example, security aside, as an engineer working on X that depends on internal tools Y, Z and W, I do not particularly care what variant of Linux is under the hood as long as I can run my favorite WM, editor and user apps. But I do want support on Y, Z and W if they misbehave. If I run a non-standard OS or distro I will likely get a lukewarm support because those teams will put debugging problems in "non-standard configurations" as a low priority.
Personally, I learned to live with most Linux distros. When I did roll out a non-standard setup I had a standby system in a standard configuration that I can show failures on before asking for help. My 2c.
Smart isn't really a key point here. You can be smart and make choices that solve your immediate needs but may not be good for the company in the long run. The distro provides for some standardization and baseline in among other variance.
> Is it a security thing?
Yes, one of the forms of standardization are audit solutions that keep track of all kinds of data about the machines. In the time I worked there, I once received outreach because a service I had started on the machine had retained an outgoing port to a third party service provider for an extended period of time. I explained what it was, and I did not get in trouble, but I also shut it down. There are many other forms of audit systems.
> Are Googlers not allowed to directly control their own computers?
It depends on your role, but many roles have root on their machines. You can of course do just about anything from there, however some things may get undone by periodic scripts if they're defensive (e.g. removal of certain software), or in other cases trigger some kind of outreach as described above. For certain other policies, you may need to file a bug with a business justification to request specific policies - you can also do this for entire teams if it applies to their work.
There's a tremendous value in having a somewhat homogenous environment. We have our own package repo and .deb packages to access internal resources. Even little stuff like editor plugins customized to our internal stuff we have packages for.
Plus, we hire a great deal of folks that aren't Linux experts. Have a sane set of defaults gets people going faster.
It makes a lot of sense to have our own distro when we're talking this kind of scale.
Not to mention all the non-engineers, even in tech roles like PM, design, etc.
Regardless of the scale, that is a recipe for having your highly capable, highly paid engineers losing hours, days, even worse - to annoying configuration issues etc. Sometimes it's the right thing to do (e.g. very early stage, no IT infra) but if you can have an IT person who is actually good/experienced at this sort of thing sort out the core problems, it's way more efficient.
It's astonishing how much collective time can be wasted on "well, it works on my machine" issues. This makes it worth coming up with some sort of systematic way to avoid (mostly) it.
> Is it a security thing? Are Googlers not allowed to directly control their own computers?
This and simplicity I think. There's definitely some controls over what kind of things you install on the MacBooks as well, though it's actually not that restrictive.
Oh boy, have I got some stories to tell you!
Although to be fair the question of who "manages" a workstation is not always as simple as just "who has the technical ability to manage it". It's the intersection of who is legally responsible for the device and it's data (this is BY FAR the most important part), who has the time to maintain it (that's often NOT the user), and who technically can maintain it according to whatever standard it has to be maintained to (this gets complicated when you factor in reporting, patching etc).
In my experience, a common approach on non-Windows systems is to give the users privileged access to do what they need to do while also monitoring that closely, managing patching for them, and if you have some legal requirement to do so; running some kind of AV for reporting purposes. This tends to strike a happy medium between giving IT/Security peace of mind to some extent while also not totally ruining the developer experience.
Deleted Comment
This is a bit like saying if you can operate on human brains, you should absolutely be able to take apart your car and put it back together.
Or your software has all these customization options
Or your product has a bunch of different configurations
What was already hard just became a thousand times more painful
https://debconf22.debconf.org/talks/11-scalable-support-for-...