News for hackers. Stuff that matters.
Re:Google’s in-house desktop Linux by nickelcitymario on Monday August 01, 2022 @10:38AM (#32306574)
Having never worked at a place the size of Google, I'm surprised that their IT team directly manages 100,000 machines. In my (apparently naive) mind, if you're smart enough to work at Google, you're smart enough to manage your own OS.

Is it a security thing? Are Googlers not allowed to directly control their own computers?

Re:Google’s in-house desktop Linux by dvirsky on Monday August 01, 2022 @01:28PM (#32309341)
(recent Xoogler here) There is quite a lot of freedom, though if you do custom stuff it's just not supported. I usually replaced the desktop, installed a custom terminal app, etc. There is also a lot of freedom in IDE choice - you can use literally whatever you want, it's just that the support is crappy if you don't use a supported one. And Vim support was IIRC completely volunteer based - there was no vim team, just people working on plugins and building releases as 20% etc. I contributed one plugin which sadly not a lot of people used, but Googlers out there - try BlazeDebugCurrentTest (it's part of the Blaze plugin) or something like that!

Actually one of my favorite Google experiences starting out was that I wanted to work with an Apple Magic Keyboard, and to get proper support for it I needed to compile some kernel driver I found on github. I asked IT support if I can do that, and the answer was "you're a SWE, review the code, make sure it looks legit and doesn't contain any security holes, and then it's your responsibility if you mess anything up". Which I did, and it worked just fine. It was in 2018 so it's not like I'm referring to some early days thing.

Re:Google’s in-house desktop Linux by adtac on Monday August 01, 2022 @01:55PM (#32309730)
vim support had only gotten better since 2018. With CiderLSP (LSP support for all of google3), it was a breeze navigating the massive codebase.
Re:Google’s in-house desktop Linux by dvirsky on Monday August 01, 2022 @02:06PM (#32309906)
Before WFH I used CLion for cpp. When we were locked down I found Cider to be annoying (I think mostly because you couldn't tab switch to a a chrome window on MacOS) so I took a couple of days to really master vim and configure all the plugins (and wrote the one plugin I was missing), and it worked just fine for a while, I would just ssh to my workstation and use that. Later on they fixed the tab switching thing so I found myself drifting more towards Cider, then Cider-V came along, and I was back at the office, so I was mostly using that, though by that time I wasn't writing cpp anymore.
Re:Google’s in-house desktop Linux by bityard on Monday August 01, 2022 @11:39AM (#32307623)
> Are Googlers not allowed to directly control their own computers?

First, if the computer is provided by the company, for company work, they're not "their own computers." They're the company's computers. And I don't know about Google, but yes it is VERY common in most mid- to large-size organizations that employees are not allowed to change things on the machines they use for work.

The IT department is usually required by either contracts or regulation to follow certain security standards and many of those prohibit end-user modification of machines. And further, these are often enforced through annual, semi-annual, or quarterly audits. Failure to follow these standards can result in the loss of a sales contract, a critical certification, or fines.

So, just like a manufacturing worker is not allowed to modify a machine on the factory floor (even to fix it, or improve it, regardless of their skill to do so), employees are not allowed to modify the organization's computing equipment.

And, it's important to note, the IT department generally has NO say in these rules since they are a business or legal requirement.

Edited to add: I've also worked in companies where the developers who write the in-house software might be masters of the business problem domain and their programming language, but have exactly ZERO knowledge of hardware or the most basic system administration principles. It doesn't cross their mind that RAM and disk space are actually finite things, for example, and that it makes for a Bad Day when one of their programs starts to consume ALL of it. These are the people who should not really be managing their own operating systems at work.

Re:Google’s in-house desktop Linux by closeparen on Monday August 01, 2022 @02:27PM (#32310219)
Which business and legal requirements, do you think, bind most small and medium enterprises to lock down workstations while engineers across Silicon Valley have root on their MacBooks from huge important companies?
Re:Google’s in-house desktop Linux by easton on Monday August 01, 2022 @08:19PM (#32313812)
Google doesn’t let devs store IP on their workstations except in very rare circumstances (per this thread and the BeyondCorp papers), and in those rare circumstances I’m willing to bet you either don’t have root or there’s way more auditing turned on.

Most bigcos are the same way, there’s a way to get root but there’s also security software or policies to counteract damage you could do. With most hardcore “zero-trust” setups you consider a device’s posture based on its current state and don’t take into account whether or not the user has admin rights (what do you care how what rights they used to install bad software, you just care that they installed bad software). No admin rights typically means nobody has time to do something better for end users or there’s a regulatory requirement (at the same time, bigcos can throw a sandboxed VM up in their data center that can’t access PII if devs need more access).

Re:Google’s in-house desktop Linux by number6 on Monday August 01, 2022 @04:29PM (#32311639)
Money for lawyers
Re:Google’s in-house desktop Linux by mistrial9 on Monday August 01, 2022 @11:46AM (#32307711)
I gave a demo in Mountain View of some excellent viz, on their big screen in some room. I wanted to make some tiny change to (plug something in) edit no not a new device, being polite, I suggested to move a device of theirs already on that network, closer to the overhead projector .. the alarm, consternation and definite NO that rippled through the three quickly-relaying employees there, was literally like an electric current. I was amazed that they couldn't change even the smallest thing, ever.

of course I was trying to get my brilliant colleague without a degree, noticed with that code; zero response, like stonewall. Later I read, that year Google got over 1million job applications.

weird place

Re:Google’s in-house desktop Linux by dsl on Monday August 01, 2022 @02:01PM (#32309816)
> I suggested to move a device of theirs already on that network, closer to the overhead projector

What you didn't know is that it wouldn't work.

There is also an expectation that you don't just randomly start changing things in shared conference rooms. If there is an issue, you open a GUTS ticket and someone comes and solves the problem. Chances are if you discovered a real issue, there are 90 other rooms with the same issue that also would be updated.

Re:Google’s in-house desktop Linux by dekhn on Monday August 01, 2022 @02:08PM (#32309933)
What were you trying to connect? Connecting to an ethernet port wouldn't work as they use ethernet authentication. Connecting to the projector's input ports instead of using Meet to present? Certainly something I wouldn't have protested if somebody external was giving a demo, back when I worked there.
Re:Google’s in-house desktop Linux by lupire on Monday August 01, 2022 @12:28PM (#32308508)
What kind of change?
Re:Google’s in-house desktop Linux by miggol on Monday August 01, 2022 @11:12AM (#32307132)
There probably a lot of brilliant minds at Google who have their very first taste of Linux there. Designers, management types, even developers, who suddenly need a Linux install to run a specific tool or dev build. Having a consistent Distro for them to just turn on and go is very useful.

If something doesn't work, they can communicate back to the (well-bearded) developer who sent them that weird build. The developer surely has some crazy esoteric distro on their bring-your-own device, but they also have desktops running the consistent distro available to them all around the office to reproduce the issue on. It's a bridge between worlds.

I work in academia and we manage our own Ubuntu spin for this reason, despite everyone we ship it to being very conventionally smart.

> if you're smart enough to work at Google, you're smart enough to manage your own OS.

I also like to think of myself as "smart" for self-managing most OS and cloud things. But the role of smarts is probably minor compared to the decade+ of experience I have doing it. When it's something I'm good at, I think it's because of intelligence. When it's something I can't do, I assume the people who can have lots of practice.

Re:Google’s in-house desktop Linux by happyopossum on Monday August 01, 2022 @12:29PM (#32308521)
> The developer surely has some crazy esoteric distro on their bring-your-own device

Nope. If you want access to more than just the basic corp resources, you’re doing it from a fully managed and approved OS on a company owned machine or VM.

Re:Google’s in-house desktop Linux by mistrial9 on Monday August 01, 2022 @11:47AM (#32307746)
the word is not "smart", the word is "autonomy" and there is none in that network environment. (see post above for real example)
Re:Google’s in-house desktop Linux by dangus on Monday August 01, 2022 @12:42PM (#32308713)
> if you're smart enough to work at Google, you're smart enough to manage your own OS.

No disrespect to you but this is a really naive take on how IT management works.

For one thing, not everyone at Google is an OS expert. There are people working for Google in marketing, sales, support, data science, graphic design, hardware design, and other fields where you can't depend on every person to "do the right thing" when it comes to "managing their own OS."

And even if they can, is that where you want them spending their time? Manually managing their OS? Don't you want them to do the job they were hired to do instead? Their computer is a work tool, not tinkertown.

Basically, the opposite of what you're saying is true: because 100,000 people work at Google, each person represents another way to potentially mess something up and cause problems for the whole company.

Assign each employee with probabilities:

- Probability of falling for a phishing scam

- Probability of installing malware

- Probability of an employee missing an announcement regarding OS/security policy

- Probability of an employee opening a support ticket with IT

These probabilities are very low for people who are computer-savvy, educated, very highly qualified people. The thing is, all you have to do is multiply those probabilities by 100,000 and now you've got a big problem.

It's much easier to run with zero IT automation if you've only got 10 people in your company and they all know each other.

Re:Google’s in-house desktop Linux by jms703 on Monday August 01, 2022 @01:43PM (#32309524)
Most of the people you listed don't run Linux desktops at Google. Of those that do, most are able to manage their own OS.

For your probabilities, you describe many issues that aren't big issues at Google. Phishing credentials is mitigated by mandatory FIDO tokens, binary whitelisting heavily reduces installs of malware, security policy isn't needed on these desktops (really just needed for mobile devices incl laptops).

Re:Google’s in-house desktop Linux by dangus on Tuesday August 02, 2022 @02:36PM (#32322613)
> Of those that do, most are able to manage their OS.

I just explained why it doesn’t matter that “most” people can manage their OS, that’s because Google has thousands of Linux users. If 1% of Google developers make a critical mistake that could be a hundred workstations or more.

On top of that, I still disagree. Many developers I’ve met are not that good at using the OS.

Remember that writing applications is a completely separate skill. You can learn to code completely separate from learning anything else about the computer. I had a college professor that taught C on a chalkboard, in this millennium.

Google’s own developer interviews are very CS and algorithm heavy and to my knowledge never test your abilities with OS management.

When you say “really just needed for mobile devices including laptops” that’s basically all devices isn’t it? You’d have to be incredibly specialized to need or prefer a desktop workstation.

I haven’t even brought up compliance controls and certifications yet, either. If you’re SOC 2 or HIPAA compliant you have to manage workstations to disable functionality and restrict system preferences. For example, there are typically restrictions on USB port data transmission and idle screen lock delay time.

Re:Google’s in-house desktop Linux by TulliusCicero on Monday August 01, 2022 @02:16PM (#32310060)
Most could, sure. I could do so. But why would I want to? It would take research and investment into something I don't give a shit about. I don't particularly care what Linux variant is being used, as long as it works. The less I have to think about it, the more I can spend time on other, more important stuff.
Re:Google’s in-house desktop Linux by joshuamorton on Monday August 01, 2022 @04:23PM (#32311572)
Software distribution becomes distinctly more complicated if you're running multiple distros. As is, I can install a lot of software from an internal apt. Saying "hey now we need to maintain an internal yum and pacman and also deploy internal packages to those" or alternatively come up with some bespoke cross platform binary distribution method seems effortful.
Re:Google’s in-house desktop Linux by anonquixey on Monday August 01, 2022 @02:07PM (#32309908)
I am a google engineer, and a fairly successful one.

I would absolutely despise it if I had to manage my own OS, just like I despise thinking about my keyboard layout or text editor.

Hacker News overrepresents the hacker type which loves the feeling of full control but I'd estimate that at least 33% of engineers, including many very talented ones, don't want that, and only want to focus on the concrete problems they are trying to solve.

Re:Google’s in-house desktop Linux by number6 on Monday August 01, 2022 @04:28PM (#32311629)
Did the even more successful engineers liked the experience more or less than you?
Re:Google’s in-house desktop Linux by mattnewton on Monday August 01, 2022 @11:10AM (#32307102)
It’s for security and for homogeneity, a lot of google is setup around the principle that “works on my machine” is terrible, and also removing needless cleverness. You have root and can run anything you want*, but you have to go out of your way to configure anything differently than others, and the result is (hopefully) it just works the same everywhere. The monorepo also runs only natively on these Linux machines through a magical fuse interface so most development is either using the web ide or ssh-ing into the Linux box if you aren’t sitting in front of it. There were big economies of scale running this way and at least this setup, definitely felt pretty great and efficient I gotta say.

* By default, a tool called “Santa” keeps a naughty and nice list of runnable programs but all it took to get a program added to the nice list is any other googler vouching for it in an automated web tool.

Re:Google’s in-house desktop Linux by dehrmann on Monday August 01, 2022 @12:09PM (#32308161)
Santa is open-source: https://github.com/google/santa
Re:Google’s in-house desktop Linux by isatty on Monday August 01, 2022 @01:18PM (#32309213)
I work at google, and I’m also someone who has used and messed around with Gentoo for like 12 years. When it comes to my day job I rather not spend time messing around with my own OS (my workflow aside) and leave it to other people. Multiply that by 100k or so engineers and the productivity boost is insane.

We’re root on both desktops and laptops though, I just rather use my time for work and I’m sure my employer would rather I do that too.

Re:Google’s in-house desktop Linux by prmoustache on Monday August 01, 2022 @04:40PM (#32311747)
> I rather not spend time messing around with my own OS (my workflow aside) and leave it to other people.

I don't really understand what "messing around mean" in your context. Exotic things like slackware or alpine aside, all major popular distros now offer unattended updates, easy upgrade path between major releases.

Your vscode, neovim, jetbrains whatever or emacs do not work differently from one distro to another and since nowadays many external stuff is either distributed as static binaries, containers or flatpacks or appimages everything is quite straightforward and not distro specific.

Re:Google’s in-house desktop Linux by ptero on Monday August 01, 2022 @10:53AM (#32306790)
I never worked at Google, but based on my experience elsewhere it is partly security and partly standardization of guaranteed minimum functionality.

For example, security aside, as an engineer working on X that depends on internal tools Y, Z and W, I do not particularly care what variant of Linux is under the hood as long as I can run my favorite WM, editor and user apps. But I do want support on Y, Z and W if they misbehave. If I run a non-standard OS or distro I will likely get a lukewarm support because those teams will put debugging problems in "non-standard configurations" as a low priority.

Personally, I learned to live with most Linux distros. When I did roll out a non-standard setup I had a standby system in a standard configuration that I can show failures on before asking for help. My 2c.

Re:Google’s in-house desktop Linux by raggi on Monday August 01, 2022 @12:40PM (#32308684)
> Having never worked at a place the size of Google, I'm surprised that their IT team directly manages 100,000 machines. In my (apparently naive) mind, if you're smart enough to work at Google, you're smart enough to manage your own OS.

Smart isn't really a key point here. You can be smart and make choices that solve your immediate needs but may not be good for the company in the long run. The distro provides for some standardization and baseline in among other variance.

> Is it a security thing?

Yes, one of the forms of standardization are audit solutions that keep track of all kinds of data about the machines. In the time I worked there, I once received outreach because a service I had started on the machine had retained an outgoing port to a third party service provider for an extended period of time. I explained what it was, and I did not get in trouble, but I also shut it down. There are many other forms of audit systems.

> Are Googlers not allowed to directly control their own computers?

It depends on your role, but many roles have root on their machines. You can of course do just about anything from there, however some things may get undone by periodic scripts if they're defensive (e.g. removal of certain software), or in other cases trigger some kind of outreach as described above. For certain other policies, you may need to file a bug with a business justification to request specific policies - you can also do this for entire teams if it applies to their work.

Re:Google’s in-house desktop Linux by dman on Monday August 01, 2022 @10:59AM (#32306888)
Google was the only employer I had where I had root on my machine, so I would say that developers have control over their machine.
Re:Google’s in-house desktop Linux by geekbird on Monday August 01, 2022 @02:09PM (#32309944)
I had root on my Linux desktop box at my university job, but I was in IT, administering a Linux service. Most of my coworkers were using Macs only.
Re:Google’s in-house desktop Linux by baobob on Monday August 01, 2022 @12:46PM (#32308777)
Was this pre or post 2009?
Re:Google’s in-house desktop Linux by compiler-guy on Monday August 01, 2022 @12:59PM (#32308955)
I have root on my Google machine right this second.
Re:Google’s in-house desktop Linux by sulam on Monday August 01, 2022 @01:04PM (#32309038)
I have root right now on my Linux and MacOS systems.
Re:Google’s in-house desktop Linux by dman on Tuesday August 02, 2022 @12:02PM (#32320437)
Post.
Re:Google’s in-house desktop Linux by jnwatson on Monday August 01, 2022 @12:34PM (#32308603)
We have root on our own (personal) boxes, so we can essentially do what we want (within reason).

There's a tremendous value in having a somewhat homogenous environment. We have our own package repo and .deb packages to access internal resources. Even little stuff like editor plugins customized to our internal stuff we have packages for.

Plus, we hire a great deal of folks that aren't Linux experts. Have a sane set of defaults gets people going faster.

It makes a lot of sense to have our own distro when we're talking this kind of scale.

Re:Google’s in-house desktop Linux by wayne on Monday August 01, 2022 @11:07AM (#32307049)
I've worked at large tech companies where very capable software engineers struggled to install or upgrade their OS, let alone be familiar with the intricacies of Linux. I was surprised at first too, but I do think there's a large class of engineers who think about code who don't think at all about the other stuff, and engineering is broad enough that that's totally fine.

Not to mention all the non-engineers, even in tech roles like PM, design, etc.

Re:Google’s in-house desktop Linux by drewzero1 on Monday August 01, 2022 @11:38AM (#32307594)
The way I see it, the engineers aren't being paid to install and repair hardware and operating system stuff. Even if they are perfectly capable, it makes sense to have IT handle that and let the engineers get on with their engineering.
Re:Google’s in-house desktop Linux by matheusmoreira on Monday August 01, 2022 @01:49PM (#32309635)
I think it depends on what the priorites are. I care a lot about my own systems but when I'm working I really don't want to have to care about anything but the activity I'm getting paid for.
Re:Google’s in-house desktop Linux by cactacea on Monday August 01, 2022 @11:49AM (#32307780)
This. There's a lot of I shaped people out there that don't have the first clue about anything beyond their day-to-day.
Re:Google’s in-house desktop Linux by poopypoopington on Monday August 01, 2022 @11:29AM (#32307424)
Software Engineers are worth more to the company when they're writing code, not when they're debugging issues with the OS. It's much more worthwhile to standardize the machines people have and let people focus on doing their actual work more effortlessly.
Re:Google’s in-house desktop Linux by danpalmer on Monday August 01, 2022 @11:05AM (#32307001)
Googler opinion, it feels like the right balance between letting me be in control while automating away the boring stuff that I don't want to deal with.
Re:Google’s in-house desktop Linux by ska on Monday August 01, 2022 @03:51PM (#32311217)
> if you're smart enough to work at Google, you're smart enough to manage your own OS

Regardless of the scale, that is a recipe for having your highly capable, highly paid engineers losing hours, days, even worse - to annoying configuration issues etc. Sometimes it's the right thing to do (e.g. very early stage, no IT infra) but if you can have an IT person who is actually good/experienced at this sort of thing sort out the core problems, it's way more efficient.

It's astonishing how much collective time can be wasted on "well, it works on my machine" issues. This makes it worth coming up with some sort of systematic way to avoid (mostly) it.

Re:Google’s in-house desktop Linux by TulliusCicero on Monday August 01, 2022 @02:11PM (#32309983)
Why would I want to manage my own Linux distro? I use it to code for work, that's it.

> Is it a security thing? Are Googlers not allowed to directly control their own computers?

This and simplicity I think. There's definitely some controls over what kind of things you install on the MacBooks as well, though it's actually not that restrictive.

Re:Google’s in-house desktop Linux by synicalx on Monday August 01, 2022 @07:24PM (#32313423)
> if you're smart enough to work at Google, you're smart enough to manage your own OS.

Oh boy, have I got some stories to tell you!

Although to be fair the question of who "manages" a workstation is not always as simple as just "who has the technical ability to manage it". It's the intersection of who is legally responsible for the device and it's data (this is BY FAR the most important part), who has the time to maintain it (that's often NOT the user), and who technically can maintain it according to whatever standard it has to be maintained to (this gets complicated when you factor in reporting, patching etc).

In my experience, a common approach on non-Windows systems is to give the users privileged access to do what they need to do while also monitoring that closely, managing patching for them, and if you have some legal requirement to do so; running some kind of AV for reporting purposes. This tends to strike a happy medium between giving IT/Security peace of mind to some extent while also not totally ruining the developer experience.

Re:Google’s in-house desktop Linux by charcircuit on Monday August 01, 2022 @03:01PM (#32310644)
It's also a matter of efficiency. I can spend more time on projects that matter instead of on system administration. The people doing the system administration are more efficient since they don't need to spend extra time figuring out what to do. It's not a matter of being smart enough to do it, but instead it's a matter of time efficiency.
Re:Google’s in-house desktop Linux by summerlight on Monday August 01, 2022 @03:32PM (#32311001)
Not everyone is interested in playing with their own customized OS configurations. Perhaps 90% of the employees would be okay if everything "just works". This is especially true due to lots of internal tooling and figuring out the right configurations for most of them would be a very painful time-consuming process for newcomers.
Re:Google’s in-house desktop Linux by yathaid on Monday August 01, 2022 @01:09PM (#32309108)
"In my (apparently naive) mind, if you're smart enough to work at Google, you're smart enough to manage your own OS."

This is a bit like saying if you can operate on human brains, you should absolutely be able to take apart your car and put it back together.

Re:Google’s in-house desktop Linux by tryauuum on Monday August 01, 2022 @04:03PM (#32311362)
I wish people never used analogies. It adds nothing to conversation and you are always inclined to pick the most ridiculous one. Neurosurgeon disassembling and reassembling a whole car, sounds indeed crazy.
Re:Google’s in-house desktop Linux by thrashh on Monday August 01, 2022 @11:52AM (#32307850)
Try doing support when everyone has a different setup

Or your software has all these customization options

Or your product has a bunch of different configurations

What was already hard just became a thousand times more painful

Re:Google’s in-house desktop Linux by duped on Monday August 01, 2022 @11:30AM (#32307449)
I've worked with Xoogler SWEs who didn't know how to use the command line. They're not hiring for computer literacy.