News for hackers. Stuff that matters.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by manojlds on Thursday February 09, 2017 @11:43AM (#13608267)
After the mess up, I don't really like seeing these posts about Gitlab. Maybe this is their problem after all.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by cookiecaper on Thursday February 09, 2017 @11:59AM (#13608463)
I think the problem has more to do with their recruiting. To be a successful distributed company, you need a disciplined and highly experienced workforce (note that this does not mean a highly educated workforce, which may actually be a contraindication).

GitLab offers a middling base salary for a role and applies modifiers for experience and city-based cost-of-living (both of which may modify the base downward; my CoL adjustment is -40%, and I live in the United States!). They explicitly state on their hiring page that they prefer people from cheaper areas. [0] [1]

On top of that, just from their headcount, it sounds like they have way too many people. 160 for a technical company like this is hard to manage when the employees are local; it's even harder to manage when employees are remote. Again, for remote work, you need mature, highly-skilled staff with a strong history of successful self-direction. You're not going to get that when you're chopping their salaries in half based on their location. Bad local wages is a major reason to work remotely in the first place.

[0] https://about.gitlab.com/jobs/developer/ ; use the calculator at the bottom of the page

[1] https://about.gitlab.com/handbook/people-operations/global-c... ; gems include "you are required to notify us when you move and we may adjust your compensation up or down" and "[a]ll things being equal we will hire people in lower cost markets vs. higher cost markets."

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by bgxor on Thursday February 09, 2017 @12:40PM (#13608932)
Those salary modifiers are atrocious. Their range in my city for a senior, very experienced engineer is comparable to what most entry level jobs actually offer around here.

I 100% think you are on the right track. The only people from my area that would take a GitLab salary just can't get a job elsewhere.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by pc86 on Thursday February 09, 2017 @12:42PM (#13608958)
It makes more sense for me to get a PO box in Washington DC and drive there to get my paycheck. Hell, it's probably cheaper to "rent" a single room out of someone's house just to have them forward the mail.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by compuguy on Thursday February 09, 2017 @01:32PM (#13609490)
Agreed, the Washington DC and the Fairfax, VA modifier should be the same. Otherwise your not going to be competitive in the Washington DC area as a whole.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by vonmoltke on Thursday February 09, 2017 @01:52PM (#13609670)
Agreed. Based on their calculator, moving from Dallas to Ft. Worth is a 20% pay cut. That's ridiculous; Dallas and Ft. Worth are essentially a single market. If any difference at all is justified it should be pretty small.

As it is, for me the calculator gives me roughly what I was making in Dallas, a bit less (at the top end) than I make now in New York, and a Lockheed entry-level salary if I were to move to Ft. Worth.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by amyjess on Friday February 10, 2017 @10:29AM (#13616005)
> Agreed. Based on their calculator, moving from Dallas to Ft. Worth is a 20% pay cut. That's ridiculous; Dallas and Ft. Worth are essentially a single market. If any difference at all is justified it should be pretty small.

That's... aggravating. You'd think their system would treat all locations in the same Core Based Statistical Area as the same. Better yet, treat all locations in the same Combined Statistical Area as the same; in fact, the definition of a CSA is a set of adjacent CBSAs with interconnected commuting patterns.

And the data is all freely available in machine-readable form from the Census Bureau's website [0], so there's no excuse to not use it.

In fact, just this past week, as a personal project, I've written a bunch of code to parse a whole ton of census data, combining CBSA/NECTA data, data from Summary File 1 (i.e. population, sorted by a number of demographics), and Gazetteer data (for land area, so I can calculate population density). Over the weekend, I'm going to take a stab at including data from the American Community Survey too.

[0] http://www.census.gov/population/metro/data/def.html

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by ernstvn on Sunday February 12, 2017 @02:51PM (#13630264)
amyjess, thanks for pointing out the CBSA; I'll admit I had not seen it before. When we set out to develop the calculator we wanted it to work globally, and focused on that. This will be a helpful way to improve it specifically in the US, so I'm adding it to https://gitlab.com/gitlab-com/organization/issues/24
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by danpalmer on Thursday February 09, 2017 @12:58PM (#13609119)
Yeah, I put in some pretty conservative estimates for my skills/experience/etc, and got out a salary estimate that is not good for my area (London, UK). I tried the city I went to university in and there is no way they would be competitive there with other companies.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by ernstvn on Thursday February 09, 2017 @01:28PM (#13609445)
Would love to get more input from you to make sure the calculator reaches its goal of providing fair market compensation in your location. Can you please send me an email to let me know your city and whatever further data you are willing to share? [email protected]
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by dragonwriter on Thursday February 09, 2017 @02:36PM (#13610111)
My local city, but the data isn't so much mine as data that the US government shares.

Sacramento, CA. Mean Salary (per latest BLS data) [0] for Software Developer, Applications: $107,540; for Web Developer: $78,050

Sacramento, CA Gitlab locality pay index: 0.39+0.25=0.51

Vs.

New York City, same BLS figures [1]: $108,770/$81,430

NYC Gitlab locality pay index: 1.00+0.25=1.25

Sac/NYC Salary Ratio (BLS): 0.989/0.958

Sac/NYC Salary Ratio (Gitlab): 0.512

Fundamentally, the rent index based methodology you are using is, I would guess, giving you below market pay almost everywhere that isn't NYC or San Francisco, assuming that you've hit the market wage for the skills you are targeting in your NYC baseline.

[0] https://www.bls.gov/oes/current/oes_40900.htm#15-0000

[1] https://www.bls.gov/oes/current/oes_35620.htm#15-0000

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by ernstvn on Thursday February 09, 2017 @07:22PM (#13612291)
Thanks; this is useful. I've made an issue ( https://gitlab.com/gitlab-com/organization/issues/23 ) to make sure we cross check BLS data with the calculator.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by tedmiston on Friday February 10, 2017 @01:03AM (#13613571)
I'm not sure what it is about small Midwestern cities but the calculator seems to dislike them. The calculated rates for Cincinnati are roughly half of market rate here. I understand the calculator will not be perfect in every city worldwide but it might be better to have no calculator as the current one is extremely discouraging even though I like GitLab.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by cookiecaper on Friday February 10, 2017 @01:35AM (#13613664)
It's based on rent. Each city is punished in proportion to how much cheaper its rent is than the rent in NYC. As NYC is one of the top 5 most expensive cities worldwide for real estate [0], any city marginally more reasonable is going to get pummeled.

[0] http://www.globalpropertyguide.com/most-expensive-cities

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by tedmiston on Friday February 10, 2017 @07:45AM (#13614978)
Perhaps it's that there is lots of cheap housing in the suburbs here, and in the greater metro area but that is outside the city proper. But most of the tech / startup activity ie concentrated in one small gentrifying neighborhood where rent is much higher (+50–100%), so coincidentally people in tech aren't buying those houses because the commute is so far from the core.

I wonder if any of these indexes breaks down a city by zip code. A good analogy for here would be if you averaged all rent over the neighborhoods in Brooklyn then used that number for a bunch of candidates in Williamsburg.

It really penalizes living in a third- / fourth-tier startup hub, for example, vs either a first- / second-tier hub or a non-hub.

Taken from another perspective maybe it's an intentional filter on the candidate pool to specific cities or countries. That wouldn't seem to be consistent with most remote philosophies but it's something to consider.

Interestingly, Buffer's salary calculator is almost the opposite with a relatively small percentage difference between Nashville or Austin vs SF or NYC for instance.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by dragonwriter on Thursday February 09, 2017 @05:43PM (#13611728)
Correction: even though the ratio calculated from it and the calculation for it are correct, I somehow mistyped the Sacramento locality index, which is 0.64, not 0.51.

Didn't notice it until after the edit window was closed.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by bgxor on Thursday February 09, 2017 @05:01PM (#13611427)
I sent you a more detailed email, but basically it is preposterous to assume a "very experienced" senior engineer in ANY part of the United States is going to be happy with making $65K.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by phil21 on Thursday February 09, 2017 @06:44PM (#13612098)
Exactly this. A very experienced engineer who is capable of working remote will likely be pretty intelligent. They'll realize a company is attempting to steal the surplus they created by living a low-cost lifestyle. I don't really understand why a company feels they have any claim to this.

Remote employees who are good at being remote have some very good options at this point of their career. Why the heck would they take less money simply because they had the foresight to move to a low cost of living location? Half the (stellar) people seeking remote work have built their entire lives around this fact - and have made very conscious decisions regarding their career and lifestyle.

Obviously it's working out for Gitlab, but I can't imagine any of my senior remote talent finding this acceptable in any way. I guess I could see it working for a couple years "converting" an engineer who is super excited to move into remote work vs. on-site. But beyond that, this policy seems extremely dangerous to me.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by jon-wood on Friday February 10, 2017 @03:31AM (#13614016)
This is precisely what turned me off applying for a job at Gitlab after initially being hugely enthusiastic. My family and I moved away from London to the city we grew up in to save some money, partly because I knew I could work remote for the same sort of money I had been making while paying considerably less on rent and living expenses. The cost of living adjustments Gitlab do would mean entirely negating that benefit.

I'm not really sure what the justification for CoL adjustments is, beyond "we can get away with it". Apparently I'm worth paying $120,000 working from home in London, but if I move 70 miles south I'm worth half that. Not only does it make little sense logically, there's no way cost of living here is less than half that in London, so its not even calculated correctly.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by ernstvn on Thursday February 09, 2017 @07:46PM (#13612419)
Thanks!
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by pc86 on Thursday February 09, 2017 @12:41PM (#13608943)
Just ran my numbers and got about 55% of my current salary assuming a lead position (what I have now, which I would probably not get with them) and "a lot of experience" which I'm on the fence about and they would probably disagree with.

Getting rid of the COL deduction by selecting NYC as my location actually makes it reasonable so something tells me not pinning the US salary floor to 1.0 is saving them money and simultaneously getting them lower quality developers.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by edblarney on Thursday February 09, 2017 @01:50PM (#13609656)
I think they are screwing this up.

By offering people medicore, local salaries, they are losing a massive opportunity to clean up on the great talent in remote areas.

Why would someone want to live in a low-end city (assuming they are mobile) - when they could live the same 'quality of life' in a much cooler place?

Assuming a degree of mobility - the whole point of living in a 2cnd tier place is that you can save a lot of money, which makes up for the '2cnd tierendess' of the place.

Montreal is 1/2 Boston. There are a good batch of great devs in Montreal. I could double my salary and move to Boston? I'd probably just do that. And still save more.

If they paid a 20% premium over other Montrealers, they'd be able to attract the best talent - and still save a lot of money over Boston/SF/NYC.

Paying 'regular market wages' for remote localities is not a winning proposition - it doesn't take advantage of the fact they are great startup, reasonably well financed etc..

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by ghaff on Thursday February 09, 2017 @06:30PM (#13612035)
Even talking about "Boston" as a single thing is tough. There's Boston/Cambridge urban folks and there are people who live an hour west or an hour north. That's an enormous difference in terms of housing prices. And even that is an oversimplification. School districts and other factors make a huge difference within that same one hour radius.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by edblarney on Friday February 10, 2017 @08:42AM (#13615260)
There's variation on a single street :)

But what neighbourhood you live in Boston is a choice.

Picking up and leaving a city is not a choice for most people.

Also the 'cost of everything except housing' is pretty consistent - everyone pays roughly the same a gallon of gas, and an identical basket of groceries.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by JabavuAdams on Thursday February 09, 2017 @10:11PM (#13612968)
There are a lot of great people who don't want to move to the US.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by cookiecaper on Thursday February 09, 2017 @01:03PM (#13609175)
Yeah, same here. If I max out the possible compensation by setting lead w/ a lot of experience, it's roughly 60% of what I make now (as a non-lead with a lot of experience).

The kicker is that I'm employed remotely by by a company based in a city whose CoL hit is even larger than the one I currently live in. The NYC range they give is plausible for me as someone who lives in a place that is much cheaper than NYC, but it's not exciting and wouldn't really motivate me to apply.

GitLab's numbers may look right on paper (and they do explain their methodology in the second link, discussing rent indexes, etc), but they're absolutely not comparable to what good people earn in the real world. Good talent commands good wages everywhere. A skilled remote worker is not going to come to GitLab for an average-for-their-area salary.

Even when working locally, talented people make much more than the average for their area. I know this because I've hired many good people and unless they're entry-level (meaning they're good but haven't had a chance to prove it yet), they don't start coming in the door until you're offering at least a 50% price premium over the reported median. In fact, we had a lot of bad candidates come in asking for around that much. The senior people I hired would make more than double the median for their nominal role. The stats on these government reports don't account for seniority, skill, niche demand, etc., and likely include some people that self-report as holding a title when they're really just aspiring to that position.

Once you're a mid-level dev, you should be pulling in a bare minimum of 80-85k, no matter where you live. I say this not as a bubble-dwelling San Franciscan (who surely finds such numbers laughable even for entry-level), but someone who has lived in various parts of "flyover country" his entire life. When local opportunities for at least this compensation are not forthcoming, go contract or remote.

There's no reason for good people to leave money on the table, and GitLab doesn't appear to get that.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by dboreham on Thursday February 09, 2017 @02:13PM (#13609890)
My wife is a compensation consultant and we have discussed this phenomenon a few times. She says that for some jobs "there is a national market" and comparisons to local salaries is therefore invalid.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by ashark on Thursday February 09, 2017 @01:14PM (#13609295)
Set it to "lead" and "a lot of experience" (I'd grade myself intermediate-to-senior and above-average experience IRL) and the top of that range is less than I make now, and I'm not big on self-promotion/networking and could probably make quite a bit more if I were. Plus their benefits are worse than what I get at my tiny company. Their comp calculation sucks for my area.

They're paying talentless Wordpress dev prices for their mid-tier devs, by local standards. This is in a mid-sized midwestern US city. LOLWUT? No one decent in my city would accept a job there.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by scaryclam on Thursday February 09, 2017 @12:59PM (#13609145)
Having that sort of calculator also encourages developers to embellish their competancy level as well, which is not going to help in finding quality developers. I'd much rather a company figure out where I am on their scale (each company I've worked at seems to have different ideas about what makes a junior/mid-level/senior/lead) than taking my word for it.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by sotojuan on Thursday February 09, 2017 @01:20PM (#13609363)
It's probably set up to attract cheap Eastern European devs.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by dragonwriter on Thursday February 09, 2017 @02:50PM (#13610246)
It's more likely set up based on an untested (and easily falsified) conjecture about the relationship between local residential rent prices and local market wages in the industry.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by sytse on Thursday February 09, 2017 @05:56PM (#13611824)
We indeed try to track local market wages. From https://about.gitlab.com/handbook/people-operations/global-c...

"In developing the compensation formula above, we looked at the compensation of our team members which had been set in the past (without the formula), and found out that there was a statistically significant correlation between compensation and the factors that are now in the formula. We purposefully chose to look for correlations with metrics that are probably causal and definitely relevant in people's lives (the rent!). This also has the advantage of letting us work with data that is readily available publicly, as opposed to trying to scour the web for market compensation rates for all roles in all locations. Perhaps surprisingly, there was a stronger correlation between compensation and rent index than with the more general cost of living index available through Numbeo (or the cost of living with rent index, for that matter); and so we moved ahead with the Rent Index."

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by dragonwriter on Thursday February 09, 2017 @06:13PM (#13611926)
The immediate problem with the method of developing your formula that is described is that it assumed that the pay of your staff before you had a systematic formula already accurately reflected prevailing market wages in the locality of every employee.

And, for US locations, BLS data is readily available publicly. While that may not have been a suitable source for a worldwide formula, it would have at least been a suitable reality check on the validity of the formula you came up with to see if it reasonably approximated the variations in market rates within the US.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by ernstvn on Thursday February 09, 2017 @07:38PM (#13612381)
You are right that we used compensation from before we had the systematic formula. However, we then repeated the exercise (of finding linear correlations with various COL indices) when we had a larger cohort but before the formula had been implemented systematically, and it still pointed to the use of the Rent Index instead of another index.

You're also right that we need the calculator to work worldwide, and I suspect that that may be part of the reason here that the calculator works well on that scale and then not always as well as it should on more local scales. We'll continue to explore how to get better on both scales; the challenge is always that we want to keep things as simple as possible also.

I've made two issues based on your comment here and the one where you shared specific data for Sacramento (thanks for that!):

- https://gitlab.com/gitlab-com/organization/issues/23 to cross check BLS data with the calculator.

- https://gitlab.com/gitlab-com/organization/issues/24 to address the global vs local question.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by dragonwriter on Thursday February 09, 2017 @11:42PM (#13613271)
BTW, I know I've been kind of hard (fair, I think, but not gentle) on you guys in this thread, and I want you to know that I really appreciate how well you've taken it and how responsive you've been.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by cookiecaper on Friday February 10, 2017 @12:08PM (#13616923)
Don't be too impressed. They have a habit of showing up in these threads and feigning concern. [0]

[0] https://news.ycombinator.com/item?id=13303948

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by pc86 on Friday February 10, 2017 @07:45AM (#13614984)
I won't quote it because I don't have the source in front of me but there is a quote elsewhere on this thread about Gitlab requiring that you tell them when you move so they can pay you less.

A) What's to stop me from getting hired in a 0.21 COL area and moving to San Francisco or Washington, DC? Are you going to fire me or pay me $20k a year because I moved?

B) My labor is not worth less and my work is not lower quality because I move to SE Asia or Eastern Europe. This is idiotic.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by mdekkers on Friday February 10, 2017 @02:07AM (#13613753)
I applied for a position at Gitlab a week ago, and didn't even get a response. Probably because I am too expensive, too old, and live in an area with strong labour laws and staff protection. It's a shame, because it would have been decent to at least get a "sorry, not interested" reply, but no response at all is just not cool. They write on their hiring page: Always leave feedback, this will help everyone to understand what happened and how you came to your decision.

Nice to "write everything down"

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by NadiaVat on Friday February 10, 2017 @10:27AM (#13615991)
@mdekkers glad we saw your post. I tried searching for you in our ATS, but could not find any application under that name. Can you email me directly on [email protected]? I'd really like to see what happened to your application and why we didn't respond.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by mdekkers on Friday February 10, 2017 @10:37AM (#13616058)
Hey Nadia, thanks for reaching out. Sure, I will send it along.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by pc86 on Friday February 10, 2017 @07:42AM (#13614967)
I don't think your local labor laws matter to Gitlab if they're based in your area.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by mdekkers on Friday February 10, 2017 @09:26AM (#13615522)
I applied for a remote position, so either "everywhere" is their area, or "nowhere".
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by sytse on Thursday February 09, 2017 @05:54PM (#13611804)
You can view how we're spread around the world on https://about.gitlab.com/team/
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by ekidd on Thursday February 09, 2017 @01:13PM (#13609286)
Hmm, in Boston they only pay 2/3rds of what Google does for somebody with my job title (not counting stock or bonuses). And as soon as you get outside of the "hot" markets, those multipliers are pretty bad. If I compare their Rochester, NY salaries with what I've actually earned working in rural college towns, well, I've gotten better paychecks in academia.

They seem like nice folks, but as an experienced remote worker (with a history of both remote employment and remote contracting), I wouldn't bother to apply based on what I currently know.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by biztos on Thursday February 09, 2017 @03:39PM (#13610712)
Wow, that is bizarre. Obviously you want to take advantage of wage differences in different markets, but why would you tell people ahead of time you're going to lowball them? Why not try to get the best you can get, and then pay whatever the actual market rate is for them?

I just checked the calculator for a major German city and I can say they won't be recruiting anybody away from the likes of Siemens. Rocket Internet maybe.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by pyb on Thursday February 09, 2017 @12:57PM (#13609111)
They have picked a strategy consisting in hiring large numbers of less experienced developers, rather than a few great ones. This is very much impossible to reverse, so now, they have to make it work with the people they have.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by dragonwriter on Thursday February 09, 2017 @01:25PM (#13609419)
> GitLab offers a middling base salary for a role and applies modifiers for experience and city-based cost-of-living

Their base is off of NYC, apparently, and they adjust based on location; from what I infer from that, if they are really middling for NYC, their modifiers are all wrong for the way market wages actually vary by location; in Sacramento, the adjusted range is below public sector pay for similar roles (salary alone, not considering benefits, etc.), much less private sector pay.

The thing is, their basic methodology is arguably somewhat sensible, but they could (at least in the US) use BLS numbers for job categories by location to get modifiers that actually reflect local markets and keep their pay consistent with respect to local markets. By choosing an index that evidently is a poor proxy for variations in market pay, they end up with pay that doesn't make sense.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by the_commentary on Thursday February 09, 2017 @01:08PM (#13609226)
I would agree with this -- Gitlab seems like a company with a great philosophy and I'm sure a lot of people would love to help them achieve their goals... but I just used the calculator for my own area and my pay would be around half what I'm getting currently. I have to assume that makes it really hard to get serious talent, and makes it more of a pass-through for people to get their foot in the door.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by dorfsmay on Thursday February 09, 2017 @12:42PM (#13608954)
FWIW, I have seen one example of hiring a remote junior person and it worked really well, BUT there was a conscious effort to do mentoring properly, and the guy running it was a mentoring guru/nut.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by ghaff on Thursday February 09, 2017 @04:49PM (#13611330)
I've worked mostly from home for a long time now through a few different jobs. (Always relatively nearby an office where I had a space but I haven't been in a job where I went in regularly for 15 years or so.) I'm mostly happy with the tradeoffs and would be fine with going 100% remote.

That said, acknowledging that a lot of things are different from when I was starting out, I can't imagine walking into my first tech industry product management job as a remote worker.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by minhajuddin on Friday February 10, 2017 @04:35AM (#13614283)
There was some previous discussion about this here: https://news.ycombinator.com/item?id=13302906 . I too had to turn down their offer of $2000 USD per month for a Senior position after the second round. This issue keeps coming up in posts about Gitlab and there doesn't seem to be any progress.

I love the Gitlab product and what they have done to Git hosting. But, their salary structure is really screwed up.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by ernstvn on Thursday February 09, 2017 @01:26PM (#13609429)
It's definitely not our intent to pay below market in any location. If our calculator shows a large negative adjustment from your current compensation, please send me an email ([email protected]) so I can review the data. We don't promise to make overnight changes to the calculator, since changes need to be robust and simple (i.e. no cherry picking city by city numbers), but we do gather data and make edits as we learn.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by cookiecaper on Thursday February 09, 2017 @02:01PM (#13609772)
The philosophy is flawed. Good people don't make average-for-their-area salaries.

Depending on which of the 5-6 applicable job titles listed by the Bureau of Labor Statistics's Occupational Employment Statistics [0] one chooses to apply, my salary is between 40% and 120% higher than the "average mean wage" (itself an inflated statistic) in my area.

This is true not just of myself but of all the good candidates I've hired. Good people don't work for average rates, especially not good people who are capable of doing remote work.

GitLab is doing itself a big disservice by targeting the "average" local wage.

[0] https://www.bls.gov/oes/current/oes_nat.htm#15-0000

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by imsofuture on Thursday February 09, 2017 @04:36PM (#13611233)
Remote employees' market is the entire world though, that's the part you're not taking into account. Your competition is the other NYC or Boston or SFO or wherever companies that are willing to hire me and pay me a generally competitive rate (very roughly 80-100% of SF pay, in my experience). Sure, you don't have to compete with a GoogFaceAzon offer, but you do have to compete with the companies similar to yours that do pay a more globally competitive rate.

There's a very important difference between: "SFO is very expensive so we have to pay people who live there more" and "we can pay people who don't live in SFO less"

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by overcast on Thursday February 09, 2017 @04:09PM (#13611004)
No one makes a salary calculator that robust without intent. I think paying twice as much, and cutting your work force in half would do you a greater service, than playing around with these cost of living calculators.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by budman1 on Friday February 10, 2017 @06:51PM (#13619969)
And be net cheaper because of 1/2 the benefit / supervision load.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by ridruejo on Thursday February 09, 2017 @12:25PM (#13608755)
Every company goes through rough patches, specially one that is growing as fast as they are doing. Very few companies are as open as transparent as they are, even in the fuckups. That is really hard to do and kudos to them
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by sytse on Thursday February 09, 2017 @12:55PM (#13609084)
Thanks. I do agree that the timing of this post is not good. The video was recorded a few weeks ago. We are eating humble pie right now and work on fixing things instead of promotion.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by coolsunglasses on Thursday February 09, 2017 @07:27PM (#13612319)
I have nothing but respect for your willingness to take these hits but it's not just about you.

This post is now fodder that can be used against me the next time I advocate for remote work at a company.

Edit: it took 10 minutes for two people to pipe up on Twitter, one specifically suggesting remote work made the ops failure more likely:

https://twitter.com/paul_snively/status/829850449247297536

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by sytse on Friday February 10, 2017 @01:32PM (#13617690)
I'm sorry for hear we're making your advocacy for remote work harder :(
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by cjbprime on Thursday February 09, 2017 @11:48AM (#13608325)
Yes, seems a little tone-deaf to publish a puff piece about GitLab at the moment.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by scrollaway on Thursday February 09, 2017 @12:33PM (#13608845)
Why? It's not a puff piece boasting about their backup plans, they're talking about their remote work strategy.

What's tone-deaf is the vitriol some people are spilling in this thread.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by sytse on Thursday February 09, 2017 @06:12PM (#13611917)
Thanks, I really appreciate that! But to be honest I don't think it's vitriol so much as that people rightly expect better from us.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by scrollaway on Thursday February 09, 2017 @07:08PM (#13612229)
Some of the reactions are understandable, but I expect most people on HN to be able to separate unrelated issues like these. Some of the comments here are absolutely unnecessarily hostile.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by jjawssd on Thursday February 09, 2017 @03:18PM (#13610511)
Spot on
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by Perihelion on Thursday February 09, 2017 @12:27PM (#13608787)
Not that this excuses anything, but maybe it'll help explain it: This interview was recorded a little while before the incident when we were unaware of some of the issues we have in process. Now that we're aware, we are working on correcting these things.

But you're right, the timing of this piece was probably not ideal.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by noir_lord on Thursday February 09, 2017 @12:53PM (#13609065)
Not ideal perhaps but on the other hand one bad mistake doesn't negate everything you do or say forever that would be extremely unforgiving.

...and it's not like any large web company hasn't had some howlers over the years.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by notcolin on Thursday February 09, 2017 @01:04PM (#13609183)
> This interview was recorded a little while before the incident when we were unaware of some of the issues we have in process.

Could you elaborate? I recently heard about GitLab probably less than a week ago and was under the impression they're a great company to work for.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by Perihelion on Thursday February 09, 2017 @01:16PM (#13609316)
When I say process, I mean some of our operational processes like making sure backups work. There was an incident with data loss last week: https://news.ycombinator.com/item?id=13537052

Like any company, we have our own business operations issues too but it's still one of the best jobs I've ever had. We're pretty open about our flaws, to the point where the CEO even has a page that lists his out: https://about.gitlab.com/handbook/people-operations/ceo-pref...

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by DigitalJack on Thursday February 09, 2017 @05:09PM (#13611493)
Read The Checklist Manifesto if you haven't already. Short book, leaves a big impression.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by connorshea on Thursday February 09, 2017 @01:13PM (#13609288)
She's referring to issues in the backup process specifically, not in the company culture (which IMO is pretty good).
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by onion2k on Thursday February 09, 2017 @12:34PM (#13608853)
Companies are run by people and consequently they're fallible. Every company makes mistakes occasionally. Gitlab handled theirs incredibly well. A more opaque company would have kept the problem, and the experience they earned handling it, to themselves. Gitlab's openness means we can all learn a little from their event. I think that's great. I wish more companies shared more when these things happen.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by cjbprime on Thursday February 09, 2017 @03:08PM (#13610428)
> A more opaque company would have kept the problem, and the experience they earned handling it, to themselves.

I liked their transparency a lot too! But it's neither possible nor acceptable to keep six hours of data loss and eighteen hours of continuous downtime "to yourself".

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by cookiecaper on Thursday February 09, 2017 @11:59AM (#13608472)
Agreed. It seems that HN did delay this a bit, but they probably should've given it at least a few more weeks.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by gukov on Thursday February 09, 2017 @03:47PM (#13610805)
Marketing 101. Flood negative with positive.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by sytse on Thursday February 09, 2017 @06:19PM (#13611968)
That was not our intention. This piece was recorded before the outage. We should asked to delay publication until after the postmortem was published. That would be the marketing 101 lesson, first show you corrected your mistake instead of talking about something else.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by geofft on Thursday February 09, 2017 @12:07PM (#13608561)
"This" being remote work?

Also, I'm a little surprised by the anti-GitLab sentiment on this thread. I thought the consensus was that they had bad luck but didn't do anything particularly more wrong than anyone else? (I may have missed some more analysis of the cause of the failure.)

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by problems on Thursday February 09, 2017 @12:17PM (#13608670)
> didn't do anything particularly more wrong than anyone else?

This is almost completely on their admin staff, maybe other people aren't willing to say it, but I will. Test your backups. Or at least make sure they're non-zero in size. It should really be Operations 101.

Whether you do this automatically or manually by setting a reminder on your calendar once a week or even month, doesn't matter. Something this simple would have solved their entire issue. I do this and we run a much smaller shop than GitLab. Heck, if we were larger I'd have hot spare database servers in another datacenter in case the primary got nuked by disk failure, network outages or mistakes.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by fsiefken on Thursday February 09, 2017 @12:55PM (#13609076)
As part of our disaster recovery plan when I was working as a sysadmin at a 150p company we had replicated server (database replicated and webfiles rsync) on hot standby, we just switched the front-facing servers manually. GitLab has a very short blurb on a similar styled HA setup, I'm not sure if and how they have implemented such themselves and if it would have helped in preventing or shortening the recent downtime. They have probably documented their own setup somewhere.

"Automated failover can be achieved with pacemaker alongside STONITH network management. Keep in mind that application servers need to be prepared for transitioning to the new network addresses.

In this situation you can also opt to synchronize the database via a database specific protocol instead of DRBD. In the documentation for each database you can find out more about the options for MySQL and the options for PostgreSQL." https://about.gitlab.com/high-availability/#filesystem-stora...

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by sytse on Thursday February 09, 2017 @05:52PM (#13611790)
We had a secondary server. The secondary we were using was basically dead.

That's what lead to the problem. We were trying to fix it, but ran the wrong procedure on the primary instead of the secondary

Thus wiping the primary, instead of what might have been left on the secondary.

The secondary was already removed at that point.

Basically the procedure was the following:

1. Secondary falls behind too much, stops replicating. At this point you need to manually re-sync the whole thing

2. A re-sync requires an empty PostgreSQL data directory, so this data was removed

3. Re-sync doesn't work, leading to the other problems

4. At some point team-member-1 thought previous re-sync attempts left data behind, so team-member-1 wiped the data directory again to be sure; except team-member-1 ran this on db1.cluster.gitlab.com (the primary) instead of db2.cluster.gitlab.com (the secondary)

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by geofft on Thursday February 09, 2017 @01:09PM (#13609249)
Synchronizing the database isn't quite what you want. It's true that in the case of an errant rm -rf it would almost certainly have helped, but it's approximately as easy to run a "DELETE FROM importantdata" and leave off the "WHERE" clause, which would get replicated. And certainly if you're using DRBD (replicate the volume, not the database), an rm -rf will get replicated.

I'm just genuinely unsure what a better outcome would have been here. (It's certainly a process failure that no backups existed other than the manual 6-hour-old snapshot, but I'm not sure you can do much better than automating that.)

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by problems on Thursday February 09, 2017 @01:10PM (#13609259)
You can easily add snapshotting using on-disk technologies like LVM or ZFS to that and achieve reliability against such an issue though, as well as being able to do a full text backup (ie: to SQL) from your replicated server at higher performance than you would on production.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by cookiecaper on Thursday February 09, 2017 @05:38PM (#13611697)
They had LVM snapshots every 24 hours. They lost 6 hours of data because someone had coincidentally triggered a snapshot 6 hours prior to the deletion event. Otherwise, they would've lost several more hours of data.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by problems on Friday February 10, 2017 @03:51PM (#13618784)
Correct me if I'm wrong, but wasn't that from staging data, not from an LVM snapshot?
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by detaro on Thursday February 09, 2017 @12:25PM (#13608757)
The "trigger" of the failure (accidentally deleting the wrong database) is bad luck that happens, but the lack of preparedness for it isn't. If I understood the document right even if the regular backups would have worked, they only happened every 24 hours (same as the staging replicas they recovered from, except one was manually created out of schedule that day).

Maybe being fully remote helps to let stuff like this "slip through the cracks". (Not great phrasing, but I don't think anyone made a conscious decision "loosing X hours of data is ok", and nobody questioned what goal the current practices could (not) achieve.) I don't know, but it seems at least a question one might ask.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by geofft on Thursday February 09, 2017 @12:35PM (#13608870)
What's the best-practice approach for taking backups that are significantly more frequent than every 24 hours, but also robust to things like an rm -rf or a DELETE FROM table;? Something like continuous data protection seems like it would be far too much data for an active database server, no?

(Or are we just saying that they should have been taking backups every 15 minutes or hour or so?)

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by cookiecaper on Thursday February 09, 2017 @05:50PM (#13611773)
With modern database servers, people will take checkpoints at some regular interval and automatically stream append-only copies of all intermediate transaction logs to a storage server some place (the same transaction logs that the database servers use for replication).

In the event of accidental deletion, the most recent snapshot will be restored and the transactions up to a certain timestamp will be replayed on top of it. If done correctly, this can prevent all but a few minutes of data loss.

AWS Aurora (and possibly other database-as-a-service providers) has this functionality built-in.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by detaro on Thursday February 09, 2017 @01:02PM (#13609167)
I'm not really qualified to confidently talk about best practices, so these points have question marks attached, since I can't judge what's possible in their setup and what's not. I guess I'm not sure if they did judge it, see last paragraph.

More frequent backups would of course help, if their impact is tolerable. They also need to be tested to actually exist and work.

At least LVM snapshots apparently are cheap enough that they can be done out-of-schedule just to get slightly newer data to staging, so they likely could have been done more often (but they probably weren't thought of as backups, which is why 24 hours seemed enough and nobody had a prepared plan to get back to production from them).

Similarly, Azure-side snapshots are mentioned as not enabled for the database hosts. Maybe that's not viable to do (they had performance issues with Azure, and I don't know how much overhead they case), maybe just something they forgot to set up.

Others have asked "why is there only one replica", which also seems like a good question (but my experience with database replication is close to non-existent, and maybe the higher load of more replicas would have caused other issues. Don't know.).

My point is more that I don't have the impression that the 24 hours are a figure that they arrived on by evaluating what "service level" they could achieve, but more a result of someone at some point setting up a backup with some interval. I don't think 24 hours is a figure where they can say "that's the best we currently can achieve", or even "that's what we planned for", but "that's what we have because that's what we have and nobody has paid closer attention". There is at the very least a cultural angle to it, which is why how they work could be relevant.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by jon-wood on Friday February 10, 2017 @03:43AM (#13614086)
One job I worked in had a slave set up with 30 minutes if replication lag intentionally applied to reduce the impact of someone accidentally nuking a table.

Then as someone else mentioned you can stream your replication log to storage elsewhere giving you the ability to do point in time restores.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by problems on Thursday February 09, 2017 @12:41PM (#13608941)
Use that live database replication along with LVM, Btrfs or ZFS snapshotting every hour and purging the old one. Take a full, proper backup every hour if you can or every day if you can't. If you can afford to, do a SQL dump at reduced priority and compress it instead of a binary copy as it's easier to check a text dump.

Anyone with more experience with large databases have anything to add or any concerns with this sort of scheme?

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by toomuchtodo on Thursday February 09, 2017 @12:37PM (#13608899)
Hourly snapshots that you purge every 24 hours (after the nightly is successfully taken).
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by cookiecaper on Thursday February 09, 2017 @12:26PM (#13608778)
It's unfair to characterize the entire organization by one failure, but their data loss issue is not attributable to some obscure, convoluted technical issue or freak accident. The simple fact is that they didn't verify that their backup and recovery infrastructure was working correctly. Moreover, they didn't have any automated monitoring confirming that the backups even appeared to be working. Those are things that every moderately-experienced IT person knows needs to be in place and occurring routinely.

If you look up the doc, they have like 6 different places where they looked for backups and each point is like "We should have these backups, checking for them now... woops, it hasn't been working."

The only reason GitLab's data loss wasn't even more disastrous is that an employee just so happened to take a manual snapshot 6 hours before the incident occurred.

Now, I'm sure we've all been in situations like that. I don't think it necessarily reflects on the individual technical employees of GitLab. I think it says more about the organizational and management resources at the top. It was their job to ensure that the appropriate resources were committed to data integrity, and they failed to do so.

90% of us probably work at places with similar problems, but publishing "the secret to [managing technical employees]" fresh on the heels of such a spectacular organizational failure is probably unwise.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by overcast on Thursday February 09, 2017 @12:37PM (#13608893)
Are you kidding? How is not ensuring your backups have EVER worked, or have even been setup, considered "bad luck", or "anything particularly more wrong". We're not even talking testing backups, literally if they have even been setup to run at all. The only reason this wasn't a complete meltdown, was that the tech decided he'd snapshot before he did work six hour earlier.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by cjbprime on Thursday February 09, 2017 @12:23PM (#13608728)
I think you're confusing sympathy and solidarity towards their engineers (which I was very glad to see) with "wow, this could happen to anyone". Quoting from their post-mortem:

> So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place.

It's not true that having zero tested working backup/restore strategies happens to everyone. It's a catastrophic failure in process. Let's not blame any specific engineers, but it seems totally appropriate to blame the CEO (who was the subject of the puff piece!) and decrease your trust in the company appropriately.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by wheelerwj on Thursday February 09, 2017 @12:19PM (#13608692)
Look, everyone is supportive of gitlab and how they recovered. but to come out boasting after being down for that long is a hard sell.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by sytse on Thursday February 09, 2017 @06:11PM (#13611912)
I totally agree that we should not be boasting but figuring out how to improve our service.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by ancarda on Thursday February 09, 2017 @12:10PM (#13608596)
It may be due to downtime earlier today (this being only days after the data loss). Things don't seem to be going well for GitLab right now.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by wehadfun on Thursday February 09, 2017 @12:40PM (#13608937)
This user account is not active or something and his tone is a little offensive but I think he makes a point. Writing everything down does nothing if no one reads it.

writing things down is not the problem... failing to read the things they write down is the problem. they probably don't have time to read, because they are too busy always writing. they can't find what they need to read in the sea of endless wasteful and pointless drivel that never should have been written down in the first place. you're all idiots.

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by snarf21 on Thursday February 09, 2017 @12:07PM (#13608565)
Scale is freaking hard. It is absurd to say they figured out the secret after a 7 layer failure on disaster recovery, even more so when one of the main reasons for source control is to have disaster recovery and the ability to rebuild old source.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by mynameisvlad on Thursday February 09, 2017 @01:30PM (#13609465)
Wat?

How is their remote working practices at all related to the database disaster?

For that matter, how are either of those related to source control and rebuilding from old source? What would rebuilding source help with when you just deleted your production DB?

Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by emondi on Thursday February 09, 2017 @02:04PM (#13609810)
Not talking about this case specifically, but it could be that remote working makes defining reponsibilities harder and that could lead to not making sure that there are proper backup/restoring procedures in place.
Re:GitLab’s Secret to Managing Employees in 160 Locations: W... by aedocw on Thursday February 09, 2017 @02:10PM (#13609864)
I don't understand why hearing some effective management practices for a distributed team would not be welcome here at any time. Their recent mistake dealt with a specific technical process and some failures around it. That incident in no way invalidates the entire company.

As a remote worker myself, I'm always keen to hear about how teams make it work. I'm also a huge advocate for distributed teams (and nobody having a commute), so personally I'm really glad this link was shared today.