News for hackers. Stuff that matters.
Re:Meta developer tools: Working at scale by optymizer on Wednesday June 28, 2023 @11:36AM (#36508019)
For what it's worth, as a Meta employee, I generally disagree with this comment. I mean, calling it 'inferior' is just laughable. It's way better than what open source projects have access to and it's way better than Amazon internal tooling (is that 'industry standard' enough?).

The sapling workflow is better than git. I was skeptical when I joined, but between no branches and excellent stack and merge support, it's just a better, more intuitive workflow.

Eden is fast. Crazy fast, in fact, if you look at the size of the monorepo. The comment about small files is odd to me. Source code are small files. The entire monorepo is just small files.

Now if we are going to complain about something, it would be the custom Android environment we've got. Android Studio integration is actually clunky, bordering on non-existent. Compared to writing vanilla Android code, which I have done for years before joining Meta, Meta folks wrapped almost every API, so all your pre-existing Android knowledge is useless. I've never been so unproductive when writing Android code.

Re:Meta developer tools: Working at scale by PreachSoup on Wednesday June 28, 2023 @11:50AM (#36508254)
I agree. The tooling is not as good as Google's(expected, Google's internal ides are incredible). But it's definitely better than the rest
Re:Meta developer tools: Working at scale by MikeTheRocker on Wednesday June 28, 2023 @11:57AM (#36508349)
Having also written Android code at Meta, I 100% agree.
Re:Meta developer tools: Working at scale by Solvency on Wednesday June 28, 2023 @02:42PM (#36510696)
What's the short version of why Meta did this, wrapping so much up in this internal nonstandard format?
Re:Meta developer tools: Working at scale by nsonha on Wednesday June 28, 2023 @03:25PM (#36511345)
I'm guessing same to why everyone else does that: they want to abstract away the platform. To some people platform intricacies are uninteresting comparing to the problem they are solving, so their abstractions aim to express their problem domain, and hide everything else.

For example for apps you may like to deal with abstractions that can express navigation and pages. They are the same across web, iOS, Android. Unlike Activity, Fragment, View (is that still a thing, I stopped doing android 10 years ago lol) and some slightly different set of abstrations over iOS.

Re:Meta developer tools: Working at scale by billjings on Wednesday June 28, 2023 @06:03PM (#36513266)
Often, performance.

When I was at Meta, if you could measure it, you could ship it and mark that as a win in your performance review. So a lot of projects got shipped off of A/B tested metric wins.

Re:Meta developer tools: Working at scale by calvinmorrison on Wednesday June 28, 2023 @01:00PM (#36509268)
Fastmail had great developer tooling. Give yourself about two minutes from a issued command via slack or CLI and you'd have a brand new tagged developer box the live system that was a self contained infrastructure in a box of everything from the imap server to the user interface. Another command you could get as many copies for whatever development branch, etc.

The entire system replicated the real system (not user data but the software) and you could spin up test addresses and accounts and all sorts trivially. Beautiful!

Re:Meta developer tools: Working at scale by mandeepj on Wednesday June 28, 2023 @03:06PM (#36511050)
> get as many copies for whatever development branch

Why would someone need a more than one copy of a dev/feature branch?

Re:Meta developer tools: Working at scale by calvinmorrison on Wednesday June 28, 2023 @03:39PM (#36511536)
Because its free? You could have a beta server spun up on the fly to do some QA while your dev server is still kicking. It's just spinning servers off of git tags
Re:Meta developer tools: Working at scale by kevan on Wednesday June 28, 2023 @01:56PM (#36510109)
>and it's way better than Amazon internal tooling (is that 'industry standard' enough?).

It's slowly changing but I wouldn't consider Amazon's tooling to be industry standard by any definition.

Re:Meta developer tools: Working at scale by zug_zug on Wednesday June 28, 2023 @04:58PM (#36512489)
>> Amazon internal tooling (is that 'industry standard' enough?).

I'm not sure, maybe?

When I say industry-standard tools I mean "Best you can purchase." So I mean a UI as good as github, a platform as good as AWS (meta's doesn't hold a candle to AWS), full-text log searching as good as splunk, chat as fast/searchable as slack (workplace doesn't hold a candle), video chat as fast/clear/clean as Zoom (workplace doesn't hold a candle)

Some problems are harder at scale (feature toggles interactions just get harder with size), but some of them are just a mess of their own making (restarting taking 30m, butterfly rules having >20m+ delays, eden not being optimized to work with buck, which needs to read tons of files out of tons of directories)

Re:Meta developer tools: Working at scale by vlovich123 on Wednesday June 28, 2023 @05:05PM (#36512595)
GitHub's developer workflow is a joke compared with the ticketing, CI, and code review tooling at Meta. I found the tools at Meta made it a lot smoother and faster to write and review code. Also, their automated flaky test detection, suppression, & ticket issuance/resolution is no joke. One of my favorite features. Their feature flagging system was also on point. Also, Workplace is a fantastic replacement of Wiki that surfaces relevant & interesting content to you. Their chat tools are way better than Google Chat / Microsoft Teams (not sure how it stacks up against Slack since I'm not a huge user of them).

I agree the observability tools I interfaced within Meta was subpar a few years ago, but I think you're being ungenerous to take that and extend it to the coding tooling specifically.

Re:Meta developer tools: Working at scale by erik_seaberg on Wednesday June 28, 2023 @05:44PM (#36513060)
Suppressing flaky tests is a little terrifying. Google would just retry them a few times, which is slow but only delays requesting a code review.
Re:Meta developer tools: Working at scale by vlovich123 on Wednesday June 28, 2023 @08:23PM (#36514695)
It determines flakiness in aggregate across all builds. It also does automatic retries but it does so for the purposes of marking a run as flaky which it then remembers. And it's your team's responsibility for making your tests stable. If they're not stable, they're not going to fail builds which is the correct response (a ticket is filed & it's your team's job to keep those tickets under control). The neat thing is also that if you fix the test and it's no longer flaky, it gets automatically added back in & the ticket is closed (in case you forgot).

https://engineering.fb.com/2020/12/10/developer-tools/probab...

Re:Meta developer tools: Working at scale by erik_seaberg on Wednesday June 28, 2023 @10:17PM (#36515551)
If a test goes from “flaky” to “never succeeds,” I absolutely want to fail builds. To me it seems pretty common for large integration tests to be flaky, and I don’t know that it’s always feasible to avoid without mocking so much that the test becomes meaningless.
Re:Meta developer tools: Working at scale by vlovich123 on Thursday June 29, 2023 @12:37AM (#36516335)
What makes you think that’s not the case?
Re:Meta developer tools: Working at scale by erik_seaberg on Thursday June 29, 2023 @01:02AM (#36516504)
Maybe I misinterpreted “flaky test detection, suppression” and “not going to fail builds.” You retry until a confirmed success before merging?
Re:Meta developer tools: Working at scale by vlovich123 on Thursday June 29, 2023 @02:18AM (#36517006)
I think flaky tests would be always retried and if they start failing 100% then it’s marked as a pure regression. But they don’t block merges if they fail (ie it might be done only after the merge). Not 100%. But generally yes - the culture is to incentivize people to fix their flaky tests and it’s one of the things EMs are judged on.

> Ultimately, our goal with PFS is not to assert that any test is 100 percent reliable, because that’s not realistic. Our goal is to simply assert that a test is sufficiently reliable and provide a scale to illustrate which tests are less reliable than they should be.

Re:Meta developer tools: Working at scale by jarjoura on Wednesday June 28, 2023 @05:49PM (#36513111)
Phabricator w/ "arc & jf" were my bae. Wish they would fully open source all of it or expose a public version of it to compete against GitHub.
Re:Meta developer tools: Working at scale by zug_zug on Wednesday June 28, 2023 @09:19PM (#36515124)
>> GitHub's developer workflow is a joke compared with the ticketing, CI, and code review tooling at Meta.

??

Re:Meta developer tools: Working at scale by vlovich123 on Wednesday June 28, 2023 @11:11PM (#36515887)
Completely consistent and unified UX and text editing (GitHub has this, atlassian doesn't). All text editing is rich multi-modal that is basically what you see in Facebook posts (i.e. you can add a bunch of images easily, it all previews immediately) w/ programmer-friendly extensions (e.g. code formatting, markdown w/ live preview etc).

The ticketing system had a rich tagging system (which GitHub has). The notifications system was completely unified and extremely useful - Workplace activity, relevant ticket updates, PRs to review, reviews, etc - all relevant and timely. From what I've seen, GitHub tends towards just being a useless uncoordinated firehose.

Finally code reviews. Having a code review for every commit is a bit to get used to but their tooling makes it super easy to post a stack (i.e. PR). Each commit in the PR can be independently reviewed & merged. This means you can land improvements progressively as code reviews complete (i.e. short cleanup commits go in faster while larger commits are reviewed longer) This also encourages people to size the commits appropriately & to be standalone.

Re:Meta developer tools: Working at scale by rsdbdr203 on Wednesday June 28, 2023 @09:49PM (#36515346)
>Their feature flagging system was also on point.

Which one... there are like 3 (or more)? The ones I worked with were the single biggest cause of SEVs. I wrote some tooling around trying to make it better, but it honestly needed to be completely replaced.

>Workplace is a fantastic replacement of Wiki that surfaces relevant & interesting content to you.

Yea? I found it horrible to have to read through post-after-post-after-post to try and keep up-to-date with what's going on. Not my way of ingesting information.

>Their chat tools are way better than Google Chat / Microsoft Teams (not sure how it stacks up against Slack since I'm not a huge user of them).

100% disagree... telling my dog to bark at the neighbor because I want something from them is better than Workplace chat. Constantly switching between internal beta and production to try and get the features I wanted, with something stable. Go ahead and send me a link to that message in a chat... oh yea, you can't :-|

Re:Meta developer tools: Working at scale by wilde on Wednesday June 28, 2023 @05:11PM (#36512672)
> Eden is fast. Crazy fast, in fact, if you look at the size of the monorepo.

You’re measuring the wrong thing. The user actions are slow and take several seconds to complete usually. No one gives a shit how big the repo is. They care how long their operation takes.

Re:Meta developer tools: Working at scale by lanza on Wednesday June 28, 2023 @09:40PM (#36515276)
There isn't a tool that can do this job faster. git takes a few seconds once you're at half a million commits. Facebook has many repos checked in in their entirety that are bigger than that.
Re:Meta developer tools: Working at scale by wilde on Thursday June 29, 2023 @07:28AM (#36519010)
That’s like saying cargo ships are fast because they carry lots of stuff. It’s still slow in absolute terms to get my stuff from China.
Re:Meta developer tools: Working at scale by optymizer on Thursday June 29, 2023 @12:51AM (#36516422)
Compared to what? Not having a monorepo? Sure, but you run into different issues with that approach. Eden is the best solution there is for a monorepo of this size.
Re:Meta developer tools: Working at scale by wilde on Thursday June 29, 2023 @07:31AM (#36519031)
Yeah compared to not having a monorepo and buying more SSDs. The separate repos are much nicer to work in, but harder for the infra folks to manage. I miss having dataswarm as a separate repo. I will miss IG’s.

I understand the tradeoff. That doesn’t mean it’s “fast”. It’s slow but the other alternatives are slower.

Re:Meta developer tools: Working at scale by nappy-doo on Wednesday June 28, 2023 @06:01PM (#36513230)
I've worked at Google and Meta (née FB). They are inferior.