Mentioning splunk, I'm assuming you are using more paid industry tools. I suspect that opens things up a bit more, but realize that most of the industry doesn't use those due to prohibitive pricing. And I don't even know what the appropriate paid tool for building would be.

We seem remarkably primed to just hate whatever tooling we have at our disposal in ways that baffles me.

Regardless, many companies underspend by an order of magnitude on dev tools. If a tool makes you 1% more efficient, it's worth them putting 1% of your annual salary into it (multiply this by engineers at the company who need the tool). Soon you'll see that a tool costing hundreds of thousands to license a year is often a steal.

So you may say "oh splunk is overkill and unfair expectation" but in their CI/CD system, if your job failed (frequent occurrence) and you wanted to search for something in there you'd manually DOWNLOAD a 300 meg file to your machine and grep it for errors.

Basically a fang engineer's time costs $3 a minute minimum, so if a query could save 1 minute then the break-even cost is paying up to $3 to run that query.

More amusing is when you see someone build a giant Elastic Search pipeline to a Kibana dashboard so that they can get what would have been table stakes metrics if they had used the normal service templates. Without a ridiculously large bill.

Or folks that think they can warp around the high cardinality traps of making a metric out of everything. Assuming if you can make a system that works for the testing environment, of course it will work when you open the floodgates.

Seen people argue that sampling shouldn't be used, "because you could miss data?" Reservoir sampling is a thing, for very real reasons.

At any rate, I don't mean to just yell about splunk, from all I've heard it is nice. I am annoyed that folks seem to ignore the OLAP and OLTP divide, such that they think your metrics system should somehow be optimized for both. At the same time.

Pulling back to builds, though. What, exactly, is the gold standard in industry? I have yet to see it. Python builds, in particular, strike me as not good.

Hear me, nameless internet stranger, that you might learn from my mistakes: This is not true. Efficiency is only worth money if it increases profits - concretely this means your efficiency gain must result in the following:

1. Delivering features faster

2. Delivering features with meaningfully higher quality

3. Delivering the same features with lower headcount costs

(2) is hard to measure, so you can generally only sell on (1) and (3). During the 80s the business world learned the hard way you can throw away a lot of money on useless efficiency. If that interests you I recommend "The Goal" by Goldratt for a fictionalized account of those learnings.

* There's also a subtler point that corporate finances may mean that even if efficiency is perfectly captured, a 1% efficiency increase may only be worth 0.1% of your salary.

I’ve been through a few savings exercises where everything is gonna go open source dashboards etc. Once the guy who understands how it all works disappears, it migrates back to splunk.

One security incident response will pay for the entire splunk infrastructure.

If you can, that seriously awesome. The point is, many can’t.

Some of this is just having worked in the industry for a while. Guess what, 15 years ago "Put all your code in a web service running on AWS" was not nearly as slam-dunk a proposition as it is now; and tools optimized for on-prem were faster and more reliable.

The other thing is that you don't always have to reinvent things to the point that Meta does. You can just wrap around an Open Source project, or even contribute to it. This is what all of my previous jobs did -- make custom tooling from industry-standard OSS building blocks; and contribute back when it made sense for the larger community.

I actually think Meta was actually trying to do this with Mercurial a while ago [1], you probably have a much better idea than us as to why that didn't work out.

----------------------------------------

[1] https://engineering.fb.com/2014/01/07/core-data/scaling-merc...

It did work out.

One guess that I can make from Sapling docs is that perhaps FB/Meta needed an ability to pull in Git repos where needed; and I can see Mercurial devs not being super enthusiastic about that being a first-class workflow in Mercurial itself.

The needs of a large internal code base are often quite different than what open source projects need.

Anther issue is that tools like Git and Mercurial need to be compatible with positively ancient repositories and can't really break backwards compatibility whereas within Meta, it is easier to move faster and deal with breaking format changes since developers's checked out repos on their laptops can be silently upgraded since the machines are well managed with Chef and can be upgraded behind the scenes. In the outside world, you can't assume people will be running the latest version and can't upgrade them.

Finally, Google is also moving away from Mercurial. Perhaps the JJ developers want to respond why (and check out Jujitsu -- it's a novel and very interesting system y'all should check out).

Sure. I presented about that at Git Merge 2022. https://github.com/martinvonz/jj#disclaimer has links to the slides and the recording from there. I'll summarize the problems we have with Mercurial here:

1. Performance/scalability. That's partly because Python is slow. Both the Mercurial project and Meta have rewritten many parts of it in C and Rust as a result. Maybe more importantly, there are many assumptions in Mercurial's design that don't scale well. We have extensions for downloading only a slice of the repo. We slice it in both file space and in version space. However, it can get very expensive to change afterwards. For example, checking out an old revision that they user hasn't previously downloaded is very slow (it requires rewriting all local revisions after that point).

2. Consistency. Mercurial was designed for local file systems, so when we store repos in our distributed file system, we run into write races that can corrupt repos.

3. Integrations. We integrate with Mercurial by running the `hg` binary and parsing the output. That's unnecessarily complicated and slow.

We also see several opportunities by switching to jj (in addition to hopefully fixing the problems above):

1. Simpler workflows. Things like: working-copy commit (no "dirty working copy" errors, for example), undo, first-class conflicts (no interrupted rebases, for example). See the GitHub project for details.

2. Cloud-based repos. The repos will be stored in a database instead of being stored in files on top of a distributed file system. That makes them much easier for our server to work with, and it opens up for many kinds of integrations that were not feasible before.

3. Simpler architecture. We designed jj from the beginning to be easy to integrate with our internal systems, so there should be much fewer workarounds.

4. Simpler code base. You can typically add a command without worrying about concurrent commands, a dirty working copy, or conflicts. An example I like to mention is how I spent about two weeks trying to implement a command for amending into an ancestor commit in Mercurial. Then I implemented a more powerful version of that (can move changes from any commit to any other commit) in an hour in jj.

Another thing is that sometimes it's truly impossible to use one-size fits all for some tools if your company scale is large enough. I've heard from friends at Amazon that the dev experience and tools are totally different in the hardware space than it is for those working in the retail or AWS orgs.

That's why big orgs are now opensourcing their tools. But it's not to clean up all "too internal" stuff from a tool.

I also felt like a lot of the developer tools looked nicer than their google counterparts at the surface but had major reliability problems under the hood where you would need to do a lot of turn it off and on again style operations.

It’s possible to run a really disciplined Slack where updates go in a ready-only channel, and people broadcast their work appropriately - but it’s really hard. Most people default to non-discoverable DMs and then stream-of-thought in public channels, which are both bad for discovery in their own way.

Posts end up being a bit more considered than a slack message, but still lightweight, and more discoverable than docs.

Shame Google nixed their “Google plus for workplace”, I would love to see this workflow used more widely.

For many, the world has moved on a lot since, and those tools feel obsolete but so embedded in practices that it’s unthinkable to apply better approaches. One example of that is tracking data sources: legacy is slow, dysfunctional, and even fairly straightforward questions time out because you have to load so much meta-data. That’s because no one really uses it: most of that information is carried by the Data engineers who have built the system, and they are over-whelmed with questions that could be answer with a good system, but derive power over analysts. They benefit from it through comments they copy in their review, so fixing it isn’t anyone’s priority.

Others have evolved because the internal demand pushed things forward. PMs want Deltoid to work, so that system has moved to be state-of-the-art or interestingly unique in many ways: scaling, not implementing MCC, integration with the metric definitions.

> most of that information is carried by the Data engineers who have built the system, and they are over-whelmed with questions that could be answer with a good system, but derive power over analysts.

Is that the kind of question you're talking about here?

As someone who vastly prefers Slack over all its competitors, I'm very interested in hearing how Meta's compares. What makes it better than Slack for you?

However the combination of the "newsgroup" and chat makes it kinda better than slack on it's own.

Would I use workplace at a new startup? probably not. I would use slack though.

I’ve seen sub 500 employee companies and Slack kicks ass, but it seems once it goes over 1k it’s like glhf managing channels and bots.

Butterfly bot integration to workplace was really something.

Workplace def overkill sub ~ 600 users though.

On the other hand, building all these connections between the disparate apps is basically my job, so it’s got its positives.

Once your company reaches a sufficient size (maybe 300+ employees?), Slack channels just become so clunky and tedious for trying to keep up with projects and discussions. Using Slack keyword notifications helps cut through the noise, but I really think applying Facebook's ranking/Groups expertise to work-related content has been the best solution I've seen so far. You're no longer inundated with unread channels that may or may not have messages relevant to you, Workplace just surfaces the relevant content (and obviously there's chat/mentions for more pressing issues).

Early career this is pretty huge for growth. I was there until 2019 though, so not sure how it is now.

My point was that the cohesion of all these internal tools makes information discovery frictionless. cross-pollination of functional space in a business is like butter because everything is built by Meta and behind their intern tool. The other companies I've worked at just don't have this. They lack the Eng capacity to build it, unfortunately.

I personally loved things breaking at times. The monorepo and all the tools were so open that enabled me to follow along and try my own fixes in some cases (sometimes being the one to fix it!).

At that point in my career, that kind of exposure was like a rocket ship for personal growth. Others shared similar sentiments.

This is a curious thing to see.

As a mere user, and speaking purely about FB not any other Meta IP…

Other than unbreaking things that break when the OS and browsers change under you, has Facebook shipped even one thing in the last five years?

Don't get me wrong, I know it takes a lot of effort to stand still — forgetting the Red Queen effect was Musk's obvious mistake with Twitter even before stuff broke — but Facebook seems completely unchanged.

But I guess that implies you didn't mean the website itself when you wrote "FB" in the comment I replied to? :)

(One of the most well known consumer brands shipping tooling rather than stuff the consumers use directly is, ah, apt for a company called "meta").

They'd be just as fast with Slack or whatever, for the most part.

EDIT: as someone posted below:

> The open-ness of code, visibility, diffs. It was perfection. Something broke in my env suddenly? OH, I just checked recently pushed diffs that affect my realm. Hey there it is, security pushed something weird. I'll just revert the part that affects me and tag them. No meeting, maybe a SEV for visibility and review, maybe not. Easy peasy.

This is the kind of empowerment developers are given and expected to handle (both the explicitly ability to revert previously committed code and the implicit responsibility that the teams code that was overridden must deal with it once they're notified rather than push back)

To your point, other experiences have shown me the ego that rears it’s ugly head when trying to move that way in an env that didn’t have that culture. At FB there was a lot of candor, but in other environments I feel like I’m going to hurt someone’s feelings in code review, or even just watching what I say in slack messages. People take work too seriously in some companies. It’s so rewarding to have fun with it.

They used to say “Nothing at FB is someone else’s problem”

I hope they still do.

Meta cut through that by focusing on measurable improvements.

The downside is it can create silos, a negatively competitive environment. IE, teams not sharing resources or credit etc because they are only looking out for themselves at the end of the day, because any measurement you don't capture is one you can't claim as yours.

I also argue it can breed short therm thinking. Meta even had special teams from what I understand that were "exempt" from the typical metrics driven review cycles because they'd create the wrong incentives.

I think the general ethos can be really powerful though, but I'd peg it to collaboration and value driven measurements (and value is loose here, I'm not strictly thinking monetary) rather than strict "user based metrics".

These days at different companies it’s all planning and hashing out all those details before implementing, and then guess what? Surprises happen anyway. And in the end we become master planners and never actually build anything. Those blueprints sure do look great though.

Abstraction layers tend to be slow.

What they should focus on is plumbing. Take what exists and connect it in a smart way.

Take React: the idea of applying functional programming and spitting out HTML was a great idea. HTML existed before react. Then they also implemented a virtual-DOM, which was the unnecessary part.

Or react-native: rendering UIs in a functional way is great, but stuffing JavaScript into everything is not necessary.

But nowadays there are plenty of solutions that do not use virtual DOM (or even use a virtual DOM solution that is just plain faster) but still allow you to use declarative rendering, basically the best of both worlds. They are just not as popular or for some people not as interesting to them as React is. Or they are content with Reacts performance already.

Phabricator was pretty sweet... and the internal version of mercurial was a dream!

The sapling workflow is better than git. I was skeptical when I joined, but between no branches and excellent stack and merge support, it's just a better, more intuitive workflow.

Eden is fast. Crazy fast, in fact, if you look at the size of the monorepo. The comment about small files is odd to me. Source code are small files. The entire monorepo is just small files.

Now if we are going to complain about something, it would be the custom Android environment we've got. Android Studio integration is actually clunky, bordering on non-existent. Compared to writing vanilla Android code, which I have done for years before joining Meta, Meta folks wrapped almost every API, so all your pre-existing Android knowledge is useless. I've never been so unproductive when writing Android code.

For example for apps you may like to deal with abstractions that can express navigation and pages. They are the same across web, iOS, Android. Unlike Activity, Fragment, View (is that still a thing, I stopped doing android 10 years ago lol) and some slightly different set of abstrations over iOS.

When I was at Meta, if you could measure it, you could ship it and mark that as a win in your performance review. So a lot of projects got shipped off of A/B tested metric wins.

The entire system replicated the real system (not user data but the software) and you could spin up test addresses and accounts and all sorts trivially. Beautiful!

Why would someone need a more than one copy of a dev/feature branch?

It's slowly changing but I wouldn't consider Amazon's tooling to be industry standard by any definition.

I'm not sure, maybe?

When I say industry-standard tools I mean "Best you can purchase." So I mean a UI as good as github, a platform as good as AWS (meta's doesn't hold a candle to AWS), full-text log searching as good as splunk, chat as fast/searchable as slack (workplace doesn't hold a candle), video chat as fast/clear/clean as Zoom (workplace doesn't hold a candle)

Some problems are harder at scale (feature toggles interactions just get harder with size), but some of them are just a mess of their own making (restarting taking 30m, butterfly rules having >20m+ delays, eden not being optimized to work with buck, which needs to read tons of files out of tons of directories)

I agree the observability tools I interfaced within Meta was subpar a few years ago, but I think you're being ungenerous to take that and extend it to the coding tooling specifically.

https://engineering.fb.com/2020/12/10/developer-tools/probab...

> Ultimately, our goal with PFS is not to assert that any test is 100 percent reliable, because that’s not realistic. Our goal is to simply assert that a test is sufficiently reliable and provide a scale to illustrate which tests are less reliable than they should be.

??

The ticketing system had a rich tagging system (which GitHub has). The notifications system was completely unified and extremely useful - Workplace activity, relevant ticket updates, PRs to review, reviews, etc - all relevant and timely. From what I've seen, GitHub tends towards just being a useless uncoordinated firehose.

Finally code reviews. Having a code review for every commit is a bit to get used to but their tooling makes it super easy to post a stack (i.e. PR). Each commit in the PR can be independently reviewed & merged. This means you can land improvements progressively as code reviews complete (i.e. short cleanup commits go in faster while larger commits are reviewed longer) This also encourages people to size the commits appropriately & to be standalone.

Which one... there are like 3 (or more)? The ones I worked with were the single biggest cause of SEVs. I wrote some tooling around trying to make it better, but it honestly needed to be completely replaced.

>Workplace is a fantastic replacement of Wiki that surfaces relevant & interesting content to you.

Yea? I found it horrible to have to read through post-after-post-after-post to try and keep up-to-date with what's going on. Not my way of ingesting information.

>Their chat tools are way better than Google Chat / Microsoft Teams (not sure how it stacks up against Slack since I'm not a huge user of them).

100% disagree... telling my dog to bark at the neighbor because I want something from them is better than Workplace chat. Constantly switching between internal beta and production to try and get the features I wanted, with something stable. Go ahead and send me a link to that message in a chat... oh yea, you can't :-|

You’re measuring the wrong thing. The user actions are slow and take several seconds to complete usually. No one gives a shit how big the repo is. They care how long their operation takes.

I understand the tradeoff. That doesn’t mean it’s “fast”. It’s slow but the other alternatives are slower.

Though we arguably rely on a lot of Apache products, whatever we use that's internal only is great to work with. Daiq** recently started supporting notebooks, which has been a game changer for us as well as for the teams we work with. Phabricator is great as well, and makes shipping stuff super easy. Only Ben**, the internal notebook solution, I find meh. Especially compared to Google Colab. But the rest has simply been a joy to work with.

For those interested, a former DE made this nice repo that maps internal tools against "real world" products: https://github.com/thijsessens/xmeta2external

However, all the other tooling around code were just superior in every way. I miss Phabricator, and mercurial and the way it seamlessly integrates into a team. I also miss all of the command line helpers that let me manage all of it.

It used to be the case that legitimate teams could form around and focus just on building internal tools for other teams if it improved engineering velocity. Not sure if that's still the case in this new era of layoff-happy Meta, but it was definitely true when I was there.

s/Meta's tooling/Meta/g

s/inferior/very inferior/g

As a backend engineer, the dev experience was just incomparably better at FB. Some things I most most are probably phabricator / stacked diffs workflows, buck dep management with everything in a monorepo, deep integration across all the the tooling etc.