squarecog

Here are a couple of talks about orchestration and choreography, workflows vs sagas, etc.

Both very technical, both with very little "sell" despite being given by a Camunda co-founder and a Temporal principal eng.

https://www.youtube.com/watch?v=zt9DFMkjkEA "Balancing Choreography and Orchestration" by Bernd Rücker

https://www.youtube.com/watch?v=EaBVzjtSK6A "Building event-driven, reactive applications with Temporal: Workflows vs Sagas" by Dominik Turnow

LLMs are Foundational Models, but not all Foundational Models are LLMs.

If you are interested in this space, it does probably make sense to look up what Stable Diffusion (StabilityAI) and Claude (Anthropic) are. Not recognizing them is fine, the pace is pretty incredible in this space and there's lots to learn, but it doesn't make the point you think you are making.

LinkedDataHub, a "RDF-native notebook", is not to be confused with LinkedIn DataHub, which is a metadata store/crawler/ui for your data systems: https://datahubproject.io/.

I should've added, there's an obvious example for the "SaaS control plane" separation, which is equivalent: "stop processing job X that is destabilizing the cluster" should be processed without needing to fight for resources with job X. Same for ACL changes, user deactivations, etc etc. It's generally a good idea to have your control stuff not be subject to whatever instabilities you might be controlling against.

These terms usually show up in the context of networking protocols. Cloudflare has a very quick explainer: https://www.cloudflare.com/learning/network-layer/what-is-th.... To make it even shorter: a control plane is where all the coordination that controls activity (data) happens. The data plane is where the data actually moves around.

Explainers seem to not cover _why_ you would want to separate these "planes". There are several reasons, and I'm no authority, but for starters: * control messages will have different expectations around them: their amount and frequency, delivery guarantees, urgency with which they are processed. Treating this traffic separately means you can engineer appropriately for data and control traffic. * last thing you want is the control message "stop processing traffic from IP x.x.x.x port y" to be stuck behind traffic from said IP/port...

In this context, the meaning is somewhat different. They are referring to administrative traffic vs "actual work" traffic. Auth, billing/accounting, configuration updates, that sort of thing. If you are running a SaaS, and your customer is very security conscious and wants none of their precious data to ever leave their VPC, you have 2 options: deploy your software into their VPC completely, making it hard to do a variety of things like upgrades, and increasing complexity; or you can separate control actions from your "worker nodes" and storage, and only deploy the latter into the VPC. You can then work on your control panels, monitor usage, continuously evolve various admin panels and config options, etc, using normal SaaS approaches while the security conscious customer knows that their core data is not leaving their virtual walls and only "bob ran a thing and stored results" goes to the vendor.

This post is about abstracting out common bits of how one implements that, and allowing SaaS offerings to provide that sort of separation easier.

Now? No idea. At the time I left in 2016, all Storm usage was replaced by Heron, that was the point of making it API-compatible.

I was in charge of the Twitter data platform team at the time we developed Heron and deprecated Storm. The Mesos component of your retelling is not quite right. Take a look at this comment I wrote around the time we started talking about Heron, addressing the same misconception: https://news.ycombinator.com/item?id=10056479

Donated to Apache in 2018. https://blog.twitter.com/engineering/en_us/topics/open-sourc... (and open-sourced in 2016, having started development in 2014).

Why is this odd?

If he started two years later and there was not a trace of the Prize work at the company, that would be an indicator that the competition was not important. If he started and could still see knock-on effects from the competition, that's an indicator that it was important.

Plus, he didn't just start at Netflix. He "took over the small team that was working and maintaining the rating prediction algorithm that included the first year Progress Prize solution."

Yeah, that sounds like he has some authority on the matter.

> So basically all research projects have to redirect funds to cover part of the "open access fees"

Yes, the cost of publishing results, as well as review, editing, etc, should be incorporated into the research budget (researchers can of course choose other means of sharing their findings than through publication). In an open access model, the publishing costs are offset by money saved in not paying for access to other articles the research project needs. I would hope that opting out means that projects have to pay fees on articles they'd otherwise get for free through this agreement, rather than riding on everyone else doing their bit.

One would imagine the fee schedule is set in the agreement UCB reached with Springer (presumably the agreement will be published at some point...)

While HDFS is indeed used for exporting old data and storing some partition mapping metadata, it's clear from the blog post that MetricsDB is much more reliant on BlobStore as well as MetricsDB-specific services.

> The servers checkpoint in-memory data every two hours to durable storage, Blobstore. We are using Blobstore as durable storage so that our process can be run on our shared compute platform with lower management overhead.

I understand your architecture criticism, and think it has merit, but I'm not sure why Apache gets dragged into that. Apache Airflow is in Python. Apache Arrow is in C. CouchDB is Erlang.

There's a ton of projects Apache Foundation hosts that fit your description but it's a mistake, I think, to confuse individual projects with Apache in general. Bad enough that people confuse the license with the foundation.

Can you explain which part, in your opinion, is facepalm worthy?

In your example conversation, the developer starts off by demonstrating that they are not good at thinking through how long things take them, by giving an impossibly optimistic estimate. Even the PM, whose job is not to figure out how to build stuff or how long building stuff takes, knows the estimate is wrong, and pushes back on it, pointing out dependencies the Dev appears to not have considered.

The dev comes back with another estimate, also provided on the spot, without actually thinking through all the tasks, problems, and potential ways to streamline / accelerate. The PM, having just had the dev's lack of thorough thinking with regards to complexity estimation amply demonstrated, asks the Dev to think more carefully, a third time.

Are we facepalming at the dev who can't be bothered to actually consider what it will take to get the thing done, or the PM who for some reason accepts the 3rd estimate, even though the dev appears to just be randomly generating numbers? Both?

I helped build Twitter's data platform, 2010-2016.

There isn't an "analysis server" and analyzing user activity is not done on a "user database backup" at Twitter's scale, though indeed that's a common way that would be done for smaller businesses.

By the way, if by user db you literally mean the db with user accounts, that's not the right data source -- you want the user _activity_ db to count active users, and for high-scale applications, those are different things. Presumably user activity updates are orders of magnitude more frequent than user object updates. You don't want to thrash your user db by constantly updating some "last seen at" field. Put that stuff somewhere else.

That said, it's true that counting is simple, it's just a Hadoop / Spark / distributed computing platform of choice job. Filter, distinct, count. It's not even hard in real-time if you have enough ram or are ok with approximate counts with bounded error, thanks to Storm, Heron, Flink, etc.

Defining what exactly constitutes an active user and catching edge cases such as this Digits thing is where things get tricky; the number of weird scenarios that cause under/overcount for what seem like reasonable and straightforward definitions would surprise you.

@baddox nailed it.

(ex-twitter engineer) I left before this project got started, and do not have any insider info on how they did it. Given what I know about the number of places tweet length assumptions were built into, it must have been a large, cross-team effort. It likely required thoughtful problem-solving.

Take, for example, search. An early iteration of Twitter search relied on this limit to pack term positions into 8 bits (Source: https://www.umiacs.umd.edu/~jimmylin/publications/Busch_etal...). 280 > 256, so if this was still around, the whole approach had to be rebuilt, and the indexes recreated. That's ... non-trivial. And that's just one subsystem.

hdf5, Feather, Arrow, protobufs, json, xml -- all solve the problem of binary representation of data on disk. They all leave the question of how to map said data to a specific problem domain up to the developer.

Projects like ONNX define said mapping for a specific domain (in ONNX's case, by agreeing on a proto schema for ML models, and its interpretation).

To use a simplistic metaphor: protobufs are the .docx format; onnx is a resume template you can fill out in Word.

Stopping when you hit 95% confidence is a classic failure mode. Yes, if you are doing classic t-test based A/B testing, you have to wait until a pre-determined threshold; otherwise, effectively, by looking at the p-value and stopping when it hits 95% confidence, what you are doing is ignoring all negative results and accepting the first positive result you see -- clearly, that's bad science. I'm simplifying the math here, but this is the general notion.

You can see a demonstration of this in practice: http://www.gigamonkeys.com/interruptus/ (sit back and watch your false positive rate, aka "bogus a/b testing results", grow).

You may be interested in what Wes McKinney has to say on the topic: http://wesmckinney.com/blog/outlook-for-2017/ (you can read/write using pyarrow)

Yes: https://twitter.com/aneeshs/status/773547228694589440

"We use GraphJet for serving contextual, personalized content recommendations in real time, such as "You May Like", emails, notifications etc"

Hi, I manage the data science team at Zymergen, and I can tell you we are working on these topics (and hiring). Shoot me an email if you'd like to chat -- dmitriy at zymergen.com

Note how the creator of Apache Crunch is on the Dataflow committer list.

co-author of blog post here.

We covered some of this, at a high level, here: https://blog.twitter.com/2015/the-what-and-why-of-product-ex...

Certainly not everything needs to be A/B tested -- depending on the nature of the change and the kind of insight one is looking to get, MAB may be better, if it applies, or a number of other approaches can be more beneficial, including not running an experiment at all. After all, Twitter itself came from a side project at the podcasting company Odeo. If Twitter was a/b tested, I'm sure it wouldn't move any podcasting metrics in a meaningful way :).

I'm obviously biased -- but I've seen really important insights gained from A/B tests, and wouldn't dismiss them too easily.

Co-author of the Twitter post here.

I don't necessarily disagree with you! Though I hope you are not calling Twitter a fly-by-night marketing startup :).

As a company, we've found A/B testing very valuable, but it certainly has limitations, and pitfalls that one must be fairly experienced to avoid. You can see some descriptions of what we do to achieve that in other posts in that series. But it's fair to say that this takes some work (like the work described in the "second control" post) to avoid badness, and it's fair to question if other methods can get us to the same end result -- ability to evaluate impact of proposed product changes -- cheaper, or more accurately.