News for hackers. Stuff that matters.
Re:twitter/the-algorithm by Irishsteve on Monday April 25, 2022 @05:12PM (#31160680)
Many people have commented that it is empty. However what they do not realize is that there has never actually been an algorithm and that is why it is empty.
Re:twitter/the-algorithm by bastardoperator on Monday April 25, 2022 @05:25PM (#31160847)
This is most likely the correct answer. I doubt there is a single piece of code or algorithm that controls all of twitter. From my limited perspective it's just a catch phrase to simplify what is likely tens of thousands of lines of code.
Re:twitter/the-algorithm by andrewmcwatters on Monday April 25, 2022 @05:29PM (#31160910)
Well, that's an algorithm. If you have to draw a diagram showing how edge cache nodes affect a user's results based on whether or not they signed in from New York or Phoenix, or if the downtime of a cluster affects the relevance rankings of tweets for a photography enthusiast, then guess what...

That's an algorithm.

Re:twitter/the-algorithm by TimTheTinker on Monday April 25, 2022 @05:41PM (#31161045)
This shouldn't be downvoted. Any black box with inputs and outputs is a function over discrete data, otherwise known as an algorithm.
Re:twitter/the-algorithm by samhw on Monday April 25, 2022 @07:06PM (#31161999)
> a function over discrete data, otherwise known as an algorithm

'A function over discrete data' is a far broader concept / far larger set than 'an algorithm'.

Re:twitter/the-algorithm by bee_rider on Monday April 25, 2022 @06:13PM (#31161432)
I think it is unfortunate that "the algorithm" has become the popularized phrase for the thing, though. Yes, clearly an algorithm which evaluates the entire state of the universe and uses that as an input is still an algorithm. But it is also clearly not what people are looking for.

I'm pretty sure people expect to be able look at The Algorithm and find the part that is trying to destroy <whatever their hobby-horse is>. I bet whatever we see will lean more toward "big dumb pile of A/B tests."

Re:twitter/the-algorithm by bastardoperator on Monday April 25, 2022 @05:46PM (#31161114)
I don't disagree, I might even argue that twitter is comprised of many algorithms. We can argue all software is algorithms, or we can just call it software.
Re:twitter/the-algorithm by andrewmcwatters on Monday April 25, 2022 @05:51PM (#31161174)
That just moves the terminology around and doesn't answer people's questions. Everyone would simply start asking, "what software or lines of code directly affect my timeline?"

The "timeline algorithm" is a perfectly normal way to describe what people ask about.

Re:twitter/the-algorithm by mhh__ on Monday April 25, 2022 @05:51PM (#31161179)
Yes it's an algorithm, but as software engineers we shouldn't pander to people who actually think there is a small flow chart dictating what happens in your search results. It's idiotic or dishonest.
Re:twitter/the-algorithm by donatj on Monday April 25, 2022 @06:46PM (#31161797)
I mean I'm sure Google has somewhere a UML chart of the entire process of ordering results. Your insistence otherwise is disingenuous at best, either that or you've never actually worked on any sort of major software, at least with any level of reliability/dependability.
Re:twitter/the-algorithm by fmobus on Tuesday April 26, 2022 @11:18AM (#31168543)
> I mean I'm sure Google has somewhere a UML chart of the entire process of ordering results.

No, there isn't.

Re:twitter/the-algorithm by mhh__ on Monday April 25, 2022 @06:56PM (#31161891)
Sure they have a chart but it's not "if !idpol -> make woke instead".

The level of discourse I'm trying to convey like what we saw in the Zuck/Pichai hearings in Congress. No understanding of the domain at all.

You or I know how software actually works, your average politician or man on the street does not.

Re:twitter/the-algorithm by ltbarcly3 on Monday April 25, 2022 @05:36PM (#31160989)
Yes there is an algorithm. It may not be concise, it may not be found in one place, and nobody alive may be able to explain it to you, but the existence of timelines that are ordered certainly implies the existence of 'the algorithm' that is used to order timelines.

I mean even if timelines were totally random, or based on some external facts, there is an algorithm that is being used to order them.

This isn't just an academic distinction. Claiming 'there is no algorithm' because the algorithm is intentionally or unintentionally obfuscated or complicated has implications if that claim of 'no algorithm' is accepted. If my algorithm for approving mortgage applications is explicitly racist, I can just spread it's functionality across myriad services owned by lots of teams, make it almost impossible to figure out how it works, and then avoid any responsibility by saying 'there is no algorithm to decide loan approvals'? That would be bullshit!

Re:twitter/the-algorithm by influx on Monday April 25, 2022 @05:13PM (#31160689)
There's not an algorithm to order timelines?
Re:twitter/the-algorithm by jonwinstanley on Monday April 25, 2022 @05:20PM (#31160778)
If it’s a list of tweets ordered based on a kajillion ML data points that varies per user is it still an algorithm?

And does every user have their own algorithm?

And could it be made readable to a human?

Re:twitter/the-algorithm by abduhl on Monday April 25, 2022 @05:22PM (#31160799)
Yes, irrelevant, and the code somewhere is readable.

Take an example like gravity: is there an algorithm for how gravity acts even though it varies per each molecule based on a kajillion other molecules? Of course there is.

Re:twitter/the-algorithm by marcinzm on Monday April 25, 2022 @05:46PM (#31161117)
If I have your DNA can I recreate a perfect copy of you as you are now including your mind? It is after all the code you were made with.
Re:twitter/the-algorithm by vasco on Monday April 25, 2022 @05:59PM (#31161278)
The point is not to recreate it or explain it, or to have a way to say "tweet X will get rank Y" but to open source what there is. It doesn't matter if no single person can explain it, it'll still be open source and readable and the types of inputs will be known. But this doesn't look like a discussion on open source and more about politics at this point.

I for one haven't worked at FAANG and think it'll be super interesting to read this code, I can't believe software engineers are complaining about a potentially super complex bleeding edge codebase for timeline recommendations / ranking being open sourced. This is going to be great reading material if it ever gets published!

Re:twitter/the-algorithm by abduhl on Monday April 25, 2022 @06:08PM (#31161387)
Nobody is trying to “recreate a perfect copy” of Twitter. Do you think that code which produces random numbers is also not an algorithm because it doesn’t recreate a perfect copy of a previous random number?
Re:twitter/the-algorithm by stjohnswarts on Monday April 25, 2022 @06:57PM (#31161899)
No you can't, you can only make a twin, but the mind is shaped by experience also and not just genetics. As is the body.
[dead]
Re:twitter/the-algorithm by zbentley on Monday April 25, 2022 @05:56PM (#31161235)
Computer systems, even AI/ML-driven ones, aren't uncomprehended the way some areas of science that we haven't developed a strong understanding of are.

Systems comprised of (or generated by) comprehensible parts can be comprehended, we just often opt not to put in that work. There's a reason model interpretability is such a significant and increasingly-demanded area of expertise.

Re:twitter/the-algorithm by ironmagma on Monday April 25, 2022 @06:04PM (#31161344)
In that case, the definition of “the algorithm” is the training set.
Re:twitter/the-algorithm by JumpCrisscross on Monday April 25, 2022 @06:18PM (#31161487)
> the definition of “the algorithm” is the training set

Or the trained model itself. There are people looking for intentional bias. But the insidiousness of the problem likely arises from unintentional bias. Letting researchers brute force the ranking models with hypotheticals could be a win win.

Re:twitter/the-algorithm by ironmagma on Monday April 25, 2022 @07:37PM (#31162257)
Is there any reason to not think that like most human-created systems it has a mixture of intentional and unintentional bias?
Re:twitter/the-algorithm by lamontcg on Monday April 25, 2022 @06:19PM (#31161505)
Twitter itself is probably the training set for twitter.
Re:twitter/the-algorithm by ironmagma on Monday April 25, 2022 @07:35PM (#31162249)
In that case, the algorithm is the labels.
Re:twitter/the-algorithm by dmonitor on Monday April 25, 2022 @05:15PM (#31160715)
Turns out, it's been RNG this entire time.
Re:twitter/the-algorithm by noobermin on Monday April 25, 2022 @05:42PM (#31161055)
If that's the case, let it be client side RNG and not an RNG run server side, thanks.
Re:twitter/the-algorithm by legowo on Monday April 25, 2022 @11:55PM (#31163779)
what ups
Re:twitter/the-algorithm by Hamuko on Monday April 25, 2022 @05:21PM (#31160790)

  ORDER BY timestamp DESC;
Re:twitter/the-algorithm by dylan604 on Monday April 25, 2022 @05:27PM (#31160878)
no self respecting PM would ever allow something so trite and simple. this was obviously copy&pasted from stackoverflow. something like this would never web scale, so obviously they need somethign written from scratch preferably in Go/Rust/Node/anythingButSQL
Re:twitter/the-algorithm by talideon on Monday April 25, 2022 @05:24PM (#31160835)
The one true algorithm!
Re:twitter/the-algorithm by krapp on Monday April 25, 2022 @05:21PM (#31160788)
There probably is not "an algorithm" on a site as large and complex as Twitter, no. There are probably dozens if not hundreds of algorithms spread throughout the codebase which affect the timeline for individual users, possibly even code entirely self-generated by ML systems.
Re:twitter/the-algorithm by captainmuon on Monday April 25, 2022 @05:28PM (#31160891)
Well sure there is "ORDER BY" something, and there are batch jobs to calculate this something from different variables. Maybe it is partially a ML black box, but I'm pretty sure they are not flying completely blind. There's got to be an internal Wiki or some PPTs where they list what goes into the mix. Number of likes, retweets. Attached media, hashtags, trending or not. Do they do sentiment analysis and use positivity as a factor? Do they do PageRank so that likes from important people have more weight? Are there manual debuffs for certain topics? Are certain posts removed or added in a second pass? These are all answerable questions in principle.
Re:twitter/the-algorithm by jvanderbot on Monday April 25, 2022 @05:41PM (#31161046)
Maybe not. It could all be event-driven, distributed, and just pulling off various prioritized queues.
Re:twitter/the-algorithm by stjohnswarts on Monday April 25, 2022 @06:58PM (#31161910)
that's still an algorithm
Re:twitter/the-algorithm by jvanderbot on Wednesday April 27, 2022 @04:41PM (#31185517)
We'll, it's ok if you feel that way given this is entirely subjective, but I disagree based on experience.

I disagree b/c: That's a system. An algorithm is designed, a system will emerge from pieces. An algorithm can be defined, a system's behavior has to be characterized post-hoc. You can characterize a system, but only as a black/gray box. An algorithm has invariants and stateful steps, a system has nearly infinite state and nearly zero meaningful invariants.

Re:twitter/the-algorithm by cellis on Monday April 25, 2022 @05:52PM (#31161194)
The trouble is, there could be literally millions of ORDER BY queries. Hell, each user could be given a fine-tuned cnn…
Re:twitter/the-algorithm by onlyrealcuzzo on Monday April 25, 2022 @05:24PM (#31160832)
In the end, it's still code that is being executed, which can be called "an algorithm". Algorithms are under no obligation to be simple or only include "one algorithm".
Re:twitter/the-algorithm by krapp on Monday April 25, 2022 @05:29PM (#31160902)
So then "open sourcing the algorithm" likely means opens sourcing the entire Twitter codebase?
Re:twitter/the-algorithm by baryphonic on Monday April 25, 2022 @05:42PM (#31161068)
At this point, what harm would that do to Twitter? The company's tech has been stagnant for years, and the only real features recently have been attempts to exploit Twitter's network effects to ape Clubhouse & Substack. (Seems successful for the former, kind of a failure for the latter.)
Re:twitter/the-algorithm by stjohnswarts on Monday April 25, 2022 @06:59PM (#31161923)
do we know? has any angy ex-twitter engineer revealed the secret sauce to the world?
Re:twitter/the-algorithm by dragonwriter on Monday April 25, 2022 @05:33PM (#31160951)
The aggregate of a set of systems that maps inputs and state to outputs and state updates is...

still a system that maps inputs and state to outputs and outputs and state updates.

A collection of algorithms is just a bigger algorithm.

Re:twitter/the-algorithm by apgwoz on Monday April 25, 2022 @05:42PM (#31161060)
Turns out they never actually scaled their mailbox pattern. As a result, what you see is what they delivered before timeout.
Re:twitter/the-algorithm by jdrc on Monday April 25, 2022 @05:22PM (#31160797)
just birds poking random tweets
Re:twitter/the-algorithm by WORMS_EAT_WORMS on Monday April 25, 2022 @05:50PM (#31161165)
This would be some pretty funny semantics... but, however it works, message is the same: Transparency for how these work

- Search results

- Comment Order

- Timeline Order

- Trends

- Human vs code

Personalization in general. Big gigantic “why” when it happens to you

Re:twitter/the-algorithm by Diesel555 on Tuesday April 26, 2022 @12:26AM (#31163927)
Of note for anyone confused and reading the comments now, the link was to an empty, but real, twitter repository at https://github.com/twitter/the-algorithm. Now it is a 404.
Re:twitter/the-algorithm by temp8964 on Monday April 25, 2022 @07:27PM (#31162189)
Here’s the play: interpret “the algorithm” in a stupid way, and then claim it doesn’t exist, and then make a joke about it on company repo.

You know what? This does demonstrate the internal problems inside Twitter and shows the need for shakeup.

Re:twitter/the-algorithm by omoikane on Monday April 25, 2022 @06:29PM (#31161617)
That sounds like a variant of this joke: https://picturesofpeoplescanningqrcodes.tumblr.com/
Re:twitter/the-algorithm by I-M-S on Tuesday April 26, 2022 @03:46PM (#31172278)
Joke's on them, in the last 2 years QR codes made the greatest upset of all times
Re:twitter/the-algorithm by andrewmcwatters on Monday April 25, 2022 @05:25PM (#31160842)
If there is a total set of data, and a subset of it is produced for users, there is an algorithm. `SELECT TOP` is an algorithm. A ML model with a trillion parameters is an algorithm.

There is no "no algorithm."

Re:twitter/the-algorithm by sverhagen on Monday April 25, 2022 @05:28PM (#31160884)
You're not wrong. But would it be useful to define better what we mean with algorithm then? Like: we mean "non-obvious algorithm"?
Re:twitter/the-algorithm by andrewmcwatters on Monday April 25, 2022 @05:32PM (#31160942)
You and I might be technical enough people to suss out little details, but I suspect the average person probably also wants to know whether or not something like Twitter testing a feature flag (Yes, really!) is affecting their timeline.

If it affects results in the slightest, I think that's what people are asking about.

Yeah, if the new senior dev screwed up a deployment and now football fans in Dallas are suddenly getting fewer posts about the most recent athlete scandal all because a regional deployment was the only thing doing relevance record keeping, then yes, that's a part of "the algorithm."

Re:twitter/the-algorithm by NaturalPhallacy on Monday April 25, 2022 @06:52PM (#31161852)
We have a leak revealing a "trends blacklist" at the very least. Some of their measures are detectable too: https://taishin-miyamoto.com/ShadowBan/
Re:twitter/the-algorithm by yifanl on Monday April 25, 2022 @05:56PM (#31161236)
As far as I'm concerned, the algorithm is and can only ever be an implementation detail. If the leadership of Twitter changes overnight, it'd be highly likely the types of content that gets recommended will subtly/not-so-subtly change... because the algorithm has been changed.

It's an interchangeable function, it would only be publicized if it's clear to leadership that it wouldn't affect their revenue if people started trying to game towards the published algorithm.

Re:twitter/the-algorithm by bberenberg on Monday April 25, 2022 @06:01PM (#31161302)
I feel like this is a line from The Program (https://www.programaudioseries.com/)
Re:twitter/the-algorithm by AviationAtom on Monday April 25, 2022 @05:48PM (#31161142)
You don't think recent news of the EU compelling social media companies to disclose their algorithms has anything to do with this?
Re:twitter/the-algorithm by layer8 on Monday April 25, 2022 @06:04PM (#31161340)
No, the repository is a joke (IME). Regarding the EU, the details haven’t even been nailed down yet, and the timeline is that it will only become a requirement in 2024 I believe. The goal also isn’t to publish “the algorithm”, but to give researchers and civil society representatives access to the training data and information about the most important factors controlling the algorithm, so they can make assessments on the basis of that. It is not about open-sourcing code.
Re:twitter/the-algorithm by AviationAtom on Monday April 25, 2022 @07:08PM (#31162019)
They don't necessarily have to publish code to the repo though, they could just as well publish papers that document their algorithm.
Re:twitter/the-algorithm by bilekas on Monday April 25, 2022 @06:06PM (#31161363)
This, personally I was thinking in light of the new upcomming sale, the current Twitter staff might decide to 'open source' some algorithm. Which would be an effective method of protest against a sale like that.

But there definitely is a relationship algo that could be considered theirs, like all social medias inflating the bubbles users all feel.

Re:twitter/the-algorithm by stjohnswarts on Monday April 25, 2022 @06:54PM (#31161871)
There's clearly an (probably shifting daily) algorithm. I'm not sure what the statement the site is trying to make. Even a random number generator that selects tweets at random for your feed is an algorithm.
Re:twitter/the-algorithm by crate_barre on Monday April 25, 2022 @05:36PM (#31160986)
Twitter is mostly an infrastructure operation imho, someone school me. It’s a scale-based problem space, how do you get all these tweets out in real time at minimum, and at best, how do you do some level of topic bucketing on top of it.
Re:twitter/the-algorithm by Karrot_Kream on Monday April 25, 2022 @05:56PM (#31161232)
Right but "the algorithm" is inextricably tied in with scaling. "the algorithm" is designed to handle updates at scale and the model probably has different parts updating to different events on different cadences.