News for hackers. Stuff that matters.
Re:Twitter open sourced the recommendation engine by jonathanmayer on Friday March 31, 2023 @03:07PM (#35391854)
Context: I teach at Princeton and study social media and recommendation systems.

From a quick skim of the repository, this is very limited transparency. The documentation gives a decent high-level overview of how Tweet recommendation works—no surprises—and the code tracks that roadmap. Those are meaningful positive steps. But the underlying policies and models are almost entirely missing (there are a couple valuable components in [1]). Without those, we can't evaluate the behavior and possible effects of "the algorithm."

[1] https://github.com/twitter/the-algorithm-ml

Re:Twitter open sourced the recommendation engine by ngrilly on Friday March 31, 2023 @03:37PM (#35392329)
What did you expect?
Re:Twitter open sourced the recommendation engine by TaylorAlexander on Friday March 31, 2023 @03:49PM (#35392527)
I don’t know if the parent’s expectations matter here. This is more about making sure others don’t misunderstand the meaning here.
Re:Twitter open sourced the recommendation engine by ngrilly on Saturday April 01, 2023 @02:51PM (#35402914)
Good point. I didn't see it like that. Thanks!
Re:Twitter open sourced the recommendation engine by meghan_rain on Friday March 31, 2023 @03:24PM (#35392131)
So why did they opensource it?
Re:Twitter open sourced the recommendation engine by daveguy on Friday March 31, 2023 @03:26PM (#35392151)
So they could pretend to be open. It's the "Open"AI model. Open-washing?
Re:Twitter open sourced the recommendation engine by cubefox on Friday March 31, 2023 @03:36PM (#35392318)
This is a very cynical take. They should be commended for publishing recommendation code at all, which no other major social network does.
Re:Twitter open sourced the recommendation engine by hanniabu on Friday March 31, 2023 @03:46PM (#35392468)
This is like FB open sourcing the compiled frontend code you can see yourself using inspect.

If we commend them for this we're helping promote and encourage this faux open source virtue signaling

Re:Twitter open sourced the recommendation engine by cubefox on Friday March 31, 2023 @04:22PM (#35393070)
No, that's very different.
Re:Twitter open sourced the recommendation engine by TaylorAlexander on Friday March 31, 2023 @03:51PM (#35392561)
Well if they say “we will open source the algorithm” and then what they really open source is a little bit of slightly relevant code that doesn’t allow us to understand the algorithm, then what we can deduce is that they are trying to weasel out of public commitments.

I can’t say for sure if that happened, but if they made a clear promise and then did something else, it’s perfectly reasonable to call that out.

Re:Twitter open sourced the recommendation engine by OJFord on Friday March 31, 2023 @09:03PM (#35395912)
Devil's advocate though: imagine you were to open source (probably with quite a short deadline) some 'algorithm' used in whatever you work on, but the rest should stay private; how would you go about that?

I don't think it's easy, there's inherently some interface(s!) where it's a hand-wavey 'get the thing from the private bit', and defining that sensibly is hard, and if you try to do it well will probably lead to a lot of meetings, scope creep, etc. - and as far as that goes it's not easy anyway, since it's highly technical and implementation-specific yet also a management/policy decision to make.

Re:Twitter open sourced the recommendation engine by jjeaff on Saturday April 01, 2023 @02:18AM (#35397647)
If you were doing it in good faith, you wouldn't need to publish the actual code. Most likely you should publish an article and a flowchart explaining how the algorithm works. Publishing a partial chunk of code just creates a story that supporters who don't understand can parrot that "they opened their algorithm".
Re:Twitter open sourced the recommendation engine by oneeyedpigeon on Saturday April 01, 2023 @06:19AM (#35399011)
Exactly. Publishing what they have is the worst of both worlds - hopefully people will create flowcharts based off it, though, although it sounds like there will still be a low level of accuracy.
Re:Twitter open sourced the recommendation engine by anyonecancode on Saturday April 01, 2023 @09:32AM (#35400182)
It depends on what your goal in open sourcing is. Are you looking to provide a base for others to build software on, and to provide a way for others to contribute back to your code? Then publishing the code makes sense.

Are you looking to build public trust in you and your organization? Then dumping a bunch of code with no context isn't going to help much, as it's not code but behavior that builds or destroys trust.

Are you looking to lean into a polarized partisan environment, pushing a narrative where its you and your supporters against an unfair group of "others"? Then a big splashy move high on symbolism and low on substance that will inspire lots of high profile, divisive media coverage is a great way to go.

Re:Twitter open sourced the recommendation engine by misiti3780 on Friday March 31, 2023 @04:20PM (#35393025)
a large portion of HN users view anything elon musk does as nefarious, which is why half of these comments are negative.
Re:Twitter open sourced the recommendation engine by guelo on Friday March 31, 2023 @04:32PM (#35393209)
Any time a billionaire buys a media company it's bad for the health of democracy.
Re:Twitter open sourced the recommendation engine by Mordisquitos on Friday March 31, 2023 @04:54PM (#35393501)
Not necessarily. What if the media company was bad for the health of democracy, and the billionaire's incompetence destroys the company's social standing and thus its ability to do more damage (even in the billionaire's own interests)?
Re:Twitter open sourced the recommendation engine by nonbirithm on Friday March 31, 2023 @05:16PM (#35393776)
Yeah, have to wonder how many people, if they had the money, would want to buy out Twitter just to wipe it out. Doesn't a huge chunk of HN hate Twitter and wish it were dead?

(Regardless I think that would be useless in the long run, since the millions of stranded users will still want another Twitter-like platform. And Twitter imploding without a designated archive will wipe out a tremendous amount of digital history.)

A lot of his decisions look pretty incompetent in the surface, like how could he not see how charging for verification devalue the system to whoever has the money?

Instead it could just be an intentional ploy to completely devalue Twitter disguised as incompetence. He can justify firing employees and charging for API access/verification as money-saving strategies, even if they're terrible strategies that have little chance of succeeding. And he could make enough people believe he's an idiot who makes things up as he goes rather than someone specifically driven or apathetic enough to run Twitter into the ground. Not to mention he was forced to buy them after changing his mind. Almost feels like a "so that's what happens" response.

I wonder how higher powers would be able to distinguish fake incompetence from real incompetence. Would they care how Twitter as a private company ends up if it's the case that it implodes from its own legitimately bad business decisions? It reminds me of how employers won't directly fire employees for discriminatory reasons, instead they make the employees' lives miserable so they're compelled to leave on their own, thus they escape scrutiny.

Re:Twitter open sourced the recommendation engine by ClumsyPilot on Friday March 31, 2023 @05:22PM (#35393853)
This is basically at the level of "9/11 was an inside job to bring down WTC 1, but WTC2 was destroyed in an unrelated but simultaneous terrorist attack"
Re:Twitter open sourced the recommendation engine by tablespoon on Friday March 31, 2023 @06:02PM (#35394264)
> Yeah, have to wonder how many people, if they had the money, would want to buy out Twitter just to wipe it out. Doesn't a huge chunk of HN hate Twitter and wish it were dead?

> (Regardless I think that would be useless in the long run, since the millions of stranded users will still want another Twitter-like platform.

If there's not an obvious successor, right when its shutdown, a lot of those people might get their habit broken and find something better to do. I know Mastodon was held up as a successor, but it's unclear to me if that's actually capable of scaling to that level.

Re:Twitter open sourced the recommendation engine by dmix on Friday March 31, 2023 @06:21PM (#35394521)
Mastodon is way too flawed to be anything but a niche tool for tech people and activists. I highly highly doubt such a system can cross the chasm. That doesn’t mean that’s a bad thing though.
Re:Twitter open sourced the recommendation engine by bobsil1 on Friday March 31, 2023 @09:34PM (#35396118)
Or, he’s as incompetent as he looks.
Re:Twitter open sourced the recommendation engine by misiti3780 on Friday March 31, 2023 @05:06PM (#35393654)
i wasnt referring to him buying twitter, i was referring to him saying he was going to open source the recommendation engine and then doing it.

i agree billionaires owning media companies is huge problem

Re:Twitter open sourced the recommendation engine by paulddraper on Friday March 31, 2023 @05:18PM (#35393808)
Which is why HN was so incensed about Bezos buying the Washington Post.
Re:Twitter open sourced the recommendation engine by mulmen on Friday March 31, 2023 @05:54PM (#35394181)
Do you believe billionaires can do good? Is their existence an existential threat to democracy?
Re:Twitter open sourced the recommendation engine by systemvoltage on Friday March 31, 2023 @06:20PM (#35394501)
We need more inequality. It’s a feature, not a bug. Entire swaths of people are convinced we need to flatten all people into one income bracket. That’s not going to end well. See USSR and Cambodian massacre.

That doesn’t mean we should propel poverty.

Midwits have been brainwashed into thinking “equity = good” with no critical thinking capability. The entire political college class screams “How can you be against equality!” followed by insinuations and fascism insults. The term “human rights” have been hijacked into socialism/communism propaganda.

It’s so deeply fucked.

Re:Twitter open sourced the recommendation engine by hooverd on Friday March 31, 2023 @06:07PM (#35394336)
Yes. There are plenty of philanthropic billionaires. Yes. That much money buys a destabilizing amount of influence.
Re:Twitter open sourced the recommendation engine by systemvoltage on Friday March 31, 2023 @06:43PM (#35394766)
Billionaires are billionaires not by literally storing cash. The rest of the society values their contributions and creations in the companies/corporations they run. Sure, they have some liquidity but the entire concept of resentment towards billionaires is essentially equal to resentment for the betterment of the world. There are some exceptions but for the most part, in a well oiled market, you can't just become a billionaire by fucking over people. See Adani and how it turns out for him: https://www.ft.com/content/5c0b6174-e66d-4fa5-89a5-6da1d69ab...
Re:Twitter open sourced the recommendation engine by pastacacioepepe on Friday March 31, 2023 @05:55PM (#35394187)
Can you name one relevant media company owned by someone from the working class?
Re:Twitter open sourced the recommendation engine by scythe on Friday March 31, 2023 @07:58PM (#35395431)
If you personally own a media company, you are by definition bourgeoisie. But see:

https://en.wikipedia.org/wiki/Media_cooperative#List_of_medi...

Re:Twitter open sourced the recommendation engine by simondotau on Friday March 31, 2023 @06:28PM (#35394592)
And when a highly scrutinised, highly visible billionaire buys it off a different bunch of billionaires which you know little about?
Re:Twitter open sourced the recommendation engine by v0idzer0 on Friday March 31, 2023 @08:02PM (#35395458)
Every major media company is owned by a billionaire
Re:Twitter open sourced the recommendation engine by systemvoltage on Friday March 31, 2023 @04:48PM (#35393432)
Cynicism is the standard operating procedure on HN for anything that tilts against their party lines. It's completely illiberal.
Re:Twitter open sourced the recommendation engine by cubefox on Friday March 31, 2023 @06:07PM (#35394340)
Yeah it seems so. I think journalists have portrayed him way too negatively. For example, they mostly ignored the Twitter files, presumably because it didn't fit their political narrative.
Re:Twitter open sourced the recommendation engine by swores on Friday March 31, 2023 @07:08PM (#35395002)
It's because there was close to zero newsworthy information in them, just nonsense being disseminated by wannabe-journalists.
Re:Twitter open sourced the recommendation engine by v0idzer0 on Friday March 31, 2023 @08:00PM (#35395450)
You clearly are oblivious as to what they contain
Re:Twitter open sourced the recommendation engine by r3trohack3r on Friday March 31, 2023 @09:27PM (#35396068)
I encourage you to watch the C-SPAN recordings of the senate sessions where they brought in Twitter employees and journalists to cover what was in the Twitter files.

From your comment it sounds like you’ve been consuming the 30s soundbytes from those hearings and the misinformation spreading around the internet.

A long list of 3 letter agencies were compiling lists of citizens and journalists and sending them to social media companies to review for ToS violations.

There is a very real threat to civil rights here. When this cannon swings around and points back at LGBTQ, racial equality, stopping the war on drugs, etc. this is going to be “not pretty.”

And the hearings covering them were unbelievably shameful. Senators talk passed the guests in the room. Refused to abandon their “sick burn” scripts regardless of where the conversation went. Insulted their guests. Went in random directions of questioning that had little to do with the root problem…

At the core of this, 3 letter agencies (seemingly across the board) have decided that it’s acceptable to ask social media companies to prevent citizens from communicating on their platforms by selectively directing the attention of their moderation teams towards individuals. Whether this is legal, or a violation of 1st amendment rights, is for sure an open question.

Only one senator directly addressed that and only briefly by saying “maybe they’re trying their best” - a statement that doesn’t exempt anyone involved from following the law.

If I were to sum up the hearings, I would like this:

Guest) Hey I think the government is breaking the law? R) Wow, I can’t believe it, look at covid! D) So Elon Musk gave these to you right? Are you even a journalist? Guest) Well they might be breaking the law R) They targeted conservatives! D) No they didn’t, this is biased Guest) But… they might be breaking the law R) COVID! D) Wait did we get these docs? They’re not in my sick burn list?

At the end of the day, they never addressed whether the government is allowed to censor citizens by weaponizing their ToS for selective enforcement and, if the government can do that, where is the line drawn? How specific are they required to be? Can a platform ban all political speech and then only selectively enforce requests from the government without doing their own moderation? How far can we launder the 1st amendment through a public-private collaboration of enforcing ToS?

Honestly I’m not sure what the hearings were really meant for, the government is unlikely to hold itself accountable. At this point I do believe the ball is in the citizen’s court to bring suit against the agencies named in the Twitter files like we did with the presidential surveillance program.

Re:Twitter open sourced the recommendation engine by jjeaff on Saturday April 01, 2023 @03:54AM (#35398200)
The government requesting that the tos of a private company be upheld seems rather mild to me. Did we get the reasons for the requests in the released files? Were they trying to reduce foreign propaganda or public health misinformation or something else important?
Re:Twitter open sourced the recommendation engine by variant on Friday March 31, 2023 @10:33PM (#35396460)
You like your government trying to tell a private company what's true and untrue?
Re:Twitter open sourced the recommendation engine by daveguy on Friday March 31, 2023 @10:48PM (#35396556)
More than I like a private company telling me what's true and untrue.
Re:Twitter open sourced the recommendation engine by misiti3780 on Friday March 31, 2023 @10:58PM (#35396624)
matt taibbi is a wanna be journalist? lol

go back to your cave.

Re:Twitter open sourced the recommendation engine by dang on Sunday April 02, 2023 @07:57PM (#35416757)
Please don't break the HN guidelines like this. It's not what this site is for, and destroys what it is for.

What's worse, if you have a true point, then posting like this actually discredits the truth and gives people a reason to reject it. That isn't in your interest and in fact hurts everyone.

https://news.ycombinator.com/newsguidelines.html

https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...

Re:Twitter open sourced the recommendation engine by mort96 on Saturday April 01, 2023 @02:23AM (#35397691)
"Way too negatively"? We're talking about one of the world's most influential people who uses their power to randomly accuse innocent normal people of being pedophiles. There is no portrayal too negative.
Re:Twitter open sourced the recommendation engine by oneeyedpigeon on Saturday April 01, 2023 @06:22AM (#35399023)
Because the way he acts gives people every right to. I agree that he may be misrepresented, but if he is, then he has to shoulder at least some of the blame.
Re:Twitter open sourced the recommendation engine by mejutoco on Saturday April 01, 2023 @09:25AM (#35400139)
The question is: are they right?
Re:Twitter open sourced the recommendation engine by 5e92cb50239222b on Friday March 31, 2023 @04:48PM (#35393427)
I still hear reverse-FUD about nvidia supposedly fully open-sourcing their Linux driver, when in reality they opened a tiny kernel portion of it that allows the main proprietary blob to connect to necessary kernel interfaces. You have to call out this bullshit when you see it.
Re:Twitter open sourced the recommendation engine by mananaysiempre on Friday March 31, 2023 @07:15PM (#35395062)
Wait, what? AFAIU what you say is true, except for the part where the “main proprietary blob” does not run on the CPU. This isn’t as glorious as an actual open-source driver would be, but it does have meaningful advantages—e.g. you now have a ghost of a chance of implementing Nvidia GPU support on a non-Linux kernel, by uploading the GPU-side blob and rewriting the CPU-side shim as required. Or is the blob license-restricted from being used line that?
Re:Twitter open sourced the recommendation engine by mort96 on Saturday April 01, 2023 @02:20AM (#35397666)
The "main proprietary blob" they're talking about is the userspace portion of the driver; the portion which does all of the heavy lifting. That definitely runs on your CPU. The only part they open-sourced is the kernel portion of the driver, which just exists to facilitate communication between the userspace driver and the hardware.
Re:Twitter open sourced the recommendation engine by philote on Friday March 31, 2023 @05:05PM (#35393642)
Hey, we can get even more cynical. Why should we trust that this code is even similar to what they run in production currently?
Re:Twitter open sourced the recommendation engine by concordDance on Saturday April 01, 2023 @06:49AM (#35399144)
I can't imagine deliberately special casing Elon's account in something they made from scratch to fool people.
Re:Twitter open sourced the recommendation engine by rakoo on Friday March 31, 2023 @05:40PM (#35394040)
Let's have reasonable goals, shall we ? "Their shit doesn't stink as bad as others'" is nothing commendable, especially after souch publicity.
Re:Twitter open sourced the recommendation engine by raiyni on Friday March 31, 2023 @07:37PM (#35395248)
They should be commended for open sourcing something they don't understand because they fired all of the people whom understood it? Elon admitted as much.
Re:Twitter open sourced the recommendation engine by LastTrain on Saturday April 01, 2023 @01:36AM (#35397461)
I’d give them more credit if they’d been honest and kept it secret then lie to my face and pretend they didn’t?
Re:Twitter open sourced the recommendation engine by jrochkind1 on Saturday April 01, 2023 @08:55AM (#35399939)
I say "why not both". Even if they are doing it only for good PR, we encourage it by giving them praise, because we should encourage things we want. (While remembering that they are not our friend, they are an entity we should pressure, and the way we pressure is by giving praise when they do things we like, and critcisim when they do not).
Re:Twitter open sourced the recommendation engine by jstummbillig on Friday March 31, 2023 @04:51PM (#35393471)
If we are willing to not assume some borderline "it's what they want you to think" conspiracy play, obviously there was always going to be a lot of highly interested and qualified people taking a very close look at this and, at some point, there was always going to be very definitive conclusion of what's the deal with what they released.

If your play was "it's some source code, hence people will think we are open, and that should be really good for us", that would make you a very special kind of idiot in this space.

Re:Twitter open sourced the recommendation engine by correlator on Friday March 31, 2023 @04:54PM (#35393499)
There is clearly a lot of information to share. It's worth considering this could be step 1 of n as opposed to assuming the worst possible intention.
Re:Twitter open sourced the recommendation engine by yurodivuie on Friday March 31, 2023 @07:00PM (#35394930)
It's healthy to have a normal amount of cynicism. They released it for a reason. "The goal of our open source endeavor is to provide full transparency to you, our users, about how our systems work."

Why be transparent (or try to appear transparent)? To convince people to trust your platform (or to recruit - which seems to be another goal of the post). Why would Twitter want or need to do this now? Well, there is a bit of context. This disclosure doesn't exist in a vacuum.

Re:Twitter open sourced the recommendation engine by mirkules on Friday March 31, 2023 @07:55PM (#35395409)
I love this take. Doomed if you do, doomed if you don't.
Re:Twitter open sourced the recommendation engine by drstewart on Friday March 31, 2023 @08:58PM (#35395873)
>It's healthy to have a normal amount of cynicism.

I agree, which is why I wonder what your motivation is to take down Twitter. You're posting about this for a reason. If I were a rival social media company, I'd probably have paid agitators to sow doubt. Especially if I didn't want to open source my own algorithm.

Re:Twitter open sourced the recommendation engine by blitzar on Saturday April 01, 2023 @04:47AM (#35398527)
I agree, which is why I wonder what your motivation is to defend Twitter. You're posting about this for a reason. If I were a social media company, I'd probably have paid agitators to defend them.
Re:Twitter open sourced the recommendation engine by justapassenger on Friday March 31, 2023 @03:32PM (#35392259)
You must be new to Musk's business practices.
Re:Twitter open sourced the recommendation engine by kzrdude on Friday March 31, 2023 @03:28PM (#35392196)
If we can't build anything with this, is it "source"?
Re:Twitter open sourced the recommendation engine by bilekas on Friday March 31, 2023 @03:46PM (#35392466)
"Does not include batteries"
Re:Twitter open sourced the recommendation engine by bradly on Friday March 31, 2023 @03:33PM (#35392277)
Because they fired most the engineers working on it?
Re:Twitter open sourced the recommendation engine by anigbrowl on Friday March 31, 2023 @03:27PM (#35392164)
PR
Re:Twitter open sourced the recommendation engine by w0m on Friday March 31, 2023 @04:04PM (#35392765)
PR and it was already leaked last week.
Re:Twitter open sourced the recommendation engine by joshspankit on Friday March 31, 2023 @04:30PM (#35393186)
That was one of Elon’s core statements when he first talked about buying Twitter. If he had gotten it out sooner there would be an easier link between the two, but if you want more context just go read the old tweets and articles from the Twitter vs Elon days.
Re:Twitter open sourced the recommendation engine by avanti on Friday March 31, 2023 @06:19PM (#35394483)
It's no secret that Twitter, like any other social media platform, is driven by user engagement and ad revenues. The more time we spend on the platform, the more valuable it becomes for them. With this new open-source algorithm, they're essentially crowdsourcing improvements to their system to better serve us the content we crave.

this move could be seen as a strategic PR play to boost their public image amidst the growing concerns around algorithmic bias and lack of transparency. By inviting the community to collaborate and address these issues, they're not only shifting some of the responsibility onto the users but also deflecting potential criticism.

Re:Twitter open sourced the recommendation engine by llx2 on Saturday April 01, 2023 @02:13AM (#35397619)
bc he have no devs anymore and thinks the community will fix it for free
Re:Twitter open sourced the recommendation engine by carstenhag on Saturday April 01, 2023 @03:15AM (#35397986)
Noone has mentioned this before - I don't know if it's really related, but afaik the European Union is thinking about requiring social media platforms to be more transparent when it comes to recommendations etc. If you can already say "hey we have a lot already online!" then maybe the laws will become less strict.
Re:Twitter open sourced the recommendation engine by eecc on Friday March 31, 2023 @03:15PM (#35391989)
Wouldn’t that make them easy prey of “spam SEO”. However, given the framework isn’t it still possible to guess the models?
Re:Twitter open sourced the recommendation engine by makeitdouble on Friday March 31, 2023 @07:23PM (#35395134)
The spam SEO issue should be dealt/thought about _before_ engaging in the whole adventure, and having to guess how it could work if decently implemented properly defeats the "open source" spirit of it.

More credits would be given if the very idea of open sourcing the algorithm hasn't already been discussed to death with predictions of the difficult points and how it probably won't happen in any sane way.

Re:Twitter open sourced the recommendation engine by jimkleiber on Sunday April 02, 2023 @06:59AM (#35409253)
Makes me wonder if a way to override people SEO hacking the algorithm is to create a market of open-source algorithms that each individual can choose and then it's not trying to hack THE algorithm but having to hack many and not knowing which algorithm an individual is using.
Re:Twitter open sourced the recommendation engine by muddi900 on Sunday April 02, 2023 @02:03PM (#35413337)
You don't have to target each 'algorithm' all at once. You can target them one at a time. Hell you can run A/B test single out the easiest targets.
Re:Twitter open sourced the recommendation engine by jimkleiber on Sunday April 02, 2023 @07:15PM (#35416377)
Yes but right now there is 100% of the users using the one algorithm (or chronological). If one doesn't know what percentage or which people are using which algorithm, it becomes harder to know which ones to try to hack to have the biggest result.
Re:Twitter open sourced the recommendation engine by eecc on Sunday April 02, 2023 @04:19PM (#35414731)
And them be pilloried for not doing it or not fast enough. Damn if you do, damn if you don’t.

I’m starting to think the broblem with Elon is mostly personal, he’s just a proxy and default wrong.

(not that I approve of his behaviors, but I can’t enjoy this whole mobbing that he’s getting; not that he cares this I’m not worried he’s getting traumatized in any way? it’s just how it’s become an identitarian trait for a certain group that irks me.)

Re:Twitter open sourced the recommendation engine by tpmx on Friday March 31, 2023 @03:11PM (#35391923)
Did you also skim the other repo, https://github.com/twitter/the-algorithm ?

From a quick clone and line-count, it has:

  235 kLOC .scala
  136 kLOC .java
  22 kLOC .py
  7 kLOC .rs
(I haven't skimmed this code.)
Re:Twitter open sourced the recommendation engine by Lord_Zero on Friday April 07, 2023 @09:32AM (#35481397)
I think you misunderstood. He's saying the training models are not there.
Re:Twitter open sourced the recommendation engine by alfor on Friday March 31, 2023 @03:25PM (#35392143)
I think this is an unseen level of transparency in internet companies and media.

The Twitter files should be enough to create a desire for more investigation into collusion and manipulation of the population.

Find what is missing, explain why it’s important and report it back.

There is no other place like Twitter. Even here I get my comments deleted if I am a bit to direct against the climate religion.

Re:Twitter open sourced the recommendation engine by bilekas on Friday March 31, 2023 @03:45PM (#35392451)
> But the underlying policies and models are almost entirely missing (there are a couple valuable components in [1]). Without those, we can't evaluate the behavior and possible effects of "the algorithm."

Haven't gone through yet, but yeah, if that's the case, all this is, is a glorified framework to plug your own in.. Not exactly what was promised.

Re:Twitter open sourced the recommendation engine by modeless on Friday March 31, 2023 @03:48PM (#35392513)
Re:Twitter open sourced the recommendation engine by simonw on Friday March 31, 2023 @04:45PM (#35393384)
Those look older to me. They all have last updated dates for October and November 2022.
Re:Twitter open sourced the recommendation engine by robopsychology on Friday March 31, 2023 @04:04PM (#35392761)
Congrats on reading several hundred thousand lines of code in 1 hour and 8 minutes :)
Re:Twitter open sourced the recommendation engine by elorm on Friday March 31, 2023 @04:39PM (#35393309)
* * *
Re:Twitter open sourced the recommendation engine by robopsychology on Friday March 31, 2023 @04:55PM (#35393528)
Context: I studied at Oxford

Fair point, I missed that when I skimmed OPs comment

Re:Twitter open sourced the recommendation engine by tpmx on Friday March 31, 2023 @04:29PM (#35393168)
We should really all just bow in awe as we are clearly inferior.
Re:Twitter open sourced the recommendation engine by robopsychology on Friday March 31, 2023 @04:33PM (#35393235)
Princeton has a Code Reading 101 that all postdocs/professors must take, however in exchange for the Secrets of Speed Reading you must acknowledge every message with where you learnt those skills.
Re:Twitter open sourced the recommendation engine by pnt12 on Friday March 31, 2023 @04:46PM (#35393396)
It's fast to read stuff when you have the domain knowledge. The weights won't be a 5kb Scala file: they'd probably be a big binary file, which is easy to search it github/locally after cloning.

Otherwise, if they are provided, someone in the thread will surely point to them.

Re:Twitter open sourced the recommendation engine by raggi on Friday March 31, 2023 @04:55PM (#35393518)
class project, 200 students, 1500 LoC each. Time for grading.

there are contexts in which this may be well practiced.

Re:Twitter open sourced the recommendation engine by culi on Friday March 31, 2023 @05:06PM (#35393658)
imagine thinking you need to read every file in a project to understand the architecture and which pieces are important for specific functionality you're looking to understand. Have you ever picked up a bugfix ticket for some code you didn't write?
Re:Twitter open sourced the recommendation engine by kadavy on Friday March 31, 2023 @04:56PM (#35393533)
For example, MostRecentCombinedUserSnapshotSource seems to be influential (such as for calculating "tweepcred"), but we can't see how it's calculated.
Re:Twitter open sourced the recommendation engine by EastSmith on Friday March 31, 2023 @05:00PM (#35393592)
FB open source algo looks much better, right? /s
Re:Twitter open sourced the recommendation engine by bobobob420 on Friday March 31, 2023 @05:24PM (#35393871)
Can i audit your classs for free?
Re:Twitter open sourced the recommendation engine by fanagra32 on Friday March 31, 2023 @05:36PM (#35394000)
Exactly. Open-washing at its finest. Just like OpenAI. "Open" just doesn't mean anything anymore at this point.

> Context: I teach at Princeton and study social media and recommendation systems.

I'm not sure how this context was relevant. Your post was perfectly able to stand on its own feet and convince through content, not through authority. If anything, this "context" removed pondus from it by appealing to authority.

Re:Twitter open sourced the recommendation engine by acdha on Friday March 31, 2023 @09:33PM (#35396113)
The context is relevant for indicating that they’ve familiar with the problem and have thought about these issues in depth. It’s also useful for not being accused of hiding their identity if someone thinks they have an unmentioned agenda. Argument from authority is bad when it’s of the form “I am an expert, therefore you shouldn’t question this claim”, not when it’s used to provide an identity to a previously-unknown name while also providing a cogent argument and supporting evidence.
Re:Twitter open sourced the recommendation engine by zhte415 on Friday March 31, 2023 @07:59PM (#35395439)
Is it valid to focus tracking a Dem/Rep split when that split is an exclusionary design for many Americans? Or is it not exclusionary in your belief? I'm curious of a social science perspective.

Ignoring the global nature of Twitter for a moment.

Re:Twitter open sourced the recommendation engine by novok on Friday March 31, 2023 @09:06PM (#35395926)
It's an open algorithm, but it's not open data! (joking)
Re:Twitter open sourced the recommendation engine by eterevsky on Saturday April 01, 2023 @03:19AM (#35398003)
I work on Google Assistant Suggestions and I don't think it's very practical to open-source an algorithm like that including the models and the underlying data. Both of them can live in separate services and be frequently updated.

I am assuming that open sourcing the code aims to increase transparency about the business logic of the ranking decisions. At the same time you don't want spammers to be able to easily run experiments against a cloned version of your system.

Re:Twitter open sourced the recommendation engine by helsinkiandrew on Saturday April 01, 2023 @04:35AM (#35398461)
> But the underlying policies and models are almost entirely missing... Without those, we can't evaluate the behavior and possible effects of "the algorithm

And neither can spammers find and test the cracks and edge cases that would allow them to break the system, that does sound reasonable to me. If they were public there would be an arms race between spammers/those wishing to game the system and Twitter engineers.

Re:Twitter open sourced the recommendation engine by ivalm on Saturday April 01, 2023 @05:13AM (#35398639)
Then don’t pretend to release “the algorithm.”
Re:Twitter open sourced the recommendation engine by helsinkiandrew on Saturday April 01, 2023 @06:28AM (#35399055)
They’re explaining how it works without giving the specifics. Much like the US military explains how the nuclear deterrent works without disclosing detailed plans and control codes.