This is a good point about what might happen, but it seems worthwhile to address and fix directly. Personally I don't see why adversarial scraping of a publicly published website should require any more ethical consideration/review than using the suggested API would. Ethical concerns should revolve around humans, not the business desires of non-human entities.

Of course this used to have the side effect of breaking significant swathes of basic browser UX, especially in areas of accessibility. I assume they’re better now than they once were, but given musk’s historical behavior I assume he won’t consider breaking something like accessibility to be bad or problematic.

I don't see why would he spend more to make website less accessible. Not fixing bugs - yes, just about everyone does so. But breaking it intentionally costs money.

Tesla's aren't THE most accessible (by cost) EV out there, but SpaceX def is.

[1] https://www.wired.com/story/twitter-layoffs-accessibility/

[2] https://www.businessinsider.com/twitter-sent-musk-risks-paid...

SpaceX is... an accessible EV? What?

No, I don't. It obviously is trying to say something about accessibility and SpaceX, but I don't see how the sense of accessbility being discussed in the thread even applies to SpaceX, much less what claim is being made.

Just because it’s the same word does not mean it’s the same meaning.

This is a comment thread discussing accessibility at Twitter, in a post about Twitter discontinuing part of its platform. It’s not a place to circlejerk Musk’s achievements, regardless how much you would want that to be so.

Explain, in exact words, what about SpaceX in any way shape or form would indicate that they know how to handle user a11y and how that would transfer over to Twitter— you know, the thing we’re actually discussing here.

Usage for reading is something like (but not identical to)

   1.sh screen_name > 1.json
   yy059 < 1.json|grep full_text > 1.txt
   less 1.txt

   unset connection
   export Connection=keep-alive
   yy059 < 1.json|grep full_text \
   |yy030 \
   |grep "https://t.co/.{10}$" \
   |uniq \
   |yy025 \
   |nc -vv h1b 80 \
   |sed -n '/location: /s///p' \
   |ahref > 1.htm
   links -no-connect 1.htm

Alternatively if I do not trust the URLs I might use a script called "www" instead of ahref. It takes URLs on stdin and fetches archive.org URLs wrapped in simple HTML to stdout, using the IA's cdx API.

  #!/bin/sh
  SCREEN_NAME=$1
  COUNT=500
  PUBLIC_TOKEN="Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA"
  GT=$(exec curl -A "" -s https://twitter.com/$SCREEN_NAME|sed -n '/gt=/{s/.*gt=//;s/;.*//p;}');
  echo "x-guest-token: $GT" >&2;
  REST_ID=$(exec curl -A "" -H "authorization: $PUBLIC_TOKEN" -H "content-type: application/json" -H "x-guest-token: $GT" -s "https://twitter.com/i/api/graphql/mCbpQvZAw6zu_4PvuAUVVQ/UserByScreenName?variables=%7B%22screen_name%22%3A%22$SCREEN_NAME%22%2C%22withSafetyModeUserFields%22%3Atrue%2C%22withSuperFollowsUserFields%22%3Atrue%7D"|sed 's/\(rest_id\":\"[0-9]*\)\(.*\)/\1/;s/.*\"//'); 
  echo "rest_id: $REST_ID" >&2;
  curl -A "" -H "authorization: $PUBLIC_TOKEN" -H "content-type: application/json" -H "x-guest-token: $GT" -s "https://twitter.com/i/api/graphql/3ywp9kIIW-VQOssauKmLiQ/UserTweets?variables=%7B%22userId%22%3A%22${REST_ID}%22%2C%22count%22%3A$COUNT%2C%22includePromotedContent%22%3Atrue%2C%22withQuickPromoteEligibilityTweetFields%22%3Atrue%2C%22withSuperFollowsUserFields%22%3Atrue%2C%22withDownvotePerspective%22%3Afalse%2C%22withReactionsMetadata%22%3Afalse%2C%22withReactionsPerspective%22%3Afalse%2C%22withSuperFollowsTweetFields%22%3Atrue%2C%22withVoice%22%3Atrue%2C%22withV2Timeline%22%3Atrue%7D&features=%7B%22dont_mention_me_view_api_enabled%22%3Atrue%2C%22interactive_text_enabled%22%3Atrue%2C%22responsive_web_uc_gql_enabled%22%3Afalse%2C%22vibe_tweet_context_enabled%22%3Afalse%2C%22responsive_web_edit_tweet_api_enabled%22%3Afalse%2C%22standardized_nudges_misinfo%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%2C%22include_rts%22%3Atrue%7D"

This solution isn't pretty but I can easily keep tabs on Twitter feeds without any need for a Twitter account, a Twitter "API key" or a so-called "modern" browser.

AAAAAAAAAAAAAAAAAAAAACHguwAAAAAAaSlT0G31NDEyg%2BSnBN5JuyKjMCU%3Dlhg0gv0nE7KKyiJNEAojQbn8Y3wJm1xidDK7VnKGBP4ByJwHPb

Maybe something will change on Feb 9. Maybe not.

Every person visiting twitter.com is using this same token. The token is neither personal nor private.

What would be the point of changing or blocking it.

Be interesting to see if that stays the same when they're charging for the API but leaving a huge loophole with this token.

Using GraphQL like this can be a dark pattern because to anyone using a "modern" browser that "tech" commpanies control it makes it seem like the website will not work without Javascript enabled. That's false, but nonetheless it gets people to enable Javascript because the website explicitly asks them to enable it. Then the website, i.e., "tech" company, can perform telemetry, data collection, surveillance, and other shenanigans.

Sometimes this practice might not be a deliberate dark pattern, it might just be developers who are using Javascript gratuitously. For example, HN search provided by Algolia uses GraphQL. HN puts URLs with pre-selected query terms and a public token ("API key") on the HN website. Everyone that uses those URLs uses the same key.

Unlike Twitter, HN istelf does not ask anyone to enable Javascript. The website works fine without it, including the Algolia search, as shown below.

Usage is

   1.sh query > 1.json

   #!/bin/sh

   curl -A "" -d '{"query":"$@","analyticsTags":["web"],"page":0,"hitsPerPage":30,"minWordSizefor1Typo":4,"minWordSizefor2Typos":8,"advancedSyntax":true,"ignorePlurals":false,"clickAnalytics":true,"minProximity":7,"numericFilters":[],"tagFilters":["story",[]],"typoTolerance":"min","queryType":"prefixNone","restrictSearchableAttributes":["title","comment_text","url","story_text","author"],"getRankingInfo":true}' "https://uj5wyc0l7x-3.algolianet.com/1/indexes/Item_production_sort_date/query?x-algolia-agent=Algolia%20for%20JavaScript%20(4.0.2)%3B%20Browser%20(lite)&x-algolia-api-key=8ece23f8eb07cd25d40262a1764599b1&x-algolia-application-id=UJ5WYC0L7X"

    #!/bin/sh

    export Connection=close;
    export Content_Type=x-www-form-urlencoded;
    export httpMethod=POST;
    x=$(echo '{"query":"'$@'","analyticsTags":["web"],"page":0,"hitsPerPage":30,"minWordSizefor1Typo":4,"minWordSizefor2Typos":8,"advancedSyntax":true,"ignorePlurals":false,"clickAnalytics":true,"minProximity":7,"numericFilters":[],"tagFilters":["story",[]],"typoTolerance":"min","queryType":"prefixNone","restrictSearchableAttributes":["title","comment_text","url","story_text","author"],"getRankingInfo":true}');
    export Content_Length=${#x};
    echo "https://uj5wyc0l7x-3.algolianet.com/1/indexes/Item_production_sort_date/query?x-algolia-agent=Algolia%20for%20JavaScript%20(4.0.2)%3B%20Browser%20(lite)&x-algolia-api-key=8ece23f8eb07cd25d40262a1764599b1&x-algolia-application-id=UJ5WYC0L7X"|(yy025;echo "$x") \
    |nc -vv h1b 80

Two important take-aways from that:

1) The attackers just pulled the old "Hey, I'm from IT service desk and need your password."

2) Apparently every engineer had god mode in production

I can't imagine what scary monsters are in that codebase.

Scraping web pages is a last resort for when proper APIs are not available.

It was useful to check to see if someone else decided to softblock me/deactivated their account (and in at least one case, has helped me to reach out to someone in need). That's not "making money from the dataset", it was a hobby project and I imagine the overwhelming majority of Twitter bots are that. (Most of the ones I used to follow generally just scraped various subreddits to tweet out/source fun art to look at).

I'm not gonna pay money to access my own data, Musk can sink in his own shit for that and I don't blame any of those other hobby bot devs for not wanting to do that either.

No attempt at making this seem like reasonable behavior and communication with developers can succeed here.

And by the way if you expect reasonable cost, you're in for a surprise. Twitter is DESPERATE for cash.

From a pure business perspective it's either Twitter dies or Elon pays the bills.

It's not only the code base. It's employees, servers, processes, data...

Even if you get twitter for free, you still need to pay to run it.

One reason Mastodon and others have trouble is that people don't want to follow random people, they want to follow the same people they did on Twitter. It's their community.

Certain communities did make a successful switch, like Apple and Tech journalist communities, but not a lot of other communities.

This seems to be taken as given, but how specifically would you expect someone to simply monetize the social graph? Just because data exists doesn't mean that anyone wants to pay for it.

As for the value of the brand itself, we could say the same thing about Yahoo in 2010.

You can really pull a lot of useful data on someone by just looking at their follow choices, self-identification, and reply/retweeting behavior. And that's before you add all the information Twitter maintains on things like profile views or video plays, or the consistency of entity identification in tweet content etc. I can give you very specific kinds of demographics like 'people who enjoy tweets about dogs and football but not cats or politics.'

Augmenting these lists with Twitter's data might improve/expand the lists bur won't create much extra value imo, because the demand side won't change.

A more sane way to do this would have been with a public tweet explaining why they were doing it (ad revenue), a 3 month lead time, and an API for helping apps get users to "upgrade" their account.

Frankly, I don't think Musk much cares for the developers. From a business perspective, he might be right here - developers aren't the ones watching ads or paying for microtransactions, users are. The developers, at his scale, are a liability. Restricting their capabilities and profiting from their struggle is just part of the modern tech stack. I hate it as much as you do, but calling it "insane" is a hysterical double-standard that crops up just too often on HN.

Nobody is making enough profit via the Twitter API to make it worth staying on. Twitter is a distant #3 social network that itself has never really made money.

This is being too generous. Facebook, Instagram, and TikTok are all far larger than Twitter. Reddit is also just barely larger than Twitter, though at that scale you have to consider who has more bots. Twitter claims slightly more MAUs than Snapchat, but since Snapchat isn't a viral network it probably has no bots to speak of, making it larger than Twitter in terms of real users.

It's a hard comparison because the value of the Apple ecosystem is so obvious and mutual. Twitter is junk on both sides; unless as a third party dev you just happened to find value in it, in which case you should pay for it.

I understand this is hurting peoples feelings, but it makes complete sense to me from all angles.

I used to use Tweetbot which has now stopped working, as a result I don't use Twitter any more, the advertisers have lost one extra person to sell to.

Which is aligned with Musk's strategy thus far, but there's also a good argument to be made that that product should have never been needed if twitter was providing a decent first party client. They should have wanted to provide that to control the advertising. Letting the third parties take that up was a poor strategy -- business-wise that is, I get that people may have liked it and have used it for a long time and are probably sad to lose it.

They bring the content users want to consume.

Then you're not in the community in which the third party elements(e.g. tweet aggregation, SSO login, fine grained access control) are vital part of. Which may be substantial, or not, and we as the sheep don't know.

When did you start using Twitter? Twitter as we know it today is largely a product of the community and third party developers. Retweets, hashtags, quote tweets, the use of a blue bird symbol; all came from either the community at large or specific third party devs. The fun bots also drive a lot of twitter engagement; honestly they're the only real thing I miss on Mastodon (while some of the most important ones did come across, a lot didn't). Third party software also makes Twitter a support system, a status reporting system... The list goes on.

However, beyond all that, even if there was a good argument for killing the API free tier (there is not), no-one sensible would do it with _one week's notice_. This will alienate even commercial users who would be willing to pay. If you're using your CRM's twitter integration to track customer complaints, say, unless you're huge the free API is probably sufficient today. With one week's notice, even assuming you hear about it the first day (most will not) good look getting a purchase order for the API within a week if your company is even remotely bureaucratic (most are).

The patron saint of Dunning-Kruger presumably thinks it's a good idea, but can't imagine anyone else does.

You don't absolutely need to have your code signed, I can give apps to my friends, they just get a warning about it not being signed and potentially unsafe when it's first installed.

Musk is trying to monetize every last aspect of current Twitter possible while cutting costs to stave off bankruptcy that his acquisition made imminent long enough to pivot Twitter into a completely different business model centering on being a payment intermediary, apparently to relitigate his firing twice from X.com before it became PayPal.

He doesn't care about the ecosystem, because its an ecosystem for a completely different busoness than the one he wants Twitter to be. He cares that use of Twitter outside of the 1st party frontends bypasses monetization, so he wants it monetized or gone.

Which is odd, because Elon is someone that definitely will remember AOL, MSN messenger and MySpace being things...

This does not follow, in Twitter's case. If Twitter's API made them money, they wouldn't have shut it down, but it's the exact opposite, it costs them money while providing no monetary benefit. So in this case, from the business perspective, it is correct to start charging for it.

Now they could spend 3 months but with debt service payments of a billion dollars a year, I highly doubt that they have the patience to wait that long.

But it provide monetary benefits by attracting and keeping users, which encourages ad spend.

Or at least it used to, until the big advertisers were driven away.

If someone is trying to run a business and doesn't understand how indirect income works they might want to consider something less challenging.

What this actually does is remove API access from small startups, solo developers, and other innovators, and reserves it for corporations.

He might not be great at decisions where people and politics are involved, but this isn’t one of those decisions. But some people here on HN really want to believe he’s an idiot, which he’s not.

Musk is the chief engineer of spacex rockets. He's not just a CEO.

Do a little research before making comments like this.

Will be interesting to observe ... will publishers pay? Will users enjoy without those?

I'd love to have free API access to all kinds of companies.

Think how much easier and faster it would be booking your travel if all airlines and hotels let you straight in to the raw pricing and availability data from their reservation systems.

Think how much money you could save if retailers exposed all their product availability and pricing data, live.

They don't, because they all regard this stuff as "commercially sensitive".

Why should Twitter be any different?

Twitter data is rarely anything to do with pricing, so that's why it's different.

I'm not after internal pricing/B2B pricing/discounting, in my thought experiment I "just" want free API access to information that's already on public websites (eg end-user/retail pricing for rooms/flights/widgets). I'm not aware of any airlines/hotels/retailers that allow even that. In bulk, it's commercially sensitive too.

But yeah, it would be great if we didn't have to use their websites to navigate the data.

Right.

So back to where we started: what's actually wrong with Twitter deciding to insist on eyeballs on twitter.com instead of scripts pointing at their free API?

In some ways it reminds me of the Hasbro/Wizards of the Coast licensing shenanigans where the business is annoying people building an ecosystem around their products which arguably adds a lot of value.

Yup, although we all know that Twitter just the latest example in a loooong line of companies to have pulled this stunt since, well, what feels like forever.

> and generate bad will towards Twitter

I'm fairly ambivalent about this aspect, and I doubt Musk cares much, it's not as though the haters weren't already hating before this latest incident.

To an extent I think this is political theater, as opposed to ineptitude, petulance, or poor financial decision-making. Twitter is an important piece of virtual territory because it's still the platform for breaking news and pretty much every media outlet publishes and promotes stories there, as well as most politicians, govt agencies and so on. Like it or not it is the most 'realtime' mass market platform.

That gives Twitter tremendous strategic importance, even though its social graph is smaller and more ephemeral than Facebook's. Size is not everything; reach matters too. That strategic importance makes it interesting to state and non-state actors, because capture of that territory is loosely equivalent to control of radio broadcasting capacity or printing presses in earlier eras. If you think of Twitter as a magically effective print and distribution store for pamphleteers with a huge secondary ecosystem around it, the shutdown of free API access is essentially a market takeover. Musk was spitballing yesterday and suggesting that from now on API access would cost $100/month and require ID to use.

(It's interesting to me that Google could have owned that space with their News product, although it has always been a one-way thing rather than socially driven. Their failure to own that space is a direct result of their aversion to user configurability and selection.)

If I woke up feeling evil (really money seeking) I might add some clause to the terms and conditions that you need to prominently display "Powered by Twitter" if you use the API and aggressively check all websites using Twitter API, one way or another.

But I guess my perspective would be considered more long term while Musks management of Twitter seems to be heavily focused on short term.

the real problem from a business standpoint is less people trying stuff out and figuring hey maybe I could actually build a business on this, time to move up to the paid tier.

Also no devs at companies have played with twitter api because devs not going to spend money to play, so never say in meeting actually we can solve this with Twitter's api, we just have pay for a licence and I can write the solution! So maybe business look for other ways to solve problems.

This. I had a brief flirtation with working with people using the Twitter API for research purposes. A very large majority of them didn't care about having the latest tweets coming off the firehose, but instead getting access to all the old data. At least at the time, the latter was much more difficult to do.

Things like using Twitter for real time sentiment analysis does require realtime data, but those applications are more likely to be for-profit, and thus could afford to pay.

I used to lead the search and historical API team at Twitter.

Probably shouldn't have landed it with $1bn/year debt payments then, eh.

It's worth remembering the market setting was very different when he made the offer. It's also worth remembering the offer was refused at the time it was made. It was only after the tech market saw a downturn, turning a good offer into an incredible one, was the offer pursued. Around that time, it also looks like he attempted to renegotiate the offer, but that was also refused.

Poor timing of the offer (only knowable in retrospect!) and shareholder obligations more or less forced this situation into existence. Musk made a good offer with a terrible contract; time, shareholders, the board, the legal system, and the stock market turned that terrible contract into a huge payout for shareholders and a signed a death warrant for Twitter.

Musk made a mistake in the way he made the offer for Twitter--but let's not pretend that the legal system isn't equally responsible for forcing the current situation. Even if the previous leadership were open to negotiating a lower buyout (and thus lower debt burden), they would have just opened themselves up to endless shareholder lawsuits had they done so. Their hands were effectively tied once the rest of the tech stock market started going down.

Might have been smart to not make it unprofitable in the first place.

https://www.statista.com/statistics/299119/twitter-net-incom...

However pretty much the entirety of their revenue stream was ads, and Musk started taking a big dump on that early 2022.

Without Musk, Twitter would likely have been profitable in 2022. No extra debt, no fleeing advertisers, no massive cuts. Some layoffs like everyone else is doing would probably have been enough.

Please back this up with an actual projection that says anything close to this. Twitter was not on track to ever be profitable before Musk bought it.

https://twitter.com/elonmusk/status/1588676939463946241

At first I assumed this was only impacting read access (apps that mimic the twitter app), but now I’m not so sure.

@isjesusreturned @badassnames @pinballmapcom etc.

Bummer.

I'll especially miss the Earthquake bots :(

https://twitter.com/elonmusk/status/1621259936524300289

This announcement and today's ban-wave that followed seemed like someone not accustomed to Twitter just knew there are bots, and thought it to be workable idea to sample global timeline and script ban duplicates, which of course is not workable at all and only maximize harm on real users.

If I looked hard enough I could probably buy 10000 verified twitter accounts with API access right now, aged 1 year or older.

I would think mobile carriers keep that in their own house

How A Twitter Fight Over Bernie Sanders Revealed A Network Of Fake Accounts

https://www.huffpost.com/entry/democratic-bot-network-sally-...

https://shareblueastroturf.netlify.app/

Besides, even if 90% of the API usage is for scraping and does not impact the bottomline, it's still a smart business decision to leave the API open if the remaining 10% are for posting content that brings in views and thus ad impressions that generate more money than the API costs to run.

Twitter may be heading closer towards becoming a ghost town.

Whether it’s Elon or someone else, this shows a total lack of understanding of what made Twitter ubiquitous.

What's with these cell phone people calling everything APPS? Hi. They're WEBSITES. Just because you might use a stupid cell phone app to access the stupid website doesn't mean it's no longer a website.

Use the right words and stop recording video vertically. I hate all of you cell phone people.

Wouldn't the right words be "smartphone people?"

Notes

Also named cellular phone, cell phone, cellphone, handphone, hand phone or pocket phone, sometimes shortened to simply mobile, cell, or just phone.

Man this generation blows. No wonder you're fine with 10 different game publisher clients, 10 different movie publisher clients, and 500 different "apps" to access websites. Keep recording vertically dude. (GEEZUS)

Lots of stuff in this thread from myself and multiple other people.

Even less for some. It seems further apps are being shut down in the run up because they have "violated Twitter Rules and policies”.

Such as movetodon [0] - a service which helps you connect to the fediverse accounts of twitter users you follow.

[0] https://mastodon.social/@Tibor/109800904950500383

End of an era; Twitter is dead. Whatever is left, it's not Twitter.

Am-I the only one surprised the site still hasn't had a major (multi-hour) outage at this point?

Considering the waves of resignations that hit the automated content filtering and bot mitigation teams, it's no wonder they are getting desperate.

  > getting fired means getting deported

Twitter has had at least one multi-hour global outage since Musk took over. This is in addition to at least one outage in Australia, plus outages of other critical services. The API outage happened before the after-the-fact announcement, login and 2FA outages kept people out of the app, etc.

For what it's worth, I cannot access Twitter right now, and DownDetector suggests I'm not alone, so we might be having another outage starting just now.

It didn’t even require doing anything to/on the page with Puppeteer. Merely using it to open a browser window and then waiting, and using that browser window by hand during the wait to go visit the site that was using Cloudflare anti-bot protection ran into the problem.

It's going to take a few scandals for this to sink in, but it's such a recipe for disaster.

On Twitter, you at least know that it's only three letter agencies reading your DMs. On Mastodon, it could be some Reddit moderator type that makes it their life mission to destroy you.

And because this must be true for every admin on every server on the internet... thank you dang (when you read this)!

Whaaaaaa but Elon is the god king who will deliver us from evil tech giants!! Are you saying he might be a greater conman then PT Barnum?

You have good points and can probably make them without resorting to insults.

While I certainly don't want to import the combative and shallow standards of Twitter 'debate' to HN, I do think we have a right tp be opinionated, and that my the term 'manchild' labels a specific sort of behavior rather than some petty or crude dig at his appearance or personal life.

Indeed, I'd wish that the HN community was more hostile to Musk's bulldozer approach, seeing what it did to both the company, and the engineers. The whole affair has been needlessly destructive -- obnoxiously so -- and his getting away with it harms the profession.

OK. I thought that was funny personally.

> While I could take a Ghandian approach of not reciprocating such negative behavior

I didn't think it was comparable to the British treatment of India under the raj, but OK.

"Manchilds" track record in this Game of Life is by and large a testament to how well he understands its rules. And, maybe, how silly they are.

I believe it's an intentional and strategic move to have every API client to start paying right away.

He probably started to realize that other people began to realize just how many Tesla bots there are.