Oh man I just smiled like the Grinch.

Me too. I'm not proud of that.

but it looks like somebody had their eyes on that issue

Expires: Tuesday, November 14, 2023 at 3:59:59 PM Pacific Standard Time

Aww man. Glad I grabbed a screenshot before it got rotated.

I had a job once, when TLS was new and half the internet was not using some form of SSL. We had to manually do keyswaps on a stack of machines to create a trust network for all the certs. IT really sucked. I made a note of the certificate expiry data and a calendar reminder 3 years into the future to start looking for a new job if I hadn't already.

Hopefully that reminder never slipped pasta your attention.

Yeah!, for context, I saw and fixed that spelling mistake of calendar / colander. This week really has been a wild ride. Can't text fast enough.

I hope you don't end up in a job where someone left on cert calendar reminder.

There is absolutely no way in hell those aren’t autorenewed. I think the answer is “quite a long time” actually. Eventually disk/machine rot in DCs will take over but it can take many months or more

Sure auto renewal but is it built to automatically roll certs across every system without any human intervention?

I would expect yes given they have tens(hundreds?) of thousands of machines alone

To be pedantic, that just mean a script exists that can roll the new cert out everywhere it needs to be. There could still be a human who needs to initiate that script being run.

Putting it in the crontab is an absolute no-brainer if you already have the script.

Everything I run at work is 5-10 years old or older. It all just hums along. Occasionally I replace a hard drive or a power supply.

You get massive DoS attacks pretty regularly? State sponsored hackers?

Now multiply that frequency against the ratio of machines Twitter has vs you.

On a modest Hadoop cluster of 300 machines with 20 disks a piece, we replaced 3 hard disks a week.

Spindles? That sounds like a ridiculous amount burn rate - it should be no more than 5-10% a YEAR. Backblaze reports only 1.6% quarterly failure rate https://www.backblaze.com/b2/hard-drive-test-data.html

Er actually missed its total 6k disks seems about right then

Is your stuff on the internet? I agree that systems that are 100% internal can get by for awhile without much maintenance, but anything on the internet is a completely different beast. Add in that Twitter is a huge target for DDoS, zero-days, etc..., and I don't see how anything just 'hums along' without near constant monitoring.

> There is absolutely no way in hell those aren’t autorenewed.

I’d take that bet.

I'd be very surprised if disk/machine rot is what it would take for such a large organization to start experiencing major problems. Strange user patterns from all this and recent features/refactors that are suddenly not monitored would hurt something in the stack within a couple weeks if it was really abandoned. Having a few dozen senior engineers/SREs familiar with big chunks of the infra could make it months though.

Exactly. Things that are abandoned will eventually fail.

Not everything is automated. There's a finite amount of time to create new automation, and that time goes first toward the things that happen hourly or daily. The things that happen monthly still get handled by humans. Not everything is in the runbook either, and what is often ends up scattered among many wiki pages and help notes in tools, so it's not easy for a newcomer to find. So those monthly things have to be done by people who remember.

They remember how to clean stuff up when a quota/capacity limit is being approached, and who to call when they need more, and they can expect a response. They remember how to recognize when their service is approaching overload, or about to enter an oscillating state, and they know how to nudge it back toward sanity before the errors start to pile up. None of that happens when whole groups are gone.

Sooner or later, a service will start to run off the rails in one way or another, and the person who inherited that service will either not recognize it or not find the solution in time. Then it's Fail Whale time.

Commenting to save this for Dec 13th!

If you click on post timestamp you can favorite the comment ;) I guess I hold twitter infra in higher regard after all these years they’ve been in business but who knows…

I hold them in high regard but honestly there's just so many moving parts the odds of anything that complex being able to ghost ship indefinitely aren't fantastic. Apparently the NOC team is gone too. If it makes it 1 week without major outages/major malfunctioning the departed devs/sres deserve all the props.

maintaining any reasonably popular web property is a constant battle of striking down spammers, scammers, and hackers. their basic server code might keep running for months, but unless their engineering teams have been doing some truly magical work up until now the quality of the content is going to rapidly degrade if there isn't active intervention to keep the bad actors suppressed.

And yet, somehow new Twitter has FAR fewer bots and spam tweets that three weeks ago. Hmm…

Edit: downvoting doesn’t change reality

What is up with the rage downvotes? It’s pretty easy to look at the replies to popular Twitter users. There are waaay fewer crypto scam bots everywhere. Why deny reality?

Those crypto bots being less pronounced probably has more to do with the entire crypto ecosystem being a bit... dying right now.

I had to eventually disable inbound PMs because the "offers to gain money working online" were just too many. So your mileage may vary.

> There is absolutely no way in hell those aren’t autorenewed.

Pretty sure a number of AWS and Azure engineers believed the same thing.

I ran certificates for a well known service with lots of users (and not many employees), and it was all manual until well after we were aquired. If you don't have very many different certificates, it's not that hard to do it manually, and it doesn't look like they have that many [1]. Digicert is pretty easy to work with on a manual basis.

Of course, even if it was automated, you still have to pay the bills, and if payroll quit as has been suggested in the thread, I'd be suprised if accounts payable is sticking around

[1] https://crt.sh/?q=twitter.com

The certificate looks like it was created on Nov 14th and is valid until Nov 14th 2023 to me.

Yeah it looks like they rotated it. It used to be this one: https://crt.sh/?q=DA26F72EACAC75CCEE794140441020848697AEB0

The new one isn’t even indexed on crt.sh yet.

My browser said that too, but that certificate may be cached from a while ago. The terminal gives a different reply:

  $ openssl s_client -servername twitter.com -connect twitter.com:443 2>/dev/null | openssl x509 -noout -dates
  notBefore=Nov 14 00:00:00 2022 GMT
  notAfter=Nov 14 23:59:59 2023 GMT

I think they rotated it. The new one isn’t in crt.sh yet, but for posterity’s sake this is the old one https://crt.sh/?q=DA26F72EACAC75CCEE794140441020848697AEB0