News for hackers. Stuff that matters.
Re:HN is up again by Cerium on Friday July 08, 2022 @04:38PM (#32026660)
I came to the same conclusion by observing that there are posts and comments from only eight hours ago.
Re:HN is up again by jbverschoor on Friday July 08, 2022 @04:43PM (#32026737)
So that means dataloss.. Probably restored from backup.

Good news for people who were banned, or for posts that didn't get enough momentum :)

edit: Was restored from backup.. so def. dataloss

Re:HN is up again by dang on Friday July 08, 2022 @09:33PM (#32030407)
8 hours of downtime, but not data loss, since there was no data to lose during the downtime.

Last post before we went down (2022-07-08 12:46:04 UTC): https://news.ycombinator.com/item?id=32026565

First post once we were back up (2022-07-08 20:30:55 UTC): https://news.ycombinator.com/item?id=32026571 (hey, that's this thread! how'd you do that, tpmx?)

So, 7h 45m of downtime. What we don't know is how many posts (or votes, etc.) happened after our last backup, and were therefore lost. The latest vote we have was at 2022-07-08 12:46:05 UTC, which is about the same as the last post.

There can't be many lost posts or votes, though, because I checked HN Search (https://hn.algolia.com/) just before we brought HN back up, and their most recent comment and story were behind ours. That means our last backup on the ill-fated server was taken after the last API update (HN Search relies on our API), and the API gets updated every 30 seconds.

I'm not saying that's a rock-solid argument, but it suggests that 30 seconds is an upper bound on how much data we lost.

Re:HN is up again by sillysaurusx on Friday July 08, 2022 @10:05PM (#32030620)
Curiosity got the better of me. Why was there a 6 ID gap between the last post and first post? The answer seems to be that admins were making posts, which is neat. (There was also one lonely Flexport job ad.)

Is your backup system tied to your API? Algolia is a third party service, and streaming the latest HN data to Algolia seems pretty similar to streaming it to a backup system.

Re:HN is up again by dang on Friday July 08, 2022 @10:06PM (#32030632)
I posted a bunch of test things and then deleted them.
Re:HN is up again by scott_s on Saturday July 09, 2022 @10:18AM (#32035331)
I love this answer so much.
Re:HN is up again by sillysaurusx on Saturday July 09, 2022 @02:25PM (#32037916)
I really wanted to ask “How did you post things if the server was down?” but perhaps some things are better left as mysteries.
Re:HN is up again by O__________O on Saturday July 09, 2022 @03:15PM (#32038335)
You could see them via HN’s API before they were deleted, nothing interesting; API was back up before the www.
Re:HN is up again by dang on Saturday July 09, 2022 @05:57PM (#32039667)
Good observation. Posting something and then seeing it show up in the API was one of the things we were testing. It exercises a lot of the code.
Re:HN is up again by dang on Saturday July 09, 2022 @05:56PM (#32039655)
The server was up for us before it was up for everybody else.
Re:HN is up again by swyx on Friday July 08, 2022 @11:25PM (#32031273)
i got that Flexport ad too.. haha kinda alarming if they are the only YC company still hiring
Re:HN is up again by dang on Saturday July 09, 2022 @06:20PM (#32039826)
Btw, job ads get queued long in advance and then the software picks the next one when it's time for a job ad. After 8 hours of being down, the software thought it was time for a job ad.
Re:HN is up again by joshuamorton on Friday July 08, 2022 @04:51PM (#32026882)
> So that means dataloss.. Probably restored from backup.

If the server went down at XX:XX, and the backup they restored from is also from XX:XX, there isn't dataloss. If the server was down for 8 hours, the last data being 8 hours old isn't dataloss, it's correct.