News for hackers. Stuff that matters.
Re:Ask HN: How does HN manage to be always online? by yodsanklai on Tuesday June 21, 2022 @06:27AM (#31822313)
(beginner question) How do they store the data? is an SQL db on overkill for such a use case? what would be the alternative? an ad-hoc filesystem based solution? then how do the two servers share the db? and is there redundancy at the db level? is it replicated somehow?
Re:Ask HN: How does HN manage to be always online? by petercooper on Tuesday June 21, 2022 @06:38AM (#31822376)
"ad-hoc filesystem based solution" is the closest of your definitions, I think. Last time I saw/heard, HN was built in Arc, a Lisp dialect, and use(s/d) a variant of this (mirrored) code: https://github.com/wting/hackernews

Check out around this area of the code to see how simple it is. All just files and directories: https://github.com/wting/hackernews/blob/master/news.arc#L16... .. the beauty of this simple approach is a lack of moving parts, and it's easy to slap Redis on top if you need caching or something.

There is a modern maintained variant at https://github.com/arclanguage/anarki/tree/master/apps/news as well if you want to spin up your own HN-a-like and have the patience.

File syncing between machines is pretty much an easily solved problem. I don't know how they do it, but it could be something like https://syncthing.net/ or even some scripting with `rsync`. Heck, a cronned `tar | gzip | scp` might even be enough for an app whose data isn't exactly mission critical.

Re:Ask HN: How does HN manage to be always online? by scottwick on Tuesday June 21, 2022 @10:38PM (#31831462)
Wow, I had no idea HN was built like that - I'm impressed. I really wish I could read the Arc code better though since I'd love to know more about the details of how data is represented on disk and when things move in and out of memory, etc.

Does anyone know of other open source applications with similar architectures like this?

Re:Ask HN: How does HN manage to be always online? by krapp on Wednesday June 22, 2022 @05:06AM (#31833576)
>Does anyone know of other open source applications with similar architectures like this?

There's a good reason everyone else just uses a relational database, and it isn't because everyone else is addicted to unnecessary complexity.

Re:Ask HN: How does HN manage to be always online? by justsomehnguy on Tuesday June 21, 2022 @11:00PM (#31831550)
> and it's easy to slap Redis on top if you need caching

With filesystem as the storage you don't even need Redis, OS would cache the most recent files anyway.

Re:Ask HN: How does HN manage to be always online? by krapp on Tuesday June 21, 2022 @06:37AM (#31822366)
Data is stored in flat text files containing Arc Lisp tables, or in RAM. There is no 'database' per se, unless they've added one and not mentioned it.

You can get the software and language HN is based on here: http://arclanguage.org

Re:Ask HN: How does HN manage to be always online? by niclo on Tuesday June 21, 2022 @06:45AM (#31822410)
I think the link is broken, it's not HTTPS
Re:Ask HN: How does HN manage to be always online? by krapp on Tuesday June 21, 2022 @06:51AM (#31822440)
Force of habit, I fixed it.
Re:Ask HN: How does HN manage to be always online? by russelg on Tuesday June 21, 2022 @09:54AM (#31823977)
I love the design similarity to HN.
Re:Ask HN: How does HN manage to be always online? by CobaltFire on Tuesday June 21, 2022 @03:31PM (#31828120)
That’s because HN is just about the only thing written in Arc, and everything else you see is a fork of an earlier version of HN.