- I wrote my own scraper for HN for reasons I no longer remember
- Scraper is called every 30 seconds by an Apps Script timer on a Google Sheet (to be dirt cheap, Vercel doesn't have free cron jobs).
- Caches the front page, plus ten comments per post into a single Firestore document (to save money on reads vs 20 reads per load)
- Home page loads via SSR with Nuxt.
- Post pages will load the rest of the comments from a separate doc for each post
- the other comments load instantly on navigation if on a halfway decent connection - but if not, you still get to see the first 10 comments to get an idea of the discussion while you wait.
- Added some Workbox stuff to get it offline-first.
- Added an IndexDB wrapper around Firestore for more control over cache behavior I think.
- It's worked now for over a year with no oversight needed and not costing a penny!
Then I got distracted because I wanted to rebuild it from scratch to not use Nuxt and never completed that rewrite. Been a while since I've worked on it, had no idea other people even knew it existed! There's definitely some bugs (I think CSS gets broken on deeply nested comments) but it worked enough for my uses so I moved on.
I did the same thing... lots of fun, but the "run out of money" part is rough! I resorted to freelancing, and now I'm going back to university.
I was curious to see more about the internals so I opened up Chrome Devtools and as soon as I clicked on the "Application" tab, it crashes Chrome and did it every time.
I just tried it and it was an amazing experience.
(I think I was able to connect to that with another phone and set up a wifi hotspot for all my devices... but it's been a while!)
HN was one of the only sites that was reasonably pleasant to use during that time. Still, I found it more fun and faster to surf via a VPS. So the VPS itself would load and render the page, and only send me the bytes over Mosh (a more efficient SSH protocol).
I tried a few terminal based browsers using this setup. The one that worked best for HN (especially comment indentation!) was w3m.
Here's the current thread in w3m: https://files.catbox.moe/zuabb4.png
I did have fast mobile data but it was expensive and limited. I watched a lot of videos on YouTube and one day realized that almost everything I watched was actually just lectures or discussions. So I could just stream the audio and save 80% of the bandwidth.
I set up a little PHP script* that took a YouTube URL and ran `youtube-dl -g f bestaudio` to give you the URL of an audio-only stream. (Directly from YouTube's servers!)
I could then just load it directly in my browser, or download it for later. (I think I should have used `worstaudio` instead.)
* That little utility page is not up anymore, but others found it very useful so I should probably make another one...
Another benefit of this is to battery life (decoding video is expensive) and being able to turn off the screen (a feature that Google banned from the app store, years before making their own (paid) version available...) Later I learned about f-droid and newpipe... good times :)