If you need more than that, you should use the Firebase-based API (https://github.com/HackerNews/API). The public dataset is also available as a Google BigQuery table: https://bigquery.cloud.google.com/dataset/bigquery-public-da....
Edit: since this subthread is not really on topic I detached it from https://news.ycombinator.com/item?id=21617478.
Instead of further optimizing the already light design surely it would be easier to do a gofund me and get a server upgrade? I certainly wouldn't mind chipping in a buck or two if it relaxes limits.
That's...sad.
I think it's awesome that there's a site this popular out there that _doesn't_ run on resume-fashionable Google/Facebook scale infrastructure that it does not require.
"Yeah boss, we're going to have to pull out memcached and replace it with a kubernates cluster of MongoDB shards. For 'reasons'..."
In the past I got the IP of my server banned from accessing HN for sending too many requests in too short of a time span. I found the unban interface that you provide, lowered the request rate of my crawler and tried again but was still sending too many requests in a limited amount of time and got the IP of my server banned again. If I recall correctly, I got it unbanned a third time and lowered the request rate even more but then got banned again and then I think it would not allow automatic unbanning.
Don't remember if I just gave up at that point or if I sent an email about it, or if I just waited some amount of time if there was a statement about how long I would have to wait before being able to use the unban interface for the IP of my server again.
Anyway, an official answer about the acceptable request rate would be nice. Perhaps put it in the FAQ?
Also, if people doing automated GET requests were to create a unique UA string for their scrapers that include a way for HN staff to get in touch, like for example (but with actual names of bot and site)
where the page on that URL would list an e-mail address for getting in touch, as well as having a statement about how to verify that a given crawler belongs to that service, would that help in not getting server IP banned automatically?