It's not really related to search, most of the cause is a pretty bizarre optimization method that reddit decided to implement fairly early on. The database structure is unusual and not very conducive to indexing (it's similar to an EAV model), so at some point they decided to basically write their own "secondary indexing"-like system that stores the "listing indexes" in memcached/Cassandra (with the data itself in PostgreSQL).

Whenever something happens that affects any listings (new post created, voting, etc), the site figures out all the listings it needs to update, and where in each listing the affected post now belongs and updates them all. So for example, if you make a new submission to /r/pics, it will go through and add the post's ID in the right spot to the "new posts in /r/pics" listing, the "hot posts in /r/pics" listing, the "new posts by yourusername" listing, and so on. As it's going through and updating all these lists, it also trims each one down to 1000 items.

It's conceptually pretty similar to a normal database indexing system, but basically maintains all the indexes "manually" and restricts them all to the top 1000 items.

If you're curious enough to dig around in the code, this is probably the main relevant file: https://github.com/reddit-archive/reddit/blob/master/r2/r2/l...

https://www.reddit.com/r/help/comments/6nxqjm/maximum_of_100...

I posted some more information about it a while ago here: https://www.reddit.com/r/help/comments/7en0uu/my_saved_posts...

Imagine I have a list of max length 5, when I initially fill it up it looks like [5, 4, 3, 2, 1]. If I save one more thing, it adds 6 at the front, then truncates the list and removes the 1 from the end, so now you have [6, 5, 4, 3, 2]. At that point, if I unsave #3, it just removes it from the list so you'd have [6, 5, 4, 2]. #1 is still saved, but nothing happens to pull it back into the list.

1000 probably seemed like a lot at the time or had reasonable performance, and it's just never been changed.

Thankfully, services like pushshift[1] exist, which has a sane API and the option to use plain elasticsearch.

[1] https://github.com/pushshift/api

"see all of my own comments": http://api.pushshift.io/reddit/comment/search?author=saurik&... (use &before=[epoch] for pagination)

"see all of the posts made to the subreddit I moderate": https://api.pushshift.io/reddit/search/submission/?subreddit...

I have a subreddit (/r/pushshift) that gives examples on how to use the API. I'm always happy to answer questions about the API and take suggestions to improve it.

I'm very very close to launching the new API which will have a lot more features and a better design than the current one.

Edit: I just realized my username on this site got truncated to uhhh ... wow.

I pulled the dumps from Bigquery and am going to load them into PG at some point here, so I can do arbitrary queries without hitting BigQuery every time. Haven't looked at how to do that in realtime though, retrospective queries are mostly what I'm interested in.

If you don't want to do that, there's also https://redditsearch.io/ - if you want to search back farther, be sure to set the toggle to "all" instead of "day"!