Web search 30M rows sub-second, cheaply

Re:Web search 30M rows sub-second, cheaply by pgaddict on Sunday November 22, 2015 @07:36PM (#10612192)

Sadly the post is very light on important details. For example, which version of PostgreSQL and which kind of indexes? Because GIN indexes do pretty much exactly what he describes as 'tag->id' (but in form that is compressed and very efficient for processing).

[ Parent | on HN ]

Re:Web search 30M rows sub-second, cheaply by jgord on Sunday November 22, 2015 @07:52PM (#10612240)

You and others make a good case - my post was to summarize what I ended up doing, as an interesting hack / datapoint - not to slam postgres [ or Redis, or Lucene/solr, for that matter ]

The main takeaway is really how fast SSDs are and how much bang for your buck you can get on a single commodity host instance [ thru some creative wrangling of data structures ]

I will endeavor to followup with a fair comparison using postgres...

Can you suggest the best approach / config to keep the top-index working set in ram ?

[ Parent | on HN ]

Re:Web search 30M rows sub-second, cheaply by pgaddict on Sunday November 22, 2015 @08:14PM (#10612310)

I think the form of the blog post (pretty much just short bullet points) makes it rather difficult to get an idea of what's the intended takeaway.

I do share your conclusion that SSDs are very good value for workloads that need to do a lot of random I/O.

Regarding the best config for PostgreSQL, it really depends on the hardware. If you really want to use boxes with just 2GB of RAM, use smaller shared buffers and let the page cache do the rest.

If you need more help with tuning PostgreSQL for the benchmark later, either ping me at tomas(at)pgaddict.com or join [email protected] (I do occasionally lurk there, but others will surely help too).

[ Parent | on HN ]

Re:Web search 30M rows sub-second, cheaply by jgord on Sunday November 22, 2015 @08:23PM (#10612332)

yeah.. I started with bullets as a draft, intending to expand into prose, but it seemed borderline readable, so I thought Id leave it as an exercise in blogging.

awesome, thanks Thomas - Ill check out your blog too.

[ Parent | on HN ]