The AWS and MongoDB Infrastructure of Parse

Re:The AWS and MongoDB Infrastructure of Parse by sandGorgon on Thursday February 09, 2017 @11:39AM (#13608209)

> At this very moment, I'm working on a classical DBMS, searching through around 200 million tuples within the order of 2 seconds (partially stored columnar using psql array types and GIN indexes).

hi, this comment was timely. I have been asking myself what to use to persist a 10 million node graph. There has been a lot of marketing around using Hbase (obscene to setup and maintain) and Cassandra (much better.. but paywalled).

My main db is postgresql and we use jsonb quite extensively. The problem is that there is a lot of statements that "a single huge jsonb table in postgresql will never scale. you will have made a horrible mistake".

Seems you have ample experience on this front - whats your take?

[ Parent | on HN ]

Re:The AWS and MongoDB Infrastructure of Parse by mhluongo on Thursday February 09, 2017 @11:55AM (#13608413)

Depends heavily on your data access patterns. If you need eg recursive queries or pattern matching, you might be better served by a graph database.

[ Parent | on HN ]

Re:The AWS and MongoDB Infrastructure of Parse by edejong on Thursday February 09, 2017 @03:36PM (#13610687)

hi sandGorgon. I agree with mhluongo on this. If you want to do things like transitive closure or connected components, you fall in the 2% section above. However, if you want to encode graphs for message systems, a simple table with parent-id might do just fine. Wrt jsonb: if you only do key->jsonb tables, you might want to consider another kind of system. Nevertheless, we've got millions of trajectories, connected to millions of annotations (links to trajectories with semi-structured jsonb), and combined with some GIN indexes, it works pretty well.

[ Parent | on HN ]