Feeding Graph databases – a third use-case for modern log management platforms

Re:Feeding Graph databases – a third use-case for modern log... by lmeyerov on Wednesday December 30, 2015 @10:28PM (#10815948)

We settled on "medium"-batching into Titan, and sharing your experience, I'm hoping the Datastax acquisition means ingest will improve.

For the bulk of our work, we do what you'd expect -- load terabytes into HDFS, (Py)Spark for straight SQL and some join helper functions, and occasionally, add some GraphX scala libraries.

I'm curious -- what did you end up doing, & for what?

[ Parent | on HN ]

Re:Feeding Graph databases – a third use-case for modern log... by rspeer on Wednesday December 30, 2015 @10:43PM (#10815999)

I maintain ConceptNet [1]. It's a large-ish, messy semantic graph linking together crowd-sourced knowledge (from Open Mind Common Sense, Wiktionary, some Games with a Purpose, etc.) with expert-created resources (WordNet, OpenCyc, etc.)

[1] http://conceptnet5.media.mit.edu

It's turned out to be a good input for machine learning about semantics, which has changed the goals of its representation a bit -- not only do I need to be able to load in data easily, I also need to be able to iterate over all of it. But some graph operations would be nice to have, too.

Many technical people I describe the project to immediately ask me what graph database I'm using, both before and after the ill-fated semester of grad school where I actually tried to use graph databases.

The answer to what I use now is: a bit of SQLite and some flat files. No need for HDFS, it still fits easily on a hard disk.

[ Parent | on HN ]