Engineers Shouldn’t Write ETL

Re:Engineers Shouldn’t Write ETL by ignoramous on Friday March 18, 2016 @12:56PM (#11313087)

Fellow amazonian here. We switched from a massively distributed datastore (not to be named) to rodb for storage and found 10x improvement, not to mention eliminating cost and other head-aches; kind of expected since rodb is an embedded db...

[ Parent | on HN ]

Re:Engineers Shouldn’t Write ETL by scottlocklin on Friday March 18, 2016 @05:49PM (#11315428)

I'd like to know what rodb is. Columnar DB? Columnar DBs often beat "big data" -I've beaten decent sized spark clusters in one thread of J.

[ Parent | on HN ]

Re:Engineers Shouldn’t Write ETL by amzthrow23424 on Friday March 18, 2016 @11:26PM (#11317141)

Read only database. It is a hand optimized/compressed database engine that is used for big "once a day" data sets.

[ Parent | on HN ]

Re:Engineers Shouldn’t Write ETL by rspeer on Saturday March 19, 2016 @01:19AM (#11317441)

I'm Googling and finding a few things called "RODB" that don't quite match your descriptions. What in particular is it?

Or are you talking about just rolling your own format to dump your data into? I've done that, but I'd still appreciate if I could use something that someone else had put the thought into. (Something like "cdb" by Daniel J. Bernstein, but that's an old 32-bit library that's limited to 4 GB files.)

[ Parent | on HN ]

Re:Engineers Shouldn’t Write ETL by ignoramous on Saturday March 19, 2016 @01:30PM (#11319435)

rodb is in-house, non-FOSS tech @ Amazon.

[ Parent | on HN ]

Re:Engineers Shouldn’t Write ETL by scottlocklin on Saturday March 19, 2016 @11:56PM (#11321729)

Thanks, that actually helps, particularly since I sell a competing product (which isn't RO).

[ Parent | on HN ]

Re:Engineers Shouldn’t Write ETL by pneumatics on Friday March 18, 2016 @01:39PM (#11313456)

What is rodb?

[ Parent | on HN ]

Re:Engineers Shouldn’t Write ETL by csears on Friday March 18, 2016 @04:27PM (#11314692)

I'm guessing he means a "Real-time Operational Database". This seems to be a generic term for system like a data warehouse that contains current data, instead of just historical data. If you are taking the output of a Spark flow and storing it in Postgres or MongoDB or HBase for applications to query, then those could be considered RODBs.

Since this is Amazon, I suspect he is referring to SPICE (or their internal version), which was released last fall as part of AWS's QuickSight BI offering...

"SPICE: One of the key ingredients that make QuickSight so powerful is the Super-fast, Parallel, In-memory Calculation Engine (SPICE). SPICE is a new technology built from the ground up by the same team that has also built technologies such as DynamoDB, Amazon Redshift, and Amazon Aurora. SPICE enables QuickSight to scale to many terabytes of analytical data and deliver response time for most visualization queries in milliseconds. When you point QuickSight to a data source, data is automatically ingested into SPICE for optimal analytical query performance. SPICE uses a combination of columnar storage, in-memory technologies enabled through the latest hardware innovations, machine code generation, and data compression to allow users to run interactive queries on large datasets and get rapid responses."

http://www.allthingsdistributed.com/2015/10/amazon-quicksigh...

[ Parent | on HN ]

Re:Engineers Shouldn’t Write ETL by sharpy on Friday March 18, 2016 @05:42PM (#11315367)

Read-only DB. Think of it as read-only memory mapped key/value store.

[ Parent | on HN ]

Re:Engineers Shouldn’t Write ETL by sshykes on Friday March 18, 2016 @01:58PM (#11313578)

Good question. Is it this [0]?

[0] http://researcher.watson.ibm.com/researcher/view_group.php?i...

[ Parent | on HN ]

Re:Engineers Shouldn’t Write ETL by ignoramous on Friday March 18, 2016 @10:33PM (#11316980)

Like rocksdb or lmdb but optimized for reads. Read-only DB.

[ Parent | on HN ]