Ask HN: How to handle 50GB of transaction data each day? (200GB during peak)

Re:Ask HN: How to handle 50GB of transaction data each day? ... by haddr on Wednesday February 24, 2016 @08:14PM (#11171684)

First of all, 50GB per day is easy. Now, maybe contrary to what they say below, do the following:

* Don't use queues. Use logs, such as Apache Kafka for example. It is unlikely to lose any data, and in case of some failure, the log with transactions is still there for some time. Also Kafka guarantees order of messages, which might be important (or not).

* Understand what is the nature of data and what are the queries that are made later. This is crucial for properly modeling the storage system.

* Be careful with the noSQL cool-aid. If mature databases, such as postgreSQL can't handle the load, choose some NoSQL, but be careful. I would suggest HBase, but your mileage may vary.

* NoSQL DBs typically limits queries that you might issue, so the modelling part is very important.

* Don't index data that you don't need to query later.

* If your schema is relational, consider de-normalization steps. Sometimes it is better to replicate some data, than to keep relational schema and make huge joins across tables.

* Don't use MongoDB

I hope it helps!

[ Parent | on HN ]

Re:Ask HN: How to handle 50GB of transaction data each day? ... by late2part on Wednesday February 24, 2016 @09:13PM (#11171921)

Here here ; +1.

Kafka = good. Postgres can do a TB; if you plan bigger than that consider Cassandra.

Also consider protobuf instead of json.gz.

[ Parent | on HN ]

Re:Ask HN: How to handle 50GB of transaction data each day? ... by stuartaxelowen on Wednesday February 24, 2016 @10:39PM (#11172221)

Or... Arrow?! :O

JK it's too early :(

[ Parent | on HN ]

Re:Ask HN: How to handle 50GB of transaction data each day? ... by OldSchoolPro on Wednesday February 24, 2016 @08:59PM (#11171863)

Interesting points.

I'm wondering about MongoDB. Have you tried it recently? It really improves with every release. We have a 20TB Mongo database and it's fast and problem free.

Also regarding the queues. In my personal experience a single RabbitMQ server with good hardware can easily handle 50GB a day.

[ Parent | on HN ]

Re:Ask HN: How to handle 50GB of transaction data each day? ... by stuartaxelowen on Wednesday February 24, 2016 @09:12PM (#11171911)

Can confirm, don't use mongo.

We're stuck with it now, and I really wish we had used Postgres for biz data and Cassandra for the high volume non-relational data. They're just materialized views on-top of your event stream anyway.

[ Parent | on HN ]

Re:Ask HN: How to handle 50GB of transaction data each day? ... by PerfectElement on Thursday February 25, 2016 @12:17AM (#11172530)

Can you expand on the MongoDB comment? I'm using it for my SaaS and don't plan on getting anywhere near 50gb per day, but I'm curious to know about its limitations.

[ Parent | on HN ]

Re:Ask HN: How to handle 50GB of transaction data each day? ... by faizshah on Wednesday February 24, 2016 @09:38PM (#11172001)

I agree but if their problem is that they want a scalable solution for processing a growing amount of data cheaply and robustly and potentially doing more complex work in the future (it's a growing business). This is basically the tagline of the current batch of hadoop ecosystem projects.

Investing in working with the hadoop ecosystem is a much better long term solution (if this is the case).

[ Parent | on HN ]

Re:Ask HN: How to handle 50GB of transaction data each day? ... by stuartaxelowen on Wednesday February 24, 2016 @10:39PM (#11172219)

IMO investing in a hadoop system constrains you - make your data accessible via S3, Kafka, and most importantly, a good data model, and you can attach the right data processing tool when a problem that requires it comes along.

[ Parent | on HN ]

Re:Ask HN: How to handle 50GB of transaction data each day? ... by faizshah on Wednesday February 24, 2016 @10:53PM (#11172271)

I agree that a good data model is key, in fact the way you structure your data is probably the most important thing.

But the reason that the hadoop ecosystem is so good is because it's a growing ecosystem with a lot of momentum and a focus on processing large and growing amounts of data at scale and in a distributed/parallel fashion. If this is what you need the ecosystem is perfect.

[ Parent | on HN ]

Re:Ask HN: How to handle 50GB of transaction data each day? ... by haddr on Thursday February 25, 2016 @08:31AM (#11174243)

If at some point of time you use hbase, using hadoop on top of that becomes a straightforward task. Both integrate very well with each other. But I think even without hadoop you can do. If the processing is relatively light (e.g. calculating some aggregates, etc) you can do it in near real time just connecting with another consumer to kafka topic.

[ Parent | on HN ]