News for hackers. Stuff that matters.
Re:Verizon to Sell Tumblr to Automattic by JulianMorrison on Tuesday August 13, 2019 @06:28AM (#20684306)
Fantasy big data: let's use Hadoop and Kafka!

Reality big data: Let's shard it across Mysql.

Re:Verizon to Sell Tumblr to Automattic by evanelias on Tuesday August 13, 2019 @10:36AM (#20686220)
Not exactly. Tumblr has a pretty huge Hadoop fleet and decently large Kafka setup too. It's just a question of OLTP vs OLAP use-cases being powered by different tech stacks.

My answer above was limited to the product backend, i.e. technologies used in serving user requests in real-time. And even then I missed a bunch of large technologies in use there, especially around search and algorithmic ranking.

Re:Verizon to Sell Tumblr to Automattic by lugg on Tuesday August 13, 2019 @11:54AM (#20687015)
That's kind of the point though. Everyone has a Hadoop/Kafka, but when it comes to actually getting things done, good ole MySQL to the rescue.

I honestly don't see the draw for Kafka. And by all means I get it, I just don't buy it. Maybe I'm just holding it wrong or something.

Re:Verizon to Sell Tumblr to Automattic by evanelias on Tuesday August 13, 2019 @12:49PM (#20687583)
It really depends on the task at hand. I'm one of the most vocally pro-MySQL commenters on HN, and have literally built my career around scaling and automating MySQL, but I still wouldn't recommend it for OLAP-heavy workloads. The query planner just isn't great at monstrous analytics queries, and ditto for the feature set (especially pre-8.0).

For high-volume OLTP though MySQL is an excellent choice.

Regarding Kafka: in many situations I agree. Personally I prefer Facebook's approach of just using the MySQL replication stream as the canonical sharded multi-region ordered event stream. But it depends a lot on the situation, i.e. a company's specific use-case, existing infrastructure and ecosystem in general.

Re:Verizon to Sell Tumblr to Automattic by lugg on Wednesday August 14, 2019 @12:36AM (#20692749)
I don't think youre quite getting my point.

Kafka is not going to replace MySQL specifically because it depends on the task at hand.

If you can't replace MySQL with Kafka, then why not just stick with whatever queue/jobs/stream infra you had before kafka. At least those solutions are quite limited in scope and easily replaceable.

At this point Kafka is a solution looking for a problem.

Re:Verizon to Sell Tumblr to Automattic by JulianMorrison on Wednesday August 14, 2019 @05:23PM (#20700086)
My feeling about Kafka is that it's a useful tool to solve the "we MUST get this data to reliable storage IMMEDIATELY" problem. And to greatly mitigate the "each item must be processed and shown to be processed, exactly once" problem.

But there are relatively few situations where that's absolutely vital. And you can solve it with good ol' SQL.