News for hackers. Stuff that matters.
Re:What every software engineer should know about Apache Kafka by realtalk_sp on Saturday May 16, 2020 @04:47PM (#23207082)
The GCP Pub/Sub API has largely replicated all the features you'd want out of Kafka (including Consumer Groups). The primary consideration at this point is cost. There's an inflection point in size (at some very large message volume) where it makes sense to start running your own Kafka cluster and hire a dedicated person or two to manage it. Most companies will never get anywhere close.

Any project just starting out should use Pub/Sub. One thing I really like is that GCP provides emulators of Pub/Sub et al for local testing. That used to be a bit of an obstacle not too long ago.

In terms of lock-in, I don't see how that applies to an AMQ. The data moving through it should only be transiently persisted, up to a week or two at most in the usual case.

If you want to avoid cloud lock-in, have DB backups, use Postgres/MySQL/etc, containerize your service(s), replicate data in object storage, etc. Common sense stuff, if that's something that's of concern.

Personally, I've seen "vendor lock-in" weaponized as an excuse for a lot of costly NIH bullshit. It's painful to reflect back on a project that could have involved literally a tenth of the time and pain it ended up taking because of that one choice alone.

Re:What every software engineer should know about Apache Kafka by gigatexal on Saturday May 16, 2020 @05:55PM (#23207545)
IMO lock-in fears are overblown. Build stuff out quickly and prove your idea and then refactor when you have customers and revenue.
Re:What every software engineer should know about Apache Kafka by SpicyLemonZest on Saturday May 16, 2020 @06:04PM (#23207590)
Refactoring a growing system while maintaining bug-for-bug compatibility is extraordinarily hard. Most people I know who've gone through such a migration never want to do it again.
Re:What every software engineer should know about Apache Kafka by halbritt on Saturday May 16, 2020 @08:09PM (#23208271)
Most people are simply bad at migrations.
Re:What every software engineer should know about Apache Kafka by gigatexal on Sunday May 17, 2020 @03:21AM (#23210284)
Wasn’t Facebook built on MySQL with the friend graph written in relational SQL and PHP? The idea was but out and then perfected later. In my head this is the same as doing it on AWS or GCP fully on their proprietary stack and then once making some coin you start a refactor or a migration in parallel to your own hardware and stack as costs warrant it.

Dropbox is famous for doing this when they did their move from S3 to pocket — an in house storage layer they built.

Re:What every software engineer should know about Apache Kafka by rcxdude on Sunday May 17, 2020 @07:54AM (#23211349)
It's not likely your product will have enough financial success to afford the huge amount of engineering effort facebook has put into improving their platform. If your system falls apart before you are making enough off of it to fix it you're going to just run out of runway.
Re:What every software engineer should know about Apache Kafka by halbritt on Monday May 18, 2020 @10:20PM (#23230681)
Or to add to that, there are often very good reasons to optimize well before facebook did. The sheer friction of making the transition to The Right Thing can often be too difficult to overcome.
Re:What every software engineer should know about Apache Kafka by SpicyLemonZest on Saturday May 16, 2020 @10:50PM (#23209101)
Ultimately, building a maintainable system requires that you allow for people being bad. If 10 smart engineers contribute to your system, and they each assume future developers will be good in their own distinct way, the resulting system will be so complicated that none of them are good enough to maintain it.
Re:What every software engineer should know about Apache Kafka by halbritt on Monday May 18, 2020 @10:20PM (#23230683)
Preferably, everything that can be is loosely coupled and extensible.
Re:What every software engineer should know about Apache Kafka by cglace on Saturday May 16, 2020 @09:47PM (#23208768)
If most are bad at migrations then what should most do?
Re:What every software engineer should know about Apache Kafka by halbritt on Saturday May 16, 2020 @09:54PM (#23208797)
Learn how to do migrations well. There's plenty of literature on the topic.
Re:What every software engineer should know about Apache Kafka by runT1ME on Saturday May 16, 2020 @07:22PM (#23208044)
I don't believe PubSub has a way to ensure linearizability between messages, where Kafka does have this by deterministic hashing of keys partitioning. Otherwise yeah, Pubsub looks pretty solid and economical.

At my company I'm guessing we push over 200M messages a day through it, and it is a breeze to maintain and stable as hell. Maybe gets harder a couple orders of magnitude more, but no complaints here.

Re:What every software engineer should know about Apache Kafka by jooz on Sunday May 17, 2020 @05:01AM (#23210696)
> The primary consideration at this point is cost.

Certainly. In my experience the operational cost of kafka is enormous. Ive seen incidents too often for a relatively small (<10 machines) cluster. I would expect that after a first long phase of finding the right configuration, to step on a phase were it runs smoothly, but I didnt see it yet.

Re:What every software engineer should know about Apache Kafka by dirtydroog on Saturday May 16, 2020 @10:31PM (#23209000)
GCP Pub/Sub is insanely expensive
Re:What every software engineer should know about Apache Kafka by bootlooped on Sunday May 17, 2020 @01:11PM (#23213380)
I don't have a lot of devops experience, but I was just struck by how cheap it appeared to me. $40 / TB? I can't even imagine how much money is sunk into managing the Kafka clusters at my employer.
Re:What every software engineer should know about Apache Kafka by nwsm on Monday May 18, 2020 @11:17AM (#23223381)
Can you explain? It doesn't seem that way at all.