Yes definitely. This thing was a big exercise in "You aren't going to need it".

> so in the end it's mostly...just an app talking to Postgres?

Yes. We solve problems for our customers. It's nice if we can do that without also creating problems for ourselves :)

> seems like they've been jumping on bandwagons before

Just wanted to point out that this system is also written in Haskell. We didn't really switch bandwagons.

Fair enough! In the end it's great that you've been able to take a step back and look at your ultimate goal (i.e. the customer problem solving). That doesn't always happen enough, and it's a good sign for your IT teams :) Also the fact that you were able to get the time to actually do this.

So there's all of that stuff, and then you think, "Oh, and it gives us an easy scaling path if we ever find our data volumes growing at an unexpectedly rapid clip." And at first you think it's just a cherry on top.

It may seem too good to be true at first, but with everyone using it, and with most the alternatives looking like they'll involve at least as much dev effort during the initial analysis, it's pretty easy to miss that.

And it's only a while later, when you're already pretty heavily invested in the ecosystem, that you start to really understand some things that the bloggers and book authors don't talk about in public. Like how Spark makes Big Data and scale-out a self-fulfilling prophecy. You will need to scale out, even if your data should fit in memory, because Spark wastes memory like a 22-year-old football player wastes money.

And maybe you're an appropriately cynical jerk, and can therefore spot that it couldn't possibly live up to the hype from a mile away. Congratulations. Hopefully you're the technical lead. Even if you are, though, too bad, because, nobody else in the meeting room is as jaded about tech as you are. Certainly not the management folks who don't program or have been out of the game for years. And, since you are cynical and jaded, that means there's still a good chance you'll go with Spark anyway. Because that's a hard argument to win - unless you've already been burned by Spark in the past, you aren't going to have enough intimate knowledge to make an argument that's concrete enough to sound convincing. And because this isn't the hill you want to die on. And because, being appropriately cynical, you realize that, at the end of the day, it's not really your problem. Spark will get the job done. It'll be inefficient, sure, but the extra server and development costs aren't coming out of your pocket, they're coming out of the pocket of the person who wants to be using Spark.

If you want a single, simple answer that's easy to sell to management, I'd refer you back to the last paragraph of my original post.

The company I work is currently undergoing a similar rewrite from firebase to Postgres (although we're doing a gradual migration), and it's amazing how much code we have been able to throw away, and how much quicker we can move in the parts of the code that have been migrated.

I had a similar thought. Postgres is a great choice, but then they also went with Haskell ... I look forward to another blog post in 2-3 years detailed all the ways that Haskell failed them and that at the day they should have just gone with an industry standard language.

FWIW they have this post from almost 3 years ago about adding Haskell to their stack. I'm guessing the time for your prediction has come and gone.

On the other hand, I find it quite encouraging that Haskell was barely mentioned. It seems they viewed the risk as "changing the projects language" rather than "using a non industry standard language".

Scala is still the least-bad option for a JVM language.

Anyone who can't be productive in Scala (not "better java", not "worse haskell", Scala) isn't someone you want on your team anyway.

It’s funny that you gloss over one the big productivity issues with Scala. Somehow everyone who joins your team should be acutely aware of your flavor of Scala.

If the community decided if wanted to be a better java or a worse Haskell then I bet more people would be productive in Scala

Read Odersky's book, write code, still have access to all JVM libraries and a lot less brain damage.

There are people that try to write worse Haskell in everything from Perl to Kotlin too. The reputation is overblown and has little to do with the actual language.

I'm sure if you are very disciplined when writing Scala then this will not happen, etc etc. But the fact that people need to decide upfront which parts of Scala to use and which ones to avoid seem like red flags to me (and in fact were red flags, in my experience).

I expect kotlin to fall off everywhere except android once loom and the rest of project amber land.

Source, please?

> Anyone who can't be productive in Scala (not "better java", not "worse haskell", Scala) isn't someone you want on your team anyway.

Why is that?