Thrift suffers pretty much all the same problems as protobufs and has many similar dysfunctional failure modes, like places that rely on serializing into Thrift structs stores in hdfs and treating that like a de facto database, with Thrift struct definitions as the schema. It is so miserable to work in code bases like that.

This is part of the original lambda architecture. What would you recommend the schema be represented in instead? Or are you preferring something like NewSQL?

I can’t tell if you mean a schema for the generated RPC or just for data storage. I’m only talking about data storage. But for RPCs, I just think don’t autogenerate them. Just expose an RPC API and let other people write code to consume it in whatever language’s packages for web requests, and don’t ever transmit things that are expected to automatically be treated as any type of object. Just send JSON or an equivalent thing in a more optimized buffer that is never allowed to be anything but a key-value store for primitive types.

On the data side, use relational database systems. Yes, even for huge modern webscale event data for apps or services with hundreds of millions of users. Don’t ever use hadoop, period.

If you’re bigger than hundreds of millions of users or otherwise are generating hundreds of billions of records per day or more, that’s the size when you might _start_ considering something different than sharded and distributed standard RDBMSs, but you are likely big enough at that point that you need an in-house, highly customized version of a distributed file store that matches your usage patterns and cost optimizations in a way that a one-size-fits-all solution cannot, and so again you should never be using hadoop.

As a result, if you find yourself relying on Thrift data serialized on a distribted filesystem as if those “flat files” are a database and map-reduce is like your de facto table scanner, it’s a red alarm bad code smell that you are growing your data scale in a horribly broken way that’s going to cause huge problems the minute you have products which need a data model or some flexibility that cannot be supported when you’ve welded data infrastructure to Thrift objects.