News for hackers. Stuff that matters.
Re:Monoliths Are the Future by WoahNoun on Thursday January 30, 2020 @05:28PM (#22196545)
Changing the schema of an upstream data source almost always breaks or requires updates to the downstream analytics system. It's an unavoidable problem whether its a microservice or a monolith; you just get to choose where you put the pain.

Consider:

Source Data -> Data Lake -> ETL Process -> Reporting DataWarehouse(s)/DataMart(s) -> User Queries

vs

Source Data -> Data Lake -> User Queries

vs

MonolithDB -> User queries

vs

MonolithDB -> ETL Process -> Reporting DataWarehouse(s)/DataMart(s) -> User Queries

A schema change in the source data should be easily updated in the ETL process in example 1. Most changes are minimal (adding, removing, renaming columns). And for a complete schema redesign in the source data, a new entry in the data lake should be created and the owners of the ETL process should decide if the new schema should be mangled to fit their existing reporting tables or to build new ones. Across the four models I outlined above, the first is by far the easiest to update and maintain, IMO.

Re:Monoliths Are the Future by satyrnein on Thursday January 30, 2020 @05:40PM (#22196676)
If the benefit of microservices is primarily an organizational one around ownership, Conway's Law, and so on, then Example 1 still seems problematic, because it's likely a different team that has to deal with the fallout.

Another strategy is that the service has an explicit API or report specification; that way, the team that owns the services also owns the problem of continuing to support that while changing their internal implementation.

Of course, whether the benefits are worth the cost is probably organization specific, just like microservices in general.