But you can have modularity within a single codebase and some languages / build-systems allow you to define very clear boundaries between your different modules. In that case, I don't think you have necessary a lot of code duplication (you might have slightly increased consumption due to the use of more indirection). But you might have to code to interfaces and explicitly eliminate coupling etc.

If you look at my portfolio[0], you’ll see it packed with things like widgets and adornments. Most of these, I use in my own projects. I eat my own dog food.

But today’s languages, with their reliance on extensibility and interfaces, makes writing standalone code assets a lot less cumbersome. I write in Swift, and there are some pretty big Swift "extension suites" out there[1]. The Apple Swift Package Manager is nice, too[2].

It's just that when I write a code module, I do it as a standalone product, with a full test suite (usually a harness[3], which often takes the form of a "full-fat" application), and a unique release management cycle.

My modules are pretty independent, so they aren't usually custom-tailored for all of their implementations; acting in a fairly general capacity.

[0] https://stackoverflow.com/story/chrismarshall

[1] https://swifterswift.com

[2] https://swiftpackageindex.com

[3] https://littlegreenviper.com/miscellany/testing-harness-vs-u...

What's really missing is the willingness to say "no". There's pressure to put fancy or trendy toys into applications to look up to date or to get experience on the latest fads. KISS and YAGNI are often ignored for eye candy or Fear of Falling Behind.

For example, fairly early in the 2000's I had a web form that only did server-side date validation. It worked just fine. But client-side date validation was becoming "expected" because everyone started doing it and the manager asked me to put it in. So I found a JavaScript component on the web, and it worked fine at first. But a few months later a bug was found in it that appeared on a certain date, and it generated a lot of phone calls from users. Those phone calls would have never existed had I gone with my gut and left it out. Few users are going to spend all day making date errors such that client-side date validation is NOT necessary. It's a me-too toy.

Warren Buffett often cites his ability to say "no" as to why he's so wealthy. He looks at the actual tradeoff numbers, not peer pressure.

I just can't conceive of how modularity, in general, causes duplicate code rather than reduces it. And this is before considering something like generic code which could potentially further reduce duplication. Every time we've been deliberate about using modularity in projects it ends up creating libraries that reduce the total amount of code across projects. Even if it's just in one project, good modularity has been used to reduce a lot of duplication.

But, for example, say we are talking about a communication module. One module may use a well-known library as a bundled dependency, and another module may use the same one. They may be smart enough to use dylibs, in which case, you have the potential for "DLL hell," or they may use static libs, in which case you have duplication of code, and maybe resource contention.

I would not consider static libraries to be duplication of code. Duplication of assemblies or compilation unit, sure, but not code. The code itself has not been duplicated.

Yeah, like my node_modules folder doesn't contain tons of duplicate code.

/s

Regarding the development experience:

I think it's a tug and back-and-forth kind of dynamic.

Modularity has exactly the kind of effect you talk about, until it doesn't.

At some point, the cost of communication (between people!) responsible for different modules can increase until it's a net negative to try to adapt a module, or even find out it exists and is suitable.

Of course, a monolithic project of comparable size has likely collapsed from its weight into a ball of mud.

Development tools have come quite a ways, but the science of getting teams to develop software together, perhaps not so much.

Yup. I think I'll put that on a T-shirt.

That said, in reality, doesn't any sort of modularity imply increased complexity?

  - you need to manage your modules and figure out where to put the code
  - you need to figure out mechanisms to share and reuse it, as well as test every single integration (breaking changes will be harder)
  - you need to invest significant resources into the Ops side of things - maintaining multiple deployments and making them talk to one another

I actually wrote a blog post on modular monoliths a while ago, that mentions using feature flags, however i'm not sure whether the tone/quality is entirely appropriate for HN: https://blog.kronis.dev/articles/modulith-because-we-need-to...

Regardless, it feels like the industry perhaps jumped over the concept of modular monoliths entirely and went straight for microservices.

Of course, if we're talking about modularity purely on the code level, then dynamic loading of code is its own can of worms, as can also be many methods of installing extensions.

That's not to say that everything on HN needs to be cut and dry, but factual discussions seem a bit more productive, in my experience. To that note, having any stats on the prevalence of modular monoliths would have been awesome, or maybe even more information about feature flags - those are often not discussed all that much!

Example: In a monolistic application you can easily send out telemetrics of different "components", e.g. player hitpoints, coins in inventory, days in game, etc. But in a microservice structure (at least the one I saw), since hitpoint service doesn't care about coins, and coins service doesn't care about days in game, to get all three in one telemetry, developers have to break the barrier and connect them in one way or another.

So essentially you still get to write them into sort of monolithtic, but uglier and needs more arguments (but still have to write them at the end because CXO needs the report).

(Long ago, the software industry wanted to copy the electronics industry, where people would assemble software from components listed in catalogs. We never quite got there, but you can see the influence of this metaphor all over the place when people talk about "components". It required the development of libre+gratuit Free online libraries to get there, causing an absolute Cambrian explosion and the left-pad problem)

Early in the GUI days, you had UI components that were pretty independent. But now they are expected to match the look and feel of given OS versions or brands such that they can no longer be truly isolated: they are expected to integrate somewhat with the OS. If people had the guts to live with "ugly", a lot of things could be simplified. I discuss saying "no" to the IT Fashion Show nearby. It may read like a git-off-my-lawn rant, but software could be cheaper if you kick fadsters off your lawn. Beauty ain't free.

But in reality, as a BI Developer (prev as a data analyst), what I really don't want to do is a join by timestamp (and a userId), pretty much the only way to join two "subsystems" that are not interconnected.

Example: Analysts want to know what's the coin balance when player starts purchasing energy bundles. Now that they don't know about each other so the telemetry only has one common field: userId. Eventually either BI developer (who develops DWH/View/etc.) or DA has to write a join by userId and timestamp to get the most accurate coin balance when player purchases an energy bundle.

SELECT *, ROW_NUMBER() OVER(PARTITION BY userId ORDER BY timestamp DESC) AS rowNum FROM A LEFT JOIN B ON A.timestamp <= B.timestamp and A.date = B.date

And then write another query to take the top row based on timestamp, and for sure this method may lose some data (e.g. what if userId doesn't have data in table B that day?).

My original answer was perhaps a bit terse and should have been more along the lines of "that is what a good schema design should fix". Your original position that "microservices make things more difficult for analysis" is absolutely true, but I guess my observation is that while you might get lucky and have everything you need in one place in your raw data, for any sufficiently complex system (especially if there are 3rd party components involved) you're going to need to do some real work to transform that data to build a repository with a flexible schema to support your analyses and make yours and your colleagues work easier and more productive. You are 100% right in that every layer of indirection makes this slightly more difficult, but unfortunately that is something you need to design for in all but the most simple systems - this is true today the same as it was true 20-30 years ago.

Hopefully you're already doing that sort of design, but my experience is that disciplined schema design for effective analysis is incredibly rare, despite it being an old, crufty, well established and documented area ala Kimball and old school Enterprise Data Warehousing and BI. Not all of it translates perfectly to the world of Big Data(TM) but the fundamental concepts of designing such that your data can be turned into easily queryable and useful measures and dimensions still very much applies.

For your example, the correct correlation ID completely depends on the context in which you need to do your analysis, and that should be ideally baked into your telemetry schema where applicable (and where it makes sense from an architectural/performance/data volume perspective), but if not, then baked into your intermediate ETL schema such that it is easily accessible for analysis. Perhaps it's a contextual or global incrementing index that more easily allows you to get the "latest" transaction per context (maybe a session or a match in your game, or an encounter, or outside of games, a shopping cart checkout flow), maybe your client passes more contextual information to the microservice to allow you to more easily hierarchically partition into your context. Maybe if you don't have that level of control downstream (like in a 3rd party service for example) you have an ETL or streaming job generating the global or local context from the raw telemetry that makes it easy to join against.

And lack of context for analysis isn't even just a microservices issue, it can even happen in any kind of modular software - in the game I'm working on right now, for example, the crafting system has no need to know about overall state from the player, hence the crafting module has no (easy) access to player state that might be relevant for downstream analysis - so even in what would be considered a monolith, getting access to player health/armor/xp/etc in the context of crafting analyses (e.g. do players craft medkits at low health more often than at high health?) is difficult and really uglies things up, much like you imply. But making available a playerStateIndex that can be used to join back to that state (managed by the player state system) is doable and introduces no real problematic module coupling, and is a repeatable and extensible pattern to use across other modules and even external services.

I've spent the better part of half of my 20+ years in the industry doing some form of data analytics, and my message to the analysts/data scientists I work with is always "if you have to do crazy gymnastics to do your analyses then our schema design needs work". You don't want to be in the (all too common) world of analysts spending 80-90% of their time doing data transformations and cleaning and 10-20% of their time doing analysis.

Yeah I figured out that if the query is over say 50 lines long and has weird marks everywhere (say DISTINCT with some windows function) then it's pretty much a bad database design.

Then I move to the data side and figured out that it actually also has something to do with the original raw data sent by the app.

So eventually _someone_ has to take the burden and do the gymnatics and frankly it totally depends on who is closer to the CXO. In our company the developers are in favor so they can do whatever they want.

>Hopefully you're already doing that sort of design, but my experience is that disciplined schema design for effective analysis is incredibly rare, despite it being an old, crufty, well established and documented area ala Kimball and old school Enterprise Data Warehousing and BI. Not all of it translates perfectly to the world of Big Data(TM) but the fundamental concepts of designing such that your data can be turned into easily queryable and useful measures and dimensions still very much applies.

EXACTLY what I'm feeling now. We are trying to work out a DWH on top of the raw telemetries (basically some simple transformations and routing done by Spark from HQ). However I really think that Kimball doesn't fit very well with the big data thing. I don't know how to say it clearly because I just moved to the data team for a few months so I'm a total newbie and everything I _feel_ could be 100% BS. I think we should start from business requirement (i.e. from our data analytics lords), and then bend the rules (Kimball will require a LOT of extra joins if what I'm reading it correctly) and keep the DWH tables sort of flattened but in a way that makes more sense than the original raw telemetry tables.

I'm not in control so I don't know what it will result into.

>For your example, the correct correlation ID completely depends on the context in which you need to do your analysis, and that should be ideally baked into your telemetry schema where applicable (and where it makes sense from an architectural/performance/data volume perspective), but if not, then baked into your intermediate ETL schema such that it is easily accessible for analysis. Perhaps it's a contextual or global incrementing index that more easily allows you to get the "latest" transaction per context (maybe a session or a match in your game, or an encounter, or outside of games, a shopping cart checkout flow), maybe your client passes more contextual information to the microservice to allow you to more easily hierarchically partition into your context. Maybe if you don't have that level of control downstream (like in a 3rd party service for example) you have an ETL or streaming job generating the global or local context from the raw telemetry that makes it easy to join against.

It's...not really easy to come up with an intermediate "common context" id sometimes. Session Id doesn't work because each session takes a lot of time and nothing else really are real-time. Since I'm the only guy in the company for requirements taking (I really think developers should have their own business analyst), it's pretty painful that each time I have to fight both sides. I have to ask the DA team to relax their requirements (realtime? No it's not possible, how about same session?) and ask the developers to bestow more connection fields (if you have this field DA will be really happy), thus the vent above :D

If the operating system and standard libraries provide better facilities for the common parts, that can go a long way to solving these problems.

Also, for cross-platform SDKs and libraries, the provider may be forced to "reinvent the wheel." That may well be worth it, as developer bandwidth, and time-to-market are important variables.

If the OS and standard libs provide those dependencies, then I don't need to bundle them.

I'm thinking of things like the Go or Python standard libraries, or OTP in Erlang. On iOS you have a rich set of built-in frameworks, etc., etc.

This can lead to some rather...overfed...applications, and four- or five-digit bug counts.

For example, I no longer trust people that describe themselves as "iOS developers" to know the first thing about actually developing for the Apple ecosystem. In some cases, they don't even know how to use Xcode.

I'm convinced that the interesting stuff happens at the interface. It's just like for neural networks: the information bottleneck makes each tool distill information inside its output.

1+1 > 2 when multiple people cooperate, thanks to this. I think the same can be applied to software. It really has plenty of advantages: code more audited, swappable parts, common interfaces, standardization on data exchange formats, testability of the components.

I would adjust that, like so:

This is actually a call for restrained, jargon-free, Product Management.