>They're afraid.
In many companies (think FAANG), engineers, especially senior engineers are incentivized/forced to show fancy design docs as part of the annual appraisal process. The more complicated the design, the more 'foresight', the better.
If it sounds kind of ridiculous (TPS reports from Office Space anyone?) it is. But on the other hand, taking a bit less cynical look, the more massive a company gets, the more the voices which demand objectivity in all these promotion/bonus multiplier processes. So a kind of obsession about such weird 'measurable' metrics gradually builds up in the name of objectivity.
And as soon as there are metrics, you can bet everyone in the system will do their best to game them (it only makes too much sense to do so).
And thus you end up with over engineered systems all over the place.
A lot of the 'not-invented-here, let's reinvent it' style culture also develops similarly - you have too many smart people in a room where the work is just not that demanding. Even if you were to get over your personal existential crisis (why am I writing yet another crud app?!), if you're the type that wants to see a promotion every other year, your're forced to invent work this way.
I think some problems are just seen as sexier than others.
I just wanted to shed a light on some unfortunate external pressures faced by a subset of engineers, which contributes to this problem. Didn't meant to come across as cynical.
My solution was to work someplace that can’t afford to waste money on bullshit. Has worked well so far.
It's a great concept, and is often what irritates people. I work at a national lab, and senior management is absolutely obsessed with metrics, often to the detriment of the purpose of the organization itself.
There needs to be some humility too - especially in our modern data driven world. All that is important, you may not be able to measure, and all that is measurable may not be important.
Certainly not in Amazon. Surely there can be exceptions but in general the company has a culture of simplifying stuff anywhere is possible.
IMHO the largest contributor to over engineering in this company is people suggesting flaws in other's designs simply to have something to contribute during a meeting. I can't remember anyone, ever, telling me to remove something from a design doc (3.5 years).
I have added unnecessary complexity to my own designs as a response to comments. Not customer driven, not data driven, but somebody at a meeting got focused on something and it ended up getting added to the design.
I've had unnecessary complexity introduced to my code during code reviews as a response to comments, too. Most often it's to make things "more testable" so as to reach an arbitrary code coverage target.
https://raw.githubusercontent.com/aws-samples/aws-refarch-wo...
Ironically, that's how you are supposed to run Wordpress on AWS.
My original comment was 100% based on my experience as a developer at Amazon.
Most of the time it's people from the team that is hiring and one or two "guests" from other teams (but usually working in the same building).
Managers also have plenty of power to influence the decision, therefore keeping a very uniform hiring bar is really difficult. (But no, it's not "completely arbitrary")
Also, I wrote that the bar dropped a lot, to the point of shifting all employees level up by one, but this does not mean that the company hires 90% of the candidates.
If a team was hiring 1 candidate every 1000 screened resumes and now it's 1 in 100 it's a whopping 10x change... but that doesn't make you a terrible engineer!
Say one slightly dubious thing and you torpedo your chances of the job. Write one slightly dubious thing on your CV, and it will be rejected without a second look.
While I’m confident that most of the people rejected for those reasons would be at least adequate at their job.
Comprehensive Unit/Integration tests, a robust type-system, pick whatever suits your style.
It's a lot easier to refactor stuff when you don't have to worry about breaking something hard to debug with a big code change.
I started out by writing unit tests for everything, which became my handhold and documentation for what the system originally did. Then I started reorganising the code into a more structured and more readable form, without changing the functionality, as proven by my unit tests. Then I started asking domain experts what exactly it should do, unit tests in hand, asking if these answers were correct. If they weren't, I changed the unit test, and then changed the code to match.
Surprisingly painless for something that by all reasonable standards was a terrible mess.
Great book BTW, should be on a top10 must read list for software developers. (#1 will always be Peopleware[2])
[1] https://www.goodreads.com/book/show/44919.Working_Effectivel... [2] https://www.goodreads.com/book/show/67825.Peopleware
Our CIO was regaling us with the story about how they were going to change our office today, make it more open, fancy like Google or Facebook, free desk. But when I asked if they’d considered dealing with the noise issues we were having, no, no they hadn’t considered that.
It just blows my mind. I just really wanted to ask them what the hell they thought they were doing modifying the office layout without asking the people who need to work there.
I also don't see how a robust type really helps there. It might even be part of the thing that needs to be refactored. Besides, many languages don't have a very robust type system.
End-to-end tests and type systems certainly have their uses, but for refactoring messy code, I don't think there's a good substitute for thorough unit tests.
Although there are different kinda of refactoring of course. The case I'm referring to was about one very messy module. This makes it very easy to unit test. If instead it's the entire architecture of your application that needs to be refactored, then you're looking at a very different case, and end-to-end tests become more important than edge cases.
End-to-end (e2e) tests the entire stack: deploy the site with database and all, run a script that visits a page, clicks and stuff, and checks if the right things become visible.
Unit tests are the other extreme: you take a single unit of code and test whether it does what it's supposed to, while mocking all communication with other units of code.
Integration tests are in between those two extremes, and probably the least understood as a result (at least by me). They look like unit tests, and don't generally test the UI, but they don't mock the other components, and ideally also set up a database to test against.
I think there's a bit of overlap between unit and integration tests; a sloppy unit test where you don't mock (some) other components but treat them as part of the unit you're testing, start to look like integration tests at some point. If you want a clear demarcation, I think you might consider a database connection essential to count as integration test.
In Ada, the programmer is discouraged from using the Ada equivalent of int directly, and is encouraged to instead introduce a subtype that reflects the specific use of int (including automatic range checking).
This isn't as natural in C++ but is still possible. Boost offers a BOOST_STRONG_TYPEDEF [0] to deliberately introduce an incompatible type. (I do recall having trouble getting it to behave, but it's been a while.)
Whether this makes sense in most mathematical code, I'm not sure, but it seems like it's an option.
[0] https://www.boost.org/doc/libs/1_73_0/boost/serialization/st...
For an example, dig into some crypto libraries. They operate on bytes all over the place performing XORs, etc. A type system isn’t really gonna help you ensure you got the correct number of AES rounds and stitched the blocks in the right order.
IMO the only systems where this “type system eliminates most tests” philosophy seriously works are the ones that don’t do anything other than pass data between components without doing anything beyond calling some serialization methods.
Which is what 90% of programmers on this website are essentially doing. And for the remaining 10% there is likely a better suited, different programming language or type system available. Even if there isn't unit tests would still be very niche and the general case would be that by default you shouldn't be unit testing.
End to end tests are really slow, but if you can get them into the 300-1600 tests per second range then i have no beef. I value tests but I seriously grudge waiting for tests.
Also, have you ever seen a project where the code is a mess but the unit tests are perfect? Even if somehow you could write unit tests that would cover for super bad code (which you can't), it is extremely unlikely that your unit tests would be that amazing.
There's quite a lot you can refactor without breaking unit tests. If you've got a single 200-line function full of nested loops with cryptic variable names, modern IDEs make it really easy to extract those loops to their own functions. Figure out what they're meant to do, give them a descriptive name, and you already improved the code a bit without breaking any unit tests. If your IDE does this well, you could even do this without unit tests, but you really will need those unit tests once you start reorganising the code making use of the excessive number of parameters those extracted functions invariably end up with.
Of course you can write unit tests for super bad code. If it's a function that returns something, it's trivial to unit test, no matter how badly written that function is. If it calls other code, you have to mock those calls and test that those mocks get called under the same conditions. If they mess with global variables, that's terrible, but even that can be mocked.
If the code uses gotos to code outside the module, somebody needs to get shot, and I guess you need a unit testing framework that can mock those gotos. I've never seen one, because nobodoy uses gotos anymore.
With end-to-end tests such as when piloting a browser, it's not really easy to get things like tracebacks into the console output for example.
If you're working on a very specific box that has well defined, well known inputs and outputs then TDD is an excellent tool.
But for anything with a non-specific "We'd like to do X and display the result on Y" it just gets in the way.
I've worked on mobile apps before with a small team, and inevitably, we'll find bugs that show up in the user interface when the user rotates their phone. It's hard to unit test for rotation changes, and it's also hard to code a rotation change into an end-to-end test on mobile. Animations are also something that's difficult to test in an automated fashion, and all the testing in the world won't be as good as showing the animation in front of a designer. So some level of manual testing is needed on mobile.
I've worked on web frontends where there would be bugs with scrolling jumping back and forth. An end-to-end test using Selenium may not catch the issue, but for a user, it can be painfully obvious. Similarly, animations are also hard to unit test on the web. So some level of manual testing is needed on the web.
The only place where I could see manual testing NOT being needed is for backend development, since the input and output to a backend system is much more controlled. You could write an end-to-end test for any scenario a user could throw at your system.
In summary, don't underestimate the value of manual QA!
Seems like that "refactoring untested code" would be a well established recipe for disaster.
Not sure if it counts as a 'way', but a static type system helps too.
I have noticed people who are extra organized in real life, who keeps every single file in a right directories after download tend to have inclination for prematured code refactoring for future use.
If these guys become code architect then i end up doing so many unnecessary things. The fundamental assumption of the refactoring gets changed very fast and the code needs to be rewritten for the majority of the cases.
From a company level I find the instructions are clear, mostly holding the same contract between services as long as possible and less schema change.
Its the engineer with subjective idea of perfect code / supporting future work makes it even more complex
As the GP says, this is about fear. Fear that if you don't plan for it now, you won't be able to deal with it if it happens. Or in architecture terms, fear that your system will be faced with a requirement that it can't cover.
Trying to plan for every eventuality is usually wasted effort. Better to build robustly, and trust in your ability to adapt.
Don't do this and solve only for the problems you have now or are about to start in the next 2 sprints.
edit: and make refactoring acceptable and part of your engineering culture so you do it often
They're afraid.
Fear: If I don't plan for all these use cases, they will be impossible! I will look foolish for not anticipating them. So let's give into that fear and over-architect just to be safe. A bit of the 'condom' argument applies: better to have it and not need it than to need it and not have it.
But the reality is that if your design doesn't match the future needs really well, you're going to have to refactor anyway. Hint: there will always be a future need you didn't anticipate! Software is a living organism that we shape and evolve over time. Shopify was a snowboard store, Youtube was a dating website, and Slack was a video game.
So my answer: relentlessly cut design features you don't need. Then relentlessly refactor your code when you discover you do need them. And don't be afraid of doing either of those things because it turns out they're both fun challenges. The best you can do is to try to ensure your design doesn't make it really hard to do anything you know or suspect you'll need in the future. Just don't start building what no one has asked for yet.