They remarked how they spent a couple of days looking over the system which was complicated and creaky, they couldn't figure it out
Now it may be the case that they didn't really try to fix it and are just using that as an excuse, but OTOH, they may actually have been stumped despite best efforts.If the system really did defeat this smart engineer's days of study, then in my opinion, and the opinion of at least one discussion I've seen on HN, the software should have been rewritten. (IMO) this really needs more clarification before scorn is passed.
So they may explain that making such-and-such a change will take two days instead of a few hours, thanks to all the FactoryFactoryFactoryAPIXMLDescriptorSchemas they have to write. That is obviously a hugely different thing to having no idea how the code works and what they need to do to add or change features.
Or for another example, say instead of understanding a system in a day, it takes a week. The system sucks. But if it is relatively stable, even six days of lost time is trivial compared to the time needed for a rewrite and the risks involved.
I am not saying that this particular rewrite of Google’s XYZ is correct or incorrect, just saying that sometimes we might be right to leave well enough alone. Development time is not free, and in addition to figuring out the ROI of writing a new system, we should consider the opportuinity cost of having Mr. Brilliant rewriting a working but terribly written system vs. inventing something else that might confer a greater benefit to his employer.
- if a programmer who thinks of him/herself and/or is thought of by others as a genius comes to you and bluntly/opaquely describes their confusion over some code, just call them out as an idiot.
Make them realize that if they are going to come and describe a problem they're having, and they're going to spend/waste your time doing it, they better have their ducks lined up and already have details ready. "Oh I understand what the code does, I just don't understand why the coder did it that particular way" is an admission that the previous statement wasn't a qualified one. It's an admission that the programmer had no intention of approaching management with a rational conversation on the topic, just a, "Hey boss I'm going to re-do this whole thing". No one wants conversations like that in a professional environment, people are too busy with actual real problems for that kind of thing.
It takes a more technical manager than most IMHO to properly adjust for that kind of behavior, however. Many managers don't understand the technology as much as those they manage (perhaps it's different in strictly programming environments), and at some point "have to trust" what someone is telling them in shorthand. That is bad in this business. It may work in other industries, but it won't work here. A technical manager needs to be able to challenge a technical idea on it's merits, not on the aura of the challenger.
It doesn't mean coders have to approach managers with a four-page paper rationalizing their need to tie their shoes differently, and as the team gets used to the increased communication and personalities, shorthand understandings relevant to the team will become used. But it should always be placed on the foundation of proof. Trust but verify.
Treat your coders like adults, hold them to that higher burden of proof standard, and hold yourself as a manager to a higher technical standard to understand it, and the entire team can avoid this kind of junk. I come from network administration, and on medium to larger sized teams I've found that at first the smarter guys hate this method because they're used to being left alone to save the day. Being challenged ruffles their feathers in the beginning. After a little while, however, they realize that they're going to earn more respect because everyone around them, or at least their boss, is actually going to understand their brilliance.
If your system is not readable and undocumented, don't blame the guy who will come and decide to rewrite it.
With these pieces of information, the decision to re-implement seems like a reasonable one.
My issue wasn't that they re-implemented the subsystem, it was that they did so without understanding all of what it originally did and so they were guaranteed to burden users of that system with a huge migration task. Folks who were previous users of the old system had to figure out what he had implemented and where their features had gone, and a series of iterative steps as folks educated this person on their particular requirements, and a series of changes, the new system was eventually a nearly complete replacement.
The opportunity cost of those developers who were rewriting their code to talk to the replacement wasn't considered, there wasn't really a solid metric for 'goodness', it had none of the old bugs but it certainly had its share of new ones, it increased the technical debt of the source code base.
So here was a brilliant engineer who did write a crap ton of code and got it working, but the path they took caused a bunch of other engineers a lot of work as well. They did not acknowledge, in any way that I could see, the burden they had imposed.
The converse, fixing a large number of problems with precise surgical changes to the code base, were not things that were being held up as the model for success. That was a sad outcome of that culture, but one I value highly both in myself and in others.
So when someone tells me they re-wrote something from scratch, I'm always curious to see how they got to that conclusion. There are many good reasons to rewrite something, whether it was a prototype that ended up being shipped and kept alive with a hodge-podge of fixes, or circumstance changes where the total code lines would be less re-doing it than amending it. I enjoy figuring out these sort of 'coral reef' architectures, emergent things that were born of the bones of a million quick fixes over the years, and I have blown up a few in my time. Sometimes the reasoning is from external sources (like a system that had inadvertently become tainted with proprietary information).
At some point Apple was taken over by beancounters who wanted reports from programmers saying how many lines of code they'd written. Woz spent a day or two refactoring something, removed 5000 lines of code, and thoughtfully submitted a form with LOC marked "-5000".
They exempted him from having to submit any more forms.
Then you wrote that, when the new system appeared, "nearly everyone" moved to it. Doesn't this suggest that the new system was actually better? Why would most of the old system's clients, for example, pay the price to move to the new system unless they believed it was improved enough to be worth the migration cost?
I'm not saying that your conclusions are wrong, just that from reading your anecdote, it's not clear how to reconcile your claim that the rewrite was an mistake with the evidence that the result of that mistake seems to be preferred by nearly everyone who had the choice to keep using the old system.
I fucking cried.
He was like, "I hoisted out some stuff that should have been behind interfaces." ... I didn't need an interface because I only had ONE and specifically made the API have a low surface area so this could occur IF IT NEEDED TO.
I sucked it up and took it in stride.
Jumping in and solving a problem is usually the last thing you want to do, but that character trait(flaw) is what people think of as a good leader. Someone who will take you over a cliff with conviction.
Does 'Compact' mean that you originally did:
if( foo ) while( foo ){ do something }
all on one line and the boss split them up into multiple lines?
I ask because I worked a job where statements were combined on single lines for ease or reading and it was a nightmare.
I was a little peeved, but looking back not so much. I was receiving pressure and what I felt were conflicting orders from the CTO and CEO, and any company that lets someone go for showing initiative is not a place I want to work. I found a more fitting job doing high-speed creative product prototyping (where they want that kind of thing) afterwords, and for more money.
(It was a startup, and I now see it as having had some funny politics going on. I also think that one of the co-founders wanted me gone since I didn't graduate from one of the "right schools." He was a huge school-snob.)
Showed the code to my manager, he left the room saying I should do some "real" sysadmin work and stop playing around.
Just for a test, I started spidering big local news sites with I think 10000 threads and left it running for about 15 minutes.
Two things happened: 1. I consumed all bandwidth, rendering Internet unusable for the whole company, and 2. got about 10% of data other 4 Java servers could gather in a day
This stunt didn't get me fired, but my manager insisted I delete the program I wrote. Nobody was interested in results I got so nothing happened.
But it was fun watching bandwidth usage RRD graphs go through the roof :)
I hear these sorts of tales occasionally and I recognize a manager who is out of their league when comes to keeping someone with your skill set engaged. If crawling the big local news sites was something useful for the company, moving your application to a hosted machine where it had access to more bandwidth would have been a start, of course a demonstration is just that, a demonstration. The next step which is a bit harder might be to crunch the data into some sort of data structure for later analysis.
If this is the kind of problem you enjoy working on and are in the SF Bay area we should talk :-)
I figured this out later when I checked their blog and he had prominently mentioned how they like to make sure their candidates are the "right people" from the "right schools." It was a very explicit statement, essentially that being from the "right school" is part of being the "right person."
I encountered this a lot in Boston actually, to the point that I developed a complex about how I couldn't really do anything because I hadn't gone to Harvard. Then I moved away and realized that the rest of the world outside this bubble doesn't care about that.
Anyhow, I was working on a system once which had a very crappy implementation. Thousands lines of code in each source file. Spaghetti code everywhere. I spent a lot of time fixing bugs in that system. Clearly it wasn't productive. I decided to completely rewrite it. Spent a couple of months working on it, but when it was done, it had a cleaner implementation and far fewer bugs. I noticed that I hardly spend any time on that system any more - an indirect indication that the decision to rewrite was the right one.
Fast forward two years. I moved on to other projects. A new guy comes along (senior than me actually). His task is to make the old system work with new requirements. He comes with an idea - a complete rewrite. Since I worked on that system before, I get to review his design. He explains to me how his new implementation is going to have various "new" features. Then I ask him if he knows that the existing implementation does all that already and more. He had to admit that he have no idea how the existing system works, so he doesn't want to touch it - thence the desire to rewrite.
I gave him my advice but he proceeded with the rewrite anyway. The old code obviously got scrapped. Now, looking back I can't see where his new code ended up. Most likely his rewrite failed, so the new implementation never saw the light of day.
The moral of the story: even if you don't like the old system, you have to understand how it works in order to create something better.
I'm currently working on a project which was written somewhat poorly with little technical oversight, the result being a minefield of code which is slightly brittle and full of dead codepaths. The old system (which works, is complicated and creaky, and I spent some days figuring out out things worked) needs rewritten and if we weren't pressed for time I'd rewrite it in a heartbeat.
Given a goal, an API spec, expected behavior one can rewrite a system which one doesn't truly understand and still do it better.
Certainly, something built by one person is rarely the best. That being said, what's wrong with occasionally taking out the trash?
The result was exactly as you describe, functions which had clear logical errors but they were never called so it didn't matter, data structures where a third of the members were never populated. It took me a couple of months of back and forth before I felt I could even begin to attack it. And after implementing the kernel half and starting in on the daemon proper, Sun discovered 'divisions' and I found myself documenting like crazy so that I could 'hand off' something that wasn't a 'networking' problem.
I particularly liked the n-level deadlock detection algorithm, that was the experience that taught me that a data structure can represent a large set of potential computations all at once.
That's one of the fundamental issue with complete rewritea: the systems you could most easily rewrite are often the one you don't need to.
I concur, but for a different reason than you seem to imply (which seems to be "being conservative with changes and goal-oriented" -- nothing wrong with that on its own though). I noticed that my own coding improved dramatically after I started working with others. This is due to several things: (1) competition and trying to be better than others, (2) once you work with several people, you take extra time to document what you are doing, which occasionally leads to you rethinking how you should be doing what you are doing in the first place, (3) don't know about you, but I am personally far more prone to taking "shortcuts" when working on my own (no one to embarrass myself in front of), (4) last but not least, when working with others, you can learn directly from others.
I have found that unless I can explain something to someone else, then I must not completely understand it. And working with another forces you to be able to communicate and explain, and that gives you both a better understanding. The result is you find things you missed.
The topic of conversation here is interviewing, and the importance of trying to develop as qualified an understanding of a potential candidate as possible in an interview. Reginald posited the 'mother of all interview questions' around the question of invention and perhaps more subtly the question of building it yourself versus leveraging the work of others.
Its a great question because it starts a lot of interesting discussions around how someone evaluates tasks they are given and allows them to talk about what tools they can bring to bear on those problems.
I related that anecdote from Google because I gave more importance in my mind to 'invention' rather than 'discovery' before I went to work at Google. Google is, in my opinion, the technical equivalent of Mecca for invention. One of the things I learned from my time there was that you could go overboard on invention. Not simply 'not invented here' syndrome, but literally lowering the barrier to creating new systems so far, that the net result is a negative rather than a positive. (Google doesn't see it that way, which I understand too, this is simply an opinion after all).
I practical terms you are absolutely correct that getting something running will have a weight. maintainability will have a weight, and maybe continuity will have a weight too. And if someone ponders those weights, gets some feedback to test their understanding of the relative importance of them, and executes based on that feedback, that is a very good thing indeed.
The person I was listening too at Google either didn't do that, or didn't understand the implications of their choices with respect to the impact on the company, or perhaps the simpler answer was they just didn't think about it. Had they thought it through, I'm sure they would have mentioned it in their talk as the talk was supposed to be about best practices and that would have been a great point to get across.
Ultimately, all software advice cashes out as 'something running right'. (Or 'make money', if you prefer. The following is true regardless of what you see as the ultimate goal.) However, we do not have perfect information about whether arbitrary changes will increase that. Our decisions and estimates are going to be approximate and heuristic.
Chuck has provided an example of how decisions can be suboptimal because employees/smart-people are rewarded for looking good, not for actually increasing the amount of things running right.
One way to correct this bias towards overwriting is to include in the decisions a fudge factor against writing new code - expressed as an anecdote summed up in 'Sometimes they are too smart and do too much'.
That said, I think it's important not to go to the opposite extreme. Personally, I'm in the camp of being too quick to rewrite things myself. And though I've learned to control my gut instinct in this regard, I've also learned that sometimes it's right.
I can name several instances where everyone would have come out better just by rewriting the damn thing up front rather than trying to fix or maintain the old system.
TL;DR - Sometimes discarding your peers' work is the best way to respect them.
There's a point where it is really really smart to throw away pieces of that system and regenerate them. You produce a lot of cruft of 20 years.
That's really the hardest part of working by yourself; it is very hard to strike the right balance between self-criticism and unwavering focus required to create something, be it code or a work of art.
I wonder if you would hire me...
True story: I went to a talk given by one of the 'engineering elders' (these were low Emp# engineers who were considered quite successful and were to be emulated by the workers :-) This person stated when they came to work at Google they were given the XYZ system to work on (sadly I'm prevented from disclosing the actual system). They remarked how they spent a couple of days looking over the system which was complicated and creaky, they couldn't figure it out so they wrote a new system. Yup, and they committed that. This person is a coding God are they not? (sarcasm) I asked what happened to the old system (I knew but was interested on their perspective) and they said it was still around because a few things still used it, but (quite proudly) nearly everything else had moved to their new system.
So if you were reading carefully, this person created a new system to 'replace' an existing system which they didn't understand and got nearly everyone to move to the new system. That made them uber because they got something big to put on their internal resume, and a whole crapload of folks had to write new code to adapt from the old system to this new system, which imperfectly recreated the old system (remember they didn't understand the original), such that those parts of the system that relied on the more obscure bits had yet to be converted (because nobody undersood either the dependent code or the old system apparently).
Was this person smart? Blindingly brilliant according to some of their peers. Did they get things done? Hell yes, they wrote the replacement for the XYZ system from scratch! One person? Can you imagine? Would I hire them? Not unless they were the last qualified person in my pool and I was out of time.
That anecdote encapsulates the dangerous side of smart people who get things done. Sometimes they are too smart and do too much. So when the question of 'invention' comes up I like to wrap it around discovery. Sort of 'Tell me about a time where you built something that you had no idea how to build it when you started.' Things I look for are how did you go about discovering the path? What resources did you engage? What options did you consider? When did you 'call the vote'[1] ? My experience has been that things built by two or three people, working as peers, are often much 'better' (meet the needs better, cover more cases, have a more consistent model) than things built by one person.
[1] 'call the vote' - in parlimentary terms when continuing debate will not help you make a better decision, it is time to stop debate and to vote on the decsion. Good leaders look for this point, and move forward once all the voices have been heard.