Good points. I feel software engineering as a field is growing a lot slower than it could, for a couple of reason.

The first is what you mention: everyone ignores lessons from the past and try to relearn everything from scratch, reinventing the wheel and best practices over and over and over, making the same mistakes again and again. Some communities also heavily encourages concepts like how experience doesn't matter, best practices aren't a thing, etc.

The second (and part of the reason the first is a thing), is the amount of people acting as experts that really aren't. Even big names and pseudo-celebrities in the field frequently have no idea what they're talking about. Once people realize this, they start questioning everything.

And thus, we're stuck in software engineering stone age. Being unable to separate the good stuff from the bullshit is holding us back a lot.

May I add to this with a third: difficulty in creating a strong system of mentoring?

Many companies have a formal mentorship program but it is often in name only (at least in my experience at 3 companies).

In a formalized apprentice program, a worker can feel okay about not knowing things and asking the person for whom they are apprenticing. Despite formal mentors, I believe many junior devs do not have a person that they feel comfortable not knowing things. Moreover, plenty of senior engineers do not make good mentors (either don't want to or don't have the strongest of communication and mentoring skills).

You'll have a crop of engineers with 0-2 years experience that don't feel comfortable not knowing something because they feel it can be a sign of technical weakness. Perfectly valid questions go unasked and a young engineer may spend 3-4 days trying to answer something that his mentor could help him/her with in 15 minutes.

Great knowledge isn't transferred optimally in this scenario.

Oh man, I relate to this feeling tremendously. I am a junior dev on an apprentice program (just shy of 1 years experience to date) and the times I have struggled with something because I let my pride, or fear of bothering people with inane questions, have been the lowest points in the last year. I always get to the otherside wishing I just talked to a senior dev - because more likely they could have saved me a stressful week.

Few random thoughts as someone who mentored many junior engineers:

- don’t care about your pride. You’re here to learn, so do everything you can to learn. Imagine even the worst possible scenario - people get fed up with you - you’ll go and find different job (benefits of being in a booming industry). Your old coworkers won’t follow you but knowledge will

- don’t think about all the time you spent looking for answers yourself as wasted. Some if it was, but you learnt also tons of other things that may not be relevant now, but will be in a future

- ask your mentors not only for direct answers to your questions but also for reason and/or how they came to that conclusion. Most important skill for you to learn is how to think about problems and how to find solutions, not the actual solutions to few problems you came across

For whatever it might be worth, here is my advice to juniors about being mentored:

It mostly is true that the only stupid question is the one you didn’t ask.

Please don’t ask the exact same question twice. Take notes if you need to. Asking for clarification or following up for more detail is fine.

If you have many questions and your mentor has other responsibilities, try not to keep interrupting them all the time. Discuss the best way to balance your needs and theirs, perhaps finding a time/place/medium where you can have a larger discussion and deal with several questions at once.

Assuming your mentor has accepted that responsibility voluntarily, they’ll probably be happy to help even with the basic stuff — we all had to start somewhere — but will appreciate it not taking over their whole day.

Also - make sure they understand your goals instead of your hyperspecific question - a lot of time can be saved by avoiding the XY problem.

http://xyproblem.info/

In my experience people with great knowledge generally have an urge to pass it on and they appreciate self initiative taken by the person learning. I have mentored a few people during my job and generally feel that my time was wasted only if they don't take initiative to learn themselves. Many a times I leave a crucial thing out of the answer and ask them to try to find for themselves as teaching is not about giving a ride but showing the way.

During the years I have found that it's not helpful to go into too much detail when teaching, as the people who are just starting won't be able to appreciate it and it may actually overwhelm them.

> In my experience people with great knowledge generally have an urge to pass it on and they appreciate self initiative taken by the person learning.

I wish my personal experience was the same, but it has been the polar opposite. In my current and previous roles, there is always too much work, which causes management to pressure development for deliverables, which results in juniors getting pushed aside. It's a terrible cycle and I have no idea how to improve it.

This is not necessarily the opposite. It is perfectly possible that the knowledgeable people you've worked with actually wanted to pass it on. They just weren't given the time to do it.

You should be angry about overtight deadlines and companies who live by those, not to the actual people. This also points to a possible solution for you: switch companies.

It's much worse than 3-4 days of wasted effort. A bad design can get so baked in it's nearly impossible to correct.

As a counterpoint to that, in some cases technology is changing so rapidly that things which were best practices no longer are and vice versa. Things that we take for granted now, like CI/CD, were not even technically possible 20 years ago. If people weren't constantly challenging best practices as new technology came out then we'd still be pushing our CRs through change acceptance boards.

Or think about "the cloud," "infrastructure as code," "immutable infrastructure," all that stuff that's considered best practices now couldn't have even been conceived of when we were spinning up new servers by driving to the data center and dropping a new 1U in the rack.

Good point, absolutely! The line is really thin between industry changing stuff (CI, cloud, distributed version control, type systems that don't suck, etc) and reinventing the wheel through arrogance or just not knowing any better (yet another testing framework, screwing up some well studied patterns :cough Java Optionals cough:)

The graybeards among us might chuckle at the thought of calling it "distributed version control" when the CI system requires a centralized copy of the repository acting as a source of truth in order to work.

CI systems only need the "centralized" copy as a reference point of what the current master is. Everyone gets their own entire copy of that repository though. If the truth suddenly changes it's known. I also know that if the copy in the "centralized" matches my own then the CI is also running exactly my code. If the centralized machine fails I can fail it over to any copy on any machine. That seems pretty distributed to me as compared to the original "distributed" version: e-mailing source code around. Version control like SVN and CVS were centralized in that me having a copy of the source and you having a copy of the source indicated nothing about whose version is correct. Moreover a failure of the central spot in the system is fatal unless you had backups as a form of pseudo-distribution whereas in today's version control a failure on any one machine is irrelevant to the preservation of the system.

I'd add to that the predilection for hiring young engineers and overlooking older ones. While it's true that older ones can get stuck in past practices, they also recognize things that have been tried before, seem like good ideas, but just don't work for non-obvious reasons.

For example, implicit declaration of variables in programming languages.

A strong team will have a mix of older and younger engineers.

Language design is its own thing with its own body of knowledge and experience. Of course, often times new languages were designed by those who could engineer their compilers and interpreted, leading to lots of mistakes that would be more apparent to a seasoned language designer.

Mentorship at the highest level (principle to senior and below) is desperately lacking. I vow myself when I reach that level to pass it along the best I can, but something I desperately searched for when I was starting.

I pair programmed for 3 months (which I consider apprenticeship / live action mentorship) with someone who was a way better engineer than me at my first "career job" out of college. I learned more in those 3 months about software engineering than in my previous two.

I'm writing the book on senior+/lead to junior/intermediate mentoring with my company. I've been floating my notes and mentoring guidelines around the department heads and other leads. I love it, personally. Department heads and managers see the value but are reluctant when they see that I'm willing to spend months with a junior or that I might myself out there for hours a week with an intermediate. If I come across a particular interesting performance problem, or something like that that is challenging and requires a certain type of experience to get into, I'll schedule time to pair with 1-3 of my team members on it so that they can hopefully benefit from the experience. The real challenge, though, is that not everyone wants to mentor. I personally think that it's critical and a core part of my responsibility but most of the other senior+/leads where I work are not engaging in much meaningful mentorship with their teams.

> I vow myself when I reach that level to pass it along the best I can, but something I desperately searched for when I was starting.

Mentoring is one of my favorite thing. Finding people who want to be mentored is...challenging though. Probably a chicken and the egg (one of the other posters mention how people are put in a situation where they can't be open about not knowing things), but so many entry level engineers just pretend they know everything.

Completely agree with you though that this kind of dynamic would drastically improve the industry.

This is a salient point. I've personally found that most junior/intermediate people want to be mentored but it should always be done carefully and with consideration. I've had a couple of junior/intermediates, though, who, like you are saying, just did not want it. I think that most of them have moved on from the company but there is 1 person who is still around and who is now the type of person who has 5 years of experience but it's all the same year.

> I learned more in those 3 months about software engineering than in my previous two.

Do you mean you learned more than you did in 2 months of school? Did you drop "years" as the last word? Did you mean it wasn't a great experience because you could have learned what you did in 2 months instead of 3?

Part of the Stone Age is that the good stuff is already out there in books nobody reads, rather than fancy Medium posts.

Do you happen to have a list of such books off the top of your head?

Do you have any book recommendations?

Yes.

This is only one of these "must-read book list", what you can do is search several of them and see which ones are quoted by almost everyone!

About "everything" I would rate the following books very highly, for their sheer usefulness:

- Thinking Fast and Slow (really do the exercises!)

- then Neuroleadership which describes the purpose of brain structures, it's somehow wider than kahneman

- A Mind for Numbers (include very useful data points for knowledge workers)

- Distinction: A Social Critique of the Judgement of Taste

- Cutting Through Spiritual Materialism

About programming I would rate the following books highly, but at the same time it seems a bit fruitless to me to read about programming, rather than spending more time reading about psychology or sociology:

- the Economics of Software Quality

- Peopleware

- Pragmatic programmer

- Mythical Man month

^you'll see that these books are hardly kept secret, it's just that people quote them but don't actually read them, or so it seems.

Part of the problem is the concept that programming is "engineering". It explicitly is not, it doesn't have a defined set of criteria for evaluating design or ensuring results.

"Doing Agile" doesn't qualify as "engineering", CMMI level <5 isn't engineering. Even CMMI level 5 is only engineering in a very small domain. Most shops claiming CMMI level 5 are only qualified in one particular aspect of software development.

So we're still in the "master/apprentice" and "journeyman guild" days of our profession. But Universities and Colleges are desperate to sell their 4 year "software engineering" degrees, so cover up the lack of rigor with a mountain of bullshit.

So the first step to really establish software development as an engineering discipline is to stop fooling ourselves. Being able to "refactor my FactoryManagerFactory class" is not engineering.

When a programmer has to sign off and take liability for any failures like a PE does on a bridge is when we'll be able to say we are "engineers" with a straight face.

I would argue that we have people who already are software engineers. People who know the machine their code is running on very well, and use this knowledge to fully optimize for that specific machine (embedded, console development).

They take what they are given, and engineer a solution that fits that piece of hardware as well as possible. No bullshit FactoryManagerFactory.

This is a bit of the "fallacy fallacy" issue. Fallacies are mental shortcuts. Fallacies are wrong when you say "X is true because Y" where Y is something like an appeal to authority. But their use does not imply "X is false because Y is a fallacy". The move isn't good because Kasparov said it's good; Kasparov said it's good because it's a good move. Which means that if you asked him, "Why?" he would give you some reasoning, which is the real Y for why X is true. Granted he might appeal to authority by saying "Because my intuition likes it", but that's a good reason to think deeper on the move in order to come up with a grounded argument

> The move isn't good because Kasparov said it's good; Kasparov said it's good because it's a good move. Which means that if you asked him, "Why?" he would give you some reasoning, which is the real Y for why X is true.

Yes, if you have access to Kasparov, you can ask his reasoning and that will help you make a decision. Some moves in chess are only good moves if you get the next five moves exactly right, so they aren't suited to beginners.

But in real life, we don't usually get access to experts. I can look up Kasporovs games but I can't ask him his reasoning. I can look at code that Dennis Ritchie writes and see how he did something, but I don't have access to him to question him on his reasoning. But he's a very good programmer with deep knowledge and a body of work to show for it, so chances are that if he's doing something some way, it's probably not a bad way. On the other hand, if I'm writing something in C--and I'm no expert in C--I aught to give it serious thought if I'm torn between two options and I'm not choosing the one that Dennis Ritchie went with. There's a good chance that I'm not seeing something that he did.

And yes, even the best people make mistakes. But unless you have good reasoning to the contrary, you probably are not going to do better than experts in most decisions.

> Some moves in chess are only good moves if you get the next five moves exactly right, so they aren't suited to beginners.

This is important. Similar issue exists in StarCraft: good builds for pro players aren't necessarily good builds for regular players. If you can't macro behind double pronged harass, don't do it. If you can't watch your ramp to block adepts shading in, don't sit a probe at the top of your ramp

Don't say "I do this because experts do this", instead study it to understand why they do. Otherwise you're going to end up cargo culting

Better to do suboptimal well & with understanding than try do optimal poorly & be confused when it doesn't pay off

I come across this one all the time, but it's usually in the form of "I know you're wrong, because person X made an argument that I don't understand, but this misunderstood summary of that argument says you're wrong".

Happens all the time with the old Donald Knuth chestnut about "premature optimization". Half the time I hear it, it's from people who don't want to optimize at all. http://www.joshbarczak.com/blog/?p=580

That Knuth quote has to be one of the quotes that is most often taken out of context to suit a current narrative.

I think this fallacy is the most widely misunderstood fallacy of all. In an argument between experts, appeal-to-authority is indeed a fallacy. If both people are indeed experts, they should be able to judge something purely on its merits. This is similar to the scientific method where debates are settled by logic and evidence, not seniority.

But if either party in the debate isn't an expert, then appealing to authority is the rational thing to do. If you yourself aren't an expert, then you lack the expertise needed to evaluate any arguments purely based on merit. You'd be far better off relying on expert advice, even if the person you're debating with has suggested something different and is more knowledge than you.

It takes a lot of objectivity and humility to know which fields you are and aren't an expert in. Overestimating your skills may feel good, but will only slow your progress and lead to bad decisions. The vast majority of the time, you're better off following expert advice.

>But if either party in the debate isn't an expert, then appealing to authority is the rational thing to do.

It might be the "rational thing to do" but it's not a logical argument -- which is what makes it a fallacy.

Fallacies can often be the logical thing to do. E.g. an "Ad Hominem Fallacy" might save your ass when the other person puts forward a perfectly logical argument, but to serve their own selfish reasons to your detriment. An "Appeal to hypocrisy" can serve to highlight when the person you debate with wants you to stop doing something (because eg. they say its bad) but keeps for themselves the benefits of keeping doing to (and thus gets the upper hand on you). And so on.

Still, like those things "Appeal to authority" is not a logical proof based on the content under discussion (and axioms, logical steps etc), but a "because X says so" on the status of person X (eg. their expertise).

That's what makes it a fallacy. Whether what that person says is true or not, or even whether it's more likely true than not, is irrelevant.

In either case, it's not a logical proof (in the formal sense) nor a definite proof (the person with the expertise could for example be lying to further their interests, or mislead for some other reason).

We're arguing over semantics here. Most people interpret a "fallacy" as something one should refrain from. Ie, "appeal-to-authority is a fallacy, therefore we should refrain from it during the course of any discussion".

I'm making the point that often times, appealing to authority is most likely to produce the correct answer.

>I'm making the point that often times, appealing to authority is most likely to produce the correct answer.

That's true for most so-called fallacies. Instead they are evolutionary shortcuts to getting the correct answer (at least at evolutionary kind of situations).

This is not true. There is a difference between logical fallacies and judgement based on probability. Circular reasoning, strawman arguments and actual appeals to authority are not the same as taking into account the likelyhood than an expert is correct.

How does committing the strawman fallacy help in producing the correct answer?

The word "fallacy" as used in the OP seems to me to be a shorthand for "belief or opinion not backed by solid observational evidence and/or reasoning". It doesn't seem to apply to the case of constructing an argument in the logical or philosophical sense.

"Appeal to authority" is a fallacy in the case where someone asserts authority where they have none, e.g. taking a scientist's advice on a subject because they're a scientist, even though their specialisation is completely unrelated to the subject at hand.

Deferring to relevant expert opinion is really the only thing you can do for the vast majority of things, since you can only be an expert in so many things. Unfortunately, "appeal to authority" is sometimes interpreted in a way so as to discredit authority in general.

That’s not what it means at all. Of course relying on false authorities is bad.

Like all “fallacies” it’s not that they are useless, it’s just that you have to be aware of the traps. Even the best experts in an area often disagree, so you can’t logically deduce that expert says X means X is true.

You're right that the actual fallacy is using an appeal to authority as proof for a statement - that is wrong.

However, it's perfectly fine to defer to an authority as a working hypothesis when you're not proving something, though - unless that authority does not have relevant expertise.

I think my point is probably better worded by https://yourlogicalfallacyis.com/appeal-to-authority

> nor is it reasonable to disregard the claims of experts who have a demonstrated depth of knowledge unless one has a similar level of understanding and/or access to empirical evidence.

But that does not dismiss that you're entirely right in that

> it is entirely possible that the opinion of a person or institution of authority is wrong; therefore the authority that such a person or institution holds does not have any intrinsic bearing upon whether their claims are true or not.

> > Appeal to authority (argument from authority, argumentum ad verecundiam) – an assertion is deemed true because of the position or authority of the person asserting it.

> You have to be careful with this one. When someone has enough credibility, sometimes you need to question yourself if you disagree.

This is true in all disciplines, and yet the fallacy exists—because there's a difference between "an assertion is deemed true because of the … person asserting it" (emphasis mine; a fallacy) versus "an assertion is given more weight because of the person asserting it" (a perfectly reasonable shortcut, as long as the assertion is still given some vetting).

This is a bad proposition and I'm afraid a wrong one. When you disagree with an expert, the expert should be able to convince you why s/he is right or at least able to articulate the reasoning. It's not a chess game where you are an opponent to the expert and the next move is supposed to be secret. The relation between you and the expert is of collaboration, training and dissemination of knowledge. If an expert fails to articulate the reasoning, I take that as a sign that advice is merely because "it would work because I saw it worked before". That often means the expert failed to analyze and learn from his/her success and is just proposing the status quo. It is often better to question this kind of expert advice and that's how progress has almost always been made throughout human history.

You also have to be careful about who the authority is and how they got to be an authority. Gary Kasparov obviously got to his place via a metric which is hard to argue against; you don't know how a senior dev got promoted to be a senior dev. I know a senior "engineer" but really a dev who's rewritten this one piece of software, first in go, then in c++, then in go, over the course of a year. There are no unit tests and no integration tests. Documentation is spotty. A second senior who has a PhD in engineering and no go experience started work and she immediately found several race conditions and complained that the code was written in a fashion that was provably incompatible with concurrency.

>If you are playing chess and Gary Kasparov comes along and tells you what your next move should be, you can disagree with him, and you might even be right, but given his expertise in chess, his move is most likely going to be better than yours and it's worth considering

If you already had a plan to win the game, but have no idea how to continue Kasparov’s line, then it doesn’t matter how strong his move is, you’re just going to be shooting yourself in the foot by making it.

In technology though, things are different. You can precisely quantify the strength of a chess move, you usually can’t even imprecisely quantify the strength of most advice about technology. In my experience, experts tend to be no better or worse at separating their own personal opinions from more objective truths.

In any case, the appeal to authority shouldn’t be necessary. If an expert is right about something, they should be able to explain to a non-expert why they’re right.

yes an no. chess is a very old, well studied game where one can improve w/ work over time. The rules don’t change and people normally agree on what are good/strong moves vs weak moves.

now, if the size of the board, how pieces move and the goal of the game would change every 3 months if Kasparov would come in and tell you that your next move is wrong it would be a pretty hard sell. Also imagine this Kasparov fellow is also into manufacturing a certain type of piece that he wants you to aggressively use to “develop” your game.

No. Absolutely not. If the person is actually an incredible authority they will have no trouble making a compelling argument or providing strong evidence.

This is never confusing unless you lack objectivity.

> If the person is actually an incredible authority they will have no trouble making a compelling argument or providing strong evidence.

This is false in many contexts.

For example with Chess, a super GM may tell you your move is bad because they have calculated a ton of possible continuations from your move and you end up in a losing endgame. If you can't follow their calculations and/or don't know enough endgame theory, you won't be able to tell if what they are saying is correct.

Similarly with say cryptography. If you don't understand enough about the field and state of the art. You may not be able to understand how some encryption might be able to be broken.

That is called bias. Again, this matter is simple when objective. People often give into appeals of authority and other fallacies because they lack the critical nature to make original decisions and instead defer to name recognition.

If this were true, then developers would have no problem selling things like unit testing or reduction of technical debt to business folk. Sometimes it's hard to explain something to someone who isn't on your level with a specific skill.

If you lack the ability to craft a persuasive argument then provide strong evidence. To an objective outsider giving into logical fallacies would make your argument weaker, not stronger.

It's just a logical fallacy because it can't be used as a concrete building block to advance a logical argument forward. Doesn't mean it's not useful, it just means that you're no longer logically air tight.

it depends greatly how Kasparov tells you what to do. if it's "do this because I'm kasparov", or "do this, because your next move could be this and it gives you this opportunity". same with software. someone telling to do this because I'm senior engineer is appeal to authority.

If I were playing chess in the park and Gary Kasporov walks by, sees a move that I'm about to make and says "that's a mistake, I'd recommend against that", then I'd be an idiot not to strongly reconsider my move, whether he explains himself or not. Sure, he makes mistakes, but Kasparov is an unimaginably better at chess than me and the chances that I make a mistake and don't see something that he does are much much greater. With chess though, it's hard to overate yourself because you can see exactly where you stand. With software engineering though, it's very easy to overestimate your abilities.