News for hackers. Stuff that matters.
Re:Don’t Do This in Production by ChuckMcM on Sunday August 19, 2018 @01:58PM (#17795390)
I was going to call out this quote : If you read this far hoping for an answer, then I’m sorry: I don’t have a simple one. This is a difficult problem to solve.

And yet there is an answer, it is hire an experienced engineer. And the author glosses over the problem by not realizing they "solved" the problem by hiring him.

It is the bad calculus where one "experienced" engineer is going to cost you maybe $150K - $200K and yet you could get anywhere from 3 to 10 junior engineers for that depending on how far out you go.

It would be awesome of there were some sort of certification for "experience" sort of equivalent to the guild master status, it isn't an education thing it is a "oh yeah I've seen that before, lets not do that here's why" kind of thing.

Re:Don’t Do This in Production by crispyambulance on Sunday August 19, 2018 @02:45PM (#17795604)
Projects fail for all kinds of reasons. Usually many reasons at the same time.

While I am sure that _some_ fail because of the abject inexperience of the staff running them, this is hardly ever the complete story. Moreover, projects do fail even with a fully experienced, fully vetted staff running the show.

The article just seems like a consultant marketing his acumen. There's nothing wrong with that. But keep in mind, he dropped in over a weekend and spent a few hours detailing problems that needed fixing and the staff subsequently made corrections and released on time.

It could easily go the other way too. The "inexperienced" team could have instead brought in another seasoned consultant who, after a few hours, got the wrong idea and led them down the wrong path precipitating an even larger slippage of ship date. THAT HAPPENS TOO and I suspect it is even more common.

Re:Don’t Do This in Production by HeavyStorm on Monday August 20, 2018 @04:23PM (#17803606)
I don't really agree. Sure, many projects fail for reasons other than developer experience but in many years of many more projects, developer skill was the most frequent reason.

Of course, that's probably more true of my environment (third world country, enterprise software) than others.

Re:Don’t Do This in Production by anyzen on Monday August 20, 2018 @06:57AM (#17798961)
> Moreover, projects do fail even with a fully experienced, fully vetted staff running the show.

Sure - but in that case they usually don't fail because of technical decisions (which is what OP was talking about). There are however many other ways a project can fail (bad management mostly).

Re:Don’t Do This in Production by yen223 on Sunday August 19, 2018 @02:08PM (#17795443)
If you aren't technical yourself, how do you separate an experienced engineer from one who, well, isn't? This was the fundamental problem that the company failed to solve, and frankly I don't think there's any easy solution around it.
Re:Don’t Do This in Production by Spooky23 on Sunday August 19, 2018 @11:35PM (#17797554)
Use referrals.

Ultimately, if you cannot identify critical staff and don’t know anyone who can, you’re screwed anyway.

Re:Don’t Do This in Production by wpietri on Tuesday August 21, 2018 @11:18AM (#17810347)
Exactly. Somebody running a company can't be an expert in everything. But they can be a good judge of people and have a wide enough network that they can get recommendations from people they trust.

A few of those trusted technologists will make a fine hiring/oversight committee. You won't need a ton of their time, so you can pay very generously for the insurance they provide.

Re:Don’t Do This in Production by memset on Sunday August 19, 2018 @02:16PM (#17795476)
Agree with this. Same applies to any expert: how do you identify an experienced marketer? Or accountant?
Re:Don’t Do This in Production by jedharris on Sunday August 19, 2018 @07:17PM (#17796763)
To hire someone with expertise to solve a problem in a domain where you aren't an expert: You find people who claim to understand the domain, and then discuss the problem with them in depth. Your goal is not to evaluate them in general but to understand how they analyze the problem and how they'd deal with it.

You may have to hire them as consultants. You have to dedicate a day or more to discussion with each of them -- maybe considerably more if your problem is complex. You have to reveal information about your plans and/or the condition of your systems that may make you anxious or embarrassed.

After you've gone over the problem separately with five or ten putative experts, you will have a pretty good idea which of them understand it (if any), and you will also know how to ask much better questions going forward.

If necessary repeat this process, using your better understanding to find better experts and gain a better map of the territory.

Once you can rank expertise in your domain reliably, you might hire one (or more) of the known experts, but the best ones will be hardest to hire. Or you may use your new meta-expertise in the domain to look for more junior hires who have an unusually good grasp of your issues. Also, you may be able to get the best of your experts to act as advisors.

This takes a lot of time and may cost some money depending on the domain. But not doing it will take more time and cost more money in the medium to long term (and even in the short term sometimes).

Re:Don’t Do This in Production by hyperpallium on Sunday August 19, 2018 @09:27PM (#17797183)
AKA become expert.

I wonder if the people who become fantastically successful simply have the ability to evaluate expertise, without having any of it. i.e. by "reading" people, in some sense.

This relies on the "expert"'s own evaluation of their expertise, which mightn't be accurate, because of Dunning–Kruger, or conversely, self-doubt. So, one would need to "read" those too.

Re:Don’t Do This in Production by jedharris on Tuesday August 21, 2018 @10:06AM (#17809645)
You definitely can't trust people's self-evaluation either way. Lots of really skilled people are self-critical. "Reading people" of course helps but it is nowhere near enough.

But you don't have to become an expert. If I hire a plumber or carpenter, I cannot do what they can. But with enough work I can understand which plumbers or carpenters will do a good job.

Re:Don’t Do This in Production by yen223 on Sunday August 19, 2018 @03:06PM (#17795683)
... or climate change scientist? Or economist? Or president?

The problem of separating experts vs non-experts might be the single biggest problem society faces right now.

Re:Don’t Do This in Production by gameswithgo on Sunday August 19, 2018 @03:16PM (#17795720)
assuming you have a generally good education and brain to start with, a lot of hard work can usually accomplish it.

Think Elon Musk learning about rockets, back before he lost his mind.

I did it once for climate change, took a couple months, some math. Got there though.

Re:Don’t Do This in Production by chmhsm on Sunday August 19, 2018 @06:31PM (#17796543)
Before he lost his mind? You mean the recent interview or before that?
Re:Don’t Do This in Production by hyperpallium on Sunday August 19, 2018 @09:23PM (#17797166)
This is the problem that credentials supposedly solve - degrees, certifications, professional body membership etc.

You don't realize the difficulty of the problem until you must hire someone outside your own areas of expertise.

Re:Don’t Do This in Production by mathattack on Sunday August 19, 2018 @07:30PM (#17796805)
Depends on the magnitude of the problem to be solved.

Small problem: Ask friends for someone they have worked with on a similar problem.

Large problem (building an org): Ask 5 VCs/CEOs/VIPs for intros to the best person they know in that role, under the explicit direction of “learning what good looks like.” Interview all 5 with open ended questions. Ask each for one for one more contact. Eventually you will learn to pattern match, and one of the folks you like will volunteer themselves or a friend, and you will know how to interview.

Re:Don’t Do This in Production by kuwze on Monday August 20, 2018 @12:01PM (#17801026)
I remember somebody on this forum saying the best way to judge someone's technical prowess is to give a simple explanation of a complicated part of their field. If you can't summarize your knowledge you don't know what you are talking about.
Re:Don’t Do This in Production by wpietri on Tuesday August 21, 2018 @11:19AM (#17810363)
Unfortunately, a non-expert can't tell the difference between a good simple explanation and glib, confident ignorance. So I think that's a good test for expertise, but it's not one that a non-expert can fully judge.
Re:Don’t Do This in Production by acjohnson55 on Sunday August 19, 2018 @09:37PM (#17797224)
References and portfolio, at the end of the day.
Re:Don’t Do This in Production by robocat on Monday August 20, 2018 @07:34AM (#17799132)
The orange juice test is one person's answer:

https://www.intercom.com/blog/the-orange-juice-test/

Re:Don’t Do This in Production by scarface74 on Sunday August 19, 2018 @08:07PM (#17796922)
Hire a recruiting company. There are recruiting companies that have a staff of full time consultants that will bring in a whole development team and manage the product.

At some point, you want to bring that knowledge in house, the recruiting company will then help you hire people or let you convert some of thier contractors (not thier full time people) to fulltime.

Re:Don’t Do This in Production by dlhavema on Sunday August 19, 2018 @08:21PM (#17796973)
We have a lot of recruiters sending us "senior full stack engineers" that dont knoe the language they are most confident in and end up being dev-ops engineers who want to get into coding. A lot of recruiting firms dont know how to to get candidates either
Re:Don’t Do This in Production by scarface74 on Sunday August 19, 2018 @08:30PM (#17797004)
I am not referring to recruiting companies that just send developers to a company. I am referring to recruiting companies that have full time staffed developers and project managers that manage the project for you in the beginning and then slowly help you build up your own staff.

Some will have a combination of onsite developers, “rural source” developers who are in a cheaper part of the US, contractors and outsourced developers but they manage the entire project.

I was offered such a position - the pay is above market but it required too much travel for me now.

Re:Don’t Do This in Production by marcinzm on Sunday August 19, 2018 @09:01PM (#17797101)
Now how do you separate out a recruitment company who has solid experience engineers on board from one that doesn't but says they do? I've seen many of the latter and many founders who thought they were getting the former in the deal.
Re:Don’t Do This in Production by scarface74 on Sunday August 19, 2018 @09:39PM (#17797230)
If you are starting a company, I would hope that you at least know what your requirements are and you could sit down with the project managers to hash out acceptance criteria and deliverables.
Re:Don’t Do This in Production by eropple on Monday August 20, 2018 @01:05AM (#17797835)
And then you find that the acceptance criteria aren't met, the deliverables don't work, and your ship date is hosed. What then?
Re:Don’t Do This in Production by scarface74 on Monday August 20, 2018 @05:01AM (#17798535)
How is that a different risk than every other project that ever existed - even those led by internal teams?
Re:Don’t Do This in Production by marcinzm on Monday August 20, 2018 @08:35AM (#17799391)
Preferably for an internal project you have a trusted delegation chain that would catch the issue early on.
Re:Don’t Do This in Production by scarface74 on Monday August 20, 2018 @09:01AM (#17799540)
Two scenarios:

Scenario #1 - you are a founder with a great idea, you hire a good software dev lead who then fills out the department. You work with the Dev lead to hash out your plan They organize it and after certain checkpoints, the dev team demos progress. As the founder, you are at the top of the chain making sure that there is a product market fit, and ensuring things are going as planned from a high level.

Scenario #2. You contract with a consulting company. Your project is led by a dev lead/project manager paid by the consulting company and they fill out the team using thier own staff.

They organize the team and after certain checkpoints, the dev team demos progress. As the founder, you are at the top of the chain making sure that there is a product market fit, and ensuring things are going as planned from a high level.

There is no difference in the delegation chain or the risks involved.

The main difference is that you are writing one check to the consulting company and they are paying the contributors and in the other case, you are paying employees individually.

The usual retort is that employees have more loyalty and ownership in the company. No they don’t. The employees at your company, if they are smart, have one eye toward thier next job just like the consultanting company has one eye toward thier next contract.

Re:Don’t Do This in Production by HeavyStorm on Monday August 20, 2018 @04:32PM (#17803685)
The goal of the recruiting company is to get candidates hired. That may or may not conflict with the ultimate goal of the client doing the hiring, which is to get the job done.

I have this problem internally at my company. Recruiting have crazy goals (hire 30 developers per month) so they push us to accept whatever they can get.

Sure, we can keep refusing the candidates but at some point you have to accept someone to get the job done. Even if you need to double check everything they do.

The reasons for that are too complex for a small post but believe me, there are many.

Re:Don’t Do This in Production by scarface74 on Monday August 20, 2018 @06:01PM (#17804405)
The post you are replying to said:

Hire a recruiting company. There are recruiting companies that have a staff of full time consultants that will bring in a whole development team and manage the product.

You are replying to a scenario that is just the opposite of my suggestion. The client company is not hiring developers at all. They are contracting out with the consulting company to manage the entire project. The founder is working with a project manager who is employed with the consulting company to develop a product. The recruiting company is doing all of the project management, development, QA, etc. The founder is hopefully coming up with acceptance criteria for the product since it was his idea.

After the project gets off the ground, then the recruiting/consulting company works with the client to develop in house expertise starting with finding a dev manager who hopefully understands the business and the technical and the dev manager works with recruiters to staff up.

You’re not hiring anyone at first. You’re basically outsourcing the entire development project to an outside agency. If you are a non technical founder, this may be the best way to go.

There is a local recruiting/consulting company that has been trying to get me to come work for them directly full time in the role of team lead, I just can’t do the travel right now.

Re:Don’t Do This in Production by contravariant on Sunday August 19, 2018 @05:33PM (#17796304)
I guess the assumption is that anyone or at least an inexperienced engineer would be able to point out an experienced engineer, while only experienced engineers are any good at picking less experienced engineers.
Re:Don’t Do This in Production by tormeh on Monday August 20, 2018 @07:06AM (#17799003)
Deloitte has pretty good technical interviews. I'm sure they can provide you interviews as a service. Probably not going to be cheap, though.
Re:Don’t Do This in Production by megaremote on Sunday August 19, 2018 @07:13PM (#17796747)
Surely there is a big clue in how long they have been working for, how many projects they have done, what their roles were.
Re:Don’t Do This in Production by itronitron on Sunday August 19, 2018 @08:26PM (#17796991)
Based on my experience interviewing over the past several years, the companies that have a 'production' system seem to only value experience with their tech stack.
Re:Don’t Do This in Production by pm90 on Sunday August 19, 2018 @09:20PM (#17797154)
That’s because they have set something up and ironed out all the bugs so it will take some convincing to move them to a different stack.

There’s no easy way around it. In fact I’ve seen some teams too eager to rip up existing production workflows every time some new technology becomes popular and that’s not productive either.

Re:Don’t Do This in Production by itronitron on Sunday August 19, 2018 @11:29PM (#17797538)
I'm not saying that they should change their stack, I'm saying that they should consider that someone having extensive experience with related technologies probably knows the domain well and can come up to speed quickly on the team's shit.
Re:Don’t Do This in Production by gameswithgo on Sunday August 19, 2018 @03:12PM (#17795708)
That is sort of a tautology solution. Hire someone who can solve the problem and you won't have the problem.

But how do you know when you have done that? Experience alone doesn't necessarily mean they can solve your problem.

Re:Don’t Do This in Production by freedomben on Sunday August 19, 2018 @10:26PM (#17797345)
Right, I think that's being missed in this thread.

Experience alone doesn't necessarily mean they can solve the problem and ship/write good code. I've worked with plenty of "senior" engineers that have 20+ years experience yet have never dealt with any serious issues. These usually work at companies where a small team of "ninjas" handles those sorts of things, and the run of the mill Dilbert coder shows up and does his thing. This is not a knock on those people. They are important in their own way, truly. But I would not want them to be the responsible party for steering the ship in a new app.

Re:Don’t Do This in Production by marcinzm on Sunday August 19, 2018 @09:11PM (#17797128)
Define experienced engineer. I've seen many on-paper experienced engineers who would be worse than a junior engineer on a project.

>oh yeah I've seen that before, lets not do that here's why

Which can actually be a pretty bad way of going about it imho if you only use your own direct experiences for that. Just like science, basing your decisions on a limited number of anecdotes is problematic because small biased subsets are not representative of the whole.

Re:Don’t Do This in Production by pm90 on Sunday August 19, 2018 @09:16PM (#17797138)
You have to make a decision based on something. Experience isn’t a bad heuristic if evaluated correctly: this is why in most serious technical interviews today, you will often get asked about your past experiences.
Re:Don’t Do This in Production by marcinzm on Monday August 20, 2018 @08:33AM (#17799381)
>You have to make a decision based on something.

And like science you can base it on the collective experience versus your personal experience. Preferably from rigorous studies but we mostly lack those in CS. So hopefully you're reading books, blog posts, studies, conference presentations, discussing with colleagues, etc.

>this is why in most serious technical interviews today, you will often get asked about your past experiences.

Interestingly the top companies seem to be focusing a lot less on experience and a lot more on problem solving/system design/etc.

Re:Don’t Do This in Production by pm90 on Monday August 20, 2018 @04:22PM (#17803589)
You bring up a good point. I do think interviewing for developers is not a field that has been studied using the scientific method, which is why we seem to have so many rules of thumbs, cargo cult traditions etc. rather than experiment-backed results. I think a few companies do study this as a science (e.g. Google) but most others are either unwilling or don't have enough resources (or some mix of the 2) to do the same.

> Interestingly the top companies seem to be focusing a lot less on experience and a lot more on problem solving/system design/etc.

As long as we're making conjectures: this is likely because a lot of them have tools and processes that are very different from what you see "in the wild". e.g. working at Google means you can leverage all their deployment infrastructure, something you likely didn't have access to outside (this is somewhat changing with k8s and cloud services).

Re:Don’t Do This in Production by majormajor on Sunday August 19, 2018 @04:49PM (#17796102)
Very few developers even trust themselves to vet the sorts of things this post is talking about. So our current level of in-industry interview training is going to make that a crapshoot.

The general reason given for non-representative algorithmic whiteboard coding problems is "you can't trust anybody's experience." But those questions are themselves not resulting in production-ready code, so you're flying blind on "can figure out a quick way to solve a problem" vs "knows what to do to turn a quick solution into something that will last."

And then there's a time-based calculation you have to make: hire fast and hope they'll learn on the fly, or slow down and delay until you find people you hope won't have to. You're hoping either way, after all. :)

Re:Don’t Do This in Production by jcadam on Monday August 20, 2018 @06:59AM (#17798972)
> hire an experienced engineer.

But "experienced" and "cultural fit" are mutually exclusive for most startups.

Re:Don’t Do This in Production by MrEfficiency on Monday August 20, 2018 @07:12AM (#17799030)
Experienced will tell you the refuse to release without X Y and Z because if B happens your server will crash.

Junior wont even know the problem and hack it together.

But hey, thats what reddit /r/askprogramming is for... right?