Hmmm. Is there any way to distinguish between these two categories? Because I agree, if someone is delegating all their work to an LLM or similar tool, cut out the middleman. Same as if someone just copy/pasted from Stackoverflow 5 years ago.
I think it is also important to think about incentives. What incentive does the newer developer have to understand the LLM output? There's the long term incentive, but is there a short term one?
Unfortunately, the use of LLMs has brought about a lot of mistrust in the workplace. Earlier you’d simply assume that a junior making mistakes is simply part of being a junior and can be coached; whereas nowadays said junior may not be willing to take your advice as they see it as sermonizing when an “easy” process to get “acceptable” results exists.
> Earlier you’d simply assume that a junior making mistakes is simply part of being a junior and can be coached; whereas nowadays said junior may not be willing to take your advice
On a positive note: I can remember mentoring some young people and watching them comb through blogs to learn about programming. I am so old that my shelf is/was full of O'Reilly books. By the time I was mentoring them, few people under 25 were reading O'Reilly books. It opened my eyes that how people changes more than what people learn. Example: Someone is trying to learning about access control modifiers for classes/methods in a programming language. Old days: Get the O'Reilly book for that programming language. Lookup access modifiers in the index. 10 year ago: Google for a blog with an intro to the programming language. There will be a tip about what access modifiers can do. Today: Ask ChatGPT. In my (somewhat contrived) example, the how is changing, but not the what.
The answer to this (throughout the ages) should be the same: read the authoritative source of information. The official API docs, the official language specification, the man page, the textbook, the published paper, and so on.
Maybe I am showing my age, but one of the more frustrating parts of being a senior mentoring a junior is when they come with a question or problem, and when I ask: “what does the official documentation say?” I get a blank stare. We have moved from consulting the primary source of information to using secondary sources (like O’Reilly, blogs and tutorials), now to tertiary sources like LLMs.
If I have a problem with a USB datastream, the last place I'm going to look is the official USB spec. I'll be buried for weeks. The information may be there, but it will take me so long to find it that it might as well not.
The first place to look is a high quality source that has digested the official spec and regurgitated it into something more comprehensible.
[shudder] the amount of life that I've wasted discussing the meaning of some random phrase in IEC-62304 is time I will never get back!
I think this is undoubtedly true from my observations. Recently, I got together over drinks with a group of young devs (most around half my age) from another country I was visiting.
One of the things I said, very casually, was, "Hey, don't sleep on good programming books. O'Reilly. Wiley. Addison-Wesley. MIT Press. No Starch Press. Stuff like that."
Well, you should've seen the looks on their faces. It was obvious that advice went over very poorly. "Ha, read books? That's hard. We'd rather just watch a YouTube video about how to make a JS dropdown menu."
So yeah, I get that "showing my age" remark. Used to be the discipline in this industry is that you shouldn't ask a question of a senior before you'd read the documentation. If you had read the documentation, man pages, googled, etc., and still couldn't come up with an answer, then you could legitimately ask for a senior mentor's time. Otherwise, the answer from the greybeards would have been "Get out of my face, kid. Go RTFM."
That system that used to exist is totally broken now. When reading and understanding technical documentation is viewed as "old school", then you know we have a big problem.
To this day, I normally search on Google first, then try an LLM... the last place that I look is the official docs if my question is about POSIX or Win32. They are just too complex and require too much base knowledge about the ecosystem. As an interesting aside, when I first learned Python, Java, and C#, I thought their docs were as approachable as Qt. It was very easy to get started with "console" programming and later expand to GUI programming.
The tangent to that is it is also changing with the how much one internalizes about the problem domain and is able to apply that knowledge later. Hard fought knowledge from the old days is something that shapes how I design systems today.
However, the tendency of people who reach for ChatGPT today to solve a problem results in them making the same mistakes again the next time since the information is so easy to access. It also results in things that are larger are more difficult... the "how do you architect this larger system" is something you learn by building the smaller systems and learning about them so that their advantages and disadvantages and how and such becomes an inherent part of how you conceive of the system as a whole. ... Being able to have ChatGPT do it means people often don't think about the larger problem or how it fits together.
I believe that is harder for a junior who is using ChatGPT to advance to being a mid level or senior developer than it is for a junior from the old days because of the lack of retention of the knowledge of the problems and solutions.
I have a friend who is a dev, a very senior one at that, who spins up 4 Claudes at once and does the whole enterprises work. Hes a "Senior AI Director" with nobody beneath him, not a single direct report, and NO knowledge of AI or ML, to my chagrin.
So now I'm whining too...
Once you’re a senior you can exercise judgement on when/how to use LLMs.
When you’re a junior you haven’t developed that judgement yet. That judgement comes from consulting documentation, actually writing code by hand, seeing how you can write a small program just fine, but noticing that some things need to change when the code gets a lot bigger.
A junior without judgement isn’t very valuable unless he/she is working hard to develop that judgement. Passing assignments through to the LLM does not build judgement, so it’s not a winning strategy.
Hot take: This reads like a person who was difficult to work with.
Senior people have responsibility, therefore in a business situation they have authority. Junior people who think they know it all don't like this. If there's a disagreement between a senior person and a junior person about something, they should, of course, listen to each other respectfully. If that's not happening, then one of them is not being a good employee. But if they are, then the supervisor makes the final call.
I saw a situation like this many years ago. The newly hired midlevel engineer thought he was smarter than the supervisor. Kept on arguing about code style, system design etc. He was fired after 6 months.
But I was friendly with him, so we kept in touch. He ended up working at MSFT for 3 times the salary.
For software, I can imagine a process where junior developers create a PR and then run through it with another engineer side by side. The short-term incentive would be that they can do it, else they'd get exposed.
Yes, it should be obvious. At least at the current state of LLMs.
> There's the long term incentive, but is there a short term one?
The short term incentive is keeping their job.
I don't mind if AI is used as a tool, but the output needs to be vetted.
That is at least for the people who don't understand what they're doing, the LLM tends to come out with something I can at least turn into something useful.
It might be reversed though for people who know what they're doing. IF they know what they're doing they might theoretically be able to put together some stackoverflow results that make sense, and build something up from that better than what gets generated from LLM (I am not asserting this would happen, and thinking it might be the case)
However I don't know as I've never known anyone who knew what they were doing who also just copy/pasted some stackoverflow or delegated to LLM significantly.
You might be specifically talking about people who copy/paste without understanding, but I think it's still OK-ish to do that, since you can't make an entire [whatever you're coding up] by copy/pasting snippets from SO like you're cutting words out of a magazine for a ransom note. There's still thought involved, so it's more like training wheels that you eventually outgrow as you get more understanding.
It at least forces you to tinker with whatever you copied over.
But I suppose my question is rhetorical. We're laying off hundreds of thousands of engineers and maming existing ones do the work of 3-4 engineers. Not much time to help the juniors.
I've learnt that saying this exact phrase does wonders when it comes to advancing your career. I used to argue against stupid ideas but not only did I achieve nothing, but I was also labelled uncooperative and technically incompetent. Then I became a "yes-man" and all problems went away.
I have seen responses to PRs that appear to be a copy and paste of my feedback into it and a copy and paste of the response and fixes into the PR.
It may be the that the developer is incorporating the mannerisms of Claude into their own speech... that would be something to delve into (that was intentional). However, more often than not in today's world of software development such responses are more likely to indicate a copy and paste of LLM generated content.
This is nothing new. People rarely have independent thoughts, usually they just parrot whatever they've been told to parrot. LLMs created common world-wide standard on this parroting, which makes the phenomenon more evident, but it doesn't change the fact that it existed before LLMs.
Have you ever had a conversation with an intelligent person and thought "wow that's refreshing"? Yeah. There's a reason why it feels so good.
If you're still asking trvia, yes. Maybe it's time to shift from the old filter and update the process?
If you can see in the job that a 30 minute PR is the problem, then maybe replace that 3rd leetcode round with 30 minutes of pair programming. Hard to chatGPT in real time without sounding suspicion.
But is the false negative for a nervous pair programmer worse than a false positive for a leetcode question? Ideally a good interviewer would be able to separate the anxiety from the actual thinking and see that this person can actually think, but that's another undervalued skill among industry.
Given how much these orgs pay, both direct to head hunters and indirect in interview time, might as well probationally hire the whoever passes the initial sniff test.
That also lets you evaluate longer term habits like punctuality, irritability, and overall not-being-a-jerkness.
Onboarding. Even with good employees, it can take a few months to get the flow of the organization, understanding the code base, and understanding the domain. Maybe a bit of technology shift too. Firing a person who doesn't appear to be preforming in the first week or two or three would be churning through that too fast.
Provisional hiring with "maybe we'll hire you after you move here and work for us for a month" is a non-starter for many candidates.
At my current job and the job previous it took two or three weeks to get things fully set up. Be it equipment, provisioning permissions, accounts, training (the retail company I worked at from '10 to '14 - they sent every new hire out to a retail store to learn about how the store runs (to get a better idea of how to build things for them and support their processes).
... and not every company pays Big Tech compensation. Sometimes it's "this is the only person who didn't say «I've got an offer with someone else that pays 50% more»". Sometimes a warm body that you can delegate QA testing and pager duty to (rather than software development tasks) is still a warm body.
Most (middle) management is an exercise in ass-covering, rather than creating healthy teams. They get easily scared when "Jira isn't green", and look someone else to blame for not doing the managing part correctly
"Bad" is vague, subjective moralist judgement. It's also easily manipulated and distorted to justify firing competent people who did no wrong.
> It’s pretty obvious when someone starts actually working if they’re going to a net positive. On the order of weeks, not months.
I feel your opinion is rather simplistic and ungrounded. Only the most egregious cases are rendered apparent in a few weeks worth of work. In software engineering positions, you don't have the chance to let your talents shine through in the span of a few weeks. The cases where incompetence is rendered obvious in the span of a few weeks actually spells gross failures in the whole hiring process, which failed to verify that the candidate failed to even meet the hiring bar.
> (...) might as well probationally hire the whoever passes the initial sniff test.
This is a colossal mistake, and one which disrupts a company's operations and the candidates' lives. Moreover, it has a chilling effect on the whole workforce because no one wants to work for a company ran by sociopaths that toy with people's lives and livelihood as if it was nothing.
If you have that kind of office politics going on, that's the issue to be solved.
>toy with people's lives and livelihood as if it was nothing.
If the employee lies about their skills, it is on them.
Real junior hiring used to mean taking someone raw, pairing them heavily for six months, and turning them into a solid mid. Now the default is “we’ll only hire someone who needs zero ramp-up” and then wonder why the market feels empty.
You’re asking these rhetorical questions as if we haven’t had centuries of precedent here, both bad and good. How does the AMA balance between neurosurgeons and optometrists? Bar associations between corporate litigators and family estate lawyers? Professional engineering associations between civil engineers and chemical engineers?
One takes the FE exam ( https://ncees.org/exams/fe-exam/ ). You will note at the bottom of the page "FE Chemical" and "FE Civil" which are two different exams.
Then you have an apprenticeship for four years as an Engineer in Training (EIT).
Following, that, you take the PE exam. https://ncees.org/exams/pe-exam/ You will note that the PE exams are even more specialized to the field.
Depending on the state you are licensed in (states tend to have reciprocal licensing - but not necessarily and not necessarily for all fields). For example, if you were licensed in Washington, you would need to pass another exam specific to California to work for a California firm.
Furthermore, there is the continuing education requirements (that are different for each state). https://www.pdhengineer.com/pe-continuing-education-requirem...
You have to take 30 hours of certified study in your field across every two years. This isn't a lot, but people tend to fuss about "why do CS people keep being expected to learn on our own?" ... Well, if we were Professional Engineers it wouldn't just be an expectation - it would be a requirement to maintain the license. You will again note the domain of the professional development is different - so civil and mechanical engineers aren't necessarily taking the same types of classes.
These requirements are set by the state licensure and part of legislative processes.
Furthermore, licensure requires state level legislation and makes it harder for employees (especially the EIT) to change jobs or move to other states for work there.
Licensure, the way that people often point to it as a way to solve the credentials problem vs interviews, isn't going to solve the problems that people think it would.
Furthermore, it is only something if there is a reason to do it. If there isn't a reason to have a licensed engineer signing off on designs and code there isn't a reason for a company to hire such.
Why should a company pay more for someone with a license to design their website when they could hire someone more cheaply who doesn't have a license? What penalties would a company have for having a secretary do some vbscripting in excel or a manager use Access rather than hiring a licensed developer?
But are you suggesting we have separate licenses for every different type of developer? We have new types coming up every few years.
The whole idea of guilds for developers is just stupid and impractical. It could never work on any long term or large scale basis.
> But are you suggesting we have separate licenses for every different type of developer? We have new types coming up every few years.
I didn’t suggest that at all and I honestly can’t figure out how you came to that interpretation unless you are hallucinating.
> The whole idea of guilds for developers is just stupid and impractical. It could never work on any long term or large scale basis.
What a convincing argument! You should get a cabinet post.
And yet welcome to leetcode grind.
If you need to fizzbuzz me, fine. But why am I still making word search solver project in my free time as if I'm applying for a college internship?
And I’m being an accelerationist hoping the whole thing collapses under its own ridiculousness.
Recruitment is broken even more than before chatgpt.
Hiring well is hard, specially if compensation isn't competitive enough to attract talented individuals who have a choice. It's also hard to change institutional hiring practices. People don't get fired by buying IBM, and they also don't get fired if they follow the same hiring practices in place in 2016.
> What are all those rounds for if we're getting engineers who aren't as valued for the team's needs at the end of the pipeline?
Software development is a multidiscinary field. It involves multiple non-overlapping skill sets, bot hard skills and soft skills. Also, you need multiple people vetting a candidate to eliminate corruption and help weed out candidates who outright clash with company culture. You need to understand that hiring someone is a disruptive activity, that impacts not only what skill sets are available in your organization but also how the current team dynamics. If you read around, you'll stumble upon stories of people who switch roles in reaction to new arrivals. It's important to get this sort of stuff right.
Well I'm still waiting. Your second paragraph seems to contradict the first. Which perfectly encapsulates the issue with hiring. Too afraid to try new things, so instead add beuracracy to leases accountability.
I think you haven't spend much time thinking about the issue. Changing hiring practices does not mean they are improve. It only means they changed. You are still faced with the task of hiring adequate talent, but if you change processes them now you don't have baselines and past experiences to guide you. You keep those baselines if you keep your hiring practices then you stick with something that is proven to work albeit with debatable optimality, and mitigate risks because your experience with the process helps you be aware of some red flags. The worst case scenario is that you repeat old errors, but those will be systematic errors which are downplayed by the fact that your whole organization is proof that your hiring practices are effective.
No, but I'd like to at least see conversation on how to improve the process. We aren't even at that point. We're just barely past acknowledging that it's even an issue.
>but if you change processes them now you don't have baselines and past experiences to guide you.
I argue we're already at this point. The reason we got past the above point of "acknowledging problem" (a decade too late, arguably) is that the baselines are failing to new technology, which is increasing false positives.
You have a point, but why does tech pick this point to finally decide not to "move fast and break things"? Not when it comes to law and ethics, but for aquiring new talent (which meanwhile is already disrupting heir teams with this AI slop?)
>those will be systematic errors which are downplayed by the fact that your whole organization is proof that your hiring practices are effective.
okay, so back to step zero then. Do we have a hiring problem? The thesis of this article says yes.
"it worked before" seems to be the antipattern the tech industry tried to fight back against for decades.
The current hiring practices are a result of acknowledging what they did before didn't work. The current ones work well enough that people don't wanna change it, the only ones who wanna change it are engineers not the companies.
Okay cool. Back to step 0. Working as planned, ignore the dumpster fire in the back. Hope they reap what they sow.
This goes beyond destructive thinking. Again, I hope the companies reap what they sow.
If you're not going to even acknowledge the issue in the article, there's no point in discussing the issue in a forum. Good day.
This is not limited to junior devs. I had the displeasure of working with a guy who was hired as a senior dev who heavily delegated any work they did. He failed to even do the faintest review of what the coding agent and of course did zero testing. At one time these stunts resulted in a major incident where one of these glorious PRs pushed code that completely inverted a key business rule and resulted in paying customers being denied access to a paid product.
Sometimes people are slackers with little to no ownership or pride in their craftsmanship, and just stumbled upon a career path they are not very good at. They start at juniors but they can idle long enough to waddle their way to senior positions. This is not a LLM problem, or caused by it.
And then in the next PR, you have to request the exact same changes
This is especially annoying when you get back a response in a PR "Yes, you're right. I have pushed the fixes you suggested."
Part of the challenge (and I don't have an answer either) is there are some juniors who use AI to assist... and some who use it to delegate all of their work to.
It is especially frustrating that the second group doesn't become much more than a proxy for an LLM.
New juniors can progress in software engineering - but they have to take the road of disciplined use of AI and make sure that they're learning the material rather than delegating all their work to it... and that delegating work is very tempting... especially if that's what they did in college.