I concur that asking devs how they use AI is a great idea.

Recently, I shared a code base with a junior dev and she was surprised with the speed and sophistication of the code. The LLM did 80+% of the "coding".

What was telling was as she was grokking the code (for helping the ~20%), she was surprised at the quality of the code - her use of the LLM did not yield code of similar quality.

I find that the more domain awareness one brings to the table, the better the output is. Basically the clearer one's vision of the end-state, the better the output.

One other positive side-effect of using "LLMs as a junior-dev" for me has been that my ambitions are greater. I want it all - better code, more sophisticated capabilities even for relatively not-important projects, documentation, tests, debug-ability. And once the basic structure is in place, many a time it is trivial to get the rest.

It's never 100%, but even with 80+%, I am faster than ever before, deliver better quality code, and can switch domains multiple times a week and never feel drained.

Sharing best AI hacks within a team will have the same effect as code-reviews do in ensuring consistency. Perhaps an "LLM chat review", especially when something particularly novel was accomplished!

Using cloud-based AI is a no-go where I work, for IP and contractual reasons. And on-premises AI is not as capable and more difficult to integrate.

Have you tried the latest open weight models? They're SO MUCH better today than they were even six months ago.

If I was in an environment that didn't allow hosted API models I'd absolutely be looking into the various Llama 3 models or Qwen2.5-Coder-32B.

What do you use so that you can throw in a set of documents and/or a nontrivial code base into an LLM workspace and ask questions about it etc.? What the cloud-based services provide goes way beyond a simple chat interface or mere code completion (as you know, of course).

I use my https://github.com/simonw/files-to-prompt tool like this:

  files-to-prompt . -e py -e md -c | pbcopy

Now I have all the Python and Markdown files from the current project on my clipboard, in Claude's recommended XML-like format (which I find works well with other models too).

Then I paste that into the Claude web interface or Google's AI Studio if it's too long for Claude and ask questions there.

Sometimes I'll pipe it straight into my own LLM CLI tool and ask questions that way:

  files-to-prompt . -e py -e md -c | \
    llm -m gemini-2.0-flash-exp 'which files handle JWT verification?'

I can later start a chat session on top of the accumulated context like this:

  llm chat -c

(The -c means "continue most recent conversation in the chat").

Thanks. Google AI Studio isn’t local, I think, is it? I’ll have to test this, but our project sizes and specification documents are likely to run into size limitations for local models (or for the clipboard at the very least ;)). And what I’d be most interested in are big-picture questions and global analyses.

No, it's not. I've not seen any local models that can handle 1m+ tokens.

I haven't actually done many experiments with long context local models - I tend to hit the hosted API models for that kind of thing.

Legal does not even want us running offline models for reasons. I assume that comes down to not knowing what offline-only means, but such is life.

Maybe they're concerned that code written with AI assistance can't be copyrighted? I've seen that idea floated in a few places.

I feel bad for your team.

Let people work how they want. I wouldn’t not hire someone on the basis of them not using a language server.

The creator of the Odin language famously doesn’t use one. He’s says that he, specifically, is faster without one.

No, it’s reasonable. If your team uses Git then it’s a valid question to establish if someone has only worked with Perforce.

They didn’t say how heavily they weight the question.

(All that said I expect that, soon, experience with the appropriate LLM tooling will be as important as having experience with the language your system is implemented in.)

Right, but using git is a team wide thing.

I can’t use perforce while my company is on git.

But if I do or do not use an LLM to assist me while coding, my team is unaffected.

If someone liked jetbrains, but your team used neovim, would you force them to use neovim?

Editors may also be a team decision in some places. Some teams are using features unique to one IDE, for example.

it can be a team decision, but it's a bad one

Then that tooling is required, like visual studio is a common one I know about in windows land.

Though nobody should care if I edited my text files with neovim as long as I still used the same toolchain as everyone else.

You hire people based on their fundamental knowledge and the ability to learn, not skills in arbitrary tools and frameworks which come and go every other day. If someone has used Perforce they will be able to get perfectly comfortable with Git by the end of their first week. So not knowing Git is an idiotic reason to reject a skilled developer. Same with programming languages, and just about every other aspect of software development.

I don't really test any specific tools or frameworks, what i'm using has changed twice just in the last year. More so, I just want to hear that the candidate has some knowledge of what the current models can do well, what they can't do, and how they're integrating it. Whether you're copying pasting code or using something like cursor is not what i'm concerned about.

Yeah, but it's oh so easy to test for, and oh so nice to have plenty of boxes checked to cover your ass if the hire goes wrong.

My expectations around productivity are going to assume you're using AI. That means stuff that might have taken a few days, i'm going to expect in a few hours or less. It's not unreasonable, i've seen over and over agian that kind of speed up. I have a lot less approval to hire people than I used to... so it's really important to me that I can extract that level of productivity out of my team.

If you're "working the way you want to" ie still handrolling all your code, you're going to find my expectations unrealistic, and that is certainly not fair to you.

Just curious, but what AI related skills do you expect them to have?

The ability to recognize and join a hype train, I presume. It’s one way to appear proactively leading-edge to marginally-informed product managers, marketers, execs and press.

That's an extremely uncharitable presumption. Although I don't agree that routine usage of AIs should be a precondition for regular software engineering jobs, there are good reasons for using LLMs besides "joining a hype train".

Nah.

I ask what their current workflow is, how they check and verify things, what their approach to prompting is etc. I'm looking to see that they've developed basic skills, have a reasonable mental model of what models can do well, what they currently can't do, and have an approach to be productive using the tools.