How to evaluate AI experience when hiring engineers

"AI experience" is on almost every engineering brief now, and almost none of them say what it means. The usual approach is to ask the candidate whether they use AI tools, hear "yes, daily," and move on. That question stopped separating candidates the moment everyone could answer it the same way. In the 2026 Stack Overflow Developer Survey, 73% of developers who use AI coding assistants or agents use them daily (Stack Overflow, October 2026). Daily use is the baseline, not a signal.

This playbook gives you a way to evaluate AI experience inside the technical interview you already run: what to ask, what to ask to see, and what to ignore. At the end you will have a 45-minute segment and a rubric with four observable behaviors.

What do you need before you start?

You need a definition of "AI experience" for this role, written down before the first interview.

• The two or three tasks on this team where AI tools are actually used (code generation in a known area, test writing, debugging, migrations).
• The review standard that applies to AI-generated code on your team. If it is "same as any other PR," write that down; it changes what you test for.
• One real, small task from your codebase or a close stand-in, with a known trap (an edge case, a stale dependency, a misleading comment).
• The tools your team runs, and agreement among interviewers that the thing being scored is judgment about output, not fluency with a product.

How do you ask about the work, not the tool?

Ask the candidate to walk you through the last piece of work where they used AI and something in the output was wrong.

Why this works: every engineer who uses these tools daily has a story about output that looked right and was not. An engineer without that story either does not use the tools much or does not check what they produce. Listen for specifics: what the model got wrong, how they noticed, what they did next.

What good looks like: "It generated a retry wrapper that swallowed the exception on the third attempt. I only caught it because I read the test it wrote and the assertion was on the log message, not the return value." Specific, mechanical, and it tells you how they read code.

What should you ask to see?

Ask for a pull request they authored where AI wrote a meaningful share of the code, and have them explain their review of it.

Why this matters: the artifact removes the gap between what people say in interviews and what they do at work. Ask them to point to the lines they changed after generation and why. If they cannot share employer work, an open-source contribution with the same structure works. What you want to see is diff-level editing of generated code, not a prompt log.

What good looks like: the candidate can point to three places where they rewrote the output, and at least one is about fit with the surrounding system rather than correctness in isolation.

How do you test judgment live?

Give the candidate your prepared task, let them use whatever AI tool they prefer, and score what they do with the output.

Why this matters: the interview should look like the job. Twenty-five minutes, screen shared, tool of their choice. You are not scoring whether they finish, but four behaviors:

1. Do they give the model context before asking, or paste the problem and hope?
2. Do they read the generated code before running it?
3. Do they find the trap you planted, and how: by reading, by testing, or by luck?
4. When the output is wrong, do they fix it by hand, re-prompt with a better constraint, or accept it?

What good looks like: the candidate pastes the relevant interface, reads the output, says "this won't handle the empty case," writes a test for it, and fixes the generated code directly. Behavior 3 found by reading scores higher than found by test failure, which scores higher than not found.

How do you score it?

Score each of the four behaviors on a three-point scale and compare notes before anyone talks about overall impression.

Why this matters: without a rubric, "AI experience" collapses back into fluency with a product, and the confident talker wins. Keep it short: 0 (not observed), 1 (observed once), 2 (consistent). A candidate who scores 2 on reading output and fixing by hand but 0 on giving context is a strong engineer who has not yet learned the tool; that is teachable in a week. The reverse profile is not.

What good looks like: two interviewers scoring the same session land within one point on each behavior.

What does not count?

Keyword lists, tool names on a résumé, prompt-engineering courses without shipped work, and "I built an app with AI" with no review history.

Why this matters: each of these measures exposure, not judgment. A list of tools tells you what the candidate has opened. A course certificate without a repository behind it tells you they watched something. A side project generated end to end is evidence of speed, which was never the hard part. The signal is in what they changed after the model wrote it.

What are the common mistakes?

• Testing the prompt, not the review. Scoring how well the candidate talks to the model instead of what they do with the answer.
• Banning the tools in the interview. Then you learn how they code without AI, which is not how they will code on your team.
• Planting a trap that requires tribal knowledge. The trap should be findable by reading carefully, not by knowing your system.
• Letting "AI experience" replace the technical interview. It is a segment inside the loop, not the loop.

How does Nexton do this?

Every engineer is interviewed by another engineer before you see the profile, and the written evaluation comes with the candidate; AI capability is assessed as a separate signal on top of that interview.

The technical interview tests whether someone can build. For the AI part, Nexton AI Ready looks at four dimensions, with the last one, output validation, carrying the most weight: engineers document real work tasks improved with AI and the reasoning behind each decision. Certifications in the program are issued by recognized AI platforms and industry bodies, not by Nexton, and they do not replace the vetting. If you want the detail of how the technical interview itself runs, see How to run engineer-to-engineer technical vetting. For how to read the certifications a candidate lists, see Which AI certifications for engineers actually mean something. And if you are rewriting the brief for 2027, How AI is reshaping engineering roles covers what to hire for.

If "AI experience" is on your brief and you are not sure how you would test it, it's worth a conversation.

FAQs

Should I let candidates use AI tools during the technical interview?
Yes, for the segment that evaluates AI experience. The interview should look like the job. Keep a separate segment, or a separate interview, for fundamentals without tools if your team needs that signal.

What if the candidate uses a different AI tool than my team?
Score the behaviors, not the product. Reading output before running it, giving context, and fixing generated code by hand transfer across tools. Product-specific fluency can be learned in days.

Is a take-home test better for this?
A take-home shows the result but hides the process, and the process is the signal. If you use one, pair it with a live walkthrough of what the model produced and what the candidate changed.

Sources: Stack Overflow 2026 Developer Survey results, October 2026. Verified October 2026.