Based on the CoderPad webinar featuring Bayo Ojuri (CoderPad), Bonnie Dilber (Zapier), and Melany Austin (GoDaddy).
“AI fluency” has quietly become a line item in nearly every technical job description. The problem: most teams added the requirement long before they had a reliable way to assess it. You can ask a candidate whether they use AI. What you cannot easily see is whether they use it well. Drawing on patterns from more than 60,000 AI-enabled technical interviews, this session laid out how to define the skill, generate real signal, and score it consistently.
Most processes are stuck at “Still” or “Crawl”
AI hiring maturity is a curve: Still, Crawl, Run. “Still” teams wrote AI fluency into the req but changed nothing about how they interview. “Crawl” teams let candidates use AI but did not adjust questions or scoring. Only “Run” teams redesigned the interview so AI use produces comparable signal. Adding the words to a posting does not move you along the curve. Changing the environment and the criteria does.
The three elements that separate signal from noise
• A realistic environment: let candidates work the way they actually will on the job, AI included, instead of measuring a world that no longer exists.
• Fresh question design: recycled problems are exactly what a model answers instantly. Fresh questions shift the test from “can you get the answer” to “how do you get to a good one.”
• Consistent evaluation criteria: a shared rubric applied the same way across candidates and interviewers turns impressions into a fair, defensible comparison.
What AI fluency looks like in an interview
• Prompts with intent, giving context and constraints
• Critically reviews AI output instead of accepting the first result
• Stays in control of the problem and can defend every decision
• Knows when not to reach for AI
The gaps are the mirror: accepting output uncritically, being unable to explain the code, or losing the thread the moment the assistant is involved.
The bar keeps moving
Talent leaders from Zapier and GoDaddy made the point that AI proficiency is a moving target. Last year’s impressive, is this year’s table stakes. The teams hiring well reset their standard on purpose as the skill and tools evolve
And in case you missed it watch the full webinar for more insights on what AI proficiency looks like now and how to hire for it.