6 Ways Customers Improved Their Technical Interviewing with Playback Mode
Most people can only hold a handful of things in their memory at the same time–famously 7, plus or minus 2. (Personally, I’m lucky to hit 3). Technical interviewing requires keeping track of dozens of elements, from the logistics of conducting the interview to evaluating the code. You’re likely to lose some important details.
That’s why CoderPad created Playback Mode to help you review your candidates’ code submission and rewatch how they arrived at their solution after the interview. Playback improves your ability to review the final code by letting you watch the interview back. You can do so in real time or by jumping to specific moments. (Protip: Playback skips from edit to edit, compressing any “dead” time, in turn saving you time.) This feature is also useful for reviewing an interview someone else conducted.
We’re evolving Playback for some exciting product updates (TBA, but really, really exciting, I promise!). So I talked with our customers to learn more about how they use Playback now and how returning to the interview helped them improve the hiring process.
My hope is that, by sharing what I found in user research, you’ll be able to refine your own technical interviews. Here are the benefits that respondents described.
1. Jogging their memory
Across several chats with customers, I kept hearing the phrase “to jog my memory.” Frequently, there was some lag time between the interview and when feedback was written, whether because of meetings, other work, or simply wanting to sit with what happened for a bit. Sometimes a hiring manager or hiring committee would ask follow-up questions days later, or a candidate would interview with the company a second time.
In each case, viewing the code and/or how it evolved allowed them to recall details and impressions from the interview.
A senior engineer for a large media company says he doesn’t start with the final code, but rather from the beginning. This serves as a way to jog his memory even days or months later.
A principal developer on an R&D team says he tries to refresh his memory by walking through how the candidate solved the problem.
Interviewers also use the timeline to spot-check specific moments during the interview, such as when the candidate did something unexpected, struggled, or when the interviewer’s attention was elsewhere.
2. Digging into key moments
Interviewers find Playback helpful to return to specific moments when there is a gap in their memory or notes, when the candidate’s actions were unclear, or when they want to understand the candidate’s thought process better.
Returning to pivotal moments in the interview can unlock a clearer picture of the candidate’s capabilities.
3. Noticing things they missed
Interviewers are splitting their focus between the candidate, the code, and the logistics of running the interview. As such, it’s common for them to want to hone in on just the code after the interview is over.
With the ability to focus on the details without competing priorities, interviewers can take even more considerations into account, such as:
- How long did it take the candidate to get through different stages of the interview?
- How much help did they really get from the interviewer?
- What was left unaddressed, and how far from a working or complete solution was a candidate?
- How was the readability, maintainability, and overall quality of code and documentation?
- Where did the candidate go wrong?
- Did I miss any red flags, e.g. signs of cheating, inefficient solutions, or poor coding practices?
4. Settling disputes amongst the hiring panel
Occasionally the hiring panel may disagree over a candidate. In these cases it’s extremely valuable to return to the hard evidence. With Playback, they were able to replay all or a portion of a candidate’s interview to strengthen their case.
5. Including supporting evidence in candidate feedback reports
To back up their assessment of a candidate, many interviewers use Playback to copy portions of the candidate’s code into their feedback report.
One Senior Machine Learning Engineer talks about what he includes in his evaluation writeup: “This is the question I used, this is the reason for my decision, and give them concrete examples [from the candidate’s code].”
6. Minimizing biases
We’re all trying to find the candidates who are going to thrive in our organization. At the same time, we want to avoid wasting time by running an interview that’s biased by our own shortcomings and causes us to miss out on quality candidates.
Why some interviewers don’t use Playback—and why they should
Some interviewers only utilized the notes they took, what was in their memory, or the impressions and conclusions formed during the interview. Their reasoning was that:
- their initial assessment was solid
- their memory was crisp
- their notes (if they took any) were sufficient
Therefore, they felt that they didn’t need to revisit the “raw evidence.” One interviewer said he doesn’t return to the code because:
“I’ve already assessed the candidate, and I’ve made my pronouncement, and then I clear my mind… I just build an image of what the candidate’s capabilities are right away.”
Skipping Playback might be a reasonable approach when assessing an interview with minimal or no coding, such as one focused on systems architecture or a coding question that’s used as a mechanism to learn how someone thinks about a problem. Usually, however, relying exclusively on our initial judgments is a strategy prone to bias.
And in the tech industry, there are still numerous factors that stand in the way of fair hiring. Mthree’s 2021 ‘Diversity in Tech’ report found that 45% of businesses haven’t yet invested in anti-bias training for hiring managers. Our hope is that Playback can serve as a way for customers to proactively create deliberate, thorough hiring best practices and ultimately reduce unconscious bias.
Playback’s utility to “jog memory” fills a gap for many interviewers, and seeing customers’ real-world usage gave us a vital perspective on how our living coding environment can continue to evolve. We invite you to continue improving technical interviews with us.