6 Ways Customers Improved Their Technical Interviewing with Playback Mode
Most people can only hold a handful of things in their memory at the same time–famously 7, plus or minus 2. (Personally, I’m lucky to hit 3). Technical interviewing requires keeping track of dozens of elements, from the logistics of conducting the interview to evaluating the code. You’re likely to lose some important details.
That’s why CoderPad created Playback Mode to help you review your candidates’ code submission and rewatch how they arrived at their solution after the interview. Playback improves your ability to review the final code by letting you watch the interview back. You can do so in real time or by jumping to specific moments. (Protip: Playback skips from edit to edit, compressing any “dead” time, in turn saving you time.) This feature is also useful for reviewing an interview someone else conducted.
We’re evolving Playback for some exciting product updates (TBA, but really, really exciting, I promise!). So I talked with our customers to learn more about how they use Playback now and how returning to the interview helped them improve the hiring process.
My hope is that, by sharing what I found in user research, you’ll be able to refine your own technical interviews. Here are the benefits that respondents described.
1. Jogging their memory
Across several chats with customers, I kept hearing the phrase “to jog my memory.” Frequently, there was some lag time between the interview and when feedback was written, whether because of meetings, other work, or simply wanting to sit with what happened for a bit. Sometimes a hiring manager or hiring committee would ask follow-up questions days later, or a candidate would interview with the company a second time.
In each case, viewing the code and/or how it evolved allowed them to recall details and impressions from the interview.
A senior engineer for a large media company says he doesn’t start with the final code, but rather from the beginning. This serves as a way to jog his memory even days or months later.
I remember code really well. It helps me remember the conversation.
A principal developer on an R&D team says he tries to refresh his memory by walking through how the candidate solved the problem.
You can see what they did, recall the thought process [they used for] problem solving.
Interviewers also use the timeline to spot-check specific moments during the interview, such as when the candidate did something unexpected, struggled, or when the interviewer’s attention was elsewhere.
2. Digging into key moments
Interviewers find Playback helpful to return to specific moments when there is a gap in their memory or notes, when the candidate’s actions were unclear, or when they want to understand the candidate’s thought process better.
Sometimes if I’m writing my notes and I realize, oh, the candidate iterated through some versions here, but I don’t quite remember where they went off track. Did they use the wrong data structure, or did I remember correctly that they made this mistake… then I’ll go back.
I want to give more importance to the thought process… [In the case of one candidate] the solution was not done. He was not able to complete it, but his thought process was good.
Returning to pivotal moments in the interview can unlock a clearer picture of the candidate’s capabilities.
3. Noticing things they missed
Interviewers are splitting their focus between the candidate, the code, and the logistics of running the interview. As such, it’s common for them to want to hone in on just the code after the interview is over.
While you’re in the interview session, you’re thinking about a lot of things: you’re looking at a resume, you’re thinking about all the other things that you want to ask, like the behavioral question and the technical questions, and then they’re solving a problem. So a lot of things are going on in your mind, but… after the interview, when you’re looking at this, your mind is focused towards [the] problem, only. That’s something that the playback helps with.
With the ability to focus on the details without competing priorities, interviewers can take even more considerations into account, such as:
- How long did it take the candidate to get through different stages of the interview?
- How much help did they really get from the interviewer?
- What was left unaddressed, and how far from a working or complete solution was a candidate?
- How was the readability, maintainability, and overall quality of code and documentation?
- Where did the candidate go wrong?
- Did I miss any red flags, e.g. signs of cheating, inefficient solutions, or poor coding practices?
4. Settling disputes amongst the hiring panel
Occasionally the hiring panel may disagree over a candidate. In these cases it’s extremely valuable to return to the hard evidence. With Playback, they were able to replay all or a portion of a candidate’s interview to strengthen their case.
I remember once that we had a dispute [about one candidate during the debriefing meeting]. Some people really liked him. But in my case, the code really was not working. They replayed [my interview with the candidate, and saw] ‘Hey, this is really not a good one.’ So CoderPad, in this situation, played a really vital role for deciding whether the candidate is good or not.
5. Including supporting evidence in candidate feedback reports
To back up their assessment of a candidate, many interviewers use Playback to copy portions of the candidate’s code into their feedback report.
One Senior Machine Learning Engineer talks about what he includes in his evaluation writeup: “This is the question I used, this is the reason for my decision, and give them concrete examples [from the candidate’s code].”
Coding style is another very important factor to make sure that they’re writing clean code, not messy code. And normally we ask questions that can be resolved within 50 lines. 100 lines of code? That is really a negative signal. It means that they’re going the wrong route.
6. Minimizing biases
We’re all trying to find the candidates who are going to thrive in our organization. At the same time, we want to avoid wasting time by running an interview that’s biased by our own shortcomings and causes us to miss out on quality candidates.
You [might] want to hear some keyword from that candidate, and if they say that keyword, you will be happy[…], But sometimes what happens, the candidate is not able to say that keyword, but he or she is correct about that particular area. [Then, it] actually makes a lot of sense to revisit our interview session, to understand that the candidate was correct, and I was thinking on a ‘monologue.’
Why some interviewers don’t use Playback—and why they should
Some interviewers only utilized the notes they took, what was in their memory, or the impressions and conclusions formed during the interview. Their reasoning was that:
- their initial assessment was solid
- their memory was crisp
- their notes (if they took any) were sufficient
Therefore, they felt that they didn’t need to revisit the “raw evidence.” One interviewer said he doesn’t return to the code because:
“I’ve already assessed the candidate, and I’ve made my pronouncement, and then I clear my mind… I just build an image of what the candidate’s capabilities are right away.”
Skipping Playback might be a reasonable approach when assessing an interview with minimal or no coding, such as one focused on systems architecture or a coding question that’s used as a mechanism to learn how someone thinks about a problem. Usually, however, relying exclusively on our initial judgments is a strategy prone to bias.
And in the tech industry, there are still numerous factors that stand in the way of fair hiring. Mthree’s 2021 ‘Diversity in Tech’ report found that 45% of businesses haven’t yet invested in anti-bias training for hiring managers. Our hope is that Playback can serve as a way for customers to proactively create deliberate, thorough hiring best practices and ultimately reduce unconscious bias.
Playback’s utility to “jog memory” fills a gap for many interviewers, and seeing customers’ real-world usage gave us a vital perspective on how our living coding environment can continue to evolve. We invite you to continue improving technical interviews with us.