How Interviewers Are Leveraging ChatGPT to Hire Developers

In our recent webinar on ChatGPT & Technical Interviews, we explored: 

• The advantages and challenges of utilizing ChatGPT in interviews
• How to assess and interpret engineers’ use of ChatGPT during an interview and on the job (what should impress you and what should set alarm bells ringing)
• Actionable tips and strategies for effectively integrating ChatGPT into your own interview process 

Over 500 of you registered to take part and on the day, the chat went wild with questions! Unfortunately, we ran out of time before we were able to provide answers to all of them. Let’s remedy that! 

Here are your questions, answered by our experts Muhannad Asfour, Chuck Daminato, Nathan Sutter, and Mathis Hammel. 

Candidate experience and ChatGPT

1. What are candidates saying about ChatGPT in interviews? – Allison

🔵 Nathan Sutter:

Some candidates are surprised, because, unfortunately, many places still do closed-book interviews. 

I haven’t personally heard any negative feedback from candidates.

🟢 Muhannad Asfour:

Some of them are a little hesitant because they’re afraid to use the “wrong” prompt. 

They worry that if they ask ChatGPT, for example, ‘how do I instantiate an array’, interviewers are going to think they don’t know what they’re talking about. With an integrated AI tab, candidates’ prompts and thought process are visible, as opposed to using Google in another window where the interviewer wouldn’t see their search queries. 

So, I think there is some nervousness there. But, the more people become used to the fact that this tool is available within the interview experience, the more comfortable they’ll be using it. I think there’ll be a shift in perspectives here.     

Identifying red flags

2. What if a candidate is secretly using ChatGPT outside of CoderPad (or the interview) in a different window? How best to detect/address this? – Jamie 

🟡 Mathis Hammel: 

If you clearly state that candidates can use ChatGPT within the interview environment, then there’s really no reason for candidates to hide the fact that they’re using AI.

That said, if they are “secretly” using AI to produce code, answer questions—that’s not necessarily a problem. I don’t think we need to dissociate code written “from scratch” and code that AI contributed to. What’s key is a candidate’s understanding of whatever AI suggests. 

If you have any doubts at all, ask candidates to explain or build on their answer. For example, LLMs are notoriously bad at security. Why not ask them if there’s anything wrong with their code, or missing from their code? Here you’re evaluating critical thinking, knowledge of best practices in development, secure coding—things that make software engineers irreplaceable by AI (yet!). 

Another great interview format is “debugging interviews”, where you provide the code and ask candidates to find the issue. This is a great way for developers to showcase their skills and something that AI has trouble doing well today.

🟠 Chuck Daminato: 

ChatGPT is imperfect, it’s not necessarily going to give you the right answer or a correctly formulated answer. I’ve had ChatGPT tell me to use libraries that no longer exist or method signatures that aren’t in a given library, for example. 

It’s how candidates react to what ChatGPT suggests, how they identify and address the bugs, errors or irrelevancies, that will indicate a deeper understanding of software development.

🟢 Muhannad Asfour: 

Let’s say your candidate is using ChatGPT secretly. I always like to ask them “Is this the most optimal solution? Why, or why not?”. 

This really exposes whether they understand the code that they’re submitting. To a greater extent, you’ll see whether a candidate can refactor that code themselves, see if there’s a more optimal way of doing it and communicate/articulate it too.  

3. Has anyone experienced an interview where a candidate relied on ChatGPT too much or used it inappropriately? – Anonymous

🔵 Nathan Sutter:

I haven’t personally experienced this, yet, but I can share some anecdotes from my team and colleagues across the industry.

I’ve heard that some candidates use ChatGPT, confident that a copy-paste answer directly from the tool will be applicable in an interview, and that interviewers would not dig further. This blew up in their faces rather quickly as they could not defend nor dig deeper into reasoning around their solution. It made for a quick determination that a candidate would not be a good fit.

I’ve also heard that interviewees can be overwhelmed by the length of ChatGPT’s response and get confused and then flustered mid interview. This isn’t a great indicator of a candidate’s capacity to problem solve, think critically, and adjust prompts!

ChatGPT and developers’ jobs

4. Can chatGPT be too much of a crutch for developers? Is it possible that engineers are too reliant on it? – Drew

🔵 Nathan Sutter: 

I personally don’t think so. 

In my opinion, anybody who is good at their job is going to use whatever tools and resources are available to make their job as efficient as possible. 

🟡 Mathis Hammel:

I think that AI tools are great for professional developers. 

I do have a contrasted opinion on AI in an educational setting though. I would worry about giving students access to these tools. I’d worry that if they’re heavily leaning on AI to complete assignments and various projects, they might miss out on learning the basics. 

It would be like allowing use of high-performing calculators or tools like Wolfram Alpha in all Math classes, to solve even the basic stuff like simple multiplications. 

I do think that you need to learn the basics before automating. 

5. Do you think GenAI tools will lower the skill level for entry level developers in the future and eventually lead to a lack of senior developers at today’s standard of expertise? – Drew

🟠 Chuck Daminato: 

I don’t feel that we’ll have this cohort of developers just regurgitating whatever the robot is telling them. 

Using ChatGPT in interviews: best practices

6. How are you using ChatGPT as the interviewer? Are you using ChatGPT to generate interview questions? Or summarize your chats with candidates? – Sachin

🟠 Chuck Daminato: 

We allow open-form usage of ChatGPT, but to date are not using it to generate our questions or summarization.

🟢 Muhannad Asfour: 

Our interview process is split into two parts. First we use CoderPad Screen to submit a coding problem relevant to the role, then we host a live “discussion” interview. 

One way we use ChatGPT currently is for the interview part of the process. Beforehand, we submit some of our interview questions to ChatGPT (say “Can you talk me through how you would, end-to-end, build a certain web application?”). The answers become a sort of “compare and contrast” with our candidates.

We then generally ask both candidates and ChatGPT a follow-up question: “Was anything missed?”. We’re looking for backup, redundancy or scalability, for example. Interestingly a lot of candidates forget about security, whereas ChatGPT doesn’t!  

Using ChatGPT for these open-end types of questions has given us a lot of food for thought.

🔵 Nathan Sutter: 

I haven’t used ChatGPT to generate interview questions so much, but I have seen colleagues use it to analyze and compare candidate’s responses. 

For example, you could have 5 candidates answer the same question and then have ChatGPT summarize the differences between their solutions. This may well highlight things that weren’t initially clearly visible. 

7. Are there AI tools that allow you to upload a job description and resume to see if the candidate is a good fit? – Sachin

🔵 Nathan Sutter: 

You can absolutely use AI tools to do this, but I’m not sure I’d say it’s a good idea. 

🟢 Muhannad Asfour: 

I would be generally interested in seeing how AI compares to the more traditional/common applicant tracking systems (ATS) that are being used now to screen resumes.

I think it could be helpful to trial it, but I would be wary of fully delegating that responsibility to ChatGPT or similar tools—for fear of missing out on good candidates.

8. What’s your take on how to practically implement ChatGPT in the interview process? – Marcelo

🟢 Muhannad Asfour: 

I’d say that step one is defining your goals. What specific objectives do you want to accomplish by incorporating ChatGPT in your interviews? What are you assessing the candidate for? What do you consider a “good” or “bad” use of ChatGPT. 

I’d also recommend getting familiar with ChatGPT before using it in any interviews. Understand how it works, the capabilities and limitations. Experiment with various questions, see what answers ChatGPT offers, and look at how you can bounce off that content. 

Finally, it’s important to set candidates up for success. Be transparent with them from the beginning. Let them know in advance that ChatGPT will be available for them to use within the interview and reassure them that you won’t knock points for them using it.

9. Are there certain industries less likely to allow an open-book interview using Google, ChatGPT or other external resources? For example, a tech job at a financial services business versus a software engineer at a software company? – Anonymous

🟠 Chuck Daminato:

This is less industry-specific, and more culture-centric at the company in question. I’ve interviewed at places that were small/growth startups and were unnecessarily strict, and at large/established banks that had an excellently collaborative and engaging interview.

It really comes down to what the drivers of the engineering culture are trying to foster (and, really, how “old school” they are; avoid them if possible!)

🟢 Muhannad Asfour:

I totally agree with Chuck on this one.

I have also experienced my fair share of interviews at various companies and it all boils down to company culture. 

Some companies believe that memorizing textbook definitions and algorithms are the unequivocal truth to measure a candidate’s competency. On the other hand, other companies take a more realistic approach and gauge the problem solving and communication abilities of the candidate rather than their memorization skills.

10. If the process for non-ChatGPT and ChatGPT-enabled interviews is pretty much the same, you still evaluate the same parameters, what are some significant benefits of using ChatGPT in an interview vs. keeping a classic interview format? – Anonymous

🟠 Chuck Daminato: 

Two points to this:

1. ChatGPT tends to get to answers quicker, so candidates (and interviewers) aren’t parsing “close but not quite what I’m looking for” solutions when doing web searches. This helps maintain a good flow in the interview process.

2. With CoderPad Interview, the embedded interface allows for a clear view of the candidate’s usage (and they can see us do the same!), plus it is included in the interview recording/history for later analysis.

ChatGPT within CoderPad

11. Which version of the model is available within CoderPad? Does/will it include plugins/web browsing? – Nate

🟡 Mathis Hammel:

We currently use version 3.5.

12. We have CoderPad and I don’t see the AI tab. Is there an option in our settings to turn it on? – Anonymous

🟡 Mathis Hammel:

An Admin has to turn on the AI tab. They can do so in “Team Settings”. You can check out what that looks like here. 

13. Can candidates also do a regular Google search within CoderPad? – Anonymous

🟡 Mathis Hammel:

There’s currently no Google search functionality within CoderPad, but that doesn’t mean that candidates can’t use Google outside of the IDE. 

Sourcing and screening with ChatGPT

14. How can I use ChapGPT to build sourcing/Boolean search strings more accurately? – Ben

🟢 Muhannad Asfour: 

I just tried this with ChatGPT to see what it would come up with.

Input to ChatGPT:

“Generate a Boolean search string to find software engineers with experience in Python and JavaScript, who have worked at Google or Facebook, and have a minimum of 5 years of experience.”

ChatGPT generated a search string like this:

`(software engineer) AND (Python OR JavaScript) AND (Google OR Facebook) AND (experience:[5 TO *])`

ChatGPT’s output looks logically correct, so I would say that it is accurate. You could improve it by being even more specific/granular with your input.

One thing to keep in mind is that the efficacy of these boolean search strings depends on the platform you’re using and the quality of the content in candidate profiles.

As for variations, I would definitely ask ChatGPT to create those for me!

15. Can I upload a template resume to ChatGPT to help find similar profiles? How can I accomplish something similar? – Ben

🟢 Muhannad Asfour:

I think if you combine ChatGPT with a scraping tool that would scrape LinkedIn and job board sites like Indeed for data, it may achieve what you’re looking for.

Although, I would caution that it would probably take a fair bit of time and tweaking to get some good results.