3 Tips To Introduce ChatGPT Into Your Technical Interviews

Autocomplete. Debuggers. Package managers.

What do they have in common?

At one point, these were new technologies that eased the lives of countless software engineers. At some point, hiring managers needed to see if their potential employees could utilize these time-saving innovations. Eventually, recruiters brought the use of these tools into their technical interviews.

We’re at that point with Artificial Intelligence (AI) and Large Language Models (LLMs) like ChatGPT, GitHub Copilot, and Google Bard. This means you should start taking steps to introduce them into your interviews.

Since this is a relatively recent technology, not all your candidates will have experience with it, and those that do will have used it for different things and at varying levels of complexity.

🔖 Related webinar: How Interviewers Are Leveraging ChatGPT to Hire Developers

Luckily for you, we have some experience with this. Below, we highlight three tips for introducing ChatGPT into your interview process.

Tip 1: Set expectations

If you tell your interviewee they can use ChatGPT or another AI tool in the interview, they may interpret your message in various ways.

Those who have never used it before may just ignore it and focus on the question. Others may use it to write every single line of code to answer a question. Others still may use it to validate their answers, generate test cases, or ask for word definitions.

There’s no objective wrong answer to how they use it, which means you’ll have to let them know what is acceptable and/or unacceptable when using ChatGPT in an interview. It’s great to set up a list of these expectations ahead of time with other interviewers so that each candidate has the same restraints to work with so they can all fairly be assessed on their use of AI tools.

An excellent general guideline is to use the same expectations you’d have if they were an engineer on your team using ChatGPT to complete a given development task. That way, you’re giving them a more realistic scenario, which increases your ability to judge their fitness with your team.

Clarity and specificity are essential when setting expectations for software developer job candidates regarding using ChatGPT in a live coding interview. Here are some example expectations you might set to make the interview more successful with AI:

1. Clearly articulate that ChatGPT is intended to be a supplementary tool during the interview. It can be used for generating coding problems, providing hints, or validating logic. However, candidates should not rely on it to write substantial portions of code.

2. Emphasize that candidates are expected to collaborate with ChatGPT as they would with a colleague. This means they can discuss problems, brainstorm solutions, and ask for suggestions on best practices.

3. Let candidates know that while ChatGPT can assist, the main evaluation will focus on their problem-solving skills and their ability to apply logical reasoning and coding practices to the questions at hand.

4. Encourage the use of ChatGPT for creating test cases or debugging code. However, candidates should verify the suggestions provided by ChatGPT and ensure they understand any code they use.

5. Make it clear that the candidates are expected to maintain integrity during the interview. If they use ChatGPT to help with a solution, they should be transparent about it and ready to discuss the reasoning behind the code.

Tip 2: Demonstrate how to use it

Before diving into the interview, take a moment to showcase how ChatGPT will be utilized. You could present a mock scenario or a typical problem they might solve using ChatGPT. For example, demonstrate how to ask ChatGPT to generate test cases and then walk through how you expect the candidate to approach them. 

This demonstration can be particularly reassuring, as it helps demystify the technology and sets a clear expectation of the candidate’s interactions with the AI. It’s also an opportunity to show the candidate that the AI’s responses are not infallible and that their critical thinking in response to ChatGPT’s prompts is a valuable part of the interview.

🤖 If you’re using CoderPad for your interviews, we simplify ChatGPT demonstrations with our AI Chat feature.

It’s also a good idea to offer a practice session to help reduce interview anxiety related to the unfamiliarity with AI. At the beginning of the interview, you can give them 5-10 minutes to prompt whatever AI tool you’re using to understand how it works before they start answering your actual interview questions.

A practice session familiarizes them with the technology and demonstrates your company’s commitment to candidate experience and fair assessment practices. It shows that you value their comfort and confidence in the interview, which can reflect positively on your company’s culture.

Tip 3: Mention its limitations

There’s an important disclaimer at the bottom of ChatGPT that you may not have noticed:

ChatGPT can make mistakes. Consider checking important information.

For a candidate, this “important information” will most likely be any code generated from the LLM, so the most apparent takeaway or reminder for candidates should always be to verify or check the response. Sometimes, this is as easy as spotting bad code; other times, issues might not be visible until the code is run or once the test cases have been applied.

Given the number of other use cases available, you may want to make the candidate aware of other limitations to make the most of their limited time working with ChatGPT during the technical interview:

1. ChatGPT may struggle with understanding complex, multi-step conversations or contexts that require significant domain expertise.

2. Since ChatGPT is trained on a vast corpus of text, there’s a risk of it generating responses that could contain or reference sensitive information.

3. The model may not always provide accurate or specific enough information for technical tasks without precise input.

4. While ChatGPT can generate code snippets, these may not always follow best practices or be error-free.

5. ChatGPT’s training data has a cutoff, making its knowledge potentially outdated. It won’t be aware of the very latest technologies or updates.

6. The AI can sometimes misunderstand the user’s intent or the requirements of a prompt, leading to irrelevant outputs.

7. ChatGPT can offer debugging assistance, but its capabilities are limited compared to specialized debugging tools and it may not understand complex bugs.

Conclusion

Incorporating ChatGPT into the technical interview process is more than just a nod to modern tech trends—it’s a strategic move that aligns with the ever-evolving landscape of software development. By setting clear expectations, demonstrating practical use, and acknowledging the tool’s limitations, you create a comprehensive, realistic, and fair assessment environment. 

Ultimately, integrating AI like ChatGPT into interviews reflects an understanding of future industry directions and a commitment to fostering a work environment that encourages continuous learning and improvement – a huge selling point when it comes to attracting the best and brightest employees to your organization.

Some parts of this blog post were written with the assistance of ChatGPT