ChatGPT & Technical Hiring: Q&A
By Nathalie Figuière
Last week, we hosted a webinar on ChatGPT & Technical Hiring.
We discussed:
- What TA professionals need to know about ChatGPT
- How our experts predict ChatGPT will impact tech hiring
- What developers (candidates and hiring managers) are saying about ChatGPT
- How ChatGPT may (or may not) help candidates cheat during the hiring process
- How TA pros could best use ChatGPT to their advantage
Over 500 of you registered to take part and on the day, the chat went wild with questions! Unfortunately, we ran out of time before we were able to provide answers to all of them. Let’s remedy that!
Here are your questions, answered.
ChatGPT and developers
1. With the advent of ChatGPT, won’t this tool replace developers at work? – Francis
💬 Nathan Sutter:
ChatGPT is really good at looking for, aggregating and presenting information to you. What it’s not good at is context, nuance, care and human judgment.
ChatGPT wouldn’t be able to, for example, spin up a version of MongoDB that is sharded in a particular way, specifically at CoderPad, integrate it with everything else and deploy it to production.
ChatGPT will lack historical context about why things were built in a certain way, care in regards to use cases that are not particularly well documented within the business, understanding of non-written learnings from customers or teams, etc.
2. Why is there this fear of AI, why do people talk about losing their jobs, as there’s still a need for human instinct. What’s your take on this? – Muhammad
💬 Nathan Sutter:
People always fear what’s new and unknown. It’s a natural and legitimate reaction, and it’s something that comes up all the time.
My bet is that the worry will evaporate as soon as people see that ChatGPT will actually make their work more interesting.
In the case of e-commerce, for example, there was an initial fear of off-the-shelf, building-block type softwares (like Shopify, BigCommerce or Wix) replacing software developers. Now, most developers have embraced this technology, as it makes their job less repetitive and more interesting.
3. […] You’ve mentioned the teams at CoderPad are using the GitHub Copilot and ChatGPT, how did you help your team overcome their fears? – Bruce
Full question: “Our company is starting to embrace tools like GitHub Copilot and ChatGPT. We have run our coding questions through the tools to see how the tools answer the questions. Some of our team is uncomfortable with this. You’ve mentioned the teams at CoderPad are using the GitHub Copilot and ChatGPT, how did you help your team overcome their fears? What was the most common fear?“
💬 Nathan Sutter:
The best place to start is to discuss with your team where the fear is coming from.
Typically, most of those who are fearful of new tools have some misconceptions about exactly what the tool can and cannot do.
“Specifically speaking to ChatGPT and Copilot, there is a lot of media hype surrounding one or both of the tools “replacing” jobs. This couldn’t be farther from the truth.”
While it’s true that ChatGPT probably can answer most technical questions at an entry-level developer’s level, it still cannot operate autonomously in a business environment to deliver goals or grow as an engineer (and therefore be more valuable to the organization over time). This is where the real value of software developers stands.
So, start by discussing what your team’s fears are based on, unpack that, and counter it with what the tools can/cannot.
ChatGPT and recruiters
4. What is the best way to use ChatGPT to help with cold calling? – Chris
💬 Nathan Sutter:
ChatGPT is very good at writing prompts, based on the potential outcomes you may be looking for.
I’d personally start by defining your goals for the cold call, and then ask ChatGPT to write you a script to potentially help sell this product/service/job/etc. to someone.
5. Do you have to be good at scripting or using APIs in order to truly use ChatGPT? – Chris
💬 Nathan Sutter:
That’s one of the cooler things about ChatGPT. Being a LLM, it can hold a full conversation with a user in their language of choice.
To get your feet wet I’d start by asking ChatGPT some questions specifically on things that are not technical at all, and having a conversation with it. This will give you a good feel for how the language and conversational model works.
🔖 Related read: How to Embrace ChatGPT in Technical Interviews
6. What’s your advice for remaining authentic in our work, when handing over part of it to a computer? – Rosie
💬 Daniel Cloudt:
It’s very hard to automate human connections. We can’t automate the caring, human, empathetic part of our jobs.
When receiving an email or a LinkedIn message, for example, candidates still care whether it’s a robot or a human writing to them. AI is getting very good at mimicking human interactions. Going forward, we’ll have to find new and creative ways to show empathy and build relationships.
I think it’s important not to favor efficiency over empathy in recruitment. We need both to do a good job and build authentic relationships with candidates and employees.
7. Are there tools being developed to offset possible hiring bias that the ChatGPT may unexpectedly introduce? – Wade
💬 Amanda Richardson:
I don’t yet know of anything that can successfully identify hiring bias. However it’s a really interesting opportunity.
I do know that some tools and plugins, Grammarly for example, are making conscious efforts to build up their apps against bias.
AI presents bias and inaccuracies, but so do humans. Maybe AI can help to mitigate that bias, rather than the other way around. In any case, we must be cognizant of whatever data sets we’re using and hold mitigating bias in recruitment as a top priority.
💬 Daniel Cloudt:
We can calibrate algorithms, but you can’t calibrate society.
Identifying bias in interviews is difficult, and it would be wonderful if technology could help with that (by, for example, reviewing interviewer notes, flagging trends, etc.).
In any case, hiring bias is innately linked to CV-based hiring and experience-based hiring. It’s an institutional bias. If you’re looking for certain experiences and interviewing with those in mind, you’re going to introduce a lot of bias. People that have experience, continue to build experience, and so on.
Hypothetical, problem-solving interviews, relevant to the job at hand, are a better, fairer way to hire developers.
Video clip: Daniel Cloudt on bias in experience-based interviews ▶️
ChatGPT, candidates and cheating
8. For tools like CoderPad Screen, are there any mechanisms in place to detect potential “cheating”? – Riaan
Full question: “For tools like CoderPad Screen, are there any mechanisms in place to detect potential “cheating”? Ultimately, screening is the first step in the process and we need to be able to “trust” the results. If a candidate, during the screening process, had to copy/paste a coding question into ChatGPT and either copy/paste (or retype) the answer back into the tool, is there something akin to plagiarism detection that can pick this up?”
Yes, we continually and automatically monitor candidate activity within our platform. We also empower you to identify any suspicious behavior.
Our Code Playback feature provides insight into how your candidate built their algorithm, when they left the environment, when they copied and pasted, etc. If you pick up on any malpractice, you have the power to flag and reject the candidate.
We regularly crawl the internet to detect question and/or solution leaks. Our plagiarism detection engine will also recognize if a candidate submits the exact code previously submitted by another candidate and trigger a notification.
Our system will pick up on any abnormal candidate performance (e.g. difficult questions completed in a fraction of the usual time).
We track candidates’ approximate geolocation so as to spot any unusual behavior (e.g. logins from different locations or devices during the test).
🔖 Related read: ChatGPT and the Future of Technical Interviews: Addressing Concerns of Increased Cheating
9. Would it be considered cheating if candidates using ChatGPT exhibit both time efficiency and a thorough understanding of the code?
Full question: “Would it be considered cheating if candidates using ChatGPT exhibit both time efficiency and a thorough understanding of the code? Considering the potential benefits of using ChatGPT in the future to enhance skills, do you believe that technical hiring teams may penalize candidates for using ChatGPT, even if they demonstrate their ability to excel in interviews or complete take-home projects?”
💬 Nathan Sutter:
“If a candidate presents a solution that is not originally their own, whether it is generated through ChatGPT, copied/pasted from StackOverflow, or written by a friend, and is still able to fully understand it, I wouldn’t consider it cheating. Dishonest if they do not disclose that at first? Yes. But cheating? No.”
Think about the signal you are trying to tease out in an interview.
Really, the question you’re trying to answer is “can this candidate problem solve effectively, in the domain they will be working in, with the team I have today”. If this person can take a solution they did not write and continue to expand upon that with an interviewer, that is a very strong signal that they will be successful in the role.
10. How is ChatGPT able to pass the bar examination and a bunch of other complex exams with high grades? – Simon
“The ChatGPT model is trained using massive amounts of data, available on the internet. It’s like giving a smart student (or candidate) unlimited resources, endless time and a robot-like memory.”
Yes, ChatGPT is able to pass the bar exam. This doesn’t mean that it’s capable of providing specific legal advice, advocating for clients in court, or handling complex legal issues that require strategic thinking and nuanced interpretation of the law.
ChatGPT and CoderPad
11. How is your approach towards ChatGPT at CoderPad different from that of your competitors? – Jean
💬 Amanda Richardson:
There are many solutions out there, it’s a competitive space and we need to earn your trust every day.
I don’t like to speak for others, but here at CoderPad, we are approaching ChatGPT with enthusiasm and excitement. We are ready to embrace this technology and we’re actively working on leveraging it to improve our product and have a greater positive impact on technical hiring.
Ultimately, if you were to present me with a tool that could make my dev team 40% more efficient, as ChatGPT may be able to do, I would spend millions of dollars on that.
Video clip: Amanda Richardson on CoderPad’s approach to ChatGPT ▶️
12. How will CoderPad act against or utilize ChatGPT to make the interview process more efficient? – Stephanie
💬 Nathan Sutter:
We have a few experiments in the works – most of which follow our basic premise of embracing the technology and using it to make devs more efficient. We’re also looking to leverage ChatGPT to help companies evaluate how developers use technology and resources.
13. Will we see less take-home technical assessments because of the fear candidates would use ChatGPT? Will the live coding interview become king? – Fadi
💬 Amanda Richardson:
Live interviews are the best way to really understand how someone thinks and reasons through problems.
Live interviews have always been a critical part of the interview process, in my opinion. There’s so much more to learn than “can someone answer this question”—as dev jobs are rarely just about answering questions.
That said, I can see questions being added to take-home technical assessments asking a candidate to write an explanation of how they got to a solution or why they chose a particular solution.
“There is still a place for asynchronous work in the interview process (after all, that mimics the job best) but I hope the evolution we will see is one that marries the two much more closely and really values how developers think and solve problems.”
14. Is it still relevant to use tools like CoderPad Screen as a way of assessing candidates? – Junior
💬 Amanda Richardson:
Absolutely. I think features like multiple-choice questions will become less relevant over time. But seeing someone code and solve a problem (through features like code playback) will always be important—and doesn’t require a live interview.
The efficiency and quality of an assessment remains high when used properly.