linkedin Skip to Main Content
Just announced! We now support spreadsheets
Back to blog

I Asked ChatGPT To Evaluate Developers’ Soft Skills: Here Is What I Learned

Interviewing

It’s zero hour. You’re in the final round of interviews with three candidates, and your manager is pressing you to pick one to fill the senior dev role you’ve had open for the last seven weeks.

You’re looking at your notes, reviewing their answers to the non-technical questions.

“What excites you most about this role?”

Candidate 1: “I’m excited to work on new projects with new technology and make an impact on the team with my extensive front-end experience.”

Candidate 2: “I hear your team is working with React, which is a framework I’ve really wanted to work with. I like the fact that your team is fully remote, as I feel like it shows a lot of trust in your employees. Plus everyone else on the team I’ve talked with said you’re a great manager!”

Candidate 3: “This job would present lots of new challenges, which I find very exciting. My love of challenges is why I volunteered to be the lead developer on the calendar app at my last company, even though I had never taken lead on a project before. The salary increase definitely helps too!”

All decent answers. Yes, Candidate 3 broke the money taboo with their salary comment, but the rest of their answer was pretty good.

So how do you assess who has the best answer? 

You could go with your gut, but that may introduce some undesired bias into your decision. 

You could go with the one that most aligns with your team’s values, but how do you glean that from 1-3 sentences?

You could ask a more experienced interviewer, but getting a hold of them may be difficult, and how do you know they have a good system for analyzing candidate answers?

You know where I’m going here. You probably want to use some kind of large language model (LLM) AI to help you analyze those answers using the years of software engineering interviewers’ wisdom those LLMs have been trained on.

ChatGPT gives us some perspective

Here’s the prompt I gave to ChatGPT to help us with that particular interview question:

When asked  “What excites you most about this role?”, a senior software development candidate responds with “[candidate answer]”. What is your feedback on that answer? Answer in 3-5 sentences.

Candidate 1

Their answer: I’m excited to work on new projects with new technology and make an impact on the team with my extensive front-end experience.

ChatGPT feedback: The candidate's response effectively showcases their enthusiasm for new projects and technology, suggesting a dedication to continual learning, which is crucial in software development. 

Their intention to leverage their extensive front-end experience to benefit the team demonstrates collaboration and an understanding of the value they can bring. 

For further improvement, the candidate could offer specific examples of new technologies they're interested in and clarify how they intend to make an impact on the team, providing a fuller understanding of their motivations and potential contributions.

Candidate 2

Their answer: I hear your team is working with React, which is a framework I’ve really wanted to work with. I like the fact that your team is fully remote, as I feel like it shows a lot of trust in your employees. Plus everyone else on the team I’ve talked with said you’re a great manager!

ChatGPT feedback: The candidate's response demonstrates enthusiasm for the role, highlighting their technical interests (React), appreciation for the remote working environment, and the value they place on positive working relationships. 

To enhance their response, they could discuss their experience with remote work and how they maintain productivity, and elaborate on how they plan to contribute using React. 

This well-rounded answer indicates their readiness for the role and their potential to add value to the team.

Candidate 3

Their answer: This job would present lots of new challenges, which I find very exciting. My love of challenges is why I volunteered to be the lead developer on the calendar app at my last company, even though I had never taken lead on a project before. The salary increase definitely helps too!

ChatGPT feedback: The candidate effectively demonstrates eagerness for new challenges and leadership, exemplified by their past role as lead developer. 

However, the focus on salary increase, while honest, might overshadow their intrinsic motivations in the eyes of an employer. 

To improve, the candidate could further detail the specific challenges they look forward to in this role and how their leadership experience would translate into value for the prospective team.

Thoughts and a pick

ChatGPT does a decent job of pulling out the relevant sentiments from these answers. You could ask it for a more thorough analysis, but that’s also more reading you have to do as an interviewer, and you surely won’t be hiring a candidate based on one question alone – you’ll have many more you need to assess.

So who would you pick? 

All the candidates seem to express enthusiasm for the role. Candidates 1 and 2 also have answers that indicate they’re interested in how they could help assist the team, while candidate 3 has leadership experience they could utilize. 

All of the candidates had room to expand on their answers by giving more details, however candidate 1 seemed to lack any specifics of why they want the role, which may mean they’re not as excited as the answer initially makes them seem.

And of course ChatGPT picked up on the salary “issue” with Candidate 3 – whether that’s a showstopper for you in the hiring process would probably need to be weighed with their other answers and technical capabilities.

If I had to pick a best answer for this question, I’d say Candidate 2 assuming they had the ability to quickly pick up React, as that last line sums up what I thought when reading their answer: “This well-rounded answer indicates their readiness for the role and their potential to add value to the team.

Question 2: Tell me about a time when you failed at something.

This is a common interview question that interviewers use to evaluate a candidates’ honesty and resilience.

Let’s see what ChatGPT has to say about these answers with this prompt:

When asked  “Tell me about a time when you failed at something”, a senior software development candidate responds with “[candidate answer]”. What is your feedback on that answer? Answer in 3-5 sentences.

Candidate 1

Their answer: During my first on-call experience as a junior developer, I got a ticket for a bug that I thought I had seen before. So thinking I knew the easy fix, I reached out to the customer and told them we’d have a fix in place in the next 24 hours. When I pushed the fix, not only was the bug still there, but it rendered an important part of the customer’s screen unusable. I was immediately mortified, but apologized and told the customer that I had elevated the ticket to a more senior developer. 

I then went over to our tech lead and explained the situation, and how I want to work to fix it as soon as possible. She worked with me for the next hour and helped me fix my mistakes. It was embarrassing, but I definitely learned not to make assumptions about how easy a fix might be. I also learned to analyze problems more thoroughly before I patch, and to definitely do more testing!

ChatGPT feedback: The candidate demonstrated humility, accountability, and a willingness to learn from their mistakes, which are all positive traits. Their answer shows their capability to handle pressure and their willingness to seek help when required, which indicates good team-playing skills. 

However, the candidate could have done better by also discussing any subsequent steps taken to prevent such incidents from happening in the future, such as proposing or implementing changes in the QA or testing procedures. 

Overall, it's a solid response showing both growth and resilience.

Candidate 2

Their answer: Creating my first app. It bombed, we only had like 15 people sign up. Ended up scrapping just six months after we published it.

ChatGPT feedback: While the candidate's response does address a failure, it lacks sufficient detail and introspection. 

As a senior software developer, it would be valuable to understand what they learned from this failure, how they adapted their approach, and what specific steps they took to try to improve the situation before the project was ultimately scrapped. 

It would also be beneficial to hear about any changes they made to their strategies or workflows as a result of this experience. Providing such insights would demonstrate resilience, adaptability, and a growth mindset, which are key characteristics for a senior role.

Candidate 3

Their answer: On my previous team, we had a feature request for a unique UI element that I had never used or seen before. I thought I could figure it out without looking at any documentation or StackOverflow, but after 6 hours of getting nothing to work, I started Googling everything I could.

I spent the next day and a half reading and trying to get the element to work, but still no luck. So at our following stand-up, I mentioned my roadblock and asked for help. Between three other developers and a quality assurance analyst we were finally able to find out what I was doing wrong and get it working. It was a hard way to learn about advanced UI elements, but I won’t soon forget them!

ChatGPT feedback: The candidate's response is thoughtful and demonstrates their ability to handle failures maturely and use them as learning opportunities. 

They show the capability to be proactive in seeking out help and openness to collaboration when they faced a roadblock, which is a positive attribute for a team player. It also reflects their humility and willingness to learn from mistakes, which are essential traits in a senior role. 

One area they could improve is mentioning their approach to similar situations in the future, such as emphasizing the importance of reading documentation or seeking help sooner. Overall, this is a well-constructed response that reflects positively on the candidate.

Thoughts and a pick

We can quickly see that Candidate 2’s answer is the least sufficient out of the three, ChatGPT addresses this in the very first feedback sentence: “While the candidate’s response does address a failure, it lacks sufficient detail and introspection.”

Candidates 1 and 3 both had detailed answers and ChatGPT gave them both similar areas to improve upon – primarily suggesting actual steps they took to address their particular failures. It also stated that they both had well-constructed responses.

So the feedback was helpful in narrowing down the best answer to two candidates. To make a pick, it is probably helpful to go back to the answer itself; for me I like the fact that Candidate 1 actually stated the specific lessons they learned, whereas Candidate 3 did not.

Candidate 1 would be my choice for this question.

Question 3: Tell me about a disagreement you had with a coworker or boss and how you resolved it.

Another common interview question where we see how candidates deal with the inevitable team conflict. This is the ChatGPT prompt we’ll use:

When asked  “Tell me about a disagreement you had with a coworker or boss and how you resolved it.”, a senior software development candidate responds with “[candidate answer]”. What is your feedback on that answer? Answer in 3-5 sentences.

Candidate 1

Their answer: One time a coworker really wanted to please a client who was demanding a really aggressive deadline for a project. As the lead engineer of the project I told him that deadline was too optimistic and it would be very unlikely the engineering team could meet those demands in the given timeframe.

We had a meeting and worked out that if we were to pursue the aggressive deadline we would need to reduce the scope of the project by eliminating complex nice-to-have items. Ultimately we would be delivering a more bare-bones experience in a shorter amount of time. My coworker took the proposal back to the client who agreed for the simpler version as speed to market was more important for them to capitalize on a potential opportunity rather than waiting for everything to be perfect.

ChatGPT feedback: The candidate's response demonstrates effective conflict resolution, negotiation, and leadership skills, which are valuable in a senior role. 

They showed an ability to communicate openly and honestly about potential project risks, and were proactive in proposing a feasible solution that took into account the client's needs and the team's capabilities. 

Furthermore, their understanding of balancing business goals and technical feasibility is commendable. 

It might be beneficial if the candidate expanded on their communication with the coworker during this process, to demonstrate how they maintain positive professional relationships. 

Overall, this response shows the candidate's strong ability to navigate disagreements and manage project expectations.

Candidate 2

Their answer: We had a request from our marketing director to create an API for our product so that a health supplement company could integrate with our ad platform to market their products on various websites. I heard of the health supplement company and knew they made a lot of unrealistic claims about their products. I brought my concerns to the director and he assured me that what they were doing was legal, but I told him it didn’t seem ethical to me. We went ahead with building out the API, but I quit shortly after that because it wasn’t a company culture I wanted to be a part of.

ChatGPT feedback: The candidate clearly demonstrates their strong ethical stance and willingness to voice their concerns, which are positive attributes. They also show that they are not afraid to make difficult decisions when their personal values conflict with the job requirements. 

However, the conclusion of the disagreement might raise concerns about their ability to navigate conflicts and find resolutions within the workplace, as the ultimate action taken was to quit the job. It might have been beneficial for the candidate to share any efforts made to address the issue before resigning, such as seeking advice from HR or escalating their concerns higher up. 

Overall, while their commitment to ethical standards is commendable, their ability to resolve conflicts may need further exploration.

Candidate 3

Their answer: I found out a coworker had made a change to a recent form feature I had done without first consulting me, and they had introduced a bug that broke the validation on one of the input fields. I confronted them and told them they had to fix it immediately, but they said they were working on another issue and probably wouldn’t have time to fix it. I then went to their manager and told them how it was inappropriate for my coworker to have made the change without consulting me, and that it needed to be fixed immediately. The manager backed up my coworker, saying that unfortunately they had higher emergency priorities and the fix to my code would have to wait. 

I ended up fixing the problem myself, but made it so that I had to be on the list of reviewers for anyone who wanted to merge code into that component so that something like this wouldn’t happen again in the future.

ChatGPT feedback: The candidate clearly demonstrates their ability to address a conflict and take action to prevent similar issues in the future, which is a positive attribute. 

However, their response could be perceived as somewhat confrontational, and they might have benefited from initially discussing the issue with the coworker in a less accusatory manner. A more collaborative approach could have potentially led to a more harmonious resolution. 

Additionally, they could mention how they communicated the new code review process to the rest of the team and whether they considered any alternative solutions. 

Overall, the candidate's response shows their assertiveness and dedication to code quality, but could be improved by demonstrating better interpersonal and communication skills.

Thoughts and a pick

Here we’re trying to see how candidates deal with the inevitable conflict of working on a team. 

Can they admit when they’re wrong? Compromise when necessary? Do they hold grudges, or are they able to move forward and not take things personally?

You basically want to see if the candidate is able to do what is best for the product, even if it means they have to take a hit to their ego.

With ChatGPT’s feedback We can see Candidate 3 might have some work to do on being willing to work with others in a non-confrontational way that would produce better long term working relationships with other coworkers: 

[…]their response could be perceived as somewhat confrontational, and they might have benefited from initially discussing the issue with the coworker in a less accusatory manner. 

While they may have been in the right, they could cause lots of trouble on your team every time there is any kind of conflict.

It’s easy to side with Candidate 2 because of their ethical stance – standing up for what is right is always commendable. However, ChatGPT’s feedback points out: 

However, the conclusion of the disagreement might raise concerns about their ability to navigate conflicts and find resolutions within the workplace, as the ultimate action taken was to quit the job. It might have been beneficial for the candidate to share any efforts made to address the issue before resigning, such as seeking advice from HR or escalating their concerns higher up. 

Whether quitting was the right move or not for this kind of conflict is kind of a gray area, but it does seem like if they truly cared about making a better product and better company that they could have escalated the issues and taken more time to explain to the marketing director how misleading their customers is not a good business practice. 

That leaves Candidate 1 who, with the exception of possibly expanding on how they maintained a professional relationship with the coworker whom they had a disagreement with, was assessed as having a great response from ChatGPT:

Overall, the candidate’s response shows their assertiveness and dedication to code quality, but could be improved by demonstrating better interpersonal and communication skills.

My pick would be Candidate 3.

Some important caveats

Keep in mind, ChatGPT can’t read body language or hear intonation, so it may miss some of the nuances that candidates convery when they give the answer – for example were they smiling wide while talking about how excited they are to be working at the new role? Or did it seem like they were just going through the motions and were rather bored while giving the answer?

Unless there’s a major red flag, you’re not going to accept or reject a candidate based on one question answer alone, so you’ll want to look at the themes of their soft skill question answers to get an idea of how they’d work with your team – if they can show initiative, kindness, honesty, and any other traits you find important in most of their answers, then you probably have a winner.

You would also be wise not to completely rely upon ChatGPT, you should still be doing your own review of the candidates answers. ChatGPT can get things wrong, and interviews already seem impersonal enough that you’re doing a disservice to your candidates if you’re not giving them your human touch.

With that said, ChatGPT does help you parse out answers that seem relatively similar, and summarize them if they’re particularly long. 

Each response to our answer analysis prompts takes a similar format that you can use when you’re reviewing ChatGPT’s feedback:

  1. It lays out what the candidate did well: This is probably the most important part for determining who had the better answers, as you can see which aligns more with whatever rubric or values you’re using to assess the candidates.
  2. It lays out what the candidate can improve upon: Maybe less important unless you’re providing the candidate with feedback after the interview. It is helpful when you have candidates to see which has more to improve upon, and if that is something they can easily address.
  3. It lays out potential issues: ChatGPT will point out any concerning behaviors or even red flags that may warn you off of selecting a candidate.

If you’re interested in using ChatGPT in your interviews, CoderPad makes it easy to use with our ChatGPT integration. You can also check out these other posts on how to use this newest dev tool to improve you and your candidates’ interview experiences.