Who’s smarter: ChatGPT or an HR pro?

IBM’s Deep Blue supercomputer took on world chess champion Garry Kasparov in a historic man-versus-machine matchup in 1996. Kasparov won that match, but in 1997, the computer reigned supreme, besting Kasparov and becoming the first computer to beat a world champ at the ultimate board game, signaling that perhaps computers can outperform even the pros.

When ChatGPT was entering the public consciousness early last year, some workers began to worry that the tool’s capabilities might eventually render them useless to employers. The HR field was not immune, and to test this, Mineral (now a division of Mitratech) set out to see if the tool was indeed smarter than their experts.

Mitratech, an HR compliance software company, helps small businesses “navigate really sticky situations,” said Mineral’s Head of HR Compliance Services and Content Susan Anderson, and make sure no one is running afoul of the law. Its team of (human) experts field more than 15,000 HR questions monthly, she said.

When data scientists performed the first experiment last year, assessing ChatGPT’s ability to advise on HR questions with OpenAI’s GPT-3.5, Anderson cut it short. The generative AI performed so poorly that the team learned what not to do when training an AI tool. It was safe to assume experts were still best suited to tackle the complicated queries facing the company’s clients.

“The answers it provides are too often inaccurate, incomplete, or misleading in a way that could be costly to a business that wants to lean on the tool to answer HR questions,” Anderson told HR Morning of the first experiment.

But OpenAI’s GPT-4 “unlocked this massive opportunity” to drive efficiency, and the team built a custom GPT-4-powered model leveraging the company’s deep knowledge base of the employment compliance landscape.

Anderson assembled a group of five compliance experts and had them use the tool to lighten their workload, measuring the quality and volume of their work, as well as the employee experience. What they learned was that by leveraging generative AI, they could more quickly wrap up work on the fundamental and rudimentary inquiries, leaving space to tackle the more complicated tasks.

“We saw that this was going to be a tool that would unlock future human potential, and do so in a way that empowered our workers to focus on high value work,” she said.

Experts also found an unexpected benefit: the tools sometimes offered “a different perspective,” allowing users to offer more developed answers.

“The experts that we worked with [for the second experiment] were very excited to continue to use the tool. They felt like it was expediting the first mile of their work. Every answer that they collaborated with was still theirs,” she said, adding that they’d use their own words, and use the tool as a starting point. “It wasn’t a game changer in terms of being able to produce more volume, but it was a game changer for our employee experience and then, the quality [of responses].”

The study found that AI assistance was helpful in answering questions around best practices and strategy. It handled questions related to background checks, exempt versus non-exempt statuses, termination decision-making, workers comp, payroll, and tax issues, according to Anderson.

The model performed much more poorly with regulatory or otherwise nuanced and complicated questions, particularly those related to health, welfare, and retirement, she said.

Who won? The experiment revealed that, unlike Deep Blue versus mortal in a chess match, everyone was a winner. Think of generative AI tools as an assist and “a great collaborative tool,” she said.

The HR experts at Mitratech found that using ChatGPT helped save time in drafting responses, and provided alternative perspectives, all of which contributed to a more efficient workflow.

Anderson believes the future of AI in HR will see more HCM providers, payroll administrators, and ATS platforms creating “copilot” experiences within their tools that leverage the abilities of generative AI in a specific area or expertise, with their specific brand voice, to help employees or clients get the contextualized answers they need, keeping humans in the loop.

“People see AI and think it’s the hammer that solves every nail,” she said. “I do believe strongly that the generative AI capabilities will be a game changer for humanity at large…And I think that the companies that are going to have success with this have the knowledge base and the workflows that are…fit for purpose.”