What Will It Take to Close the AI Evaluation Gap in Employee Listening?

Key Takeaway: AI capabilities are evolving faster than our ability to measure them — which makes rigorous evaluation more important than ever. For employee listening, meaningful AI evaluation means accounting for the complexity of the environment, the degree of autonomy a model can handle, and the complexity of the output it produces. The goal isn't to determine whether a model can complete a task. It's to understand when, how, and under what conditions we can trust its output. That measurement work is what makes it possible to use AI responsibly and effectively in the workplace.

In September, we published our first Leaderboard Refresh for PYX-Voice, our benchmark for how well frontier AI models analyze and interpret employee survey data. After publishing that first-of-its-kind benchmark in July, we were eager to see whether performance had shifted between the models available over the summer and the newer models released just a couple of months later.

The result was illuminating: of the 13 models we tested in September, nine surpassed the performance of July's top performer. The pace of development in AI is at once exhilarating and terrifying. How incredible for any technology to develop at this pace! But how concerning in a world without guardrails and clear standards.

As many in the field have said, the technology is advancing faster than any of us can adequately measure it. And as we at PYX Labs have found, operationalizing and measuring performance is exceptionally complex, difficult, and time-consuming — particularly in highly subjective contexts.

Vincent Sunn Chen wrote in Closing the Evaluation Gap in Agentic AI, "this evaluation gap is one of the most important problems in AI." He argues that while AI benchmarks are only part of the solution, they are essential components of a broader measurement toolkit.

In reading Chen's piece, I was immediately drawn to the conceptual model he lays out for the three core dimensions of benchmarks — a welcome structured framework in a field of overwhelming complexity and nuance: environment complexity, autonomy horizon, and output complexity.

It begged the question: how does this framework apply to our work at PYX Labs — and, more specifically, to our PYX-Voice benchmark?

Here is how Chen's three dimensions map to the specific domain of employee listening:

Let me unpack each one.

How complex is the environment surrounding employee survey analysis and interpretation?

Very complex. Employee feedback reflects an individual's own needs and the systemic patterns of the people around them, and the language shifts across companies, teams, and people — so a benchmark worth trusting has to capture that context, not just the words on the page. This is Chen's first dimension, environment complexity.

Organizational contexts are dynamic, with individual perspectives interacting within complicated systemic patterns. Employee feedback reflects not only a person's own needs and preferences, but also the ways the people around them interact with the broader internal and external business environment.

The language of employee feedback differs across companies, teams, and people. At the same time, day-to-day shifts change that context in real time. And beyond the complexity influencing the feedback itself, another layer applies to its analysis and interpretation: determining what matters and what will actually change business outcomes.

So what does this mean for our benchmark approach?

We aim to capture as much of this complexity as we possibly can.

Domain specificity

We work with our in-house I/O psychologists to document unwritten, tacit knowledge in clear, transparent language. To the extent that we're able, we strive to create clear standards and terms that replace vague gut checks about what "good" looks like in the domain. That expertise isn't incidental — it draws on two decades of working with real employee-experience data, which is what lets us say with authority what "good" looks like when a model interprets how people feel at work.

As we evolve, we're working to become increasingly granular about the facets of performance and to ensure we have adequate coverage of each.

Context complexity

When creating PYX-Voice, we crafted three "worlds" — datasets representing roughly 17,500 real employees across three organizations — that combined real-world data with consultant-crafted context, including business priorities and relevant research questions.

As we evolve the benchmark, there is an opportunity to add even more contextual information to better reflect what real I/O psychologists do when interpreting employee feedback. Our team members discussed some of these ideas in a recent debrief of the benchmark's development.

Human and AI interaction

There is another layer of complexity: how humans and AI actually interact.

To simplify our approach for the first benchmark, we imagined the prompts were written by a well-informed HR practitioner who uploaded survey data, business context, and a list of 84 tasks to a frontier model. The benchmark then evaluated how well the models completed those tasks.

But the reality of how HR practitioners partner with models is more complex. There is often iteration, additional context, clarification, and back-and-forth narrative shaping.

This begs an important question: how can we better understand that dynamic? And when do these interactions improve performance versus degrade it?

What are the elements of these interactions that allow for greater accuracy, efficiency, and novel insight?

These are questions we'll need to explore as benchmarks become more representative of real-world AI use.

How long can an AI system work autonomously before human oversight becomes essential?

Parts of survey analysis and interpretation can already be delegated to a model today, provided there is a quality-assurance step to catch overclaims and fabrication. What we haven't solved yet is exactly what that QA step should be. This is Chen's second dimension, autonomy horizon.

A large part of our motivation for understanding models' efficacy at these tasks is to answer a practical question: what should practitioners be doing themselves, and what can they reasonably delegate to a model to streamline their workflows?

In other words, how long can a model work usefully before it starts making things up, making mistakes, or simply adding no value?

Perhaps the most important aspect of this dimension is the idea Chen calls "non-stationary goals and environments": as the context surrounding a task evolves, the approach and output need to shift accordingly.

Consider a simple example.

On Monday, an HR practitioner might assign an LLM a task based on their current understanding of the organization:

"Our company strategy will prioritize customer retention next year. I need to find the process improvement suggestions that will help us do that."

By Friday, the context may have changed dramatically:

"Our company is preparing for a significant M&A event, and the intellectual property on our Product team is the most important asset we have. I need to figure out how to support the Product team's engagement and the tools that will help them document institutional knowledge."

The employee feedback stayed the same. The goals for interpreting and acting on that feedback changed dramatically.

That distinction matters for employee listening. If the meaning and usefulness of an analysis depend on the organizational context in which it is performed, evaluating AI performance cannot simply be about whether a model produces a technically plausible answer. We also need to understand whether it can adapt its reasoning as the goals and environment change — holding on to what remains relevant and letting go of what no longer applies.

What happens when the desired output becomes too complex to measure?

Past a certain point, measuring the output becomes as hard as producing it. The more nuanced and multi-layered the task, the more models struggle — and the harder it becomes to define what a "good" answer even looks like. This is Chen's third dimension, output complexity.

What we've found so far is, unsurprisingly, that the more complex and nuanced the task output needs to be, the more LLMs struggle to produce it.

When we ask models to synthesize multiple data points across quantitative and qualitative survey items and provide a coherent narrative, something breaks down along the way. When we ask models to "bubble up" the three most important themes in a business unit, they tend to surface insights that lack meaning, substance, or practicality.

Improved prompts help. But the more complex the task, the more opportunities there are for error.

There is another aspect of this that Chen articulates particularly well: "as agents produce more complex work, evaluation must become more complex too — and the gap is growing."

In fact, as we set out to build PYX-Voice, we were surprised to find that measuring some of these complex tasks became akin to a sweater unraveling by a single thread.

For instance, we might develop a "gold standard" response with one I/O psychologist, only to find that the next had a different take. Once we landed on a new "gold standard output," we found it difficult to articulate specific criteria. There was infinite potential for what the output could look like in response to the prompt. How could we feasibly account for all those possibilities?

Then, once we finalized what felt like reasonable criteria, we hit another roadblock: our human graders applied those criteria alongside their own difficult-to-define judgment of what "good" looks like. Our LLM Judge applied the criteria in its own way, based on the context and fine-tuning it had received.

The result was low agreement between our human graders and our LLM grader. That meant we could not yet confidently measure performance on the task.

When we discussed this dilemma on our recent webcast, we talked about something I/O measurement experts know well: to measure something, you have to break it down into the smallest, most discrete construct possible.

Consider a task as complex as developing an executive summary for a CHRO about holistic employee feedback across an organization — one that takes into account what matters most to that CHRO based on the current business strategy and industry conditions.

To measure that output, we need to break it down into the tiniest possible chunks of performance:

• What knowledge, skills, abilities, and other characteristics are needed to achieve the desired output?
• What are the structural elements of a good response?
• What information is accurate or inaccurate based on the underlying data?
• How do those different pieces come together to create meaning and impact?

It's no small feat. And we're not there yet.

But these are exactly the types of questions we'll need to explore as a field if we want to confidently assess whether these tools can enable responsible, informed AI use.

What can benchmarking teach us about responsible AI in employee listening?

That "how good is this model?" is the wrong question. Good benchmarks don't just score a model — they give us a shared language for asking better questions: good at what, under what conditions, with what autonomy, producing what kind of output, and by whose definition of good. That's what makes AI usable responsibly at work.

Chen's framework gives us a useful way to wrap our minds around an exceptionally complex topic.

The more we work on benchmarks like PYX-Voice, the more we see that evaluating AI in the workplace is not simply a matter of asking a model a question and checking whether the answer "looks right."

The environment matters. The degree of autonomy matters. The complexity of the output matters.

And, perhaps most importantly, the breadth, depth, and quality of measurement itself matters.

Our September leaderboard refresh showed just how quickly model performance can change. Nine of the 13 models we tested surpassed the performance of July's top performer. That pace of development makes rigorous evaluation even more important.

Benchmarks won't answer every question about AI capability. But they can give us a common language for asking better ones: identifying where models are useful today, understanding where human judgment remains essential, and measuring how those boundaries change over time.

For those of us working at the intersection of AI, employee listening, and organizational science, that measurement challenge is also an opportunity.

If we can make the work of evaluation more rigorous, transparent, and nuanced, we can move beyond simply asking "how good is this model?"

We can start asking much more useful questions:

Good at what? Under what conditions? With what level of autonomy? Producing what kind of output? And according to whose definition of good?

Those are the questions that will help us understand not only what AI can do, but how we can use it responsibly in the real-world environments where people and organizations actually work.

—

About the author. Sarah Foster is Director of PYX Labs and an I-O psychologist. She led the design of the PYX-Voice benchmark and the expert criteria it grades against.

Reference: Chen, V. S. (n.d.). Closing the evaluation gap in agentic AI. Snorkel AI. https://benchmarks.snorkel.ai/closing-the-evaluation-gap-in-agentic-ai/