The messy business of managing people at work: Is AI the solution?

Hiring and managing people at work effectively has never been easy. In recruitment, the task has become more complicated as online applications, and more recently generative AI, have led to a surge in the volume of applicants. As a result, screening candidates has become more cumbersome, leading employers to rely increasingly on technology to assist the process. Recruitment is thus a prime example of the “paradox of automation”[i], where each problem that technology tries to solve creates a new problem to be solved.

This need, coupled with billions of dollars in investment into AI-assisted tools, has seen a transformation in human resource management. Yet the rush to automate HR functions is outpacing our understanding of whether these systems actually work.

In a recent ILO working paper, my co-author, Hannah Johnston, and I, evaluate the effectiveness of different AI applications across four core HR functions: recruitment, compensation, scheduling, and performance management. To assess the AI tools that are applied in these areas, we apply a framework based on the following three parameters:

1. the objective – what is the system trying to achieve,
2. the data it is trained on and uses, and
3. how it is programmed

The quality of each of these parameters differentiates systems that work well from those that do not.

Defining the objective is not straightforward

While defining a system’s objective may appear straightforward, in practice, it is not. For neutral and straightforward aims – such as determining the shortest route between two locations – it is easy to rely on identifiable and relevant variables, and the findings are easy to interpret. But most human resource functions involve the “fleshy, messy, indeterminate stuff of everyday life”[ii] which creates practical challenges. Sure, we want to hire the best candidate for a job, but how do we define a good candidate? Yes, we want employees that work hard, are engaged and are competent, but how do we identify the data needed for an AI system to effectively assess performance?

One organization that decided to use AI for recruitment defined their objective as selecting candidates with a “growth mindset”. As a result, the AI system was programmed to identify and score candidates in their interview according to their use of words such as ‘growth’, ‘development’, and ‘learning’. This complex human quality was thus reduced to a word-frequency count, raising fundamental questions about whether the system was measuring anything meaningful at all.

Data quality and suitability are key

Data limitations are the focus of much of the critique of AI, with three salient concerns. First, there is the question of data quality. AI systems rely on training data to learn the connections and patterns that provide the foundation upon which decisions are made. When these data are poor quality, AI systems yield poor quality outputs. As the saying goes: ‘garbage in, garbage out’.

A second issue concerns the suitability of the data. In bespoke AI systems that are developed internally, training data consist of the organization’s past operations. In contrast, ‘off-the-shelf’ systems developed for sale to third parties use data compiled from available sources, including sometimes purchased from the growing data market. Often these data are not reflective of the procuring organization’s characteristics, raising concerns about their representativeness.

Data also need to be meaningful. Because AI systems run on data, there is a continual search for more data to be included in the model. For recruitment, this can mean incorporating information from candidates’ social media activity, but it has also involved adding information from other assessments, including cognitive, personality or situational judgement tests, often in the form of games. One game had candidates inflate a digital balloon in exchange for a reward, purportedly to measure candidates’ propensity for risk-taking behavior and reaction time.[iii] Similarly, applications that measure employees’ online activity by recording keystrokes, taking random screenshots, or monitoring online presence can demonstrate whether a worker is active online. But data on screentime shouldn’t be equated with how well a person does their job.

It is important to remember that AI systems do not have theory – rather, they build and predict patterns through data. Thus, there is a need for a “human resource” theory to guide the selection of data to be collected. Otherwise, if the data feeding the system are meaningless, how is an HR manager supposed to interpret the results?

The examples also show that the oft-cited critique that data aren’t representative, while valid, is not the sole concern. One could incorporate more representative training data of people’s scores in the digital ballon game, but that would still not address whether the data constitutes a predictable indicator of performance.

Transparency in decision-making is critical for understanding results

Algorithms are a central decision-making feature at the core of AI systems. They can be defined as a set of rules executed through computer programming code with a particular aim or objective. While the most basic algorithms merely execute a list of prescribed instructions, these instructions nonetheless reflect the view of the person who programmed it and thus the intentional or unintentional biases that the person may have. Machine learning algorithms, like those used in AI systems, are still developed and deployed by humans, but include an additional layer of opacity. As the system evolves, it can become fraught, with their precise features eluding even those who built them.

In one striking example, researchers found that even a gender-neutral STEM job advertisement on a social media platform was shown disproportionately to men — not because of explicit targeting, but because the platform's algorithm "learned" that it was cheaper to advertise to men and thus optimised accordingly.[iv]

What are HR managers to do?

In many of the AI for HR applications, the systems operate with a high level of autonomy and automation that requires little of those who deploy them. The work of HR professionals is essentially outsourced to off-the-shelf, ‘easy to use’, prepackaged technology. But herein lies the problem: the systems may be ‘easy to use’, but the users have a poor understanding of what they are using , how it works, or what the results mean.

Overcoming this shortcoming requires that HR managers participate in the design, implementation and oversight of AI systems. And for HR functions that pertain to how work is conducted and evaluated (i.e., work scheduling and performance management), it is also necessary to involve the workers themselves. There are examples of how this has been done effectively. One large multinational spent two years iterating an AI recruitment system, involving HR professionals throughout, and ultimately adopted a hybrid human-AI model with explainable results.[v] In another example, a telecommunications firm co-designed a scheduling system with its field technicians, resulting in a 10% productivity improvement and a more than one-third reduction in mental health absences.[vi]

Involving all stakeholders in the design of AI systems provides a means to address the three pillars of AI systems: to clarify the objective, to ensure that the data used are suitable for that purpose and context as well as representative and transparent, and to ensure that the programing accounts for various conditions that the system is likely to encounter.

Such meaningful stakeholder engagement will lengthen the development and implementation time and it will also require that HR professionals develop more comprehensive and deeper understanding of AI. This investment is both necessary and worthwhile.