Palqee Prisma found statistically relevant bias towards migrants in AI resume ranking tool, where traditional fairness metrics couldn't.
Challenge:
Applicant Tracking Systems (ATS) are increasingly used to automate the hiring process, but concerns about their potential bias, particularly towards migrants, remain prevalent. These systems often disregard explicit protected attributes like gender and nationality but may still exhibit bias due to subtle, context-related factors. Impactmania, part of the EU funded FINDHR initiative to tackle Fairness and Intersectional Discrimination in AI HR tools, sought to determine whether ATS could be biased against migrant applicants based on nuanced resume attributes.
Solution:
"To investigate potential biases, we utilised Palqee prisma, a tool designed to enhance the transparency and trustworthiness of AI models. Prisma allows for a contextual understanding of why an AI system makes certain decisions, which is crucial for identifying and addressing biases." - Impactmania.
The project focused on resumes tailored for a U.K.-based job advertisement, analysing the impact of three subtle attributes—Language Proficiency, Overseas Work Experience, and International Education—on ATS rankings.
Approach:
Palqee prisma Configuration: We set up prisma with parameters specific to the U.K. job market, targeting the role of a Project Manager.
Attribute Identification & Classification: Prisma classified a diverse set of resumes, based on the identified subtle attributes covering both local (U.K.) and international candidates.
Categorization: Resumes were classified as either 'local' or 'international,' enabling a thorough analysis of potential biases.
Comparison: Prisma evaluates results among resumes.
Results:
Our evaluation revealed a statistically significant bias towards migrant applicants. Key findings include:
Score Disparities: International resumes consistently received lower scores than local ones, with an average decline of 2-3 points for international resumes containing subtle attributes, compared to a 1-point decrease for local resumes.
Impact of International Education: This attribute had the most pronounced negative effect, with 80% of international resumes scoring lower than their local counterparts.
Statistical Significance: The differences in scores between local and international resumes were statistically significant, particularly for resumes with subtle attributes, suggesting that the ATS favoured applicants with local backgrounds.
Conclusion:
The experiment underscored the limitations of traditional bias detection methods, which often overlook the nuanced factors that can influence ATS decisions. By integrating Palqee prisma into the bias assessment process, it provided a clearer understanding of the ATS's behaviour, highlighting the potential risks for migrant applicants. This case study demonstrates the importance of contextual analysis in mitigating AI bias, particularly in hiring practices, and supports the need for more robust AI governance and compliance mechanisms.