AI systems used for candidate screening, shortlisting, assessment, promotion, task allocation, monitoring or termination may be classified as high-risk under the AI Act. A company that uses them cannot simply rely on the provider's documentation: it must define the purpose, human oversight, input data, monitoring, rights and the criteria for suspending the system.
Which uses may be high-risk
Annex III covers systems intended for:
- Recruitment and selection.
- Targeted job advertising.
- Screening and evaluation of applications.
- Decisions on working conditions.
- Promotion or termination.
- Task allocation based on behaviour or traits.
- Monitoring and performance evaluation.
Classification must examine the specific use and the exceptions in Article 6. HR tools should not be assumed high-risk by default, nor should a "copilot" be assumed exempt simply because of its name.
Roles
The manufacturer is usually the provider, and the company using the system is the deployer. A company can become a provider if it changes the intended purpose, markets the system under its own name, or makes a substantial modification in the cases set out in the Regulation.
The contract does not replace a genuine analysis of control.
AI Act and GDPR
The AI Act does not replace the GDPR or employment law. The company needs:
- A defined purpose.
- A legal basis.
- Information in line with Articles 13 and 14 GDPR.
- Data minimisation.
- Proportionate retention.
- Security.
- An assessment of automated decisions.
- A DPIA where high risk is likely.
- Employee representation where applicable.
The AI Act's "high-risk" classification and the high risk referred to in Article 35 GDPR are distinct concepts, even though they frequently overlap.
Automated decisions
Article 22 GDPR protects people from certain decisions based solely on automated processing that produce legal or similarly significant effects. Human review is only genuine if the reviewer has the competence, information, time and authority to change the outcome.
Rubber-stamping a default recommendation, or reviewing hundreds of applications in seconds, is not effective review.
System inventory
The inventory should record:
- Name, version and provider.
- Purpose.
- Employment stage.
- Population affected.
- Data and sources.
- Outputs and the resulting decision.
- Users and supervisors.
- Integration with the ATS/HRIS.
- Retention and logs.
- Metrics and limits.
- Incidents and changes.
Embedded AI features activated through an update must also be inventoried.
Candidate and workforce data
The review should cover:
- CVs and forms.
- Tests.
- Video or voice.
- Activity and productivity.
- Communications.
- Location.
- Health or disability.
- Personality or emotion inferences.
- Social networks and external sources.
Seemingly neutral variables can act as proxies. Necessity must be justified variable by variable, not by availability.
Assessing the provider
Request:
- Intended purpose and prohibited uses.
- Classification and its rationale.
- Validation data and population.
- Metrics broken down by group.
- False positives and false negatives.
- Oversight instructions.
- Logs.
- Version changes.
- Security.
- Incidents.
- Sub-processors.
- Documentation for GDPR and the AI Act.
A claim of "no bias" is not evidence.
Bias and quality
Comparing overall accuracy is not enough. Rates must be analysed by relevant group and by stage:
| Metric | Question |
|---|---|
| Selection | Who is excluded? |
| False negative | Who is wrongly rejected? |
| False positive | Who is favoured without justification? |
| Calibration | Does the score mean the same for everyone? |
| Coverage | Which applications go unprocessed? |
| Override | When does a person correct it? |
Groups and methods are determined on a sound legal and ethical basis, avoiding the creation of new risks.
Human oversight
The supervisor must be able to see:
- Relevant data.
- Source.
- Score and its limits.
- Uncertainty.
- Main factors.
- Alternatives.
- Override history.
Using the output as the sole reason is prohibited where the design requires judgement. Overrides are monitored to detect over-reliance or failures.
Transparency
Candidates and staff need clear information about:
- Use of AI.
- Purpose.
- Data.
- Role in the decision.
- Consequences.
- Rights.
- Human contact.
- How to challenge the decision.
This should not be buried in a lengthy policy. Explanations must enable people to act on them.
Deployer obligations
For high-risk systems, Article 26 requires use in line with the instructions, competent oversight, control over the input data under its control, monitoring, and action in response to risks or incidents.
The company keeps logs for the applicable period when they are under its control, and meets the specific employment information obligations where they apply.
Coordinated assessments
You may need:
- A DPIA.
- A fundamental rights impact assessment in the cases set out in Article 27.
- The provider's conformity assessment.
- Employment and equality analysis.
- Security.
A shared inventory is used, but each instrument keeps its own requirements.
Involvement and governance
Involve HR, legal, the DPO, security, equality, employee representatives and management. The business owner should not approve this alone.
The policy defines permitted, prohibited and escalation uses. Example: AI may rank applications for review, but must not automatically reject them without approved conditions.
60-day plan
Days 1 to 15
Inventory, classification and pausing uses with no owner.
Days 16 to 30
Provider, GDPR, DPIA and bias assessment.
Days 31 to 45
Oversight, transparency, logs and training.
Days 46 to 60
Pilot, testing, consultation and decision.
Common mistakes
- Relying on the provider's label.
- Using public data without a legal basis.
- Inferring emotions or personality without necessity.
- Measuring only overall accuracy.
- Calling an automatic rubber-stamp "human review."
- Failing to inform people.
- Ignoring version changes.
- Keeping CVs and scores indefinitely.
- Not allowing challenges.
- Deploying without internal representation.
Checklist
- Use and classification.
- Contractual and actual roles.
- Purpose, legal basis and minimisation.
- Provider and metrics.
- Bias and quality by group.
- Effective oversight.
- Transparency and rights.
- DPIA/FRIA where applicable.
- Logs and incidents.
- Training and review.
Frequently asked questions
Is all HR AI high-risk?
No. It depends on the use and its classification. Selection and employment decisions covered by Annex III usually require a more rigorous analysis.
Can AI reject candidates automatically?
This requires analysis under the AI Act, Article 22 GDPR and employment law. In many scenarios, genuine human intervention and safeguards are required.
Can a public CV be used freely?
No. Public availability does not remove the need for a purpose, legal basis, information and rights.
Who is liable, the provider or the company?
Both have obligations depending on their role. The company that deploys the system retains its own responsibilities.
Official sources consulted
- Regulation (EU) 2024/1689.
- AI Act Service Desk: Article 26.
- AEPD (Spanish DPA): GDPR compliance for AI processing.
- AEPD (Spanish DPA): assessing human intervention in automated decisions.
Summum Consultoría can coordinate classification, the DPIA, the provider assessment and employment-related controls.