AI Evaluation Lead

Requirements of the role

The Data, Technology and Innovation (DTI) division enables the FCA to be a digital-first, data-led smart regulator by delivering a secure, agile, and cost-effective technology and data ecosystem that drives better decisions, transparency, and operational efficiency.

Working alongside the wider AI Programme (which will continue to oversee/coordinate AI activity across the FCA), the department will partner with business leads to shape and deliver work in priority areas — Authorisations, SPC, EMO and Anti‑Money Laundering.

Role responsibilities

• Define and own evaluation frameworks for GenAI outputs covering quality measures such as accuracy relevance robustness and hallucination rates
• Design curate and govern test datasets and benchmarks to ensure consistent model and solution performance assessment over time
• Support development of automated evaluation pipelines and operational reporting to embed assurance into delivery
• Identify assess and mitigate risks in model behaviour for example bias errors safety concerns and edge cases with clear escalation and control recommendations
• Manage delivery through well-defined work packages setting priorities operating standards and performance objectives including line management of Business Analysts
• Engage senior stakeholders to understand strategic priorities build a pipeline of scoped and prioritised projects and translate needs into analytics led solutions with clear business value
• Work in the public interest protecting 40 million UK consumers who rely on financial services and supporting long term economic growth from an industry contributing 12% of UK economic output
• Manage digital and data-led change by encouraging innovative experiments and working with senior stakeholders while empowering a diverse team to collaborate openly

View on member website