AI Evaluation Engineer- 6 months - London - Inside IR35

Key Responsibilities

• Define and implement end-to-end evaluation strategy for generative AI conversational systems (LLMs, RAG pipelines, agents)
• Establish evaluation metrics and quality benchmarks for model and system performance
• Build test datasets and benchmark models (GPT variants, open-source LLMs) and prompt strategies against them
• Implement automated evaluation pipelines to support continuous testing and monitoring
• Conduct human-in-the-loop evaluation for qualitative validation of model outputs

What You Will Ideally Bring

• Strong understanding of LLMs, RAG architecture, and prompt engineering
• Python skills for data analysis and building evaluation pipelines
• Experience with prompt evaluation tools (eg PromptTools, DeepEval, or similar)
• Experience in evaluation framework design, NLP quality assessment, and test dataset/benchmark creation
• Experience with LLM output evaluation and scoring, plus safety, risk and compliance testing
• Experience with tooling and automation for evaluation pipelines and experimentation frameworks
• Familiarity with Responsible AI principles

Contract Details

• Duration: 6 months
• Rate: £600/day - Inside IR35
• Location: London EC1M