AI Evaluation Engineer

Develop and maintain AI evaluation frameworks, regression test suites and automated evaluation pipelines. Build datasets, quality dashboards and evaluation tools to improve the performance of AI agents and LLM-powered applications. Design evaluation methods such as LLM-as-a-Judge, multi-turn conversations, tool evaluation and agent workflow analysis. Integrate AI evaluation into CI/CD and model release processes. Analyse production issues and improve datasets, prompts, evaluation logic and AI workflows. Conduct experiments to optimise model accuracy, retrieval performance, latency and cost. Build reusable platform capabilities and developer tools to support multiple AI products. Prepare technical documentation and collaborate with cross-functional engineering teams. Requirements Bachelor's Degree in Computer Science, Software Engineering or a related discipline. Minimum 2 years of relevant experience in AI Platform Engineering, LLMOps, Machine Learning Engineering or Software Engineering. Experience with most of the following technologies: LangSmith, LangGraph / LangChain, FastAPI, Temporal, Grafana, Redis, React / TypeScript, EvalsHub or similar AI evaluation tools Experience with LLM evaluation, Retrieval-Augmented Generation (RAG), semantic search, embeddings or model benchmarking is preferred. Strong analytical and problem-solving skills. Good communication and stakeholder management skills. Able to work independently and collaboratively in a fast-paced environment. Interested candidates are encouraged to submit their resumes outlining their relevant experience and achievements to sherriyang(@)talentvis.com or click apply! **We regret to inform that only shortlisted candidates would be notified** EA License No: 04C3537 EA Personnel No: R22106683 EA Personnel Name: Yang Hui Shan, Sherri