QA Automation Engineer

About the Role We are seeking a highly skilled QA Automation Engineer to safeguard the quality, reliability, performance and safety of our AI-powered data products. In this role, you will architect intelligent, test automation frameworks, design rigorous evaluation systems for ML/LLM outputs and build quality infrastructure that scales alongside rapidly evolving AI capabilities. You'll be a key quality gatekeeper for AI products used in production, ensuring models behave safely, consistently, and as intended — before, during, and after release. Roles & Responsibilities Test Automation & Framework Engineering Architect and build scalable, modular automation frameworks using Python, Pytest, Playwright, Selenium, or other custom-built tooling. Automate testing across UI, APIs, microservices, and AI models to enable fast, high-confidence releases. Build internal bots or agents that auto-generate test cases and run intelligent, AI-assisted exploratory testing. Champion best practices in test architecture, design patterns and maintainable, reusable test code. AI/ML & LLM Evaluation Own end-to-end test strategy and execution for AI/ML models, GenAI, LLMs, and multi-agent systems, spanning functional, adversarial, safety, and security testing. Design evaluation frameworks to measure hallucination rate, bias, toxicity, and prompt-response consistency. Curate and maintain golden datasets; apply both deterministic rule-based checks and LLM-as-a judge techniques to evaluate output quality at scale. Data Validation Validate data pipelines, training/inference data quality, feature stores, schema contracts to prevent data leakage or drift. Verify embedding quality, RAG retrieval relevance and the accuracy of agentic tool- and function calling. CI/CD & Production Reliability Embed automated test suites into CI/CD pipelines, defining data-driven release gates based on AI quality metrics. Establish real-time production monitoring and dashboards to detect model drift, concept drift and anomalies. Continuously triage failures and tune performance to reduce pipeline execution time. Stakeholder Collaboration Partner with Data Scientists, ML Engineers, and Product Managers to define measurable acceptance criteria. Translate testing and quality metrics into actionable outcomes for both technical and business stakeholders. Who we are looking for Work Qualifications At least 5 years of hands-on experience in QA, Test Automation, or SDET roles, including some experience in testing AI/ML, LLM or agentic systems. Technical Experience Programming & Frameworks Strong proficiency in Python with solid software engineering fundamentals Hands-on experience with Pytest, unittest, or equivalent test frameworks; comfortable writing maintainable, well-documented test code. Proficient in SQL for querying, validating and profiling test data. Comfortable working with structured/semi-structured data formats and large-scale datasets Automation Expertise Proven experience with API and UI test automation Experience designing end-to-end, data-driven and keyword-driven automation frameworks. Testing Proficiency Strong grasp of test design techniques Experience with risk-based testing and defining measurable quality/exit criteria Functional Skills Excellent communication skills with the ability to translate complex technical concepts for both technical and non-technical stakeholders Comfortable working in cross-functional, fast-moving environments where priorities may evolve.