Databricks (Delta Lake, Pyspark) AI Dev

Position Overview The Data & AI Engineer role is responsible for building scalable data pipelines, developing AI agents, and delivering end-to-end data and AI applications on the enterprise data platform, with a primary focus on Databricks. This role bridges data engineering, AI engineering, and data science to deliver production-grade solutions that power analytics, automation, and intelligent applications across Suntory Beverage & Food’s global operations. As part of the GHQ AAA team, you are expected to contribute to the organisation’s enterprise AI architecture by delivering reusable, scalable, and high-performance data and AI solutions. Key Responsibilities Data Pipeline Development Design, develop, and maintain scalable data pipelines using Databricks (Delta Lake, PySpark, workflows). Build robust ETL/ELT processes to ingest, transform, and serve data from multiple enterprise sources (SAP, external data, APIs). Ensure data quality, reliability, and performance optimisation across pipelines. Implement data models aligned with analytics and AI use cases (e.g., feature-ready datasets). Collaborate with data governance teams to ensure compliance with enterprise data standards. AI Agent Development Design and develop AI agents and GenAI solutions (e.g., knowledge assistants, automation agents) using Databricks and Azure AI capabilities. Implement Retrieval-Augmented Generation (RAG), prompt engineering, and orchestration logic for enterprise AI use cases. Integrate agents with enterprise data sources, vector databases, and APIs. Collaborate with business teams to identify and deliver AI-driven automation and productivity use cases. Ensure scalability, performance, and responsible AI practices in agent deployment. Databricks App Development Develop end-to-end data and AI applications using Databricks (e.g., notebooks, dashboards, apps, APIs). Build interactive analytics or AI-driven applications for business users (e.g., demand planning tools, decision support apps). Expose data and AI services via APIs (e.g., Databricks Model Serving, API integration). Work with front-end or BI teams (e.g., Power BI, apps) to integrate backend logic into user-facing solutions. Optimise application performance and ensure scalability for enterprise usage. Data Science Development Support development of machine learning models (forecasting, optimisation, classification). Perform feature engineering and data preparation for modelling. Collaborate with Data Scientists to productionise models within Databricks environment. Integrate trained models into pipelines and applications for real-time or batch inference. Ensure alignment between modelling outputs and business requirements. Experiences & Skills Requirements Bachelor’s or Master’s degree in Computer Science, Data Engineering, Data Science, or related field. 1–3 years of experience in data engineering, AI engineering, or related roles. Strong hands-on experience with Databricks (Delta Lake, PySpark, workflows, model serving) . Proficiency in Python and SQL for data processing and application development. Experience building data pipelines and ETL processes in large-scale environments. Practical experience in AI/GenAI development (LLMs, RAG, prompt engineering) . Familiarity with API development and integration for data and AI services. Understanding of machine learning concepts and model deployment workflows . Experience with MLOps / DataOps practices (CI/CD, pipeline automation, monitoring) is preferred. Knowledge of Azure ecosystem (ADLS, ADF, Azure AI, API Management) is a plus. Strong problem-solving skills and ability to work across cross-functional teams. Experience in FMCG, supply chain, or commercial analytics is an advantage.