Senior Data Engineer (Databricks)

About the Role We are looking for an experienced Senior Data Engineer to design, build, and optimize scalable data platforms and pipelines that enable data-driven decision-making. You will work closely with our client's stakeholders to develop reliable, secure, and high-performance data solutions while driving best practices in Data Engineering, CI/CD, and Site Reliability Engineering (SRE). This role is ideal for engineers who are passionate about building modern data platforms, ensuring data quality, and delivering resilient cloud-based data solutions. Key Responsibilities Data Pipeline Development & Architecture Translate business data requirements into technical solutions. Design, develop, and maintain scalable batch and real-time data pipelines. Build data ingestion pipelines to collect, cleanse, transform, and integrate data from multiple source systems. Design reusable, scalable, and optimized data models to support analytics and reporting. Develop secure mechanisms for users and applications to access enterprise data. Research, evaluate, and implement new technologies to improve the organization's data infrastructure. Data Quality, Testing & Validation Design and implement data quality frameworks to ensure data accuracy, completeness, and consistency. Develop and execute functional, regression, and exploratory testing for data pipelines. Build automated testing capabilities to improve deployment quality and reduce production defects. Perform data reconciliation and end-to-end validation across multiple systems. Support User Acceptance Testing (UAT) and collaborate with business stakeholders to validate solutions. Operations, Governance & Continuous Improvement Monitor and maintain databases and ETL platforms, including performance tuning, capacity planning, troubleshooting, and issue resolution. Collaborate with infrastructure and security teams to ensure scalable and secure data platforms. Work with data stewards to establish and enforce data governance standards and best practices. Maintain metadata repositories and data catalogues. Implement data security controls, including access management, encryption, and data masking. Drive continuous improvement in CI/CD pipelines, automation, and Site Reliability Engineering (SRE) practices. Requirements Qualifications Bachelor's Degree in Computer Science, Information Technology, Software Engineering, or a related discipline. Minimum 5 years of experience in Data Engineering or related roles. Technical Skills Strong understanding of Data architecture Data modelling Data structures and algorithms Database design and data storage concepts Hands-on experience designing and maintaining batch and streaming data pipelines. Strong programming skills in Python SQL Shell Scripting Experience with workflow orchestration tools such as Apache Airflow Azure Data Factory Hands-on experience with Databricks. Experience working with cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform (GCP). Data Quality & Testing Experience implementing data quality frameworks and validation processes. Experience developing automated testing for data pipelines. Familiarity with defect tracking and test management tools such as JIRA. DevOps & CI/CD Experience building and maintaining CI/CD pipelines. Familiarity with Docker Git Terraform Understanding of infrastructure automation and deployment best practices. Experience implementing data governance, security, and compliance standards. Preferred Qualifications Candidates with the following experience will have an advantage: Government sector projects. Data governance, data infrastructure, or data security initiatives. Climate or Weather domain projects. Required Technical Skills Python SQL Shell Scripting Apache Airflow Azure Data Factory Databricks AWS / Azure / Google Cloud Platform (GCP) ETL Development Data Pipelines Data Modelling Data Quality Testing Data Governance CI/CD Docker Git Terraform Site Reliability Engineering (SRE) JIRA