Data Engineer

Key Responsibilities Integration & Workflow Engineering Analyse and re-platform existing workflow-based integrations from one platform to another while maintaining functional parity. Implement orchestration logic including triggers, conditional routing, retries, exception handling, and state management. Design and implement automation patterns to handle platform limitations (e.g., looping, batching, pagination, throttling). Ensure workflows are idempotent, fault tolerant. Support environment-based deployments (DEV / UAT / PROD) with configuration-driven designs. Data Engineering & ETL Design and build ETL pipelines for ingesting flat files (e.g., CSV) into relational databases. Handle schema validation, basic schema evolution, data quality checks, and error reconciliation. Optimize data ingestion for performance, scalability, and reliability. Collaborate with application teams to understand upstream and downstream data dependencies. Cloud & Big Data Platform Contributions Build and maintain data pipelines on cloud-based big data platforms using distributed processing frameworks. Contribute to Lakehouse-style data storage that supports both batch and streaming data. Work with modern table formats that support incremental processing, versioning, and historical queries. Support use cases such as append-heavy datasets, high-write event data, and analytical queries. Operations, Quality & Observability Implement logging, monitoring, and alerting for workflows and data pipelines. Support operational readiness including runbooks, deployment procedures, and rollback strategies. Participate in root-cause analysis and continuous improvement of data pipelines. Ensure adherence to data governance, security, and compliance standards. Required Skills & Experience Technical Skills Strong experience with workflow orchestration / integration platforms. Solid understanding of ETL concepts and hands-on experience with file-based ingestion. Proficient in SQL and working knowledge of relational databases (e.g., SQL Server or equivalent). Experience with distributed data processing frameworks (e.g., Spark). Familiarity with streaming and batch data processing concepts. Practical experience with cloud platforms and managed data services. Understanding of CI/CD principles for data and integration workflows. Programming & Automation Experience with scripting or programming languages commonly used in data engineering (e.g., Python). Ability to build reusable utilities for batching, retries, pagination, and error handling. Experience with REST APIs and structured data formats (JSON, CSV). Desired (Good to Have) Exposure to modern Lakehouse table formats supporting incremental processing and time travel. Experience with stream processing engines (e.g., Flink or Spark Streaming). Familiarity with query engines used for analytical access. Experience working in regulated environments (banking, financial services, or manufacturing). Knowledge of data observability and quality frameworks. Experience Level Typically, 3–5 years of relevant experience in data engineering, integration engineering, or similar roles. Proven ability to work independently on moderately complex problems while collaborating within a larger team. Soft Skills & Competencies Strong analytical and problem-solving skills. Ability to understand existing systems and translate requirements into working solutions. Clear communication with technical and non-technical stakeholders. Detail-oriented with a focus on reliability and maintainability. Comfortable working across multiple teams and applications.