Information Technology - Lead Data Engineer

Job Description The lead data engineer is a senior software developer with strong softwareengineering skills who is responsible for building custom open-source-baseddata ingestion and MLOps platforms. He/she has deep appreciation of thecomplexity of the data engineering process, such as the challenges of dataingestion involving large or near-real-time datasets, the maintenance of highdata quality, and the importance of automation for increasing pipelinerobustness and reducing the need for human intervention. Responsibilities • Be an effective distributed-system implementer in the following coreactivities: o Design and develop data engineering services and their ecosystem usingdistributed databases (relational, columnar, graph, in-memory); orchestration(Apache Airflow); and distributed stream/batch data processing (Kafka, Kinesis,Spark). oDesign and develop MLOps production pipelines; provide technical support todata scientists/ML engineers by getting their ML/DL models deployed at scaleand meeting SLAs on both cloud and on-premises GPU and CPU instances. o Design data models for mission-critical, high-volume, near-real-time/batchdata; build idempotent/atomic production data pipelines to make data ingestionmore fault tolerant. o Design and develop intuitive, highly automated, self-service data platformfunctions for business users. o Design, build, and operate scalable and reliable data pipelines on theDatabricks platform. • Explore, evaluate and champion the introduction of next-generationtechnologies in the data-ingestion workflow. Participate in project planningand provide technical guidance on cloud architecture for data projects. Requirements • BS in Computer Science or other related discipline is required. Advanceddegrees in Computer Science (PhD, MS) are highly desirable. •5+ years of relevant industry experience in some or most of the followingtechnical areas: o Advanced programming skills in Python. Conversant with data structures andalgorithm design. o Experience in building data pipelines (including data collection,warehousing, processing, analysis, monitoring, and governance) usingopen-source data ingestion platforms. o Intermediate-level knowledge and experience with AWS cloud components andbest practices. Good understanding in deploying data stores such as S3,RedShift, Elasticache, PostgreSQL, and EMR. o Hands on experience with Databricks workspace, cluster management, AI Agentcapabilities, and job orchestration o Prior experience in modern software development is required (such as webfrontend UI, backend API microservices, understanding of CI/CD and Scrum/Kanbanagile development). Strong grasp on object-oriented or functional programming(using e.g. Python, Java, Scala, or C#).