Data & AI Platform Engineer

Design, build, productionize, and support scalable document parsing and content extraction pipelines on a modern hybrid cloud platform, handling structured and unstructured data including PDFs, PPTX, Excel, Word, images, scanned documents, videos, audios, charts, tables etc Design and implement OCR, VLM, and multimodal AI pipelines leveraging state-of-the-art models (e.g., GPT- [4/5]o, vision-language models, Text-to-Speech) for accurate content extraction, layout understanding, and table/chart interpretation Develop and optimize embedding models, including fine-tuning and domain adaptation to support organization-specific terminology, keywords, and semantic search use cases Implement knowledge ingestion, indexing, and retrieval pipelines for RAG (Retrieval-Augmented Generation) and agentic GenAI systems, ensuring high recall, precision, and low-latency retrieval Make an Impact by: Build and maintain data ingestion pipelines for batch and streaming data sources using tools/library like Kafka, LlamaIndex, LangChain, PyMuPDF, MSAL etc. Implement layout-aware parsing to accurately preserve document structure, hierarchy, and reading order (using OCR, VLM, and multimodal AI solutions). Extract accurate numeric and semantic data from complex charts and irregular tables while preventing hallucinations. Assist in integrating data from diverse source systems (files, APIs, databases, streaming) Implement, fine-tune, and optimize embedding models to support organization-specific terminology, improving semantic search, retrieval accuracy, and downstream GenAI performance. Help maintain metadata and pipeline documentation for transparency and traceability Participate in integrating pipelines with tools such as Microsoft Fabric, Databricks, Delta Lake, and other platform components Build and maintain knowledge base and RAG solution on cloud (Azure) Implement and operate knowledge base storage, lifecycle management and embedding/vectorization Contribute to automation efforts using version control and CI/CD workflows Apply basic data governance and access control policies during implementation. Skills for Success: Bachelor’s degree in Computer Science, Engineering, or a related field 1–3 years of experience in data engineering or data platform development. Fresh graduates encouraged to apply as well. Proven ability to independently build basic batch or streaming document processing pipelines Hands-on experience with Python, PyTorch, PEFT, LoRA, OCR, VLM, LLM, prompting, Text-to-Speech models and SQL for data transformation and validation Familiarity with Apache Spark (especially PySpark) and large-scale data processing concepts Experience with implementing knowledge base and RAG solutions for agentic AI use cases Self-starter with strong problem-solving skills and a keen attention to detail Able to work independently while collaborating effectively with senior engineers and other stakeholders Strong documentation and communication skills