[CLOSED] Management of MSO Data Lake

You will assist in managing and optimising the data architecture and analytical environment that the Data Analytics team uses on a day to say basis. This role involves: i) Streamlining redundant / duplicated code scripts / repositories into common functions / packages that can be immediately leveraged by the team, ii) Coordinating with internal IT departments, external IT vendors (AWS) to enable solutions that meet the team’s data requirements, and iii) Implementing improvements in the data pipeline for municipal insights, ensuring fully automated data cleaning and processing for downstream data products and report. The responsibilities include: i) Identifying core functions commonly used by the team across all scripts and create modular functions that the team can ‘plug and play’ within their own scripts without having to write their own code. ii) Understand and map out the requirements of the Analytics team, and work with our IT department (CISD) and AWS (IT vendor) to enable these services (i.e. LLM functionality, Machine Learning deployment) for the team’s usage. iii) Convert existing Python scripts into AWS Glue jobs so that these scripts can be executed automatically without human intervention. The ideal candidate should be fluent in Python programming language and technical knowledge of AWS data services (i.e. Athena, Glue, S3).