Data Engineer - Commodities

Data Engineer - Commodities

Millennium is a top tier global hedge fund with a strong commitment to leveraging innovations in technology and data science to solve complex problems for the business. We are assembling a strong Commodity Technology team to build our next generation in-house commodity technology platform and associated ecosystem of tools, applications and systems.Commodity Technology provides a dynamic and fast-paced environment with excellent growth opportunities and projects involving cutting edge technologies.

Responsibilities

• Develop data and analytics platform using standard tools for batch and stream processing, container management, observability, and persistence
• Work closely with traders, quants, and other technologists globally to understand system development requirements
• Develop software solutions in an agile fashion using modern software development practices e.g. comprehensive testing, version control practices, CI/CD, etc.
• Test and deploy software solutions in an automated fashion using CI/CD practices
• Ensure DevOps style management and operation of DEV/UAT/PROD deployment environments

Mandatory Requirements

• 5+ years’ of professional experience with Python software development
• Experience with Airflow, event streaming (e.g., Kafka) and processing (e.g., Flink), big data table formats like Iceberg or Delta Lake, and Kubernetes
• Working knowledge of traditional and cloud SQL databases

Preferred Requirements

• Familiarity with quantitative finance and electronic trading concepts.
• Experience using AWS technologies such as S3, Athena, SQS, Batch, Lambda
• Knowledge of the various Python dependency management and packaging tools (e.g., Poetry) and data quality, observability, and testing tools (e.g., Great Expectations, Soda)