Senior Data Engineer

Technical Requirements:

• Skilled in designing and building complex distributed systems.
• Proficient in modern batch and streaming data ecosystems.
• Expertise in Java, Scala, or Python, along with SQL and Bash.
• Capable of leading, organizing, and managing team projects.
• Experienced in project management and reporting.
• Mentorship abilities for junior team members on engineering best practices and code quality.

Responsibilities:

• Develop and oversee data platform and assist its users.
• Collaborate with Traders and Researchers to identify and integrate new data sources for analysis to enhance trading opportunities.
• Design and create data solutions for discovering, purchasing, organizing, tracking, and managing data rights and quality to meet the strategic needs of trading teams.
• Continuously monitor data ingestion pipelines and data quality to ensure system stability, reliability, and data integrity. Also, contribute to the development and enhancement of monitoring and quality control systems and processes.

Qualifications:

• Over 7 years of experience in modern data technologies and building data-heavy distributed systems.
• Advanced proficiency in Java, Scala, or Python with a track record of producing high-quality, maintainable code.
• In-depth knowledge of SQL and Bash.
• Experience in utilizing and developing cloud-native technologies for scalable data processing.
• Previous work with both batch and streaming data systems, with a clear understanding of their limitations.
• Familiarity with various data processing technologies like Flink, Spark, Polars, Dask, etc.
• Knowledge of diverse data storage technologies such as S3, RDBMS, NoSQL, Delta/Iceberg, Cassandra, Clickhouse, Kafka, etc.
• Proficiency with various data formats and serialization systems including Arrow, Parquet, Protobuf/gRPC, Avro, Thrift, JSON, etc.
• Experienced in managing intricate data ETL pipelines using tools like Kubernetes, Argo Workflows, Airflow, Prefect, Dagster, etc.
• Prior experience in schema governance and managing schema evolution.
• Experience in developing quality control processes to identify and rectify data discrepancies.
• Motivation to mentor junior team members and uphold high standards of engineering practices and code quality.
• Strong problem-solving skills in technical contexts.
• Demonstrated ability to thrive in an agile, fast-paced environment, managing multiple tasks and projects efficiently in a trading context.