SOFTWARE ENGINEER ( HADOOP )

Looking for a Software Engineer with knowledge in Hadoop and Spark. Experience with data mining and stream processing technologies (Kafka, Spark Streaming, Akka Streams).

Responsibilities:

1. Translate functional and technical requirements into detailed specifications running on AWS using services EC2, ECS, RDS Aurora MySQL, SQS, SNS, KMS, Athena.
2. Migrate current Prometheus and DARQ applications to manage multi-account AWS cloud environment containers, scalable computing platforms Docker (ECS), Kubernetes.
3. Develop Spark code using Scala and Spark-SQL/Streaming for faster testing and processing of data.
4. Create, optimize, and troubleshoot complex SQL queries to retrieve and analyze data from databases such as Redshift, Oracle, MS SQL Server, MySQL, and PostgreSQL.
5. Design ETL transformations and jobs using Pentaho Kettle Spoon designer 5.7.12 and Pentaho Data Integration Designer and schedule them on ETL WFE application Carte Server.
6. Design, code, test, and customize RHQ reports for Market systems data and provide data quality solutions to meet client requirements.
7. Develop various complex queries for different data sources Nasdaq Data Warehouse, Revenue Management System and work on performance tuning of queries.
8. Create scripts for automation processes for data ingestion.
9. Build and deploy artifacts (RPMs) and services using GitLab pipelines to Dev, QC, and Prod AWS accounts in cloud platform.
10. Validate DARQ data, reports and manage document libraries on collaboration site using Confluence.

Requirements:

The minimum education requirements to perform the above job duties are a Bachelor’s degree in Computer Science, Information Technology, or a related technical field.