AWS Data Bricks Spark/Scala Engineer

Job Title: AWS Data Bricks Spark/Scala Engineer

Job ID: 2023-11933

Job Location: Morris Plains, NJ (remote to start); Remote, Memphis, TN (preferred), Morris Plains, NJ, Austin, TX (preferred), St. Louis, MO

Employment Type: W2

Duration: Long Term

Work Eligibility: All Work Authorizations are Permitted

Key Technology: DataBricks, AWS, Spark

Job Responsibilities:

• Drive mentality of building well architected applications for AWS Cloud.
• Drive the mentality of quality being owned by the entire team.
• Identify code defects and work with developers to address quality issues in product code.
• Find bottlenecks and thresholds in existing code through the use of automation tools.
• Articulate clear business objectives aligned to technical specifications and work in an iterative agile pattern daily.
• Ownership over your work task and comfortable interacting with all levels of the team and raising challenges when necessary.

Skills and Experience Required:

• 6-8 years of strong experience in Spark, Python, Shell Scripting, PostgreSQL, Hadoop, AWS (S3, EC2, SNS, SQS, Lambda, ECS, Glue, IAM, and CloudWatch), Databricks (Delta lake, Notebooks, Pipelines, cluster management, Azure/AWS integration).
• In-depth knowledge on Hadoop architecture and its components. Troubleshooting Hadoop Yarn jobs is a must. Implement Hadoop job orchestration using Shell scripting and Airflow.
• In-depth understanding of Spark Architecture including Spark Core, Spark SQL, Data Frames, Spark Streaming, RDD caching, Spark MLib.
• Strong experience in writing complex and effective SQLs and Stored Procedures. Strong query optimization skills to improve performance of ETL process.
• Strong hands-on cloud experience with AWS suite.
• Demonstrated work experience in Hadoop, Spark, and Hive.
• Strong development experience on Databricks components.
• Strong hands-on experience on Spark is a must.
• Work experience on Hadoop as data warehouse/Data Lake implementations.
• Moderate to senior Terraform development experience needed.
• Healthcare domain knowledge is a plus.