Azure Data Engineer - Databricks

Responsibilities:

1. Assist with deployment, configuration, and management of Azure Cloud environment.
2. Assist with migration efforts of existing ETL jobs into Azure/Databricks cloud environment.
3. Assist ETL staff with following best practices for Cloud, maximizing efficiency of cloud resources, and general knowledge sharing of cloud expertise with AA Cloud Staff.
4. Assist with operationalizing deployments and support of Cloud services for ETL Operations, including standardizing and automating processes and workflows, creating documentation/knowledge articles, and assisting Operations staff with limited experience in Cloud.
5. Provide written and oral presentations to high-level management on the status of current efforts.
6. Develop and implement dashboards to monitor performance, executor health, and credits consumption.
7. Configure alerts when job runtimes exceed thresholds or data latencies are higher than expected.
8. Develop procedures for monitoring dashboards and alerts.
9. Integrate Databricks/Azure monitoring/alerting within the monitoring environment.
10. Provide advisory services and recommendations for issues that arise during the period of performance.

Education: A degree from an accredited College or University in the field of services is preferred.

Required Experience: 10+ years' experience with at least 6+ years hands-on experience in the following Required Technical Skills:

Required Technical Skills:

1. Strong skills and experience in Cloud Operations support in Azure.
2. Strong experience supporting large-scale/enterprise-level Cloud environments.
3. Strong experience supporting several cloud services related to compute, network, databases, etc.
4. Experience in using Databricks or other Spark-based platforms.
5. Knowledge of or experience in using any of the following: Sqoop, Oozie, Flume.
6. Fluency in at least one scripting language (Python, Perl, Ruby, or equivalent).
7. Ability to troubleshoot cloud resource problems and perform complex system tests.
8. Develop and maintain accurate documentation for internal procedures and services.
9. Ability to automate solutions to repetitive problems/tasks.
10. Ability to use a wide variety of open-source technologies and cloud services.
11. Integration of Git in continuous deployment.
12. Experience with DevOps monitoring tools.
13. Ability to break down complex networking concepts and present them concisely.
14. Shell scripting experience.
15. PowerShell experience.
16. Knowledge of best practices and IT operations in an always-up, always-available service.
17. Creative thinking skills.
18. Detail-oriented personality.
19. Ability to communicate well with other members of the development team.
20. Strong communication skills (verbal/written) to deliver technical insights and interpret data reports to clients, also aiding in understanding and serving client requirements.

Strong understanding of cloud networking at scale.