Sr Data Engineer - Pyspark / Databricks

Dice is the leading career destination for tech experts at every stage of their careers. Our client, Orion Business Innovations Careers, is seeking the following. Apply via Dice today!

Sr Data Engineer - Pyspark / Databricks/Senior PySpark Developer

1. The Senior PySpark Developer should have at least 8 years of extensive experience in designing, building, and deploying Python and PySpark-based applications.
2. Familiarity with Databricks and Delta Lake is essential. Experience in designing, developing, deploying, and testing within these platforms is crucial.
3. Proficiency in ETL processes using Python, PySpark, and Azure data factory.
4. Knowledge of Azure services such as ADLS, Synaptics, PowerBi and MS Fabric will be advantageous.
5. Develop and maintain data processing pipelines using Spark in Databricks notebooks. Optimize data workflows for performance, scalability, and reliability. Experience in finetuning application performance required.
6. Work closely with team of Analysts, pyspark developers / QA to understand data requirements and deliver actionable insights.
7. Ensure compliance with data governance and security standards.

1. Strong proficiency in Python and PySpark.
2. A Senior PySpark Developer should have a solid understanding of system design principles and experience with data modeling.
3. Experience with Apache Spark and big data processing.
4. Effective verbal and written communication skills are essential for presenting detailed designs and collaborating with cross-functional teams.
5. Combine technical expertise with problem-solving passion to deliver technical solutions for large-scale transformation projects.

1. Data Lake/ ADLS Gen 2
2. Azure Databricks
3. PySpark, Python
4. Azure data factory