Lead Databricks Engineer

About Persistent

We are a trusted Digital Engineering and Enterprise Modernization partner, combining deep technical expertise and industry experience to help our clients anticipate what’s next. Our offerings and proven solutions create a unique competitive advantage for our clients by giving them the power to see beyond and rise above. We work with many industry-leading organizations across the world, including 12 of the 30 most innovative global companies, 60% of the largest banks in the US and India, and numerous innovators across the healthcare ecosystem.

Our disruptor’s mindset, commitment to client success, and agility to thrive in the dynamic environment have enabled us to sustain our growth momentum by reporting $328.2M revenue in Q1 FY25, delivering 16% Y-o-Y growth. Our 23,500+ global team members, located in 19 countries, have been instrumental in helping the market leaders transform their industries. We’re also pleased to share that Persistent has been recognized as a Leader in the ISG Provider Lens™ Digital Engineering Services Quadrant Report for the U.S. and Europe 2024. We were also recognized for excellence in governance and executive leadership in Institutional Investor’s 2024 Asia Executive Team Survey . Throughout our market-leading growth, we’ve maintained a strong employee satisfaction score of 8.2/10.

About Position:

Role: Lead Databricks Engineer

Location: Iselin NJ (Hybrid 2-3 days from office)

Experience: 12+

What You'll Do:

Strong knowledge on data bricks architecture and tools

Have experience of task and wf jobs creations in data bricks.

Deep understanding of distributed computing and how to use spark for data processing.

SQL and Pyspark : strong command over querying databases and proficiency in pyspark.

Cloud platform: Preferred Azure for data bricks deployment.

5+ years of experience in data engineering with Databricks and Spark

Proficient in SQL and Python and Pyspark

Experience with Azure Databricks Medallion Architecture with DLT, Iceberg

Financial/Corporate Banking context would be a plus

Experience with data integration and ETL tools, such as Azure Data Factory

Experience with Azure cloud platform and services

Experience with data warehouse and data lake concepts and architectures

Good to have experience with big data technologies, such as Kafka, Hadoop, Hive, et

Expertise You'll Bring:

Design, develop, and maintain data pipelines using Databricks and Spark, and other cloud technologies as needed

Optimize data pipelines for performance, scalability, and reliability

Ensure data quality and integrity throughout the data lifecycle

Collaborate with data scientists, analysts, and other stakeholders to understand and meet their data needs

Troubleshoot and resolve data-related issues, and provide root cause analysis and recommendations

Document data pipeline specifications, requirements, and enhancements, and communicate them effectively to the team and management

Create new data validation methods and data analysis tools, and share best practices and learnings with the data engineering community

Implement ETL processes and data warehouse solutions, and ensure compliance with data governance and security policies

Benefits:

Competitive salary and benefits package

Culture focused on talent development with quarterly promotion cycles and company-sponsored higher education and certifications.

Opportunity to work with cutting-edge technologies.

Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards

Annual health check-ups

Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents

Our company fosters a values-driven and people-centric work environment that enables our employees to:

· Accelerate growth, both professionally and personally

· Impact the world in powerful, positive ways, using the latest technologies

· Enjoy collaborative innovation, with diversity and work-life wellbeing at the core

· Unlock global opportunities to work and learn with the industry’s best

Let’s unleash your full potential at Persistent - persistent.com/careers