Senior Azure Data Engineer

Responsibilities

1. Architect, design, develop and engineer end-to-end data pipelines across multiple data sources and systems of record.
2. Ensure data quality, integrity, security, and completeness throughout the data lifecycle.
3. Develop, design data models, data structures and ETL jobs for data acquisition and transformation purposes.
4. Develop a deep understanding of the data sources, implement data standards, maintain data quality and master data management.
5. Develop data services and APIs.
6. Work closely with the cloud service providers to ensure completeness and alignment with the service offerings.
7. Manage and maintain cloud-based data and analytics platform.
8. Engage in quick proof of concepts and proof of value in prototyping data and analytics solutions and derive viability.
9. Understand Unified Data Governance and Data Quality Management solutions for managing and governing enterprise data.
10. Interact with business stakeholders to understand requirements and translate them into technology solutions.

Qualifications

1. Bachelor’s degree in computer science, computer engineering, or other related field of study with 10+ years of relevant experience.
2. Experience in Azure Cloud platform service offerings.
3. Expertise in Azure SQL with strong knowledge of writing complex SQL queries, stored procedures, tuning, and query optimizations.
4. Experience with Azure API Management Service (APIMS) and developing Azure Functions for exposing API/RESTful data services.
5. Experience working on real-time data capture, processing, transformation, and storing using technologies like Azure Event Hubs, Azure Service Bus, and Azure Stream Analytics.
6. Knowledge of creating pipelines and complex data transformations and manipulations using one of the languages .Net/C#, Python, Java, R, or Scala with Databricks/Spark.
7. Experience in NoSQL Databases and Big Data technologies including Azure CosmosDB or other Document stores.
8. Experience with Azure Purview for automating and managing Data catalog, classification, lineage, and metadata management across enterprise data.
9. Experience with configuring rules, exceptions, and approval workflows for Data Quality Management using Profisee.
10. Experience working with different data storage options including ADLS Gen2 and Azure BLOB storage.
11. Understanding of different data formats including Parquet, Avro, CSV, ORC, etc.
12. Understanding of MPP databases and maintaining large amounts of data processing is a plus.
13. Experience in creating, scheduling, and monitoring Azure Data Factory (ADF) pipelines.
14. Experience with Azure Synapse Analytics and Azure Analysis Services, Power BI, and business intelligence technologies is a big plus but not mandatory.
15. Past working experience in a fast-paced and agile environment.
16. Experience working with an on-shore/off-shore model.
17. Build and meet project timelines and manage delivery commitments with proper communication to management.
18. Collaborate and communicate with key business lines, technology partners, vendors, and architects.

Compensation

The base pay range for this position is USD $175,000.00/Yr. - USD $200,000.00/Yr. Exact offers will be determined based on job-related knowledge, skills, experience, and location.