Zapata Technology is seeking an experienced Data Engineer for a hybrid role in San Diego, California. The Data Engineer will lead the migration of our data infrastructure from Confluent with manual ETL processes to Apache NiFi. This role will involve working with PostgreSQL, MongoDB or Cassandra, and optionally Neo4j, within a Kubernetes environment. The ideal candidate will have extensive experience in data engineering, ETL processes, and managing data pipelines in cloud-native environments.
This role is contingent upon contract award.
Compensation: 125-150K
Responsibilities:
- Plan, coordinate, and execute the migration from Confluent with manual ETL processes to Apache NiFi, ensuring minimal downtime and data integrity during the migration.
- Design and implement automated ETL workflows using Apache NiFi to improve the efficiency and reliability of data processing.
- Work with PostgreSQL, MongoDB, or Cassandra to ensure seamless data integration and management, optimizing database performance and reliability.
- Deploy and manage all components within a Kubernetes environment to ensure scalability, high availability, and efficient resource utilization.
- Collaborate closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions while documenting processes, workflows, and configurations.
- Implement monitoring solutions to track the performance of data pipelines and databases, continuously optimizing workflows for performance and cost-efficiency.
Qualifications:
- Proven experience in data engineering, with a focus on ETL processes and data migration projects.
- Proficiency in Apache NiFi for ETL automation.
- Strong knowledge of PostgreSQL, MongoDB, or Cassandra.
- Familiarity with Neo4j and its integration (preferred but not mandatory).
- Experience with Kubernetes for container orchestration.
- Understanding of data architecture, data modeling, and data integration best practices.
Soft Skills:
- Excellent problem-solving abilities and attention to detail.
- Strong communication skills, with the ability to work collaboratively in a team environment.
- Ability to manage multiple tasks and projects simultaneously, meeting deadlines effectively.
Preferred Qualifications:
- Experience in cloud platforms such as AWS, Google Cloud, or Azure.
- Familiarity with additional data processing tools and frameworks.
- Certification in Kubernetes or related technologies.
- Ability to incorporate Neo4j into the data architecture if required.
Bachelor’s degree in Information Technology or a related field, and two to three years of experience is preferred.
Position Type:
Full time position
Clearance:
Secret security clearance. Employment is contingent on having or obtaining the required active security clearance or successfully passing the required background check, as well as other factors, including, but not limited to, drug screens.
AAP/EEO Statement:
Equal Opportunity Employer – M/F/Disabled/Veteran
Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, sexual orientation, gender identity, disability or protected veteran status.
Zapata Technology, Inc. provides reasonable accommodation to applicants who are veterans or who have disabilities and are unable to fully use our company application system.
#J-18808-Ljbffr