Data Engineer (1409)

Summary:

The Data Engineer is part of a skilled IT team that will work independently and collectively with other engineers to identify and test potential enhancements and resolve technical issues with the rapid scaling of an intelligence application.

Essential Functions:

• Collaboratively develop and implement custom-developed ETL scripts; and support software products as needed.
• Ensure data quality, consistency, and integrity throughout the pipeline.
• Create, monitor, and adjust data ingestion flows.
• Process raw data by transforming it into a usable form (ETL).
• Rapidly create data pipelines to support training exercise with little prior notification to connect fielded systems to cloud-based infrastructure as well as other fielded systems.
• Develop for and work with the ZIngest system, an ingestion system developed by Zapata Technology.
• Perform System Administration functions such as database management, data management, and fixing issues that arise, including the installation of peripheral software and software updates.
• Perform Software Engineering functions such as full life cycle database design and software development systems analysis to include requirements gathering; development; testing; documentation; supporting technical writers and testers; configuration; maintenance of developed software projects.
• Manage lifecycle development of software, gaining and understanding requirements, executing actual development, documentation, maintenance, and expansion of deliverables as the need arises.
• Implement rapid features and API capabilities to meet evolving customer needs for data access and dissemination.
• Interface with the customer(s) to create ingestion feeds for unique systems with ability to implement solutions with few prior known details/documentations about said systems.
• Use source-code control to track and protect changes to the code baseline.
• Perform Infrastructure Management functions such as: Design, architect, and implement scalable infrastructure on multiple IC networks and application “nodes” using DevSecOps practices. Manage containerization efforts (Docker, Podman) and orchestration. Upgrade and migrate applications to containers to allow the required scalability and move legacy monolith applications to a microservices type environment. Document Infrastructure administration, tear down, and creation.
• Assist with Automation and Security tasks: Implement Infrastructure as Code (IaC) using Puppet, Ansible or similar technology to meet security mandated change management guidelines. Ensure application and database compliance with DISA and GISA standards and regulations. Design and implement API security to protect unauthorized access to data and reduce exposed endpoints. Review API code and designs to identify potential security risks. Implement Role Based Access Control (RBAC) for APIs to meet basic security requirements. Document API endpoints using OpenAPI documentation standards. Work with Cyber-Security personnel to document security vulnerabilities and any mitigations to reduce risks that cannot be removed.

Job Qualifications:

• Knowledge of Agile programming and development concepts including Jira or similar.
• Knowledge of DevSecOps processes including Infrastructure as Code using Puppet, Ansible, etc.
• Knowledge of containerization and container orchestration including networking using Docker, Podman, or Kubernetes.
• Experience with SIGINT and All Source Intelligence reports standards such as USMTF and lifecycle requirements management development.
• Experience with the full DataOps lifecycle including ingestion and ETL processes into databases, data lakes, or data warehouses.
• Ability to look at the big picture while evaluating software requests in order to determine where existing requirements intersect or can be used in conjunction with new products as needed.
• Ability to estimate the technical debt of the creation of new features or systems.
• Interpret specifications, troubleshoot, and define needed software solutions.
• Detail-oriented and organized; able to understand information systems and ensure accuracy of work.
• Grasp of Software Engineering concepts, especially scripting in Python and Groovy.
• Linux System administration knowledge: know the Linux command line, service management, software configuration concepts in lieu of virtual machine constraints.
• Ability to work with the customer(s) to find out specifications and formats for the data.
• Ability to convert raw data into a human-readable and easily viewable format.
• Ability to convert data and fix malformed data. This requires an understanding of XML, JSON, and regular expressions.

Preferred Qualifications:

• An understanding of pub-sub messaging systems including RabbitMQ, Pulsar and Kafka.
• A solid understanding of and experience in NiFi data flow, from configuration to creating new flows and making new processors.
• Experience with troubleshooting the NiFi software.
• Knowledge of NiFi, with ability to keep the cluster running.
• Ability to write scripts in Groovy and Python within NiFi and Linux.
• Ability to write new processors in NiFi using Java.

Certifications Required:

• IAT Level III certification is needed for this role, but having a level II with ability to obtain level III in agreed upon time may be accepted.

Education/Experience include:

A bachelor’s degree with 9 to 12 years of experience in performing similar or related work is required. Four years of experience may be substituted for a bachelor’s degree.

Clearance Type:

TS/SCI clearance required.

AAP/EEO Statement:

Zapata Technology is an Equal Opportunity/Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard to race, color, genetic information, creed, religion, sex, pregnancy, sexual orientation, gender identity, national origin, age, protected veteran status, or disability status.

AAP/EEO Employer: Minorities/Females/Disabled/Veterans