Data Engineer II (Pipelines and Python)

Data Engineer II, Data Pipelines

Recorded Future harvests data from many public web sources and structured data feeds. Data rich web sources such as public forums, social media sites and digital marketplaces provide cornerstone information for current cyber intelligence. As an engineer, you will join a team of motivated people working on building reliable data pipelines that ingest raw data and convert it into actionable analytics to identify and combat cyber security threats.

What you’ll do as a Data Engineer:

• You will lead development, productionalization and maintenance of high volume data extraction via custom harvesters, APIs, and data stores.
• Create well monitored production data pipelines which convert raw data into well organized and actionable information.
• Maintain and upgrade existing data flow production systems.
• Collaborate with junior engineers and senior engineers through design reviews, code reviews, pair programming, and other software engineering best practice collaboration.

What you should bring to the Data Engineer role:

• 3+ years of python programming: You are comfortable writing production-grade data processing infrastructure in python. Your debugging skills are strong.
• 3+ years of high volume data management: You have a strong skill set for working with large, complex data structures. You have experience transforming large datasets in Python, querying with modern datastores such as MongoDB and ElasticSearch, and developing corresponding APIs.
• Well versed in both computer programming and data engineering: You want to review as well as question existing code, and can suggest modifications and improvements.