About the role
We are looking for a skilled & experienced Software Engineer to join our AV Infrastructure & DataOps, focusing on critical automation tools and pipelines for our autonomous vehicle software stack. This team is at the forefront of our efforts to streamline and optimize our processes for the development, validation & deployment of Gatik’s autonomous vehicle software. As a Data Engineer, you will be responsible for developing and maintaining scalable data pipelines, ensuring efficient data collection, processing, and storage. You will optimize workflows, maintain data security and compliance, and support the data needs of the organization. This role requires strong Python skills, experience with data storage solutions and processing frameworks, and familiarity with cloud platforms. Strong problem-solving, communication, and collaboration abilities are essential.
As a member of our team, you'll have the opportunity to work on cutting-edge technologies and contribute to groundbreaking projects that are shaping the future of transportation. If you're passionate about automation, innovation, and making a meaningful impact, we invite you to join us on this exciting journey.
This role is onsite 5 days a week at our Mountain View, CA office!
What you'll do
- Create, optimize, and maintain scalable data pipelines to collect, process, and store large volumes of data from various sources.
- Implement ETL (Extract, Transform, Load) processes to ensure data quality and integrity.
- Design and implement data architectures that support efficient data storage, retrieval, and analysis.
- Integrate data from multiple sources, including APIs, databases, and third-party services.
- Optimize data processing workflows for maximum efficiency and speed.
- Implement techniques such as data partitioning and indexing to improve query performance.
- Ensure data security, privacy, and compliance with relevant regulations.
- Implement access controls, encryption, and other security measures.
- Work closely with data scientists, analysts, and other stakeholders to understand data needs and provide necessary support.
- Develop tools and frameworks to enable data analysis and reporting.
- Monitor the performance and health of data systems and pipelines.
- Troubleshoot and resolve issues to ensure data availability and reliability.
- Maintain comprehensive documentation of data systems, pipelines, and processes.
- Ensure documentation is up to date and accessible to relevant team members.
- Stay updated on the latest data engineering tools and technologies.
- Evaluate and recommend new tools and technologies to improve data engineering processes.
What we're looking for
- 5-7 years of industry experience in data engineering or related roles.
- Strong proficiency in Python, with experience in developing data pipelines and ETL processes.
- Experience with data storage solutions (e.g., SQL, NoSQL, cloud storage).
- Proficiency with data processing frameworks (e.g., Apache Spark, Apache Beam).
- Familiarity with cloud platforms (e.g., AWS, Azure, Google Cloud).
- Strong problem-solving skills and the ability to troubleshoot complex data issues.
- Ability to optimize data workflows for performance and scalability.
- Excellent communication and collaboration skills.
- Ability to work effectively in a fast-paced, dynamic environment.
- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
Bonus
- Experience with containerization and orchestration tools (e.g., Docker, Kubernetes).
- Familiarity with data visualization and business intelligence tools (e.g., Tableau, Power BI).
- Knowledge of machine learning and data science concepts.