Duties and Responsibilities
- Create data pipelines with Python, SQL, and Spark
- Identify opportunities for improving existing systems to ensure scalability
- Improve visibility into existing data, making it easy to query and visualize
- Design and build schema and access plans for backend features that require an internal data store
- Perform QA with all code, including unit, integration, and / or regression tests
- Create data quality checks for ingested and post-processed data
- Ensure alerting and monitoring of automated pipelines
- Maintain and enhance existing data products
- Bring new perspectives to data engineering problems and take initiative
- Maintain a clean and secure python environment
Skills
- Fluent in SQL
- Proficient with python; understands the importance of maintaining
- Comfortable with python dependency management tools (conda, venv, etc.)
- Capable of building and running cost-efficient Spark jobs
- Experience with Cloud Technologies (AWS, Azure, or GCP), AWS preferred (Kinesis, SQS, Redshift, Athena, S3, and Lambda)
- Understands database systems, both relational (postgres, mysql, etc.) and non-relational (mongodb, cassandra, etc.)
- Familiarity with event systems (kafka, SQS, etc.)
- Capable of auditing and securing data
- Experience with data analytics platforms like Databricks or equivalent platform
- Proficient with git, GitHub, pull requests, continuous integration and continuous deployment.
What we offer:
- A collaborative and supportive work environment, recognized as a Great Place to Work.
- Cell phone allowance of $100
- Monthly parking pass reimbursed
- Medical, dental, and visions benefits
- Life insurance
- Unlimited PTO
- Bring your pet to work
- Your pet insurance is covered (up to $100)
- 401k with Company match
- Annual performance-based bonus
Spot Pet Insurance is the fastest growing pet insurance company in North America. Our commitment to an exceptional end-to-end customer experience and our data-driven approach have quickly established us as a leading pet insurance provider. We're dedicated to providing pet parents with peace of mind by offering accessible and comprehensive coverage so their furry companions can lead happier, healthier lives. To demonstrate this, we recently joined forces with MrBeast to find homes for 100 homeless pets and committed to giving each of them pet insurance for life! Along the way, we’ve created a company culture that allows our employees to thrive, with perks like daily free meals, a pet-friendly office, and ridiculously fun company events every quarter. Our dedication to fostering a positive and rewarding work environment for our team has even earned us a Great Place to Work certification.
The ideal candidate is passionate about designing cost efficient and robust data pipelines. The data engineer will work on SPOT’s backend team and work directly with the backend engineers on building and improving data warehouses, access, and processes. They will also be expected to provide guidance and direction for improving or replacing existing data workflows for the data team.