Data & AI Engineer (Python)

Job Description:

As the Data & AI Engineer, we want you to join our innovative team, working with our client, an innovator in the social media world! In this role, you will design, build, and maintain robust and efficient data pipelines that collect, process, and store data from various sources, including user interactions, listing details, and external data feeds!

Key Responsibilities:

• Develop data models that enable the efficient analysis and manipulation of data for merchandising optimization. Ensure data quality, consistency, and accuracy.
• Build scalable data pipelines (SparkSQL & Scala) using Airflow scheduler/executor framework.
• Collaborate with Data Scientists, Product Managers, and Software Engineers to define data requirements and deliver data solutions that drive merchandising and sales improvements.
• Participate with the broader Data Engineering community to influence tooling and standards to improve culture and efficiency.

Required Skills:

• 5+ years of relevant industry experience.
• Shown experience in Python programming language.
• Experience with distributed processing technologies and frameworks, such as Hadoop, Spark, Kafka, and distributed storage systems (e.g., HDFS, S3).
• Expertise with ETL schedulers such as Apache Airflow, Luigi, Oozie, AWS Glue or similar frameworks.
• Knowledge of data warehousing concepts and hands-on experience with relational databases (e.g., PostgreSQL, MySQL) and columnar databases (e.g., Redshift, BigQuery, HBase, ClickHouse).