Data Engineer Python

Job Title: Python Data Engineer

Location: Houston, TX (ONSITE ROLE)

Duration: Long term contract

Job Description: We are looking for a talented Data Engineer with expertise in Python data processing. The ideal candidate will have a strong background in Python API development, parallel data processing, and distributed systems design. You will be responsible for building and maintaining systems that handle large-scale data processing tasks, ensuring high performance and scalability.

Key Responsibilities:

1. Python API Development:
   • Develop and maintain RESTful APIs using Python web frameworks such as FastAPI or Django.
   • Collaborate with front-end developers to integrate user-facing elements with server-side logic.
2. Parallel Data Processing:
   • Utilize Pandas, NumPy, and other libraries to process large datasets efficiently.
   • Implement multithreading, multiprocessing, and asynchronous programming techniques.
   • Optimize data processing pipelines to handle millions of rows with minimal latency.
3. Distributed Systems Design:
   • Design and implement distributed systems with a focus on scalability and reliability.
   • Understand and apply core concepts such as load balancing and task queues.
   • Use Docker to containerize applications and manage dependencies.
   • (Preferred) Experience with Kubernetes for container orchestration.
4. Technical Communication:
   • Clearly articulate complex technical concepts to team members and stakeholders.
   • Document system designs, processes, and code effectively.
   • Collaborate with cross-functional teams to align on project goals and deliverables.

Must-Have Qualifications:

1. Experience in Python Web Frameworks:
   • Proficiency with FastAPI, Django, or similar frameworks.
   • Understanding of RESTful API principles and best practices.
2. Docker Knowledge:
   • Ability to create and manage Docker Files.
   • Experience with containerization for deployment and development workflows.
3. Systems Design Understanding:
   • Basic knowledge of load balancing, task queues, and distributed system concepts.
   • Ability to design systems that are scalable and maintainable.
4. Concurrent and Parallel Computing Skills:
   • Proficiency in multithreading and multiprocessing without relying solely on external libraries or frameworks.
   • Familiarity with asynchronous programming, particularly asyncIO in Python.
5. Communication Skills:
   • Excellent technical communication abilities.
   • Experience collaborating in team environments and conveying complex ideas clearly.

Preferred Qualifications:

1. Education:
   • BS or MS in Computer Science.
2. Advanced Data Processing Tools:
   • Experience with Polars, PySpark, or similar tools.
   • Handling of large-scale data processing tasks efficiently.
3. Distributed Computing Experience:
   • Hands-on experience with distributed architectures in Docker.
   • Familiarity with concepts like task queuing, MapReduce, and saga patterns.
4. Kubernetes Experience:
   • Knowledge of container orchestration using Kubernetes.
   • Experience deploying and managing applications in a Kubernetes cluster.
5. Problem-Solving at Scale:
   • Demonstrated ability to solve complex problems using parallel or distributed computing.
   • Innovative thinking beyond single-threaded processes.