Data Engineer Python

Company:  https:/wayup.com/sitemap.xml
Location: Houston
Closing Date: 31/10/2024
Salary: £100 - £125 Per Annum
Hours: Full Time
Type: Permanent
Job Requirements / Description

Job Title: Python Data Engineer

Location: Houston, TX (ONSITE ROLE)

Duration: Long term contract

Job Description: We are looking for a talented Data Engineer with expertise in Python data processing. The ideal candidate will have a strong background in Python API development, parallel data processing, and distributed systems design. You will be responsible for building and maintaining systems that handle large-scale data processing tasks, ensuring high performance and scalability.

Key Responsibilities:

  1. Python API Development:
    • Develop and maintain RESTful APIs using Python web frameworks such as FastAPI or Django.
    • Collaborate with front-end developers to integrate user-facing elements with server-side logic.
  2. Parallel Data Processing:
    • Utilize Pandas, NumPy, and other libraries to process large datasets efficiently.
    • Implement multithreading, multiprocessing, and asynchronous programming techniques.
    • Optimize data processing pipelines to handle millions of rows with minimal latency.
  3. Distributed Systems Design:
    • Design and implement distributed systems with a focus on scalability and reliability.
    • Understand and apply core concepts such as load balancing and task queues.
    • Use Docker to containerize applications and manage dependencies.
    • (Preferred) Experience with Kubernetes for container orchestration.
  4. Technical Communication:
    • Clearly articulate complex technical concepts to team members and stakeholders.
    • Document system designs, processes, and code effectively.
    • Collaborate with cross-functional teams to align on project goals and deliverables.

Must-Have Qualifications:

  1. Experience in Python Web Frameworks:
    • Proficiency with FastAPI, Django, or similar frameworks.
    • Understanding of RESTful API principles and best practices.
  2. Docker Knowledge:
    • Ability to create and manage Docker Files.
    • Experience with containerization for deployment and development workflows.
  3. Systems Design Understanding:
    • Basic knowledge of load balancing, task queues, and distributed system concepts.
    • Ability to design systems that are scalable and maintainable.
  4. Concurrent and Parallel Computing Skills:
    • Proficiency in multithreading and multiprocessing without relying solely on external libraries or frameworks.
    • Familiarity with asynchronous programming, particularly asyncIO in Python.
  5. Communication Skills:
    • Excellent technical communication abilities.
    • Experience collaborating in team environments and conveying complex ideas clearly.

Preferred Qualifications:

  1. Education:
    • BS or MS in Computer Science.
  2. Advanced Data Processing Tools:
    • Experience with Polars, PySpark, or similar tools.
    • Handling of large-scale data processing tasks efficiently.
  3. Distributed Computing Experience:
    • Hands-on experience with distributed architectures in Docker.
    • Familiarity with concepts like task queuing, MapReduce, and saga patterns.
  4. Kubernetes Experience:
    • Knowledge of container orchestration using Kubernetes.
    • Experience deploying and managing applications in a Kubernetes cluster.
  5. Problem-Solving at Scale:
    • Demonstrated ability to solve complex problems using parallel or distributed computing.
    • Innovative thinking beyond single-threaded processes.
#J-18808-Ljbffr
Apply Now
Share this job
https:/wayup.com/sitemap.xml
  • Similar Jobs

  • Python Data Engineer

    Houston
    View Job
  • Data Engineer / Python Data Enigneer

    Houston
    View Job
  • Python Engineer

    Houston
    View Job
  • Software Engineer- Python

    Houston
    View Job
  • Senior Software Engineer (Python)

    Houston
    View Job
An error has occurred. This application may no longer respond until reloaded. Reload 🗙