Principal Machine Learning Engineer

MD Anderson is expanding our Enterprise MLOps and Analytics Platform capabilities to enhance support for MLOps and ModelOps, facilitating the operationalization, monitoring, and management of in-house and third-party AI solutions. This expansion is integral to strengthening our overall AI Governance framework.

We are actively seeking a Principal MLOps Engineer to lead and establish a team of MLOps engineers tasked with designing and enhancing our Enterprise MLOps and Analytics Platform. The Principal MLOps Engineer will be pivotal in managing the operations and overseeing the entire MLOps and ModelOps tech stack, ensuring a robust end-to-end lifecycle management and governance of machine learning models throughout the organization. As the principal figure in MLOps, this role involves defining the team dynamics, shaping the culture, and implementing processes and technologies essential for supporting ML solutions within our hybrid data and compute framework. Collaborating closely with IT, cybersecurity, and compliance teams, the Principal MLOps Engineer will be instrumental in creating a secure and compliant infrastructure for the scalable management of AI/ML models.

Key Responsibilities

1. Lead and mentor a team of MLOps Engineers to create a scalable MLOps and Analytics platform within a hybrid compute environment, including Kubernetes and Azure.
2. Design, implement, and oversee CI/CD pipelines, ensuring the infrastructure is conducive to ML model training, deployment, and monitoring while upholding security, scalability, reliability, and performance.
3. Develop, refine, and standardize Model Governance integrations, performance tracking for bias and impact, and a model catalog with standardized scorecards and deployments.
4. Innovate automated validation, deployment, observability, and management tools for scalable and reproducible AI solutions.
5. Design fallback and decommissioning strategies for AI solutions to ensure operational continuity.
6. Deliver training on AI solutions to enhance understanding and application across the organization.
7. Engage with technology trends, contribute to tech communities, and foster a culture of continuous learning and innovation.

Technical Expertise

1. Demonstrate deep understanding of the AI/ML Platform infrastructure and cloud architecture.
2. Experience developing and deploying AI/ML algorithms into production.
3. Strong proficiency in Python and C++ or C#, complemented by experience with machine learning libraries such as TensorFlow, PyTorch, and Scikit-learn.
4. Knowledge of DevOps practices, CI/CD pipelines, including tools like Azure DevOps or Git Actions.
5. Proficiency in working with containers such as Docker and container orchestration systems like Kubernetes. Familiarity with process orchestration/DAGs tools.
6. Experience with data, code, and model artifact management processes and MLOps tools.
7. Experience with on-premises, cloud-based, and hybrid computing environments, as well as cloud-native tools and services.
8. Knowledge of ISO standards for software and/or AI development lifecycle management.

Analytical Skills

1. Experience with project management methodologies (e.g. SAFe agile, PRINCE2, Lean methodology).
2. Deep understanding of the AI/ML Model Lifecycle Management.
3. Proficient in decision-making, problem-solving, and the successful execution of AI/ML solutions in a healthcare environment.
4. Manage AI/ML projects throughout their lifecycle, ensuring timely delivery, budget adherence, and quality standards compliance.
5. Experience leading an ML engineering and/or data scientist team focused on developing, deploying, and maintaining production-ready models.
6. Experience working closely with third-party vendors and partners to integrate new AI solutions into existing infrastructure and workflows.
7. Experience implementing risk identification and mitigation strategies, including contingency planning, to prevent project delays and complications.
8. Preference for working knowledge of hospital workflows.
9. Preference for experience with healthcare data privacy and security protocols, such as HIPAA.
10. Preference for experience with flowcharts, business process models, mapping tools, data flow diagrams, and process flow diagrams.

Professionalism: Oral and Written

1. Work closely with data scientists, ML engineers, software engineers, and other stakeholders to understand requirements and integrate machine learning models into the overall system.
2. Create and maintain comprehensive documentation for CI/CD pipelines, deployment processes, and infrastructure configurations.
3. Experience reporting on project progress, impact, and risks to leadership and stakeholders, providing strategic advice to help prioritize AI/ML solutions use-cases.
4. Experience stakeholder management to drive adoption, address concerns, and prioritize solution support.

Education Required

Bachelor's degree in Computer Science, Software Engineering, Data Science, Physics, Math & Statistics, or another related engineering discipline.

Preferred Education

Master's Level Degree

Experience Required

Seven years of experience in machine learning engineering, data science, data engineering, and/or software engineering. With Master's degree, five years' experience required. With PhD, three years of experience required.

Preferred Experience

Experience working on production quality healthcare focused machine learning solutions.

Proficiency

Proficiency in architecting, implementing, and using MLOps solutions.

Onsite Presence

Is Not Required

Equal Employment Opportunity

It is the policy of The University of Texas MD Anderson Cancer Center to provide equal employment opportunity without regard to race, color, religion, age, national origin, sex, gender, sexual orientation, gender identity/expression, disability, protected veteran status, genetic information, or any other basis protected by institutional policy or by federal, state or local laws unless such distinction is required by law. Learn more .

Additional Information

• Requisition ID: 165964
• Employment Status: Full-Time
• Employee Status: Regular
• Work Week: Days
• Minimum Salary: US Dollar (USD) 160,500
• Midpoint Salary: US Dollar (USD) 203,000
• Maximum Salary: US Dollar (USD) 245,500
• FLSA: exempt and not eligible for overtime pay
• Fund Type: Hard
• Work Location: Remote (within Texas only)
• Pivotal Position: Yes
• Referral Bonus Available?: Yes
• Relocation Assistance Available?: Yes
• Science Jobs: No