MLOps Programmer Analyst
General Position Summary: Participate in the creation, implementation, and production support of healthcare software applications, with a focus on AI and large language model and machine learning operations (LLMOps/MLOps). The primary responsibility is to develop and support custom generative AI applications that improve care delivery and health system operations. Specific tasks will include running an experimental framework to determine the optimal prompt engineering approaches, tuning prompts, and collaborating with subject matter experts (SMEs) for evaluations and results. This role requires evaluating model outputs in production - particularly when ground metrics are absent, monitoring for issues such as model drift and hallucinations, and optimizing for offline and online metrics. Other responsibilities can include evaluating vendor-developed AI/ML features and managing health data pipelines into the Precision Medicine Analytics Platform. Opportunities for growth include planning and prioritizing software projects, and informing department strategy on AI/ML.
Job Scope/Complexity: Typically understands project life cycle and has responsibility for sections of life cycle. Under guidance, responsible for full life cycle of small and/or parts of a large project. Once priorities are set, perform tasks independently with regular process updates.
Job Responsibilities: The responsibilities listed below are typical examples of the work performed by this position. Not all duties assigned to this position are included, nor is it expected that everyone in this position will be assigned every job responsibility.
ANALYSIS AND REQUIREMENTS GATHERING
- Under guidance, define business or clinical problems by meeting with customers to observe and understand current processes and the issues related to those processes. Provide written documentation of findings to share with the customer and IT colleagues.
- Translate identified problems into system requirements by researching technology and leveraging design principles.
- Write user stories and acceptance criteria by translating the requirements into functional and technical descriptions for use by engineers.
- Develop code using appropriate languages and tools to provide solutions for defined user stories and tasks.
- Create and document test scenarios to validate and verify application functionality. Automate testing suites using appropriate harnesses/tools.
- Write and maintain documentation by writing audience-appropriate materials to serve as references.
- Monitor changes and resolve problems by responding to problems as they occur, following change management policies and procedures. Proactively ensure the solution satisfies customer requirements and provides a smooth transition to the solution.
- Implement monitoring tools and conduct regular reevaluation of AI/ML applications to ensure ongoing performance meets standards for accuracy and efficiency.
Required Education: Bachelor's degree required. Additional experience may be substituted for education.
Required Experience: One year of related work experience with computer systems and applications. Additional education may substitute for experience.
Preferred Job Qualifications:
- Experience in data science, machine learning, data engineering
- Experience in some of the following languages and tools: Java, Python, Azure, Databricks
- Experience in machine learning frameworks, for example PyTorch, TensorFlow, ONNX
- Experience with Large Language Models, Generative AI, model development/inference patterns
- Experience with statistical analysis and data visualization tools
- Experience designing scalable LLM solutions with distributed computing