Global IT QA Software Engineer Senior Manager – Gen AI/ LLM

This is a GenAI Quality Engineer Senior Manager position within BCG's Quality Engineering Center of Expertise (QE CoE). The QE CoE resides within BCG’s Cloud & Platform Engineering (CPE) Portfolio. The role of CPE includes developing and supporting new and enhanced applications, as well as integrating and configuring third party software applications used by our Global HR, Finance, Knowledge, Marketing, Legal, etc. functions. The responsibilities of CPE include enabling efficient processes within the Global Functions; leading and implementing system change initiatives; and supporting the functional user community to optimize the management of the business.

As a GenAI Quality Engineer Senior Manager, you will oversee the quality engineering processes for AI/ML/LLM models and systems. You will be responsible for developing and implementing testing frameworks to ensure the performance, accuracy, and reliability of generative AI solutions. Key duties include designing comprehensive test plans and utilizing advanced testing methodologies to identify and resolve defects.

You will collaborate with cross-functional teams including data scientists, software engineers, and product owners to ensure that AI/ML/LLM models meet specified standards and user requirements. Additionally, you will monitor and analyze model performance, provide feedback for continuous improvement, and stay updated on the latest advancements in AI technology and quality engineering practices.

This role requires strong technical expertise in AI/ML/LLMs, programming, and testing tools, along with leadership and project management skills to drive the quality engineering strategy and ensure the successful deployment of AI/ML/LLM solutions.

YOU'RE GOOD AT

• Developing comprehensive test plans
• Implementing advanced testing methodologies
• Identifying and resolving defects
• Collaborating with cross-functional teams
• Ensuring AI model performance and reliability
• Analyzing model performance metrics
• Providing feedback for continuous improvement
• Staying updated on AI technology advancements
• Applying strong technical expertise in AI/ML/LLMs
• Utilizing programming and testing tools
• Managing quality engineering strategies
• Overseeing successful AI/ML/LLM solution deployment
• Communicating effectively with stakeholders
• Problem-solving and critical thinking
• Project management
• Understanding AI/ML/LLM model standards and requirements
• Conducting risk assessments
• Enhancing testing frameworks
• Maintaining high standards of AI/ML/LLM model accuracy and performance

YOU BRING (EXPERIENCE & QUALIFICATIONS)

• Bachelor’s degree in computer science, Engineering, or a related field
• Extensive experience in quality engineering or software testing
• Proven experience in testing Gen AI, LLMs, Artificial Intelligence, Machine Learning, or Natural Language Processing based systems for scale and robustness.
• Strong background in AI and machine learning
• Proficiency in programming languages (e.g., Python, Java)
• Experience with AI/ML/LLM testing tools and frameworks e.g. DeepEval or equivalent.
• Solid understanding of AI model development and deployment
• Proven leadership and team management skills
• Excellent problem-solving and analytical abilities
• Strong communication and collaboration skills
• Knowledge of industry standards and best practices in AI/LLM quality engineering
• Experience with performance and reliability testing

YOU'LL WORK WITH BCG’s information technology group collaboratively delivers the latest digital technologies that enable our consultants to lead and our business to grow. For our IT jobs, we seek individuals with expertise in the areas of IT infrastructure, application development, business systems, collaborative and social technologies, information security, and project leadership.