LLM Ops Engineer at Pattern

Pune, Maharashtra, India

Apply Now

Not SpecifiedCompensation

Mid-level (3 to 4 years)Experience Level

Full TimeJob Type

UnknownVisa

Technology, Artificial IntelligenceIndustries

Requirements

Strong Python programming skills
Experience with generative AI models and tools (OpenAI, Anthropic, Bedrock, etc)
Knowledge of fundamental statistical concepts and tools in data science such as: heuristic and non-heuristic measurements in NLP (BLEU, WER, sentiment analysis, LLM-as-judge, etc), standard deviation, sampling rate, and a high level understanding of how modern AI models work (knowledge cutoffs, context windows, temperature, etc)
Familiarity with AWS
Understanding of prompt engineering concepts
People skills: expected to frequently collaborate with other teams to help perfect their AI workflows
3-5 years of experience in LLM/AI Ops, MLOps, Data Science, or MLE

Responsibilities

Collaborate with data scientists and software engineers to integrate an LLM Ops platform (Opik by CometML) for existing AI workflows
Identify valuable performance metrics (accuracy, quality, etc) for AI workflows and create on-going sampling evaluation processes using the LLM Ops platform that alert when metrics drop below thresholds
Cross-team collaboration to create datasets and benchmarks for new AI workflows
Run experiments on datasets and optimize performance via model changes and prompt adjustments
Debug and troubleshoot AI workflow issues
Optimize inference costs and latency while maintaining accuracy and quality
Develop automations for LLM Ops platform integration to empower data scientists and software engineers to self-serve integration with the AI workflows they build
Monitor, evaluate, and optimize AI/LLM workflows in production environments
Ensure reliable, efficient, and high-quality AI system performance by building out an LLM Ops platform that is self-serve for the engineering and data science departments

Skills

Key technologies and capabilities for this role

PythonOpenAIAnthropicBedrockAWSPrompt EngineeringBLEUWERSentiment AnalysisLLM-as-judgeMLOpsLLM OpsOpikCometML

Questions & Answers

Common questions about this position

What experience level is required for this LLM Ops Engineer role?

The role requires 3-5 years of experience in LLM/AI Ops, MLOps, Data Science, or MLE.

What are the key technical skills needed for this position?

Strong Python programming skills, experience with generative AI models and tools like OpenAI, Anthropic, Bedrock, knowledge of statistical concepts in data science such as BLEU, WER, and LLM-as-judge, familiarity with AWS, and understanding of prompt engineering are required.

Is this a remote position, or is there a location requirement?

This information is not specified in the job description.

What is the salary or compensation for this role?

This information is not specified in the job description.

What soft skills are important for success in this role?

Strong people skills are required, as you will frequently collaborate with other teams like data scientists and software engineers to perfect their AI workflows.

Pattern

Ecommerce acceleration and competitive analysis

About Pattern

Pattern.com focuses on ecommerce acceleration, particularly on Amazon, by providing a free digital shelf management software that analyzes data from millions of searches and products. This tool uses machine learning to give brands insights into their competition, helping them optimize their product listings. Unlike competitors, Pattern.com offers both free tools and premium services, including market insights and consultancy, to help brands grow on Amazon. The company's goal is to empower brands to enhance their visibility and sales on one of the largest online marketplaces.

Lehi, UtahHeadquarters

2013Year Founded

$269.4MTotal Funding

GROWTH_EQUITY_VCCompany Stage

Data & Analytics, Consulting, Consumer GoodsIndustries

1,001-5,000Employees