LLM Ops Engineer at Pattern

Pune, Maharashtra, India

Apply Now

Not SpecifiedCompensation

Mid-level (3 to 4 years)Experience Level

Full TimeJob Type

UnknownVisa

Technology, Artificial IntelligenceIndustries

Requirements

Strong Python programming skills
Experience with generative AI models and tools (OpenAI, Anthropic, Bedrock, etc)
Knowledge of fundamental statistical concepts and tools in data science such as: heuristic and non-heuristic measurements in NLP (BLEU, WER, sentiment analysis, LLM-as-judge, etc), standard deviation, sampling rate, and a high level understanding of how modern AI models work (knowledge cutoffs, context windows, temperature, etc)
Familiarity with AWS
Understanding of prompt engineering concepts
People skills: expected to frequently collaborate with other teams to help perfect their AI workflows
3-5 years of experience in LLM/AI Ops, MLOps, Data Science, or MLE

Responsibilities

Collaborate with data scientists and software engineers to integrate an LLM Ops platform (Opik by CometML) for existing AI workflows
Identify valuable performance metrics (accuracy, quality, etc) for AI workflows and create on-going sampling evaluation processes using the LLM Ops platform that alert when metrics drop below thresholds
Cross-team collaboration to create datasets and benchmarks for new AI workflows
Run experiments on datasets and optimize performance via model changes and prompt adjustments
Debug and troubleshoot AI workflow issues
Optimize inference costs and latency while maintaining accuracy and quality
Develop automations for LLM Ops platform integration to empower data scientists and software engineers to self-serve integration with the AI workflows they build
Monitor, evaluate, and optimize AI/LLM workflows in production environments
Ensure reliable, efficient, and high-quality AI system performance by building out an LLM Ops platform that is self-serve for the engineering and data science departments

Skills

Python

OpenAI

Anthropic

Bedrock

AWS

Prompt Engineering

BLEU

WER

Sentiment Analysis

LLM-as-judge

MLOps

LLM Ops

Opik

CometML

Pattern

Ecommerce acceleration and competitive analysis

About Pattern

Pattern.com focuses on ecommerce acceleration, particularly on Amazon, by providing a free digital shelf management software that analyzes data from millions of searches and products. This tool uses machine learning to give brands insights into their competition, helping them optimize their product listings. Unlike competitors, Pattern.com offers both free tools and premium services, including market insights and consultancy, to help brands grow on Amazon. The company's goal is to empower brands to enhance their visibility and sales on one of the largest online marketplaces.

Lehi, UtahHeadquarters

2013Year Founded

$269.4MTotal Funding

GROWTH_EQUITY_VCCompany Stage

Data & Analytics, Consulting, Consumer GoodsIndustries

1,001-5,000Employees