Senior MLOps Engineer – LLMOps at TRM Labs

United States

Apply Now

Not SpecifiedCompensation

Senior (5 to 8 years)Experience Level

Full TimeJob Type

UnknownVisa

Blockchain, AI/ML, CybersecurityIndustries

Requirements

Write high-quality, maintainable software — primarily in Python, but value engineering ability over language familiarity
Strong background in scalable infrastructure, including containerization and orchestration (e.g. Docker, Kubernetes)
Strong background in infrastructure-as-code and deployment (e.g. Terraform, CI/CD pipelines)
Strong background in monitoring and logging frameworks (e.g. Datadog, Prometheus, OpenTelemetry)
Understand and implement MLOps best practices, including model versioning and rollback strategies
Understand and implement MLOps best practices, including automated evaluation and drift detection
Understand and implement MLOps best practices, including scalable model and agent serving infrastructure (e.g. vLLM, Triton, BentoML)
Deploy and maintain LLM and agentic workflows in production, including monitoring cost, latency, and performance
Deploy and maintain LLM and agentic workflows in production, including capturing traces for analysis and debugging
Deploy and maintain LLM and agentic workflows in production, including optimizing prompt/response flows with real-time data access
Demonstrate strong ownership and pragmatism, balancing infrastructure elegance with iterative delivery and measurable impact
Rapid issue resolution: identify and resolve critical onsite issues in minutes to hours, create virtual war rooms, implement fixes, and share lessons

Responsibilities

Build reusable CI/CD workflows for model training, evaluation, and deployment — integrating Langfuse, GitHub Actions, and experiment tracking, etc
Automate model versioning, approval workflows, and compliance checks across environments
Build out a modular and scalable AI infrastructure stack — including vector databases, feature stores, model registries, and observability tooling
Partner with engineering and data science to embed AI models and agents into real-time applications and workflows
Continuously evaluate and integrate state-of-the-art AI tools (e.g. LangChain, LlamaIndex, vLLM, MLflow, BentoML, etc.)
Drive AI reliability and governance, enabling experimentation while ensuring compliance, security, and uptime
Build and enhance AI/ML Model Performance
Ensure data accuracy, consistency and reliability, leading to better model training and inferencing
Deploy infrastructure to support offline and online evaluation of LLMs and agents — including regression testing, cost monitoring, and human-in-the-loop workflows
Enable researchers to iterate quickly by providing sandboxes, dashboards, and reproducible environments

Skills

Key technologies and capabilities for this role

MLOpsLLMOpsCI/CDGitHub ActionsLangfusemodel trainingmodel evaluationmodel deploymentvector databasesfeature storesmodel registriesobservabilityexperiment tracking

Questions & Answers

Common questions about this position

What programming languages or skills are required for this Senior MLOps Engineer role?

The role requires writing high-quality, maintainable software primarily in Python, with a strong background in scalable infrastructure including containerization and orchestration such as Docker and Kubernetes.

What specific tools and technologies will I work with in this position?

You'll build CI/CD workflows integrating Langfuse, GitHub Actions, and experiment tracking; work with vector databases, feature stores, model registries; and evaluate tools like LangChain, LlamaIndex, vLLM, MLflow, and BentoML.

What is the company culture like at TRM Labs?

TRM Labs features a mission-driven, fast-paced team of experts in law enforcement, data science, engineering, and financial intelligence, tackling complex global challenges daily.

Is this Senior MLOps Engineer position remote or office-based?

This information is not specified in the job description.

What makes a strong candidate for this Senior MLOps Engineer – LLMOps role?

Strong candidates write high-quality, maintainable software primarily in Python, have a strong background in scalable infrastructure like containerization and orchestration, and can build robust AI/ML pipelines with tools for deployment, observability, and governance.

TRM Labs

Blockchain intelligence for financial crime detection

About TRM Labs

TRM Labs focuses on blockchain intelligence to assist financial institutions, cryptocurrency businesses, and government agencies in identifying and investigating financial crimes and fraud related to cryptocurrencies. The company utilizes advanced data engineering, data science, and threat intelligence to provide tools and insights that help clients combat illicit activities in the crypto sector. TRM Labs serves a wide range of clients, including banks and regulatory bodies, and generates revenue through subscription-based services and customized solutions. Its goal is to enhance the security and integrity of the global financial system by offering comprehensive blockchain analytics and investigative capabilities.

San Francisco, CaliforniaHeadquarters

2018Year Founded

$146MTotal Funding

SERIES_BCompany Stage

Data & Analytics, FintechIndustries

201-500Employees

Benefits

Remote Work Options

Annual Company Offsite

Professional Development Budget

Risks

Increased scrutiny in the UK could affect operations and partnerships in the region.

Collaboration with Tether and TRON may expose TRM Labs to reputational risks.

Cyberattack vulnerabilities in crypto exchanges could impact TRM Labs' clients.

Differentiation

TRM Labs specializes in blockchain intelligence for financial crime detection and investigation.

The company collaborates with major firms like PayPal, Visa, and government agencies.

TRM Labs offers subscription-based services and customized solutions for diverse client needs.

Upsides

Partnership with Four Inc. could increase government contracts in U.S. public sector.

Success in freezing $100 million in assets highlights effectiveness in public-private partnerships.

Collaboration with DeFi platforms like 1inch enhances influence in decentralized finance.

Land your dream remote job 3x faster with AI

Try Jobo Free