Machine Learning Systems Engineer, Encodings and Tokenization at Anthropic

The Machine Learning Systems Engineer will design, develop, and maintain tokenization systems used across Pretraining and Finetuning workflows, optimize encoding techniques to improve model training efficiency and performance, collaborate closely with research teams to understand their evolving needs around data representation, build infrastructure that enables researchers to experiment with novel tokenization approaches, implement systems for monitoring and debugging tokenization-related issues in the model training pipeline, create robust testing frameworks to validate tokenization systems across diverse languages and data types, identify and address bottlenecks in data processing pipelines related to tokenization, and document systems thoroughly while communicating technical decisions clearly to stakeholders across teams.