Machine Learning Engineer (Distributed Training) at CloudWalk

Machine Learning Engineer (Distributed Training)

CloudWalk

São PauloBrazil

Description

Who we are:

CloudWalk is a fintech company reimagining the future of financial services. We are building intelligent infrastructure powered by AI, blockchain, and thoughtful design. Our products serve millions of entrepreneurs across Brazil and the US every day, helping them grow with tools that are fast, fair, and built for how business actually works. Learn more at cloudwalk.io.

Who We’re Looking For:

We’re looking for a Machine Learning Engineer to own and evolve our distributed training pipeline for large language models. You’ll work inside our GPU cluster to help researchers train and scale foundation models using frameworks like Hugging Face Transformers, Accelerate, DeepSpeed, FSDP, and others. Your focus will be distributed training: from designing sharding strategies and multi-node orchestration to optimizing throughput and managing checkpoints at scale.

This role is not research - it's about building and scaling the systems that let researchers move fast and models grow big. You’ll work closely with MLOps, infra, and model developers to make our training runs efficient, resilient, and reproducible.

What You'll Do:

Own the architecture and maintenance of our distributed training pipeline;

Train LLMs using tools like DeepSpeed, FSDP, and Hugging Face Accelerate;

Design and debug multi-node/multi-GPU training runs (Kubernetes-based);

Optimize training performance: memory usage, speed, throughput, and cost;

Help manage experiment tracking, artifact storage, and resume logic;

Build reusable, scalable training templates for internal use;

Collaborate with researchers to bring their training scripts into production shape.

What We’re Looking For:

Expertise in distributed training: Experience with DeepSpeed, FSDP, or Hugging Face Accelerate in real-world multi-GPU or multi-node setups;

Strong PyTorch background: Comfortable writing custom training loops, schedulers, or callbacks;

Hugging Face stack experience: Transformers, Datasets, Accelerate - you know the ecosystem and how to bend it;

Infra literacy: You understand how GPUs, containers, and job schedulers work together. You can debug cluster issues, memory bottlenecks, or unexpected slowdowns;

Resilience mindset: You write code that can checkpoint, resume, log correctly, and keep running when things go wrong;

Collaborative builder: You don’t mind digging into other people’s scripts, making them robust, and helping everyone train faster.

Bonus Points:

Experience with Kubernetes-based GPU clusters and Ray;

Experience with experiment tracking (MLflow, W&B);

Familiarity with mixed precision, ZeRO stages, model parallelism;

Comfort with CLI tooling, profiling, logging, and telemetry;

Experience with dataloading bottlenecks and dataset streaming.

How We Hire:

Online assessment: technical logic and fundamentals (Math/Calculus, Statistics, Probability, Machine Learning/Deep Learning, Code)

Technical interview: deep dive into distributed training theory and reasoning (no code)

Cultural interview

If you are not willing to take an online quiz, do not apply.

If you’ve trained LLMs before - or helped others do it better - this role is for you. Even if you don’t check every box, if you’re confident working with distributed compute and real-world LLM workloads, we want to hear from you.

Technologies

GoPyTorchFoundationMLflowBlockchainTransformersDeepSpeed

Nice to Have

Experience managing Kubernetes-based GPU clusters and employing tools like Ray for orchestration.Familiarity with experiment tracking tools like MLflow and Weights & Biases (W&B).Understanding of mixed precision training, ZeRO stages, and model parallelism techniques.Proficiency with command-line tooling for profiling, logging, and monitoring machine learning workflows.Ability to identify and resolve data loading bottlenecks and implement dataset streaming.

Must Have

Expertise in distributed training with DeepSpeed, FSDP, or Hugging Face Accelerate in multi-GPU or multi-node environments.Strong PyTorch programming skills, including the ability to write custom training loops and implement callbacks.Experience with the Hugging Face ecosystem, including Transformers and Datasets, with the ability to integrate it effectively into projects.Proficient understanding of GPU infrastructure, container technology, and job scheduling, with the ability to troubleshoot related issues.A mindset focused on resilience, ensuring code can checkpoint, resume, log, and recover from errors during training processes.Ability to collaborate effectively with researchers, refining their training scripts for production readiness.

Apply for this position