Distributed Training: DDP

Medium

deep-learning-libraries Meta NVIDIA Amazon

How does PyTorch's DistributedDataParallel (DDP) scale training across multiple GPUs?

Multiple Choice

It splits the model layers across GPUs, with each GPU computing one layer It replicates the model on each GPU, partitions the data, and synchronizes gradients across GPUs using all-reduce after each backward pass It trains different models on different GPUs and picks the best one It uses a single GPU but simulates multiple GPUs in software

Correct!

Incorrect — try again next time!

Upgrade to Pro to unlock solutions and explanations

Distributed Training: DDP

Multiple Choice

Related problems

Deep-dive guides