Gradient Accumulation

It prevents gradient explosion by clipping gradient values It sums gradients over multiple mini-batches before performing a weight update, simulating a larger effective batch size without requiring more GPU memory It stores all historical gradients for momentum computation It accumulates data samples into larger files for faster I/O