The question
In pix2pix the generator’s BatchNorm layers use each image’s own statistics at test time instead of the running averages collected during training. I implemented pix2pix from scratch (U-Net generator, PatchGAN discriminator, CMP facades) and wanted to understand why, and whether it is actually necessary.
The reasoning I worked out
Normalizing each image by its own statistics acts like instance normalization: it strips image-specific contrast and color, which fits an image-to-image translation task, since the output should follow the input’s structure rather than the training set’s average color statistics. There is also a batch-size angle. This model trains at batch size 1, so during training every activation is always normalized by its own image. My working hypothesis was that switching to global running stats at inference shifts activations away from what the network was trained on, which would explain the color shifts and artifacts you see if you do it.
The experiment
I retrained the same model at batch size 8 and compared inference on running stats against per-image test-time stats. The larger batch mitigated the problem: running-stats inference looked better than it did at batch size 1, but still did not match test-time stats.
What I learned
The result is consistent with the hypothesis: training at batch size 1 only ever exposes the network to per-image-normalized activations, while larger batches expose it to small offsets around shared batch statistics and buy back some tolerance, which is why the gap shrinks without fully closing. One caveat: train() mode also enables the generator’s dropout layers, so this comparison does not isolate BatchNorm statistics on their own. Either way, a design detail that looks arbitrary in the paper turns out to matter, and the interesting question is what the network was ever exposed to during training.