Why do my testicles change size in different temperatures?

Yep, it's the temperature. In the cold, they shrink closer to the body to try to get some of the warmth, and in the heat, they hang lower, in order to try to keep cool. If your testicles overheat, the sperm dies or becomes less mobile.