The existence and stability of atoms rely on the fact that neutrons are more massive than protons. The measured mass difference is only 0.14% of the average of the two masses. A slightly smaller or larger value would have led to a dramatically different universe. Here, we show that this difference results from the competition between electromagnetic and mass isospin breaking effects. We performed lattice quantum-chromodynamics and quantum-electrodynamics computations with four nondegenerate Wilson fermion flavors and computed the neutron-proton mass-splitting with an accuracy of 300 kilo-electron volts, which is greater than 0 by 5 standard deviations. We also determine the splittings in the Σ, Ξ, D, and Ξcc isospin multiplets, exceeding in some cases the precision of experimental measurements.