While electromagnetic and up-down quark mass difference effects on octet baryon masses are very small, they have important consequences. The stability of the hydrogen atom against beta decay is a prominent example. Here, we include these effects by adding them to valence quarks in a lattice QCD calculation based on Nf=2+1 simulations with five lattice spacings down to 0.054 fm, lattice sizes up to 6 fm, and average up-down quark masses all the way down to their physical value. This allows us to gain control over all systematic errors, except for the one associated with neglecting electromagnetism in the sea. We compute the octet baryon isomultiplet mass splittings, as well as the individual contributions from electromagnetism and the up-down quark mass difference. Our results for the total splittings are in good agreement with experiment.