We report on the identification of the highest redshift submillimetre-selected source currently known LESS J033229.4-275619. This source was detected in the Large Apex Bolometer Camera (LABOCA) Extended Chandra Deep Field-South (ECDF-S) Submillimetre Survey (LESS), a sensitive 870-mu m survey (sigma(870 mu m) similar to 1.2 mJy) of the full 30 x 30 arcmin(2) ECDF-S with the LABOCA on the Atacama Pathfinder Experiment telescope. The submillimetre emission is identified with a radio counterpart for which optical spectroscopy provides a redshift of z = 4.76. We show that the bolometric emission is dominated by a starburst with a star formation rate of similar to 1000 M-circle dot yr(-1), although we also identify a moderate luminosity active galactic nucleus (AGN) in this galaxy. Thus it has characteristics similar to those of z similar to 2 submillimetre galaxies (SMGs), with a mix of starburst and obscured AGN signatures. This demonstrates that ultraluminous starburst activity is not just restricted to the hosts of the most luminous (and hence rare) quasi-stellar objects at z similar to 5, but was also occurring in less extreme galaxies at a time when the Universe was less than 10 per cent of its current age. Assuming that we are seeing the major phase of star formation in this galaxy, then we demonstrate that it would be identified as a luminous distant red galaxy at z similar to 3 and that the current estimate of the space density of z > 4 SMGs is only sufficient to produce greater than or similar to 10 per cent of the luminous red galaxy population at these early times. However, this leaves open the possibility that some of these galaxies formed through less intense, but more extended star formation events. If the progenitors of all of the luminous red galaxies at z similar to 3 go through an ultraluminous starburst at z greater than or similar to 4 then the required volume density of z > 4 SMGs will exceed that predicted by current galaxy formation models by more than an order of magnitude.