The semi-device-independent approach provides a framework for prepare-and-measure quantum protocols using devices whose behavior does not need to be characterized or trusted, except for a single assumption on the dimension of the Hilbert space characterizing the quantum carriers. Here, we propose instead to constrain the quantum carriers through a bound on the mean value of a well-chosen observable. This modified assumption is physically better motivated than a dimension bound and closer to the description of actual experiments. In particular, we consider quantum optical schemes where the source emits quantum states described in an infinite-dimensional Fock space and model our assumption as an upper bound on the average photon number in the emitted states. We characterize the set of correlations that maybe exhibited in the simplest possible scenario compatible with our new framework, based on two energy-constrained state preparations and a two-outcome measurement. Interestingly, we uncover the existence of quantum correlations exceeding the set of classical correlations that can be produced by devices behaving in a purely pre-determined fashion (possibly including shared randomness). This feature suggests immediate applications to certified randomness generation. Along this line, we analyze the achievable correlations in several prepare-and-measure optical schemes with a mean photon-number constraint and demonstrate that they allow for the generation of certified randomness. Our simplest optical scheme works by the on-off keying of an attenuated laser source followed by photocounting. It opens the path to more sophisticated energy-constrained semi-device-independent quantum cryptography protocols, such as quantum key distribution.