We show that by gauging the Schr\"odinger algebra with critical exponent $z$ and imposing suitable curvature constraints, that make diffeomorphisms equivalent to time and space translations, one obtains a geometric structure known as (twistless) torsional Newton-Cartan geometry (TTNC). This is a version of torsional Newton-Cartan geometry (TNC) in which the timelike vielbein $\tau_\mu$ must be hypersurface orthogonal. For $z=2$ this version of TTNC geometry is very closely related to the one appearing in holographic duals of $z=2$ Lifshitz space-times based on Einstein gravity coupled to massive vector fields in the bulk. For $z\neq 2$ there is however an extra degree of freedom $b_0$ that does not appear in the holographic setup. We show that the result of the gauging procedure can be extended to include a St\"uckelberg scalar $\chi$ that shifts under the particle number generator of the Schr\"odinger algebra, as well as an extra special conformal symmetry that allows one to gauge away $b_0$. The resulting version of TTNC geometry is the one that appears in the holographic setup. This shows that Schr\"odinger symmetries play a crucial role in holography for Lifshitz space-times and that in fact the entire boundary geometry is dictated by local Schr\"odinger invariance. Finally we show how to extend the formalism to generic torsional Newton-Cartan geometries by relaxing the hypersurface orthogonality condition for the timelike vielbein $\tau_\mu$.