We derive an action whose equations of motion contain the Poisson equation of Newtonian gravity. The construction requires a new notion of Newton-Cartan geometry based on an underlying symmetry algebra that differs from the usual Bargmann algebra. This geometry naturally arises in a covariant 1/c expansion of general relativity, with c being the speed of light. By truncating this expansion at subleading order, we obtain the field content and transformation rules of the fields that appear in the action of Newtonian gravity. The equations of motion generalize Newtonian gravity by allowing for the effect of gravitational time dilation due to strong gravitational fields.