Nonparametric star formation histories (SFHs) have long promised to be the "gold standard" for galaxy spectral energy distribution (SED) modeling as they are flexible enough to describe the full diversity of SFH shapes, whereas parametric models rule out a significant fraction of these shapes a priori. However, this flexibility is not fully constrained even with high-quality observations, making it critical to choose a well-motivated prior. Here, we use the SED-fitting code Prospector to explore the effect of different nonparametric priors by fitting SFHs to mock UV–IR photometry generated from a diverse set of input SFHs. First, we confirm that nonparametric SFHs recover input SFHs with less bias and return more accurate errors than do parametric SFHs. We further find that, while nonparametric SFHs robustly recover the overall shape of the input SFH, the primary determinant of the size and shape of the posterior star formation rate as a function of time (SFR(t)) is the choice of prior, rather than the photometric noise. As a practical demonstration, we fit the UV–IR photometry of ~6000 galaxies from the Galaxy and Mass Assembly survey and measure scatters between priors to be 0.1 dex in mass, 0.8 dex in SFR100 Myr, and 0.2 dex in mass-weighted ages, with the bluest star-forming galaxies showing the most sensitivity. An important distinguishing characteristic for nonparametric models is the characteristic timescale for changes in SFR(t). This difference controls whether galaxies are assembled in bursts or in steady-state star formation, corresponding respectively to (feedback-dominated/accretion-dominated) models of galaxy formation and to (larger/smaller) confidence intervals derived from SED fitting. High-quality spectroscopy has the potential to further distinguish between these proposed models of SFR(t).