We probe the influence of particle size polydispersity on field-induced structures and structural transitions in magnetic fluids (ferrofluids) using phase contrast optical microscopy, light scattering and Brownian dynamics simulations. Three different ferrofluids containing superparamagnetic nanoparticles of different polydispersity indices (PDIs) are used. In a ferrofluid with a high PDI (∼0.79), thin chains, thick chains, and sheets are formed on increasing the in-plane magnetic field, whereas isotropic bubbles, and hexagonal and lamellar/stripe structures are formed on increasing the out-of-plane magnetic field over the same range. In contrast, no field-induced aggregates are seen in the sample with low polydispersity under the above conditions. In a polydisperse sample, bubbles are formed at a very low magnetic field strength of 30 G. Insights into the structural evolution with increasing magnetic field strength are obtained by carrying out Brownian dynamics simulations. The crossovers from isotropic, through hexagonal columnar, to lamellar/stripe structures observed with increasing field strength in the high-polydispersity sample indicate the prominent roles of large, more strongly interacting particles in structural transitions in ferrofluids. Based on the observed microstructures, a phase diagram is constructed. Our work opens up new opportunities to develop optical devices and access diverse structures by tuning size polydispersity.