We identify a class of random, dense $n\times n$ matrices for which LU factorization with any form of pivoting produces a growth factor of at least $n/(4 \log n)$ for large $n$ with high probability. The condition number of the matrices can be arbitrarily chosen and large growth also happens for the transpose. No previous matrices with all these properties were known. The matrices can be generated by the MATLAB function \verb"gallery('randsvd',..)", and they are formed as the product of two random orthogonal matrices from the Haar distribution with a diagonal matrix having only one diagonal entry different from $1$, which lies between $0$ and $1$ (the ``one small singular value'' case). Our explanation for the large growth uses the fact that the maximum absolute value of any element of a Haar distributed orthogonal matrix tends to be relatively small for large $n$. We verify the behavior numerically, finding that for partial pivoting the actual growth is significantly larger than the lower bound, and much larger than the growth observed for random matrices with elements from the uniform $[0,1$] or standard normal distributions. We show more generally that a rank-$1$ perturbation to an orthogonal matrix producing large growth for any form of pivoting also generates large growth under reasonable assumptions. Finally, we demonstrate that GMRES-based iterative refinement can provide stable solutions to $Ax = b$ when large growth occurs in low precision LU factors, even when standard iterative refinement cannot.