After successfully diversifying during the Paleocene, the descendants of the first wave of mammals that survived the end-Cretaceous mass extinction waned throughout the Eocene. Competition with modern crown clades and intense climate fluctuations may have been part of the factors leading to the extinction of these archaic groups. Why these taxa went extinct has rarely been studied from the perspective of the nervous system. Here, we describe the first virtual endocasts for the archaic order Tillodontia. Three species from the middle Eocene of North America were analyzed: Trogosus hillsii, Trogosus grangeri, and Trogosus castoridens. We made morphological comparisons with the plaster endocast of another tillodont, Tillodon fodiens, as well as groups potentially related to Tillodontia: Pantodonta, Arctocyonidae, and Cimolesta. Trogosus shows very little inter-specific variation with the only potential difference being related to the fusion of the optic canal and sphenorbital fissure. Many ancestral features are displayed by Trogosus, including an exposed midbrain, small neocortex, orbitotemporal canal ventral to rhinal fissure, and a broad circular fissure. Potential characteristics that could unite Tillodontia with Pantodonta, and Arctocyonidae are the posterior position of cranial nerve V3 exit in relation to the cerebrum and the low degree of development of the subarcuate fossa. The presence of large olfactory bulbs and a relatively small neocortex are consistent with a terrestrial lifestyle. A relatively small neocortex may have put Trogosus at risk when competing with artiodactyls for potentially similar resources and avoiding predation from archaic carnivorans, both of which are known to have had larger relative brain and neocortex sizes in the Eocene. These factors may have possibly exacerbated the extinction of Tillodontia, which showed highly specialized morphologies despite the increase in climate fluctuations throughout the Eocene, before disappearing during the middle Eocene.