Towards Better Use of Ontological Structure in the Evaluation of Automated ICD Coding

Since the release of the MIMIC-III dataset there have been numerous attempts at automating ICD-9 coding through artificial neural networks. The problem, cast as a large-scale multi-label classification, is generally evaluated with standard precision, recall, and F1 measures without regard for the rich ontological structure. In this work we argue for hierarchical evaluation of the predictions of these models, propose a set of metrics for such evaluation. We describe a structural issue in the representation of the hierarchy in prior art and propose an alternative representation based on the levels of the ontology. We also propose further avenues of research involving the proposed ontological representation.