Edinburgh Research Explorer

Mining source code repositories at massive scale using language modeling

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Related Edinburgh Organisations

Open Access permissions

Open

Original languageEnglish
Title of host publicationMining Software Repositories (MSR), 2013 10th IEEE Working Conference on
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Pages207-216
Number of pages10
ISBN (Print)978-1-4799-0345-0
DOIs
Publication statusPublished - 2013

Abstract

The tens of thousands of high-quality open source software projects on the Internet raise the exciting possibility of studying software development by finding patterns across truly large source code repositories. This could enable new tools for developing code, encouraging reuse, and navigating large projects. In this paper, we build the first giga-token probabilistic language model of source code, based on 352 million lines of Java. This is 100 times the scale of the pioneering work by Hindle et al. The giga-token model is significantly better at the code suggestion task than previous models. More broadly, our approach provides a new "lens" for analyzing software projects, enabling new complexity metrics based on statistical analysis of large corpora. We call these metrics data-driven complexity metrics. We propose new metrics that measure the complexity of a code module and the topical centrality of a module to a software project. In particular, it is possible to distinguish reusable utility classes from classes that are part of a program's core logic based solely on general information theoretic criteria.

    Research areas

  • Complexity theory, Entropy, Java, Measurement, Predictive models, Software, Training

Download statistics

No data available

ID: 10623280