A Case Study on Machine Learning for Synthesizing Benchmarks

Andrés Goens; Alexander Brauckmann; Sebastian Ertel; Chris Cummins; Hugh Leather; Jeronimo Castrillon

doi:10.1145/3315508.3329976

A Case Study on Machine Learning for Synthesizing Benchmarks

Andrés Goens, Alexander Brauckmann, Sebastian Ertel, Chris Cummins, Hugh Leather, Jeronimo Castrillon

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Good benchmarks are hard to find because they require a substantial effort to keep them representative for the constantly changing challenges of a particular field. Synthetic benchmarks are a common approach to deal with this, and methods from machine learning are natural candidates for synthetic benchmark generation. In this paper we investigate the usefulness of machine learning in the prominent CLgen benchmark generator. We re-evaluate CLgen by comparing the benchmarks generated by the model with the raw data used to train it. This re-evaluation indicates that, for the use case considered, machine learning did not yield additional benefit over a simpler method using the raw data. We investigate the reasons for this and provide further insights into the challenges the problem could pose for potential future generators.

Original language	English
Title of host publication	Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages
Editors	Tim Mattson, Abdullah Muzahid, Armando Solar-Lezama
Place of Publication	New York, NY, USA
Publisher	Association for Computing Machinery, Inc
Pages	38–46
Number of pages	9
ISBN (Print)	9781450367196
DOIs	https://doi.org/10.1145/3315508.3329976
Publication status	Published - 22 Jun 2019
Event	40th ACM SIGPLAN Conference on Programming Language Design and Implementation - Phoenix, United States Duration: 24 Jun 2019 → 26 Jun 2019 https://pldi19.sigplan.org/home

Conference

Conference	40th ACM SIGPLAN Conference on Programming Language Design and Implementation
Abbreviated title	PLDI 2019
Country/Territory	United States
City	Phoenix
Period	24/06/19 → 26/06/19
Internet address	https://pldi19.sigplan.org/home

Keywords / Materials (for Non-textual outputs)

Machine Learning
Benchmarking
Synthetic program generation
CLGen
Generative models

Access to Document

10.1145/3315508.3329976Licence: All Rights Reserved

https://dl.acm.org/doi/abs/10.1145/3315508.3329976Licence: All Rights Reserved

Cite this

Goens, A., Brauckmann, A., Ertel, S., Cummins, C., Leather, H., & Castrillon, J. (2019). A Case Study on Machine Learning for Synthesizing Benchmarks. In T. Mattson, A. Muzahid, & A. Solar-Lezama (Eds.), Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages (pp. 38–46). Association for Computing Machinery, Inc. https://doi.org/10.1145/3315508.3329976

Goens, Andrés ; Brauckmann, Alexander ; Ertel, Sebastian et al. / A Case Study on Machine Learning for Synthesizing Benchmarks. Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages. editor / Tim Mattson ; Abdullah Muzahid ; Armando Solar-Lezama. New York, NY, USA : Association for Computing Machinery, Inc, 2019. pp. 38–46

@inproceedings{7328aaef3f6049d99c7929b0f041d934,

title = "A Case Study on Machine Learning for Synthesizing Benchmarks",

abstract = "Good benchmarks are hard to find because they require a substantial effort to keep them representative for the constantly changing challenges of a particular field. Synthetic benchmarks are a common approach to deal with this, and methods from machine learning are natural candidates for synthetic benchmark generation. In this paper we investigate the usefulness of machine learning in the prominent CLgen benchmark generator. We re-evaluate CLgen by comparing the benchmarks generated by the model with the raw data used to train it. This re-evaluation indicates that, for the use case considered, machine learning did not yield additional benefit over a simpler method using the raw data. We investigate the reasons for this and provide further insights into the challenges the problem could pose for potential future generators.",

keywords = "Machine Learning, Benchmarking, Synthetic program generation, CLGen, Generative models",

author = "Andr{\'e}s Goens and Alexander Brauckmann and Sebastian Ertel and Chris Cummins and Hugh Leather and Jeronimo Castrillon",

year = "2019",

month = jun,

day = "22",

doi = "10.1145/3315508.3329976",

language = "English",

isbn = "9781450367196",

pages = "38–46",

editor = "Tim Mattson and Abdullah Muzahid and Armando Solar-Lezama",

booktitle = "Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages",

publisher = "Association for Computing Machinery, Inc",

note = "40th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2019 ; Conference date: 24-06-2019 Through 26-06-2019",

url = "https://pldi19.sigplan.org/home",

}

Goens, A, Brauckmann, A, Ertel, S, Cummins, C, Leather, H & Castrillon, J 2019, A Case Study on Machine Learning for Synthesizing Benchmarks. in T Mattson, A Muzahid & A Solar-Lezama (eds), Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages. Association for Computing Machinery, Inc, New York, NY, USA, pp. 38–46, 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, Phoenix, Arizona, United States, 24/06/19. https://doi.org/10.1145/3315508.3329976

A Case Study on Machine Learning for Synthesizing Benchmarks. / Goens, Andrés; Brauckmann, Alexander; Ertel, Sebastian et al.
Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages. ed. / Tim Mattson; Abdullah Muzahid; Armando Solar-Lezama. New York, NY, USA: Association for Computing Machinery, Inc, 2019. p. 38–46.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - A Case Study on Machine Learning for Synthesizing Benchmarks

AU - Goens, Andrés

AU - Brauckmann, Alexander

AU - Ertel, Sebastian

AU - Cummins, Chris

AU - Leather, Hugh

AU - Castrillon, Jeronimo

PY - 2019/6/22

Y1 - 2019/6/22

N2 - Good benchmarks are hard to find because they require a substantial effort to keep them representative for the constantly changing challenges of a particular field. Synthetic benchmarks are a common approach to deal with this, and methods from machine learning are natural candidates for synthetic benchmark generation. In this paper we investigate the usefulness of machine learning in the prominent CLgen benchmark generator. We re-evaluate CLgen by comparing the benchmarks generated by the model with the raw data used to train it. This re-evaluation indicates that, for the use case considered, machine learning did not yield additional benefit over a simpler method using the raw data. We investigate the reasons for this and provide further insights into the challenges the problem could pose for potential future generators.

AB - Good benchmarks are hard to find because they require a substantial effort to keep them representative for the constantly changing challenges of a particular field. Synthetic benchmarks are a common approach to deal with this, and methods from machine learning are natural candidates for synthetic benchmark generation. In this paper we investigate the usefulness of machine learning in the prominent CLgen benchmark generator. We re-evaluate CLgen by comparing the benchmarks generated by the model with the raw data used to train it. This re-evaluation indicates that, for the use case considered, machine learning did not yield additional benefit over a simpler method using the raw data. We investigate the reasons for this and provide further insights into the challenges the problem could pose for potential future generators.

KW - Machine Learning

KW - Benchmarking

KW - Synthetic program generation

KW - CLGen

KW - Generative models

U2 - 10.1145/3315508.3329976

DO - 10.1145/3315508.3329976

M3 - Conference contribution

SN - 9781450367196

SP - 38

EP - 46

BT - Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages

A2 - Mattson, Tim

A2 - Muzahid, Abdullah

A2 - Solar-Lezama, Armando

PB - Association for Computing Machinery, Inc

CY - New York, NY, USA

T2 - 40th ACM SIGPLAN Conference on Programming Language Design and Implementation

Y2 - 24 June 2019 through 26 June 2019

ER -

Goens A, Brauckmann A, Ertel S, Cummins C, Leather H, Castrillon J. A Case Study on Machine Learning for Synthesizing Benchmarks. In Mattson T, Muzahid A, Solar-Lezama A, editors, Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages. New York, NY, USA: Association for Computing Machinery, Inc. 2019. p. 38–46 doi: 10.1145/3315508.3329976

A Case Study on Machine Learning for Synthesizing Benchmarks

Abstract

Conference

Keywords / Materials (for Non-textual outputs)

Access to Document

Fingerprint

Cite this