We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:


References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Software Engineering

Title: Stop Building Castles on a Swamp! The Crisis of Reproducing Automatic Search in Evidence-based Software Engineering

Authors: Zheng Li
Abstract: The evidence-based approach has increasingly been employed to synthesize empirical findings from the primary research in software engineering. Nevertheless, the reproducibility of evidence-based software engineering (EBSE) studies seems to be underemphasized. In our investigation into the automatic search of 311 sample studies, more than 50% of the search strings are not reusable; about 87.5% of the search activities (e.g., search field settings) are unrepeatable; and more than 95% of the whole automatic search implementations are unreproducible. Considering that searching is a cornerstone of an EBSE study, we are afraid that the reproducibility of the current secondary research could be worse than we can imagine. By analyzing and reporting the root causes of the aforementioned observations, we urge collaboration and cooperation among all the stakeholders in our community to improve the research reproducibility in EBSE.
Subjects: Software Engineering (cs.SE)
Cite as: arXiv:2103.01381 [cs.SE]
  (or arXiv:2103.01381v1 [cs.SE] for this version)

Submission history

From: Zheng Li [view email]
[v1] Tue, 2 Mar 2021 00:40:49 GMT (109kb,D)

Link back to: arXiv, form interface, contact.