We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.SE

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Software Engineering

Title: Measuring Discrimination to Boost Comparative Testing for Multiple Deep Learning Models

Abstract: The boom of DL technology leads to massive DL models built and shared, which facilitates the acquisition and reuse of DL models. For a given task, we encounter multiple DL models available with the same functionality, which are considered as candidates to achieve this task. Testers are expected to compare multiple DL models and select the more suitable ones w.r.t. the whole testing context. Due to the limitation of labeling effort, testers aim to select an efficient subset of samples to make an as precise rank estimation as possible for these models. To tackle this problem, we propose Sample Discrimination based Selection (SDS) to select efficient samples that could discriminate multiple models, i.e., the prediction behaviors (right/wrong) of these samples would be helpful to indicate the trend of model performance. To evaluate SDS, we conduct an extensive empirical study with three widely-used image datasets and 80 real world DL models. The experimental results show that, compared with state-of-the-art baseline methods, SDS is an effective and efficient sample selection method to rank multiple DL models.
Comments: ICSE'2021
Subjects: Software Engineering (cs.SE); Machine Learning (cs.LG)
Cite as: arXiv:2103.04333 [cs.SE]
  (or arXiv:2103.04333v2 [cs.SE] for this version)

Submission history

From: Linghan Meng [view email]
[v1] Sun, 7 Mar 2021 12:03:59 GMT (1287kb,D)
[v2] Tue, 9 Mar 2021 13:10:13 GMT (1286kb,D)

Link back to: arXiv, form interface, contact.