We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: There is Limited Correlation between Coverage and Robustness for Deep Neural Networks

Abstract: Deep neural networks (DNN) are increasingly applied in safety-critical systems, e.g., for face recognition, autonomous car control and malware detection. It is also shown that DNNs are subject to attacks such as adversarial perturbation and thus must be properly tested. Many coverage criteria for DNN since have been proposed, inspired by the success of code coverage criteria for software programs. The expectation is that if a DNN is a well tested (and retrained) according to such coverage criteria, it is more likely to be robust. In this work, we conduct an empirical study to evaluate the relationship between coverage, robustness and attack/defense metrics for DNN. Our study is the largest to date and systematically done based on 100 DNN models and 25 metrics. One of our findings is that there is limited correlation between coverage and robustness, i.e., improving coverage does not help improve the robustness. Our dataset and implementation have been made available to serve as a benchmark for future studies on testing DNN.
Subjects: Machine Learning (cs.LG); Software Engineering (cs.SE); Machine Learning (stat.ML)
Cite as: arXiv:1911.05904 [cs.LG]
  (or arXiv:1911.05904v1 [cs.LG] for this version)

Submission history

From: Dong Yizhen [view email]
[v1] Thu, 14 Nov 2019 02:36:40 GMT (4869kb,D)

Link back to: arXiv, form interface, contact.