We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CR

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Cryptography and Security

Title: Why Most Results of Socio-Technical Security User Studies Are False

Authors: Thomas Gross
Abstract: Background. In recent years, cyber security user studies have been scrutinized for their reporting completeness, statistical reporting fidelity, statistical reliability and biases. It remains an open question what strength of evidence positive reports of such studies actually yield. We focus on the extent to which positive reports indicate relation true in reality, that is, a probabilistic assessment.
Aim. This study aims at establishing the overall strength of evidence in cyber security user studies, with the dimensions -- Positive Predictive Value (PPV) and its complement False Positive Risk (FPR), -- Likelihood Ratio (LR), and -- Reverse-Bayesian Prior (RBP) for a fixed tolerated False Positive Risk.
Method. Based on $431$ coded statistical inferences in $146$ cyber security user studies from a published SLR covering the years 2006-2016, we first compute a simulation of the a posteriori false positive risk based on assumed prior and bias thresholds. Second, we establish the observed likelihood ratios for positive reports. Third, we compute the reverse Bayesian argument on the observed positive reports by computing the prior required for a fixed a posteriori false positive rate.
Results. We obtain a comprehensive analysis of the strength of evidence including an account of appropriate multiple comparison corrections. The simulations show that even in face of well-controlled conditions and high prior likelihoods, only few studies achieve good a posteriori probabilities.
Conclusions. Our work shows that the strength of evidence of the field is weak and that most positive reports are likely false. From this, we learn what to watch out for in studies to advance the knowledge of the field.
Comments: Open Science Framework: this https URL, 19 pages, Author's copy of the work. The work was supported by the ERC Starting Grant CASCAde, GA no. 716980
Subjects: Cryptography and Security (cs.CR); Human-Computer Interaction (cs.HC)
Cite as: arXiv:2109.10839 [cs.CR]
  (or arXiv:2109.10839v1 [cs.CR] for this version)

Submission history

From: Thomas Gross [view email]
[v1] Wed, 22 Sep 2021 16:59:06 GMT (1984kb,D)

Link back to: arXiv, form interface, contact.