We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.SE

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Software Engineering

Title: Groot: An Event-graph-based Approach for Root Cause Analysis in Industrial Settings

Abstract: For large-scale distributed systems, it's crucial to efficiently diagnose the root causes of incidents to maintain high system availability. The recent development of microservice architecture brings three major challenges (i.e., operation, system scale, and monitoring complexities) to root cause analysis (RCA) in industrial settings. To tackle these challenges, in this paper, we present Groot, an event-graph-based approach for RCA. Groot constructs a real-time causality graph based on events that summarize various types of metrics, logs, and activities in the system under analysis. Moreover, to incorporate domain knowledge from site reliability engineering (SRE) engineers, Groot can be customized with user-defined events and domain-specific rules. Currently, Groot supports RCA among 5,000 real production services and is actively used by the SRE teamin a global e-commerce system serving more than 185 million active buyers per year. Over 15 months, we collect a data setcontaining labeled root causes of 952 real production incidents for evaluation. The evaluation results show that Groot is able to achieve 95% top-3 accuracy and 78% top-1 accuracy. To share our experience in deploying and adopting RCA in industrial settings, we conduct survey to show that users of Grootfindit helpful and easy to use. We also share the lessons learnedfrom deploying and adopting Grootto solve RCA problems inproduction environments.
Subjects: Software Engineering (cs.SE)
Journal reference: Proceedings of the thirty-sixth IEEE/ACM international conference on Automated software engineering, 2021
Cite as: arXiv:2108.00344 [cs.SE]
  (or arXiv:2108.00344v2 [cs.SE] for this version)

Submission history

From: Hanzhang Wang [view email]
[v1] Sun, 1 Aug 2021 00:33:52 GMT (6274kb,D)
[v2] Wed, 22 Sep 2021 16:16:23 GMT (6798kb,D)

Link back to: arXiv, form interface, contact.