We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Electrical Engineering and Systems Science > Systems and Control

Title: ConEx: Efficient Exploration of Big-Data System Configurations for Better Performance

Abstract: Configuration space complexity makes the big-data software systems hard to configure well. Consider Hadoop, with over nine hundred parameters, developers often just use the default configurations provided with Hadoop distributions. The opportunity costs in lost performance are significant. Popular learning-based approaches to auto-tune software does not scale well for big-data systems because of the high cost of collecting training data. We present a new method based on a combination of Evolutionary Markov Chain Monte Carlo (EMCMC) sampling and cost reduction techniques to cost-effectively find better-performing configurations for big data systems. For cost reduction, we developed and experimentally tested and validated two approaches: using scaled-up big data jobs as proxies for the objective function for larger jobs and using a dynamic job similarity measure to infer that results obtained for one kind of big data problem will work well for similar problems. Our experimental results suggest that our approach promises to significantly improve the performance of big data systems and that it outperforms competing approaches based on random sampling, basic genetic algorithms (GA), and predictive model learning. Our experimental results support the conclusion that our approach has strongly demonstrated potential to significantly and cost-effectively improve the performance of big data systems.
Subjects: Systems and Control (eess.SY); Machine Learning (cs.LG); Software Engineering (cs.SE)
Cite as: arXiv:1910.09644 [eess.SY]
  (or arXiv:1910.09644v2 [eess.SY] for this version)

Submission history

From: Rahul Krishna [view email]
[v1] Thu, 17 Oct 2019 20:37:07 GMT (1668kb,D)
[v2] Mon, 22 Jun 2020 21:13:24 GMT (2875kb,D)

Link back to: arXiv, form interface, contact.