Title: Avoiding a Cluster Catastrophe: Retention Efficiency and the Binary Black Hole Mass Spectrum

Abstract: The population of binary black hole mergers identified through gravitational waves has uncovered unexpected features in the intrinsic properties of black holes in the universe. One particularly surprising and exciting result is the possible existence of black holes in the pair-instability mass gap, $\sim50-120~M_\odot$. Dense stellar environments can populate this region of mass space through hierarchical mergers, with the retention efficiency of black hole merger products strongly dependent on the escape velocity of the host environment. We use simple toy models to represent hierarchical merger scenarios in various dynamical environments. We find that hierarchical mergers in environments with high escape velocities $(\gtrsim300~\mathrm{km\,s}^{-1})$ are efficiently retained. If such environments dominate the binary black hole merger rate, this would lead to an abundance of high-mass mergers that is potentially incompatible with the empirical mass spectrum from the current catalog of binary black hole mergers. Models that efficiently generate hierarchical mergers, and contribute significantly to the observed population, must therefore be tuned to avoid a "cluster catastrophe" of overproducing binary black hole mergers within and above the pair-instability mass gap.