Title: Instabilities near ultrastrong coupling in an optomechanical cavity

Abstract: With artificially engineered systems, it is now possible to realize the coherent interaction rate, which can become comparable to the mode frequencies, a regime known as ultrastrong coupling (USC). We experimentally realize a cavity-electromechanical device using a superconducting waveguide cavity and a mechanical resonator. In the presence of a strong pump, the mechanical-polaritons splitting can nearly reach 81% of the mechanical frequency, overwhelming all the dissipation rates. Beyond the USC limit, the steady-state response becomes unstable. We systematically measure the boundary of the unstable response while varying the pump parameters. The unstable dynamics display rich phases, such as self-induced oscillations, period-doubling bifurcation, period-tripling oscillations, and ultimately leading to the chaotic behaviour. The experimental results and their theoretical modelling suggest the importance of residual nonlinear interaction terms in the weak-dissipative regime.