Title: Homodyne coherent quantum noise cancellation in a hybrid optomechanical force sensor

Abstract: In this paper, we propose an experimentally viable scheme to enhance the sensitivity of force detection in a hybrid optomechanical setup assisted by squeezed vacuum injection, beyond the standard quantum limit (SQL). The scheme is based on a combination of the coherent quantum noise cancellation (CQNC) strategy with a variational homodyne detection of the cavity output spectrum in which the phase of the local oscillator is optimized. In CQNC, realizing a negative-mass oscillator in the system leads to exact cancellation of the backaction noise from the mechanics due to destructive quantum interference. Squeezed vacuum injection enhances this cancellation and allows sub-SQL sensitivity to be reached in a wide frequency band and at much lower input laser powers. We show here that the adoption of variational homodyne readout enables us to enhance this noise cancellation up to $40 ~\mathrm{dB}$ compared to the standard case of detection of the optical output phase quadrature, leading to a remarkable force sensitivity of the order of $10^{-19} \mathrm{N}/\sqrt{\mathrm{Hz}}$, around 2-order enhancement compared to the standard case. Moreover, we show that at nonzero cavity detuning, the signal response can be amplified at a level three to five times larger than that in the standard case without variational homodyne readout, improving the signal-to-noise-ratio (SNR). Finally, the variational readout CQNC developed in this paper may be applied to other optomechanical-like platforms such as levitated systems and multimode optomechanical arrays or crystals as well as Josephson-based optomechanical systems.