Title: Scalar-tensor mixed polarization search of gravitational waves

Abstract: An additional scalar degree of freedom for a gravitational wave is often predicted in theories of gravity beyond general relativity and can be used for a model-agnostic test of gravity. In this letter, we report the direct search for the scalar-tensor mixed polarization modes of gravitational waves from compact binaries in a strong regime of gravity by analyzing the data of GW170814 and GW170817, which are the merger events of binary black holes and binary neutron stars, respectively. Consequently, we obtain the constraints on the ratio of scalar-mode amplitude to tensor-mode amplitude: $\lesssim 0.20$ for GW170814 and $\lesssim 0.068$ for GW170817, which are the tightest constraints on the scalar amplitude in a strong regime of gravity before merger.