Title: Noncollinear Antiferromagnetic Order and Effect of Spin-Orbit Coupling in Spin-1 Honeycomb Lattice

Abstract: Motivated by the recently synthesized insulating nickelate Ni$_2$Mo$_3$O$_8$, which has been reported to have an unusual non-collinear magnetic order of Ni$^{2+}$ $S=1$ moments with a nontrivial angle between adjacent spins, we construct an effective spin-1 model on the honeycomb lattice, with the exchange parameters determined with the help of first principles electronic structure calculations. The resulting bilinear-biquadratic model, supplemented with the realistic crystal-field induced anisotropy, favors the collinear N\'eel state. We find that the crucial key to explaining the observed noncollinear spin structure is the inclusion of the Dzyaloshinskii--Moriya (DM) interaction between the neighboring spins. By performing the variational mean-field and linear spin-wave theory (LSWT) calculations, we determine that a realistic value of the DM interaction $D\approx 2.78$ meV is sufficient to quantitatively explain the observed angle between the neighboring spins. We furthermore compute the spectrum of magnetic excitations within the LSWT and random-phase approximation (RPA) which should be compared to future inelastic neutron measurements.