Title: Unitary coupled-channels model for three-mesons decays of heavy mesons

Abstract: A unitary coupled-channels model is presented for investigating the decays of heavy mesons and excited meson states into three light pseudoscalar mesons. The model accounts for the three-mesons final state interactions in the decay processes, as required by both the three-body and two-body unitarity conditions. In the absence of the Z-diagram mechanisms that are necessary consequences of the three-body unitarity, our decay amplitudes are reduced to a form similar to those used in the so-called isobar-model analysis. We apply our coupled-channels model to the three-pions decays of a1(1260), pi2(1670), pi2(2100), and D0 mesons, and show that the Z-diagram mechanisms can contribute to the calculated Dalitz plot distributions by as much as 30% in magnitudes in the regions where f0(600), rho(770), and f2(1270) dominate the distributions. Also, by fitting to the same Dalitz plot distributions, we demonstrate that the decay amplitudes obtained with the unitary model and the isobar model can be rather different, particularly in the phase that plays a crucial role in extracting the CKM CP-violating phase from the data of B meson decays. Our results indicate that the commonly used isobar model analysis must be extended to account for the final state interactions required by the three-body unitarity to reanalyze the three-mesons decays of heavy mesons, thereby exploring hybrid or exotic mesons, and signatures of physics beyond the standard model.