Title: Ground states of atomic Fermi gases in a two-dimensional optical lattice with and without population imbalance

Abstract: We study the ground state phase diagram of population balanced and imbalanced ultracold atomic Fermi gases with a short range attractive interaction throughout the crossover from BCS to Bose-Einstein condensation (BEC), in a two-dimensional optical lattice (2DOL) comprised of two lattice and one continuum dimensions. We find that the mixing of lattice and continuum dimensions, together with population imbalance, has an extraordinary effect on pairing and the superfluidity of atomic Fermi gases. In the balanced case, the superfluid ground state prevails the majority of the phase space. However, for relatively small lattice hopping integral $t$ and large lattice constant $d$, a pair density wave (PDW) emerges unexpectedly at intermediate coupling strength, and the nature of the in-plane and overall pairing changes from particle-like to hole-like in the BCS and unitary regimes, associated with an abnormal increase in the Fermi volume with the pairing strength. In the imbalanced case, the stable polarized superfluid phase shrinks to only a small portion of the entire phase space spanned by $t$, $d$, imbalance $p$ and interaction strength $U$, mainly in the bosonic regime of low $p$, moderately strong pairing, and relatively large $t$ and small $d$. Due to the Pauli exclusion between paired and excessive fermions within the confined momentum space, a PDW phase emerges and the overall pairing evolves from particle-like into hole-like, as the pairing strength grows stronger in the BEC regime. In both cases, the ground state property is largely governed by the Fermi surface topology. These findings are very different from the cases of pure 3D continuum, 3D lattice or 1DOL.