Title: The Hall conductance, topological quantum phase transition and the Diophantine equation on honeycomb lattice

Abstract: We consider a tight-binding model with the nearest neighbour hopping integrals on the honeycomb lattice in a magnetic field. Assuming one of the three hopping integrals, which we denote t_a, can take a different value from the two others, we study quantum phase structures controlled by the anisotropy of the honeycomb lattice.For weak and strong t_a regions, respectively, the Hall conductances are calculated algebraically by using the Diophantine equation. Except for a few specific gaps, we completely determine the Hall conductances in these two regions including those for subband gaps. In a weak magnetic field, it is found that the weak t_a region shows the unconventional quantization of the Hall conductance, \sigma_{xy}=-(e^2/h)(2n+1), (n=0,\pm 1,\pm 2,...), near the half-filling, while the strong t_a region shows only the conventional one, \sigma_{xy}=-(e^2/h)n,(n=0,1,2,...). From topological nature of the Hall conductance, the existence of gap closing points and quantum phase transitions in the intermediate t_a region are concluded. We also study numerically the quantum phase structure in detail, and find that even when t_a=1, namely in graphene case, the system is in the weak t_a phase except when the Fermi energy is located near the van Hove singularity or the lower and upper edges of the spectrum.