Title: Combinatorial Information Theory: I. Philosophical Basis of Cross-Entropy and Entropy

Abstract: The three main theoretical bases of the concepts of entropy and cross-entropy - information-theoretic, axiomatic and combinatorial - are critically examined. It is shown that the combinatorial basis, proposed by Boltzmann and Planck, is the most fundamental (most primitive) basis of these concepts, since it provides (i) a derivation of the Kullback-Leibler cross-entropy and Shannon entropy functions, as simplified forms of the multinomial distribution subject to the Stirling approximation; (ii) an explanation for the need to maximize entropy (or minimize cross-entropy) to find the most probable realization; and (iii) the means to derive entropy and cross-entropy functions for systems which do not satisfy the multinomial distribution, i.e. which fall outside the domain of the Kullback-Leibler and Shannon measures. The information-theoretic and axiomatic bases of cross-entropy and entropy - whilst of tremendous importance and utility - are therefore seen as secondary viewpoints, which lack the breadth of the combinatorial approach. Appreciation of this reasoning would permit development of a powerful body of "combinatorial information theory", as a tool for statistical inference in all fields (inside and outside science). The essential features of Jaynes' analysis of entropy and cross-entropy - reinterpreted in light of the combinatorial approach - are outlined, including derivation of probability distributions, ensemble theory, Jaynes relations, fluctuation theory and Jaynes' entropy concentration theorem. New results include a generalized free energy (or ``free information'') concept, a generalized Gibbs-Duhem relation and phase rule. Generalized (combinatorial) definitions of entropy and cross-entropy, valid for any combinatorial system, are then proposed and examined in detail.