Title: Optimal Prototype Filter Design Based on New Mathematical Model of GFDM System

Abstract: We introduce a new perspective and mathematical model for the generalized frequency division multiplexing (GFDM) to analyze the effect of self-interferences and remove their detrimental impacts. Using this model, we express the GFDM signal in a vector form where the self-interferences are represented using two independent terms. For the first time, we model these interferences in terms of prototype filter parameters in the frequency domain and derive their analytical expressions. Building on these expressions, we propose the idea of optimal filter design as one of the solutions to annihilate self-interference in GFDM systems. We introduce the relationship between the required bandwidth for each subcarrier and the number of subsymbols in the proposed optimal prototype filter and show how the prototype filter optimally exploits the total system bandwidth. We derive analytical expressions to evaluate the bit error rate (BER) of the system and show how it outperforms its recent counterparts in the literature. We also perform computer simulations to verify the analytical results and find that the analytical results match well with those of the computer simulations.