Title: Introducing 4D Geometric Shell Shaping for Mitigating Nonlinear Interference Noise

Abstract: Four dimensional geometric shell shaping (4D-GSS) is introduced as an approach for closing the nonlinearity-caused shaping gap. This format is designed at the spectral efficiency of 8 b/4D-sym and is compared against polarization-multiplexed 16QAM (PM-16QAM) and probabilistically shaped PM-16QAM (PS-PM-16QAM) in a 400ZR-compatible transmission setup with high amount of nonlinearities. Reach increase and nonlinearity tolerance are evaluated in terms of achievable information rates and post-FEC bit-error rate. Numerical simulations for a single-span, single-channel show that 4D-GSS achieves increased nonlinear tolerance and reach increase against PM-16QAM and PS-PM-16QAM when optimized for bit-metric decoding (RBMD). In terms of RBMD, gains are small with a reach increase of 1.7% compared to PM-16QAM. When optimizing for mutual information, a larger reach increase of 3% is achieved compared to PM-16QAM. Moreover, the introduced GSS scheme provides a scalable framework for designing well-structured 4D modulation formats with low complexity.