Title: Optimal smoothing in nonparametric mixed-effect models

Abstract: Mixed-effect models are widely used for the analysis of correlated data such as longitudinal data and repeated measures. In this article, we study an approach to the nonparametric estimation of mixed-effect models. We consider models with parametric random effects and flexible fixed effects, and employ the penalized least squares method to estimate the models. The issue to be addressed is the selection of smoothing parameters through the generalized cross-validation method, which is shown to yield optimal smoothing for both real and latent random effects. Simulation studies are conducted to investigate the empirical performance of generalized cross-validation in the context. Real-data examples are presented to demonstrate the applications of the methodology.