### Current browse context:

cs.LG

### Change to browse by:

### References & Citations

# Computer Science > Machine Learning

# Title: On the Provable Generalization of Recurrent Neural Networks

(Submitted on 29 Sep 2021 (v1), revised 9 Dec 2021 (this version, v2),

*latest version 26 Jan 2022*(v4))Abstract: Recurrent Neural Network (RNN) is a fundamental structure in deep learning. Recently, some works study the training process of over-parameterized neural networks, and show that over-parameterized networks can learn functions in some notable concept classes with a provable generalization error bound. In this paper, we analyze the training and generalization for RNNs with random initialization, and provide the following improvements over recent works:

1) For a RNN with input sequence $x=(X_1,X_2,...,X_L)$, previous works study to learn functions that are summation of $f(\beta^T_lX_l)$ and require normalized conditions that $||X_l||\leq\epsilon$ with some very small $\epsilon$ depending on the complexity of $f$. In this paper, using detailed analysis about the neural tangent kernel matrix, we prove a generalization error bound to learn such functions without normalized conditions and show that some notable concept classes are learnable with the numbers of iterations and samples scaling almost-polynomially in the input length $L$.

2) Moreover, we prove a novel result to learn N-variables functions of input sequence with the form $f(\beta^T[X_{l_1},...,X_{l_N}])$, which do not belong to the "additive" concept class, i,e., the summation of function $f(X_l)$. And we show that when either $N$ or $l_0=\max(l_1,..,l_N)-\min(l_1,..,l_N)$ is small, $f(\beta^T[X_{l_1},...,X_{l_N}])$ will be learnable with the number iterations and samples scaling almost-polynomially in the input length $L$.

## Submission history

From: Lifu Wang [view email]**[v1]**Wed, 29 Sep 2021 02:06:33 GMT (33kb)

**[v2]**Thu, 9 Dec 2021 14:32:34 GMT (31kb)

**[v3]**Mon, 3 Jan 2022 15:23:58 GMT (32kb)

**[v4]**Wed, 26 Jan 2022 17:10:50 GMT (34kb)

Link back to: arXiv, form interface, contact.