Title: On Peterson's open problem and representations of the general linear groups

Abstract: Fix $\mathbb Z/2$ is the prime field of two elements and write $\mathcal A_2$ for the mod $2$ Steenrod algebra. Denote by $GL_d:= GL(d, \mathbb Z/2)$ the general linear group of rank $d$ over $\mathbb Z/2$ and by $\mathscr P_d$ the polynomial algebra $\mathbb Z/2[x_1, x_2, \ldots, x_d]$ as a connected unstable left $\mathcal A_2$-module on $d$ generators of degree one. We study the {\it Peterson hit problem} of finding a minimal set of $\mathcal A_2$-generators for $\mathscr P_d.$ Equivalently, we need to determine a basis for the $\mathbb Z/2$-vector space $$Q\mathscr P_d := \mathbb Z/2\otimes_{\mathcal A_2} \mathscr P_d \cong \mathscr P_d/\mathcal A_2^+\mathscr P_d$$ in each degree $n\geq 1.$ Note that this space is a representation of $GL_d$ over $\mathbb Z/2.$ The problem for $d= 5$ is not yet completely solved, and unknow in general.
In this work, we give an explicit solution to the hit problem of five variables in the generic degree $n = r(2^t -1) + 2^ts$ with $r = d = 5,\ s =8$ and $t$ an arbitrary non-negative integer. An application of this study to the cases $t = 0$ and $t = 1$ shows that the Singer algebraic transfer of rank $5$ is an isomorphism in bidegrees $(5, 5+8)$ and $(5, 5+21).$ Here, the Singer transfer of rank $d$ is a $\mathbb Z/2$-algebra homomorphism from $GL_d$-coinvariants of certain subspaces of $Q\mathscr P_d$ to the cohomology groups of the Steenrod algebra, ${\rm Ext}_{\mathcal A_2}^{d,d+*}(\mathbb Z/2, \mathbb Z/2).$ It is one of the useful tools for describing these cohomology groups.