We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Why MDAC? A Multi-domain Activation Function

Abstract: In this study, a novel, general and ingenious activation function termed MDAC is proposed to surmount the troubles of gradient vanishing and non-differentiable existence. MDAC approximately inherits the properties of exponential activation function (such as Tanh family) and piecewise linear activation function (such as ReLU family). Specifically, in the positive region, the adaptive linear structure is designed to respond to various domain distributions. In the negative region, the combination of exponent and linearity is considered to conquer the obstacle of gradient vanishing. Furthermore, the non-differentiable existence is eliminated by smooth approximation. Experiments show that MDAC improves performance on both classical models and pre-training optimization models in six domain datasets by simply changing the activation function, which indicates MDAC's effectiveness and pro-gressiveness. MDAC is superior to other prevalent activation functions in robustness and generalization, and can reflect excellent activation performance in multiple domains.
Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL); Neural and Evolutionary Computing (cs.NE)
Cite as: arXiv:2111.13858 [cs.LG]
  (or arXiv:2111.13858v1 [cs.LG] for this version)

Submission history

From: Zhenhua Wang [view email]
[v1] Sat, 27 Nov 2021 10:05:12 GMT (514kb)

Link back to: arXiv, form interface, contact.