We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Deep discriminative to kernel generative modeling

Abstract: The fight between discriminative versus generative goes deep, in both the study of artificial and natural intelligence. In our view, both camps have complementary value, so, we sought to synergistic combine them. Here, we propose a methodology to convert deep discriminative networks to kernel generative networks. We leveraged the fact that deep models, including both random forests and deep networks, learn internal representations which are unions of polytopes with affine activation functions to conceptualize them both as generalized partitioning rules. From that perspective, we used foundational results on the relationship between histogram rules and kernel density estimators to obtain class conditional kernel density estimators from the deep models. We then studied the trade-offs we observed from implementing this strategy in low-dimensional settings, both theoretically and empirically, as a first step towards understanding. Theoretically, we show conditions under which our generative models are more efficient than the corresponding discriminative approaches. Empirically, when sample sizes are relatively high, the discriminative models tend to perform as well or better on discriminative metrics, such as classification rates and posterior calibration. However, when sample sizes are relatively low, the generative models outperform the discriminative ones even on discriminative metrics. Moreover, the generative ones can also sample from the distribution, obtain smoother posteriors, and extrapolate beyond the convex hull of the training data to handle OOD inputs more reasonably. Via human experiments we illustrate that our kernel generative networks (Kragen) behave more like humans than deep discriminative networks. We believe this approach may be an important step in unifying the thinking and approaches across the discriminative and generative divide.
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Data Structures and Algorithms (cs.DS); Neurons and Cognition (q-bio.NC); Machine Learning (stat.ML)
Cite as: arXiv:2201.13001 [cs.LG]
  (or arXiv:2201.13001v4 [cs.LG] for this version)

Submission history

From: Jayanta Dey [view email]
[v1] Mon, 31 Jan 2022 05:07:16 GMT (16278kb,D)
[v2] Sun, 6 Feb 2022 14:38:23 GMT (16262kb,D)
[v3] Mon, 14 Feb 2022 14:14:47 GMT (16263kb,D)
[v4] Fri, 25 Mar 2022 03:41:23 GMT (16263kb,D)

Link back to: arXiv, form interface, contact.