Generative vs Discriminative Models
Given a setup with $X$ and $Y$ as input and output variable respectively.
The modeling to predict $Y$ from $X$ can be done in multiple ways.
Following are a few common ones:
- We learn the probability model $P(X | Y)$ and use Bayes’ theorem to predict
$P(Y | X) \propto P(X | Y) P(Y)$.
- It is a comprehensive modeling of the distribution of $X$ for every $Y$.
It requires significantly high amount of data in comparison with discriminative
modeling to achieve good predictive capability.
- We can sample using the distribution and generate synthetic $X$ for a given $Y$.
- Examples: ngram or neural language modeling and Naive Bayes models
- We directly learn $P(Y | X)$ and use it at the time of prediction.
- We are learning a model to just disambiguate the classes (i.e., $Y$) and not the
representation of the classes.
- In contrast with generative models, discriminative models are less demanding in terms of
- Examples: logistic regression, SVMs and tree based models