Shane Chu

Local representations vs. Distributed representations

Local and distributed representation is one of the most basic and useful dichotomies to think about problem formulations.

We can understand the difference between local vs. distributed representation in the following picture from Bhaskar Mitra:

In (a) the local representation, we represent different concepts in the model using a distinct set of parameters. The picture illustrates a local representation using one-hot encodings, where each possible one-hot vector represents a single concept (e.g., banana, mango, dog).

In (b) the distributed representation, we represent different concepts in the model using various combinations of parameters.

It makes a big difference in practice when deciding which representation to use for modeling tasks. The number of concepts one could represent in a local representation is linear in the number of parameters. The number of concepts one could describe in a distributed representation is roughly exponential in the number of parameters (See this at the 26:53 mark by Ruslan Salakhutdinov). So a distributed representation model can use fewer parameters to describe much more complicated concepts.