TopIntroduction
A basically important problem in Image Engineering (IE), Pattern Recognition (PR), and Computer Vision (CV) is how to find a suitable representation of multivariate data.
In many cases, the primitive data sets or observations are organized as data matrices (or tensors), and described by linear (or multi-linear) combination models. From the algebraic perspective, the formulation of dimensionality reduction can be regarded as decomposing the original data matrix into two factor matrices. The canonical methods, such as principal component analysis (PCA), linear discriminant analysis (LDA), independent component analysis (ICA), and vector quantization (VQ) et al., are the exemplars of such low-rank approximations. They differ from one another in the statistical properties attributable to the different constraints imposed on the component matrices and their underlying structures; however, they have something in common that there is no constraint in the sign of the elements in the factorized matrices. In other words, the negative component or the subtractive combination is allowed in the representation.
In contrast, a new paradigm of factorization — Non-negative Matrix Factorization (NMF) is quite different in this aspect. NMF is a recently developed, biologically inspired method for nonlinearly finding purely additive, sparse, linear, and low-dimension representations of non-negative multivariate data to consequently make latent structure, feature or pattern in the data clear (Lee, 1999).
NMF makes all representation components non-negative (only purely additive representations are allowable) and nonlinearly implements dimension reduction. Psychological and physiological evidence for NMF is that perception of the whole is based on perception of its parts, which is compatible with the intuitive notion of combining parts to form a whole (Lee, 1999), therefore it is considered to grasp the essence of intelligent or biological data representation in some degree.
Far beyond a mathematical exploration, the philosophy underlying NMF, which tries to formulate a feasible model for learning object parts, is closely relevant to perception mechanism. While the parts-based representation seems intuitive, it is indeed based on physiological and psychological evidence: perception of the whole is based on perception of its parts (Paatero, 1997). In fact there are two complementary connotations in non-negativity — non-negative component and purely additive combination. On the one hand, the negative values of both observations and latent components are physically meaningless in many kinds of real world data, such as image analysis tasks. Meanwhile, the discovered prototypes commonly correspond with certain semantic interpretation.
Besides, NMF usually produces a sparse representation of data, which has been shown to be a useful middle ground between a completely distributed representation and a unary representation (Field 1994). The non-negativity constraint will lead to sort of sparseness naturally (Lee, 1999), which is proved to be a highly effective representation distinguished from both the completely distributed and the solely active component description (Field, 1994).
When NMF is interpreted as a neural-network learning algorithm depicting how the visible variables are generated from the hidden ones, the parts-based representation is obtained from the additive model. A positive number indicates the presence and a zero value represents the absence of some event or component. This conforms nicely to the dualistic properties of neural activity and synaptic strengths in neurophysiology: either excitatory or inhibitory without changing sign (Lee, 1999).