Latent variable models have a broad set of applications in domains such as social networks, natural language processing, computer vision and computational biology. Training them on a large scale is challenging due to non-convexity of the objective function. We propose a unified framework that exploits tensor algebraic constraints of the (low order) moments of the models. This versatile framework is guaranteed to estimate the correct model consistently and the spectral decomposition (matrix/tensor decomposition) proposed are embarrassingly parallel and has global convergence guarantees using SGD despite the non-convexity of the objective function. Topic Modeling will be discussed extensively, as well as user commonality inference in the large-scale social network using Mixed Membership Stochastic Blockmodel, and convolutional dictionary learning for text paraphrase embedding learning.