Over the past few decades, a lot of attention has been drawn to large-scale streaming data analysis, where researchers are faced with huge amount of high-dimensional data acquired in a stream fashion. In this case, conventional algorithms that compute the result from scratch whenever a new data comes are highly inefficient. To handle this problem, we propose a new incremental regularized least squares algorithm that is applied to supervised dimensionality reduction of large-scale streaming data with focus on linear discriminant analysis. Experimental results on real-world data sets demonstrate the effectiveness and efficiency of our algorithms.