ImageNet dataset with more than 14 million images and 21,000 classes makes the problem of visual classification more difficult to deal with. One of the most difficult tasks is to train a fast and accurate visual classifier on several multi-core computers with limited individual memory resource. In this paper we address this challenge by extending both state-of-the-art large scale linear classifier (LIBLINEAR-CDBLOCK) and non-linear classifier (Power Mean SVM) for large scale visual classification tasks in these following ways: (1) an incremental learning method for Power Mean SVM, (2) a balanced bagging algorithm for training binary classifiers. Our approach has been evaluated on the 100 largest classes of ImageNet and ILSVRC 2010. The evaluation shows that our approach can save up to 82.01 % memory usage and the learning process is much faster than the original implementation and LIBLINEAR SVM.