We propose the new multi-class of bagged proximal support vector machines (MC-Bag-PSVM) for handling the ImageNet challenging problem with very large number of images and a thousand classes. Our MC-Bag-PSVM trains in the parallel manner ensemble binary PSVM classifiers used for the One-Versus-All (OVA) multi-class strategy on multi-core computer with GPUs. The binary PSVM model is constructed by bagged binary PSVM models built in under-sampling training dataset. The numerical test results on ILSVRC 2010 dataset show that our MC-Bag-PSVM algorithm is faster and more accurate than the state-of-the-art linear SVM algorithm. An example of its effectiveness is given with an accuracy of 75.64% obtained in the classification of ImageNet-1000 dataset having 1,261,405 images in 2048 deep features into 1,000 classes in 29.5 min using a PC Intel(R) Core i7-4790 CPU, 3.6 GHz, 4 cores and Gigabyte GeForce RTX 2080Ti 11 GB GDDR6, 4352 CUDA cores.