This paper addresses several critical agricultural classification problems, e.g. grain discoloration and medicinal plants identification and classification, in Vietnam via combining the idea of knowledge transferability and state-of-the-art deep convolutional neural networks. Grain discoloration disease of rice is an emerging threat to rice harvest in Vietnam as well as all over the world and it acquires specific attention as it results in qualitative loss of harvested crop. Medicinal plants are an important element of indigenous medical systems. These resources are usually regarded as a part of culture’s traditional knowledge. Accurate classification is preliminary to any kind of intervention and recommendation of services. Hence, leveraging technology in automatic classification of these problems has become essential. Unfortunately, building and training a machine learning model from scratch is next to impossible due to the lack of hardware infrastructure and finance support. It painfully restricts the requirements of rapid solutions to deal with the demand. For this purpose, the authors have exploited the idea of transfer learning which is the improvement of learning in a new prediction task through the transferability of knowledge from a related prediction task that has already been learned. By utilizing state-of-the-art deep networks re-trained upon our collected data, our extensive experiments show that the proposed combination performs perfectly and achieves the classification accuracy of 98.7% and 98.5% on our collected datasets within the acceptable training time on a normal laptop. A mobile application is also deployed to facilitate further integrated recommendation and services.