In recent years, with the strong and outstanding development of the Internet, the need to refer to the feedback of previous customers when shopping online is increasing. Therefore, websites are developed to allow users to share experiences, reviews, comments and feedback about the services and products of businesses and organizations. The organizations also collect user feedback about their products and services to give better directions. However, with a large amount of user feedback about certain services and products, it is difficult for users, businesses, and organizations to pay attention to them all. Thus, an automatic system is necessary to analyze the sentiment of a customer feedback. Recently, the well-known pre-trained language models for Vietnamese (PhoBERT) achieved high performance in comparison with other approaches. However, this method may not focus on the local information in the text like phrases or fragments. In this paper, we propose a Convolutional Neural Network (CNN) model based on PhoBERT for sentiment classification. The output of contextualized embeddings of the PhoBERT’s last four layers is fed into the CNN. This makes the network capable of obtaining more local information from the sentiment. Besides, the PhoBERT output is also given to the transformer encoder layers in order to employ the self-attention technique, and this also makes the model more focused on the important information of the sentiment segments. The experimental results demonstrate that the proposed approach gives competitive performance compared to the existing studies on three public datasets with the opinions of Vietnamese people.