Chilies and their products have become popular in everyday life. However, in Vietnam today, their care and harvesting of them are done mainly by manual methods. As a part of the project to develop an automatic harvesting robot, this research focuses on detecting and locating chili fruits in three-dimensional space. First, the Yolov5 model is fine-tuned and trained to detect the fresh chili fruit object. Then, a stereo camera is used to capture an image and determine the disparity of the fruits in the left/right images using matching methods. The distance and 3D location are estimated by the triangle method via the disparity. In addition, an image calibration method is also implemented to obtain the camera’s focal length parameters as well as to reduce the camera’s fisheye effect, increasing estimation accuracy. The model can detect a chili at a size of 9.4 × 55 mm up to a distance of 600 mm and an image resolution of 480 × 480 pixels. In further studies, this position data will be transferred to the robotic arm to perform necessary operations such as harvesting and
avoiding obstacles. The positioning accuracy as well as the ability to detect small fruits would be improved to enhance the working ability of the whole system.