In a previous study, a fresh chili destemming system has been proposed, and the TCS3200 color sensor was used as low-cost equipment in the arrangement section of the system. However, the working and installing parameters have not yet been investigated in detail. This study focused on investigating the optimal working conditions of TCS3200 on the fresh chili destemming system. The study included three experiments including the effect of the red and green color channels, the effect of the height between the sensor and fruits, and the effect of the running velocity on the received signal level of the sensor. A total of 260 ripen fruits were chosen to test. The results showed that the red channel generated a stronger signal than the green one. The optimal height from the sensor to the fruits was obtained at 25 mm, and the maximum conveyor velocity is limited to 100 mm/s. This study helps to determine the optimal operating parameters and enhance the working ability of a low-cost sensor. However, the high-speed pushing actuating mechanism should be developed in further works.