Lack of diseases detection in plants frequently results in the spread of diseases that are difficult to treat and expensive. Rapid diseases recognition enables farmers to control the diseases with appropriate treatment. This study aims to support chili farmers in identifying chili plant diseases based on leaf images. This work presents a CNN design based on several existing CNN architectures that have been fine-tuned to achieve the highest possible accuracy. The study found that the ResNet101 model with the Tanh activation function, SGD optimizer, and Reduced Learning Rate (ReduceLR) schedule, achieved a peak classification accuracy of 99.53%. This significant improvement demonstrates the potential of using advanced CNN techniques and fine-tuning strategies to enhance model accuracy in agricultural applications. The implications of this study extend to the field of precision agriculture, suggesting that the proposed model can be integrated into smart farming systems to improve the timely and efficient control of chili leaf diseases. Such advancements not only enhance crop yields but also contribute to sustainable agricultural practices and the economic stability of chili farmers.