The personality in the present world plays a critical role in social interactions, the use of modern technologies, and individuals' success. Therefore, in the last two decades, the study of automatic personality perception and automatic personality recognition has become more prevalent than speech processing. These studies have shown that personality traits affect acoustic features. However, the intrinsic imbalanced distribution of personality classes across the dataset is an issue mentioned in most previous studies and the classification results suffer from it. In this paper, an innovative supervised k-fold cross-validation method was proposed to cope with the problem of affecting the imbalanced distribution of data across different classes. The classification outcomes showed better performance in comparison with three traditional data balancing methods. Moreover, the obtained results of the proposed evaluation method indicated that the proposed method acts as a k-fold cross-validation method if the data distribution is balanced; otherwise, it will improve the classification results.