Novel Genre Classification on Imbalanced Data using Machine Learning and Transformers

Dania Qomariyah, Muljono Muljono

Abstract


Novel genre classification based on plot summaries can support book search and reading recommendation systems. However, genre distributions in real-world datasets are generally imbalanced. This study compares Random Forest, Logistic Regression, linear Support Vector Machine (SVM), and Bidirectional Encoder Representations from Transformers (BERT) using 4,446 cleaned records out of an initial 4,657 records after removing conflicting labels and duplicate entries. The classical machine learning models employed TF-IDF representations with 500 features and unrestricted features incorporating bigrams, while BERT was evaluated using sequence lengths of 256 and 512 tokens, with both weighted and unweighted cross-entropy loss. Evaluation was conducted using a fixed 70/15/15 train-validation-test split and stratified k-fold cross-validation with k = 3, 5, and 10. Under the fixed split, random resampling reduced Random Forest accuracy from 60.42% to 45.88% without improving macro-F1, whereas class weighting increased macro-F1 from 0.424 to 0.586. In cross-validation, BERT with a 512-token sequence length and class-weighted loss achieved the highest average macro-F1 scores (0.809–0.816) and accuracy (77.3%–78.7%), outperforming SVM, which achieved macro-F1 scores of 0.752–0.766 and accuracy of 75.1%–76.1%. At k = 10, the largest F1-score improvements were observed in the romance, travel, and sports genres, although improvements were also evident in larger genres. Across all three k values, the training time per fold for this BERT configuration was approximately 5,300–6,600 times that of SVM, considering training time only. SVM with unrestricted TF-IDF features remained a strong and computationally efficient baseline. These findings indicate that BERT delivers improved classification performance at the cost of substantially higher training time, suggesting that model selection should consider the priorities and computational constraints of the intended application.

Keywords


class imbalance; novel genre classification; random forest; synopsis; transformer

Full Text:

PDF

References


E. M. N. Subroto and M. Faisal, “A Multi-Label Book Genre Classification: Comparison of Machine Learning Techniques and Problem Transformation Methods,” Jurnal Ilmiah Kursor, Vol. 13, No. 2, Dec. 2025, DOI: 10.21107/kursor.v13i2.389.

X. Yang and Z. Zhang, “Enhancing Book Genre Classification with BERT and InceptionV3: A Deep Learning Approach for Libraries,” PeerJ Comput. SCI., Vol. 11, p. e2934, Jun. 2025, DOI: 10.7717/peerj-cs.2934.

A. Sethy, A. K. Rout, A. Uriti, and S. P. Yalla, “A Comprehensive Machine Learning Framework for Automated Book Genre Classifier,” Revue d’Intelligence Artificielle, Vol. 37, No. 3, pp. 745–751, Jun. 2023, DOI: 10.18280/ria.370323.

P. Mahanani and C. Fibriani, “Novel Genre Classification based on Synopsis using the Random Forest Algorithm,” SISTEMASI, Vol. 15, No. 1, p. 296, Jan. 2026, DOI: 10.32520/stmsi.v15i1.5815.

K. Hamaguchi, Y. Matsuda, and J. Kuroiwa, “Potential of Contextual Representations for Genre Classification of Modern Japanese Literary Works using BERT,” Nonlinear Theory and Its Applications, IEICE, Vol. 17, No. 3, pp. 805–821, 2026, DOI: 10.1587/nolta.17.805.

E. C. Garrido-Merchan, R. Gozalo-Brizuela, and S. Gonzalez-Carvajal, “Comparing BERT Against Traditional Machine Learning Models in Text Classification,” Journal of Computational and Cognitive Engineering, Vol. 2, No. 4, pp. 352–356, Apr. 2023, DOI: 10.47852/bonviewJCCE3202838.

N. Aydın, O. A. Erdem, and A. Tekerek, “Comparative Analysis of Traditional Machine Learning and Transformer-based Deep Learning Models for Text Classification,” Politeknik Dergisi, Vol. 28, No. 2, pp. 445–452, Mar. 2025, DOI: 10.2339/politeknik.1469530.

Y.-C. Lin, S.-A. Chen, J.-J. Liu, and C.-J. Lin, “Linear Classifier: An often-Forgotten baseline for Text Classification,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Stroudsburg, PA, USA: Association for Computational Linguistics, Aug. 2023, pp. 1876–1888. DOI: 10.18653/v1/2023.acl-short.160.

J. A. Benítez-Andrades, J.-M. Alija-Pérez, M.-E. Vidal, R. Pastor-Vargas, and M. T. García-Ordás, “Traditional Machine Learning Models and Bidirectional Encoder Representations from Transformer (BERT)–based Automatic Classification of Tweets about Eating Disorders: Algorithm Development and Validation Study,” JMIR Med. Inform., Vol. 10, No. 2, p. e34492, Feb. 2022, DOI: 10.2196/34492.

S. Henning, W. Beluch, A. Fraser, and A. Friedrich, “A Survey of Methods for Addressing Class Imbalance in Deep-Learning based Natural Language Processing,” in Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, Stroudsburg, PA, USA: Association for Computational Linguistics, 2023, pp. 523–540. DOI: 10.18653/v1/2023.eacl-main.38.

Atharva Inamdar, “Book Genre Prediction,” Kaggle. Accessed: Oct. 14, 2026. [Online]. Available: https://www.kaggle.com/datasets/athu1105/book-genre-prediction

M. C. Hinojosa Lee, J. Braet, and J. Springael, “Performance Metrics for Multilabel Emotion Classification: Comparing Micro, Macro, and Weighted F1-Scores,” Applied Sciences, Vol. 14, No. 21, p. 9863, Oct. 2024, DOI: 10.3390/app14219863.

A. Kunaefi, Z. Abidin, and R. Kusumawati, “Klasifikasi Berita Hoaks Bahasa Indonesia menggunakan IndoBERT Fine-Tuning dengan Pendekatan Focal Loss pada Data tidak Seimbang,” JIPI (Jurnal Ilmiah Penelitian dan Pembelajaran Informatika), Vol. 10, No. 2, pp. 1706–1714, May 2025, DOI: 10.29100/jipi.v10i2.7811.

J. Adhikari, “Leveraging Book Genre Classification using Machine Learning,” DESIDOC Journal of Library & Information Technology, Vol. 45, No. 3, pp. 169–176, May 2025, DOI: 10.14429/djlit.20026.

K. Lee et al., “Deduplicating Training Data Makes Language Models Better,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Stroudsburg, PA, USA: Association for Computational Linguistics, 2022, pp. 8424–8445. DOI: 10.18653/v1/2022.acl-long.577.

V. Solovyev, M. Solnyshkina, A. Ten, and N. Prokopyev, “A BERT-based Classification Model: The Case of Russian Fairy Tales,” Journal of Language and Education, Vol. 10, No. 4, pp. 98–111, Dec. 2024, DOI: 10.17323/jle.2024.24030.

D. Arroyan, M. R. Handayani, K. Umam, and M. A. Ulinuha, “Evaluasi Kinerja Random Forest, SVM, dan Transformer untuk Klasifikasi Komentar Judi Online di Youtube,” JUSTIN (Jurnal Sistem dan Teknologi Informasi), Vol. 14, No. 1, pp. 60–69, 2026, DOI: 10.26418/justin.v14i1.94059.

G. F. S. Medantoro and M. Muljono, “Comparative Analysis of IndoBERT and Classic Machine Learning Models for Sentiment Classification of Education Policy on Social Media X,” Journal of Applied Informatics and Computing, Vol. 10, No. 1, pp. 548–557, Feb. 2026, DOI: 10.30871/jaic.v10i1.11723.

J. Wang, J. Wei, F. Tian, and Y. Wei, “A Comparative Study of Machine Learning Models for Sentiment Analysis of Transboundary Rivers News Media Articles,” Soft comput., Vol. 28, No. 23–24, pp. 13331–13347, Dec. 2024, DOI: 10.1007/s00500-024-10357-2.

M. Siino, I. Tinnirello, and M. La Cascia, “Is Text Preprocessing Still Worth the Time? A Comparative Survey on the Influence of Popular Preprocessing Methods on Transformers and Traditional Classifiers,” Inf. Syst., Vol. 121, p. 102342, Mar. 2024, DOI: 10.1016/j.is.2023.102342.

S. U. Hassan, J. Ahamed, and K. Ahmad, “Analytics of Machine Learning-based Algorithms for Text Classification,” Sustainable Operations and Computers, Vol. 3, pp. 238–248, Jan. 2022, DOI: 10.1016/j.susoc.2022.03.001.

K. Ghosh, C. Bellinger, R. Corizzo, P. Branco, B. Krawczyk, and N. Japkowicz, “The Class Imbalance Problem in Deep Learning,” Mach. Learn., Vol. 113, No. 7, pp. 4845–4901, Jul. 2024, DOI: 10.1007/s10994-022-06268-8.

S. F. Taskiran, B. Turkoglu, E. Kaya, and T. Asuroglu, “A Comprehensive Evaluation of Oversampling Techniques for Enhancing Text Classification Performance,” SCI. Rep., Vol. 15, No. 1, p. 21631, Jul. 2025, DOI: 10.1038/s41598-025-05791-7.

R. Widayanti and F. C. Kasih, “Comparative Analysis of Baseline IndoBERT, Class-Weighted IndoBERT, and SMOTE with Support Vector Machine for Handling Imbalanced Sentiment Classification in Indonesian,” Jurnal Teknik Informatika (Jutif), Vol. 7, No. 3, pp. 2857–2875, Jun. 2026, DOI: 10.52436/1.jutif.2026.7.3.5692.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Stroudsburg, PA, USA: Association for Computational Linguistics, 2019, pp. 4171–4186. DOI: 10.18653/v1/N19-1423.




DOI: https://doi.org/10.32520/stmsi.v15i11.7069

Article Metrics

Abstract view : 0 times
PDF - 0 times

Refbacks

  • There are currently no refbacks.


Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.