eHyPRETo: Enhanced Hybrid Pre-Trained and Transfer Learning-based Contextual Relation Classification Model
DOI:
https://doi.org/10.56294/sctconf2024758Keywords:
Pre-trained Model, RC (Relation Classification), Contextual, ELECTRA, RoBERTa, Domain Adaptable, eHyPREToAbstract
Introduction: relation classification (RC) plays a crucial role in enhancing the understanding of intricate relationships, as it helps with many NLP (Natural Language Processing) applications. To identify contextual subtleties in different domains, one might make use of pre-trained models.
Methods: to achieve successful relation classification, a recommended model called eHyPRETo, which is a hybrid pre-trained model, has to be used. The system comprises several components, including ELECTRA, RoBERTa, and Bi-LSTM. The integration of pre-trained models enabled the utilisation of Transfer Learning (TL) to acquire contextual information and complex patterns. Therefore, the amalgamation of pre-trained models has great importance. The major purpose of this related classification is to effectively handle irregular input and improve the overall efficiency of pre-trained models. The analysis of eHyPRETo involves the use of a carefully annotated biological dataset focused on Indian Mosquito Vector Biocontrol Agents.
Results: the eHyPRETo model has remarkable stability and effectiveness in categorising, as evidenced by its continuously high accuracy of 98,73 % achieved during training and evaluation throughout several epochs. The eHyPRETo model's domain applicability was assessed. The obtained p-value of 0,06 indicates that the model is successful and adaptable across many domains.
Conclusion: the suggested hybrid technique has great promise for practical applications such as medical diagnosis, financial fraud detection, climate change analysis, targeted marketing campaigns, and self-driving automobile navigation, among others. The eHyPRETo model has been developed in response to the challenges in RC, representing a significant advancement in the fields of linguistics and artificial intelligence
References
1. Agarwal B, Poria S, Mittal N, Gelbukh A, and Hussain A. Concept-level sentiment analysis with dependency-based semantic parsing: a novel approach. Cognitive Computation, 7, pp. 487-499. https://doi.org/10.1007/s12559-014-9316-6v.
2. Ahmad H, Asghar MU, Asghar MZ, Khan A, and Mosavi, AH. A hybrid deep learning technique for personality trait classification from text. IEEE Access, 9, pp. 146214-146232. https://doi.org/10.1109/ACCESS.2021.3121791.
3. Aydoğan M, and Karci A. Improving the accuracy using pre-trained word embeddings on deep neural networks for Turkish text classification. Physica A: Statistical Mechanics and its Applications, 541, p.123288. https://doi.org/10.1016/j.physa.2019.123288.
4. Clark K, Luong MT, Le QV, and Manning, CD. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555, pp. 1-18. https://doi.org/10.48550/arXiv.2003.10555.
5. Devlin J, Chang MW, Lee K, and Toutanova K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, pp. 1-16. https://doi.org/10.48550/arXiv.1810.04805.
6. Du M, He F, Zou N, Tao D, and Hu X. Shortcut learning of large language models in natural language understanding. Communications of the ACM, 67(1), pp. 110-120. https://doi.org/10.1145/3596490.
7. Formisano E, De Martino F, and Valente G. Multivariate analysis of fMRI time series: classification and regression of brain responses using machine learning. Magnetic resonance imaging, 26(7), pp. 921-934. https://doi.org/10.1016/j.mri.2008.01.052.
8. Geng Z, Chen G, Han Y, Lu G, and Li F. Semantic relation extraction using sequential and tree-structured LSTM with attention. Information Sciences, 509, pp. 183-192. https://doi.org/10.1016/j.ins.2019.09.006.
9. Gou Z, and Li Y. Integrating BERT embeddings and BiLSTM for emotion analysis of dialogue. Computational Intelligence and Neuroscience, pp. 1-8. https://doi.org/10.1155/2023/6618452.
10. Harnoune A, Rhanoui M, Mikram M, Yousfi S, Elkaimbillah Z, and El Asri B. BERT based clinical knowledge extraction for biomedical knowledge graph construction and analysis. Computer Methods and Programs in Biomedicine Update, 1, pp. 100042. https://doi.org/10.1016/j.cmpbup.2021.100042.
11. He X, Chen W, Nie B, and Zhang M. Classification technique for danger classes of coal and gas outburst in deep coal mines. Safety science, 48(2), pp. 173-178. https://doi.org/10.1016/j.ssci.2009.07.007.
12. Herzig J, Nowak PK, Müller T, Piccinno F, and Eisenschlos, JM. TaPas: Weakly supervised table parsing via pre-training. arXiv preprint arXiv:2004.02349, pp. 4320 - 4333. https://doi.org/10.18653/v1/2020.acl-main.398.
13. Hui B, Liu L, Chen J, Zhou X, and Nian Y. Few-shot relation classification by context attention-based prototypical networks with BERT. EURASIP Journal on Wireless Communications and Networking, pp. 1-17. https://doi.org/10.1186/s13638-020-01720-6.
14. Arel I, Rose DC, and Karnowski TP. Deep machine learning-a new frontier in artificial intelligence research [research frontier]. IEEE computational intelligence magazine, 5(4), pp. 13-18. https://doi.org/10.1109/MCI.2010.938364.
15. Joshi M, Chen D, Liu Y, Weld DS, Zettlemoyer L, and Levy O. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the association for computational linguistics, 8, pp. 64-77. https://doi.org/10.1162/tacl_a_00300.
16. Lan Z, Chen M, Goodman S, Gimpel K, Sharma P, and Soricut R. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942, pp. 1-17. https://doi.org/10.48550/arXiv.1909.11942.
17. Liu T, Zhang X, Zhou W, and Jia W. Neural relation extraction via inner-sentence noise reduction and transfer learning. arXiv preprint arXiv:1808.06738, pp. 2195- 2204. https://doi.org/10.18653/v1/d18-1243.
18. Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, and Stoyanov V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, pp. 1-13. https://doi.org/10.48550/arXiv.1907.11692.
19. Mehta SB, Chaudhury S, Bhattacharyya A, and Jena A. Handcrafted fuzzy rules for tissue classification. Magnetic Resonance Imaging, 26(6), pp. 815-823. https://doi.org/10.1016/j.mri.2008.01.021.
20. Qin Q, Zhao S, and Liu C. A BERT-BiGRU-CRF model for entity recognition of Chinese electronic medical records. Complexity, pp. 1-11. https://doi.org/10.1155/2021/6631837.
21. Sanh V, Debut L, Chaumond J, and Wolf T. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, pp. 1-5. https://doi.org/10.48550/arXiv.1910.01108.
22. Sun Q, Xu T, Zhang K, Huang K, Lv L, Li X, Zhang T, and Dore-Natteh D. Dual-channel and hierarchical graph convolutional networks for document-level relation extraction. Expert Systems with Applications, 205, pp. 117678. https://doi.org/10.1016/j.eswa.2022.117678.
23. Sun Q, Zhang K, Huang K, Xu T, Li X, and Liu Y. Document-level relation extraction with two-stage dynamic graph attention networks. Knowledge-Based Systems, 267, pp .110428. https://doi.org/10.1016/j.knosys.2023.110428.
24. Tan KL, Lee CP, Anbananthen KSM, and Lim KM. RoBERTa-LSTM: a hybrid model for sentiment analysis with transformer and recurrent neural network. IEEE Access, 10, pp. 21517-21525. https://doi.org/10.1109/ACCESS.2022.3152828.
25. Xu R, Chen T, Xia Y, Lu Q, Liu B, and Wang X. Word embedding composition for data imbalances in sentiment and emotion classification. Cognitive Computation, 7, pp. 226-240. https://doi.org/10.1007/s12559-015-9319-y.
26. Yang Z, Dai Z, Yang Y, Carbonell J, Salakhutdinov RR, and Le QV. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, pp. 1-11.
27. Yi R, and Hu W. Pre-trained BERT-GRU model for relation extraction. In Proceedings of the 8th international conference on computing and pattern recognition, pp. 453-457. https://doi.org/10.1145/3373509.3373533.
28. Yu Q, Wang Z, and Jiang K. Research on text classification based on bert-bigru model. In Journal of Physics: Conference Series, 1746(1), pp. 012019). IOP Publishing. https://doi.org/10.1088/17426596/1746/1/012019.
29. Yu, S., Su, J. and Luo, D., 2019. Improving bert-based text classification with auxiliary sentence and domain knowledge. IEEE Access, 7, pp.176600-176612. https://doi.org/10.1109/ACCESS.2019.2953990.
30. Zhang Z, Han X, Liu Z, Jiang X, Sun M, and Liu Q. ERNIE: Enhanced language representation with informative entities. arXiv preprint arXiv:1905.07129, pp. 1-11, https://doi.org/10.18653/v1/p19-1139.
31. Zhao Y, Wan H, Gao J, and Lin Y. Improving relation classification by entity pair graph. In Asian Conference on Machine Learning, pp. 1156-1171.
32. Zhou W, Huang K, Ma T, and Huang J. Document-level relation extraction with adaptive thresholding and localized context pooling. In Proceedings of the AAAI conference on artificial intelligence, 35(16), pp. 14612-14620. https://doi.org/10.1609/aaai.v35i16.17717.
Published
Issue
Section
License
Copyright (c) 2024 Jeyakodi G, Sarojini B, Shanthi Bala P (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
The article is distributed under the Creative Commons Attribution 4.0 License. Unless otherwise stated, associated published material is distributed under the same licence.
