STUDY ON DEEP LEARNING MODELS FOR THE CLASSIFICATION OF VR SICKNESS LEVELS
Haechan NA
lye7009@koreatech.ac.krKorea University of Technology and Education (Korea, Republic of)
Yoon Sang KIM
Korea University of Technology and Education, Institute for Bioengineering Application Technology, Department of Computer Science and Engineering, BioComputing Lab (Korea, Republic of)
https://orcid.org/0000-0002-0416-7938
Abstract
Virtual Reality (VR) sickness is often accompanied by symptoms such as nausea and dizziness, and a prominent theory explaining this phenomenon is the sensory conflict theory. Recently, studies have used Deep Learning to classify VR sickness levels; however, there is a paucity of research on Deep Learning models that utilize both visual information and motion data based on sensory conflict theory. In this paper, the authors propose a parallel merging of a Deep Learning model (4bay) to classify the level of VR sickness by utilizing the user's motion data (HMD, controller data) and visual data (rendered image, depth image) based on sensory conflict theory. The proposed model consists of a visual processing module, a motion processing module, and an FC-based VR sickness level classification module. The performance of the proposed model was compared with that of the developed models at the time of design. As a result of the comparison, it was confirmed that the proposed model performed better than the single model and the merged (2bay) model in classifying the user's VR sickness level.
Supporting Agencies
Keywords:
VR sickness, Cyber sickness, Deep Learning, LSTM, ResNet, Sensory Conflict TheoryReferences
Du, M., Cui, H., Wang, Y., & Duh, H. B. L. (2021). Learning from deep stereoscopic attention for simulator sickness prediction. IEEE Transactions on Visualization and Computer Graphics, 29(2), 1415-1423. https://doi.org/10.1109/TVCG.2021.3115901
DOI: https://doi.org/10.1109/TVCG.2021.3115901
Google Scholar
Falkowicz, K., & Kulisz, M. (2024). Prediction of buckling behaviour of composite plate element using artificial neural networks. Advances in Science and Technology. Research Journal, 18(1). https://doi.org/10.12913/22998624/177399
DOI: https://doi.org/10.12913/22998624/177399
Google Scholar
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual Learning for image recognition. IEEE conference on computer vision and pattern recognition (CVPR) (pp. 770-778). IEEE. https://doi.org/10.1109/CVPR.2016.90
DOI: https://doi.org/10.1109/CVPR.2016.90
Google Scholar
Jeong, D., Paik, S., Noh, Y., & Han, K. (2023). MAC: multimodal, attention-based cybersickness prediction modeling in virtual reality. Virtual Reality, 27(3), 2315-2330. https://doi.org/10.1007/s10055-023-00804-0
DOI: https://doi.org/10.1007/s10055-023-00804-0
Google Scholar
Karpiński, R., Krakowski, P., Jonak, J., Machrowska, A., & Maciejewski, M. (2023). Comparison of selected classification methods based on Machine Learning as a diagnostic tool for knee joint cartilage damage based on generated vibroacoustic processes. Applied Computer Science, 19(4), 136-150. https://doi.org/10.35784/acs-2023-40
DOI: https://doi.org/10.35784/acs-2023-40
Google Scholar
Keshavarz, B., Peck, K., Rezaei, S., & Taati, B. (2022). Detecting and predicting visually induced motion sickness with physiological measures in combination with Machine Learning techniques. International Journal of Psychophysiology, 176, 14-26. https://doi.org/10.1016/j.ijpsycho.2022.03.006
DOI: https://doi.org/10.1016/j.ijpsycho.2022.03.006
Google Scholar
Kulisz, M., Kujawska, J., Cioch, M., Cel, W., & Pizoń, J. (2024). Comparative analysis of Machine Learning methods for predicting energy recovery from waste. Applied Sciences, 14(7), 2997. https://doi.org/10.3390/app14072997
DOI: https://doi.org/10.3390/app14072997
Google Scholar
Kundu, R. K., Islam, R., Quarles, J., & Hoque, K. A. (2023). LiteVR: Interpretable and lightweight cybersickness detection using explainable AI. 2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR) (pp. 609-619). IEEE. https://doi.org/10.1109/VR55154.2023.00076
DOI: https://doi.org/10.1109/VR55154.2023.00076
Google Scholar
LaViola, Jr, J. J. (2000). A discussion of cybersickness in virtual environments. ACM Sigchi Bulletin, 32(1), 47-56. https://doi.org/10.1145/333329.333344
DOI: https://doi.org/10.1145/333329.333344
Google Scholar
Lim, H. K., Ji, K., Woo, Y. S., Han, D. U., Lee, D. H., Nam, S. G., & Jang, K. M. (2021). Test-retest reliability of the virtual reality sickness evaluation using electroencephalography (EEG). Neuroscience Letters, 743, 135589. https://doi.org/10.1016/j.neulet.2020.135589
DOI: https://doi.org/10.1016/j.neulet.2020.135589
Google Scholar
Monteiro, D., Liang, H. N., Tang, X., & Irani, P. (2021). Using trajectory compression rate to predict changes in cybersickness in virtual reality games. 2021 IEEE international symposium on mixed and augmented reality (ISMAR), (pp. 138-146). IEEE. https://doi.org/10.1109/ISMAR52148.2021.00028
DOI: https://doi.org/10.1109/ISMAR52148.2021.00028
Google Scholar
Ng, A. K. T., Chan, L. K. Y., & Lau, H. Y. K. (2020). A study of cybersickness and sensory conflict theory using a motion-coupled virtual reality system. Displays, 61, 101922. https://doi.org/10.1016/j.displa.2019.08.004
DOI: https://doi.org/10.1016/j.displa.2019.08.004
Google Scholar
Shimada, S., Ikei, Y., Nishiuchi, N., & Yem, V. (2023a). Study of cybersickness prediction in real time using eye tracking data. 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) (pp. 871-872). IEEE. https://doi.org/10.1109/VRW58643.2023.00278
DOI: https://doi.org/10.1109/VRW58643.2023.00278
Google Scholar
Shimada, S., Pannattee, P., Ikei, Y., Nishiuchi, N., & Yem, V. (2023b). High-frequency cybersickness prediction using Deep Learning techniques with eye-related indices. IEEE Access, 11, 95825-95839. https://doi.org/10.1109/ACCESS.2023.3312216
DOI: https://doi.org/10.1109/ACCESS.2023.3312216
Google Scholar
Shodipe, O. E., & Allison, R. S. (2023). Modelling the relationship between the objective measures of car sickness. 2023 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE) (pp. 570-575). IEEE. https://doi.org/10.1109/CCECE58730.2023.10289000
DOI: https://doi.org/10.1109/CCECE58730.2023.10289000
Google Scholar
Wang, J., Liang, H. N., Monteiro, D. V., Xu, W., Chen, H., & Chen, Q. (2020). Real-time detection of simulator sickness in virtual reality games based on players' psychophysiological data during gameplay. 2020 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) (pp. 247-248). IEEE. https://doi.org/10.1109/ISMAR-Adjunct51615.2020.00071
DOI: https://doi.org/10.1109/ISMAR-Adjunct51615.2020.00071
Google Scholar
Wen, E., Gupta, C., Sasikumar, P., Billinghurst, M., Wilmott, J., Skow, E., Dey. A., & Nanayakkara, S. (2024). VR. net: A real-world dataset for virtual reality motion sickness research. IEEE Transactions on Visualization and Computer Graphics, 30(5), 2330-2336. https://doi.org/10.1109/TVCG.2024.3372044
DOI: https://doi.org/10.1109/TVCG.2024.3372044
Google Scholar
Yang, A. H. X., Kasabov, N., & Cakmak, Y. O. (2022). Machine Learning methods for the study of cybersickness: A systematic review. Brain Informatics, 9(1), 24. https://doi.org/10.1186/s40708-022-00172-6
DOI: https://doi.org/10.1186/s40708-022-00172-6
Google Scholar
Younis, M. C. (2024). Prediction of patient’s willingness for treatment of mental illness using Machine Learning approaches. Applied Computer Science, 20(2), 175-193. https://doi.org/10.35784/acs-2024-23
DOI: https://doi.org/10.35784/acs-2024-23
Google Scholar
Zhao, J., Tran, K. T., Chalmers, A., Hoh, W. K., Yao, R., Dey, A., Wilmott, J., Lin, J., Billinghurst, M., Lindeman, & Rhee, T. (2023). Deep Learning-based simulator sickness estimation from 3D motion. 2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) (pp. 39-48). IEEE. https://doi.org/10.1109/ISMAR59233.2023.00018
DOI: https://doi.org/10.1109/ISMAR59233.2023.00018
Google Scholar
Authors
Haechan NAlye7009@koreatech.ac.kr
Korea University of Technology and Education Korea, Republic of
Authors
Yoon Sang KIMKorea University of Technology and Education, Institute for Bioengineering Application Technology, Department of Computer Science and Engineering, BioComputing Lab Korea, Republic of
https://orcid.org/0000-0002-0416-7938
Statistics
Abstract views: 89PDF downloads: 35
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
All articles published in Applied Computer Science are open-access and distributed under the terms of the Creative Commons Attribution 4.0 International License.
Most read articles by the same author(s)
- Hae Chan Na, Yoon Sang Kim, A STUDY ON AN AR-BASED CIRCUIT PRACTICE , Applied Computer Science: Vol. 20 No. 1 (2024)
Similar Articles
- Islam MOHAMED, Mohamed EL-WAKAD, Khaled ABBAS, Mohamed ABOAMER, Nader A. Rahman MOHAMED, PUPIL DIAMETER AND MACHINE LEARNING FOR DEPRESSION DETECTION: A COMPARATIVE STUDY WITH DEEP LEARNING MODELS , Applied Computer Science: Vol. 20 No. 4 (2024)
- Boutkhil SIDAOUI, PREDICTING STATES OF EPILEPSY PATIENTS USING DEEP LEARNING MODELS , Applied Computer Science: Vol. 20 No. 2 (2024)
- Thanh-Lam BUI, Ngoc-Tien TRAN, NAVIGATION STRATEGY FOR MOBILE ROBOT BASED ON COMPUTER VISION AND YOLOV5 NETWORK IN THE UNKNOWN ENVIRONMENT , Applied Computer Science: Vol. 19 No. 2 (2023)
- Behnaz ESLAMI, Mehdi HABIBZADEH MOTLAGH, Zahra REZAEI, Mohammad ESLAMI, Mohammad AMIN AMINI, UNSUPERVISED DYNAMIC TOPIC MODEL FOR EXTRACTING ADVERSE DRUG REACTION FROM HEALTH FORUMS , Applied Computer Science: Vol. 16 No. 1 (2020)
- Kevin Joy DSOUZA, Zahid Ahmed ANSARI, HISTOPATHOLOGY IMAGE CLASSIFICATION USING HYBRID PARALLEL STRUCTURED DEEP-CNN MODELS , Applied Computer Science: Vol. 18 No. 1 (2022)
- Manikandan SRIDHARAN, Delphin Carolina RANI ARULANANDAM, Rajeswari K CHINNASAMY, Suma THIMMANNA, Sivabalaselvamani DHANDAPANI, RECOGNITION OF FONT AND TAMIL LETTER IN IMAGES USING DEEP LEARNING , Applied Computer Science: Vol. 17 No. 2 (2021)
- Hawkar ASAAD, Shavan ASKAR, Ahmed KAKAMIN, Nayla FAIQ, EXPLORING THE IMPACT OF ARTIFICIAL INTELLIGENCE ON HUMANROBOT COOPERATION IN THE CONTEXT OF INDUSTRY 4.0 , Applied Computer Science: Vol. 20 No. 2 (2024)
- Saheed ADEWUYI, Segun AINA, Aderonke LAWAL, Adeniran OLUWARANTI, Moses UZUNUIGBE, AN OVERVIEW OF DEEP LEARNING TECHNIQUES FOR SHORT-TERM ELECTRICITY LOAD FORECASTING , Applied Computer Science: Vol. 15 No. 4 (2019)
- Saheed A. ADEWUYI, Segun AINA, Adeniran I. OLUWARANTI, A DEEP LEARNING MODEL FOR ELECTRICITY DEMAND FORECASTING BASED ON A TROPICAL DATA , Applied Computer Science: Vol. 16 No. 1 (2020)
- Olutayo BOYINBODE, Paul OLOTU, Kolawole AKINTOLA, DEVELOPMENT OF AN ONTOLOGY-BASED ADAPTIVE PERSONALIZED E-LEARNING SYSTEM , Applied Computer Science: Vol. 16 No. 4 (2020)
You may also start an advanced similarity search for this article.
