HAND MOVEMENT DISORDERS TRACKING BY SMARTPHONE BASED ON COMPUTER VISION METHODS
Marko Andrushchenko
marko.andrushchenko@nure.uaKharkiv National University of Radio Electronics (Ukraine)
https://orcid.org/0000-0003-1722-2390
Karina Selivanova
Kharkiv National University of Radio Electronics (Ukraine)
Oleg Avrunin
Kharkiv National University of Radio Electronics (Ukraine)
https://orcid.org/0000-0002-6312-687X
Dmytro Palii
National Pirogov Memorial Medical University (Ukraine)
https://orcid.org/0000-0001-6537-6912
Sergii Tymchyk
Vinnytsia National Technical University (Ukraine)
https://orcid.org/0000-0003-2977-1602
Dana Turlykozhayeva
Al-Farabi Kazakh National University, Scientific Research Institute of Experimental and Theoretical Physics (Kazakhstan)
https://orcid.org/0000-0002-7326-9196
Abstract
This article describes the development of a cost-effective, efficient, and accessible solution for diagnosing hand movement disorders using smartphone-based computer vision technologies. It highlights the idea of using ToF camera data combined with RG data and machine learning algorithms to accurately recognize limbs and movements, which overcomes the limitations of traditional motion recognition methods, improving rehabilitation and reducing the high cost of professional medical equipment. Using the ubiquity of smartphones and advanced computational methods, the study offers a new approach to improving the quality and accessibility of diagnosis of movement disorders, offering a promising direction for future research and application in clinical practice.
Keywords:
healthcare, information medical technologies, image analysis, computer vision, artificial intelligence, motion disordersReferences
Apple Machine Learning Research (n.d.). Deploying Transformers on the Apple Neural Engine [https://machinelearning.apple.com/research/neural-engine-transformers].
Google Scholar
Apple Inc. (n.d.). Streaming Depth Data from the TrueDepth Camera. Apple Developer Documentation [https://developer.apple.com/documentation/avfoundation/additional_data_capture/streaming_depth_data_from_the_truedepth_camera].
Google Scholar
Apple Developer Documentation. Streaming Depth Data from the TrueDepth Camera | Apple Developer Documentation [https://developer.apple.com/documentation/avfoundation/additional_data_capture/streaming_depth_data_from_the_truedepth_camera,2023] (accessed 3 Dec. 2023).
Google Scholar
Assimp.org. The Asset-Importer Library Home [https://www.assimp.org] (accessed 3 Dec. 2023).
Google Scholar
Avrunin O. G. et al.: Research Active Posterior Rhinomanometry Tomography Method for Nasal Breathing Determining Violations. Sensors 21, 2021, 8508.
Google Scholar
Avrunin O. et al.: Improving the methods for visualization of middle ear pathologies based on telemedicine services in remote treatment. IEEE KhPI Week on Advanced Technology, KhPI Week, 2020, 347–350.
Google Scholar
GitHub [https://github.com/googlesamples/mediapipe/tree/main/examples/hand_landmarker/ios] (accessed 19 Feb. 2024).
Google Scholar
Google for Developers. Hand landmarks detection guide [https://developers.google.com/mediapipe/solutions/vision/hand _landmarker#model] (accessed 19 Feb. 2024).
Google Scholar
Gupta S., White M.: Improved On-Device ML on Pixel 6, with Neural Architecture Search. Google Research Blog [https://blog.research.google/2021/11/improved-on-device-ml-on-pixel-6-with.html] (accessed 8 Nov. 2021).
Google Scholar
Kim B., Neville Ch.: Accuracy and Feasibility of a Novel Fine Hand Motor Skill Assessment Using Computer Vision Object Tracking. Scientific Reports 13(1), 2023, 1–14 [https://doi.org/10.1038/s41598-023-29091-0].
Google Scholar
Lin T.-Y. et al.: Feature Pyramid Networks for Object Detection [https://arxiv.org/pdf/1612.03144.pdf].
Google Scholar
Liu W. et al.: SSD: Single Shot MultiBox Detector [https://arxiv.org/pdf/1512.02325.pdf].
Google Scholar
Liang M. et al.: Deep Continuous Fusion for Multi-Sensor 3D Object Detection. 2020 [https://arxiv.org/abs/2012.10992] (accessed 19 Feb. 2024).
Google Scholar
Liang M. et al.: Multi-Task Multi-Sensor Fusion for 3D Object Detection [https://openaccess.thecvf.com/content_CVPR_2019/papers/Liang_MultiTask_MultiSensor_Fusion_for_3D_Object_Detection_CVPR_2019_paper.pdf].
Google Scholar
Muhammad B. S., Chai D.: RGB-D Data-Based Action Recognition: A Review. Sensors 21(12), 2021, 4246–4246 [https://doi.org/10.3390/s21124246].
Google Scholar
On-Device, Real-Time Hand Tracking with MediaPipe [https://blog.research.google/2019/08/on-device-real-time-hand-tracking-with.html] (accessed 19 Feb. 2024).
Google Scholar
Romanyuk O. et al.: A function-based approach to real-time visualization using graphics processing units. Proc. SPIE 11581, 2020, 115810E [https://doi.org/10.1117/12.2580212].
Google Scholar
Selivanova K. Avrunin O.: Method of Hand Movement Disorders Determination based on the Surgeon's Laparoscopic Video Recording. 3rd KhPI Week on Advanced Technology – KhPIWeek, 2022, 1–4 [https://doi.org/10.1109/KhPIWeek57572.2022.9916457].
Google Scholar
Selivanova K. et al.: The tracking system of a three-dimensional position of hand movement for tremor detection. Proc. SPIE 11581, 2020, 115810I [https://doi.org/10.1117/12.2580330].
Google Scholar
Sokol Y. et al.: Using medical imaging in disaster medicine. IEEE 4th International Conference on Intelligent Energy and Power Systems, IEPS 2020, 2020, 287–290.
Google Scholar
Taeger J. et al.: Utilization of Smartphone Depth Mapping Cameras for App-Based Grading of Facial Movement Disorders: Development and Feasibility Study. JMIR mHealth and uHealth 9(1), 2021, e19346 [https://doi.org/10.2196/19346].
Google Scholar
Turlykozhayeva D. et al.: Routing Algorithm for Software Defined Network Based on Boxcovering Algorithm. 10th International Conference on Wireless Networks and Mobile Communications – WINCOM, 2023.
Google Scholar
Tymkovych M. et al.: 3D scanning technologies by optical RealSense cameras for SIREN-based 3D hand representation. Proc. SPIE 12985, 2023, 129850O [https://doi.org/10.1117/12.3022737].
Google Scholar
Urban S. et al.: On the Issues of TrueDepth Sensor Data for Computer Vision Tasks Across Different IPad Generations. 2022 [https://arxiv.org/abs/2201.10865] (accessed 26 Nov. 2023).
Google Scholar
WójcikW. et al.: Information Technology in Medical Diagnostics II. Taylor & Francis Group. CRC Press, Balkema Book. London, 2019.
Google Scholar
Wójcik W. et al.: Information Technology in Medical Diagnostics. CRC Press, 2017.
Google Scholar
Zhang F. et al.: MediaPipe Hands: On-device Real-time Hand Tracking. 2006 [https://arxiv.org/abs/2006.10214].
Google Scholar
Authors
Marko Andrushchenkomarko.andrushchenko@nure.ua
Kharkiv National University of Radio Electronics Ukraine
https://orcid.org/0000-0003-1722-2390
Authors
Karina SelivanovaKharkiv National University of Radio Electronics Ukraine
Authors
Oleg AvruninKharkiv National University of Radio Electronics Ukraine
https://orcid.org/0000-0002-6312-687X
Authors
Dmytro PaliiNational Pirogov Memorial Medical University Ukraine
https://orcid.org/0000-0001-6537-6912
Authors
Sergii TymchykVinnytsia National Technical University Ukraine
https://orcid.org/0000-0003-2977-1602
Authors
Dana TurlykozhayevaAl-Farabi Kazakh National University, Scientific Research Institute of Experimental and Theoretical Physics Kazakhstan
https://orcid.org/0000-0002-7326-9196
Statistics
Abstract views: 570PDF downloads: 214
Most read articles by the same author(s)
- Oleg Avrunin, Yana Nosova, Ibrahim Younouss Abdelhamid, Oleksandr Gryshkov, Birgit Glasmacher, USING 3D PRINTING TECHNOLOGY TO FULL-SCALE SIMULATION OF THE UPPER RESPIRATORY TRACT , Informatyka, Automatyka, Pomiary w Gospodarce i Ochronie Środowiska: Vol. 9 No. 4 (2019)
- Oleg Avrunin, Yana Nosova, Sergii Zlepko, Ibrahim Younouss Abdelhamid , Nataliia Shushliapina, ASSESSMENT OF THE DIAGNOSTIC VALUE OF THE METHOD OF COMPUTER OLFACTOMETRY , Informatyka, Automatyka, Pomiary w Gospodarce i Ochronie Środowiska: Vol. 9 No. 3 (2019)
- Maksym Tymkovych, Oleg Avrunin, Karina Selivanova, Alona Kolomiiets, Taras Bednarchyk, Saule Smailova, CORRESPONDENCE MATCHING IN 3D MODELS FOR 3D HAND FITTING , Informatyka, Automatyka, Pomiary w Gospodarce i Ochronie Środowiska: Vol. 14 No. 1 (2024)
- Valerіi Kryvonosov, Oleg Avrunin, Serhii Sander, Volodymyr Pavlov, Liliia Martyniuk, Bagashar Zhumazhanov, A USAGE OF THE IMPEDANCE METHOD FOR DETECTING CIRCULATORY DISORDERS TO DETERMINE THE DEGREE OF LIMB ISCHEMIA , Informatyka, Automatyka, Pomiary w Gospodarce i Ochronie Środowiska: Vol. 13 No. 4 (2023)
- Oleg Avrunin, Yana Nosova, Nataliia Shushliapina, Ibrahim Younouss Abdelhamid, Oleksandr Avrunin, Svetlana Kyrylashchuk, Olha Moskovchuk, Orken Mamyrbayev, ANALYSIS OF UPPER RESPIRATORY TRACT SEGMENTATION FEATURES TO DETERMINE NASAL CONDUCTANCE , Informatyka, Automatyka, Pomiary w Gospodarce i Ochronie Środowiska: Vol. 12 No. 4 (2022)