HAND MOVEMENT DISORDERS TRACKING BY SMARTPHONE BASED ON COMPUTER VISION METHODS

Marko Andrushchenko

marko.andrushchenko@nure.ua
Kharkiv National University of Radio Electronics (Ukraine)
https://orcid.org/0000-0003-1722-2390

Karina Selivanova


Kharkiv National University of Radio Electronics (Ukraine)

Oleg Avrunin


Kharkiv National University of Radio Electronics (Ukraine)
https://orcid.org/0000-0002-6312-687X

Dmytro Palii


National Pirogov Memorial Medical University (Ukraine)
https://orcid.org/0000-0001-6537-6912

Sergii Tymchyk


Vinnytsia National Technical University (Ukraine)
https://orcid.org/0000-0003-2977-1602

Dana Turlykozhayeva


Al-Farabi Kazakh National University, Scientific Research Institute of Experimental and Theoretical Physics (Kazakhstan)
https://orcid.org/0000-0002-7326-9196

Abstract

This article describes the development of a cost-effective, efficient, and accessible solution for diagnosing hand movement disorders using smartphone-based computer vision technologies. It highlights the idea of using ToF camera data combined with RG data and machine learning algorithms to accurately recognize limbs and movements, which overcomes the limitations of traditional motion recognition methods, improving rehabilitation and reducing the high cost of professional medical equipment. Using the ubiquity of smartphones and advanced computational methods, the study offers a new approach to improving the quality and accessibility of diagnosis of movement disorders, offering a promising direction for future research and application in clinical practice.


Keywords:

healthcare, information medical technologies, image analysis, computer vision, artificial intelligence, motion disorders

Apple Machine Learning Research (n.d.). Deploying Transformers on the Apple Neural Engine [https://machinelearning.apple.com/research/neural-engine-transformers].
  Google Scholar

Apple Inc. (n.d.). Streaming Depth Data from the TrueDepth Camera. Apple Developer Documentation [https://developer.apple.com/documentation/avfoundation/additional_data_capture/streaming_depth_data_from_the_truedepth_camera].
  Google Scholar

Apple Developer Documentation. Streaming Depth Data from the TrueDepth Camera | Apple Developer Documentation [https://developer.apple.com/documentation/avfoundation/additional_data_capture/streaming_depth_data_from_the_truedepth_camera,2023] (accessed 3 Dec. 2023).
  Google Scholar

Assimp.org. The Asset-Importer Library Home [https://www.assimp.org] (accessed 3 Dec. 2023).
  Google Scholar

Avrunin O. G. et al.: Research Active Posterior Rhinomanometry Tomography Method for Nasal Breathing Determining Violations. Sensors 21, 2021, 8508.
  Google Scholar

Avrunin O. et al.: Improving the methods for visualization of middle ear pathologies based on telemedicine services in remote treatment. IEEE KhPI Week on Advanced Technology, KhPI Week, 2020, 347–350.
  Google Scholar

GitHub [https://github.com/googlesamples/mediapipe/tree/main/examples/hand_landmarker/ios] (accessed 19 Feb. 2024).
  Google Scholar

Google for Developers. Hand landmarks detection guide [https://developers.google.com/mediapipe/solutions/vision/hand _landmarker#model] (accessed 19 Feb. 2024).
  Google Scholar

Gupta S., White M.: Improved On-Device ML on Pixel 6, with Neural Architecture Search. Google Research Blog [https://blog.research.google/2021/11/improved-on-device-ml-on-pixel-6-with.html] (accessed 8 Nov. 2021).
  Google Scholar

Kim B., Neville Ch.: Accuracy and Feasibility of a Novel Fine Hand Motor Skill Assessment Using Computer Vision Object Tracking. Scientific Reports 13(1), 2023, 1–14 [https://doi.org/10.1038/s41598-023-29091-0].
  Google Scholar

Lin T.-Y. et al.: Feature Pyramid Networks for Object Detection [https://arxiv.org/pdf/1612.03144.pdf].
  Google Scholar

Liu W. et al.: SSD: Single Shot MultiBox Detector [https://arxiv.org/pdf/1512.02325.pdf].
  Google Scholar

Liang M. et al.: Deep Continuous Fusion for Multi-Sensor 3D Object Detection. 2020 [https://arxiv.org/abs/2012.10992] (accessed 19 Feb. 2024).
  Google Scholar

Liang M. et al.: Multi-Task Multi-Sensor Fusion for 3D Object Detection [https://openaccess.thecvf.com/content_CVPR_2019/papers/Liang_MultiTask_MultiSensor_Fusion_for_3D_Object_Detection_CVPR_2019_paper.pdf].
  Google Scholar

Muhammad B. S., Chai D.: RGB-D Data-Based Action Recognition: A Review. Sensors 21(12), 2021, 4246–4246 [https://doi.org/10.3390/s21124246].
  Google Scholar

On-Device, Real-Time Hand Tracking with MediaPipe [https://blog.research.google/2019/08/on-device-real-time-hand-tracking-with.html] (accessed 19 Feb. 2024).
  Google Scholar

Romanyuk O. et al.: A function-based approach to real-time visualization using graphics processing units. Proc. SPIE 11581, 2020, 115810E [https://doi.org/10.1117/12.2580212].
  Google Scholar

Selivanova K. Avrunin O.: Method of Hand Movement Disorders Determination based on the Surgeon's Laparoscopic Video Recording. 3rd KhPI Week on Advanced Technology – KhPIWeek, 2022, 1–4 [https://doi.org/10.1109/KhPIWeek57572.2022.9916457].
  Google Scholar

Selivanova K. et al.: The tracking system of a three-dimensional position of hand movement for tremor detection. Proc. SPIE 11581, 2020, 115810I [https://doi.org/10.1117/12.2580330].
  Google Scholar

Sokol Y. et al.: Using medical imaging in disaster medicine. IEEE 4th International Conference on Intelligent Energy and Power Systems, IEPS 2020, 2020, 287–290.
  Google Scholar

Taeger J. et al.: Utilization of Smartphone Depth Mapping Cameras for App-Based Grading of Facial Movement Disorders: Development and Feasibility Study. JMIR mHealth and uHealth 9(1), 2021, e19346 [https://doi.org/10.2196/19346].
  Google Scholar

Turlykozhayeva D. et al.: Routing Algorithm for Software Defined Network Based on Boxcovering Algorithm. 10th International Conference on Wireless Networks and Mobile Communications – WINCOM, 2023.
  Google Scholar

Tymkovych M. et al.: 3D scanning technologies by optical RealSense cameras for SIREN-based 3D hand representation. Proc. SPIE 12985, 2023, 129850O [https://doi.org/10.1117/12.3022737].
  Google Scholar

Urban S. et al.: On the Issues of TrueDepth Sensor Data for Computer Vision Tasks Across Different IPad Generations. 2022 [https://arxiv.org/abs/2201.10865] (accessed 26 Nov. 2023).
  Google Scholar

WójcikW. et al.: Information Technology in Medical Diagnostics II. Taylor & Francis Group. CRC Press, Balkema Book. London, 2019.
  Google Scholar

Wójcik W. et al.: Information Technology in Medical Diagnostics. CRC Press, 2017.
  Google Scholar

Zhang F. et al.: MediaPipe Hands: On-device Real-time Hand Tracking. 2006 [https://arxiv.org/abs/2006.10214].
  Google Scholar

Download


Published
2024-06-30

Cited by

Andrushchenko, M., Selivanova, K., Avrunin, O., Palii, D., Tymchyk , S., & Turlykozhayeva, D. (2024). HAND MOVEMENT DISORDERS TRACKING BY SMARTPHONE BASED ON COMPUTER VISION METHODS. Informatyka, Automatyka, Pomiary W Gospodarce I Ochronie Środowiska, 14(2), 5–10. https://doi.org/10.35784/iapgos.6126

Authors

Marko Andrushchenko 
marko.andrushchenko@nure.ua
Kharkiv National University of Radio Electronics Ukraine
https://orcid.org/0000-0003-1722-2390

Authors

Karina Selivanova 

Kharkiv National University of Radio Electronics Ukraine

Authors

Oleg Avrunin 

Kharkiv National University of Radio Electronics Ukraine
https://orcid.org/0000-0002-6312-687X

Authors

Dmytro Palii 

National Pirogov Memorial Medical University Ukraine
https://orcid.org/0000-0001-6537-6912

Authors

Sergii Tymchyk  

Vinnytsia National Technical University Ukraine
https://orcid.org/0000-0003-2977-1602

Authors

Dana Turlykozhayeva 

Al-Farabi Kazakh National University, Scientific Research Institute of Experimental and Theoretical Physics Kazakhstan
https://orcid.org/0000-0002-7326-9196

Statistics

Abstract views: 27
PDF downloads: 22


License

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.


Most read articles by the same author(s)