OUTPUTS

Phase 2 – Prototype development and testing (January 2023 – December 2023)

This section presents the main outputs for the activities which took place during Phase 2 – Prototype development and testing of the TRAVEE project.

Results:

(1) Report on hand monitoring algorithms and BCI algorithms

In the hand monitoring process, the system provided by the Oculus Quest 2 VR headset is used. This is a commercial system that provides satisfactory results for the rehabilitation session process.

An important advantage provided by the Oculus Quest 2 VR headset system is that it comes built-in with palm and finger tracking, eliminating the need for additional hardware devices. This makes the TRAVEE system easily usable independently, such as in telemedicine applications.

The closed fist gesture, recognized by the system.

The inertial unit developed by the UTI team has 9 degrees of freedom (9DOF) and contains a processing core that consolidates primary information from sensors (accelerometer, gyroscope, and magnetometer). This processing aims to provide the TRAVEE system with data on orientation and linear acceleration, contributing to the integration and calculation of displacement velocity. Additionally, the device has the capability to identify the current usage state, allowing differentiation between situations where it is placed on a static surface (e.g., a table) or held steady on the patient’s limb.

The architecture of the inertial device

The movement of a limb or even the contraction of a single muscle leads to changes in brain activity at the cortical level. In fact, even the preparation for a movement or imagining the movement leads to changes in the so-called sensorimotor rhythms. For discriminating these changes through BCI, the CSP method has proven to be very efficient, as it eliminates disturbances in EEG signals by correlating them with neighboring electrodes. The result provided by the BCI system after classification is further conveyed via HDC both to the FES (Functional Electrical Stimulation) system, which induces the corresponding hand movement, and to the server, which can use the information for other subsystems.

The hardware structure of the BCI-controlled stimulation system

(2) Algorithm Testing Report

Tests were conducted for the following categories of algorithms/hardware:

  • BCI Algorithm Testing: In order to achieve the best control accuracy, six time windows of 2 seconds each were selected from the training data for creating spatial filters and the classifier. The set with the smallest cross-validation error was chosen to generate online feedback. The optimal window for variance calculation was selected as 1.5 seconds.

The model used for data recording and online feedback generation

  • FES Algorithm Testing: The functional electrical stimulation system was tested for real-time prescription of hand extensor stimulation parameters based on signals from the input block. For positive values received as feedback from the BCI system, the algorithm activates electrical stimulation on the channels in the first set of 4 (Ch 1:4), while for negative values, it activates stimulation on the channels in the second set of channels (Ch 5:8)

The parameter mask of the neurostimulator control block

  • IMU Testing: Since the IMU device was developed entirely within the project, the testing process included a sequence of steps for its progressive validation – testing the hardware along with its corresponding firmware, testing communication and Bluetooth interactions, validating integration with HDC, and validating the outputs provided by the IMU device
  • Hand Tracking Algorithm Testing: One aspect of testing the hand tracking by the Oculus Quest 2’s video tracking system was performance. In this regard, factors considered included latency, static positional accuracy, path offset, angular error, and lighting conditions

(3) Description of the system’s skeleton

The TRAVEE system consists of 7 hardware nodes, on which 8 different applications are executed, containing a total of 30 software systems. All of these can be viewed in the system architecture diagram.

The architecture diagram of the TRAVEE system, which illustrates the hardware nodes, the software applications on these nodes, and the systems implemented within the applications

On the therapist’s personal computer, there are 5 software nodes:

  • A web interface designed to provide the therapist with control over the TRAVEE system, accessible through a web client such as Mozilla Firefox, Microsoft Edge, or Google Chrome;
  • A communication system between the systems managing the hardware nodes represented by the EEG headset, the FES system, the IMU sensor, and the Kinect camera, as well as the control panel in the web interface and the application on the VR headset;
  • Three separate systems, one for managing the EEG headset and the FES system, one for managing the IMU sensor, and one for managing the Kinect camera. These hardware devices are used by the TRAVEE system during the rehabilitation sessions.

The Server hardware node has two software nodes, represented by a server application that stores information persistently in the database and manages the security of communication with the web application and the VR headset application, and the database service.

The VR headset hardware node contains the application used by the patient to visualize the virtual environment in which the rehabilitation sessions takes place.

(4) Description of the exercises and evaluation results

There are 5 types of recovery exercises: the exercise involving individual movements, the game-based exercise called ‘Ball Touching,’ the game-based exercise called ‘Fruits’, the game-based exercise called ‘Path’, and the game-based exercise called ‘Rocket Launch.’

The exercise involving individual movements offers the following movements for both hands: arm abduction-adduction, arm anteversion-retroversion, forearm flexion-extension, arm flexion-extension, finger flexion-extension, wrist flexion-extension, thumb opposition, digit-to-digit grasp, forearm pronation-supination, and shoulder raising.

Exercises through individual movements

The game-based exercise called ‘Ball Touching’ requires the patient to touch multiple balls, which approach from various positions in the world.

The game-based exercise called ‘Ball Touching’. The VR headset application features a virtual environment containing a stadium

The game-based exercise called ‘Fruits’ involves the patient viewing a virtual environment where fruits appear either to the left or right of the patient and optionally a basket in front of the patient at the level of their pelvis. The patient must collect all the fruits and place them in the basket through the following process:

  1. The patient touches one of the fruits with the hand on the side where the fruit is located, left or right;
  2. At this moment, the fruit is attached to the hand and follows the hand’s position in space;
  3. The patient must move the hand to which the fruit is attached above the basket, at which point the fruit detaches from the hand and falls into the basket.

The game-based exercise called ‘Fruits.’ The VR headset application features a 3D model of the basket in which the fruits must be placed

The game-based exercise called ‘Path’ involves the patient moving their hand along a pre-established curve, referred to as the path, which they must touch with their palm throughout the entire route. Touching an unpaved area of the curve creates a marking to complete that section. The path is completed when the entire path is marked. At the end of the game, a success message appears in front of the patient.

The game-based exercise called ‘Path’. The VR headset application displays the curve in the form of a branch, which is visible to the patient

The game-based exercise called ‘Rocket Launch’ involves the patient performing a specific movement when a ‘little monster’ appears in the virtual environment, approaching the patient from the front. When the patient performs the movement, a rocket is launched from the patient’s position, automatically moving towards the little monster and making it disappear.

The game-based exercise called ‘Rocket Launch’. 3D models of the ‘little monsters’ that move towards the patient.

For all exercises, the configuration parameters are detailed, along with the exercise behavior in case the parameter values are modified during the execution of an exercise, and how the play area is controlled, specific to each type of exercise.

The functional testing of the BCI&FES system included: electrode placement for FES, testing functional electrical stimulation parameters, attaching the EEG headset with electrodes, recording data for BCI system calibration, calculating spatial filters and the classifier, and performing exercises with visual feedback and induced movements using FES.

Non-functional testing of a prototype is an essential stage in evaluating its performance and characteristics in various conditions and contexts. Within the testing of exercises with the BCI&FES system, various methods have been used to assess and validate the performance and characteristics of the BCI&FES system, such as performance testing, failover testing, usability testing, security testing, disaster recovery testing, efficiency testing, and reliability testing

(5) Testing and Evaluation Report

The pilot study was conducted at INRMFB and involved 13 medical professionals specialized in neuromotor recovery and 30 patients with neuromotor deficits.

The patients were selected by the INRMFB team from medically eligible inpatients. The patients were informed about the nature and objectives of the pilot study and expressed their consent to participate through a standard informed consent form.

The involved patients had varying demographic characteristics and clinical profiles and were at different stages of functional recovery.

Recovery with the TRAVEE system was supplementary to the patients’ recovery activities using other conventional means.

The patients underwent a variable number of sessions (between 1 and 5), on the same day or on different days.

Feedback forms were collected from patients (after each session) and from therapists (after the completion of the group of 30 patients), and the medical impact of the pilot study on each patient was individually analyzed.

As a result of the pilot study:

  • the concrete benefits and the very high acceptance rate of the TRAVEE system were highlighted
  • numerous ideas for improving the system emerged, which are synthesized in a later chapter and led to the refinement of the prototype
  • selection criteria for patients who can benefit from the virtual reality training program within theproject were established

(6) Description of the Complete Prototype

(6.1) Technical Capabilities of the Complete Prototype

The system, through the web application interface, allows the management of data related to the rehabilitation session process, such as user information, institutions, VR headsets, doctor and patient accounts. This information is used to facilitate the rehabilitation sessions and analyze the progress of each patient’s use of the TRAVEE system.

The web application communicates with the server application and the database service to persistently store the actions performed by the therapist in the TRAVEE system. This process aims to facilitate further analysis of the process to improve technical capabilities and the results of rehabilitation sessions. Additionally, the web application ensures user access rights verification for system users. There are four user role classes: administrator, institution administrator, doctor, and default headset account, each with different responsibilities within the system.

Interface for Viewing the List of VR Headsets

The process of conducting a recovery session is supported by two different applications: the web application, which provides a control panel for the therapist conducting the session, and the VR headset application, used by the patient to visualize an attractive virtual environment for performing recovery exercises.

The web application provides the therapist with interfaces for viewing the list of sessions associated with the institution they belong to, creating a session, and conducting a session. In the session interface, the therapist has access to starting an exercise based on the configuration parameters, modifying these parameter values during the exercise, completing the exercise, and ending the session.

The interface used for conducting an exercise within a recovery session

The system offers 5 different types of recovery exercises: the exercise involving individual movements, the game-based exercise called ‘Touching the Balls’, the game-based exercise called ‘Fruits’, the game-based exercise called ‘The Path’, and the game-based exercise called ‘Rocket Launch.’

The EEG signals related to the motor cortex of the brain are acquired through the gUSBamp device, using 16 electrodes placed according to the 10-20 EEG map standard. The system integrates algorithms for EEG signal processing and information extraction for classification. To discriminate between two classes of motor imagery (e.g., imagining the movement of the right hand versus imagining the movement of the left hand), the model utilizes the Common Spatial Patterns (CSP) method. This method is based on the simultaneous diagonalization of two covariance matrices. It performs a transformation that maximizes the variance of samples for one condition while simultaneously minimizing the variance of samples for the other condition.

To assist in performing the movements, a neurostimulator is used, which takes stimulation commands from the computer and translates them into electrical signals designed to stimulate the neuromuscular system of the affected limb. Channels 1:4 of the neurostimulator are assigned for stimulating the right upper limb, while channels 5:8 are assigned for stimulating the left upper limb.

The HDC component interfaces with auxiliary hardware devices used for (1) collecting data from the environment or the patient and (2) providing feedback during neuro-motor recovery exercises. The HDC component can be seen as a ‘glue logic’ between the Web Dashboard, HMD, and the auxiliary hardware devices integrated into the system. In the current stage, the HDC has been adapted to interface with the BCI & FES system, IMU devices, and haptic feedback units.

The architecture of the HDC component

(6.2) Testing the complete prototype

The procedure for functional and non-functional testing of VR exercises and games, as well as the associated therapist interface, was described and illustrated in (4).

The elements covered by the procedure were applied to each system prototype before being sent to patient piloting at INRMFB. This resulted in a very low error rate during piloting.

During testing, the BCI system detected the subject’s imagination of hand movement at approximately one second from the cue presentation. The FES system responded with a delay of approximately 30 milliseconds from the trigger signal sent by the BCI system.

Functional Electrical Stimulation (FES)

The HDC component was functionally validated in two stages: (1) through simulation of interactions with the web interface and HMD, and (2) by generating command and data flows within the complete prototype.

As it is a device developed internally by the UTI team, the IMU device underwent an extensive testing process, typical for the development of complex hardware-software systems (unit testing, integration testing, etc.).

(7) Description of the telemedicine system

The server hardware node contains two software nodes: the web application and the database service.

The database service is responsible for the persistent storage of information in the system. For the telemedicine component, the entities in the database include prescriptions created by therapists for autonomous home-based exercise sessions by patients, along with entities containing information about the prescribed exercises. Additionally, all elements within the session execution component, which is the largest component in the entire system, are closely related to the telemedicine component since it is directly used for conducting exercises prescribed by therapists.

The server application is responsible for sending information to the database service for persistent storage and for verifying access rights in the system, both for users with the role of doctors who input prescriptions and for the default accounts of VR headsets.

The web application interface used by therapists allows them to view a list of prescriptions associated with the institution to which a doctor-user belongs. Additionally, such a user is permitted to create and edit prescriptions by selecting a patient associated with their institution and a list of activities. They can specify, for each activity, the desired configuration parameters and the maximum duration of the activity.

Interface for Creating and Editing a Prescription

“Another important component provided to users with the role of a doctor is the ability to view the results of a telemedicine activity by accessing the session management component, where information about the progress of exercises performed by patients is saved.

The VR headset application is autonomous and, by accessing the server application, it detects the existence of prescriptions associated with the VR headset on which the application is running. At this point, the patient has the option to select, by orienting the headset with its center marked by a point in a menu interface, the execution of the prescription and one of the prescribed exercises.

The recovery exercises and games are conducted similarly to their clinical execution.

(8) Description of the Final Prototype

(8.1) Technical Capabilities of the Final Prototype

The version implemented in the final prototype includes modifications compared to the one in the complete prototype. These changes are made to improve the usability of the interfaces provided by the TRAVEE system, the quality of the experience during recovery sessions, and also to refine the system for the use of all components in the pilot and clinical study phase that will take place in the next stage of the project.

The hardware architecture of the final system

The web application interface used by the therapist contains modifications aimed at compacting the displayed information and facilitating navigation between the project’s components. Additionally, the final prototype includes interface elements to facilitate communication with the HDC application, which is designed for the use of additional hardware devices beyond the VR headset. In the final prototype, access to the telemedicine component is introduced, unlike the complete prototype, which contains this component privately for laboratory testing purposes. This change is made to enable therapists to use the telemedicine component. The exercise execution interface contains new elements that improve the visualization and control of exercise configuration parameters.

The VR headset application contains several modifications to enhance the visual quality of the virtual environments used by patients during recovery exercises. Furthermore, the improvements include more diversified exercise control by introducing new types of games, such as the ability to bring fruits to the mouth for the “Fruits” game and the ability to use both hands alternately or randomly for individual motion exercises and games called ‘Ball Touching’ and “Rocket Launch”. The control of the play area is enhanced for all games by completely modifying or introducing new configuration parameters that provide superior control over the exercise execution area. Finally, the communication system with the HDC application has been perfected to enable communication between this application, the VR headset application, and the web application.

(8.2) Testing the final prototype

The procedure for functional and non-functional testing of VR exercises and games, as well as the associated therapist interface, is similar to the one described and illustrated in (6.2).

The elements covered by the procedure were applied to each prototype of the system before being sent to the ongoing patient pilot study at INRMFB. This resulted in an extremely low error rate during pilot testing.

Following the redesign, the HDC component underwent a functional validation process similar to the one described in (6.2): (1) through simulations of interactions with the web interface and the HMD, and (2) by generating command and data flows within the final prototype.

The functional diagram of the HDC as an intermediary between the HMD and the hardware monitoring and stimulation devices for the patient

Following the software improvements, the IMU device was evaluated in terms of energy consumption and data reporting stability.

(9) Writing 6 scientific articles

Voice Metrics for Discourse Quality Analysis

Nicolae Jinga, Florica Moldoveanu, Alin Moldoveanu, Anca Morar, Irina Mocanu, Alexandru Butean

On Authentication in Virtual Reality Environments for Rehabilitation and Psychotherapy Systems

Florina Ungureanu, Bianca Andreea Bordea, Robert Gabriel Lupu, George Vieriu

Immersive Phobia Therapy through Adaptive Virtual Reality and Biofeedback

Alin Moldoveanu, Oana Mitruț, Nicolae Jinga, Cătălin Petrescu, Florica Moldoveanu ,Victor Asavei, Ana Magdalena Anghel, Livia Petrescu

Biophysical Signal Processing for Automatic Anxiety Classification in a Virtual Reality Exposure Therapy System

Nicolae Jinga, Catalin-Dumitru Petrescu, Oana Mitrut, Alin Moldoveanu, Florica Moldoveanu and Livia Petrescu

Gamification of post-stroke neuromotor rehabilitation exercises using hand tracking in VR

Robert Florin Apavaloaiei, Stefan Daniel Achirei

Gamified Virtual Reality in Post-stroke Neurorehabilitation: A Systematic Review

Alin Moldoveanu, Iulia-Cristina Stănică, Ana-Magdalena Anghel, Mara-Ilinca-Mădălina Brezeanu

(10) Update of the TRAVEE project website

Phase 1 – Initiation (June 2022 – December 2022)

This section presents the main outputs for the activities which took place during Phase 1 – Initiation of the TRAVEE project.

Results:

(1) State of the art report regarding relevant technologies

During the first phase, taking into consideration the rapid technological progress, a series of studies have been elaborated to identify the advancements and the current innovative solutions in the fields of interest of the project (HCI – Human Computer Interaction, haptification, serious virtual reality and augmented reality games, BCI, FES, IMU). The following studies have been elaborated:

  • A detailed study of the virtual and augmented reality solutions considering the imposed low costs constraints
  • A detailed study of BCI hardware equipment
  • A detailed study of monocular video tracking solutions
  • A detailed study of telemedicine solutions
  • A detailed study of physical recovery exercises and their gamification
  • A detailed study of robotic glove equipment
  • A detailed study of equipment containing IMU sensors, for tracking different limbs
  • A detailed study of interactions based on emotional response between the patient and the virtual therapist interface

(2) The User Requirements Document (URD)

The URD has been elaborated starting from the detailed requirements expressed by all the members of the consortium, according to the established project scope, written in the project proposal document and considering the state of the art in technological advancements, evaluated previously.

The URD has detailed the following aspects:

  • Actors (name, short description and summary of functions accessed, expectations, required competencies)
  • Generic hardware components (role, implementation options, examples, minimal requirements)
  • Use cases (name, summary, pre and post conditions, main flow, exceptions, attributes, constraints, etc.)

System specifications defined by use cases

(3) Architectural Design Document (ADD)

The ADD describes in a clear and consistent manner the high-level architecture of the TRAVEE system.

This document was elaborated starting from the user requirements specified previously, in the URD document and after studies, selection and preliminary experiments with existing technologies.

The level of detail of the architecture was established with respect to the chosen development model (detailed design activities for individual modules of the system, which are to be developed during the future Agile sprints), and the expertise of the partners.

The UML diagram of the TRAVEE system containing the hardware and software modules of the system

(4) Report of recovery exercises and how to define them

During the first phase, a detailed study regarding the domain of gamification of neuromotor recovery exercises has been elaborated.

The selection process of the most relevant exercises and of the optimal forms of gamification is extremely important. The considered approach proposes a combination of four elements:

  • The current perspective of the medical partners and the gamification principles proposed by them
  • The perspective of the ICT partners
  • The results obtained during the previous TRAVEE project (2015-2018)
  • The conclusions of the study regarding the gamification of the neuromotor recovery exercises

An important aspect is the desire of using incipient forms of gamification even in the early recovery stage, followed by advanced forms (mini games with gradual difficulty) during the advanced recovery stage.

The perspectives and the selection of exercises and gamification associated forms, will be correlated, and completed according to the results of the previous study.

In the end, considering the limited availability of resources for implementation, it is important to bound the forms of gamification used, within a manageable range which can be covered, but at the same time having consideration for what is most suitable considering the preferences and average profiles of the INRMFB patients. Thus, the partners consider applying some preference questionnaires on patients, at the beginning of the second phase.

(5) Writing 4 scientific articles

On the Design of a Modular Wearable Solution for Assistive Technologies

Nicolae Alexandru Botezatu, Robert Gabriel Lupu, Simona Caraiman

Vocabulary enrichment in mother language after preschoolers’ interaction with a social robot

Nicoleta Laura Popa, Oana Alexandra Clim, Adina Boaca, Cristian-Tiberius Axinte, Robert-Gabriel Lupu, Georgiana Juravle

Evaluation of non-textual metrics for public speaking analysis and training

Nicolae Jinga, Alin Moldoveanu, Florica Moldoveanu, Anca Morar, Alexandru Butean

A systematic survey of gamification strategies in VR-based neuromotor rehabilitation

Mara Ilinca Mădălina Chirașcu, Iulia Stănică, Alin Moldoveanu, Florica Moldoveanu

(6) Development of the TRAVEE project website