OUTPUTS

Phase 3 – System Refinement, Clinical Study, and Continuation Preparation (January 2024 – June 2024)

This section presents the main results for the activities that took place during Phase 3 – System Refinement, Clinical Study, and Continuation Preparation of the TRAVEE project.

Results:

(1) Clinical study report

The purpose of the study was to demonstrate the improvement of active mobility in the paralyzed upper limb in post-stroke patients with sequelae, through training using the TRAVEE system. This is a complex technological assistance system that allows adaptation to the functional level and the phase of medical rehabilitation in which each patient is.

It is well known that motor rehabilitation is of crucial importance in the partial or total recovery of impaired motor control. Rehabilitation techniques are based on the repetitive and intensive execution of simple motor tasks. These requirements can become demanding in terms of economic costs and the involvement of both staff and patients. They can lead to lack of adherence and abandonment of the rehabilitation process.

In addition to this, increasing the patient’s motivation, interest, and curiosity towards the therapeutic methods used is essential. These aspects are important because the repetitive application of monotonous training methods, lacking variety and novelty, and ignoring the playful aspect of the learning process can lead to fatigue, boredom, lack of participation, and abandonment of the process, with detrimental consequences on the level of motor rehabilitation.

The main objectives of the study were:

1. evaluation of the (active and passive) mobility of the upper limb following training with the TRAVEE system
2. analysis of achieving the initial therapeutic goals set using Goal Attainment Scaling (GAS).

The secondary objectives were:

1. the safety of training with the TRAVEE system
2. correlation of the WHODAS score with combined medical rehabilitation training and training with the TRAVEE system
3. evaluation of the degree of spasticity following medical rehabilitation training and training with the TRAVEE system
4. evaluation of the degree of pain on the NRS (Numerical Rating Scale) following medical rehabilitation training and training with the TRAVEE system
5. patient satisfaction
6. satisfaction of the medical rehabilitation team

Related objectives:

1. correct functioning, efficiency, and usability of web interfaces for therapists
2. obtaining additional feedback from patients to improve the system and enhance their experience

Throughout the study, each subject was carefully observed and analyzed by the mixed team, with the individual data obtained allowing for the continuous improvement of the system with real-life data. Given the complexity of post-stroke sequelae and their interdependence with numerous medical and non-medical factors, the TRAVEE system still requires the development of functions that enable individual adaptation.

(2) Report on system improvements

IMPROVEMENTS TO THE WEB PLATFORM

The web platform has undergone multiple improvements aimed at simplifying the process of managing information within the platform and conducting a recovery session. Among the improvements made, we mention the following:

• Development of a game-based exercise management system at the institutional level
• The display of questionnaire questions has been improved by adding text content that describes the scale of a question
• Expansion of the feedback questionnaire by introducing a set of 4 new questions specific to the use of EEG, FES, and haptic devices
• Introduction of the ability to filter VR devices based on their use at home
• Introduction of the ability to search for a patient by name when creating a new session
• Addition of a system that allows specifying specific buttons for a game-based exercise during the exercise
• Addition of a system of constraints for displaying and activating the configuration parameters of a game-based exercise based on the values of other configuration parameters of the exercise
• Addition of a reporting system

 

IMPROVEMENTS TO THE VR APPLICATION

In the current phase, improvements were made to the game-based exercises piloted in the previous phase, and four new game-based exercises were developed:

• The game-based exercise “Diorama”

• The game-based exercise “Fires”

• The game-based exercise “Piano”

• The game-based exercise “Cooking”

All game-based exercises piloted in the previous phase have been technically refined. The few issues encountered during the piloting process have been resolved to provide therapists and patients with a pleasant experience in the neuromotor recovery process.

Additionally, the game-based exercise “Rocket” has been modified to support communication with a haptic hardware device in the form of a bracelet containing vibrating motors. When the hand movement is detected at the moment a rocket can be launched, the VR application sends a signal to the bracelet, via the HDC, requesting it to vibrate.

TELEMEDICINE IMPROVEMENTS

Improvements have been made to the telemedicine system. From the web platform perspective, the prescription management system has been refined to allow for easy creation and editing of prescriptions. Additionally, the process of editing game-based exercises included in a session has been improved.

HDC IMPROVEMENTS

To optimize the interaction logic between the Web Application – HDC, and HDC – VR Application, as well as to eliminate some unwanted behaviors identified during the piloting process, the following modifications have been made to the HDC component:

• Representation of command payloads in JSON format
• Removal of information regarding which limb is being used in the exercise, as it is not relevant for HDC
• Redefinition of the HDC-VR headset connection as a generic interaction with a hardware component to simplify the internal implementation logic
• Implementation of a detection filter for sequences of values indicating the beginning of measurement windows for the left/right limb at the BCI subsystem level
• Addition of the capability to associate a haptic feedback unit with a limb

(3) Technical Continuation Plan of the Project

The following research directions and improvements have been synthesized into the technical continuation plan of the project.

BCI SYSTEM

A possible research direction proposes advancing the motor imagery component of the BCI system for post-stroke patients from TRL 4 to TRL 6 or higher, which is essential for early recovery. This BCI component needs to be improved in terms of capabilities, accuracy, setup time, ergonomics, flexibility, and cost, and integrated into the TRAVEE system. The system will use transfer learning methods to improve the classification of EEG signals, which are weak and noise-affected.

The proposed neural network architecture may include: Convolutional Neural Networks (CNN) – Capturing spatial relationships between EEG channels; Recurrent Neural Networks (RNN) – Managing temporal dependencies between consecutive segments; Transformer – Global analysis of EEG signals.

Innovations and advantages of the proposed BCI system include: increased accuracy; significant reduction in recalibration time; continuous model improvement; adaptability to different EEG devices; the potential for using low-cost devices.

The project will focus on integrating this component into the existing TRAVEE system, which already includes other components at TRL 6 or higher, and will benefit from the current software infrastructure and clinical logistics of TRAVEE.

WEB INTERFACE FOR THERAPISTS

The web interface for therapists has been satisfactorily evaluated by the therapists who were involved in its use, but it can undergo improvements from several perspectives. Direct improvements are represented by those that ease the use of the platform, particularly for the web interface of the telemedicine system, which is in the clinical study phase at the time of writing this document.

Another important improvement for the web interface for therapists is the expansion of the reporting and statistics system. The platform currently offers a set of reports that summarize the results of game-based exercises performed by patients during recovery sessions. To this, reports and progress graphs at the level of patients and conducted activities should be added.

COMMUNICATION BETWEEN SYSTEM DEVICES

A necessary improvement is the introduction of a system that ensures synchronization of the system state between the devices used at a given time. Due to the use of wireless systems in a clinical context or at home, which can experience packet loss or connection loss, it is necessary to introduce a system that ensures the transmission and reception of messages between the devices communicating during a session. This improvement ensures the maintenance of the state of the devices and the system as a whole.

GAME-BASED EXERCISES

Another important research direction is to increase the set of game-based exercises provided by the TRAVEE system by introducing new exercises similar to the existing ones and by designing and developing new types of exercises that use physical props and offer the possibility of conducting an exercise with multiple patients in the same virtual world.

AUTOMATIC DETECTION AND ADAPTATION OF THE WORK AREA

One of the most important development directions is the introduction of a system that allows the automatic adaptation of the work area for game-based exercises. A significant part of the therapists’ activity during a recovery session with the TRAVEE system is adjusting the exercises to meet the needs of the patient performing the recovery session. Thus, an automatic adaptation system assists therapists by simplifying the process of conducting a session. In addition, this system supports the home recovery process by automatically adapting the work space, without the need for remote interaction with a therapist.

(4) The User Manual for the TRAVEE System

The user manual is a comprehensive document addressed to end-users such as therapists and system administrators.

The manual details the use of all system functionalities and includes the following sections:

• System Architecture (physical components and software modules)
• Web Application for Therapists
• HDC Application (for controlling specialized hardware devices)
• Server
• VR Application
• BCI Application
• FES Application
• Haptic Application
• Recovery Exercises and Games

(5) User Evaluation Report of the Complete Prototype

Patient Feedback

The feedback from all patients has been remarkably good, right from the first use of VR.

After each recovery day using TRAVEE, patients completed a feedback form.

Therapist Feedback

So far, 19 therapists from INRMFB have been involved in the piloting, all with relevant medical specializations, medical grades, and experience covering a full range.

All therapists have highly appreciated the TRAVEE system from all perspectives evaluated through the feedback form.

(6) Writing 5 scientific articles

Gamified Virtual Reality in Post-stroke Neurorehabilitation: A Systematic Review

Iulia Cristina Stănică, Alin Moldoveanu, Ana Magdalena Anghel, Andrei Cristian Lambru, Victor Asavei, Delia Cinteză

The Impact of Virtual Reality Training in post Stroke Patients – a TRAVEE Experience

Delia Cinteză, Miruna Săndulescu, Claudia-Gabriela Potcovaru, Iulia Ghinea, Ionela Oprea, Andrei Cristian Lambru, Alin Dragoș Bogdan Moldoveanu, Daniela Poenaru

Robotic assisted immersive therapy system for stroke rehabilitation

Robert Gabriel Lupu, Adrian Burlacu, Ovidiu Catarama, Ionut Sopon

Emotion-based interaction between virtual therapist and patients – a systematic review

Iulia-Cristina Stanica, Alin Moldoveanu

TRAVEE – Recuperare neuromotorie gamificată și adaptabilă in realitate virtuală

Alin Moldoveanu, Andrei Cristian Lambru, Ana Magdalena Anghel, Iulia Stănică, Florica Moldoveanu, Robert Gabriel Lupu, Victor Asavei, Andreea Dumitrescu, Ovidiu Chiriac, Andrei Baltescu, Delia Cinteză

(7) Update of the TRAVEE project website

Phase 2 – Prototype development and testing (January 2023 – December 2023)

This section presents the main outputs for the activities which took place during Phase 2 – Prototype development and testing of the TRAVEE project.

Results:

(1) Report on hand monitoring algorithms and BCI algorithms

In the hand monitoring process, the system provided by the Oculus Quest 2 VR headset is used. This is a commercial system that provides satisfactory results for the rehabilitation session process.

An important advantage provided by the Oculus Quest 2 VR headset system is that it comes built-in with palm and finger tracking, eliminating the need for additional hardware devices. This makes the TRAVEE system easily usable independently, such as in telemedicine applications.

The inertial unit developed by the UTI team has 9 degrees of freedom (9DOF) and contains a processing core that consolidates primary information from sensors (accelerometer, gyroscope, and magnetometer). This processing aims to provide the TRAVEE system with data on orientation and linear acceleration, contributing to the integration and calculation of displacement velocity. Additionally, the device has the capability to identify the current usage state, allowing differentiation between situations where it is placed on a static surface (e.g., a table) or held steady on the patient’s limb.

The movement of a limb or even the contraction of a single muscle leads to changes in brain activity at the cortical level. In fact, even the preparation for a movement or imagining the movement leads to changes in the so-called sensorimotor rhythms. For discriminating these changes through BCI, the CSP method has proven to be very efficient, as it eliminates disturbances in EEG signals by correlating them with neighboring electrodes. The result provided by the BCI system after classification is further conveyed via HDC both to the FES (Functional Electrical Stimulation) system, which induces the corresponding hand movement, and to the server, which can use the information for other subsystems.

(2) Algorithm Testing Report

Tests were conducted for the following categories of algorithms/hardware:

• BCI Algorithm Testing: In order to achieve the best control accuracy, six time windows of 2 seconds each were selected from the training data for creating spatial filters and the classifier. The set with the smallest cross-validation error was chosen to generate online feedback. The optimal window for variance calculation was selected as 1.5 seconds.

• FES Algorithm Testing: The functional electrical stimulation system was tested for real-time prescription of hand extensor stimulation parameters based on signals from the input block. For positive values received as feedback from the BCI system, the algorithm activates electrical stimulation on the channels in the first set of 4 (Ch 1:4), while for negative values, it activates stimulation on the channels in the second set of channels (Ch 5:8)

• IMU Testing: Since the IMU device was developed entirely within the project, the testing process included a sequence of steps for its progressive validation – testing the hardware along with its corresponding firmware, testing communication and Bluetooth interactions, validating integration with HDC, and validating the outputs provided by the IMU device
• Hand Tracking Algorithm Testing: One aspect of testing the hand tracking by the Oculus Quest 2’s video tracking system was performance. In this regard, factors considered included latency, static positional accuracy, path offset, angular error, and lighting conditions

(3) Description of the system’s skeleton

The TRAVEE system consists of 7 hardware nodes, on which 8 different applications are executed, containing a total of 30 software systems. All of these can be viewed in the system architecture diagram.

On the therapist’s personal computer, there are 5 software nodes:

• A web interface designed to provide the therapist with control over the TRAVEE system, accessible through a web client such as Mozilla Firefox, Microsoft Edge, or Google Chrome;
• A communication system between the systems managing the hardware nodes represented by the EEG headset, the FES system, the IMU sensor, and the Kinect camera, as well as the control panel in the web interface and the application on the VR headset;
• Three separate systems, one for managing the EEG headset and the FES system, one for managing the IMU sensor, and one for managing the Kinect camera. These hardware devices are used by the TRAVEE system during the rehabilitation sessions.

The Server hardware node has two software nodes, represented by a server application that stores information persistently in the database and manages the security of communication with the web application and the VR headset application, and the database service.

The VR headset hardware node contains the application used by the patient to visualize the virtual environment in which the rehabilitation sessions takes place.

(4) Description of the exercises and evaluation results

There are 5 types of recovery exercises: the exercise involving individual movements, the game-based exercise called ‘Ball Touching,’ the game-based exercise called ‘Fruits’, the game-based exercise called ‘Path’, and the game-based exercise called ‘Rocket Launch.’

The exercise involving individual movements offers the following movements for both hands: arm abduction-adduction, arm anteversion-retroversion, forearm flexion-extension, arm flexion-extension, finger flexion-extension, wrist flexion-extension, thumb opposition, digit-to-digit grasp, forearm pronation-supination, and shoulder raising.

The game-based exercise called ‘Ball Touching’ requires the patient to touch multiple balls, which approach from various positions in the world.

The game-based exercise called ‘Fruits’ involves the patient viewing a virtual environment where fruits appear either to the left or right of the patient and optionally a basket in front of the patient at the level of their pelvis. The patient must collect all the fruits and place them in the basket through the following process:

1. The patient touches one of the fruits with the hand on the side where the fruit is located, left or right;
2. At this moment, the fruit is attached to the hand and follows the hand’s position in space;
3. The patient must move the hand to which the fruit is attached above the basket, at which point the fruit detaches from the hand and falls into the basket.

The game-based exercise called ‘Path’ involves the patient moving their hand along a pre-established curve, referred to as the path, which they must touch with their palm throughout the entire route. Touching an unpaved area of the curve creates a marking to complete that section. The path is completed when the entire path is marked. At the end of the game, a success message appears in front of the patient.

The game-based exercise called ‘Rocket Launch’ involves the patient performing a specific movement when a ‘little monster’ appears in the virtual environment, approaching the patient from the front. When the patient performs the movement, a rocket is launched from the patient’s position, automatically moving towards the little monster and making it disappear.

For all exercises, the configuration parameters are detailed, along with the exercise behavior in case the parameter values are modified during the execution of an exercise, and how the play area is controlled, specific to each type of exercise.

The functional testing of the BCI&FES system included: electrode placement for FES, testing functional electrical stimulation parameters, attaching the EEG headset with electrodes, recording data for BCI system calibration, calculating spatial filters and the classifier, and performing exercises with visual feedback and induced movements using FES.

Non-functional testing of a prototype is an essential stage in evaluating its performance and characteristics in various conditions and contexts. Within the testing of exercises with the BCI&FES system, various methods have been used to assess and validate the performance and characteristics of the BCI&FES system, such as performance testing, failover testing, usability testing, security testing, disaster recovery testing, efficiency testing, and reliability testing

(5) Testing and Evaluation Report

The pilot study was conducted at INRMFB and involved 13 medical professionals specialized in neuromotor recovery and 30 patients with neuromotor deficits.

The patients were selected by the INRMFB team from medically eligible inpatients. The patients were informed about the nature and objectives of the pilot study and expressed their consent to participate through a standard informed consent form.

The involved patients had varying demographic characteristics and clinical profiles and were at different stages of functional recovery.

Recovery with the TRAVEE system was supplementary to the patients’ recovery activities using other conventional means.

The patients underwent a variable number of sessions (between 1 and 5), on the same day or on different days.

Feedback forms were collected from patients (after each session) and from therapists (after the completion of the group of 30 patients), and the medical impact of the pilot study on each patient was individually analyzed.

As a result of the pilot study:

• the concrete benefits and the very high acceptance rate of the TRAVEE system were highlighted
• numerous ideas for improving the system emerged, which are synthesized in a later chapter and led to the refinement of the prototype
• selection criteria for patients who can benefit from the virtual reality training program within theproject were established

(6) Description of the Complete Prototype

(6.1) Technical Capabilities of the Complete Prototype

The system, through the web application interface, allows the management of data related to the rehabilitation session process, such as user information, institutions, VR headsets, doctor and patient accounts. This information is used to facilitate the rehabilitation sessions and analyze the progress of each patient’s use of the TRAVEE system.

The web application communicates with the server application and the database service to persistently store the actions performed by the therapist in the TRAVEE system. This process aims to facilitate further analysis of the process to improve technical capabilities and the results of rehabilitation sessions. Additionally, the web application ensures user access rights verification for system users. There are four user role classes: administrator, institution administrator, doctor, and default headset account, each with different responsibilities within the system.

The process of conducting a recovery session is supported by two different applications: the web application, which provides a control panel for the therapist conducting the session, and the VR headset application, used by the patient to visualize an attractive virtual environment for performing recovery exercises.

The web application provides the therapist with interfaces for viewing the list of sessions associated with the institution they belong to, creating a session, and conducting a session. In the session interface, the therapist has access to starting an exercise based on the configuration parameters, modifying these parameter values during the exercise, completing the exercise, and ending the session.

The system offers 5 different types of recovery exercises: the exercise involving individual movements, the game-based exercise called ‘Touching the Balls’, the game-based exercise called ‘Fruits’, the game-based exercise called ‘The Path’, and the game-based exercise called ‘Rocket Launch.’

The EEG signals related to the motor cortex of the brain are acquired through the gUSBamp device, using 16 electrodes placed according to the 10-20 EEG map standard. The system integrates algorithms for EEG signal processing and information extraction for classification. To discriminate between two classes of motor imagery (e.g., imagining the movement of the right hand versus imagining the movement of the left hand), the model utilizes the Common Spatial Patterns (CSP) method. This method is based on the simultaneous diagonalization of two covariance matrices. It performs a transformation that maximizes the variance of samples for one condition while simultaneously minimizing the variance of samples for the other condition.

To assist in performing the movements, a neurostimulator is used, which takes stimulation commands from the computer and translates them into electrical signals designed to stimulate the neuromuscular system of the affected limb. Channels 1:4 of the neurostimulator are assigned for stimulating the right upper limb, while channels 5:8 are assigned for stimulating the left upper limb.

The HDC component interfaces with auxiliary hardware devices used for (1) collecting data from the environment or the patient and (2) providing feedback during neuro-motor recovery exercises. The HDC component can be seen as a ‘glue logic’ between the Web Dashboard, HMD, and the auxiliary hardware devices integrated into the system. In the current stage, the HDC has been adapted to interface with the BCI & FES system, IMU devices, and haptic feedback units.

(6.2) Testing the complete prototype

The procedure for functional and non-functional testing of VR exercises and games, as well as the associated therapist interface, was described and illustrated in (4).

The elements covered by the procedure were applied to each system prototype before being sent to patient piloting at INRMFB. This resulted in a very low error rate during piloting.

During testing, the BCI system detected the subject’s imagination of hand movement at approximately one second from the cue presentation. The FES system responded with a delay of approximately 30 milliseconds from the trigger signal sent by the BCI system.

The HDC component was functionally validated in two stages: (1) through simulation of interactions with the web interface and HMD, and (2) by generating command and data flows within the complete prototype.

As it is a device developed internally by the UTI team, the IMU device underwent an extensive testing process, typical for the development of complex hardware-software systems (unit testing, integration testing, etc.).

(7) Description of the telemedicine system

The server hardware node contains two software nodes: the web application and the database service.

The database service is responsible for the persistent storage of information in the system. For the telemedicine component, the entities in the database include prescriptions created by therapists for autonomous home-based exercise sessions by patients, along with entities containing information about the prescribed exercises. Additionally, all elements within the session execution component, which is the largest component in the entire system, are closely related to the telemedicine component since it is directly used for conducting exercises prescribed by therapists.

The server application is responsible for sending information to the database service for persistent storage and for verifying access rights in the system, both for users with the role of doctors who input prescriptions and for the default accounts of VR headsets.

The web application interface used by therapists allows them to view a list of prescriptions associated with the institution to which a doctor-user belongs. Additionally, such a user is permitted to create and edit prescriptions by selecting a patient associated with their institution and a list of activities. They can specify, for each activity, the desired configuration parameters and the maximum duration of the activity.

“Another important component provided to users with the role of a doctor is the ability to view the results of a telemedicine activity by accessing the session management component, where information about the progress of exercises performed by patients is saved.

The VR headset application is autonomous and, by accessing the server application, it detects the existence of prescriptions associated with the VR headset on which the application is running. At this point, the patient has the option to select, by orienting the headset with its center marked by a point in a menu interface, the execution of the prescription and one of the prescribed exercises.

The recovery exercises and games are conducted similarly to their clinical execution.

(8) Description of the Final Prototype

(8.1) Technical Capabilities of the Final Prototype

The version implemented in the final prototype includes modifications compared to the one in the complete prototype. These changes are made to improve the usability of the interfaces provided by the TRAVEE system, the quality of the experience during recovery sessions, and also to refine the system for the use of all components in the pilot and clinical study phase that will take place in the next stage of the project.

The web application interface used by the therapist contains modifications aimed at compacting the displayed information and facilitating navigation between the project’s components. Additionally, the final prototype includes interface elements to facilitate communication with the HDC application, which is designed for the use of additional hardware devices beyond the VR headset. In the final prototype, access to the telemedicine component is introduced, unlike the complete prototype, which contains this component privately for laboratory testing purposes. This change is made to enable therapists to use the telemedicine component. The exercise execution interface contains new elements that improve the visualization and control of exercise configuration parameters.

The VR headset application contains several modifications to enhance the visual quality of the virtual environments used by patients during recovery exercises. Furthermore, the improvements include more diversified exercise control by introducing new types of games, such as the ability to bring fruits to the mouth for the “Fruits” game and the ability to use both hands alternately or randomly for individual motion exercises and games called ‘Ball Touching’ and “Rocket Launch”. The control of the play area is enhanced for all games by completely modifying or introducing new configuration parameters that provide superior control over the exercise execution area. Finally, the communication system with the HDC application has been perfected to enable communication between this application, the VR headset application, and the web application.

(8.2) Testing the final prototype

The procedure for functional and non-functional testing of VR exercises and games, as well as the associated therapist interface, is similar to the one described and illustrated in (6.2).

The elements covered by the procedure were applied to each prototype of the system before being sent to the ongoing patient pilot study at INRMFB. This resulted in an extremely low error rate during pilot testing.

Following the redesign, the HDC component underwent a functional validation process similar to the one described in (6.2): (1) through simulations of interactions with the web interface and the HMD, and (2) by generating command and data flows within the final prototype.

Following the software improvements, the IMU device was evaluated in terms of energy consumption and data reporting stability.

(9) Writing 6 scientific articles

Voice Metrics for Discourse Quality Analysis

Nicolae Jinga, Florica Moldoveanu, Alin Moldoveanu, Anca Morar, Irina Mocanu, Alexandru Butean

On Authentication in Virtual Reality Environments for Rehabilitation and Psychotherapy Systems

Florina Ungureanu, Bianca Andreea Bordea, Robert Gabriel Lupu, George Vieriu

Immersive Phobia Therapy through Adaptive Virtual Reality and Biofeedback

Alin Moldoveanu, Oana Mitruț, Nicolae Jinga, Cătălin Petrescu, Florica Moldoveanu ,Victor Asavei, Ana Magdalena Anghel, Livia Petrescu

Biophysical Signal Processing for Automatic Anxiety Classification in a Virtual Reality Exposure Therapy System

Nicolae Jinga, Catalin-Dumitru Petrescu, Oana Mitrut, Alin Moldoveanu, Florica Moldoveanu and Livia Petrescu

Gamification of post-stroke neuromotor rehabilitation exercises using hand tracking in VR

Robert Florin Apavaloaiei, Stefan Daniel Achirei

Gamified Virtual Reality in Post-stroke Neurorehabilitation: A Systematic Review

Alin Moldoveanu, Iulia-Cristina Stănică, Ana-Magdalena Anghel, Mara-Ilinca-Mădălina Brezeanu

(10) Update of the TRAVEE project website

Phase 1 – Initiation (June 2022 – December 2022)

This section presents the main outputs for the activities which took place during Phase 1 – Initiation of the TRAVEE project.

Results:

(1) State of the art report regarding relevant technologies

During the first phase, taking into consideration the rapid technological progress, a series of studies have been elaborated to identify the advancements and the current innovative solutions in the fields of interest of the project (HCI – Human Computer Interaction, haptification, serious virtual reality and augmented reality games, BCI, FES, IMU). The following studies have been elaborated:

• A detailed study of the virtual and augmented reality solutions considering the imposed low costs constraints
• A detailed study of BCI hardware equipment
• A detailed study of monocular video tracking solutions
• A detailed study of telemedicine solutions
• A detailed study of physical recovery exercises and their gamification
• A detailed study of robotic glove equipment
• A detailed study of equipment containing IMU sensors, for tracking different limbs
• A detailed study of interactions based on emotional response between the patient and the virtual therapist interface

(2) The User Requirements Document (URD)

The URD has been elaborated starting from the detailed requirements expressed by all the members of the consortium, according to the established project scope, written in the project proposal document and considering the state of the art in technological advancements, evaluated previously.

The URD has detailed the following aspects:

• Actors (name, short description and summary of functions accessed, expectations, required competencies)
• Generic hardware components (role, implementation options, examples, minimal requirements)
• Use cases (name, summary, pre and post conditions, main flow, exceptions, attributes, constraints, etc.)

(3) Architectural Design Document (ADD)

The ADD describes in a clear and consistent manner the high-level architecture of the TRAVEE system.

This document was elaborated starting from the user requirements specified previously, in the URD document and after studies, selection and preliminary experiments with existing technologies.

The level of detail of the architecture was established with respect to the chosen development model (detailed design activities for individual modules of the system, which are to be developed during the future Agile sprints), and the expertise of the partners.

(4) Report of recovery exercises and how to define them

During the first phase, a detailed study regarding the domain of gamification of neuromotor recovery exercises has been elaborated.

The selection process of the most relevant exercises and of the optimal forms of gamification is extremely important. The considered approach proposes a combination of four elements:

• The current perspective of the medical partners and the gamification principles proposed by them
• The perspective of the ICT partners
• The results obtained during the previous TRAVEE project (2015-2018)
• The conclusions of the study regarding the gamification of the neuromotor recovery exercises

An important aspect is the desire of using incipient forms of gamification even in the early recovery stage, followed by advanced forms (mini games with gradual difficulty) during the advanced recovery stage.

The perspectives and the selection of exercises and gamification associated forms, will be correlated, and completed according to the results of the previous study.

In the end, considering the limited availability of resources for implementation, it is important to bound the forms of gamification used, within a manageable range which can be covered, but at the same time having consideration for what is most suitable considering the preferences and average profiles of the INRMFB patients. Thus, the partners consider applying some preference questionnaires on patients, at the beginning of the second phase.

(5) Writing 4 scientific articles

On the Design of a Modular Wearable Solution for Assistive Technologies

Nicolae Alexandru Botezatu, Robert Gabriel Lupu, Simona Caraiman

Vocabulary enrichment in mother language after preschoolers’ interaction with a social robot

Nicoleta Laura Popa, Oana Alexandra Clim, Adina Boaca, Cristian-Tiberius Axinte, Robert-Gabriel Lupu, Georgiana Juravle

Evaluation of non-textual metrics for public speaking analysis and training

Nicolae Jinga, Alin Moldoveanu, Florica Moldoveanu, Anca Morar, Alexandru Butean

A systematic survey of gamification strategies in VR-based neuromotor rehabilitation

Mara Ilinca Mădălina Chirașcu, Iulia Stănică, Alin Moldoveanu, Florica Moldoveanu

(6) Development of the TRAVEE project website