TRAVEE – Virtual Therapist with Augmented Feedback for Neuromotor Recovery (UEFISCDI contract number 83PTE from 30/06/2022)
SC INFO WORLD SRL
Victor Asavei (June 2022 – June 2023)
Cristian Lambru (July 2023 – June 2024)
Total Budget Project:
Public budget : 1.190.763,00 RON
Project implementation duration:
UEFISCDI – Transfer to the economic operator (PTE) – Submission code: PN-III-P2-2.1-PTE-2021-0634 http://uefiscdi.gov.ro/
The project is imagined as a continuation of the previous TRAVEE-PNII 1/2014 (2014-2017) project, during which several innovative approaches (augmented feedback, virtual therapist, haptic feedback) have been implemented and validated in both laboratory and clinical environments, for the post-stroke recovery of upper afflicted limbs of patients.
At the moment, the project aims to market this technology at affordable prices for the patients, with emphasis on obtaining a minimal cost and customizable solution, in order to help during all stages of recovery, both with upper and lower afflicted limbs.
Thus, this project will develop a neuromotor recovery system (for people who suffered either cerebrovascular stroke, various cerebral trauma, or brain surgery) for both upper and lower limbs, which can be used both in early and advanced recovery stages, at affordable costs, customizable to accommodate a wide range of physical support and automatic feedback, including visual augmented feedback – an innovative concept introduced, implemented, and validated previously.
The TRAVEE project aims to develop a cost effective, affordable IT solution, for the early and advanced recovery of stroke afflicted lower and upper limbs.
During the early stage of recovery, the focus is on assisting the patient to regain muscular control in the afflicted limbs. The recovery contains a specialized pre-set of simple exercises according to patient specific stage and evolution. During this stage, the neuroplasticity is at its peak potential, thus strong and sustained feedback is essential. That is why, the system includes the original idea of visually augmented feedback, previously implemented, and successfully tested (for the upper limbs), during the TRAVEE-PNII 1/2014 project clinical study. The previous project study results show that the use of visual augmented feedback effectively helps the patients surpass the blockage situations which occur specifically during this early stage of recovery.
During this stage of recovery, the system uses Virtual Reality technology (VR) and affordable devices such as: recovery gloves, brain computer interface (BCI), functional electrical stimulation (FES), inertial measurement unit bracelet (IMU).
The patient’s avatar (a virtual 3D model inside a virtual environment) is actualized using the information received from the IMU sensor (for actual movements) or BCI (for anticipatory motor imagery, until the patient succeeds doing the actual movement).
The augmented feedback encompasses the replication of an enhanced movement of the patient’s limb on the avatar, movement which corresponds with the intention and expectations of the patient and not with the actual result. Very low amplitude movements are represented using a gradual amplification factor so that the patient perceives them, and their cortex can process them as being a constant and significant good result. Hence, the causal learning feedback loop is closed, avoiding blockages of recovery, and attributing great positive psychological value to the perception of the obtained virtual results, encouraging the patient to continue training intensively.
The second stage of recovery aims to help the patient regain mobility and precise muscular control, through a series of motivational exercises of handling and interacting with various virtual elements from the virtual/augmented environment. Games which require ample limb movements, but also precise and accurate handling of virtual objects are to be implemented at this stage. For the tracking of limbs or fingers, the RGB flow is sufficient from the patient perspective, the tracking techniques will be based on artificial intelligence as well as real-time tracking of the 3D space of the hand using a monocular RGB camera. The evolution of the patient recovery can be followed remotely by the therapist, who will have the possibility to change the exercises of the patient or adjust their degree of difficulty, through a minimalistic telemedicine solution.
In Phase 2 from 2023 of the project, the following activities have been performed:
- Establishment/creation/adaptation of the EEG signal processing algorithm for determining motor imagery activity based on the selected EEG device/devices
- Creation of the limb tracking algorithm based on sensors
- Creation of the functional electrical stimulation algorithm
- Testing the algorithms for the new hardware
- Development of the patient interface
- Development of the therapist interface
- Development of the telemedicine system
- Refinement based on testing
- Creation/implementation of recovery exercises
- Evaluation of exercises based on testing
- Continuous testing and evaluation by therapists and selected patients: patient interface, therapist interface, exercises
- Implementation of the core system flow
- Implementation of motion augmentation algorithms
- Implementation, optimization, and integration of BCI algorithms
- Optimization and integration of motion tracking algorithms based on IMU sensors
- Implementation of the functional electrical stimulation algorithm
- Creation of the database with defined recovery exercises
- Testing the laboratory prototype
- Implementation of the telemedicine system
- Integration of components into the telemedicine system
- Integration of advanced functions into the prototype and system refinement
- Preparation of patient testing procedures
- Dissemination of results in journals and conferences
In Phase 1 from 2022 of the project, the following activities have been performed:
- A detailed study of the virtual and augmented reality solutions considering the imposed low costs constraints
- A detailed study of BCI hardware equipment
- A detailed study of monocular video tracking solutions
- A detailed study of telemedicine solutions
- A detailed study of physical recovery exercises and their gamification
- A detailed study of robotic glove equipment
- A detailed study of equipment containing IMU sensors, for tracking different limbs
- A detailed study of interactions based on emotional response between the patient and the virtual therapist interface
- Establishing functional and non-functional requirements for the TRAVEE system, considering new hardware features and algorithm capabilities
- Establishing of the high-level architecture
- Defining the recovery exercises
- Dissemination of results through academic journals and conferences