Description

Project Characterization Document: A Robot-Based Cognitive Support System for a Smart Home
 
This project focuses on the development of an interactive smart home system, which combines a TEMI (Android-based robotic communication) robot with a local large language model (LLM), to assist in cognitive tests for patients and provide real-time responses. The goal of the system is to identify various scenarios through a camera, understand the situation, and take appropriate actions by generating voice responses using the robot.

Background
At the Institute's Smart Home Laboratory, many studies are conducted to test patients' ability to perform independent daily tasks (ADL), especially among older populations or those with disabilities. The COFFE test is a common protocol for assessing patients' cooking and preparation skills. During the tests, it is important to provide guidance and support to patients if necessary, especially in cases of cognitive difficulty or potential risks.

System Requirements
1.	Real-time scenario identification: The system should be able to analyze video from a camera installed in the test environment and in the robot, and identify various actions and situations, such as a patient who is curated or intends to use a metal tool in the microwave.
2.	Integration with LLM: Using a local LLM model (e.g.LLAMA-VISION) to interpret the scenarios presented on the camera and make decisions based on predefined scenarios and responses.
3.	Voice Response Usinga TEMI Robot: The robot will provide appropriate voice responses to the patient, such as instructions, warnings, or encouragement, based on the decisions made by the LLM.
4.	Personalization: The ability to adjust the level of difficulty and response issues according to the patient's condition.

Features and mechanisms
•	Object and Movement Recognition: An LLM model will analyze the video in real-time, identify objects such as kitchen utensils or the patient himself, and track his movements.
•	Decision Engine: The system will include an algorithm that links the predefined scenarios with the data estimated by the LLM, and makes an appropriate decision for the response.
•	Speech synthesis: The TEMI robot will convert LLM's textual answers into speech using speech synthesis technology, to provide clear and accessible voice responses.
•	User-friendly interface: Intuitive UI design for researchers to manage the system, define new scenarios, and track performance.
Advantages
•	Improve users' experience on cognitive tests by providing real-time support and guidance.
•	Customizing the tests to different cognitive levels, allowing for a more accurate assessment of patients' abilities.
•	Potential for expansion to a variety of household tasks, helping to maintain the independence of adults or people with disabilities.
•	Innovative research in the field of human-robot interaction in home environments.
Challenges
•	Fine-tuning of an LLM model to identify complex scenarios from the available data.
•	Seamless integration between the TEMI robot and the LLM system, ensuring real-time response.
•	Maintaining the privacy and safety of patients while interacting with the robot.

Conclusion and Next Steps
This project offers an innovative approach to integrating artificial intelligence and robotics in smart home environments, with the goal of improving cognitive tests and providing personalized support to patients. Next steps will include designing the system architecture, training LLM models, developing a user interface, and comprehensive testing in real environments.

Knowledge Requirement for Project Participants
In order to carry out the described project, several basic areas of knowledge and skills are required:
•	Artificial Intelligence and Machine Learning: Understanding machine learning techniques, including training models for object recognition, natural language processing (NLP), and decision-making. Knowledge of models such asLLM (e.g., LLAMA) and techniques such as Vision Transformer (ViT) will be an advantage.
•	Image and video processing: Skill in image and video processing, including object recognition, motion tracking, and scene understanding. Knowledge of popular image processing libraries like OpenCV can be useful.
•	Software development: Skills in software development, including an appropriate programming language (such asPython).
•	Operating the robot: Knowledge of the Android operating system.
•	Communication and Collaboration: Ability to work in a multidisciplinary team, communicate technical ideas, and collaborate with experts in various fields.
The optimal team for this project will include a combination of experts in the fields of artificial intelligence, image processing, programming in the Android environment, and clinical research. A collaboration between computer scientistsand researchers in the field of geriatricswill ensure a holistic approach to the development and execution of the project.
A Robot-Based Cognitive Support System for a Smart Home

ROBOTICS AND AI

Description

Emphasis in project execution

The project is has cooperation with the industry and combines meeting deadlines while being creative and focused on the task

THE PROJECT IS AT FULL CAPACITY
