Description

Real-Time Voice and Speech-Based Client Profiling

General Description:
Students will build a system that listens to real conversations between clients and agents, and captures what’s going on emotionally and behaviorally beneath the surface.

It looks at tone, tempo, speaking patterns, and word choice — all in real time.
The system also learns over time, using past interactions with the same client to build a fuller behavioral profile. This means it can go from detecting how someone feels right now, to understanding what kind of person they are, what affects them, and how they like to communicate. It's a step toward truly adaptive, emotionally intelligent AI that can personalize every interaction, as it happens and in the long run. 

Why It Matters:
In customer service, one size doesn't fit all. A stressed client doesn’t need the same response as a cheerful one, and someone who likes to talk things through needs a different approach than someone who prefers quick answers. 
By helping systems adapt to individual clients in real time and over time, this project brings us closer to emotionally aware, human-centered AI.

Project Scope:
Students will develop a pipeline that processes real customer call recordings, extracting both voice-based features (e.g., pitch, tone, rhythm) and speech-based signals (e.g., choice of words, emotional language, engagement cues). The system will be used to:
● Detect the client’s emotional state in real time
● Track emotional dynamics during the call—how emotions shift and how they respond to the agent’s behavior
● Generate a voice & speech-based profile of the client’s beh
avior
● Identify the psychological and communication style (e.g., do they prefer to listen or talk? are they direct or indirect?)
● Surface emotional and behavioral triggers—what makes the client feel frustrated, engaged, calm, etc.
● Enrich the profile with persistent attributes gathered across multiple calls—combining real-time signals with insights from past interactions
● Lay the foundation for personalized agent responses based on client type and mood
The final result will be a working prototype capable of analyzing live calls and gradually building a personalized client profile to support better service, empathy, and communication.

Student Requirements:
● Interest in behavioral AI and voice/speech technologies
● Desire to learn from industry mentors and gain experience working on real-world tech
● Active participation in weekly check-ins
● Enthusiasm for building tech that supports human development
● Ability to conduct independent research
● Personal responsibility for assigned tasks and deadlines
● Good teamwork and collaboration skills
● Interest in psychology is a plus
Development Tools
Python, Jupiter Notebook, Tensorflow, Git Librosa, OpenSMILE (for audio analysis) NLTK or spaCy (for speech content analysis)

Deliverables:
By the end of the project, students are expected to produce:
● A working prototype that can:
○ Analyze a single call in real time
○ Extract emotional and behavioral signals from voice and speech
○ Track emotional shifts during the conversation
○ Store and continuously update client profiles across multiple calls
● A defined set of client attributes, such as psychotype, communication style, and emotional triggers — with logic for how each is identified
● Technical documentation, including:
○ System architecture and data pipeline
○ Profiling methodology and attribute extraction logic
○ Recommendations for future improvements
● Demo presentation showcasing use cases and system adaptability to different client types
● GitHub repository with all working code and supporting materials
Project Coordinators
Veronika Shpareva veronika@voxspec.com Josh (Yegor) Zaslavskiy yz@voxonics.com
Additional Notes
• Each project team may include up to 4 students. • Please include your contact details at the end of the document for student inquiries: - Full name - Email address.


One Pager – Project Summary:
One-page overview with problem, solution, team members, and mentor.

GitHub Link – Source Code:
Clean, organized code with clear comments.

YouTube Link – Project Video:
Screen recording explaining the code, app, and site. Upload and share the link in a separate document.

Specification Document (English):
Detailed, professional spec in English. Important for grading.

PowerPoint Presentation:
Includes team info, requirements, screenshots, graphs, and mockups.


For any questions, you may also contact: mayyalih@gmail.com
Real-Time Voice and Speech-Based Client Profiling

AI and Machine Learning

Description

Emphasis in project execution

The project is has cooperation with the industry and combines meeting deadlines while being creative and focused on the task
