Master's Thesis
Virtual Group Conversations for Adults with Sudden Speech Impairment
Summary
Sudden speech impairment can mentally and physically tax individuals, negatively affecting their quality of life. The recent introduction of artificial intelligence used within applications for speech and speech therapy practices presents a rich opportunity to encourage and improve virtual social group conversations for senior adults with sudden speech impairment.
Drawing from communication and learning theories and guided by Multimedia Learning Theory Principles, this investigation explores how a personalized AI interface could seamlessly support and facilitate inclusive virtual social group conversations for older adults with sudden speech impairment. The resulting studies examine how personalized gestural, verbal, and auditory cues can combat the heightened and complex challenges speech-impaired senior adults face as speakers and listeners in virtual video group conversations. These studies focus on situations where speech-impaired adults converse with non speech-impaired family and friends through design explorations of turn-taking dynamics, response feedback, adjusting airtime, and active listening.
Overview
How can personalized visual, auditory and gestural cues during a virtual group conversation encourage active listening by non speech impaired participants and encourage participation by speech impaired participants to create an inclusive and supportive environment for speech impaired users?
Research Question
Programs
After Effects, Adobe Illustrator, Adobe InDesign, Adobe Photoshop, Figma, Miro
Virtual Group Conversations for Adults with Sudden Speech Impairment
This scenario video explores how personalized visual, auditory and gestural cues during a virtual group conversation encourage active listening by non-speech-impaired participants to create a supportive and inclusive environment for speech-impaired users. I worked to create an inclusive and not overly restrictive experience for speech-impaired and non-speech-impaired users.
The following video is a brief look at a conversation between Jim and his friends, 6 months after Jim’s stroke.
Conceptual Framework
The conceptual framework that provides the groundwork for this study is based on the Communication Accommodation Theory (CAT) and Social Learning Theory (SLT). The conversation dynamics between two individuals are shown. However, for the purpose of this study, this framework was extended to incorporate multiple individuals in a group scenario. The framework also demonstrates how individuals accommodate others during conversation, while also learning from and modeling the behavior of those they are talking to.
Precedents
Current precedents include speech therapy applications, AI driven voice recognition applications, sonar tracking through smart watches or smart phones and applications to improve public speaking. However, there are currently no applications or programs that are focused on assisting in social group conversations either in person or virtually.
Persona 1: Pat Smith
Sudden speech impairment caused by stroke can be very different for different people. So I created two personas to further explore and expand my studies.
My first persona is Pat, a grandma who is nervous around technology but very much wants to continue interacting with her family especially since she is not very mobile after her stroke. Since her stroke, Pat has weekly virtual conversations with her family members. Pat is my persona for study 1 and 2 with the conversations for these studies occurring 4 months after Pat’s stroke.
Persona 2: Jim Robinson
My second persona is Jim. Having been a computer programmer before his stroke, Jim is excited to work with new technology. He was very social before his stroke and is extremely frustrated at his decreased ability to socialize. To help Jim out, a few of Jim’s friends agree to virtually meet weekly. Jim is my persona for study 3 and 4 with the conversations for these studies occurring 6 months after Jim’s stroke.
Visualization of a Year
This visualization depicts how it would look if the AI assistant was used in the weekly group video conversations for about a year. The visualization highlights main events such as when the stroke occurs and when the study takes place. Human and AI interactions and the speech-impaired user’s speech abilities are also traced throughout the year.
Study 1: Personalized Gestural Cues to Alternate Turn-Taking
My first study explores how personalized gestural cues can affect turn-taking dynamics.
The important events in turn-taking during a conversation are how to add to or enter the conversation, clarity of who goes next once someone ends a turn and, specifically for Pat, gaining clarity and context if confused during a turn. I use gestures commonly used by English speakers in-person conversations as well as gestures currently used by individuals with aphasia
Study 2: Personalized Visual Cues to Improve Response Feedback
My second study explores how personalized visual cues can affect response feedback.
To respond successfully, Pat needs to understand what is being talked about, especially if the context has changed. She also needs keywords of the current speaker to understand what they are saying. Finally, Pat needs writing and drawing options to help her communicate/respond.
Study 3: Personalized Auditory Cues to Adjust Airtime
My third study explores how auditory cues and regulated speaking speed can adjust airtime of non speech impaired users to give speech impaired participants time to understand and add to the conversation.
In order to be successful, Jim needs more time between speakers to understand and enter the conversation and more time to communicate
Future Studies
I did a few explorations of how this interface might translate to a phone within my final scenario. However, further studies and testing are needed to gauge the effectiveness of these designs. These studies would also benefit from design explorations of the video conferencing AI-assisted interface for other standard communication devices, such as smartphones and tablets.
