NC

For more information on the research check out the NC Research blog. Link

We live in a world where interacting with machines has become commonplace. Thus, it is only natural that humans engage in dialogue with these technological entities in our current age. People ask questions to gather information or give commands to various devices, including AI speakers, smartphones, and even automobiles. In these scenarios, the specific wake words we employ to summon these machines—such as “Hey Siri,” “Alexa,” and “Hey Google”—are known as “keywords.” Keyword Spotting (KWS), a technology that recognizes these wake words, is one of the core technologies in human-machine interaction. Lately, research has been focusing on the customization of these keywords, which involves formulating distinct keywords tailored to each individual.

This trend underscores users’ inclination to engage with machines in a manner that resonates with their individual preferences. This inclination is anticipated to gain further momentum as we enter the era of digital humans. In response, NC is actively delving into the realm of User-Defined Keyword Spotting (UDKWS) technology, designed to recognize a wide array of personalized keywords used by individuals.

The strategy for developing this technology might appear straightforward at first glance. Could it be as simple as providing a User-Defined Keyword Spotting model for training with the desired keywords of users? However, certain challenges complicate this process. In order to train a model capable of distinguishing between keywords and non-keywords, a substantial amount of training speech data is essential. Yet, acquiring such data is far from being an easy task. Moreover, external non-technical complications can emerge when users personally record additional keywords while modifying the existing keyword list. While customization may offer significant performance improvements, it also brings forth the potential for user inconvenience and privacy concerns. Furthermore, as users typically prioritize swift responses, models embedded within the device hold more appeal than a server-client structure. Consequently, the preferences of users negate the feasibility of deploying a high-performance speech spotting model with continuous background operation, thereby limiting the utilization of resources.

NC has adopted a Zero-Shot Keyword Spotting model to address the diverse range of keywords essential for technologies such as customized digital humans. What sets this model apart from previous studies on customized keywords is its focus on pronunciation*. While preceding technologies mainly grasped the binary distinction of “same or different” for pairs of words (Word 1 vs. Word 2), the model proposed by NC surpasses this by comparing the pronunciation patterns of these two words. This advancement empowers the model to discern even subtle differences in words sharing similar pronunciations. In short, the model is built through training on the congruity between speech and phoneme information from large general verbal output databases, facilitating the assessment of speech-text equivalence. Based on this, PhonMatchNet was proposed, which can determine the verbal output status for arbitrary keywords. By applying this innovation, the seamless incorporation of user-specific keywords becomes viable, obviating the task of amassing and training additional speech data, as demonstrated in the zero-shot KWS approach. This significantly reduces the time and cost required for comprehensive learning of a singular keyword. Significantly, this research also harbors potential applications in assessing pronunciation proficiency within second language education. Given its high practicality, the model is on course to have a patent application submitted by the end of August.

For more information on the research check out the NC Research blog. Link

NC embarked on further research to amplify the responsiveness of these speech spotting models, rendering them more user-friendly. In this pursuit, a platform framework was developed, tailored for training and spotting keywords, and designed to operate seamlessly within a web browser environment. A specific case study was then presented, showcasing how this framework effectively overcame the prior limitations of deep learning models being restricted to server-based execution.
  

When it comes to creating models that exhibit rapid responsiveness within the device, the concept of data lightweighting emerges as a necessity. To achieve this, the initial step involved the integration of a speech embedding model into the architecture of a keyword spotting system. This leveraged pre-existing information about the distinctive attributes of audio signals, enabling the training of the keyword spotting model even when faced with limitations in available training data. Furthermore, recomputation was applied exclusively to areas influenced by newly introduced audio input. The model was also enhanced to a stream-able format, making use of the results from previous frame calculations and effectively removing duplicate computations.
  

However, even with this lightweight keyword model, operating it in the conventional manner that previously required a server-based execution environment posed a challenge. This is because deep learning models and libraries are written in a format that web browsers cannot understand. However, the definition of digital humans set by NC necessitates their ability to run effortlessly on the most versatile platform. Therefore, to address this issue, WebAssembly was utilized to enable real-time inference in web browsers, rendering the deep learning-related code intelligible to currently deployed web browsers. With this environment, deep learning models can be run in web browsers. In short, users can create keyword models and independently operate them through a web browser on any operating system or platform. Also, since the model can be run directly within the web environment without intermediary servers, users can anticipate a markedly accelerated response speed.

The previously mentioned study, titled “PhonMatchNet: Phoneme-Guided Zero-Shot Keyword Spotting for User-Defined Keywords,” and the current study both place a significant emphasis on the concept of a “keyword.” The “Personalized Digital Human” that NC aims to develop refers to digital humans tailored to each individual. Both the aforementioned studies revolve around the idea of creating digital humans capable of swiftly acquiring individual names with minimal training data, while maintaining commendable performance levels at a cost-efficient scale. As a result, digital humans can now be addressed by names given them by users. Moreover, they are primed for interaction in near real-time through platforms that are more accessible and familiar to us, thus providing users with an elevated and refined personalized experience.

In the future, NC envisions the refinement of digital humans for swifter and more efficient operation through diverse optimization techniques. An added advantage lies in the infinite potential for the expansion of digital humans, as they are not limited to languages such as English, Korean, Japanese, Chinese, Spanish, etc. Creating the most robust model against both internal and external noise is also crucial. Even in extreme environments, such as the noise resulting from the integration of video information with text and sound, the introduced model demonstrates its robustness.

For more information on the research check out the NC Research blog. Link

The way humans and machines have communicated until now has been vertical and one-sided, which complicates its classification as genuine interaction. This interaction paradigm has been confined to transactional exchanges where humans pose queries or provide directives, and machines reciprocate with responses. However, the vision at NC encompasses the development of digital humans equipped not only to decode human emotions but also to foster more profound and substantial interactions. This ambitious objective requires the integration of emotion recognition capabilities. Yet, perceiving emotions poses a challenge even for humans, as it entails more than simple voice recognition. It also encompasses understanding various factors such as gestures, expressions, context, and then making comprehensive assessments. In other words, rather than considering modalities like voice and facial expressions individually, research on multimodal technologies* was necessary. Given this foundation, the exploration of multimodal emotion recognition was initiated, focusing on two key modalities: text and speech.

During this research, three challenges were effectively addressed. Initially, the challenge in the realm of multimodal emotion recognition emerged from the scarcity of paired data that encompassed multiple modalities, rather than just a single modality. Moreover, the collection and annotation of such data incur significant costs. To overcome the data scarcity issue, an approach was employed that utilizes pre-trained speech and text Self-Supervised Learning (SSL) models with unlabeled large-scale data.

Next, in multimodal emotion recognition, identifying the “key elements” posed a difficulty when assessing emotions through two modalities, namely speech and text. For instance, in the context of emotion recognition within the sentence “Argh, the stock market plummeted and it’s driving me insane,” the text identifies negative words like “plummeted” and “insane,” while the speech focuses on the sigh at the beginning of the sentence and the frail expression in the intonation. In the context of emulating a human-like cognitive mechanism, it was essential to design a network that intuitively focuses on different aspects of each modality to recognize emotions. To achieve this, a fusion network with a new focus-attention mechanism was designed, which helped identify important parts for emotion recognition in each modality. Finally, metric learning techniques were applied to ensure distinct classification spaces for different emotions and similar classification spaces for related emotions. Through these methods, the design for emotion classification was enhanced, and the accuracy and performance of emotion recognition were improved, as demonstrated (with the highest performance reaching 80 and the research paper performance at 78).

One of the most important goals in the advancement of digital humans is to enable them to engage in emotionally attuned conversations akin to human interactions. This research paper introduces an empathetic interactive feature that detects users’ emotions and responds in a natural and fitting manner. This is achieved by imitating how humans perceive multimodal information. This feature goes beyond the limitations of goal-oriented conversations, enabling a more natural understanding and reflection of human emotions in conversations. Consequently, it empowers digital humans to establish a horizontal rapport with users, effectively evolving into true collaborative partners. The emotion recognition module based on multimodal inputs incorporates both speech and text information. Moving forward, NC aims to expand its research, covering existing fields such as Vision and NLP, to develop digital humans with advanced emotion recognition capabilities.