Language：English   Publishing type：Research paper (international conference proceedings)   Publisher：IEEE  

Dialogue robots powered by large language models can generate advanced utterances. However, the interaction performance is not yet perfect, and the robots sometimes cause problems during interactions. In this study, to detect anomalies in human-robot interactions, we created a dataset and constructed anomaly detection models. For the dataset creation, we collected videos of human-robot interactions in a framework where humans intervene when a dialogue breakdown occurs and labeled the scenes where humans intervened as anomalies. Using this dataset, we built classification models and deep metric learning models utilizing encoders for video, audio, and multimodal information. The results showed that we could successfully train the models and achieve an accuracy and F1-score of over 80%. The performance of the deep metric learning models surpassed that of the classification models, thus demonstrating the importance of separating the differences between classes. We also clarified the importance of audio information.

DOI： 10.1109/ro-man60168.2024.10731184

researchmap

Authorship：Corresponding author   Language：English   Publishing type：Research paper (scientific journal)   Publisher：Springer Science and Business Media LLC  

The demand for therapeutic robots to alleviate mental health problems is growing. Studies have shown that people’s mental health improves when they are touched. Consequently, therapeutic robots are designed to alleviate stress through robot’s autonomous touch. However, robot’s autonomous touch can sometimes cause discomfort to recipients. This paper proposes a semi-autonomous touch method that merges robot’s autonomous touch with user-operated touch to mitigate discomfort while maintaining the sensation of being touched by another person. We conducted an experiment involving 24 participants who were touched on the neck by robots under three conditions: robot’s autonomous touch, user-operated touch, and the proposed semi-autonomous touch method condition. Additionally, the study investigated participants’ impressions of the robot in each condition. The results showed that semi-autonomous touch condition mitigated discomfort more effectively compared with the robot’s autonomous touch method condition. It also enhanced the feeling of being touched by another person entity and suppressed interaction boredom compared with the user-operated touch method condition. Participants reported higher trustworthiness and perceived friendliness in robots utilizing the semi-autonomous touch method compared to those with autonomous touch method condition. These findings indicate that robots featuring the proposed semi-autonomous touch method can provide a comforting experience, leveraging the therapeutic benefits of being touched by another person, and underscore their potential in mental health applications.

DOI： 10.1038/s41598-024-67964-0

researchmap

Authorship：Lead author   Language：English   Publishing type：Research paper (international conference proceedings)   Publisher：IEEE  

Interactive systems that enable people to easily access mental health care in their daily lives are being studied. Self-instruction training, which is a kind of cognitive-behavioral therapy, is expected to be effective for general-purpose mental health care, but the difficulty for users to create self-instructional texts by themselves has hindered its widespread use. Furthermore, it is not sufficient to simply use a computer-assisted method for the training in which the computer automatically generates the self-instructional texts. This is because previous studies have shown that users do not feel self-instructional texts that are not created by themselves are their thoughts, and consequently cannot internalize the content of the texts, reducing the effectiveness of the training. In this study, we focused on the interface design that facilitates users to internalize automatically generated self-instructional texts, toward the realization of a self-instructional training support system. We proposed an interface that shows the real-time captured video of a user's face like a mirror and superimposes a speech bubble containing a self-instructional text to be connected to the user's face and confirmed the interface facilitates the user's internalization of the text through online experiments. The contribution of this paper is to provide a novel interface design for computer-assisted self-instructional training.

DOI： 10.1109/compsac61105.2024.00334

researchmap

Other Link： https://hdl.handle.net/2237/0002011511

Authorship：Lead author   Language：English   Publishing type：Research paper (international conference proceedings)   Publisher：ACM  

While research and applications of operated social robots that can be used as operators’ avatars have been conducted, a para-operated social robot has also been proposed, in which the robot and its operator exist in the same space, and the operator, robot, and interlocutor can interact with each other. From the findings of human-human-robot interaction research, it is known that the robot’s behavior influences the interpersonal relationships of the people involved in the dialog. However, there is no knowledge of how the operator is perceived by the interlocutor when the operations by the operator are unveiled. This paper reports preliminary results from two online surveys conducted to investigate the following question considering the robot-operator distance; Does the impression of the operator improve in unveiled para-operation when the operator operates the robot and has it say favorable utterances? The results of the two video surveys conducted at different distances between the robot and the operator show that the impression of the operator was improved by the robot’s favorable humorous utterances when the distance was close, but not when the distance was further apart. We consider that the findings contribute to the potential application of unveiled para-operated robots and suggest the importance of considering the distance factor in the influence of robots on human-human relationships.

DOI： 10.1145/3623809.3623924

researchmap

Language：English   Publishing type：Research paper (international conference proceedings)   Publisher：ACM  

In recent years, a framework of parallel conversations has been proposed to facilitate efficient conversations through cooperation between humans and dialogue systems. This approach aims to enable simultaneous conversations with multiple users by enabling the system to handle basic conversation and human operators to intervene when problems arise in the system’s conversation. Previous studies on parallel conversations have primarily focused on delegating simple exchanges such as greetings and acknowledgments to the system, with humans taking over for more complex interactions like providing guidance. Recent advancements in large language models may change this situation, enabling dialogue systems to engage in more advanced interactions. In this study, to examine which interventions will be made when large language models are utilized, we placed six dialogue robots based on large language models in an actual facility and conducted a field experiment involving parallel conversations for about a month. Our analysis of the collected data on dialogues and interventions showed that the most frequent interventions were made for supporting interactions when the system failed to react to the user utterances, indicating the limitations of using large language models alone and clarifying our next steps for facilitating smoother parallel conversations.

DOI： 10.1145/3623809.3623863

researchmap

Language：English   Publishing type：Research paper (international conference proceedings)   Publisher：ACM  

Dialogue systems are widely utilized in chatbots and call centers. However, it is often difficult for such systems to deliver fully autonomous dialogue. For users to have a better dialogue experience, a framework for human-system collaborative dialogue is proposed in which a human operator takes over the dialogue when needed, engaging in conversation with the user instead of the system (we call this process intervention). Operators join the dialogue in the middle; therefore, it is believed that dialogue summarization can be helpful for interventions. However, it is currently unclear whether dialogue summarization is actually useful. Therefore, in this study, we aim to investigate the usefulness of dialogue summaries for interventions through a field experiment conducted at an actual facility combining an aquarium and a zoo. The results of the field experiment revealed that dialogue summaries were more useful for intervention than dialogue history. Furthermore, we found no differences in the word categories included in the operator utterances during interventions irrespective of whether the dialogue history or dialogue format summary was presented to the operators, suggesting that dialogue format summary has content similar to that of dialogue history but improves the usefulness in intervention.

DOI： 10.1145/3623809.3623889

researchmap

Authorship：Lead author,　Corresponding author   Language：English   Publishing type：Research paper (scientific journal)   Publisher：Nature Portfolio  

Studies on social robots that can communicate with humans are increasingly important. In particular, semi-autonomous robots have shown potential for practical applications in which robot autonomy and human teleoperation are jointly used to accomplish difficult tasks. However, it is unknown how the attitude represented in the autonomous behavior of the robots affects teleoperators. Previous studies reported that when humans play a particular role, their attitudes align with that role. The teleoperators of semi-autonomous robots also play the role given to the robots and may assimilate their autonomous expression. We hypothesized that the attitude of teleoperators may align with that of robots through teleoperation. To verify this, we conducted an experiment with conditions under which a participant operated a part of the body of an android robot that autonomously expressed a preferential attitude toward a painting and a condition under which they did not. Experimental results demonstrated that the preferential attitude of participants who teleoperated the android aligned statistically significantly more with that of the robot in comparison to those who did not teleoperate it, thereby supporting our hypothesis. This finding is novel regarding attitude change in teleoperators of semi-autonomous robots and can support the implementation of effective human-robot collaboration systems.

DOI： 10.1038/s41598-022-13829-3

researchmap

Authorship：Lead author,　Corresponding author   Language：Japanese   Publishing type：Research paper (scientific journal)   Publisher：Human Interface Society  

While the potential of adopting robots interacting with humans in stores was shown in the previous studies, however robots that can work with people in stores and provide services have not taken root in society yet. In this paper, we realized an android robot which can help the duties of the salespersons at a boutique as well as provide mental support by cooperating with them. We considered that it is important for such robots to not only serve the customers but also support the salespersons. Therefore, we first interviewed salespersons at a boutique to find out what they expect from the robot. The result of the survey showed that they mainly expect three points from the robot: 1) reducing the workload of salesperson, 2) providing mental support for salesperson, and 3) assisting salesperson to establish a good relationship with the customer. To deal with these issues, we implemented an onsite-operated android. Through an 11-day field experiment in a boutique, in which the salespersons used the android, it was observed that the android could contribute to the salespersons in some situation of the work, and more possibilities for the android to contribute to the salespersons in some different aspects was investigated.

DOI： 10.11184/his.22.3_275

researchmap

Authorship：Lead author,　Corresponding author   Language：Japanese   Publishing type：Research paper (scientific journal)   Publisher：The Japanese Society for Artificial Intelligence  

In this study, we developed and evaluated a dialogue system which enables an android robot to have a chat with users on Niconico Live provided by Dwango Co., Ltd. which is a live streaming service. In Niconico Live, broadcasters can talk to users who write comments displayed on the video stream. Therefore, by using Niconico Live chat, we eliminated speech recognition errors which can occur during speech conversations. In addition, the dialogue system can keep consistency of conversation by selecting the comment to which it can correctly respond because many comments are shown simultaneously on the video stream. The dialogue system was designed as a retrieval-based one which finds the appropriate response to user's utterance from a dialogue corpus. Therefore, we collected a dialogue corpus containing 4,460 pairs of comments and robot responses by teleoperating the android robot talking with users, as a first step. In the next step, we completed the dialogue system on Niconico Live integrating the dialogue corpus into it. To evaluate the performance of the dialogue system, we recorded the conversation between the android and users while running the designed system. After that, we showed the recorded conversation to evaluators and asked them how they feel about the naturalness and consistency of the conversation. Results of the experiment indicate that Niconico Live users perceived the responses of the dialogue system to be natural and found the chat with the android entertaining. Through this study, we demonstrated the applicability of the dialogue system on Niconico Live. However, it is difficult to discuss its effectiveness when applying it to other situations or other communication media such as a humanoid robot or a virtual agent. Therefore, as a future work, conducting a comparative experiment might lead to better understanding of the effectiveness of the dialogue system for androids.

DOI： 10.1527/tjsai.DSH-G

researchmap

Event date： 2024.3

Language：Japanese   Presentation type：Poster presentation  

Venue：Shizuoka University   Country：Japan  

researchmap

Event date： 2024.3

Language：Japanese   Presentation type：Oral presentation (general)  

Venue：Shizuoka University   Country：Japan  

researchmap

Event date： 2023.9

Language：Japanese   Presentation type：Poster presentation  

Venue：Future University Hakodate   Country：Japan  

researchmap

Event date： 2023.8

Language：Japanese   Presentation type：Oral presentation (general)  

Venue：Toyohashi University of Technology   Country：Japan  

researchmap

Event date： 2023.6

Language：Japanese   Presentation type：Oral presentation (general)  

Venue：KUMAMOTO-JO HALL + Online   Country：Japan  

An autonomous dialogue agent is expected to engage in live chat like Virtual YouTubers or to give classes and lectures in application of one-to-many dialogue where the agent interacts with multiple interlocutors forming a community. In one-to-one dialogue, the effectiveness has been shown that an agent interacts with its interlocutor adapting to their characteristics. However, in one-to-many dialogue with community members, no method has been established to realize an agent with the dialogue ability adapted to the characteristics of the community. In this study, we proposed a two-step method to build such an agent: (1) collecting dialogue examples from chat in which community members operate an agent to interact with the other members, (2) developing the agent by using these examples. We conducted experiments to verify this method with our developed system enabling dialogue examples collection and agent implementation. The results suggested that it is possible to collect dialogue examples that reflect the characteristics of a community and to realize an agent capable of dialogue adapted to the characteristics.

researchmap

Other Link： https://www.jstage.jst.go.jp/article/pjsai/JSAI2023/0/JSAI2023_2K4GS903/_article/-char/ja

Event date： 2023.6

Language：Japanese   Presentation type：Oral presentation (general)  

Venue：KUMAMOTO-JO HALL + Online   Country：Japan  

A semi-autonomous robot is becoming important because it can accomplish practical tasks through collaboration between the robot’s autonomy and human operation. However, it is still unclear how a semi-autonomous robot is recognized by its operator who collaborates with it, and understanding this point is critical for application of the robot.For example, the attribution of responsibility for task failure or success may be affected depending on whether an operator treats the robot as a sidekick or a mere tool.In this study, we developed a psychological scale to measure whether an operator feels a sense of operation (tool-likeness) or direction (human-likeness) when they command it. Then, we conducted the experiment to investigate the hypothesis that the degrees of abstractness of the command affect the operator's recognition of a semi-autonomous robot. The results showed that high and low abstractness tended to evoke a sense of direction and operation, respectively.

researchmap

Other Link： https://www.jstage.jst.go.jp/article/pjsai/JSAI2023/0/JSAI2023_3I5OS4c04/_article/-char/ja

Event date： 2019.1

Language：English   Presentation type：Poster presentation  

Venue：Osaka   Country：Japan  

researchmap

Event date： 2017.5

Language：Japanese   Presentation type：Oral presentation (general)  

Country：Japan  

researchmap

Other Link： https://www.jstage.jst.go.jp/article/pjsai/JSAI2017/0/JSAI2017_2N23/_article/-char/ja/