Faculty Profiles - HASEGAWA, Yasuhisa

Endoscope Automation Framework with Hierarchical Control and Interactive Perception for Multi-Tool Tracking in Minimally Invasive Surgery Open Access

Fozilov K., Colan J., Davila A., Misawa K., Qiu J., Hayashi Y., Mori K., Hasegawa Y.

Sensors Vol. 23 ( 24 ) 2023.12

Language：English Publishing type：Research paper (scientific journal) Publisher：Sensors

In the context of Minimally Invasive Surgery, surgeons mainly rely on visual feedback during medical operations. In common procedures such as tissue resection, the automation of endoscopic control is crucial yet challenging, particularly due to the interactive dynamics of multi-agent operations and the necessity for real-time adaptation. This paper introduces a novel framework that unites a Hierarchical Quadratic Programming controller with an advanced interactive perception module. This integration addresses the need for adaptive visual field control and robust tool tracking in the operating scene, ensuring that surgeons and assistants have optimal viewpoint throughout the surgical task. The proposed framework handles multiple objectives within predefined thresholds, ensuring efficient tracking even amidst changes in operating backgrounds, varying lighting conditions, and partial occlusions. Empirical validations in scenarios involving single, double, and quadruple tool tracking during tissue resection tasks have underscored the system’s robustness and adaptability. The positive feedback from user studies, coupled with the low cognitive and physical strain reported by surgeons and assistants, highlight the system’s potential for real-world application.

DOI： 10.3390/s23249865

Closed-Loop Optimization of Soft Sensor Morphology Using 3D Printing of Electrically Conductive Hydrogel International coauthorship Open Access

Sojiro Sugiura, David Hardman, Thomas George Thuruthel, Yasuhisa Hasegawa, Fumiya Iida

Advanced Intelligent Systems Vol. 5 ( 10 ) 2023.10

A Shared Control Framework for Enhanced Grasping Performance in Teleoperation Reviewed International journal

Zhu Y., Jiang B., Chen Q., Aoyama T., Hasegawa Y.

IEEE Access Vol. 11 page： 69204 - 69215 2023.7

Authorship：Last author Language：English Publishing type：Research paper (scientific journal) Publisher：IEEE Access

Remote teleoperation has shown significant advancements since the first teleoperation system was proposed by Goertz in the 1940s. In recent years, the research on shared control methodologies in which the robot assists the operators in accomplishing the desired tasks has gained extensive attention. One such important task in teleoperation is object grasping. In this paper, we propose a shared control framework to enhance the teleoperated grasping performance. The proposed framework is built upon a virtual reality device-based direct teleoperation system. In this framework, a template matching-based object point cloud compensation is introduced for multi-angle grasping pose generation. Then, the feasible grasping candidates are selected considering joint constraints-aware manipulability. Finally, the grasping assistance is achieved by trajectory blending with dynamic authority adjustment. To validate the performance of the proposed framework, we carried out experimental evaluations. The output results indicate improved grasping performance in terms of reduced task completion time, linear trajectory, and workload.

DOI： 10.1109/ACCESS.2023.3292410

Natural Grasp Intention Recognition Based on Gaze in Human-Robot Interaction International coauthorship International journal

Yang B., Huang J., Chen X., Li X., Hasegawa Y.

IEEE Journal of Biomedical and Health Informatics Vol. 27 ( 4 ) page： 2059 - 2070 2023.4

Language：English Publishing type：Research paper (scientific journal) Publisher：IEEE Journal of Biomedical and Health Informatics

Objective: While neuroscience research has established a link between vision and intention, studies on gaze data features for intention recognition are absent. The majority of existing gaze-based intention recognition approaches are based on deliberate long-term fixation and suffer from insufficient accuracy. In order to address the lack of features and insufficient accuracy in previous studies, the primary objective of this study is to suppress noise from human gaze data and extract useful features for recognizing grasp intention. Methods: We conduct gaze movement evaluation experiments to investigate the characteristics of gaze motion. The target-attracted gaze movement model (TAGMM) is proposed as a quantitative description of gaze movement based on the findings. A Kalman filter (KF) is used to reduce the noise in the gaze data based on TAGMM. We conduct gaze-based natural grasp intention recognition evaluation experiments to collect the subject's gaze data. Four types of features describing gaze point dispersion (f_{var}), gaze point movement (f_{gm}), head movement (f_{hm}), and distance from the gaze points to objects (f_{d_{j}}) are then proposed to recognize the subject's grasp intentions. With the proposed features, we perform intention recognition experiments, employing various classifiers, and the results are compared with different methods. Results: The statistical analysis reveals that the proposed features differ significantly across intentions, offering the possibility of employing these features to recognize grasp intentions. We demonstrated the intention recognition performance utilizing the TAGMM and the proposed features in within-subject and cross-subject experiments. The results indicate that the proposed method can recognize the intention with accuracy improvements of 44.26% (within-subject) and 30.67% (cross-subject) over the fixation-based method. The proposed method also consumes less time (34.87 ms) to recognize the intention than the fixation-based method (about 1 s). Conclusion: This work introduces a novel TAGMM for modeling gaze movement and a variety of practical features for recognizing grasp intentions. Experiments confirm the effectiveness of our approach. Significance: The proposed TAGMM is capable of modeling gaze movements and can be utilized to process gaze data, and the proposed features can reveal the user's intentions. These results contribute to the development of gaze-based human-robot interaction.

DOI： 10.1109/JBHI.2023.3238406

Micro-manipulation Interface that Presents Surface Force Tactile Sensation According to the Cell Deformation Reviewed

Saito Sumiwa, Aoyama Tadayoshi, Funabora Yuki, Takeuchi Masaru, Hasegawa Yasuhisa

Journal of the Robotics Society of Japan Vol. 41 ( 3 ) page： 322 - 325 2023.4

Authorship：Last author Language：Japanese Publishing type：Research paper (scientific journal) Publisher：The Robotics Society of Japan

<p>The microinjection technique is widely applied in fields such as biology, medicine, and intracytoplasmic sperm injection (ICSI). Microinjection process requires special skill and, thus, generally ICSI is conducted by embryologists. However, the shortage of embryologists has become a problem in clinics, and there is a need for an easy injection system that does not rely on embryologists. In this study, we propose a micromanipulation interface that assists to perceive cell extension for easy injection system. The proposed interface provides 2-dimensional force/tactile feedback through a fabric actuator with air pressure based on real-time visual sensing of cell deformation. The effectiveness of the proposed interface is verified through puncturing experiments on perceiving cell extension perfumed by inexperienced subjects. </p>

DOI： 10.7210/jrsj.41.322

CiNii Research

A Concurrent Framework for Constrained Inverse Kinematics of Minimally Invasive Surgical Robots Open Access

Jacinto Colan, Ana Davila, Khusniddin Fozilov, Yasuhisa Hasegawa

Sensors Vol. 23 ( 6 ) page： 3328 2023.3

Language：English Publishing type：Research paper (scientific journal)

Minimally invasive surgery has undergone significant advancements in recent years, transforming various surgical procedures by minimizing patient trauma, postoperative pain, and recovery time. However, the use of robotic systems in minimally invasive surgery introduces significant challenges related to the control of the robot’s motion and the accuracy of its movements. In particular, the inverse kinematics (IK) problem is critical for robot-assisted minimally invasive surgery (RMIS), where satisfying the remote center of motion (RCM) constraint is essential to prevent tissue damage at the incision point. Several IK strategies have been proposed for RMIS, including classical inverse Jacobian IK and optimization-based approaches. However, these methods have limitations and perform differently depending on the kinematic configuration. To address these challenges, we propose a novel concurrent IK framework that combines the strengths of both approaches and explicitly incorporates RCM constraints and joint limits into the optimization process. In this paper, we present the design and implementation of concurrent inverse kinematics solvers, as well as experimental validation in both simulation and real-world scenarios. Concurrent IK solvers outperform single-method solvers, achieving a 100% solve rate and reducing the IK solving time by up to 85% for an endoscope positioning task and 37% for a tool pose control task. In particular, the combination of an iterative inverse Jacobian method with a hierarchical quadratic programming method showed the highest average solve rate and lowest computation time in real-world experiments. Our results demonstrate that concurrent IK solving provides a novel and effective solution to the constrained IK problem in RMIS applications.

DOI： 10.3390/s23063328

Toward Autonomous Robotic Minimally Invasive Surgery: A Hybrid Framework Combining Task-Motion Planning and Dynamic Behavior Trees International journal Open Access

Khusniddin Fozilov, Jacinto Colan, Kosuke Sekiyama, Yasuhisa Hasegawa

IEEE Access Vol. 11 page： 91206 - 91224 2023

Language：English Publishing type：Research paper (scientific journal)

The growing need for high levels of autonomy in Autonomous Robotic Surgery Systems (ARSS) calls for innovative approaches to reduce surgeons' cognitive load, optimize hospital workflows, and ensure efficient task-level reasoning and adaptation during execution. This paper presents a novel hybrid framework that synergistically combines Task-Motion Planning and Dynamic Behavior Trees for ARSS in Minimally Invasive Surgery. Our approach is designed to address the challenges of coordinating multiple surgical tools within a small workspace, thereby making complex surgical tasks like multi-throw suturing feasible and efficient. Through an extensive evaluation in simulation across diverse initial conditions and noise scenarios, the proposed method demonstrates improved success rates, reduced execution times, and fewer regrasps compared to standalone approaches. Furthermore, it showcases robustness under increased noise conditions. By applying our framework to a complex multi-throw suturing task, we illustrate its capability to seamlessly handle comprehensive suturing tasks, including needle picking, insertion, extraction, and the handover of the needle between Patient Side Manipulators. The results suggest that our hybrid approach not only enhances ARSS autonomy but also adapts effectively to unexpected environmental changes, laying the groundwork for its potential applicability in real-world surgical robotics.

DOI： 10.1109/ACCESS.2023.3308619

OpenRST: An Open Platform for Customizable 3D Printed Cable-Driven Robotic Surgical Tools Open Access

Jacinto Colan, Ana Davila, Yaonan Zhu, Tadayoshi Aoyama, Yasuhisa Hasegawa

IEEE Access Vol. 11 page： 6092 - 6105 2023

Language：English Publishing type：Research paper (scientific journal)

Robot-assisted minimally invasive surgery (RMIS) has been shown to be effective in improving surgeon capabilities, providing magnified 3D vision, highly dexterous surgical tools, and intuitive human-robot interfaces for high-precision tool motion control. Robotic surgical tools (RST) are a critical component that defines the performance of an RMIS system. Current RSTs still represent a high cost, with few commercially available options, which limits general access and research on RMIS. We aim to take advantage of recent progress in biocompatible 3D printing and contribute to the development of RMIS technologies, presenting an open platform for low-cost, biocompatible, and customizable RSTs. The proposed design concept consists of a 3-DOF end-effector with a decoupled wrist mechanism, a tool interface module, and a tool drive unit. We validated our end-effector design using Finite Element Analysis (FEA) to confirm that stress generated by high grip forces is maintained below the material yield stress. Validation experiments showed that the proposed RST could provide up to 10N grip forces and up to 3N pulling forces. The proposed control framework exhibited a mean absolute positioning tracking error of approximately 0.1 rad. Finally, we also demonstrated the use of the proposed RST in two surgical training tasks: pick-and-place and stitching. The designs and software control framework are open-access and freely available for customization and fast development at https://github.com/jcolan/OpenRST.

DOI： 10.1109/ACCESS.2023.3236821

Micromanipulation System Capable of Simultaneously Presenting High-Resolution and Large Field-of-View Images in Real-Time Open Access

Tadayoshi Aoyama, Sarau Takeno, Kenta Yokoe, Kazuki Hano, Masaki Takasu, Masaru Takeuchi, Yasuhisa Hasegawa

IEEE Access Vol. 11 page： 34274 - 34285 2023

Improvement in the Manipulability of Remote Touch Screens Based on Peri-Personal Space Transfer International journal Open Access

Kento Yamamoto, Yaonan Zhu, Tadayoshi Aoyama, Masaru Takeuchi, Yasuhisa Hasegawa

IEEE Access Vol. 11 page： 1 - 1 2023

Gaze and Environmental Context-Guided Deep Neural Network and Sequential Decision Fusion for Grasp Intention Recognition International coauthorship International journal Open Access

Bo Yang, Xinxing Chen, Xiling Xiao, Pei Yan, Yasuhisa Hasegawa, Jian Huang

IEEE Transactions on Neural Systems and Rehabilitation Engineering Vol. 31 page： 3687 - 3698 2023

Auditory Feedback for Enhanced Sense of Agency in Shared Control Reviewed

Tomoya Morita, Yaonan Zhu, Tadayoshi Aoyama, Masaru Takeuchi, Kento Yamamoto, Yasuhisa Hasegawa

Sensors Vol. 22 ( 24 ) page： 9779 2022.12

An immersive micro-manipulation system using real-time 3D imaging microscope and 3D operation interface for high-speed and accurate micro-manipulation Reviewed International journal

Kenta Yokoe, Tadayoshi Aoyama, Toshiki Fujishiro, Masaru Takeuchi, Yasuhisa Hasegawa

ROBOMECH Journal Vol. 9 ( 1 ) page： 16 2022.12

Auditory Feedback for Enhanced Sense of Agency in Shared Control Reviewed International journal

Tomoya Morita, Yaonan Zhu, Tadayoshi Aoyama, Masaru Takeuchi, Kento Yamamoto, Yasuhisa Hasegawa

Sensors Vol. 22 ( 24 ) page： 9779 - 9779 2022.12

Authorship：Last author Language：English Publishing type：Research paper (scientific journal) Publisher：MDPI AG

There is a growing need for robots that can be remotely controlled to perform tasks of one’s own choice. However, the SoA (Sense of Agency: the sense of recognizing that the motion of an observed object is caused by oneself) is reduced because the subject of the robot motion is identified as external due to shared control. To address this issue, we aimed to suppress the decline in SoA by presenting auditory feedback that aims to blur the distinction between self and others. We performed the tracking task in a virtual environment under four different auditory feedback conditions, with varying levels of automation to manipulate the virtual robot gripper. Experimental results showed that the proposed auditory feedback suppressed the decrease in the SoA at a medium level of automation. It is suggested that our proposed auditory feedback could blur the distinction between self and others, and that the operator attributes the subject of the motion of the manipulated object to himself.

DOI： 10.3390/s22249779