研究者詳細 - 長谷川　泰久

Endoscope Automation Framework with Hierarchical Control and Interactive Perception for Multi-Tool Tracking in Minimally Invasive Surgery Open Access

Fozilov K., Colan J., Davila A., Misawa K., Qiu J., Hayashi Y., Mori K., Hasegawa Y.

Sensors 23 巻 ( 24 ) 2023年12月

記述言語：英語掲載種別：研究論文（学術雑誌）出版者・発行元：Sensors

In the context of Minimally Invasive Surgery, surgeons mainly rely on visual feedback during medical operations. In common procedures such as tissue resection, the automation of endoscopic control is crucial yet challenging, particularly due to the interactive dynamics of multi-agent operations and the necessity for real-time adaptation. This paper introduces a novel framework that unites a Hierarchical Quadratic Programming controller with an advanced interactive perception module. This integration addresses the need for adaptive visual field control and robust tool tracking in the operating scene, ensuring that surgeons and assistants have optimal viewpoint throughout the surgical task. The proposed framework handles multiple objectives within predefined thresholds, ensuring efficient tracking even amidst changes in operating backgrounds, varying lighting conditions, and partial occlusions. Empirical validations in scenarios involving single, double, and quadruple tool tracking during tissue resection tasks have underscored the system’s robustness and adaptability. The positive feedback from user studies, coupled with the low cognitive and physical strain reported by surgeons and assistants, highlight the system’s potential for real-world application.

DOI： 10.3390/s23249865

Closed-Loop Optimization of Soft Sensor Morphology Using 3D Printing of Electrically Conductive Hydrogel 国際共著 Open Access

Sugiura S., Hardman D., George Thuruthel T., Hasegawa Y., Iida F.

Advanced Intelligent Systems 5 巻 ( 10 ) 2023年10月

記述言語：英語掲載種別：研究論文（学術雑誌）出版者・発行元：Advanced Intelligent Systems

Soft sensing technologies provide a novel alternative for state estimation in wearables and robotic systems. They allow one to capture intrinsic state parameters in a highly conformable manner. However, due to the nonlinearities in the materials that make up a soft sensor, it is difficult to develop accurate models of these systems. Consequently, design of these soft sensors is largely user defined or based on trial and error. Since these sensors conform and take the shape of the sensing body, these issues are further exacerbated when they are installed. Herein, a framework for the automated design optimization of soft sensors using closed-loop 3D printing of a recyclable hydrogel-based sensing material is presented. The framework allows direct printing of the sensor on the sensing body using visual feedback, evaluates the sensor performance, and iteratively improves the sensor design. Following preliminary investigations into the material and morphology parameters, this is demonstrated through the optimization of a sensorized glove which can be matched to specific tasks and individual hand shapes. The glove's sensors are tuned to respond only to particular hand poses, including distinguishing between two similar tennis racket grip techniques.

DOI： 10.1002/aisy.202300152

A Shared Control Framework for Enhanced Grasping Performance in Teleoperation 査読有り国際誌

Zhu Y., Jiang B., Chen Q., Aoyama T., Hasegawa Y.

IEEE Access 11 巻頁： 69204 - 69215 2023年7月

担当区分：最終著者記述言語：英語掲載種別：研究論文（学術雑誌）出版者・発行元：IEEE Access

Remote teleoperation has shown significant advancements since the first teleoperation system was proposed by Goertz in the 1940s. In recent years, the research on shared control methodologies in which the robot assists the operators in accomplishing the desired tasks has gained extensive attention. One such important task in teleoperation is object grasping. In this paper, we propose a shared control framework to enhance the teleoperated grasping performance. The proposed framework is built upon a virtual reality device-based direct teleoperation system. In this framework, a template matching-based object point cloud compensation is introduced for multi-angle grasping pose generation. Then, the feasible grasping candidates are selected considering joint constraints-aware manipulability. Finally, the grasping assistance is achieved by trajectory blending with dynamic authority adjustment. To validate the performance of the proposed framework, we carried out experimental evaluations. The output results indicate improved grasping performance in terms of reduced task completion time, linear trajectory, and workload.

DOI： 10.1109/ACCESS.2023.3292410

Natural Grasp Intention Recognition Based on Gaze in Human-Robot Interaction 国際共著国際誌

Yang B., Huang J., Chen X., Li X., Hasegawa Y.

IEEE Journal of Biomedical and Health Informatics 27 巻 ( 4 ) 頁： 2059 - 2070 2023年4月

記述言語：英語掲載種別：研究論文（学術雑誌）出版者・発行元：IEEE Journal of Biomedical and Health Informatics

Objective: While neuroscience research has established a link between vision and intention, studies on gaze data features for intention recognition are absent. The majority of existing gaze-based intention recognition approaches are based on deliberate long-term fixation and suffer from insufficient accuracy. In order to address the lack of features and insufficient accuracy in previous studies, the primary objective of this study is to suppress noise from human gaze data and extract useful features for recognizing grasp intention. Methods: We conduct gaze movement evaluation experiments to investigate the characteristics of gaze motion. The target-attracted gaze movement model (TAGMM) is proposed as a quantitative description of gaze movement based on the findings. A Kalman filter (KF) is used to reduce the noise in the gaze data based on TAGMM. We conduct gaze-based natural grasp intention recognition evaluation experiments to collect the subject's gaze data. Four types of features describing gaze point dispersion (f_{var}), gaze point movement (f_{gm}), head movement (f_{hm}), and distance from the gaze points to objects (f_{d_{j}}) are then proposed to recognize the subject's grasp intentions. With the proposed features, we perform intention recognition experiments, employing various classifiers, and the results are compared with different methods. Results: The statistical analysis reveals that the proposed features differ significantly across intentions, offering the possibility of employing these features to recognize grasp intentions. We demonstrated the intention recognition performance utilizing the TAGMM and the proposed features in within-subject and cross-subject experiments. The results indicate that the proposed method can recognize the intention with accuracy improvements of 44.26% (within-subject) and 30.67% (cross-subject) over the fixation-based method. The proposed method also consumes less time (34.87 ms) to recognize the intention than the fixation-based method (about 1 s). Conclusion: This work introduces a novel TAGMM for modeling gaze movement and a variety of practical features for recognizing grasp intentions. Experiments confirm the effectiveness of our approach. Significance: The proposed TAGMM is capable of modeling gaze movements and can be utilized to process gaze data, and the proposed features can reveal the user's intentions. These results contribute to the development of gaze-based human-robot interaction.

DOI： 10.1109/JBHI.2023.3238406

細胞の変形度を想起させる力触覚提示が可能な布状アクチュエータを用いた微細操作インタフェース査読有り

齋藤澄和, 青山忠義, 舟洞佑記, 竹内大, 長谷川泰久

日本ロボット学会誌 41 巻 ( 3 ) 頁： 322 - 325 2023年4月

A Concurrent Framework for Constrained Inverse Kinematics of Minimally Invasive Surgical Robots Open Access

Colan J., Davila A., Fozilov K., Hasegawa Y.

Sensors 23 巻 ( 6 ) 頁： 3328 2023年3月

記述言語：英語掲載種別：研究論文（学術雑誌）出版者・発行元：Sensors

Minimally invasive surgery has undergone significant advancements in recent years, transforming various surgical procedures by minimizing patient trauma, postoperative pain, and recovery time. However, the use of robotic systems in minimally invasive surgery introduces significant challenges related to the control of the robot’s motion and the accuracy of its movements. In particular, the inverse kinematics (IK) problem is critical for robot-assisted minimally invasive surgery (RMIS), where satisfying the remote center of motion (RCM) constraint is essential to prevent tissue damage at the incision point. Several IK strategies have been proposed for RMIS, including classical inverse Jacobian IK and optimization-based approaches. However, these methods have limitations and perform differently depending on the kinematic configuration. To address these challenges, we propose a novel concurrent IK framework that combines the strengths of both approaches and explicitly incorporates RCM constraints and joint limits into the optimization process. In this paper, we present the design and implementation of concurrent inverse kinematics solvers, as well as experimental validation in both simulation and real-world scenarios. Concurrent IK solvers outperform single-method solvers, achieving a 100% solve rate and reducing the IK solving time by up to 85% for an endoscope positioning task and 37% for a tool pose control task. In particular, the combination of an iterative inverse Jacobian method with a hierarchical quadratic programming method showed the highest average solve rate and lowest computation time in real-world experiments. Our results demonstrate that concurrent IK solving provides a novel and effective solution to the constrained IK problem in RMIS applications.

DOI： 10.3390/s23063328

Toward Autonomous Robotic Minimally Invasive Surgery: A Hybrid Framework Combining Task-Motion Planning and Dynamic Behavior Trees 国際誌

Fozilov K., Colan J., Sekiyama K., Hasegawa Y.

IEEE Access 11 巻頁： 91206 - 91224 2023年

記述言語：英語掲載種別：研究論文（学術雑誌）出版者・発行元：IEEE Access

The growing need for high levels of autonomy in Autonomous Robotic Surgery Systems (ARSS) calls for innovative approaches to reduce surgeons' cognitive load, optimize hospital workflows, and ensure efficient task-level reasoning and adaptation during execution. This paper presents a novel hybrid framework that synergistically combines Task-Motion Planning and Dynamic Behavior Trees for ARSS in Minimally Invasive Surgery. Our approach is designed to address the challenges of coordinating multiple surgical tools within a small workspace, thereby making complex surgical tasks like multi-throw suturing feasible and efficient. Through an extensive evaluation in simulation across diverse initial conditions and noise scenarios, the proposed method demonstrates improved success rates, reduced execution times, and fewer regrasps compared to standalone approaches. Furthermore, it showcases robustness under increased noise conditions. By applying our framework to a complex multi-throw suturing task, we illustrate its capability to seamlessly handle comprehensive suturing tasks, including needle picking, insertion, extraction, and the handover of the needle between Patient Side Manipulators. The results suggest that our hybrid approach not only enhances ARSS autonomy but also adapts effectively to unexpected environmental changes, laying the groundwork for its potential applicability in real-world surgical robotics.

DOI： 10.1109/ACCESS.2023.3308619

OpenRST: An Open Platform for Customizable 3D Printed Cable-Driven Robotic Surgical Tools Open Access

Colan J., Davila A., Zhu Y., Aoyama T., Hasegawa Y.

IEEE Access 11 巻頁： 6092 - 6105 2023年

記述言語：英語掲載種別：研究論文（学術雑誌）出版者・発行元：IEEE Access

Robot-assisted minimally invasive surgery (RMIS) has been shown to be effective in improving surgeon capabilities, providing magnified 3D vision, highly dexterous surgical tools, and intuitive human-robot interfaces for high-precision tool motion control. Robotic surgical tools (RST) are a critical component that defines the performance of an RMIS system. Current RSTs still represent a high cost, with few commercially available options, which limits general access and research on RMIS. We aim to take advantage of recent progress in biocompatible 3D printing and contribute to the development of RMIS technologies, presenting an open platform for low-cost, biocompatible, and customizable RSTs. The proposed design concept consists of a 3-DOF end-effector with a decoupled wrist mechanism, a tool interface module, and a tool drive unit. We validated our end-effector design using Finite Element Analysis (FEA) to confirm that stress generated by high grip forces is maintained below the material yield stress. Validation experiments showed that the proposed RST could provide up to 10N grip forces and up to 3N pulling forces. The proposed control framework exhibited a mean absolute positioning tracking error of approximately 0.1 rad. Finally, we also demonstrated the use of the proposed RST in two surgical training tasks: pick-and-place and stitching. The designs and software control framework are open-access and freely available for customization and fast development at https://github.com/jcolan/OpenRST.

DOI： 10.1109/ACCESS.2023.3236821

Micromanipulation System Capable of Simultaneously Presenting High-Resolution and Large Field-of-View Images in Real-Time

Aoyama T., Takeno S., Yokoe K., Hano K., Takasu M., Takeuchi M., Hasegawa Y.

IEEE Access 11 巻頁： 34274 - 34285 2023年

記述言語：英語掲載種別：研究論文（学術雑誌）出版者・発行元：IEEE Access

Microinjection technology is widely used in biotechnological processes such as gene manipulation and microinsemination. Generally, microinjection is performed under an optical microscope while viewing the video of the targets in real-time. In the microinjection process, multiple oocytes must be placed in the same droplet, for which the injection must be performed multiple times. Furthermore, the target must be observed at different magnifications during injection and the oocyte transfer operation. Hence, the operators are required to change the magnification and light intensity repeatedly. These tasks are complicated and are a burden for the operators. Thus, the accuracy, reproducibility, and productivity of microinjection depend on the operator's skill level. In this study, to reduce the burden on the operator and simplify the operation of microinjection, we propose a micromanipulation system that enables both wide-range and high-resolution video presentation with free viewpoint selection. The proposed micromanipulation system is based on a view-expansion microscope system with simultaneous multi-view imaging using a galvanometer mirror and a high-speed vision system. We verify the effectiveness of the proposed system by evaluating the resolution of images presented to the operator and through the experiments involving moving microbeads performed by inexperienced subjects. The experimental results indicate that our micromanipulation system streamlines the task of micromanipulation by eliminating the need to change the objective lens.

DOI： 10.1109/access.2023.3264785

Improvement in the Manipulability of Remote Touch Screens Based on Peri-Personal Space Transfer 国際誌

Yamamoto K., Zhu Y., Aoyama T., Takeuchi M., Hasegawa Y.

IEEE Access 11 巻頁： 1 - 1 2023年

記述言語：英語掲載種別：研究論文（学術雑誌）出版者・発行元：IEEE Access

Current remote-control interfaces are difficult to operate intuitively while viewing the entire remote display and require familiarity with the operation. The space within an individual's reach, called the peri-personal space (PPS), assists them in planning their physical movements. Hence, manipulating objects outside one's PPS is more difficult than doing so inside it. Therefore, this study aimed to develop an interface that creates the illusion of an operator's finger being transferred to a remote display and transfers the PPS to the virtual finger in the display. To transfer the PPS to a manipulated object, it is necessary to enhance the user's sense of agency and ownership toward the manipulated target by ensuring it is similar in shape and motion to the user's body. Long-term input testing confirmed that by transferring the PPS to a virtual finger, there is no significant difference, as suggested by a t-test between the coefficients of the learning curves outside and inside the PPS. Furthermore, when users learned to manipulate the virtual finger outside the PPS under conditions that impaired their sense of agency, even after two weeks of learning, the PPS was not continuously transferred to the virtual finger for more than seven minutes from the start of the virtual finger operation. The experimental results demonstrated that the interface enabled the display located outside the PPS to be operated with the same degree of operability as inside the PPS from the start of the operation, even with a short learning period.

DOI： 10.1109/access.2023.3271003

Gaze and Environmental Context-Guided Deep Neural Network and Sequential Decision Fusion for Grasp Intention Recognition 国際共著国際誌 Open Access

Yang B., Chen X., Xiao X., Yan P., Hasegawa Y., Huang J.

IEEE Transactions on Neural Systems and Rehabilitation Engineering 31 巻頁： 3687 - 3698 2023年

記述言語：英語掲載種別：研究論文（学術雑誌）出版者・発行元：IEEE Transactions on Neural Systems and Rehabilitation Engineering

Grasp intention recognition plays a crucial role in controlling assistive robots to aid older people and individuals with limited mobility in restoring arm and hand function. Among the various modalities used for intention recognition, the eye-gaze movement has emerged as a promising approach due to its simplicity, intuitiveness, and effectiveness. Existing gaze-based approaches insufficiently integrate gaze data with environmental context and underuse temporal information, leading to inadequate intention recognition performance. The objective of this study is to eliminate the proposed deficiency and establish a gaze-based framework for object detection and its associated intention recognition. A novel gaze-based grasp intention recognition and sequential decision fusion framework (GIRSDF) is proposed. The GIRSDF comprises three main components: gaze attention map generation, the Gaze-YOLO grasp intention recognition model, and sequential decision fusion models (HMM, LSTM, and GRU). To evaluate the performance of GIRSDF, a dataset named Invisible containing data from healthy individuals and hemiplegic patients is established. GIRSDF is validated by trial-based and subject-based experiments on Invisible and outperforms the previous gaze-based grasp intention recognition methods. In terms of running efficiency, the proposed framework can run at a frequency of about 22 Hz, which ensures real-Time grasp intention recognition. This study is expected to inspire additional gaze-related grasp intention recognition works.

DOI： 10.1109/TNSRE.2023.3314503

Auditory Feedback for Enhanced Sense of Agency in Shared Control 査読有り

Tomoya Morita, Yaonan Zhu, Tadayoshi Aoyama, Masaru Takeuchi, Kento Yamamoto, Yasuhisa Hasegawa

Sensors 22 巻 ( 24 ) 頁： 9779 2022年12月

An immersive micro-manipulation system using real-time 3D imaging microscope and 3D operation interface for high-speed and accurate micro-manipulation 査読有り国際誌

Yokoe K., Aoyama T., Fujishiro T., Takeuchi M., Hasegawa Y.

ROBOMECH Journal 9 巻 ( 1 ) 頁： 16 2022年12月

担当区分：最終著者記述言語：英語掲載種別：研究論文（学術雑誌）出版者・発行元：ROBOMECH Journal

The use of intracytoplasmic sperm injection (ICSI), an assisted reproductive technique (ART), is increasing widely. ICSI is currently performed by specially skilled embryologists. However, with the increasing demand for ART, the shortage of skilled embryologists has become a problem. Therefore, we propose an immersive micromanipulation system that requires no special skills for efficient and accurate micromanipulation. Our proposed system is composed of a real-time three-dimensional (3D) imaging microscope and 3D operation interfaces. The 3D operation interfaces are stationary pen-type or wearable glove-type interfaces. In this system, an operator wearing a head-mounted display (HMD) and using 3D operation interfaces is immersed in a virtual micromanipulation space. The operator can move the pipettes by 3D operation interface and freely change the viewpoint. We verified that the proposed system improves the speed and accuracy of operating a pipette through two types of experiments with subjects.

DOI： 10.1186/s40648-022-00228-6

Auditory Feedback for Enhanced Sense of Agency in Shared Control 査読有り国際誌

Morita T., Zhu Y., Aoyama T., Takeuchi M., Yamamoto K., Hasegawa Y.

Sensors 22 巻 ( 24 ) 頁： 9779 - 9779 2022年12月

担当区分：最終著者記述言語：英語掲載種別：研究論文（学術雑誌）出版者・発行元：Sensors

There is a growing need for robots that can be remotely controlled to perform tasks of one’s own choice. However, the SoA (Sense of Agency: the sense of recognizing that the motion of an observed object is caused by oneself) is reduced because the subject of the robot motion is identified as external due to shared control. To address this issue, we aimed to suppress the decline in SoA by presenting auditory feedback that aims to blur the distinction between self and others. We performed the tracking task in a virtual environment under four different auditory feedback conditions, with varying levels of automation to manipulate the virtual robot gripper. Experimental results showed that the proposed auditory feedback suppressed the decrease in the SoA at a medium level of automation. It is suggested that our proposed auditory feedback could blur the distinction between self and others, and that the operator attributes the subject of the motion of the manipulated object to himself.

DOI： 10.3390/s22249779