研究者詳細 - 駒水　孝裕

MMEnsemble: Imbalanced Classification Framework Using Metric Learning and Multi-sampling Ratio Ensemble

その他リンク： https://dblp.uni-trier.de/db/conf/dexa/dexa2023-2.html#Komamizu23

Komamizu, T

DATABASE AND EXPERT SYSTEMS APPLICATIONS, DEXA 2021, PT II 12924 巻頁： 176 - 188 2021年

In classification, class imbalance is a factor that degrades the classification performance of many classification methods. Resampling is one widely accepted approach to the class imbalance; however, it still suffers from an insufficient data space, which also degrades performance. To overcome this, in this paper, an undersampling-based imbalanced classification framework, MMEnsemble, is proposed that incorporates metric learning into a multi-ratio undersampling-based ensemble. This framework also overcomes a problem with determining the appropriate sampling ratio in the multi-ratio ensemble method. It was evaluated by using 12 real-world datasets. It outperformed the state-of-the-art approaches of metric learning, undersampling, and oversampling in recall and ROC-AUC, and it performed comparably with them in terms of Gmean and F-measure metrics.

DOI： 10.1007/978-3-030-86475-0_18

Correction to: Computational measurement of perceived pointiness from pronunciation (Multimedia Tools and Applications, (2023), 83, 9, (26183-26210), 10.1007/s11042-023-15732-z)

その他リンク： https://dblp.uni-trier.de/db/conf/dexa/dexa2021-2.html#Komamizu21

Matsuhira C., Kastner M.A., Komamizu T., Ide I., Hirayama T., Kawanishi Y., Doman K., Deguchi D.

Multimedia Tools and Applications 83 巻 ( 9 ) 頁： 26211 - 26212 2024年3月

出版者・発行元：Multimedia Tools and Applications

The original publication of this article contains the following errors: missing ORCID of authors incorrect author contribution statement pronunciation symbols were not shown correctly in both online and PDF versionsthe gamma symbol "Γ" were incorrectly displayed as "0" in the PDF version missing ORCID of authors incorrect author contribution statement pronunciation symbols were not shown correctly in both online and PDF versions the gamma symbol "Γ" were incorrectly displayed as "0" in the PDF version The original article has been corrected.

DOI： 10.1007/s11042-023-17657-z

Image-Collection Summarization Using Scene-Graph Generation With External Knowledge

Phueaksri, I; Kastner, MA; Kawanishi, Y; Komamizu, T; Ide, I

IEEE ACCESS 12 巻頁： 17499 - 17512 2024年

出版者・発行元：IEEE Access

Summarization tasks aim to summarize multiple pieces of information into a short description or representative information. A text summarization task summarizes textual information into a short description, whereas an image collection summarization task summarizes an image collection into images or textual representation in which the challenge is to understand the relationship between images. In recent years, scene-graph generation has shown the advantage of describing the visual contexts of a single-image, and incorporating external knowledge into the scene-graph generation model has also given effective directions for unseen single-image scene-graph generation. While external knowledge has been implemented in related work, it is still challenging to use this information efficiently for relationship estimation during the summarization. Following this trend, in this paper, we propose a novel scene-graph-based image-collection summarization model that aims to generate a summarized scene-graph of an image collection. The key idea of the proposed method is to enhance the relation predictor toward relationships between images in an image collection incorporating knowledge graphs as external knowledge for training a model. With this approach, we build an end-to-end framework that can generate a summarized scene graph of an image collection. To evaluate the proposed method, we also build an extended annotated MS-COCO dataset for this task and introduce an evaluation process that focuses on estimating the similarity between a summarized scene graph and ground-truth scene graphs. Traditional evaluation focuses on calculating precision and recall scores, which involve true positive predictions without balancing precision and recall. Meanwhile, the proposed evaluation process focuses on calculating the F-score of the similarity between a summarized scene graph and ground-truth scene graphs, which aims to balance both false positives and false negatives. Experimental results show that using external knowledge to enhance the relation predictor achieves better results than existing methods.

DOI： 10.1109/ACCESS.2024.3360113

RecipeMeta: Metapath-enhanced Recipe Recommendation on Heterogeneous Recipe Network

Shi J., Komamizu T., Doman K., Kyutoku H., Ide I.

Proceedings of the 5th ACM International Conference on Multimedia in Asia, MMAsia 2023 2023年12月

出版者・発行元：Proceedings of the 5th ACM International Conference on Multimedia in Asia, MMAsia 2023

Recipe is a set of instructions that describes how to make food. It can help people from the preparation of ingredients, food cooking process, etc. to prepare the food, and increasingly in demand on the Web. To help users find the vast amount of recipes on the Web, we address the task of recipe recommendation. Due to multiple data types and relationships in a recipe, we can treat it as a heterogeneous network to describe its information more accurately. To effectively utilize the heterogeneous network, metapath was proposed to describe the higher-level semantic information between two entities by defining a compound path from peer entities. Therefore, we propose a metapath-enhanced recipe recommendation framework, RecipeMeta, that combines GNN (Graph Neural Network)-based representation learning and specific metapath-based information in a recipe to predict User-Recipe pairs for recommendation. Through extensive experiments, we demonstrate that the proposed model, RecipeMeta, outperforms state-of-the-art methods for recipe recommendation.

DOI： 10.1145/3595916.3626430

NarSUM 2023 Chairs Welcome

Kankanhalli M.S., Patras I., Liu J., Wong Y., Komamizu T.

NarSUM 2023 - Proceedings of the 2nd Workshop on User-centric Narrative Summarization of Long Videos, Co-located with: MM 2023 2023年10月

NarSUM '23: The 2nd Workshop on User-Centric Narrative Summarization of Long Videos

Kankanhalli M.S., Patras I.Y., Liu J., Wong Y., Komamizu T., Yamazaki S., Stephen K., Kansal K.

MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia 頁： 9731 - 9733 2023年10月

出版者・発行元：MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia

With video capture devices becoming widely popular, the amount of video data generated per day has seen a rapid increase over the past few years. Browsing through hours of video data to retrieve useful information is a tedious and boring task. Video Summarization technology has played a crucial role in addressing this issue. It is a well-researched topic in the multimedia community. However, the focus so far has been limited to creating summary to videos which are short (only a few minutes). This workshop aims to call for researchers on relevant background to focus on novel solutions for user-centric narrative summarization of long videos. This workshop will also cover important aspects of video summarization research like what is "important"in a video, how to evaluate the goodness of a created summary, open challenges in video summarization, etc.

DOI： 10.1145/3581783.3610946

An Automatic Labeling Method for Subword-Phrase Recognition in Effective Text Classification

Kimura Y., Komamizu T., Hatano K.

Informatica (Slovenia) 47 巻 ( 3 ) 頁： 315 - 326 2023年8月

掲載種別：研究論文（学術雑誌）出版者・発行元：Informatica (Slovenia)

The deep learning-based text classification methods perform better than traditional ones. In addition to the success of the deep learning technique, multi-Task learning (MTL) has come to become a promising approach for text classification; for instance, an MTL approach in text classification employs named entity recognition as an auxiliary task and has showcased that the task helps to improve the text classification performance. Existing MTL-based text classification methods depend on the auxiliary tasks using supervised labels. Obtaining such supervision labels requires additional human and financial costs in addition to those for the main text classification task. To reduce these additional costs, we propose an MTL-based text classification framework on supervised label creation by automatically labeling phrases in texts for the auxiliary recognition task. A basic idea to realize the proposed framework is to utilize phrasal expressions consisting of subwords (called subword-phrases). To the best of our knowledge, no text classification approach has been designed on top of subword-phrases because subwords only sometimes express a coherent set of meanings. The novelty of the proposed framework is in adding subword-phrase recognition as an auxiliary task and utilizing subword-phrases for text classification. It extracts subword-phrases in an unsupervised manner using the statistics approach. To construct labels for effective subword-phrase recognition tasks, extracted subword-phrases are classified based on document classes to ensure that subword-phrases dedicated to some classes can be distinguishable. Experimental evaluation for text classification using five popular datasets showcased the effectiveness of the subword-phrase recognition as an auxiliary task. It also showed that comparing various labeling schemes in recent studies indicated insights for labeling common subword-phrases among several document classes.

DOI： 10.31449/inf.v47i3.4742

Computational measurement of perceived pointiness from pronunciation

Matsuhira, C; Kastner, MA; Komamizu, T; Ide, I; Hirayama, T; Kawanishi, Y; Doman, K; Deguchi, D

MULTIMEDIA TOOLS AND APPLICATIONS 83 巻 ( 9 ) 頁： 26183 - 26210 2023年8月

出版者・発行元：Multimedia Tools and Applications

Sound symbolism is a well-researched topic of psycholinguistics, which tries to comprehend the connection between the sound of a word and its meanings. The Bouba-Kiki effect, one form of sound symbolism, claims that people perceive the pronunciation of “Kiki” as pointier than that of “Bouba.” There is no research that focuses on modeling such perception, i.e., how pointy a pronunciation sounds to humans, through computational and data-driven approaches. To address this, this paper first proposes the novel concept of “phonetic pointiness” defined as how pointy a shape humans are most likely to associate with a given pronunciation. We then model this phonetic pointiness from computational and data-driven approaches to calculate a score for an arbitrary pronunciation. There are three proposed models: a referential model, an expressive model, and a combined model, which integrates the previous two. The idea comes from an existing psycholinguistic classification of two types of sound symbolisms: referential symbolism and expressive symbolism, where the former relates to vocabulary knowledge, while the latter is based on pure human intuition. The proposed models are constructed only with image and language data available on the Web, therefore not requiring task-specific human annotations. We evaluate these models through a crowd-sourced user study, finding a promising correlation between human perception and the phonetic pointiness calculated by the proposed models. The results indicate that human perception can be modeled better by combining both types of sound symbolisms. Furthermore, by observing the behaviors of the models, we show several possible use-cases, such as product naming and psycholinguistic research, which can be a useful insight to further studies and applications.

DOI： 10.1007/s11042-023-15732-z

Image Impression Estimation by Clustering People with Similar Tastes

Kojima Banri, Komamizu Takahiro, Kawanishi Yasutomo, Doman Keisuke, Ide Ichiro

IEICE Proceeding Series 78 巻頁： P1-14 2023年7月

記述言語：英語出版者・発行元：The Institute of Electronics, Information and Communication Engineers

This paper proposes a method for estimating impressions received from images according to the personal attributes of users, so that they can find the desired images based on their tastes. A previous study taking into account gender and age as personal attributes showed promising results. However, it also showed that users sharing the same gender and age do not necessarily share similar tastes. Therefore, other attributes should be considered to well capture users' personal tastes. However, taking more attributes into account leads to a problem that insufficient amounts of data are served to classifiers, due to explosion of the number of combinations of attributes. To tackle this problem, we propose an aggregation-based method to condense training data for impression estimation while personal attribute information is taken into account. For evaluation, a dataset of 4,000 carpet images annotated with 24 impression words by crowd-workers was prepared, which contained 273k annotations. Experimental results showed that the use of combinations of personal attributes improved the accuracy of impression estimation. This indicates that combinations of personal attributes are helpful to estimate impressions of individual viewers to images.

DOI： 10.34385/proc.78.p1-14

Towards Achieving Lightweight Deep Neural Network for Precision Agriculture with Maize Disease Detection

Padeiro Carlos-Victorino, Komamizu Takahiro, Ide Ichiro

IEICE Proceeding Series 78 巻頁： P1-23 2023年7月

記述言語：英語出版者・発行元：The Institute of Electronics, Information and Communication Engineers

Agriculture is the pillar industry of human sur- vival. However, various crop diseases reduce the hu- man food supply and lead to starvation and death in the worst cases. Experts perform visual symptoms ob- servation for crop disease diagnosis. Which process is time-consuming and expensive. Also, the process has significant risk of human error due to subjective per- ception. Convolutional Neural Networks (CNN) use image processing techniques to show great potential in plant disease detection. However, it requires thou- sands of channels to learn rich features, resulting in large models requiring powerful computing, power sup- ply, and high bandwidth, making it more expensive and difficult for farmers to acquire. Therefore, deploying these solutions on resource-constrained devices is de- sirable to make them more accessible. Thus, we pro- pose a lightweight object detection CNN that can run on resource-constrained devices to detect crop diseases. Channel pruning is applied to optimize resource use by removing unimportant channels and filter weights to reduce network parameters, inference time, and the number of FLOPS. Experimental results with object de- tector, Faster R-CNN with two backbones, ResNet-50, and EfficientNet-B7, show significant improvement in model efficiency, keeping high accuracy.

DOI： 10.34385/proc.78.p1-23

Small Object Detection for Birds with Swin Transformer

Huo Da, Kastner Marc-A., Liu Tingwei, Kawanishi Yasutomo, Hirayama Takatsugu, Komamizu Takahiro, Ide Ichiro

IEICE Proceeding Series 78 巻頁： TE-3 2023年7月

記述言語：英語出版者・発行元：The Institute of Electronics, Information and Communication Engineers

Object detection is the task of detecting objects in an image. In this task, the detection of small objects is particularly difficult. Other than the small size, it is also accompanied by difficulties due to blur, occlusion, and so on. Current small object detection methods are tailored to small and dense situations, such as pedestrians in a crowd or far objects in remote sensing scenarios. However, when the target object is small and sparse, there is a lack of objects available for training, making it more difficult to learn effective features. In this paper, we propose a specialized method for detecting a specific category of small objects; birds. Particularly, we improve the features learned by the neck; the sub-network between the backbone and the prediction head, to learn more effective features with a hierarchical design. We employ Swin Transformer to upsample the image features. Moreover, we change the shifted window size for adapting to small objects. Experiments show that the proposed Swin Transformerbased neck combined with CenterNet can lead to good performance by changing the window sizes. We further find that smaller window sizes (default 2) benefit mAPs for small object detection.

DOI： 10.34385/proc.78.te-3

MVA2023 Small Object Detection Challenge for Spotting Birds: Dataset, Methods, and Results

Kondo Yuki, Ukita Norimichi, Yamaguchi Takayuki, Hou Hao-Yu, Shen Mu-Yi, Hsu Chia-Chi, Huang En-Ming, Huang Yu-Chen, Xia Yu-Cheng, Wang Chien-Yao, Lee Chun-Yi, Huo Da, Kastner Marc-A., Liu Tingwei, Kawanishi Yasutomo, Hirayama Takatsugu, Komamizu Takahiro, Ide Ichiro, Shinya Yosuke, Liu Xinyao, Liang Guang, Yasui Syusuke

IEICE Proceeding Series 78 巻頁： TE-1 2023年7月

記述言語：英語出版者・発行元：The Institute of Electronics, Information and Communication Engineers

Small Object Detection (SOD) is an important machine vision topic because (i) a variety of real-world applications require object detection for distant objects and (ii) SOD is a challenging task due to the noisy, blurred, and less-informative image appearances of small objects. This paper proposes a new SOD dataset consisting of 39,070 images including 137,121 bird instances, which is called the Small Object Detection for Spotting Birds (SOD4SB) dataset. The detail of the challenge with the SOD4SB dataset is introduced in this paper. In total, 223 participants joined this challenge. This paper briefly introduces the awardwinning methods. The dataset, the baseline code, and the website for evaluation on the public testset are publicly available.

DOI： 10.34385/proc.78.te-1

[D21] 歴史情報としての法令データベースの構築

佐野智也, 外山勝彦, 駒水孝裕, 増田知子

デジタルアーカイブ学会誌 7 巻 ( s2 ) 頁： s142 - s145 2023年

記述言語：日本語出版者・発行元：デジタルアーカイブ学会

国家・社会制度に関する政策は、法令を通して制度化されるため、日本社会の動きは、法令情報を介して捉えることができる。本研究は、制定・改正などを通じた法令の連続的変遷を把握し、日本の国家・社会運営の長期的変化を調査するための研究基盤の確立を目指すものである。その最初の目標として、明治以降の全法令を検索可能なオープンデータベースシステムの構築を進めているが、現在、明治 19（1886）年から平成 29（2017）年までに公布された法律と勅令のXML文書化を完了し、それらの全文検索が可能なデータベースの構築を終えた。本報告では、既存のデータベースの問題点について述べた上で、構築したデータベースを説明する。

DOI： 10.24506/jsda.7.s2_s142

Visual Passage Score Aggregation for Image Retrieval

Komamizu T.

Proceedings - 2023 IEEE 6th International Conference on Multimedia Information Processing and Retrieval, MIPR 2023 頁： 37 - 42 2023年

出版者・発行元：Proceedings - 2023 IEEE 6th International Conference on Multimedia Information Processing and Retrieval, MIPR 2023

This paper proposes an effective image retrieval method. Recent image retrieval approaches attempt to construct a single global feature, including local features of an image. In contrast, this paper proposes multiple features for each image. The basic idea is that a target object in a query image is not necessarily in a major part of a database image; therefore, its single feature may include noisy information from the surroundings of the target object. To deal with this, this paper proposes a Visual Passage Score Aggregation framework (VPSA). VPSA first decomposes an image into several pieces of images, called Visual Passages. Based on visual passages, VPSA aggregates relevance scores of visual passages for ranking. VPSA is efficient in the retrieval phase because an ordinary nearest neighbor search is used. The experiment revealed that VPSA showed superior or comparable performance to the state-of-the-art methods, and it takes a shorter time in the retrieval phase.

DOI： 10.1109/MIPR59079.2023.00021

Towards Captioning an Image Collection from a Combined Scene Graph Representation Approach

Phueaksri, I; Kastner, MA; Kawanishi, Y; Komamizu, T; Ide, I

MULTIMEDIA MODELING, MMM 2023, PT I 13833 巻頁： 178 - 190 2023年

掲載種別：論文集(書籍)内論文出版者・発行元：Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Most content summarization models from the field of natural language processing summarize the textual contents of a collection of documents or paragraphs. In contrast, summarizing the visual contents of a collection of images has not been researched to this extent. In this paper, we present a framework for summarizing the visual contents of an image collection. The key idea is to collect the scene graphs for all images in the image collection, create a combined representation, and then generate a visually summarizing caption using a scene-graph captioning model. Note that this aims to summarize common contents across all images in a single caption rather than describing each image individually. After aggregating all the scene graphs of an image collection into a single scene graph, we normalize it by using an additional concept generalization component. This component selects the common concept in each sub-graph with ConceptNet based on word embedding techniques. Lastly, we refine the captioning results by replacing a specific noun phrase with a common concept from the concept generalization component to improve the captioning results. We construct a dataset for this task based on the MS-COCO dataset using techniques from image classification and image-caption retrieval. An evaluation of the proposed method on this dataset shows promising performance.

DOI： 10.1007/978-3-031-27077-2_14

Towards Achieving Lightweight Deep Neural Network for Precision Agriculture with Maize Disease Detection

Padeiro, CV; Komamizu, T; Ide, I

2023 18TH INTERNATIONAL CONFERENCE ON MACHINE VISION AND APPLICATIONS, MVA 頁： 1 - 6 2023年

掲載種別：研究論文（国際会議プロシーディングス）出版者・発行元：Proceedings of MVA 2023 - 18th International Conference on Machine Vision and Applications

Agriculture is the pillar industry of human survival. However, various crop diseases reduce the human food supply and lead to starvation and death in the worst cases. Experts perform visual symptoms observation for crop disease diagnosis. Which process is time-consuming and expensive. Also, the process has significant risk of human error due to subjective perception. Convolutional Neural Networks (CNN) use image processing techniques to show great potential in plant disease detection. However, it requires thousands of channels to learn rich features, resulting in large models requiring powerful computing, power supply, and high bandwidth, making it more expensive and difficult for farmers to acquire. Therefore, deploying these solutions on resource-constrained devices is desirable to make them more accessible. Thus, we propose a lightweight object detection CNN that can run on resource-constrained devices to detect crop diseases. Channel pruning is applied to optimize resource use by removing unimportant channels and filter weights to reduce network parameters, inference time, and the number of FLOPS. Experimental results with object detector, Faster R-CNN with two backbones, ResNet-50, and EfficientNet-B7, show significant improvement in model efficiency, keeping high accuracy.

DOI： 10.23919/MVA57639.2023.10215815

Small Object Detection for Birds with Swin Transformer

その他リンク： https://dblp.uni-trier.de/db/conf/mva/mva2023.html#PadeiroKI23

Huo, D; Kastner, MA; Liu, T; Kawanishi, Y; Hirayama, T; Komamizu, T; Ide, I

2023 18TH INTERNATIONAL CONFERENCE ON MACHINE VISION AND APPLICATIONS, MVA 頁： 1 - 5 2023年

掲載種別：研究論文（国際会議プロシーディングス）出版者・発行元：Proceedings of MVA 2023 - 18th International Conference on Machine Vision and Applications

Object detection is the task of detecting objects in an image. In this task, the detection of small objects is particularly difficult. Other than the small size, it is also accompanied by difficulties due to blur, occlusion, and so on. Current small object detection methods are tailored to small and dense situations, such as pedestrians in a crowd or far objects in remote sensing scenarios. However, when the target object is small and sparse, there is a lack of objects available for training, making it more difficult to learn effective features. In this paper, we propose a specialized method for detecting a specific category of small objects; birds. Particularly, we improve the features learned by the neck; the sub-network between the backbone and the prediction head, to learn more effective features with a hierarchical design. We employ Swin Transformer to upsample the image features. Moreover, we change the shifted window size for adapting to small objects. Experiments show that the proposed Swin Transformer-based neck combined with CenterNet can lead to good performance by changing the window sizes. We further find that smaller window sizes (default 2) benefit mAPs for small object detection.

DOI： 10.23919/MVA57639.2023.10216093