Authorship：Corresponding author   Language：English   Publisher：The Institute of Image Information and Television Engineers  

<p>Compressing the sign information of discrete cosine transform (DCT) coefficients is an intractable problem in image coding schemes due to the equiprobable characteristics of the signs. To overcome this difficulty, we propose an efficient compression method for the sign information called “sign retrieval.” This method is inspired by phase retrieval, which is a classical signal restoration problem of finding the phase information of discrete Fourier transform coefficients from their magnitudes. The sign information of all DCT coefficients is excluded from a bitstream at the encoder and is complemented at the decoder through our sign retrieval method. We show through experiments that our method outperforms previous ones in terms of the bit amount for the signs and computation cost. Our method, implemented in Python language, is available from https://github.com/ctsutake/dsr.</p>

DOI： 10.3169/mta.12.110

Open Access

Scopus

CiNii Research

Other Link： https://arxiv.org/abs/2209.10712

Publisher：2024 IEEE International Conference on Visual Communications and Image Processing, VCIP 2024  

A light field is represented as a set of multi-view images captured from a dense 2-D array of viewpoints. To treat a light field as being continuous, we represent it as a neural radiance field (NeRF), which is a learned representation of a 3-D scene. NeRFs are renowned for their ability to reconstruct a target 3-D scene with compelling visual quality, but they are slow to train. A solution for this problem is to use a tiny neural network and trainable volumetric features as the scene representation, which is considered the baseline of our research. For further acceleration, we propose a method for warm-starting the per-scene training by setting good initial values for the trainable parameters. To this end, we introduce another encoder network to obtain the initial volumetric features from the target light field. Starting with the appropriate initial values, our method can achieve better rendering quality with fewer training iterations than the baseline.

DOI： 10.1109/VCIP63160.2024.10849784

Scopus

Publisher：Proceedings of SPIE - The International Society for Optical Engineering  

An event camera adopts a bio-inspired sensing mechanism that can record the luminance changes over time. The recorded information, called events, are detected asynchronously at each pixel in the order of microseconds. Events are quite useful for framerate upsampling of a video, because the information between the low-framerate video frames (key-frames) can be supplemented from the events. We propose a method for framerate upsampling from events on the basis of an unsupervised approach; our method does not require ground-truth high-framerate videos for pre-training but can be trained solely on the key-frames and events taken from the target scene. We also report some promising experimental results with a fast moving scene captured by a DAVIS346 event camera.

DOI： 10.1117/12.3018663

Web of Science

Scopus

Publisher：Proceedings of SPIE - The International Society for Optical Engineering  

We propose a data compression method for a light field using a compact and computationally-efficient neural representation. We first train a neural network with learnable parameters to reproduce the target light field. We then compress the set of learned parameters as an alternative representation of the light field. Our method is significantly different in concept from the traditional approaches where a light field is encoded as a set of images or a video (as a pseudo-temporal sequence) using off-the-shelf image/video codecs. We experimentally show that our method achieves a promising rate-distortion performance.

DOI： 10.1117/12.3018716

Web of Science

Scopus

Publisher：Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition  

We propose a computational imaging method for time-efficient light-field acquisition that combines a coded aperture with an event-based camera. Differentfrom the conventional coded-aperture imaging method, our method applies a sequence of coding patterns during a single exposure for an image frame. The parallax information, which is related to the differences in coding patterns, is recorded as events. The image frame and events, all of which are measured in a single exposure, are jointly used to computationally reconstruct a light field. We also designed an algorithm pipeline for our method that is end-to-end trainable on the basis of deep optics and compatible with real camera hardware. We experimentally showed that our method can achieve more accurate reconstruction than several other imaging methods with a single exposure. We also developed a hardware prototype with the potential to complete the measurement on the camera within 22 msec and demonstrated that light fields from real 3-D scenes can be obtained with convincing visual quality. Our software and supplementary video are available from our project website.

DOI： 10.1109/CVPR52733.2024.02354

Scopus

Publisher：2024 IEEE International Conference on Visual Communications and Image Processing, VCIP 2024  

A light field is usually represented as a set of multi-view images captured from a two-dimensional (2-D) array of viewpoints and requires a large amount of data compared with a standard 2-D image. We propose a 2-D compatible light-field compression method for encoding a light field as a 2-D monocular image and subsidiary data. In terms of the image quality, we prioritize the central image (regarded as the 2-D monocular image) over the other images in the light field, because the light field is considered an extension of the 2-D monocular image. To this end, we encode and decode the monocular image using a standard image codec and introduce a learned encoder and decoder pair for the subsidiary data. Experimental results indicate that our method achieved promising rate-distortion performance, especially for extremely low bit-rate ranges. Even though our method requires only a small amount of subsidiary data compared with those for the monocular image, the entire light field can be reconstructed with reasonable visual quality.

DOI： 10.1109/VCIP63160.2024.10849860

Scopus

Language：English   Publisher：The Institute of Image Information and Television Engineers  

<p>We propose a method for compressively acquiring a light field video using a single camera equipped with an optical aperture-exposure coding mechanism. The aperture-exposure coding is applied to each exposure time, enabling the embedding of the information of a light field video (a 5-D volume) into a single observed image (a 2-D measurement). Temporally-successive images obtained from the camera are used to computationally reconstruct the light field video at a faster frame rate than that of the camera. We also developed a hardware prototype to validate our method on real 3-D time-varying scenes. Using our method, we can obtain a light field video with 5 × 5 viewpoints over 4 temporal sub-frames (100 views in total) per each observed image. By repeating the capture and reconstruction processes over time, we can acquire a light field video of arbitrary length at 4 × the frame rate of the camera. To the best of our knowledge, we are the first to propose a method of joint angular-temporal compression for light-field acquisition, achieving a finer temporal resolution than that of the camera. A supplementary video is available from https://youtu.be/FAujrak8Dok.</p>

DOI： 10.3169/mta.12.22

Open Access

Scopus

CiNii Research

Language：Japanese   Publisher：一般社団法人電子情報通信学会  

DOI： 10.1587/ieiceissjournal.28.3_7

CiNii Research

Language：Japanese   Publisher：3次元画像コンファレンス実行委員会  

DOI： 10.60374/sanjigen.31.0_13

CiNii Research

Language：Japanese   Publisher：3次元画像コンファレンス実行委員会  

DOI： 10.60374/sanjigen.31.0_31

CiNii Research

Publisher：IEEE Access  

We propose a method for reconstructing a continuous light field of a target scene from a single observed image. Our method takes the best of two worlds: joint aperture-exposure coding for compressive light-field acquisition, and a neural radiance field (NeRF) for view synthesis. Joint aperture-exposure coding implemented in a camera enables effective embedding of 3-D scene information into an observed image, but in previous works, it was used only for reconstructing discretized light-field views. NeRF-based neural rendering enables high quality view synthesis of a 3-D scene from continuous viewpoints, but when only a single image is given as the input, it struggles to achieve satisfactory quality. Our method integrates these two techniques into an efficient and end-to-end trainable pipeline. Trained on a wide variety of scenes, our method can reconstruct continuous light fields accurately and efficiently without any test time optimization. To our knowledge, this is the first work to bridge two worlds: camera design for efficiently acquiring 3-D information and neural rendering.

DOI： 10.1109/ACCESS.2023.3314340

Open Access

Web of Science

Scopus

Web of Science

Web of Science

Web of Science

Language：English   Publisher：The Institute of Image Information and Television Engineers  

<p>A light field contains a large amount of data because it is represented as a dense set of multi-view images. We propose a method of compressing a light field as a newly emerging representation called a multiplane image (MPI), a graphics-oriented representation composed of a stack of semi-transparent images. Our method was constructed on a deep convolutional neural network (CNN), which was trained to generate an MPI from the given light field. To draw out the potential of this representation, we trained the CNN to be overfitted to the target light field. We also encouraged spatial smoothness to make the MPI easier to compress. Despite being in the early stage of development, our method has already achieved promising rate-distortion performance.</p>

DOI： 10.3169/MTA.11.27

Scopus

CiNii Research

Language：English   Publisher：The Institute of Image Information and Television Engineers  

<p>A multiplane image (MPI) is a useful 3-D representation composed of a stack of semi-transparent images, from which arbitrary views can be rendered with little computational cost. In this paper, we tackled the problem of super-resolution for MPIs, where a high-resolution MPI is inferred from a lower resolution one. By analyzing the anti-aliasing condition for the light field that would be produced from an MPI, we clarified that such a high-resolution MPI should have smaller sampling intervals over not only the spatial dimension but also the depth dimension. On the basis of this analysis, we constructed a learning-based method to transform a low-resolution MPI into a higher resolution one with depth resolution enhancement. Tested on BasicLFSR dataset, our method achieved 30.54 dB on average, which was 1.29 dB higher than the case without depth resolution enhancement. Visual results indicated that our method can accurately restore high-frequency components. Although super-resolution techniques have been studied extensively for images, videos, and light fields, this is the first work to address the problem of direct super-resolution for MPIs.</p>

DOI： 10.3169/MTA.11.34

Open Access

Scopus

CiNii Research

Language：English   Publisher：The Institute of Electronics, Information and Communication Engineers  

<p>A light field (LF), which is represented as a set of dense, multi-view images, has been used in various 3D applications. To make LF acquisition more efficient, researchers have investigated compressive sensing methods by incorporating certain coding functionalities into a camera. In this paper, we focus on a challenging case called snapshot compressive LF imaging, in which an entire LF is reconstructed from only a single acquired image. To embed a large amount of LF information in a single image, we consider two promising methods based on rapid optical control during a single exposure: time-multiplexed coded aperture (TMCA) and coded focal stack (CFS), which were proposed individually in previous works. Both TMCA and CFS can be interpreted in a unified manner as extensions of the coded aperture (CA) and focal stack (FS) methods, respectively. By developing a unified algorithm pipeline for TMCA and CFS, based on deep neural networks, we evaluated their performance with respect to other possible imaging methods. We found that both TMCA and CFS can achieve better reconstruction quality than the other snapshot methods, and they also perform reasonably well compared to methods using multiple acquired images. To our knowledge, we are the first to present an overall discussion of TMCA and CFS and to compare and validate their effectiveness in the context of compressive LF imaging.</p>

DOI： 10.1587/transinf.2022PCP0003

Web of Science

Scopus

CiNii Research

Language：English   Publisher：The Institute of Electronics, Information and Communication Engineers  

<p>Inspired by the framework of algorithm unrolling, we propose a scalable network architecture that computes layer patterns for light field displays, enabling control of the trade-off between the display quality and the computational cost on a single pre-trained network.</p>

DOI： 10.1587/transinf.2022PCL0002

Web of Science

Scopus

CiNii Research

Publisher：Optics Express  

Integral three-dimensional (3D) displays can display naturally viewable 3D images. However, displaying 3D images with high pixel density is difficult because the maximum pixel number is restricted by the number of lenses of a lens array. Therefore, we propose a method for increasing the maximum pixel density of 3D images by optically synthesizing the displayed images of an integral 3D display and high-definition two-dimensional display using a half mirror. We evaluated the improvements in 3D image resolution characteristics through simulation analysis of the modulation transfer function. We developed a prototype display system that can display 3D images with a maximum resolution of 4K and demonstrated the effectiveness of the proposed method.

DOI： 10.1364/OE.469045

Open Access

Web of Science

Scopus

PubMed

Language：Japanese   Publisher：3次元画像コンファレンス実行委員会  

DOI： 10.60374/sanjigen.30.0_75

CiNii Research

Language：Japanese   Publisher：3次元画像コンファレンス実行委員会  

DOI： 10.60374/sanjigen.30.0_39

CiNii Research

Language：Japanese   Publisher：3次元画像コンファレンス実行委員会  

DOI： 10.60374/sanjigen.30.0_71

CiNii Research

Publisher：Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition  

We propose a method for compressively acquiring a dynamic light field (a 5-D volume) through a single-shot coded image (a 2-D measurement). We designed an imaging model that synchronously applies aperture coding and pixel-wise exposure coding within a single exposure time. This coding scheme enables us to effectively embed the original information into a single observed image. The observed image is then fed to a convolutional neural network (CNN) for light-field reconstruction, which is jointly trained with the camera-side coding patterns. We also developed a hardware prototype to capture a real 3-D scene moving over time. We succeeded in acquiring a dynamic light field with 5x5 viewpoints over 4 temporal sub-frames (100 views in total)from a single observed image. Repeating capture and reconstruction processes over time, we can acquire a dynamic light field at 4x the frame rate of the camera. To our knowledge, our method is the first to achieve a finer temporal resolution than the camera itself in compressive light-field acquisition. Our software is available from our project webpage.11https://www.fujii.nuee.nagoya-u.ac.jp/Research/CompCam2

DOI： 10.1109/CVPR52688.2022.01921

Web of Science

Scopus

Web of Science

Publisher：Proceedings of the International Display Workshops  

We propose a method of displaying two different 3D scenes on a single layered light-field display, where the layer patterns are optimized for the two scenes simultaneously. We demonstrate that both scenes can be displayed in high quality when the viewing zones for them are separated sufficiently.

Scopus

Publisher：Proceedings of the International Display Workshops  

We propose an image generation method to display three-dimensional (3D) images with high maximum pixel density and improved subjective quality on a two-dimensional image synthetic integral 3D display. In addition to the target light field image, weight maps obtained from the depth information were used to generate the images.

Scopus

Language：Japanese   Publisher：3次元画像コンファレンス実行委員会  

DOI： 10.60374/sanjigen.29.0_97

CiNii Research

Language：Japanese   Publisher：3次元画像コンファレンス実行委員会  

DOI： 10.60374/sanjigen.29.0_93

CiNii Research

Publisher：Proceedings - International Conference on Image Processing, ICIP  

Over the past decade, nonlinear image compression techniques based on neural networks have been rapidly developed to achieve more efficient storage and transmission of images compared with conventional linear techniques. A typical nonlinear technique is implemented as a neural network trained on a vast set of images, and the latent representation of a target image is transmitted. In contrast to the previous nonlinear techniques, we propose a new image compression method in which a neural network model is trained exclusively on a single target image, rather than a set of images. Such an overfitting strategy enables us to embed fine image features in not only the latent representation but also the network parameters, which helps reduce the reconstruction error against the target image. The effectiveness of our method is validated through a comparison with conventional image compression techniques in terms of a rate-distortion criterion.

DOI： 10.1109/ICIP42928.2021.9506367

Web of Science

Scopus

Publisher：Proceedings of SPIE - The International Society for Optical Engineering  

This paper attempts to improve denoising efficiency of BM3D technique for videos, i.e., VBM3D. VBM3D constructs 3D cubes from target video frames by a block matching algorithm that minimizes the residual matching error. However, such a cube formation results in sacrificing the pixel correlation in the temporal direction. This paper thus modifies this step to preserve the sub-pixel alignment, which makes the Fourier coefficients of each cube located on a vicinity of a certain plane in the 3-D Fourier domain. Then, SURE-shrinkage technique is separately applied to the inside and outside of the vicinity of the plane to denoise each cube. The experimental results given in this paper demonstrate the validity of our approach.

DOI： 10.1117/12.2591104

Web of Science

Scopus

Publisher：Proceedings - International Conference on Image Processing, ICIP  

A light field (LF), which is represented as a set of dense multiview images, has been utilized in various 3D applications. To make LF acquisition more efficient, researchers have investigated compressive sensing methods by incorporating modulation or coding functions into the camera. In this work, we investigate a challenging case of compressive LF acquisition in which an entire LF should be reconstructed from only a single coded image. To achieve this goal, we propose a new modulation scheme called factorized modulation that can approximate arbitrary 4D modulation patterns in a factorized manner. Our method can be hardwareimplemented by combining the architectures for coded aperture and pixelwise coded exposure imaging. The modulation pattern is jointly optimized with a CNNbased reconstruction algorithm. Our method is validated through extensive evaluations against other modulation schemes.

DOI： 10.1109/ICIP42928.2021.9506797

Web of Science

Scopus

Authorship：Lead author   Language：Japanese   Publishing type：Research paper (scientific journal)  

DOI： 10.3169/itej.74.198

Language：English   Publishing type：Research paper (scientific journal)  

DOI： 10.1109/access.2019.2913024

Authorship：Lead author   Language：English   Publishing type：Research paper (scientific journal)  

DOI： 10.1587/transinf.2017EDP7201

Authorship：Lead author   Language：English   Publishing type：Research paper (scientific journal)  

DOI： 10.1587/transinf.2015EDP7244