arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Heming Sun

125 papers · 2005–2026

2005: 12006: 02007: 02008: 02009: 02010: 02011: 02012: 22013: 12014: 42015: 42016: 42017: 32018: 42019: 102020: 142021: 142022: 212023: 162024: 122025: 102026: 5
2005peak 21 in 20222026

Frequent co-authors

Papers

showing 1–25 of 125 RSS

Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules

Zhengxue Cheng, Heming Sun, Masaru Takeuchi +1 more

eess.IV · · cited 1,363× · arXiv ↗ · more like this ·

Image compression is a fundamental research field and many well-known compression standards have been developed for many decades. Recently, learned compression methods exhibit a fast development trend with promising results. However, there is still a performance gap between learned compression algorithms and reigning compression standards, especially in terms of widely used PSNR metric. In this paper, we explore the remaining redundancy of recent learned compression algorithms. We have found accurate entropy models for rate estimation largely affect the optimization of network parameters and thus affect the rate-distortion performance. Therefore, in this paper, we propose to use discretized Gaussian Mixture Likelihoods to parameterize the distributions of latent codes, which can achieve a more accurate and flexible entropy model. Besides, we take advantage of recent attention modules…

Learned Image Compression with Mixed Transformer-CNN Architectures

Jinming Liu, Heming Sun, Jiro Katto

eess.IV · · cited 508× · arXiv ↗ · more like this ·

Learned image compression (LIC) methods have exhibited promising progress and superior rate-distortion performance compared with classical image compression standards. Most existing LIC methods are Convolutional Neural Networks-based (CNN-based) or Transformer-based, which have different advantages. Exploiting both advantages is a point worth exploring, which has two challenges: 1) how to effectively fuse the two methods? 2) how to achieve higher performance with a suitable complexity? In this paper, we propose an efficient parallel Transformer-CNN Mixture (TCM) block with a controllable complexity to incorporate the local modeling ability of CNN and the non-local modeling ability of transformers to improve the overall architecture of image compression models. Besides, inspired by the recent progress of entropy estimation models and attention modules, we propose a channel-wise entropy…

Deep Convolutional AutoEncoder-based Lossy Image Compression

Zhengxue Cheng, Heming Sun, Masaru Takeuchi +1 more

cs.CV · · cited 219× · arXiv ↗ · more like this ·

Image compression has been investigated as a fundamental research topic for many decades. Recently, deep learning has achieved great success in many computer vision tasks, and is gradually being used in image compression. In this paper, we present a lossy image compression architecture, which utilizes the advantages of convolutional autoencoder (CAE) to achieve a high coding efficiency. First, we design a novel CAE architecture to replace the conventional transforms and train this CAE using a rate-distortion loss function. Second, to generate a more energy-compact representation, we utilize the principal components analysis (PCA) to rotate the feature maps produced by the CAE, and then apply the quantization and entropy coder to generate the codes. Experimental results demonstrate that our method outperforms traditional image coding algorithms, by achieving a 13.7% BD-rate decrement on…

A Fast QTMT Partition Decision Strategy for VVC Intra Prediction

Yibo Fan, Junan Chen, Heming Sun +2 more

Signal Processing · IEEE Access · · cited 118× · DOI ↗ · more like this ·

Different from the traditional quaternary tree (QT) structure utilized in the previous generation video coding standard H.265/HEVC, a brand new partition structure named quadtree with nested multi-type tree (QTMT) is applied in the latest codec H.266/VVC. The introduction of QTMT brings in superior encoding performance at the cost of great time-consuming. Therefore, a fast intra partition algorithm based on variance and Sobel operator is proposed in this paper. The proposed method settles the novel asymmetrical partition issue in VVC by well balancing the reduction of computational complexity and the loss of encoding quality. To be more concrete, we first terminate further splitting of a coding unit (CU) when the texture of it is judged as smooth. Then, we use Sobel operator to extract gradient features to decide whether to split this CU by QT, thus terminating further MT partitions.…

Energy Compaction-Based Image Compression Using Convolutional AutoEncoder

Zhengxue Cheng, Heming Sun, Masaru Takeuchi +1 more

Computer Vision and Pattern Recognition · IEEE Transactions on Multimedia · · cited 96× · DOI ↗ · more like this ·

Image compression has been an important research topic for many decades. Recently, deep learning has achieved great success in many computer vision tasks, and its use in image compression has gradually been increasing. In this paper, we present an energy compaction-based image compression architecture using a convolutional autoencoder (CAE) to achieve high coding efficiency. Our main contributions include three aspects: 1) we propose a CAE architecture for image compression by decomposing it into several down(up)sampling operations; 2) for our CAE architecture, we offer a mathematical analysis on the energy compaction property and we are the first work to propose a normalized coding gain metric in neural networks, which can act as a measurement of compression capability; 3) based on the coding gain metric, we propose an energy compaction-based bit allocation method, which adds a…

Learning Image and Video Compression through Spatial-Temporal Energy Compaction

Zhengxue Cheng, Heming Sun, Masaru Takeuchi +1 more

eess.IV · · cited 87× · arXiv ↗ · more like this ·

Compression has been an important research topic for many decades, to produce a significant impact on data transmission and storage. Recent advances have shown a great potential of learning image and video compression. Inspired from related works, in this paper, we present an image compression architecture using a convolutional autoencoder, and then generalize image compression to video compression, by adding an interpolation loop into both encoder and decoder sides. Our basic idea is to realize spatial-temporal energy compaction in learning image and video compression. Thereby, we propose to add a spatial energy compaction-based penalty into loss function, to achieve higher image compression performance. Furthermore, based on temporal energy distribution, we propose to select the number of frames in one interpolation loop, adapting to the motion characteristics of video contents.…

Deep Residual Learning for Image Compression

Zhengxue Cheng, Heming Sun, Masaru Takeuchi +1 more

eess.IV · · cited 81× · arXiv ↗ · more like this ·

In this paper, we provide a detailed description on our approach designed for CVPR 2019 Workshop and Challenge on Learned Image Compression (CLIC). Our approach mainly consists of two proposals, i.e. deep residual learning for image compression and sub-pixel convolution as up-sampling operations. Experimental results have indicated that our approaches, Kattolab, Kattolabv2 and KattolabSSIM, achieve 0.972 in MS-SSIM at the rate constraint of 0.15bpp with moderate complexity during the validation phase.

Fast QTMT Partition Decision Algorithm in VVC Intra Coding based on Variance and Gradient

Junan Chen, Heming Sun, Jiro Katto +2 more

Signal Processing · · cited 70× · DOI ↗ · more like this ·

Quadtree with nested multi-type tree (QTMT) partition structure in Versatile Video Coding (VVC) contributes to superior encoding performance compared to the basic quad-tree (QT) structure in High Efficiency Video Coding (HEVC). However, the improvement of performance leads to an un-avoidable increase of computational complexity. To achieve a balance between coding efficiency and compression quality, we propose a fast intra partition algorithm based on variance and gradient to solve the rectangular partition problem in VVC. First, further splitting of smooth areas is terminated. Then, QT partition is chosen depending on the gradient features extracted by Sobel operator. Finally, one partition from five possible QTMT partitions is directly chosen by computing the variance of variance of sub-CUs. The theoretical basis of our method is that a homogeneous area tends to be predicted with a…

A Low-Complexity HEVC Intra Prediction Algorithm Based on Level and Mode Filtering

Heming Sun, Dajiang Zhou, Satoshi Goto

Signal Processing · · cited 53× · DOI ↗ · more like this ·

HEVC achieves a better coding efficiency relative to prior standards, but also involves increased complexity. For intra prediction, complexity is especially intensive due to a highly flexible coding unit structure and a large number of prediction modes. This paper presents a low-complexity intra prediction algorithm for HEVC. A fast preprocessing stage based on a simplified cost model is proposed. Based on its results, a level filtering scheme reduces the number of prediction unit levels that requires fine processing from 5 to 2. To supply level filtering decision with appropriate thresholds, a fast training method is also designed. A mode filtering scheme further reduces the maximum number of angular modes to be evaluated from 34 to 9. Complexity reduction from HM 3.0 is over 50% and stable for various sequences, which makes the proposed algorithm suitable for real-time applications.…

A Pipelined 2D Transform Architecture Supporting Mixed Block Sizes for the VVC Standard

Yibo Fan, Yixuan Zeng, Heming Sun +2 more

Signal Processing · IEEE Transactions on Circuits and Systems for Video Technology · · cited 46× · DOI ↗ · more like this ·

For the next-generation video coding standard Versatile Video Coding (VVC), several new contributions have been proposed to improve the coding efficiency, especially in the transformation operations. This paper proposes a unified $32\times 32$ block-based transform architecture for the VVC standard that enables 2D Discrete Sine Transform-VII (DST-VII) and Discrete Cosine Transform-VIII (DCT-VIII) of all sizes. It mainly gives three contributions: 1) The N-Dimensional Reduced Adder Graph (RAG-n) algorithm is adopted to design the minimal adder-oriented computational units. 2) The storage of the asymmetric transform units can be realized in the dual-port SRAM-based transpose memory. 3) The pipelined 2D transformations of mixed block sizes are achieved with the throughput rate of 32 samples per cycle. The synthesis results indicate that this architecture can reduce area by up to 73.1%…

Performance Comparison of Convolutional AutoEncoders, Generative Adversarial Networks and Super-Resolution for Image Compression

Zhengxue Cheng, Heming Sun, Masaru Takeuchi +1 more

eess.IV · · cited 38× · arXiv ↗ · more like this ·

Image compression has been investigated for many decades. Recently, deep learning approaches have achieved a great success in many computer vision tasks, and are gradually used in image compression. In this paper, we develop three overall compression architectures based on convolutional autoencoders (CAEs), generative adversarial networks (GANs) as well as super-resolution (SR), and present a comprehensive performance comparison. According to experimental results, CAEs achieve better coding efficiency than JPEG by extracting compact features. GANs show potential advantages on large compression ratio and high subjective quality reconstruction. Super-resolution achieves the best rate-distortion (RD) performance among them, which is comparable to BPG.

Approximate DCT Design for Video Encoding Based on Novel Truncation Scheme

Heming Sun, Zhengxue Cheng, Amir Masoud Gharehbaghi +2 more

Signal Processing · IEEE Transactions on Circuits and Systems I Regular Papers · · cited 36× · DOI ↗ · more like this ·

This paper presents an energy- and area-efficient architecture for approximated discrete cosine transform (DCT). Due to the good compression ability, DCT is widely exploited in signal processing. However, it is computationally intensive especially for large transform sizes. In this paper, we have reduced the computation cost of DCT by truncating a couple of least significant bits (LSB), most significant bits (MSB), and zero columns. First, considering that the contribution of LSBs is weakened because of the final right shift operation, we have eliminated the computation process for some LSBs. For the addition of the remaining LSBs, a parallel carry propagation adder is proposed to reduce the calculation latency. Second, owing to the phenomenon that high-frequency components are quite small in natural scenes, a couple of MSBs are selectively truncated according to their positions. Third…

Perceptual Quality Study on Deep Learning based Image Compression

Zhengxue Cheng, Pinar Akyazi, Heming Sun +2 more

eess.IV · · cited 29× · arXiv ↗ · more like this ·

Recently deep learning based image compression has made rapid advances with promising results based on objective quality metrics. However, a rigorous subjective quality evaluation on such compression schemes have rarely been reported. This paper aims at perceptual quality studies on learned compression. First, we build a general learned compression approach, and optimize the model. In total six compression algorithms are considered for this study. Then, we perform subjective quality tests in a controlled environment using high-resolution images. Results demonstrate learned compression optimized by MS-SSIM yields competitive results that approach the efficiency of state-of-the-art compression. The results obtained can provide a useful benchmark for future developments in learned image compression.

Q-LIC: Quantizing Learned Image Compression with Channel Splitting

Heming Sun, Lu Yu, Jiro Katto

eess.IV · · cited 28× · arXiv ↗ · more like this ·

Learned image compression (LIC) has reached a comparable coding gain with traditional hand-crafted methods such as VVC intra. However, the large network complexity prohibits the usage of LIC on resource-limited embedded systems. Network quantization is an efficient way to reduce the network burden. This paper presents a quantized LIC (QLIC) by channel splitting. First, we explore that the influence of quantization error to the reconstruction error is different for various channels. Second, we split the channels whose quantization has larger influence to the reconstruction error. After the splitting, the dynamic range of channels is reduced so that the quantization error can be reduced. Finally, we prune several channels to keep the number of overall channels as origin. By using the proposal, in the case of 8-bit quantization for weight and activation of both main and hyper path, we can…

14.7 A 4Gpixel/s 8/10b H.265/HEVC video decoder chip for 8K Ultra HD applications

Dajiang Zhou, Shihao Wang, Heming Sun +8 more

Signal Processing · · cited 27× · DOI ↗ · more like this ·

8K Ultra HD is being promoted as the next-generation digital video format. From a communication channel perspective, the latest high-efficiency video coding standard (H.265/HEVC) greatly enhances the feasibility of 8K by doubling the compression ratio. Implementation of such codecs is a challenge, owing to ultra-high throughput requirements and increased complexity per pixel. The former corresponds to up to 10b/pixel, 7680×4320pixels/frame and 120fps - 80× larger than 1080p HD. The latter comes from the new features of HEVC relative to its predecessor H.264/AVC. The most challenging of them is the enlarged and highly variable-size coding/prediction/transform units (CU/PU/TU), which significantly increase: 1) the requirement for on-chip memory as pipeline buffers, 2) the difficulty in maintianing pipeline utilization, and 3) the complexity of inverse transforms (IT). This paper presents…

Learned Image Compression with Fixed-point Arithmetic

Heming Sun, Lu Yu, Jiro Katto

Computer Vision and Pattern Recognition · · cited 26× · DOI ↗ · more like this ·

Learned image compression (LIC) has achieved superior coding performance than traditional image compression standards such as HEVC intra in terms of both PSNR and MS-SSIM. However, most LIC frameworks are based on floating-point arithmetic which has two potential problems. First is that using traditional 32-bit floating-point will consume huge memory and computational cost. Second is that the decoding might fail because of the floating-point error coming from different encoding/decoding platforms. To solve the above two problems. 1) We linearly quantize the weight in the main path to 8-bit fixed-point arithmetic, and propose a fine tuning scheme to reduce the coding loss caused by the quantization. Analysis transform and synthesis transform are fine tuned layer by layer. 2) We exploit look-up-table (LUT) for the cumulative distribution function (CDF) to avoid the floating-point error.…

Multistage Spatial Context Models for Learned Image Compression

Fangzheng Lin, Heming Sun, Jinming Liu +1 more

cs.CV · · cited 25× · arXiv ↗ · more like this ·

Recent state-of-the-art Learned Image Compression methods feature spatial context models, achieving great rate-distortion improvements over hyperprior methods. However, the autoregressive context model requires serial decoding, limiting runtime performance. The Checkerboard context model allows parallel decoding at a cost of reduced RD performance. We present a series of multistage spatial context models allowing both fast decoding and better RD performance. We split the latent space into square patches and decode serially within each patch while different patches are decoded in parallel. The proposed method features a comparable decoding speed to Checkerboard while reaching the RD performance of Autoregressive and even also outperforming Autoregressive. Inside each patch, the decoding order must be carefully decided as a bad order negatively impacts performance; therefore, we also…

An 8K H.265/HEVC Video Decoder Chip With a New System Pipeline Design

Dajiang Zhou, Shihao Wang, Heming Sun +8 more

Signal Processing · IEEE Journal of Solid-State Circuits · · cited 25× · DOI ↗ · more like this ·

8K ultra-HD is being promoted as the next-generation video specification. While the High Efficiency Video Coding (HEVC) standard greatly enhances the feasibility of 8K with a doubled compression ratio, its implementation is a challenge, owing to ultrahigh-throughput requirements and increased complexity per pixel. The latter comes from the new features of HEVC. At the system level, the most challenging of them is the enlarged and highly variable-size coding/prediction/transform units, which significantly increase the requirement for on-chip memory as pipeline buffers and the difficulty in maintaining pipeline utilization. This paper presents an HEVC decoder chip featuring a system pipeline that works at a nonunified and variable granularity. The pipeline saves on-chip memory with a novel block-in-block-out queue system and a parameter delivery network, while allowing overhead-free and…

A Reconfigurable Multiple Transform Selection Architecture for VVC

Zhijian Hao, Heming Sun, Guoqing Xiang +3 more

Signal Processing · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · · cited 22× · DOI ↗ · more like this ·

Video coding plays an important role in the highly information-based world as videos contribute the largest part of network traffic. The latest video coding standard Versatile Video Coding (VVC) introduces a new transform scheme multiple transform selection (MTS), which brings considerable coding gains at the expense of high coding complexity. In this article, we propose a reconfigurable MTS architecture that supports all transform types in VVC with square and rectangular sizes ranging from <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$4\times $ </tex-math></inline-formula> 4 to 32 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times32$ </tex-math></inline-formula> . Firstly, we explore the features of three…

Improving Multiple Machine Vision Tasks in the Compressed Domain

Jinming Liu, Heming Sun, Jiro Katto

Computer Vision and Pattern Recognition · 2022 26th International Conference on Pattern Recognition (ICPR) · · cited 22× · DOI ↗ · more like this ·

There is a growing number of images that are analyzed by machines rather than just humans. Recently, most machine vision tasks are based on decoded images which require an image compression (encoding/decoding) framework. However, using the decoded images in the pixel-domain has two drawbacks: 1) the complexity is high for the decoder part, 2) the accuracy (e.g., mIoU, mean absolute error, and average precision) of machine vision tasks will be degraded since decoded images only aim to optimize the human perceived quality (e.g., PSNR) so that information required for machine vision tasks will be lost during the decoding process. In this paper, we improve the machine vision tasks in the compressed domain. 1) A gate module is utilized to effectively select some compressed-domain features. 2) Knowledge distillation is introduced to improve the accuracy. 3) A training strategy is explored to…

End-to-end Learned Image Compression with Fixed Point Weight Quantization

Heming Sun, Zhengxue Cheng, Masaru Takeuchi +1 more

eess.IV · · cited 22× · arXiv ↗ · more like this ·

Learned image compression (LIC) has reached the traditional hand-crafted methods such as JPEG2000 and BPG in terms of the coding gain. However, the large model size of the network prohibits the usage of LIC on resource-limited embedded systems. This paper presents a LIC with 8-bit fixed-point weights. First, we quantize the weights in groups and propose a non-linear memory-free codebook. Second, we explore the optimal grouping and quantization scheme. Finally, we develop a novel weight clipping fine tuning scheme. Experimental results illustrate that the coding loss caused by the quantization is small, while around 75% model size can be reduced compared with the 32-bit floating-point anchor. As far as we know, this is the first work to explore and evaluate the LIC fully with fixed-point weights, and our proposed quantized LIC is able to outperform BPG in terms of MS-SSIM.

QA-Filter: A QP-Adaptive Convolutional Neural Network Filter for Video Coding

Chao Liu, Heming Sun, Jiro Katto +2 more

Signal Processing · IEEE Transactions on Image Processing · · cited 21× · DOI ↗ · more like this ·

Convolutional neural network (CNN)-based filters have achieved great success in video coding. However, in most previous works, individual models were needed for each quantization parameter (QP) band, which is impractical due to limited storage resources. To explore this, our work consists of two parts. First, we propose a frequency and spatial QP-adaptive mechanism (FSQAM), which can be directly applied to the (vanilla) convolution to help any CNN filter handle different quantization noise. From the frequency domain, a FQAM that introduces the quantization step (Qstep) into the convolution is proposed. When the quantization noise increases, the ability of the CNN filter to suppress noise improves. Moreover, SQAM is further designed to compensate for the FQAM from the spatial domain. Second, based on FSQAM, a QP-adaptive CNN filter called QA-Filter that can be used under a wide range of…

COUGH: A Challenge Dataset and Models for COVID-19 FAQ Retrieval

Xinliang Frederick Zhang, Heming Sun, Xiang Yue +2 more

cs.CL · · cited 21× · arXiv ↗ · more like this ·

We present a large, challenging dataset, COUGH, for COVID-19 FAQ retrieval. Similar to a standard FAQ dataset, COUGH consists of three parts: FAQ Bank, Query Bank and Relevance Set. The FAQ Bank contains ~16K FAQ items scraped from 55 credible websites (e.g., CDC and WHO). For evaluation, we introduce Query Bank and Relevance Set, where the former contains 1,236 human-paraphrased queries while the latter contains ~32 human-annotated FAQ items for each query. We analyze COUGH by testing different FAQ retrieval models built on top of BM25 and BERT, among which the best model achieves 48.8 under P@5, indicating a great challenge presented by COUGH and encouraging future research for further improvement. Our COUGH dataset is available at https://github.com/sunlab-osu/covid-faq.

Learning in Compressed Domain for Faster Machine Vision Tasks

Jinming Liu, Heming Sun, Jiro Katto

Computer Vision and Pattern Recognition · 2021 International Conference on Visual Communications and Image Processing (VCIP) · · cited 19× · DOI ↗ · more like this ·

Learned image compression (LIC) has illustrated good ability for reconstruction quality driven tasks (e.g. PSNR, MS-SSIM) and machine vision tasks such as image understanding. However, most LIC frameworks are based on pixel domain, which requires the decoding process. In this paper, we develop a learned compressed domain framework for machine vision tasks. 1) By sending the compressed latent representation directly to the task network, the decoding computation can be eliminated to reduce the complexity. 2) By sorting the latent channels by entropy, only selective channels will be transmitted to the task network, which can reduce the bitrate. As a result, compared with the traditional pixel domain methods, we can reduce about 1/3 multiply-add operations (MACs) and 1/5 inference time while keeping the same accuracy. Moreover, proposed channel selection can contribute to at most 6.8%…

Fast Algorithm and VLSI Architecture of Rate Distortion Optimization in H.265/HEVC

Heming Sun, Dajiang Zhou, Landan Hu +2 more

Signal Processing · IEEE Transactions on Multimedia · · cited 19× · DOI ↗ · more like this ·

In H.265/high efficiency video coding (HEVC) encoding, rate distortion optimization (RDO) is an important cost function for mode decision and coding structure decision. Despite being near-optimum in terms of coding efficiency, RDO suffers from a high complexity. To address this problem, this paper presents a fast RDO algorithm and its very large scale implementation (VLSI) for both intra- and inter-frame coding. The proposed algorithm employs a quantization-free framework that significantly reduces the complexity for rate and distortion optimization. Meanwhile, it maintains a low degradation of coding efficiency by taking the syntax element organization and probability model of HEVC into consideration. The algorithm is also designed with hardware architecture in mind to support an efficient VLSI implementation. When implemented in the HEVC test model, the proposed algorithm achieves 62%…