arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Gerhard Tech

26 papers · 2008–2024

2008: 12009: 22010: 52011: 12012: 52013: 12014: 12015: 12016: 12017: 02018: 12019: 02020: 02021: 42022: 22023: 02024: 1
2008peak 5 in 20102024

Frequent co-authors

Papers

showing 1–25 of 26 RSS

Overview of the Multiview and 3D Extensions of High Efficiency Video Coding

Gerhard Tech, Ying Chen, Karsten Müller +3 more

Computer Vision and Pattern Recognition · IEEE Transactions on Circuits and Systems for Video Technology · · cited 421× · DOI ↗ · more like this ·

The High Efficiency Video Coding (HEVC) standard has recently been extended to support efficient representation of multiview video and depth-based 3D video formats. The multiview extension, MV-HEVC, allows efficient coding of multiple camera views and associated auxiliary pictures, and can be implemented by reusing single-layer decoders without changing the block-level processing modules since block-level syntax and decoding processes remain unchanged. Bit rate savings compared with HEVC simulcast are achieved by enabling the use of inter-view references in motion-compensated prediction. The more advanced 3D video extension, 3D-HEVC, targets a coded representation consisting of multiple views and associated depth maps, as required for generating additional intermediate views in advanced 3D displays. Additional bit rate reduction compared with MV-HEVC is achieved by specifying new…

3D High-Efficiency Video Coding for Multi-View Video and Depth Data

Karsten Müller, Heiko Schwarz, Detlev Marpe +9 more

Signal Processing · IEEE Transactions on Image Processing · · cited 353× · DOI ↗ · more like this ·

This paper describes an extension of the high efficiency video coding (HEVC) standard for coding of multi-view video and depth data. In addition to the known concept of disparity-compensated prediction, inter-view motion parameter, and inter-view residual prediction for coding of the dependent video views are developed and integrated. Furthermore, for depth coding, new intra coding modes, a modified motion compensation and motion vector coding as well as the concept of motion parameter inheritance are part of the HEVC extension. A novel encoder control uses view synthesis optimization, which guarantees that high quality intermediate views can be generated based on the decoded data. The bitstream format supports the extraction of partial bitstreams, so that conventional 2D video, stereo video, and the full multi-view video plus depth format can be decoded from a single bitstream.…

3D video coding using the synthesized view distortion change

Gerhard Tech, Heiko Schwarz, Karsten Müller +1 more

Signal Processing · Fraunhofer-Publica (Fraunhofer-Gesellschaft) · · cited 82× · DOI ↗ · more like this ·

In 3D video, texture and supplementary depth data are coded to enable the interpolation of a required number of synthesized views for multi-view displays in the range of the original camera views. The coding of the depth data can be improved by analyzing the distortion of synthesized video views instead of the depth map distortion. Therefore, this paper introduces a new distortion metric for 3D video coding, which relates changes in the depth map directly to changes of the overall synthesized view distortion. It is shown how the new metric can be integrated into the rate-distortion optimization (RDO) process of an encoder, that is based on high-efficiency video coding technology. An evaluation of the modified encoder is conducted using different view synthesis algorithms and shows about 50% rate savings for the depth data or 0.6 dB PSNR gains for the synthesized view.

Mixed resolution coding of stereoscopic video for Mobile devices

Heribert Brust, Aljoša Smolić, Karsten Mueller +2 more

Signal Processing · Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) · · cited 64× · DOI ↗ · more like this ·

This paper presents results for full and mixed resolution stereo video coding including subjective and objective evaluation for 3DTV on mobile devices. For this, objective as well as subjective tests have been carried out with different sequences at different image sizes and evaluated on two different stereoscopic displays. The subjective tests showed, that for uncoded sequences full resolution was rated better, however for coded sequences at low bit rates mixed resolution was rated better. An unsharp masking algorithm for up-sampling the lower resolution view only partially improved the overall quality because coding artifacts were strengthened as well. Objective tests showed that the optimum bit rate distribution for mixed resolution stereo video coding is 30% to 35% of the total bit rate for the lower resolution view (half horizontal and half vertical). Finally, tests showed that…

3D video coding using advanced prediction, depth modeling, and encoder control methods

Heiko Schwarz, Christian Bartnik, Sebastian Bosse +9 more

Signal Processing · Fraunhofer-Publica (Fraunhofer-Gesellschaft) · · cited 48× · DOI ↗ · more like this ·

The presented approach for 3D video coding uses the multiview video plus depth format, in which a small number of video views as well as associated depth maps are coded. Based on the coded signals, additional views required for displaying the 3D video on an autostereoscopic display can be generated by depth image based rendering techniques. The developed coding scheme represents an extension of HEVC, similar to the MVC extension of H.264/AVC. However, in addition to the well-known disparity-compensated prediction advanced techniques for inter-view and inter-component prediction, the representation of depth blocks, and the encoder control for depth signals have been integrated. In comparison to simulcasting the different signals using HEVC, the proposed approach provides about 40% and 50% bit rate savings for the tested configurations with 2 and 3 views, respectively. Bit rate reductions…

3D video coding with depth modeling modes and view synthesis optimization

Karsten Müller, Philipp Merkle, Gerhard Tech +1 more

Signal Processing · Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) · · cited 42× · OpenAlex ↗ · more like this ·

This paper presents efficient coding tools for depth data in depth-enhanced video formats. The method is based on the high-efficiency video codec (HEVC). The developed tools include new depth modeling modes (DMMs), in particular using non-rectangular wedgelet and contour block partitions. As the depth data is used for synthesis of new video views, a specific 3D video encoder optimization is used. This view synthesis optimization (VSO) considers the exact local distortion in a synthesized intermediate video portion or image block for the depth map coding. In a fully optimized 3D-HEVC coder, VSO achieves average bit rate savings of 17%, while DMMs gain 6%in BD rate, even though the depth rate only contributes 10% to the overall MVD bit rate.

3D video formats and coding methods

Karsten Müller, Philipp Merkle, Gerhard Tech +1 more

Computer Vision and Pattern Recognition · Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) · · cited 23× · DOI ↗ · more like this ·

The introduction of first 3D systems for digital cinema and home entertainment is based on stereo technology. For efficiently supporting new display types, depth-enhanced formats and coding technology is required, as introduced in this overview paper. First, we discuss the necessity for a generic 3D video format, as the current state-of-the-art in multi-view video coding cannot support different types of multi-view displays at the same time. Therefore, a generic depth-enhanced 3D format is developed, where any number of views can be generated from one bit stream. This, however, requires a complex framework for 3D video, where not only the 3D format and new coding methods are investigated, but also view synthesis and the provision of high-quality depth maps, e.g. via depth estimation. We present this framework and discuss the interdependencies between the different modules.

Sharp, bright, three-dimensional: open profiling of quality for mobile 3DTV coding methods

Dominik Strohmeier, Gerhard Tech

Computer Vision and Pattern Recognition · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · · cited 22× · DOI ↗ · more like this ·

The choice of the right coding method is a critical factor in the development process of mobile 3D television and video. Several coding methods are available and each of these is based on a different approach. These differences lead to method specific artefacts - content and bit rate are as well important parameters for the performance. In our study,w e evaluated Simulcast,Multi View Coding,Mixed Resolution Stereo Coding and Video + Depth Coding. Therefore each method has been optimized at a high and a low bit rate using parameters typical for mobile devices. The goal of the study was to get knowledge about the optimum codign method for mobile 3DTV,but also to get knowledge about the underlying rationale of quality perception. We used Open Profiling of Quality (OPQ) for comparison. OPQ combines quantitative rating and sensory profiling of the content. This allowed us to get a preference…

Fast Partitioning for VVC Intra-Picture Encoding with a CNN Minimizing the Rate-Distortion-Time Cost

Gerhard Tech, Jonathan Pfaff, Heiko Schwarz +4 more

Computer Vision and Pattern Recognition · Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) · · cited 21× · DOI ↗ · more like this ·

This paper presents a CNN to reduce the encoding time of a VVC-based intra-picture encoder. For encoding a 32 × 32 block, the CNN estimates two partitioning parameters that restrict the allowed coding block width and height. To estimate them such that the encoder skips testing inefficient partitioning modes, we train the CNN as follows: First, we generate training data by encoding sequences without the CNN. While encoding, we test all combinations of the two parameters for each 32 × 32 block and store the resulting Lagrangian rate-distortion-time (RDT) cost. We use the recorded cost to derive the loss function when training the CNN. Consequently, the CNN is trained such that it minimizes the Lagrangian RDT cost. Our CNN reduces the encoding time by 50% with a bit rate increase of 0.9%, which outperforms existing CNN-based approaches. Our generic training approach could also be applied…

Extension of High Efficiency Video Coding (HEVC) for multiview video and depth data

Heiko Schwarz, Christian Bartnik, Sebastian Bosse +9 more

Signal Processing · Fraunhofer-Publica (Fraunhofer-Gesellschaft) · · cited 19× · DOI ↗ · more like this ·

This paper presents an approach for 3D video coding that uses a format in which a small number of views as well as associated depth maps are coded and transmitted. At the receiver side, additional views required for displaying the 3D video on an autostereoscopic display can be generated based on the corresponding decoded signals by using depth image based rendering (DIBR) techniques. In terms of coding technology, the proposed coding scheme represents an extension of High Efficiency Video Coding (HEVC), similar to the Multiview Coding (MVC) extension of H.264/AVC. Besides the well-known disparity-compensated prediction, advanced techniques for inter-view and inter-component prediction, the representation of depth blocks, and the encoder control for depth signals have been developed and integrated. In comparison to simulcasting the different signals using HEVC, the proposed approach…

Mixed resolution coding with inter view prediction for mobile 3DTV

Heribert Brust, Gerhard Tech, Karsten Mueller +1 more

Signal Processing · Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) · · cited 12× · DOI ↗ · more like this ·

This paper presents results on mixed resolution stereo video coding including inter view prediction. The objective tests show that coding one view of a stereo pair at a lower resolution outperforms coding of a low pass filtered view at base resolution. For both coding methods inter view prediction produces additional coding gains. Two different prediction approaches have been compared: One uses the lower resolution view as base view, which is used to predict the second view, the other uses the higher resolution view as base view. The prediction is realized by up-sampling or down-sampling the decoded base view before predicting. It turned out that equal results are achieved for low bit rates, while predicting from high resolution to low resolution view performs better for medium and high bit rates. During coding, different QP combinations for left and right view have been compared to…

Optimization and comparision of coding algorithms for mobile 3DTV

Gerhard Tech, Aljoša Smolić, Heribert Brust +5 more

Computer Vision and Pattern Recognition · Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) · · cited 12× · DOI ↗ · more like this ·

Different methods for coding of stereo video content for mobile 3DTV are examined and compared. These methods are H.264/MPEG-4 AVC simulcast transmission, H.264/MPEG-4 AVC Stereo SEI message, mixed resolution coding, and video plus depth coding using MPEG-C Part 3. The first two methods are based on a full left and right video (V+V) representation, the third method uses a full and a subsampled view and the fourth method is based on a one video plus associated depth (V+D) representation. Each method was optimized and tested using professional 3D video content. Subjective tests were carried out on a small size autostereoscopic display that is used in mobile devices. A comparison of the four methods at two different bitrates is presented. Results are provided in average subjective scoring, PSNR and VSSIM (Video Structure Similarity).

Partial Depth Image Based Re-Rendering for Synthesized View Distortion Computation

Gerhard Tech, Karsten Müller, Heiko Schwarz +1 more

Computer Vision and Pattern Recognition · IEEE Transactions on Circuits and Systems for Video Technology · · cited 11× · DOI ↗ · more like this ·

3D video systems transmit depth maps in order to render synthesized views (SVs) at a receiver. To anticipate this purpose when processing a depth map, a sender-side depth processing algorithm (DPA), e.g. a depth encoder, can also render the SVs, compute their SV distortion (SVD), and adapt to it. This requires a low-complexity algorithm as computational resources are usually limited. We propose such an algorithm in this paper. First, we discuss a measure that relates a depth change to an SVD change using rendering. Then, we present an optimized process combining basic rendering steps, as warping, occlusion handling, interpolation, hole filling, and blending. Furthermore, we analyze which parts of an SV are affected by a depth change and modify the process to re-render only them. The resulting algorithm is significantly less complex than an unoptimized rendering-based variant and…

Evaluation of view synthesis algorithms for mobile 3DTV

Gerhard Tech, Karsten Müller, Thomas Wiegand

Signal Processing · Fraunhofer-Publica (Fraunhofer-Gesellschaft) · · cited 10× · DOI ↗ · more like this ·

View synthesis algorithms suitable for mobile devices are presented and discussed. The focus is on low complexity rendering methods that enable line wise processing. A full pixel accurate rendering algorithm is compared to two algorithms using sub pixel accuracy: One sub pixel accurate method applies a linear interpolation, while the other method utilizes rendering in an up-sampled domain. An analysis of the rendered views and the corresponding stereo views is given. In the evaluation especially aspects on 3D impression, hole filling in disoccluded areas and flickering were investigated. Finally, optimized versions of the evaluated algorithms are presented and their computational complexities are discussed.

Adaptive Differential Filters for Fast and Communication-Efficient Federated Learning

Daniel Becking, Heiner Kirchhoffer, Gerhard Tech +4 more

cs.LG · · cited 8× · arXiv ↗ · more like this ·

Federated learning (FL) scenarios inherently generate a large communication overhead by frequently transmitting neural network updates between clients and server. To minimize the communication cost, introducing sparsity in conjunction with differential updates is a commonly used technique. However, sparse model updates can slow down convergence speed or unintentionally skip certain update aspects, e.g., learned features, if error accumulation is not properly addressed. In this work, we propose a new scaling method operating at the granularity of convolutional filters which 1) compensates for highly sparse updates in FL processes, 2) adapts the local models to new data domains by enhancing some features in the filter space while diminishing others and 3) motivates extra sparsity in updates and thus achieves higher compression ratios, i.e., savings in the overall data transfer. Compared…

Rate-Distortion-Time Cost Aware CNN Training for Fast VVC Intra-Picture Partitioning Decisions

Gerhard Tech, Jonathan Pfaff, Heiko Schwarz +4 more

Computer Vision and Pattern Recognition · Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) · · cited 8× · DOI ↗ · more like this ·

This paper presents a new method for fast VVC intra-picture encoding using a CNN. The CNN operates on the original samples of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{32}\times \mathbf{32}$</tex> blocks. Given a current block, it derives for each of the block's multi-type trees (MTTs), which are nested in quad-tree (QT) nodes, a parameter pair. The parameter pairs constrain the minimum width and height of the sub-blocks in their MTTs. This enables the CNN to control the number of tested MTT splits with fine granularity. To skip modes while maintaining the rate-distortion (RD) performance, we train the CNN considering the Lagrangian rate-distortion-time (RDT) cost caused by the derived parameters. First, we generate training data by encoding; when reaching a quad-tree node in a <tex…

CNN-based parameter selection for fast VVC intra-picture encoding

Gerhard Tech, Jonathan Pfaff, Heiko Schwarz +4 more

Computer Vision and Pattern Recognition · Fraunhofer-Publica (Fraunhofer-Gesellschaft) · · cited 6× · DOI ↗ · more like this ·

This paper presents two new methods for fast VVC intra-picture encoding. Both are based on an approach that uses a CNN for blockadaptive parameter estimation. The parameters restrict the multitype-tree (MTT) partitionings tested by the encoder. The methods aim for an improvement of the approach by further constraints with additional parameters. Adding parameters increases the time required for training data generation exponentially. This raises the question which parameters to add and how. To explore further partitioning restrictions, the first method adds parameters controlling the block sizes the MTT can start from. Although this leads to four parameters, we can exploit that some of their combinations are invalid. To investigate whether testing fewer prediction and transform modes is feasible, the second method adds a single parameter that restricts their number jointly. The paper…

Neural Network Coding of Difference Updates for Efficient Distributed Learning Communication

Daniel Becking, Karsten Müller, Paul Haase +6 more

Computer Vision and Pattern Recognition · IEEE Transactions on Multimedia · · cited 5× · DOI ↗ · more like this ·

Distributed learning requires a frequent communication of neural network update data. For this, we present a set of new compression tools, jointly called differential neural network coding (dNNC). dNNC is specifically tailored to efficiently code incremental neural network updates and includes tools for federated BatchNorm folding (FedBNF), structured and unstructured sparsification, tensor row skipping, quantization optimization and temporal adaptation for improved context-adaptive binary arithmetic coding (CABAC). Furthermore, dNNC provides a new parameter update tree (PUT) mechanism, which allows to identify updates for different neural network parameter sub-sets and their relationship in synchronous and asynchronous neural network communication scenarios. Most of these tools have been included into the standardization process of the NNC standard (ISO/IEC 15938-17) edition 2. We…

Block based Rate-Distortion analysis for quality improvement of synthesized views

Pablo Carballeira, Gerhard Tech, Julián Cabrera +4 more

Signal Processing · Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) · · cited 5× · DOI ↗ · more like this ·

We present a preliminary study on the Rate-Distortion (RD) gain that can be achieved applying RD optimization techniques in a multiview plus depth encoder. We consider the use of Multiview Video Coding (MVC) for both, color and depth sequences, and evaluate the improvement that can be obtained allowing a quantization parameter (QP) assignment on a macroblock basis compared to the use of a fixed QP for the whole sequence. The optimization criterion is the minimization of the distortion of the synthesized views generated at the receiver. Our motivation for this criterion is to capture the impact of depth coding according to its final purpose: the generation of virtual views. Since a unique objective quality metric for view synthesis artifacts evaluation has not been set yet, the performance of several algorithms for quality evaluation of the target synthesized view have been compared.…

Learning-based encoder algorithms for VVC in the context of the optimized VVenC implementation

Gerhard Tech, Valeri George, Jonathan Pfaff +5 more

Computer Vision and Pattern Recognition · Fraunhofer-Publica (Fraunhofer-Gesellschaft) · · cited 3× · DOI ↗ · more like this ·

Versatile Video Coding (VVC) is the most recent and efficient video-compression standard of ITU-T and ISO/IEC. It follows the principle of a hybrid, block-based video codec and offers a high flexibility to select a coded representation of a video. While encoders can exploit this flexibility for compression efficiency, designing algorithms for fast encoding becomes a challenging problem. This problem has recently been attacked with data-driven methods that train suitable neural networks to steer the encoder decisions. On the other hand, an optimized and fast VVC software implementation is provided by Fraunhofer’s Versatile Video Encoder VVenC. The goal of this paper is to investigate whether these two approaches can be combined. To this end, we exemplarily incorporate a recent CNN-based approach that showed its efficiency for intra-picture coding in the VVC reference software VTM to…

Synthesized View Distortion Based 3D Video Coding for Extrapolation and Interpolation of Views

Gerhard Tech, Heiko Schwarz, Karsten Müller +1 more

Signal Processing · Fraunhofer-Publica (Fraunhofer-Gesellschaft) · · cited 3× · DOI ↗ · more like this ·

In 3D video coding, the Multi-View Video plus Depth (MVD) representation format enables the interpolation of virtual intermediate views in between the original recorded camera views, as well as the extrapolation of views outside this range using view synthesis algorithms. For this, MVD includes depth maps as additional information to be transmitted. The performance of the associated depth data coding can be improved by considering the distortion of synthesized views in the rate-distortion optimization of the encoder. Therefore, this paper evaluates coding gains achieved with encoder optimization with respect to virtual view positions. The impact of the number and the positions of virtual views used as reference in the encoder side optimization process on the quality of the synthesized views is analyzed. Coding results for view interpolation and view extrapolation are provided. Moreover…

Diffusion filtering of depth maps in stereo video coding

Gerhard Tech, Karsten Müller, Thomas Wiegand

Computer Vision and Pattern Recognition · Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) · · cited 2× · DOI ↗ · more like this ·

A method for removing irrelevant information from depth maps in Video plus Depth coding is presented. The depth map is filtered in several iterations using a diffusional approach. In each iteration smoothing is carried out in local sample neighborhoods considering the distortion introduced to a rendered view. Smoothing is only applied when the rendered view is not affected. Therefore irrelevant edges and features in the depth map can be damped while the quality of the rendered view is retained. The processed depth maps can be coded at a reduced rate compared to unaltered data. Coding experiments show gains up to 0.5 dB for the rendered view at the same bit rate.

History Dependent Significance Coding for Incremental Neural Network Compression

Gerhard Tech, Paul Haase, Daniel Becking +7 more

Artificial Intelligence · 2022 IEEE International Conference on Image Processing (ICIP) · · cited 1× · DOI ↗ · more like this ·

This paper presents an improved probability estimation scheme for the entropy coder of Incremental Neural Network Coding (INNC), which is currently under standardization in ISO/IEC MPEG. More specifically, the paper first analyzes the compression performance of INNC and how the bitstream size relates to the neural network (NN) layers. For the layers requiring the most bits, it analyzes the coded NN weight updates and their temporal dependencies. Major finding is that the probability of a significant (i.e., non-zero) update for a weight can depend considerably on whether the weight has been updated before. Based on this finding, the paper proposes a new probability estimation scheme: Depending on whether a significant update has been received before (i.e., based on the weight’s history), the entropy coder models the probability for a current significant update differently. This scheme…

Improved Prediction Via Thresholding Transform Coefficients

Michael Schäfer, Jonathan Pfaff, Jennifer Rasch +6 more

Signal Processing · Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) · · cited 1× · DOI ↗ · more like this ·

This paper presents a thresholding method for processing the predicted samples in the state-of-the-art High Efficiency Video Coding (HEVC) standard. The method applies an integer-based approximation of the discrete cosine transform to an extended prediction block and sets transform coefficients beneath a certain threshold to zero. Transforming back into the sample domain yields the improved prediction signal. The method is incorporated into a software implementation that is conforming to the HEVC standard and applies to both intra and inter predictions. Consequently, bit-rate savings ranging from 2.3% to 8.0% have been measured in terms of the Bjøntegaard-Delta bit rate (BD-rate).

Adaptive kernel algorithm for FPGA-based speckle reduction

Gerhard Tech, R. Schwann, Goetz Kappen +2 more

Computer Vision and Pattern Recognition · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · · cited 1× · DOI ↗ · more like this ·

Image quality from ultrasound and optical coherence tomography (OCT) is degraded by speckle patterns, which limit the detection of small features and cause a loss of image contrast. To reduce speckle patterns, a novel adaptive kernel algorithm suitable for linearly scaled and log-compressed OCT and ultrasound images is presented. This algorithm combines region growing with a stick based approach. For each direction from the center of a square window to its border, a stick length is selected depending on a homogeneity criterion. Such a set of sticks forms an individual filter kernel for each image pixel. The current kernel size is observed to detect outliers in speckle. If the kernel size drops below a threshold, an outlier is assumed and the filter output is corrected by using a median filter in a second filtering stage. A new homogeneity model is presented that incorporates two…