5 Test Methodologies for Immersive Audio Systems of TS 26.118 (Codec Quality Characterization Test)

26.2593GPPRelease 17Subjective test methodologies for the evaluation of immersive audio systemsTS

5.1 Introduction

This clause specifies the Codec Quality Characterization Test for the audio profiles in TS 26.118. The Codec Quality Characterization Test is based on the test method defined in [2]. The Codec Quality Characterization Test assesses the Basic Audio Quality attribute at different bit-rates for a given audio profile.

NOTE: The reference and hidden reference for the Codec Quality Characterization Test are rendered to the loudspeakers with the reference renderer of the audio profile under test. Because the reference renderer may include degradations to the immersive audio quality, including a reduction on the number of audio streams, care will be taken when evaluating the results.

5.2 Experimental Design

The experimental design of the Codec Quality Characterization Test is such that all assessors rate all Anchor/Reference and Test Conditions. To control for possible presentation order biases, the presentation order of the samples is fully randomized during the experiment (double-blind test). To minimize listener fatigue, the following constraints on the experimental design are defined:

– Each Test Material shall be no longer than 12 s in duration.

– No more than four Codec Operating Points shall be tested for each test material.

– Each experiment shall contain no more than 10 Test Materials.

5.3 Selection of Assessors

The selection of assessors shall follow the guidelines in [2] clause 4.1. Only experienced assessors shall participate in the experiment and the test administrator shall employ pre- and post-screening according to [2] clause 4.1. The final test results shall include assessments from at least 10 experienced assessors that have passed both pre- and post-screening.

5.4 Test Materials

Critical audio materials representing typical virtual reality content shall be used as Test Materials. Each test shall include at least 3 channel-based, 3 object-based and 3 scene-based Test Materials. In the event a Test Material is a hybrid format, the primary category to which the Test Material belongs to (distributed among channels, objects and scene-based) shall be indicated in the Test Report.

All Test Materials shall be provided as either 24-bit integer or 32-bit PCM float signals with a sampling rate of 48 kHz.

5.5 Content Presentation

The content presentation and grading process are according to [2] clauses 5.3 and 5.4.

5.6 Listening Environment

The listening environment should comply with [3] clauses 8.2 and 8.3.

5.7 Listening System

The listening system shall be loudspeaker-based. The loudspeaker layout is layout J described in [4] Annex 1.

5.8 Listening Level

The listening level is according to [2] clause 8. The listening level is adjusted with channel-based content.

5.9 Anchor/Reference Conditions

All Codec Quality Characterization Tests shall include one Hidden Reference and two Anchors. The two Anchors are 3.5kHz and 7kHz low-pass filtered versions of the Reference condition, as described in [2] clause 5.1.

The Reference and Hidden Reference conditions are the source test Materials rendered to the loudspeaker setup through the Reference Renderer of the Audio Profile under test with the coding bypassed.

5.10 Test Conditions

The Test Conditions shall be generated by encoding, decoding and rendering the test Materials with the target operating points of:

– 128 kbps (for First Order Ambisonics contents only)

– 256 kbps

– 384 kbps

– 512 kbps

A +/- 10% variation from the target operating points is acceptable. The actual bit-rate for each Test Condition shall be reported with an accompanying justification for the target operating point deviation. The renderer used for the Test Conditions shall be the same renderer used for the Anchor and Reference Conditions.

5.11 Attributes

The Codec Quality Characterization Test shall assess the Basic Audio Quality attribute described in [2] clause 6.4.

5.12 Test Report and Presentation of Results

The Test Report shall provide the Mean and 95 % Confidence Intervals (t-distribution) for each test Condition, Hidden Reference and Anchors. All results provided shall be post-screened results.