7 Conformance Testing
26.4443GPPCodec for Enhanced Voice Services (EVS)Release 17Test sequencesTS
7.1 Bit-exact Conformance
For an implementation to be declared conformant according to the bit-exact conformance method, the encoder and decoder output sequences of the implementation shall match bit-exactly the test sequences provided in the corresponding ZIP files according to the attached Readme.txt file and in accordance with clause 6. This applies for all implementations based on TS 26.442 [7], TS 26.443 [8] or TS 26.452 [12]. For fixed-point implementations based on TS 26.442 [7] and TS 26.452 [12] this is the only conformance method.
7.2 Non-Bit-exact Conformance
7.2.1 Overview
If an implementation under test is based on the reference floating–point code (TS 26.443 [8]) and the output sequences are not bit-exact to the test sequences according to clause 6, the non bit-exact conformance testing process defined here shall be used to test the conformance.
A conformant floating-point implementation of the EVS codec shall be compliant to the reference specification in TS 26.443 [8] by implementing all the algorithmic steps of the EVS codec, further specified in 3GPP TS 26.445 (Detailed Algorithmic Description) [2], 3GPP TS 26.451 (Voice Activity Detection (VAD)) [3], 3GPP TS 26.449 (Comfort Noise Generation (CNG)) [4], 3GPP TS 26.450 (Discontinuous Transmission (DTX)) [5], 3GPP TS 26.447 (Packet Loss Concealment (PLC) of Lost Packets) [6], and 3GPP TS 26.446 (AMR-WB Interoperable Function) [10].
If a floating-point implementation uses the Jitter Buffer Management (JBM) according to TS 26.448 [11], the implementation shall be compliant to the reference specification in TS 26.443 [8] by implementing all the algorithmic steps of 3GPP TS 26.448 (Jitter Buffer Management (JBM)).
An implementation shall be tested for non-bit-exact conformance using three specific tests:
– Decoder test comparing the implementation decoder with TS 26.443 [8] decoder.
– Encoder test comparing the implementation encoder with TS 26.443 [8] encoder.
– MOS-LQO verification comparing the implementation with TS 26.442 [7] implementation.
All three tests shall pass for the implementation to be declared conformant. The tests are described in more details in Annex A.
Figure 7.1 shows the flow chart of the non bit-exact conformance process.
Figure 7.1: Non bit-exact conformance process
7.2.2 Running the Tests
The executables and scripts illustrating how to run the tests are included in the ZIP of the floating-point test sequences. Annex A provides more details on the tests. In the case of discrepancy between the procedure described in Annex A and the scripts provided in the ZIP file, the procedure of the scripts provided prevail.
For the encoder and decoder tests, instructions on how to operate the implementation under test to run the tests are contained in the text file testvec\Readme.md. The scripts require an additional tool, ResampAudio.exe (instructions on how to download it are contained in the Readme file). The decoder and encoder test scripts are run by executing two Bourne-Shell scripts:
– Readme_dec_snr.sh.txt: batch file describes how to run the decoder test
– Readme_enc_sh.txt: batch file describes how to run the encoder test
The reference files for SNR and MLD encoder and decoder tests are also included in the zip file.
For the MOS-LQO verification, instructions on how to operate the implementation under test to run the test are contained in the text file mos-lqo\README.md. The test script is run by executing the mos-lqo.py Python® script, which in turn executes several Bourne-Shell scripts. The test script requires additional tools:
– P.863 [16] implementation, compatible with version 2 of [16]
– CopyAudio.exe (instructions on how to download it are contained in the Readme file)
For the MOS-LQO test an additional database, based on [17], [13], [14] is used, as explained in Annex A.3.2.
The implementation will be declared conformant if all three tests (encoder, decoder, MOS-LQO) are passed.
Annex A: Tools Description (normative)