16 Summary of results from characterization Phase 1 and 2
3GPP46.008Half rate speechPerformance Characterization of the GSM Half Rate speech codecRelease 17TS
The whole set of individual and global data were extensively analysed and discussed within the TCH-HS expert group. The effects of different factors and their interactions were subject to analysis of variance (ANOVA). Tables 10 to 14 report the results obtained in 9 experiments.
16.1 Summary of Results From Characterization Phase 1
The whole set of individual and global data were extensively analysed and discussed within the TCH-HS expert group. The effects of different factors and their interactions were subject to analysis of variance (ANOVA). Tables 10 to 14 report the results obtained in 9 experiments.
Table 10: Summary of Characterization Phase 1 Results – Differential Q values
|
Diff Q (dB) |
|||
|
exc. (UPCM, No-IRS) Exp. 1 – 3 |
EP3 (A law-IRS and UPCM, No-IRS) Exp. 1 – 3 |
Noise only Exp. 4 and 5 |
All Exp. 1 – 5 |
|
-1,09 |
-1,45 |
-3,01 |
-2,29 |
NOTE: The figures indicate DQ values in dB averaged over input level,
where DQ = QHR – QFR.
Table 11: Summary of Characterization Phase 1 Results (Exp. 1, 2 and 3)
|
Single |
Encoding |
Conditions |
Tandeming |
Conditions |
All |
All |
||
|
Audio part |
EP0 |
EP0/1 |
EP0/1/2 |
EP0/1/2/3 |
EP0 |
EP0/1 |
exc. EP3 |
|
|
1.A-Law IRS |
+0,01 |
-0,32 |
-0,43 |
+0,12 |
-0,32 |
-0,34 |
-0,41 |
+0,02 |
|
2.NoIRS, LinearPCM |
-2,16 |
-2,13 |
-1,82 |
-0,90 |
-4,98 |
-4,50 |
-2,49 |
-1,62 |
|
1 and 2. |
-1,08 |
-1,22 |
-1,12 |
-0,39 |
-2,65 |
-2,42 |
-1,45 |
-0,80 |
NOTE: Dependence on Specific Conditions without Background Noise. The figures indicate DQ values in dB averaged over input level,
where DQ = QHR – QFR.
Table 12: Summary of Characterization Phase 1 Results (Exp. 4 and 5)-
|
Audio part |
Office Babble |
Vehicle |
Traffic |
No Tandeming |
With Tandeming |
All |
|
A-Law IRS |
-2,64 |
-3,19 |
-3,20 |
-2,10 |
-3,93 |
-3,01 |
NOTE: Differential Q values in Noise Conditions. The figures indicate DQ values in dB averaged over input level,
where DQ = QHR – QFR.
Table 13: Summary of Characterization Phase 1 Results – Significant differences
(Experiment 1 to Experiment 5)
|
Laboratory |
Experiment 1 |
Experiment 2 |
Experiment 3 |
Experiment 4 |
Experiment 5 |
|
BT |
HR = FR |
HR < FR |
HR < FR |
HR < FR |
x |
|
CNET |
x |
x |
x |
x |
HR < FR |
|
CSELT |
HR = FR |
HR = FR |
HR = FR |
x |
HR < FR |
|
Deutsche Telekom |
HR = FR |
HR = FR |
HR < FR |
HR < FR |
x |
|
Global |
HR = FR |
HR < FR |
HR < FR |
HR < FR |
HR < FR |
NOTE: See legend in subclause 16.2 for symbol explanation.
16.2 Summary of Results From Characterization Phase 2
Table 14: Summary of Characterization Phase 2 Results (Experiment 6 to Experiment 9)
|
Subject: |
qdu |
Tandeming with other Standards |
Talker Dependency |
DTX Functions |
|
Laboratory |
Experiment 6 |
Experiment 7 |
Experiment 8 |
Experiment 9 |
|
BT |
HR+any < any+HR |
see clause 9 |
DTX operation appears to be satisfactory. |
|
|
CNET |
HR+any < any+HR |
DTX fairly satisfactory, concerns over CNI. |
||
|
CSELT |
HR = FR HR+HR < FR+FR |
DTX satisfactory, concerns over CNI and comfort noise quality. |
||
|
DBP |
HR = FR HR+HR = FR+FR |
see clause 9 |
DTX fairly satisfactory, concerns over comfort noise quality. |
Legend
Symbol Definition
= no significant difference at the 95 % confidence level
HR Half rate codec
FR Full rate codec
x Experiment not performed by laboratory
HR<FR HR significantly worse than FR at the 95 % confidence level.
any All tested codecs, except HR (G.726 [2], G.728 [3], and FR)
The candidate codec performed equally well or slightly worse than the full rate for most cases, the overall figure of merit being -0,8 dB (weighted) signal-to-quantization distortion (without taking into account the noise conditions). The requirement was to provide a half rate standard with speech quality approximately equivalent to the GSM full rate codec with 1 dB of tolerance in terms of equivalent (weighted) signal-to-quantization distortion.
Under UPCM No IRS audio part conditions, particularly when tandemed, the full rate performed consistently better than the half-rate.
In environmental noise conditions, formal tests using two different methods, ACR and DCR were used to determine the difference in performance between the full and half-rate systems. It was found that differential Q (dB) values (comparing full and half-rate codecs) are more pronounced when using the DCR procedure than when using the ACR procedure, leading to a larger measured difference between the systems in the DCR experiments. The half-rate always performed worse than the full rate under the noise conditions, often with the difference in performance falling outside the -3 dB limit.
TCH-HS significantly improved the methodology for measuring subjectively the performance of candidate codec. Since the most important requirements set by SMG and tested by TCH-HS were met by the optimized algorithm, SMG approved the optimized codec.
Further information can be found in annex A reference 9).
16.3 Conclusion
A subjective test methodology for the quality assessment of ETSI’s half rate algorithm has been implemented, based on listening opinion tests.
The test methodology reflected international telephony assessment methods that are described extensively in the ITU-T Series P Recommendations, and that have shown to be suitable for characterizing both the GSM full rate and half rate algorithm performance. Results of tests conducted by several organizations showed consistency when normalized to Equivalent Q, in terms of the relative performance of the half rate algorithm and full rate RPE-LTP, removing the effect of differences in absolute performance, due to different languages, interpretation of quality scales, etc.
By considering the average performance across all countries, it was concluded that the half rate algorithm performance was comparable to RPE-LTP in all the experimental conditions tested, except for tandeming and background noise conditions, and met the initial requirements set out by SMG.
The results confirmed that the performance of the half rate codec falls short of that normally experienced on the PSTN.
For network planning purposes, it is proposed that the same rules will be adopted as for RPE-LTP, which will be adequate for most applications.
Annex A:
Bibliography
1) IEEE Proceedings. GLOBECOM’84, Vol. 2, pp.778-785, G. Williams and H. Suyderoud (1984): "Subjective performance evaluation of the 32-kbit/s ADPCM algorithm".
2) IEEE Proceedings GLOBECOM ’86, Houston, Texas, pp. 599-604, G. Modena, A. Coleman, P. Coverdale, P. Usai (1986): "Subjective performance evaluation of the 64 kbit/s (7 kHz) audio coder".
3) A. Coleman, N. Gleiss, P. Usai, Speech Communication, Vol. 7, , pp.151-166, (June 1988): "A subjective testing methodology for evaluating Medium Rate Codecs for Digital Mobile Radio applications".
4) IEEE Proceedings GLOBECOM ’89, Dallas, Texas, pp. 1075-1079, A. Coleman, N. Gleiss, J. Sotscheck, H.Scheuermann, P. Usai (1989): "Subjective performance evaluation of the RPE-LTP codec for the Pan-European cellular digital mobile radio system".
5) C. South et alii (1993), Speech Communication Vol. 13, pp.113-132: "Subjective Test Methodology for a 16 kbit/s Speech Coder".
6) ITU-T Recommendation P.80 (1992): "Methods for subjective determination of transmission quality".
7) ITU-T Recommendation P.81 (1988): "Modulated Noise Reference Unit",
8) ITU-T Recommendation P.83 (1992): "Subjective performance assessment oftelephone-band and wideband digital codecs"
9) ICC ’95, Session 2.2.2, P. Usai et alii (June 1995): "Subjective performance evaluation of the GSM Half Rate Speech Coding Algorithm (with voice signals)".
Annex B:
Change history
|
Change history |
|||||
|
SMG No. |
TDoc. No. |
CR. No. |
Section affected |
New version |
Subject/Comments |
|
SMG#16 |
4.0.0 |
ETSI Publication |
|||
|
SMG#20 |
5.0.0 |
Release 1996 version |
|||
|
SMG#27 |
6.0.0 |
Release 1997 version |
|||
|
SMG#29 |
7.0.0 |
Release 1998 version |
|||
|
SMG#31 |
8.0.0 |
Release 1999 version |
|||
|
Change history |
|||||||
|
Date |
TSG # |
TSG Doc. |
CR |
Rev |
Subject/Comment |
Old |
New |
|
03-2001 |
11 |
Version for Release 4 |
4.0.0 |
||||
|
06-2002 |
16 |
Version for Release 5 |
4.0.0 |
5.0.0 |
|||
|
12-2004 |
26 |
Version for Release 6 |
5.0.0 |
6.0.0 |
|||
|
06-2007 |
36 |
Version for Release 7 |
6.0.0 |
7.0.0 |
|||
|
12-2008 |
42 |
Version for Release 8 |
7.0.0 |
8.0.0 |
|||
|
12-2009 |
46 |
Version for Release 9 |
8.0.0 |
9.0.0 |
|||
|
03-2011 |
51 |
Version for Release 10 |
9.0.0 |
10.0.0 |
|||
|
09-2012 |
57 |
Version for Release 11 |
10.0.0 |
11.0.0 |
|||
|
10-2012 |
Replacement of figures 1 and 2 |
11.0.0 |
11.0.1 |
||||
|
09-2014 |
65 |
Version for Release 12 |
11.0.1 |
12.0.0 |
|||
|
12-2015 |
70 |
Version for Release 13 |
12.0.0 |
13.0.0 |
|||
|
Change history |
|||||||
|
Date |
Meeting |
TDoc |
CR |
Rev |
Cat |
Subject/Comment |
New version |
|
03-2017 |
SA#75 |
Version for Release 14 |
14.0.0 |
||||
|
06-2018 |
SA#80 |
– |
– |
– |
– |
Version for Release 15 |
15.0.0 |
|
2020-07 |
– |
– |
– |
– |
– |
Update to Rel-16 version (MCC) |
16.0.0 |
|
2022-04 |
– |
– |
– |
– |
– |
Update to Rel-17 version (MCC) |
17.0.0 |