A7.2 Statistical testing of 2 D position error and TTFF for A-GPS and A-GNSS Minimum Performance test cases

3GPP51.010-1Mobile Station (MS) conformance specificationPart 1: Conformance specificationTS

A7.2.1 Test Method

Each test is performed in the following manner:

a) Setup the required test conditions.

b) Measure the 2D position and Time to First Fix repeated times. Start each repetition after having applied the message ‘RESET MS POSITIONING STORED INFORMATION’. This ensures that each result is independent from the previous one. The results, measured, are simplified to:

good result, if the 2D position and TTFF are ≤ limit.

bad result, if the 2D position or TTFF or both are > limit

c) Record the number of results (ns) and the number of bad results (ne)

d) Stop the test at a pass or an fail event.

e) Once the test is stopped, decide according to the pass fail decision rules ( A7.2.4.2)

A7.2.2 Error Ratio (ER)

The Error Ratio (ER) is defined as the ratio of bad results (ne) to all results (ns).

(1-ER is the success ratio)

A7.2.3 Test Design

A statistical test is characterised by:

Test-time, Selectivity and Confidence level

A7.2.3.1 Confidence level

The outcome of a statistical test is a decision. This decision may be correct or in-correct. The Confidence Level CL describes the probability that the decision is a correct one. The complement is the wrong decision probability (risk) D = 1-CL

A7.2.3.2 Introduction: Supplier Risk versus Customer Risk

There are two targets of decision:

a) A measurement on the pass-limit shows, that the DUT has the specified quality or is better with probability CL (CL e.g.95%) This shall lead to a “pass decision”

The pass-limit is on the good side of the specified DUT-quality. A more stringent CL (CL e.g.99%) shifts the pass-limit further into the good direction. Given that the quality of the DUTs is distributed, a greater CL passes less and better DUTs.

A measurement on the bad side of the pass-limit is simply “not pass” (undecided)

aa) Complementary:

A measurement on the fail-limit shows, that the DUT is worse than the specified quality with probability CL.

The fail-limit is on the bad side of the specified DUT-quality. A more stringent CL shifts the fail-limit further into the bad direction. Given that the quality of the DUTs is distributed, a greater CL fails less and worse DUTs.

A measurement on the good side of the fail-limit is simply “not fail”.

b) A DUT, known to have the specified quality, shall be measured and decided pass with probability CL. This leads to the pass limit.

For CL e.g. 95%, the pass limit is on the bad side of the specified DUT-quality. CL e.g.99% shifts the pass-limit further into the bad direction. Given that the DUT-quality is distributed, a greater CL passes more and worse DUTs.

bb) A DUT, known to be an (ε0) beyond the specified quality, shall be measured and decided fail with probability CL.

For CL e.g.95%, the fail limit is on the good side of the specified DUT-quality.

NOTE: the different sense for CL in (a), (aa) versus (b), (bb).

NOTE: for constant CL in all 4 bullets (a) is equivalent to (bb) and (aa) is equivalent to (b).

A7.2.3.3 Supplier Risk versus Customer Risk

The table below summarizes the different targets of decision.

Table A7.2.3.3 Equivalent statements

Equivalent statements, using different cause-to-effect-directions,

and assuming CL = constant >0.5

cause-to-effect-directions

Known measurement result  estimation of the DUT’s quality

Known DUT’s quality  estimation of the measurement’s outcome

Supplier Risk

A measurement on the pass-limit shows, that the DUT has the specified quality or is better

(a)

A DUT, known to have an (ε0) beyond the specified DUT-quality, shall be measured and decided fail

(bb)

Customer Risk

A measurement on the fail-limit shall shows, that the DUT is worse than the specified quality

(aa)

A DUT, known to have the specified quality, shall be measured and decided pass

(b)

NOTE: The bold text shows the obvious interpretation of Supplier Risk and Customer Risk.
The same statements can be based on other DUT-quality-definitions.

A7.2.3.4 Introduction: Standard test versus early decision concept

In standard statistical tests, a certain number of results (ns) is predefined in advance of the test. After ns results the number of bad results (ne) is counted and the error ratio (ER) is calculated as ne/ns.

Applying statistical theory, a decision limit can be designed, against which the calculated ER is compared to derive the decision. Such a limit is one decision point and is characterised by:

– D: the wrong decision probability (a predefined parameter)

– ns: the number of results (a fixed predefined parameter)

– ne: the number of bad results (the limit based on just ns)

In the formula for the limit, D and ns are parameters and ne is the variable. In the standard test ns and D are constant. The property of such a test is: It discriminate between two states only, depending on the test design:

– pass (with CL) / undecided (undecided in the sense: finally undecided)

– fail (with CL) / undecided (undecided in the sense: finally undecided)

– pass(with CL) / fail (with CL) (however against two limits).

In contrast to the standard statistical tests, the early decision concept predefines a set of (ne,ns) co-ordinates, representing the limit-curve for decision. After each result a preliminary ER is calculated and compared against the limit-curve. After each result one may make the decision or not (undecided for later decision) The parameters and variables in the limit-curve for the early decision concept have a similar but not equal meaning:

– D: the wrong decision probability (a predefined parameter)

– ns: the number of results (a variable parameter)

– ne: the number of bad results (the limit. It varies together with ns)

To avoid a “final undecided” in the standard test, a second limit must be introduced and the single decision co-ordinate (ne,ns) needs a high ne, leading to a fixed (high) test time. In the early decision concept, having the same selectivity and the same confidence level an “undecided” does not need to be avoided, as it can be decided later. A perfect DUT will hit the decision coordinate (ne,ns) with ne=0. This test time is short.

A7.2.3.5 Standard test versus early decision concept

For Supplier Risk:
The wrong decision probability D in the standard test is the probability, to decide a DUT in-correctly in the single decision point. In the early decision concept there is a probability of in-correct decisions d at each point of the limit-curve. The sum of all those wrong decision probabilities accumulate to D. Hence d<D

For Customer Risk:
The correct decision probability CL in the standard test is the probability, to decide a DUT correctly in the single decision point. In the early decision concept there is a probability of correct decisions cl at each point of the limit-curve. The sum of all those correct decision probabilities accumulate to CL. Hence cl<CL or d>D

A7.2.3.6 Selectivity

There is no statistical test which can discriminate between a limit-DUT-quality and a DUT-quality which is an (ε0) apart from the limit in finite time and confidence level CL>1/2. Either the test discriminates against one limit with the results pass (with CL)/undecided or fail (with CL)/undecided, or the test ends in a result pass (with CL)/fail (with CL) but this requires a second limit.

For CL>0.5, a (measurement-result = specified-DUT-quality), generates undecided in test “supplier risk against pass limit” (a in clause A7.2.3.2) and also in the equivalent test against the fail limit (aa in clause A7.2.3.2)

For CL>0.5, a DUT, known to be on the limit, will be decided pass for the test “customer risk against pass limit” (b in clause A7.2.3.2) and also in the equivalent test against fail limit (bb in clause A7.2.3.2).

This overlap or undecided area is not a fault or a contradiction, however it can be avoided by introducing a Bad or a Good DUT quality according to:

– Bad DUT quality: specified DUT-quality * M (M>1)

– Good DUT quality: specified DUT-quality * m (m<1)

Using e.g M>1 and CL=95% the test for different DUT qualities yield different pass probabilities:

Figure A7.2.3.6: Pass probability versus DUT quality

A7.2.3.7 Design of the test

The test is defined according to the following design principles:

1. The early decision concept is applied.

2. A second limit is introduced: Bad DUT factor M>1

3. To decide the test pass:

Supplier risk is applied based on the Bad DUT quality

To decide the test fail

Cusomer Risk is applied based on the specified DUT quality

The test is defined using the following parameters:

1. Specified DUT quality: ER = 0.05

2. Bad DUT quality: M=1.5 (selectivity)

3. Confidence level CL = 95% (for specified DUT and Bad DUT-quality)

This has the following consequences:

a) A measurement on the fail limit is connected with 2 equivalent statements:

A measurement on the fail-limit shows, that the DUT is worse than the specified DUT-quality

A DUT, known to have the specified quality, shall be measured and decided pass

A measurement on the pass limit is connected with the complementary statements:

A measurement on the pass limit shows, that the DUT is better than the Bad DUT-quality.

A DUT, known to have the Bad DUT quality, shall be measured and decided fail

The left column is used to decide the measurement.

The right column is used to verify the design of the test by simulation.

The simulation is based on the two fulcrums A and B only in Figure A7.2.3.6. There is freedom to shape the remainder of the function.

b) Test time

1. The minimum and maximum test time is fixed.

2. The average test time is a function of the DUT’s quality.

3. The individual test time is not predictable (except ideal DUT).

c) The number of decision co-ordinates (ne,ns) in the early decision concept is responsible for the selectivity of the test and the maximum test time. Having fixed the number of decision co-ordinates there is still freedom to select the individual decision co-ordinates in many combinations, all leading to the same confidence level.

A7.2.4 Pass fail decision

A7.2.4.1 Numerical definition of the pass fail limits

ne

nsp

nsf

ne

nsp

nsf

ne

nsp

nsf

ne

nsp

nsf

0

77

NA

43

855

576

86

1525

1297

129

2173

2050

1

106

NA

44

871

592

87

1540

1314

130

2188

2067

2

131

NA

45

887

608

88

1556

1331

131

2203

2085

3

154

NA

46

903

625

89

1571

1349

132

2218

2103

4

176

NA

47

919

641

90

1586

1366

133

2233

2121

5

197

NA

48

935

657

91

1601

1383

134

2248

2139

6

218

42

49

951

674

92

1617

1401

135

2263

2156

7

238

52

50

967

690

93

1632

1418

136

2277

2174

8

257

64

51

982

706

94

1647

1435

137

2292

2192

9

277

75

52

998

723

95

1662

1453

138

2307

2210

10

295

87

53

1014

739

96

1677

1470

139

2322

2227

11

314

100

54

1030

756

97

1692

1487

140

2337

2245

12

333

112

55

1046

772

98

1708

1505

141

2352

2263

13

351

125

56

1061

789

99

1723

1522

142

2367

2281

14

369

139

57

1077

805

100

1738

1540

143

2381

2299

15

387

152

58

1093

822

101

1753

1557

144

2396

2317

16

405

166

59

1108

839

102

1768

1574

145

2411

2335

17

422

180

60

1124

855

103

1783

1592

146

2426

2352

18

440

194

61

1140

872

104

1798

1609

147

2441

2370

19

457

208

62

1155

889

105

1813

1627

148

2456

2388

20

474

222

63

1171

906

106

1828

1644

149

2470

2406

21

492

237

64

1186

922

107

1844

1662

150

2485

2424

22

509

251

65

1202

939

108

1859

1679

151

2500

2442

23

526

266

66

1217

956

109

1874

1697

152

2515

2460

24

543

281

67

1233

973

110

1889

1714

153

2530

2478

25

560

295

68

1248

990

111

1904

1732

154

2544

2496

26

577

310

69

1264

1007

112

1919

1750

155

2559

2513

27

593

325

70

1279

1024

113

1934

1767

156

2574

2531

28

610

341

71

1295

1040

114

1949

1785

157

2589

2549

29

627

356

72

1310

1057

115

1964

1802

158

2603

2567

30

643

371

73

1326

1074

116

1979

1820

159

2618

2585

31

660

387

74

1341

1091

117

1994

1838

160

2633

2603

32

676

402

75

1357

1108

118

2009

1855

161

2648

2621

33

693

418

76

1372

1126

119

2024

1873

162

2662

2639

34

709

433

77

1387

1143

120

2039

1890

163

2677

2657

35

725

449

78

1403

1160

121

2054

1908

164

2692

2675

36

742

465

79

1418

1177

122

2069

1926

165

2707

2693

37

758

480

80

1433

1194

123

2084

1943

166

2721

2711

38

774

496

81

1449

1211

124

2099

1961

167

2736

2729

39

790

512

82

1464

1228

125

2114

1979

168

2751

2747

40

807

528

83

1479

1245

126

2128

1997

169

2765

NA

41

823

544

84

1495

1263

127

2143

2014

42

839

560

85

1510

1280

128

2158

2032

NOTE: The first column is the number of bad results (ne)
The second column is the number of results for the pass limit (nsp)
The third column is the number of results for the fail limit (nsf)

A7.2.4.2 Pass fail decision rules

Having observed 0 bad results, pass the test at ≥77 results, otherwise continue

Having observed 1 bad result, pass the test at ≥106 results, otherwise continue

Having observed 2 bad results, pass the test at ≥131 results, otherwise continue

etc. until

Having observed 6 bad results, pass the test at ≥218 results, fail the test at ≤ 42 results, otherwise continue

Having observed 7 bad results, pass the test at ≥238 results, fail the test at ≤ 52 results, otherwise continue

etc. until

Having observed 168 bad results, pass the test at ≥2751 results, fail the test at ≤2747 results, otherwise continue

Having observed 169 bad results, pass the test at ≥2765 results, otherwise fail

NOTE: an ideal DUT passes after 77 results. The maximum test time is 2765 results.

A7.2.4.3 Background information to the pass fail limits

There is freedom to design the decision co-ordinates (ne,ns).

The binomial distribution and its inverse is used to design the pass and fail limits. Note that this method is not unique and that other methods exist.

Where

fail(..) is the error ratio for the fail limit

pass(..) is the error ratio for the pass limit

ER is the specified error ratio 0.05

ne is the number of bad results. This is the variable in both equations

M is the Bad DUT factor M=1.5

df is the wrong decision probability of a single (ne,ns) co-ordinate for the fail limit.
It is found by simulation to be df = 0.004

clp is the confidence level of a single (ne,ns) co-ordinate for the pass limit.
It is found by simulation to be clp = 0.9975

qnbinom(..): The inverse cumulative function of the negative binomial distribution

The simulation works as follows:

A large population of limit DUTs with true ER = 0.05 is decided against the pass and fail limits.

clp and df are tuned such that CL (95%) of the population passes and D (5%) of the population fails.

A population of Bad DUTs with true ER = M*0.05 is decided against the same pass and fail limits.

clp and df are tuned such that CL (95%) of the population fails and D (5%) of the population passes.

This procedure and the relationship to the measurement is justified in clause A7.2.3.7. The number of DUTs decrease during the simulation, as the decided DUTs leave the population. That number decreases with an approximately exponential characteristics. After 169 bad results all DUTs of the population are decided.

NOTE: The exponential decrease of the population is an optimal design goal for the decision co-ordinates (ne,ns), which can be achieved with other formulas or methods as well.

Annex 8:
Void

Annex 9 (normative):
GAN certificate