## A7.2 Statistical testing of 2 D position error and TTFF for A-GPS and A-GNSS Minimum Performance test cases

3GPP51.010-1Mobile Station (MS) conformance specificationPart 1: Conformance specificationTS

## A7.2.1 Test Method

Each test is performed in the following manner:

a) Setup the required test conditions.

b) Measure the 2D position and Time to First Fix repeated times. Start each repetition after having applied the message ‘RESET MS POSITIONING STORED INFORMATION’. This ensures that each result is independent from the previous one. The results, measured, are simplified to:

good result, if the 2D position and TTFF are ≤ limit.

bad result, if the 2D position or TTFF or both are > limit

c) Record the number of results (ns) and the number of bad results (ne)

d) Stop the test at a pass or an fail event.

e) Once the test is stopped, decide according to the pass fail decision rules ( A7.2.4.2)

## A7.2.2 Error Ratio (ER)

The Error Ratio (ER) is defined as the ratio of bad results (ne) to all results (ns).

(1-ER is the success ratio)

## A7.2.3 Test Design

A statistical test is characterised by:

Test-time, Selectivity and Confidence level

### A7.2.3.1 Confidence level

The outcome of a statistical test is a decision. This decision may be correct or in-correct. The Confidence Level CL describes the probability that the decision is a correct one. The complement is the wrong decision probability (risk) D = 1-CL

### A7.2.3.2 Introduction: Supplier Risk versus Customer Risk

There are two targets of decision:

a) A measurement on the pass-limit shows, that the DUT has the specified quality or is better with probability CL (CL e.g.95%) This shall lead to a “pass decision”

The pass-limit is on the good side of the specified DUT-quality. A more stringent CL (CL e.g.99%) shifts the pass-limit further into the good direction. Given that the quality of the DUTs is distributed, a greater CL passes less and better DUTs.

A measurement on the bad side of the pass-limit is simply “not pass” (undecided)

aa) Complementary:

A measurement on the fail-limit shows, that the DUT is worse than the specified quality with probability CL.

The fail-limit is on the bad side of the specified DUT-quality. A more stringent CL shifts the fail-limit further into the bad direction. Given that the quality of the DUTs is distributed, a greater CL fails less and worse DUTs.

A measurement on the good side of the fail-limit is simply “not fail”.

b) A DUT, known to have the specified quality, shall be measured and decided pass with probability CL. This leads to the pass limit.

For CL e.g. 95%, the pass limit is on the bad side of the specified DUT-quality. CL e.g.99% shifts the pass-limit further into the bad direction. Given that the DUT-quality is distributed, a greater CL passes more and worse DUTs.

bb) A DUT, known to be an (ε0) beyond the specified quality, shall be measured and decided fail with probability CL.

For CL e.g.95%, the fail limit is on the good side of the specified DUT-quality.

NOTE: the different sense for CL in (a), (aa) versus (b), (bb).

NOTE: for constant CL in all 4 bullets (a) is equivalent to (bb) and (aa) is equivalent to (b).

### A7.2.3.3 Supplier Risk versus Customer Risk

The table below summarizes the different targets of decision.

Table A7.2.3.3 Equivalent statements

 Equivalent statements, using different cause-to-effect-directions, and assuming CL = constant >0.5 cause-to-effect-directions Known measurement result  estimation of the DUT’s quality Known DUT’s quality  estimation of the measurement’s outcome Supplier Risk A measurement on the pass-limit shows, that the DUT has the specified quality or is better (a) A DUT, known to have an (ε0) beyond the specified DUT-quality, shall be measured and decided fail (bb) Customer Risk A measurement on the fail-limit shall shows, that the DUT is worse than the specified quality (aa) A DUT, known to have the specified quality, shall be measured and decided pass (b)

NOTE: The bold text shows the obvious interpretation of Supplier Risk and Customer Risk.
The same statements can be based on other DUT-quality-definitions.

### A7.2.3.4 Introduction: Standard test versus early decision concept

In standard statistical tests, a certain number of results (ns) is predefined in advance of the test. After ns results the number of bad results (ne) is counted and the error ratio (ER) is calculated as ne/ns.

Applying statistical theory, a decision limit can be designed, against which the calculated ER is compared to derive the decision. Such a limit is one decision point and is characterised by:

– D: the wrong decision probability (a predefined parameter)

– ns: the number of results (a fixed predefined parameter)

– ne: the number of bad results (the limit based on just ns)

In the formula for the limit, D and ns are parameters and ne is the variable. In the standard test ns and D are constant. The property of such a test is: It discriminate between two states only, depending on the test design:

– pass (with CL) / undecided (undecided in the sense: finally undecided)

– fail (with CL) / undecided (undecided in the sense: finally undecided)

– pass(with CL) / fail (with CL) (however against two limits).

In contrast to the standard statistical tests, the early decision concept predefines a set of (ne,ns) co-ordinates, representing the limit-curve for decision. After each result a preliminary ER is calculated and compared against the limit-curve. After each result one may make the decision or not (undecided for later decision) The parameters and variables in the limit-curve for the early decision concept have a similar but not equal meaning:

– D: the wrong decision probability (a predefined parameter)

– ns: the number of results (a variable parameter)

– ne: the number of bad results (the limit. It varies together with ns)

To avoid a “final undecided” in the standard test, a second limit must be introduced and the single decision co-ordinate (ne,ns) needs a high ne, leading to a fixed (high) test time. In the early decision concept, having the same selectivity and the same confidence level an “undecided” does not need to be avoided, as it can be decided later. A perfect DUT will hit the decision coordinate (ne,ns) with ne=0. This test time is short.

### A7.2.3.5 Standard test versus early decision concept

For Supplier Risk:
The wrong decision probability D in the standard test is the probability, to decide a DUT in-correctly in the single decision point. In the early decision concept there is a probability of in-correct decisions d at each point of the limit-curve. The sum of all those wrong decision probabilities accumulate to D. Hence d<D

For Customer Risk:
The correct decision probability CL in the standard test is the probability, to decide a DUT correctly in the single decision point. In the early decision concept there is a probability of correct decisions cl at each point of the limit-curve. The sum of all those correct decision probabilities accumulate to CL. Hence cl<CL or d>D

### A7.2.3.6 Selectivity

There is no statistical test which can discriminate between a limit-DUT-quality and a DUT-quality which is an (ε0) apart from the limit in finite time and confidence level CL>1/2. Either the test discriminates against one limit with the results pass (with CL)/undecided or fail (with CL)/undecided, or the test ends in a result pass (with CL)/fail (with CL) but this requires a second limit.

For CL>0.5, a (measurement-result = specified-DUT-quality), generates undecided in test “supplier risk against pass limit” (a in clause A7.2.3.2) and also in the equivalent test against the fail limit (aa in clause A7.2.3.2)

For CL>0.5, a DUT, known to be on the limit, will be decided pass for the test “customer risk against pass limit” (b in clause A7.2.3.2) and also in the equivalent test against fail limit (bb in clause A7.2.3.2).

This overlap or undecided area is not a fault or a contradiction, however it can be avoided by introducing a Bad or a Good DUT quality according to:

– Bad DUT quality: specified DUT-quality * M (M>1)

– Good DUT quality: specified DUT-quality * m (m<1)

Using e.g M>1 and CL=95% the test for different DUT qualities yield different pass probabilities:

Figure A7.2.3.6: Pass probability versus DUT quality

### A7.2.3.7 Design of the test

The test is defined according to the following design principles:

1. The early decision concept is applied.

2. A second limit is introduced: Bad DUT factor M>1

3. To decide the test pass:

Supplier risk is applied based on the Bad DUT quality

To decide the test fail

Cusomer Risk is applied based on the specified DUT quality

The test is defined using the following parameters:

1. Specified DUT quality: ER = 0.05

2. Bad DUT quality: M=1.5 (selectivity)

3. Confidence level CL = 95% (for specified DUT and Bad DUT-quality)

This has the following consequences:

a) A measurement on the fail limit is connected with 2 equivalent statements:

 A measurement on the fail-limit shows, that the DUT is worse than the specified DUT-quality A DUT, known to have the specified quality, shall be measured and decided pass

A measurement on the pass limit is connected with the complementary statements:

 A measurement on the pass limit shows, that the DUT is better than the Bad DUT-quality. A DUT, known to have the Bad DUT quality, shall be measured and decided fail

The left column is used to decide the measurement.

The right column is used to verify the design of the test by simulation.

The simulation is based on the two fulcrums A and B only in Figure A7.2.3.6. There is freedom to shape the remainder of the function.

b) Test time

1. The minimum and maximum test time is fixed.

2. The average test time is a function of the DUT’s quality.

3. The individual test time is not predictable (except ideal DUT).

c) The number of decision co-ordinates (ne,ns) in the early decision concept is responsible for the selectivity of the test and the maximum test time. Having fixed the number of decision co-ordinates there is still freedom to select the individual decision co-ordinates in many combinations, all leading to the same confidence level.

## A7.2.4 Pass fail decision

### A7.2.4.1 Numerical definition of the pass fail limits

 ne nsp nsf ne nsp nsf ne nsp nsf ne nsp nsf 0 77 NA 43 855 576 86 1525 1297 129 2173 2050 1 106 NA 44 871 592 87 1540 1314 130 2188 2067 2 131 NA 45 887 608 88 1556 1331 131 2203 2085 3 154 NA 46 903 625 89 1571 1349 132 2218 2103 4 176 NA 47 919 641 90 1586 1366 133 2233 2121 5 197 NA 48 935 657 91 1601 1383 134 2248 2139 6 218 42 49 951 674 92 1617 1401 135 2263 2156 7 238 52 50 967 690 93 1632 1418 136 2277 2174 8 257 64 51 982 706 94 1647 1435 137 2292 2192 9 277 75 52 998 723 95 1662 1453 138 2307 2210 10 295 87 53 1014 739 96 1677 1470 139 2322 2227 11 314 100 54 1030 756 97 1692 1487 140 2337 2245 12 333 112 55 1046 772 98 1708 1505 141 2352 2263 13 351 125 56 1061 789 99 1723 1522 142 2367 2281 14 369 139 57 1077 805 100 1738 1540 143 2381 2299 15 387 152 58 1093 822 101 1753 1557 144 2396 2317 16 405 166 59 1108 839 102 1768 1574 145 2411 2335 17 422 180 60 1124 855 103 1783 1592 146 2426 2352 18 440 194 61 1140 872 104 1798 1609 147 2441 2370 19 457 208 62 1155 889 105 1813 1627 148 2456 2388 20 474 222 63 1171 906 106 1828 1644 149 2470 2406 21 492 237 64 1186 922 107 1844 1662 150 2485 2424 22 509 251 65 1202 939 108 1859 1679 151 2500 2442 23 526 266 66 1217 956 109 1874 1697 152 2515 2460 24 543 281 67 1233 973 110 1889 1714 153 2530 2478 25 560 295 68 1248 990 111 1904 1732 154 2544 2496 26 577 310 69 1264 1007 112 1919 1750 155 2559 2513 27 593 325 70 1279 1024 113 1934 1767 156 2574 2531 28 610 341 71 1295 1040 114 1949 1785 157 2589 2549 29 627 356 72 1310 1057 115 1964 1802 158 2603 2567 30 643 371 73 1326 1074 116 1979 1820 159 2618 2585 31 660 387 74 1341 1091 117 1994 1838 160 2633 2603 32 676 402 75 1357 1108 118 2009 1855 161 2648 2621 33 693 418 76 1372 1126 119 2024 1873 162 2662 2639 34 709 433 77 1387 1143 120 2039 1890 163 2677 2657 35 725 449 78 1403 1160 121 2054 1908 164 2692 2675 36 742 465 79 1418 1177 122 2069 1926 165 2707 2693 37 758 480 80 1433 1194 123 2084 1943 166 2721 2711 38 774 496 81 1449 1211 124 2099 1961 167 2736 2729 39 790 512 82 1464 1228 125 2114 1979 168 2751 2747 40 807 528 83 1479 1245 126 2128 1997 169 2765 NA 41 823 544 84 1495 1263 127 2143 2014 42 839 560 85 1510 1280 128 2158 2032

NOTE: The first column is the number of bad results (ne)
The second column is the number of results for the pass limit (nsp)
The third column is the number of results for the fail limit (nsf)

### A7.2.4.2 Pass fail decision rules

Having observed 0 bad results, pass the test at ≥77 results, otherwise continue

Having observed 1 bad result, pass the test at ≥106 results, otherwise continue

Having observed 2 bad results, pass the test at ≥131 results, otherwise continue

etc. until

Having observed 6 bad results, pass the test at ≥218 results, fail the test at ≤ 42 results, otherwise continue

Having observed 7 bad results, pass the test at ≥238 results, fail the test at ≤ 52 results, otherwise continue

etc. until

Having observed 168 bad results, pass the test at ≥2751 results, fail the test at ≤2747 results, otherwise continue

Having observed 169 bad results, pass the test at ≥2765 results, otherwise fail

NOTE: an ideal DUT passes after 77 results. The maximum test time is 2765 results.

### A7.2.4.3 Background information to the pass fail limits

There is freedom to design the decision co-ordinates (ne,ns).

The binomial distribution and its inverse is used to design the pass and fail limits. Note that this method is not unique and that other methods exist.

Where

fail(..) is the error ratio for the fail limit

pass(..) is the error ratio for the pass limit

ER is the specified error ratio 0.05

ne is the number of bad results. This is the variable in both equations

M is the Bad DUT factor M=1.5

df is the wrong decision probability of a single (ne,ns) co-ordinate for the fail limit.
It is found by simulation to be df = 0.004

clp is the confidence level of a single (ne,ns) co-ordinate for the pass limit.
It is found by simulation to be clp = 0.9975

qnbinom(..): The inverse cumulative function of the negative binomial distribution

The simulation works as follows:

A large population of limit DUTs with true ER = 0.05 is decided against the pass and fail limits.

clp and df are tuned such that CL (95%) of the population passes and D (5%) of the population fails.

A population of Bad DUTs with true ER = M*0.05 is decided against the same pass and fail limits.

clp and df are tuned such that CL (95%) of the population fails and D (5%) of the population passes.

This procedure and the relationship to the measurement is justified in clause A7.2.3.7. The number of DUTs decrease during the simulation, as the decided DUTs leave the population. That number decreases with an approximately exponential characteristics. After 169 bad results all DUTs of the population are decided.

NOTE: The exponential decrease of the population is an optimal design goal for the decision co-ordinates (ne,ns), which can be achieved with other formulas or methods as well.

Annex 8:
Void

Annex 9 (normative):
GAN certificate