246:
298:
219:
395:
307:
158:) with respect to an analyst-supplied background measure. Shannon relative information is used because it is scale invariant, tail insensitive, slow, and familiar. Parenthetically, measures with physical dimensions, such as the standard deviation, or the width of prediction intervals, raise serious problems, as a change of units (meters to kilometers) would affect some variables but not others. The product of statistical accuracy and informativeness for each expert is their
386:
413:
76:
120:
all steps in the expert elicitation process for scientific review. This made visible wide spreads in expert assessments and teed up questions regarding the validation and synthesis of expert judgments. The nuclear safety community later took onboard expert judgment techniques underpinned by external validation . Empirical validation is the hallmark of science, and forms the centerpiece of the classical model of
284:
those other experts who perform well? Data on this question is sparse, but in a few studies experts were asked to rate each other, and the âgroup mutual rankingsâ were negatively correlated the rankings in terms of performance . Reference compared performance of various weighting schemes, including âcitation weightsâ based on expertsâ citation numbers and found performance comparable to equal weighting.
230:
22:
494:, which becomes unmanageable. Recent research suggests that using 80% of the calibration variables for the training set is a good compromise of competing interests. Figure 6 shows the ratios of combined scores for PW / EW per study, aggregated over all splits with 80% of the calibration variables in the training set.
185:
is documented, and their names and affiliations are part of the reporting. However, to encourage candid judgments, individualsâ responses are not exchanged within the group and association of names with assessments is not reported in the open literature, but is preserved to enable peer review by the problem owner.
257:
The best argument for validation of expert judgment is the expert judgment data itself. Whereas the pre-2006 data contains wide variations in numbers of experts and numbers of calibration variables, the 33 independent professionally contracted post-2006 elicitations are more uniform in design, better
202:
reputed to have performed well in the past. Scientists and engineers, in contrast, are typically averse to any methodology which eschews empirical validation. Most invocations of expert judgment do not attempt any form of validation, as if the predicate âexpertâ were validation enough. The classical
180:
While some mathematicians and decision analysts regard combining expert judgments as a mathematical problem, the classical model regards expert combination as more akin to an engineering problem. A bicycle obeys Newton's Laws but does not follow from them. It is designed to optimize performance under
275:
There is however, a bright side: 93 of the 320 experts would not be rejected as statistical hypotheses at the 5% level. Figure 3 shows the statistical accuracy of the best expert and second best expert for each of the 33 studies. âBestâ is defined in terms of the individual's combined score, which
214:
Finding good calibration variables is difficult, and requires a deep dive into the subject matter at hand. The quality of the calibration, and the performance on calibration variables, buttresses the credibility of the whole study. At the end of the day, the problem owner will ask: âif expert A has
119:
Expert
Judgment (EJ) denotes a wide variety of techniques ranging from a single undocumented opinion, through preference surveys, to formal elicitation with external validation of expert probability assessments. Recent books are . In the nuclear safety area, Rasmussen formalized EJ by documenting
210:
predicted by their performance on âalmanac questionsâ. Experienced and inexperienced experts performed similarly on questions outside their field, but the experienced experts were much better on questions from their field. Hence, validation must be based on assessments of uncertain quantities from
184:
Real expert judgment studies differ in many ways from research or academic exercises. The experts are typically recruited in a traceable peer nomination process based on their knowledge of and engagement with the subject of the study; they may receive remuneration. In all cases, expertsâ reasoning
128:
coordinates workshops. Application areas include nuclear safety, investment banking, volcanology, public health, ecology, engineering, climate change and aeronautics/aerospace. For a survey of applications through 2006 see and give exhortatory overviews. A recent large scale implementation by the
283:
An oft heard suggestion is âWhy not ask the experts to weight each other?â They often know each other or each other's work, and they may well concur on whose opinion should weight heaviest. A moment's reflection councils caution, however. Could an expert with poor performance be able to identify
265:
In this summary, 227 of the 320 experts have a statistical accuracy score less than 0.05, which is the traditional rejection threshold for simple hypothesis testing. Half of the experts score below 0.005, and roughly one third fall into the abysmal range below 0.0001. These numbers challenge the
279:
25 of the 33 studies have at least one, and usually two or more experts whose statistical accuracy is acceptable. Simply identifying those experts and relying on them would be a big improvement over un-validated expert judgment (spotting good performers without measuring performance is a fool's
188:
Elicitations typically last several hours; the elicitation protocol is formalized and is part of the public reporting. Elicitation styles differ among practitioners, including face-to-face interviews, with or without plenary briefing and training, and "supervised plenary". Remote elicitation is
149:
or probability with which one would falsely reject the hypotheses that an expert's probability assessments were statistically accurate. A low value (near zero) means it is very unlikely that the discrepancy between an expert's probability statements and observed outcomes should arise by chance.
431:
Since the variables of interest are rarely observed within the time frame of the studies, out-of-sample validation mostly reduces to cross validation, whereby the model is initialized on a subset of the calibration variables (training set) and scored on the complimentary set (test set). The
162:. With an optimal choice of a statistical accuracy threshold beneath which experts are unweighted, the combined score is a long run âstrictly proper scoring ruleâ: an expert achieves his long run maximal expected score by and only by stating his true beliefs. The classical model derives
513:
places greater demands on the analyst, both with regard to generating meaningful calibration variables and explaining the methods and results. Finding competent and experienced analysts is the greatest bottleneck for applications. Refinement of performance measures, and improvements in
197:
Since experts are invoked when quantities of interest are uncertain, the goal of structured expert judgment is a defensible quantification of uncertainty. Confronted with uncertainty, society at large will always harken to prophets, oracles, pundits, blue ribbon panels,
261:
The p-values are sensitive to the power of the statistical test, and hence to the number of calibration variables. These numbers are roughly comparable for experts in the post-2006 data. Figure 2 shows the p-values of all post-2006 experts, arranged from best to worst.
215:
very good performance on the calibration variables, whereas expert B has very poor performance, am I going to ignore that difference?â If the owner's answer is âyesâ then the calibration variables have failed in their purpose and the effort has been for naught.
181:
constraints. Similarly expert judgment combination is viewed as a tool for enabling rational consensus by optimizing performance measures under mathematical and decision theoretic constraints. The theory of rational consensus is summarized in .
252:
P-values of best (blue diamond) and second best (red square) experts, in terms of combined score. Diamonds and squares on the same vertical line belong to the same study. The thick orange horizontal line denotes the traditional 5% rejection
885:
Aspinall, W.P.; Loughlin, S.C.; Michael, F.V.; Miller, A.D.; Norton, K.C.; Sparks, R.S.J.; Young, S.R. (2002). "The
Montserrat Volcano Observatory: its evolution, organisation, role and activities". In Druitt, T.H.; Kokelaar, B.P. (eds.).
203:
model's emphasis on validation is its distinguishing feature. Virtually all validation data with real experts and real applications (as opposed to academic exercises) has been generated by practitioners with the classical model.
258:
resourced, better documented and better lend themselves to aggregate presentation. The data comprise in total 320 experts. Figure 1 shows the distribution of experts over the number of assessed calibration variables.
1045:
Flandoli, F.; Giorgi, E.; Aspinall, W.P.; Neri, A. (2011). "Comparison of a new expert elicitation model with the
Classical Model, equal weights and single experts, using a cross-validation technique".
406:
Out of sample p-values (left) and combined scores (right) of PW and EW, aggregated over same sized training sets, as percentage of all calibration variables, and aggregated over the 33 post-2006 studies
336:. Averaging quantiles is easier to compute than averaging distributions, and is frequently employed by unwary practitioners. Averaging quantiles is mathematically equivalent to Harmonically Weighted (
924:
Wadge, G.; Aspinall, W.P. (2014). "A Review of
Volcanic Hazard and Risk Assessments at the Soufrière Hills Volcano, Montserrat from 1997 to 2011". In Wadge, G.; Robertson, R.E.A.; Voight, B. (eds.).
909:
Aspinall, W.P. (2006). "Structured elicitation of expert judgment for probabilistic hazard and risk assessment in volcanic eruptions". In Mader, H.M.; Coles, S.G.; Connor, C.B.; Connor, L.J. (eds.).
440:
of the real study. If the test set is small then the ability to resolve differences in combination schemes is small. That said, considered all splits of training/test sets, and showed that
129:
World Health
Organization is described in . A long running application at the Montserrat Volcano Observatory is described in . The classical model scores expert performance in terms of
834:
Hald, T.; Aspinall, W.P.; Devleesschauwer, B.; Cooke, R.M.; Corrigan, T.; Havelaar, A.H.; Gibb, H.; Torgerson, P.; Kirk, M.; Angulo, F.; Lake, R.; Speybroeck, N.; Hoffmann, S. (2015).
97:
836:"World Health Organization estimates of the relative contributions of food to the burden of disease due to selected foodborne hazards: a structured expert elicitation"
211:
the expertsâ field, to which we know, or will know, the true values within the time frame of the study. Such quantities are called âcalibrationâ or âseedâ variables.
276:
accounts for both statistical accuracy and informativeness, but is driven by statistical accuracy. The plot arrangement is from best to worst of the best experts.
1073:
Cooke, R.M.; Aspinall, W.P. (2013). "Quantifying scientific uncertainty from expert judgement elicitation". In Hill, L.; Rougier, J.C.; Sparks, R.S.J. (eds.).
332:
not superior in-sample there would be little point in conducting out-of-sample validation. suggest that averaging experts' quantiles might be superior to
432:
difficulty is in choosing the training/test set split. If the training set is small, then the ability to resolve expert performance is small and the
206:
One of the first studies with experienced and inexperienced experts showed that expert performance on questions from their field of expertise was
1160:
Eggstaff, J.W.; Mazzuchi, T.A.; Sarkani, S. (2014). "The Effect of the Number of Seed
Variables on the Performance of Cooke's Classical Model".
1088:
Cooke, R.M.; ElSaadany, S.; Xinzheng Huang, X. (2008). "On the
Performance of Social Network and Likelihood Based Expert Weighting Schemes".
1125:
Lichtendahl Jr., K.C.; Grushka-Cockayne, Y.; Winkler, R.L. (July 2013). "Is It Better to
Average Probabilities or Quantiles?".
525:
All mathematical details, an overview of post-2006 applications, recent publications and links to data are available online
590:
245:
88:
1226:
155:
145:â while precision âis a description of random errorsâ. In the classical model statistical accuracy is measured as the
51:
324:
Data used to gauge expert performance can also be used to measure performance of combination schemes. With regard to
619:
Cooke, R.M. (2012). "Uncertainty
Analysis Comes to Integrated Assessment Models for Climate ChangeâŚand Conversely".
394:
372:
is statistically accurate would be rejected at the 0.05 level. In 8 cases rejection would be at the 0.001 level.
199:
121:
328:
these are in-sample comparisons, as the validation data is also used to initialize the combination model. Were
969:
Cooke, R.M.; Wittmann, M.E.; Rothlisberger, J.D.; Rutherford, E.S.; Zhang, H.; Mason, D.; Lodge, D.M. (2014).
463:
as function of training set size. Both scores increase due to loss of statistical power in the test set, but
318:
Comparison of statistical accuracy (top) and combined scores (bottom) of 33 post-2006 expert judgment studies
297:
926:
The
Eruption of Soufrière Hills Volcano, Montserrat, from 2000 to 2010: Geological Society Memoirs, Vol. 39
385:
218:
1097:
890:
The eruption of Soufrière Hills Volcano, Montserrat, from 1995 to 1999. Geological Society, London, Memoir
138:
970:
306:
1189:"Expert forecasting with and without uncertainty quantification and weighting: What do the data say?"
1102:
785:"Evaluation of a Performance-Based Expert Elicitation: WHO Global Attribution of Foodborne Diseases"
576:
The Good Judgment Project: A Large Scale Test of Different Methods of Combining Expert Predictions
32:
1018:
Cooke, R.M. (2015). "Messaging climate change uncertainty (with supplementary Online Material)".
724:
488:
calibration variables the total number of splits (excluding the empty set and the entire set) is
151:
971:"Structured expert judgment to forecast species invasions: Bighead and Silver Carp in Lake Erie"
895:
591:"Reactor safety study. An assessment of accident risks in U. S. commercial nuclear power plants"
942:
Cooke, R.M.; Mendel, M.; Thijs, W. (1988). "Calibration and Information in Expert Resolution".
658:
534:
412:
1142:
1000:
867:
816:
765:
716:
1200:
1169:
1134:
1107:
1055:
1027:
990:
982:
951:
887:
857:
847:
806:
796:
755:
708:
681:
650:
628:
601:
541:) software for processing expert judgment data with the classical model. Freely available.
455:
statistical accuracy score. Figure 5 (left) shows out-of-sample statistical accuracy of
75:
888:
862:
835:
811:
784:
1220:
1205:
1188:
955:
728:
651:
172:
783:
Aspinall, W.P.; Cooke, R.M.; Havelaar, A.H.; Hoffmann, S.; Hald, T. (1 March 2016).
340:) combinations of distributions. Figure 4 shows the p-values and combined scores of
852:
801:
1173:
1111:
1059:
685:
229:
1146:
672:
Cooke, R.M.; Goossens, L.H.J. (2008). "TU Delft Expert Judgment Data Base".
41:
1138:
1004:
871:
820:
769:
720:
225:
Number of experts assessing number of calibration variables, post-2006 data
21:
995:
1031:
632:
538:
471:
is better able to resolve expert performance it approaches the in-sample
986:
236:
P-values of experts from post-2006 studies, arranged from best to worst
146:
125:
574:
Ungar, L.; Mellers, B.; Satopää, V.; Tetlock, P.; Tetlock, J. (2012).
653:
Experts in Uncertainty; Opinion and Subjective Probability in Science
760:
743:
712:
605:
189:
rarely used, but some recent studies use online face-to-face tools.
475:. The combined score (right) shows that out-of-sample dominance of
699:
Aspinall, W.P. (2010). "A route to more tractable expert advice".
411:
244:
228:
217:
266:
assumption that the predicate âexpertâ is a guarantee of quality
69:
15:
514:
cross-validation methods and software would also be welcome.
177:
combinations, as well as with individual expert assessments.
427:
for training sets sized at 80% of the calibration variables
368:
is best in 26 cases. In 18 cases (55%) the hypothesis that
522:
501:
is demonstrably superior to simple combination schemes (
91:, as style of writing is unsuitable for Knowledge (XXG).
93:
46:
36:
561:
Trusting judgments; How to get the best out of experts
657:. New York Oxford: Oxford University Press. pp.
563:. Cambridge: Cambridge University Press. p. 214.
352:
for the 33 post-2006 studies, arranged according to
1075:Risk and Uncertainty assessment in Natural Hazards
1187:Cooke, R.M; Marti, H.D.; Mazzuchi, T.A. (2021).
150:Informativeness is measured as Shannon relative
928:. London: Geological Society. pp. 439â456.
509:) that do not use performance data. .However,
1077:. Cambridge University Press. pp. 64â99.
8:
937:
935:
137:. These terms should not be confused with â
674:Reliability Engineering & System Safety
644:
642:
436:of each training set poorly resembles the
1204:
1162:Reliability Engineering and System Safety
1101:
1090:Reliability Engineering and System Safety
1048:Reliability Engineering and System Safety
994:
861:
851:
810:
800:
759:
269:with regard to uncertainty quantification
360:has the best combined score in 3 cases,
551:
742:Sutherland, W.J.; Burgman, M. (2015).
451:There is an out-of-sample penalty for
166:combinations. These are compared with
7:
1193:International Journal of Forecasting
578:. AAAI Fall Symposium Series (RSS).
467:increases faster. As out-of-sample
133:(sometimes called calibration) and
14:
913:. London: IAVCEI. pp. 15â30.
141:â. Accuracy âis a description of
87:to comply with Knowledge (XXG)'s
1206:10.1016/j.ijforecast.2020.06.007
393:
384:
305:
296:
170:combinations, and recently with
74:
20:
479:grows with training set size.
1:
419:Ratios of combined scores of
956:10.1016/0005-1098(88)90011-8
853:10.1371/journal.pone.0145839
802:10.1371/journal.pone.0149817
156:Kullback Leibler divergence
1243:
1174:10.1016/j.ress.2013.07.015
1112:10.1016/j.ress.2007.03.017
1060:10.1016/j.ress.2011.05.012
686:10.1016/j.ress.2007.03.005
911:Statistics in Volcanology
164:Performance Weighted (PW)
122:probabilistic forecasting
598:Wash-1400 (Nureg-75/014)
589:Rasmussen, N.C. (1975).
376:Out-of-sample validation
100:may contain suggestions.
85:may need to be rewritten
35:, as no other articles
1139:10.1287/mnsc.1120.1667
559:Burgman, M.A. (2016).
523:http://rogermcooke.net
497:Performance weighting
428:
254:
237:
226:
139:accuracy and precision
1020:Nature Climate Change
415:
248:
232:
221:
168:Equally Weighted (EW)
1032:10.1038/nclimate2466
975:Conservation Biology
744:"Use experts wisely"
649:Cooke, R.M. (1991).
633:10.1038/nclimate2466
288:In-sample validation
131:statistical accuracy
627:(3 2013): 467â479.
1227:Survey methodology
1127:Management Science
987:10.1111/cobi.12369
429:
255:
238:
227:
54:for suggestions.
44:to this page from
754:(7573): 317â318.
707:(7279): 657â674.
143:systematic errors
115:
114:
89:quality standards
68:
67:
1234:
1211:
1210:
1208:
1184:
1178:
1177:
1157:
1151:
1150:
1133:(7): 1594â1611.
1122:
1116:
1115:
1105:
1085:
1079:
1078:
1070:
1064:
1063:
1054:(7): 1292â1310.
1042:
1036:
1035:
1015:
1009:
1008:
998:
966:
960:
959:
939:
930:
929:
921:
915:
914:
906:
900:
899:
893:
882:
876:
875:
865:
855:
831:
825:
824:
814:
804:
780:
774:
773:
763:
739:
733:
732:
696:
690:
689:
669:
663:
662:
656:
646:
637:
636:
616:
610:
609:
595:
586:
580:
579:
571:
565:
564:
556:
493:
487:
397:
388:
309:
300:
126:European Network
110:
107:
101:
78:
70:
63:
60:
49:
47:related articles
24:
16:
1242:
1241:
1237:
1236:
1235:
1233:
1232:
1231:
1217:
1216:
1215:
1214:
1186:
1185:
1181:
1159:
1158:
1154:
1124:
1123:
1119:
1103:10.1.1.486.2435
1087:
1086:
1082:
1072:
1071:
1067:
1044:
1043:
1039:
1017:
1016:
1012:
968:
967:
963:
941:
940:
933:
923:
922:
918:
908:
907:
903:
884:
883:
879:
846:(1): e0145839.
833:
832:
828:
795:(3): e0149817.
782:
781:
777:
761:10.1038/526317a
741:
740:
736:
713:10.1038/463294a
698:
697:
693:
680:(93): 657â674.
671:
670:
666:
648:
647:
640:
621:Climatic Change
618:
617:
613:
606:10.2172/7134131
593:
588:
587:
583:
573:
572:
568:
558:
557:
553:
548:
531:
520:
492:
489:
486:
483:
410:
409:
408:
407:
400:
399:
398:
390:
389:
378:
322:
321:
320:
319:
312:
311:
310:
302:
301:
290:
243:
241:Validation data
195:
135:informativeness
111:
105:
102:
92:
79:
64:
58:
55:
45:
42:introduce links
25:
12:
11:
5:
1240:
1238:
1230:
1229:
1219:
1218:
1213:
1212:
1179:
1152:
1117:
1096:(5): 745â756.
1080:
1065:
1037:
1010:
996:2027.42/110571
981:(1): 187â197.
961:
931:
916:
901:
877:
826:
775:
734:
691:
664:
638:
611:
581:
566:
550:
549:
547:
544:
543:
542:
530:
527:
519:
516:
490:
484:
448:out-of-sample
402:
401:
392:
391:
383:
382:
381:
380:
379:
377:
374:
364:is best in 4.
314:
313:
304:
303:
295:
294:
293:
292:
291:
289:
286:
242:
239:
194:
191:
160:combined score
113:
112:
82:
80:
73:
66:
65:
52:Find link tool
28:
26:
19:
13:
10:
9:
6:
4:
3:
2:
1239:
1228:
1225:
1224:
1222:
1207:
1202:
1198:
1194:
1190:
1183:
1180:
1175:
1171:
1167:
1163:
1156:
1153:
1148:
1144:
1140:
1136:
1132:
1128:
1121:
1118:
1113:
1109:
1104:
1099:
1095:
1091:
1084:
1081:
1076:
1069:
1066:
1061:
1057:
1053:
1049:
1041:
1038:
1033:
1029:
1025:
1021:
1014:
1011:
1006:
1002:
997:
992:
988:
984:
980:
976:
972:
965:
962:
957:
953:
949:
945:
938:
936:
932:
927:
920:
917:
912:
905:
902:
897:
892:
891:
881:
878:
873:
869:
864:
859:
854:
849:
845:
841:
837:
830:
827:
822:
818:
813:
808:
803:
798:
794:
790:
786:
779:
776:
771:
767:
762:
757:
753:
749:
745:
738:
735:
730:
726:
722:
718:
714:
710:
706:
702:
695:
692:
687:
683:
679:
675:
668:
665:
660:
655:
654:
645:
643:
639:
634:
630:
626:
622:
615:
612:
607:
603:
599:
592:
585:
582:
577:
570:
567:
562:
555:
552:
545:
540:
536:
533:
532:
528:
526:
524:
517:
515:
512:
508:
504:
500:
495:
480:
478:
474:
470:
466:
462:
458:
454:
449:
447:
444:outperformed
443:
439:
435:
426:
422:
418:
414:
405:
396:
387:
375:
373:
371:
367:
363:
359:
355:
351:
347:
343:
339:
335:
331:
327:
317:
308:
299:
287:
285:
281:
277:
273:
271:
270:
263:
259:
251:
247:
240:
235:
231:
224:
220:
216:
212:
209:
204:
201:
193:Why validate?
192:
190:
186:
182:
178:
176:
175:Weighted (HW)
174:
169:
165:
161:
157:
153:
148:
144:
140:
136:
132:
127:
123:
117:
109:
99:
95:
90:
86:
83:This article
81:
77:
72:
71:
62:
59:November 2016
53:
48:
43:
39:
38:
34:
29:This article
27:
23:
18:
17:
1196:
1192:
1182:
1165:
1161:
1155:
1130:
1126:
1120:
1093:
1089:
1083:
1074:
1068:
1051:
1047:
1040:
1023:
1019:
1013:
978:
974:
964:
947:
943:
925:
919:
910:
904:
889:
880:
843:
839:
829:
792:
788:
778:
751:
747:
737:
704:
700:
694:
677:
673:
667:
652:
624:
620:
614:
597:
584:
575:
569:
560:
554:
521:
510:
506:
502:
498:
496:
481:
476:
472:
468:
464:
460:
456:
452:
450:
445:
441:
437:
433:
430:
424:
420:
416:
403:
369:
365:
361:
357:
353:
349:
345:
341:
337:
333:
329:
325:
323:
315:
282:
278:
274:
268:
267:
264:
260:
256:
249:
233:
222:
213:
207:
205:
200:crowd wisdom
196:
187:
183:
179:
173:Harmonically
171:
167:
163:
159:
142:
134:
130:
118:
116:
103:
94:You can help
84:
56:
30:
1199:: 378â387.
894:. pp.
152:information
944:Automatica
546:References
50:; try the
37:link to it
1168:: 72â82.
1147:1526-5501
1098:CiteSeerX
950:: 87â94.
729:205052636
535:EXCALIBUR
491:2−2
417:Figure 6:
280:errand .
253:threshold
106:July 2022
98:talk page
40:. Please
1221:Category
1026:: 8â10.
1005:25132396
872:26784029
840:PLOS ONE
821:26930595
789:PLOS ONE
770:26469026
721:20090733
529:Software
518:Websites
404:Figure5:
356:scores.
316:Figure4:
250:Figure3:
234:Figure2:
223:Figure1:
863:4718673
812:4773223
539:website
147:p-value
1145:
1100:
1003:
870:
860:
819:
809:
768:
748:Nature
727:
719:
701:Nature
348:, and
96:. The
33:orphan
31:is an
725:S2CID
594:(PDF)
482:With
423:over
1143:ISSN
1001:PMID
898:â92.
868:PMID
817:PMID
766:PMID
717:PMID
459:and
453:PWâs
154:(or
124:. A
1201:doi
1170:doi
1166:121
1135:doi
1108:doi
1056:doi
1028:doi
991:hdl
983:doi
952:doi
858:PMC
848:doi
807:PMC
797:doi
756:doi
752:526
709:doi
705:463
682:doi
678:117
659:321
629:doi
625:117
602:doi
208:not
1223::
1197:37
1195:.
1191:.
1164:.
1141:.
1131:59
1129:.
1106:.
1094:93
1092:.
1052:96
1050:.
1022:.
999:.
989:.
979:29
977:.
973:.
948:24
946:.
934:^
896:71
866:.
856:.
844:11
842:.
838:.
815:.
805:.
793:11
791:.
787:.
764:.
750:.
746:.
723:.
715:.
703:.
676:.
641:^
623:.
600:.
596:.
511:PW
507:HW
505:,
503:EW
499:PW
477:PW
473:PW
469:PW
465:PW
461:EW
457:PW
446:EW
442:PW
438:PW
434:PW
425:EW
421:PW
370:HW
366:PW
362:HW
358:EW
354:PW
350:HW
346:EW
344:,
342:PW
338:HW
334:EW
330:PW
326:PW
272:.
1209:.
1203::
1176:.
1172::
1149:.
1137::
1114:.
1110::
1062:.
1058::
1034:.
1030::
1024:5
1007:.
993::
985::
958:.
954::
874:.
850::
823:.
799::
772:.
758::
731:.
711::
688:.
684::
661:.
635:.
631::
608:.
604::
537:(
485:n
108:)
104:(
61:)
57:(
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.
