AMT Adaptivní maticový test Version 27.00 Stru ná verze Mödling, b ezen 2008 Copyright © 1999 by SCHUHFRIED GmbH Auto i testu: L. F. Hornke, S. Etzel & K. Rettig Auto i manuálu: L. F. Hornke, S. Etzel; K. Rettig P eklad: S. Hoskovcová
OBSAH 1. 2.
STRUČNÝ POPIS METODY ............................................................... 3 OBSAHOVÝ POPIS METODY ............................................................. 5
2.1. 2.2.
Formy testu................................................................................................... 5 Popis proměnných........................................................................................ 5
3.
EVALUACE .................................................................................... 8
3.1. 3.2. 3.3.
4.
Objektivita..................................................................................................... 8 Možnost zkreslení výsledků.......................................................................... 8 Férovost ....................................................................................................... 8
ADMINISTRACE TESTU .................................................................... 9
4.1. 4.2.
Instrukce a a nácvik...................................................................................... 9 Testování...................................................................................................... 9
5.
INTERPRETACE VÝSLEDKŮ TESTU .................................................. 11
5.1. 5.2. 5.3. 5.4.
Interpretace – obecná doporučení.............................................................. 11 Interpretace – doporučení pro dopravně psychologickou diagnostiku ........ 11 Interpretace – hlavní proměnná AMT ......................................................... 11 Další zobrazení výsledků............................................................................ 12
6.
VYUŽITÍ V DOPRAVNÍ PSYCHOLOGII ................................................ 13
7.
LITERATURA ............................................................................... 16
2
1. STRUČNÝ POPIS METODY Autoři: Lutz F. Hornke, Stefan Etzel a Klaus Rettig ve spolupráci s Anjou Küppersovou Použití: AMT je neverbální metoda sloužící k měření obecné inteligence ve smyslu usuzování. AMT je vhodný pro osoby starší 14 let. Hlavní oblasti využití: personální psychologie, dopravní psychologie, letecká psychologie, pedagogická psychologie. Teoretické zázemí: Položky jsou podobné, jako u klasických maticových metod. Rozdíl spočívá v tom, že se při jejich konstrukci vycházelo z detailní analýzy kognitivních procesů, které probíhají při řešení tohoto typu úloh. Nejdříve bylo připraveno 289 položek, které byly prověřeny třemi velkými studiemi v Katovicích, Moskvě a Vídni na rozsáhlých vzorcích. Položková analýza byla provedena pomocí dichotomního pravděpodobnostního testového modelu podle Rasche (viz. Hornke, Küppers a Etzel, 2000). Výsledná databáze položek umožňuje jednak adaptivní zadávání testu se všemi přednostmi moderní počítačově podporované diagnostiky: doba testu je kratší, přesnost měření vyšší a motivace probandů vyšší díky výběru úloh přiměřených výkonu probanda. Administrace: Položky jsou vybírány adaptivně, to znamená, že proband dostane po úvodní fázi ke zpracování takové položky, které svou obtížností odpovídají jeho výkonnosti. Přeskočení úlohy nebo návrat k předchozímu zadání není možné. Výběr z osmi možností řešení snižuje pravděpodobnost, že proband správnou odpověď zvolí náhodně. Formy testu: Existují čtyři formy testu S1, S2, S3 a S11, které se liší podle přednastavené přesnosti (standardní chyba měření) odhadu parametrů testované osoby a také v obtížnosti startovních položek. Standardní chyba měření je nastavená u formy testu S1 na 0,63, u S2 na 0,44, u S3 na 0,39 a u S11 na 0,63 (to odpovídá reliabilitě 0,70, 0,83, 0,86 a 0,70). Vyhodnocení: Výsledkem testování je odhad „obecné inteligence“ probanda. Hodnota je vypočítána pomocí Raschova modelu a metody Maximum Likelihood. Dále je udáván percentil dosaženého výkonu ve vztahu k referenčnímu vzorku. Spolehlivost (reliabilita): Reliabilita ve smyslu vnitřní konzistence je dána na základě platnosti Raschova modelu. U čtyř forem testu je standardní chyba měření nastavená na 0,63, 0,44, 0,39 a 0,63. To odpovídá reliabilitě 0,70, 0,83, 0,86 a 0,70. Přesnost měření je platná pro všechny probandy na všech škálách; to je hlavní a rozhodující přednost oproti běžným psychometrickým testům podle klasické teorie testování: všichni probandi jsou posuzování se stejnou spolehlivostí!
3
Platnost (validita): Sommer a Arendasy (2005; Sommer, Arendasy & Häusler, 2005) potvrdili pomocí konfirmatorní faktorové analýzy, že tento test spolu s testy induktivního a deduktivního myšlení sytí faktor fluidní inteligence (Gf). Fluidní inteligence je navíc osvědčená jako faktor inteligence s nejvyšší výpovědní hodnotou o obecné inteligenci. Studie v oblasti dopravní a letecké psychologie prokazují kriteriální validitu metody. Normy: Normy jsou podloženy evaluačním vzorkem 1356 probandů, jakož i standardizačním vzorkem N=461 osob. Doba provedení: Podle zvolené formy testu je to 20 až 60 minut (včetně fáze instrukce a zácviku).
4
2. OBSAHOVÝ POPIS METODY 2.1.
FORMY TESTU
Pro účely různých cílů diagnostiky byly sestaveny tři standardizované formy testu AMT. Tyto se liší přesností měření a tím pádem i délkou testování. Forma testu S1: screening Pomocí této formy lze v krátkosti získat hrubý přehled o výkonnosti diagnostikované osoby. Standardní chyba měření je u této varianty =0,63. Test je podle dosavadních zkušeností ukončen v průměru po asi 13 úlohách. Forma testu S2: standard Nastavená standardní chyba měření je u této varianty = 0,44. V průměru proband zpracuje asi 23 úloh. Forma testu S3: precizní Pomocí této formy testu je možné učinit přesnější výpověď o skutečné výkonnosti diagnostikovaných osob. Tato forma testu se nabízí v případě, kdy očekáváme malé rozdíly mezi zkoumanými osobami nebo musíme roztřídit osoby, které mají velmi podobné výsledky. Standardní chyba měření je =0,39. Vyšší přesnost měření s sebou nese potřebu zpracovat větší počet úloh: v průměru jich proband zpracuje asi 30. Forma testu S11: krátká forma pro dopravně-psychologické vyšetření Podobně jako forma testu S1 umožňuje tato forma hrubý přehled o schopnostech vyšetřované osoby. Od formy S1 se liší tím, že dopravně-psychologická forma začíná snadnějšími položkami a obtížnost se stupňuje pomaleji. Standardní chyba měření = 0,63. Doba administrace testu je omezena na 20 minut.
2.2.
POPIS PROMĚNNÝCH
Hlavní proměnné Obecná inteligence Zjišťovaný parametr Ө můžeme chápat také jako z-skór. Přepočet na percentily je založen na standardizačním vzorku. Percentil vyjadřuje umístění probanda vzhledem k standardizačnímu vzorku. Vedlejší proměnné Počet zpracovaných položek Tato proměnná udává, kolik položek testovaná osoba zpracovala. Tento počet může být různý podle toho, jak se proband chová a podle konvergence odhadu algoritmu. Závisí na schopnosti
5
probanda: nadprůměrně resp. podprůměrně schopné osoby budou muset zpracovat více úloh, než průměrně schopný proband. U osob, které dokáží vyřešit všechny příklady nebo žádnou úlohu, se test po 10 úlohách přeruší. V takovém případě se za parametr považuje obtížnost položky nejtěžší resp. nejsnazší úlohy. Maximálně může být zadáno 35 úloh; pokud v tomto okamžiku ještě není dosaženo nastavené standardní chyby měření, pak se test ukončí. V rámci adaptivního testu není pro srovnávání osob počet zpracovaných položek vůbec vhodný. Platný je pouze odhadovaný osobní parametr Ө a příslušná standardní chyba měření resp. dosažený percentil! Doplňkové proměnné Doba zpracování Doba zpracování je pro celý test udávána v minutách a sekundách. V protokolu testu se navíc dokumentují doby zpracování jednotlivých položek. Na základě protokolu testu je možné zpětně identifikovat položky, při jejichž zpracování se proband odchýlil ze svého průměru a tyto je možno s probandem prodiskutovat.
6
Obraz 1: Protokol testu AMT V tomto příkladu je nápadné, že proband poměrně snadnou položku 10 (β=-1,440) zpracovával více jak dvě minuty. Kromě toho položku 11 viděl pouhých 7 sekund. V obou případech se ptáme, zda přitom proband usilovně přemýšlel nebo se stalo něco jiného, co by mohlo zpochybnit hodnotu testu. Takový pohled na věc nám testy ve verzi tužka-papír neumožní.
7
3. EVALUACE 3.1.
OBJEKTIVITA
Objektivita administrace Nezávislost na osobě administrátora je dána, pokud chování probanda během testu a tím pádem i výsledek testu nezávisí na náhodných nebo systematických variacích chování administrátora testu (Kubinger, 2003). Administrace AMT přes počítač zajišťuje všem probandům stejné zadání, které je nezávislé na administrátorovi testu. Objektivita vyhodnocení Registrace dat a výpočet proměnných, jakož i výpočet standardních skórů probíhá automaticky bez podílu lidského faktoru. Tím pádem můžeme vyloučit chyby ve výpočtech. Objektivita interpretace Vzhledem k tomu, že výkon probanda porovnáváme s normou, je objektivita interpretace zajištěná (Lienert & Raatz, 1994). Kvalita interpretace je také závislá na tom, s jakou pečlivostí budou dodržena doporučení kapitoly „Interpretace výsledku testu“.
3.2.
MOŽNOST ZKRESLENÍ VÝSLEDKŮ
Pokud je test odolný k záměrnému zkreslení výsledků, pak neumožňuje, aby proband určitou volbou odpovědí mohl ovlivnit resp. kontrolovat konkrétní výsledek testu (Kubinger. 2003). Podobně jako u všech výkonových testů není u AMT možné, aby proband záměrně dosahoval vyšších výsledků ve svůj prospěch. U testu typu multiple choice musíme počítat s tím, že proband zvolí správnou odpověď náhodně. Pravděpodobnost, že by se správná odpověď dala zvolit náhodně, je minimalizována nabídkou osmi možností odpovědi a navíc volbou „neznám odpověď“ – tato volba je hodnocená jako „chybná“ odpověď.
3.3.
FÉROVOST
Férový test nesmí systematicky diskriminovat určité skupiny probandů na základě jejich sociokulturního zázemí (Kubinger, 2003). Podle dosavadních zkušeností můžeme AMT označit za férový test: instrukce a fáze zácviku je dostatečná i pro osoby, které nemají zkušenost s počítači a potřebují si nacvičit zadávání odpovědí prostřednictvím počítače.
8
4. ADMINISTRACE TESTU AMT se má fázi instrukce (včetně fáze nácviku) a následnou vlastní fázi testování.
Obraz 2: Položka AMT použitá pro instrukci
4.1.
INSTRUKCE A NÁCVIK
Proband dostane informaci o základních aspektech ovládání Vienna Test System a seznámí se s vstupním zařízením podle své volby (klávesnice/myš/světelné pero/dotyková obrazovka) . Poté začne instrukce testu AMT. Během instrukce dostane proband prostřednictvím obrazovky vysvětlení, jak bude test probíhat. Pomocí několika tréninkových úloh se vysvětlí problém, který se bude řešit a způsob, jak zadávat odpověď. Přeskakování jednotlivých úloh není možné. Při chybné odpovědi je proband vyzván, aby znovu zvážil svou odpověď a zvolil správné řešení. Pokud proband na tři pokusy nezvolí u tréninkové položky správnou odpověď, fáze instrukce se přeruší; v takovém případě musí přiměřeně zasáhnout administrátor testu. Teprve poté, co proband dosáhl určitého počtu správných odpovědí, začne vlastní fáze testování.
4.2.
TESTOVÁNÍ
Jak již bylo popsáno výše, probíhá zadání testových úloh na základě adaptivní strategie testování. Výběr každé další položky se řídí podle aktuálního odhadu úrovně výkonu diagnostikované osoby. Odpovědi na položky volí proband mezi osmi nabízenými možnostmi. Pokud není schopen najít řešení úlohy, může zvolit pole s popiskem „neznám odpověď“. Tato odpověď je vyhodnocována jako „chybná odpověď“.
9
V testu se pokračuje až po dosažení určité standardní chyby měření, která se liší podle zvolené formy testu. Testování může být ve výjimečných případech přerušeno, pokud bylo prvních deset položek za sebou vyřešeno chybně, resp. všechny položky byly vyřešeny správně. V takových případech není možný přesný odhad výkonnosti probanda na základě chybějící variance dat. Jako osobní parametr se v takových případech zvolí obtížnost nejsnazší, resp. nejobtížnější položky z databáze položek. Dalším kritériem ukončení testu je pevně daný počet položek. U některých osob může kvůli extrémně výrazné schopnosti dojít k tomu, že pro jejich úroveň není k dispozici dostatek vhodných položek. To může vést k tomu, že nastavené standardní chyby měření nelze dosáhnout v rámci zadaných 35 položek nebo jejich menšího počtu. Test se pak z praktických důvodů ukončí s poněkud vyšší standardní chybou měření.
10
5. INTERPRETACE VÝSLEDKŮ TESTU 5.1.
INTERPRETACE – OBECNÁ DOPORUČENÍ
Celkově se dá říci, že výsledek v rozmezí 0. až 16. percentil je pro danou proměnnou výrazně podprůměrný. Osoba s takovým výsledkem je ve srovnání s referenčním vzorkem svým výkonem podprůměrná. 16. až 24. percentil lze považovat za mírně podprůměrný výsledek dané proměnné. Osoba s takovým výsledkem je ve srovnání s referenčním vzorkem svým výkonem podprůměrná až průměrná. Výsledný 25. až 75. percentil můžeme považovat za průměrný pro danou proměnnou. Výkon odpovídá v tomto případě výkonu většiny referenční populace 76. až 84. percentil vypovídá o mírně nadprůměrném výsledku proměnné. 84. a vyšší percentil ukazuje na výrazně nadprůměrný výsledek dané proměnné. Osoba s takovým výsledkem je ve srovnání s referenčním vzorkem svým výkonem nadprůměrná. Každý standardní skór se vztahuje k použitému referenčnímu vzorku.
5.2.
INTERPRETACE – DOPORUČENÍ PRO DOPRAVNĚ PSYCHOLOGICKOU DIAGNOSTIKU
V Rakousku a Německu jsou interpretační vodítka zakomponována do platných směrnic pro vydávání potvrzení o psychické způsobilosti pro řízení motorového vozidla a lze je nalézt v dokumentu Bundesanstalt für Straßenwesen, 2000, S. 16 Abschnitt 2.5.. Pro využití v ČR a SR se lze inspirovat v zařazení řidičů do skupin podle toho, zda se jedná o běžného řidiče nebo řidiče se zvýšenou zodpovědností. Skupina 1 – řidiči bez zvýšené zodpovědnosti - zahrnuje řidiče, kde je mezní hodnotou, pod kterou by proband neměl klesnout, 16. percentil. Pro skupinu 2 – řidič se zvýšenou zodpovědností - je mezní hodnotou 33. percentil. Podrobnější popis obou skupin řidičů naleznete v manuálu k testové baterii Expertní systém TRAFFIC.
5.3.
INTERPRETACE – HLAVNÍ PROMĚNNÁ AMT
Hlavní proměnná obecná inteligence představuje schopnost neverbálně induktivně usuzovat. Osoba, která dosáhla vysokých výsledků (PR>84.) v této proměnné, je obzvláště dobře schopná poznat zákonitosti, resp. pravidelnost a z toho odvozená pravidla. Osoby, které dosahují nadprůměrných percentilů, dokáží abstrahovat zákonitosti na základě své zkušenosti a vyvodit z toho důsledky pro své budoucí chování.
11
5.4.
DALŠÍ ZOBRAZENÍ VÝSLEDKŮ
Protokol testu Testovací protokol: Položka
Odpveď
Čas
1
2-
00:57
2
3-
3
ItS
PAR
VI
REL
LWk
-0.243
--
(-- ... --)
--
31%
00:26
-1.305
--
(-- ... --)
--
57%
5+
00:25
-1.937
-1.941
(-- ... --)
0.354
71%
4
8-
00:58
-1.930
-2.572
(-- ... --)
0.389
71%
5
4+
00:14
-2.574
-2.081
(-4.839 ... 0.678)
0.491
82%
6
6+
00:33
-2.213
-1.729
(-3.864 ... 0.406)
0.546
77%
7
8-
00:37
-1.784
-2.055
(-3.959 ... -0.151)
0.585
68%
8
2+
00:32
-1.908
-1.761
(-3.428 ... -0.094)
0.623
71%
9
4+
00:55
-1.788
-1.522
(-3.060 ... 0.016)
0.648
68%
10
4-
00:39
-1.528
-1.739
(-3.145 ... -0.333)
0.677
62%
11
7+
00:21
-1.825
-1.552
(-2.873 ... -0.231)
0.697
69%
12
4+
00:19
-1.565
-1.373
(-2.646 ... -0.099)
0.711
63%
13
8+
01:06
-1.381
-1.203
(-2.437 ... 0.030)
0.723
59%
14
2+
00:22
-1.197
-1.040
(-2.246 ... 0.166)
0.732
54%
15
3+
00:26
-1.039
-0.884
(-2.064 ... 0.296)
0.739
50%
16
4-
00:40
-0.867 (M)
-1.028
(-2.115 ... 0.059)
0.760
46%
Poznámka: Odpoveď: 1...8 = Zvolený obrázok, 9 = "Neviem riešenie", 0 = Na položku nebola uvedená žiadna odpoveď (pri prerušení testu); +: Správne riešenie, -: Nesprávne riešenie, doba: Doba spracovania v minútach:sekundách; ItS: Obtiažnosť položky (<0=ľahšia, >0=ťažšia;, (M )=motivačná položka); PAR: Aktuálne odhadnutý parameter osoby (<0=horší, >0=lepší, --=nie je možný žiadny odhad); Interval dôveryhodnosti (VI) udáva, v ktorej oblasti sa nachádza skutočný parameter výkonnosti s 5% pravdepodobnosťou omylu. Spoľahlivosť výsledkov testu (REL) je jednou z dolných hraníc merania presnosti A nachádza sa medzi hodnotami 0 (žiadna presnosť merania) a 1 (optimálna presnosť merania). LWk je individuálna pravdepodobnosť vyriešenia určitej úlohy.
Obraz 11 Protokol testu Z protokolu testu je možné vyčíst detailní informace o tom, jak byl test zpracován, jako například, jaká odpověď byla na kterou otázku zvolena, zda byla odpověď dobrá nebo chybná, kolik času proband potřeboval na zodpovězení otázky. To může být užitečné v případě, že chceme informaci, ve které fázi testu měl proband více problémů s řešení než jindy. Jako alternativu protokolu testu je možné získat výstup ve formě grafického diagramu adaptivního průběhu testu.
12
6. VYUŽITÍ V DOPRAVNÍ PSYCHOLOGII Forma testu S11 byla vytvořena zvláště pro použití v dopravní psychologii. Jak už bylo uvedeno, je S11 velmi ekonomickou formou testu pro provedení screeningu inteligence. Navíc je adaptivní algoritmus modifikován tak, aby se volbou položek nenavodil pocit vysoké náročnosti hned na začátku testu. S ohledem na věk a úroveň vzdělání se zvolí položka, kterou by proband podobného věku a úrovně vzdělání splnil správně asi na 75%. Tento postup má zajistit motivaci probanda, zvláště pokud je starší nebo má nízkou úroveň vzdělání. Proband nemá být frustrován nebo zúzkostněn hned na začátku testování. Další zvláštností formy testu S11 je, že umožňuje testování zaměřené na dopravně psychologické otázky. V AMT/S11 se tento požadavek realizuje tak, že v oknu „možnosti“ nastavíte, zda má test být optimalizován pro skupinu 1 (řidiči bez zvýšené odpovědnosti) resp. skupinu 2 (řidiči se zvýšenou zodpovědností). V obou případech trvá test tak dlouho, dokud se nepotvrdí, že výkon s vysokou statistickou pravděpodobností (95%) leží nad hraniční hodnotou relevantní pro dopravně psychologické vyšetření (IQ 70 resp. parametr -2,6 pro skupinu 1 a IQ 85 resp. parametr – 1,8 pro skupinu 2) nebo dokud se neuplatní další kritérium AMT pro přerušení testu.
Obraz 12: Okno pro nastavení mezních kritérií při dopravně psychologické diagnostice. Horní volba „vypnuto“ je přednastavená – nejsou nastavena žádná mezní kritéria pro dopravně psychologickou diagnostiku.
13
Adaptívny priebehový diagram: Parameter 1 0 -1 -2 -3 -4 -5
1
2 3 4 5 Správne zodpovedané položky
6
7
8
9
10
11
12
13
14
15
16
Položka
Nesprávne zodpovedané položky Odhadca pre osobné parametre Interval dôvery (5% pravdepodobnosť omylu)
Obraz 13: Zobrazení adaptivního průběhu části testování pomocí AMT. Konfidenční interval leží celý nad mezní hodnotou (v tomto případě IQ 70) a proband vyplnil 6 položek, což je minimální počet položek pro test. Kritérium je splněné a test se proto přeruší.
14
Pokud používáme test tímto způsobem, můžou další mezní kritéria – závisle na schopnostech probanda – vést k podstatnému zkrácení testování. Obraz 14 představuje průměrný čas testování, který je potřebný pro posouzení obecné inteligence řidiče skupiny 1.
Obraz 14: Očekávaná doba testování jako funkce obecné inteligence probanda při vyšetření řidiče skupiny 1. Pokud proband dosáhne výkonu na úrovni asi 30. percentilu, lze předpokládat dosažení stanoveného kritéria a testování se znatelně zkrátí. Při výkonech ležících pod 30. percentilem není možné předčasné rozhodování (ani s tím spojené přerušení testu). Test pokračuje, dokud není dosaženo cílové reliability. Tak je tomu zpravidla po asi 11 položkách.
15
7. LITERATURA Andrich, D. (1995). Review of the book Computerized adaptive testing: A primer. Psychometrika, 4, 615-620. Arendasy, M., Hornke, L. F., Sommer, M., Häusler, J., Wagner-Menghin, Gittler, Bognar & Wenzl, M. (2005). Manual Intelligenz-Struktur-Batterie (INSBAT). Mödling: SCHUHFRIED GmbH. Backhaus, K., Erichson, B., Plinke, W. & Weiber, R. (2004). Multivariate Analysemethoden: Eine anwendungsorientierte Einführung. Berlin: Springer. Byrne, B. M. (1989). A primer of LISREL. Basic applications and programming for confirmatory factor analytic models. New York: Springer. Hambleton, R.K. & Swaminathan, H. (1985). Item response theory - principles and applications. Boston: Kluwer-Nijhoff Publishing. Gustafsson, J. E. (1984). A unifying model for the structure of intellectual abilities. Intelligence, 8, 179-203. Häusler, J (2004). AdapSIM Software. Wien: Eigenverlag Heckhausen, H.. (1989). Motivation und Handeln. Berlin: Springer Hornke, L.F. (1976). Grundlagen und Probleme adaptiver Testverfahren. Frankfurt: Haag + Herchen. Hornke, L.F. (1993). Mögliche Einspareffekte beim computergestützten Testen. Diagnostica, 39, 109-119. Hornke, L.F. (1999). La prise de Décision Basée sur le Testing adaptif (DÉBAT). Psychologie et Psychométrie, 20, 181-192. Hornke, L.F. & Habon, M.W. (1984). Erfahrungen zur rationalen Konstruktion von Test-Items. Zeitschrift für Differentielle und Diagnostische Psychologie, 5, 203-212. Hornke, L.F. & Habon, M.W. (1986). Rule-based item bank construction and evaluation within the linear logistic framework. Applied Psychological Measurement, 10, 369-380. Hornke, L.F. & Rettig, K. (1988). Regelgeleitete Itemkonstruktion unter Zuhilfenahme kognitionspsychologischer Überlegungen. In K.D. Kubinger (Hrsg.). Moderne Testtheorie. Weinheim und München: Psychologie Verlagsunion. Hornke, L.F., Etzel, S. & Küppers, A. (2000). Konstruktion und Evaluation eines adaptiven Matrizentests. Diagnostica, 46, 182-188.
16
Kubinger, K. D. (2003). Gütekriterien. In K. D. Kubinger & R. S. Jäger (Hrsg.), Schlüsselbegriffe der psychologischen Diagnostik (S. 195-204). Weinheim: Psychologie Verlags Union. Lienert, G.A. (1989). Testanalyse und Testkonstruktion. Weinheim : Beltz Lienert, G.A., Raatz, U. (1994). Testaufbau und Testpraxis. Weinheim: Beltz. Rettig,
K. & Hornke, L.F. (1990). Adaptives Testen. Managementdiagnostik (S. 444-450). Göttingen: Hogrefe.
In
W.
Sarges
(Hrsg.),
Rost, J. (1996). Lehrbuch Testtheorie, Testkonstruktion. Bern: Huber. Sommer, M. & Arendasy, M. (2005). Theory-based construction and validation of a modern computerized intelligence test battery. Budapest: EAPA 2005 Abstracts. Sommer, M., Arendasy, M., Schuhfried, G. & Litzenberger, M. (2005). Diagnostische Unterscheidbarkeit unfallfreier und mehrfach unfallbelasteter Kraftfahrer mit Hilfe nichtlinearer Auswertemethoden. Zeitschrift für Verkehrssicherheit, 51, 82-86. Sommer, M., Arendasy, M., Hansen, H.-D., & Schuhfried, G. (2005). Personalauswahl mit Hilfe von statistischen Methoden der Urteilsbildung am Beispiel der Flugpsychologie. Untersuchungen des Psychologischen Dienstes der Bundeswehr, 40, 39-64. Sommer, M.; Häusler, J. (2004). Motivation Stabilizing Items in Computerized Adaptive Testing: Psychometric and Psychological Effects. Malaga: EAPA 2004 Abstracts Sympson, J.B. & Hetter, R.D. (1985). Controlling item exposure rates in computerized adaptive testing. Papers presented at the Annual Conference of the Military Testing Association. San Diego: Military Testing Association. Undheim, J. O. & Gustafsson, J. E. (1987). The hierarchical organisation of cognitive abilities: Restoring general intelligence through the use of linear structural relations (LISREL). Multivariate Behavioral Research, 22, 149-171.
17
COMPLETE VERSION OF THE MANUAL IN ENGLISH LANGUAGE
18
1. SUMMARY Authors: Lutz F. Hornke, Stefan Etzel and Klaus Rettig with the assistance of Anja Küpper.s Application: This AMT is a non-verbal test for assessing general intelligence as revealed in the ability to think inductively. It is suitable for subjects aged 13 and over. Main area of application: personnel selection and development, educational psychology, clinical and health psychology, neuropsychology, traffic psychology, aviation psychology, sport psychology. Theoretical background: The items resemble classical matrices, but in contrast to these they are constructed on the basis of explicit psychologically-based principles involving detailed analysis of the cognitive processes used in solving problems of this type. A total of 289 items were created and they were evaluated in three extensive studies involving large numbers of people in Katowice (Poland), Moscow and Vienna. The items were analysed using the Rasch dichotomous probabilistic test model and the corresponding characteristic values were estimated for the items (cf. Hornke, Küppers & Etzel, 2000).The resulting item pool means that the test can be presented adaptively and that it has all the advantages of modern computerized test procedures: shorter administration time but improved measurement precision, and high respondent motivation because the items presented are appropriate to the respondent’s ability. Administration: Items are presented adaptively – that is, after an initial phase the respondent is presented only with items of a level of difficulty which is appropriate to his ability. It is not possible to omit an item or to go back to a preceding one. The eight alternative answers to each question reduce the probability of successful guesswork. Test forms: There are four test forms S1, S2, S3 and S11; they differ in respect of the pre-set precision (standard measurement error) of the person parameter estimate and in the level of difficulty of the first item. The standard measurement error is set at 0.63 for test form S1, 0.44 for S2, 0.39 for S3 and 0.63 for S11 (corresponding to reliabilities of 0.70, 0.83, 0.86 and 0.70). Scoring: The test yields an estimate of the respondent’s general intelligence. The estimate is produced on the basis of the Rasch model according to the maximum likelihood method. A percentile ranking with reference to a norm sample is also given.
19
Reliability: Because of the validity of the Rasch model, reliability in the sense of internal consistency is given. For the four test forms it has been set at a standard measurement error (SEM) of 0.63, 0.44, 0.39 and 0.63, corresponding to reliabilities of 0.70, 0.83, 0.86 and 0.7. This reliability applies to all respondents and at all scale levels. This is the central and significant advantage over other widely-used psychometric tests based on classical test theory: all respondents are assessed with equal reliability. Validity: According to Hornke, Etzel and Küppers (2000; Hornke, 2002), the construction rational correlates at 0.72 with the difficulty parameters. In addition, Sommer and Arendasy (2005; Sommer, Arendasy & Häusler, 2005) demonstrating using a confirmatory factor analysis that this test, together with tests of inductive and deductive thinking, loads onto the factor of fluid intelligence (Gf). Fluid intelligence was found to be the intelligence factor with the highest gloading. A number of studies carried out in the fields of traffic and aviation psychology also confirm the test’s criterion validity. Norms: Norm data is available for an evaluation sample of N=1356 respondents and for a norm sample of N=461 respondents. Time required for the test: Between 20 and 60 minutes (including instruction and practice phase), depending on test form.
20
2. DESCRIPTION OF THE TEST 2.1.
THEORETICAL BACKGROUND
This AMT is innovative in two respects: the items are constructed on the basis of rules that take into account the insights of cognitive psychological research. administration of the test is adaptive – that is, information on item parameters is used to select from the large item pool and present to the subject only those items that will provide substantial information about him/her. (1) Construction of the items: The item format of the AMT that of classical matrix tasks. Each test item consists of a stimulus part made up of nine fields, which is displayed in the upper part of the screen. Eight of these fields contain geometric patterns, while the final field contains a question mark. The eight patterns stand in some particular relationship to each other. The respondent's task is to identify these relationships and “replace” the question mark logically (correctly) with one of the eight patterns (alternative answers) that are presented in the lower part of the screen (see Fig. 1). Each item has only one correct solution.
Fig. 1: Sample item from the AMT
In this case the shapes in each row are the same but the colours are different. No. 4 is therefore the correct solution. It requires the an identity operation and a variation operation, both of which are applied to a horizontal row. These tasks, in contrast to those of the classical matrix test, are based on an explicitly formulated set of rules. These rules were developed in response to published criticisms of existing matrix tests and detailed analysis of the cognitive processes involved in solving classical matrix items (Hornke & Habon in 1984, 1986; Hornke & Rettig, 1988). 266 items were then created for the AMT item pool.
21
(2) Test theory model: The construction of the AMT and practical administration of the test is based on the Rasch model (Rasch 1960; Hambleton & Swaminathan 1985, Rost, 1996). According to the Rasch model, the probability that a respondent i with ability θi will give a correct response (Xij=1) to item j with difficulty βj for is given by . . . . . . . .exp(θi - βj) . .P(Xij=1) =.--------------------. . . . . . . . . . . . . .1 + exp(θi - βj) If the item difficulties are know, the test can be administered adaptively. There are many possible ways of formulating the test algorithm (Rettig & Hornke 1990). The following section describes the procedure implemented in the AMT (see Fig.1). INITIAL PHASE: The first item presented is normally one of medium difficulty, since at this stage information about the respondent’s ability is normally not available. An item of medium difficulty thus represents the most appropriate challenge for the respondent. The answer to the initial item is recorded and scored. It is not (yet) possible to make a definitive estimate of the person paramter by the maximum likelihood method used here if only one answer or only correct or incorrect answers have been given. Further items are presented until at least one correct and one incorrect answer have been given. Until then, and starting from the initial item of medium difficulty, the test proceeds by working either upwards or downwards through the item pool, with the difficulty level varying by a constant amount at each stage. In the highly unlikely event that after 10 items the respondent’s anwers are all correct or all incorrect, the test is stopped. The difficulty of the most difficult or the easiest item used is then taken as the estimate of the person parameter. MAIN PHASE: If both correct and incorrect answers have been given, the individual’s ability score can be calculated. The procedure is as follows: After each further item the individual ability θ is estimated by the maximum likelihood method (ML) from all the k answers given up to that point. This is done by maximising the likelihood function with regard to θ. . . k. . xi. . 1-xi . . Π ( Pi. (1-Pi). . ) . . i=1 . The technical details of this process are described by Hambleton and Swaminathan (1985). In addition, the standard error of measurement (SEM) is determined for this ML estimate of the person parameter θ. The STOP RULES described below are then scanned; if none of them applies, the next item is selected and presented. The item selected as the next (k+1) item is that which, from the items not yet presented, has.the minimum absolute distance |βk - θ| from the individual ability score θ as estimated from the responses to the items so far presented. The answer to this item is recorded and a renewed estimate of the individual score is calculated by the ML method. The procedure is then repeated until one of the STOP criteria applies.
22
Ist noch ein geeignetes, bisher nicht verwendetes Item vorhanden ? (verschieden in Anfangs-/ Hauptphase)
STOP
ja Präsentation des ausgewählten Items, Registrierung der PB- Antwort
Nur Hauptphase: Schätzung des Personenparameters; Prüfung der Abbruchkriterien (insbesondere Berechnung des Schätzfehler)
nein
Abbruchkriterium erreicht ?
ja STOP
Fig. 2: The AMT test algorithm based on the Rasch model
STOP RULES: If the respondent’s ability θ can be estimated by the ML method (i.e. the subject has given at lease one correct and one incorrect response), the test is stopped as soon as one of the following criteria applies (Table 1): Table 1: Test termination criteria # 1 2 3
Reason for termination Maximum number of correct answers (MAXCORR = 10) achieved (initial phase) Maximum number of incorrect answers (MAXFALSE =10) achieved (initial phase). A maximum number of items (MAXITEMS = 30) has been exceeded (independent of measurement error)
5
No further items within an acceptable distance of θ are available The critical value for the measure of conformity has been exceeded – the respondent’s working style is too uneven for an assessment of his general intelligence to be made The measurement precision requirements (depending on the test form) have been met The time taken has exceeded the set maximum (depending on the test form)
6 9 10
Items used once are flagged and cannot be used again.
23
(3) Advantages of adaptive testing: The aim is to adapt the test to the respondent’s performance level – that is, to “tailor” it to his or her requirements. Binet and Simon adopted this approach in 1908 when they designed series of intelligence tests that were graduated according to age. However, it was not until 1960 that the theoretical basis for comparing the performance of two individuals who complete sets of items that are partically or totally different was formulated by George Rasch. In drawing up the dichomotomous logistic test model he laid the foundation for probabilistic test theory. Adaptive presentation requires powerful computers for successful implementation. Such computers are able to perform the detailed and therefore time-consuming calculations involved in the “customisation” process; they need to work out how well the subject is currently performing and on the basis of this information select and present the next appropriate test item. When compared with traditional test methods, the advantages of adaptive testing are: The best possible balance between test length and precision of measurement is achieved. More accurate results are obtained from fewer items. The respondent is on the whole neither underchallenged because the items are too easy, nor overchallenged by ones that are too difficult. This increases test motivation.
2.2.
ADAPTIVE TESTING
The AMT item pool was constructed from scratch following Hornke and Habon (1984, 1986); it contains 289 matrix items. These items were analysed in a large-scale empirical study involving N=1356 participants (Hornke, Küppers, & Etzel, 2000). All items were analysed using the one-parameter logistic test model of Rasch and the difficulty parameters β were estimated;.they vary between -3.45 and 4.99 with a mean of -0.131, a median of -0.092, and a standard deviation of 1.325.
Fig. 3: Frequency distribution of AMT item difficulty
24
Simulation studies show that the indicated reliabilities can be obtained with the average numbers of items shown. It is readily apparent that in order to obtain the required reliability at all scale points very varying numbers of items are required. The higher the precision requirements, the more items are needed. Table 2: Number of items required for different levels of reliability. (NUSE represents the number of items required, SEM is the standard error of measurement associated with a particularly level of reliability; the maximum number of items permitted by the simulation program is 99). Anforderungen an die Reliabilität beim AMT und die Auswirkung auf die Anzahl zu bearbeitender Items Angeforderte Mittelwert Median Spannweite Minimum Maximum Perzentile
Reliabilität SEM
5 10 20 30 40 50 60 70 80 90 95
NUSE60 ,60 ,63 12,94 13,00 10 12 22 12 12 12 12 12 13 13 13 14 14 15
NUSE70 ,70 ,55 16,50 16,00 10 15 25 15 15 16 16 16 16 16 17 17 18 19
NUSE75 ,75 ,50 19,30 19,00 14 18 32 18 18 18 19 19 19 19 20 20 21 22
NUSE80 ,80 ,44 23,54 23,00 59 22 81 22 22 22 23 23 23 23 24 24 25 26
NUSE85 ,85 ,39 30,52 30,00 71 28 99 29 29 29 30 30 30 30 31 31 32 34
NUSE90 ,90 ,32 44,58 44,00 57 42 99 42 43 43 43 44 44 44 45 46 47 49
NUSE95 ,95 ,22 87,36 86,00 15 84 99 84 84 84 85 85 86 87 88 89 93 99
For a reliability level of 0.80 (AMT Form S2), the average number of items required is 23. 95% of the population can be expected to complete the test with between 22 and 26 items. For all the AMT forms an average of 13, 23 or 30 items should suffice. Conventional tests do not achieve such consistency of precision, as Green (1970) makes clear in Figure 3(a); the Y axis of this graph is a function of the standard error of measurement. It is clear from this that classical, published tests differentiate well in the middle of the scale but are far from adequate at either end of the distribution.
25
(b) Benötigte Items bei Reliabiltität .80
(a) 90 80 70 60 50 40 30 20 10 0 -4
-2
0
2
4
Geschätztes THETA bei Reliabilität .80
Fig. 4: (a) Comparison of the confidence interval widths of an adaptive test and two conventional tests (from Green, 1970, p. 187; see also Hornke, 1976, p. 252; “tailored tests” are adaptive tests). (b): Distribution of the number of items required at a reliability of 0.80. Here: distribution of the required numbers of items with regard to the adaptively determined THETA scores
The AMT, by contrast, maintains the required reliability of 0.80 across the entire scale. In all the cases shown the standard error of measurement is smaller than 0.44; with only very few exceptions the number of items required remains in the range 20 – 25. This demonstrates that the AMT item pool contains sufficient items at every difficulty level to guarantee the same reliability for every respondent (irrespective of his or her ability). An individual case (see Fig. 5) illustrates the possible course of a test. The true value of THETA is known from the simulation to be 2.0. After 23 items an estimated THETA score of 1.79 is obtained, based on the answers given and a standard error of measurement of 0.44. The graph of the THETA estimates shows clearly that the estimates converge towards a limit. The thin lines (inner line = THETA±SEM; outer line = THETA±1.96*SEM ≈ 95% confidence interval) show that with each further item confidence increases. The protocol also shows that 11 out of 23 items – approximately 50% – were solved correctly, as is typically the case in adaptive testing.
26
Fig. 5: Protocol of an adaptive testing simulation study. (
is the solution (correct or incorrect) to item of the item bank presented at position in the test; <sTh> is the estimated value of Theta with standard error measurement <SEM>; the symbols and ο represent respectively correct and incorrect answers, with the position of the symbol indicating the difficulty of the item)
Except in the initial phase there are always sufficient suitable items available, as the symbols and ο and the thicker line representing the course of the intermediate THETA estimates make clear.
2.3.
ITEM EXPOSURE CONTROL
The use of adaptive testing does, however, introduce some problems. Progress through the test is completely deterministic. If the test always starts with the same item, there are only two items that can be presented in second place, four possible items in the third position, and so on. This means that some items will be presented very much more frequently than others. This is particularly true of the items of medium difficulty, which play a particulary important role in the item pool. These items become public knowledge disproportionately quickly; in order to guard against falsification of the test they must be frequently replaced or the item pool must be enlarged. An alternative is to make the course of the test probabilistic rather than strictly deterministic. This reduces the risk of a respondent practising the route through the test and learning it by heart, as in “coached faking”.
27
In order to arrive at an optimal solution, the Item Exposure Control Parameters must be calculated in such a way that all items are as far as possible presented with equal frequency while also optimising measurement precision. This can be achieved by using a simulative adaptation algorithm (Simpson & Hetter, 1985). It was made technically possible by means of the software AdapSIM (Häusler, 2004). The quality of the IEC solutions arrived at can be described by two measurements: 1. Consistency of oscillation Since the IEC solution does not converge but becomes a gentle oscillation, this measurement (by analogy with Cronbach’s Alpha) can be used to describe the internal consistency of the solution. 2. Stability of the IEC solution: This corresponds to the correlation between two independently estimated IEC solutions.
Fig. 6: Graph showing overexposure as a function of the adaptive process
28
Table 3: Adaptation of the IEC solution Step 1 2 3 4 5 6 7 8 9 10 11 12 13
Max. overexposure 11.93 11.085 10.265 9.232 6.34 3.375 2.631 2.203 1.98 1.85 1.865 1.86 1.742
Mean test length 24.14 24.11 24.03 24.02 23.96 24.06 24.02 24.05 24 24.04 24.09 24 23.96
Reliability 0.823 0.821 0.831 0.829 0.814 0.831 0.831 0.817 0.83 0.827 0.821 0.826 0.825
Quality of the IEC solution: Oscillation consistency: IEC=0.929 Stability of the IEC solution: RSTAB=0.982
2.4.
MOTIVATION AS REQUIRED
Since adaptive testing offers each respondent approximately the same probability of finding a solution, the experience of success in the test can be regarded as constant for everyone. All respondents will be able to solve around 50% of the items presented. Dependin on the respondent’s motivational needs, however, this can sometimes be too little (Andrich, 1995). Heckhausen (1989), for example, found that for a success-oriented person a 70 – 80% probability of arriving at the correct solution was optimally motivating. A deviation from a 50% solution probability for a few items is not necessarily accompanied by a loss of test economy. According to Sommer & Häusler (2004), the addition of 25% motivational items, each with a solution probability of 80%, does not have a detrimental effect on test length, since it means that respondents work in a more motivated fashion. Motivation items are used in the AMT primarily to counteract a demotivated and highly impulsive working style. If a respondent answers a number of successive items incorrectly and his working times fall short of a critical value, motivation items are presented until the respondent’s working style is normalised.
29
2.5.
TEST STRUCTURE
Item presentation in the AMT is adaptive. The respondent can choose one of nine alternative answers, making his selection by means of mouse, computer keyboard or touch screen. It is not possible to omit an item or to go back to a preceding one.
2.6.
TEST FORMS
To meet the needs of different assessment goals and application purposes, three standardised versions of the AMT have been created. They vary in terms of their precision of measurement and thus in terms of their length. Test form S1: Screening This test form can be used to obtain a brief general summary of a candidate’s ability. The standard measurement error of this form is SEM=0.63. The test is normally completed after on average 13 items. Test form S2: Standard The pre-set standard measurement error of this form is SEM=0.44. The respondent will work an average of 23 items. Test form S3: Precision This form can be used to make more precise statements about the respondent’s actual ability. It is particularly useful if the differences between respondents are expected to be very small or if classification decisions are being made in a situation in which the class boundaries lie very close together. The standard error of measurement is pre-set at a level of SEM=0.39. As a result of the higher level of precision a larger number of items needs to worked; an average of about 30 items will need to be presented. Test form S11: Traffic psychological short form Like form S1, this test form gives a general indication of a respondent’s ability. It differs from the S1 form in that it uses easier start items, thus providing a gentler introduction to the test. The standard error of measurement for this form is SEM=0.63. In addition, a maximum test length of 20 minutes is prescribed. All four versions can make a useful contribution to a decision-making process. Even relatively “imprecise” measurements can be used to make decisions if one of the boundaries of the ability assessed (i.e. one of the limits of the confidence interval) just meets the decision-making point (Hornke, 1999). To quote an example from the test protocol given above: after 12 items with a SEM=0.66 it would be entirely appropriate – with a risk of 2.5% - to decide that the respondent is not going to achieve the critical cut-off point of 2.75 and therefore cannot be assigned to the category of capable respondents.
30
2.7.
DESCRIPTION OF VARIABLES
Main variables General intelligence The estimated person parameter Ө can be can be viewed as corresponding approximately to a z-score and can be interpreted as such. Scores are converted into percentile ranks on the basis of the norm sample. Ө is an estimate of the respondent’s general intelligence; the respondent’s position within the norm sample is described by his percentile rank. Subsidiary variables Number of items solved This variable indicates how many items were worked by the respondent. It can be different for each test, depending on the respondent’s behaviour and the convergence of the estimation algorithm. It is dependent on the respondent’s ability respondents of above or below average ability may sometime need to complete more items than those of average ability. If respondents are able to solve either all or none of the problems, the test is aborted after 10 items. In such cases the item difficulty of the most difficult or the easiest item is taken to be the person parameter. A maximum of 35 items can be presented; if the pre-set standard error of measurement has still not been reached at this point, the test is ended. In an adaptive test the number of items solved cannot be used to compare participants. The only information that is meaningful for this purposes is the estimated person parameter Ө and the associated standard error of measurement or percentile rank. Additional variables Working time The time to work the whole test is given in minutes and seconds. The test protocol also documents the time taken to work each item. The test protocol can be used to identify items that were not worked in a typical manner; these items can if necessary be discussed with the respondent.
31
Fig. 7: AMT test protocol
In this example it is striking that the respondent spent more than two minutes on Item 10, which was in fact an easy item (=-1.440). Item 11, on the other hand, was viewed for only seven seconds. Both cases give rise to the question of whether serious cognitive processes were actually taking place or whether something else was happening that might render the test score questionable. This type of retrospective consideration of quality issues is not possible with paper-and-pencil tests.
32
3. EVALUATION 3.1.
OBJECTIVITY
Administration objectivity Test administrator independence exists when the respondent’s test behaviour, and thus his test score, is independent of variations (either accidental or systematic) in the behaviour of the test administrator (Kubinger, 2003). Since administration of the Adaptive Matrices Test is computerised, all subjects receive the same information, presented in the same way, about the test. These instructions are independent of the test administrator. Similarly, test presentation is identical for all respondents. Scoring objectivity The recording of data and calculation of variables is automatic and does not involve a scorer. The same applies to the norm score comparison. Computational errors are therefore excluded. Interpretation objectivity Since the test has been normed, interpretation objectivity is given (Lienert & Raatz, 1994). Interpretation objectivity does, however, also depend on the care with which the guidelines on interpretation given in the chapter “Interpretation of Test Results” are followed.
3.2.
RELIABILITY
The use can select the level of reliability (in the sense of internal consistency) by selecting the critical standard error measurement. Four standard forms are currently available, based on reliabilities of 0.70, 0.83, 0.86 and 0.70 (CritSEM=0.63, 0.44, 0.39 and 0.63) Adaptive testing ensures that each respondent is assessed with the same precision of measurement. A longitudinal study involving 82 respondents (48% men, 52% women) aged between 17 and 78 (m=44; s=17) who completed Form S1 yielded a retest reliability of r=0.74 and a stability over a period of three months of r=0.62.
3.3.
VALIDITY
Construct validity Construct validity exists when it can be demonstrated that a test implements particular theoryled approaches. Content (logical) validity is closely linked to the construction rational. The focus here is on cognitive operations that were incorporated into the item construction. According to Hornke, Etzel and Küppers (2000), the construction rational correlates at 0.72 with the difficulty parameters and thus demonstrates the construct validity of the AMT. Further evidence of the AMT’s construct validity comes from a study by Sommer and Arendasy (2005) of the factor structure of the Intelligence Structure Battery.(INSBAT: Arendasy, Hornke, Sommer, Häusler, Wagner-Menghin, Gittler, Bognar & Wenzl, 2005), in which the AMT was included. The allocation of the subtests in accordance with the Cattell-Horn-Carroll model provided the theoretical basis. The authors also compared the fit of this model with a pure gfactor model. The pure g-factor model assumes that the relationships between the individual subtests can be explained solely be a general factor Table 4 shows the global fit indices for the two models.
33
Table 4: Model quality criteria of the Cattell-Horn-Carroll model 2
2
Model
χ
df
p
χ /df
CFI
RSMEA
CHC model g-factor
153.88 345.48
85 90
<0.001 <0.001
1.81 3.84
0.95 0.78
0.06 0.12
The fit of the model was assessed using the χ2 test, the χ2/df, the CFI and the RSMEA. An adequate degree of agreement between the empirical covariance matrix and the model matrix is indicated by the following values: χ2 not significant, χ2/df < 2 (Byrne, 1989), RSMEA <0.08 and CFI >0.90 (Backhaus et al., 2004). As Table 4 shows, the CHC model provides a sufficiently good approximation to the empirical data, while the g-factor model does not adequately describe the data. In addition the CHC model fits the data significantly better (∆2 [5] = =191.60; p < 0.001) than the pure g-factor model. Fig. 8 shows the standardised factor loadings for the CHC model.
Fig. 8: Standardised loadings of the CHC model. WS: Lexical Knowledge subtest; VP: Verbal Production subtest; AD: Algebraic Reasoning subtest; ASF: Computational Estimation subtest; AK: Arithmetical Competence subtest; ANF: Adaptive Numerical Flexibility Test;.NID: Numerical Inductive Reasoning subtest; VDD: Verbal Deductive Reasoning subtest; VIK: Visual Short-term Memory subtest; VEK: Verbal Short-term Memory subtest; LZG: Long-term Memory subtest; RV: Spatial Perception subtest; Gc: crystallized intelligence; Gq: quantitative reasoning; Gf: fluid intelligence; Gstm: short-term memory; Gltm: long-term memory; Gv: visual processing capacity; G: general intelligence
34
All the factor loadings are statistically significant and at a level which can be regarded as medium to high. As Fig. 8 shows, the AMT and all the tests of inductive and deductive reasoning load onto the factor of fluid intelligence (Gf). The loadings of the g-factor onto the individual factors of the second stratum of the CHC model largely correspond to the theoretical expectations derived from previous empirical findings with samples representative of the general population. At .99 the loading of the g factor onto Gf is high. It does not differ significantly from 1 (∆2 [1] = 1.03; p = 0.311). This result implies that the g factor and Gf are indistinguishable and it is thus in accordance with the results of earlier work by Gustafsson (1984; Undheim & Gustafsson, 1987). Criterion validity Criterion validity exists when a test correlates with an external criterion relevant to the purpose of the investigation. Studies of criterion validity have been carried out in the fields of traffic and aviation psychology. A study by Sommer, Arendasy, Schuhfried and Litzenberger (2005) showed that a test battery containing the AMT could distinguish at a significant level between accident-free drivers and drivers who had been involved in accidents in which they had been at fault. In addition, the AMT correlated at r=0.242 with the global assessment of driving behaviour in the Vienna Driving Test. Further evidence of the AMT’s criterion validity emerged from a study of the criterion validity of a range of ability tests relevant to aviation psychology (Sommer, Arendasy, Hansen & Schuhfried, 2005). N=99 male applicants for pilot training completed a comprehensive battery of ability tests that included the AMT. The global assessment of performance in a standardised flight simulator served as the criterion variable. This test battery enabled success in the flight simulator to be correctly predicted for 90% of the subjects. This corresponds to a validity coefficient of R=0.79. The AMT contributed to the predictive model with a relative relevance of 18%.
3.4.
SCALING
Respondents’ scores are compared with the norm sample on a scale with a mean of 0.000, a median of 0.002 and a standard deviation of 0.890. For interpretation purposes they can be converted into percentile ranks relating to a norm sample.
3.5.
ECONOMY
Since the Adaptive Matrices Test is a computerised procedure, it is very economical to administer and score. The administrator’s time is saved because the instructions at the beginning of the test are standardised and raw and norm values are calculated automatically. With regard to administration time there is a connection between precision of measurement on the one hand and the number of items that need to be worked, and hence the respondent’s time, on the other. An increase in reliability requires an increase in the average number of items solved. However, compared with a classical test of fixed length of comparable reliability, the average number of items to be worked is always smaller in an adaptive test.
35
3.6.
USEFULNESS
"A test is useful if it measures a personality trait for the assessment of which there is a practical need. A test therefore has a high degree of usefulness if it cannot be replaced by any other test” (Lienert 1994, p.13). The AMT provides a precision measurement of general intelligence, a factor that is relevant to many psychological assessment situations. The procedure can therefore be considered useful.
3.7.
REASONABLENESS
Reasonableness describes the extent to which a test is free of stress for the test subject; the respondent should not find the experience emotionally taxing and the time spent on the test should be proportional to the expected usefulness of the information gained (Kubinger, 2003). The adaptive mode of presentation makes a significant contribution towards the reasonableness of this test since it largely prevents weaker respondents being presented with items that are too difficult for them or strong candidates have to work items that are too easy. The AMT can be used without reservation with groups of individuals who have no severe intellectual impairment. The instructions make use of a learning program that ensures that respondents understand the task, thus avoiding the risk of respondents having to use the first items of the actual test to “teach themselves” how to work the test.
3.8.
RESISTANCE TO FALSIFICATION
A test that meets the meets the quality criterion of resistance to falsification is one that can prevent a respondent answering questions in a manner deliberately intended to influence or control his test score (e.g. Kubinger, 2003). As with all performance tests, the test scores of the AMT cannot be deliberately manipulated by respondents to their advantage. In a multiple choice test there is always a possibility that a respondent will arrive at the correct solution by guesswork. In the AMT the likelihood of guessing the correct answer is kept to a minimum by providing eight possible answers for each item as well as the alternative statement “I don’t know the solution”. This statement is scored as in incorrect answer.
3.9.
FAIRNESS
If tests are to meet the quality criterion of fairness, they must not systematically discriminate against particular groups of respondents on the grounds of their sociocultural background (Kubinger, 2003). Experience to date indicates that the AMT test is fair. In particular, individuals with little computer experience are not disadvantaged, because the instruction phase provides sufficient opportunity for respondents – even if they have not previously used a computer – to practice the input of responses.
36
4. NORMS The norms were obtained by calculating the mean percentile rank PR(x) for each test score THETA observed in the norm sample using the following formula (Lienert & Raatz, 1994): PRx = 100 ⋅
cum fx − fx 2 N
cum fx corresponds to the number of respondents who have achieved the test score THETA or a lower score, fx is the number of respondents with the test score THETA, and N is the size of the sample. Norm sample A norm sample representative is available consisting of N=461 individuals (220 men, 241 women) aged between 18 and 81 (mean = 37.0, standard deviation = 14.5). The data were obtained between 2005 and the beginning of 2006 in the standardisation laboratory of Schuhfried GmbH. Evaluation sample The previous norming carried out with the evaluation sample during test development (previously “Adults”; N=1356) has, however, been retained (see below) in order that any comparison or effect studies currently under way can be completed. In any new studies use of the norm sample is recommended. The norming of the evaluation sample is based on the data of 1356 respondents (580 men and 776 women aged between 15 and 80 who were tested at various locations (Kattowitz, Poland; Moscow, Russia & Vienna, Austria). Statistiken THETA N
Gültig Fehlend
Mittelwert Standardfehler des Mittelwertes Median Standardabweichung Schiefe Standardfehler der Schiefe Kurtosis Standardfehler der Kurtosis Minimum Maximum
1356 0 ,000
160
,024
120
140
,002a ,890 ,022
100 80 60
,066
40
-,335
20 0 -2,5
,133 -2,506 2,721
-1,5 -2,0
a. Aus gruppierten Daten berechnet
-,5 -1,0
,5 0,0
1,5 1,0
2,5 2,0
THETA-Schätzungen der Normstichprobe
Fig. 9:.Distribution characteristics of the THETA scores of the norm sample
37
Table 5: Percentile rank and z distribution of the Ө scores Percentile rank
AMT-θ
z-score
5 10 15 20 25 30 35 40 45
-1.415 -1.182 -0.950 -0.778 -0.645 -0.499 -0.365 -0.240 -0.111
-1.589 -1.328 -1.067 -0.874 -0.725 -0.560 -0.410 -0.270 -0.125
Percentile rank 50 55 60 65 70 75 80 85 90 95
38
AMT-θ 0.002 0.123 0.231 0.355 0.477 0.631 0.757 0.923 1.173 1.517
z-score 0.003 0.138 0.259 0.398 0.535 0.709 0.850 1.036 1.318 1.705
5. TEST ADMINISTRATION The AMT consists of an instruction and practice phase and the test phase itself.
Fig. 10: Instruction item from the AMT
5.1.
INSTRUCTION AND PRACTICE PHASE
General issues concerning use of the Vienna Test System are first explained to the respondent, and the chosen input device is introduced (keyboard/mouse/light pen/ touch screen). The specific instructions for the AMT then begin. The manner in which the test is to be worked is explained on-screen. Practice items illustrate the formulation of the problems and the answer format. It is not possible to omit the practice examples. If the respondent gives a wrong answer to a practice example, he is alerted to this; the system requests him to reconsider the solution and make another attempt to select the correct answer. After three incorrect answers to a practice item the instruction phase is aborted and the administrator should make an appropriate intervention. The respondent must give a certain number of correct answers before proceeding to the test phase.
5.2.
TEST PHASE
As described above, the presentation of test items is based on an adaptive test strategy. The choice of the next item is always determined by the current estimate of the respondent’s performance level. The respondent records his answer to an item by choosing one of the eight alternative solutions provided. If he is unable to solve an item he can select the box beside the statement “I don’t know the solution”. This answer is always scored as incorrect.
39
The test continues until the standard error of measurement falls below the predefined level associated with the test form used. Occasionally the test may be ended by the system; this occurs if 10 successive items are answered either correctly or incorrectly. When this happens the respondent’s ability cannot be estimated because there is insufficient variance in the data; in such cases the item difficulty of the easiest or the most difficult item in the item pool is taken as the person parameter estimate. Another cancellation criterion involves the number of items presented to the respondent. For individuals at the extremes of the scale it is possible that there may be insufficient items available that are suitable for this part of the ability range. In this case it may not be possible to achieve the pre-set standard error of measurement with 35 or fewer items. For practical reasons the test is then ended with a somewhat higher standard error of measurement.
40
6. INTERPRETATION OF TEST RESULTS 6.1.
GENERAL NOTES ON INTERPRETATION
A percentile rank of <16 can in general be regarded as below average. An individual with such a result can be regarded as having below-average ability in comparison to the reference population used. A percentile rank of 16 – 24 can be regarded as below average to average. In comparison to the reference population used, an individual with a percentile rank in this range demonstrates below average to average ability. A percentile rank between 25 and 75 is an average score. The ability of an individual whose score is in this range is in broad terms typical of that of the reference population. Percentile ranks between 76 and 84 indicate average to above average ability in comparison to the reference population used. Percentile ranks >84 reflect a clearly above average result. In comparison to the reference population, individuals with percentile ranks in this range demonstrate above average ability.
6.2.
NOTES ON INTERPRETATION IN TRAFFIC-PSYCHOLOGICAL ASSESSMENT
Guidelines on the interpretation of percentile ranks in the context of traffic-psychological assessment can be found in the reporting guidelines of the Bundesanstalt für Straßenwesen (Federal Highway Research Institute) (Bundesanstalt für Straßenwesen, 2000, p. 16 section 2.5.). Depending on whether the assessment relates to a driver of Group 1 or Group 2, percentile ranks of 16 (Group 1) and 33 (Group 2) are regarded as critical cut-off values.
6.3.
INTERPRETATION OF THE MAIN VARIABLES OF THE AMT
The main variable “general intelligence” measures non-verbal logical inductive reasoning ability. Individuals with a high score (PR>84) on this variable are therefore particularly good at identifying patterns and regularities and applying the rules derived from them. Individuals with an above-average percentile rank are able to abstract regularities from their learning experience and deduce consequences for future behaviour.
41
6.4.
ADDITIONAL OUTPUT OF RESULTS
Test protocol Testprotokoll: Item
Antwort
Zeit
PAR
VI
REL
LWk
1
2-
00:03
0.000
ItS
--
(-- ... --)
--
1%
2
2-
00:01
-2.291
--
(-- ... --)
--
8%
3
2-
00:00
-3.247
--
(-- ... --)
--
18%
4
2-
00:01
-3.974
--
(-- ... --)
--
31%
5
2+
00:00
-4.299
-4.712
(-- ... --)
0.385
38%
6
2-
00:01
-4.548
-5.252
(-- ... --)
0.403
44%
7
2+
00:00
-4.525
-4.639
(-7.218 ... -2.060)
0.523
44%
8
2+
00:01
-4.175
-4.233
(-6.163 ... -2.303)
0.584
35%
9
2-
00:00
-3.189
-4.383
(-6.213 ... -2.554)
0.606
17%
10
2-
00:00
-2.629
-4.465
(-6.238 ... -2.691)
0.619
10%
11
2-
00:00
-2.574
-4.534
(-6.267 ... -2.802)
0.628
10%
12
2-
00:01
-2.495 (M)
-4.593
(-6.298 ... -2.888)
0.636
9%
13
2-
00:00
-2.480 (M)
-4.647
(-6.325 ... -2.969)
0.642
9%
14
2-
00:00
-2.461 (M)
-4.696
(-6.351 ... -3.040)
0.648
9%
15
2-
00:00
-2.454 (M)
-4.741
(-6.381 ... -3.101)
0.653
9%
16
2-
00:01
-2.434 (M)
-4.784
(-6.409 ... -3.158)
0.657
9%
Anmerkung(en): Antwort: 1...8 = Gewählte Figur, 9 = "Ich weiß die Lösung nicht", 0 = Item wurde nicht beantwortet (bei Testabbruch); +: Richtige Lösung, -: Falsche Lösung; Zeit: Bearbeitungszeit in M inuten:Sekunden; ItS: Itemschwierigkeit: <0=leichter, >0=schwieriger; PAR: Aktuell geschätzter Personenparameter (<0=schlechter, >0=besser, --=keine Schätzung möglich); Das Vertrauensintervall (VI) gibt an in welchem Bereich der wahre Leistungsparameter mit 5% Irrtumswahrscheinlichkeit liegt. Die Reliabilität (REL) ist eine untere Grenze für die M essgenauigkeit und liegt zwischen 0 (keine M essgenauigkeit) und 1 (optimale M essgenauigkeit). LWk ist die individuelle Lösungswahrscheinlichkeit für eine bestimmte Aufgabe.
Fig. 11: Test protocol
The test protocol provides detailed information on how the test was worked; it shows, for example, how each item was answered, whether the answer was correct or incorrect, and how long the subject took to answer each item. This can be used to investigate whether a higher than average number of problems arose at any particular point during the test. A diagram of the adaptive process can be viewed as an alternative to the test protocol.
42
7. APPLICATION IN TRAFFIC PSYCHOLOGY Test form S11 has been specially developed for use in traffic psychology. As already described in Section 2.6, S11 is a very economical test form designed for intelligence screening. In addition, the adaptive algorithm governing item selection has been modified to eliminate the possibility of a respondent feeling overchallenged right at the start of the test. The start item is selected on the basis of the respondent’s age and education; it is an item that respondents of comparable age and educational level have approximately a 75% likelihood of solving correctly. This ensures that older respondents or those with a lower level of education do not become frustrated at the outset, thereby becoming demotivated or anxious about the test as a whole. A further distinguishing feature of Form S11 is that it enables decision-oriented testing to be carried out in the context of traffic-psychology-related investigations. Decision-oriented testing is based on the view that testing only makes sense if the tests used relate to the decision to be made. Only then can the assessor be certain of obtaining a test result that can be interpreted in a manner that makes a useful contribution to the decision-making process. The AMT/S11 implements this requirement by enabling the user to specify in the Options window whether the test should be optimised to relate to Group 1 (drivers without increased responsibility) or Group 2 (drivers with increased responsibility). In both cases the test is continued until the point at which there is a high statistical certainty (95%) that the latent dimension (in this case general intelligence) lies above the threshold values specified for traffic psychological purposes (IQ 70 or parameter -2.6 for Group 1 and IQ 85 or parameter -1.8 for Group 2) or until one of the other termination criteria used in the AMT applies (see Section 2.1).
Fig. 12: Option window for the traffic-psychological cancellation criteria of decision-oriented testing. The top option is set by default; this does not involve any additional traffic-psychological cancellation criteria.
43
Adaptives Verlaufsdiagramm: 1
Parameter
0 -1 -2 -3 -4 -5 -6
1 Richtig beantwortete Items
2
3
4
5
6
Item
Falsch beantwortete Items Schätzer für den Personenp arameter Vertrauensintervall (5% Irrtumswahrscheinlichkeit) Cut-off Wert
Fig. 13: Graph of the adaptive process in an AMT test session. The test terminates once the overall confidence interval lies above the cut-off score (in this case IQ 70) and at least 6 items have been presented.
Used in this way, these additional cancellation criteria may – depending on the respondent’s ability – significantly reduce the test length. Fig. 14 shows the average test length needed to arrive at a conclusion about the general intelligence of a Group 1 driver.
Fig. 14: Expected test length as a function of the respondent’s general intelligence for an investigation in Group 1. Above an ability level of roughly PR 30 the decision-oriented procedure carried out in connection with a traffic-psychology-related investigation leads to a noticeable reduction in test length. Where the ability level is less than PR 30, a quicker than normal decision (and associated cancellation of the test) is not possible. The test is continued until the specified target reliability is achieved. This is usually the case after approximately 11 items.
44
REFERENCES Andrich, D. (1995). Review of the book Computerized adaptive testing: A primer. Psychometrika, 4, 615-620. Arendasy, M., Hornke, L. F., Sommer, M., Häusler, J., Wagner-Menghin, Gittler, Bognar & Wenzl, M. (2005). Manual Intelligenz-Struktur-Batterie (INSBAT). Mödling: SCHUHFRIED GmbH. Backhaus, K., Erichson, B., Plinke, W. & Weiber, R. (2004). Multivariate Analysemethoden: Eine anwendungsorientierte Einführung. Berlin: Springer. Byrne, B. M. (1989). A primer of LISREL. Basic applications and programming for confirmatory factor analytic models. New.York: Springer. Hambleton, R.K. & Swaminathan, H. (1985). Item response theory - principles and applications. Boston: Kluwer-Nijhoff Publishing. Gustafsson, J. E. (1984). A unifying model for the structure of intellectual abilities. Intelligence, 8, 179-203. Häusler, J (2004). AdapSIM Software. Vienna: self-pubished Heckhausen, H. (1989). Motivation und Handeln. Berlin: Springer Hornke, L.F. (1976). Grundlagen und Probleme adaptiver Testverfahren. Frankfurt: Haag + Herchen. Hornke, L.F. (1993). Mögliche Einspareffekte beim computergestützten Testen. Diagnostica, 39, 109-119. Hornke, L.F. (1999). La prise de Décision Basée sur le Testing adaptif (DÉBAT). Psychologie et Psychométrie, 20, 181-192. Hornke, L.F. & Habon, M.W. (1984). Erfahrungen zur rationalen Konstruktion von Test-Items. Zeitschrift für Differentielle und Diagnostische Psychologie, 5, 203-212. Hornke, L.F. & Habon, M.W. (1986). Rule-based item bank construction and evaluation within the linear logistic framework. Applied Psychological Measurement, 10, 369-380. Hornke, L.F. & Rettig, K. (1988). Regelgeleitete Itemkonstruktion unter Zuhilfenahme kognitionspsychologischer Überlegungen. In K.D. Kubinger (Eds). Moderne Testtheorie. Weinheim and Munich: Psychologie Verlagsunion. Hornke, L.F., Etzel, S. & Küppers, A. (2000). Konstruktion und Evaluation eines adaptiven Matrizentests. Diagnostica, 46, 182-188. Kubinger, K. D. (2003). Gütekriterien. In K. D. Kubinger & R. S. Jäger (Eds.), Schlüsselbegriffe der psychologischen Diagnostik (p. 195-204). Weinheim: Psychologie Verlags Union.
45
Lienert, G.A. (1989). Testanalyse und Testkonstruktion. Weinheim : Beltz Lienert, G.A., Raatz, U. (1994). Testaufbau und Testpraxis. Weinheim: Beltz. Rettig, K. & Hornke, L.F. (1990). Adaptives Testen. In W. Sarges (Ed.), Managementdiagnostik (p. 444-450). Göttingen: Hogrefe. Rost, J. (1996). Lehrbuch Testtheorie, Testkonstruktion. Bern: Huber. Sommer, M. & Arendasy, M. (2005). Theory-based construction and validation of a modern computerized intelligence test battery. Budapest: EAPA 2005 Abstracts. Sommer, M., Arendasy, M., Schuhfried, G. & Litzenberger, M. (2005). Diagnostische Unterscheidbarkeit unfallfreier und mehrfach unfallbelasteter Kraftfahrer mit Hilfe nichtlinearer Auswertemethoden. Zeitschrift für Verkehrssicherheit, 51, 82-86. Sommer, M., Arendasy, M., Hansen, H.-D., & Schuhfried, G. (2005). Personalauswahl mit Hilfe von statistischen Methoden der Urteilsbildung am Beispiel der Flugpsychologie. Untersuchungen des Psychologischen Dienstes der Bundeswehr, 40, 39-64. Sommer, M.; Häusler, J. (2004). Motivation Stabilizing Items in Computerized Adaptive Testing: Psychometric and Psychological Effects. Malaga: EAPA 2004 Abstracts Sympson, J.B. & Hetter, R.D. (1985). Controlling item exposure rates in computerized adaptive testing. Papers presented at the Annual Conference of the Military Testing Association. San Diego: Military Testing Association. Undheim, J. O. & Gustafsson, J. E. (1987). The hierarchical organisation of cognitive abilities: Restoring general intelligence through the use of linear structural relations (LISREL). Multivariate Behavioral Research, 22, 149-171.
8.
46
