24/10/2012
PEMBUATAN DAN MENELAAH SOAL MCQ (ITEM DEVELOPMENT AND REVIEW)
Komite Bersama Uji Kompetensi Dokter Indonesia
What is “Good” Assessment?
1
24/10/2012
Professional authenticity
A simple model of competence Does
Behaviour
Shows how
Knows how
Cognition Knows Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S63-S7.
Climbing the pyramid......
Does
Performance assessment in vivo: Mini-CEx, Video, Audits…..
Shows how Shows how
Performance assessment in vitro: OSCE, SP-based test…..
Knows how Knows how
(Clinical) Context based tests: MCQ, essay type, oral…..
Knows Knows
Factual tests: MCQ, essay type, oral…..
2
24/10/2012
Sulitkah membuat soal MCQ?
Tijauan1: SKDI
Tinjauan 2
Tinjauan 3
Tinjauan 4
Tinjauan 5
Tinjauan 6
Tinjauan 7
Ketrampilan dasar klinis
Kognitif
Recall
Reproduksi
Pertumbuhan, perkembangan, dan degenerasi
Promosi kesehatan dan pencegahan penyakit
Individu
Aplikasi biomedis, behavior, clinical, & epidemiologi pada kedokteran
Procedural skill
Reasoning
Kepala dan leher
Kelainan genetik dan kongenital
Penapisan/Dia gnosis
Keluarga
Komunikasi efektif
Afektif
Saraf & Perilaku
Penyakit Infeksi dan Imunologi
Manajemen/T erapi
Masyarkat
Manajemen masalah kesehatan primer
Kulit, otot, tulang, jaringan lunak
Penyakit neoplasma
Rehabilitasi
Penelusuran, kritisi, dan manajemen informasi
Darah & Sistem Kekebalan
Penyakit akibat trauma atau kecelakaan
Aspek hukum dan etika
Profesionalisme, moral, dan etika praktik kedokteran
Jantung, Pembuluh darah & Sistem limfatik
Kesadaran, pemeliharaan, dan pengembangan personal
Saluran cerna, hepatobilier, dan pankreas Saluran Pernapasan Ginjal dan saluran Urogenital
Blue Print UKDI
Endokrin dan metabolisme
3
24/10/2012
KESALAHAN PADA SOAL PILIHAN (MCQ)
GANDA
Hanya menguji kemampuan mengingat – sering hanya hal yang trivia Tidak menguji aplikasi atau integrasi pengetahuan Tidak menguji kemampuan daya nalar dan pemecahan masalah Tidak konsisten dengan prinsip-prinsip pendidikan masa kini Lebih meningkatkan kemampuan memorisasi dari pada pemahaman
HAL-HAL YANG HARUS DIPERHATIKAN DALAM MEMBUAT SOAL “MCQ” Hindari pertanyaan “recall” simpel : jawaban yang dapat diperoleh dari suatu halaman buku ajar Hindari pertanyaan yang mendorong memorisasi sederhana Selalu menguji kemampuan aplikasi, integrasi, atau pemahaman Gunakan skenario yang relevan sebagai stem
4
24/10/2012
PRINSIP-PRINSIP DASAR
Nilai aplikasi dan integrasi pengetahuan Gunakan skenario klinik untuk menyajikan masalah Uji kemampuan daya nalar dan pemecahan masalah Fokuskan pada materi yang penting – hindari “trivia” Sediakan satu jawaban “paling benar “ Buatlah distraktor homogen Hindari memorisasi fakta sederhana
PANDUAN UMUM KONSTRUKSI MCQ (1) 1. Yakinkan bahwa setiap soal dapat dijawab tanpa harus melihat pilihan jawaban atau yakinkan bahwa pilihan jawaban adalah 100 % benar atau salah. 2. Usahakan agar data atau informasi penting seoptimal mungkin dicantumkan pada soal (stem), sedangkan pilihan jawaban usahakan sesingkat mungkin.
5
24/10/2012
MULTIPLE-CHOICE QUESTION FORMATS A-type One-best-answer Questions
Multiple-choice Questions
R-type Extended matching
Standard True-false-type True-false Questions K-type Complex True False
COMPONENTS OF AN A-TYPE QUESTION A
35-year old man has had a stomach ache all afternoon. He ate the following lunch: two big McDonalds hamburgers, an ice cream shake, large fries.
Which
is the most likely diagnosis?
A. Appendicitis B. Bowel obstruction C. Colon cancer D. Pancreatitis E. Too much lunch
Vignette, Scenario, Or Stem Lead-in
Options A, B, C, and D are Distractors Option E is the Key
6
24/10/2012
SAMPLE GOOD QUESTION AND THE „COVER THE OPTIONS‟ RULE
A 32-yo man has a 4-day history of progressive weakness in his extremities. He has been healthy except for an upper respiratory tract infection 10 days ago. His temperature is 100 F, BP 130/80, pulse 94, respirations 42 and shallow. He has symmetric weakness of both sides of the face and the proximal and distal muscles of the extremities. Sensation is intact. No deep tendon reflexes can be elicited; the plantar responses are flexor. Which of the following is the most likely diagnosis?
A. B. C. D. E.
Acute disseminated encephalomyelitis Guillain-Barré syndrome Myasthenia gravis Poliomyelitis Polymyositis
SAMPLE FLAWED QUESTION AND THE „COVER THE OPTIONS‟ RULE
Which of the following is true about pseudogout? A. It occurs frequently in women. B. It is seldom associated with acute pain in a joint. C. It may be associated with a finding of chondrocalcinosis. D. It is clearly hereditary in most cases. E. It responds well to treatment with allopurinol.
7
24/10/2012
PANDUAN UMUM KONSTRUKSI MCQ (2)
Hindari informasi berlebihan pada soal. Hindari soal-soal yang bersifat menjebak dan terlalu kompleks serta tidak relevan. Gunakan pilihan jawaban yang secara gramatikal konsisten dan secara logis kompatibel dengan soal.
PANDUAN UMUM KONSTRUKSI MCQ (3)
Hindari kata-kata absolut seperti, selalu, tidak pernah, dan semua pada pilihan jawaban, dan hindari pula istilah yang tidak jelas (meragukan) seperti biasanya atau sering. Hindari kalimat atau frase negatif seperti KECUALI, TIDAK, atau BUKAN. Fokuskan soal pada konsep-konsep penting, dan hindari hanya menguji kemampuan mengingat fakta.
8
24/10/2012
PANDUAN UMUM KONSTRUKSI MCQ (4)
Fokuskan soal-soal pada konsep penting dan kasuskasus yang sering serta memiliki potensi masalah serius. Setiap soal diusahakan agar menilai aplikasi pengetahuan, tidak hanya menguji daya ingat terhadap fakta terisolasi. Seluruh distraktor (pengecoh= jawaban salah) dibuat homogen (dari sisi gramatikal, taksonomi).
PANDUAN UMUM KONSTRUKSI MCQ (5)
Hindari kesalahan teknis membuat soal, seperti :
Pilihan jawaban terlau panjang, kompleks, atau ganda. Istilah yang rancu pada pilihan jawaban. Bahasa pada jawaban tidak parallel. Pilihan jawaban tidak menurut urutan logik. Bukan salah satu di atas atau semua benar digunakan sebagai pilihan jawaban
9
24/10/2012
PANDUAN UMUM KONSTRUKSI MCQ (6)
Pilihan pada jawaban memberikan petunjuk logik pada jawaban yang benar. Pilihan jawaban yang benar adalah frase atau kalimat terpanjang. Penggunaan kata-kata yang diulang atau memiliki makna sama baik pada soal maupuin pilihan jawaban. Strategi konvergens pada pilihan jawaban.
Item Development Item writer
Local Item Bank Administrator
Regional Item Bank Administrator
National Item Review
Item Bank 2
Panel expert
National Item Bank Administrator
Item Bank 1
Non scored items
Item Bank 3
Item Selection process
Question book
Item Bank 4
10
24/10/2012
A 60-year-old man is brought to the emergency department by the police, who found him lying unconscious on the sidewalk. After ascertaining that the airway is open, the first step in management should be intravenous administration of
A. B. C. D. E.
Examination of cerebrospinal fluid Glucose with vitamin B1 (thiamine) CT scan of the head Phenytoin diazepam
Grammatical cues: one or more distracters don’t follow grammatically from the stem
A 60-year-old man is brought to the emergency department by the police, who found him lying unconscious on the sidewalk. Ascertaining that the airway is open, tfirst step in management should be intravenous administration of:
A. B. C. D. E.
Examination of cerebrospinal fluid Glucose with vitamin B1 (thiamine) CT scan of the head Phenytoin diazepam
11
24/10/2012
In patients with advanced dementia, Alzheimer’s type, the memory defect
A.
can be treated adequately with phosphatidylcholine (lecithin)
B.
could be a sequela of early parkinsonism
C.
is never seen in patients with neurofibrillary tangles at autopsy
D.
is never severe
E.
possibly involves the cholinergic system
Absolute terms: terms such as “always” or “never” are used in options
In patients with advanced dementia, Alzheimer’s type, the memory defect
A.
can be treated adequately with phosphatidylcholine (lecithin)
B.
could be a sequela of early parkinsonism
C.
is never seen in patients with neurofibrillary tangles at autopsy
D.
is never severe
E.
possibly involves the cholinergic system
12
24/10/2012
Secondary gain is
A.
synonymous with malingering
B.
a frequent problem in obsessive-compulsive disorder
C.
a complication of a variety of illnesses and tends to prolong many of them
D.
never seen in organic brain damage
Long correct answer: correct answer is longer, more specific, or more complete than other options
Secondary gain is
A.
synonymous with malingering
B.
a frequent problem in obsessive-compulsive disorder
C.
a complication of a variety of illnesses and tends to prolong many of them
D.
never seen in organic brain damage
13
24/10/2012
A 58-year-old man with a history of heavy alcohol use and previous psychiatric hospitalization is confused and agitated. He speaks of experiencing the world as unreal. This symptom is called
A.
depersonalization
B.
derailment
C.
derealization
D.
focal memory deficit
E.
signal anxiety
Word repeats : a word or phrase is included in the stem and in the correct answer
A 58-year-old man with a history of heavy alcohol use and previous psychiatric hospitalization is confused and agitated. He speaks of experiencing the world as unreal. This symptom is called
A.
depersonalization
B.
derailment
C.
derealization
D.
focal memory deficit
E.
signal anxiety
14
24/10/2012
Local anesthetics are most effective in the A.
Anionic form, acting from inside the nerve membrane
B.
Cationic form, acting from inside the nerve membrane
C.
Cationic form, acting from outside the nerve membrane
D.
Uncharged form, acting from inside the nerve membrane
E.
Uncharged form, acting from outside the nerve membrane
Convergence strategy : the correct answer includes the most elements in common with the other options
Local anesthetics are most effective in the A.
Anionic form, acting from inside the nerve membrane
B.
Cationic form, acting from inside the nerve membrane
C.
Cationic form, acting from outside the nerve membrane
D.
Uncharged form, acting from inside the nerve membrane
E.
Uncharged form, acting from outside the nerve membrane
15
24/10/2012
Peer review committees in HMOs may move to take action against a physician‟s credentials to care for participants of the HMO. There is an associated requirement to assure that the physician receives due process in the course of these activities. Due process must include which of the following
A. Notice, an impartial forum, council, a chance to hear and confront evidence against him/her B. Proper notice, a tribunal empowered to make decision, a chance to confront witnesses against him/her, and a chance to present evidence in defense C. Reasonable and timely notice, impartial panel empowered to make a decision, a chance to hear evidence against himself/herself and to confront witnesses, and the ability to present evidence in defense
Options are long, complicated, or double
Peer review committees in HMOs may move to take action against a physician‟s credentials to care for participants of the HMO. There is an associated requirement to assure that the physician receives due process in the course of these activities. Due process must include which of the following
A. Notice, an impartial forum, council, a chance to hear and confront evidence against him/her B. Proper notice, a tribunal empowered to make decision, a chance to confront witnesses against him/her, and a chance to present evidence in defense
C. Reasonable and timely notice, impartial panel empowered to make a decision, a chance to hear evidence against himself/herself and to confront witnesses, and the ability to present evidence in defense
16
24/10/2012
Severe obesity in early adolescence
A.
Usually responds dramatically to dietary regimens
B.
Often is related to endocrine disorders
C.
Has a 75% chance of clearing spontaneously
D.
Shows a poor prognosis
E.
Usually responds to pharmacotherapy and intensive psychotherapy
Frequency terms in the options are vague (e.g. rarely, usually)
Severe obesity in early adolescence
A.
Usually responds dramatically to dietary regimens
B.
Often is related to endocrine disorders
C. Has a 75% chance of clearing spontaneously D.
Shows a poor prognosis
E.
Usually responds to pharmacotherapy and intensive psychotherapy
17
24/10/2012
In a vaccine trial, 200 2-year-old boys were given a vaccine against a certain disease and the monitored for five years for occurrence of the disease. Of this group, 85% never contacted the disease. Which of the following statements concerning these results is correct?
A.
No conclusion can be drawn, since no follow-up was made of nonvaccinated children
B.
The number of cases (ie, 30 cases over five years) is too small for statistically meaningful conclusions
C.
No conclusions can be drawn because the trial involved only boys
D.
Vaccine efficacy (%) is calculated as 85 – 15/100
Language in the options is not parallel; options are in non-logical order In a vaccine trial, 200 2-year-old boys were given a vaccine against a certain disease and the monitored for five years for occurrence of the disease. Of this group, 85% never contacted the disease. Which of the following statements concerning these results is correct?
A.
No conclusion can be drawn, since no follow-up was made of nonvaccinated children
B.
The number of cases (ie, 30 cases over five years) is too small for statistically meaningful conclusions
C.
No conclusions can be drawn because the trial involved only boys
D. Vaccine efficacy (%) is calculated as 85 – 15/100
18
24/10/2012
The diagnosis of a large ovarian cyst is most strongly suggested by
A.
an anterior dullness, lateral tympany
B.
a decreased peristalsis
C.
a fluid wave
D.
a shifting dullness
E.
none of the above
None of the above is used as an option The diagnosis of a large ovarian cyst is most strongly suggested by
A.
an anterior dullness, lateral tympany
B.
a decreased peristalsis
C.
a fluid wave
D.
a shifting dullness
E.
none of the above
19
24/10/2012
Arrange the parents of the following children with Down’s syndrome in order of highest to lowest risk of occurrence. Assume that the maternal age in all cases is 22 years and that a subsequent pregnancy occurs within 5 years. The karyotypes of the daughters are :
I. II. III. IV. V.
46, XX, -14, +T (14q21q) pat 46, XX, -14, +T (14q21q) de novo 46, XX, -14, +T (14q21q) mat 46, XX, -21, +T (14q21q) pat 47, XX, -21, +T (21q21q) (parents not karyotyped)
A. B. C. D. E.
III, IV, I, V, II IV, III, V, I, II III, I, IV, V, II IV, III, I, V, II III, IV, I, II, V
Stems are tricky or unnecessarily complicated Arrange the parents of the following children with Down’s syndrome in order of highest to lowest risk of occurrence. Assume that the maternal age in all cases is 22 years and that a subsequent pregnancy occurs within 5 years. The karyotypes of the daughters are :
I. II. III. IV. V.
46, XX, -14, +T (14q21q) pat 46, XX, -14, +T (14q21q) de novo 46, XX, -14, +T (14q21q) mat 46, XX, -21, +T (14q21q) pat 47, XX, -21, +T (21q21q) (parents not karyotyped)
A. B. C. D. E.
III, IV, I, V, II IV, III, V, I, II III, I, IV, V, II IV, III, I, V, II III, IV, I, II, V
20
24/10/2012
VIGNETTE: KLINIK, MASALAH KEDOKTERAN KELUARGA (1)
Sebuah vignette antara lain berisi deskripsi kondisi pasien, dan skenario klinik dengan beberapa informasi, yaitu:
Usia, Jenis kelamin (e.g. laki-laki 45 tahun) Tempat pelayanan (IGD)
VIGNETTE: KLINIK, MASALAH KEDOKTERAN KELUARGA (2)
NBME, p.38 or p.56
Keluhan utama Durasi
Riwayat Pasien Diagnostik fisik
• +/- hasil diagnostik • +/- terapi awal, temuan selanjutnya
21
24/10/2012
REFERENCE: Case and Swanson – Constructing Written Test Questions
www.nbme.org
22
