24/10/2012
Tim teknis
pass
fail
competent incompetent
Avoid false positive or false negative
1
24/10/2012
A standard is a statement about whether performance is good enough for a particular purpose
The standard defines the boundary between passing and failing; separates candidates who are competent from those who are not competent The numerical answer to: “How much is enough?” “How good is good enough?”
A Policy Decision The standard is based on judgments about examinees’ performances. Judgments are an expression of our professional, social and educational values.
2
24/10/2012
C H A R A C T E R I S T I C S
The method has to be: Defensible Credible Supported by a body of evidence Feasible Acceptable to all stakeholders
6 Langkah Penentuan Batas Kelulusan 1. Menentukan tipe standar 2. Menentukan metode penentuan nilai batas kelulusan 3. Memilih ‘judges’ 4. Proses penentuan batas kelulusan oleh ‘judges’ 5. Menghitung batas kelulusan 6. Menentukan follow up (Norcini, 2003)
3
24/10/2012
1. Menentukan tipe standar
Relative standards Norm-referenced Patokan Acuan Normal / PAN
• Absolute standards • Criterion-referenced • Patokan Acuan Baku / PAB
Relative standardsFeatures Absolute standards • Dinyatakan dalam jumlah atau persentase peserta yang lulus (mis: 100 peserta terbaik, 30% terbaik) • Performan seorang peserta tergantung pada performan peserta lain • Menstimulus kompetisi • Lebih mudah dilakukan • Tidak tepat untuk ujian berbasis kompetnsi
• Dinyatakan dalam persentase jawaban benar (mis: 65% jawaban benar) • • • •
Performan seorang peserta tidak tergantung pada performan peserta lain Menstimulus kolaborasi Lebih sulit dilakukan Tepat untuk ujian berbasis kompetensi
4
24/10/2012
2. Menentukan metode penentuan nilai batas kelulusan Relative standards
Relative methods
Absolute standards
Absolute methods 1. Keputusan didasarkan pada item soal 2. Keputusan didasarkan pada peserta ujian
Compromise Methods Contoh Hofstee’s method
3. Memilih ‘judges’
• Kredibilitas nilai batas kelulusan sangat tergantung pada kredibilitas panel ‘judges’, sehingga perlu diperhatikan: 1. Kapabilitas, komitmen dan kredibilitas judges 2. Aspek keterwakilan dari judges
5
24/10/2012
Step: 4. Proses penentuan batas kelulusan oleh ‘judges’ 5. Menghitung batas kelulusan 6. Menentukan follow up
STANDARD SETTING DALAM OSCE
6
24/10/2012
Borderline Regression Method • Increasingly popular in OSCE and performance based examination • In each station each examinee is assessed using two methods – Check list based method – Global performance
• Data from each station is aggregated • A regression equation is developed using global performance is independent variable and check list marks as dependent variable • Cut-mark (standard) is the boundary between borderline and pass
Global mark • Profesional Judgement : tentang kompetensi dan penampilan kandidat secara keseluruhan dalam station tertentu
• Global score sebaiknya tidak berkaitan/ independen dengan marking sheet / checklist • Penguji memutuskan apakah seorang kandidat melakukan semua instruksi secara fluently dan melakukan semua ketrampilan secara independent
7
24/10/2012
Borderline Regression Method
Items Explain to pt Position pt Inspection of skin Temperature change Pulsa- Femoral tion Popliteal Post. tibial Dors. pedis Capillary refill Total
Full mark Mark 1 1 1 1 2 2 1 1 1 1 1 1 1 0 1 1 1 1 10 9
Put the global rating from the overall performance in OSCE Fail (1) Borderline (2) Pass (3)
a
Outstanding (4)
Borderline Regression Method
Items Explain to pt Position pt Inspection of skin Temperature change Pulsa- Femoral tion Popliteal Post. tibial Dors. pedis Capillary refill Total
Full mark Mark 1 1 1 1 2 1 1 0 0 1 1 1 1 0 1 1 1 0 10 5
Put the global rating from the overall performance in OSCE Fail (1) Borderline (2)
a
Pass (3) Outstanding (4)
8
24/10/2012
excellent • Excellent performance of skill. Outstanding demonstration of technical and non-technical aspects of skill. Air of confidence and fluent
Very good pass • Very good performance of skill. Majority of the technical aspects of the skill demonstrated. Few minor and non-essential omission/errors. Examiner more than satisfied that candidate has passed station
9
24/10/2012
Clear pass • Acceptable performance of skill. Despite omission /errors demonstrated in performance of skill-safe to progress. At times can be formulaic in approach.
Borderline : ‘pass doubtful’ • Patchy performance of skill. Examiner undecided whether to pass or fail candidate. Demonstrated some aspects of the skills however omissions and inaccuracies occurred in their performance of the skill. Often formulaic in approach and struggled with performing skill
10
24/10/2012
Clear fail • Performance of skill did not come up to a passing standard. Appeared disorganised. Unsafe and unsuitable to progress.
Borderline Regression Methods
11
24/10/2012
Global Rating dalam Checklist OSCE UKDGI • 3 = Pass (excellent , very good) • 2 = Borderline (doubtful) • 1 = Fail (clear fail)
Deskripsi global mark OSCE UKDGI • Pass/lulus : penampilan kandidat dalam mengerjakan semua instruksi cukup lancar, tidak melakukan kesalahan yang fatal dan menggunakan hampir semua alat atau bahan yang essential dalam satu tindakan • Borderline/pass doubtful : penampilan kandidat mengalami banyak keragu-raguan, terdapat beberapa kesalahan dalam melakukan tindakan sehingga penguji mengalami keraguan dalam memberi nilai
12
24/10/2012
• Fail / tidak lulus : penampilan dalam melakukan suatu tindakan tidak memenuhi persyaratan standar, tidak terorganisir, tidak aman
CONTOH SCORE CANDIDAT PADA STATION 7 TRY OUT OSCE OKT 2011
NO 1 2
3
4 5 6
7
ITEM
FULL MARK
MARK
Memperkenalkan diri Identitas pasien Keluhan utama Riwayat penyakit saat ini Riwayat gigi tiruan Riwayat penyakit Rencana perawatan
2
0
2
2
2
2
Put the global rating from the overall performance in OSCE
2
0
Fail (1)
2
2
2
0
2
2
TOTAL SCORE
14
8
Borderline (2)
V
Pass (3)
13
24/10/2012
STATION 7 16 14
R square : 0.616
12
Total Score
10 8
7,96
Total Score Predicted Total Score
6
Linear (Predicted Total Score)
4 2 0 0
1
2
3
Performance Scale
14
