TEST RELIABILITY AND MEASUREMENT ERRORS IN ARTIFICIAL INTELLIGENCE-GENERATED (AI-G) AND TEACHER-MADE TESTS: A COMPARATIVE STUDY

Abstract

Reliability and Standard Error of Measurement (SEM) can enhance quality of test items by assessing the consistency, accuracy, and precision of test items. This study investigated the differences in the reliability and standard error of measurement of artificial-intelligence-generated and teacher-made tests in measuring students’ achievement in Business Studies. This research raised two research questions and two hypotheses tested at 0.05 significance level. The study adopted a quasi-experimental, post-test-only non-equivalent group design. A multistage sampling technique selected a sample size of 969 junior secondary school two Business Studies students in Ogun State public secondary schools. Data collected was from four forms of 50-item Multiple-Choice Questions (ChatGPT, Gemini, Claude and Teacher-made). The tests went through review by seven subject matter experts. KR-20 reliability coefficients of the pilot test instruments were Teacher-made 0.78, ChatGPT 0.85, Claude AI 0.70 and Google Gemini 0.73. Data were analyzed using mean, standard deviation, Feldt’s test and Bootstrap SEM referencing. Findings showed that ChatGPT and Google Gemini tests had similar reliability and SEM to the Teacher-Made test. Based on the findings, recommendations are Google Gemini and ChatGPT can be used to generate multiple-choice test items in Business Studies.