Comparison of Objective and Subjective Methods on Determination of Differential Item Functioning

Research objective is comparing the objective methods often used in literature for determination of differential item functioning (DIF) and the subjective method based on the opinions of the experts which are not used so often in literature. Mantel-Haenszel (MH), Logistic Regression (LR) and SIBTEST are chosen as objective methods. While the data of an extensive examination in Turkey applied for objective methods, the data that are obtained from Expert Opinions Form used to evaluate the items of the same examination. The data obtained from 5077 female and 5271 male students are used for the objective methods, and 23 experts' opinions are used for subjective method. The concordance between the objective and subjective methods is calculated by using the compatibility rate and Cohen's kappa coefficient in the research. While the highest concordance related to the existence of DIF is obtained between MH and SIBTEST methods (.90; κ=0,79) and the lowest concordance is between LR and SIBTEST methods (.75; κ=0,50) in objective methods, When the concordance of the objective method with the subjective methods is examined, at least moderate concordance (.75; κ=0,47) is obtained in the decision. When items which have DIF is examined according to DIF level, three items indicate low level of DIF and one item indicates moderate or high level of DIF for both methods. In addition, in subjective method, a decision study is made on the number of the experts presenting opinion within the generalizability theory and the acceptable reliability value is reached with 13 experts' opinions.