Effects of Language on CATS Performance

The University of Puerto Rico, Mayaguez (UPRM) is an officially bilingual university where engineering classes may be taught in Spanish, English, or a combination of both languages. Spanish is the home language of 91% of undergraduates at UPRM. Because of low performance on the Concept Assessment Tool for Statics (CATS) (around 29% compared to the results of other institutions, which ranged from 30-70%), a Spanish version of CATS (CATS-S) was developed. Ten Hispanic senior civil engineering students were recruited and divided into two groups, control and experimental. The control group was assigned the original CATS version, while the experimental group received the CATS-S. For both instruments, questions were given in open-ended format. The control group scored an average of 51% on CATS, and the experimental group scored an average of 24% on CATS-S. That is, native-Spanish speakers did worse on CATS-S than CATS. This study seeks to understand the role of language in the performance of these students by comparing their performance on CATS and CAT-S, answering the following questions: 1. What are the linguistic discrepancies between CATS and CATS-S? How might these discrepancies lead to differences in performance? 2. Can the poor performance of students on CATS-S be explained by the linguistic discrepancies found in question 1? Linguistic analysis took place by looking at syntactic and semantic differences between the English on CATS and the Spanish in CATS-S. This analysis led to the identification of the following linguistic discrepancies: (1) semantic assonance, (2) inconsistent terminology, and (3) changed information. Performance by the control group on each item of CATS was compared with performance by the experimental group on each item of CATS-S. We deduced that where there was a major discrepancy in performance between the control and experimental groups, language could be playing a role in performance. If both the control and experimental group did poorly on a question, we deduced that this was likely because of a lack of conceptual knowledge, not the language of the question. This analysis determined that lack of conceptual knowledge was the main reason for poor performance on both CATS and CATS-S, but that language did play a role in poor performance on certain CATS-S questions. Suggestions for a revised version of CATS-S and a next phase of testing are made.