Teaching quality in higher education: Do student evaluation of teaching questionnaires allow a reliable and valid assessment of teaching quality?

Improving teaching quality is a relevant topic in society to nurture the students’ innate potential in the best possible way. One often applied tool for assessing teaching quality in higher education is student evaluations of teaching (SETs) that are used as a criterion for making important decisions in higher education such as employing teachers, distributing funds, and making changes in the curriculum. Despite their effect on decisions it is relatively unclear if SETs are a valid and reliable assessment of teaching quality. Therefore, the current dissertation examines the influence of student characteristics that are not conceptually related to teaching quality in two ways. First the variance of students was estimated through cross-classified multilevel models. This type of analysis allows a direct estimation of variance attributed to the students while separating it from the residual variance. Second selected student characteristics were added as predictors to estimate their effect on SETs and therefore to enhance the interpretation of their validity. In all studies a standardized German questionnaire was applied as the SET instrument. The first study investigated the reliability of SETs as measurement of teaching quality which was operationalized as interrater reliability calculated from intra-class correlations. It was assessed by comparing the variance components of teachers, courses, and students. The instrument can be considered reliable only when a high proportion of variance is explained by teachers and courses, however not by students. The study revealed that teachers and courses were large sources of variance in different dimensions of the questionnaire. This result suggests that SETs are reliable instruments if a sufficient number of students (at least 24) evaluate a teacher and course. Moreover, the study also revealed students and the interaction of students and teachers as relevant sources of variance. This finding implies that student characteristics and the individual fit between students and teachers can affect SETs and should be considered in the ongoing discussion of the instrument’s validity. The latter two studies examined the validity of SETs. The second study focused on student’s characteristics, content expectations and prior subject interest. Both variables are theoretically unrelated to teaching quality because they are outside of the teacher’s sphere of influence. Therefore they should not affect SET results. This means that the inclusion of both variables as predictors into cross-classified multilevel models should not lead to significant fixed effects. The second study revealed significant but weak effects of both variables on SETs and thus did not pose strong threats to the validity of SETs. The third study also addressed the validity of SETs, extending the research of prior subject interest by measuring it twice, once at the beginning of the course and a second time retrospectively at the time of evaluation. Additionally, likability of the teacher perceived by students was measured once at the beginning of the course and a second time at the typical time of evaluation toward the end of the semester. Similar to the results from Study 2 a weak effect was found of prior subject interest on the validity of SETs. However, the likability of teachers had a strong effect on the validity of SETs at both times of measurement. This result was interpreted as a strong effect of likability, which is theoretically unconnected to teaching quality and thus undermines the validity of SETs. The results of the three studies suggest that SETs are a reliable instrument given a sufficient number of evaluations. Only the likability of the teacher perceived by students strongly affected the SETs and hence was a solid threat to its validity as a measurement of teaching quality. Conclusively, decisions should not be justified solely by SETs.