Testing invariance between web and paper students satisfaction surveys: A case study

Abstract

Purpose: This paper studied the measurement invariance (MI) across web-based and paper-based surveys to evidece if both techniques of data collection can be regarded as equivalent.

Design/methodology/approach: We develop a multigroup confirmatory factor analysis (MGCFA) with Maximum Likelihood Estimation to asses meassurement invariance  of the Job Diagnostic Survey (JDS) adapted to teaching, with data collected from paper and web surveys. Sample from paper surveys was constituted by 294 student of a Spanish public university in the academic years 2007-08, 2008-09 and 2009-10. Internet surveys were administered through an open source survey application called LimeSurvey. We received 241 completed questionnaires.

Findings: Results show that metric invariance, covariance invariance, variance of latent factors invariance and measurement errors invariance can be established between two groups. We can conclude that both methods of collecting data can be considered equivalent.

Research limitations/implications: This study was done with a particular sample and strict focus questionnaire and we might not generalize the findings. It should be extended in the future to include other universities and graduate students.

Originality/value: Results showed that the factor structures remained invariant across the internet-based and paper-based groups, that is to say, both methods of collecting data can be considered equivalent, with the same factor structure, factor loadings, measurement errors of factors and the same reliability. These findings are useful for researchers since they add a new sample in which web and paper questionnaires are equivalent and for teachers to desire to change the teaching methodology at university, encourage students’ participation and teamwork through active methodologies.