Soubhik Barari, Ph.D., Eric Newsom, M.S.Ed., Ji Eun Park, Ph.D. and Susan M Paddock, Ph.D.
Higher education has long been viewed as a stepping-stone for individuals to achieve professional and personal goals. To demystify the options within the higher education landscape, prospective students and their families often turn to college rankings. This is especially true in an era of increasingly expensive college costs (Franke, 2017). Several major college rankings providers have recently revised their methodology to better reflect these cost concerns, along with making other updates. While this effort may be laudable, any proposal for substantive changes in rankings raises the question of the underlying qualities encapsulated by the rankings and whether the ascribed methodology can accurately capture those qualities.
Given these potential issues, we conducted a methodological review of five prominent college rankings systems: U.S. News & World Report (USNWR), Wall Street Journal (WSJ), Forbes’ Top College list (Forbes), The Times Higher Education World University Ranking (THE), and the QS World University Ranking (QS).
Our overarching approach is to assess the construct validity of college rankings. Construct validity is a crucial property in the development of numerical scales, such as rankings, that aim to quantitatively describe otherwise abstract qualities of social entities such as ‘market value’ or ‘educational quality.’ Our assessment includes a review of conceptual, data, and methodological factors within this framework that are central to developing a methodologically sound ranking of colleges, drawing examples from our five illustrative ranking systems. Here, we summarize five key issues spanning the conceptual, data, and methodological aspects of construct validity along with an initial recommendation to address each (expanded further in Section 5)