Title: Validating Wordscores

Abstract: Wordscores is a popular quantitative text scaling method to estimate parties' positions on a priori specified dimensions, without requiring the researchers to read or even understand the language in the documents they are analysing. This study tries to establish whereas Wordscores is able to deliver this promise by conducting a rigorous validation of its output using the Euromanifestos of 164 parties across 23 countries. We assess content validity by looking at the scored words in their context, criterion validity by comparing the Wordscores output to expert surveys and other judgemental estimates of party positions, and construct validity by using the Wordscores estimates to predict party membership in the European Parliament groups. We conclude that, despite the promises, Wordscores fails to deliver valid party positions, and outline three conditions under which its performance can be improved.