The PISA project has in the last decade steadily increased its influence on educational policies in participating countries. Tables of PISA scores and country rankings are often taken at face value, not only in media, but also by policymakers and politicians. The PISA undertaking is a well-funded multinational “techno-scientific” machinery, there are few critical voices and very limited informed debate and critical research. This article will raise some of the concerns that would require further, more systematic scrutiny. In this article, I will start by raising some basic questions about why OECD has initiated the PISA project. The intentions of PISA are, not surprisingly, strongly related to the overall political aims OECD and the underlying commitment to a competitive global free market economy. Examples are given on how PISA exerts influence on educational policy. Then I will provide critical points of two categories. The first category relates to the PISA project as such. These problems are inherent in the PISA undertaking, and hence cannot be “fixed”. I will argue that it is impossible to construct a test that in a fair and objective way can be used across countries and cultures to assess the quality of learning in “real-life” situations with “authentic texts”. Problems arise when the brave intentions of the PISA framework are translated to concrete test items to be used in a great variety of languages, cultures and countries. The requirement of “fair testing” implies by necessity that all local, current and topical issues must be excluded. This runs against most current thinking in e.g. science education, where “science in context” and “a localized curriculum” are ideals promoted by e.g. UNESCO, educators as well as in many national curricula.