Alice goes down the rabbit hole (illustration by Tove Jansson, © Moomin Characters, reproduced with permission)

There’s a scene in Alice’s Adventures in Wonderland where she has just eaten from the Caterpillar’s mushroom in the hopes of returning her body (which has shrunk to a mere three inches) to its usual size. The attempt goes awry, with her neck extending to a disproportionate degree. With her head now well above the tree line, Alice encounters the Pigeon, who accusingly squawks ‘Serpent!’ ‘But I’m not a serpent, I tell you’, Alice responds indignantly, pointing out that she is, in fact, ‘a little girl’. ‘A likely story indeed!’ the Pigeon demurs. ‘I’ve seen a good many little girls in my time, but never one with such a neck as that! No, no! You’re a serpent; and there’s no use denying it. I suppose you’ll be telling me next that you never tasted an egg!’ Alice is compelled to admit that she has indeed eaten eggs, but insists: ‘little girls eat eggs quite as much as serpents do, you know’. ‘I don’t believe it’, the Pigeon says; ‘but if they do, why, then, they’re a kind of serpent, that’s all I can say’ (Carroll [1865] 1980, 48–49).

Alice and the Pigeon (illustration by Tove Jansson. © Moomin Characters, reproduced with permission)

At the heart of this encounter is the question of what differentiates a girl from a serpent. Although nonsensical in form, it serves to make explicit both the act of classification and the standards it entails. What is the standard for girldom? For the Pigeon, one can’t be a girl if one has an extremely long neck; one can’t be a girl if one eats eggs. Once made explicit, these standards appear to Alice as quite ridiculous, although she is hard pressed to convince the Pigeon of that, primarily because she can’t articulate how she knows she is a girl.

Yet, if Lewis Carroll reveals the absurdity of such standards, he also illustrates the dilemmas posed by their absence. This is nowhere more apparent than when Alice enters the Caucus-race. To initiate the race, the Dodo marks out a rough circle: ‘The exact shape doesn’t matter’, he breezily informs the assembled crowd. The participants are placed haphazardly along the course and then, without further ado, they begin running about all over the place. Approximately half an hour later, the Dodo declares that the race is now over and the participants quiz him about who has won. The Dodo ponders this for a few moments and rules that ‘Everybody has won, and all must have prizes’ (p. 26, emphasis in original). Clearly, without standards (for starting, for stopping, for the race track itself), the race has become something else entirely.

Alice in the Caucus-race (illustration by Tove Jansson. © Moomin Characters, reproduced with permission

Whether intended as a commentary on standards or not, I can think of few better illustrations of their potential absurdities than Carroll’s work. Take, for example, journal standards—a topic in which I have become increasingly interested. While quantifying, classifying, and formalizing practices are a core feature of contemporary life (Lampland and Star 2009; Busch 2011), we are currently witnessing an unprecedented concern with formalized academic standards, and accompanying practices of quantification, with individual scholars increasingly caught in a web of metrics that attempt to quantify their ‘quality’ and ‘impact’ (see Strathern 2000; Shore 2008; Burrows 2012; Bell 2015). In what follows, I present some cultural artifacts that I hope tell a story about journal standards, inspired by Carroll’s work and Martha Lampland and Susan Leigh Star’s 2009 book Standards and Their Stories.

Exhibit 1. The high-profile journal

This is a screen shot from the website of the journal Social Science & Medicine. I draw your attention to the ‘Journal Metrics’ section and the four scores listed within it. Each of these scores represents a distinct (albeit interconnected) standard, although the most influential is the journal’s ‘impact factor’. This number means that papers published in Social Science & Medicine in 2012 and 2013 received an average of three citations in 2014. Journal impact factors are tabulated on an annual basis and journals are ranked accordingly in the Journal Citation Reports. While impact factor can theoretically be calculated for any journal over three years of age, Web of Science (formerly the Institute for Scientific Information) invented it. Thus, only the journals it indexes can use it.

Exhibit 2. The Thomson Reuters selection process

(http://wokinfo.com/essays/journal-selection-process/)

To be indexed in a Web of Science database (for example, the Social Sciences Citation Index), journal editors must submit an application demonstrating that their journal meets certain standards. The description provided at the link details these standards and includes statements such as ‘It would appear that to be comprehensive, an index of the scholarly literature might be expected to cover all journals published … [but] a relatively small number of journals publish the majority of significant scholarly results’. As one might expect of a commercial database, these standards lean strongly towards factors such as the journal’s commercial viability and indexability. Citation analysis is also conducted. To quote from the description, ‘Thomson Reuters looks for citations to the journal itself, as expressed by impact factor and/or total citations received’; in other words, to get an impact factor, a journal should already effectively have one.

Exhibit 3. Article on the ERA journal-ranking system

(http://www.theaustralian.com.au/higher-education/end-of-an-era-journal-rankings-dropped/story-e6frgcjx-1226065864847)

This article details the demise of the journal-ranking system in Australia proposed under ERA (Excellence in Research Australia), the national research-evaluation framework. Under this system, all academic journals were ranked using four tiers of ‘quality’ rating: A*, A, B, and C; naturally, impact factor was a key consideration in the rankings individual journals received. It was ultimately dropped, with the then minister of Innovation, Industry, Science and Research admitting that ‘There is clear and consistent evidence that the rankings were being deployed inappropriately within some quarters. … One common example was the setting of targets for publication in A and A* journals by institutional research managers’. Despite their formal disavowal, according to various reports (see for example Bowrey 2013), individual institutions are still using these rankings as a management tool.

Exhibit 4. Analysis: Is ‘impact factor’ a valid proxy for scholarly impact?

(http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1781329/)

This article discusses the debate about whether impact factor is a valid proxy for the impact of any given journal article, and discusses the results of the authors’ analysis of a random sample of 323 articles selected from 17 social work journals. They found that ‘The IFS [impact factor score] was the best predictor of both short- and long-term impact’. On the basis of this analysis, the authors conclude: ‘Whether the disciplines should dump the IFS [impact factor score] or not is a question that deserves further empirical attention, but if your tenure and promotion committee uses this approach in making academic employment decisions, you may want to gather the relevant literature and forward it to them’. 

Exhibit 5. Publish or Perish software

(http://www.harzing.com/pop.htm)

‘Publish or Perish’ is a software package that retrieves and analyzes academic citations. The description on their website states: ‘Are you applying for tenure, promotion or a new job? Do you want to include evidence of the impact of your research? Is your work cited in journals which are not ISI [Institute for Scientific Information, now Web of Science] listed? Then you might want to try Publish or Perish, designed to help individual academics to present their case for research impact to its best advantage’.

Exhibit 6. Impact Factor Services for International Journals

(http://ifsij.com/)

Impact Factor Services for International Journals is a new organization – one of a growing number – that operates as an impact-factor measuring service for international journals. According to their website, ‘We are not claiming that our impact factor is better than other agencies providing it, we just calculate the impact factor according to our own process and methods’. In fact, for fees between USD$25–40/year, nine distinct kinds of ‘factors’ can be evaluated: Impact Factor, Journal Total Quality, Journal Regularity, Journal Indexing, Journal Popularity, Journal Technical Quality, Journal Citation, Journal Editorial Quality, and Journal Originality.

Taken together, these exhibits demonstrate the core attributes of standards: their integrated and nested qualities, their connection with particular ethics and values (in this case, competitive and commercial ones), the ways they quickly become naturalized, and their very real effects on the ground (Lampland and Star 2009). Although Exhibit 6 has typically been taken as an affirmation of the need for standards, it is clearly a consequence of the fetishization of the impact factor as a measure of quality. As Star and Lampland (2009) observe, people don’t just passively comply with standards – they develop workarounds and engage in formal rather than substantive compliance. Put in Lewis Carroll’s terms, standards won’t necessarily get you red roses; they’ll get you white roses painted red.

The Queen’s gardeners painting the roses red so she doesn’t chop off their heads (illustration by Tove Jansson. © Moomin Characters, reproduced with permission)

Acknowledgements

The illustrations were created in 1966 by the Finnish author and illustrator Tove Jansson for a Swedish edition of Alice’s Adventures in Wonderland (Alice i Underlander). Thanks are due to Eileen Moyer for the invitation to submit a piece to The Nightstand, Jenna Grant for her help with nailing down a focus, and Denielle Elliott and Erin Martineau for their feedback and suggestions.   

About the Author

Kirsten Bell is a Research Associate in the Department of Anthropology at the University of British Columbia in Vancouver, Canada. Her growing interest in quantifying, classifying, and formalizing practices in the academy is largely due to her role as a co-editor of Critical Public Health (yes, it has an impact factor; yes, the impact factor is prominently featured on the journal’s website), and the behind-the-scenes access it has afforded to the interlocking standards that increasingly dictate the form – and content – of academic knowledge production. 

References

Bell, Kirsten. 2015. ‘An Academic Diary in the Strictest Sense of the Term’. Centre for Imaginative Ethnography, Literary Experiments in Ethnography Series, 30 June. http://imaginativeethnography.org/imaginings/literary-experiments-in-ethnography/d-june-30-kirsten-bell/.

Bowrey, Kathy. 2013. ‘Audit Culture: Why Law Journals Are Ranked and What Impact This Has on the Discipline of Law’. Legal Education Review 23, no. 2: 291–308. http://www.austlii.edu.au/au/journals/LegEdRev/2013/14.html

Burrows, Roger. 2012. ‘Living with the H-Index? Metric Assemblages in the Contemporary Academy’. The Sociological Review 60, no. 2: 355–72. http://dx.doi.org.10.1111/j.1467-954X.2012.02077.x.

Busch, Lawrence. 2012. Standards: Recipes for Reality. Cambridge: MIT Press.

Carroll, Lewis. (1865) 1980. Alice’s Adventures in Wonderland: Through the Looking Glass and Other Writings. Sydney: Book Sales International.

Lampland, Martha, and Susan Leigh Star, eds. 2009. Standards and Their Stories: How Quantifying Practices Shape Everyday Life. Ithaca, NY: Cornell University Press.

Shore, Cris. 2008. ‘Audit Culture and Illiberal Governance: Universities and the Politics of Accountability.’ Anthropological Theory 8, no. 3: 278–98. http://dx.doi.org/10.1177/1463499608093815.

Strathern, Marilyn, ed. 2000. Audit Cultures: Anthropological Studies in Accountability, Ethics and the Academy. London: Routledge.