25 August 2010

Melody Dye on the "Politics of Ideas" (part 3)

Melody Dye at Child's Play continues with part 3 of her series on "the politics of ideas" (her description).

Her first two installments are here and here; they critique anonymous peer review as currently practiced in cognitive science and psychology (but with obvious relevance to other fields, including, although Dye hasn't mentioned this, to the humanities).

I have linked to and discussed these earlier installments here and here.

Dye introduces her third post as follows:
In earlier posts, I discussed how certain crooked editorial practices can effectively subvert the review process, and how lack of transparency in review breeds precisely the kind of culture that anonymous-review was designed to undermine.  Today, I address the question of why these problems exist in the first place, and explore how changing the culture of review may also change the culture of the field — for the better.
She explicitly limits herself in this post to the "cultures" of psychology and cognitive science, but the problems she describes are not limited to those fields, and her comments are of special relevance to the humanities, which are, it seems to me, even more caught up in a culture of "camps."

Some quotes:
In psychology, there are certain entrenched camps that will only cite each other with any regularity, and will, as a matter of course, not cite researchers outside their group, even when they conduct research on the same topics.  This is a clever means of ensuring that scientists with similar ideas rise to the top, while competitors stay down. Since citation rates are often taken as a measure of merit and impact, well-cited papers will then become even better cited papers.
This will sound depressingly familiar to my readers in the academic humanities.

To investigate the make-up and history of these "camps," she points to Neurotree, which constructs "family trees" for a given researcher (Steven Pinker, in her linked example).

(So far as I know, there is no similar "genealogical" resource for my former field of musicology.  I constructed a similar genealogy of scholars of 18th-century music in my lamentably unpublished introduction to the collection Music in Eighteenth-Century Austria. It was quite enlightening.)

Regarding these genealogies, Dye comments:
With many cognitive scientists, there are surprisingly close parallels between their citation records and their ‘ancestral’ relationships (both top-down and across the page).  I say surprising, because while you might expect a certain degree of continuity, the degree you find – particularly among researchers at elite institutions – is almost astonishing.

[...]

...the relationships established between senior colleagues and their youthful protégés can appear downright incestuous.  As happens quite frequently, the research questions of one generation are unloaded straight on down to the next, without criticism or comment.  Surely the young camp of researchers has new methods up their sleeves, and maybe new math, but they’re humming along to the strains of an all too familiar tune.  And this happens even when it’s clear that “the same old song” isn’t solving anyone’s problems anymore.
The old guard is naturally (in a human sense) resistant to considering that they may have spent their
professional lives chasing chimeras. Thus, as Dye writes:
And so this is something like what happens: The elder scientists stay close at the heels of the young, and handpick the bright new stars and talents that best embody their visions of the future (–which, by necessity, are visions of the past).  These chosen few are then shepherded through the hiring and tenure processes at the top schools, and cited and quoted by their guardians in all the best journals, and nominated for the most prestigious awards (–of the kind that not just anyone can apply for, of course).  And in this way, the senior scientists keep a steady grip on the coming age, which holds the bright and steady promise of their chosen descendants, who will hold the reigns over the next generation.
And this feeds directly into the publication process, including peer review.

There is much more, including a story (from Dye's own experience) of the world of art galleries, in which a wealthy "gallerist" describes that world to her as one of "insider trading" and legal and socially acceptable "money laundering."

More quotes:
Given that psychology has yet to establish a firm tradition of inquiry, it is critical that we discover — empirically — what the best theoretical modes and investigative approaches are in grappling with the study of mind.  This cannot be resolved by fiat, and it should not be decided by politics or popular theories, which may well turn out to be wrong.

No: what is required is a culture of honest and competent reviewing, which would allow for the dissemination of research on the basis of scientific rigor and advance.

But such a culture does not exist in psychology, and it cannot, so long as we maintain a system of anonymous peer review.  The system was put in place to protect us from the spite and vindictiveness of individuals, but it has become a system that perpetuates and intensifies our politics, stratifies our ranks, and allows us to forget the humanness of each other, and the joint purpose that we, as scientists, share.
So far as I can see, this passage (and indeed, her entire post) would be equally valid (and perhaps even more so) if one substituted the name of any field of the humanities (including "musicology") for "psychology." (Although, in undeserved deference to the general distaste among scholars in the humanities for the label "scientist," we can also change "scientists" to "scholars" in our recasting of Dye's sentences.)

She goes on to offer specific suggestions on how to reform the review process in psychology. She makes a powerful case that reviewers should not be anonymous, and that reviews and responses should be made available online.

Anyone interested in the politics of academic fields, including the humanities, and the corruption of the process of review should read this courageous post.
Bookmark and Share

11 comments:

  1. It is in my view an illusion to search for an objective evaluation process in humanities. It is always a subjective choice in the end, and at its best. Great scholars in humanities are those who encourage the individuality of their pupils their specific "search desire" etc., it is only the individuality of the researcher and the originality of the research question, which is able to enrich this science in opposition to (mechanic) methods of natural science. The underlying ethic is human rights.

    ReplyDelete
  2. Many thanks for the comment (and good luck with your blog; I've subscribed).

    The position you outline is, indeed, very widely held among scholars in the humanities (it would be interesting to do a survey).

    However, writings in the humanities are full of statements that are empirically verifiable or falsifiable, and these writings are also (lamentably) full of statements that are logically incoherent. One can hold these statements to account, just as is done (at least ideally) in the sciences and social sciences; it is not "all a matter of opinion."

    In that way, the humanities do not differ from any other field, except that in the humanities, ordinary standards of verifiability and coherence are all too often disregarded, even in peer review.

    Some in the humanities like to pretend that the humanities lie in some "meta" realm, that is not subject to these standards, and from which one can "critique" or "theorize" about those standards without being subject to them.

    But there is no such meta-realm of discourse from which such critiques can be made. The notion that such a meta-realm exists is an illusion--a convenient one, as it turns out, for those who would distort and manipulate the systems of review, evaluation, and hiring for their own benefit or that of their students.

    I'll be writing more about this here as time goes on.

    ReplyDelete
  3. you said "However, writings in the humanities are full of statements that are empirically verifiable or falsifiable, and these writings are also (lamentably) full of statements that are logically incoherent. One can hold these statements to account, just as is done (at least ideally) in the sciences and social sciences; it is not "all a matter of opinion."

    Of course yes, but would this not be a matter of form rather than a matter of objecitivity in choice. And it could be put in a catalogue of basic formal requests or rules. It might even exist already somehow, but there seems to be no time to execute it because of constant underfunding in the humanities.
    At the turn of the 20th century in the arts in Europe, new schools were formed outside academic institutions. It is maybe necessary to form new groups within the humanities now (outside wellknown institutions), who hold up to classic academic standards again. (pls dont ask where the money should come from :))

    ReplyDelete
  4. money:
    http://cordis.europa.eu/fp7/dc/index.cfm?fuseaction=UserSite.FP7CallsPage

    ReplyDelete
  5. money, its this one>
    New Call for proposals - ERC Starting Grants
    Press release "New opportunities for early-career top researchers from anywhere in the world: €661 Mio for new "ERC Starting Grant" call (20/7/2010

    http://erc.europa.eu/

    ReplyDelete
  6. I'm not sure I entirely understand the first part of your point ("Of course yes....").

    But thanks very much for the link to the ERC call for proposals.

    ReplyDelete
  7. You are welcome. (Of course...): It is from a European point of view.

    ReplyDelete
  8. formal criterias in the evaluation process>
    - plagiarism (which is an increasing problem)
    - correct citation
    - there should be some criterias to evaluate statistic data
    - correct logic development of thesis and statement of defense: this would actually mean
    to redevelop rhetoric art
    ...etc.

    low funding as reason for bad quality in this process occurs probably only within underfunded and overcrowded European universities

    ReplyDelete
  9. All of those criteria seem important, especially the last: the logic and rhetoric of argument, a topic to which I've given a lot of thought.

    I find also that there is a very strong tendency in the humanities to make broad general statements without giving any (or sufficient) evidence to support them.

    Scholars in the humanities are generally hopeless with statistics, of course.

    I think we might also add to your list the checking of verifiable facts.

    I gathered that you were writing from a European perspective ;-)

    I lived in Vienna for several years, and taught at Cardiff University for a couple of years, but I'm now somewhat out of touch with the situation in European universities, particularly in the humanities.

    ReplyDelete
  10. Reapply!
    http://www.cardiff.ac.uk/jobs/music/lecturer---senior-lecturer-eighteenth-century-studies-2196.html

    ReplyDelete
  11. Hmmm, that's interesting.

    To be honest, I became so tired of having my applications ignored, even when it sounded as if the job description had been written with me in mind, that I gave up applying. I've also been more or less abandoned by most of those who might have acted as references for me.

    Still, I need a job, I belong in a classroom, and Cardiff wouldn't be so bad if I had a car (which I didn't last time).

    ReplyDelete