Open Scholarship

In one of the readings for this week, Kathleen Fitzpatrick considers how digital-based works and open scholarship fit poorly into current tenure processes. She surprised me by not simply discussing how new forms of scholarship could be accommodated within the standard review process for tenure or promotion but by pointing out how the review process itself could be reformed to absorb changes in the scholarly world on an ongoing basis.

One of the gems in her recommendations is this:

(4) Dance with the one you brought.

In the same way that the work demands to be dealt with on its own terms, it’s crucial that tenure review processes engage with the candidates we’ve actually hired, rather than trying to transform them into someone else. While it’s tempting to advise junior scholars to take the safer road to tenure by adhering to traditional standards and practices in their work, such advice runs the risk of derailing genuinely transformative projects. Particularly when candidates have been hired into positions focused on new forms of research and teaching, or when they have been hired because of the exciting new paths they’re creating, those candidates must be supported in their experimentation. In creating that support, it’s particularly important to guard against doubling the workload on the candidate by requiring them both to complete the project and to publish about the project, or worse, to complete the project and do traditional work as well. This is a recipe for exhaustion and frustration; candidates should be encouraged to focus on the forms of their work that present the greatest promise for impact in their fields.

I thought it an interesting perspective on how to allow junior faculty to engage in high-risk work that suits their scholarly expertise. The current tenure system probably helps skew the production of scientific work in a way that fits the “game”. As Novak, Spies, and Motyl put it, “Novelty and positive results are vital for publishability but not for truth.” Behavior that supports scientific accuracy is often at odds with behavior that supports professional success.

Smaldino and McElreath in “The natural selection of bad science”:

Institutional guidelines for evaluation at least partly determine how researchers devote their energies, and thereby shape the kind of science that gets done.

Smaldino and McElreath attempt to address why, in spite of decades of attempted reform, scientists continue to publish underpowered research (experiments that use so few trials or subjects that they have a high likelihood of producing spurious results). They conclude that educating scientists about good statistical design does not work. Research reform will require institutional-level changes in how publications are incorporated into evaluation for professional advancement.

 

Similar issues arise in the evaluation of collaborative research, particularly in team science. The NIH suggests writing expectations right into a tenure-track offer letter for anyone who is expected to engage in collaborative research. The reality is that many university faculty and administrative staff are actively opposed to allowing their faculty engage in collaborative research.

Kara Hall drew attention to this in a presentation she gave when accepting an award for her contributions to the science of team science. She includes quips she has heard about collaborative science, such as “We value team scientists – they just shouldn’t get tenure.” She later reveals (spoiler alert) that these quotes all came from high-powered administrators, faculty, and members of funding agencies; the quote I included came from a university dean.

Dr. Hall advocates creating institutional-level recognition and incentives to drive team science, though she recognizes the challenges in creating evaluation criteria. I found one example that she used particularly poignant. A 2009 New York Times Magazine article she cites described basketball player Shane Battier as the “No-Stats All-Star.”

Here we have a basketball mystery: a player is widely regarded inside the N.B.A. as, at best, a replaceable cog in a machine driven by superstars. And yet every team he has ever played on has acquired some magical ability to win.

He was acquired by the Houston Rockets when they started applying Moneyball-style stats to locate the most undervalued players. What they found was that he doesn’t dribble or shoot especially well, and while his defense is decent, where he excels is in making his teammates play better at both offense and defense:

Battier’s game is a weird combination of obvious weaknesses and nearly invisible strengths. When he is on the court, his teammates get better, often a lot better, and his opponents get worse — often a lot worse.

How do we introduce Moneyball-style evaluation into the tenure and promotion review processes so that we can recognize and promote scholars whose talent is to amplify the efficacy of their colleagues? The traditional review process rewards the near-term success of individuals rather than recognizing other qualities that may promote the long-term success of the institution.

 

To wrap up, back to Kathleen Fitzpatrick:

We have long treated the tenure review, and to a lesser extent the review for promotion to full, as a threshold exercise: an assessment of whether the candidate has done enough to qualify.

(…)

Assistant professors run the pre-tenure period as a race and, making it over the final hurdle, too often collapse, finding themselves exhausted, without focus or direction, depressed to discover that what is ahead of them is only more of the same. The problem is not the height of the hurdles or the length of the track; it’s the notion that the pre-tenure period should be thought of as a race at all, something with a finish line at which one will either have won or lost, but will in any case be done. I believe that we can find a better means of supporting and assessing the careers of junior faculty if we start by approaching the tenure review in a different way entirely, thinking of it not as a threshold exercise but instead as a milestone, a moment of checking in with the progress of a much longer, more sustained and sustainable career.

Hear, hear.