Few humanities disciplines have been untouched by the growing availability of digital resources and computational methods. The variety and form of new types of scholarly methods and resources has made it more difficult than it was previously for some humanities departments to assess research submitted for hiring, annual evaluation, tenure, and promotion. This has meant that some evaluation committees simply do not consider digital work as research, or feel incapable of evaluating it. As a result, they sidestep or outsource evaluation to external reviewers, which has then led to the perception that researchers are discouraged from pursuing digital projects even when they are hired to work in areas like new media studies or humanities computing. For these reasons the following guidelines on the evaluation of digital work have been adopted by scholarly digital humanities associations like the CSDH/SCHN and ACH to provide recommendations on evaluation practices.
The Value of Digital Scholarship
Programs of research, which are by nature exploratory, may require faculty members to take up modes of research that depart from methods they have previously used, therefore the form the resulting scholarship takes should not prejudice its evaluation. Original works in new media forms, whether digital or other, should be evaluated as scholarship following best practices if so presented. Likewise, researchers should be encouraged to experiment with new forms when disseminating knowledge, confident that their experiments will be fairly evaluated.
Examples of digital work include interactive visualizations, original research tools, well documented and curated datasets, multimedia art, web installations, hybrid teaching web sites, serious games and scholarly electronic editions.
Guidelines for Evaluation
Recognizing that work published in new media can be difficult to assess, we recommend the following guidelines:
- Chairs and faculty should communicate regularly about their work and, where new forms of evaluation are called for, they should discuss possible forms of assessment and documentation. Where a faculty member expects that a major part of their tenure or promotion dossier will be digital work, they should discuss ways of documenting the scholarship early in the process. Ideally a chair and faculty member could agree at the beginning of a project about how it should be documented and assessed for the purposes of tenure and promotion.
- Chairs and faculty should get advice on ways of documenting the impact of digital work they are not familiar with. For example, instructional technology projects can be documented with the help of local teaching and learning centres. The CSDH/SCHN (csdh-schn.org/) and ACH (org) can provide such advice or connect faculty and chairs to others who can.
- Chairs should ask faculty to deposit work in the institutional archive as part of (and as a signal for) evaluation. Faculty should plan to deposit their work. Deposit can act as the trigger and the documentation for evaluation. Deposit practices would have the added advantage of ensuring that new media work is archived.
- New media work should be reviewed in the form it was intended to be read/used for the purposes of tenure and promotion. Documentation of the reception and use of the work is also valuable, but is not a substitute for the review of a work in its original form by colleagues.
- Digital work is often collaborative and interdisciplinary because of the skills needed. Collaboration should not be evaluated as simply the sum of regular contributions, nor should credit be calculated by dividing by the number of contributors. The work that goes into building and leading interdisciplinary teams should be recognized as it can foster innovation. In addition, the management of large collaborations should be recognized as scholarly work when documented. The director of a project, even if they don’t write a line of code or text, contributes in the management of the project from the articulation of its goals to the design of its outcomes. Just as we recognize the scholarly impact of a director, editor, dramaturge, or designer, so should we recognize new media leadership, and recognize that contributions to digital scholarship are not limited to content creation.
- Faculty who create innovative works that experiment with form can be expected to also present on their innovations whether at conferences or in associated writings. It is not enough to do it, creators also need to share their work in appropriate ways, be open to critique, and they can be expected to explain their innovation to others.
- Pedagogical innovations are expected to be formally and appropriately assessed if they are to be treated as scholarship of pedagogy. It is not enough to have done something, you need to ask how it has contributed new knowledge.
- Failure can be a form of scholarship when documented so that it informs others. Just as negative scientific results are considered scholarship when properly documented, so failed digital media experiments should be recognized as contributions. Otherwise we discourage honest appraisal and learning.
Some forms of documentation that both chairs and faculty can use to assist in the open and fair assessment of the quality of digital work include:
- New forms of peer review. There are new forms of peer-review and juried exhibit emerging to recognize quality digital work. These will vary by discipline and the form the digital work takes, but faculty should inform themselves of the venues for review and avail themselves of those appropriate.
- Grants. Competitive grants can be thought of as a rigorous review of the ideas behind a proposed digital work or as a form of competitive commission. Either way, they are a sign that a scholar is submitting her ideas for competitive review.
- Public Presentations. As mentioned above, faculty who create new media works should also present to their departments, at conferences and at other venues about that work. Peer-review conference papers on a project can be considered as a form of peer-review of the project itself. Departmental talks increase the understanding in the unit desired for evaluation. Invited talks, especially international talks, increase the impact of digital work.
- Expert Review. People familiar with new media can be found, both at the University of Alberta, and elsewhere. Faculty and chairs should be encouraged to commission short expert reviews of new works where other forms of peer-review are difficult. Just as we commission colleagues to observe innovative teaching, we can also commission colleagues to reflect on a new work. For that matter we can also ask experts to walk us through how they interpret a new media work. Digital scholarly associations can help find experts and suggest guidelines for review.
- Statistics. For some types of work that are online it is possible to gather data about who and how many people are accessing the work. This data can be helpful when contextualized.
- Narrative. Faculty should consider developing short narrative explanations of their working practices that contextualize the final work in a research trajectory that colleagues can understand. Such narratives, when well written and illustrated, can help chairs and others understand what is significant and how much work went into a seemingly simple digital work. Explain what worked and what didn’t and how you thought through the project.
Digital media have not only expanded the forms that research can take, but research practices are also changing in the face of digital distribution and open access publishing. In particular we are being called on to preserve research data and to share new knowledge openly. Universities that have the infrastructure should encourage faculty to deposit not only digital works, but also curated datasets and preprint versions of papers/monographs with documentation in an open access form. These can be deposited with an embargo in digital archives as part of good practice around research dissemination and preservation. The deposit of work, including online published work, even if it is available elsewhere, ensures the long-term preservation by ensuring that there are copies in more than one place. Further, libraries can then ensure that the work is not only preserved, but is discoverable in the long term as publications come and go.
Here are some resources to help with assessment:
- MLA Guidelines for Evaluating Work in Digital Humanities and Digital Media: http://www.mla.org/guidelines_evaluation_digital
- MLA issue of Profession with articles on Evaluating Digital Scholarship: http://www.mlajournals.org/toc/prof/2011/1
- AHA Guidelines for the the Evaluation of Digital Scholarship: http://www.historians.org/teaching-and-learning/digital-history-resources/evaluation-of-digital-scholarship-in-history
- Journal of Digital Humanities “Short Guide to Evaluation of Digital Work”: http://journalofdigitalhumanities.org/1-4/short-guide-to-evaluation-of-digital-work-by-geoffrey-rockwell/
- See also the other articles in the issue of the JDH on Closing the Evaluation Gap: http://journalofdigitalhumanities.org/1-4/
 These are adapted from the MLA Guidelines for Evaluating Work in Digital Humanities and Digital Media. See <http://www.mla.org/guidelines_evaluation_digital>