Finding appropriate definitions and measures of research quality for transdisciplinary and applied natural resource management research

Research increasingly seeks not only to generate knowledge, but also to have impact. In this context, traditional academic definitions of research quality may be insufficient. This article presents a protocol for the systematic review of new and emerging definitions, criteria and indicators of research quality in applied, inter- and transdisciplinary contexts. It seeks to clarify arguments for or against expanding the definitions of research quality and to identify appropriate definitions and measures, with an emphasis on natural resource management research. The primary research question is: What are appropriate criteria and indicators for defining and measuring the quality of transdisciplinary research in natural resource management research? The proposed review will be based on literature sourced from a search of Scopus, Web of Knowledge and Google Scholar. The search terms and strings to be used were developed and tested iteratively, based on a benchmark set of references and incrementally refined searches designed to be comprehensive and to reduce irrelevant results. To select relevant articles, two reviewers will independently perform three rounds of screening by scanning (1) titles, (2) abstracts and (3) articles. The articles selected will then be reviewed for the following: arguments for or against expanding definitions of research quality; purposes for research quality evaluation; proposed principles of research quality; proposed criteria for research quality assessment; proposed indicators and measures of research quality; and proposed processes for evaluating transdisciplinary research. The results will be synthesized to provide an overview of the literature, to summarize the arguments and approaches for expanding definitions of research quality, and to identify and discuss the main purposes, principles, indicators and measures of research quality in transdisciplinary and applied contexts.

The relationship between science and society is changing.Boundaries between public and private research agencies, other knowledge institutions and civil society are no longer clear or distinct (Spaapen et al. 2007).Research aims to generate new knowledge and, increasingly, to promote and facilitate the use of that knowledge to enable change, solve problems and support innovation (Clark and Dickson 2003) -in other words, to have impact.Transdisciplinary approaches transcend disciplinary and institutional boundaries to contextualize research around the interests of stakeholders and to foster a more socially robust knowledge (Nowotny et al. 2003).New problem-driven approaches seek to create and apply knowledge in support of decision making for sustainable development.Such approaches are grounded in the belief that knowledge that is co-produced through close collaboration between scholars and practitioners is more useful and more used (Holling 1993;Clark and Dickson 2003;Berkes 2009).The research itself can be complex, spanning diverse goals and approaches and involving multiple actors from different backgrounds and multiple scales (Bergmann et al. 2005;Walter et al. 2007;Klein 2008).
Effective evaluation of quality in research is critical for the funding, management, ongoing development and advancement of research methods, projects and programs.Funders demand excellence from the research organizations they support; research managers guide their programs with the aim of achieving excellence, and they expect excellence from the programs, projects and individual scientists in their charge; scientists refer to their organizations' definitions and measures of research excellence in their own planning and research strategies.But how to define quality in research is a continuous challenge, particularly in a transdisciplinary context.Conventional research evaluation frameworks use criteria that are largely discipline specific (Tijssen 2003).Current approaches evaluate individual researchers, programs and research units primarily on the basis of publications, citations and peer assessment.Ongoing critiques of well-established evaluation criteria highlight the shortcomings and challenges of conventional approaches, which emphasize bibliometric and discipline-specific peerreview criteria, but often neglect to engage the innovative advances and process contributions of transdisciplinary research (Nowotny et al. 2003;Donovan 2008;Stiftel et al. 2008;De Jong et al. 2011).
The changing nature of research, with its greater focus on complex problems, more and larger partnerships across disciplines (interdisciplinary), and more engagement with a broader range of stakeholders (transdisciplinary), demands new approaches to research evaluation.While current criteria may remain relevant, additional criteria that address the innovative approaches and the diversity of actors, outcomes and long-term social impacts of transdisciplinary research are needed.As Tijssen (2003, 93) put it, "Clearly, in view of its strategic and policy relevance, developing and producing generally acceptable measures of 'research excellence' is one of the chief evaluation challenges of the years to come."Clear criteria are needed for research quality evaluation to foster excellence while supporting innovation.As Carew andWickson (2010, 1154)  There is a growing body of literature that addresses questions of evaluation and measurement of quality for research that explicitly aims to contribute to social change.Many countries have national research reviews, and many of those have begun to consider quality criteria beyond traditional disciplinary measures (Donovan 2008;KNAW 2009;REF 2011;ARC 2012;TEC 2012).There is also a body of literature that critiques those national reviews, including how they consider societal relevance and impact.Literature is emerging that reviews and synthesizes knowledge and best practice in research evaluation in a transdisciplinary context, and that proposes criteria and approaches for scientific quality evaluation (Defila and DiGiulio 1999;Bergmann et al. 2005;Wickson et al. 2006;Klein 2008;Carew and Wickson 2010;ERIC 2010).There is also a growing literature reporting empirical analyses of research projects or programs that includes attention to and innovations for accommodating broader definitions of research quality (Bergmann et al. 2005;De Jong et al. 2011;Spaapen and Drooge 2011).Promising concepts such as benchmarking, evaluating research in context, and assessing process and structural variables have been proposed by various authors (Defila and DiGiulio 1999;Bergmann et al. 2005;ERIC 2010;De Jong et al. 2011).This literature is growing quickly, and it is diverse, appearing in a wide range of journals, books and other media.
There is a need to collate and synthesize new approaches that can be used along with established science quality criteria, to develop a comprehensive set of research quality criteria and a framework appropriate for evaluating transdisciplinary research.In response to this need, we will conduct a study to review the literature that discusses the definitions of research quality and research excellence, and the principles, criteria and indicators for assessing the quality of applied, inter-and transdisciplinary research.The study will identify the quality definitions and criteria employed and synthesize these to produce a stateof-the-art guide to assessing research quality in a transdisciplinary context.The literature on research quality is rich.In order to adequately capture and synthesize the volume and diversity of this available literature, we will use a systematic review approach.In the present paper, we propose a review protocol for peer review and critique.

Objective of the review
The purpose of this study is to contribute to the more effective use and understanding of science, particularly in the realm of natural resource management.The objective is to use a systematic review of current knowledge of and approaches to the evaluation of research quality in an applied, inter-or transdisciplinary context to identify appropriate criteria and indicators for defining and measuring research quality.The primary research question is: What are appropriate criteria and indicators for defining and measuring transdisciplinary research quality in natural resource management research?

Search terms
A list of key terms developed from a preliminary literature review served as the starting point for developing the search terms that will be used when searching the databases.Initial search term strings were developed, tested and refined based on resulting search outputs through trial searches in Scopus, Web of Knowledge and Google Scholar.We then conducted search term scoping trials to identify and test candidate search strings that would yield comprehensive retrieval of relevant articles without excessive irrelevant results.A benchmark list of highly relevant papers was selected from our initial literature review to test the comprehensiveness of candidate search strings.Given the depth and breadth of literature that addresses research quality in some way, we had to use a time-intensive iterative process of search term refinement.As we encountered more relevant articles, we adapted our terms and search strings accordingly.Multiple search strings were modified and refined until benchmark papers were captured and additional searches became increasingly redundant.The resulting search terms are listed in Table 1, with the search strings in Table 2.Because of the large number of terms used to describe various aspects of the evaluation of research quality, our search strategy favors high sensitivity over specificity to ensure that we capture the relevant information.

Database searches
Three databases will be searched: Scopus, Web of Knowledge (WoK) 1 and Google Scholar.
1 The edition of WoK accessed: Science Citation Index Expanded (1989-present) Scopus and WoK cover a broad range of science and social science journals but they differ enough in their coverage to warrant searching both.In trial searches, Scopus typically returned from 30% to 50% more records than WoK and many of the additional records were considered to be relevant to this review.Although WoK returned fewer results, a substantial number of the titles it returned were not found in Scopus.Given its limited search options and ambiguous search algorithms, Google Scholar is not as well suited to a comprehensive systematic literature search.We therefore chose to use a unique set of more targeted search terms in this large database to capture relevant literature that do not surface in the search of Scopus and WoK.

Targeted journal searches
Based on expert recommendations and our own preliminary literature review, we identified a number of relevant journals that will be comprehensively searched using the search strings identified in Search String #4 in Table 2. Searching individual journals allows us to use broader, more inclusive search strings that would be unmanageable in the overall database search.
If other relevant journals become known to us throughout the review, they will be added and searched using the same search strings.

Specialist and supplementary searches
To ensure the inclusion of all relevant publications, we will scan the references in the articles that pass our third screening stage and follow those citations to include any additional relevant literature not found through our search.A limited selection of "gray" literature (i.e., published and unpublished documents that do not pass through the scientific peer-review system) will be identified from bibliographies of relevant publications (including literature reviews) and expert recommendations.
Search results from WoK and Scopus will be combined in Microsoft Excel, and duplicates will be removed.As Google Scholar results cannot be exported to Excel, these results will be imported into RefWorks and then exported to Excel, where they will be combined with the rest of the results with duplicates removed.Papers that pass the first two screenings (described in Section 2.6) will then be shared among the team in Mendeley.

Limitations
At this stage, the review will be limited to Englishlanguage sources.Possible collaboration with international partners to include more languages, such as Spanish and French, will be considered based on the results of the English-language review.

Inclusion criteria
The review will focus on literature that explicitly discusses the need for research evaluation criteria that go beyond traditional disciplinary criteria and include considerations of societal relevance and impact.We are seeking papers that review, critique, discuss and/or propose criteria, indicators or measures for the evaluation of quality in applied or transdisciplinary research, as relevant to natural resource management research.Through our preliminary literature review, we identified four types of documents that contain the following types of information: 1. Documents that describe, discuss or critique a research evaluation framework, at any scale.This will include discussions of national research evaluations, as well as university and other research organizations' evaluations that raise the need for nontraditional quality definitions, criteria, and/or that suggest alternative approaches.2. Literature reviews/syntheses that provide an overview of knowledge and best practice in research evaluation and make recommendations for criteria or processes in the evaluation of scientific quality in an applied or transdisciplinary context.3. Documents that critique existing quality criteria and make recommendations for alternative or additional criteria, especially in an applied or transdisciplinary context.4. Empirical analyses of research projects or programs that provide recommendations for research evaluation in an applied or transdisciplinary context.
Three rounds of screening for inclusion (1.title; 2. abstract; 3. full article) will be carried out in parallel by two reviewers using the inclusion/ exclusion criteria in Tables 3 and 4. Each reviewer will screen all articles included in each round and the results will be compared for agreement.Any disagreement in the final inclusion/exclusion of papers will be resolved by discussions between reviewers to reach consensus.In the case of disputes, a third reviewer will be consulted.
Four rounds of trial screenings were undertaken by four members of the research group to refine inclusion and exclusion criteria and to assess inter-reviewer agreement.In each trial, 100 titles were randomly selected from the output of search strings.Team members individually reviewed the titles and abstracts and marked them for inclusion or exclusion.After the third trial, eight papers were chosen for a full review to connect information in the abstracts with the content of the full article and do trial runs of extracting data from articles.The team met to discuss the results of every trial and, for all disagreements, hear from each party regarding their decision-making process and criteria.Through this process, the nuances in the inclusion criteria were revealed and inclusion criteria further clarified.Each individual's interpretation of the literature and decisionmaking process were illuminated and the team's individual processes had coalesced into a true team process by the fourth and ultimate trial.Through this process we determined that consensus was the best way to proceed with this review.In round one, titles that are obviously irrelevant will be excluded (see inclusion criteria in Table 3).A conservative strategy will be used, erring on the side of inclusion in cases of uncertainty.In the second round of screening, abstracts for all articles included in the first round will be read and assessed based on the inclusion criteria listed in Table 4. Again, in cases of uncertainty, papers will be included in the next round.In the final screening, reviewers will scan whole articles and apply the inclusion/exclusion criteria in Table 4.
At each stage of screening, records will be kept of all papers screened, included and excluded.

Critical appraisal
In systematic reviews, studies are appraised to ensure that they are adequate for answering the research question and to assess the methods, results and conclusions of each individual study for susceptibility to bias that could influence the outcome of the review (Petticrew and Roberts 2006).The majority of papers expected to be included in this review will be theoretical and review papers, and therefore do not have explicit methods that can be appraised using established quality assessment frameworks.
Rather than focus on methodological validity, our quality assessment, adapted for the types of papers we expect to be appraising, considers relevance (relevance to our research question), transparency (clarity and logic of how knowledge was generated), significance of the contribution (are new ideas offered?) and generalizability (is the context specified?Do the ideas apply in other contexts?).These criteria are summarized in the critical appraisal framework in Table 5, adapted from Spencer et al. (2003).Study quality based on the guidelines outlined in Table 5 will be assessed separately by both reviewers, with their assessments compared for agreement.Disagreements will be discussed until consensus is reached on the inclusion or exclusion of papers.Outstanding disagreements will be resolved by involving a third reviewer.

Data extraction
The aim of the review is to collate, analyze and synthesize rationales for broader definitions and criteria of research quality in a transdisciplinary or applied research context.The relevant literature is expected to cover a broad range of issues, from multiple perspectives.The review seeks information on the following topics: • arguments for or against expanding definitions of research quality; • purposes for research quality evaluation; • proposed principles of research quality; • proposed criteria for research quality assessment; • proposed indicators and measures of research quality; • proposed processes for evaluating transdisciplinary research.
Information and ideas will be extracted using the data extraction guide in Table 6.We will collect and collate information and ideas about the context, conceptual frameworks, definitions, measures and critiques of research quality.This will be supported by the use of a data extraction guide, which was developed using a generic template from Cooper (2010) and modified for the requirements of this review.To develop the data extraction guide, reviewers worked through a variety of relevant papers, including all papers in our benchmark list, to identify the types of information to be extracted and to ensure the guide was clear and relevant to our purpose.Any changes to the data extraction guide Brian Belcher, Katherine Rasmussen, Matthew Kemshaw and Deborah Zornes Background 1 2 |

Table 3 . Inclusion criteria for title screen
Finding appropriate definitions and measures of research quality for transdisciplinary and applied natural resource management research | 7

Table 4 . Inclusion criteria for abstract and full article screens
• Discusses research outcomes (diffusion, uptake, use, impact) as an indicator or consequence of research quality