What do criteria and indicators assess ? An analysis of five C & I sets relevant for forest management in the Brazilian Amazon

The primary challenge when attempting to capture the full potential of tropical forests in rural development is to increase human well-being while maintaining the ecological functions of these forests. Sustainable forest management has been identified as one of the most promising strategies to achieve this end. Since UNCED, in 1992, efforts to develop instruments for supporting sustainable forest management have been intensified (UNCED 1992), and researchers have focused on criteria and indicators (C&I) as the main tool to assess sustainability (Grayson and Maynard 1997). C&I are designed to deliver transparent and cost-effective information sets that are required to establish sustainability concepts and for evaluating and implementing sustainable forest management at global, regional and local levels. Prabhu et al. (1998) have suggested that C&I represent a form of communication network. Fundamental to the concept of C&I is the rule that no single criterion or indicator constitutes, by itself, a complete measure of sustainability. An individual criterion or indicator needs to be considered within the context of other C&I of the system. It is useful to think of arrays of C&I as information nodes on areas of concern, which together provide a full picture of the state of forests and current sustainability trends at the hierarchical level of principles What do criteria and indicators assess? An analysis of five C&I sets relevant for forest management in the Brazilian Amazon

and criteria (Figure 1).Thereby it is possible to disaggregate the complex concept of sustainability into assessable and communicable elements.Following the definition of the Centre for International Forestry Research (CIFOR 1999a) principles and criteria provide the framework for managing forests in a sustainable fashion, without themselves being a direct measure of performance.Criteria are the intermediate points to which the information provided by indicators can be integrated and where an interpretable assessment crystallises.Indicators and verifiers define what information is delivered to evaluate the criteria.While indicators are seen as variable components used to infer the status of a particular criterion, verifiers contain data or information that enhances the specificity or the ease of assessment of an indicator.Since UNCED, more than 100 countries in six continents have participated in the development of C&I (Wijewardana et al. 1997).Numerous international, national and local initiatives have defined C&I sets, particularly as support for public policies at the level of Forest Management Units (FMU), and as a requirement for certification, monitoring and auditing of management activities (CIFOR 1999 a , Kanowski et al. 2000, FAO 2001).
Although all C&I sets aim at evaluating sustainability, they are diverse in content and structure, because of the specific underlying conditions that influence their development.In order to systematise the reasons for this diversity, three groups of factors were established (Figure 2): (1) differences of external frame-conditions, including all factors that indirectly influence the individuals involved in the process of C&I definition; (2) characteristics of the C&I development mechanism, covering specific work conditions and attributes of individuals directly involved; and (3) demands raised by specific application of C&I in relation to financial and human resources, as well as time constraints (Pokorny et al. 2002).
As a consequence, there is a wide variety of C&I sets showing specific items and structure.This diversity is confusing to potential clients of C&I, which reduces their acceptance and hinders their utility as support for implementation of sustainable forest management.In order to halt this erosion of confidence in C&I it is essential to provide knowledge and awareness about the existing specificities, similarities and differences between one C&I set and another.Against this background the diversity of five C&I sets relevant for the Brazilian Amazon were systematically analysed.This paper presents key findings about the content, information used for the assessment, specificity and practicability of the analysed C&I sets.Using these results, the paper provides recommendations for those who are interested in developing and/or applying C&I.

THE ANALYSED C&I SETS
In order to ensure geographical compatibility, the analysis of C&I sets related to forest management concentrated on one specific region, the Brazilian Amazon.Due to its unique biodiversity and extent, this region represents one of the world's most important remaining tropical forest resources (Lele et al. 2000).Although various national and local initiatives have already started to develop regional adopted C&I, e.g. the Brazilian Institute for the Environment and the Natural Renewable Resources (IBAMA 1997), the Institute for Environmental Studies in the Amazon (IPAM 2001), the Brazilian Association for Technical Norms (ABNT) and Acre's State Secretariat for Forests and Extractivism (SEFE), to date the number of C&I sets applied in monitoring or auditing tools is fairly restricted.Thus, only the following five C&I sets were selected and analysed in depth.Table 1 shows key characteristics of the selected C&I sets.The selected C&I sets show heterogeneous objectives, key actors, degree of detail and structures.The sets of CIFOR, FSC and ITTO have international relevance.The set of Tarapoto is still in development, but will strongly influence all countries of the Amazon region.The C&I set for Adaptive Collaborative Management is not yet related to any practical applications.

DIVERSITY OF CONTENT
The first step was to compare the content of the five C&I sets.A method for a systematical comparative analysis based on frequencies was designed to clarify what was being assessed.The C&I of all sets was scanned and the contents listed, which were then aggregated into three thematic levels: a general level composed of six categories informing about the general thematic focus of the C&I sets, an intermediary level of generalisation subdividing the six thematic categories into 27 themes, and finally, various topics presenting the most detailed level of analysis (Table 2).This provided an adequate level of generality, which enabled a general comparison in addition to a sufficient degree of detail for a more in-depth analysis.Each item of the five C&I sets was classified within these three levels with the possibility to relate each item to several categories.A calculation was made of how many items were related to the different themes in order to describe and compare the content of the C&I sets.
The results of this frequency analysis shown in Figure 3 provide two interesting features.First, all C&I sets address the complex assessment of sustainability by considering a wide variety of aspects reflecting an attempt for integrity, and secondly, there are strong differences in the manner and intensity of evaluating these contents.The C&I sets use a very different number of indicators to evaluate the different themes.Using the number of items as an indicator of intensity the following observations were made in relation to the six thematic categories: • generally, the evaluation of frame conditions outside of the FMU plays a minor role.Only the CIFOR sets, specifically the ACM one, considered to some extent aspects of legislation and public policy, economic and sociocultural frame conditions, and infrastructure • C&I regarding relationships between the FMU and external parameters such as 'respect to legislation' or 'communication of actors outside the FMU', and 'education' were found in all C&I sets.The FSC set showed the highest quantity of items related to this category, specifically those regarding acceptance of external defined regulations • internal mechanisms assessing relationships between actors directly related to the FMU were generally given little attention.The Tarapoto set did not evaluate this category at all and only the sets of ACM and FSC paid some attention to it • with the exception of the ACM set, all C&I sets focus on forest operations.A large number of C&I referred to checking the existence and quality of documents, specifically maps, information about the FMU, plans and strategy papers mainly on operational activities.The assessment of pre-exploitation operations is much more intensive than the assessment of exploitation and postexploitation operations.The FSC set spent significantly more C&I in evaluating technical forest management aspects than all the others • in general, the socioeconomic impacts of forest management were not considered very intensively, but there were strong differences between the sets.The highest frequencies were detected in the assessment of human well-being, mainly on issues of health, security and culture.Quality of information systems, level of conflicts, as well as financial results were assessed by all sets • With regard to evaluation of environmental impacts, the CIFOR set alone showed more C&I than all the other sets put together.Its emphasis was laid on the assessment of quality of flora, ecosystems and landscape.Only few sets considered genetic aspects, quality of fauna and assessment of soil and water.
Most of the C&I sets show a clear thematic focus.The FSC set concentrates on the evaluation of forest operations.The ACM set focuses on socioeconomic results of forest management, mainly about human well being of actors inside and outside the FMU.The CIFOR set attends very strongly the environmental results of forest management.The sets of ITTO and Tarapoto stay at a more general level and did not evaluate all the themes.The diversity of content becomes even clearer, if the C&I sets are compared in relation to the topics at the lowest level of thematic categorisation.Figure 4 shows the proportion of coincidence between the C&I sets for the 148 defined topics.The figure indicates that the five C&I sets consider different topics for their assessment.Comparing the FSC set with the Tarapoto set, only 27% of topics evaluated were common to both sets.In contrast, the sets of FSC, CIFOR and ITTO showed a significantly higher degree of similarity with up to 80% of the topics being the same.Only 12 (8%) of all topics were evaluated by all sets.Of course, the C&I sets vary not only with regard to the assessment intensity but also present quite different topics to evaluate the sustainability of forest management.

INFORMATION USED FOR THE ASSESSMENT
Another useful aspect to describe differences between C&I sets is the type of information used to assess a specific content.The data and information used for the assessment of the criteria is defined through I&V.Eight categories of information were defined in order to systematise the I&V in relation to the information they were targeted at (Table 3), and applied for all I&V in the five C&I sets.Their frequencies as given in Figure 5.
The proportion of I&V belonging to different categories of information differed substantially between the C&I sets.The assessment related to the FSC set was based mainly on information about the existence of documents and the application of guidelines.The latter aspect was also very important for the ITTO set.The Tarapoto set looked additionally for information about socioeconomic impacts.For the ACM set information about the conditions inside the FMU played a key role.The CIFOR set showed a strong emphasis for the assessment of ecological impacts.In

Category of information Description
Technical guidelines Recommends something that should be done in different areas such as forest operation, social, communication etc.

Application of guidelines
Evaluates if a specific guideline, norm, regulation or agreement defined in the C&I set or elsewhere has achieved its target.

External conditions
Evaluates aspects outside the responsibility of the FMU.

Internal conditions
Evaluates aspects under the responsibility of the FMU.

Environmental impacts
Evaluates the environmental impacts of FMU´s activities.

Sociocultural impacts
Evaluates the sociocultural impacts of FMU´s activities.

Socioeconomic impacts
Evaluates the socioeconomic impacts of FMU´s activities.
Existence of documents Checks on the existence of documents, equipment, programs, reports etc.

FIGURE 6 Proportion of I&V belonging to different characters of information, used to assess criteria related to environmental impact
general the proportion of I&V related to information about socioeconomic impacts was small.I&V looking for information about sociocultural impacts were nearly absent.
The information used to assess criteria related to the thematic category of environmental impacts was analysed in more detail due to its fundamental importance in evaluating sustainability.Figure 6 indicates that only the two CIFOR sets focused on information concerning environmental impacts for assessing the criteria of this thematic category.In the other C&I sets, I&V of other information categories dominate.The sets of FSC and ITTO focus strongly on information concerning the technical guidelines, mainly for 'Reduced Impact Logging'.In addition, they search for information about the existence of documents.It is remarkable that the Tarapoto set relies on information about socioeconomic impacts to assess ecological impacts.
The analysis shows a high proportion of I&V seeking information about the existence of documents as well as technical guidelines and their application.Information about impacts, although highly relevant for the evaluation of sustainability, and therefore frequently mentioned at the criteria level, are rare at the I&V level.Even information about environmental impacts was not used to a notable degree by all sets under analysis, and I&V related to information on socioeconomic and sociocultural impacts are seldom mentioned.There is a discrepancy between a common sense interpretation and the content of criteria and the information gathered at the I&V level.In other words, what is assessed is frequently not what was intended.

PRACTICABILITY AND SPECIFICITY
The last step consisted of an analysis of practicability and specificity of the items shown by the five C&I sets.I&V were assessed in terms of clarity, practicability, transparency and objectivity of the assessment result (Table 4).Again, the frequencies for the different classes were calculated.
The analysis revealed acute deficiencies for all C&I sets.As shown in Figure 7 only half of the analysed I&V were categorised as simple to be assessed.The assessment of the remaining half seemed too difficult or time consuming.A similar outcome resulted for specificity.Only half of the I&V were categorised sufficiently specific to enable a clear and objective assessment.Over 40% of the items showed deficiencies in relation to clarity, quantification and/or qualification.Only one third of all I&V were categorised as specific and at the same time simple to assess.More than half of the I&V under analysis showed deficiencies in definition causing difficulties in assessment.Around 14% of the I&V did not seem to be of practical use at all, either because the wording was confusing and/or their assessment in practice was not feasible.
To better understand what contents were affected by these substantial deficiencies in specificity and practicability, the quality of I&V were analysed in relation to the different categories of information (Table 3) and to the areas of interest (policy, ecology, social and production of goods and services) as defined by CIFOR (1999b).
As a result of this detailed analysis Figure 8 shows that large parts of the I&V in all areas of interest present deficiencies in quality, in particular for the assessment of ecological aspects.Poor quality was also detected in I&V of all categories of information.Specifically for the assessment of impacts only very few I&V of good quality were found, whereas the I&V related to the information categories 'Existence of documents', 'Assessment of technical guidelines' and 'Evaluation of frame conditions inside the FMU' have a better quality.
Comparison between the five C&I sets showed a significantly higher proportion of I&V of good quality in CIFOR, ACM and FSC sets than in ITTO and Tarapoto sets.The FSC set in particular, mainly because of the high proportion of I&V related to assessment of technical aspects, showed a better result.The main methodological element of this study was the analysis of frequencies based on a categorisation of descriptive parameters of content and quality of the C&I of five sets relevant for forest management in the Amazon region.In the interpretation the general limitations of using the results of quantitative analysis for qualitative interpretation has to be considered.In addition, the systematic categorisation of the C&I is problematic.In an attempt to define clear and unambiguous categories for the studied parameters, it was necessary to find a compromise between specificity and comparability.Although the definitions for some parameters were still not sharp enough, the continuous adaptation to the requirements detected by the team during the categorisation process resulted in an acceptable outcome, which is shown by the fact that the categorisation by different persons was very similar.The deficiencies of the analysed C&I sets made the categorisation difficult, mainly because of three reasons: (1) lack of specificity of items, which often diminished the clarity of the content; (2) confusion within C&I sets as to the hierarchic concept; and finally (3) the fact that more than half of all I&V were related to more than one thematic category.The last aspect resulted in pseudo quality, because the generally negative characteristic of the mix increased the frequency of I&V appearing in the different categories.
In spite of these methodological difficulties, the study revealed interesting points about the analysed C&I sets.Even though all C&I sets address the social, technical, ecological and economic dimensions of sustainability, they show strongly different thematic foci.This diversity in content is not always understandable, even if the underlying objectives for the sets are taken into account.It did not seem possible to reconstruct the specific underlying conditions that influenced the development of the set.In interpreting sustainability as the sum of the assessed C&I, the understanding of sustainability varied strongly between the sets.But these differences seem to be coincidental, because all approaches attempted to consider all dimensions of sustainability.Most astonishing is the fact that all C&I sets considered social, economic and environmental impacts at the principle and criteria level, but in only a few cases were these impacts really measured.There was a general lack of validity.Additional strong deficiencies in relation to specificity and practicability of I&V were detected.Many I&V did not appear to be amenable to objective assessment.This was true mainly for I&V when assessing the ecological impacts.Thus the C&I sets had rather the character of checklists, which did not offer means for assessment of gathered information.
As a consequence of these quality deficiencies, the results of C&I driven assessment processes depend to a high degree on the subjectivity of the evaluator and his/ her underlying experiences, values and interests.Thus, although there are differences in content, it seems more likely that one evaluator would come to the same result using different C&I sets, rather than two independent evaluators using the same C&I set would produce the same or similar results.This lack of reliability reduces the transparency and objectivity of C&I sets and consequently the confidence in and the value of the assessment result.
Considering the function of C&I as a tool for facilitating communication among stakeholder and assessment of sustainable forest management, these results are very important.There is a general need for improvement to develop the full operational potential of C&I.To increase objectivity and transparency, C&I have to more clearly and unambiguously reflect what is really assessed.To achieve this more attention needs to be focused on production of an easy, understandable and more specific wording of the items.In addition it has to be confirmed that the principles and criteria really express what the related items at lower hierarchical levels assess.For example, the situation has to be avoided where a criterion gives the impression that environmental impacts are assessed, although the related indicators only search information about the existence of certain documents or the application of guidelines for 'Reduced Impact Logging'.Finally, all items not assessable in a given context, such as genetic diversity or in some cases biodiversity of the fauna, have to be eliminated.The fact that most of the analysed C&I sets considered the different

Number of indicators and verifiers
What do criteria and indicators assess?
topics with very different frequencies indicates another opportunity to limit the number of items.The possibility of eliminating indicators has to be evaluated, and the identification of key indicators is recommended.Finally care has to be taken to ensure a clear hierarchical structure of the C&I sets by checking for repetitions and unclear relations.It is possible to significantly improve the quality of C&I sets by discussing in detail verifiers and assessment methods.
Awareness about what exactly is measured in the field would help to avoid unnecessary complexity and support a clearer definition of the contents at all hierarchical levels.
The analysis shows that the assessment of sustainability is a complex task.A great deal of information is needed on a large number of aspects.Independent from the detected deficiencies, results in C&I sets tend to be complex, which diminishes their practicability and consequently their acceptance.The varying topics and assessment intensities of the studied C&I sets indicate the underlying need for less complexity.The exclusion of topics due to lack of interest, relevance or practicability question the integrative approach of C&I for assessing sustainability.In fact, most C&I sets, although putting sustainability as an assessment goal, focus on the assessment of specific aspects such as the application of technical guidelines for 'Reduced Impact Logging', ecological impacts, legality etc.To avoid misunderstandings and to facilitate the development of less complex, more practicable C&I sets, it is recommended that the objective of the assessment is identified more clearly, and greater consideration is given to the potential clients and frame-conditions for its application.

FIGURE 2
FIGURE 2The key factors for diversity of C&I sets

FIGURE 3 FIGURE 4
FIGURE 3 Frequencies of C&I of the five selected sets in relation to the six thematic categories

FIGURE 7
FIGURE 7 Proportion of I&V of the five analysed C&I sets in relation to practicability and specificity

FIGURE 8
FIGURE 8 Quality of indicators and verifiers in relation to the different areas of interest and information categories

Principle Criterion Indicator Verifier Hierarchical Level Hierarchical Organization Principle Criterion Indicator Verifier Hierarchical Level Hierarchical Organization
FIGURE 1 Hierarchical organisation of C&I

TABLE 1
Characterisation of the five C&I sets analysed in the study * Number of principle (P), criteria (C); indicator (I), verifier (V) and sum (?) forum of representatives of member countries with the contribution of environmental NGOs in addition to timber trade associations.The revised version, analysed in this study, was published in 1998.The objective of the C&I set is to provide information to support decision makers to elaborate adequate political strategies for the implementation of sustainable forest management and to facilitate the communication between research and practice.•TheTarapotoprocess, initiated in 1995 by the Amazonian countries of Brazil, Bolivia, Colombia, Ecuador, Guyana, Peru, Suriname and Venezuela, and unified in the Tratado de Cooperación Amazónica (TCA), aims at the development and implementation of C&I based national and regional monitoring systems(TCA 1995).As a starting point for national consultations in each of the TCA countries, a C&I set has been defined in the city of Tarapoto in Peru in 1995.
was founded as a result of the International Conference of the United Nations about Trade and Development (UNCTAD) in order to promote the sustainable management of tropical forest by supporting the fair trade of tropical timber.The members of ITTO represent nearly 95% of the tropical timber market.The C&I set presented by ITTO in 1992 marked the starting point for many C&I initiatives.This C&I set, related to the sustainable production of timber, was negotiated by an internal The Tarapoto process is based on the evaluation of these C&I in accordance to their practicability.Brazil carried out the national consultancy in the year 2000.The study used this preliminary C&I set, because the elaboration of the final version under consideration of the national evaluations is still in progress(TCA 2001).

TABLE 2
Thematic categories used to systematise the content of items from the five C&I sets

TABLE 4
Classes used for the systematisation of indicators and verifiers in relation to specificity and practicability DeficientOnly subjective assessment possible, because an unambiguous qualitative characterisation is missing (e.g.. adequate conditions; wearing of individual protection equipment; existence of plans; etc.) and/or no values, locals or limits defined, (e.g.: sufficient, adequate; significant; etc.) and/or unclear editing of text.ConfusingInterpretation not possible, because of confusing redaction.