Considerations for ecosystem receptors

This stage stipulates the type of change to look for in the indicators. It’s important to know what management actions will be taken if an effect is detected at Step 5 of the framework (refer to Chapter 10 of Downes et al. 2002). The management goals will provide much of this context.

For example, if a slightly to moderately disturbed ecosystem is being managed to conserve the population of a recreationally important fish, and the threat is a persistent contaminant that could reduce fish fecundity, then the decision criteria for the indicators need to be set at values that are smaller than those that begin to affect the reproduction of the fish. This will allow sufficient lead time for managers to act before the population of fish is affected.

This process requires a significant level of scientific judgement. The actual values used as decision criteria will depend on a number of modifying factors, such as chemical speciation of toxicants, or knowledge of the sensitivity of the fish species from laboratory testing or field studies.

Defining an acceptable level of change

When setting levels of protection for high conservation or ecological value ecosystems or slightly to moderately disturbed ecosystems, we recommend a criterion of ‘no change beyond natural variability’ for ecosystem receptors.

Operationally, you will still need to:

• stipulate how much change can be expected under ‘natural’ conditions (natural variation constitutes an acceptable level of change in the ecosystem)
• decide on an effect size
• ensure that sampling is intensive enough (to detect effects larger than the acceptable natural changes in the chosen indicators, and avoid Type II errors).

The determination of the acceptable level of change may have both scientific and social or cultural elements.

See also:

• Field sampling program
• Monitoring program objectives
• Pilot studies

If you are new to environmental assessment, defining an acceptable level of change may seem weak, especially when some stakeholders insist there must be no change in the indicator.

You cannot set a criterion of ‘no change’ because change is an essential component of all functioning ecosystems. Proof of no change would require you to prove a null hypothesis, which is operationally impossible.

It is possible to state some level of change in an indicator below which it is not important to reject the null hypothesis of no change. This requires stakeholders to be explicit about what level of change in the indicator is regarded as harmless or acceptable. In formal terms, this process involves specifying an effect size in Stage 2.

This stage translates the identified changes into quantifiable effect sizes.

Indicator values vary naturally in space and time, and estimates of their true values can only be made via samples. Some observed changes in the indicators are likely to be ecologically trivial, or important components of ecosystem function.

If the water quality monitoring objective is to detect non-trivial changes or trends in the indicators soon enough to allow management to act, then the monitoring program needs to be sensitive enough to detect modest rather than large changes in those indicators.

Selecting widely used default monitoring protocols — out of habit or convenience — is not sufficient. The selected monitoring programs must have the sensitivity required to assess achievement or otherwise of the management goal.

You need to identify the maximum amount of change in the indicator that is tolerable before you can conclude that there has been important or unacceptable change that requires a management response. This level of ecologically important change is sometimes called the critical effect size, or ‘effect size’ for short.

Effect size comprises its form and magnitude (Toft & Shea 1983, Mapstone 1995).

The form of an effect is the statistical measure (e.g. mean, variance) that is expected to differ between reference and disturbed conditions, and the pattern of differences or trends that needs to be detected (Stewart-Oaten et al. 1986, Green 1989, Underwood 1991, 1992).

The magnitude of an effect is the size of those differences or trends that would be considered ecologically important to detect.

Setting an effect size for ecosystem receptors can be difficult because there is often little information about the relationships between contaminants and biological indicators in field conditions, especially in Australia and New Zealand.

We make some suggestions about how to proceed although this is not an exhaustive list. Other strategies may arise as you gain more experience in planning monitoring programs.

If an indicator has intrinsic socioeconomic value (e.g. a commercially or recreationally important species), then effect sizes can be set to ensure sustainable use of that indicator. However, many biological indicators have been selected because they are:

• more sensitive than commercial species, or
• thought to be ecologically important rather than of economic value.

For example, seagrass is not used directly by humans in Australia and New Zealand but is an important indicator because of the habitat it provides for many other species.

Existing research on similar stressors, preferably in comparable regions, can provide information about the relationship between the indicator and size of the potential effect, especially if existing effects can be found on a gradient from mild to extreme.

For example, a variety of wastewater treatment plants may be present in a river basin with differing degrees of sewage treatment. Pilot data relating indicator levels and type of treatment could be used in stakeholder consultations to correlate stakeholders’ expectations of acceptable sewage treatment with change in the indicator. In some cases, statistical or simulation models can use these data to estimate how much an indicator might change under different scenarios (Paul et al. 2016).

For many indicators in Australian and New Zealand ecosystems, however, such background data are unlikely to be available. This will inevitably involve some judgement by people planning the monitoring program, and an arbitrary but conservative effect size will need to be specified (e.g. Humphrey et al. 1995). This should be done explicitly, at the beginning of the program (as a feedback between Steps 2 and 3 of the Water Quality Monitoring Framework).

Sometimes the desired effect size requires impractically large sample sizes, and the trade-offs between error rates needs to be discussed iteratively with stakeholders (refer to Stage 3). Any change to the effect size later in the program must be openly and explicitly negotiated and fully justified on scientific grounds (Step 9 of the Water Quality Monitoring Framework).

Recognising that some procedures have implicit effect sizes

For some of the procedures in the Water Quality Guidelines, the effect size and error rates are implicit in the methods, making them less amenable to the procedure of using scalable decision criteria that we describe here.

For example, in the AUSRIVAS procedure for rivers, notions of effect size and error rates are inherent in the way the summary indexes are compared with the reference conditions.

This stage assesses the risk of making a Type I error (false alarm) and a Type II error (false sense of security) in the light of the consequences or costs of making either of those errors.

Having stipulated an effect size, the stakeholders then need to minimise the risk of 2 potential statistical outcomes — the Type I and Type II errors. The challenge is to ensure that sufficient data are collected to detect the change stipulated in the effect size while, on the other hand, not expending too many resources on sample sizes that will detect ecologically trivial changes in the indicator. Inevitably, resources are scarce so all monitoring programs will need to balance these 2 errors.

The first potential error is to declare that an effect has occurred (effect size has been exceeded) when there has actually been no ecologically important change.

Conventionally, the Type I error rate, α, has been set at 0.05 or smaller (Toft & Shea 1983, Fairweather 1991, Mapstone 1995). Some have argued for conservative or default values for α and the Type II error rate, β, but ideally these rates should be negotiated rather than accepted uncritically (Greenland et al. 2016, Wasserstein & Lazar 2016). The most important part of this negotiation is to ensure that the balance between these 2 types of errors is acceptable to your involved stakeholders, which could occur at Steps 2, 4 or 5 of the Water Quality Management Framework.

To this end, the Water Quality Guidelines recommend Mapstone’s (1995) proposal that the ratioof Type I and Type II errors is negotiated when refining the design of the monitoring program. This process requires iteration between stakeholders, but should be transparent, accountable and, above all, take place before the final monitoring or assessment program is implemented (Mapstone 1995).

Choosing error rates α and β involves 4 steps:

• Establish the relative importance or cost of the consequences of each type of error.
• Set the ratio of the critical Type I and Type II errors relative to their costs. If there is insufficient information to estimate the costs of the errors, then Mapstone (1995) suggested they should be weighted equally.
• Negotiate with the stakeholders the desired values of α and β with reference to the ratio established in the previous step.
• Design a sampling program to meet the desirable Type II error rate, β (established in the previous step), given the effect size specified earlier. This allows the sample size and details of the design to be finalised.

Mapstone (1995) detailed 2 alternative decision procedures that can be followed once data have been collected and analysed.

Ideally, these negotiations should consider a number of potential indicators simultaneously. In the process of balancing Type I and Type II errors, some indicators will inevitably prove much more costly than others if the 2 error rates are to be kept low. In such cases, stakeholders are faced with a choice:

• replace the costly indicators with others that will detect the stipulated effect size more cheaply, or
• increase the sizes of both errors while maintaining the ratio between the errors (if the costly indicators must be included in the program for some reason).

The only way to reduce the sizes of these errors is to increase the sampling intensity. Maintaining the ratio between the errors ensures that Type I errors are not minimised at the expense of increasing Type II errors. The monitoring program should not lose power to detect an important change at the expense of being conservative about the probability of incorrectly declaring that an important change has occurred.