AMP Toolbox
Step 5: Evaluate and adjust policies
Within the scope of this step, it is anticipated that evaluation and adjustments are planned at the measure level (not at the level of the Programme of Measures). Such evaluations make evaluation at the policy level more pertinent and, if relevant, may lead to preparing a new policy cycle.
For policies aiming to achieve GES within a limited time span, such as the MSFD and the MAP EcAp initiative, evaluation primarily comes down to measuring the remaining gaps between the GES (defined by targets and indicators) and the current state of the environment, against the time left for full achievement of the targets. Other parameters could also be assessed according to the MSFD, e.g. socio-economic impacts, costs, benefits and sustainability, as well as policy acceptance by stakeholders, side effects and all key factors affecting policy performances, as identified in Step 3. To determine the current state of gaps, the monitoring programme designed in Steps 2 and 3 and implemented in Step 4 should have produced the necessary data.
More information
A well designed monitoring programme according to the MFSD covers essential elements for making a policy adaptive, such as the need to identify the cause of any changes in order to identify possible corrective measures to repair deviations from the desired progress in achieving or maintaining GES.
Why is this step necessary?
A regular review and adjustment of measures, even when the policy is performing well, can help address emerging issues and trigger important policy adjustments. Evaluation provides the opportunity to learn lessons from the decision-making process as well as from the responses of the ecosystem to the measures. Learning from major unexpected social or institutional problems linked with the functioning of the ecosystems often requires reassessing the problem from Step 1.
Technical learning about the structure and functions of the ecosystem often requires simple monitoring adjustment, as presented in
For policies aiming to achieve GES within a limited time span, such as the MSFD and the MAP EcAp initiative, evaluation primarily comes down to measuring the remaining gaps between the GES (defined by targets and indicators) and the current state of the environment, against the time left for full achievement of the targets. Other parameters could also be assessed according to the MSFD, e.g. socio-economic impacts, costs, benefits and sustainability, as well as policy acceptance by stakeholders, side effects and all key factors affecting policy performances, as identified in Step 3. To determine the current state of gaps, the monitoring programme designed in Steps 2 and 3 and implemented in Step 4 should have produced the necessary data.
More information
A well designed monitoring programme according to the MFSD covers essential elements for making a policy adaptive, such as the need to identify the cause of any changes in order to identify possible corrective measures to repair deviations from the desired progress in achieving or maintaining GES.
Why is this step necessary?
A regular review and adjustment of measures, even when the policy is performing well, can help address emerging issues and trigger important policy adjustments. Evaluation provides the opportunity to learn lessons from the decision-making process as well as from the responses of the ecosystem to the measures. Learning from major unexpected social or institutional problems linked with the functioning of the ecosystems often requires reassessing the problem from Step 1.
Technical learning about the structure and functions of the ecosystem often requires simple monitoring adjustment, as presented in
Figure 1 Adaptive management displayed as a cycle, showing technical learning and social/institutional learning (adapted from Willams et al., 2014)
Who should be engaged?
Well designed policies should designate competent authorities for policy evaluation. Evaluation should be conducted by a group outside the implementation team to ensure objectivity. Stakeholders should be involved in the evaluation process, as they are in the position to alert others to unintended effects in the field and to propose corrective measures.
How should this step be carried out?
Who should be engaged?
Well designed policies should designate competent authorities for policy evaluation. Evaluation should be conducted by a group outside the implementation team to ensure objectivity. Stakeholders should be involved in the evaluation process, as they are in the position to alert others to unintended effects in the field and to propose corrective measures.
How should this step be carried out?
Two levels of evaluation are recommended, one addressing implementation and management procedures and the other addressing policy outcomes. To determine the success with regard to achieving the policy outcome the remaining distance from or the progress towards GES should be assessed with reference to the policy plan and the policy timeline.
A carefully planned monitoring programme is required to provide the information to enable such an evaluation, including the collection of data on social, economic and environmental impacts of measures or unwanted side effects. A well designed monitoring programme should encompass monitoring for single measures as well as monitoring for the policy as a whole. Key questions should include:
• Why the outcome differs from the outcome foreseen in the appraisal;
• How effective the measure has been in view of the objectives, and why;
• The cost effectiveness of the measure; and
• What the results imply for future management or policy decisions.
A report of results should be brought to the notice of competent authorities to amend the measures or the policy. Timing of evaluations depends on the dynamic of the cause-and-effect processes relative to the measures and also on the importance, complexity or controversy of the issues at stake.
Some policies provide for periodic revisions of the policy as a whole (e.g. six- year cycles for the MSFD). However, it is advisable to evaluate these policies at the measure level more often than every six years. Frequent evaluations at the measure level prepare the way for evaluation at the policy level and make it more pertinent.
Key activities
Information on activities
What should be the outcome?
Evaluation
• An assessment of current conditions and trends
• Foresight analysis and integrated assessment of the ongoing policy to detect emerging issues and to investigate hidden cause-effect relationships
• Policy pilot review process
Adjustments
What should be the outcome?
Evaluation
• An assessment of current conditions and trends
• Foresight analysis and integrated assessment of the ongoing policy to detect emerging issues and to investigate hidden cause-effect relationships
• Policy pilot review process
Adjustments
- An overview of policy objectives in need of modification or adjustment based on what was learned during evaluation. It is useful to make a distinction between social and institutional learning and technical learning.
- Relevant policy authorities should be informed in case of major social and institutional impacts or changes, the consequences of which exceed the possibilities of simple policy adjustment. In that case, a complete revision of the policy should be prepared. Hedging actions are implemented if possible.
- In case of technical learning, changes in monitoring plans or policy instruments are designed and implemented.
Further reading
- Bhadwal, S., Barg, S., Swanson, D., (2009), Automatic policy adjustments, Chapter 5 in: Swanson, D., Bhadwal, S., (Eds), (2009), Creating adaptive policies, a guide for policymaking in an uncertain world, IDRC, Ottawa, Canada.
- HS Treasury UK, 2011. The Green Book, Appraisal and Evaluation in Central Government.
- European Commission, 2009, Impact assessment guidelines (Document SEC (2009) 92).
- European Commission, DG EuropeAid, Evaluation Method X (Note that the list of toll include some basic tools which are in the EAF tool box, with more solid references).
- MFSD, Annex V.
- Swanson, D., Bhadwal, S. (Eds), (2009), Creating adaptive policies, a guide for policymaking in an uncertain world, IDRC, Ottawa, Canada, (accessed 2.13.12).
- Williams, B.K., Brown, E.D., (2014), Adaptive Management: From More Talk to Real Action. Environmental Management 53, 465–479. doi:10.1007/s00267-013-0205-7.
Example
Several problems or issues can be addressed with the AMP Toolbox, in this case marine litter in the Mediterranean and Black Sea is used as an example. For more details see flag example.
Several problems or issues can be addressed with the AMP Toolbox, in this case marine litter in the Mediterranean and Black Sea is used as an example. For more details see flag example.
Information on examples
Evaluation and adjustments are key aspects of adaptive policies. This step involves investigating whether and to what extent the policy is effective and how much of the problem has been addressed (“Evaluate on-going policy”) and what more needs to be done (“Adjust the policy”).
Evaluate: Data recorded with monitoring plan should facilitate knowledge accumulation about the sources, transport, fluxes and impacts of marine litter; and increase the confidence of the models or the scenario planning methods. Apart for this technical learning, the plan should also facilitate cyclical assessment and revision of the targets, as well as the rest of the elements of the policy.
Evaluation and adjustments are key aspects of adaptive policies. This step involves investigating whether and to what extent the policy is effective and how much of the problem has been addressed (“Evaluate on-going policy”) and what more needs to be done (“Adjust the policy”).
Evaluate: Data recorded with monitoring plan should facilitate knowledge accumulation about the sources, transport, fluxes and impacts of marine litter; and increase the confidence of the models or the scenario planning methods. Apart for this technical learning, the plan should also facilitate cyclical assessment and revision of the targets, as well as the rest of the elements of the policy.
Adjustments: can be performed following briefly the processes described in Steps 2, 3 and 4; or the whole cycle for fundamental changes. For more details see flag example.