ATTENTION: This page is intended to be viewed online and may not be printed or copied.

7. Performance Evaluation and Continual Monitoring

7.1 General

Once an organization has established a SCRM program including processes for identifying and treating risks, it should implement a monitoring program and evaluate plans, procedures, and capabilities through periodic review, testing, post-incident reports, and other exercises. It should check the conformity and effectiveness of the program, and establish, implement, and maintain procedures for monitoring and taking corrective action as necessary. This includes reviewing other organizational changes that may affect SCRM.

As the first step of performance evaluation, self-assessment is effective. Self-assessment can provide an overall view of the performance of an organization and degree of maturity of the management and it can be also applied to SCRM. It can also give the metric of performance level. Furthermore, it can help identify areas for improvement and/or innovation and to determine priorities for subsequent actions. Therefore, the organization should do self-assessment as performance evaluation. A maturity model self-assessment tool is given in Annex K.

Above all, organizations should test their plans periodically. People learn best by doing, hence regular testing of risk treatment (security, crisis, and continuity) plans is necessary to ensure they will work when needed. Organizations may test plans in four ways, including:

1. An orientation or “walk-through” to acquaint teams with the plan and their roles and responsibilities in it.

2. A “tabletop” exercise to reinforce the logic and content of the plan and to integrate its decision-making processes and provide “hands-on” experience. This may entail presenting a team with a scenario and related events and posing problems to solve. The exercise is designed to provoke constructive discussion and familiarize participants with the plan, their roles and responsibilities, and possible gaps in the plan.

3. A functional test that creates simulations involving group interaction in actual disruptions in order to validate the key planning components and strategies. Such tests may include evacuation procedures.

4. A full-scale test to evaluate the plan and response through interaction of suppliers and supply-chain partners.

Table 2 provides an overview of key properties of the four testing scenarios. The design of the exercise and test should be based on risks identified in the risk assessment process.

Table 2: Overview of Key Properties of the Four Exercise and Testing Scenarios

SCRM plans should be tested at least annually to achieve desired SCRM objectives (not limited to those elements required by regulation). Exercising and testing should incorporate changes in plans or operating conditions.

Plans, like risks necessitating them, and risk treatments should be monitored over time. Risk management is a dynamic process addressing operations in an ever changing environment. Therefore, the adequacy and appropriateness of plans needs to be continually monitored and adapted to changing conditions.

7.2 Exercising and Testing, and Adjusting the Plan

The goals and expectations of tests and exercises should include:

• Validating effectiveness of SCRM plans and opportunities for improvement;

• Testing capacity (e.g., abilities of an emergency communication system, generator capacity, or back up information technology systems);

• Reducing time to accomplish a crisis response process (e.g., repeating exercises so as to shorten the incident management cycle such as response and recovery times);

• Increasing awareness and knowledge among persons working on behalf of the organization about the risk-management plan; and

• Incorporating lessons learned from previous tests and actual incidents.

Testing should occur at regularly scheduled intervals. It should evolve over time, starting as a relatively simple program. Future scenarios should increase in complexity as exercising and testing needs develop further. These can consist of individual or group drills, table top exercises, and fully functional hands on exercises. Testing of this nature should involve suppliers, customers, and other stakeholders as appropriate. Exercise and testing requirements should be embedded within the procurement contract terms and integrated into the supplier management processes.

Figure 12 provides a framework for exercises and testing. Testing, like the SCRM process, begins with establishing the context, and, like the SCRM process, is a cyclical process. Both involve planning, following through on the plans, checking their performance, acting to improve their performance, and reconsideration of how the results, as well as considering changes in the organizational context and how they might reshape the context, scope, and boundaries of SCRM for an organization.

Figure 12: Framework for Exercises and Testing

The first step in testing, evaluating, and adjusting SCRM programs should be setting of goals and expectations. Testing can keep response teams and persons working on behalf of the organization effective in their duties, clarify their roles, and reveal weaknesses in the SCRM program that should be corrected. In addition to testing the efficacy of risk treatment processes and identifying opportunities for improvement, goals of the exercise and testing regime may include:

• Awareness and training of persons working on behalf of the organization;
• Capacity testing;
• Reducing the time necessary to accomplish a SCRM process (enhanced response times);
• Team building;
• Solicit stakeholder input and testing assumptions of risk assessment process;
• Identification of persons for leadership roles in SCRM procedures; and
• Improved coordination with first responders and other stakeholders.

In defining goals and expectations, it is important to consider that the scope of testing should be planned to develop over time. Early tests could include evaluating individual components of risk treatment plans. As the exercises and tests evolve, they should become increasingly complex, covering the entire scope of SCRM plans and the interactions of components as well as including external participation by public safety and emergency responders.

Top management commitment and participation is essential for a successful exercise and testing program in planning, staging, and debriefing. A commitment to the exercise and testing program lends credibility and authority to the entire SCRM process. Exercises should be planned considering the risks to the organization as identified in the risk assessment as well as the inherent risks of the exercise itself. Timelines, metrics, and feasibility also should be considered during the planning process.

There are multiple roles that exercise and test participants perform. All participants should understand their roles in the exercise and the exercise should involve all participants. As part of the exercise, participants should be allowed to interact and discuss issues and lessons. Documentation and communication protocols should be clearly established for the exercise to provide the necessary data for evaluation. Emergency communications should also be developed if problems arise during the conduct of the exercise.

After completion, the exercise should be critically evaluated with the participation of top management. The evaluation should include, among other things, an assessment of how well the goals and objectives of the test were achieved, the effectiveness of participation, and whether the SCRM plans will function as anticipated in the case of a real crisis. An after action report should be created as a reference to catalog measures of success, opportunities for improvement, and lessons learned for subsequent exercises. Future exercising and testing, as well as the SCRM program itself, should be modified as necessary based on the exercise results. The exercise should be a driver for continual improvement of the SCRM program.

7.3 Tracking Change

Some risks, such as those posed by hurricanes and tornadoes, may not change much over time other than frequency and perhaps intensity. Other risks that organizations face, such as those inherent in their processes, suppliers, or their regulatory environment, can change. As a result, firms need to monitor risks and how to address them over time. The example below reviews the nature of regulatory risks and how organizations can respond to and monitor it.

While perhaps not obvious at first, regulations can create significant supply-chain risks. They can affect import and export documentation and compliance requirements, as well as shipment safety and security issues, thereby affecting shipment costs and creating the risk for delays and financial penalties. Regulations can affect the countries or states in which an organization may work, as well as those in which its suppliers may work.

Some recent examples of U.S. regulations affecting supply-chain processes include the requirement of the Transportation Security Administration for screening of all cargo on passenger jets, U.S. Customs and Border Protection's requirement for new data elements on the Importer Security Filing (ISF) regulation for all ocean shipments, and Customs regulations requiring use of a high-security bolt seal on all ocean shipments. The air-cargo screening requirement adds costs for new screening facilities as well as new risks of delay at points where adequate screening capacity might not exist. The ISF reporting requirement adds costs for compliance and shipment-delay risks if reporting is not done properly. The high-security bolt requirement can also add risk of delays or even rejection of a shipment should shippers fail to comply. Compliance failure in any of these or other regulations could also result in financial penalties, embarrassing news coverage, or even loss of license to do business.

To summarize, failure to monitor, shape, and respond to new regulations can pose significant risks for the supply chain. Below are some guidelines and current best practices for an organization seeking to minimize such risks. Like all recommendations in this Standard, these are meant primarily as guidelines to provoke thought, and from which organizations may wish to select for adaptation to their own circumstances. An effective risk-mitigation program for legislative and regulatory requirements should help an organization monitor proposed or pending regulations, participate in the process shaping final regulations, plan and respond to changes in regulation, avoid compliance penalties, and ensure the smooth flow of incoming and outgoing shipments.

In monitoring risks, organizations should seek to become aware early of proposed legislative and regulatory initiatives, understand how they might affect their business, and share with internal decision makers to determine a response. Some means to do this include establishing a “government affairs” function or assigning individual responsibility to monitor proposed legislation and regulations, creating an internal network of individuals who monitor regulatory issues, joining trade associations that monitor these and subscribe to their newsletters and bulletins, and developing other external contacts to monitor legislative and regulation changes. Monitoring should include assessing the risk of emerging regulation, tracking compliance with existing regulations, and identifying the points of the supply chain that will be affected by regulations. Annex J provides some sample regulatory and compliance requirements, points along the supply chain they may affect, and what control, if any, an organization may have over them.

To shape regulations, organizations should seek to participate in the legislative and rulemaking process. They may develop an internal process for tracking and responding to regulatory notices, using this process to identify the consequences of new regulations and to offer preferred alternatives. They might establish an internal capacity, or hire an external consultant or lobbyist, to represent the organization in the development of legislation or regulations. Joining and participating in industry associations provides another means for interacting with political or government-agency leaders who shape legislation and regulations. Organizations may seek opportunities for volunteering to participate on industry advisory committees or other outreach events that government agencies use in developing and seeking feedback on regulatory changes.

In responding to regulations, organizations should prepare in advance to avoid or mitigate the risks, including costs, delays, and penalties inherent in new regulations. While monitoring and seeking to shape pending regulatory requirements, organizations should develop, with early executive support and funding, an internal process or team of cross-functional representatives to analyze pending regulations and plan how to address each one. For new regulations, organizations should communicate details to partners and help them prepare to support the new requirements. New requirements may also require organizations to update their contractual terms and conditions with their supply chain partners. Developing and implementing plans to monitor the supply chain as new regulations go into effect can ensure that compliant processes are in place and working.

New regulations, like other evolving areas with which an organization should contend, can create significant risks for supply chains. These risks may range from costs to delays to compliance penalties to still other areas. To be resilient, a supply chain should have the capacity to monitor, shape, and respond to evolving areas such as new regulations.

7.4 Monitoring and Reviewing the Risk Management Program

A SCRM program is not a once off process; rather, it is an ongoing, dynamic, and living process. As a result, the organization should establish and maintain a process for monitoring and reviewing the SCRM program to:

• Update risk assessments as needed;

• Identify and evaluate the effect on the risk assessment and management of the changes in context, assumptions, and other factors that may change over time due to internal and external circumstances;

• Evaluate the effectiveness of risk treatments; and

• Evaluate the actual effectiveness after exercise and the manifestation of undesirable and disruptive events.

The Plan-Do-Check-Act model provides a good method for ongoing monitoring, review, and improvement of the risk assessment process.

Figure 13 shows one potential set of processes to ensure risk management becomes an integral part of running any business. The key factors are to: include a review of risks and risk treatments into on-going business meetings, incorporate risk information into annual business planning, and ensure mechanisms are in place to identify new and emerging risks.

Figure 13: Integrating Risk Management into Business Operations

Effective SCRM is essential to a successful business. As globalization increases, so too do the interdependencies and complexities between suppliers, logistics providers, and a successful enterprise. A breakdown in any part of the supply chain connecting these entities can potentially lead to catastrophic consequences.

The guidelines in this Standard are intended to assist in the crucial task of establishing an effective SCRM program tailored to the unique characteristics of each organization. These principles should be integrated into the other key corporate procedures and policies that address procurement and general risk management including supplier-management routines.

While no risk management program can fully predict, mitigate, or prevent all risks or consequences, organizations that proactively implement a supply chain risk-management program will be more resilient and prepared for the day when a "risk" becomes "real."

Next: Annex A