Security and Resilience in Organizations and their Supply Chains

ATTENTION: This page is intended to be viewed online and may not be printed or copied.

Annex A

(informative)

A.10 Performance Evalutaion

A.10.1 Introduction

Performance evaluation involves the measurement, monitoring, and evaluation of the organization’s management of risk, legal and regulatory compliance, health and safety, and human rights performance. The organization should have a systematic approach for measuring and monitoring its risk management performance on a regular basis. Metrics assure the organizations policy, objectives, and targets are achieved, as well as elucidate areas for improvement.

To measure and monitor the organization’s risk management performance, a set of performance indicators should be developed to measure both the management systems and its outcomes. Measurements can be either quantitative or qualitative, and should be directly related to the risk assessment and security and resilience objectives and targets. Performance indicators can be management, operational, or economic indicators. Indicators should provide useful information to identify both successes and areas requiring correction or improvement.

The ORMS should provide procedures for defining metrics, collection of data, and analysis of data collected. Metrics should be established to monitor and measure the effectiveness of the ORMS, and identify areas for improvements to enhance performance to preemptively prevent potential undesirable and disruptive events. Knowledge gained from this information can be used to implement corrective and preventive action.

Key characteristics are those that the organization needs to consider to determine how it is managing its significant risks, achieving objectives and targets, and improving security and resilience performance.

When necessary to ensure valid results, measuring equipment should be calibrated or verified at specified intervals, or prior to use, against measurement standards traceable to international or national measurement standards. Where no such standards exist, the basis used for calibration should be recorded.

A.10.2 Evaluation of Compliance

The organization should be able to demonstrate that it has evaluated compliance with the legal, regulatory, and contractual obligations as well as voluntary commitments identified, including applicable permits or licenses.

The organization should be able to demonstrate that it has evaluated compliance with the other identified requirements to which it has subscribed.

A.10.3 Exercises and Testing

Exercising and testing scenarios should be designed using the events identified in the risk assessment. Exercising and testing can serve as an effective training tool, and can be used to validate the assumptions and conclusion of the risk assessment.

Exercising ensures that technology resources function as planned, and that persons working on the organizations behalf are adequately trained in their use and operation. Exercising can keep persons working on the organizations behalf effective in their duties, clarify their roles, and identify areas for improvement in the ORMS plans and procedures. Exercising can reveal weaknesses in the ORMS that should be corrected. A commitment to exercising lends credibility and authority to the ORMS.

The first step in exercises and testing should be the setting of goals and expectations. A critical goal is to determine whether certain prevention and response processes work and how they can be improved. The organization should use exercises and the documented results of exercises to ensure the effectiveness and readiness of the ORMS – specifically, its risk treatment plans, team readiness, and facilities – to perform and validate its risk and business management functions.

Benefits of exercising and testing include:

• Validation of planning scope, assumptions, and strategies;

• Examine and improve competence of persons working on behalf of the organization;

• Capacity testing (e.g., the capacity of a call-in or call-out phone system);

• Increase efficiency and reduce the time necessary for accomplishment of a process (e.g., using repeated drills to shorten response times); and

• Awareness and knowledge for internal and external stakeholders about the ORMS and their roles.

The organization should design exercise scenarios to evaluate the risk treatment plans. An exercise schedule and timeline for periodically exercising the ORMS and its components should be established. Exercising and testing should be realistic, evaluate the capabilities and capacities of ORMS, and assure the protection of people and assets involved. The scope and detail of the exercises should mature based on the organization’s experience, resources, and capabilities. Early tests may include checklists, simple exercises, and small components of the ORMS. Examples of increasing maturity of exercises include:

• Orientation: Introductory, overview, or education session;
• Table Top: Practical or simulated exercise presented in a narrative format;
• Functional: Walk-through or specialized exercise simulating a scenario as realistically as possible in a controlled environment; and
• Full Scale: Live or real-life exercise simulating a real-time, real-life scenario.

There are several roles that exercise participants may fill. All participants should understand their roles in the exercise. The exercise should involve all organizational participants defined by the scope of the exercise; where appropriate, external stakeholders may be included. As part of the exercise, a review should be scheduled with all participants to discuss issues and lessons learned. This information should be documented in a formal exercise report which should be reviewed by top management. Updates should be made to plans and procedures, and corrective and preventive measures expeditiously implemented.

Design of tests and exercise should be evaluated and modified as necessary. They should be dynamic, taking into account changes to the ORMS, personnel turnover, actual incidents, and results from previous exercises. Lessons learned from exercises and tests, as well as actual incidents experienced, should be built into future exercises and test planning for the ORMS.

Exercise and test results should be documented, used during debriefs, added to lessons learned, and retained as records.

A.10.4 Nonconformities, Corrective and Preventive Action

The organization should establish effective procedures to ensure that non-fulfillment of a requirement, inadequacies in planning approach, incidents, near misses, and weaknesses associated with the ORMS (its plans and procedures) are identified and communicated in a timely manner to prevent further occurrence of the situation, as well as to identify and address root causes. The procedures should enable ongoing detection, analysis, and elimination of actual and potential causes of nonconformities.

An investigation should be conducted of the root cause(s) of any identified nonconformity in order to develop a corrective action plan for immediately addressing the problem to mitigate any consequences, make changes needed to correct the situation and to restore normal operations, and take steps to prevent the problem from recurring by eliminating cause(s). The nature and timing of actions should be appropriate to the scale and nature of the nonconformity and its potential consequences.

Sometimes, a potential problem may be identified, but no actual nonconformity exists. In this case, a preventive action should be taken using a similar approach. Potential problems can be extrapolated from corrective actions for actual nonconformities, identified during the internal ORMS audit process, analysis of industry trends and events, or identified during exercise and testing. Identification of potential nonconformities can also be made part of routine responsibilities of persons aware of the importance of noting and communicating potential or actual problems.

Establishing procedures for addressing actual and potential nonconformities and for taking corrective and preventive actions on an ongoing basis helps to ensure reliability and effectiveness of the ORMS. The procedures should define responsibilities, authority, and steps to be taken in planning and carrying out corrective and preventive action. Top management should ensure that corrective and preventive actions have been implemented and that there is systematic follow-up to evaluate their effectiveness.

Corrective and preventive actions that result in changes to the ORMS should be reflected in the documentation, as well as trigger a revisit of the risk assessment related to the changes to the system to evaluate the effect on plans, procedures, and training needs. Changes should be communicated to affected stakeholders.

A.10.4.1 Corrective Action
The organization should take action to eliminate the cause of nonconformities associated with the implementation and operation of the ORMS to prevent their recurrence. The documented procedures for corrective action should define requirements for:

• Identifying any nonconformities;
• Determining the causes of nonconformities;
• Evaluating the need for actions to ensure that nonconformities do not recur;
• Determining and implementing the corrective action needed;
• Recording the results of action taken; and
• Reviewing the corrective action taken and the results of that action.

A.10.4.2 Preventive Action
The organization should take action to prevent potential nonconformities from occurring. Preventive actions taken should be appropriate to the potential impact of nonconformities.

The documented procedure for preventive action should define requirements for:

• Identifying potential nonconformities and their causes;

• Determining and implementing preventive action needed;

• Recording results of action taken;

• Reviewing preventive action taken;

• Identifying changed risks and ensuring that attention is focused on significantly changed risks;

• Ensuring that all those who need to know are informed of the non-conformity and preventive action put in place; and

• The priority of preventive actions based on results of risk assessments.

A.10.5 Internal Audit

It is essential to conduct internal audits of the ORMS to ensure that the ORMS is achieving its objectives, that it conforms to its planned arrangements, that it has been properly implemented and maintained, and to identify opportunities for improvement. Internal audits of the ORMS should be conducted at planned intervals to determine and provide information to top management on appropriateness and effectiveness of the ORMS, as well as to provide a basis for setting objectives for continual improvement of ORMS performance.

The organization should establish an audit program (see ANSI/ASIS SPC.2-2014 for guidance) to direct the planning and conduct of audits, and identify the audits needed to meet the program objectives. The program should be based on the nature of the organization’s activities, in terms of its risk assessment, the results of past audits, and other relevant factors.

An internal audit program should be based on the full scope of the ORMS; however, each audit need not cover the entire system at once. Audits may be divided into smaller parts, so long as the audit program ensures that all organizational units, activities, and system elements – and the full scope of the ORMS – are audited in the audit program within the auditing period designated by the organization.

The `results of an internal ORMS audit can be provided in the form of a report, and used to correct or prevent specific nonconformities and provide input to the conduct of the management review.

Internal audits of the ORMS can be performed by personnel from within the organization or by external persons selected by the organization, working on its behalf. In either case, the persons conducting the audit should be competent and in a position to do so impartially and objectively. In smaller organizations, auditor independence can be demonstrated by an auditor being free from responsibility for the activity being audited.

NOTE: If an organization wishes to combine audits of its ORMS with quality, safety, or environmental audits, the intent and scope of each should be clearly defined. Third-party conformity assessment, performed by a body that is independent of the organization, provides confidence to internal and external stakeholders that the requirements of this Standard are being met. The value of certification is the degree of public confidence and trust that is established by an impartial and competent external assessment.

A.10.6 Management Review

Management review provides top management with the opportunity to evaluate the continuing suitability, adequacy, and effectiveness of the ORMS. The management review should cover the scope of the ORMS, although not all elements of the ORMS need to be reviewed at once, and the review process may take place over a period of time. The management review will enable top management to address need for changes to key ORMS elements, including:

• Policy;
• Resource allocations;
• Risk appetite and risk acceptance;
• Objectives and targets; and
• Security and resilience strategies.

Review of the implementation and outcomes of the ORMS by top management should be regularly scheduled and evaluated. While ongoing system review is advisable, formal review should be structured, appropriately documented, and scheduled on a suitable basis. Persons who are involved in implementing the ORMS and allocating its resources should be involved in the management review. In addition to the regularly scheduled management system reviews, the following factors can trigger a review and should otherwise be examined once a review is scheduled:

• Risk Assessment: The ORMS should be reviewed every time a risk assessment is completed for the organization. The results of the risk assessment can be used to determine whether the ORMS continues to adequately address the risks facing the organization.

• Sector/Industry, Contractual, and Political Trends: Significant changes in sector/industry, contractual, and political trends should initiate an ORMS review. General trends and best practices in the sector/industry and in security and resilience planning techniques can be used for benchmarking purposes.

• Regulatory Requirements: New regulatory requirements may require a review of the ORMS.

• Event Experience: A review should be performed following an undesirable or disruptive event, whether the prevention, mitigation, or response plans were activated or not. If the plans were activated, the review should take into account the history of the plan itself, how it worked, why it was activated, etc. If the plans were not activated, the review should examine why not, and whether this was an appropriate decision.

• Test and Exercise Results: Based on test and exercise results, the ORMS should be modified as necessary.

Continual improvement and ORMS maintenance should reflect changes in the risks, activities, and operation of the organization that will affect the ORMS. The following are examples of procedures, systems, or processes that may affect the plan:

• Policy changes;
• Hazards and threat changes;
• Changes to the organization and its business processes;
• Changes in assumptions in risk assessment;
• Personnel changes (employees and contractors) and their contact information;
• Subcontractor and supply chain changes;
• Process and technology changes;
• Systems and application software changes;
• Lessons learned from exercising and testing;
• Lessons learned from external organizations’ undesirable and disruptive events;
• Issues discovered during actual invocation of the plan;
• Changes to external environment (new client needs, political changes, relations with local communities, etc.); and
• Other items noted during review of the plan and identified during the risk assessment.

Next: Annex A - Maturity Model for the Phased Implementation