ATTENTION: This page is intended to be viewed online and may not be printed or copied.

Annex A

(informative)

A. Risk Assessment Methods, Data Collection, and Sampling

A.1 General

The challenge with optimizing risk assessment to achieve the assessment objectives is time. The assessor needs to develop an assessment strategy, or “path”, to collect data in a representative, logical, and methodical manner. Effective risk assessment planning is necessary to make efficient use of time to provide a complete picture of risks and the level of risk. The RTL is responsible for the effective planning and application of assessment strategy and methods. The RTL has the responsibility for oversight of conducting the assessment activities.

A.2 Types of Interactions

There are two types of interactions between the assessment team and the organization being assessed during the course of the risk assessment. In assessing risk, the assessment team will examine policies, procedures, human activities, technologies (including information systems), and the interfaces between human and technological activities. Types of interactions include:

• Human interaction – between assessment team and the organization being assessed (including internal and external stakeholders):

  • Conducting interviews;
  • Completing checklists, surveys, and questionnaires with stakeholder participation;
  • Conducting document review with stakeholder participation;
  • Exercises, gaming, workshops, and scenario analysis;
  • Sampling; and
  • Undercover investigations, hot lines, whistleblower and grievance programs, and intelligence resources.

• Minimal human interaction – assessment team review of equipment, technologies, policies, procedures, facilities and documentation:

  • Conducting document review (e.g., records, data analysis);
  • Physical examination and tests of risk control measures;
  • Observation of work performed;
  • Conducting on-site visit;
    
  • Completing checklists; and
  • Sampling (e.g., products, equipment).

A.3 Assessment Paths

Assessments typically involve multiple interdependent processes. The assessor may therefore segment the assessment by using tracing or discovery techniques and/or segment the assessment by risk, threat, or consequence type; activities or functions; value generator; or department. Examples of assessment paths include:

• Tracing: Chronologically tracking a process or risk event:

  1. Follow the path of an activity forward or backward through a processes starting at the beginning, end or middle; and

  2. Follow the path of a risk event forward or backward through a sequence of causes and effects, starting at the before, during or after the event.

• Process Method: Test a sequence of steps, or interactions of activities and processes:

  1. Use flowcharts of process flow diagrams;

  2. Evaluate process controls, interactions, effectiveness, and opportunities for improvement;

  3. Objectives Method: Focuses on specific objectives and the associated risks;

  4. Risk Source Method: Focuses on specific risk sources;

  5. Department Method: Focuses on a department, division, or functional level;

  6. Requirement Method: Focuses on needs and requirements of stakeholders (e.g., supply chain partners); and

  7. Discovery Method: Random assessment.

Assessment trails can be used to better understand risk and the identify root causes of weaknesses, as well as identify opportunities for improvement. This involves as progressive series of “why” and “what if” questions to identify root causes. The assessor should keep detailed notes of the assessment trail and recognize when the trail is heading for a dead-end.

A.4 Sampling

A.4.1 General
During an assessment, it is not always practical, in time or cost terms, to evaluate all available information. Sampling, the process or technique of selecting a representative part of a population for the purpose of determining parameters or characteristics of the whole population, may be necessary to adequately assess the risk. The method and rationale for sampling and the numbers of samples from the population should be tailored to the circumstances of the assessment to achieve the assessment objectives. The sampling approach should provide a level of confidence that the assessment objectives are achieved.

Completely random sampling may not always be appropriate. For example, in areas of known operational deficiencies, high information uncertainty, or higher risk the assessor should select more samples. Considerations in selecting sample size and sample selection include (but is not limited to):

• Major areas and issues related to risk;
• Areas of previous risk events, emerging risks, and historic weaknesses;
• Elements serving as foundations of the risk and business management system;
• Interactions between elements of the management system;
• Issues known to be of greater significance to the organization and its stakeholders;
• Activities liked to legal, regulatory or liability related issues;
• Activities and functions where resources are overtaxed;
• Complexity and interdependency of critical activities; and
• New or significantly changed activities.

In order to assure that conclusions are correct in assessing risk, it is important to understand the confidence factor that the results are unbiased and consistent with a sampling of the entire population. Successful sampling is based on focused problem definition. In sampling, this includes defining the population from which the sample is drawn. A population can be defined as including all people or items with a specific characteristic that needs to be understood.

Sampling should consider the steps in Figure 14:

Figure 14: Sampling Process

A.4.2 Sampling Methods
The selection of an appropriate sample should be based on both the sampling method and the type of data required. There are two types of sampling methods:

• Non-statistical sampling:

  • Relies on the knowledge, skills and experience of the assessment team;

  • Focuses on areas where previous problems have been found or areas for specific improvements;

  • Can be used to identify a root cause of a problem;

  • Emphasizes areas of high risk or high interest to the organization and its stakeholders;

  • Cannot make generalization about an entire population; and

  • No statistical estimate of the effect of uncertainty in the findings of the assessment and the conclusions reached.

• Statistical sampling:

  • Sample selection process based on probability theory;

  • Ensures each item of a population has an equal chance of being selected;

  • Used when conclusions about a population are required;

  • Attribute-based sampling is used when there are only two possible sample outcomes for each sample (e.g., correct/incorrect or pass/fail);

  • Variable-based sampling is used when the sample outcomes occur in a continuous range; and

  • Provides statistical estimate of the effect of uncertainty in the findings of the assessment and the conclusions reached.

A.4.3 Examples of Sampling Methods
Examples of non-statistical sampling methods include:

• Judgmental sampling: based on deliberate choice and excludes any random process.

• Convenience sampling: using those who are willing to volunteer, or cases which are presented as a sample.

• Haphazard sampling: samples are selected based on convenience but preferably should still be chosen as randomly as possible.

Examples of statistical sampling methods include:

• Random sampling: ensures every member of the population has an equal chance of selection.

• Systematic sampling: after randomly selecting a starting point in the population between 1 and n, every nth unit is selected, where n equals the population size divided by the sample size.

• Stratified sampling: the population is sub-divided into homogenous groups, for example regions, size or type of establishment. The strata can have equal sizes or there may be a higher proportion in certain strata.

• Cluster/Block sampling: units in the population can often be found in groups or clusters. The population that is being sampled is divided into groups called clusters.

A.4.4 Sample Size and Margin of Error
In statistical sampling it is important to understand the level of confidence. Any percentage less than 100% is possible, but in order to have meaningful results, the numbers should be close to 100%. Common levels of confidence are 90%, 95% and 99%. The value of α is determined by subtracting our level of confidence from one, and writing the result as a decimal. So a 95% level of confidence would correspond to a sampling risk of 5%, meaning the assessor is willing to accept the risk that 5 out of 100 of the samples examined will not reflect the actual values if the entire population was examined.

Next: Annex B