At a major insurance company headquartered in the Midwestern United States, the assistant vice president for corporate security has used an environmental risk metric for the past 12 years to help the company decide where to place office facilities around the country. The company owns or leases hundreds of facilities across the United States. Corporate security regularly collects a suite of data, assigns weights to various factors, and develops a numeric score that places each facility into a low, medium, or high category of risk. For each risk category, written policy specifies a cluster of security measures that should be in place at the site. Exceptions can be granted, but the systematic approach results in uniformity and in efficiency in decision-making and security systems contracting. Most importantly, the metrics-based approach helps senior management understand the level of risk in site selection and make informed decisions on risk management. In addition, over time, the metrics have steered the corporation toward having a smaller percentage of its locations in high-risk sites.

This example illustrates how security professionals can use metrics to determine what works, measure the value of security operations, and demonstrate security's alignment with its organization's objectives. To help security managers use metrics more effectively, the ASIS Foundation funded research to create tools for discovering, developing, assessing, improving, and presenting security metrics. By using the tools, security professionals may be better positioned to manage their operations, measure their effectiveness, and communicate with senior management.

Metrics are measurements or other objective indicators collected over time to guide decision-making. The term is sometimes used interchangeably with measurements, analytics, and performance measures. With metrics, security managers can speak to senior leaders in familiar business language, offering measurable results that correlate with investment. Without compelling metrics, security managers and their budgets rely largely on the intuition of company leadership.

Two years ago, the ASIS Foundation implemented a new structure for assessing and overseeing security research. The first test of that structure was a proposal for research on security metrics, says Linda F. Florence, Ph.D, CPP, president, ASIS Foundation Board of Trustees. "The ASIS International Defense and Intelligence Council had a special interest in the topic, having made several presentations on metrics at the ASIS Annual Seminar and Exhibits. The council formed a vision of what the security field needed, found researchers who could perform the work, and helped the researchers develop a proposal for ASIS Foundation funding."

The Foundation Research Council approved the proposal, and the Foundation sought and received funding from the ASIS Board of Directors. The result was the ASIS Foundation Metrics Research Project. The Foundation awarded a grant to Global Skills X-Change (GSX) and Ohlhausen Research to undertake the project. GSX specializes in applying validation, measurement, and standards development techniques to produce business tools. Ohlhausen Research, Inc., conducts research in security, criminal justice, and technology.

DEPTH PERCEPTION

The project's research team consisted of the author as principal investigator; subject matter expert and former Director of Information Protection for the U.S. Air Force Daniel McGarvey; Senior Analyst Megan Poore; and Technical Advisor Lance Anderson, Ph.D.

Throughout the research, which be­gan in 2013, the ASIS Defense and Intelligence Council ensured that the security practitioner's point of view was represented by serving on the project's advisory board and expert panel.

The researchers gained insights into security metrics through a systematic review of the literature, an online sur­vey of ASIS members, and lengthy fol­low- up interviews by phone. In addition, the research team was guided by an advisory board and an expert panel composed of security professionals with experience in the use of metrics. The project was completed in the spring of 2014.

The research found many books, articles, and reports discussing reasons to use metrics, characteristics of existing metrics, and methods for communicating metrics. Among the most valuable resources on security metrics were George Campbell's Measures and Metrics in Corporate Security: Communicating Business Value and Mary Lynn Garcia's The Design and Evaluation of Physical Protection Systems, as well as numerous articles in both Harvard Business Review and MIT Sloan Management Review—the latter on business metrics generally.

This noted, most sources that examine security metrics operate at a conceptual level only. The literature has few specific strategies for developing or evaluating security metrics. Likewise, descriptions of empirically sound security metrics with statistical justification and evidence are scarce.

To uncover specific uses of security metrics and to gain an understanding of the different ways in which security professionals may be using metrics, the research team invited more than 3,000 ASIS members to participate in an online survey. The survey's 20 questions asked about metrics collection, comparison to external benchmarks, return on investment, sharing and presentation of metrics, and alignment with organizational risks and objectives. The survey also examined the particulars of metrics usage among respondents.

The 297 respondents demonstrated a high degree of interest in metrics. Of the respondents who said they are not using security metrics, 78 percent said they would use metrics if they knew more about how to create and use them effectively. More than half of all respondents asked for more information from ASIS regarding metrics.

Respondents provided the research team with a detailed view of the many ways that security professionals are using metrics today, including focusing on topics, reporting data, sharing with the C-suite, aligning with organizational risk, and using a dashboard tool.

Metrics topics. Respondents were asked which aspects of the security program they measure. The top five categories were security incidents, criminal incidents and investigations, cost against budget, security training and education, and guarding performance, which includes turnover and inspections.

Reporting. Eighty percent of respondents who use metrics provide their metric findings to persons outside the security department. Recipi­ents of the information include senior management (79 percent of those who share metrics outside the security department), managers of other departments (59 percent), supervisors (51 percent), and people who report to the security department (47 percent). Those who share metrics provide the information quarterly (43 percent), monthly (40 percent), or annually (17 percent).

Sharing. Respondents who share metrics with C-suite personnel were asked which elements they share. The top choices were security incidents (80 percent), cost against budget (62 percent), criminal incidents and investigations (57 percent), regulatory compliance (44 percent), and risk analysis process (40 percent).

Alignment. Eighty percent of respondents who use metrics said that their metrics are tied to, aligned with, or part of the larger organizational risk process or organiza­tional objectives. For example, some metrics protect the company's most important product line; other metrics may support business continuity, compliance, risk management, or client satisfaction. One respondent explained that top management sets broad goals and writes plans while se­cu­rity metrics demonstrate how effective those plans are.

Dashboard tool. Forty-four percent of respondents who use metrics perform their data collection, review, or sharing via a security management dashboard tool.

This research makes it possible to clearly define security's role and contribution to the organization at the tactical, organizational, and strategic levels. The report provides a working metrics tool that can help practitioners use metrics in the most effective manner.

IN THE TOOL BELT

GSX and Ohlhausen Research studied the current uses of security metrics and created several resources for practition­ers. The Security Metrics Evaluation Tool (Security MET) helps security pro­fessionals develop, evaluate, and improve security metrics. A library of metric descriptions, each evaluated according to the Security MET criteria, provides valuable resources. Guidelines for using metrics can help security professionals inform and persuade senior management.

The tools, especially the Security MET, are designed to help security managers assess and refine metrics that they are using or considering, based on an intimate knowledge of conditions at their organization, in a manner guided by scientific assessment methods.

Security MET. The Security MET is meant to aid and empower the security manager, not to dictate any particular security decision. By providing a standard for scientific measurement, it offers guidance for improving the inputs that go into the security professional's own decision-making process.

The Security MET is a written instrument that security managers can use to assess the quality of specific security metrics. Users can determine whether an existing or proposed metric possesses scientific validity, organizational rele­vance (such as clear alignment with corporate risks or goals), return on investment, and practicality.

The tool was developed through a comprehensive, iterative process that involved synthesizing scientific literature, reviewing security industry standards, and obtaining input from metrics experts on the project's advisory board and expert panel. Many of the criteria come from the field of psychometrics, which is concerned with the measurement of mental traits, abilities, and processes. The psychometric literature addresses the measurement of complex human behaviors, including sources of error inherent in social and organizational situations. In addition, through its connection with legal guidelines and case law, psychometric theory provides ways to address complicated legal issues related to fairness and human error.

The tool presents nine criteria for evaluating a security metric. The criteria fall into three groups: technical, operational, and strategic.

Technical. The technical criteria include reliability, validity, and generaliz­ability. Reliability means the degree to which the metric yields consistent scores that are unaffected by sources of measurement error. Validity refers to the degree to which evidence based on theory or quantitative research supports drawing conclusions from the metric. Generalizability means the degree to which conclusions drawn from the metric are consistent and applicable across different settings, organizations, timeframes, or circumstances.

Operational. Operational criteria include the monetary and nonmonetary costs associated with metric development and administration, as well as timeliness and the extent to which metric data can be manipulated, coached, guessed, or faked by staff.

Strategic. Strategic criteria include return on investment, organizational relevance, and communication. Return on investment is the extent to which a metric can be used to demonstrate cost savings or loss prevention in relation to relevant security spending. Organizational relevance is the extent to which the metric is linked to organizational risk management or a strategic mission, objective, goal, asset, threat, or vulnerability relevant to the organization—in other words, linked to the factors that matter the most to senior management. Communication refers to the extent to which the metric, metric results, and metric value can be communicated easily, succinctly, and quickly to key stakeholders, especially senior management.

A score sheet is presented at the end of the Security MET. The instrument is easy to score and imposes little to no time burden on staff. Lower scores on particular criteria show where a metric has room for improvement.

Here's an example of how the Security MET can be used to evaluate a real-life metric. At a major financial services firm, employees were being robbed of their mobile phones on the sidewalks all around the office as they came to work, when they went outside for lunch, or when they left to go home. The firm identified hot spots and times for phone theft and applied extra security measures. After reaching a maximum of 40 thefts in a two-month period, the number soon declined to zero.

Evaluating the metric with the Security MET provides some valuable insights. The metric—the number of mobile phone thefts—is highly reliable, as it is based on incident reports from employee victims, police reports, and video surveillance. Its validity appears to be confirmed by the outcome—that problem was eliminated. Collecting the data has little marginal cost, as the company already tracks and trends security incidents. Its organizational relevance is high, as it aligns with the firm's goal of attracting workers to the central business district. As for communication, it is a straightforward metric that is easy to explain. In terms of return on investment, it is hard to quantify the value of keeping employees safe and continuing to attract new employees.

Thus, while the metric appears to present a reasonable return on investment, the Security MET helps the user see that developing clear proof of ROI would be one way to strengthen this particular metric. The addition of a short survey asking employees if they feel more se­cure and would recommend the company to others would provide validation for both the solution and the metric.

Metrics library. The researchers de­veloped 16 summaries of metrics currently in use in the security field. The summaries were developed primarily through telephone interviews with on­line survey respondents. The summaries may serve as examples for security pro­fessionals who are considering ways to use metrics. (See box on page 58 for a complete list of topics.)

The library presents a three- to four-page summary of each metric. In addition, each metric is evaluated by several metrics experts, using the Security MET. The metrics library is presented in the full project report.

These real-world metrics come from a variety of industries including defense/aerospace, energy/oil, finance, government, insurance, manufacturing, pharmaceuticals, real estate management, retail, security services, shipping/logistics, and telecommunications.

Some of the metrics are more sophisticated and detailed than others, providing a range of examples for potential users to consider. The metrics are not presented as models of perfection. Rather, they are authentic examples that security professionals can follow, refine, or otherwise adapt when developing their own metrics.

Guidelines. A key task in this research was to develop guidelines for effectively using security metrics to persuade senior management. What would make those presentations more compelling? Several recommendations emerged.

Present metrics that are aligned with the organization's objectives or risks or that measure the specific issues in which management is most interested. One of the most important measures is return on investment (ROI).

Present metrics that meet measurement standards. A metric may be more persuasive to senior management if it has been properly designed from a scientific point of view and has been evaluated against a testing tool, such as the Security MET, or established measurement and statistical criteria.

Tell a story. If the metric is prevention-focused, a security professional can make the metric compelling by naming the business resources threatened, stat­ing the value of those resources, and describing the consequences if the event occurs. Another part of a compelling story is the unfolding of events over time. Metrics can show progress toward a specific strategic goal.

Use graphics and keep presentations short. Senior managers may be interested in only a few key measures. While security professionals may choose to monitor many metrics via a dashboard interface, they should create a simpler dashboard for senior management. Some security professionals said they limit their presentations to five minutes.

Present metric data regularly. As data ages it becomes more historical, less actionable, and thus potentially less valuable. The research does not suggest an optimal interval for sharing security metrics with senior management, but the survey shows that 83 percent of security professionals who share metrics outside the department do so at least quarterly. 

Future steps for helping security professionals improve their use of metrics include a webinar sponsored by the ASIS Defense and Intelligence Council and the further development of the metrics library. Other ideas under consideration include metrics training for security practitioners, the development of a tool for creating a metric from scratch and implementing it in an organization, and the creation of a library of audited— not merely self-reported—metrics.

The best security practice is evi­dence-based; without research, practitioners must rely on anecdotal information to make decisions. The ASIS Foundation continues to seek ideas for research projects that would increase security knowledge and help security professionals perform their work more effectively.

The complete project report, Persuading Senior Management with Effective, Evaluated Security Metrics, is available as a free download. The 196-page report contains the full text of the Security MET, the library of metric summaries (with evaluations), guidelines for presenting metrics to senior management, the project's literature review, and detailed results of the online survey.

Florence says, "We are proud to brand this quality research with the ASIS Foundation logo and share the findings with our members and the security profession as a whole. This research will help propel security from an industry to a profession, where we belong."  

Peter E. Ohlhausen is president of Ohlhausen Research, Inc., and served as principal investigator for the ASIS Foundation Metrics Research Project. He is a member of ASIS.