Metrics and the Maturity Mindset
Close your eyes and imagine yourself throwing darts at a dartboard. Any wagers on accuracy?
In the physical security space—that place where guards, gates, and badges once ruled—using metrics alone to measure risk and present value to the enterprise is similar to throwing darts blindfolded. While cybersecurity is critical, the physical security of people and property remains essential to strategic and tactical risk management for most organizations. What security teams often fail to recognize is that it’s essential to understand how mature you want to be in a variety of physical security domains and build an enterprise security risk management strategy around those maturity levels. Measuring metrics alone is simply cataloguing the completion of activity without a view to security risk management maturity or a clearly articulated strategy. That trio—a maturity mindset, a clearly defined strategy, and metrics measurement—is fundamental to effectiveness.
The Enterprise Security Risk Management team at Caterpillar Inc., headquartered in Peoria, Illinois, has joined forces with security experts at Ernst & Young LLP (EY) to demonstrate the value of having a maturity mindset. (See “Maturity Model 101” on page 38 for more on the process.) Not only does it help protect the people, products, property, information, and brand at Caterpillar, it also is central to making sure the security team and strategy are predictive and poised for future challenges and opportunities.
WHAT’S WRONG WITH METRICS?
Collecting data is valuable, of course, but the emphasis on metrics in the security discipline is sometimes misguided. Security teams can end up doing a good job of executing on a bad process. Metrics may look great, but if they measure an immature or broken process, they really don’t answer the questions that should be asked. For example, IT security might be proud to have cleaned 17,000 viruses out of the system in its efforts to be compliant, when it actually missed 5,000 viruses due to inadequate process or scope. The numbers don’t show the lack of effectiveness because the process is broken. Knowing how mature you want to be is what makes the difference because maturity targets translate into specific activities, programs, and projects to achieve the desired state, and metrics help measure against maturity.
For example, when Caterpillar Enterprise Security first began using EY’s cybersecurity maturity model, a decision was made not to be extremely mature in terms of evolving prevention technologies. Instead, the team wanted to become best-in-class in detect-and-respond maturity, assuring the ability to quickly recognize any serious network attacks and mitigate risk effectively. The objective was to give management reasonable assurance that the cybersecurity program would not become a money pit, spending wildly to prevent attacks that, frankly, are unavoidable in today’s climate. That picture for executives was literally worth a thousand words—the board and senior executives value the model as an excellent snapshot of where the security function is in time and where it is trying to be, as well as how it compares to peers in other industries such as financial services or transportation. Success in using the cybersecurity maturity model to communicate effectively with the C-suite—something with which physical security professionals often struggle—indicated it was time to apply the same effort and analysis to protecting people and property.
WHY IS A MATURITY MODEL BETTER?
In April 2013, Caterpillar and EY engaged eight CSOs from globally recognized companies and other industry experts in face-to-face and virtual meetings over nearly nine months to agree on domains, subdomains and definitions most relevant to physical security. The varied viewpoints and needs among the group led to interesting discussions—some more complex than others. For example, those with a more global footprint noted that the term “investigations” carries different meaning in some parts of the world and should be changed to “inquiry and investigations.” Some of the subdomains emerged from these discussions, assuring the ability to weight each area with more granularity and better reflect how various security organizations operate in different industries or parts of the world. Ultimately, the group agreed on nine domains, some with subdomains.
EY then developed a comprehensive questionnaire and interview guide with hundreds of questions related to each area. An independent assessment team executed the model among key stakeholders at Caterpillar for each of the nine domains to plot the first set of physical security maturity results. For example, consider the Crisis Management domain. The interviewer asks a variety of questions, including “Is a Crisis Management Plan in place?”; “Is there an assembled crisis management team?”; and “Does management have sufficient program oversight?” The assessment then follows with the 1–5 ratings. (See “Maturity Levels” 101 on page 38.)
Leadership visibility or support of the Crisis Management program would indicate a Defined (3) rating, yet only formal engagement from executives will garner an Optimized (5) rating. Having metrics and reporting requirements that are defined and integrated into annual evaluations is an indicator that the program is Managed (4), but not until these are reported to executive leadership on a regular basis is it possible to achieve a rating of Optimized (5).
With regard to the Crisis Management Team, ratings may vary based on roles and responsibilities, certifications and training, whether or not cross-functional members are included, and who has ultimate decision-making authority. When it comes to integration into the company’s disaster recovery plan, having no processes for integration merits an Initial/Ad Hoc (1) rating; a maturity target of Defined (3) might be sufficient for the security function if these crisis planning areas are handled effectively elsewhere in the enterprise.
Over the next couple of years, the assessment team refined the questionnaire to clearly delineate the future targets for each subdomain and to make it more Caterpillar-specific where needed to provide a more detailed picture that was still easy to comprehend. Caterpillar continues to raise the bar for various levels of maturity, and this tool also helps adapt to changes in the threat landscape—adjusting capabilities and technology resources as suggested by the desired future state and the output of the tool.
Caterpillar’s Physical Security Maturity Model has focused attention around two aspects of its physical security programs: First, is the maturity level of each area correct, or do some need additional attention? Secondly, do some areas need additional funding, and, if so, how can it be applied to advance the maturity? In the Crisis Management example, Caterpillar moved from a Managed (4) to Optimized (5) maturity rating by reporting metrics in this area to the executive office on a regular basis. To improve its maturity rating in the General Training and Awareness subdomain of Awareness, Caterpillar Enterprise Security budgeted for an annual Security Awareness Week that promotes awareness of both physical and cybersecurity among employees globally to move the maturity needle.
The maturity model has created a template for discussion with executive management that is simple to use and visual—it clarifies communication. The tool also is used for discussion with executives and the board to reflect progress and also to highlight areas needing additional investment. The visual representation (see “Maturity Model in Action,” page 38) tells a story quickly, capturing executive attention, and it provides a level of context that management can grasp more immediately. Once the executive office has this picture of where Enterprise Security stands, a detailed discussion follows as a corollary to this picture and facilitates more effective decision making. The tool has reinforced the security team’s emphasis on a risk-based approach to providing security of people and property across the enterprise.
ARE THERE COLLATERAL BENEFITS?
Interestingly, the Enterprise Security team is finding that the maturity model also provides a platform for telling its story—helping executives better understand what the Enterprise Security organization does. Each time the maturity model is presented, it creates an opportunity to talk about the team’s services and the value the team adds to the enterprise. For some CSOs, the maturity model could help to provide a justification for expanding or increasing the portfolio of services or areas of responsibility.
A physical security maturity model also is an excellent tool for building security risk management collaboration across the enterprise. It helps security teams better understand where there are overlaps and recognize that not everything in the model is owned by the security organization. It presents a picture of security capabilities and needs, regardless of who owns them—from facilities to employee health and safety to human resources to legal. To drive change, stakeholders have to agree to engage annually on what’s needed to move toward the future state and achieve maturity levels.
WHAT’S NEXT?
Caterpillar and EY are still accumulating information and evolving the Caterpillar Physical Security Maturity Model questionnaire and implementation process, expecting it to follow the same path as the cybersecurity maturity model in becoming a slide rule for risk acceptance, risk mitigation, and security investments. It is quickly becoming an effective tool for gaining faster agreement among business leaders about how much risk they are willing to accept for their operations, whether in Illinois, Ireland, or India. Using this tool to present a clear picture of where Enterprise Security was, where it is, and where the function wants to go demonstrates to executives where their investments will have the greatest impact.
Moving forward, Enterprise Security at Caterpillar will integrate maturity of both physical and information security into these discussions. This will give management a perspective on decisions being made in each area and unified Enterprise Security strategies. In the longer term, the plan is to converge the two models into one to present a unified Enterprise Security Risk Management roadmap. And, as EY collects data over time from other companies using the tool, it will show how Caterpillar security compares against its peers, and eventually provide a broader view across the entire industry.
--
Tim Williams, CPP, is CSO of Caterpillar Inc. He is a current member of ASIS International and a past president. Tom Schultz is an executive director at Ernst & Young LLP.
