Security Metrics Maturity Model for Operational Security

CHAPTER ONE: Introduction

In this chapter, the key concepts and definitions by well known security authorities on security metrics is introduced and discussed. Then the issues and motivation that brings about this research topic is explained. Thereafter, the end result which is the objectives is put forth. To achieve these objectives, the goals are briefly outlined. There is also a section that explains the scope of the research and limitations for this work. Finally, the research flow on the chapters is explained.

1.1 Introduction

Information Technology (IT) is continuously evolving at faster rate and enterprises are always trying to keep pace with the changes. So does the threats. As the complexity of IT increase, the unprecedented threat environment and security challenges also have increased multi fold over the years. Security Managers and CSOs with the blessings of top management keep investing and on security solutions to protect from ever increasing adversaries. But getting the blessings is not always an easy task for them as management normally does not see the direct benefit. Convincing on security investment is also part of challenges for Security Managers and CSOs.

As part of the convincing process, the Security Metrics (SM) plays a vital role in any organization. It helps the management to have a close to comprehensive view of their organizational security posture. SM provides some measurement on how secure the organization is. However, how accurate is the information provided by the SM? Can the management take the SM as a final view of their respective organizational security posture? Can SM ensure the investment made for security is worth? A good SM should be able to answer accurately or provide some qualified response for the questions that management have.

SM receiving many attention lately as IT Security is no more an option. With multitude of attacks from adversaries and many regulatory requirements, organizations are spending on security investment to ensure they are protected and stay competitive in markets. The greatest push factors for the metrics awareness are the recent amplified regulatory requirement, greater demand for transparency and accountability. Additionally there are many internal factors that driving organization to justify security investments, security and business objectives alignment and finally to fine-tune effectiveness and efficiency of organizational security programs.

Much has been written and researched on SM on various aspects from data collection, analysis to measurement method etc. A considerable number of research efforts have been emerging in best practices, methodologies, framework, tool and techniques are being recommended and adopted to mature the security metrics. However, relatively little has been reported and proven on quality and matured metrics one has to follow and put in practice. Moreover security cannot be measured as a universal concept due to the complexity, uncertainty, non-stationary, limited observability of operational systems, and malice of attackers [VERENDEL V, 2010]. More has to be researched in the area of security metrics.

Many interpretations and meanings of Security Metrics have been found on the Internet. Some examples taken from well know publications and researchers are as follows:

According to the National Institute of Standards and Technology (NIST), “Metrics are tools designed to facilitate decision-making and improve performance and accountability through collection, analysis and reporting of relevant performance-related data” [NIST-SP, 2001].

Whereas SANS in its “A Guide to Security Metrics, SANS Security Essentials GSEC Practical Assignment”, Shirley C. Payne says that “Measurements provide single point-in-time views of specific, discrete factors, while metrics are derived by comparing to a predetermined baseline two or more measurements taken over time. Measurements are generated by counting: metrics are generated from analysis. In other words, measurements are objective raw data and metrics are either objective or subjective human interpretations of those data”. [SHIRLEY C. PAYNE, 2006] She also further describes what would be considered a “useful” metric:

“Truly useful metrics indicate the degree to which security goals, such as data confidentiality, are being met and they drive actions taken to improve an organization’s overall security program.”

Yet another one practical definition by Andrew Jaquith, states that “Metrics is a term used to denote a measure based on a reference and involves at least two points, the measure and the reference. Security in its most basic meaning is the protection from or absences of danger. Literally, security metrics should tell us about the state or degree of safety relative to a reference point and what to do to avoid danger”. [JAQUITH (1), 2007]

[M. SWANSON, 2003] highlights some of the key uses of security metrics in an organization. They are (not limited to):-

• Enabling organizations to verify compliance level against internal and external institutions. (e.g. laws, regulations, standards, contractual obligations) and internal ones (e.g. organizational policies and procedures

• Provide visibility and increasing transparency on accountability with regards to specific security controls and facilitating detection.

• Provide effectiveness and efficiency of security management by providing better visibility on security posture at high and granular level, helping in security strategies and display trends.

• Helping management to decide better on security investments in terms of allocating resources, product and services.

Having a right security metrics is a paramount in gauging a security posture of an organization. Most of the SM concerns coins from the correctness and effectiveness. Correctness denotes assurance that the security-enforcing mechanisms have been rightly implemented (i.e. they do exactly what they intended to do, such as performing some calculation). Effectiveness denotes assurance that the security-enforcing mechanisms of the systems meet the stated security objectives (i.e. they do not do anything other than what is intended for them to do, while satisfying expectation or resiliency).[BARABANOV et al, 2011]

Organizations faced with many security metrics options to be used. The security managers and CSOs bombarded with large set of related, unrelated, heterogeneous security metrics by different source or assets within the organization. How will they make these metrics to be more meaningful and eventually reduce risks and support strategic security decisions? Therefore, the decision makers should be furnished a proper security metrics guidelines that encompass the right type of measurement / data to choose, correct way of analyzing and interpreting and any other recommendations.

This research, therefore will explore further on the existing security metrics recommendations currently in practice. In order to improve the current security metrics, more research efforts are needed and focused in the area of good estimators, human element reduction, obtaining more systematic and speedy means to obtain meaningful measurements and better understanding of composition of security mechanisms. [LUNDHOLM et al, 2011]

Therefore, this research will explore the identification of quality security elements to determine matured security metrics as there are many areas within IT security that contributes to an organizational security posture. This mainly involves providing weight-age for each and every element. Thereafter the elements are then prioritized and finally sum up to provide a final security posture of an organization. Some of the key domains within security are cryptography, operational security, physical security, application security, telecommunication security and many more.

The research will identify elements within these domains that play a vital role in an organization to produce a security metrics report for management. These elements are further scrutinized and qualified to be part of the security metrics. The scrutinization and qualification is done through various researches done by previous researchers. The systematic techniques will provide a guided recommendation for near optimal security metrics for an organization.

The key questions for this research will be what is acceptable security metrics element or measurement for a domain? How accurately these parameters are obtained? How effective are they? As a whole how matured are the metrics? How these various elements and parameters can be used to provide an accurate and convincing security posture report for an organization in a practical manner?

To go further explaining this research, imagine this scenario: A key security personnel of an organization presenting a finding of the company’s security posture. She/he talking about how good the security in place, how good is the security fortress, how impenetrable the security perimeter and so on. To support his claims he throws some PowerPoint slides with security metrics. The management was like awed and feeling comfortable with the presentation and they felt secure doing their business. But then there are few questions from the floor on the accuracy, quality, completeness and maturity of the metrics. How confidence is the security metrics presented?

Hence a proper model that supports the claim is needed. The model will substantiate the claims of the security personnel on her/ his findings. Therefore this research will look into the ways of substantiating by proposing a maturity model.

The end result of this research will be guiding principles that leads Security Managers to produce a convincing and close to accurate report for C Level management of an organization. This research will look into various studies done on existing measurements and security elements for Security Metrics and produce a method that will portray the maturity of security metrics used in an organization.

1.2 Problem Statement

The lack of clear guidance on security measurements that represent a security posture of an organization has been always a problem despite many researches done in the area. Despite many methods and definition in the area of security metrics were introduced, nothing is strikingly clear that enable organization to adopt and implement in their respective organization particularly in operational security. There are many theoretical and more to academia texts available in this area [JAQUITH, 2007, M. SWANSON, 2003, CIS-SECMET, 2012]. Organizations still lack of precise knowledge of practicable and effective security metrics in the operational security settings.

1.3 Motivation

There is an obvious need in guiding organization to the right direction in implementing their respective organizational security program. There is paucity exist in the mode of guiding process for organization to implement security program with the right metrics to monitor their operational activities. The main incentive behind proposing a matured security metrics for operational security is a workable solution and guide for matured security metrics for any organization. Organizations need a model to look into the type of metrics used in their security program and a model to chart their metrics improvement program. Hence the solution will be an asset for organizations in implementing reliable and practical security metrics. This paper will answer question like “Are incidents declining and improving security over time? If yes or no, how reliable are the answers?” Is my metrics are correct and reliable if not how can I improve it? Further, the paper will provide some practical top down approach in approaching security metrics in an operational environment.

Another motivation for this paper is the findings from the [PONEMON, 2010], who claims many researches lack of guidance, impractical in operational environment and purely formal treatment as no empirical support as a whole.

In the end, through some findings of this paper, organizations will be able to gauge the return on investment on security investments. They should be able to measure successes and failures of past and current security investments and well informed on future investments.

1.4 Objectives

The problem statement and motivations bring the objective for this work. The objectives for this project will be:

a. To provide security metric quality taxonomy for operational security

b. To devise methods for matured security metrics for operational security

To achieve these objectives, the methodology and goals used for this work would be:

1. Conduct a literature review on existing research works and state of the art
2. Identify the key operational areas based industry expert inputs
3. Develop a taxonomy based on the key operational areas
4. Identify the key criteria or parameters that make a good quality metrics
5. Identify on how to categorize or rank the metrics to represent the maturity of a metric
6. Develop a method to guide for a quality security metrics
7. Develop a metric score card to represent maturity level
8. Develop a Security Metrics Maturity Index (SM-Mi)

1.5 Scope of Work

For the purpose of this research only a certain area of operational security is identified. Also to be more focused, to give a better view and example, we will choose few important and popular metrics among security practitioners. The research is aim to provide a very practical approach in operational security metrics for an organization, but is not meant to be treated as an exhaustive guide or resource. Metrics prioritization is out of the scope of this research as organizations have various different business objectives and goals. These decide and dictate the type of metrics to be used and emphasized as such metrics will not be discussed [BARABANOV, 2011].

1.7 Thesis Layout

The research consists of 6 chapters; the first chapter will describe some security concepts and motivation for this topic. The second chapter will delve into the related works done in this area. This chapter will identify some key research findings and what is lacking in them and how some of the information will help for this thesis. As for the research methodology and proposed framework, chapter 3 will explain this. Chapter 4 will identify and explain in detail the formulation of proposed metrics and taxonomy for operational security in the form of techniques. Meanwhile Chapter 5 will discuss a case study based on the solution proposed. Chapter 6 will be a brief chapter that summarize the research and will discuss on future direction of this research.