Enterprise Architecture is being increasingly adopted in organizations these days. Yet the research regarding the measurement of value generated by EA is limited (Kaisler, et al. 2005). This paper is a work-in-progress paper that provides an insight into the EA value measurement process. Achieving strategic alignment between business and IT is one of the major concerns the top management feel organizations are facing today. In this paper we try to address this problem through EA and try to demonstrate how EA helps in achieving strategic alignment. We start off with performance management and investigate how it affects various facets of organization, why it is essential for organizations and other related topics. During the last 20 years, both EA and performance management have evolved as two separate lines of activities many times loosely touching each other but never fully integrated. One of the main contributions of this paper would be to extend the ideas laid out and to develop an integrated model that can be a foundation for measurement process in a broad range of enterprises. They can use this research to improve the efficiency and effectiveness of the performance management process itself and to make the Enterprise Architecture a living-actionable means for organizations. This paper serves as an initial step in identifying the key areas of concerns in achieving the integration and a kick start by value measurement process of EA. Here, we discuss the current state of practice of Enterprise Architecture (EA) in detail, its framework, benefits from EA implementation in organizations and also value measurement for an EA through goal driven approach. We also propose survey methodology in order to examine above mentioned concepts in industrial world and to strengthen our understanding of metrics in organizations, to suggest a comprehensive list of metrics categories, look into individual metrics themselves used by successful companies implementing EA and the need for having a comprehensive regulatory procedure for reviewing the set of metrics chosen. Through this paper we are trying to tie theory and practice together which to our best of knowledge is missing in most EA research articles that talk about EA value measurement.
Get your grade
or your money back
using our Essay Writing Service!
Performance management, Enterprise Architecture (EA), business strategy, strategic alignment, metrics, Goal Question Metric (GQM), Representational Theory of Measurement.
Organizations these days are realizing the increasing need for having effective business processes and achieving strategic alignment of business and IT. The top concern for organizations is how to align IT with their business needs. Many academic articles have addressed these issues in the past but putting it into practice in a real and dynamic environment is far from reality. 'IT and Business Alignment' remains the top management concern as per leading survey reports . Managing vast amount of data, avoiding redundancy, achieving effective and efficient flow of information across organization and cross organization would be some of the top priorities organizations have . These are some of the growing needs of service industry today to survive in the revolutionary market. This has led to the increasing adoption of Enterprise Architecture practice in organizations.
Companies have to realize the fact that they cannot manage things if they don't measure it properly. Metrics are the means used to clearly and consistently indicate performance. Performance management deals with managing and maintaining metrics. There has been much research carried on the need for metrics and measurement in Enterprise Performance Management programs [3, 4]. Metzenbaum in her article discuss the difficulty in determining what constitutes appropriate measures and targets as well as difficulty in determining how often to measure . In this paper, we propose that without reliable, consistent measures, accurate communication of performance trends and targets is not possible. In order to achieve this we propose a conceptual model based on well known and widely adopted IS theories - Goal Question Metric (GQM) approach and Representational theory of measurement. This model serves the purpose of strategic transformation and EA value measurement initiative. In order to complement our literature findings, we have designed a survey instrument and further report the results with an attempt to validate our assertions to gauge market trends.
The paper is outlined as follows: Section 2 provides an overview on background literature or prior work. In this, we discuss the process of performance management, its definition, importance of performance management system, various contemporary models, and how companies have adopted them to achieve their target goals, some limitations of each of these models. We will also discuss the meaning, importance, framework and business perspectives of enterprises architecture and the how it supports business-IT alignment.
Always on Time
Marked to Standard
Section 3 deals with the introduction to metrics and the role played by performance metrics in organizations and the growing need for comprehensive metrics for achieving organizational objectives.
Next in section 4, we have our research questions and the design flow. The design flow involves the steps we follow through the process of our research design.
In section 5, we review the existing IS theories relevant to our research. This includes reviewing GQM model and representational theory of measurement. GQM is one of the widely applied IS theories to determine software metrics in IT field but less common in EA field. We extend this approach with some additions to EA value measurement to determine EA metrics in addition to measuring strategic alignment achieved by implementing EA.
In Section 6 we discuss our research methodology where we suggest a conceptual model for measuring the value of EA combining the IS theories mentioned above with few enhancements to extend it to EA. This section also involves analysis of our model through example that explains the working of our model.
Section 7 includes some of the challenges that organizations encounter during the process of measurement.
The last Section 8 discusses about conclusion and future work.
In this section, we provide a detailed background on various topics which includes performance management models (Balanced Scorecard, performance model for time-based companies), limitations of these models, concepts related to EA including EA frameworks, importance of business-IT alignment. We review the existing literature related to these topics and lay a foundation for developing an integrated performance driven EA model.
Performance Reporting and Performance Management
Performance management has become crucial to any business ranging from small business enterprises (SMEs) to large profit making corporations to achieve organization wide success 
Organizations today are investing huge amount of money on performance related activities . In order to remain competitive, they are adopting various performance measures to enhance the value of their intangible and intellectual assets.
Majority of the companies are inclined towards traditional system of reporting and management structure where business processes are not systematically measured.  Measurement is still viewed as a department related or decentralized activity neglecting the big picture of organization as a whole.
One of the related works and widely accepted one in corporate performance measurement is Balanced Scorecard (BSC)  by Kaplan and Norton. BSC measures corporate performance across four areas: financial, customers, business processes, learning and growth . Figure 1 illustrates BSC framework. BSC emphasizes the need for companies to understand the need for improving their intangible assets to achieve better performance through improved customer relationship, develop innovation in products and services, effective knowledge management, increased throughput of high-quality products and instilling a need for continuous learning and growth cycle in organizations .
Before BSC was introduced, organizations were mainly focusing only on financial measures. One of the major contributions of BSC is extending the area of focus to organization's vision and strategy in addition to financial measurement to envision future goals and achieve long term business objectives.
Some of the drawbacks of BSC are: It addresses issues in performance measurement only to some extent in that it lacks a clear direction for business process performance measurement . Also it limits non-financial measures to customer satisfaction which is not a comprehensive list of measures for companies looking for innovation . It also does not consider factors like suppliers, culture and other goal related factors.
Figure 1: Balanced Scorecard approach (Adapted from Robert Kaplan et al., "The Balanced Scorecard: Translating Strategy into Action", Harvard Business Review 1996)
CUI Song et al., have proposed an integrated Performance model for time-based companies.  They argue that better management of time is one of the critical success factors that determine company's performance. They mention that by effective management of time, organizations can increase their customer service, reduce costs, and efficiently manage resources. This model addresses performance management from the perspective of time and hence we feel it lacks to address the broader picture of organization in its entirety.
In order to achieve better performances, companies must understand the clear demarcation between Performance management and Performance reporting.
According to Gartner, performance reporting refers to reporting on how well a process or organization performs. This is usually measured in terms of service levels effectiveness, data volumes and cost of goods and services. These are considered to be important metrics but they are also lagging metrics (historical in nature) i.e, they do not have any kind of future state associated with them.
This Essay is
a Student's Work
This essay has been submitted by a student. This is not an example of the work written by our professional essay writers.Examples of our work
In contrast performance management process involves set of methodologies to identify the right set of metrics, organize and monitor the results and use them towards comprehensive initiatives to address organization as a whole across all levels. It involves management of people, process and technology to achieve desirable outcomes for the welfare of the entire organization.
Thus, we can infer that major portion of performance management is mostly inclined towards indicating future performance and therefore employs leading metrics like trends, variances, benchmarking measures, change management, scorecard reporting and tracking. The whole purpose of performance management measure is not just defining metrics but developing solid business case to identify business objectives and communicating the same to line of shareholders to set a clear vision. This needs to be clearly understood by the management in order to achieve the objectives set to the fullest possible extent.
Overview of Enterprise Architecture
According to Anne Lapkin et al., (2008) in their Gartner research article clearly define what Enterprise Architecture means as follows :
"processes that help organization translate its business mission and strategy into effective organizational change through clear understanding of enterprise's current as-is state and work towards attaining a better future or to-be state through evolution."
Companies today are trying to be more agile in an effort to respond to internal or external drivers rapidly. This can be achieved by adopting a solid architecture. Many large organizations are realizing the strategic impact and business value EA generates . Main expectations of enterprise architecture will be to reduce complexity and achieve better accuracy in articulating strategy, having a precise scoping for projects (with present and future projections), seamless integration of software services (both homogeneous and heterogeneous ) and planning procedures . They should also involve methods that are necessary to cater to the needs of all the stakeholders (customers, suppliers, employees, shareholders, the public, collaborators, and Government) to remain transparent in the business. Kurpjuweit and Winter (2007) claim that EA assists stakeholders of the organization in decision making processes like effective project planning, design and documentation for reusability, clear communication between IT and management staff etc.
EA is widely implemented by enterprises to cope with the increasing demands of the business and establish an integrated approach towards managing people, process and technology. It helps in establishing effective and efficient governance not only in individual departments but also across entire organization and cross-organization. It also helps management prioritize their goals and be more transparent in communicating the same to staff across organization which reduces complexity and misunderstandings .
According to Ross et al. (2006), EA is being widely adopted and has become a well established business process management system in recent years.
Its main contribution is in aligning IT with business needs and help setting up effective governance to achieve better agility . Some of the identified benefits of EA include:
assists in better decision making process of the enterprise 
enhance business through mergers and acquisitions
better alignment of IT and business, better predictability of project costs (includes acquisition cost, operational cost and maintenance cost)
helps in avoiding information redundancy in business processes and better utilization of resources 
Moving on, let us see what EA comprises.
There are four architectural sections (Figure 2) that are commonly used as part of overall enterprise architecture and are described below :
Business Architecture: defines key business processes, organizational strategy and models, governance structure used by organization to meet its goals.
Application Architecture: defines specifications for building applications that serve business purposes. It provides an idea of the application systems that needs to be deployed to establish the necessary correlation with the business processes of organization.
Data (Information) Architecture: involves data definition, categorization and sharing of data through well defined Service Level Agreements (SLAs) between stakeholders. Information architecture also includes managing data sources and provides accountability for information needs of the organization.
Infrastructure / Technical Architecture: deals with the IT infrastructure like hardware and networks that supports the information, applications and business dealt in the above architectures. It also includes establishing standards for technologies and methods used to operate Application architecture and support application needs.
Figure 2: Enterprise Architecture model
Role of EA in business-IT alignment
Today, companies are realizing the increasing need for aligning IT along with their business needs in order to reduce the gap that exists between the two and to achieve potential returns from IT investments. Today, IT has been increasingly viewed not only as business enabler but also as business reformer . IT assets contribute to a major portion of company's performance. Studies in the past have revealed that the health of the company is highly determined by IT assets including strategic alignment and other intangible metrics that add value . Rising business mergers and acquisitions, increasing economic crisis, unstable governance are some of the situations that lead to dynamic business environment . Strategic alignment has a significant role to play in performance improvement processes of organizations and allows companies to be more productive and competitive in these dynamic market conditions 
Figure 3 represents 'Strategic Alignment model' first suggested by Henderson and Venkatraman almost a decade ago.
This model is useful for analyzing the objectives of IT department and objectives of the organization as a whole. As illustrated this model has two distinct areas - Business and IT which further has two parts associated with each .
Under business section, the model suggests inclusion of business strategy and organizational infrastructure. Business strategy further includes scope of the business, governance and competencies. The scope of the business includes anything that is part of the business ranging from customers, products and services, suppliers to external factors like market share, competitors and other environmental factors. Governance includes governmental rules, compliance initiatives and other regulatory measures. Business competencies include innovation, improved R&D index, strategy momentum, new product development and other research related activities.
Under organizational infrastructure we have the organizational skills, the administrative structure (which includes the management roles and reporting structure) and organizational processes.
IT strategy can be further divided into IT strategy and IT infrastructure. IT strategy comprises of the technology related competencies, IT scope and IT governance. In recent years IT strategy has been increasingly applied to support not only infrastructure management but also in risk management, performance related activities and other integration tasks.
IT scope includes the capabilities of information technology like IT systems and services that are crucial for the organization.
IT competencies includes technology related attributes that contribute towards quality control and other high level competencies like reliability, consistency, agility etc for better alignment with the business needs of the organization.
IT governance includes effective SLA's (Service Level Agreements) in partnerships, risk management and mitigation, other IT related regulations.
IT infrastructure includes IT architecture (applications and technological standards and rules driving towards integration of services and products towards a common business goal), IT processes include development and maintenance related activities, IT skills refer to the human resource capabilities and intellectual skills.
According to Gilbert (2007) there are many aspects of strategic alignment that needs to be studied in depth. They include addressing the issue of strategic alignment in Small and Medium Enterprises (SME), alignment in Multi Business Company (MBC), the fit between governance and business strategy and many other areas which needs further research and needs to be explored in greater detail.
Figure 3: Strategic Alignment model (Adapted: "Strategic Alignment: A model for organizational transformation through IT" Oxford University Press)
Even though the concept of alignment is not new, there are many questions left unanswered. There are limited studies done that show how to achieve practical alignment in organizations which is also one of the major concerns of organizations. According to Luftman et al. alignment of business and IT results in transformation and this should serve as the base to improve organizational performance in addition to addressing a particular issue .
In the sections below we show how EA can be a means to achieve business-IT alignment.
Enterprise architecture is a body of knowledge used to integrate various constructs of organization - people, process and technology to deliver the right information to the right people at the right time. EA frameworks help in systematic documentation of business processes and IT assets 
Developing a framework for EA includes defining information systems and how these systems interact in achieving organizational goals . EA frameworks play a major role in achieving benefits of EA. Some of them include:
EA frameworks embed a list of standards for systems within them.
They assist the architects in tools selection and enterprise wide decision making process and establishing governance .
Supports integration and accounting for information resources on a global scale .
Some organizations have adopted already existing EA frameworks while majority of other organizations have developed a framework of their own (hybrid framework) with some customization to primary frameworks. Some of the primarily used EA frameworks by organizations include Zachman framework , The Open Group Architecture Framework (TOGAF) , Federal Enterprise Architecture Framework (FEAF)  adopted by the Government of USA, Department of Defense Architecture Framework (DoDAF)  used by the Dept of Defense.
Zachman framework for EA was proposed by John Zachman in 1987. He is considered to be the father of EA. This framework basically deals with six constructs or six basic questions: what, how, when, why, where and who . Even though this framework is first of its kind and considered a solid framework for EA, it does not provide a comprehensive solution in that it provides a high level view of establishing process. It lacks direction regarding the sequence and implementation and has no explicit compliance rules defined as there is no standard industrial methodology adopted . However some believe that following it in complete will automatically fulfill all compliance requirements .
The main feature of TOGAF when compared to Zachman framework is it incorporates a formal process (series of steps) for building EA .
FEA was developed by the Chief Information Officer (CIO) in the US Federal Council  mainly assisting information sharing across various Federal agencies to achieve integration . One of the complementary efforts to FEA was The Clinger-Cohen Act of 1996  which was introduced in the Federal Government in order to maximize the benefits realized from IT. One of the main concerns raised regarding this act was it did not necessarily address IT governance completely.
DoDAF was built mainly to handle mission critical situations and used to implement EA in defense.
Hybrid framework: is combining the features from different primary EA framework and building a customized framework to suit the individual business needs of organization. Nowadays, the tendency of companies is towards implementing hybrid frameworks.
Through our survey research we are trying to capture information regarding the rationale behind organizations developing a hybrid framework (it can be based on the views or abstractions in these frameworks ) and the reason for hybrid technique being so popular in the EA community. We also try to find out what elements are taken from these primary frameworks and used in building a hybrid framework for EA.
Measurement is not a new topic to us. We deal with measurement in some or the other form in our daily lives. Starting from weather forecast to measuring speed almost all activities involve measurement. When talking about metrics for business processes the whole concept of starts with the expert's statement "You cannot manage what you cannot measure. And you cannot improve something which you cannot manage properly" . According to Andy Neely (1995) the term metric refers to series of measurement steps - defining a measure, how the measurement will be done (may involve mathematical calculations), who will be doing the calculation and the source/origin of data.
There has been many efforts to formalize the process of software measurement but none of them have specified how exactly to calculate the measure. In the course of this effort many software tools have been built which gives different values of measures for the same program . The process of measurement is so complex that there is not just one model that serves as a best fit for all measures. Every organization has its own customized measurement process depending on the needs and management decisions. Ross et al (1993) mention in their article that there had been a long ongoing need for having a metrics program for software improvement process but there is very little research done on evaluating these metrics program .
A systematic and solid corporate performance management process is needed for companies to survive in this information age . And an important pat of performance management is metrics. Performance metrics gives us a picture of the health of the company, identify the strengths and weaknesses across segments and also detect processes (eg. fault detection, risk analysis) that deviate from the normal behavior . Organizations need metrics to assess and improve customer satisfaction, time-to-market factors and other innovational processes for improving their performance measures . There has been extensive research study carried out to show the impact of these metrics, customer metrics  in particular on financial performance and organizational performance overall . Accounting for tangible assets is very common and an easy process which every company does through financial reporting but when it comes to intangible assets, there is no single standard or a comprehensive reporting technique to officially report them . Many research studies have identified various constructs of customer metrics like observable or tangible constructs and unobservable or intangible constructs and how they affect organizational performance. Experts believe that the easiest way of collecting data to measure these unobservable customer metrics is through customer surveys.
RESEARCH QUESTIONS AND DESIGN
Our research question mainly revolves around EA value measurement and how organizations view it. In order to address this we have our research questions laid out as follows:
How do organizations measure the value of an Enterprise Architecture program? What metrics are used in the process?
What are the issues / challenges organizations face during the course of EA value measurement process?
Figure 4 provides the design flow for our research and the steps involved. We started off with extensive interdisciplinary literature review which includes not just academic journals and leading conference papers but also business press releases to get ongoing picture of the real world situation.
Designing the survey instrument and pilot testing follows the next step in our research. To our best of knowledge our survey is first of its kind and is designed to capture details of frameworks used with EA program, EA value measurement apart from capturing demographics of the industry and EA personnel. This will be sent to EA experts who lead EA profession in their respective organizations. These responses are crucial for our study and will form the basis for our understanding and explanation of various aspects related to EA.
After the survey is completed, the data from the respondents will be collected and analyzed using various statistical methods to draw important implications from them.
The next stage in the process is to use the survey results to verify the conceptual model we developed for EA value measurement to draw necessary implications and make comparisons of theory and practice.
The results from the comparison will then be reported using standard reporting techniques which organizations can use to assess the value of EA program or to assist them make necessary changes to their measurement process itself.
Figure 4: Research methodology
The survey instrument is divided into four sections. First two section deals with demographic information of the organization and about the individuals.
In the next section of the survey, we try to capture details regarding frameworks for enterprise architecture. Here we try to find answers related to adoption techniques of frameworks for EA. Discussion related to this section will not be done in this article.
The last section of the survey deals with EA value measurement. Here we probe questions like what are the organizational goals for EA program, what are the metrics organizations utilize to evaluate their EA program. This includes categories of metrics and individual metrics themselves. Also we try to find answers for questions like what are the challenges organizations face when collecting information related to these metrics, who evaluates/reviews these metrics, how satisfied the organization is with information on these metrics.
REVIEWING RELEVANT INFORMATION SYSTEMS (IS) THEORIES
There has been a lot of research associated with software metrics and many software tools have been developed using these software metrics . But when it comes to EA value metrics, organizations lack robust measurement techniques. And also there are very few studies and methods suggested to carry out EA value measurement [26, 39].
Experts claim that adopting EA methodology will improve the efficiency and effectiveness of the business processes but there is no standard techniques to measure its value and is still a big concern in the industrial world. There has been research models proposed to test empirically the extent to which EA frameworks serve the purpose of EA value measurement. However we claim that this does not address the issue in its entirety. For example, researchers have extended task-technology fit theory to assess the efficiency of EA frameworks . But this directly does not answer all questions dealing with EA value measurement.
In order to address this issue, we propose a framework constructed using two theories from IS as our underlying base for value measurement - Representational theory of measurement and Goal Question Metric (GQM) which organizations can adopt to build their EA value measurement model. The framework we propose can also be used by organizations to redesign their EA value measurement techniques and enhance evaluation techniques to help in providing optimal feedback for assessment.
As stated in previous sections, we will be using the survey instrument (questions) for testing our framework in order to have a robust working model and also to add a practical flavor to our research design.
Tackling the problem of indirect measurement
The main goal of EA is to achieve enterprise wide integration and achieve optimal performance using the shared resources, make optimal use of knowledge base consisting of varying intellectual capabilities of stakeholders of the organization and utilizing other organizational strengths.
As discussed before corporate performance assessment has been one of the major concerns of companies today . Despite huge amount of expenses on performance measurement of integration projects, organizations are unable to achieve complete benefits from these expenses. There are various methodologies/standards laid out for developing software quality metrics. One of them is IEEE standard . Some of the concepts of measurement included in IEEE standard mentioned by Cem Kaner et al (2004) are as follows :
Attribute - is physical or perceptual property of an entity that is measurable
Metric - is the measurement function used to describe the attribute
Direct metric - this metric is independent of the measure of any other attribute. In simple terms it is the fundamental measurement as against derived measurement.
Software quality metric - using this metric we can represent the software data as a numerical value. Later on we can analyze how the quality of the software associated with a given attribute gets affected using this relation.
The authors also suggest that determining the list of quality factors is very important in order to develop a set of metrics for any given project. It is also necessary that each quality factor be associated with a direct metric or a set of direct metrics. They also suggest that each direct metric in turn will serve as the quantitative representation of a quality factor.
We know that throughout the course of EA process, the role played by humans is very crucial (eg, stakeholders form an important part in EA value measurement process). Many standards developed for measuring the software processes neglect humans and carry out the measurement in order to make the system simple and easy to create a metric. But when we consider humans as the main actors of the system, these techniques fail as the system is no longer linear and simple. It becomes a dynamic and multi-dimensional system. And hence, it involves lot of indirect or derived measures.
In order to address this issue, Cem Kaner et al (2004) suggest that defining a metric should involve answering a question first, assigning attributes that can answer this question and finally defining measures that relates to these attributes. This correlates to GQM approach commonly adopted in defining software measures which we will discuss in our later sections.
Gupta .S et al (2006) show ways to compute numerical values for various customer metrics like customer acquisition, customer equity, customer retention. They also discuss computing cross selling, customer lifetime value etc associated with customer seen as an important stakeholder of the company . In this paper the authors also examine the link between observable and unobservable customer metrics, how perceptual customer metrics influences company's financial performance and how observable metrics influence company's performance.
Using Representational Theory of Measurement
Analyzing the properties of an object necessarily involves 3 basic operations that include: measurements, evaluations, preferences and using the representational theory of measurement we can analyze these operations .
For carrying out measurement of an object, we need to assign numbers to properties of objects or entities that need to be judged which may include real world events in terms of empirical relationship . To summarize this, a given observed or empirical phenomena can be assigned / mapped to a numerical value using measurement procedure as shown in figure 5.
Figure 5: Representational Theory of measurement
The representational theory of measurement thus suggests the need for having a mathematical calculation when dealing with metrics which we try to utilize in our study of EA metrics.
Goal Question Metric Approach (GQM)
Goal Question Metric (GQM) (Basili, et al. 1994) is one of the widely used approaches used to measure software measures . Goal Question Metric (Figure 5) is an analytical model built based on the organizational goals to carry out the measurement process in a systematic way. It uses bottom-up approach to analyze and interpret the metrics data and it uses a top-down approach for defining metrics . To adopt this method, organizations must be clear of their goals and have them laid out clearly before any measurement process. The main purpose of using GQM is to define a measurement process which involves explicit rules and a formal procedure to clearly understand what the output is once we are done with the measurement activity. Since the metrics are linked to goals through questions, the GQM approach proves to be an efficient technique in arriving at a set of metrics with a definite and intended use .
The GQM has its origin from evaluating and identifying defects for a series of projects in the NASA Space Flight environment . It has various paradigms associated with it starting from goal setting to quality improvement paradigm and its application to various software and information settings . It can be customized to wide variety of measurement processes at multi-organization level and the success of the projects in the past where GQM is applied proves it to be very efficient methodology .
There are many customized measurement frameworks developed from GQM approach to suit varying organizational needs. Some of them include: Patrik et al (2006) have adopted GQM to build and test change management and requirements management business process systems . Barclay C et al (2009) have extended GQM approach to determine the contribution of IS projects by developing a Project Objectives Measurement Model (POMM) . Robert W Stoddard (1996) extended the concept of GQM to analyze software reliability . Janaina et al (1999) developed GQM-PLAN tool to develop metrics for software quality measurement process .
The premise of GQM is every goal an organization has is associated with a set of questions related to that goal and these questions in turn are answered through quantitative measures (metrics)
According to Basili et al (1994) there are three levels in GQM model: "Conceptual, Operational and Quantitative level" each of which individually correlates to Goal, Question and Metric. Brief explanations of each of these follows:
GOAL - Any organization having a formally laid out corporate performance management process should have clearly identified set of goals in order to initiate a value measurement process. In order to do this, they need to first identify objects for which goal is defined. Some of the objects are identified as products (includes design artifact, requirements document, programs etc), processes (describes the action done on products like testing, designing etc) and resources (includes items utilized by processes like hardware, software, man power etc). When organizations define a goal, they should keep in mind various factors that go into the definition of measurement like the purpose, issue, process and the viewer (Basili et al 1994).
QUESTION - Every goal is characterized by a set of questions and these questions help us to assess the quality and extent to which a goal has been accomplished.
METRIC - Metrics help in answering the questions asked above in quantitative fashion. They can be subjective or objective depending on how the measurement process is done. For example, metrics defined only for objects considered are said to be objective like size of the code, number of employees working on a particular project, time elapsed etc. If they measure the qualitative part then they are subjective like measuring customer satisfaction, efficiency of a program, throughput of a task etc.
Figure 5: Goal Question Metric Approach (Adapted from Basili, et al - "The Goal Question Metric Approach" (1994).)
Even though GQM approach is one of the widely accepted approaches for software measures, it needs some modifications in order to suit EA value assessment. Some of the limitations of GQM are as follows:
GQM does not explicitly address the issue of measuring business-IT alignment which is one of the shortcomings when using this approach for EA value assessment.
In the traditional GQM approach, goals are being defined for 'Project' which is the primary object of measurement . This limits its use when we define goals for 'Organization'.
GQM inherently does not involve any evaluation techniques or feedback loop for evaluating metrics. This limits its design for EA program where evaluation plays a major role.
In the next section we discuss how we overcome these issues and extend the GQM to build a conceptual model to suit organizational needs.
CONCEPTUAL MODEL FOR EA VALUE MEASUREMENT
There are many industrial reports that have tried to provide a solution for measuring the value of EA by defining set of metrics for EA. To mention some:
Leganza, G. (2002) mentions in one of the Forrester articles that EA metrics need to be classified under two categories: activity metrics (metrics associated with workload characteristics of EA group) and value metrics (metrics that show enterprise Architecture value to organization) . There are concerns raised regarding this article that activity metrics do not necessarily communicate value of EA or progress to the fullest extent. And value metrics cannot be directly correlated to sponsor's goals if there is no clear linkage between the two .
Scott, J. (2009) in his Forrester article mentions five essential metrics for EA value measurement as: metric for measuring strategy alignment, metric to measure financial stability, measuring improved customer satisfaction, metrics related to EA skills and EA process improvement .
We argue that the categories of metrics provided by these articles are very high level and do not necessarily include all the metrics used for EA evaluation. Through our research we are trying to study thoroughly and suggest a set of EA value metrics for addressing the issue of EA evaluation in industry. Our task is to provide a solution for organizations to define metrics for their EA initiative and consequently establish an effective EA value assessment program.
In order to answer our research question 1, we propose a conceptual model to evaluate the performance of EA. Through our extensive literature review process we have identified various categories of metrics as: financial, IT, customer, supplier, business/strategy, growth/innovation, compliance/regulatory. Each category of metrics in turn consists of individual metrics which can be associated questions to link to a specific goal. Through the survey we try to capture metrics under each of these categories and report them accordingly. We believe that having EA experts as our respondents for the survey, we can arrive at a comprehensive list of EA value metrics.
According to the literature analysis of GQM approach, one of the missing features of this method is it does not address the issue of strategic alignment explicitly. In order to measure the strategic alignment in particular and to measure the value of EA program in general we propose a conceptual model (Figure 6) which serves both the purpose.
Roche et al (1997) have extended GQM to business process measurement by suggesting a method called "Measurement Application Method" . Here the authors argue that measurement must be treated as not as product / producing metrics but as a process. Without having clear process modeling it is difficult to produce efficient metrics. They also state that improving the process automatically improves the product. Through our discussion we could infer that, EA program has been evolving as an enabler of IT. The role played by EA architects is becoming increasingly important in improving IT efficiency and effectiveness. EA provides direction for IT projects, helps in cost reduction and reusability of projects through shared services. Hence linking EA goals to IT goals of organization is very crucial for EA value assessment. One way to achieve this is through metric-based reporting which we have included in our model.
Another major feature of our model is 'Business need based Evaluation'. This is particularly important for any EA program as evaluating the results against business objectives through continuous feedback (iterative process) is an important step in the transitional phase of developing to-be architecture from as-is architecture in organizations. There are various process models suggested which stress the need for EA management to be an ongoing process with continuous feedback for evaluation . As shown in the figure, each level in our model has feedback network tied to it where the metrics developed during each iteration can be gauged / evaluated depending on the output produced (i.e., how far the goals have been accomplished). Further, this information can be used to make decisions regarding next set of organizational goals (to-be state).
As shown in the figure 6, GQM approach is applied to various levels of an organization - Strategic, Tactical, Process/Program level, Operational level each of which have their goals established in a hierarchical fashion in order to address organization as a whole.
Let us try and understand each layer in the organizational information system of our model in detail:
Strategic Level: Measuring strategic alignment is made easy and well defined with goals defined at the strategic level where we have business and IT strategies clearly laid out. This is depicted explicitly in our model at the strategic level. Typically in organizations, business strategy drives the IT strategy. Business goals and IT goals originate from each of these strategies. Further, each goal is linked to a set of questions which address long term strategic objectives. And these questions help in arriving at the set of metrics. Metrics derived from this layer include business/strategy metrics.
Tactical level: This layer is sometimes referred to as middle management. The goals here deal with short or mid term of organizations. Growth/innovation metrics, compliance/regulatory metrics of EA projects are defined in this particular level.
Program level: this layer deals with the goals that enable establishing a sound program master plan, setting up developmental activities and business operations. Financial metrics pertaining to EA could be a major part of metrics resulting from this layer.
Operational level: this layer is the bottom most layer in organizational system. It deals with goals concerning IT and related service operations. This layer is characterized by daily operations and is more structured. Some of the metrics defined in this layer include IT metrics, customer metrics, and supplier metrics.
Figure 6: Conceptual model for EA value measurement using GQM approach
One other important feature of this framework is, since each level is inter connected, one can consider defining goals considering the entire organization as opposed to the traditional GQM approach where the primary object of measurement is 'Project' and the goals are defined considering this. This serves the purpose of addressing and accounting for goals considering organization as a whole.
Consider a simple example to demonstrate the working of our model:
Consider an operational Goal 'A' - "to achieve sales effectiveness without dropping price and therefore margin for items/services sold" by sales team in organization 'X' which has the following constructs:
Purpose - Achieve/Improve
Issue - without dropping price and margin
Object - sales effectiveness
Viewer - from sales manager's viewpoint
Below is the table that corresponds to the above goal:
What is the current list price of goods and services?
List price revenue
List price indicates what the revenue would have been if the goods and services are sold without discount
What is the discount given on items/services?
What is the sales effectiveness?
Sales price Index
= 1- (Total discount revenue / Total list price revenue)
CHALLENGES TO EA VALUE MEASUREMENT
Based on our literature review of industrial whitepapers and research articles, we have identified that there are many challenges organizations face in EA value measurement process. A Forrester survey conducted in 2007 is one of the evidences for this. The survey was taken by more than 50 European enterprise architects and they revealed that the EA program was developed based on specific goals but the metrics developed to achieve those efforts were ill-defined or did not exist in reality . Majority of the EA teams were not very clear on objectives. They also lacked a formal review procedure and approval process for EA metrics chosen. Some of these issues are management based and we will try to address these issues through our survey questions.
CONCLUSION AND FUTURE WORK
We have suggested a conceptual model involving both qualitative and quantitative measurement to evaluate the value of EA through extended GQM approach. The motivation behind this research is the increasing need for a comprehensive value management program in the field of EA. We agree that there could be various other techniques that serve similar purpose and our method is just one of such research ideas. One of the interesting future works of this research would be to look into the business process or business case that is involved in developing the metrics for EA and how it varies with respect to organizational context. And also look into the cost associated with collecting EA metrics data.
The success of our model depends on the right set of questions asked. The metrics resulting from our model depends on these questions and hence we feel that this can be one of the limitations of our model. If the company has ill defined goals, then it would result in a wrong set of metrics or would lead to undesirable results.
I would like to thank to my advisor Dr. Brian Cameron for extending his invaluable support and guidance in writing this paper. Also my special thanks to John Dodd from Industry Advisory Council for sharing his ideas and contributing towards this research.