This essay has been submitted by a student. This is not an example of the work written by our professional essay writers.
The collection, preservation, and forensic analysis of physical evidence are often crucial to the successful investigation and prosecution of crimes. The Federal Bureau of Investigation's (FBI) laboratory, located in Quantico, Virginia, is one of the largest and most comprehensive forensic laboratories in the world. The laboratory not only supports FBI investigations, but also provides forensic and technical services to federal, state, local, and foreign law enforcement agencies. The FBI's laboratory annually conducts over one million examinations involving analyses of physical evidence ranging from blood and other biological materials to explosives, drugs, and firearms. Laboratory examiners also provide expert witness testimony on the results of forensic examinations.
To keep a record of evidence provided to the laboratory for analysis, the FBI uses the Evidence Control System (ECS), created in 1978. The Laboratory Division converted this antiquated system to a database in 1998, but the ECS still has limited functionality. One FBI programmer developed the current version of ECS, and as new releases of database software become available, the database has been upgraded. The FBI currently uses Microsoft's Access 2002 as the ECS database software.
The ECS system represents an “in and out” tracking system. Evidence is entered into the system when it arrives at the laboratory, and the system documents: (1) the control number for the evidence,
(2) when an analysis has been performed on the evidence, and
(3) when the evidence leaves the laboratory. Except for this information in the ECS, the laboratory relies completely on paper documentation that follows a piece of evidence as it passes through the laboratory's various sections. Each section of the laboratory enters data into its own computers. However, these files are immediately printed out and paper copies, rather than an electronic file, are relied on to track the evidence and the work performed. In addition, the data entered into a section's individual computers are not linked to provide an overall management view of where the evidence is located, what analyses have been completed, or how long each step of the process is taking.
One laboratory official described the current system as very limited, and stated that when evidence is returned to the originator, its departure from the laboratory is not always entered into the ECS. As a result, FBI managers are unable to identify with certainty the evidence contained in the laboratory at any point in time or its progress in being examined and analyzed. Moreover, another laboratory official stated that only one person is familiar with the ECS database, a programmer from the FBI's Information Technology Operations Division (ITOD). The laboratory employee who created the original system has retired. The official also pointed out that despite available technology, the FBI continues to use a labor-intensive manual system. Each laboratory unit enters the same routine information, such as case number, date collected, and the submitting agency, for each item of evidence as it is passes from one unit to another for continued processing.
In comparison to the laboratory's limited database, modern commercial-off-the-shelf (COTS) laboratory information systems can provide many useful functions, including: the ability to track evidence throughout the analysis process; Internet capabilities that allow external agencies to review and request information about evidence they have submitted; extensive reporting, workload analysis, and responses to ad-hoc querying; on-line help; and data searching.
The FBI's laboratory hired a contractor in 1998 to assist in the development of requirements for an information management system to replace the ECS. The contractor also evaluated COTS systems. However, the FBI's Laboratory Division was unable to fund the project at that time.
In 2002, the Laboratory Division reprogrammed funds to replace the ECS with a modern information system. The system requirements developed by the contractor in 1998 were updated and validated through Joint Application Development (JAD)
Technical – Functional Requirements
Technical – Performance Plan
The FBI's evaluation of the JusticeTrax proposal cited some strengths but also areas of risk. Examples of JusticeTrax's strengths were: (1) It had a mature COTS system used by organizations with missions similar to the FBI's, including the Royal Canadian Mounted Police Forensic Services Laboratory; and (2) LIMS was already integrated with bar-code scanner and printers that could be provided for testing within 15 days and for implementation within 45. Although the committee assessed LIMS as meeting the laboratory's mission- critical needs, the evaluation also identified two key risks in addition to an ambitious delivery schedule: (1) because JusticeTrax is based in Arizona, it needed to hire employees to work on the project in Virginia, train them, and have them obtain security clearances within the timeframe proposed; and (2) the JusticeTrax product required significant customization of its software to meet the FBI's requirements such as security standards, migrating data from the ECS, and providing the capability to issue alerts and notices. Another concern was that JusticeTrax did not have the capability to provide web-browser connectivity immediately, but instead proposed converting its LIMS product to a web-based application in early 2004.
JusticeTrax LIMS Product Selected
Based on its evaluation of the six proposals received in response to its RFP, the FBI awarded JusticeTrax a $4.3 million contract in September 2003 to customize its LIMS product for the FBI's laboratory.
On May 20, 2005, the FBI's Information Management Project Review Board (IMPRB), one of the review boards established in the LCMD, reviewed the LIMS project. During the review, laboratory officials described the history of LIMS, including the laboratory's need for an information management system and the delays experienced in trying to implement the LIMS project. At the time of the review, JusticeTrax had already trained the FBI's would-be LIMS users. Although LIMS was functional, it had not yet been brought online because it did not meet all of the FBI's security requirements. The review board also learned that although JusticeTrax's basic LIMS was a COTS system, the software had undergone extensive modification so that about 95 percent of the FBI's version of LIMS was based on custom code. A member of the IMPRB doubted the project would pass the FBI's security certification and accreditation testing. The FBI's Security Division provides C&A, authorizing the deployment and operation of a system, only if it deems a system secure based on its testing and evaluation. FBI officials agreed that if LIMS could not pass C&A, then the project should be cancelled. The IMPRB expressed additional concerns about project risks, including the fact that the Visual FoxPro code used for JusticeTrax's LIMS is old technology and whether the small firm could adequately support the system into the future. The IMPRB recommended that a Red Team be assembled to review the LIMS project and consider alternative approaches.16
The FBI formed a LIMS Red Team in July 2005 with representatives of the Laboratory Division, the Office of General Counsel, the Office of the CIO, the Finance Division, and the ITOD. The team held meetings from July through October 2005 and presented its findings, conclusions, and recommendations to the FBI's CIO in October. From the beginning of its review, the Red Team identified serious technical deficiencies with LIMS, which included:
- The requirement for a web-browser interface had not been satisfied;
- There were security vulnerabilities associated with administrative shares (auditable records);
- The transmission between client and server interface was inherently insecure; and
- The technical architecture was not suitable to ensure chain of custody requirements.
The Red Team recommended terminating the JusticeTrax LIMS contract because the system could not pass C&A. The team also suggested that BizFlow, a product the FBI is licensed to use, might be a suitable alternative.17 According to the Red Team, BizFlow has the capability to integrate workflows with information management, create and replicate forms, provide formatted and customizable reports, and handle bar-coding equipment.
Certification and Accreditation
As the IT review board predicted, C&A testing led to the termination of the LIMS contract. As part of the LCMD, C&A is the FBI's management control for ensuring the adequacy of computer systems' security. The C&A testing and evaluation process is designed to ensure the FBI's systems are designed securely and remain secure throughout their life cycle. If the Security Division's testing and evaluation determine that a new system is secure, the Security Division provides accreditation and approves the system to enter into operations within the FBI's IT architecture.
The LIMS RFP required security to be part of the system. However, due to several high-profile espionage-related security breaches within the FBI, the FBI strengthened C&A requirements after the September 2003 award of the LIMS contract. The specifics were not available to JusticeTrax until the FBI provided the results of the FBI's Security Division's Certification Test Report to JusticeTrax in August 2005. The report stated that LIMS failed testing in four key areas: (1) password storage, (2) auditing capability, (3) control of grand jury evidence, and (4) shared directory (information sharing outside the laboratory).
In September 2005, the Security Division began testing for a second Certification Test Report after JusticeTrax provided patches to the LIMS software based on the first report. The FBI performed tests to ensure that the system was at an approved baseline security configuration and that the system presented little or no risk to FBI systems or data. However, the Security Division identified 14 vulnerabilities according to the ease of exploiting the system. The 14 findings ranged from “requires expert-level knowledge to exploit the vulnerability to gain access to the system” to “does not require tools or expert-knowledge to exploit and gain access to the system.” The significance level, meaning impact if exploited, for all 14 vulnerabilities was rated high.18
Termination of the Project
By October 2005, it became clear to the FBI that LIMS would not meet the FBI's security and other requirements. The FBI gave JusticeTrax an opportunity to correct the system's deficiencies, but those efforts were unsuccessful. Eventually, after 28 months of effort, the FBI terminated the LIMS contract.
On October 4, 2005, the FBI issued a Cure Notice to Justice Trax stating that the LIMS software application was not able to successfully pass the FBI's Security C&A Testing.19 In the Cure Notice, the FBI identified two outstanding concerns: (1) system security, and
(2) the lack of a fully functional web-browser interface. JusticeTrax attempted to correct the security flaws, but the FBI's Security Division did not accept the corrections. JusticeTrax planned to provide the web browser at a later date.
Based on the Certification Test Report and its finding that LIMS posed a very high security risk, the Security Division recommended on October 17, 2005, that LIMS not be accredited. The C&A process found that the system's vulnerabilities could not be mitigated due to the inherent design of the software. Therefore, the certifier recommended against granting an approval to operate the system.20
At the end of October 2005, the FBI issued a Stop-work Order to JusticeTrax. According to the Federal Acquisition Regulation, situations may occur during contract performance that cause the government to order a suspension of work, or a work stoppage. A Stop-work Order may be issued in any negotiated fixed-price or cost-reimbursement supply, research and development, or service contract due to advancement in the state-of-the-art, production or engineering breakthroughs, or realignment of programs.
In January 2006, the FBI issued a contract termination letter to JusticeTrax. In March 2006, the FBI and JusticeTrax agreed to terminate the contract. The FBI agreed to pay JusticeTrax an additional $523,932, and the contractor waived any claims arising from the contract.
The FBI's CIO noted to the OIG that the LIMS contract was awarded before the FBI's IT investment management controls were implemented through the LCMD. He stated that in his opinion, the LIMS project demonstrates the success of the FBI's LCMD because the FBI terminated the project after the IMPRB review and the C&A process showed that the LIMS system's serious deficiencies could not be corrected. The CIO noted that the LCMD process now requires project managers to come before review boards so that the FBI's divisions no longer manage IT projects in isolation. The CIO stated that the controls provided by the LCMD help to detect problems earlier in a project's life cycle.
JusticeTrax officials stated that in their opinion, the failure of the LIMS project was due to the FBI's lack of communication, information sharing, and resources. They also stated that the FBI did not provide a “champion,” that is, an FBI official who would work to ensure the success of the project. Finally, JusticeTrax officials said that the FBI insisted on requirements, especially regarding system security, that were not specified in the contract. Although the contract included a provision for security, JusticeTrax officials stated that details for the C&A requirements were never provided. After reviewing the requirements in the contract, we agree that the security requirements were too general to provide enough detail on how to meet the requirements.
Laboratory Division's New Review Process
In addition to the FBI's LCMD, the Laboratory Division had established in October 2005 a division-wide Major Acquisition Review Committee (MARC) to strengthen the oversight of the Laboratory Division's acquisitions, including IT investments. The MARC will assist Laboratory managers to ensure that Laboratory projects adhere to all Department of Justice and FBI requirements for sound project and financial management. The MARC mirrors the LCMD, but covers all projects rather than only the IT projects covered by the LCMD. The purpose of the MARC is to:
- review and approve Laboratory Division investments that meet the following thresholds: acquisition requests totaling $250,000 or more, IT requests totaling $50,000 or more, and all projects totaling $100,000 or more;
- ensure that the requests are aligned with the Laboratory Division Strategic and Program Plans;
- ensure that the requests have been included in the Laboratory Division's Fiscal Year Spend Plan;
- ensure that acquisition rules, regulations, and requirements have been appropriately adhered to;
- ensure that project management standards and practices are being implemented and appropriately reviewed;
- ensure that all IT requests are properly prepared and are aligned with the FBl's Enterprise Architecture, and adhere to the Office of the CIO's requirements; and
- ensure resolution of concerns affecting the acquisition project (e.g., mission alignment, requirements, technology, security, information sharing, funding, and risks).
The base year of the LIMS contract was September 2003 to September 2004, with a $1.6 million budget. The base year could be extended by four 1-year contract options, bringing the total contract budget to $4.3 million.
Prior to the Red Team's decision to recommend termination, the FBI paid JusticeTrax a total of $856,219 in personnel, training, and equipment costs. This included $205,136 in hardware that the Laboratory Division purchased from JusticeTrax that can be used by the FBI laboratory separate from LIMS.21 During our audit, we reviewed and verified that all expenses were supported by invoices.
When the FBI terminated the LIMS contract, the FBI and JusticeTrax agreed to a settlement of $523,932. Therefore, the FBI spent a total of $1,380,151 on the LIMS contract as shown in the table below.
FBI Payments to JusticeTrax
Personnel and training
Source: FBI data
The FBI wasted $1,175,015 on the LIMS project: $1,380,151 paid to JusticeTrax less the reusable equipment totaling $205,136.22
The FBI Laboratory Division's need for an information management system remains. To fulfill the need, the FBI is considering other COTS systems. For example, the Red Team that evaluated JusticeTrax's LIMS recommended Bizflow software, which is used for workflow and information management. The FBI purchased Bizflow to use within the FBI in general, but the software has not yet gone through C&A testing or other LCMD processes. Alternative solutions might also be found in other Department of Justice components' or other federal agencies' laboratory information systems. For example, the FBI has obtained information from the Drug Enforcement Administration on its ongoing project to acquire a system for managing evidence. The Bureau of Alcohol, Tobacco, Firearms and Explosives is also expected to deploy a new laboratory information system in the spring of 2006 that has been under development for over 5 years.
We concluded that the FBI's inability to implement the LIMS system and its loss of nearly $1.2 million in the attempt was a shared responsibility between the FBI and JusticeTrax. The project began before the FBI had established its ITIM processes. When those processes were implemented, they helped identify problems with the project that ultimately led to terminating the contract before losing additional money. Still, the FBI did not do its homework before awarding the contract, including adequately identifying and assessing the risks in selecting JusticeTrax, and in vastly modifying the company's COTS LIMS product. The FBI had a responsibility to not only ensure that JusticeTrax understood the system requirements, but that JusticeTrax also had the technical capacity to fulfill the requirements.
In addition, the FBI did not adequately document for JusticeTrax the security requirements for certification and accreditation of the LIMS software. To the extent security requirements evolved, those changes should have been made clear through contract modifications, if necessary. The FBI also should have identified the citizenship problem of the JusticeTrax president, foreseen the security clearance requirements for JusticeTrax personnel, and assessed the problems and delays inherent in requiring major modifications to tailor a COTS system — especially one based on an outdated code. A firmly managed schedule, and cost, technical, and performance benchmarks, would have raised danger signs early in the project and perhaps led to resolution much more rapidly. Among the FBI's weaknesses were: (1) the lack of established IT management processes to ensure a sound project and identify problems early, and (2) not designating a project manager to oversee the project. Also, two key contracting personnel, both of whom were involved in the development of the LIMS requirements, left the project only 4 months after the contract was awarded. This lack of continuity and institutional knowledge likely contributed to the poor outcome of the LIMS project.
Because JusticeTrax did not provide personnel with security clearances to work on the system, and its president was not a U.S. citizen, JusticeTrax contributed to the early delays in starting the project. It was incumbent upon JusticeTrax to meet all FBI requirements for the system, including mandatory security protections. However, JusticeTrax has a legitimate point that some details of the requirements were unknown at the start of the project.
JusticeTrax's use of outdated code made modifications difficult and time-consuming, and JusticeTrax did not properly assess its ability to perform the work required to adapt its system to operate in the FBI environment. Also, while JusticeTrax intended to make its system web-based, the delays in the project prevented that before the contract was terminated.
Because JusticeTrax was unable to mitigate unacceptable security vulnerabilities, the FBI had no choice but to terminate the LIMS contract. As a result, the FBI's Laboratory Division continues to lack a modern system to track evidence through the laboratory and otherwise manage its laboratory operations because it is difficult to determine the location and status of evidence at any given point in time or to determine how long the process is taking. We believe the FBI should consider adopting a COTS workflow system for its laboratory information system or an acceptably secure information management system used by another federal law enforcement entity.
We agree with FBI officials who stated that the FBI's LCMD should prevent problems such as those encountered with LIMS if the processes are applied as intended with detailed requirements for the contracting process, management oversight boards, and other controls to ensure troubled projects are identified sooner and can be remedied.
We recommend that the FBI:
1. Consider whether a COTS workflow system or laboratory information management systems in use or under development within the federal government will meet the needs of the FBI laboratory.
2. Ensure that any project to provide a laboratory information management system not only follows the FBI's LCMD but is overseen by an experienced IT project manager.
3. Establish cost controls to ensure that training or other expenses are not incurred prematurely in the development of a successor to the LIMS project.
STATEMENT ON COMPLIANCE WITH LAWS AND REGULATIONS
This audit assessed the status of the FBI's Laboratory Information Management System (LIMS) project. In connection with the audit, we reviewed management processes and records to obtain reasonable assurance that the FBI's compliance with laws and regulations that, if not complied with, in our judgment, could have a material effect on FBI operations. Compliance with laws and regulations applicable to the FBI's LIMS project is the responsibility of the FBI's management.
Our audit included examining, on a test basis, evidence about laws and regulations. The specific laws and regulations against which we conducted our tests are contained in the relevant portions of the Federal Acquisition Regulation.
Our audit identified no areas where the FBI was not in compliance with the laws and regulations referred to above. With respect to transactions that were not tested, nothing came to our attention that caused us to believe that FBI management was not in compliance with the laws and regulations cited above.
STATEMENT ON INTERNAL CONTROLS
In planning and performing our audit of the FBI's Laboratory Information Management System (LIMS) project, we considered the FBI's internal controls for the purpose of determining our audit procedures. This evaluation was not made for the purpose of providing assurance on the internal control structure as a whole. However, we noted certain matters that we consider to be reportable conditions under the Government Auditing Standards.
Reportable conditions involve matters coming to our attention relating to significant deficiencies in the design or operation of the management control structure that, in our judgment, could adversely affect the FBI's ability to manage its LIMS project. During our audit, we identified the following management control concerns.
- The FBI's Laboratory Division remains without an information management system to aid laboratory mangers in overseeing the operations of the laboratory.
- The FBI initially lacked an Information Technology Investment Management process, but has corrected that deficiency.
Because we are not expressing an opinion on the FBI's internal control structure as a whole, this statement is intended solely for the information and use of the FBI in managing its IT investments. This restriction is not intended to limit the distribution of this report, which is a matter of public record.
OBJECTIVES, SCOPE, AND METHODOLOGY
The primary objectives of the audit were to: (1) determine the status of the LIMS project; (2) assess the information technology investment management process used for LIMS; (3) assess project management and other management controls; and (4) determine project costs.
Scope and Methodology
The audit was performed in accordance with the Government Auditing Standards and included tests and procedures necessary to accomplish the audit objectives. We conducted work at the FBI Laboratory Division in Quantico, Virginia; FBI Headquarters in Washington, D.C.; and JusticeTrax corporate headquarters in Mesa, Arizona.
We interviewed officials from the FBI and JusticeTrax. The FBI officials interviewed were from the Laboratory Division, Office of the Chief Information Officer, Office of General Counsel, Finance Division, and Criminal Justice Information Services. Additionally, we reviewed FBI documents on the LIMS project and budget, and prior GAO and OIG reports.
To determine the current status of the LIMS project, the Information Technology Investment Management processes used, and the extent of project management and other management controls, we interviewed FBI personnel and reviewed correspondence between the FBI and JusticeTrax. To determine LIMS project costs, we examined the contract budget, cost spreadsheets, and product invoices.
PRIOR REPORTS ON THE FBI'S INFORMATION TECHNOLOGY
Below is a listing of relevant reports concerning the FBI's information technology (IT) systems. These include reports issued by the Department of Justice Office of the Inspector General (OIG) and the Government Accountability Office (GAO).
OIG Reports on the FBI's IT
OIG reports issued over the past 15 years have highlighted issues concerning the FBI's utilization of IT, including its investigative systems. In 1990, the OIG issued The FBI's Automatic Data Processing General Controls, which found that:
- The FBI's phased implementation of its 10-year Long Range Automation Strategy, scheduled for completion in 1990, was severely behind schedule and may not be accomplished;
- The FBI's Information Resources Management program was fragmented and ineffective, and the FBI's Information Resources Management official did not have effective organization-wide authority;
- The FBI had not developed and implemented a data architecture; and
- The FBI's major mainframe investigative systems were labor intensive, complex, untimely, and non-user friendly, and few agents used them.
- defining and developing IT investment boards,
- following a disciplined process of tracking and overseeing each project's cost and schedule milestones over time,
- identifying existing IT systems and projects,
- identifying the business needs for each IT project, and
- using defined processes to select new IT project proposals.
In September 2003, the OIG issued The Federal Bureau of Investigation's Implementation of Information Technology Recommendation, which outlined the FBI's continued need to address the recommendations made by oversight organizations concerning its IT strategies. The report stated that although OIG audits found repeated deficiencies in the FBI's IT control environment and lack of compliance with information security requirements, the FBI leadership appeared to be committed to enhancing controls to ensure that recommendations were implemented in a consistent and timely manner. Additionally, the report noted that the FBI established a system to facilitate the tracking and implementation of OIG recommendations.
In May 2004, the OIG issued The FBI DNA Laboratory: A Review of Protocol and Practice Vulnerabilities. In this report the OIG findings focused on two general types of vulnerabilities that became apparent during the review: (1) protocol vulnerabilities and practice, and (2) operational vulnerabilities. As a result of the vulnerabilities, one of the 35 OIG recommendations was that the FBI Laboratory Division implement an information management system. The OIG noted that laboratory management had begun to lay the groundwork for the implementation of a system in 2002. Given the benefits that such a system would bring to evidence tracking and chain-of-custody documentation, the OIG recommended the successful implementation of an information management system as one of the laboratory's top administrative priorities.
In February 2006, the OIG issued The FBI's Pre-Acquisition Planning for and Controls over the Sentinel Case Management System. Sentinel is part of the FBI's IT modernization project to replace the FBI's antiquated case management system. The report noted the FBI has taken steps to address its past mistakes in IT investments and to adequately plan for the development of Sentinel.
External Reports on the FBI's IT
The GAO has issued several reports and related testimony that highlight deficiencies with the FBI's IT environment. In a review of the Department's Campaign Finance Task Force, the GAO reported in May 2000 that the FBI lacked an adequate information system that could manage and interrelate the evidence that had been gathered in relation to the Task Force's investigations. Also, as part of a government-wide assessment of federal agencies, the GAO reported in February 2002 that the FBI needed to fully establish the management foundation that was necessary to successfully develop, implement, and maintain an Enterprise Architecture.
In September 2003, the GAO issued Information Technology: FBI Needs an Enterprise Architecture to Guide Its Modernization Activities. This report reiterated the GAO's finding made in the May 2002 report on the Department's Campaign Finance Task Force that the FBI did not have an Enterprise Architecture, although it had begun efforts to develop one. Additionally, the GAO found that the FBI still did not have the processes in place to effectively develop, maintain, and implement an Enterprise Architecture.
In September 2004, the GAO issued Information Technology: Foundational Steps Being Taken to Make Needed FBI Systems Modernization Management Improvements. This report stated that although improvements were underway and more were planned, the FBI did not have an integrated plan for modernizing its IT systems. Each of the FBI's divisions and other organizational units that manage IT projects performed integrated planning for its respective IT projects. However, the plans did not provide a common, authoritative, and integrated view of how IT investments could help optimize mission performance, and they did not consistently contain the elements expected to be found in effective systems modernization plans. The GAO recommended that the FBI limit its near-term investments in IT systems until it developed an integrated systems and modernization plan and effective policies and procedures for systems acquisition and investment management. Additionally, the GAO recommended that the FBI's Chief Information Officer (CIO) be provided with the responsibility and authority to effectively manage information technology FBI-wide.
In September 2005, the GAO issued Information Technology: FBI Is Taking Steps to Develop an Enterprise Architecture, but Much Remains to be Accomplished. This report stated that the FBI managed its Enterprise Architecture program in accordance with many best practices, but other such practices had yet to be adopted. These best practices, which are described in GAO's Enterprise Architecture management maturity framework, are those necessary for an organization to have an effective architecture program. In addition, the FBI relied heavily on contractor support to develop its Enterprise Architecture. However, it did not employ effective contract management controls in doing so.
In September 2005, the GAO issued testimony entitled, Information Technology: FBI is Building Management Capabilities Essential to Successful System Deployments, but Challenges Remain. This testimony stated that the FBI had made important progress in establishing IT management controls and capabilities that GAO's research and experience show are key to exploiting technology to enable transformation. These included centralizing IT responsibility and authority under the CIO and establishing and beginning to implement management capabilities in the areas of enterprise architecture, IT investment management, systems development and acquisition life cycle management, and IT human capital. In addition:
- The FBI had developed an initial version of its enterprise architecture and is managing its architecture activities in accordance with many key practices, but it had yet to adopt others (such as ensuring that the program office has staff with appropriate architecture expertise).
- The FBI was in the process of defining and implementing investment management policies and procedures. For example, it was performing assessments of existing systems to determine if any could be better used, replaced, outsourced, or retired, but these assessments had yet to be completed.
- The FBI had issued an agency-wide standard life cycle management directive, but it had yet to fully implement this directive on all projects. Also, certain key practices, such as acquisition management, required further development.
- The FBI had taken various steps to bolster its IT workforce, but it had yet to create an integrated plan based on a comprehensive analysis of existing and needed knowledge, skills, and abilities. According to the CIO, the FBI intended to hire a contractor develop an implementation plan. The CIO also intended to establish a management structure to carry out the plan.
- The challenge for the FBI is to build on these foundational capabilities and implement them effectively on the program and project investments it has underway and planned.
THE FBI'S LIFE CYCLE MANAGEMENT DIRECTIVE
According to the FBI's Chief Information Officer (CIO), since the inception of the Life Cycle Management Directive (LCMD), all FBI information technology (IT) programs and projects have been reviewed and managed according to the processes described in the LCMD. New IT programs and projects have been managed according to this IT Systems Life Cycle from inception and will be managed through retirement or replacement, while existing IT programs and projects are reviewed and placed within an appropriate IT Systems Life Cycle phase according to their maturity and other factors.
Systems Life Cycle Phases
The LCMD has established nine phases that occur during the development, implementation, and retirement of IT projects. During these phases, specific requirements must be met for the project to obtain the necessary FBI management approvals to proceed to the next phase. The approvals occur through seven control gates, where management boards meet to discuss and approve or disapprove a project's progression to future phases of development, implementation, or retirement. The nine phases of development, implementation, and retirement are as follows:
Concept Exploration — Identifies the mission need, develops and evaluates alternate solutions, and develops the business plan.
Requirements Development — Defines the operational, technical and test requirements, and initiates project planning.
Acquisition Planning — Allocates the requirements among the development segments, researches and applies lessons learned from previous projects, identifies potential product and service providers, and secures funding.
Source Selection — Solicits and evaluates proposals and selects the product and service providers.
Design — Creates detailed designs for system components, products, and interfaces and initiates test planning.
Development and Test — Produces and tests all system components, assembles and tests all products, and plans for system testing.
Implementation and Integration — Executes functional, interface, system, and integration testing, provides user training, and accepts and transitions the product to operations.
Operations and Maintenance — Maintains and supports the product, and manages and implements necessary modifications.
Disposal — Shuts down the system operations and arranges for the orderly disposition of system assets.
Control Gate Reviews
The seven control gate reviews provide management control and direction, decision-making, coordination, confirmation of successful performance of activities, and determination of a system's readiness to proceed to the next life cycle phase. Decisions made at each control gate review dictate the next step for the IT program or project and may include: allowing an IT program or project to proceed to the next segment or phase, directing rework before proceeding to the next segment or phase, or terminating the IT program or project. The FBI's Investment Project Review Board (IMPRB) — comprised of 12 representatives from each FBI division at the Assistant Director level and 4 representatives from the Office of the Chief Information Office, including the CIO — is responsible for approving an IT project's passing through each control gate. The seven control gate reviews that represent the approval of an IT project are as follows:
Gate 1 — System Concept Review approves the recommended system concept of operations.
Gate 2 — Acquisition Plan Review approves the Systems Specification and Interface Control documents and the approach and resources required to acquire the system as defined in the Acquisition Plan.
Gate 3 — Final Design Review approves the build-to and code-to documentation and associated draft verification procedures, ensures that the design presented can be produced and that when built is expected to meet its design-to specification at verification.
Gate 4 — Deployment Readiness Review approves the readiness of the system for deployment in the operational environment.
Gate 5 — System Test Readiness Review verifies readiness to perform official system-wide data gathering verification testing for either qualification or acceptance.
Gate 6 — Operational Acceptance Review approves overall system and product validation by obtaining customer acceptance and determining whether the Operations & Maintenance organization agrees to, and has the ability to, support continuous operations of the system.
Gate 7 — Disposal Review authorizes termination of the Operations and Maintenance Phase and disposes of system resources.
At each control gate, executive-level reviews determine system readiness to proceed to the next phase of the IT systems life cycle. Evidence of readiness is presented and discussed at each control gate review in the form of deliverables, checklists, and documented decisions. Regardless of the development model used for a particular program or project, all control gate reviews should be performed unless an agreement is made to skip or combine reviews. Depending upon the development model employed, programs or projects may pass through the control gates more than once.
The control gate reviews also provide executive-level controls to ensure that IT projects are adequately supported and reviewed before a project receives additional funding. Five executive-level review boards serve as the decision authority for the control gate reviews:
- Investment Management Project Review Board (IMPRB) leads the System Concept Review and the Acquisition Plan Review and ensures all IT acquisitions are aligned and comply with FBI policies, strategic plans, and investment management requirements.
- Technical Review Board leads the Final Design Review and ensures IT systems comply with technical requirements and meet FBI needs.
- Change Management Board leads the Deployment Readiness Review, System Test Readiness Review, Operational Acceptance Review, and the Disposal Review, and controls and manages developmental and operational efforts that change the FBI's operational IT environment.
- Enterprise Architecture Board ensures IT systems comply with Enterprise Architecture requirements.
- IT Policy Review Board establishes, coordinates, maintains and oversees implementation of IT policies.
LCMD Project-Level Reviews
Project-level reviews help determine a project's readiness to proceed to the next phase of the project life cycle. Each project-level review provides information to the executive-level control gates as data is developed and milestones are completed. They include the following:
- Mission Needs Review is a technical progress review that approves the set of mission goals that will be satisfied throughout the project.
- System Specification Review is a technical progress review to approve the System Specification and External Interface Control Documents. The review is the decision point to proceed with the development of an Acquisition Plan, the allocation of system requirements to segment specifications, and the development of Project Plans that will execute the acquisition.
OFFICE OF THE INSPECTOR GENERAL ANALYSIS AND SUMMARY OF ACTIONS NECESSARY TO CLOSE REPORT
The OIG provided a draft of this audit report to the FBI on April 28, 2006, for its review and comment. The FBI provided a written response, dated May 31, 2006, which we included as
Appendix 5 of this final report. The FBI concurred with the three recommendations in the audit report and also provided comments regarding three general issues in the report. Our analysis of the FBI's response follows.
FBI's General Comments
1. In its response, the FBI states that the purpose of LIMS was to enhance the processes and procedures currently in place in the laboratory by improving efficiencies and automation. Although we agree with this statement, it does not reflect the full impact that the implementation of the LIMS project would have had on the laboratory. As noted in the report, laboratory officials stated that the paper-based system currently being used by the laboratory is very limited in what information it can provide to enhance the management of evidence as it passes through the laboratory. LIMS would have allowed the FBI to electronically trace evidence as it passes through the lab and provide workflow data needed to better manage the laboratory.
The FBI's response also states that our report implies the laboratory's operations are not effective or adequate and points out that the FBI's laboratory is one of the largest and most comprehensive forensic laboratories in the world. Our audit report recognizes the significant amount of work performed at the FBI laboratory and does not question the work that is performed on evidence within the laboratory. However, the size and scope of the laboratory do not demonstrate the effectiveness or adequacy of the management of the evidence held within the laboratory. Our audit concludes that the management of evidence as it passes through the laboratory would have been significantly enhanced had a laboratory information management system been fully and effectively implemented.
The FBI's response also states that improvements to the laboratory's information management system are required, rather than the establishment of a new system. The FBI is currently utilizing a Microsoft Access database to document when a piece of evidence is received, when a test has been completed on the evidence, and when it is released from the laboratory. However as pointed out in the report, the release of a piece of evidence is not always documented adequately. As a result, laboratory management cannot determine what evidence is contained within the laboratory at any given point in time. Additionally, the database system utilized by the laboratory also cannot reasonably pinpoint where a piece of evidence is at any given point in time. While we agree that the laboratory has an information management system in place, the system has limited functionality. This limited functionality led the FBI to enter into the LIMS contact to acquire a more effective system. We believe that the FBI either needs to make significant improvements to the existing information management system or acquire a new system that provides laboratory management the ability to more effectively manage laboratory operations.
2. The FBI response states that our report implies the FBI had singular control over the system development and process, although the report acknowledges that the vendor also bears some responsibility for the project's difficulties. As the response suggests, our audit found that both the FBI and the contractor were responsible for the outcome of the LIMS project. However, the FBI was solely responsible for establishing the system requirements and ensuring that the contractor met those requirements. We noted in the report that the FBI has recently made significant strides in the development and management of information technology projects. However, the LIMS project did not benefit from these new management practices.
The FBI's response also notes that the contract termination settlement is far less than the full contract amount. We agree. However, the FBI incurred costs in addition to the settlement amount, such as the personnel involved in the development, management, and termination of the project. More important is the fact that despite having worked on the development of an information management system since 1998 and reprogramming funds from other Laboratory Division programs in order to pay for the project, the FBI's laboratory remains without a modern system.
3. The FBI requests that the vendor's name and specific dollar amounts of the project be redacted from the report to protect the future business opportunities of the vendor and future requests for proposal issued by the FBI on similar projects. After careful review and consideration of the FBI's request, we have decided to not redact the information for the following reasons: (1) the contractor's name and the dollar amounts paid to JusticeTrax are public information;
(2) the public has a right to know the name of the system contractor; and (3) our report is clear that both the FBI and JusticeTrax were responsible for contributing to LIMS' failed implementation. For example, we fault the FBI for not adequately documenting system security requirements and for its overall poor project management, and we fault JusticeTrax for not meeting the FBI's security requirements once they were established and for not providing the web-enablement capabilities for the LIMS software as required by the contract. Therefore, we believe that our report is accurate as to which party was responsible for the various system implementation failures. Finally, because the name of the contractor and the dollar amounts paid to it are public information, we do not agree that disclosing the information in this report is inappropriate or will have an effect on future FBI request for proposals.