Print Email Download Reference This Send to Kindle Reddit This
submit to reddit

Critical Care Response Teams: An Evaluability Assessment

BACKGROUND

The Critical Care Secretariat (CCS) was established in June 2004 by the Government of Ontario. Its key function is the effective prioritization of the recommendations outlined in the Ontario Critical Care Steering Committee Final Report into a highly integrated 7-part program. Together, these programs enhance accountability, empowerment and coordination at the hospital, LHIN (Local Health Integration Networks) and Provincial levels to support new and ongoing improvements to critical care Access, Quality and System Integration.

Transforming critical care service delivery is integral to keeping Ontarians healthy, ensuring better access and reduced wait times, and providing an environment within which all healthcare practitioners can deliver their best.

Based on the information put forth from the CCS, this initiative claims that it will: establish a province-wide system for critical care performance measurement and quality improvement. CCS will also improve on the degree of standardization of best practices and critical care training across Ontario. Furthermore, this initiative increases efficient and effective delivery of critical care service delivery through operational management processes and surge capacity response planned, developed, and coordinated at the LHIN level.

As a provincial strategy that enables and encourages all Ontario hospitals providing critical care services to be involved, the Critical Care Strategy continues to be thoroughly informed by and in continuous communication with clinical and administrative leaders from across Ontario. These include the Critical Care Expert Advisory Panel, Critical Care LHIN Leaders Provincial Table, and the Trauma Expert Advisory Panel. The Secretariat's key mandate is to work closely with the field to implement programs that improve access, quality and integration of critical care services to meet the needs of critically ill patients.

One of the key functions includes, among others, the implementation of the strategy's 7 funded programs. The programs, their mandates and key deliverables are described below:

Program

Mandate

Key Deliverables

Critical Care Information System

Enable evidence-based decision-making to support system-wide capacity planning and targeted performance improvement initiatives through data collection, analysis and reporting.

Pilot CCIS in 7 hospitals

Rollout CCIS to additional hospitals

Design and build the provincial Critical Care Information System (CCIS)

Critical Care Response Teams

Bring critical care expertise out of the ICU to patients throughout the hospital 24/7 to improve patient outcomes and efficiency of resource utilization.

Demonstration Project to test the CCRT model in 4 paediatric sites

Demonstration Project to test alternate CCRT models in smaller hospitals

Launch Intensivist-led CCRTs in larger hospitals that meet criteria

System-Level Training Initiatives

Expand the skill and capacity of existing Critical Care Health Professionals, enhancing pre-and post-ICU care, and supporting surge capacity response plans.

Provide ACES training to over 100 physicians in community hospitals

Conduct provincial training needs assessment with CC LHIN leaders

Provide training to over 350 RNs and RTs in support of CCRT Expansion

Program

Mandate

Key Deliverables

Performance Improvement Collaborative

Employ innovative approaches to achieve quality benchmarks defined at the hospital, LHIN and provincial level, and cultivate a culture of ongoing accountability and performance improvement in critical care services delivery.

Conduct an annual online hospital application process for Coaching Teams

Deploy Performance Improvement Coaching Teams into 40+ hospital sites

Establish Performance Improvement Coaching Teams to address priorities

Ethical Issues of Access

Establish specific/actionable and medically relevant admission, discharge and triage (ADT) policies supporting Ontario's critical care healthcare providers and patients.

Establish committee of clinician advisors to prepare a draft of ADT policies

Submit Green Paper and ADT policies draft to the Blue Ribbon Panel for the development of formal recommendations to the Government of Ontario

Research initiatives to contribute towards the development of a Green Paper

Health Human Resource Investments

Address shortages in key health human resources. Establish, and support the achievement of provincially recognized standards in critical care nurse training and education programs.

Establish provincial standards in critical care nurse training and core competencies

Develop college and hospital incentives to upgrade programs to meet standards

Promote achievement of Standards by nurses through flexible, accessible programs

Funding to support critical care nurse training, and to raise the number number of intensivists trained from 8 to 18 annually

Program

Mandate

Key Deliverables

Surge Planning and Capacity Management

Increase the number of critical care to address key pressure points, and develop alternatives for medically stable, chronically ventilated patients.

$10.25M to fund 14 ICU beds and 4 step-down beds in 8 hospitals.

Implement the recommendations of the Final Report of the Chronic Ventilation Task Group for managing chronic care patients

$19.50M to fund 27 ICU beds and 6 Chronic-Vent beds in 9 hospitals.

Rapid Response Teams

The program under evaluation is the Critical Care Response Teams (CCRT), one of the 7 funded programs of the Critical Care Secretariat.

The Institute for Healthcare Improvement has recommended that hospitals implement rapid response teams as 1 of 6 strategies to reduce preventable in-hospital deaths. In response, significant financial and personnel resources have been invested in implementing CCRTs, despite limited published data supporting their effectiveness (Iyengar).

Critical care response teams have been described in the medical literature since 1995 and promote a hospital-wide approach to preventive strategies aimed at patients at risk for unexpected death, cardiac arrest and unplanned intensive care unit (ICU) admissions (Upadhye). Teams are typically composed of intensive care physicians, nurses and respiratory therapists who are available 24/7 to bring critical care expertise to the bedside of patients outside of intensive care units and in the hospital ward. The purpose of CCRTs is to improve the early care of critically ill or deteriorating patients in order to improve outcomes and potentially reduce the rate of, or shorten the duration of, ICU admissions.

The mandate of the critical care response teams, as seen in the table above, is to bring critical care expertise out of the ICU to patient throughout the hospital 24/7 to improve patient outcomes and efficiency of resource allocation. The key deliverables that the Critical Care Secretariat describes include: Launch Intensivist led CCRTs in larger hospitals that meet criteria, a pilot project to test the CCRT model in 4 paediatric sites and a second pilot project to test alternate CCRT models in smaller hospitals.

EVALUABILITY ASSESSMENT

The framework for conducting the evaluability assessment revolved around addressing three critical issues as identified by Dunn in the Impact Assessment Primer Series report. These are:

1. Is it plausible to expect outcomes at this stage? Do stakeholders share a clear understanding of how the program operates and are there logical links from program activities to intended outcomes?

2. Is it feasible to measure the outcomes? Is it possible to measure the intended outcomes, given the resources available for the evaluation and the program implementation strategy?

3. Would an evaluation be useful? Are there specific needs that the assessment will satisfy and can it be designed to meet those needs?

EVALUABILITY ASSESSMENT FINDINGS AND RECOMMENDATIONS

1. Is it plausible to expect outcomes at this stage? Do stakeholders share a clear understanding of how the program operates and are there logical links from program activities to intended outcomes?

The Program’s key deliverables have been achieved and disseminated. Rapid response teams have been implemented in 26 Ontario hospitals and have been running for at least 12 months at most sites.

One of the desired outcomes is to improve patient outcomes. There have been mixed perceptions on improving patient outcomes and clinical benefits obtained directly from the implementation on CCRTs. Several studies demonstrate an association between CCRT implementation and improved hospital outcomes [Bellomo, DeVita, Buist]. However; there are also various negative trials [Hillman, Kenward, Chan]. Despite the conflicting evidence, many institutions and health systems have continued to fund CCRT implementations due to perceived benefits extending beyond those evaluated in the published research [Lokajner, Jones].

In terms of improved resource allocation as a result of CCRT implementation, access to health care expenditures by department and hospital would have to be made available to the evaluators. A thorough cost benefit analysis will be necessary in order for a clear understanding of operation and effect on this particular intended outcome.

Various operational issues have been identified pointing to the shortcomings in some CCRTs when responding to calls. The Institute for healthcare improvement claims that upon a review of the literature:

“..our experience reveals that there are three main systemic issues contribute to the problem:  1) Failures in planning (includes assessments, treatments, goals) 2) Failure to communicate (patient to staff, staff to staff, staff to physician, etc.) 3) Failure to recognize deteriorating patient condition. These fundamental problems can often lead to a failure to rescue.”

The establishment of CCRTs stands to impact this if identified in a timely fashion therefore making their unnecessary deaths preventable.

Program Logic Model

INPUTS

What we invest

Create CCRTs in 26 Ontario hospitals, as part of the Critical Care Strategy. The Ontario government invested $29.4 million into this initiative. The Minister stated that the creation of CCRTs in Ontario "demonstrates the government's willingness and commitment to ensuring that not only Ontarians receive the critical care they deserve, but that they receive it in the most timely manner possible."

Developing coaching teams and making these teams available at no cost to healthcare providers, the MOHLTC is enabling expert knowledge to be transferred throughout the system in a collaborative and coordinated manner.

OUTPUTS

What we do (Activities)

Nurses, physician and respiratory therapists collaborative education programs and coaching teams.

Demonstration Project to test the CCRT model in 4 paediatric sites

Demonstration Project to test alternate CCRT models in smaller hospitals

Launch Intensivist-led CCRTs in larger hospitals that meet criteria

Who we reach (Participation)

Patients on the general hospital ward before becoming critically ill.

OUTCOMES

Short

Bring critical care expertise out of the ICU to patients throughout the hospital 24/7 to improve patient outcomes and efficiency of resource utilization.

Medium

Mediate the procedural issues by means of CCRTs that cause “failure to rescue” in turn improving patient outcomes.

Establish a province-wide system for critical care performance measurement and quality improvement, improve on the degree of standardization of best practices and critical care training across Ontario, increase efficient and effective delivery critical care service delivery via operational management processes, and surge capacity response planning coordinated at the LHIN level and, flatten the demand curve, mitigating, as far as possible, the need for increased capacity investments and provide clarity where investment is required.

Long-term

Better access to critical care, reduced wait times, improved patient outcomes and reduce health care costs.

2. Is it feasible to measure the outcomes? Is it possible to measure the intended outcomes, given the resources available for the evaluation and the program implementation strategy?

It was found that the desired objective and outcomes lack specificity and are not measurable unless the definitions are refined. “Improved patient outcomes” is a robust and unfocused outcome to strive for in its mandate. I propose that this be measured in terms of 1) numbers of code blue or cardiac arrests, 2) admissions to the ICU 3) re-admissions to the ICU within 48 hours of discharge, 4) post operative complications, and 5) hospital deaths. These measurements, when taken together, help to better inform evaluators on patient outcome and its improvement or deterioration.

It may be possible to measure as well certain process measures. These measures will inform us on implemented changes and may provide an indication on outcome measures. Numbers of calls to CCRT teams, time to respond, numbers of interventions done by CCRT teams which are not standard practice on the wards are some suggested measure which could be taken and may tell us whether changes are actually taking place and perhaps leading to improvement.

Finally, the ability to collect data is it anticipated will vary greatly across sites as 26 Ontario hospitals vary greatly in level of care provided, influence by the city, LHIN, and dominant medical education institution.

3. Would an evaluation be useful? Are there specific needs that the assessment will satisfy and can it be designed to meet those needs?

Given the stage of the program at hand, there is very little opportunity for program modification. Formative evaluation is therefore not appropriate. The focus at this stage should rather be one of demonstrating that the program generates sufficient impacts to merit the resources used to implement it. I recommend instead comparative summative information to be gathered in order for the Ministry of Health to ensure that the rapid response teams provide the results that the funds were intended to address within the population of Ontarians. At this stage of the program accountability and overall effectiveness and impact are the focus of the evaluation.

Evaluations provide valuable insight into how programs are operating, the extent to which they are serving their intended beneficiaries, their strengths and weaknesses, their cost-effectiveness, and potentially productive directions for the future. This enables judgments to be reached about how well policies and practices are working. This is influenced by explicit goals, objectives, and standards. It may allow for current operating standards to be changed.

There is variability between the various Ontario hospitals that have implemented CCRTs in terms of their ability to collect data to measure the intended outcomes. The evaluation will therefore be designed in such a way to make the data collection process as easy as possible which will be cost effective while remaining valuable and informative. The measurement of patient outcomes will allow conclusions to be drawn on CCRT effectiveness and in turn their cost effectiveness.

EVALUATION DESIGN

The goal-oriented approach uses program-specific goals and objectives as the criteria for determining success in an objective manner. The evaluator tries to measure the extent to which goals are attained and specific objectives are accomplished. This must be done by clearly clarifying the links between activities and outcomes, by stipulating the desired results and the objectives in clearly stated, measurable terms. It allows for logical connections to be made between activities, outcomes and the procedures for measuring results.

Ideally an experimental design using a true control group would be used in order to allow strong conclusions to be drawn about any positive or negative effects detected during the evaluation as a result of the rapid response teams. This type of design, however, will not be possible due to rapid response teams already having been implemented across Ontario and running for well over a year. A non-equivalent control group would be the next best design to consider. Some of the key characteristics to control for in Ontario hospitals should include level of care provided, city, LHIN, and dominant medical education institution. Because most of the larger hospitals have already implemented CCRTs and there is great variability between the smaller hospitals, this will also not be possible at this stage.

An alternative approach to evaluation of community studies is to use a quasi-experimental design in which program outcomes are examined over time in the intervention unit (Cook). If data can be collected at several points in time before and after introduction of the intervention, a trend analysis can be conducted for the outcomes of interest. Using as large a number of measurement points as possible will allow for an effective regression methodology to analyze the data. Piecewise regression can identify both sudden changes due to an intervention, as well as more gradual changes over time (Gilings).

Therefore I recommend a single group time series design with measurements dating back 18 months prior to implementation and 18 months after implementation. As the program has already been implemented, some of the data to be collected will be done via retrospective chart and record review. Consistency in the definitions of outcomes will be of utmost importance.

TIME (Months)

-18

-12

-6

Baseline

+6

+12

+18

Hospital

O

O

O

X

O

O

O

Methods

Patient outcomes will be measured in terms of 1) numbers of code blue or cardiac arrests, 2) admissions to the ICU 3) re-admissions to the ICU within 48 hours of discharge, 4) post operative complications, and 5) hospital deaths. Measurements will be taken from the monthly reports produced by the Critical Care Information System (CCIS) (MOHLTC). CCIS includes scorecards and performance measurement and management tools that support decision-making at local and system levels. These reports improve the efficiency of critical care resource allocations and will serve as the primary data base for the evaluation of CCRTs. Obtaining these measures of patient outcomes from hospital charts directly would prove to be a time consuming and expensive endeavour. The only cost which our client will encounter will be that of obtaining the data from these pre-existing sources and the services of a statistician for conducting the piecewise regression analysis.

To ensure cost effectiveness this data shall be collected from a representative cohort as evaluating all 26 hospitals will be time consuming and expensive. The cohort should include at least one pediatric site, and one site both large and small from each LHIH. As the data is available in pre-existing databases and CCRTs have been running in most hospitals for over 8 months, an evaluation of CCRT effectiveness in improving patient outcomes shall be possible in a very short time period.

It is recommended that the evaluation measures later be incorporate as part of the routine followed by the CCRT team during a call. This would minimize the resources required for the task and enhance timeliness of detection while at the same time yield useful information regarding outcome and process measures.

CONCLUSION

In conclusion, transforming critical care service delivery is integral to keeping Ontarians healthy and ensuring better access and reduced wait times. This provides an environment within which all healthcare practitioners can deliver their best. Critical care response teams aim to improve patient outcomes by bringing critical care expertise out of the intensive care unit. Evaluations provide valuable insight into how programs are operating, the extent to which they are serving their intended beneficiaries, their strengths and weaknesses, their cost-effectiveness, and potentially productive directions for the future. The evaluation proposed in this evaluability assessment report, is a time efficient and cost effective manner of obtaining valuable information as to whether critical care response teams are actually saving lives and improving outcomes.

Print Email Download Reference This Send to Kindle Reddit This

Share This Essay

To share this essay on Reddit, Facebook, Twitter, or Google+ just click on the buttons below:

Request Removal

If you are the original writer of this essay and no longer wish to have the essay published on the UK Essays website then please click on the link below to request removal:

Request the removal of this essay.


More from UK Essays

Doing your resits? We can help!