Training Evaluation for Nursing Quality Development

 SECTION 3- PROCEDURES

This section is aimed at describing the research methodology. It is divided into three parts which include; 1) explain the context of the case study; 2) design of action research process; and 3) describe the procedure used in designing the model of evaluation procedures.

This project will create a model for training evaluation for the Nursing Quality Development Unit, The General Hospital, Thailand as a case study. The Case Study is a research approach focused on understanding the dynamics that exist within a single setting (Eisenhardt, 1989). Fundamentally, the Case Study provides a thorough examination of an individual case to allow one to correctly gain insight of the occurrence from the thorough examination of a single case (Becker, 1970). Fidel (1984) asserted that the Case Study endeavours, on the one hand, to attain a detailed understanding of the phenomenon being studied but to also simultaneously develop more general conjectural statements as regards consistencies in the recognized phenomena.

The Context of Nursing Quality Development Unit, The General Hospital

The role of the Nursing Quality Development Unit, General Hospital includes the five dimensions of innovative practice in nursing: quality improvement, research and continuing professional education, risk management, monitoring, evaluation, and consultation.

Team members are three clinical supervisors and five clinical educators who are professionals in their fields and relish the prospect of interacting and serving as a resource to the whole organization. The task of the clinical supervisor includes an extensive range of activities such as:

• Working with registered nurses to learn new skills, finding new research and creating advanced tactics to meet the needs of a tasking healthcare environment; and
• Creating advanced educational experiences.

Nursing Quality Development Unit team members have a common objective to:

• Ensure nursing services offers the public an all-inclusive, consistent, evidence-based approach to clinical practice
• Encourage professional development and education of the nursing staffs
• Assess and manage nursing quality and practice
• Assess and manage nursing-sensitive outcomes
• Review existing proof and design nursing performance advancement and research projects (Nursing Quality Development Unit 2012).

Methodology

Action research or formative research will be utilized in formulating the evaluation model for the continuing registered nurses’ training programs. This kind of research is proposed for designing a model that will manage instructional processes via the reflective procedure of unbiased problem solving (Soultana & Stamatina, 2013; Reigeluth & Frick, 1999).

The Action research method agrees that “themes” must take the place of hypotheses. An organization researches on how to introduce a specific information into the system (Checkland & Holwell, 1998), Also, action research is viewed as a more enabling experience for the researcher as individual intuitions and essential evaluations of complex circumstances can lead to; including an appreciation of the role which researchers themselves have played (Elliott 1991). The investigator worked at the hospital which is a case study of this project for eight years as a registered nurse and clinical educator who was involved in the matters of continuing professional education and evaluation.  Of equal significance, action research is used in health care organizations owing to the fact that the procedure is based on maintaining the focus on the `real world’, as opposed to the controlled environments. It recognizes the issues which social actors are faced with daily, and tries to challenge complex organizational practices (Kelly & Simpson, 2001).

Design of action research process

 

Figure 2:

Process

The Action research procedure involves five stages and they are as follows:

Stage 1: Analytical context and the existing evaluation model.

In this stage, the investigator will examine the context and the model that is presently been employed for evaluation purposes in the ongoing training at The Police General Hospital, Thailand. As Heritage (2013) states, reviewing different sources and decisions on necessary skills or concepts of domain is relevant before deciding and concluding on how the advancement will be introduced. The steps followed in this stage are:

Review of the document.

The reason for doing this is to provide an overall impression of the hypothetical background as well as the associated experimental data for this program context as supported by Lorenc et al. (2014). The official papers that will be put under review include:

The program evaluation theory and

The existing evaluation model

Interview: 

In this stage, participants include every staff member of the Nursing Quality Development Unit, The Police General Hospital partaking in the training program, and the human resource management division. This method utilizes the reflection on the framework and experience. It additionally analyses facts prior to settling on the type of development model for future use (Bond, 2013).

Stage 2: Designing the model

Following the content analysis during which the data will aid in changing the needed evaluation model based on standards, criteria, and indicators. This stage encompasses:

Designing of an evaluation model: 

The training evaluation model will be developed by making use of the analysed data from the first stage.

Expert review:

Asking professionals in program evaluation to evaluate the first model, authenticate it, and provide feedbacks. This professional utilizes the formative evaluation to assess the training. Hence, the team can add a content professional, instructors or even a designer (Triantafillou et al., 2003).

Stage 3: Model testing

The first model draft will go through testing at the Nursing Quality Development Unit, where the supervisor will be tasked with evaluating the model and providing feedbacks.

Stage 4: Revised Model.

The new information gained from the reflection and from the feedback in the last step will be useful in terms of refining the model.

Stage 5: Validated and adjusted Model

Modules or factors will be authenticated and modified until the model is satisfactory and illustrative of the real model. There will be a professional appraisal conducted in a similar manner as in stage 2 and Personalized evaluation will likewise be conducted in the same manner as it was carried out in stage 3

SECTION 4- FINDINGS

The results for each stage of action research methodology are detailed in this section. The results given include the training evaluation model, logic model, program training logic model, standard and criteria for Nursing Quality Development Unit, and The General Hospital used to conduct the training programs for registered nurse.

Results (Phase 1: Some preliminary information)

Document review

When it comes to developing the training program evaluation model, there seems to be need for more preliminary information for both the existing model and the theory of training program evaluation. Alkin, Christie, and Vo (2012) claimed that evaluation is a process of gathering information. Some of the past program evaluation are inadequate since they fail to cover the extensive array of activities needing information (Alkin, Christie, & Vo, 2012). The main aim of taking such a step is to give an overview of the academic background of pertinent collected data so that the model of training program evaluation system can be correctly built.

The program evaluation theory

The purpose of theory in program evaluation is a significant but overlooked area in evaluation practice and theory. Program theory has a number of import function that can advance our capability to simplify from specific evaluations, add to social science theory, and reach agreement in evaluation planning (Bickman, 1987). In the current step, the theory of evaluation in the training program will be reviews to finding the suitable standard for model development. According to Phillips (1997), every case of evaluation and measurement is unique, the procedure and concept that works for one case may or may not work in another case. Techniques can be borrowed but the results cannot be borrowed (Phillips, 1997). Evaluation is conducted for numerous reasons. Generally, they can be classified into two groups, to advance training process and to determine whether or not to keep going (Phillips, 1997). Also, the exact reason include;  to decide whether a program meets the aim, to identify the strengths and weaknesses, to decide cost, benefit, and to collate data to help in making future program decisions (Stufflebeam & Coryn, 2014).  The theory of program evaluation is validating the model of evaluation’s ability to aid the readers in comprehending how and why a program works or fails to work. Weiss (1997) claimed that being aware of just the result, regardless of whether we know them with absolute certainty, does not give us sufficient information to inform program improvement or policy revision. Thus program evaluation has to get theory to be a framework and carry out the evaluation methodically.  Rogers, Petrosino, Huebner, & Hacsi (2000) stated that evaluation will offer considerably more reliable information when they address mechanisms that utilize theory-based evaluation. Moreover, Weiss (1997) claimed that evaluation is not a simple process and that we need credible theories as a direction from the basis.

The existing training evaluation model

Nursing Quality Development Unit has been conducting training for registered nurses since 2002. The two key reasons of these trainings are to meet the quality assurance standard. It is extensively believed that quality assurance is nowadays of foremost importance to providers of goods and services in the society particularly in the health sector (Strauss, 1978). Quality can be described as the degree of similarity between the purpose of education and the actually approved care as the result of education (Strauss, 1978). Also, nurses require a particular kind of extra training to assist them with the needed knowledge and skills which will empower them to ascertain and respond to the needs of patients (Parle, Maguire, & Heaven, 1997). Hence, numerous training programs have been carried out by the Nursing Quality Development Unit for registered nurses as regards the maintenance of nurse’s quality and in order to meet the standard of QA. Another reason is to help nurses to meet the education regulation of the Nursing and Midwifery Council which requires 50 hours of continuing nursing education for renewal of licensing every five years (Thailand Nursing and Midwifery, 2014). However, up till now, evaluations of these training programs are restricted to the end of training surveys and are usually fixated on recognizing the various problems of the programs. Stufflebeam and Coryn (2010) expounded on the fact that evaluation can offer complete and comprehensive information that measures the evaluation of the training program though the program evaluation needs to be consistent and standardized. Moreover, regular training program design have not adhered to logical steps in their design and development, for instance, training programs are focused on the interests and the need of individuals and not to meet the needs of the organization. Thus, an evaluation should be planned as a program design (Morrison et al., 2010).

 

Interview

In a couple of evaluations, the program theory has been extensively designed but the evaluator is based on a review of research literature on identical programs or pertinent causal mechanisms, through deliberations with key informants, through a review of program documentation or through observation of the program itself (Lipsey & Pollard, 1989). The investigator interviewed the supervisors of Nursing Quality Development Unit through telephone interviews (Skype) and used semi-structured questions, the summary of outcomes are as follows;

Lack of evaluation skills.

The evaluated toolkit and training programs are a relevant approach to developing nurses’ skills and knowledge. Ironically, the basic skills of evaluation are conduct evaluation; discovering and translating information, communicating with trainee are the same as the main skills of a decent evaluator (Posavac, 2015).  Every staff of Nursing Quality Development Unit are Registered Nurses, they lack basic evaluation knowledge and skills in evaluation and assessment.

Lack of support

This is also a problematic area and a tough one to handle. This is usually contingent upon the organizational culture (Hofstede, Neuijen, Ohayv, & Sanders, 1990). The valuation philosophy of hospital still needs to better the behaviour of policy makers and key management members who do not place importance on training evaluation. A superior evidence base on which to strengthen performance measurement policy is necessary, and new ideas should undergo thorough evaluation (Smith, 2009). This might help to proffer a few insights for points to include in any ‘training evaluation awareness’ campaign that the organization plans to run.

Not planning ahead and no appropriate tools.

The successful evaluations merely need painstaking planning, the aptitude to ask decent questions, and a rudimentary grasp of how to locate relevant answers (Guskey, 2002). For valuable results to make considerate, responsible decisions as regards expert development processes and effects, the facility needs to make efficient plans and tools to provide relevant information that they can use (Bushnell, 1990).

In these conditions, the training program evaluation carried out by Nursing Quality Development Unit requires the interactive form of evaluation. Evidence from the evaluation may determine future development of plans for closing the gap of need between hospital, nurses and evaluator from quality assurance institute. Furthermore, the suggestions from evaluation will additionally inform the organization for policymaking on the enhancement or adapt program, validate accountability, and rationalize funding. The results of the evaluation will likewise furnish the hospital, as program owners, with information for the program training development.

Results (Phase 2: Design a model and validate the model)

Designing evaluation model: 

The aim of this phase is to develop a training evaluation model that can guide evaluations of an extensive range of clinical and continuing professional development. The project’s ultimate aim is to assist the supervisors of the Nursing Quality Development Unit to accomplish more regularity and control over the hundreds of continual registered nurses’ training program for which it gives funding, and simultaneously, raise the ability of other training programs to carry out their own training evaluations.

After appraising the program evaluation theory as well as the main features of the current training evaluation model, this phase will clarify how the model was improved by the training program evaluation and then appraised by professionals. Evaluating training necessitates a methodical and thorough approach to decide whether the training is effective. This is a step-by-step debate of each component of the model. It involves modest valuations of participant satisfaction; evaluations of skills, knowledge, and attitudes learned; more difficult and challenging evaluations of behavioural and organizational modifications that may be attributable to the training experience; and the constantly disregarded job of objectively evaluating instructors  (Freeth, Hammick, Reeves, Koppel, & Barr, 2008).

Selected training evaluation model

There is an unending argument in the field of evaluation as regards which model is more ideal to enable the methods involved. Eseryel (2002) expounded the point that diverse approaches to evaluation of training deliberated in this paper show that the undertakings involved in evaluation of training are difficult and not regularly well defined. Since evaluation undertakings in training situations involve numerous objectives related to multiple levels, evaluation should perchance be seen as a combined activity between training designers, training managers, trainers, and probably many more.

The investigator chose  Phillips’ Evaluation Model  (1983) for creating the training evaluation model because Phillips’ Evaluation Model  facilitates decision makers to choose, from numerous options, the package that will best improve the general efficiency of a training program (Liong, 2005). According to Bushnell (1990), the Phillips’ Evaluation Model as a system-based model can easily discover if training programs are realizing their true objectives. It also allows them to distinguish the kinds of modifications they ought to effect in order to advance course design, content, and delivery. Perhaps most essentially, it notifies them on whether trainees really gained the necessary knowledge and skills (Bushnell, 1990). The Phillips’ Evaluation Model recognizes seven steps of evaluation which are defined in more details as follows:

Step 1 Collect pre-program data on performance and/or skill levels

Step2 Collect post-program data on performance and/or skill levels

Step 3 Isolate the effects of training from other positive and negative performance

           Influencers

Step 4 Convert the data into monetary value

Step 5 Compute the rates of delivering the training program

Step 6 Calculate the ROI (return on investment)

Step 7 Pinpoint and enumerate the intangible benefits.

Customizing and expanding on Phillips’ Evaluation Model 

Figure 3: Training evaluation Model (First Draft)

Step 1: Conduct needs analysis

A needs analysis for training is the initial step in designing a training and evaluation program.  A documented needs assessment also aids the evaluator in assessing the reason for the training program (Arthur Jr, Bennett Jr, Edens, & Bell, 2003).  The needs assessment gathers information from nurses on what training is required and the best approach to have it delivered.  A need assessment can also be used to discover whether a reputable training meets the needs of the hospital.  A training needs assessment assesses preferred or actual behaviour and includes such methods as surveys, interviews, observations, and document reviews (Gupta, 2011).

Step 2: Design Training Plan

Designing a training plan is the second important step in creating a training program.  The training plan can be viewed as a management plan for multiple different phases of training goals, management/administration, and evaluation (Thomas, 2003).  The training plan has to be included in the evaluation model as this plan should be accessible to the evaluator for a review.  In a newly commenced training, an evaluation plan ought to be developed as part of this phase (Garavan & O′ Cinneide, 1994). The point of origin for creating any training is to ascertain, articulate, and document the objectives of the organisation.  This aspect of the plan uses the outcomes of the needs assessment phase (Laird, Holton, & Naquin, 2003).   This aspect of the plan includes instituting and projecting the budgets of the training and resources needed to deliver and oversee the training.

Step 3: Develop and test the training syllabus

Creating a training syllabus takes time, skill, and dedication (Fink, 2013).  It is imperative that the aim of the training be openly acknowledged so that the training goals can be refined, explicit curriculum materials developed, and teaching methods and media designed (Romiszowski, 2016).  The training syllabus development phase joins together all the information developed in the first two phases—needs assessment and training plan.  This phase provides information as regards the identification of need, aim of the training and expected results, length of training, target population, sources of financing, and management plan (Rossi, Lipsey, & Freeman, 2003).

Step 4: Develop and evaluation plan for training

Evaluating the training and the trainers is a crucial, frequently overlooked, aspect of the training development effort.  An Evaluation plan is a guide on what you need to assess, what information is essential, and who you are evaluating for (Moore & Dutton, 1978).The evaluation plan should enumerate the main evaluation questions and the comprehensive monitoring questions that assist with answering the evaluation questions. This lets the Nursing Quality Development Unit to ascertain the information they have to gather, and how best to go about this. Contingent upon the facts of the training and evaluation plan, they can pinpoint the individuals in charge of various tasks, and timelines. The plan out to be easily understood by anyone partaking in the project at any time and be detailed on the occurrences as regards monitoring and evaluation (Tweddle, Robinson, Pocock, & Roy, 2012). It is likewise imperative to remember that there are several types of evaluation. An evaluation plan should preferably be completed at the planning stage of a project, prior to commencing the implementation phase. This will let the Nursing Quality Development Unit to plan beforehand and data collection tasks that you may have to embark on, including pre-intervention surveys.

Step 5: Conduct training

Following the testing and modifications of course materials, the next phase of the training development process involves implementing the course.  How often a course is delivered is determined by the availability of attendees, trainers, and funding.

Step 6: Conduct evaluation

The fundamental aspect of the training process is the gathering of data. It is prudent to utilize standardized data collection tools, which are proven to be effective in real life situations, and enhance these if needed to capitalize on data quality (Phillips, 1997). Besides, using experienced data collectors, giving data collectors training on a particular job or tool and/or overseeing data collection across numerous data collectors can additionally help to decrease partiality or inaccuracies in the  acquired data (Polkinghorne, 2005). Data collection is not essentially the only duty of evaluators; the staffs of unit can help gather the data on behalf of the evaluator (Bringle & Hatcher, 1996).

Step 7: Revising the Training

The final step in the model is revision of the training.  The evaluation will have enlightened the evaluators and major stakeholders on the thought of the participants regarding the training, what knowledge they gained, how their performance and/or knowledge transformed, and perhaps how the organization profited from the training (Russ-Eft & Preskill, 2009).  This information can then be used to adapt the training as required to make it more effective, more goals-oriented, or more suitable to its audience.  Revision of the training can include something as easy as including more breaks or a larger classroom, to providing regionally-based training facilities, to bring in completely new instructors or changing the entire curriculum.  It is essential that once the training is revised, the evaluation carries on so that the training can be continually updated and improved. (Bradley & Connors, 2007)

Expert review

A professional review involves a usability professional using his/her knowledge and expertise of testing the model. The professional will identify problems and suggest solutions to enhance usability (Brink & Morse, 1991). The professional employed by the Nursing Quality Development Unit to review the evaluation model is a person from the fields of evaluation, assessment and, nursing with extensive experience in nursing education and the highest level of understanding in assessment and evaluation.

Generally, the professional is supposed to possess skills and knowledge suitable to the areas of activities in which they are appointed to help evaluate (Berliner, 1986). The professional must additionally possess an advanced level of professional experience in the public or private sector in at least one of the following aspects or activities: research in the applicable scientific and technological fields; administration, management or evaluation of projects; technology transfer and innovation; use of the research results and technological development projects; and human resources (Jones, Armour, & Potrac, 2003). Additionally, the professional is expected to provide feedbacks and comments to the proposers (Blacker, Macchetto, Meylan, Stanghellini, & van der Marel, 2002).

It accepts that some professionals in a certain substantive region are adept at comprehensive examination and evaluation  model that could not be conducted through any other process (Shepard, 1993). The professional reviewed the evaluation model in figure 3 for validation and gave the following recommendation and feedback;

1) Appropriate design; the model kept to a logical step in their design and development based on evaluation theory.

2) The model match between evaluation methodology and Thai health care system.

3) As the Nursing Quality Development Unit have not used the evaluation model and evaluation before and the staffs of the Nursing Quality Development Unit lack evaluation and assessment skill, the model needs to include the workshop in order to provide skills and knowledge as well as direction to the staffs or supervisor.

4) According to the limited skill of the staffs, the example or the additional form or tools need to be provided as assistance to the staffs.

5) Cost and benefit in the model need to be discussed with the policy makers or the top management group for possible budget support for the training.

6) Before revising the model there need to discuss with the Nursing Quality Development Unit about the uncontrollable variable which can impact the outcome of the evaluation process.

Results (Phase 3: Test the model)

The first draft for the evaluation model in figure 3 have been tested at the Nursing Quality Development Unit, which used the One-to One evaluation method with three of the supervisors who were involved in the training and evaluated the training program. They evaluated the model and provided feedbacks as follows;

1. Step 1,2,3,5 and 7 will be conducted by the staff of the Nursing Quality Development Unit and the professor of the Partnership University.
2. In activity 3.2, and 3.3 the instructors who are the external source will provide the method and the media depend on the objective and on the instructors’ teaching plan and methods.
3. Step 4 and 6 are the major concern at the Nursing Quality Development Unit since it is a small organization, all staffs are nurses lacked evaluation know-how, the additional of explanation, example and packaging of tools for example criteria, standard, indicator , logic model and theory of change for the revised version.
4. In practical, the unit will have certain issues about upper management level staffs and head ward attitude as well as on the budget support of the evaluation. The investigator should provide the direction or guideline to solve or avoid these issues.
5. In case the staffs of the Nursing Quality Development unit cannot conduct an effective evaluation due to uncontrollable factors, the investigator should provide the option for readability.

Results (Phase 4: Refine Model)

From the professional and supervisor feedback and recommendations in step 3, the investigator has adjusted the model in step 3 and 4 as shown in figure 4 below;

Figure 4: Training and evaluation model (Revised)

The explanation in each step of developing the evaluation plan helps to assist the user with the details below;

Identify purpose of the evaluation

The two aspects of the evaluation serve as a basis for evaluation planning, focus, design, and interpretation and use of results. The aim of an evaluation impacts the identification of stakeholders for the evaluation, selection of specific evaluation questions, and the timing of evaluation activities. It is important that the program is clear about its intended objectives as regards the evaluation. Purpose priorities in the plan can help form the link between objectives and intended use of evaluation information (Rossi, Lipsey, & Freeman, 2003).

Develop evaluation Framework and logic model

An evaluation Framework also called a statement of theory, theory of program logic, or theory of program action  lays out the connections between the program strategy and tactics and the desired outcomes (Roehl, 2002). Evaluation offers overall general framework for evaluations across various programs or diverse evaluations of a single program for instance; output evaluation, and process evaluation. This includes program theory of change, program logic model and management processes (Funnell & Rogers, 2011).

1. Theory of change

In this study, the Kurt Lewin theory is employed as the theory of change in figure 5 shown below;

Theory of Change Model: Lewin’s Change Theory.

The Change Theory of Nursing was developed by Kurt Lewin (1947), who is recognized as the father of social psychology. This theory is his most influential theory. He theorized a three-stage model of change called unfreezing-change-refreeze model that needs prior learning in order to be rejected and replaced (Vioral, 2014). Numerous generations of management theorists have advanced this model, especially in relation to the possibilities for stability and an end to change. The relation between changes in professional practices and wide-ranging social change will not be linear or simple though the Kurt Lewin three-stage model gives an outline that helps us to visualize, plan and manage each of the stages of change (Cummings & Worley, 2014).

 

 

 

Program Evaluation – Program Logic Model

 

	Assumptions	Resource/ Inputs	Activities / Processes	Participants / Stakeholders	Immediate Outputs	Short-Term Outcomes	Long-Term Outcomes (5 -10 years)
PHASE I:Unfreeze – “ready to change”	The effective training need direction from organization, and training plan which supports the needs of nurses and the facility.	Existing Training plan Instrument for Survey Questions for interview	Survey the needs of nurses. Interview policy makers. Develop training plan.	Supervisors Nurses Policy makers	Needs of nurses in training. Training direction Training plan	Use Training plan as a guideline in professional development Policy maker and nurses display satisfaction in training program	Increased number of nurses who have knowledge, skills, attitude and confidence in providing quality nursing care   Changes in patient health care improvements Change in Hospital quality improvement Change in population level system improvements
PHASE II:Change	The knowledge and skills in evaluation and assessment are essential for supervisors and head ward.	Supervisors Head ward 5 days’ training 1 day workshop for head wards. Course material and equipment Budget from hospital	Workshop for Head wards Training for supervisors on the evaluation skills	Supervisors Head wards	#of supervisors enrolled in program # of head wards engaged with workshop # of supervisors that passed the exam and test % of workshop program and training satisfaction evaluated by supervisors and head wards	Trained Evaluator Increased skill and knowledge and confidence in evaluator of supervisors Increased incorporation of skills in evaluation of  Head wards
	The continued professional improvement for registered nurses will provide quality nursing care.	Nurses Training plan Training program Nurses training record and database Course material and equipment Budget from hospital	Professional Training course for registered nurse	Supervisors Nurses	#of nurses enrolled in program # of nurses that Complete training # of nurses that pass the exam and test % of program satisfaction evaluated by nurses and supervisors	Trained nurses Increased skill, knowledge and confidence in providing nursing care Increased incorporation of skills in assessment
PHASE IIIRefreeze	Supervising and evaluating provides effective direction to ensure the delegated practice of the Nurse is safe, correct and meets the standard.	Supervisors Nurses Head wards Quality control standard Time Nursing Planning	Supervising Evaluating	Supervisors Nurses Head wards	Nurses identify appropriate action to take #of incident reports # of quality nursing care plan # supervisors report # of evaluation report	Reduce incident report Increase standard of nursing care Increase quality of nursing care plan

Criteria and Standard

Phase I: Unfreeze

Activity	Criteria	Standard
Survey the needs of nurses.	The needs of registered nurses in training.	95 % of registered nurses participated in survey.
Interview policy makers.	Direction of training and evaluation.	Directions, opinions and recommendations for improvement of the effectiveness of the program training and evaluation.
Develop training and evaluation plan.	The effective training and evaluation plan.	The realizable training and evaluation plan developed from the needs of registered nurses and based on the policy.

Phase II: Change

Activity	Criteria	Standard
1 day workshop for Head wards for understanding the evaluation process in the same direction.	Effectiveness of formal feedback from head wards.	More than 90% satisfaction on the workshop. More than 90% gave positive response to content of the training and evaluation. More than 98% increased their confidence to engage in evaluation at their wards.
5 days’ evaluation training for supervisors.	Positive result in the test	Pass 75% of test on basic knowledge on evaluation And More than 20% overall of the test result which is higher than pre-test.
	Positive result in the assignment in evaluation.	Pass 75% of assignments.
	Positive feedback from trainees (supervisors).	More than 90% of trainees gave positive response about increase in their confidence to work as an evaluator.
	Positive feedback from the head wards.	90% of trained supervisors demonstrated appropriate action and confidence evaluating (in the first month after training).

Activity	Criteria	Standard
Professional Training course for registered nurses.	Number of registered nurses who use training plan for continuing development education.	95% of registered nurses have individual plans for continuing professional education. 95% of registered nurses attend training from their individual plan.
The continual professional training courses will be conducted based on the needs of hospitals and registered nurses and each course on the criteria and standard may be adapted based on the purpose and method of training.	Positive result from test.	Pass 75% of test on basic knowledge And More than 20% overall of the test result higher than pre-test.
	Positive result from assignment.	Pass 75% of assignments.
	Effectiveness of formal feedback from trainees (registered nurse).	More than 90% satisfied with the training course More than 90% gave positive response to content of the training course.
	Number of the trainees to finish the course.	90% of the trained finished the course.

Phase III: Refreeze

Activity	Criteria	Standard
Supervisor visits ward to evaluate registered nurse’s nursing care and provide feedback.	Positive feedback from registered nurse and head ward.	More than 90% give positive response to supervising and evaluating process. 90% of registered nurses use feedbacks from supervisors to improve their nursing care. Decrease of more than 10% of incident reports.

 

 

Selected evaluation method

Evaluation plans ought to detail how evaluation data will be collected for various areas of the evaluation. These options are important for finalizing evaluation tasks related to gathering data, analysing data, reporting data, and taking decisions as regards the evaluation (Patton, 2011).

Design evaluation instrument

The development of a proper evaluation instrument is usually the most disturbing aspect of evaluating a gain in knowledge and actions of the program participants. The concerns are regarding how to design an efficient instrument that collects the information, and has a valuable response rate and consistent responses (Colton & Covert, 2007).

Determine and develop program content

The needs of nurses and aims of the training program are key features when deciding subject content. The content may be dependent upon subject-matter professionals who choose what trainees need to know in order to achieve the aims. This step needs painstaking review by suitable staffs of the unit, for instance supervisors or trainers. Some changes may be required based on the qualifications of the trainers who will present the program and on the training budget as well (Phillips, 1997).

Test program and revise

After the program is developed, a test run may be required. Testing is especially important when the program will be repeated with many participants as opposed to a one-time program. This step affords us the chance to test at least to a certain level, some of the evaluation methods (Phillips, 1997).

Results (Phase 5: Validate and adjust components of model)

The professional and supervisors of the Nursing Quality Development Unit validate and modify mechanisms or parameters until the model reaches a satisfactory stage and epitomizes reality.

The expert review

The professional review carried out in the same manner as step 2; the expert is happy with the refined model in figure 4.

One-to-one evaluation

One-to-one evaluation carried out in the same manner as in step 3.1; the supervisor is also happy with the refined model in figure 4 and the additional details of theory of change and program logic model in figure 5 and table 1.

Summary

The Nursing Quality Development Unit has been carrying out training programs for registered nurses since 2002. However, the evaluation is limited to the end of training surveys and is usually fixated on identifying the problems of the program. The evaluation plan and tactic have not been developed for the evaluation system. The problems pertinent to evaluation such as the absence of an evaluator, staffs lack of knowledge and skills in evaluation, attitude of policy makers and head wards on the evaluation, pressure from the top to make contributions, lack of commitment, uncontrollable variables like self-motivation of participants, and the workplace environment.

The evaluation for training program for registered nurses model is developed from HRD (Human resource development model (Phillip, 1983) and integrated with the training plan for the Nursing Quality Development Unit.

The first draft was received by the expert and evaluated by supervisor of the Nursing Quality Development Unit and then used for feedback and recommendations for the revised version. In the revision model, the investigator provides detailed step by step and adjusts the model according to the suggestions. Also, logic model and theory of change were designed to include the example of instruments and evaluation question with standards and criteria. The final draft was carried out for expert review and one-to-one evaluation and revised until it was satisfactory and reliable.

SECTION 5- DISCUSSION

This section presents a discussion on the findings and contributions and the summary of the probable results of the training evaluation and also identifies the limitations of the current work. This Section is divided into four parts. The first part discusses the probable results of the training evaluation; the second presents the discussion of the evaluation model, the third discusses the limitations of the current work while the fourth brings the study to a conclusion.

Summary of the expected outcomes of training evaluation

The assured quality of the Thailand healthcare system relies on laws and regulations of the country to ensure that the Thai citizen can access quality in the health care system. In light of this, all registered nurses should be afforded the opportunity for continuing professional education and training to meet the standard of assuring quality. Together with the Change Theory of nursing which was developed by Kurt Lewin (1947), the General Hospital anticipated the profound changes from the training program in three levels such as individual, organizational and National. This evaluation program designed to examine the competence and effectiveness of the program and an investigation of its influence on the wellbeing of the citizens, concentrating on the outcome evaluation. The General Hospital will gain relevant decision-making information that will support the extension of the future coverage of the program for all registered nurses and a decision on the improvement or modification of a program, demonstrating accountability, and justifying the funding.

Figure 6: Summary of outcomes in training evaluation

 

Discussion

Quality assurance is now a major consideration for providers of goods and services in the society most especially in the e-health sector. Also, continuing professional training is one of the essentials for healthcare facilities to develop knowledge, and skill for registered nurses to provide quality care and to meet the standard of quality assurance. It is no doubt then that evaluation strategies have become one of the most important challenges which could help improve their training within the limited factors of the hospital which include the staffs’ lack of evaluation knowledge and skills, top management lack of emphasis on training evaluation and the problems with budget for evaluation.

An evaluation model of the Nursing Quality Development Unit, adopted from the Philips evaluation model and logic model and linked to the construct of Nursing and Midwifery Council measurement indicators and regulatory frameworks for assuring quality health care system in Thailand is proposed. Action research method is used to develop an evaluation model for training measurement indicators in a hospital. This is due to a better comprehension of the critical intangible assets with the registered nurse manifestation in activities being according to their continuing professional training and contribution to overall quality assurance ultimately. Based on this coherent and consistent model for training evaluation for quality assurance, it would be possible to predict the consequences of the decisions made based on the information given by the Nursing Quality Development Unit.

An illustrative example is given of the proposed model. Although the assessing criteria may be affected by the contingent factors of different training courses, it is the adaptation of the training evaluation model to those factors for the assessment of their training contribution in a hospital.

This would certainly go a long way towards addressing a number of criticisms of evaluation aspects of the hospital. However, with the support from this study, the training evaluation model should continue to be considered as the important process for quality assurance for the hospital. The evaluation model could thus, provide an insight into the improvement of the development of continuing professional education reporting for quality assurance.

Limitation

Although this study was carefully prepared, it is important to note the limitations of the studies involved in this study. First of all, as previously discussed, an important limitation in this study is the study sample selected. The case study unit is in overseas, which means the investigator was unable to collect data at the site. Thus, the use of technology to conduct the interview and evaluation might not be accurate as it would be if conducted with the participants at the site. Novick (2008) reported certain drawbacks of the telephone interviews which included limited telephone coverage in certain areas, the need for short interview durations, and the absence of visual or nonverbal cues as well as loss or distortion of verbal data (Novick, 2008). The second reason is that, the study was conducted within limited time which was not enough for the investigator to repeat the step of testing the model. Action research is time-consuming and complex both to conduct and to report. If the researcher can repeat the process, it will increase the efficiency as well as improve the rigours of the study (Ballantyne, Hughes, & Mylonas, 2002). Thirdly, the study in some stages involved organization policy but the reliable data was unable to be provided by the participants which can be a significant obstacle in the findings. Rapoport (1970) draws attention to the conducted action research of the risk of the researcher becoming over-involved in the situation or of being used as a tool in organisational politics (Rapoport, 1970).

Conclusion:

As an investigator who was working as a registered nurse and as an educator who was involved with the training and the supervisor who responded in continuing training program for registered nurses, the selected nursing organizations in this research have the same opinion that the training program evaluation must be a developed model for effectiveness, efficiency and worthiness. In this project, the Phillip’s evaluation model has been used and modified to create the evaluation model for continuing professional training for registered nurses combined with the logic model and the theory of change, the example of standard, indicators and criteria also provided in this study. Under the belief that the findings of this research will be able to respond to the requirement of the Nursing Quality Development Unit standards of Nursing and Midwifery Council as well.