Determining Validity of Research Framework

CHAPTER SEVEN

FRAMEWORK VALIDATION

7.1 Introduction

This chapter focuses on the evaluation and validation of the framework. The research validity and method deployed to maximise the validity will be discussed in the first part of this chapter. Also, the eternal framework validity will then be discussed.  The second part of this chapter will analyse and discuss the results obtained from the evaluation of the framework.   The framework evaluation has been carried out by academics and experts practitioners and has measured the suitability and effectiveness of the framework in the construction sector.

7.2 Validity

The validation process started at the first stage of the research,  making sure that the research participants, measuring instruments and context were exclusively accomplishing the research aim.  A number of writers have contended that validity cannot be applied to qualitative research. At the same time, have understood the need for some qualifying check or measure for their research (Golafshani 2003). Clearly there are no direct tests validating qualitative research; however, this does not imply that there are no guidelines (Patton 1990). Through the stages of this research, the effort has been made to guarantee that: (1) the measuring instrument is measuring the concept being referred to and no other concepts; and (2) there is an accurate measurement of the concept (Bailey 1994). The measuring instruments used in the interview method were the interview questions; also the method used in analysing the data collected and building the framework. Therefore, it was important to ensure that all research propositions were clear and effectively measured. Also, the method of analysis required the provision of right inferences.

Qualitative research has often been portrayed by flexibility and uncertainty, and tension between creativity and rigour. Therefore at the interview analysis and the stages of framework building,  a critical compromise was carefully considered (Pyett 2003). The analysis of interviews engaged “a deductive process” whereby the author was “his most trenchant critic” (Cook and Campbell 1976, p: 229).  This is known as internal validity.  However, in this study, an open viewpoint of validity was used whereby different techniques enhanced the analysis and understanding of the practice of knowledge capture and retrieval and prompted more reliable and valid development of the framework (Golafshani 2003). These methods included:

• An in-depth revision of the instruments used for data collection and the method of analysis
• Carrying out a peer review of interviews themes and methods of analysis
• Carrying out a pilot study; assessing the language of the questions; definitions and terms; clarity of the purpose of questions for interview
• Construct a descriptive and relational framework for the interview analysis to make sure there is the consistency of analysis criteria and accurate interpretation and inferences.

7.3 External Validity

Developing a  framework for the capture and retrieval of WLC knowledge in a construction project is based on a small number of cases, and does not mean this framework can be suitable and effective to other construction organisations practising WLC in the construction sector. However, researchers are also interested in the generalisation of the findings beyond the investigated cases (Schwab 1999). The justification for carrying out an in-depth interview is that people involved in a matured practice have knowledge that would somehow or another not be accessible to the researcher. Therefore, it is the quality of the understanding that is vital, instead of the number of respondents who share it (Wainwright 1997).

While researching a few cases may accomplish excellent validity by giving a significant understanding of the practice in those cases; they have been broadly criticised as lacking external validity (De Vaus 2001). The generalising of findings to or across a target population sample is known as external validity (Pedhazur and Schmelkin 1991).  There are no efficient or verifiable methods to carry out external validation based solely on a  single examination of a research relationship (Schwab 1999). However, a definitive way to assess the validity of the proposed framework is to test such principles in reality, yet within research practice, this is rarely conceivable (Pyett 2003).

Framework testing by the application method,  it could take quite a while before the reasonable judgment of the framework validity could be reached. Regardless of the possibility that this was to take a short time, there would be a need to access to a huge sample (representing the industry) and the framework implementation in this sector. However, in this particular case, this was viewed difficult.  Nevertheless, a few other methods were able to amplify the framework validity in this study as outlined:

• Select example case studies organisations (of current and best practices) from the target population in the interview stage.
• The pilot study provided a solid ground for understanding the current practice of KM; and therefore it was utilised as guiding principles in using the best practice to suit the industry, which boosted the generalising or external framework validity.
• Looking for the evaluations of the suitability and effectiveness of the framework in the construction sector by experts practitioners and academics.

7.4 Framework evaluation

Testing the general framework feasibility by applying it to a large number of organisations representing the construction sector have been difficult in this research.  The framework evaluation by experts was viewed as another alternative for evaluation.  in fact, this method was deemed as appropriate with regards to risk,  finance and value.

Assuming that 100 randomly selected organisations are a representative sample and the resources required are accessible in those organisations,  it would take six months for the testing and implementation of the framework (practically it would take years). However what if the framework did not work?  This would mean lost time of 600 months (100 organisation X 6 months) and therefore significant effort.  Therefore, in this case, the evaluation method by experts could be deemed the most appropriate method even though the  “generalise by implement” method was deemed possible.

The reason for the evaluation method accordingly was not to anticipate that the framework was suitable for the organisation of the participated experts;  to obtain the expert’s evaluation in regards to the frameworks suitability in the industry.  The suitability of the framework was not seen adequate however because the suitable framework did not necessary means it would add value. Therefore, effectiveness was another criterion the evaluation sought to answer.

7.4.1 Participants

• Justification

Based on their experience,  the suitability and the effectiveness of the framework can be judged by experts by using the right tool techniques, required resources and methods. The time and effort required for the framework implementation can also be estimated and matched with the current practice.

The participants involved in the framework evaluation is made up of two groups of experts:  academic and practitioners. The practitioners are involved with  WLC practice in the construction project and therefore have a better understanding of the daily work. Also, their participation in a project which requires WLC practice means they can work with other organisation who practice WLC in the construction sector and can understand the verity of work environments.

Also, academics are expected to have a deep understanding of different methodological viewpoints, as they know about observation and analysis.  Further,  the academics rely greatly on the construction industry as a major source of information,  so does not imply that they will be isolated. Also, many academics are involved in the practice.

• Sample

A total number of sixteen experts were divided equally between the academics and the practitioner who participated in the framework evaluation. The participants year of experience ranges from eight to thirty-eight years, accounting for the total of three hundred and thirteen years of experience. The eight practitioners were from the three case study organisations who participated in the pilot and main interview of the study. The focus on practitioners was on those in the field with experience in WLC practice because they had an in-depth understanding of the WLC practice. Also,  participating academics were from four universities in the UK, working in the built environment and information and knowledge management departments.

Table 7.1: Characteristics of participating experts in the framework evaluation

Because the framework evaluation required a seminar presentation,  this process was thought to be long, and it was expected a low rate of response. Therefore, it was decided to reach the target of participants by sending invites to a member of staff with the three case study organisations.  The total number invitations forwarded was 300 of which 120 were sent to academics; and 180 to the three case study organisations.

7.4.2 Method of Evaluation

Emails was used a tool to invite experts and included;

• An overview of the research project
• Request to participate in the evaluation of the framework for the purpose of validity
• Request on the best times which suits each of the case study organisation for a 40mins seminar presentation

The framework was presented to each case study organisation using  Microsoft Office PowerPoint 2007.  The presentation was interactive in which participants lessened the explanation; while diagram parts were moved in each presentation slides accordingly.  The seminar presentation was arranged into ten slides which took a narrative sequence of logic since it began by clarifying the parts of the framework independently and afterwards presented the strategic framework.

Figure 7.1: A screenshot of the PowerPoint presentation

The questionnaire was designed and hosted using the online service provider: “Survey Monkey’ and consisted of two parts. Firstly, details about participants including name, organisation, job title, business and size of the organisation.

secondly,  the framework evaluation and this was  included in  two sections

• Score based questions to the rate the suitability and effectiveness of the developed framework
• Seeking the opinions of the participants about the advantages, limitations and ways of improving the framework.

It was decided to present the framework diagrammatically showing the strategic, tactical and operational levels.   The presentation gave a more in-depth description; however diagrams were sent to four research students to assess their readability, and the outcomes affirmed that such diagrams could convey the concept and details of the framework if the reader had extensive experience in construction management. One of the research students who took part in the pilot study was a medical doctor, dissociated from the construction industry, who found the framework unreadable, Although he has a good understanding of the major concept. Respondents were also sent an attachment of the presentations slides via email.

7.5 Results

7.5.1 Suitability of framework

Participants were being asked to show their rating for suitability for each of the components of the framework; where 1 implied unsuitable and 5 suitable. The answer from the analysis uncovered that the levels of suitability for all approaches were considered far above the ground.

As shown in Table 7.2,  3.74 out of 5 is the lowest mean value for the architecture of the knowledge base. Although it is not indicated in the value that the architecture of knowledge base is a 100% suitable, rather, it shows that it is likely going to be more suitable than unsuitable. The circumstance with regards to knowledge capture, retrieval and structure approach much better as they all have a score of  4 out of 5.  The skewness measure has a negative value which shows  that the mass of distribution concentrates on the side of suitable.

Table 7.2: Suitability of the framework components

Figure 7.2 shows that the majority of the participants gave a 4 or 5 rating of the suitability of all the framework components. With regards to the architecture of the knowledge base, the suitability was rated under 3 by two participants (i.e. 2). The normal curve illustrates that the approximation of the average of all components almost falls in the 4 rate of suitability. All the indications provide evidence that those components are suitable for the construction organisations.

7.5.2 Effectiveness of framework

Participants were being asked to show their rating for effectiveness for each of the components of the framework; where 1 implied to be ineffective and 5 effective. The mean value of effectiveness for all components as shown is (Table 7.3) are above 3.75 and ranges between 3.87 and 4.18. The skewness measure has a negative value which shows that the mass of the distribution is concentrated on the side of effectiveness.

Table 7.3: Effectiveness of the framework components

As shown on the histograms shown in Figure 7.3 shows that the majority of experts participants evaluated all components to a score 4 or 5 rates of effectiveness. Knowledge retrieval thus turns out slightly more effective than the other framework components. The normal curve makes it clear the approximation of the average rate of all components which almost lies in the 4 rate of effectiveness. All the indicators together provide evidence that those components will be effective when implemented.

7.5.3 The Entire Framework Suitability

Experts participants were asked to indicate their rate of suitability for the entire framework using the same criteria as mentioned above. As shown in Table 7.4, the mean value of suitability (3.93) and the negative value of the skewness measure provide an encouraging indication for the external validity. In Figure 7.5,  the pie chart shows that the majority of experts participants (62%) rate the suitability 4 out of 5; and the vast majority (81.3%) are either giving the framework a rate of 5 or 4. Furthermore, the histogram in Figure 7.4 shows that only one expert gave a rating of suitability under 3 (i.e. 2). Therefore, the indications of suitability are positive and show that this framework is considered suitable.

Table 7.4: The entire framework suitability

7.5.4 Effectiveness of the Entire Framework

Expert participants were asked to indicate their rate of effectiveness for the entire framework using the same criteria as mentioned above. The mean value (4) and the skewness measure value (negative) which shows that the framework is evaluated to be effective as shown in Table 7.5. A third of the expert’s participant shows a full mark of effectiveness, and 40% scored the framework a rate of 4 for effectiveness as shown in Figure 7.7. However, as shown in Figure 7.6, three experts rate it a 3 and one a 2. Nonetheless, the normal curve is pointing to an approximation of effectiveness at the rate of 4. Indications show that the framework is effective.

Table 7.5: Effectiveness of the entire framework.

7.5.5 Experts Opinion about the Framework

The expert participants were requested to express their opinion about the framework regarding suitability, effectiveness, advantages and limitations. This could give additional assessment not covered in the closed-ended questions. Twelve of the feedbacks were received and had been listed below:

• It looks good to me. The problem is getting it work in practice.
• Great model and l like it, particularly the knowledge forms and the categorisation. This will help save time when searching for knowledge
• I think the overall seminar presentation was good and it covered a lot of the things we do not consider as worth pointing out in our day to day project delivery implementation.
• It is very hard to make a judgment on these key issues just on the presentation. However, I am quite inspired by the approach to knowledge capture. Although the presentation is great, however, the information, it provides is insufficient.
• It seems very general.
• The framework works well in suitability and effectiveness; and in use of collaboration tools, intranet tools with knowledge management within organisations. It is essential for a large organisation that staff can find and utilise answers as well as gaining access to the subject matter, experts and sharing information on similar projects and programmes. The advantages are that knowledge is shared rather than lost or protected by individuals. The saying that knowledge is power is true of individuals; However in organisations that are able and willing to capture knowledge and share this with the wider team rather than isolating it with individual project members.
• The framework is suitable and effective in many areas and an insightful and logical structure. There is some duplication, but that would seem to be unavoidable.
• Leadership support is important, with top-bottom commitment. Co-operation/sharing is not always a “natural” inclination in individuals.
• The framework seems to be highly advanced and would require an IT  system to enhance its processes and to encourage people to use it. If all members of the organisation are contributing positively to the framework; and most of them have the ability to capture, retrieve, and then re-use the captured knowledge, then the system will be for sure highly effective. In reality, I think we share the knowledge “sometimes”, but even when we do so, we do not document or capture them properly. As a result, we lose whatever we have shared as we cannot capture, structure and re-use all the knowledge captured by only relying on our brain capabilities. We as practitioners need such a framework to assist us to have a better understanding of the knowledge so we can use it appropriately. From an overall point of view, I admire you for this excellent presentation, and it was informative and well structured. We have learned a lot from this insightful presentation about knowledge management. There is no doubt that this framework is needed for business success.
• Although the framework appears to be complicated on its operational side; I am certain that if the  KM operation depends on this model, the result will be relevant and highly efficient.
• The framework makes sense from a project management perspective. However, knowledge capture and retrieval are complex and resource intensive activities and can be onerous to the project team if proper incentives are not aligned with the leadership’s expectation of knowledge sharing.
• I think if your framework is properly applied, it will be extremely effective. Also, it is highly suitable. Regarding its advantages, I think it is simple to read and understand by any person who has a basic level of knowledge management and its management.

The issues mentioned in the comments can be grouped into four categories: Firstly, those which highlight the role of knowledge capture and retrieval.

Secondly, those who stress the importance and need for such a framework, due to the awareness of the problem, and thus confirm the motives for this research.

Thirdly, indications that the framework may require a huge effort and construction projects already involved in demanding work. This issue has been discussed in the statement of the problem (section 1.4) whereby the problem lies on the capture and not the retrieval side. To minimise the problem, the project review was adopted as the main technique for knowledge capture which already exists in projects, rather than introducing a new approach. Additionally, the daily based reviews were rejected in this research for the same reason. Furthermore, a strategy for motivation has been pinpointed in the framework to ensure the application of knowledge capture.

Fourthly and most importantly, a suitable and effective framework will also require careful application and effort. This has been said by several experts, and therefore considered a critical point, because the good theory does not necessary guarantee successful application.

Finally, an expert described the framework to be very general. As aIn fact the framework is considering a particular area to investigate, related to knowledge capture and retrieval of project management processes, in the context of the project. Models inside the framework have provided precise details regarding what to do in particular scenarios. Also, knowledge capture and retrieval cannot be tackled in isolation from issues such as.

Other points were also considered related to the advantages of the framework; and impressions about the presentation.

7.5.6 Framework Improvement

Another question was put forward in regards to how the framework can be improved. The feedback from the expert participants are listed below:

• By trial and error.
• The framework can be improved after application
• l did not analyse the presentation critically to make rational input in this direction
• I would rather read the whole PhD thesis and make my suggestions
• What issues should be addressed? Profitability, time and teamwork?
• There is no such thing as ideal knowledge framework. Therefore the knowledge framework can also be improved. Evolvement of a framework can be made by lessons learnt from previous projects; receiving constructive criticism and using good management techniques such as benefits and stakeholder management. Furthermore being clear about the benefits of knowledge capture and retrieval, with the onward transmittal/access of results by stakeholders, communication and using the framework communication channels for optimum use. The framework is excellent for understanding the needs and requirements of this age, and that knowledge should be shared for the benefit of all within an organisation; rather than being kept and possibly lost by the few, when the project team is dissolved or re-assigned.
• Show an example of how it will operate in practice.
• Working hand in hand with the IT system which enhances its process and records all captured knowledge after structuring it.

Firstly, improvement is a continuous process, and the most appropriate way to improve the framework is through the application.

Secondly, improvement can be achieved by addressing time, profitability and teamwork which have already been addressed in this study. Profitability is a significant factor that can declare the importance of knowledge capture and retrieval of the project and is, therefore, one of the areas to be embarked upon. However, the development of a model to measure economic profit alone is enough to do a research project.

Thirdly, the exertion of effort in linking the IT system with knowledge capture and retrieval approaches. The knowledge base is a component of the framework, and there is a need to train people on the use of the knowledge base which links knowledge capture and retrieval approaches.

Finally, the use of labels to distinguish between compulsory and optional activities and resources. This has been presented differently in this research as the majority of the activities and resources were thought to be important if knowledge was required to be captured and retrieved successfully. It was decided only to emphasise activities or resources that were suggested but not deemed necessary.

7.6 Summary

This chapter has set out to determine the validity of the framework. It has demonstrated that validity relies on of the inherent validity of the methods of data collection, analysis, and the inferences led to the construction of the framework. Critical in-depth revision, piloting, peer reviews and a fixed framework for analysis were the methods used in this research to maximise its validity.

For the purpose of generalising (external validity) the framework, a number of factors were considered early on in this project. These included using a randomly selected sample in the survey; choosing a sample of best practice from the target population and comparing the results of best practice with the current practice obtained in the survey to assess the ability of and utilise the solution for the target population. Further steps were taken to test the generalisability of the framework by inviting practitioner and academic experts to evaluate its suitability and effectiveness. The results have indicated that the framework and its components are applicable and effective.