Non-Cognitive Skills in the Problem-Based Learning Environment

Project-Based Learning (PBL) can trace its origins back over a hundred years to educators like John Dewey who observed that we learn more when we are engaged in experiential situations that stimulate our curiosity. Traditional curricula focus on content specific knowledge which generally requires students to make use of intellectual skills like thinking, reasoning and memorization. These skills are collectively referred to as cognitive skills.  Assessment of cognitive skills has been studied and examined in exhaustive detail over the past decades. An entire industry of consultants and assessment organizations has evolved to provide standardized tests, evaluations, studies and analysis of cognitive data to assist educators around the world. My question focuses on the realm of non-cognitive skills in a PBL environment. I want to know: How can non-cognitive skills based rubrics be effectively used to improve teacher assessment of student readiness to develop independent projects in PBL? My exploration of articles, books, studies and reports raised other questions which required some answers. Those questions are: What is the difference between non-cognitive and 21st Century skills? Why are non-cognitive or 21st Century skills necessary for PBL? How can non-cognitive or 21st Century skills be assessed?

Findings

Non-cognitive vs. 21st Century skills

 The initial review of literature started with searching for information on 21st Century skills. The search started to yield a variety of other terms such as: non-cognitive skills, non-cognitive constructs, soft skills, people skills, human skills, character traits, social skills, emotional intelligence, personal skills, personal qualities, survival skills, and socio-emotional skills. This variety of terms required a better understanding of the similarities and differences regarding these terms.

 The United States Department of Labor identified cognitive and non-cognitive skills as key components for competencies of all industry workers. The non-cognitive skills are also referred to as soft skills that make up a person’s personality. These soft skills enable workers to work together and be successful in a work environment (ACT, 2014). 

Student achievement has also been linked to non-cognitive skills or attributes but both Kafka (2016) and Bjorklund-Young (2016) agreed that defining and measuring these skills has been challenging.  Egalite, Mills & Greene (2016) actually referred to non-cognitive skills as the “softer side” of student learning. The capstone paper on Creative Problem Solving by Guenin (2015) and the book by Little & Ellison (2015) used common terminology referring to 21st Century skills as critical for student success. In his article, Hardie (2015) equates 21st Century skills to survival skills for students. In the case study by Wismath & Orr (2014), the work patterns of 133 students at a university were studied focusing on the 21st Century skills of collaborative communication and problem solving.

Some of the literature reviewed revealed authors freely mixing or interchanging terms without providing clear definitions to distinguish one from another. Kirabo (2018) equated non-cognitive skills to both socio-emotional skills and learning mindsets. In her dissertation, Rutledge (2016) talked about non-cognitive, 21st Century, soft or character skills. She used the terms social, behavioral, and emotional skills. Rutledge attempted to bring these terms together by stating that they all embrace personality characteristics and traits. She also mentioned intrapersonal and interpersonal skills. The all-encompassing term she used for these skills and traits is “personal qualities”. In the study using behavioral measures of non-cognitive skills, West, Kraft, Finn, Martin, Duckworth, Gabrieli & Gabrieli (2016) mentioned 21st Century skills but emphasized that non-cognitive skills embrace a wider set of competencies.

The research done by Bjorklund-Young (2016) was probably the most enlightening regarding the use of different terms, sometimes referring to the same skills, or attributes. She pointed out that the identified non-cognitive skill sets depend upon the field of study that examines those skills. Psychologists classify skills in terms commonly used in psychology like neuroticism and extraversion. Educators tend to classify the non-cognitive skills based on their relationship to student academic success.

The variety of terms and definitions encountered, indicated a need to research a foundational work regarding 21st Century skills. In their book, 21st Century Skills Learning For Life in Our Times, Trilling & Fadel (2009) related their expertise with this subject. They both co-chaired the Partnership for 21st Century Skills (P21) Standards, Assessment and Professional Development Committee. This committee produced the foundational 21st Century learning framework. The center of the framework is the category of Learning and Innovation skills. Within this set of skills are the three sets of critical 21st Century skills of critical thinking & problem solving, communication & collaboration, and creativity & innovation.

Project-Based Learning and Non-cognitive or 21st Century skills

 Meyer & Wurdinger (2015) agreed with both Little & Ellison (2015) and Deys & Deys (2015) that project-based learning nurtures the development of 21st Century skills like communication, critical thinking and collaboration. They pointed out that problem solving is a key constructivist element in project-based learning. They also emphasized project-based learning provides students with a vehicle to learn collaboration and communication skills as they work with others and share their ideas with their peers, teachers and members of their community. Little & Ellison (2015) also stated that schools utilizing a PBL curriculum tend to integrate community building with students into their culture. This community building promotes the personal attributes and characteristics which help students develop 21st Century skills.

 Project-Based Learning provides a process which requires students to not only conduct research but to organize their information. This skill helps students surmise what evidence they will need to create. The ability to take this step is an important part of problem-solving.  Creativity is the 21st Century skill that students then utilize to advance their concept of evidence into reality. The actual creation of this evidence is the primary vehicle for students to demonstrate knowledge, comprehension and competence in their learning (Pete & Fogarty, 2018).

Assessment of non-cognitive or 21st Century skills

 The ACT fact sheet on the ACT WorkKeys assessment stated that this assessment measures both cognitive and non-cognitive skills. The purpose is to help employers determine the ability of a student to be successful in either professional or skilled labor positions in the workforce. The non-cognitive skills the assessment measures are fit, performance, and talent.  The assessment asks students questions about their interests, activities, personal values and attitudes that might occur in workplace environments (ACT, 2014).

 The methods and types of measurements used to assess non-cognitive skills take different approaches. Kirabo (2018) and Egalite et al. (2016) used behavioral measures to assess the non-cognitive skills of groups of students. The study report by Kirabo stands out from many of the other studies that were researched in the writing style. Kirabo’s style of writing seemed to indicate a bias toward his use of behaviors to measure these skills as the best approach to use. He discussed using data about student suspensions and on-time grade progression as behavioral measures. Kirabo even titled one section of the report as The Predictive Power of Behavior and followed it up with a nice colorful bar chart titled Behavior is a Stronger Predictor of Student Success. He mentioned gathering all of the raw data and then using his own set of calculations to arrive at his conclusions. It is difficult to accept his conclusions because he provided almost no information regarding how he conducted his calculations.  

In contrast to the study by Kirabo (2018), the study by West et al. (2016) actually mentioned that measuring non-cognitive skills is a very elusive task. The article by Pieratt (2019) pointed out that even trying to develop a good definition of some non-cognitive skills can be a challenge. Assessing a skill can be difficult if there aren’t clear definitions and examples. Pieratt used an example of asking teachers to tell you what successful small group work looks like. The most common response she obtained is similar to the statement, “I can’t really tell you what it looks like but I can tell you what it does not look like”.

 Reliable results from scientific research indicate that current measures developed for non-cognitive skills can help predict success for students in both school and in the workplace.   Many of these measures, however, have only been developed for use in scientific research. The actual ability to reliably use these skill measurement tools for educational purposes are questionable as not all of the measurement tools translate well into the world of education. (“Measures of”, 2015).

 In her article about teaching strategies, Harshbarger (2018) talked about how to measure social skills. She recommended using versions of learning scales with increasing levels of rigorous versions of standards as students’ progress in skill development. Harshbarger also stated that students should use Likert scales to measure their own level of engagement in the development of these skills. In her dissertation, Rutledge (2016) actually discussed personalized learning programs like project-based learning environments. These types of environments usually include regular meetings between students and teachers regarding long term projects.  This type of prolonged interaction tends to promote strong student-teacher relationships.  Rutledge stated that a tool teachers can use to measure a student’s non-cognitive skills is this relationship building process. She revealed that questionnaires or tasks are frequently the most common forms of assessment used for measuring non-cognitive skills. The questionnaires either have the teacher respond based on judgments and observations about a student or they are self-reports that the students provide about themselves. Rutledge pointed out that a potential issue with teacher judgments can be a lack of objectivity. Likewise, a potential issue with student self-reports could be students skewing data by trying to second guess the questionnaire. Students may assume there are correct and incorrect answers and try to provide what they think is the correct answer.

 A potentially unique approach to assessing non-cognitive skills originated with the development of a new measurement tool by a non-profit organization called the Independent School Data Exchange (INDEX). This approach is mentioned in the book Loving Learning: How Progressive Education Can Save Americas Schools (Little & Ellison, 2015). INDEX worked with the Educational Testing Service (ETS) to develop an online tool to measure student character traits. These traits, or skills, include teamwork, creativity, ethics, resilience, curiosity, and time management. This tool is called the Mission Skills Assessment (MSA). The INDEX and ETS websites, unfortunately, provide almost no information about the MSA. One report did examine the MSA and confirmed the six non-cognitive skills it measures. The assessment is used for students in Grades 6 to 8 and utilizes a student self-report scale, a situational judgment test and a teacher report scale. This research report stated that the majority of the assessment tools are Likert type scales and self-reports. Schools using this assessment do not receive reports on individual students but rather aggregate student data to report on the skills across groups of students (Petway, Rikoon, Brenneman, Burrus, & Roberts, 2016). Information about the MSA was scarce in the searches conducted. The available information focused on descriptions of the types of tools used. There were no samples or examples of the actual measurement tools available in any of the searches. Unlike the ACT, ETS has not released examples of previous year versions for review or practice.

Rubrics and non-cognitive or 21st Century skills

 Rubrics seem to be one of the more common assessment tools mentioned in searches regarding non-cognitive and 21st Century skills. In their article about rubrics, Thordarson & Gallagher (2019) made the point that a rubric is a simple scoring tool which provides a means to measure different criteria based on different levels of achievement. They do point out that educators have typically only used it as a scoring tool to determine grades. Thordarson & Gallagher believe that this approach misses the real potential for helping a student improve. They stated that teachers should use the rubric during the formative assessment process to provide specific feedback to students as they develop skills. They warn against using text heavy analytic rubrics with lots of rows and columns because teachers, and students, do not like to use them. Thordarson & Gallagher recommend using a single point rubric. The center column lists the learning target or skill which is the expectation for the student. The column to the left is used for feedback on things that need improvement. The column on the right is feedback on things the student did which exceed the expectation. Thordarson & Gallagher mentioned that while rubrics are good for assessing skills, students should use reflective practices to assess mindset growth. Gonzalez (2017) agreed with Thordarson & Gallagher (2019) regarding the use of a single-point rubric as a good tool for assessing skills. The advantage of using a single-point rubric is focusing on just one skill and its different components. Bernstein & Isaac (2018) mentioned that the majority of analytic style rubrics tend to omit non-cognitive skills as an evaluation component. 

 In her book, Boss (2018) said that the Buck Institute for Education recommends teachers use two sets of rubrics to assess a project. The first set should be for assessing content knowledge. The second set is to assess the success skills or 21st Century skills like collaboration, creative thinking, communication, and creativity. She stated that a separate rubric should be created to assess each of the 21st Century skills. Boss mentioned that teachers should create checklists for observable behaviors related to the 21st Century skills. Teachers can use these checklists when they observe students engaging in the project process. 

 The Organisation for Economic Cooperation and Development (OECD) is an international agency that studies and analyzes evidence-based solutions to a variety of issues around the world. The Programme for International Student Assessment (PISA) is a worldwide, multiyear study, under OECD, of different educational topics to evaluate educational systems. In 2015, PISA published an in depth study of 15-year-old students focusing on the subject of collaborative problem solving. PISA started by developing a specific definition of collaborative work. PISA developed a collaborative problem solving assessment and even identified different competency levels for the criteria they wanted to measure (OECD, 2017).

 The conclusion of this literature review is best summed up by two academic studies and one dissertation. In their study, Flake & Petway (2018) stated that even with increasing interest in non-cognitive skills by both scientific and educational communities, there has actually been very little research regarding how to assess those skills. Rutledge (2016) agreed with Flake & Petway (2018) by calling for more research to develop assessments for the variety of non-cognitive skills. The findings of the study by Egalite et al. (2016) highlighted both the strengths and weaknesses of existing tools for measuring non-cognitive skills. There is a scarcity of measurement tools to evaluate these skills. What are the best measurement tools for non-cognitive skills in a project-based learning environment? Probably the measurement tools recommended by the professionals currently focused on PBL. The only other alternative may be the tools devised and reported in PISA 2015 as PISA focused on finding evidence-based solutions. Specific information on how and where these tools were used in a school in the United States was not available in the report.  

References

• ACT. (2014). Key Facts Cognitive and noncognitive skills Why do Act WorkKeys assessments measure two types of foundational skills? KEY FACTS Cognitive and noncognitive skills Why do ACT WorkKeys assessments measure two types of foundational skills? Retrieved from http://www.act.org/content/dam/act/unsecured/documents/WK-Brief-KeyFacts-CognitiveandNoncognitiveSkills.pdf
• Bernstein, A. G., & Isaac, C. (2018). Critical Thinking Criteria for Evaluating Online Discussion, 12(2). Retrieved from https://eric.ed.gov/?q=”Critical+Thinking+Criteria+for+Evaluating+Online+Discussion”&pr=on&ft=on
• Bjorklund-Young, A. (2016, June). John Hopkins School of Education Institute for Education Policy. Retrieved from https://edpolicy.education.jhu.edu/wp-content/uploads/2016/06/NoncognitiveskillsmastheadFINAL.pdf
• Boss, S. (2018). Chapter 5 Assess Student Learning. In J. Larmer (Ed.), Project Based Teaching: How to Create Rigorous and Engaging Learning Experiences (pp. 110–112). Alexandria, VA: ASCD.
• Deys, K. L., & Deys, J. L. (2015, November 30). Challenging the Wikipedia Mentality: Encouraging Original Inquiry through Freire’s Problem-Posing Method, 27(4), 55-63. Retrieved from https://eric.ed.gov/?q=brainstorming&ff1=dtySince_2015&ff2=subCooperative+Learning&id=EJ1123229
• Egalite, A. J., Mills, J. N., & Greene, J. P. (2016, March). The Softer Side of Learning: Measuring Students’ Non-Cognitive Skills, 19(1), 27-40. Retrieved from http://dx.doi.org.libweb.uwlax.edu/10.1177/1365480215616313
• Flake, J. K., & Petway, K. T. (2018, May 22). Methodologies for Investigating and Interpreting Student–Teacher Rating Incongruence in Noncognitive Assessment. Retrieved from https://onlinelibrary.wiley.com/doi/full/10.1111/emip.12201
• Gonzalez, J. (2017, September 22). Your Rubric Is a Hot Mess; Here’s How to Fix It. Retrieved from http://www.brilliant-insane.com/2014/10/single-point-rubric.html
• Guenin, S. (2015, May 26). Creative Problem Solving: A High School Curriculum Satisfying the Needs of 21ST Century Learner. Retrieved from https://scholarworks.umb.edu/cct_capstone/338/
• Hardie, E. (2015, June). When Students Drive Improvement, 72, 92-96. Retrieved from http://www.ascd.org/publications/educational-leadership/summer15/vol72/num09/When-Students-Drive-Improvement.aspx
• Harshbarger, D. (2018, March 5). Building Students’ Noncognitive Skills. Retrieved from https://www.edutopia.org/article/building-students-noncognitive-skills
• Kafka, T. (2016, January). CCRC Teachers College, Columbia University. Retrieved from https://ccrc.tc.columbia.edu/images/a-list-of-non-cognitive-assessment-instruments.pdf
• Kirabo, J. C. (2018, December 4). The Full Measure of a Teacher: Using value-added to assess effects on student behavior, 19(1), 62-68. Retrieved from https://www.educationnext.org/full-measure-of-a-teacher-using-value-added-assess-effects-student-behavior/
• Little, T., & Ellison, K. (2015). Metrics of the Heart. In Loving learning: how progressive education can save Americas schools (1st ed., pp. 146–147). New York, NY: W.W. Norton & Company.
• Measures of students’ ‘non-cognitive’ skills for teacher evaluation, school accountability, or student diagnosis. (2015, May 13). Retrieved from https://www.sciencedaily.com/releases/2015/05/150513102629.htm
• Meyer, K., & Wurdinger, S. (2015, November 30). Students’ Perceptions of Life Skill Development in Project-Based Learning Schools, 2(1), 91-114. Retrieved from https://eric.ed.gov/?q=”project+based+learning”+AND+”problem+solving”&pr=on&ff1=dtySince_2015&id=EJ1127481
• OECD (2017), PISA 2015 Results (Volume V): Collaborative Problem Solving, PISA,
  • OECD Publishing, Paris. http://dx.doi.org/10.1787/9789264285521-en
• Pete, B. M., Fogarty, R. (2018). Everyday problem-based learning: quick projects to build problem-solving fluency (pp. 74–121). Alexandria, VA: ASCD.
• Petway, K. T., Rikoon, S. H., Brenneman, M. W., Burrus, J., & Roberts, R. D. (2016, April 30). Development of the Mission Skills Assessment and Evidence of Its Reliability and Internal Structure. Research Report. ETS RR-16-19. Retrieved from https://eric.ed.gov/?q=”Mission+Skills+Assessment”&id=EJ1124781
• Pieratt , J. (2019, April 4). 3 Steps to Supporting the Crusade for Better Student Collaboration, 14(22). Retrieved from http://www.ascd.org/ascd-express/vol14/num22/3-steps-to-supporting-the-crusade-for-better-student-collaboration.aspx
• Rutledge, J. (2016, November 30). Measuring What Matters: How Noncognitive Skills Are Captured, Stored, and Utilized in Personalized Learning Environments. Retrieved from https://eric.ed.gov/?q=measuring+noncognitive+skills&ff1=dtySince_2015&id=ED579135
• Rutledge, J. R. (2017). Measuring what matters: how noncognitive skills are captured, stored, and utilized in personalized learning environments (dissertation). ProQuest LLC., Ann Arbor, MI.
• Thordarson, K., & Gallagher, A. (2019, March). Will Design Thinking Kill the Rubric?, 61(3), Retrieved from http://www.ascd.org/publications/newsletters/education-update/mar19/vol61/num03/Will-Design-Thinking-Kill-the-Rubric¢.aspx
• Trilling, B., & Fadel, C. (2009). What Are 21st Century Skills? In 21st Century Skills Learning For Life in Our Times (1st ed., pp. 45–60). San Francisco, CA: Jossey-Bass.
• Wagner, T. (2002). Making the Grade: Reinventing America’s Schools (1st ed.). New York, NY: RoutledgeFalmer.
• West, M. R., Kraft, M. A., Finn, A. S., Martin, R. E., Duckworth, A. L., Gabrieli, C. F. O., & Gabrieli, J. D. E. (2016, March). Promise and Paradox: Measuring Students’ Non-Cognitive Skills and the Impact of Schooling, 38(1), 148-170.  Retrieved from http://dx.doi.org.libweb.uwlax.edu/10.3102/0162373715597298
• Whitehurst, G. J. “R. (2018, August 24). Grading soft skills: The Brookings Soft Skills Report Card, 2(4). Retrieved from https://www.brookings.edu/research/grading-soft-skills-the-brookings-soft-skills-report-card/
• Wismath, S. L., & Orr, D. (2014, November 30). Collaborative Learning in Problem Solving: A Case Study in Metacognitive Learning, 6(3). Retrieved from https://eric.ed.gov/?q=EJ1084619&id=EJ1084619