Metacognition and Learning in Adulthood

Developmental Testing Service, LLC

Metacognition and learning in adulthood

Prepared in response to tasking from ODNI/CHCO/IC Leadership Development Office by Dr. Theo L. Dawson1, Developmental Testing Service, LLC

Saturday, August 23, 2008

35 South Park Terrace, Northampton, MA 01060 T «Phone» F «Phone» «Email» http:devtestservice.com Developmental Testing Service, LLC

Metacognition and learning in adulthood

Contents

Metacognition 2 Developmental Testing Service, LLC

Metacognition and learning in adulthood1

What is metacognition? Metacognition is thinking about thinking. Metacognitive skills are usually conceptualized as an interrelated set of competencies for learning and thinking, and include many of the skills required for active learning, critical thinking, reflective judgment, problem solving, and decision-making. Adults whose metacognitive skills are well developed are better problem-solvers, decision makers and critical thinkers, are more able and more motivated to learn, and are more likely to be able to regulate their emotions (even in difficult situations), handle complexity, and cope with conflict. Although metacognitive skills, once they are well-learned, can become habits of mind that are applied in a wide variety of contexts, it is important for even the most advanced adult learners to “flex their cognitive muscles” by consciously applying appropriate metacognitive skills to new knowledge and in new situations.

According to Flavell (1979), who coined the term, metacognition is a regulatory system that includes (a) knowledge, (b) experiences, (c) goals, and (d) strategies. Metacognitive knowledge is stored knowledge or beliefs about (1) oneself and others as cognitive agents, (2) tasks, (3) actions or strategies, and (4) how all these interact to affect the outcome of any intellectual undertaking. Metacognitive experiences are conscious cognitive or affective experiences that concern any aspect of an intellectual undertaking. Knowledge is considered to be metacognitive (as opposed to simply cognitive) if it is used in a strategic manner to meet a goal. According to Sternberg {, 1986 #14899) it is "figuring out how to do a particular task or set of tasks, and then making sure that the task or set of tasks are done correctly" (p. 24).

Some of the concepts commonly associated with the metacognition literature are shown in Table 1.

1Theo L. Dawson, Ph.D. (UC Berkeley, 1998), is a respected cognitive developmental psychologist. Her work focuses on the description of learning sequences—the actual pathways through which people learn complex concepts and skills—and the design of developmental assessments of these skills. She has worked closely with members of the IC since 2002, when she, along with her colleague Dr. Kurt Fischer (Harvard Graduate School of Education), conducted a study of the sequences through which critical thinking, leadership reasoning, and decision making skills develop in adulthood. Since then, her work with the IC has involved assisting with the design of leadership curricula, evaluating the extent to which curricula support leader development, and helping to define skill levels for a range of leadership and analyst standards.

Metacognition 3 Developmental Testing Service, LLC

Table 1: Metacognition concepts Concept Description Metacognitive knowledge Includes a person’s beliefs about intra-individual differences, inter-individual differ- about persons ences, and universals of cognition Metacognitive knowledge The information available to apply to a cognitive activity and an individual’s knowl- about tasks edge about the task demands of a given situation Metacognitive knowledge Awareness of and beliefs about available strategies about strategies Level of conscious aware- Retrieval and construction of metacognitive knowledge can be either conscious or ness of metacognitive unconscious knowledge Limits of metacognitive Can be accurate or inaccurate, may not be activated when needed, may not have knowledge much influence when it is activated, and may not have a beneficial effect when it is influential Duration of metacognitive Can be momentary or lengthy, as when we are consciously grappling with a chal- experiences lenging problem Occurrence of metacogni- Most likely to occur when one is engaged in intentional, reflective intellectual activi- tive experiences ties such as problem-solving and learning Effects of metacognitive Can lead one to establish new goals and to revise or abandon old ones, can cause experiences one to add to one’s existing metacognitive knowledge base, and can activate strategies that would otherwise have remained inactivated Memory-monitoring, self- Other terms for metacognition regulation, meta-reasoning, consciousness/awareness, and auto- consciousness/self- awareness Transfer Using a metacognitive skill learned in one context to solve a problem in another context. Cognition A general term for thinking, sometimes distinguished from metacognition, which is thinking about thinking. Ill-structured problems Ill-structured problems are open-ended, with unclear goals, multiple solutions, and no “right” answers—like many of the problems we face in the workplace

Double loop learning Learning how the way one defines and solves problems can be a source of problems (Argyris, 1991).

Models of metacognition The term metacognition is both a general term for “thinking about thinking,” and a term used by a particular group of researchers to describe their field. There are numerous models of metacognition, far too many to describe here. To complicate things further, metacognition is a central component of several skill sets that are central to education and the workplace, including (1) reflective judgment, (2) critical thinking, (3) decision making, and (4) problem solving. (Some researchers argue that these are components of metacognition.) Table 1 provides brief descriptions of some of the more prominent metacognitive models.

Metacognition 4 Developmental Testing Service, LLC

Table 2: Models of metacognition Finding Implication Reference Self regu- (J. G. Borkowski & Thorpe, 1994; lated learning Como, 1986; Ghatala, 1986; (lower levels) Schloemer & Brenan, 2006; Barry J. Zimmerman, 1990) Note: Watch out for applications of this model that focus exclusively on teaching students more effective ways to memorize facts.

Self-regulated learners approach educational tasks with confidence, diligence, and resourcefulness; are aware when they know a fact or possess a skill and when they do not; proactively seek out information when needed and take the necessary steps to master it; find a way to suc- ceed even when they encounter obstructions; view learning as a systematic and controllable process; accept responsibility for their achievement outcomes; and monitor the effectiveness of their learning methods or strategies. Self-regulated learning strategies include self-evaluation, organization and transformation, goal setting and planning, information seeking, record keeping, self-monitoring, environmental structuring, giving self-consequences, rehears- ing and memorizing, seeking social assistance, and re- viewing. (paraphrased from Zimmerman, 1990) In addition to metacognition, motivation and behavior are considered to be components of self-regulated learning.

Metacognition 5 Developmental Testing Service, LLC

Finding Implication Reference Critical think- The Foundation for Critical Thinking ing (all lev- (http://criticalthinking.org) els) This model includes standards of thought (clarity, accuracy, relevance, logicalness, breadth, preci- sion, significance, completeness, fairness, and depth), which are applied to the elements of thought (shown on the left), resulting in the development of intellectual virtues. This is an iterative cycle, which is repeated as reasoning increases in complexity over the course of development. Critical thinking and creative thought are linked in this model (Paul & Elder, 2006).

Reflective The reflective judgment model is a very well-researched (Kitchener & Fischer, 1990; Kitch- judgment model of the development of thinking about thinking. ener & King, 1990; P. K. Wood, 1997) Double loop Argyris brings metacognition to the workplace by highlight- {Argyris, 1991 #14906; Argyris, learning (up- ing the importance of becoming conscious of the conse- 1982 #14625; per levels) quences of one’s actions. He uses real-life examples from the workplace to demonstrate methods for teaching upper level managers to think effectively about their thinking.

Metacognition 6 Developmental Testing Service, LLC

Metacognition research

Table 3: Metacognition and learning Finding Implication Reference Metacognitive skills develop. They can be learned. (Baer, Hollenstein, Hofstetter, Fuchs, & Reber-Wyss, 1994; Brown, Bransford, Ferrara, & Cam- pione, 1983; John H. Flavell, 1979; Ruth Garner & Alexander, 1989) Adults often fail to monitor their Adults can benefit from metacogni- (J. H. Flavell, 1981; Ruth Garner & thinking. tive training. Alexander, 1989; Glenberg, Wilkin- son, & Epstein, 1982) Students who have been taught It is possible to produce better (J. Borkowski, Carr, & Pressely, metacognitive (self-regulated learn- learners by teaching metacognitive 1987; Bransford, Sherwood, Vye, & ing) skills learn better than students skills. Rieser, 1986; Carr, Kurtz, Schnei- who have not been taught these der, Turner, & Borkowski, 1989; R. skills. Garner, 1990; Hascher & Oser, 1995; Mace, Belfiore, & Hutchinson, 2001; Pressley & Ghatala, 1990; B. J. Zimmerman & Schunk, 2001) Students with good metacognitive It is possible to produce better criti- (Bransford et al., 1986; Ewell- skills are better critical thinkers, cal thinkers, problem-solvers, and Kumar, 1999; Heath, 1983) problem-solvers, or decision makers decision makers by teaching meta- than students who are not. cognitive skills. People whose thinking is more Metacognition and cognitive com- (Swanson & Hill, 1993; Vukman, complex tend to have better meta- plexity are related. 2005) cognitive skills. In most people, the development of Since cognitive complexity and (Fischer & Pruyne; Fischer, Yan, & cognitive complexity progresses at metacognition are related, we might Stewart) different rates in different knowledge expect metacognition to be more domains, depending upon experi- advanced in more developed knowl- ence and learning in particular do- edge domains. mains. Cognitive development involves Because metacognitive skills in- (Bransford et al., 1986) both knowledge acquisition and volve the conscious structuring of (largely unconscious) knowledge knowledge, they are likely to be structuring. (If there is no knowledge more developed in areas of greater to organize, then there is no devel- knowledge. opment.) Content knowledge is more easily Learning environments should in- (Glaser, 1984) accessed in real-world situations if clude opportunities for students to students learn how new concepts reflectively apply new concepts and and procedures can function as tools in real-world contexts. tools for solving relevant problems. Both content knowledge and meta- Learning may be enhanced when (Bransford et al., 1986; Perkins, cognitive skills are essential for instruction (1) provides explicit con- 1987) learning. tent knowledge while (2) asking students to use metacognitive skills to operate on that knowledge. Metacognitive training can increase Increased self-confidence and a (McCombs & Marzano, 1990; students’ self-confidence and sense sense of increased personal re- Schunk, 1990) of personal responsibility for their sponsibility may provide motivation own development. for learning.

Metacognition 7 Developmental Testing Service, LLC

Finding Implication Reference Metacognitive training can increase Training in metacognitive skills may (Bandura, 1986; Hofer & Yu, 2003; students’ motivation to learn. enhance students’ sense of self Sperling, Howard, Staley, & DuBois, efficacy, thus increasing their moti- 2004) vation to learn. Moving adult learners to a point of Metacognitive strategies should be (Ruth Garner & Alexander, 1989) acknowledging that old routines no embedded in assignments and longer work as well as new, classroom activities across the curriculum at every level of instruction instructed ones takes time and many demonstrations of the superi- ority of the new routines. “Intelligent novices” can use general Once adults have gained expertise (Bruer, 1993; Mathan & Koedinger, metacognitive skills to figure out and learned how to use a range of 2005); Garner, 1989 #14918} how to obtain knowledge in an un- metacognitive skills in one domain, familiar domain. they can use some of their metacognitive skills to more rapidly learn in another domain. Students receiving intelligent novice Teachers or “intelligent tutors” can (Mathan & Koedinger, 2005) feedback acquire a deeper concep- support the use of existing meta- tual understanding of domain princi- cognitive strategies in new knowl- ples and demonstrate better transfer edge areas by providing feedback and retention of skills over time than that reminds students to employ students who do not receive such metacognitive strategies they have feedback. used in familiar knowledge areas. When students perceive an empha- Classrooms in which “covering the (Ames & AfIher, 1988) sis on mastery goals in their class- content” is emphasized over under- room, they report using more meta- standing can deprive students of the cognitive learning strategies. opportunity to learn and master learning skills. Use of concept maps helped adult Concept mapping, used well, is a (Daley, 2002) students develop thinking skills, useful metacognitive skill. promoted growth in understanding their learning processes, and fos- tered understanding of knowledge construction. Repeated experiences of dyadic Active engagement in thinking (Kuhn, Shaw, & Felton, 1997) discussions within the classroom about a topic enhances the quality improved reasoning skills (over con- of reasoning about that topic. trols). Informal learning is enhanced in Training in the use of metacognitive (Enos, Kehrhahn, & Bell, 2003) managers who employ a wide range strategies may increase informal of metacognitive strategies. learning in less metacognitively so- phisticated managers. Students in problem based learning Problem based learning environ- (Sungur & Tekkaya, 2006) classrooms have been found to ments may enhance metacognitive have higher levels of intrinsic goal skills relative to conventional in- orientation, task value, use of elabo- structional environments. ration learning strategies, critical thinking, metacognitive self- regulation, effort regulation, and/or peer learning compared with control-group students.

Metacognition 8 Developmental Testing Service, LLC

Finding Implication Reference Gifted learners have been found to Gifted learners, because they learn (Dresel & Haugwitz, 2005) employ fewer metacognitive strate- easily, may not need to employ gies than less gifted students. metacognitive strategies to excel. This could result in reasoning defi- cits in later life.

Table 4: Metacognition and critical thinking Concept Resources Critical thinking is the ability to (1) identify and formulate important questions and The Foundation for Critical problems; (2) gather and assess information; (3) test proposed conclusions Thinking against relevant criteria and standards; (4) think within alternative systems of (http://criticalthinking.org) thought, assessing their assumptions, implications and practical consequences; and (5) communicate effectively, without appealing to logical fallacies or manipu- lating others.

Table 5: Metacognition and reflective judgment Description Implications Source Reflective judgment is metacogni- (Hofer, 2004) tion. Beliefs about learning significantly Students whose reflective judgment skills (B. K. Hofer, 1999; Barbara K. impact the quality of learning strate- are more developed are likely to be better Hofer, 2000; Paulsen & Feld- gies and learning outcomes in gen- learners. man, 2005; Schommer, 1990, eral. 1993; Schommer, Crouse, & Rhodes, 1992)(Bråten & Stromso, 2005; Muis, 2007; P. Wood & Kardash, 2002)

Metacognition 9 Developmental Testing Service, LLC

Figure 1: Model of metacognitive skills and conditions for their development and use

Metacognition 10 Developmental Testing Service, LLC

Table 6: Metacognition and problem solving Finding Implications Source Effective problem solving depends Students whose reasoning skills are more de- Bransford, 1986 #6747; on the nature and organization of veloped are likely to be better learners. Chi, 1988 #982} the knowledge available to individuals. Psychological biases interfere with Psychological biases may be ameliorated (Duchon, Ashmos, & problem-solving through metacognition Dunegan, 1991)

Table 7: Metacognition and psychological biases Unconscious knowledge structuring takes place every time we learn something. These natural structuring processes have limitations that (1) are ineffective in some situations or (2) produce cognitive biases. Metacognition can be employed to counter these biases.

Note: Although it is clearly part of the subtext of much research in metacognition (especially critical thinking), the subject of psychological bias is rarely raised explicitly.

Bias Implications Source Overestimate People tend to overestimate the depth of their explanatory knowl- (Mills & Keil, 2004) explanatory edge (how well they understand concepts), which can produce deci- knowledge sion errors.

Table 8: Metacognition and mindfulness Finding or comment Source “Mindfulness begins by bringing awareness to current experience—observing and at- (Bishop et al., 2004) tending to the changing field of thoughts, feelings, and sensations from moment to moment—by regulating the focus of attention. This leads to a feeling of being very alert to what is occurring in the here-and-now. It is often described as a feeling of being fully present and alive in the moment.” (p. 232) “Mindfulness is further defined by an orientation to experience that …[involves] making a commitment to maintain an attitude of curiosity about where the mind wanders.” (p. 233) Mindfulness can be thought of as creating an optimally receptive state for new learning (Garland, 2007; Tho- and experience, increasing the likelihood that appropriate metacognitive skils will be mas, 2006) selected and employed. Mindfulness practice requires the activation of metacognitive knowledge, monitoring, (Wells, 2005) and control.

Metacognition 11 Developmental Testing Service, LLC

Figure 2: Metacognitive skills are an important component of interpersonal intelligence

Metacognition 12 Developmental Testing Service, LLC

Teaching and learning metacognition

Figure 3: Learning model for the workplace, emphasizing motivation, assessment, and metacognition (adapted from http://www.bbi.school.nz/philosophy/learningtolearn.html)

Table 9: Teaching and learning metacognitive skills: The research Finding Reference Metacognitive skills can be taught. (J. Borkowski et al., 1987; Bransford et al., 1986; R. Garner, 1990; Hascher & Oser, 1995) Metacognitive skills learned in one context are not automatically transferred to another (Ericsson, Chase, & context. Faloon, 1980)

Metacognitive skills are learned and applied more effectively in supported active learning contexts than in direct instruction contexts.

An application of metacognition to diversity issues. Could be useful as a starting place (Byars-Winston & for discussion. Fouad, 2006)

The role of formative assessment in self-regulated learning. (Nicol & MacFarlane- Dick, 2006)

Metacognition 13 Developmental Testing Service, LLC

Table 10: Teaching and learning metacognitive skills: Activities Activity Description Resource Identifying cogni- When students monitor their learning, they can become aware of (Nickerson, Perkins, & tive errors potential problems, including errors in encoding, operations, and Smith, 1985) goals. Errors in encoding include missing important data or not separating relevant from irrelevant data. Errors in operations include failing to select the right sub-skills to apply or failing to divide a task into subparts. For example, some math students will jump right to what they think is the final calculation to get the desired answer. Errors in goal seeking include misrepresenting the task and not understanding the criteria to apply. Problems with cognitive load include being unable to handle the number of sub-skills necessary to do a task, or not having enough automatic, internalized sub- skills. A good way to discover what kind of errors managers are making in their thinking processes is to have them unpack their thinking by explaining, step by step, how they are approaching a given task. This not only allows the instructor to diagnose possible errors, it provides managers with an opportunity to describe their thinking processes, which, by itself, develops their metacognitive abilities. Strategic learning Working in groups, have students generate questions about the (Simpson & Nist, 2000) content being learned and attempt to answer them. Have students produce written or verbal summaries of course texts. Require students to create examples, make analogies, and explain relationships between concepts. Then, engage students in the use of organizing strategies like concept maps, network representa- tions, and other graphic organizers. Self-regulatory These questions are designed to follow instruction on a particular (McInerney, McInerney, metacognitive topic and precede instruction on the next topic: (a) "What did I learn & Marsh, 1997) questions about this topic?" (monitoring) (b) "With what did I have difficulty?" (monitoring) (c) "What types of things can I do to deal with this difficulty?" (problem solving-planning) (d) "What specific action(s) am I going to take this week to solve any difficulties?" (planning) Before assigning this task for the first time, instructors should work through an example with the group.

Metacognition 14 Developmental Testing Service, LLC

Activity Description Resource Cromley summa- Skills need to be taught in the context in which they will be used. (Cromley, 2000) rizes the research For example, if students are learning to lead in a high security situa- findings, then tion, they need to practice confronting the kinds of issues that come translates them up in high security situations, not just "general" skills. into practical ac- Reading skills are subject-specific—understanding what you read in tivities for the a technical report does not guarantee that you will read well in an- classroom. This is other subject area. an excellent resource for adult Problem-solving skills in one subject area are different from those in educators. other areas. Problem-solving skills need to be taught separately for each subject. Since problem-solving skills do not automatically transfer from one subject area to another, instructors need to show students how to transfer these skills and give them lots of practice. Students need more and better mental models of the world in order to learn and master new information and skills. Thinking skills, such as inferring unstated facts, need to be taught explicitly in the classroom; they do not develop on their own (except in a very few students). These strategies need to be practiced over and over again. Most adult learners have a very limited number of strategies for understanding new material or solving problems. Teaching them more strategies can help them learn much better. Learning lasts when the student understands the material, not just memorizes it. Information needs to be presented in small chunks so that working memory can process it. Students need immediate practice to move information from working memory to long-term memory. It is impossible to remember without associating new information with what you already know. Authors provide (From the abstract) This paper proposes an expanded conception (Holton & Clarke, 2006) an overview of of scaffolding with four key elements: scaffolding and its (i) scaffolding agency – expert, reciprocal, and self-scaffolding; effects on learning. (ii) scaffolding domain – conceptual and heuristic scaffolding; (iii) the identification of self-scaffolding with metacognition; and (iv) the identification of six zones of scaffolding activity; each zone distinguished by the matter under construction and the relative posi- tioning of the participant(s) in the act of scaffolding.

Metacognition 15 Developmental Testing Service, LLC

Figure 4: Model of formative assessment and self-regulated learning

Metacognition 16 Developmental Testing Service, LLC

Table 12: Metacognition fads Fad Claim Comments Source

Learning Different people have Adjusting teaching style to accommo- (Cromley, 2000; Curry, 1990; styles different learning styles date different learning styles has not Stahl, 2002) (that can be measured been shown to impact learning. Moreo- with survey-type instru- ver, catering to learning preferences ments) and these have may limit versatility, having a negative an impact on learning. effect on learning over time.

Emotional Bringing intelligence to “Much of the data necessary for demon- {Landy, 2005 intelligence bear upon emotion. strating the unique association between #14980} EI and work-related behavior appears to reside in proprietary databases, pre- venting rigorous tests of the measurement devices or of their unique predictive value. For those reasons, any claims for the value of EI in the work setting cannot be made under the scien- tific mantle ” (p. 411)

Resources

Table 11: Metacognition curricula Resource Description Level Resources IDEAL problem The IDEAL problem solving system is “based on evidence Pre- (Bransford, Hay- solving: Identify, that successful problem-solvers actively attempt to (a) management nes, Stein, Lin, & Define, Explore, identify problems that others may have overlooked; (b) NashvilleRead, Act, and Look, develop at least two sets of contrasting goals for any prob- 1998) and Learn. lem and define them explicitly; (c) explore strategies and continually evaluate their relevance to their goals; (d) an- ticipate the effects of strategies before acting on them; and (e) look at the effects of their efforts and learn from them.” (p. 3) TV411—a na- Focuses, in part on helping adults manage their own learn- Pre- (D'Amico & tional television ing using metacognitive skills. management Capehart, 2001) series aimed at adult learners

Curricula to en- A handbook offering a number of activities designed for Pre- (Carman & hance students adult learners. Many of these are appropriate for pre- management Askov, 1994) higher order managers, especially if they are contextualized (free from cognitive skills Eric). Cognitive Strat- An instructional approach that has been shown to enhance Pre- (Scheid, 1993) egy Instruction learning by emphasizing the development of thinking skills management (CSI) and processes. The objective is to enable all students to become more strategic, self-reliant, flexible, and productive learners. Using critical A useful resource for students of critical thinking in the Pre- http://www.unisa thinking to gain workplace. management net.unisa.edu.au knowledge and /Resources/nursi understanding ng/Critical%20thi nking/Critical%2 0thinking.htm

Metacognition 17 Developmental Testing Service, LLC

Resource Description Level Resources Designing meta- Several research-based suggestions for creating learning (Lin, 2001) cognitive activi- activities with a strong metacognitive component ties

Table 12: Metacognition measures Resource Description Resources Lectical Reflective An online assessment of reflective judgment skill (for man- http://devtestservice.com Judgment Assessment agers). Provides reliable developmental scores in ¼ level (LRJA01) increments, as well as personalized feedback, including developmentally informed learning suggestions. Intended for personal use, coaching, and course evaluations. Reasoning About Cur- The RCI is an online assessment of the capacity to recog- http://www.umich.edu/~re rent Issues Test (RCI) nize and endorse statements that reflect the attributes of f- reflective thinking. It is not an assessment of reflective judg/reasoningaboutcurre judgment skill, and it is not intended as an assessment of ntissuestest.html individuals, but can be used to assess group trends. International Critical An assessment of students’ knowledge about critical think- http://www.criticalthinking. Thinking Basic Con- ing concepts—the extent to which they have learned these org/courses/critical_think cepts and Understand- concepts as they are presented in Elder and Paul’s model. It g_test1.cfm ing Online Test is not an assessment of critical thinking ability. Reliability information is not provided. International Critical A pen and paper assessment of critical thinking skill that can http://www.criticalthinking. Thinking Test be adapted to any subject area. Scoring is done by instruc- org/assessment//ICAT- tors, and is based on rubrics. info.cfm Motivated Strategies Based on the idea of self-regulated learning (SRL), this self- (Pintrich, Smith, Garcia, & for Learning Question- report instrument is “designed to assess motivation and use Mckeachie, 1993) naire (MSLQ) of learning strategies by college students. The motivation scales tap into three broad areas: (1) value (intrinsic and extrinsic goal orientation, task value), (2) expectancy (control beliefs about learning, self-efficacy); and (3) affect (test anxiety). The learning strategies section is comprised of nine scales which can be distinguished as cognitive, metacognitive, and resource management strategies. The cognitive strategies scales include (a) rehearsal, (b) elaboration, (c) organization, and (d) critical thinking. Metacognitive strategies are assessed by one large scale that includes planning, monitoring, and regulating strategies. Resource management strategies include (a) managing time and study environment; (b) effort management, (c) peer learning, and (d) help-seeking.” (abstract)

Metacognition 18 Developmental Testing Service, LLC

Resource Description Resources Structure of Observed Method for evaluating learning/reasoning outcomes that (Biggs & Collis, 1982) Learning Outcomes involves 5 levels of learning. Is often used to scaffold the (SOLO) development of scoring rubrics. Rubrics are high inference and inter-rater agreement is difficult to maintain. Nonethe- less, when inter-rater agreement is properly controlled, the rubrics are reliable enough for course evaluations (not usually for individual evaluation). 1. Pre-structural Learners acquire bits of unconnected information, which have no organization and make no sense. 2. Uni-structural Learners make simple and obvious connections, but show little evidence that their significance has been grasped. 3. Multi-structural Learners make a number of connections, but meta- connections between them are missed, as is their significance for the whole. 4. Relational Learners appreciate the significance of the parts in relation to the whole. 5. Extended abstract Learners make connections both within and beyond the subject area, showing they are able to generalize and transfer the principles and ideas.

Metacognition 19 Developmental Testing Service, LLC

References Ames, C., & AfIher, J. ( 1988). Achievement goals in the classroom: Students' learning strategies and motivation processes. Journal of Educational Psychology., 80, 260-267. Argyris, C. (1991). Teaching smart people how to learn. Harvard Business Review, May-June. Baer, M., Hollenstein, A., Hofstetter, M., Fuchs, M., & Reber-Wyss, M. (1994). How d expert and novice writers differ in their knowledge of the writing process and its regulation (metacognition) from each other, and what are the differences in metacognitive knowledge between writers of different ages? Paper presented at the Annual Meeting of the American Educational Research Association. Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Prentice-Hall, Englewood Cliffs, NJ. Biggs, J., & Collis, K. (1982). Evaluating the quality of learning: The SOLO taxonomy (structure of the observed learning outcome). New York: Academic Press. Bishop, S. R., Lau, M., Shauna, S., Carlson, L., Anderson, N. D., Carmody, J., et al. (2004). Mindfulness: A proposed operational definition. Clinical Psychology: Science and Practice, 7(3), 230-241. Borkowski, J., Carr, M., & Pressely, M. (1987). "Spontaneous" strategy use: Perspectives from metacognitive theory. Intelligence, 11, 61-75. Borkowski, J. G., & Thorpe, P. K. (1994). Self-regulation and motivation: A life-span perspective on underachievement at any point during the life span (B. P. D & A. E. L, Trans.). In B. J. Z. Dale H. Schunk (Ed.), Self-regulation of learning and performance: Issues and educational applications. (pp. 45-73). U Notre Dame, Notre Dame, IN, US: Lawrence Erlbaum Associates, Inc, Hillsdale, NJ, US. Bransford, J. D., Haynes, A. F., Stein, B. S., Lin, X., & NashvilleRead, N. (1998). The IDEAL workplace: Strategies for improving learning, problem solving, and creativity. Bransford, J. D., Sherwood, R., Vye, N. J., & Rieser, J. (1986). Teaching thinking and problem solving. American Psychologist, 41(10), 1078-1089. Bråten, I., & Stromso, H. (2005). The relationship between epistemological beliefs, implicit theories of intelligence, and self-regulated learning among Norwegian postsecondary students. British Journal of Educational Psychology, 75, 539–565. Brown, A. L., Bransford, J. D., Ferrara, R. A., & Campione, J. C. (1983). Learning, remembering, and understanding. In J. H. Flavell & E. M. Markman (Eds.), Carmichael's manual of child psychology (Vol. 1, pp. 77-166 ). New York: Wiley. Bruer, J. (1993). Schools for Thought: A science of learning in the classroom. Cambridge: MIT Press. Byars-Winston, A. M., & Fouad, N. A. (2006). Metacognition and multicultural competence: Expanding the culturally appropriate career counseling model. Career Development Quarterly, 54, 187-201. Carman, P. S., & Askov, E. N. (1994). Development of a curriculum to enhance adult learners higher order skills. University Park, PA: Pennsylvania State Universityo. Document Number) Carr, M., Kurtz, B. E., Schneider, W., Turner, L. A., & Borkowski, J. G. (1989). Strategy acquisition and transfer among German and American children: Environmental influences on metacognitive development. Developmental Psychology, 25, 765-771. Como, L. (1986). The metacognitive control components of self-regulated learning. Contemporary Educational Psychology, 11, 333-346. Cromley, J. (2000). Learning to think: Learning to learn. Washington, DC: National Institute for Literacyo. Document Number) Curry, L. (1990). One critique of the research on learning styles. Educational Leadership, 48, 50-56. D'Amico, D., & Capehart, M. A. (2001). Letting Learners Lead: Theories of Adult Learning and TV411. Focus on Basics, 5(B), 35-41. Daley, B. J. (2002). Facilitating learning with adult students through concept mapping. Journal of Continuing Higher Education, 50(1), 21-32. Dresel, M., & Haugwitz, M. (2005). The relationship between cognitive abilities and self-regulated learning: Evidence for interactions with academic self-concept and gender. High ability studies, 16(2), 201-218. Duchon, D., Ashmos, D., & Dunegan, K. J. (1991). Avoid decision making disaster by considering psychological bias. Review of Business, 13. Enos, M. D., Kehrhahn, M. T., & Bell, A. (2003). Informal learning and the transfer of learning: How managers develop proficiency. Human Resource Development Quarterly, 14(4), 369-. Ericsson, K., Chase, W., & Faloon, S. (1980). Acquisition of a memory skill. Science and Engineering Ethics, 208, 1181-1182. Ewell- Kumar, A. (1999). The influence of metacognition on managerial hiring decision making: Implications for management development Dissertation Abstracts International Section A: Humanities and Social Sciences, 59(10-A). Fischer, K. W., & Pruyne, E. Reflective thinking in adulthood: Development, variation, and consolidation. In J. Demick (Ed.), Handbook of Adult Development.

Metacognition 20 Developmental Testing Service, LLC

Fischer, K. W., Yan, Z., & Stewart, J. B. Adult cognitive development: Dynamics in the developmental web. In J. Valsiner & K. Connoly (Eds.), Handbook of developmental psychology (pp. 491-516). Thousand Oaks, CA: Sage. Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive-developmental inquiry. American Psychologist, 34(10), 906-911. Flavell, J. H. (1981). Cognitive monitoring. In W. P. Dickson (Ed.), Children's oral communication skills (pp. 35-60). New York: Academic. Garland, E. L. (2007). The meaning of mindfulness: A second-order cybernetics of stress, metacognition, and coping. Complementary Health Practice Review, 12(1), 15-30. Garner, R. (1990). When children and adults do not use learning strategies: Toward a theory of settings. Review of Educational Research, 60, 517-529. Garner, R., & Alexander, P. A. (1989). Metacognition: Answered and unanswered questions. Educational Psychologist, 24(2), 143-158. Ghatala, E. S. (1986). Strategy-monitoring training enables young learners to select effective strategies. Educational Psychologist, 21, 43-54. Glaser, R. (1984). Education and thinking: The role of knowledge. American Psychologist, 39, 93-104. Glenberg, A. M., Wilkinson, A. C., & Epstein, W. (1982). The illusion of knowing: Failure in the self-assessment of comprehension. Memory & Cognition, 10, 597-602. Hascher, T. A., & Oser, F. (1995). Promoting autonomy in the workplace--A cognitive-developmental intervention. Paper presented at the Annual Meeing of the American Educational Research Association. Heath, S. B. (1983). Ways with words: Language, life, and work in communities and classrooms. Cambridge, MA: Cambridge University Press. Hofer, B. K. (2004). Epistemological understanding as a metacognitive process: Thinking aloud during online searching. Educational Psychologist, 39(1), 43-55. Hofer, B. K., & Yu, S. L. (2003). Teaching self-reglated learning through a "learning to learn" course. Teaching in Psychology, 30(1), 30-33. Holton, D., & Clarke, D. (2006). Scaffolding and metacognition. International Journal of Mathematical Education in Science and Technology,, 37(2), 127-143. Kitchener, K. S., & Fischer, K. W. (1990). A skill approach to the development of reflective thinking. Contributions to Human Development, 21, 48-62. Kitchener, K. S., & King, P. M. (1990). The reflective judgment model: ten years of research. In M. L. Commons, C. Armon, L. Kohlberg, F. A. Richards, T. A. Grotzer & J. D. Sinnott (Eds.), Adult development (Vol. 2, pp. 62-78). New York: Praeger. Kuhn, D., Shaw, V., & Felton, M. (1997). Effects of dyadic interaction on argumentative reasoning. Cognition and Instruction, 15(3), 287-315. Lin, X. (2001). Designing metagognitive activities. Educational Technology, Research, & Development, 49, 23-40. Mace, F. C., Belfiore, P. J., & Hutchinson, J. M. (2001). Operant theory and research on self regulation. In B. J. Zimmerman & D. H. Schunk (Eds.), Self-regulated learning and academic achievement (pp. 39-65). Mahwah, NJ: Eribaum. Mathan, S., & Koedinger, K. R. (2005). Fostering the intelligent novice: Learning from errors with metacognitive tutoring. Educational Psychologist, 40(4), 257-265. McCombs, B. L., & Marzano, R. J. (1990). Putting the self into self-regulating learning: The self as agent in integrating will and skill. Educational Psychologist, 25(51-69). McInerney, V., McInerney, D. M., & Marsh, H. W. (1997). Effects of metacognitive strategy training within a cooperative group learning context on computer achievement and anxiety: An aptitude treatment interaction study. Journal of Educational Psychology., 89(4), 686-695. Mills, C. M., & Keil, F. C. (2004). Knowing the limits of one's understanding: The development of an awareness of an illusion of explanatory depth. Journal of Experimental Child Psychology, 87, 1-32. Muis, K. R. (2007). The role of epistemic beliefs in self-regulated learning. Educational Psychologist, 42(3), 173-190. Nickerson, R. S., Perkins, D. N., & Smith, E. E. (1985). The teaching of thinking. Hillsdale, NJ: Erlbaum. Nicol, D. J., & MacFarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199-218. Paul, R., & Elder, L. (2006). Critical thinking: The nature of critical and creative thought. Journal of Developmental Education, 30(2), 34-35. Perkins, D. (1987). Knowledge as design: Teaching thinking through content. In J. B. R. Sternberg (Ed.), Teaching thining skills: Theory and practice. New York: W. H. Freeman. Pintrich, P. R., Smith, D. A. F., Garcia, T., & Mckeachie, W. J. (1993). Reliability and predictive validity of the Motivated Strategies for Learning Questionnaire (MSLQ). Educational and Psychological Measurement, 53, 801- 813. Pressley, M., & Ghatala, E. S. (1990). Self-regulated learning: Monitoring learning from text. Educational Psychologist, 25, 9-33. Schloemer, P., & Brenan, K. (2006). Journal of Education for Business, 81-87.

Metacognition 21 Developmental Testing Service, LLC

Schunk, D. H. (1990). Goal setting and self-efficacy during self-regulated learning. Educational Psychologist, 25, 71- 86. Simpson, M. L., & Nist, S. L. (2000). An update on strategic learning: It’s more than textbook reading strategies. Journal of Adolescent and Adult Literacy, 43, 528-541. Sperling, R. A., Howard, B. C., Staley, R., & DuBois, N. (2004). Metaognition and self-regulated learning constructs. Educational Research and Evaluation, 10(2), 117-139. Stahl, S. A. (2002). Different strokes for different folks? In L. Abbedutto (Ed.), Taking sides: Clashing on controversial issuesin educational psychology (pp. 98-107). Guilford, CT: McGraw-Hill. Sungur, S., & Tekkaya, C. (2006). Effects of problem-based learning and traditional instruction on self-regulated learning. Journal of Educational Research, 99(5), 307-317. Swanson, L., & Hill, G. (1993). Metacognitive aspects of moral reasoning and behavior. Adolescence, 28, 711-735. Thomas, D. C. (2006). Domain and development of cultural intelligence: The importance of mindfulness. Group & Organization Management, 31(1), 78-99. Vukman, K. B. (2005). Developmental differences in metacognition and their connections with cognitive development in adulthood. Journal of Adult Development, 12(4), 211-221. Wells, A. (2005). Detached mindfulness In cognitive therapy: A metacognitive analysis and ten techniques. Journal of Rational-Emotive and Cognitive-Behavior Therapy, 23(4), 337-355. Wood, P., & Kardash, C. A. (2002). Critical elements in the design and analysis of studies of epistemology. In B. K. Hofer & P. R. Pintrich (Eds.), Personal epistemology: The psychology of beliefs about knowledge and knowing (pp. 231–260). Maswah, NJ: Lawrence Erlbaum. Wood, P. K. (1997). A secondary analysis of claims regarding the Reflective Judgment interview: Internal consistency, sequentiality and intra-individual differences in ill-structured problem-solving. In Higher education: Handbook of theory and research (Vol. 12, pp. 243-312). New York: Agathon. Zimmerman, B. J. (1990). Self-regulated learning and academic achievement: An overview. Educational Psychologist, 25(1), 3-17. Zimmerman, B. J., & Schunk, D. H. (2001). Reflections on theories of self-regulated learning and academic achievement. In B. J. Zimmerman & D. H. Schunk (Eds.), Self-regulated learning and academic achievement: Theoretical perspectives (pp. (pp. 289-307). Eribaum: Mahwah, NJ.

Metacognition 22