EARLI09 Symposium "Eliciting metacognitive thinking – exploring methods, data and techniques for analysis"
www.earli2009.orgAmsterdam 2009 | Fostering Communities of Learners | August 25-29
Symposium (Meta-cognition)
Eliciting metacognitive thinking – exploring methods, data and techniques for analysis
Participants
- Chairperson: Kate Wall, Newcastle University, United Kingdom
- Organiser: Kate Wall, Newcastle University, United Kingdom
- Discussant: Steve Higgins, Durham University, United Kingdom
- Papers: Barbara Hofer; Helen Askell-Williams & Mike Lawson; Margarida Romero; Marion Tillema, Huub van den Bergh, Gert Rijlaarsdam & Ted Sanders
The aim of this symposium is to consider the different ways in which data contributing to knowledge about metacognition is elicited and analysed. The key focus is an exploration of methods for data collection, representation, analysis and interpretation. It therefore raises questions about on-line and off-line conceptions, and considers the affordances and constraints of the qualitative and quantitative approaches presented through the different papers in the session, in order to stimulate debate about the implications of the different methods for future research in the field. This is of educational significance because meta-cognition is an important area for understanding and improving learning and instruction, yet there is no consensus about the contribution that the different approaches might make. It is based in the belief there is a need to re-examine some of the concepts and epistemological understandings commonly associated with the field particularly in terms of the implicit or explicit assumptions about metacognitive knowledge which the various approaches have. These include the assumptions that such knowledge is stable (Brown and Palincsar, 1982), conscious (Schmitt & Newby, 1981); statable (Flavell, 1979); interactive (Schraw & Moshman, 1995).
Epistemic Metacognition: Theoretical and Methodological Issues
- author: Barbara Hofer, Middlebury College, United States
The study of "personal epistemology, " the investigation of individual conceptions of knowledge and knowing, has been explored as a set of beliefs, a cognitive process, and a cognitive developmental trajectory. All are shown to have powerful influences on learning. Epistemic evaluations are also evoked metacognitively, although this has been underexplored. In this four-year multi-method study of epistemological development in adolescence we are exploring epistemic metacognition during online learning for a simulated science assignment. The study utilizes a think-aloud protocol, followed by a retrospective interview, and the completion of a set of survey instruments. A model for situating epistemic understanding as an aspect of metacognition will be presented, along with a discussion of methodological considerations and the cognitive demands of prompting metacognition among different adolescent age groups.
In a variety of learning situations, both formal and informal, individuals are likely to be metacognitively processing the credibility and veracity of new information. As they read or listen to the news, pursue a topic of interest online, attend a lecture, hear a political speech, or tackle an assigned reading for a course, they may be considering who and what to believe. The evaluation of knowledge claims, source of authority, and means of justification are aspects of "personal epistemology" (Hofer & Pintrich, 1997, 2002), a construct that refers to individuals' conceptions of knowledge and knowing. Personal epistemology has been addressed as a set of beliefs (e.g., Schommer, 1993), as an aspect of cognitive and intellectual development (Perry, 1970, King & Kitchener, 1991, Kuhn, Cheney, & Weinstock, 2000), and a set of fine-grained resources (Hammer & Elby, 2004), all of which have been shown to influence learning. Recent work also suggests that epistemic understanding may be activated metacognitively (Hofer, 2004a). The majority of research on personal epistemology is with college students and less is known about this process during earlier years. The purpose of our current work is to advance and refine an understanding of epistemological development during adolescence, exploring the process from multiple paradigms (Hofer, 2004b) with potential application in secondary education. Theoretically, this work is based on an expansion of metacognition to include epistemic processes.
Figure 1: Expanding Metacognition to Encompass Epistemic Processes
Metacognition – Existing Components
Metacognition – Epistemic Components
Metacognitive Knowledge
Knowledge about cognition and
strategies, tasks and contexts, self as
learner or thinker
Beliefs about the Nature of Knowledge:
Certainty of Knowledge
Tacit beliefs about whether knowledge is
certain or tentative and evolving
Simplicity of Knowledge
Tacit beliefs about whether knowledge is a
simple collection of facts or complex and
interrelated
Beliefs about Self as Knower
Epistemological Dispositions, Need for
Cognition, Need for Epistemic Closure, etc.
Metacognitive Judgments and Monitoring
Monitoring comprehension and learning,
judging task difficulty
(e.g., "Do I understand this?")
Beliefs about the Nature of Knowing:
Evaluating Source of Knowledge
("How do I know this?" "How does this fit
with my own experience?")
Determining Justification for Knowing
("Do I judge this to be credible?" "Is there
evidence to support this claim?" "Can I
reconcile theory and evidence?")
Self-regulation and Control of Cognition
(e.g., "Do I need to read this again?")
Regulating cognition during knowledge construction
Self regulation of knowing as affected by volition, interest, motivation, thinking dispositions, intellectual values, and beliefs
("Do I know what I need to know or do I need to know more?" "How will I go about this?")
(Hofer, 2004)
Empirically, this current study is a multi-method four-year project involving individual data collection with adolescents at two time points, one year apart, as described below.
Method
Participants
Students from central Vermont public schools, in 8th (n=31), 10th (n=28), and 12th (n=27) grades have been the initial participants in this study (and 6th graders will be included in the next round of data collection). Students were randomly selected and compensated for their involvement.
Procedure
Participants are assessed individually, with a think-aloud protocol during online searching, a retrospective interview about the search, and an epistemological development interview. Following this process, participants are sent a link to a web-based survey. Individuals are contacted one-year later for a follow-up study.
Measures
1. Epistemic metacognition: think aloud protocol and retrospective interview. Students are given a hypothetical task for a science assignment and asked to conduct a computer search for resources for their paper while thinking aloud, a method of tapping cognitive and metacognitive processes (Ericsson & Simon, 1993). Students are told in advance that at the end of the search they will be asked to explain what they know about the topic as well as to support their choices of the 3 most valuable sources. In a retrospective interview, students are asked to reflect back on their search, and are queried about epistemic aspects of the search (e.g., what makes a source credible).
2. Epistemic development interview. This is based on existing interview protocols, involving responses to ill-structured problems (King & Kitchener, 1994; Kuhn, 1991). Examples reflect current scientific issues, followed by prompts that assess epistemological positions (e.g., "Can you ever know for certain?" "What if experts disagree?"). Questions regarding justification, sources of authority, and the development of knowledge within science follow, as well as comparisons between science and history.
Web-based questionnaire:
1. Domain general beliefs, adapted from a review of models (Hofer & Pintrich, 1997)
2. Domain specific beliefs (Hofer, 2000), in both history and science
3. Motivation and interest scales from the MSLQ (Pintrich, Smith, Garcia, & McKeachie, 1993)
4. Need for Cognition (Cacioppo & Petty, 1982)
Data Analyses
Coding of the data from the online searching tasks has been guided by methodology for analyzing verbal data (Chi, 1997) and think-aloud protocols (Ericsson & Simon, 1993). Interview transcripts have been coded by a team of three raters. This approach includes coding for dimensions of epistemic understanding (certainty, simplicity, source, and justification) and for epistemic level (absolutist, multiplist, evaluativist) within each of the dimensions. Primary attention has been given to emergent and transitional levels of epistemic development beyond the traditional 3-level scheme (Kuhn, et al., 2000).
Discussion of Methodological Issues
The purpose of this paper is not to focus on current findings from the study but to address instead the methodological issues involved in a multi-method study of metacognition and epistemology in adolescents across a wide age span. Issues discussed will include 1) the relation among findings from the think-aloud protocol, the retrospective interview, the epistemic interview, and the surveys; 2) the cognitive demands of the think-aloud protocol and the expectations of the prompted metacognitive tasks; 3) developmental findings in regard to responses to these particular tasks. The learning from this project may be of use to researchers interested in methods for assessing either metacognition or epistemic understanding during adolescence, and the overall findings of the study are likely to help guide educators at the secondary level, particularly in regard to enhancing individual online learning.
Embedding cognitive and metacognitive strategy instruction into Year 9 science lessons.
- author: Helen Askell-Williams, Flinders University, Australia
- author: Mike Lawson, Flinders University, Australia
Over three school terms we collaborated with a class teacher to design and deliver, to a Year 9 science class, direct instruction and guided practice in applying cognitive and metacognitive strategies to promote subject-matter learning. Key design components included finding ways of: embedding cognitive and metacognitive instruction into subject-matter instruction; finding common ground between the teacher's and our mental models of desirable instructional goals; designing instruments to gather information about students' cognition and metacognition without unduly disrupting regular lessons; and representing, analysing and reporting the collected data to diverse audiences.
We created research tools for collecting data in ways that would not disrupt the flow of, but rather, would add value to, the subject-matter instruction. Tools included guided written response learning mini-protocols and prompted concept maps. We observed lessons, and collected students' work samples, academic results, and audio-taped interviews. In addition, as this study was a component of a large scale Australian Research Linkage Grant study, we had access to questionnaire data both from the intervention group and also from a broader sample of over 2000 students. We designed a framework to assess the quality of students' responses, and created student "learning capital" profiles to illustrate the variable quality of students' cognitive and metacognitive knowledge. Finally, we compare the intervention group with the external reference group, thus providing a broader foundation for interrogating the students' knowledge about effective learning actions.
The study provides evidence about instruments, such as mini protocols, that can be used in regular class settings for both instructional and data collection purposes. The evidence of student learning strategy knowledge provided by the evaluative framework and the student learning capital profiles can inform future instructional designs. The study also illustrates constraints and affordances in moving learning strategy knowledge from the periphery to integration with the science curriculum.
Learners need both content-based and process-based knowledge to successfully engage in problem solving for learning. However, students' knowledge varies widely about strategies that allow them to exert effective management of their learning. An increasing body of evidence points to the efficacy of explicitly teaching students strategies for cognitive elaboration of subject matter and for metacognitive regulation.
The research literature contains numerous examples of cognitive and metacognitive strategy instruction, including guiding protocols for elaborating text-based information; mnemonics for prompting elaborated note-taking; matrices and checklists to cue metacognitive activity; pre-worked-examples to scaffold the steps in problem solving; advance organisers to assist in setting up a mental model that has the potential to be elaborated; and reflective learning journals. However, standing beside the research literature there is also commentary indicating that the uptake of such procedures in regular classrooms is not as frequent as it might be.
Our interest lies with ways of integrating cognitive and metacognitive learning strategy instruction into the regular classroom, over extended periods of time, and in ways that are sustainable once researchers such as ourselves leave the scene. In this study we worked with a classroom teacher to design and deliver learning strategy instruction integrated with identified modules of the regular science curriculum such as body systems, matter and carbon compounds.
Aims
The aims of our study wereto investigate instructional tools and techniques for embedding cognitive and metacognitive strategy instruction into regular science lessons with a view to improving the quality of students' strategic learning actions and their science learning outcomes. to investigate facilitators and barriers to embedding explicit cognitive and metacognitive strategy instruction into traditional subject-matter-focused curricula.
Method
Over three school terms in 2008 we attended one or two science lessons per week in a class of 28 Year 9 students. In some lessons we observed the lesson and assisted students as requested with their class work. The purposes of these visits were to observe the delivery of the lessons, to build working relationships with the students, and to familiarize students with our presence in the classroom. In addition to our general classroom presence, on pre-arranged occasions, but still in regularly scheduled lessons, we delivered explicit learning strategy instruction and guided practice of cognitive and metacognitive strategies for learning, specifically designed to have application to the science subject-matter of the lesson (e.g. digestive systems; carbon compounds). Conversations with our collaborating classroom teacher indicated that our learning interventions in the class would need to be brief, and not unduly add to the workload of himself, or the students. Thus, we created purpose designed instructional tools, including written response learning mini-protocols and guided concept maps. These instructional tools also served as data collection instruments.
In collaboration with the class teacher, we gradually transferred the learning strategy instruction from us to him, with a view to embedding learning strategy instruction into his regular delivery of the science curriculum. We also conducted interviews with the students, and collected students' written learning mini-protocols, concept maps, work samples and academic results. To provide an external comparison, (and as part of a larger scale research Australian Research Council study) we also administered a questionnaire about cognitive and metacognitive strategic knowledge to over 2000 students across four schools.
Table One summarises the classroom interventions, instruments and the data collected
Table One
Intervention
Component
Pre-intervention data collection
Instructional Intervention
Post intervention data collection
1. Attention focusing
Guided Mini-protocol: Identifying Key Ideas
Direct instruction and
guided practice using mini-protocol
Guided Mini-protocol: Identifying Key Ideas
2. Organising
Guided Mini-protocol: Selecting & Sorting Key ideas
Direct instruction and
guided practice using mini-protocol
Guided Mini-protocol: Selecting & Sorting Key ideas
3. Relating & Integrating
Direct instruction and guided practice in diagramming and concept mapping
Guided Diagramming & Concept mapping
4. Summative subject-matter knowledge
Non-directed diagramming task
Non-directed diagramming tasks
5.Metaconitive monitoring
Guided Mini-protocol:
Assessing Understanding
Guided Mini-protocol:
Assessing Understanding
5. Living and Learning at School Questionnaire
Administered late 2007 to large student cohort
Administered late 2008 to large student cohort
6. In-class interviews
Students give verbal accounts of their learning actions
Students give verbal accounts of their learning actions
Students give verbal accounts of their learning actions
Results
We created student learning profiles that displayed students' relative strengths and weakness in learning strategy knowledge. For example, Figure One illustrates a learning profile that compares the profiles of two boys who differ in their self-reports the cognitive and metacognitive strategies that they employ.
Changes over time were observed in the quality of some, but not all, students' mini-protocols, concept maps and diagrams. It is more difficult to make claims for causal relationships between improved cognitive and metacognitive strategies and direct measures of student learning outcomes, such as scores on summative tests. An area of clear change was the instructional behaviour of the teacher, who progressively adopted learning strategy language and explicit instruction into his science lessons. Comparisons of questionnaire data with the large external reference group highlighted both the typical nature of the intervention class, and also the relatively low level of utilization of some key strategies for learning, (such as discussion and questioning) in identifiable sub-groups of students.
Significance
Even at the early stages of dissemination of the results of this study, it has become apparent that students, teachers and school administrators find our evidence from detailed inquiry into students' learning capital, and in-class methods for improving learning capital, to be invaluable for informing teaching classroom practices. Thus, this research contributes to knowledge about methodologies, and facilitators and barriers to, the implementation of instructional designs that include integrated subject-matter and learning strategy instruction into authentic classrooms.
Time Management Strategic Episodes (TMSE), an online assessment approach to metacognition in collaborative context
- author: Margarida Romero, Universite de Limoges, France
We aim to analyze metacognition in a long-term project activity. However, metacognition has been mostly analyzed and assessed in individual learning contexts within short-time tasks. More recent studies have starting considering metacognition in collaborative settings or as social product (Goos et al., 2002; Vauras et al., 2003) and even, as a socially shared product (Hurme, Palonen et Järvelä, 2006). The methodology we propose for long term metacognition assessment is based on the two first levels of the Stahl (2005) model, considering the individual level and the team level. We propose to study the two first levels through the Time Management Strategic Episodes (TMSE) as an interaction sequence including six phases: (1) the context or problem identification, (2) the action proposals, (3) the proposals evaluations and (4) the decision making phase. Modeling metacognition through the TMSE phases could contribute to (1) easily identify the qualitative contribution of each student beyond their quantitative participation, and (2) help to characterize the nature of the metacognitive collaborative decision making process.
Introduction
Time management is perceived as a major difficulty for students engaged on distance project based learning (Dewey, 1966; Kehoe et al., 1998). A project is a long term complex task, that engages specific high order skills (Perkins, 1992; Stites, 1998) as long-term planning, short and long term regulation and other metacognitive skills allowing to pursue their learning objectives during a long period of time. However, metacognition has been traditionally analyzed and assessed in individual learning contexts, within short-term tasks. More recent studies have starting considering metacognition in collaborative settings or as social product (Goos et al., 2002; Vauras et al., 2003) and even, as a socially shared product (Hurme, Palonen et Järvelä, 2006).
Methodology
In a collaborative context we can study metacognition through the interactions among the group members, introducing elicitation approaches as thinking aloud protocols (Azevedo et Cromley, 2004; Mengelkamp et Bannert, 2007) or other additive techniques as thinking templates (Wall et Higgins, 2006). In computer supported learning, tools could help learners to explicit their metacognition knowledge and experiences. Moreover, in distance learning context where group interaction is mainly based on written based tools as forums, emails or chats, there's no need to record and transcript interactions, because the group conversation already take places in log able tools. Evermore, the lack of physical shared context on distance learning promotes the elicitation of context during the interaction. In this context, an authentic online assessment (task dependent measure) based on interaction logs could be the basis for online metacognition assessment (Veenman et al, 2006), a more reliable approach than off-line approach, which analyzes metacognition before or after the task. Opting for the online approach we decided to analyze the online regulation processes during the chat activity; studying individually the behavior of each group participant. During all the data collection phase, we aimed not interfere in any decision making process.
The group has 4 weeks to upload their work into the learning management system (LMS). During this period the group was free to plan and regulate their work. They organized 9 chats, of which we have analyzed one of them. The methodology we propose for the analysis is based on the two first levels of the Stahl's model (2005), considering the individual level and the team level. We didn't consider the third level, the learning community level. We propose to study this two first levels through the Time Management Strategic Episodes (TMSE), that we could define as a sequence of interactions concerning a metacognitive collaborative decision making process. At the macro level, we visualize the project metacognition though the main metacognitive episodes, which are considered without eliciting every metacognitive process included in the interaction level. Analyzing this episode level we could visualize the main metacognitive collaborative strategies at a project level and going to the micro level only for specific episodes.
We defined the Time Management Strategic Episodes (TMSE) as an interaction sequence including six phases: (1) the context or problem identification, (2) the action proposals, (3) the proposals evaluations, and (4) the decision making phase. The added value of each phase is increasingly important. Consequently we consider that a context identification contribution (TMSE phase 1) is less relevant than an (2) Action proposition.
Findings
The analysis of individual contributions to TMSE collective regulation phases allows to consider relevant interindividual differences, both in the amount of contributions of each student (quantitative involvement), and in the TMSE phases they contributed more.
Individual involvement in TMSE phase's analysis indicates 3 involvement levels. Students less participative to group regulation are students E (t-test = -1, 95, dl = 6, p = 0.09) and F (t-test = -1, 8, dl = 6, p = 0.12). Students A (t-test = -0.19, dl = 6, p = 0.85) and C (t-test = 0.36, dl = 6, p = 0.73) contribution is mean. Students B (t-test = 0.9, dl = 6, p = 0.40) and D (t-test = 0.9, dl = 6, p = 0.40) have maximum levels of pertinent contribution to TMSE group regulation.
Students B and D have contributed in more pertinent TMSE phases. Student B is more involved in high added value TMSE phase 2 (action proposition) than student D. In the other hand, Student A has participated more importantly in phase 1 (Context identification).
TMSE phase's participation shows important inter individual participation variations. Within the students who have participated more in the group regulation process (B and D), student B has contributed more importantly to group objectives. Student B is the group leader with a clearly proactive attitude. Student D, having the same participation level in social regulation as B, had an important involvement in phase 3 (action propositions judgment), playing the role of student B evaluator.
The study aims to propose a methodology for metacognition online methodology, the TMSE. For this reason, the results of this observation, based in one chat interaction within a project team (N=6), must be considered as exploratory.
Theoretical and educational significance of the research
The significance of the research is based on the methodological approach to the elicitation of metacognition in collaborative long-term activities. Modeling metacognition through the TMSE phases could contribute to (1) easily identify the qualitative contribution of each student beyond their quantitative participation, and (2) help to characterize the nature of the metacognitive collaborative decision making process.
During the activity, the use of TMSE could help tutors and teachers to identify learner's qualitative contribution to the group work and contribute to better evaluate the individuals in collaborative tasks.
The role of metacognition during writing in mother tongue and foreign language
- author: Marion Tillema, Utrecht University, Netherlands
- author: Huub van den Bergh, Utrecht University, Netherlands
- author: Gert Rijlaarsdam, University of Amsterdam, Netherlands
- author: Ted Sanders, Utrecht University, Netherlands
Recent work by Tillema, Van den Bergh, Rijlaarsdam & Sanders (2008) on the topic of writing in a mother language, suggests that a high degree of metacognitive control allows pupils to adapt the way they execute a writing task to the specific demands of the task in question. In this presentation, we apply this notion of adaptation, working under metacognitive control, to the task of writing in a foreign language. 10 ninth-grade pupils each wrote two argumentative essays in their mother tongue (L1: Dutch) and two essays in a foreign language (FL: English) under think aloud conditions. The think aloud protocols were coded by means of a coding scheme which included various types of 'metacognitive' categories, such as self instructions, goal setting and metacommentary, as well as 'cognitive' categories, such as generating ideas, formulating and revising. All writing processes, which consist of combinations of cognitive activities, were modelled by means of a polynomial model. The essays were rated by five raters on four criteria of text quality. For writing in the L1, we found that pupils with a high degree of metacognitive control (based on their think aloud protocols) are better able to work adaptively: if necessary, they will use different writing processes for different tasks. This is reflected in the correlation between their two writing processes. For writing in a foreign language, we found that, in general, pupils showed less metacognitive activities than in the L1. In addition, it seemed that, the larger the loss of metacognitive control in the FL, the larger the loss of text quality (as compared to the L1). It seems that a higher degree of metacognitive control enables pupils to adjust their writing processes to the specific demands of FL writing tasks.
Writing requires the handling of several cognitive activities, such as generating ideas, formulating and revising (Flower & Hayes, 1980; Hayes & Flower, 1980; Hayes, 1996). These cognitive activities are considered to be governed and orchestrated by a metacognitive capacity (Braaksma, Rijlaarsdam, Van den Bergh and Van Hout-Wolters, 2004).
Carey and Flower (1989) stressed the recursive nature of writing: while performing a writing task, writers recurrently need to construct mental representations of their texts. As the text develops, these representations change.
This was empirically investigated by Rijlaarsdam and Van den Bergh (1996), Van den Bergh and Rijlaarsdam (2001) and Van Weijen (2008), who took the writing process into account in their explanations underlying text quality. Their explanations put the moment at which cognitive activities are applied central. For example, Van den Bergh and Rijlaarsdam (2001) found that the effectiveness of applying the activity of formulating (and therefore not applying any other activity) is at its peak at about one third of the writing process. Van Weijen (2008) found that the effectiveness of generating, on the other hand, decreases from the moment the writing process begins. Following Van den Bergh and Rijlaarsdam (2001) and Van Weijen (2008), we view the writing process as the distribution of cognitive activities (such as generating ideas, formulating, and revising) over the time that the writing task takes up.
We present the results of a study in which 10 ninth-grade pupils wrote two argumentative essays in their mother language (L1: Dutch) and two essays in a foreign language (FL: English) under think aloud conditions. An elaborate coding scheme was used for coding the think aloud transcripts, which included various types of 'metacognitive' categories, such as self instructions, goal setting and metacommentary, as well as 'cognitive' categories, such as generating ideas, formulating and revising. All writing processes (i.e. the way cognitive activities are distributed over time) were modelled by means of a polynomial model. The essays were rated by five raters on four criteria of text quality.
The results show a complex interaction between metacognitive capabilities, writing skill and adaptive behaviour. We found that those writers who performed well (text quality scores of half a standard deviation or more above the average) on both tasks had completely different writing processes during both tasks (r
When (unaccomplished) learners of a foreign language write a text in a foreign language, the text quality deteriorates as compared to the quality of texts the same writer produced in his or her mother language (Harris & Silva, 1993). But the extra burden of a foreign language does not have the same effect on every writer. Whereas for some writers text quality is hardly affected by having to write in a foreign language, other writers may witness a major deterioration in the quality of their texts (Schoonen, Van Gelderen, De Glopper, Hulstijn, Simis, Snellings & Stevenson, 2003).
Our results seem to offer an explanation for these differences in performance loss. We found that, in general, pupils showed less metacognitive activities in the FL than in the L1. In addition, the results of our study suggest that, the larger the loss of metacognitive control in the FL, the larger the loss of text quality (as compared to the L1). It seems that a higher degree of metacognitive control enables pupils to adjust their writing processes to the specific demands of FL writing tasks.
