Assessment and Teaching of 21st Century Skills

Assessment and Teaching of 21st Century Skills

Assessment and Teaching of
21st Century Skills
Claire Scoular
Assessment Research Centre
University of Melbourne
2nd April 2014
San Jose, Costa Rica
Outline
1. Background of ATC21S
2. ATC21S: The Latin American Chapter
3. Costa Rican results and outcomes
4. The future of the project in Costa Rica
2
SECTION 1 :
Background of ATC21S
CHANGE BEGINS WITH GLOBAL PARTNERSHIP
ATC21S is a joint effort to mobilize international,
educational, governmental and business communities to
bring 21st-century skills into the classroom.
4
ATC21S GOAL
To develop new assessment approaches
matched to new 21C skills and to advise
systems, schools, and teachers on the uses of
assessment data to help students develop
higher order performances
5
ATC21S PROCESS
Phase 1
Phase 2
Conceptualize
C21 skills and
education
output needs
Skill
identification
and
hypotheses
Phase 4
Phase 3
Task
development
Pilot studies
and trials
Phase 5
Dissemination
scale and policy
formation
6
SECTION 1 : Phase 1
Conceptualize C21 Skills
PREPARING STUDENTS TO SUCCEED IN THE
21ST CENTURY
• Changes in employment trends
• Increase in workforce by 2050
• For Latin America expected increase of 26% (Wang)
8
PREPARING STUDENTS TO SUCCEED IN THE
21ST CENTURY
“We are surrounded by
technology, and kids are digital
natives. They go into the
classroom, and they don’t see
the same use of technology in
their education as they might in
their outside lives. If education
doesn’t start to address that
issue, there will be a disconnect
in what kids see in their lives
growing up and when they get
out into the workplace, which is
one of the motivations for us to
join this project.”
e are surrounded by technology, and kids are
digital natives. They go into the classroom, and
they don’t see the same use of technology in
their education as they might in their outside
lives. If education doesn’t start to address that
issue, there will be a disconnect in what kids see
in their lives growing up and when they get out
into the workplace, which is one of the
motivations for us to join this project.”
— Michael Timms, National Project
Manager ATC21S for USA
• — Michael Timms, National Project
Manager ATC21S for USA9
TRANSFORM EDUCATION FOR THE 21ST
CENTURY
10
PHASE 1: CONCEPTUALIZING THE SKILLS
 Assemble experts
 Define methods
 Explore practical and
technical needs in the
classroom
 Create a new framework
11
12
SECTION 1 : Phase 2
Theoretical and Conceptual
Framework
SELECTED SKILLS
WAYS OF WORKING
AND THINKING
TOOLS FOR WORKING AND
LIVING IN THE WORLD
Collaborative problem
solving
ICT digital literacy
Personal and social responsibility
through social networking
14
PROCESS OF ASSESSMENT DEVELOPMENT
Definition of constructs
Collaborative Problem-solving
ICT Literacy
Formulation of developmental continua
Identify the dimension/s
Characterise the evidence
Draft three profile levels
Formulation of tasks
Teacher view
Cognitive laboratory
Pilot
Trial
15
COLLABORATIVE PROBLEM SOLVING:
Definition
Collaborative problem solving as a 21st Century Skill
may be seen as a combination of its component parts.
These are collaboration, problem solving and the new
technologies currently believed to characterise the
21st century
• Collaboration – working together with a common
goal
– Communication
– Cooperation
• Problem solving
– Knowledge
– Strategies
16
16
Purpose, functions and outcomes
• Why?
– When different expertise, knowledge or skills are
both needed and available
• What?
– Formulate and express ideas; argue and defend
– Value, review, evaluate
• Outcomes
– Problem solution
– Enhanced skills and knowledge
17
COLLABORATIVE PROBLEM SOLVING
FRAMEWORK
Collaborative
problem
solving
Cognitive
Social
Participation
Perspective
taking
Social
regulation
Task
regulation
18
Knowledge
buiilding
18
Social skills strands
Participation
Action
Perspective
taking
Adaptive
responsiveness
Social
regulation
Negotiation
Self evaluation
Interaction
Task completion
Audience
awareness
Transactive
memory
Responsibility
initiative
19
Cognitive skills strands
Task regulation
Knowledge building
Resource management
Knowledge acquisition
Information collection
Systematicity
Represents relationships
Ambiguity tolerance
Identifies consequences
Organisation
Goal setting
Hypothesises
20
ICT Literacy
Intellectual Capital
Social Capital
Producer
Consumer
21
SECTION 1 : Phase 3
Task Development
Process
Theoretical
Framework
Interpret
Task Design
Score
&
Intervention
Report
23
24
Laughing Clowns
25
25
Indicative Behaviours
Skill
Behaviour in task
Interaction
- Chatting with partner
Presence of chat before allowing partner to
make a move
Audience awareness
- Adapts contributions to increase
understanding for partner
Number of ball moves attempted before
stopping and waiting for partner to
respond
Reponsibility initiative
- Takes responsibility for progress for the
group task
Communicating with partner before first
half of balls have been used
Resource management
Realises that balls are meant to be shared
and only uses allocated half
Relationships - Identifies patterns between
information
Both students coming to agreement on
how their machine works
26
Balance Beam
27
Indicative Behaviours
Skill
Behaviour in task
Action
- Active in scaffolded tasks
Student A passes B a mass
Task completion – undertaking part of task
individually
Follows instructions, moves 100g to position
4
Responsiveness – responding to
contributions of others
Realising that some masses cannot balance.
If Student A resends 50 or 500, B returns it
immediately
Systematicity
- Implements possible solutions to a
problem
Trying out all combinations of masses on
different beam positions
Solution
- Correct answer
Number of successful balances achieved (3
optimum)
28
Skill progressions
SOCIAL
LEVELS
Level 4
PARTICIPATION
The student initiates and promotes interaction with their partner; sharing resources,
information and ideas.
The student perseveres in solving the task as shown by repeated attempts and/or
multiples strategies
Level 3
The student is actively participating in the task in familiar and unfamiliar contexts.
The student interacts with their partner more (i.e. responds to communication cues
and requests) and shares information and/or resources with partner.
Level 2
The student is actively participating in the task when it is familiar and/or scaffolded.
There is limited interaction with partner and this is mainly prompted by task
instructions.
Level 1
The student participates in the task but is working quite independently
Task activity is usually directed by system instructions
29
Reporting to Students: Learning Readiness
Real time reports based on
empirically developed learning
progressions. They will identify
a point of readiness to learn
for each student.
This type of report can be
linked to teaching
interventions associated with
readiness to learn as indicated
by the black bar in the spine of
the chart.
30
SECTION 1 : Phase 4
Pilot studies and trials
SECTION 1 : Phase 5
Dissemination & Policy
Formation
33
Dissemination
Resources will now be placed in the public
domain
Atc21s.org
34
Trialing Tools
Concept Checking
Cognitive Laboratories
Pilots
Trials
Teacher pedagogy workshops
35
SECTION 2 :
THE LATIN AMERICAN CHAPTER
Objectives and Benefits to Costa Rica
• Validate the assessment through pilot and trials
• Results could continue the agenda and lead Latin
America in 21st Century skills
• Contributing to large international project
• Gain ideas about future implementation of 21C
skills
• Evaluate curriculum
• Regional dialogues with peers as well as feedback
from stakeholders in the global initiative
37
ASSESSMENT OF COMPETENCIES OF THE 21ST
CENTURY
Three exemplary contributions:
– Language adaptions
– Customization of ICT Literacy Tasks
– Localized website
38
LANGUAGE ADAPTIONS
39
LANGUAGE ADAPTIONS
40
LANGUAGE ADAPTIONS
Componente cognitivo
Descripciones de nivel
Nivel 5 Resolución refinada de problemas
En este nivel, los estudiantes pueden reconocer errores previos y transferir y usar ese conocimiento en las
secciones más complejas de las pruebas. Reconocen qué recursos han causado errores previos y los evitan.
Están desarrollando confianza en el manejo de las pruebas y esto les permite corregir o superar las
respuestas/acciones de sus parejas. Los estudiantes comparten con sus parejas recursos apropiados y usan
estrategias consistentes que han identificado como efectivas en pasos anteriores de la solución del problema.
Todo el planeamiento y la mayoría de las estrategias implementadas se realizan dentro del contexto del
equipo.
Nivel 4 Trabajo eficiente
En este nivel, los estudiantes comienzan a resolver problemas de manera colaborativa y a usar estrategias de
cooperación más efectivas. En este momento pueden simplificar el problema y revisar varias veces la ejecución
de secciones previas de las pruebas para determinar si la regla es transferible a las páginas o secciones
siguientes. En este nivel, el intercambio de información entre estudiantes incrementa y se hace más eficiente.
Su secuencia y el juicio y la error acciones requieren pocos intentos y por lo tanto se concluyó en una cantidad
óptima de tiempo / intento. Las completan en un tiempo óptimo. Las acciones parecen estar bien pensadas y
planeadas, y cada acción parece tener un propósito. Pueden identificar causas y efectos y usar estrategias
adecuadas para lograr una vía de solución correcta. El estudiante dirige y guía a su pareja hacia acciones y
recursos apropiados.
Nivel 3 Planeamiento y ejecución estratégicos
En este nivel, los estudiantes pueden completar tareas de menor dificultad de manera correcta e
independiente. Son capaces de completar el último paso o sección y ayudar a su pareja en este proceso.
Incrementan su trabajo juntos al planear estrategias de trabajo, establecer objetivos y compartir sus recursos.
El estudiante tiende a enfocarse en los recursos relevantes y desecha otros que no fueron beneficiosos en
intentos anteriores. Son capaces de llegar a respuestas de común acuerdo cuando esto es necesario. En
pruebas con varias secciones el estudiante tiende a identificar estrategias y recursos exitosos y aplicar las
41
mismas reglas repetidamente. El estudiante continúa intensificando la investigación secuencial y el
ICT TASK CUSTOMIZATION
Poetry
42
ICT TASK CUSTOMIZATION
Antarctic Trek
43
LOCALIZED WEBSITE
44
Trialing the assessment tools
Trials 2011 (data collection CPS)
16 elementary and high schools
90 11-year-old students, 222 13-year-olds, and 188 15-year-olds
Trials 2013 (ICT and system testing)
9 high schools
776 students
45
SECTION 3 :
SPECIFIC COSTA RICAN
OUTCOMES
ALL COUNTRIES COMBINED
Valid CPS construct
Item Separation reliability
0.999
Person separation reliability
0.814
Regression variable
-0.182
47
ALL COUNTRIES COMBINED
Two separate dimensions (social & cognitive)
Social
Cognitive
Separation reliability
0.998
Person separation
reliability
0.754
0.770
Mean Latent Ability
0.560
-0.824
Correlation between
dimensions
0.788
48
Summary of data in Costa Rica
• 575 students were scored
• Sample only not representative
• Focus is to validate the test not to compare
countries
49
COUNTRY COMPARISON
Costa Rica
All other Countries
Mean of Latent Ability (StD Error)
-0.121 (0.021)
0.145 (0.01)
EAP-PV reliability (Similar to Cronbach’s alpha)
0.711
0.783
Reliability of Item Separation
0.993
0.999
The psychometric properties of the tasks in Costa Rica were
similar as in other countries, indicating that these tasks can
be completed in Costa Rica with little to no language,
cultural, or country specific bias.
50
Differences in the difficulty of the items were minor
or non-existent
Out of 150 items only two were more difficult for
CR than other countries
Hexagons - Answering the final answer correctly
Small Pyramids – Responding to questions from partner
Difficulty = Frequency
Could be explained by time constraints
51
Two items were easier for CR than other countries
Sunflower - presence of chat when growth appears
Survey (self assessment) - Replying to chats
However, there are many items that assess each skill
and these differences are based on single items so they
are not representative of overall ability in a specific skill
There is no particular task that is more difficult for Costa
Rica than the other countries.
52
SECTION 4:
IMPLICATIONS AND FUTURE
DIRECTIONS
Now we have…
Two skill constructs
Assessments to test skills
Share results for teaching
54
Now we need…
Strategies for teaching
Resources for teaching
PD for teaching
Developmental assessments
55
Domain specific or general curriculum
Expected to teach subject content and develop 21st
century skills
•
assess and intervene in both key learning area and general
capabilities
Required to think beyond discipline expertise
•
•
current teacher training mainly discipline-based pedagogy
most effective methods of teaching skills in one discipline
may not be same in another
56
EMBEDDED INTO CURRICULUM : Australian Curriculum
GENERAL CAPABILITIES
Literacy
ICT Capability
Numeracy
Critical and Creative
Thinking
Personal and Social
Capability
Ethical
Understanding
LEARNING
AREAS
Science
History
Math
Intercultural
Understanding
English
Australian Curriculum and Reporting Authority, ACARA (2013).
57
EMBEDDED INTO CURRICULUM : Singapore Curriculum
http://www.moe.gov.sg/media/press/2010/03/moe-to-enhance-learning-of-21s.php
58
EMBEDDED INTO CURRICULUM : Costa Rica
SEXUALIDAD
ESPAÑOL
MATEMÁTICAS
ÉTICA, ESTÉTICA Y CIUDADANÍA PRIMARIA
(MÚSICA, ARTES PLÁSTICAS, EDUCACIÓN FÍSICA,
CÍVICA Y ESTUDIOS SOCIALES)
VIDA COTIDIANA Y ARTES INDUSTRIALES
ÉTICA, ESTÉTICA Y CIUDADANÍA SECUNDARIA
(MÚSICA, ARTES PLÁSTICAS, CÍVICA, EDUCACIÓN FÍSICA)
CONVIVENCIA
59
SUSTAINABILITY
60
61
62
63
Poetry
64
Antarctic Trek
65
Why engage in the assessments?
Establish a baseline for
academic standards
Influence curricula at local, regional,
national and global levels
Generate new approaches to
Influence curricula at local, regional,
evaluation and teaching
national and global levels
and
66
Claire Scoular
[email protected]
Esther Care
[email protected]
www.unimelb.edu.au/arc
www.atc21s.org
www.actrc.org
© Copyright The University of Melbourne 2014