Embedding assessment in course design: the case of a research methods course

In my last post on designing a new approach to research methods within a masters course I adopted a framework based on the use of threshold concepts. The focus of this theoretical approach is in aiding students to move between three stages in their understanding of ‘threshold’ concepts together with attendant knowledge and skills. This is a transition between:

  • pre-liminal
  • liminal
  • post-liminal

Part of the difficulty involved in assessing student understanding and learning within this framework is the complexity involved in making explicit the level of prior learning at the beginning of the module and the subsequent trajectory which individuals follow as they transcend each level. Land and Meyer (2010) argue that assessment should focus on the conceptual difficulties which students face whilst accepting that these are a natural part of the learning process, as such they argue:

‘… if, as we maintain, the transformations occasioned by threshold concepts are important, and require a rather different way of looking at the curriculum, then it follows that such transformations will require a more nuanced and generative model of assessment to help us purposefully identify variation in progress and understanding between individual learners.’ (Land and Meyer, 2010: 63)

They make clear that each individual will move through the process of ‘liminal shift’ at different rates and in different ways and therefore it is important that assessment is used to guide and establish for both teacher and student what the difficulties and emerging understandings are within their thinking and learning. This requires both a framework for understanding learning and also a way of attempting to make that learning visible to both teacher and student. One example which is given as a medium for achieving this is the use of concept mapping, drawing on the work of Kinchin and Hay (2000) (another very detailed consideration of concept mapping is that developed by Novak, 1998). Again, Land and Meyer (2010) suggest that the use of tools such as concept mapping allow for four distinct advantages in relation to making student understanding and learning explicit:

  1. they allow us to uncover what each student knows rather than attempting to anticipate this as might be the case in any test scenario where questions are created by the tutor thereby bounding and anticipating ‘correct answers’;
  2. they allow students to demonstrate what knowledge they possess, and importantly how they have arranged that knowledge within their own minds;
  3. they become a narrative on the developing understanding of the student rather than a series of basic sequential snapshots;
  4. they allow us to see which concepts remain resistant to change within the minds of students, whilst at the same time understanding how interrelations between concepts may have changed within the students’ thinking.

Importantly, the use of various tools such as concept mapping encourage the introduction of self-explanation theory which, dialogue with the self to encourage externalising of learning (for further consideration see Chi et al (1989) http://chilab.asu.edu/papers/ChiBassokLewisReimannGlaser.pdf, and for some approaches to developing self-explanation see Hausmann et al (2009) http://www.lrdc.pitt.edu/pubs/Abstracts/HausmannSelf-Explanation.pdf). In all of this consideration is the idea of understanding learning through the lens of variation theory. Theory of Variation is based on the work of Marton and Booth (1997) which argues there is no single way to understand, experience or think about a particular phenomenon, an argument which is based upon a phenomenographic tradition. Tong (2012: 3) emphasises that:

‘In learning, individual students make sense of new concepts in different ways, according to their existing understandings and frameworks of knowledge. This requires teachers to engage closely with their students to grasp the variations in understandings and knowledge so they can take account of this diversity in structuring the learning activities in a lesson (Marton and Tsui, 2004).’

Therefore, importance is given to the ‘object of learning’ which might be a particular concept or area of knowledge. Once identified, the object of learning needs to be understood in relation to its critical features, i.e. its particular characteristics which differentiate it from any other object of learning. Consultation with students through some medium is important here so as to understand the variation in prior learning and understanding which exists prior to teaching. Having understood both student conceptualisations and the features of the particular object of learning, activities can be designed which emphasise the features which are important in aiding student understanding. Ways in which this can be developed include the use of comparison making by use of contrast, separation, generalisation and fusion. In relation to assessment the central aspect here is the ability for both the teacher and student to be able to externalise learning thereby allowing networks of knowledge and understanding to be interrogated to allow for targeted and appropriate pedagogic approaches and subject content. It is in this way the assessment can begin to take the form of an embedded and constant element of learning rather than a staged snapshot.

By bringing together threshold concepts, variation theory and the principle of assessment as learning more practical ways of tracking student understanding become possible. Approaches might include regular concept mapping, the use of reflective diaries, the development of portfolios and the possibility of more technologically led capture of self-explanation. One example of this might be the use of screen capture technology such as Jing (http://www.techsmith.com/jing.html) or Screencast-O-Matic (http://www.screencast-o-matic.com/ ). This is following on from the work of Carl Simmons at Edge Hill University who has already made very positive use of video feedback to students. Students can be asked to create a PowerPoint slide or other artefact explaining a particular object of learning within, or at the end of, a session including a narration captured whilst the screen capture software is running which is then sent to the tutor. This allows for the capture of levels of conceptual difficulty and understanding, occurring as the students are involved in their learning, and embedding the notion of self-explanation. The advantage of this approach is that each reflection is a maximum of five minutes long (Jing only allows 5 minute captures), thereby allowing tutors to judge levels of misconception or emergent understanding very quickly. If linked to the use of simple podcast feedback on the part of the tutor very rapid support can be offered in a rich feedback environment. In addition, where a large proportion of students are obviously still struggling with the concepts and knowledge involved the tutor gains a more nuanced understanding of where the problems exist and can therefore adjust the features within the object of learning to aid in more targeted teaching in future sessions.

The use of tools for capturing the trajectory of student learning allow individuals to reflect upon those areas where they have begun to feel more confident whilst also highlighting areas where understanding is still partial at best. It also allows for more in-depth and nuanced discussions between individual students and tutors as there is a richer vein of evidence for understanding student learning, misconceptions and potential next steps.

By bringing together threshold concepts, attendant bodies of knowledge and systems based on assessment as learning and variation theory, a clearer, deeper and more critical framework can be developed in support of student learning. If ‘progression’ is seen from the perspective of trajectories from pre-liminal to post-liminal states then informal assessment becomes more focused on discussions concerning concepts, knowledge, and the relational aspects between them. This then gives a far more focused approach to regular feedback to students from tutors and also offers a clear framework for ever more critical approaches to self-explanation. In addition, where formal assessment is required, in our own case through the production of a conference poster and a formal written assignment assessment, assignment outlines can use the same language as that adopted within the wider course, and the feedback provided likewise; in this sense whilst still essentially summative these formal assessments then become a far more obvious extension of the course itself.


Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, M. W., & Glasser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science, 13, 145-182.

Hausmann, R., Nokes, T., VanLehn, K., & Gershman, S. (2009). The design of self-explanation prompts: The fit hypothesis. Proc. 31st Annual Conference of the Cognitive Science Society. pp. 2626–2631.

Kinchin, I. & Hay, D. (2000) ‘how a qualitative approach to concept map analysis can be used to aid learning by illustrating patterns of conceptual development.’ Educational Research, 42(1), 43-57

Land, R. and Meyer, J.H.F. (2010) ‘Threshold Concepts and Troublesome Knowledge (5): Dynamics of Assessment.’ In Threshold Concepts and Transformational Learning, J.H.F. Meyer, R. Land and C. Baillie (eds), Rotterdam: Sense Publishers, 61-79.

Marton, F. & Booth, S. (1997). Learning and awareness. Mahwah, New Jersey: Lawrence Erlbaum Associates, Publishers.

Marton, F. & Tsui, A.B.M. (2004). Classroom discourse and the space of learning. Mahwah,

NJ: Lawrence Erlbaum Associates, Inc., Publishers.

Novak, J.D. (1998) Learning, creating and using knowledge: Concept maps as facilitative tools in schools and corporations. Abingdon: Routledge.

Tong, S.Y.A. (2012) ‘Applying the Theory of Variation in Teaching Reading.’ Australian Journal of Teacher Education, 37(10), 1-19 (accessed at http://ro.ecu.edu.au/cgi/viewcontent.cgi?article=1800&context=ajte)