OBJECTIVES

  1. To devise a procedure for workplace curriculum descriptions based on competencies and a (more or less comprehensive) set of EPAs.
  2. To devise a procedure for the identification of markers of progress for learners in workplaces, within health and teacher education, related to predefined competencies and EPAs.
  3. Establishing a literature review on workplace-based indicators of progress for visualisation

DESCRIPTION OF WORK AND ROLE OF PARTNERS

This Work Package defines what data are needed about trainees and are most valid for workplace-based assessment and feedback. The main participants in this WP are the educational institutes. They research the overall collection of data that will be captured in a portfolio system, i.e., a dossier of evidence collected over time, which demonstrates students’ and trainees’ development and performances. Technical universities UM and UR participate as linking pins to WPs 3, 4 and 5. UMCU will lead WP2.

Task 2.1 Competency-based workplace curriculum mapping with EPAs

Valid feedback and assessment are based on a clear description of competencies or EPAs that the trainee
is intended to develop. Existing curricula must be redefined, based on concepts of competencies and EPAs. Consequently, the first task in WP2 is to develop a framework for describing competencies and EPAs at the level of the curriculum. This procedure can be compared to a job analysis procedure. It requires broad local consensus of what constitutes the most important units of professional practice (Mulder, Ten Cate, Daalder, & Berkvens, 2010; Ten Cate, 2013). The procedure to be followed at multiple sites (general undergraduate medical programs in Utrecht, Berlin and San Francisco and postgraduate courses of anaesthesiology and nephrology and UMC Utrecht and Charité University Berlin, undergraduate veterinary courses in Utrecht and Budapest
and teacher education courses in Utrecht and Tartu) is directed at i) analysing the profession and identifying comprehensive EPAs (estimated number will be less than 50 per course) ii) connecting EPAs with the most suitable competency-framework (e.g. CanMEDS or ACGME for the medical domain; VetPro for the veterinarian domain and SBL for the teacher education domain). A one or two day kick-off meeting will be held with Partners in Utrecht or Amsterdam to align all Partners with this procedure; UCSF will be informed in a separate workshop in California. Regular teleconferences will be held to monitor progress.

Task 2.2 Identification of natural markers for professional development

The next step is to make a selection of realistic evidence at the workplace that represents the competencies or EPAs to be assessed and can serve as an input to the overarching portfolio system. This task implies that a wide sample will be taken from across workplace setting situations within the health and teacher professions. It will be ensured that the data directly map back to the competencies and EPAs aimed at in the curriculum (the deliverable produces by task 2.1 will be used for that). The tasks and evidence sought will be as authentic as possible. Some main methods of gathering data that will be explored are: video registrations, results of serious games (SIMMED) and simulation exercises which are directly relevant to workplace learning; such as direct observations of skills, observations of patient and clinical encounters, multi-source feedback. All markers of developmental progress should be suitable for justified entrustment decisions, even if used primarily for feedback purposes. t is fundamental that the tasks or evidence depicted in the portfolio align well with the EPAs of Task 2.1, at an appropriate level of generality.
The procedure to be followed is i) elaboration of EPAs, e.g. as described in Ten Cate & Young (2012) and Ten Cate (2013), ii) for each EPA relevant experts and learners are consulted to generate creative markers for progress (Dreyfus-based milestones as described in the Paediatrics Milestones project of the American Board of Paediatrics may serve as an example), iii) within each MVT domain (Med, Vet & Teach) consensus will be sought to arrive at generalisable weighted markers that will be suitable to translate to Learner Analytics input, i.e. the “student models” of WP4. A face to face meeting will be held at the start of this phase.

To ensure that the outputs of the work of WP2 are correctly translated and understood, an interpretivist requirements engineering methodology (Badii, 2008) will be deployed by UR which utilises a number of techniques to deeply engage the end-users (in this case the medical and educational instructors/teachers and trainees) using a variety of instruments and protocols such as interactive and video-based probes as well as interviews and questionnaires to elicit and conclude the stakeholders’ prioritised requirements according to pre-defined and known filters such as Emerging Technology Environment and State-of-Pedagogic-Practice.

Task 2.3 Literature review on workplace-based indicators of progress for visualisation

The results of Task 2.1 and 2.2 will be supplemented with a collaborative literature review on indicators of progress to inform entrustment decisions and feedback that is suitable to be represented in an LA-based visualisation. The visualisations aim to provide feedback to the trainees based on data collected, theoretical foundation and the variables. This overview will be included in this literature study supplemented with a survey among trainees and supervisors. Formalised entrustment decisions translated into so-called “digital badges” will be considered as an option (http://dmlcompetition.net/ Competition/4/badges-about.php).
The deliverable of this task will be used to select a set of visualisations that are created in Work Package 5 and implemented in Work Package 6.