The WATCHME project aims to improve workplace-based feedback and assessment and professional development by means of learning analytics. It aims to solve the efficiency and quality problems of workplace-based feedback and assessment by means of an electronic portfolio system, that is enhanced with i) student models that monitor the learners’ competency development, ii) a Just-in-Time feedback module, and iii) visualisation tools that inform learners, teachers and institutes just in time as well as on an analytical level. The project structure follows an iterative approach from understanding the user´s needs and context to the specification of requirements, the development and test of operational prototypes and finally to a tool improving LA and workplace–based feedback.
Impact: Quality and efficiency of workplace-based feedback and assessment will increase, as well as the development of professional expertise throughout the domains.
To achieve the objectives, the scientific and technical activities are divided in 7 work packages (WPs) with test beds in three professional fields at universities throughout Europe. The overall strategy of the work plan within WATCHME follows an iterative approach using various cycles of a waterfall approach-like nature. The methodology is realised through the following interaction of WPs:
- WP1 provides the financial and administrative project management including risk management, quality assurance and progress monitoring.
- WP2 will start the development cycle by providing the necessary information from the educational specialists in the domains of human medicine, veterinary medicine, and teacher education, on the competency framework and relevant markers for work-place based assessment
- WP3 takes this information and initially performs the translation into functional and non-functional requirements necessary to specify the overall framework architecture. It will also provide and/or develop the work place-based assessment instruments necessary to collect data from the learning environments, integrate the portfolio system with modules on student model (WP4) and visualisation (WP5) and manage the learning-analytics platform and portfolio system during the trials (WP6).
- WP4 deals with the development of the student model, which needs to be capable of addressing and complying with the domains as per WP2. The requirements and architecture specification of WP3 will serve as a starting point and results will be delivered back to WP3 for integration with the portfolio system before being evaluated in WP6.
- WP5 addresses the visualisation and Just-in-Time feedback aspects of the WATCHME project. Again, taking the requirements and specification of WP3 as a starting point, this WP will develop the visualisation and feedback modules ready for integration into the LA enhanced portfolio system to be tested in WP6.
- WP6, then, performs the various evaluation cycles of the WATCHME project. It is planned that three major evaluation cycles will take place, midway and towards the end of the project, as well as continuous assessment of the development. Evaluation cycles will involve formative and summative evaluation approaches within the three learning domains human medicine, veterinary medicine, and teacher education. The results of the evaluations are fed back into the technical WPs WP3-5, which then start the next iteration in the methodology.
- WP7 finally provides the dissemination and exploitation work for the WATCHME project, which are important pillars for the impact realisation for the WATCHME results.
Continuing education at the workplace has seen increasing demand as a crucial means of acquiring the requisite professional knowledge and skills. Electronic Portfolios (e-portfolios) for workplace training provide a global view of each trainee’s progress but offer no dynamic feedback to exploit the rich learning assessment data that could be analysed to support responsive adaptation for more efficient and rewarding training.
WATCHME effectively deploys Learning Analytics (LA) tools to deliver personalised learning; supporting learner empowerment, enhanced Quality-of-Experience, flexibility and mobility plus efficiency gains through workplace-based feedback and assessment. This will be achieved by developing, implementing and evaluating a mobile, electronic portfolio-based system that utilises LA and student models to provide data-driven and model-based tailored feedback via a visualisation dashboard. The multi-sourced integrated deployment of authentic workplace data including self-reporting, narratives, qualitative and quantitative data e.g. videos and serious educational games enables the WATCHME trainees to exercise and thereby sharpen their learnt skills by responding to realistically situated judgment calls in a motivational and safe environment.
A Multi-entity Bayesian Network will be used for the student models, enabling an aggregation of data and tailored Just-in-Time feedback, visualised via intuitive interfaces. The close cooperation of educational researchers and ICT developers will deliver a tool to map critical professional tasks linked to their requisite competencies and markers for feedback and assessment in the workplace integrated with an easy access electronic portfolio system for data collection and LA-driven tools to inform trainees, teachers and supervisors about the trainee’s progress and deliver personalised Just-in-Time feedback with intuitive visualisation tools.
Progress of WATCHME – a brief status overview
Within the first year the consortium managed to develop a curriculum mapping procedure based on EPAs that is valid and feasible at multiple sites in the WatchMe project and its content was validated with experts on site. Workplace-based markers have been defined and the eportfolio architecture and configuration was made to start off the implementation of the eportfolio system including a description of the architecture design and assessment tools. A student model specification has been made with an overview of options for algorithms. The second year focussed on the next step within the iterative approach and the WATCHME consortium finalized the requirements and implemented them in prototypes. The first test whether the prototypes meets user and organizational requirements have been started, which will last in the next and final year of the project. The prototypes will be further developed and improved.