Project report: AI in Assessment and Feedback (TeacherMatic Pilot)
Project co-managed by:
Adele Cushing (Senior Digital Education Consultant, Birkbeck)
Sarah Sherman (Director, Bloomsbury Learning Exchange)
Sponsored by:
Brett Lucas (Head of Digital Education, Birkbeck)
Participating institutions: University of London (Worldwide); London School of Hygiene and Tropical Medicine; City St George’s; University College London; Birkbeck, University of London
Executive summary
The Bloomsbury Learning Exchange (BLE) and Birkbeck jointly participated in a Jisc project on the use of AI in assessment and feedback during 2025-26. Up to 50 participants from faculties and professional services across five institutions used TeacherMatic, an education-focused AI tool built on OpenAI technology, for exploratory use. The pilot identified benefits in using TeacherMatic as a controlled, education-focused tool to support feedback drafting, rubric development, question generation, AI literacy and cross-institutional shared practice. Key risks and constraints relate to accuracy at higher levels of study, confidence in assessment reliability, the need for continued human academic judgement, uncertain time savings, limited evaluation data and practicalities of Moodle integration. The strongest case for future use is therefore a human-led support tool rather than an autonomous assessment system.
_____________________________
Project overview
Introduction
This cross-organisational Jisc project was co-led by the Bloomsbury Learning Exchange (BLE) and Birkbeck, University of London to explore TeacherMatic’s potential in supporting higher education assessment and feedback activity. 50 user licences were distributed across a variety of educational levels (levels 4 through 7/8) and assessment types in the above mentioned institutions (see Appendix) e.g., multiple choice, short-answer questions, long-answer essays, within academically agreed legislative and ethical frameworks. Participants freely gave time, with some citing a desire to further understand AI generally, and others keen to evaluate its role in assessment.
Staff perspectives were gathered through a benchmarking survey, focus-group-style project meetings (three times a term), an evaluation form and podcast interviews. Of the 50 licence users, 35 completed an initial baseline survey, three quarters of whom had previously used AI in teaching. The survey indicated a 50/50 split for previous use of AI in assessment and feedback, such as drafting questions, rubrics and formative feedback. Participants who had previously used AI predominantly referred to ChatGPT followed by Copilot for administrative support and rubric generation.
Project Management
The central project team consisted of four Birkbeck colleagues and the BLE Director (the 'Project Leads'). This small team met fortnightly for 30 minutes to check on progress and plan updates and information for participants.
Each partner institution had a pilot lead, with whom the Project Leads directed all communications. The pilot leads were responsible for identifying and liaising with the participating academics from their institution. In some cases, pilot leads were also participants.
Meetings and Communication
All participants met three times a term; as the year progressed, additional participants joined to coincide with their use of the platform. Each meeting followed a similar format: a reminder of project aims and intended outputs (introducing the project to newcomers and reinforcing to existing participants); round-table updates on pilot participant progress, eliciting feedback to pass to TeacherMatic developers and Jisc, plus sharing practice with future participants. All meetings were recorded, and notes and slide-decks were circulated to all participants via the pilot leads.
Documentation and project outputs were stored in a shared One Drive folder for all participants. The Project Leads used an MS Teams space to centralise communication between themselves. Each Pilot Lead was also encouraged to provide a Teams space for their institution's participating academics.
Training
A basic training plan was developed as follows:
Details about the training sessions:
Lessons learned from running these sessions have been consolidated into a Training Plan (available here); this can be applied to the adoption of any GenAI platform.
Moodle Integration
An academic year is a long time in AI development. Features and workflows that were challenging at the start improved during the project, benefiting participants who joined later. The pilot community shared practice around data processing and ethics, assessment workflows and content generation. TeacherMatic developers also enhanced the system in response to feedback. Findings from project meetings and staff evaluations completed by 10 users identified the following benefits, risks, and constraints.
Benefits
The project demonstrates that TeacherMatic succeeds as a usable educational AI platform but has not yet earned trust as an assessment tool for high-stakes academic work in university education. Staff recognise value in content generation, feedback drafting, and efficiency gains, but concerns around marking validity, academic judgement, consistency, and subject expertise currently limit adoption. The strongest opportunity is in supporting human assessors rather than automating assessment decisions.
There is an opportunity to use the platform as a tool to support new career academics in developing feedback styles and a consistency in delivery tone. This could also be helpful as staff development for all who teach.
This project was initiated to develop local understanding of generative AI tools. Participating in a Jisc-supported project gave participating institutions an opportunity to explore new technology and raise their profile in a national programme through published outcomes. Running the project as a joint activity meant we were able to extend levels of study, discipline areas and involved many more collaborators than if one institution had worked alone.
Despite doubts regarding AI reliability generally, research suggests students are still using it. The importance of education to support their development of critical thinking around such systems should therefore not be ignored. Key to this is the understanding of AI functionality and limitations by all staff. TeacherMatic’s controlled hosting environment may continue to be useful in developing AI literacy, further investigation into assessment workflows and facilitating discussions on ethical use and critical thinking. There is an opportunity to integrate the tool into AI literacy and ethics workshops as well as existing staff development such as digital skills courses and teacher training. Also, although this project focused on assessment and feedback, TeacherMatic has a broader suite of tools for teaching, assessment, feedback, content creation, administration and management.
Although the project indicated that TeacherMatic has potential value as a managed, human-led support tool for assessment and feedback, pilot evaluation (10 responses) was inconclusive in justifying a long-term investment in the tool at individual member institutions. As a result, Birkbeck will lead an extended investigation through 2026-27 to build on enthusiasm from their participants, and continue to explore the platform beyond the assessment generators.
Project Outputs
The following recommendations are for both future users of TeacherMatic and for the developers of the platform itself.
Some resources that have informed this work:
Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö. and Mariman, R. (2024) Generative AI Without Guardrails Can Harm Learning: Evidence from High School Mathematics. The Wharton School Research Paper. Available at: https://doi.org/10.2139/ssrn.4895486 (Accessed: 19 June 2026).
Benzo, R. (2026) A (Double) Dual Reality Framework. Presentation delivered at AI and the Future of the University: Critical Perspectives, organised by Birkbeck Law School, hosted by Craig Reeves and Victoria Ridler, Birkbeck, University of London, London, 5 June.
Fazi, M.B. (2019) ‘Can a machine think (anything new)? Automation beyond simulation’, AI & Society, 34(4), pp. 813–824. Available at: https://doi.org/10.1007/s00146-018-0821-0 (Accessed: 19 June 2026)
Pearson, J. (2026) How do we make sure things in FE don't just "get worse"? Association of Colleges. Available at: https://www.aoc.co.uk/news-campaigns-parliament/news-views/aoc-blogs/how-do-we-make-sure-things-in-fe-dont-just-get-worse (Accessed: 19 June 2026)
Schultze-Gerlach, T., Kumar, V. S., McKeown, G., O'Connor, P. A., Rychlowska, M., & Sparemblek, K. (2024). Using Large Language Models to augment (rather than replace) human feedback in higher education improves perceived feedback quality. Pre-print. Queen's University Belfast. https://doi.org/10.31234/osf.io/tvcag (Accessed: 25 June 2026)
Welsh, K. and Milne, L. (21 Jan 2026) “Cautious, Transparent and Responsible: Shaping AI for Education at the University of Chester”, TeacherMatic Webinar
Appendix
Summary of TeacherMatic pilot module representation by organisation, faculty/school or area, example module titles and level of study.
Adele Cushing (Senior Digital Education Consultant, Birkbeck)
Sarah Sherman (Director, Bloomsbury Learning Exchange)
Sponsored by:
Brett Lucas (Head of Digital Education, Birkbeck)
Participating institutions: University of London (Worldwide); London School of Hygiene and Tropical Medicine; City St George’s; University College London; Birkbeck, University of London
Executive summary
The Bloomsbury Learning Exchange (BLE) and Birkbeck jointly participated in a Jisc project on the use of AI in assessment and feedback during 2025-26. Up to 50 participants from faculties and professional services across five institutions used TeacherMatic, an education-focused AI tool built on OpenAI technology, for exploratory use. The pilot identified benefits in using TeacherMatic as a controlled, education-focused tool to support feedback drafting, rubric development, question generation, AI literacy and cross-institutional shared practice. Key risks and constraints relate to accuracy at higher levels of study, confidence in assessment reliability, the need for continued human academic judgement, uncertain time savings, limited evaluation data and practicalities of Moodle integration. The strongest case for future use is therefore a human-led support tool rather than an autonomous assessment system.
_____________________________
Project overview
Introduction
This cross-organisational Jisc project was co-led by the Bloomsbury Learning Exchange (BLE) and Birkbeck, University of London to explore TeacherMatic’s potential in supporting higher education assessment and feedback activity. 50 user licences were distributed across a variety of educational levels (levels 4 through 7/8) and assessment types in the above mentioned institutions (see Appendix) e.g., multiple choice, short-answer questions, long-answer essays, within academically agreed legislative and ethical frameworks. Participants freely gave time, with some citing a desire to further understand AI generally, and others keen to evaluate its role in assessment.
Staff perspectives were gathered through a benchmarking survey, focus-group-style project meetings (three times a term), an evaluation form and podcast interviews. Of the 50 licence users, 35 completed an initial baseline survey, three quarters of whom had previously used AI in teaching. The survey indicated a 50/50 split for previous use of AI in assessment and feedback, such as drafting questions, rubrics and formative feedback. Participants who had previously used AI predominantly referred to ChatGPT followed by Copilot for administrative support and rubric generation.
Project Management
The central project team consisted of four Birkbeck colleagues and the BLE Director (the 'Project Leads'). This small team met fortnightly for 30 minutes to check on progress and plan updates and information for participants.
Each partner institution had a pilot lead, with whom the Project Leads directed all communications. The pilot leads were responsible for identifying and liaising with the participating academics from their institution. In some cases, pilot leads were also participants.
Meetings and Communication
All participants met three times a term; as the year progressed, additional participants joined to coincide with their use of the platform. Each meeting followed a similar format: a reminder of project aims and intended outputs (introducing the project to newcomers and reinforcing to existing participants); round-table updates on pilot participant progress, eliciting feedback to pass to TeacherMatic developers and Jisc, plus sharing practice with future participants. All meetings were recorded, and notes and slide-decks were circulated to all participants via the pilot leads.
Documentation and project outputs were stored in a shared One Drive folder for all participants. The Project Leads used an MS Teams space to centralise communication between themselves. Each Pilot Lead was also encouraged to provide a Teams space for their institution's participating academics.
Training
A basic training plan was developed as follows:
- An Initial introductory training to the TeacherMatic platform.
- Targeted training sessions were organised and delivered for all pilot participants, with focus on specific AI generators featured in the project including Rubric generator, Multiple choice questions generator, Feedback generator (two versions), Assignment generator, and Worksheet generator.
- Individual training sessions were delivered upon request for specific use/purpose.
- Additional support documents, TeacherMatic videos and recordings were circulated after training sessions.
Details about the training sessions:
- Two initial introductory training sessions were provided by TeacherMatic themselves to offer an overview of the most popular AI generators.
- Two organisational training sessions provided to City St George (12 participants) and Birkbeck (14 participants) by the Project’s Training Specialist during March 2026.
- Seven individual training sessions were delivered to Birkbeck academic staff, which focused on the Multiple Choice Questions generator, Rubric generator, Feedback including Advanced Feedback and the two versions of Assignment AI generators. There are three versions of Feedback generators, original Feedback generator allows single submission, Enhanced feedback generator allows 5 submissions at a time and Advanced Feedback generator allows bulk submissions based on single assignment criteria.
Lessons learned from running these sessions have been consolidated into a Training Plan (available here); this can be applied to the adoption of any GenAI platform.
Moodle Integration
- The TeacherMatic Moodle integration (LTI 1.3) was installed on the BLE’s shared Moodle instance (version 4.5). Site administration access was provided to enable appropriate permissions.
- The setting up of TeacherMatic and Moodle integration was straight forward; the TeacherMatic documentation/guide was clear to follow.
- On Moodle, TeacherMatic shows the same landing page/interface as in a browser. However, it did not reduce any operational steps or time. For example, feedback/activities still had to be exported and uploaded back into Moodle.
An academic year is a long time in AI development. Features and workflows that were challenging at the start improved during the project, benefiting participants who joined later. The pilot community shared practice around data processing and ethics, assessment workflows and content generation. TeacherMatic developers also enhanced the system in response to feedback. Findings from project meetings and staff evaluations completed by 10 users identified the following benefits, risks, and constraints.
Benefits
- A user-friendly interface for content generators, with input forms that capture context such as level of study. This reduced the need for complex prompting and helped users understand the information needed to produce useful outputs.
- A controlled environment for drafting feedback, rubrics and question banks using lecturer-provided materials.
- Rubric generators supported reflection on marking criteria, differentiation between levels and the distribution of marks. Some lecturers created rubrics for the first time, while others reviewed and modified existing rubrics.
- Comparison of AI-generated assessment and feedback with human academic judgement encouraged critical reflection on assessment practice and AI limitations.
- Feedback and question generation reduced the effort required for some tasks and supported greater consistency in tone, wording and structure.
- TeacherMatic can assist a lecturer in providing additional feedback for students; its depth, relevance and accuracy still require academic oversight.
- Subject-related outputs were generally more suitable for Level 4 work, although there were some exceptions. At Level 5 and above, outputs could be too simplistic, particularly where assessment required synthesis, critical analysis or specialist disciplinary understanding.
- Marking did not always align with academic judgement. In one Level 7 trial, marking was initially over-generous but became overly severe when the lecturer instructed the system to mark more strictly.
- Feedback sometimes focused too heavily on issues with limited weighting, while overlooking points the lecturer considered central to the assessment. This raised concerns about marking validity and the system’s interpretation of assessment criteria.
- Outputs could be generic, shallow, irrelevant or insufficiently nuanced. Participants identified particular limitations for dissertations, research-based assessments and subjects requiring interpretation or specialist knowledge.
- Consistency in tone, structure and wording did not necessarily result in consistency of academic judgement. Marks could vary according to the instructions, contextual material and refinements supplied by the lecturer.
- Improved results required detailed lecturer input, including assessment guidance, examples and repeated refinement. In the Level 7 trial, closer alignment was achieved only after the lecturer developed additional marking guidance and continued to revise the outputs.
- TeacherMatic could support feedback drafting, comparison and moderation, but the pilot did not establish it as a reliable independent marker. Final marks and feedback therefore remained the responsibility of the academic.
- The system did not necessarily save time. Workload was sometimes redistributed to preparing materials, refining instructions and checking outputs rather than reduced.
- The multi-institutional project did not have sufficient time or resources to investigate the Moodle integration fully, limiting assessment of its practical value.
The project demonstrates that TeacherMatic succeeds as a usable educational AI platform but has not yet earned trust as an assessment tool for high-stakes academic work in university education. Staff recognise value in content generation, feedback drafting, and efficiency gains, but concerns around marking validity, academic judgement, consistency, and subject expertise currently limit adoption. The strongest opportunity is in supporting human assessors rather than automating assessment decisions.
There is an opportunity to use the platform as a tool to support new career academics in developing feedback styles and a consistency in delivery tone. This could also be helpful as staff development for all who teach.
This project was initiated to develop local understanding of generative AI tools. Participating in a Jisc-supported project gave participating institutions an opportunity to explore new technology and raise their profile in a national programme through published outcomes. Running the project as a joint activity meant we were able to extend levels of study, discipline areas and involved many more collaborators than if one institution had worked alone.
Despite doubts regarding AI reliability generally, research suggests students are still using it. The importance of education to support their development of critical thinking around such systems should therefore not be ignored. Key to this is the understanding of AI functionality and limitations by all staff. TeacherMatic’s controlled hosting environment may continue to be useful in developing AI literacy, further investigation into assessment workflows and facilitating discussions on ethical use and critical thinking. There is an opportunity to integrate the tool into AI literacy and ethics workshops as well as existing staff development such as digital skills courses and teacher training. Also, although this project focused on assessment and feedback, TeacherMatic has a broader suite of tools for teaching, assessment, feedback, content creation, administration and management.
Although the project indicated that TeacherMatic has potential value as a managed, human-led support tool for assessment and feedback, pilot evaluation (10 responses) was inconclusive in justifying a long-term investment in the tool at individual member institutions. As a result, Birkbeck will lead an extended investigation through 2026-27 to build on enthusiasm from their participants, and continue to explore the platform beyond the assessment generators.
Project Outputs
- Case studies and guidance on use in specific disciplines
Case studies are publicly available here - Podcasts about the Project and Outcomes
Podcasts are publicly available here to hear. - A network of ‘champions’ was established from the outset comprising project participants, pilot leads and the project team. The group came together for a hybrid meeting three times per term. A Teams chat space for each institution was used for local communications.
- The BLE established has a Special Interest Group for GenAI, in which participants from this project have been involved. More information is here. This Group, which shares effective practices, will continue to meet in the future and will conduct workshops and events for colleagues across the Bloomsbury consortium.
- Develop a human-led AI adoption framework and adaptable training plans for Generative AI use in HE. Completed and available here.
The following recommendations are for both future users of TeacherMatic and for the developers of the platform itself.
- Prioritise assessment reliability over productivity features. The largest barrier to TeacherMatic adoption is lack of confidence in marking accuracy, grading consistency, and academic judgement in outputs. Focus future development on rubric design and differentiation and evidence-based assessment rather than additional generators.
- Position TeacherMatic as a drafting and moderation assistant rather than an autonomous marker. Pilot evidence suggests greater acceptance when the tool supports academics rather than replaces academic judgement.
- Develop discipline-specific assessment models. Respondents repeatedly highlighted problems with nuanced and specialist subject areas. Future testing should focus on discipline-sensitive content and academic literacy.
- Introduce conversational refinement and learning capabilities. Users want to provide examples of high-quality work, refine outputs iteratively, and understand how the system reaches conclusions. A more interactive workflow would address many current frustrations.
- Include more training and offer wider community practice. Training was consistently praised and helped participants engage productively with the platform; we recommend TeacherMatic offers more high-quality, embedded training.
Some resources that have informed this work:
Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö. and Mariman, R. (2024) Generative AI Without Guardrails Can Harm Learning: Evidence from High School Mathematics. The Wharton School Research Paper. Available at: https://doi.org/10.2139/ssrn.4895486 (Accessed: 19 June 2026).
Benzo, R. (2026) A (Double) Dual Reality Framework. Presentation delivered at AI and the Future of the University: Critical Perspectives, organised by Birkbeck Law School, hosted by Craig Reeves and Victoria Ridler, Birkbeck, University of London, London, 5 June.
Fazi, M.B. (2019) ‘Can a machine think (anything new)? Automation beyond simulation’, AI & Society, 34(4), pp. 813–824. Available at: https://doi.org/10.1007/s00146-018-0821-0 (Accessed: 19 June 2026)
Pearson, J. (2026) How do we make sure things in FE don't just "get worse"? Association of Colleges. Available at: https://www.aoc.co.uk/news-campaigns-parliament/news-views/aoc-blogs/how-do-we-make-sure-things-in-fe-dont-just-get-worse (Accessed: 19 June 2026)
Schultze-Gerlach, T., Kumar, V. S., McKeown, G., O'Connor, P. A., Rychlowska, M., & Sparemblek, K. (2024). Using Large Language Models to augment (rather than replace) human feedback in higher education improves perceived feedback quality. Pre-print. Queen's University Belfast. https://doi.org/10.31234/osf.io/tvcag (Accessed: 25 June 2026)
Welsh, K. and Milne, L. (21 Jan 2026) “Cautious, Transparent and Responsible: Shaping AI for Education at the University of Chester”, TeacherMatic Webinar
Appendix
Summary of TeacherMatic pilot module representation by organisation, faculty/school or area, example module titles and level of study.
Organisation |
Faculties, schools or areas involved |
Module titles |
Module by level of study |
Birkbeck |
Four broad areas represented: Faculty of Humanities and Social Sciences; Faculty of Business and Law; Faculty of Science; Professional Services. Six schools or areas were included across these categories. |
Introduction to Digital Culture; Working With Digital Media; Media, Technology and Culture; Crime, Inequality & Social Change; Marketing Simulation; Principles of Marketing; Introduction to Cognition; Research Methods; Start to Teach MicroTeach Session. |
13 modules/activities listed: Level 4 (3); Level 5 (4); Level 5/6 (2); Level 6/7 (1); Level 7 (2); level not specified (3). |
City St George's |
School of Policy & Global Affairs represented across three areas: Department of Economics; Department of International Politics; Policy and Global Affairs. |
Economics; Policy and Corporate Affairs; Security Studies; Emerging Issues in International Politics. |
4 modules listed: Level 4 (1); Level 5 (2); Levels 4/5/6 (1). |
London School of Hygiene and Tropical Medicine |
The School of Infectious and Tropical Diseases |
Biology of Infectious Diseases – (online delivery); Mycology - (in person delivery) |
2 modules listed: Level 7 |
UCL |
UCL Institute of Education was represented across four areas: Culture, Communication and Media; Curriculum, Pedagogy and Assessment; Education, Practice & Society; Learning and Leadership. |
Digital Design Thinking and Making; Design & Use of Technology in Education; Geography PGCE; MA Geography Education; Citizenship PGCE; What is Education; Sociology and Research Methodology; Gender, Education and Development; Learning Design for Blended and Online Education; MA Educational Leadership. |
10 entries listed: Level 7 (7); Level 6/7 (2); Level 7/8 (1). |
University of London Worldwide |
Five areas represented: Faculty of Historical Studies and Social Sciences; Student Life; Online Education; Careers Service; Worldwide/distance learning provision. |
MSc Global Environment and Sustainability. Other entries were not linked to named modules and were recorded as N/A. |
6 entries listed: Level 7 (1); Levels 4/5/6 (5). |
