TEACHERMATIC PILOT FOR ASSESSMENT AND FEEDBACK for Knowledge Management, Organisational Learning and Innovation module
Alistair Cumming, Associate Lecturer in the Department of People, Work and Organisational Psychology at Birkbeck, University of London
Background: This pilot was conducted on a Level 7 MSc 3,000-word Report style assignment – requiring analysis and recommendations based on practices within a real organisation known to the students (so it should be harder for students to use AI to manufacture analysis).
Process: I tried 3 iterations (as below) – all using the Advanced feedback generator
My goals in the pilot: To help me frame the actual summative feedback I gave to students – to speed up this process and ensure that I didn’t miss things across students (i.e. things that I had mentioned to one as important was also mentioned on another). I also was interested whether it could help me confirm the validity of my marking as a sort of second marker.
First trial
I uploaded an extended information about the brief giving indications of structural expectations, course overview and readings and guidance I had provided to students including marking structure. I then added separately the rubric I use and some indications of Grade related feedback. I tried out 3 student assignments, but the results were disappointing and did not compare well to my own feedback and marks.
Second trial
I adjusted slightly the files I had added in Trial 1 to make sure that it got clearer guidance on the rubrics and marking. I then tried out 5 student assignments and compared them to my own feedback and marks. This made slight improvements but was still disappointing. I then started to test out the same assignment and submissions in CoPilot to see how the latter fared. CoPilot was certainly far better as it was easier to share my own feedback and achieve closer alignment. I then asked it for its marking lessons learned and then added these into the ‘Refine feedback’ section. That did help TeacherMatic get closer to what I was looking for and highlighted that this ‘Refine feedback’ could be used for something like the dialogue I get with CoPilot (except that it doesn’t directly respond, so it more one way than two way) – except it is student submission specific so is not carried over to another student automatically. It was also still not really picking up the points I wanted it to so I stopped using it for a bit as I was getting frustrated with the limited value it was adding.
Third trial
After trialling a few more assignments in CoPilot and getting closer alignment, I asked it to give me a summary guide for markers and AI that would help it when marking these assignments. I then added in the general instructions and brief as I had in Trial 1, but instead of adding the rubric, I added in the guidelines created by CoPilot. I also later added some specific text in the ‘Refine’ section to get the feedback structure and style I wanted. I tried this on 8 assignments and the results were far better and closer to the feedback I would want to give although often I needed a further Refinement to get to this - I realised here that the Refine section could be used even more specifically to adjust the feedback by highlighting points that I disagreed with and so get the TeacherMatic feedback even closer to my own views.
In terms of using it to speed up my actual feedback writing, it worked better when I gave specific prompts based on my own feedback, but it was still rather hard work to get to this point. Marks were still often too high (even when set at ‘Standard’) and even when the marks were broadly OK, the feedback didn’t really hit the right themes to align with it such that the feedback offered was still in need of a fair bit of editing. TeacherMatic also wasn’t always good at working out where my suggested (Refine) feedback themes were most relevant ie it would sometimes bring it into every criterion whether relevant or not. It particularly seemed to struggle with the nuances of quite different organisational contexts (e.g. micro-organisations vs larger ones) and take a holistic view of each assignment in its own unique context.
Overall conclusions:
Clearly Trial 3 showed the potential of TeacherMatic better, but I needed a lot of work with CoPilot first to help me define the style – content – themes that matter and present them in a form that TeacherMatic could understand and act on - and even then, I needed to refine it further to get closer to what I felt the mark and feedback should be.
What I missed compared to CoPilot was the ability to share my own final actual feedback so that it could see what I was crediting – what I didn’t like – how I liked to phrase things etc and align closer to this progressively over time. I could also ask it questions and so understand why it did certain things and check its understanding ie a two-way exchange that TeacherMatic didn’t offer me.
In terms of my original goals to help me frame the summative feedback I gave to students:
Final thoughts: Maybe, not being an essay, and being at Level 7, TeacherMatic rather struggled with the complexity of the report (i.e. the inter-connectedness of the different elements of the report and understanding which bits were particularly key to coherence and strength of argument) and the depth and nuance of feedback I was seeking to provide. In its current form, I certainly wouldn’t use it again for summative Marking / Feedback. It might be OK for Formative assessments, but I don’t have such assignments. CoPilot has its issues, but compared to TeacherMatic in this pilot, it provided me with far richer and more relevant feedback; got to this quicker and largely appeared to get better with use. TeacherMatic only did this (in part) when I fed in my own learning from using CoPilot. If I had relied entirely just on TeacherMatic, I think I would have given up a lot earlier.
Alistair Cumming, Associate Lecturer in the Department of People, Work and Organisational Psychology at Birkbeck, University of London
Background: This pilot was conducted on a Level 7 MSc 3,000-word Report style assignment – requiring analysis and recommendations based on practices within a real organisation known to the students (so it should be harder for students to use AI to manufacture analysis).
Process: I tried 3 iterations (as below) – all using the Advanced feedback generator
My goals in the pilot: To help me frame the actual summative feedback I gave to students – to speed up this process and ensure that I didn’t miss things across students (i.e. things that I had mentioned to one as important was also mentioned on another). I also was interested whether it could help me confirm the validity of my marking as a sort of second marker.
First trial
I uploaded an extended information about the brief giving indications of structural expectations, course overview and readings and guidance I had provided to students including marking structure. I then added separately the rubric I use and some indications of Grade related feedback. I tried out 3 student assignments, but the results were disappointing and did not compare well to my own feedback and marks.
- Marking tended to be over-generous
- Asking it to be less generous, tended to make it go to the other extreme
- Feedback was often far too specific on issues of limited weighted significance or far too short – and I often disagreed with the points it seemed to have focused on
Second trial
I adjusted slightly the files I had added in Trial 1 to make sure that it got clearer guidance on the rubrics and marking. I then tried out 5 student assignments and compared them to my own feedback and marks. This made slight improvements but was still disappointing. I then started to test out the same assignment and submissions in CoPilot to see how the latter fared. CoPilot was certainly far better as it was easier to share my own feedback and achieve closer alignment. I then asked it for its marking lessons learned and then added these into the ‘Refine feedback’ section. That did help TeacherMatic get closer to what I was looking for and highlighted that this ‘Refine feedback’ could be used for something like the dialogue I get with CoPilot (except that it doesn’t directly respond, so it more one way than two way) – except it is student submission specific so is not carried over to another student automatically. It was also still not really picking up the points I wanted it to so I stopped using it for a bit as I was getting frustrated with the limited value it was adding.
Third trial
After trialling a few more assignments in CoPilot and getting closer alignment, I asked it to give me a summary guide for markers and AI that would help it when marking these assignments. I then added in the general instructions and brief as I had in Trial 1, but instead of adding the rubric, I added in the guidelines created by CoPilot. I also later added some specific text in the ‘Refine’ section to get the feedback structure and style I wanted. I tried this on 8 assignments and the results were far better and closer to the feedback I would want to give although often I needed a further Refinement to get to this - I realised here that the Refine section could be used even more specifically to adjust the feedback by highlighting points that I disagreed with and so get the TeacherMatic feedback even closer to my own views.
In terms of using it to speed up my actual feedback writing, it worked better when I gave specific prompts based on my own feedback, but it was still rather hard work to get to this point. Marks were still often too high (even when set at ‘Standard’) and even when the marks were broadly OK, the feedback didn’t really hit the right themes to align with it such that the feedback offered was still in need of a fair bit of editing. TeacherMatic also wasn’t always good at working out where my suggested (Refine) feedback themes were most relevant ie it would sometimes bring it into every criterion whether relevant or not. It particularly seemed to struggle with the nuances of quite different organisational contexts (e.g. micro-organisations vs larger ones) and take a holistic view of each assignment in its own unique context.
Overall conclusions:
Clearly Trial 3 showed the potential of TeacherMatic better, but I needed a lot of work with CoPilot first to help me define the style – content – themes that matter and present them in a form that TeacherMatic could understand and act on - and even then, I needed to refine it further to get closer to what I felt the mark and feedback should be.
What I missed compared to CoPilot was the ability to share my own final actual feedback so that it could see what I was crediting – what I didn’t like – how I liked to phrase things etc and align closer to this progressively over time. I could also ask it questions and so understand why it did certain things and check its understanding ie a two-way exchange that TeacherMatic didn’t offer me.
In terms of my original goals to help me frame the summative feedback I gave to students:
- Even Trial 3 with Refinements didn’t markedly help me speed up this process (even if I ignore all the extra work in setting up the tool) as I was having to steer it on each student
- It was reasonable at ensuring that I didn’t miss things across students (ie things that I had mentioned to one as important was also mentioned on another) – but I sometimes had to prompt it for this.
- In terms of confirming the validity of my marking as a sort of second marker, the marking it gave was generally too generous and if I challenged it, then it was often rather too harsh, so it didn’t really help me in my assessing my own judgement.
Final thoughts: Maybe, not being an essay, and being at Level 7, TeacherMatic rather struggled with the complexity of the report (i.e. the inter-connectedness of the different elements of the report and understanding which bits were particularly key to coherence and strength of argument) and the depth and nuance of feedback I was seeking to provide. In its current form, I certainly wouldn’t use it again for summative Marking / Feedback. It might be OK for Formative assessments, but I don’t have such assignments. CoPilot has its issues, but compared to TeacherMatic in this pilot, it provided me with far richer and more relevant feedback; got to this quicker and largely appeared to get better with use. TeacherMatic only did this (in part) when I fed in my own learning from using CoPilot. If I had relied entirely just on TeacherMatic, I think I would have given up a lot earlier.
