• Messer, Marcus
• Brown, Neil C. C.
• Kölling, Michael
• Shi, Miaojing

Abstract

Research into automated grading has increased as Computer Science courses grow.<br/>Dynamic and static approaches are typically used to implement these graders, the most common implementation being unit testing to grade correctness.<br/>This paper expands upon an ongoing systematic literature review to provide an in-depth analysis of how machine learning (ML) has been used to grade and give feedback on programming assignments.<br/>We conducted a backward snowball search using the ML papers from an ongoing systematic review and selected 27 papers that met our inclusion criteria.<br/>After selecting our papers, we analysed the skills graded, the preprocessing steps, the ML implementation, and the models’ evaluations.<br/><br/>We find that most the models are implemented using neural network-based approaches, with most implementing some form of recurrent neural network (RNN), including Long Short-Term Memory, and encoder/decoder with attention mechanisms.<br/>Some graders implement traditional ML approaches, typically focused on clustering.<br/>Most ML-based automated grading, not many use ML to evaluate maintainability, readability, and documentation, but focus on grading correctness, a problem that dynamic and static analysis techniques, such as unit testing, rule-based program repair, and comparison to models or approved solutions, have mostly resolved.<br/>However, some ML-based tools, including those for assessing graphical output, have evaluated the correctness of assignments that conventional implementations cannot.

Topics

• inclusion
• clustering
• machine learning