Research Questions and Findings
RQ1. What PMD-related problems are typically selected by students?
Among the 1,230 analyzed issues, the most common PMD ruleset category concerns ‘Code Style’, representing 32.3% of the issues.
RQ2. What type of issues typically takes longer to be fixed?
‘Design’ PMD ruleset category takes longer to resolve. Unlike other PMD category, this requires going beyond one or few instructions, and ChatGPT has a limited understanding of the broader context and entirety of the codebase.
RQ3. To what extent was ChatGPT successful in addressing the students' debugging needs?
Overall, this assignment helps cultivate ana- lytical and critical thinking skills as students engage in the debugging process. In addition, it teaches students to be skep- tical towards the use of ChatGPT by shedding light on the limitations of ChatGPT in identifying and solving problems. It highlights the importance of incorporating other tradi- tional static analysis tools and techniques to improve the accuracy and efficiency of their predictions. Furthermore, the involvement of a human-in-the-loop, capable of comprehend- ing code, can be highly valuable and desirable.
