Case Study on ChatGPT’s Performance in Assisting Students with Physics Tests

Authors

DOI:

https://doi.org/10.26618/jpf.v13i1.16624

Keywords:

artificial intelligence, chatGPT, physics concept understanding

Abstract

The rapid development of artificial intelligence (AI), particularly ChatGPT, has sparked interest in its application in education. This study aims to investigate the potential of ChatGPT in helping students understand and solve physics problems, focusing on the Test of Understanding Graphs in Kinematics and the Determining and Interpreting Resistive Electric Circuit Concepts Test. The study involved 25 physics education students who completed these tests independently and with ChatGPT's assistance. The results revealed that students with a strong foundational understanding and reflective abilities interacted more effectively with ChatGPT, leading to improved answers and deeper conceptual understanding. In contrast, students with weaker prior knowledge tended to accept ChatGPT’s answers without critical reflection, perpetuating errors. Furthermore, ChatGPT showed limitations in interpreting image-based questions, reading scales, and providing consistent responses to concept-specific queries. These findings suggest that while ChatGPT has the potential to enhance learning, it requires thoughtful integration, particularly in helping students develop critical thinking and problem-solving skills. Teachers are encouraged to use ChatGPT’s limitations to design assessments that minimize the risk of cheating and foster deeper understanding. In conclusion, this study underscores the importance of combining AI tools with strong conceptual foundations and active reflection to optimize learning outcomes in physics education. Future research should focus on refining strategies for using AI in education to address its current limitations and enhance its effectiveness in complex learning scenarios.

References

Abbas, N., Ali, I., Manzoor, R., Hussain, T., & Hussain, M. H. L. (2023). Role of artificial intelligence tools in enhancing students’ educational performance at higher levels. Journal of Artificial Intelligence Machine Learning and Neural Network, 3(5), 36–49. https://doi.org/10.55529/jaimlnn.35.36.49

Akavova, A., Temirkhanova, Z., & Lorsanova, Z. (2023). Adaptive learning and artificial intelligence in the educational space. E3s Web of Conferences, 451, 06011, 1-4. https://doi.org/10.1051/e3sconf/202345106011

Amin, B. D., Sahib, E. P., Harianto, Y. I., Patandean, A. J., Herman, & Sujiono, E. H. (2020). The interpreting ability on science kinematics graphs of senior high school students in South Sulawesi, Indonesia. Jurnal Pendidikan IPA Indonesia, 9(2), 179–186. https://doi.org/10.15294/jpii.v9i2.23349

Beichner, R. (1994). Testing student interpretation of kinematic graphs. American Journal of Physics, 62, 750-762. https://doi.org/10.1119/1.17449

Chiu, T. K. F., Moorhouse, B. L., Chai, C. S., & Ismailov, M. (2023). Teacher support and student motivation to learn with Artificial Intelligence (AI) based chatbot. Interactive Learning Environments, 32(7), 3240–3256. https://doi.org/10.1080/10494820.2023.2172044

Cooper, G. (2023). Examining science education in ChatGPT: an exploratory study of generative artificial intelligence. Journal of Science Education and Technology, 32, 444–452. https://doi.org/10.1007/s10956-023-10039-y

Eaton, P., & Willoughby, S. D. (2018). Confirmatory factor analysis applied to the force concept inventory. Physical Review Physics Education Research, 14, 1-11. https://doi.org/10.1103/physrevphyseducres.14.010124

Engelhardt, P. V., & Beichner, R. J. (2004). Students’ understanding of direct current resistive electrical circuits. American Journal of Physics, 72, 98–115. https://doi.org/10.1119/1.1614813

Farrokhnia, M., Banihashem, S. K., Noroozi, O., & Wals, A. (2023). A SWOT analysis of ChatGPT: implications for educational practice and research. Innovations in Education and Teaching International, 61(3), 460–474. https://doi.org/10.1080/14703297.2023.2195846

Gregorcic, B., & Pendrill, A. M. (2023). ChatGPT and the frustrated socrates. Physics Education, 58,1-9. https://doi.org/10.1088/1361-6552/acc299

Hashem, R., Ali, N., Zein, F. E., Fidalgo, P., Khurma, O. A. (2023). AI to the rescue: Exploring the potential of chatgpt as a teacher ally for workload relief and burnout prevention. Research and Practice in Technology Enhanced Learning, 19, 1-26. https://doi.org/10.58459/rptel.2024.19023

Henderson, R., & Stewart, J. (2018). Racial and ethnic bias in the force concept inventory. Conference Proceedings, 172–175. https://doi.org/10.1119/perc.2017.pr.038

Hidaayatullaah, H. N. (2022). The science literacy profile based on madrasah students’ misconceptions on science concepts. Proceedings of the International Conference on Madrasah Reform, 111-116. https://doi.org/10.2991/assehr.k.220104.017

Hikmatiar, H., Sya’bania, N., Jayadin., Kasman, R. A., Imranah., Sahlan., & Saputra, S. (2024). The effectiveness of Chatgpt in completing astronomy lectures: Building awareness of its use. Jurnal Pendidikan Fisika, 12(2), 121-130. https://doi.org/10.26618/jpf.v12i2.13587

Hoa, N. Q. (2023). AI and plagiarism: Opinion from teachers, administrators and policymakers. Proceedings of the Asiacall International Conference, 4, 75–85. https://doi.org/10.54855/paic.2346

Ivanjek, L., Morris, L., Schubatzky, T., Hopf, M., Burde, J. P., Haagen-Schützenhöfer, C., Dopatka, L., Spatz, V., & Wilhelm, T. (2021). Development of a two-tier instrument on simple electric circuits. Physical Review Physics Education Research, 17, 1-15. https://doi.org/10.1103/PhysRevPhysEducRes.17.020123

Kasepalu, R., Prieto, L. P., Ley, T., & Chejara, P. (2022). Teacher artificial intelligence-supported pedagogical actions in collaborative learning coregulation: A wizard-of-oz study. Frontiers in Education, 7, 1-15. https://doi.org/10.3389/feduc.2022.736194

Khan, R. A., Jawaid, M., Khan, A. R., & Sajjad, M. (2023). ChatGPT - Reshaping medical education and clinical management. Pakistan Journal of Medical Sciences, 39(2), 605-607. https://doi.org/10.12669/pjms.39.2.7653

Kim, S.-W. (2023). Change in attitude toward artificial intelligence through experiential learning in artificial intelligence education. International Journal on Advanced Science Engineering and Information Technology, 13(5), 1953–1959. https://doi.org/10.18517/ijaseit.13.5.19039

Linuwih, S. (2013). Konsepsi alternatif mahasiswa calon guru fisika tentang gaya-gaya yang bekerja pada balok. Jurnal Pengajaran MIPA, 18(1), 69-77. https://doi.org/10.18269/jpmipa.v18i1.259

Maries, A., & Singh, C. (2013). Exploring one aspect of pedagogical content knowledge of teaching assistants using the test of understanding graphs in kinematics. Physical Review Physics Education Research, 9, 1-14. https://doi.org/10.1103/physrevstper.9.020120

Phage, I. (2018). Undergraduate physics students’ conceptual understanding in the learning of kinematics using a blended approach. Ijaedu- International E-Journal of Advances in Education, 4(11), 199–204. https://doi.org/10.18768/ijaedu.455623

Phage, I. B., Lemmer, M., & Hitge, M. (2017). Probing factors influencing students’ graph comprehension regarding four operations in kinematics graphs. African Journal of Research in Mathematics, Science and Technology Education, 21(2), 200–210. https://doi.org/10.1080/18117295.2017.1333751

Polverini, G., & Gregorcic, B. (2024). Performance of ChatGPT on the test of understanding graphs in kinematics. Physical Review Physics Education Research, 20, 1-16. https://doi.org/10.1103/PhysRevPhysEducRes.20.010109

Remoto, J. P. (2023). ChatGPT and other AIs: Personal relief and limitations among mathematics-oriented learners. Environment and Social Psychology, 9(1), 1-13. https://doi.org/10.54517/esp.v9i1.1911

Suganda, T., Kusairi, S., Azizah, N., & Parno, P. (2020). The correlation of isomorphic, open-ended, and conventional score on the ability to solve kinematics graph questions. Jurnal Penelitian & Pengembangan Pendidikan Fisika, 6(2), 173–180. https://doi.org/10.21009/1.06204

Sutopo, S., Parno, P., & Angin, S. L. (2017). Pemahaman mahasiswa tentang multi representasi konsep percepatan. Jurnal Riset dan Kajian Pendidikan Fisika, 4(2), 48-53. https://doi.org/10.12928/jrkpf.v4i2.6551

Wang, T., Lund, B. D., Marengo, A., Pagano, A., Mannuru, N. R., Teel, Z. A., & Pange, J. (2023). Exploring the Potential Impact of Artificial Intelligence (AI) on International Students in Higher Education: Generative AI, Chatbots, Analytics, and International Student Success. Applied Sciences, 13(11), 1-15. https://doi.org/10.3390/app13116716

Warsono, W., Nursuhud, P. I., Darma, R. S., Supahar, S., Oktavia, D. A., Setiyadi, A., & Kurniawan, M. A. (2020). Multimedia learning modules (MLMs) based on local wisdom in physics learning to improve student diagram representations in realizing the nature of science. International Journal of Interactive Mobile Technologies (Ijim), 14(6), 148-158. https://doi.org/10.3991/ijim.v14i06.11640

Zavala, G., Tejeda, S., Barniol, P., & Beichner, R. J. (2017). Modifying the test of understanding graphs in kinematics. Physical Review Physics Education Research, 13(2), 20111, 1-16. https://doi.org/10.1103/PhysRevPhysEducRes.13.020111

Downloads

Additional Files

Published

2025-01-20

Issue

Vol. 13 No. 1 (2025): PENDIDIKAN FISIKA

Section

Articles

License

Copyright:

Authors who publish with this journal agree to the following terms:

1. Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution-ShareAlike 4.0 International License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.

2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.

3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.

Licence:

Authors are free to:

1. Share: Copy and redistribute the material in any medium or format

2. Adapt: Remix, transform, and build upon the material for any purpose, even commercially.

The licensor cannot revoke these freedoms as long as the authors follow the license terms, which include the following:

1. Attribution: You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.

2. ShareAlike: If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.

3. No additional restrictions: You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.

Jurnal Pendidikan Fisika  is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.