Prompting Without Principles: Are Students Transferring Software Engineering Knowledge to LLM Use?

Generative AI (GenAI), in particular large language models (LLMs), have rapidly become part of software engineers’ toolboxes. While the software engineering community has been actively debating efficacy and extent of use of GenAI tools, little is known about how early-career developers apply these tools and transfer their classroom training into practice. To address this gap, we conducted a controlled study with graduate students enrolled in a software refactoring course in the Master of Software Engineering program at Carnegie Mellon University’s Department of Electrical and Computer Engineering. The study compared how students solved a library upgrade problem using an ad hoc non-LLM approach versus an automated approach supported by LLMs. Our findings show that students often failed to apply disciplined software engineering practices when relying on LLMs, simply feeding the LLMs error text from builds rather than using them to understand issues and explore solutions. Our results highlight a dual challenge for educators: reinforcing foundational software engineering skills and their transfer to GenAI use, while also introducing new competencies such as effective prompting and navigating LLM-driven workflows.