Jul 2026· Annual Conference on Innovation and Technology in Computer Science Education· pp. 107-113· 1 citation· 34 references
Computer Science
TL;DR
This paper presents a virtual learning environment that integrates a secure sandbox with access to multiple commercial and open-source LLMs to support experiential learning in AI security.
Abstract
Computing education increasingly focuses on teaching secure coding and developing secure web applications. Now, with the rise of generative AI, we face new challenges, particularly the potential misuse of this technology for identifying and exploiting software vulnerabilities. This paper presents a virtual learning environment that integrates a secure sandbox with access to multiple commercial and open-source LLMs to support experiential learning in AI security. The lab offers scenario-based exercises that cover web, code, and system-level vulnerabilities. Students craft adversarial prompts, test LLM-generated exploits, and analyse models behaviour using established metrics such as attack success rate (ASR), exploit generation accuracy, and refusal rate. Our expert validation revealed distinct model behaviours: GPT-4o achieved the highest ASR (77.5% in web testing), demonstrating consistent exploit generation, while Claude exhibited the highest safety refusal rate (32.56%). By engaging with these observable outcomes, students develop a critical understanding of LLM capabilities, limitations, and ethical risks. We argue that AI-integrated virtual labs are essential for preparing students to work responsibly with emerging AI-assisted security tools.
Investigating the security vulnerabilities present in AI-generated JWT authentication code for Java Spring Boot Representational State Transfer Application Programming Interfaces (REST API) reinforces that AI-generated JSON Web Token (JWT) authentication code requires dedicated security review.
Hoang Long Nguyen, Mezid Hmudda, Benjamin Powley· 0 citations
AI coding agents powered by LLMs are increasingly integrated into real-world software development, where they generate, edit, and execute code with autonomous access to local files and tools. Coding agents inherit security risks from both the LLM backbone, where adversarial prompts, poisoned training data, and backdoor triggers can cause models to emit insecure or attacker-chosen code, and their agentic architecture, where tool-using autonomy enables induced misuse of external APIs, data exfiltration, and persistent compromise of development environments. This paper presents a systematic evaluation of malicious issue requests against state-of-the-art coding agents (Cursor, Claude Code, and Codex Desktop), powered by two major model families (OpenAI GPT-5.3 Codex/GPT-5.4 and Anthropic Sonnet 4.6). Our novel benchmark IssueTrojanBench contains malicious issues that are constructed based on four novel attack categories (i.e., embedded as malicious instructions in issues), six delivery vectors (e.g., PDF, or issue comment), and further augmented by perturbations. Our results reveal critical vulnerabilities in the as-deployed modern coding agents, i.e., 66.5% of the malicious issues from IssueTrojanBench penetrate all the guardrails (agent- and LLM-level) of coding agents. Our further analysis shows that rejection is almost entirely from LLMs rather than the agent frameworks, with GPT models broadly vulnerable and Sonnet 4.6 exhibiting more selective, risk-aware blocking of high-impact actions. Our evaluation also highlights that the current agent-level defense strategy offers limited additional protection for coding agents. Our findings highlight the urgent need for stronger agent- and model-level safety mechanisms to protect AI coding agents.
Large Language Models (LLMs) are increasingly integrated into software development workflows, yet their ability to autonomously generate secure authentication code remains uncertain. This paper evaluates the security architecture of authentication systems generated by five prominent AI coding assistants through a bi-modal assessment framework combining static code analysis and dynamic penetration testing, mapped to NIST SP 800-63B guidelines. The study examines model behavior across four prompting strategies Basic, Secure, NIST-Based, and Reprompting to reflect varying levels of developer guidance. Empirical results demonstrate that code generated from functional or generically secure prompts consistently omits critical protections, particularly concerning brute-force resistance, session management, and robust password handling. While providing explicit, single-shot NIST context significantly improves compliance, the findings reveal that this remains structurally inadequate. Instead, iterative Reprompting: forcing models into a contextual self-auditing loop is strictly required to achieve a comprehensive, defense-in-depth security architecture. Ultimately, this study proves that current AI coding assistants do not produce secure-by-default applications, dictating that enterprise deployments must transition from single-shot prompt engineering to continuous, standards-driven verification pipelines.
Ishpuneet Singh, Shreyas Mahajan, Gurjot Singh et al.· 0 citations
With increasing reliance on computing systems and the growing frequency of cybersecurity incidents, it is important for CS undergraduates to develop foundational security skills before entering professional roles. In particular, students should be able to recognize and reason about potential security vulnerabilities in software. However, existing approaches to integrating security into the CS curriculum often emphasize narrow areas such as secure coding or highly technical topics like cryptography or software security, rather than fostering a broader perception of security threats. In this paper, we examine how undergraduates conceptualize and identify security threats by analyzing how they attempt to find ''attacks'' in other students' code. We conducted a think-aloud study with 15 CS undergraduates at a US-based R1 institution who had no formal training in computer security. Participants analyzed peer-developed text-based video game implementations to identify potential vulnerabilities, drawing on their prior experience implementing a similar game in an earlier ''Build-It'' task. Our analysis shows that students employed systematic, hypothesis-driven strategies, including unit testing, edge-case exploration, and controlled experimentation, while also drawing on prior experiences both inside and outside the classroom. Although most students attempted to validate whether an attack was successful, several stopped after identifying a single vulnerability, leaving additional issues unexplored. Based on these findings, we offer recommendations for CS instructors and curriculum committees on integrating foundational security concepts into programming assignments to help students better recognize and reason about computer security threats.
Michelle Jensen, Matthew Berland, Rahul Chatterjee· Annual Conference on Innovat...· 0 citations
The increasing deployment of Large Language Models (LLMs) in critical infrastructure has introduced a class of security risks that remain insufficiently characterised and poorly tooled in practice. Attack vectors including prompt injection, jailbreaking, code execution facilitation, covert data exfiltration, and training data poisoning present quantifiable threats to systems that rely on LLM outputs, yet no lightweight, provider-agnostic tool exists to measure these risks systematically. This paper presents VectorSec, an open-source, web-based LLM security scanner built with Python and Dash. The tool executes a structured test suite of 320 adversarial prompts spanning 16 vulnerability categories aligned with the OWASP Top 10 for LLMs. Each response is scored through a four-stage pipeline combining pattern matching, sentiment analysis, semantic similarity, and secondary LLM verification. Evaluation across three open-source models demonstrates that pattern-matching-only filtering misses approximately 15% of High and Critical findings that semantic verification correctly identifies, motivating the multi-layer design. VectorSec provides interactive dashboard reporting, real-time progress tracking, and audit-ready PDF and CSV exports, lowering the barrier to structured LLM security assessment in operational settings.
M. Yamin· International Conference on...· 0 citations
With the increasing usage of AI-generated code in software development workflows, new security challenges and concerns arise. This paper analyzes five LLMs: ChatGPT, Claude, Gemini, DeepSeek, and Grok in three phases of security assessments against web vulnerabilities listed by the OWASP Top 10. Phase 1 (December 2025) evaluated 13 prompts in XSS, Authentication, API Security, Hardcoded Secrets, and React Server Component CVE. Phase 2 (March 2026) evaluated 11 prompts using newly disclosed CVEs, complex authentication, and API security, with updated models. Phase 3 (May 2026) revisited phase one prompts against current models. Across all 185 samples, 49 out of the 120 Phase 1 and 2 samples (40.8%) were identified as vulnerable, 65 (54.2%) as secure and 6 (5.0%) partially vulnerable. The CVE-based prompts recorded the highest percentage of vulnerability (76%). No failures were observed in the authentication prompts with simple patterns, while complex access control resulted in 45% failures. Phase 3 (65 samples) re-ran Phase 1 prompts on current models; no progress was observed; the partial result rate went up from 7.7% to 27.7%. The results revealed hedged responses from the updated models. All results were cross-validated using Semgrep static analysis.
Malak Mansour, Anas AlMajali· IEEE Jordan Conference on Ap...· 0 citations
