Protecting Everyday Users from Phishing: An Empirical Evaluation & Comparison of Large Language Models
Abstract
While general-purpose large language models (LLMs) represent a new frontier in cybersecurity, their effectiveness for specialized tasks such as phishing detection requires further empirical validation. This research systematically compares the effectiveness of three industry-leading models, which were selected by the re-searcher based on their availability at the time of analysis: Google Gemini, OpenAI GPT, and Anthropic Claude. Utilizing a balanced dataset of 2,000 emails, the study evaluated classification performance based on precision, recall, and F1-scores. Quantitative results reveal a critical performance trade-off: while models like Claude and Gemini achieved perfect or near-perfect recall in identifying phishing emails, they exhibited lower precision by incorrectly flagging many legitimate emails as malicious. Claude emerged as the most robust performer with an overall accuracy of 96.15%. Qualitative analysis revealed a trust-verify para-dox, where users value LLMs as diagnostic assistants but remain skeptical of black-box outputs, demanding evidence-based reasoning and explainability. This research concludes that while LLMs serve as a valuable secondary advisory tool for enhancing user vigilance, they are not ready for fully autonomous deployment without human oversight.