Skip to content
Open access

PhishLens: Calibration-Aware Selective Routing with Verified Local-LLM Evidence Selection for Phishing Website Detection

Oct 2026 · International Journal of Engineering and Management Research · 0 citations · 28 references

Abstract

Large language models (LLMs) can explain phishing verdicts in natural language, but using them as the detector adds latency and cost, and their explanations can cite evidence that is absent or wrong. We present PhishLens, a phishing website detector in which frozen classical models set every phishing label and probability, while a small local LLM is used only to select explanatory evidence for uncertain cases. A LightGBM model on 46 URL and static-HTML features, with a 20-feature URL-only Random Forest fallback, produces the label and a calibration-checked probability. Cases whose probability falls inside a frozen uncertainty band are routed to a local Qwen3 (4B) model that may return only ranked identifiers of verified evidence and one approved safety-action code; a deterministic verifier checks every output, and any failed output is replaced by a deterministic explanation. The LLM cannot change the probability or label, and submitted URLs are never visited. Detector features, models, calibration choices, classification thresholds and routing bands were selected on registrable-domain-grouped Phish360 partitions and frozen before the final detector evaluations. The later ID-selection protocol was assessed only in a 20-case development-validation pilot and did not alter any detector or reported test prediction. On the untouched Phish360 test set, the URL+HTML pathway reached 94.58% accuracy, MCC 0.8869, F1 0.9316, PR-AUC 0.9798 and ECE 0.0298. In a second-look external evaluation on CompPhish v3, it reached 87.64% accuracy, MCC 0.7517, F1 0.8672, PR-AUC 0.9441 and ECE 0.0635. The routing policy would avoid LLM invocation for 80.55% of external cases, although the routed share rose from the 10% validation budget to 19.45%. In a separate 40-case validation pilot, all 40 Qwen outputs passed deterministic verification, with a Wilson 95% confidence interval of 91.2%–100%. Exploratory feedback did not establish that Qwen explanations were more useful than deterministic explanations. Calibration and routing budgets must therefore be re-checked under dataset shift.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.