Jena D. Hwang

XYBench: Can LLMs Respond Pragmatically to Queries with Misconceptions?

When non-expert users ask LLMs for assistance, their queries can often have misconceptions (e.g.,"How do I parse XML with regex?"). In such cases, often referred to as the XY-problem, LLMs must identify the misconception ("regex are fragile") and meaningfully direct the user toward a pragmatic solution that will addres...