Continuum: Automated Construction and Retrieval of Software Decision Knowledge Graphs from Developer-AI Conversations

Abstract

Developer-AI conversations, through coding assistants and chat-based programming aids, produce rich architectural decision traces (trigger, context, alternatives considered, decision, rationale) that are lost when the chat session ends. Conversation logs are unstructured, not indexed for retrieval, and disconnected from one another, so a team member joining weeks later cannot recover why a technology was chosen or what alternatives were considered. We present Continuum, an end-to-end system that captures and serves software decision knowledge from developer-AI conversation logs. The system integrates five engineered stages: (1) LLM-based decision extraction with chain-of-thought prompting, (2) a 7-stage cascading entity resolution pipeline combining cached lookups, a curated canonical dictionary (534 technology-name mappings), fuzzy matching, and embedding similarity, (3) a Neo4j knowledge graph with polyglot persistence across PostgreSQL and Redis, (4) a GraphRAG retrieval layer combining BM25 fulltext and NV-EmbedQA vector search via Reciprocal Rank Fusion with K-hop subgraph expansion, and (5) an agent integration layer via the Model Context Protocol (MCP) that enables coding agents to both query prior decisions and contribute new ones. The contribution is the end-to-end integration: prior work has explored individual pieces (decision extraction, knowledge graphs, GraphRAG), but not a deployable system that ties them together for developer-AI conversations. We report a preliminary evaluation on a synthetic corpus of 200 generated developer-AI conversations across 9 technical domains, using a held-out train/test methodology. On our synthetic dataset, the 7-stage entity resolver attains 97.2% accuracy on 981 test mentions (95% CI: [96.2%, 98.3%]), outperforming exact-match (74.6%) and fuzzy-only (76.7%) baselines under McNemar's test (p < 0.001), with B-cubed F1 = 0.979. Two of the authors independently judged 40 sampled decision traces, yielding extraction precision of 75.0% (all categories) and 97.3% when exploratory non-decisions are excluded; on the 10-item overlap, Cohen's kappa = 0.07 reflects genuine disagreement on what qualifies as a "decision" in open-ended AI chat. On 40 graph-derived queries, hybrid GraphRAG retrieval achieves full recall across decision-search, tech-comparison, and entity-lookup query types (MRR = 0.775); a manual relevance assessment by the two reviewers on 10 natural-language queries yielded a mean rating of 2.2/3.0. As a real-log case study, we apply the pipeline to the full Claude Code session history from the build of Vibe Voyager, a public agent-built browser game (github.com/shehral/vibe); from 77 segmented and scrubbed sub-conversations the system extracts 379 decision traces and 290 unique canonical entities, with the decisions splitting 65% agent-made vs. 35% human-made (Section 4.2). We are transparent about the limitations of this evaluation (synthetic corpus, small annotator pool, proxy ground truth for 150 conversations, graph-derived queries) and discuss them in Section 4.3. Overall, these results indicate that architectural decision knowledge can be recovered from developer-AI conversations with high entity-resolution accuracy and demonstrated feasibility on real logs, while robust measurement of extraction quality and generalization to multi-developer settings remain open.