The motivation is the way real literature review actually works: you start from one paper you care about, read what it cites, then read what those cite, building out the surrounding context of a research area. DendriteRAG automates that "dendritic" expansion. You register a seed paper and the system recursively crawls its references to a configurable depth, growing a personalized knowledge base without you manually hunting down each citation. The second guiding principle is local-first privacy — original PDFs, the metadata database, and the vector index all stay on the user's machine, and the language model runs locally rather than through a cloud API.
Architecturally it is split into independent local components that talk over HTTP. A Main API acts as the orchestrator (FastAPI plus SQLite): it serves the web UI, handles user requests, manages the crawl-and-parse queue, and coordinates calls to the model and the vector service. A separate RAG API is a focused vector-store server (FastAPI plus ChromaDB with sentence-transformer embeddings) that does only two things — accept text chunks and answer similarity queries — which keeps it independently reusable. A local LLM host does the language work, and a plain JavaScript and Tailwind web UI provides seed entry, queue management, and chat over the knowledge base.
The interesting engine is reference resolution. Rather than relying on structured citation metadata that PDFs rarely expose cleanly, the parser feeds overlapping text chunks to the local model with a strict extraction prompt, pulling bibliographic references out of messy free text into structured records. Because chunk overlap produces many partial and duplicate entries, it runs a deduplication pass — an LLM-based cleaner that merges variants of the same reference, with a simpler programmatic fallback. Resolved references are then looked up through a Semantic Scholar client that returns titles, authors, DOIs, abstracts, years, and open-access PDF links, and matched open-access PDFs are downloaded for ingestion. Each layer is separated into its own service module, so the parsing, scholarly-lookup, vector, and database concerns stay independently testable.
Honest context: this is an early-stage project — the repository is a small set of initial commits establishing the structure and core features, and the README itself marks database migration setup as not yet wired in. It depends on a running local model host with a downloaded model and is explicitly designed for single-user, local operation rather than scale or multi-user serving. Embeddings use a compact general-purpose sentence-transformer model, which is a pragmatic default rather than a domain-tuned scientific embedder. The design is clean and the recursive-crawl idea is fully present in code; what it is not yet is a hardened, long-running production system.