Knowledge Base

Overview

The Knowledge Base is a vector-searchable repository of organizational knowledge that powers AI features across NetStacks. It stores documents from multiple sources, breaks them into searchable chunks, generates vector embeddings, and provides both semantic search and full-text search capabilities. AI Chat and NOC Agents use the Knowledge Base through RAG (Retrieval-Augmented Generation) to provide answers grounded in your organization's specific documentation, policies, and procedures.

Document Sources

How It Works

Document Ingestion Pipeline

When a document is added to the Knowledge Base, it goes through a multi-stage ingestion pipeline:

1. Upload / Fetch — The document is uploaded directly or fetched from an external source (URL, Confluence, SharePoint). The raw content and a content hash are stored for deduplication.
2. Sanitization — The content passes through the credential sanitization pipeline to remove any embedded credentials, API keys, or other sensitive data before further processing.
3. Chunking — The document is split into smaller, semantically meaningful chunks. The chunking strategy can be configured per document source.
4. Embedding — Each chunk is converted into a vector embedding using the configured embedding model. The embedding captures the semantic meaning of the text.
5. Storage — Chunks and their embeddings are stored in PostgreSQL using pgvector. A tsvector column is also generated for full-text search support.

Document state transitions through the pipeline: pending → processing → indexed (or error if ingestion fails). Only documents in the indexed state are included in search results.

Chunking Strategies

Search Capabilities

The Knowledge Base supports three search modes:

• Vector Similarity Search — Uses pgvector's cosine distance operator (<=>) to find chunks that are semantically similar to the query. Returns results ordered by similarity score (1 - cosine distance).
• Full-Text Search — Uses PostgreSQL's tsvector and ts_rank for BM25-style keyword matching. Finds chunks containing exact terms from the query.
• Hybrid Search — Combines vector similarity and full-text search for the best results. Useful when queries contain both conceptual meaning and specific technical terms.

RAG Integration

When AI Chat or a NOC Agent needs organizational context, the system automatically queries the Knowledge Base, retrieves the most relevant chunks, and includes them in the LLM prompt. This grounds the AI's responses in your actual documentation rather than general knowledge.

Setting Up the Knowledge Base

Follow these steps to build your organization's knowledge base and connect it to AI features.

1. Navigate to AI > Knowledge Base — Open the Knowledge Base management page from the main navigation.
2. Add document sources — Choose how to add documents:
   • Upload files — Drag and drop PDF, DOCX, TXT, or Markdown files. Each file becomes a separate document.
   • Add URLs — Paste URLs of web pages or documentation sites to scrape and index.
   • Add Web Sources — Point a web crawler at a documentation site to automatically discover and index pages using breadth-first search with configurable depth and page limits.
   • Connect Confluence — Enter your Confluence instance URL, authentication credentials, and select spaces or pages to sync.
   • Connect SharePoint — Configure your SharePoint tenant, authentication, and select document libraries to sync.
3. Configure chunking strategy — Select the appropriate chunking strategy for each source type. Use heading-based for structured documentation, semantic for articles, and fixed-size for unstructured content.
4. Monitor ingestion progress — Watch the document state transition from pending to processing to indexed. Failed documents show an error message with details.
5. Test search — Use the search interface to test queries against your knowledge base. Verify that relevant chunks are returned with good similarity scores.
6. Connect to AI features — AI Chat and NOC Agents automatically use the Knowledge Base for RAG-powered responses. No additional configuration is needed once documents are indexed.

You can also use AI to generate documentation automatically from your existing data:

Code Examples

Adding a Document Source via API

Upload a document to the Knowledge Base with metadata and category tags:

# Upload a PDF document to the knowledge base
curl -X POST https://controller.example.com/api/v1/knowledge/documents \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "title": "BGP Peering Policy - AS65001",
    "description": "Standard BGP peering policy for all edge routers",
    "source_type": "upload",
    "content_type": "application/pdf",
    "category": "network-policy",
    "tags": ["bgp", "peering", "policy", "edge-routers"],
    "requires_review": false
  }'

Chunking Strategy Comparison

The same document produces different chunk boundaries depending on the strategy:

Original Document: "BGP Peering Policy" (2,400 tokens)

--- Fixed-Size (512 tokens, 50 token overlap) ---
Chunk 1: Tokens 1-512    (Introduction + partial Peer Requirements)
Chunk 2: Tokens 463-974  (Peer Requirements continued)
Chunk 3: Tokens 925-1436 (Prefix Limits + partial Route Filtering)
Chunk 4: Tokens 1387-1898 (Route Filtering + partial Monitoring)
Chunk 5: Tokens 1849-2400 (Monitoring + Escalation)

--- Semantic (max 600 tokens) ---
Chunk 1: Introduction paragraph (180 tokens)
Chunk 2: Peer Requirements section (420 tokens)
Chunk 3: Prefix Limits and Filtering (580 tokens)
Chunk 4: Route Monitoring (390 tokens)
Chunk 5: Escalation Procedures (410 tokens)

--- Heading-Based ---
Chunk 1: "1. Introduction" (180 tokens)
Chunk 2: "2. Peer Requirements" (520 tokens)
Chunk 3: "3. Prefix Limits" (340 tokens)
Chunk 4: "4. Route Filtering" (480 tokens)
Chunk 5: "5. Monitoring" (390 tokens)
Chunk 6: "6. Escalation" (410 tokens)

Vector Search Query

Search the Knowledge Base for semantically similar content:

# Search for documents about BGP peering with AS65001
curl -s https://controller.example.com/api/v1/knowledge/search \
  -H "Authorization: Bearer $TOKEN" \
  -G -d "query=What is our BGP peering policy with AS65001?" \
  -d "limit=5" | jq '.'

# Example response
[
  {
    "chunk": {
      "id": "chunk-a1b2c3...",
      "document_id": "doc-9f8e7d...",
      "chunk_index": 1,
      "content": "All BGP peers in AS65001 must meet the following requirements: minimum 1Gbps link capacity, valid IRR records in RADB or ARIN, signed ROAs for all announced prefixes...",
      "token_count": 420
    },
    "document_title": "BGP Peering Policy - AS65001",
    "document_category": "network-policy",
    "similarity": 0.92
  },
  {
    "chunk": {
      "id": "chunk-d4e5f6...",
      "document_id": "doc-9f8e7d...",
      "chunk_index": 2,
      "content": "Prefix limits for AS65001 peers: maximum 500 IPv4 prefixes, maximum 100 IPv6 prefixes. Exceeding limits triggers a warning at 80% and session teardown at 100%...",
      "token_count": 340
    },
    "document_title": "BGP Peering Policy - AS65001",
    "document_category": "network-policy",
    "similarity": 0.87
  }
]

RAG-Powered AI Chat Response

When a user asks about organizational policy, AI Chat retrieves knowledge base context to provide an accurate, grounded response:

User: What is our maximum prefix limit for BGP peers in AS65001?

[Knowledge Base Search: 3 relevant chunks retrieved, similarity > 0.85]

AI:   Based on your BGP Peering Policy document, the maximum prefix limits
      for AS65001 peers are:

      - IPv4: 500 prefixes maximum
      - IPv6: 100 prefixes maximum

      The policy specifies a warning threshold at 80% (400 IPv4 / 80 IPv6)
      and automatic session teardown at 100%. You can configure these limits
      on Cisco IOS with:

      router bgp 65001
        address-family ipv4 unicast
          neighbor 10.0.1.2 maximum-prefix 500 80 restart 15

      This sets a 500 prefix limit with warning at 80% and automatic restart
      after 15 minutes if the session is torn down.

      Source: BGP Peering Policy - AS65001 (network-policy)

Questions & Answers

What document formats does the knowledge base support?

The Knowledge Base supports PDF, DOCX, TXT, and Markdown file uploads. For external sources, it can scrape web pages by URL, sync Confluence pages and spaces, sync SharePoint document libraries, and automatically ingest incident reports and terminal session transcripts. Each source type has its own content extraction and chunking pipeline.

How does vector search work?

Vector search converts your query into a vector embedding using the same model that embedded the document chunks. It then uses pgvector's cosine distance operator to find chunks whose embeddings are most similar to your query embedding. Results are ranked by similarity score (1.0 = identical meaning, 0.0 = completely unrelated). This allows the search to find conceptually similar content even when the exact words differ.

What chunking strategy should I use?

Use heading-based chunking for structured documentation with clear heading hierarchy (SOPs, policies, runbooks). Use semantic chunking for well-written articles and procedures where paragraphs are natural boundaries. Use fixed-size chunking for unstructured content like log files, raw text, or documents without clear structure. When in doubt, semantic chunking is a good default.

Can I connect to Confluence or SharePoint?

Yes. The Knowledge Base supports both Confluence and SharePoint as external document sources with automatic sync. For Confluence, provide your instance URL and API token, then select spaces or specific pages to index. For SharePoint, configure your tenant with OAuth credentials and select document libraries. Both sources support automatic re-sync on a configurable schedule to keep indexed content up to date.

How often are external sources re-synced?

External sources (Confluence, SharePoint, URLs) are re-synced on a configurable schedule. The default is daily. During re-sync, the system checks content hashes for changes and only re-processes documents that have been modified since the last sync. This minimizes embedding computation costs while keeping content current.

How does the knowledge base improve AI responses?

The Knowledge Base powers RAG (Retrieval-Augmented Generation) for AI Chat and NOC Agents. When you ask a question, the system searches the Knowledge Base for relevant chunks and includes them in the LLM prompt as context. This grounds the AI's response in your actual organizational documentation rather than general knowledge, producing answers that reference your specific policies, procedures, device configurations, and historical incidents.

Troubleshooting

Related Features

• AI Chat — Uses the Knowledge Base through RAG to provide organization-specific answers grounded in your documentation.
• NOC Agents — Autonomous agents that query the Knowledge Base for organizational context during network event investigation.
• LLM Configuration — Configure the embedding model used for vector search and the LLM providers that power RAG responses.
• Web Sources — Automatically crawl and ingest documentation sites into the Knowledge Base using breadth-first search.
• MCP Servers — Model Context Protocol servers that can provide additional data sources for knowledge enrichment.