KBTDapp - Knowledge Base AI

100% Local AI-Powered Knowledge Base with RAG, Quiz Generation & Privacy-First Design

An interactive AI-powered application for building, querying, and learning from your own private knowledge base - entirely on your machine. No cloud APIs, no data sharing, completely free to run.

• 🔒 Privacy-First: All AI processing happens locally - your data never leaves your machine
• 💰 Completely Free: Uses open-source Ollama models (no API costs)
• 🚀 Fast & Powerful: Optimized for Apple Silicon (M1/M2/M3/M4) with native ARM64 support
• 📚 Multi-Source Ingestion: Import from files, Confluence, web scraping, or direct text
• 🧠 Smart RAG System: Retrieval Augmented Generation with semantic search (pgvector)
• 🎯 AI Quiz Generation: Automatically generates practice quizzes from your knowledge base
• 🎨 Beautiful UI: Clean, modern interface with responsive design

Chat interface that answers questions using your knowledge base through RAG (Retrieval Augmented Generation). Features:

• Semantic similarity search with pgvector
• Configurable match thresholds and result counts
• Source citations for every answer
• 5 personality styles (professional, cheerful, mentor, pirate, custom)
• Feedback learning system (upvote/downvote answers to improve future responses)

Import and index content from multiple sources into a searchable vector database:

• File Upload: PDF, Word (DOCX/DOC), text files, images (with OCR via Tesseract.js)
• Confluence API: Import single pages or entire page trees (with version tracking)
• Web Scraping: 4 modes - single URL, multiple URLs, sitemap.xml, or recursive crawling
• Direct Text: Paste content directly into the knowledge base

AI-generated quizzes for knowledge retention and practice:

• Auto-extracts topics from your knowledge base
• Generates multiple choice and true/false questions
• Instant feedback with detailed explanations
• Progress tracking and score history

• Frontend & Backend: Next.js 14 with TypeScript
• Database: PostgreSQL 17 with pgvector extension (native installation)
• AI/LLM: Ollama (local LLM - runs llama3, nomic-embed-text)
• Styling: Tailwind CSS
• Vector Search: pgvector for semantic similarity search

Before you begin, ensure you have:

1. macOS (this guide is for macOS)
2. Homebrew - Install
3. Node.js 20+ - brew install node or Download

# Clone the repository
git clone https://github.com/adher-code/knowledge-base-ai.git
cd knowledge-base-ai

# Install dependencies
npm install

Download the latest release from GitHub and extract it.

npm install

# Install PostgreSQL 17
brew install postgresql@17

# Install pgvector extension
brew install pgvector

# Start PostgreSQL service
brew services start postgresql@17

# Wait a moment, then create database
/opt/homebrew/opt/postgresql@17/bin/createdb kbtdapp

# Initialize database with schema
/opt/homebrew/opt/postgresql@17/bin/psql kbtdapp < init-db.sql

# Install Ollama
brew install ollama

# Start Ollama service
brew services start ollama

# Pull required AI models
ollama pull llama3
ollama pull nomic-embed-text

# Copy example file
cp .env.example .env.local

# Get your username
whoami

# Edit .env.local and set DB_USER to the output from above
nano .env.local
# or
code .env.local  # if using VS Code

For detailed setup instructions (including Confluence integration), see SETUP_SECRETS.md

npm run dev

Open http://localhost:3000 in your browser.

kbtdapp/
├── app/                    # Next.js app directory
│   ├── api/               # API routes (future)
│   ├── layout.tsx         # Root layout
│   ├── page.tsx           # Home page
│   └── globals.css        # Global styles
├── components/            # React components
├── lib/                   # Utility libraries
│   ├── db.ts             # Database connection and helpers
│   ├── ollama.ts         # Ollama client
│   └── rag.ts            # RAG implementation
├── init-db.sql           # Database initialization script
└── package.json          # Dependencies

npm run dev      # Start development server
npm run build    # Build for production
npm run start    # Start production server
npm run lint     # Run ESLint

# Connect to database
/opt/homebrew/opt/postgresql@17/bin/psql kbtdapp

# View knowledge base content
/opt/homebrew/opt/postgresql@17/bin/psql kbtdapp -c "SELECT * FROM knowledge_base;"

# Stop PostgreSQL
brew services stop postgresql@17

# Restart PostgreSQL
brew services restart postgresql@17

# List installed models
ollama list

# Test llama3
ollama run llama3

# Check Ollama status
curl http://localhost:11434/api/tags

This project uses git-flow. See CLAUDE.md for details.

# Start new feature
git flow feature start <feature-name>

# Finish feature
git flow feature finish <feature-name>

Upload documents directly from your local filesystem with automatic text extraction:

• PDF (.pdf) - Portable Document Format
• Word (.docx, .doc) - Microsoft Word documents
• Text (.txt, .md) - Plain text and Markdown files

• Drag-and-drop interface
• Automatic text extraction from all formats
• File validation (type and size)
• Max file size: 10MB
• 100% local processing - files never leave your machine

1. Navigate to /knowledge-base
2. Find the orange "Upload Files" section
3. Drag and drop a file or click "Choose File"
4. Click "Upload File" to process

The system will automatically extract text, generate embeddings with Ollama, and store in your local database.

The application includes a comprehensive web scraping system for importing public web content into your knowledge base.

1. Single URL: Scrape one specific page
2. Multiple URLs: Batch scrape a list of URLs (one per line)
3. Sitemap: Parse sitemap.xml and scrape all listed pages
4. Crawl: Start at a landing page and recursively follow internal links (up to 5 layers deep)

• Smart Content Extraction: Auto-detects main content or use custom CSS selectors
• Metadata Extraction: Title, description, keywords, author, publish date, Open Graph tags
• Rate Limiting: 2-second delay between requests (max 50 pages per batch)
• Ethical Scraping: Respects server resources, avoids DoS-style behavior
• Link Following: Crawl mode automatically follows same-domain links
• 100% Local Processing: All content processed locally with Ollama embeddings

1. Navigate to /knowledge-base
2. Scroll to the green "Web Scraping" section
3. Choose your scraping mode:
   • Single: Enter one URL
   • Multiple: Paste URLs (one per line)
   • Sitemap: Enter sitemap.xml URL with optional limit
   • Crawl: Enter starting URL and depth (0-5 layers)
4. Optionally add a CSS selector for targeted content
5. Click "Start Web Scraping"

• Static HTML only: JavaScript-rendered SPAs require headless browser (see BACKLOG.md)
• Max 50 pages: Per request to prevent server overload
• Same-domain only: Crawl mode only follows internal links

# Check if PostgreSQL is running
brew services list | grep postgresql

# Restart PostgreSQL
brew services restart postgresql@17

# Check database exists
/opt/homebrew/opt/postgresql@17/bin/psql -l | grep kbtdapp

# Check Ollama status
brew services list | grep ollama

# Restart Ollama
brew services restart ollama

# Test connection
curl http://localhost:11434/api/tags

1. ✅ Local environment setup complete
2. ✅ Learning Mode (Confluence + Web scraping)
3. 🚧 Support Mode (RAG-based chat) - in progress
4. ⏳ Teaching Mode (quizzes)

Verify your setup with the comprehensive test suite (28 tests):

node scripts/test-all.js

Expected output: ✅ All tests passed (28/28)

Test coverage:

• Database connectivity (PostgreSQL + pgvector)
• Ollama services (llama3 + nomic-embed-text)
• Document CRUD operations
• Vector similarity search
• All API endpoints
• RAG pipeline end-to-end

See TESTING_GUIDE.md for manual testing procedures.

• QUICKSTART.md - 3-step startup guide
• START.md - Detailed setup and service management
• SETUP_SECRETS.md - Environment variables setup (including Confluence)

• FILE_UPLOAD_FEATURE.md - File upload documentation
• OCR_IMAGE_FEATURE.md - Image OCR with Tesseract.js

• FAQ.md - Frequently asked questions
• TESTING_GUIDE.md - Testing procedures
• MIGRATION_GUIDE.md - Move to a new laptop

• CLAUDE.md - Comprehensive technical documentation
• SECURITY.md - Security policy and credential management
• BACKLOG.md - Feature roadmap

Optimized for Apple Silicon (M1/M2/M3/M4):

• Embedding generation: 2-3x faster with Neural Engine
• LLM inference: 3-5x faster vs Intel
• Overall RAG queries: 2-4 seconds (vs 8-12 seconds on older Macs)

Benchmark on M4 Mac:

• Question embedding: ~0.5 seconds
• Vector search: ~0.1 seconds
• Answer generation: ~1-3 seconds
• Total RAG pipeline: 2-4 seconds

Contributions are welcome! Here's how to get started:

1. Fork the repository on GitHub
2. Clone your fork: git clone https://github.com/YOUR_USERNAME/knowledge-base-ai.git
3. Create a feature branch: git checkout -b feature/your-feature-name
4. Make your changes and test thoroughly
5. Commit your changes: git commit -m "Add your feature"
6. Push to your fork: git push origin feature/your-feature-name
7. Open a Pull Request on GitHub

Before submitting:

• Run tests: node scripts/test-all.js
• Run linter: npm run lint
• Test build: npm run build
• Update documentation if needed

See SECURITY.md for security guidelines.

See BACKLOG.md for planned features:

• Headless browser support for JS-rendered sites (Puppeteer/Playwright)
• Document chunking for large files
• Batch embedding generation
• Admin dashboard for learned patterns
• Export/import learned patterns
• Spaced repetition algorithm for Teaching Mode
• Multi-language support

This project is licensed under the MIT License - see the LICENSE file for details.

TL;DR: You can freely use, modify, and distribute this software, even for commercial purposes.

• 100% Local: Everything runs on your machine (PostgreSQL + Ollama)
• No Cloud APIs: No data sent to OpenAI, Anthropic, or any external AI service
• No Telemetry: We don't collect usage data or analytics
• Your Data Stays Yours: Database and models stored locally

See SECURITY.md for credential management best practices.

Built with:

• Next.js 14 - React framework
• PostgreSQL 17 - Database
• pgvector - Vector similarity search
• Ollama - Local LLM runtime
• Llama 3 - 8B parameter chat model
• nomic-embed-text - Embedding model
• Tesseract.js - OCR engine

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Documentation: See Documentation section above

If you find this project useful, please consider giving it a star on GitHub!

Made with ❤️ for the privacy-conscious AI community