Developer guide

WebWhisper Documentation

Index your sources once, then connect your own interface to grounded answers.

What WebWhisper does

WebWhisper turns public website pages and supported uploaded files into a searchable, per-account knowledge collection. You configure an OpenAI or Gemini provider key, ingest your content, then ask questions through the dashboard or the API.

The chat uses retrieval-augmented generation (RAG): it embeds the question, searches that account's indexed chunks using vector similarity and (when enabled) PostgreSQL full-text search, and gives relevant context to the selected language model. The response and source references are returned when available. If the indexed content does not support an answer, the pipeline can return a no-relevant-content response instead of treating the question as grounded.

Each account currently has one isolated collection. Multiple named knowledge bases are not implemented.

How the platform works

  1. Create an account and sign in. Email verification is required only when enabled by the deployment.
  2. Configure an OpenAI or Gemini API key in Dashboard Settings.
  3. Add website URLs in URL Manager, or upload PDF, DOCX, TXT, Markdown, or supported image files in Documents.
  4. Start the scrape or document-processing job from its dashboard action. URL extraction and document parsing are queued through Celery.
  5. Extracted text is staged, split into chunks, embedded with the configured provider, and stored in PostgreSQL with pgvector. PostgreSQL full-text search is also available for hybrid retrieval.
  6. Ask a question in Chat or send it to the API. WebWhisper retrieves chunks scoped to the account, applies configured guardrails, and calls the selected provider.
  7. The chat response includes an answer, source list, and session ID. Sources may be empty when retrieval finds no usable source.

The crawler currently defaults to one page per URL and zero crawl depth. Image processing requires a configured vision model. The API offers chat against your account's collection; it does not expose endpoints to create multiple collections.

API authentication and keys

Create a key in Dashboard → API Keys. The raw key is displayed only once. Store it in a server-side environment variable or secret manager; do not put it in browser JavaScript, source control, or URLs.

The API base URL is the backend origin. Local development defaults to http://localhost:8000; deployments must use the backend URL configured for that environment. The frontend's NEXT_PUBLIC_API_URL setting is for frontend-to-backend calls and should not be used to expose secrets.

Preferred authentication headers:

  • X-API-Key: ww_…
  • Authorization: Bearer ww_…

The legacy api_key query parameter is accepted, but is discouraged because URLs may be recorded in logs and referrers.

Public chat endpoint

POST/api/v1/chat

Request JSON (the session identifier is optional):

{
  "message": "What is the return policy?"
}

To continue a conversation, pass a real session UUID as session_id; without it, the API creates a session for the turn.

Successful responses contain:

{
  "answer": "The indexed content says ...",
  "sources": ["https://example.com/returns"],
  "session_id": "returned-session-uuid"
}

Typical responses include 200 for success, 401 for a missing/invalid/revoked key, 403 for a quota or access restriction, 400 when an LLM key is not configured, and 429 when the per-key rate limit is exceeded.

curl
curl --fail-with-body --request POST "$WEBWHISPER_API_URL/api/v1/chat" \
  --header "Content-Type: application/json" \
  --header "X-API-Key: $WEBWHISPER_API_KEY" \
  --data '{"message":"What is the return policy?"}'

Build your own knowledge-base chatbot

  1. Create and verify a WebWhisper account if verification is enabled.
  2. Add URLs or supported documents in the dashboard and wait for their processing status to complete.
  3. Set your OpenAI or Gemini provider key in Dashboard Settings and generate an API key under API Keys.
  4. Keep the WebWhisper key on your server, for example as WEBWHISPER_API_KEY; set WEBWHISPER_API_URL to the backend origin for the current environment.
  5. Install the client library with python -m pip install requests.
  6. Call POST /api/v1/chat from your backend. Your frontend should call your backend, not WebWhisper with the secret key.
  7. Render answer and, when present, the items in sources. Keep the returned session_id if you want follow-up messages to continue the same conversation.
  8. Handle timeouts, authentication failures, quota responses, and 429 responses. Respect Retry-After when supplied, and use bounded retries with backoff.
  9. Test with a question whose answer is present in your indexed sources and inspect the returned sources.
Python with requests
import os
import requests

base_url = os.environ["WEBWHISPER_API_URL"].rstrip("/")
api_key = os.environ["WEBWHISPER_API_KEY"]

try:
    response = requests.post(
        f"{base_url}/api/v1/chat",
        headers={"X-API-Key": api_key},
        json={"message": "What is the return policy?"},
        timeout=(3.05, 60),
    )
    response.raise_for_status()
    result = response.json()
except requests.Timeout:
    raise SystemExit("WebWhisper timed out; retry with backoff.")
except requests.RequestException as exc:
  raise SystemExit(f"Could not reach WebWhisper: {exc}")
except requests.HTTPError as exc:
    status = exc.response.status_code
    if status == 401:
        raise SystemExit("API key is missing, invalid, or revoked.")
    if status == 403:
        raise SystemExit("A plan quota or access restriction was reached.")
    if status == 429:
        retry_after = exc.response.headers.get("Retry-After")
        raise SystemExit(f"Rate limited; Retry-After={retry_after}")
    raise

print(result["answer"])
for source in result.get("sources", []):
    print("Source:", source)

This architecture lets your application own the UI and user experience while WebWhisper handles source ingestion, chunking, retrieval, provider calls, and session-aware answers.

Limits and error handling

The public chat endpoint is limited by API key; the default configuration is 60 requests per minute per key. The default free-plan seed is 25 URLs, 25 documents, 100 MB storage, 20 MB scraped content, 200,000 tokens per month, 200 chats per day, and retrieval top-k of 8. Plan rows and administrator overrides control actual account quotas, so these defaults are not a guarantee for every account. Deployment hard caps may apply as well.

Uploads default to a 50 MB maximum file size, and upload and scraping actions have separate per-user rate limits. The deployment can change these settings. A 429 response indicates throttling; a 403 can indicate an exceeded plan quota; 401 means the API key must be checked or rotated.

  • 400: request cannot be processed, such as missing provider configuration.
  • 401: API key is missing, invalid, or revoked.
  • 403: access or plan quota restriction.
  • 429: per-key request limit exceeded.
  • 5xx or network timeout: temporary service or upstream provider failure; retry cautiously with backoff.

Environment configuration

Configure these values in the backend application or secret manager. Do not commit credentials.

  • WEBWHISPER_API_URL: backend origin used by your integration.
  • WEBWHISPER_API_KEY: key created from the WebWhisper dashboard.
  • NEXT_PUBLIC_API_URL: frontend API origin for a WebWhisper deployment; it is public configuration and must not contain secrets.
FastAPI backend
import os

import requests
from fastapi import FastAPI, HTTPException

app = FastAPI()
API_URL = os.environ["WEBWHISPER_API_URL"].rstrip("/")
API_KEY = os.environ["WEBWHISPER_API_KEY"]

@app.post("/ask")
def ask(question: str):
    try:
        response = requests.post(
            f"{API_URL}/api/v1/chat",
            headers={"X-API-Key": API_KEY},
            json={"message": question},
            timeout=(3.05, 60),
        )
    except requests.Timeout:
        raise HTTPException(status_code=504, detail="Knowledge service timed out")
    except requests.RequestException:
        raise HTTPException(status_code=502, detail="Knowledge service unavailable")

    if response.status_code == 401:
        raise HTTPException(status_code=502, detail="Knowledge service key rejected")
    if response.status_code == 429:
        raise HTTPException(status_code=503, detail="Knowledge service rate limited")
    if not response.ok:
        raise HTTPException(status_code=502, detail="Knowledge service request failed")
    return response.json()

Install and run the example in your application environment:

python -m pip install fastapi uvicorn requests
export WEBWHISPER_API_URL="http://localhost:8000"
export WEBWHISPER_API_KEY="<key-from-dashboard>"
uvicorn app:app --reload

Then send a test request to your local wrapper:

curl --request POST "http://localhost:8000/ask?question=What%20is%20the%20return%20policy%3F"

For production, use your deployment's HTTPS API origin and a managed secret store. The local URL above is only for local development.