docs-translator

Self-hosted document translation service that preserves layout. Upload a PDF, DOCX, TXT, or Markdown file and get back the same file translated into any language — with paragraphs, headings, tables, fonts, and even figures kept in place.

Backed by FastAPI + Celery + PostgreSQL, powered by either the local Claude CLI (no API key needed) or the OpenAI API.

This repository contains the backend only. The Next.js frontend lives in a separate repository.

Most translation tools throw away layout. This one has three independent pipelines, each tuned for a different trade-off:

Each pipeline shares auth, uploads, and Celery infrastructure but writes to different tables and ships as a different task. They don't compete — they cover different problems.

• Models: TranslationJob, TranslationChunk
• Worker: app.workers.tasks.run_translation
• Flow: parse → paragraph-aware chunking (max 2500 chars, sentence-fallback) → translate chunk by chunk → progress flushed after every chunk
• Idempotent retries: chunk rows use deterministic uuid5(NAMESPACE_OID, f"{job_id}:{i}") IDs with ON CONFLICT DO UPDATE, so retries never duplicate
• DOCX layout preservation on download: the download endpoint re-opens the original .docx and swaps each paragraph's text in place, preserving headings, run formatting, and tables — falls back to a plain generated DOCX on any error

• Models: PdfJob, PdfBlock
• Worker: app.pdf_translator.pipeline.run_pdf_translation
• Three explicit phases:
  1. ANALYZING — PyMuPDF extracts paragraphs, span bboxes, and font metadata into pdf_blocks
  2. TRANSLATING — skips non-translatable blocks (pure numbers/labels), checks Redis cache, batches 10 paragraphs per Claude call, expects strict JSON array response
  3. RENDERING — redacts original span bboxes in IMAGE_NONE mode so diagrams survive, then reinserts translated text with a font-shrink ladder (95%→50%) and bbox expansion only into free space, never into a neighbour block
• Resilience: translate_batch_with_retry does exponential backoff, then split-and-retry (halves the batch until each half succeeds or is size 1), then up to 2 additional passes over FAILED blocks. Items the translator finally gives up on keep the source text so the page still renders.
• Celery task has time_limit=6h, soft_time_limit=5h (600-page books take 1–3 hours)

• Models: BookJob, BookElement
• Worker: app.book_translator.pipeline.run_book_translation
• Built for large books where translated text overflows original bboxes. Instead of forcing text back into page geometry, it reflows into a fresh DOCX.
• Three phases:
  1. ANALYZING — extracts ordered TEXT paragraphs + rasterized IMAGE regions for figures/tables/diagrams; running headers/footers and page numbers are dropped; median body font size is stored so the builder can map larger paragraphs to Word Heading 1/2/3 styles
  2. TRANSLATING — same infrastructure as pipeline 2 (Redis cache, batch retries)
  3. BUILDING — builder.py emits one DOCX in strict seq_index order, re-inserting PNG images in place
• DOCX → PDF on demand: primary output is DOCX; requesting format=pdf triggers conversion via docx2pdf (Windows/MS Word COM) or LibreOffice headless (Docker). Result is cached for subsequent downloads.

Three logical DBs on the same Redis instance:

• /0 — app cache (used by PDF translation cache)
• /1 — Celery broker
• /2 — Celery result backend

app/services/translator.py is the only call site for generic translation. It dispatches to either:

• claude_client.py — subprocess claude -p --output-format text, reads ~/.claude for auth. Docker mounts the host ~/.claude into the worker container.
• openai_client.py — OpenAI SDK with tenacity retries

Selected by the TRANSLATION_PROVIDER env var (claude default, or openai).

git clone https://github.com/Mirxonjon/docs-translator-backend.git
cd docs-translator-backend

cp .env.example .env
# Edit .env: at minimum set JWT_SECRET

docker compose up -d --build
docker compose logs -f api worker

The API container automatically runs alembic upgrade head and seeds the admin user.

Services:

• API: http://localhost:8000/v1
• Swagger: http://localhost:8000/docs
• Flower (Celery UI): http://localhost:5555

Note: docker-compose.yml does not ship a Postgres service — containers connect to the host Postgres via host.docker.internal. Run Postgres yourself (local install or a separate container), then start the rest.

python -m venv .venv
.venv\Scripts\activate          # Windows
# source .venv/bin/activate     # macOS/Linux

pip install -r requirements.txt
cp .env.example .env

# Bring up Redis (Postgres runs on your host)
docker compose up -d redis

alembic upgrade head
python -m app.db.seed           # creates admin@docs.example.com / Admin123!

# Terminal 1: API
make api                        # or: uvicorn app.main:app --reload

# Terminal 2: Celery worker
make worker

# Terminal 3 (optional): Flower
make flower

All settings are environment variables. See .env.example for the full list. Highlights:

GET /v1/health/ → {app, postgres, redis, timestamp}

# 1. Login
TOKEN=$(curl -s -X POST http://localhost:8000/v1/auth/login \
  -H 'Content-Type: application/json' \
  -d '{"email":"admin@docs.example.com","password":"Admin123!"}' | jq -r .access_token)

# 2. Upload a PDF
DOC_ID=$(curl -s -X POST http://localhost:8000/v1/documents/upload \
  -H "Authorization: Bearer $TOKEN" \
  -F "file=@./contract.pdf" | jq -r .id)

# 3. Kick off a layout-preserving PDF translation
JOB_ID=$(curl -s -X POST http://localhost:8000/v1/pdf/jobs/ \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d "{\"document_id\":\"$DOC_ID\",\"target_lang\":\"uz\"}" | jq -r .id)

# 4. Poll progress
curl -s http://localhost:8000/v1/pdf/jobs/$JOB_ID \
  -H "Authorization: Bearer $TOKEN" | jq '{status, progress, done_blocks, total_blocks}'

# 5. Download the translated PDF
curl -s -o translated.pdf -H "Authorization: Bearer $TOKEN" \
  http://localhost:8000/v1/pdf/jobs/$JOB_ID/download

make test          # pytest
make lint          # ruff check app tests
make format        # black app tests
make migrate       # alembic upgrade head
make revision m="add field foo"   # autogenerate migration

Job status enums are stored as Postgres ENUM types. Adding a new value requires an Alembic migration that updates the enum — a plain model change isn't enough.

Both PDF pipelines re-insert translated text via PyMuPDF, which needs system fonts that cover the target script (Cyrillic, Uzbek ʻ, CJK, etc.). The Dockerfile installs fonts-noto* for this; on a bare host venv without Noto installed, the renderer will silently fall back to tofu/boxes.

• Admin: admin@docs.example.com / Admin123!
• User: user@docs.example.com / User123!

Change these immediately in any deployment.

Issues and pull requests are welcome. If you're adding a new provider, follow the pattern in app/services/translator.py — dispatch by env var, keep the facade contract stable.

MIT — see LICENSE.