Skip to content

How to maintain the database

Everything lives in the repository: the database db/premal.sqlite, the schema db/schema.sql, the vocabulary db/vocab.json, the tool tools/paperdb.py and the generated pages in wiki/. The PDFs under papers/ are not tracked by git; the database is what carries the extracted details.

Adding papers

  1. Drop the PDF into the right folder under papers/ (see the collections). For the PPML collection use papers/references/ppml/<topic>/<firstauthor>-<year>_<short-title>.pdf.
  2. Run the local pipeline:

    python3 tools/paperdb.py scan      # new file: hash, text, identifiers, title guess
    python3 tools/paperdb.py link      # attach it to an existing paper (same DOI/arXiv/ePrint/title) or create one
    python3 tools/paperdb.py enrich    # optional, needs network: Crossref, arXiv, Semantic Scholar
    python3 tools/paperdb.py tag       # keyword mentions
    python3 tools/paperdb.py build-docs && mkdocs build
    
  3. Check the new page under Papers and fix anything wrong with set (see below).

If the same work already exists, link merges the new file into that paper (same SHA-256, same arXiv/ePrint id, same curated entry or identical normalised title). The duplicates page lists what was merged and what looks similar but was not.

Editing details by hand

python3 tools/paperdb.py set KEY title "Correct title" --reviewed
python3 tools/paperdb.py set KEY authors "Given Family; Given Family"
python3 tools/paperdb.py set KEY paper_type survey
python3 tools/paperdb.py set KEY notes "Preprint of the journal version"
python3 tools/paperdb.py relate KEY1 KEY2 version_of

--reviewed marks the paper as reviewed: enrich and link will not overwrite its fields again. Any SQL works too, e.g. sqlite3 db/premal.sqlite and then INSERT INTO paper_details(paper_id, field, value, source) VALUES (42, 'datasets', 'MNIST', 'manual');.

Structured (LLM) extraction

paper_details rows with source = 'llm' were produced by reading excerpts of each PDF with a language model. They are marked unreviewed on the pages. To redo or extend them:

python3 tools/paperdb.py excerpts /tmp/excerpts --missing-only   # writes one text file per paper + _schema.json
# ... have the excerpts read and out/<key>.json written for each ...
python3 tools/paperdb.py import-details /tmp/excerpts

Updating the curated tables

docs/data/schemes.json, docs/data/methods.json and docs/data/sok_references.js stay the source of the scheme table, the systems page and the reference list. After editing them run link and build-docs.

Searching

python3 tools/paperdb.py search "programmable bootstrapping"    # FTS5 over the page text
python3 tools/paperdb.py search "GELU NEAR/5 approximation"

Rebuilding from scratch

rm db/premal.sqlite db/fulltext.sqlite
python3 tools/paperdb.py init && python3 tools/paperdb.py scan && python3 tools/paperdb.py link
python3 tools/paperdb.py enrich && python3 tools/paperdb.py tag && python3 tools/paperdb.py build-docs