How to maintain the database¶
Everything lives in the repository: the database db/premal.sqlite, the schema db/schema.sql,
the vocabulary db/vocab.json, the tool tools/paperdb.py and the generated pages in wiki/.
The PDFs under papers/ are not tracked by git; the database is what carries the extracted details.
Adding papers¶
- Drop the PDF into the right folder under
papers/(see the collections). For the PPML collection usepapers/references/ppml/<topic>/<firstauthor>-<year>_<short-title>.pdf. -
Run the local pipeline:
python3 tools/paperdb.py scan # new file: hash, text, identifiers, title guess python3 tools/paperdb.py link # attach it to an existing paper (same DOI/arXiv/ePrint/title) or create one python3 tools/paperdb.py enrich # optional, needs network: Crossref, arXiv, Semantic Scholar python3 tools/paperdb.py tag # keyword mentions python3 tools/paperdb.py build-docs && mkdocs build -
Check the new page under Papers and fix anything wrong with
set(see below).
If the same work already exists, link merges the new file into that paper
(same SHA-256, same arXiv/ePrint id, same curated entry or identical normalised title).
The duplicates page lists what was merged and what looks similar but was not.
Editing details by hand¶
python3 tools/paperdb.py set KEY title "Correct title" --reviewed
python3 tools/paperdb.py set KEY authors "Given Family; Given Family"
python3 tools/paperdb.py set KEY paper_type survey
python3 tools/paperdb.py set KEY notes "Preprint of the journal version"
python3 tools/paperdb.py relate KEY1 KEY2 version_of
--reviewed marks the paper as reviewed: enrich and link will not overwrite its fields again.
Any SQL works too, e.g. sqlite3 db/premal.sqlite and then
INSERT INTO paper_details(paper_id, field, value, source) VALUES (42, 'datasets', 'MNIST', 'manual');.
Structured (LLM) extraction¶
paper_details rows with source = 'llm' were produced by reading excerpts of each PDF with a language model.
They are marked unreviewed on the pages. To redo or extend them:
python3 tools/paperdb.py excerpts /tmp/excerpts --missing-only # writes one text file per paper + _schema.json
# ... have the excerpts read and out/<key>.json written for each ...
python3 tools/paperdb.py import-details /tmp/excerpts
Updating the curated tables¶
docs/data/schemes.json, docs/data/methods.json and docs/data/sok_references.js stay the source of the
scheme table, the systems page and the reference list. After editing them run link and build-docs.
Searching¶
python3 tools/paperdb.py search "programmable bootstrapping" # FTS5 over the page text
python3 tools/paperdb.py search "GELU NEAR/5 approximation"
Rebuilding from scratch¶
rm db/premal.sqlite db/fulltext.sqlite
python3 tools/paperdb.py init && python3 tools/paperdb.py scan && python3 tools/paperdb.py link
python3 tools/paperdb.py enrich && python3 tools/paperdb.py tag && python3 tools/paperdb.py build-docs