Tier 2 standards sync: ISO, IEC & IEEE¶
Scholar MCP caches ISO, IEC, and IEEE standards metadata locally. The source of truth
is the community-maintained Relaton YAML dumps at
relaton/relaton-data-iso,
relaton/relaton-data-iec, and
relaton/relaton-data-ieee, read
from their v2 branch. Each repository's main branch no longer receives data.
Only a sync replaces a synced record. A lookup through get_standard or
resolve_standard_identifier caches what it fetched, but never over a synced
record, so a synced standard stays in search_standards and never expires
between syncs.
Running a sync¶
scholar-mcp sync-standards # all registered bodies
scholar-mcp sync-standards --body ISO # only ISO
scholar-mcp sync-standards --body IEEE # only IEEE
scholar-mcp sync-standards --force # re-sync even if upstream SHA is unchanged
Exit codes: 0 on success (or no-op), 1 on hard failure, 3 on partial failure
(some bodies succeeded, some did not).
After upgrading to a release that corrects synced records
A fix to how records are read from the Relaton dumps only reaches the local
copy on the next sync. Such a release forgets the upstream commit recorded
for ISO, IEC, and IEEE when the server first opens its cache, so the next
sync-standards run rewrites every record even if upstream has not
changed. Until that run,
the records keep what the earlier release stored. Release 3.0 needs this
once: ISO and IEC titles synced before it carry only the first part of the
title.
After upgrading to the release that reads the v2 branch
Earlier releases synced each repository's main branch, whose data stops
in 2025 or early 2026, and reported every ISO record as published. The
first sync-standards run after the upgrade rewrites every record from
v2 and deletes the synced identifiers v2 no longer lists, mostly drafts
that have since advanced and IEEE drafts that v2 names differently. Until
that run, search and lookups answer from the main data. Standards that
were only fetched live, never synced, are fetched again on their next lookup.
Running a sync from Docker¶
The scholar-mcp binary is on PATH inside the container, so no bash -c or
uv run wrapper is needed.
Exec into a running container (uses the existing container's open DB and env):
docker compose exec -u appuser scholar-mcp scholar-mcp sync-standards
The -u appuser flag is required because the container's default user is root
(the entrypoint uses gosu to drop privileges for the main process, but exec
bypasses the entrypoint). Running without it would create root-owned files in the
shared volume.
One-off via Docker Compose (goes through the entrypoint → drops to appuser):
docker compose run --rm scholar-mcp scholar-mcp sync-standards
The docker compose run form is preferred when running outside of an already-running
stack, it picks up the correct UID/GID from the entrypoint and inherits all env vars
from the environment: / env_file: declarations in compose.yml.
A sync can run while the server keeps answering. The cache opens its
database in WAL mode, so lookups read without waiting for the sync, and a
lookup that writes waits for the sync's current batch to finish, up to 30
seconds, rather than failing. WAL keeps two files beside cache.db,
cache.db-wal and cache.db-shm. Copy all three, or stop the server first,
when you back up the cache.
To sync only a specific body:
docker compose exec -u appuser scholar-mcp scholar-mcp sync-standards --body ISO
docker compose run --rm scholar-mcp scholar-mcp sync-standards --body ISO
Cron / systemd scheduling¶
Daily resync is plenty; Relaton dumps move slowly and the SHA check skips the tarball fetch when nothing has changed.
0 3 * * * /usr/local/bin/scholar-mcp sync-standards >> /var/log/scholar-mcp-sync.log 2>&1
Or with systemd:
# /etc/systemd/system/scholar-mcp-sync.service
[Unit]
Description=Sync scholar-mcp Tier 2 standards
[Service]
Type=oneshot
ExecStart=/usr/local/bin/scholar-mcp sync-standards
# /etc/systemd/system/scholar-mcp-sync.timer
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target
GitHub rate limiting¶
The sync makes 2 GitHub API calls per body (/commits/{branch} and /tarball/{sha}).
Unauthenticated clients get 60 requests per hour per IP, more than enough for daily
cron. For repeated --force testing, set SCHOLAR_GITHUB_TOKEN to a personal access
token (no scopes required for public-repo reads) to lift the limit to 5,000 req/hr:
export SCHOLAR_GITHUB_TOKEN=ghp_...
Live-fetch fallback¶
A get_standard("ISO 9001:2015") call on an unsynced cache triggers a single-file
fetch from raw.githubusercontent.com. Results are cached with a 90-day TTL (standards
rarely change) and can be replaced by running sync-standards to populate the full
synced copy.
Joint ISO/IEC standards¶
Standards like ISO/IEC 27001:2022 and ISO/IEC 15408:2022 appear in both dumps. The
loader stores each joint identifier once, and the row reflects the dump that last
wrote it. Its body is that dump's body, "ISO" or "IEC", not "ISO/IEC": the
records list identifiers of their own dump's type only, so the loader can't tell a
joint document from the record.
Troubleshooting¶
"removal pass aborted"¶
A Relaton dump keeps a withdrawn edition at its withdrawn stage, so an identifier
that disappears from it was renamed or advanced; the ISO, IEC and IEEE sync
deletes such records. When the new dump lists fewer than half as many records as
the sync held before, it looks truncated and the sync deletes nothing. Run
--force after confirming upstream state is sane.
"withdrawal pass aborted" or "removal pass aborted"¶
The CC sync marks records that disappear from its source as withdrawn. The
CEN table lists withdrawn standards itself, so the CEN sync deletes a record
whose entry left the table, which means it was never a harmonised standard.
Either sync skips that pass when more than 50% of prior identifiers are
missing, which protects the whole cache from a broken edit. Run --force after
confirming upstream state is sane.
Rate-limit errors (HTTP 403)¶
Indicates the unauthenticated limit has been exhausted. Set SCHOLAR_GITHUB_TOKEN or
wait an hour.
Inspecting last sync¶
Open the SQLite cache database (default $SCHOLAR_MCP_CACHE_DIR/scholar.db) and query
the standards_sync_runs table:
SELECT body, upstream_ref, added, updated, unchanged, withdrawn, errors,
datetime(started_at, 'unixepoch') AS started,
datetime(finished_at, 'unixepoch') AS finished
FROM standards_sync_runs;
To see how many standards are cached per body:
SELECT source, COUNT(*) AS count, datetime(MAX(synced_at), 'unixepoch') AS last_synced
FROM standards
WHERE source IS NOT NULL
GROUP BY source;
IEEE¶
IEEE metadata comes from the relaton/relaton-data-ieee repository.
Supported identifier forms¶
IEEE 1003.1-2024,IEEE 802.11-2020, plain IEEE standardsIEEE Std 1588-2019, theStdtoken is accepted (upstream convention)IEC/IEEE 61588-2021, joint standards hosted in the IEEE repoISO/IEC/IEEE 42010-2011, triple-joint standards (ISO + IEC + IEEE committees)
For joint standards the stored body field reflects the true joint nature (such as "ISO/IEC/IEEE"), while dispatch still routes through the "IEEE" fetcher key. Per-body filename conventions differ between repositories: ISO/IEC use lowercase-hyphen (iso-9001-2015.yaml), IEEE uses uppercase-underscore (IEEE_1003.1-2024.yaml, ISO_IEC_IEEE_42010-2011.yaml).
Not yet supported (filed follow-ups)¶
ANSI/IEEE Std 754-1985and other historical ANSI-co-branded identifiers, see #127AIEE 11-1937and other pre-IEEE AIEE identifiers, see #127IEEE P802.11-REVmeunpublished drafts, see #128- Explicit
body="IEC/IEEE"/body="ISO/IEC/IEEE"search dispatch, see #129. Today, use identifier lookup orbody="IEEE"which returns joints too. - IEEE Xplore authenticated full-text, see #92
Common Criteria¶
Common Criteria metadata is sync-only, there's no live API for the CC portal. Run:
scholar-mcp sync-standards --body CC
to populate the local cache with core CC documents and Protection Profiles.
Core CC documents (~15 records)¶
The core CC documents (CC:2022, CC:2017 (CC 3.1 Revision 5), and the Common Evaluation Methodology (CEM)) are stored in a hard-coded table in _sync_cc.py. Updates land manually when CCRA publishes a new release (every ~5 years).
CC core documents that are also published as ISO/IEC 15408 / 18045 (parts 1-3) write two records under one source="CC" sync, one with body="CC" (CC:2022 Part 1), one with body="ISO/IEC" (ISO/IEC 15408-1:2022). Both records share the same free CC PDF as full_text_url. The related field on each cross-links to the other.
Why two records? CC and ISO/IEC 15408 are parallel publications, not a joint committee output. Existing citations use either form (CC:2022 or ISO/IEC 15408) but never a joined form. To keep returned metadata matching what the LLM looked up, we store one record per publishing body.
To prevent collision with the existing ISO loader, the ISO loader has a small denylist (_RELATON_SKIP_SLUGS) covering the 15408 / 18045 family, those records are owned exclusively by the CC loader, which has the freely downloadable PDFs (vs. ISO's paywalled metadata).
Protection Profiles (~500 records)¶
Loaded from https://www.commoncriteriaportal.org/pps/pps.csv. Identifier extraction uses per-scheme regex with a composite fallback:
| Scheme | Code | Identifier example |
|---|---|---|
| German BSI | DE | BSI-CC-PP-0099-V2-2017 |
| Korean KECS | KR | KECS-PP-0822-2017 |
| French ANSSI | FR | ANSSI-CC-PP-2014/01 |
| US NIAP | US | NIAP-PP-... / PP_..._v3.1 |
| Spanish CCN | ES | CCN-PP-0058-2021 |
| Other | * | CC PP {scheme}-{name} (composite fallback) |
Supported identifier forms¶
CC:2022,CC:2017,CC 3.1 R5,CC 3.1 Revision 5CC:2022 Part 1,Common Criteria 2022 Part 1CEM:2022,Common Evaluation Methodology 2022ISO/IEC 15408-1:2022(resolves to the CC-owned dual record with free PDF)BSI-CC-PP-0099-V2-2017,KECS-PP-0822-2017, etc.
Not yet supported (filed follow-ups)¶
- ~6700 certified product certifications from sec-certs JSON, see #131
- CC Supporting Documents and Guidance Documents, see #132
- CEM Supplements and Application Notes, see #133
- Auto-discovery of new core documents from the portal HTML, see #134
CEN/CENELEC (European Norms)¶
CEN/CENELEC harmonised standards metadata is sync-only. Run:
scholar-mcp sync-standards --body CEN
to populate the local cache. Unlike ISO/IEC/IEEE (which sync from upstream Relaton repositories), CEN standards load from a curated hard-coded table covering the most-cited harmonised standards from the major EU directives.
Covered directives¶
| Directive | Shorthand | Example standards |
|---|---|---|
| Electromagnetic Compatibility 2014/30/EU | EMC | EN 55032, EN 61000-series |
| Radio Equipment 2014/53/EU | RED | EN 300 328, EN 301 489-series |
| Machinery 2006/42/EC | Machinery | EN ISO 12100, EN ISO 13849-1 |
| Medical Devices 2017/745 | MDR | EN ISO 13485, EN ISO 14971 |
| Toy Safety 2009/48/EC | Toys | EN 71-series |
| General Product Safety 2023/988 | GPSR | EN 14988, EN 1130 (childcare articles) |
| Low Voltage 2014/35/EU | LVD | EN 62368-1, EN 60335-1 |
A record is "harmonised" while the Official Journal cites it. A
withdrawn one is "withdrawn", with the date its presumption of conformity
ended in withdrawn_date and its replacement in superseded_by, as
EN 62061:2005 lists EN IEC 62061:2021. The Cyber Resilience Act has no
harmonised standards yet, so the table has no CRA entries. The Machinery
Directive's list ends on January 20, 2027, when the Machinery Regulation
(EU) 2023/1230 replaces it.
Supported identifier forms¶
EN 55032:2015, plain European NormEN ISO 13849-1:2023, ISO standard adopted as ENEN IEC 62443-3-3:2020, IEC standard adopted as ENEN ISO/IEC 27001:2022, ISO/IEC joint adopted as EN
All dispatch to body="CEN". No cross-linking to the ISO/IEC records, the EN ISO / EN IEC prefix is self-documenting.
Table maintenance¶
The _HARMONISED_STANDARDS table in _sync_cen.py needs periodic review when the EU Commission issues new decisions. A scheduled workflow checks it against the Official Journal each quarter and files what it finds as an issue labelled cen-review. It follows the procedure in docs/design/cen-table-review.md. If EUR-Lex infrastructure stabilises, #137 would automate this via Formex XML parsing.
Not yet supported¶
- Full CEN/CENELEC catalogue beyond harmonised standards
- Live scrape fallback from
standards.cencenelec.eu, see #138 - EUR-Lex Formex XML automated sync, see #137
Auto-enrichment¶
When get_citations, get_references, or get_citation_graph return S2 citation records, the StandardsEnricher automatically checks each citation's title. If the title is predominantly a standards identifier (the resolved canonical form covers >50% of the title text), the enricher attaches a standard_metadata field containing the full StandardRecord.
Which citations trigger enrichment¶
Citations whose title IS a standard, short, identifier-dominated titles:
| Citation title | Coverage | Triggers? |
|---|---|---|
RFC 9000 |
100% | Yes |
ISO 9001:2015 |
100% | Yes |
NIST SP 800-53 Rev. 5 |
100% | Yes |
IEEE 802.11-2020 |
100% | Yes |
EN 55032:2015 |
100% | Yes |
Citations that mention a standard but are papers ABOUT it:
| Citation title | Coverage | Triggers? |
|---|---|---|
Implementing ISO 27001 in healthcare: a review |
~20% | No |
A survey of NIST post-quantum candidates |
~15% | No |
What standard_metadata contains¶
A StandardRecord dict, the same shape returned by get_standard:
identifier, canonical form (such as"RFC 9000","ISO 9001:2015"). A W3C specification carries its common name where the server knows one ("WCAG 2.1") and the short name W3C gives it otherwise ("html401");get_standardaccepts either.title, full standard titlebody, standards body ("IETF","ISO","IEEE","CC","CEN", etc.)status,"published","withdrawn","superseded","draft","harmonised", etc. An ISO, IEC or IEEE record takes it from the stage the Relaton dump records; a cancelled ISO project reports"withdrawn".superseded_by, a list of the identifiers that replace this edition, each oneget_standardaccepts. An RFC that later RFCs obsolete is"superseded"and lists them: RFC 2616 givesRFC 7230toRFC 7235. A withdrawn ISO or IEEE edition whose dump record names its replacement does the same: ISO/IEC 27001:2005 givesISO/IEC 27001:2013. An IEEE replacement keeps the "Std" of its identifier (IEEE Std 1588-2019), andget_standardanswers it from the synced copy. A W3C specification that another one replaced lists it too, as Web Forms 2.0 listsHTML5, while a retired one with no named successor, such as HTML 4.01, is"superseded"with an empty list. An ETSI deliverable whose newest edition ETSI marks as replaced lists the other ETSI standards that replaced it (ETSI EN 300 328-1givesETSI EN 300 328); a historical or withdrawn one is"withdrawn". A NIST publication on CSRC that NIST withdrew lists what replaced it, asNIST SP 800-53 Rev. 4listsNIST SP 800-53 Rev. 5andFIPS 186-4listsFIPS 186-5; one withdrawn with no replacement named is"withdrawn". CSRC lists about 550 of the more than 11,000 NIST publications, and the rest always read"published". The server reads the CSRC list along with the catalogue every 90 days, so a withdrawal can take that long to show. NIST identifiers spell out a part, volume or version, asNIST SP 800-57 Part 1 Rev. 5andNIST SP 800-160 Vol. 2, and keep a numbered edition such asNIST SP 800-73-4. A publication on CSRC that the NIST catalogue lacks, such asFIPS 186-3or the 2005NIST SP 800-53, has a record built from the CSRC list: its title, date, status and link, with no abstract (scopeis empty). The CSRC white papers (CSWP) have no records. An IEC edition that IEC marks as revised or replaced lists the editions that followed it, as IEC 60335-2-91:2008 listsIEC 62841-4-4:2020. About a quarter of them name no successor in the IEC dump and keep an empty list, and a few list every part of the family that replaced them. A withdrawn IEC edition lists nothing. The firstsync-standardsrun after upgrading to the release that fills these lists reads the IEC dump again, even if it has not changed; until that run, IEC editions keep an empty list. The other bodies leave the list empty, so an empty list doesn't show that an edition is current. IETF, NIST, W3C and ETSI records cached by an earlier release are fetched again on their next lookup, so they pick up these fields.full_text_url, direct link to the freely downloadable document (when available)full_text_available,Truefor Tier 1 sources (IETF, NIST, W3C, ETSI) and CCrelated, cross-linked identifiers (such as CC ↔ ISO/IEC 15408)
Tier 2 bodies require sync¶
ISO, IEC, IEEE, CC, and CEN metadata comes from the synced cache. If sync-standards hasn't been run, Tier 2 citations won't get enriched (Tier 1 citations, RFC, NIST SP, WCAG, ETSI, always work via live API lookup).