Bounded RIS Reference Import

Every reference-format import (CSV/TSV, EndNote XML, RIS) now runs through one streaming record pipeline; see Streaming record pipeline for the passes, the batch hand-off, the limits and the error model (S10, #4597).

Outcome

Researchers can manually select and import a RIS reference file without first converting it to SyRF CSV or EndNote XML. The first delivery is intentionally limited to a safe, reviewable RIS path through the existing search-upload workflow.

This brief replaces the broad proposal in PR #2628. That proposal is retained in Git history as research input, but its architecture decisions and delivery claims are not authoritative.

Current State

PR #2971 delivered the bounded RIS slice on 2026-08-30. PR #3004 then closed the study-export formula-safety gap exposed during final review:

• the upload dialog offers explicit EndNote XML, CSV, and RIS choices; RIS is manually selected and there is no format auto-detection;
• LibraryFileType.Ris is append-only value 5; historical persisted values remain unchanged;
• RIS travels through the existing signed-upload, object-storage notification, parser-registry, and Study persistence path;
• validation is based on RIS structure rather than extension or MIME claims;
• supported input encodings are strict UTF-8, UTF-8 BOM, UTF-16 LE/BE with BOM, and Windows-1252;
• the original upload is retained, while successful imports produce a bounded augmented RIS object for the current workflow;
• fatal validation, persistence, storage, and cancellation paths do not intentionally expose partially imported studies;
• warnings are bounded and sanitised;
• study CSV/TSV exports use SyRF's reversible marker for dangerous spreadsheet prefixes while preserving generated finite negative numbers as numeric values;
• TY maps only to the current Study reference type and never to PRISMA acquisition-source provenance;
• PubMed format (NBIB) import is implemented in the backend (LibraryFileType.PubmedNbib = 6, see PubMed format (NBIB) import) but is not offered in the upload dialog yet;
• Zotero RDF import is implemented in the backend (LibraryFileType.ZoteroRdf = 7, see Zotero RDF import) and is offered only through the search import wizard;
• BibTeX, CSL-JSON, external CSV profiles, and format auto-detection remain unimplemented follow-ups;
• the approved future data direction uses system-scoped pmPublication records and immutable project-scoped Citation values, but that model is not yet implemented.

Streaming record pipeline (every reference format)

Owner direction D8 (S10, #4597): parsers and processors stream, so very large libraries import in bounded memory, and every format shares one error model. One orchestrator, RecordPipelineImport<TRecord> (ParserImplementations/NewParser/RecordPipelineImport.cs), runs every reference-format import. A format plugs in a RecordPipelineFormat<TRecord>: an IPipelineRecordReader<TRecord> (streams records, throws RecordPipelineRefusalException for a structural refusal), a processor factory (IRecordProcessor<TRecord, Study>, given the diagnostics builder), an IAugmentedOutputWriter<TRecord> (synchronous writes, see below), and optionally a pre-check (the spreadsheet header).

Format	Parse implementation	Reader	Processor	Augmented output
CSV/TSV	SpreadsheetRecordProcessorParseImplementation	NewSpreadsheetReader.ReadDataRowsAsync	StudySpreadsheetRecordProcessor	leading Study Id column
EndNote XML	EndnoteRecordProcessorParseImplementation	NewEndnoteReader.ReadRecordsAsync	StudyEndnoteRecordProcessor	one SyRF-id per record
RIS	RisRecordProcessorParseImplementation	RisRecordReader (RisLineSource decodes)	StudyRisRecordProcessor	U1 - SyRF-ID: per record
PubMed NBIB	NbibRecordProcessorParseImplementation	NbibRecordReader.ReadRecordsAsync	StudyNbibRecordProcessor	GN - SyRF-ID: per record
Zotero RDF	ZoteroRdfRecordProcessorParseImplementation	ZoteroRdfReader (pass 0 builds ZoteroRdfIndex)	StudyZoteroRdfRecordProcessor	dc:description SyRF-ID: per item, every other node kept

PubmedXmlParseImplementation (legacy PubMed XML) and LivingSearchJson are out of scope and unchanged.

Passes

1. Pre-check (CSV/TSV only). The header row and the screening column map are checked; any error refuses the import.
2. Pass 1, inspect and validate. The source is streamed once and nothing is kept: the reader's refusals, the record count, and every record mapped by the processor and thrown away (its warnings go to a discarded builder). Record errors go to one ImportErrorCollector. Any error returns a FileParseResult with the errors and a null URL: no output is opened and no study exists. Progress reports the records read every 1,000 (at most 100 reports at the record limit) under ParsePhase.Validating, with total 0 meaning "not yet known"; the save closure's handler records them with SearchImportJob.UpdateValidationProgress, which counts them as parsed without fixing the total (an empty file's total really is 0). The upload dialog and the Processing page show it as "N records checked" (a count, no percentage) while the job is Parsing with no total; the dialog then shows "Saving studies... (saved/total)" with a bar for pass 2. When pass 1 succeeds it reports every record with the total, (count, count), flagged FinalUpdate so the 500 ms throttle never drops it: the job's parse is complete and it moves to Saving before the first batch, whatever the last checkpoint was (a 2,500-record file's last checkpoint is 2,000). A reader can ask for pass 1 to start again once (RecordPipelineRestartException): RIS does when a file without a byte-order mark is not valid UTF-8, and reads it as Windows-1252.
3. Pass 2, map, write and save. The source is reopened; each record is mapped, written to the augmented file synchronously and added to a batch; every SaveBatchSize (200) studies the batch goes to the study processor as a StudyBatch. Pass 2's progress is the saving progress the save closure reports per batch; the pipeline itself reports only the final update, with the output URL, when the output is complete. As a backstop, the save closure counts every record as parsed in its saving updates (they all were, in pass 1), so the first saving update the throttle lets through also completes the parse if no parse report reached the job. A cancellation that lands between the passes opens no output. A cancellation seen inside a mapping call, or while the upload stream opens, still ends as CancelledEarly, never as a failure or a record error.

Memory is one batch plus one record (plus the diagnostics aggregate's bit sets), whatever the file size. RecordPipelineBoundedMemoryTests imports generated files through a source generated as it is read: RIS at 256 MiB / 100,000 records grew the live heap by 2.6 MiB, EndNote XML at 100,000 records (59 MiB) by 1.0 MiB, CSV at 50,000 rows by 1.0 MiB; no read exceeds 64 KiB.

Study processor contract

Documented on IStudyParseImplementation.ParseStudiesAsync: the study processor is called zero or more times, each with one batch, in file order, never concurrently, only after the whole file passed pass 1. ProjectManagementService.ParseReferenceFile's save closure keeps a running saved count, takes the total from the StudyBatch (a legacy single call with every study still uses its own count), marks the final saving update only when saved equals the total, and keeps the S3c settled-status check before every batch of 200 (the closure still chunks defensively). The statistics import fence is admitted before the first batch, as before (.claude/rules/materialized-stats.md).

Error model

• Refusals (structural: not this format, encoding, malformed, a limit) are one error and replace any record errors. RIS keeps its whole-file precedence while streaming: the size limit, then an encoding error, then null characters, then the first structural error in file order (the rest of the file is drained so the earlier kinds are still found), then a missing final ER or an empty file, then the generated file's size.
• Record errors are collected for the whole file: at most 100 errors, then one "{k} more records have errors." error counting the failed records not kept. Record numbers are 1-based (spreadsheets count data rows).
• Mapping exceptions in pass 1 (a cleaner's regex timeout, an unexpected value) become that record's MalformedRecord error, "Record n could not be read."; only the exception type is logged.
• Failures in pass 2 (storage, the study processor, the S3c stop, a record that maps differently than in pass 1, a file with a different record count) are logged with the record number, the partial output is deleted, the exception propagates and ParseReferenceFile's failure path deletes the saved studies. A failed upload or deletion during that clean-up is logged and never replaces the root cause.
• Cancellation at any point returns CancelledEarly; in pass 2 the partial output is deleted (a failed deletion is logged) and the caller deletes the saved studies.
• Messages never echo bibliographic values (the one pre-existing exception is noted under follow-ups).

Limits (refused in pass 1)

Format	Bytes	Records	Other
RIS	256 MiB, input and generated (D9)	100,000 (D9)	every RisReaderLimits bound above
EndNote XML	256 MiB	100,000	a record nesting more than 64 elements deep
CSV/TSV	none	none	header and screening map pre-check
PubMed NBIB	64 MiB, input and generated	50,000	every NbibReaderLimits bound below
Zotero RDF	256 MiB, input and generated	100,000 items	index: 500,000 nodes / 64 Mi characters (below)

The browser refuses RIS files over 256 MiB (MAX_RIS_FILE_BYTES). The search-upload signing endpoint has no size condition, so the parser's pass-1 limits are the server-side bound.

Synchronous output

BlockingStream does not override WriteAsync, so an async write queues on the thread pool while the concurrent upload blocks a worker on the same stream; on a small pool that deadlocks. Every IAugmentedOutputWriter writes synchronously; Stream-based writers get the upload stream from RecordPipelineFormat.OutputStream, and RIS uses the stream's cancellable write so a stalled upload unblocks on cancellation.

Adding a format

Write a reader that streams records and throws RecordPipelineRefusalException (with a sanitised ParseError) for refusals, a processor that returns record errors rather than throwing and reports warnings to the builder it is given, and an output writer; wire them in a RecordPipelineFormat from the format's IStudyParseImplementation. Pass 1 and pass 2 must see the same records: a reader that needs state (an encoding, an index of shared nodes for Zotero RDF) keeps it on the reader instance, which serves one import.

Parity and rollback

The CSV and EndNote parity suites ran unchanged against the pipeline, and the PubMed NBIB suite (NbibRecordProcessorParseImplementationTests) runs green against it with three by-design changes: a record error opens no output (it used to be opened and deleted), and cancellation is shown between batches rather than at a per-record progress report. The RIS suite (RisParserParityTestBase) and a 73-case corpus (RisParityCorpus: encodings, line endings, every refusal and competing refusals, record errors, re-import, warnings) ran green against both the legacy whole-file parser and the pipeline before RisParseImplementation/RisDocumentReader were removed; the corpus is pinned to the legacy parser's renderings in Fixtures/Ris/parity.golden.txt. Changes by design: a record error now refuses the import before any output is opened (CSV and EndNote opened and then dropped it), record errors are capped at 100 for CSV too, the study processor receives batches, a pass-2 failure deletes the partial output, and a clean-up failure after cancellation is logged rather than thrown. No feature flag (D2): parity-tested, same studies, output and warnings; rollback is reverting the PR, and no stored data changes shape.

Delivered MVP Boundary

The first independently shippable slice is manual RIS import only.

Included

• a new append-only LibraryFileType.Ris value;
• an explicit RIS option in the existing upload dialog;
• complete web, API, object-storage metadata, notifier, parser-registry, and current Study persistence wiring;
• structural RIS validation based on file content, independent of extension and MIME claims;
• bounded file, record, line, field, diagnostic, and in-memory behaviour;
• documented support for UTF-8, UTF-8 BOM, UTF-16 BOM, and Windows-1252 input;
• conservative mapping into fields already available on Study;
• bounded, sanitised diagnostics without raw reference content or personal data;
• cancellation and all-or-nothing failure behaviour;
• retention of the original uploaded file so a future Citation migration can reprocess it;
• safe CSV generation for any imported values later emitted to spreadsheet software;
• automated fixtures and regression coverage for existing EndNote XML and CSV import.

The current default ceilings are 256 MiB for input and generated output and 100,000 records (raised from the delivered 10 MiB and 50,000 by owner decision D9 once RIS streamed, S10), 32 Ki characters per physical line, 128 Ki characters per field, 512 fields and 4,096 physical lines per record, 128 continuation lines per field, 100 stored diagnostics, and 8 MiB of mapped UTF-8 text per Study. Generated output uses an approximately 64 KiB bounded producer/consumer queue.

Not Delivered by This Slice

• NBIB, BibTeX, or CSL-JSON import;
• automatic format detection or a format-choice wizard;
• external Web of Science, Scopus, EBSCO, or other CSV profiles;
• PMID, MeSH, language, volume, issue, page, or other schema expansion;
• pmPublication/Citation migration or global identifier reconciliation;
• retirement or renumbering of historical file-type values;
• staged or partially visible imports for files above the bounded MVP limits.

Architecture and Data Rules

Compatibility

Existing persisted file-type values remain readable. RIS is appended after all historical values; existing numeric assignments must never be reordered or reused.

The MVP may adapt parsed RIS records into the current Study sink. It must not introduce the obsolete pmReference/ImportRecord proposal or pretend the approved pmPublication/Citation direction is already implemented.

PRISMA Semantics

RIS publication types such as TY describe the cited work. They must not populate PRISMA acquisition-source provenance.

PRISMA sourceType and sourceName describe where a systematic search was conducted and remain properties of the SystematicSearch/search-import context. Generic RIS content cannot safely infer them.

Retention Gate

The gate was verified on 2026-08-30: current object-storage lifecycle rules do not expire original objects under the project-scoped Imported Search Libraries prefix, and the completed SearchImportJob retains its OriginalFileUrl. The implementation must preserve both behaviours. Any future lifecycle change that could expire originals before Citation backfill requires the minimal pmPublication/Citation persistence kernel first.

Validation and Resource Safety

The parser must reject malformed or mismatched content even when the extension, MIME type, or manual selection says RIS. Production uses fixed ceilings for:

• uploaded bytes;
• records per file;
• characters per physical line and per field;
• fields and continuation lines per record;
• stored diagnostics;
• studies held in memory (one batch of 200 since S10; previously every parsed study).

Unknown tags produce one warning per distinct tag per record, naming the tag (for example LA), and are omitted from the current Study; the original upload keeps them. Invalid optional PY/Y1, DO and UR values are omitted with a warning that names the tag that held them. Structural errors are fatal. Fatal errors and cancellation leave no partially imported studies. Logs and API diagnostics identify the record and a sanitised reason category; malformed physical-line errors may also identify a line number. They do not echo titles, abstracts, notes, author contact details, or arbitrary raw values.

Delivered parser limits (RisReaderLimits)

Relocated from the root CLAUDE.md; these are the concrete ceilings behind the list above.

RIS acceptance is structural, not extension/MIME based: records require non-empty TY, ER, and a title (TI or T1). Supported encodings are strict UTF-8 (with or without BOM), BOM-marked UTF-16 LE/BE, and defined Windows-1252 bytes. Both input and generated UTF-8 output are bounded to 256 MiB (D9), so re-import remains valid; legacy encodings that would expand beyond that output cap fail before study processing. The parser is also bounded to 100,000 records, 32 Ki characters per physical line, 128 Ki characters per field, 512 fields and 4,096 physical lines per record (with one field/line reserved for generated U1 unless a replaceable U1 already exists), 128 continuation lines per field, 100 diagnostics, and 8 MiB of mapped UTF-8 text per Study. Generated augmented RIS is encoded through a fixed 16 KiB buffer into a four-entry backpressure queue (about 64 KiB of queued output), retains the original upload for future backfill, and adds/replaces only full U1 - SyRF-ID:<guid> markers. The producer queue, stream reader, and multipart requests share the import cancellation token so stalled output can abort and roll back.

Validation (every bound above, and every record's title and mapped size) completes in the pipeline's first pass, before anything is written or saved; the second pass maps, writes and saves in batches. Failures and cancellation are all-or-nothing: save pipelines are drained, the partial generated output is deleted and the saved studies are deleted by the import's failure path. Persisted/UI warnings are bounded and sanitised, including an explicit truncation sentinel; raw stream lines are never logged. Alongside that capped sample, the job stores an uncapped ImportDiagnostics aggregate (SearchImportJob.ImportDiagnostics, built by ImportDiagnosticsBuilder): the total warning count, the number of records with warnings, and one group per (code, field, cause) with its occurrence count, affected-record count (each record counted once) and up to five example record numbers, labelled with the tag's name, such as "Language (LA)". The aggregate is capped at 50 groups (GroupsTruncated), which keeps it under 20 KB of BSON inside the Project document. Jobs imported before the aggregate existed, and parsers that do not build one, have no aggregate, and the web falls back to the warning sample. The searches page and the upload dialog show the aggregate as "import notes", one per group, never as a warning count. Not feature-flagged: the change is additive (new optional fields; legacy documents and DTOs keep working) and only rewords how already-shown warnings are summarised.

Each warning's Field and Cause roll out in two steps, because the previous build reads stored ParseWarning documents with a strict class map and would throw on unknown elements while old and new pods run side by side. Step one (this release): Field/Cause are [BsonIgnore], so they exist in memory and on the API response straight after a parse but are not stored (they are null after a reload); the durable field and cause live in ImportDiagnostics, which old builds tolerate because SearchImportJob captures unknown elements. ParseWarning also gains [BsonIgnoreExtraElements], so this release can read warnings that a later release stores with more elements. Step two (a later release, once no pod older than step one can run): remove the [BsonIgnore] to persist Field/Cause on each warning.

Generated CSV formula prefixes use a doubled apostrophe/U+2063 protocol marker on both headers and values, escaping literal markers before browser/server restoration. BibTeX, CSL-JSON, import auto-detection, extended CSV, and bibliographic schema expansion remain separate follow-ups.

Pipeline placement

The existing search-import pipeline supports EndNote XML, CSV/TSV, and a manually selected RIS format (LibraryFileType.Ris = 5; persisted enum values are append-only). RIS follows the normal browser upload -> API signing -> S3 notifier -> project-management parser registry -> pmStudy persistence path. It does not introduce pmReference or ImportRecord: TY maps only to Study.ReferenceType, while acquisition provenance remains the systematic search/reference file identity so citation types never become PRISMA source provenance.

Conservative Field Mapping

The MVP maps only fields supported by the current Study model:

RIS data	Current SyRF field	Rule
TI/T1	Title	Prefer TI, then T1; do not concatenate conflicting titles
AU/A1	Authors	Preserve author order; join using the existing Study representation
AB/N2	Abstract	Prefer AB, then N2
PY/Y1	Year	Accept a valid four-digit year only
JO/JF/T2	Journal	Use a deterministic documented precedence
DO	DOI	Strip a recognised DOI URL/prefix and validate conservatively
UR	URL	Accept an absolute HTTP(S) URL only
KW	Keywords	Preserve order and repeated tags within current field limits
TY	Reference type	Keep publication type separate from search-source provenance
L1	PDF relative path	Every L1 value of a record, in order, is classified by the shared AttachmentReferenceClassifier; the first relative PDF becomes PdfRelativePath, stored exactly as written (never URL-decoded). internal-pdf://<n>/<name>.pdf (EndNote) and files/<n>/<name>.pdf (Zotero) are both accepted

L1 attachment handling

Mapping lives in RisAttachmentMapper, which StudyRisRecordProcessor (the streaming RIS mapper) calls for each record. Every other L1 reference produces a warning with Field = L1 and a cause; messages name the kind of reference, never the path.

Reference	Cause	Warning code
Further PDF after the first (duplicates by normalised path are ignored)	AttachmentAdditionalNotAttached	UnsupportedFieldOmitted
Non-PDF (for example an .html snapshot)	AttachmentNotPdf	UnsupportedFieldOmitted
file://, drive-letter or leading / or ~/ path	AttachmentLocalPathRefused	InvalidOptionalFieldOmitted
http, https, ftp, ftps address	AttachmentRemoteRefused	InvalidOptionalFieldOmitted
Path containing .. (or a dots-and-spaces segment)	AttachmentTraversalRefused	InvalidOptionalFieldOmitted
Other schemes, control characters	AttachmentUnsupported	InvalidOptionalFieldOmitted

L2 (full-text link) and L4 (figure) are not attachments: they remain "not imported" and are never fetched. UR stays the URL source. Record numbers in messages are 1-based. There is no feature flag: the change only populates a field that was always empty for RIS and cannot affect existing studies.

Unsupported values remain available only in the retained original file until the approved Citation model can represent them.

Delivered Acceptance Criteria

1. A user can select a valid RIS file and complete the existing import workflow into current Study records.
2. Content that is not structurally valid RIS is rejected even if its filename or MIME type claims RIS.
3. Valid fixtures from at least three independent exporters pass, covering LF/CRLF, supported encodings, repeated tags, continuation lines, and unknown tags.
4. Empty, truncated, malformed, oversized, overlong-field, over-record-limit, and excessive-diagnostic fixtures fail safely and predictably.
5. Cancellation or fatal parsing produces no partial imported studies.
6. Historic enum values and EndNote XML/CSV import behaviour remain covered by regression tests.
7. No raw bibliographic or personal values appear in logs or user-facing diagnostics.
8. Study CSV/TSV exports neutralise spreadsheet formula prefixes for imported user-controlled values.
9. The original RIS object remains project-authorised and available for future backfill for the agreed retention period.
10. Focused unit, integration, UI, API-contract, notifier, resource-bound, and exact-head CI checks pass before merge; staging evidence is recorded separately in the delivery pull request.

Delivered Critical Path

1. Confirm original-object retention and authorisation behaviour.
2. Append the shared enum and wire the explicit format choice through every service boundary.
3. Implement bounded decoding, structural validation, parsing, and conservative mapping.
4. Add fixtures and focused integration/regression tests.
5. Verify the complete upload-to-persistence path and staging behaviour.

EndNote XML Import

EndNote XML runs on the record pipeline (EndnoteRecordProcessorParseImplementation in ParserImplementations/NewParser/): NewEndnoteReader streams the records and StudyEndnoteRecordProcessor is the single mapper. The legacy EndnoteXmlParseImplementation (whole-document load, EndNote-only) is deleted. No feature flag: parser changes ship unflagged (search-import plan, D2); rollback is reverting the PR, and no stored data changes shape.

Any producer, refused structures

There is no source-app check: EndNote, Zotero's EndNote XML export, any other producer, or none, import when the structure is EndNote XML. A record is any element named record (no namespace) at any depth. The reader ignores DTDs, comments and processing instructions, never resolves anything external, and reads every record whether or not whitespace separates them. The pipeline's first pass checks, counts and maps every record before any output is written; these refuse the whole import (nothing saved, no output opened, null URL):

Input	Error code	Message
First non-whitespace character (after a BOM) is not <: tagged text (%0 …), .enl/.enlx, RIS, empty	UnsupportedFileType	"This file isn't EndNote XML. If it came from EndNote, export it again as XML …"
Root rdf:RDF	UnsupportedFileType	"This is an RDF file, not EndNote XML."
Root PubmedArticleSet	UnsupportedFileType	"This is PubMed XML; export from PubMed in PubMed format (.nbib)."
Malformed XML	MalformedRecord	"The XML is malformed at line L, position P[, in record n]."
No record elements	MissingXmlElements	"No EndNote records were found."
More than 256 MiB, or more than 100,000 records	ResourceLimitExceeded	"The EndNote XML file exceeds the 256 MiB upload limit." / "… exceeds the 100000 record limit."
A record nesting elements more than 64 levels deep (the record element is level 1)	ResourceLimitExceeded	"Record n nests elements more than 64 levels deep."
A record with no title after the fallbacks	MissingRequiredField	"Record n has no title." (up to 100, then "k more records have errors.")

Only an XmlException and the limits above are refusals. Any other exception while reading, such as a storage client's mid-stream (IOException, AmazonS3Exception, HttpRequestException), fails the import through the pipeline's normal failure path so it can be retried; it is never reported as a malformed file.

A structural refusal replaces any record errors found before it. Cancellation during the second pass deletes the partial augmented file and the saved studies. The augmented file is <xml><records> with each record as read plus exactly one SyRF-id (an id from an earlier SyRF import is replaced), written synchronously to the upload stream.

Nesting cap

NewEndnoteReader builds each record with DepthLimitedXElementReader (shared by the XML reference readers; Zotero RDF adopts it in #4614), not XNode.ReadFromAsync. ReadFrom costs time quadratic in nesting depth: on main before this cap, a 350 KB synthetic file whose second record nested 50,000 elements inside <notes> imported after 1 m 15 s (5,000 levels: 384 ms), so a small crafted upload could tie up a worker. The capped reader builds the same element in one step per node and stops at the first element past NewEndnoteReader.MaxRecordDepth (64): that file is now refused in about 10 ms, before anything deeper is built. EndNote writes at most five levels (record, urls, pdf-urls, url, style); 64 leaves a wide margin for other producers. A record at the cap imports as before.

Mapping and cleaning

Text fields (never paths, URLs or DOIs) join EndNote <style> runs, then remove inline markup exporters leave as literal text (i, b, u, em, strong, sup, sub, sc, span, a, br, p, div, font, jats:*, mml:*, keeping inner text; <br>, </p>, </div> and <jats:p> become a paragraph break in the abstract and a space elsewhere), then decode character references exactly once. Single-line fields collapse all whitespace; the abstract keeps paragraph breaks (LF). A value changed by markup removal or decoding adds a ValueNormalised note.

EndNote data	SyRF field	Rule
titles/title → alt-title → short-title → translated-title	Title	First non-empty; none refuses the record
contributors/authors/author → secondary-authors/author	Authors	Order kept, empty skipped, trailing corporate comma trimmed; editors only when there are no authors, with a note
periodical/full-title → alt-periodical/full-title → titles/secondary-title → periodical/abbr-1	Publication name	Every reference type (D5): book sections and proceedings get the book/proceedings title; alternate is abbr-1, else the name
abstract	Abstract	Paragraphs kept
dates/year → dates/pub-dates/date	Year	First four-digit token 1000-2999; otherwise omitted with a note
electronic-resource-num → a doi.org related/web URL	DOI	RIS DOI normalisation; several lines (merged records): the first valid DOI is used; a non-DOI value is omitted with a note
urls/related-urls → web-urls → text-urls	URL	First absolute HTTP(S) URL; never pdf-urls or image-urls
urls/pdf-urls/url	PDF path	Shared attachment rules (below)
auth-address	Author address	Distinct values joined with ;
keywords/keyword	Keywords	Split on line breaks, de-duplicated case-insensitively
ref-type/@name	Reference type	Else the element text when not purely numeric

Every other non-empty child of record (and of titles, contributors, dates, urls, periodical, alt-periodical) is reported as FieldNotImported with the element name, labelled for display ("Volume (volume)"); database, source-app, rec-number, foreign-keys, ref-type and SyRF-id are structural. Diagnostics use the shared aggregate: 100 sampled warnings, one group per element and cause with affected-record counts.

Attachments

Each pdf-urls/url value is classified with AttachmentReferenceClassifier (paths never URL-decoded). The first relative PDF becomes PdfRelativePath, stored without the internal-pdf:// prefix and without a leading slash (1234567890/Example.pdf; the legacy parser stored /1234567890/Example.pdf; link building, PdfPathNormalizer matching and risk-of-bias selection treat both the same). Further PDFs are reported (AttachmentAdditionalNotAttached), non-PDF files reported (AttachmentNotPdf), and absolute local paths, web links, .. paths and other schemes refused with a counted note; notes never contain the path.

PubMed format (NBIB) import

Delivered in the search import programme (plan §4.4, slice S5). The backend accepts PubMed's own text export ("PubMed" format, usually saved as .nbib or .txt) as LibraryFileType.PubmedNbib = 6 (append-only). It is inert until offered: the current upload dialog does not list it; the search import wizard will offer it behind searchImportWizard. The parser itself is not feature-flagged (programme decision D2): it is additive and only runs for that file type.

It follows the same upload -> API signing (.nbib, application/nbib) -> S3 notifier -> parser registry -> pmStudy path as RIS, and shares RIS's accepted encodings, DOI normalisation (ReferenceValueParsing) and bounded UTF-8 output writer.

Streaming record pipeline

PubMed import runs on RecordPipelineImport (S10; see Streaming record pipeline): NbibRecordReader streams one record at a time (IPipelineRecordReader), StudyNbibRecordProcessor maps one record and NbibAugmentedOutputWriter writes the returned file through the shared BoundedUtf8Writer. Nothing holds the whole file or every study: the reader keeps a 16 KiB read buffer, the current line and the current record, and studies are saved in batches of 200 (a 26 MiB, 50,000-record file grew the live heap by 0.9 MiB in RecordPipelineBoundedMemoryTests).

Each read opens a fresh stream from storage:

1. Encoding (pass 1). A byte-order mark is sniffed from the first bytes (UTF-8, UTF-16 LE/BE; UTF-32 is refused). A file without one is read once to check it is strict UTF-8 throughout, including sequences split across reads; otherwise it is read as Windows-1252. The reader keeps that decision and asks the pipeline to read the file again (RecordPipelineRestartException).
2. Inspection (pass 1). The whole file is checked for structure and every bound, records are counted, the returned file's size is totalled, and every record is mapped (title, mapped size) and thrown away. Any refusal or record error refuses the whole import here: no output is opened and no study exists. Up to 100 record errors are reported, then "{k} more records have errors.".
3. Parse (pass 2). Each record is mapped, written to the returned file with synchronous writes and saved in batches. Cancellation, or a failure, deletes the partial returned file and the saved studies.

Lines end at CR, LF or CRLF, including a CRLF split across reads; a line over the length bound is refused before it is held.

Format

A tag line is a 2-4 character tag left-justified in four characters, a hyphen at index 4, then a space and the value (PMID- 12345, TI - Title, LID - 10.1000/x [doi]); a value-less tag (GR -) is allowed. A line starting with whitespace continues the previous field and is joined with one space (trailing spaces trimmed). Blank lines separate records. A record normally starts with PMID-; a record whose first tag is not PMID is still accepted, and a second PMID with no blank line before it starts a new record. Official tags are those in PubMed's PubMed Format tags table.

Mapping

PubMed tag	Study field	Rule
TI, then BTI, then TT	Title	Required. Using TT adds a "values tidied" note. Missing: the import is refused with the record number and PMID
AB	Abstract	Structured labels kept as written; OAB is not added
FAU, else AU; CN	Authors	Document order. FAU is "Surname, Given"; AU is "Surname Initials". CN is a corporate author. Investigators (IR, FIR) and editors are not authors
AD	Author address	Distinct values joined with "; "
DP, else DEP	Year	First four characters as digits, 1000-2999; otherwise omitted with a note
JT; TA	Publication name; alternate name	Each falls back to the other
LID or AID ending [doi]	DOI	[doi] removed, then RIS's DOI normalisation; invalid values omitted with a note. No [doi] identifier: no DOI, no note
PMID	URL	https://pubmed.ncbi.nlm.nih.gov/<PMID>/. Missing or non-numeric: no URL, with a note
PT	Reference type	First value
OT, then MH	Keywords	Each in order; MeSH * major-topic markers removed, subheadings kept; case-insensitive duplicates dropped
Other official tags	none	Kept in the original and returned files; one diagnostic group per tag ("Volume (VI)", "Language (LA)")
Tags not in PubMed's table	none	One diagnostic group per tag: "Tag ZZ is not a PubMed tag and was not imported."

Notes go into the same ImportDiagnostics aggregate as RIS (100-warning sample, one group per tag with affected-record counts, labelled with PubMed's official tag name). Values are never echoed; a PMID appears in a refusal only when it is a plain number.

No attachments. PubMed format has no attachment tags: PdfRelativePath is always empty, and nothing in the import matches PDFs from URLs or identifiers.

Returned file

Each record is written back (as UTF-8, preceded by a blank line, as PubMed does) with GN - SyRF-ID:<study id> after its last line. GN is PubMed's General Note tag, so reference managers keep it. On re-import a GN - SyRF-ID:<guid> line without a continuation is replaced, never duplicated; any other GN value is the user's and is kept. The generated size is checked against the file bound before any study is processed.

Limits (NbibReaderLimits)

Bound	Value	Compared with RIS
File (input and generated output)	64 MiB	RIS 256 MiB since D9 (it was 10 MiB, which a full 10,000-citation PubMed export does not fit); PubMed itself exports at most 10,000 per file
Records	50,000	RIS 100,000 since D9
Characters per line / per field	32 Ki / 128 Ki	same
Fields / lines per record	32,768 / 65,536	RIS 512 / 4,096; consortium records carry hundreds of FAU/AU/AD and IR/FIR lines (a real trial record has 474 tags on 916 lines)
Continuation lines per field	2,048	RIS 128 (about 10,000 characters at PubMed's 80-column wrap); raised so the field-character bound limits long abstracts
Characters held per record	8 Mi	RIS bounds a record by its line and field limits; bounds the one record the reader holds
Mapped UTF-8 text per Study	8 MiB	same; exceeding it is a record error
Diagnostics sample	100	same

One field and line per record stay reserved for the generated GN line unless a replaceable one exists. Any limit breach refuses the whole import.

Refusals

The whole import is refused, and no study is created, for the cases below, all found in pass 1 before any output is opened. Record errors are collected (up to 100); a refusal is the only error.

Case	Error code
No records (empty or blank file)	UnsupportedFileType
RIS content (TY - first), CSV, EndNote tagged text or any first line that is not a tag line	UnsupportedFileType
PubMed XML (message: export with the PubMed format instead)	UnsupportedFileType
XML whose dialect cannot be seen: an over-long first content line starting with <, or an XML prolog followed by an over-long line (minified XML; blank lines before it do not matter). The message is hedged: "The file is XML, not PubMed format. If it is PubMed XML, export the citations from PubMed with the PubMed format …"	UnsupportedFileType
A line that is neither a tag line, an indented continuation nor blank (message gives line and record numbers)	MalformedRecord
A continuation with no field before it; null characters	MalformedRecord
A record without TI, BTI or TT (record error; message gives the record number and PMID)	MissingRequiredField
UTF-32, malformed UTF-8/UTF-16, bytes undefined in Windows-1252	UnsupportedEncoding
Any limit above	ResourceLimitExceeded

Accepted encodings are RIS's: strict UTF-8 with or without BOM, UTF-16 LE/BE with BOM, Windows-1252.

Zotero RDF import

Slice S6 of the search import programme (plan). The backend accepts Zotero's own RDF export (the Zotero RDF translator, usually saved as .rdf) as LibraryFileType.ZoteroRdf = 7 (append-only; signed as .rdf, application/rdf+xml). It is offered only through the search import wizard (behind searchImportWizard); the parser itself is not flagged (D2).

Passes and the index

Zotero writes an item's journal (or book, series, section) and attachments either nested inside the item or as top-level nodes that items refer to with rdf:resource (urn:issn:…, #item_N), before or after the items, and nests a shared node the first time with an rdf:about and refers to it afterwards. So ZoteroRdfReader reads the file three times, never loading the whole document:

1. Pass 0, index (the first read of pass 1). Every top-level node is streamed once; the container and z:Attachment nodes with an rdf:about, wherever they sit, go into ZoteroRdfIndex with only what mapping needs (a container's title, alternative title, DOI, the names of its other fields and its own isPartOf; an attachment's path, z:linkMode and whether link:type is PDF), and every link:link / dcterms:isPartOf reference is noted. When the file has been read, nodes nothing references are dropped, so the index grows with the referenced containers and attachments, not with the rest of the file. While reading it holds at most 500,000 nodes and 64 Mi characters (unreferenced nodes count until they are dropped); past either the import is refused (ResourceLimitExceeded). Each top-level node is built from the reader iteratively and refused as soon as it nests more than 64 elements deep (XNode.ReadFrom is quadratic in depth), and its isPartOf chain is checked against 8 levels before any container is read, so nothing recurses further and each nested container is read once. The reader then asks the pipeline to read the file again (RecordPipelineRestartException).
2. Pass 1 streams the items (top-level nodes with a z:itemType other than attachment or note), maps each through the index and applies every refusal and record error.
3. Pass 2 maps, writes and saves in batches; every other top-level node (attachments, journals, notes, collections) is written to the returned file in document order.

At the item limit with a journal and three attachments per item (400,000 referenced nodes, 22 Mi characters) the index costs about 98 MiB of live heap (RecordPipelineBoundedMemoryTests, held to that figure plus 25%); 50,000 items without attachments import in +0.7 MiB.

Refusals

Case	Error code	Message
Not XML, a root other than rdf:RDF, or rdf:RDF without the Zotero export namespace (arbitrary RDF, EndNote XML, RIS)	UnsupportedFileType	"This isn't a Zotero RDF export."
Malformed XML (bytes that do not decode, an unsupported encoding, bad names, entities or namespace declarations), or a DTD (prohibited; nothing external is resolved)	MalformedRecord	"The XML is malformed at line L, position P."
No items	MissingXmlElements	"No Zotero items were found."
Over 256 MiB, over 100,000 items, an index over its bound, a returned file over 256 MiB	ResourceLimitExceeded	names the limit
Elements nested more than 64 deep in one node, or dcterms:isPartOf chains more than 8 deep (Zotero writes at most about 12 and 3)	ResourceLimitExceeded	"The Zotero RDF file nests elements more than 64 levels deep." / "… nests journals, books or series more than 8 levels deep."
An item without a title	MissingRequiredField	"Item n has no title." (a record error: up to 100, then "k more records have errors.")

Only an XmlException (every malformed input above, decoding included, is one) becomes a refusal. Any other exception while reading, a storage client's (an IOException, AmazonS3Exception or HttpRequestException mid-stream) or a SyRF bug, fails the import through the pipeline's normal failure path, so a transient storage error is never reported as a malformed file.

Mapping

Text is cleaned as EndNote XML's is (Zotero's rich-text tags such as <i>, <b>, <sup>, <span class="nocase"> removed, entities decoded once, whitespace collapsed; the abstract keeps paragraphs), with a ValueNormalised note when markup was removed.

Zotero RDF	Study field	Rule
dc:title	Title	Required
bib:authors/rdf:Seq/rdf:li	Authors	foaf:Person (foaf:surname, foaf:givenName) or foaf:Organization/foaf:name, in order; bib:editors only when there are no authors, with a note
dcterms:isPartOf (by reference or nested)	Publication name	The first journal, book, periodical or series title (a section is followed to its book); dcterms:alternative is the alternate, each falling back to the other (D5)
dc:identifier "DOI 10…" on the item or its journal; else a doi.org dcterms:URI	DOI	RIS normalisation; Zotero puts the article DOI on the journal node
dc:identifier/dcterms:URI/rdf:value	URL	First absolute HTTP(S) URL that is not doi.org; else https://pubmed.ncbi.nlm.nih.gov/<z:PMID>/ for a numeric PMID
dc:date	Year	First four-digit token 1000-2999
dcterms:abstract	Abstract	Paragraphs kept
dc:subject text and z:AutomaticTag/rdf:value	Keywords	Trimmed, de-duplicated ignoring case
z:itemType	Reference type	journalArticle "Journal Article", bookSection "Book Section", others title-cased

Every other item child with content (bib:pages, z:language, dc:publisher, dc:rights, z:PMCID, an "Extra" dc:description, an ISSN or ISBN identifier, …) and the journal's own fields (prism:volume, prism:number, …) are reported as FieldNotImported per field, labelled ("Pages (bib:pages)"); notes attached to an item (dcterms:isReferencedBy a bib:Memo) are one "Notes (notes)" note. Diagnostics use the shared aggregate (100 sampled warnings, "Additional Zotero RDF warning diagnostics were omitted after the configured limit was reached.").

Attachments

Each link:link resolves to a z:Attachment. The path is z:path (its rdf:resource or text), or, when there is none, the rdf:resource child of older exports; it is stored exactly as Zotero wrote it (never URL-decoded; spaces, Unicode and % kept), the same rule as the browser extractor. It is a PDF when link:type is application/pdf or the path ends .pdf. z:linkMode 3 (linked URL) has no file and is refused (AttachmentRemoteRefused); every other path goes through AttachmentReferenceClassifier, so absolute linked files are refused and attachments: base-directory paths are unsupported. The first PDF is attached; further PDFs, non-PDF files and refused or missing attachments are reported with Field = link:link, never with the path.

Returned file

The whole document is written back: the original root with its namespace declarations (a default xmlns included), every top-level node in order, and each item with exactly one <dc:description>SyRF-ID:<study id></dc:description>. A dc:description from an earlier SyRF import is replaced; the user's own (Zotero's "Extra") is kept. The returned file's size (the input plus one marker per item) is checked against 256 MiB in pass 1.

Sequenced Follow-ups

1. Audit new-upload handling for historical LivingSearchJson and add direct PubMed XML fixtures.
2. Define PMID identity and Citation persistence (PubMed format import stores the PMID only as the study URL until then).
3. Add confidence-based format detection with a manual override.
4. Add known external CSV profiles after formula-safety coverage.
5. Add CSL-JSON.
6. Reassess BibTeX libraries and its macro/nesting threat model before implementation.
7. Implement the approved pmPublication/Citation migration, backfill, provenance, privacy, and cross-project authorisation rules.
8. Add a byte and record limit for EndNote XML: done in S10 (256 MiB / 100,000 records).
9. Spreadsheet record errors still quote the cell for an unparseable year ("'x' can not be parsed…"); the pipeline's other messages never echo values. Reword it in a follow-up (it changes a user-visible message).

Resolved Delivery Gate

• Original project-scoped reference uploads are retained without a configured expiry, and completed import jobs keep the original object URL for future Citation backfill.