I have built a system with Hermes to solve a concrete problem posed to me by a University department: moving from model-assisted exploration to a PRISMA review that is methodologically defensible, technically auditable, and deliverable as a real editorial package.
First, my plea in defense. It is not an agent that writes papers (well, yes, but I don’t want to pick more fights with academics). It is an infrastructure with rules, files, commands, and material controls. In this part I am not delivering the packaged agent ready to use —that will come in another article—, but the discussion on how to improve it. I will share it, but I need some room so that whoever uses it can replicate my exact results.
I am not going to enter the university debate about whether or not to use this kind of system. The only thing I humbly intend here is to show how it is built inside so you can replicate it, improve it, or contribute whatever you want to the model.
General architecture of Hermes for PRISMA reviews.
The real problem I wanted to solve with HERMES
The first thing I wanted to solve by building this agent system was the fragility of reviews made with a single model (whether from OpenAI, Anthropic, Qwen…). A model can summarize, compare, and propose argument lines very well, but that does not amount to a serious systematic review. If the process leaves no search trail, inclusion criteria, exclusion reasons, full-text confirmation, and verifiable editorial closure, the result may sound good and be methodologically weak. That is what I wanted to attack with this system.
PRISMA 2020 insists on documenting how studies are identified, selected, and included. PRISMA-S tightens that requirement at the search layer. That is the foundation on which everything else stands.
The second problem has been operational. A serious review is not solved in a single model call. It requires a protocol, bibliographic acquisition, DOI auditing, screening, PDF retrieval, full-text reading, deep extraction, figure generation, manuscript writing, cross-review, packaging, and synchronization.
If all of that lives only in the contextual memory of a conversation, the system is fragile by definition. That is why Hermes separates state, process, and persistent artifacts. What matters is not that it «remembers», but that it re-reads (the Obsidian Vault) what it wrote down in order to generate sound decisions.
Sources: Page et al. 2021 (BMJ n71) · Rethlefsen et al. 2021 (PRISMA-S)
From copilot to replicable system.
The methodological principles it is built on
Hermes is not built around the question «what can an agent do in this research», but around the question «what minimum conditions must a review meet for me to trust it».
I have worked on six rules:
- PRISMA 2020 as the reporting framework.
- PRISMA-S as the search transparency framework.
- DOI-first whenever the identifier exists.
- Mandatory full text for the final corpus.
- Recomputation of counts from real artifacts.
- Editorial closure only with a PASS gate, not with the subjective impression that «it looks pretty good already».
These rules materialize in the base skill prisma-systematic-review/SKILL.md, which works as an operational and methodological contract. I insist on this point because in many agent systems the interesting part lives in invisible prompts. Not here: here the methodology is a versionable, auditable piece.
The token consumption decision is to require legible full text for final inclusion, as well as a model that supports this kind of paper and extracts all the text well.
An abstract helps to discard quickly, but it is not enough to declare that a study belongs to the serious corpus. It is a methodological frontier, not a style preference.
Sources: PRISMA Statement · Cochrane Handbook ch. 4
Non-negotiable guardrails.
The architecture used by Hermes
The architecture rests on four layers.
The first is a minimal surface through Telegram, because I don’t want operations to depend on an interface.
System in operation from Telegram.
The second is the Docker runtime with two services: hermes-agent runs the gateway and the main work; hermes-prisma-watchdog monitors live reviews and resumes phases when it detects stagnation.
The third layer is the model policy. Primary model: Qwen 3.6 served by NaN Builders. Fallback chain on Ollama Cloud: DeepSeek v4-pro, Kimi K2.6, Qwen3-coder, and Gemma4. This chain lives in a config.yaml with all its ingredients (temperature, top-p, max tokens, frequency penalty, response format…).
The fourth layer is orchestration, with complete_review.py, publication_autopilot.py, and publication_gate.py.
There is another layer that usually stays hidden and must be mentioned: the skills. My installation has ten local research skills (I will share them with the complete package).
On the road to a «publishable PRISMA» I heavily use five: systematic review, state, academic review, integrity, and change mapping. The architecture is not a super-agent that does everything, but specialized layers with small contracts. That makes it maintainable and scalable for my use case.
Key artifacts: config.yaml, docker-compose.yml, start-gateway.sh, prisma-watchdog.py, 10 skills in hermes-home/skills/research
High-level architecture.
How a review starts: intake, question, and criteria
A good review gets biased earlier at intake than later in the discussion. That is why the system forces freezing as early as possible the question, the time window, the admissible study type, the desired final N, the inclusion and exclusion criteria, and the expected editorial framework.
It is not a formality. If the topic is born as «let’s see what comes out», the rest of the system only automates ambiguity.
What adds value is that the phase leaves a visible trail from the very first minute. protocol/intake.md, protocol/eligibility-criteria.md, and protocol/search-strategy.md appear. What matters is not only that they exist, but that another person can open them and check whether the bias originates in the question, in the search, or in the interpretation.
There is a decision I like to mention: if I don’t have a specific journal to publish in (templates), Hermes falls back to a generic-common-core mode (a template that would pass a first filter). That decision later affects the LaTeX template, the type of editable package, and the tone of the editorial closure. It is not cosmetic. It is a way of not faking alignment with a journal I have not yet chosen as a publication venue.
Key artifacts: protocol/intake.md, protocol/eligibility-criteria.md, protocol/search-strategy.md
Sources: Cochrane Handbook ch. 4 · Kitchenham et al. 2009
Reproducible commands.
The questions and commands that freeze the methodological frontier
This phase is where the review stops being an intention and becomes a methodological machine. The research question, the unit of analysis, the time window, the inclusion and exclusion criteria, and the quality threshold stop being tacit decisions. They become fixed in artifacts that later condition the entire journey of the system. That matters because, in a scientific review, much of the bias does not appear at the end, when the text is being written, but much earlier, in the way what counts as an eligible study and what stays out is defined.
The first key component is bootstrap_topic_review.py. Its value is not only in creating folders, but in materializing the starting point of the review. From it come protocol/intake.md, the search strategy, and the initial project structure. In other words, this script does not prepare a directory: it fixes the methodological contract of the review. If later the corpus turns out biased, I can go back to that origin and check whether the problem was born in the formulation of the question, in the breadth of the search, or in the harshness of the criteria.
The second component is doi_audit.py. Here the goal is not «cleaning CSV», but stabilizing the bibliographic identity of the corpus. When sources arrive from OpenAlex, Crossref, Semantic Scholar, or arXiv, the normal thing is to find duplicates, missing DOIs, minimal title variants, and inconsistent metadata. This script turns that mess into three critical views: which records have an identifiable DOI, which are duplicated, and which remain ambiguous. In practical terms, this reduces one of the most frequent errors in semi-automatic reviews: believing that more studies have been found than actually exist or, even worse, losing traceability of why a paper was merged, discarded, or marked as uncertain.
The third element is complete_review.py. This file matters because it does not solve a specific task, but governs the entire sequence: acquisition, screening, full-text retrieval, extraction, and the jump to the editorial layer. Its real utility is that it turns a collection of loose operations into a flow with explicit dependencies. Extraction does not start if the corpus is not sufficiently fixed, a study is not accepted without full-text evidence, and publication is not reached if intermediate artifacts are missing. That makes automation not simply fast, but disciplined.
The fourth component is review_runtime_state.py or, failing that, the material state that Hermes serializes as runtime-state. Here the value is not in «seeing which phase I am in», but in preserving operational context without depending on conversational memory. When the review is interrupted, the system can re-read what is already done, what is missing, what blockages exist, and which artifacts support each transition. That greatly changes the nature of the work: continuity no longer depends on verbally remembering where I had left off, but on re-reading a persistent state that the system can interpret and reuse.
What matters, therefore, is not that Hermes runs four scripts, but that each one closes a different methodological uncertainty. bootstrap_topic_review.py fixes the scope. doi_audit.py stabilizes the identity of the corpus. complete_review.py imposes the order of the process. runtime-state preserves the operational memory. Together they make the system not improvise what should have been defined at the beginning. From that moment on, Hermes no longer tries to guess which review I want to build: it executes a review whose rules, limits, and transitions have already been written.
The methodological frontier is fixed before importing the first record and is left in writing. From there the system does not guess what I want to do: it executes what I specified from the very first moment.
Key artifacts: bootstrap_topic_review.py · doi_audit.py · complete_review.py · review_runtime_state.py
Maieutics first.
From heterogeneous sources to a «DOI-first» universe
Hermes connects to several sources because none is sufficient on its own.
At startup, bootstrap_topic_review.py queries OpenAlex, Crossref, Semantic Scholar, and arXiv. It does so with direct HTTP calls and parsing of JSON or Atom responses. The result is not «a search in the abstract», but a set that can be exported with a technical trail.
The first artifact of this phase is searches/search-log.csv: it stores query, source, date, and retrieved volume. Then records/master-records.csv, records/doi-index.csv, records/duplicates.csv, and records/missing-doi.csv appear.
The first is the normalized raw universe. The second consolidates the canonical key by DOI. The third preserves detected collisions. The fourth points out what cannot be resolved by DOI. When there is no DOI, Hermes uses a stable hash identifier and generates a RID.
Here the review stops being a collection of searches and starts to resemble a documentary base to start working with.
Sources: Rethlefsen et al. 2021 (PRISMA-S)
Universe normalized by DOI.
How Hermes decides: OK, KO, or if more is needed
The inclusion and exclusion logic is not a magic algorithm. It is a mixed sequence of four layers: minimum rules, model-assisted decision, decision canonization, and audit with recomputation. The system does not just ask a model whether something «seems relevant».
The most visible rule is in complete_review.py, where it sets a minimum threshold of useful text. If the full text does not reach it, it is not a sufficient basis for final inclusion. In the same script lives canonicalize_screening_decision, which normalizes inclusion and exclusion variants so that include, include_ft, exclude, excluido, excluir, or maybe are reduced to a consistent set. That normalization matters because PRISMA counts are not written by hand afterwards: they are recalculated from canonized decisions.
A study becomes OK when it passes minimum rules, confirms the focus in full text, and leaves a traceable justification in CSV. It becomes KO when it fails those criteria or the exclusion reason is clear, and it stays at «more is needed» when quick screening is not enough. That logic is closer to ROBIS or AMSTAR 2 than to a simple relevance ranking.
Sources: Whiting et al. 2016 (ROBIS) · Shea et al. 2017 (AMSTAR 2)
Four layers to decide, not a single question to the model.
Deep extraction: from PDF to a record with real traction
Deep extraction takes the review to another level. It is not enough for me to know that an article is «experimental» or that it analyzes. I want to know what it names (within the research question posed), which benchmark it uses, which sample it describes, which variables it employs, which method it reports, and in which theory it positions itself. All of that goes into extraction/extraction-table.csv, with fields such as work_type, empirical_type, design_detail, models_or_systems_studied, benchmark_dataset_or_corpus, dependent_variables, independent_variables, and theoretical_framework.
When an initial extraction falls short, Hermes does not resign itself to a record without data. It resorts to refresh_extraction_depth.py, which re-reads the text extracted from the PDF and forces a deeper layer. This solves a problem I have lived through several times: articles that on a first pass appeared as an «experimental study», when the complete document did offer names about the research question, scales, and more useful metrics to add. I would mark this as very important if you are going to build your own.
I organize each study into four layers: common core (work type, year, DOI, authors, publication, keywords, abstract), empirical block (design, country, sample, method, variables, instruments), technical block (models, datasets, benchmarks, corpus, tasks), and theory and synthesis block (theoretical framework, key findings, extraction confidence). Comparability is born from the extraction schema, not from a pretty table at the end.
Key artifacts: extraction/extraction-table.csv, refresh_extraction_depth.py
Empirical, technical, theoretical, and core zones.
Figures, tables, and visual evidence
I did not want Hermes to treat article figures as lateral noise. In many topics, a figure concentrates architecture, experimental design, or conceptual relationships that the abstract does not clearly capture. That is why I added an explicit visual evidence layer with models that support viewing and processing them.
The key script is prepare_paper_figures.py. It does not only export images: it separates scientific figures extracted from the PDFs, diagnostic page renders, and visual assets generated for the manuscript. That separation was important because, without it, a complete rendered page can be confused with a reusable scientific figure (divide well and you will conquer).
The result lives in several paths: figures/extracted/ for detected scientific figures, figures/page-renders/ for supporting renders, and figures/manifest.csv, figure-catalog.md, and figure-ranking.csv for traceability. The manuscript’s own figures carry the 686f6c61 signature (in my case) to distinguish them from those of the corpus. Mixing reuse of others’ evidence and own visual synthesis is bad methodological practice and bad editorial practice.
Sources: Wilkinson et al. 2016 (FAIR) · ACM Artifact Review
Visual plans: manifest and catalog.
From corpus to publishable manuscript
When the corpus is consolidated, the challenge stops being documentary and becomes editorial. The central piece is publication_audit.py. That script recomposes sections, cleans inconsistencies, normalizes citations and references, integrates tables and figures, detects obvious gaps, and generates the manuscript. In practice it is the APA and assembly agent of the system, even if it does not carry that name in the tree.
The three artifacts it leaves are paper/manuscript/publication-ready.md, paper/manuscript/publication-ready.tex, and paper/manuscript/publication-ready.pdf. The first is the editable version. The second, technical output for academic continuity in LaTeX. The third, compiled version ready to circulate. Added to that is paper/references/references.generated.bib with the BibTeX bibliography, and main-common-core.tex when there is no target journal.
Here I impose my obsessions (which are not few). Every relevant study must have something more than a table row. That is why the final manuscript includes per-study records with one or two paragraphs before the compact table, and expanded blocks of discussion, theoretical implications, original contribution, conclusions, and future lines. The result is not an automatic output: it is a piece that an academic person recognizes as a manuscript that is «more deterministic» and provides greater validity.
Key artifacts: paper/manuscript/publication-ready.{md,tex,pdf}, paper/references/references.generated.bib, main-common-core.tex
From corpus to manuscript in PDF, MD, and .TEX.
Cross-review, integrity, and editorial gate
This layer is what most differentiates Hermes from a conventional flow with several LLMs. I do not accept a manuscript because a model has written a solid narrative. I accept it when it passes a cumulative circuit of review, integrity, and artifact validation.
publication_peer_review.py launches the cross-review and writes paper/review/review-manifest.csv, paper/review/peer-review-overview.md, and each reviewer’s material.
The policy is in reviewer-models.csv: Reviewer A works with Qwen 3.6 on NaN, Reviewer B with DeepSeek v4-pro, and Kimi K2.6 remains as the first editorial fallback. check_manuscript_integrity.py looks for weak claims, inconsistencies, and support gaps. build_revision_roadmap.py turns that review into a matrix of actionable changes. publication_gate.py checks whether the required artifacts exist and whether the closure is legitimate. publication_autopilot.py is the loop that iterates over those layers until closing or documenting a real blockage.
This system is closer to the structured evaluation logic of ROBIS and AMSTAR 2, and to the artifact review spirit of ACM, than to the idea of «taking one last look». And it avoids a mistake: delivering something that looks finished when it has not passed the controls that justify calling it publishable.
Sources: Whiting et al. 2016 (ROBIS) · Shea et al. 2017 (AMSTAR 2) · ACM Artifact Review
Review, integrity, and validation.
Final packages, operational memory, and limits found
When the gate returns PASS, Hermes does not end in a loose folder. It generates two packages with different functions: paper/package/publication-package.zip as the general editorial package and paper/package/publication-latex-editable.zip as editable academic continuity.
The distinction matters because not every reader needs the same thing. At the same time, sync_review_to_obsidian.py synchronizes the review to the Obsidian vault and keeps a navigable version for later research: records, overview, visual library, notes, and PRISMA trail.
Example vault in Obsidian.
Operational continuity does not depend on remembering a conversation. It depends on state files like notes/runtime-state.md and notes/runtime-state.json, on manifests, on ZIPs, and on a watchdog that knows how to re-read and resume. The real memory of the system is outside the chat. From the FAIR logic, this brings the review closer to a findable, accessible, and interoperable asset.
Material memory.
I do not want to close the article pretending that Hermes is perfect for this goal. It still depends on the quality of the sources, on the recoverability of the full text, on the availability of cloud models, and on the possibility of extracting complex figures or tables without loss.
Even with the hardened gate, I find improvable areas: a formal risk of bias layer, better per-journal templates, more control over meta-analysis when the corpus allows it, and even finer automation of the step between publication-ready and submission-ready. For that evolution I keep working to deliver and publish something you can install, use, and improve.
Where I want to keep improving.
Sources: Wilkinson et al. 2016 (FAIR) · Wohlin 2014
(If you made it this far, you deserve a round of applause. Thank you for your time)
References
- Page, M. J., McKenzie, J. E., Bossuyt, P. M., et al. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ, 372, n71.
- Page, M. J., Moher, D., Bossuyt, P. M., et al. (2021). PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ, 372, n160.
- Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., et al. (2021). PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Systematic Reviews, 10, 39.
- Cochrane Handbook for Systematic Reviews of Interventions, Chapter 4: Searching for and selecting studies.
- Kitchenham, B., Brereton, O. P., Budgen, D., Turner, M., Bailey, J., and Linkman, S. (2009). Systematic Literature Reviews in Software Engineering: A Systematic Literature Review. Information and Software Technology, 51(1), 7-15.
- Whiting, P., Savovic, J., Higgins, J. P. T., et al. (2016). ROBIS: A new tool to assess risk of bias in systematic reviews.
- Shea, B. J., Reeves, B. C., Wells, G., et al. (2017). AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ, 358, j4008.
- Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., et al. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data, 3, 160018.
- Wohlin, C. (2014). Guidelines for Snowballing in Systematic Literature Studies and a Replication in Software Engineering.
- ACM. Artifact Review and Badging, current policy.
Originally published on X on May 10, 2026.