# RAGFlow nightly: knowledge and native Memory test

Date: 2026-10-20. Status: completed. Scope: a small retained-case test and a separate native Agent integration probe.

RAGFlow nightly can write conversation memory and use it in a new Agent session. It preserves the tested original records and exposes links from extracted memories to raw messages. The fixed comparison remains **11 complete answers out of 13**, the same result as RAGFlow 1.0.0-rc1. The two failures still omit a current delivery decision that exists in both raw and extracted memory. This is a retrieval gap.

The native Agent probe also exposes two limits. One historical answer includes later policy without a clear date boundary. All three native final responses have `reference: null`. Memory IDs in answer text are not verified original-source citations. These findings support an internal pilot with explicit source checks. They do not establish customer acceptance or justify a complete ELF migration.

## Version and conditions

- Official `nightly-arm64`, published on 2026-10-19, pinned to digest `sha256:002f50e053e9c5081610c9b0bb9f07a190cb240e1c983cadc9f0e4364dec9153`.
- Runtime version: `v1.0.0-rc2-48-g7eb673bfc`. Source: [7eb673bfc3ab1cc22532fe23a1e9bb85c34e48bd](https://github.com/infiniflow/ragflow/commit/7eb673bfc3ab1cc22532fe23a1e9bb85c34e48bd).
- This was the latest published official ARM64 image at selection. The AMD64 nightly advanced during setup. This report does not claim to test that later image or current main.
- Docker ran ARM64 without emulation. The RAGFlow image and source were not patched.
- Extraction and answers used `openai/gpt-6-luna`, OpenAI routing, medium reasoning, and the full 128,000 output-token allowance. Hy-Memory was not tested.
- Embeddings used `qwen/qwen3-embedding-8b`, Nebius routing, and the full **4096 dimensions**. Both knowledge and Memory indexes declare indexed 4096-dimensional cosine vectors. All 27 fresh provider vectors have that width. Exact text matches reused 53 previously paid vectors from the same model and provider.
- The original six documents, eight conversation records, two projects, and thirteen answer attempts were reused. The inputs and oracle are byte-identical to the retained comparison. A separate calibration preceded formal calls.
- Formal retrieval used at most six knowledge hits and six Memory hits, a 32 KiB context allowance, and the same shared answer reader as the previous run. Native `source_id` links supplied raw parents. No context entry was dropped; the largest context was 4,906 bytes.
- No scored answer was repeated to improve its result. No returned text was silently cut. All formal responses ended with `finish_reason: stop`.

See [protocol](protocol.json), [source pin](source-pin.json), [runner provenance](runner-provenance.json), and [vector mappings](embedding-index-mapping.json).

## Fixed comparison

| Check | RAGFlow 1.0.0-rc1, retained result | Tested nightly |
| --- | ---: | ---: |
| Complete answers after semantic review | 11/13 | 11/13 |
| Complete answers with bound quotes | 11/13 | 11/13 |
| Individual quotes match supplied context | 26/26 | 28/28 |
| Uploaded knowledge text reads back exactly | 6/6 | 6/6 |
| Original conversation envelopes read back exactly | 8/8 | 8/8 |
| Extracted-to-raw parent links resolve | 21/21 | 18/18 |

Quote counts and extracted-memory counts depend on generation. A change in those counts alone is not a quality improvement or regression. A matching quote proves binding to the supplied context; it does not prove that every claim is entailed or that every requested fact was answered. Exact readback proves preservation for this sequence, not enforced write protection, retention, tenant isolation, or arbitrary recovery.

The retained baseline is [2026-10-15-native-kb-memory.md](../../docs/evidence/benchmarking/2026-10-15-native-kb-memory.md). It is historical evidence and was not changed or rerun for other products.

The two incomplete answers are `post/memory-0-0` and `post/combined-001`. Both require Larch's current delivery mode, `STREAM-0-286`.

- Raw message 11 states that the August 15 decision replaces `BATCH-0-741` with `STREAM-0-286`.
- Extracted episodic message 13 retains the decision, its date, subject, and `source_id: 11`. It remains active.
- Both failed contexts omit that decision. They contain later records and records from the other project.
- The reader returns the supported retry policy and reports the delivery mode as unknown. The decisive fact was lost before answer generation.

Descending `valid_at` ordering, together with mixed event and import dates, is a plausible cause. This run did not change the sort policy, so it does not prove a specific fix. Changing only the writer model would not directly address the observed missing retrieval of an already correct extracted fact.

Formal ingestion, updates, restart, and thirteen answers took **175.981 seconds**. Mean answer time was **4.74 seconds**; median was **4.32 seconds**. Image downloads and initial setup are excluded. See [formal summary](formal-summary.json), [semantic review](review-ragflow.json), [source audit](audit-formal-ragflow.json), and [failure diagnosis](failure-diagnosis.json).

## Separate native Agent probe

The official Message component wrote two ORBIT-NIGHTLY conversation records. The first set `POLL-71` for staging with Mira as owner. The second conditionally approved `STREAM-93` only after `AUDIT-62` passes, retained `POLL-71` until then, and prohibited production rollout. A native Agent then answered three frozen questions in three distinct new sessions. It used the native Memory Retrieval tool on every question. The probe did not inject the answers into Agent context.

| Check | Result |
| --- | --- |
| Message component creates raw memory | 2/2 |
| Extraction tasks complete | 2/2 |
| Original records read back exactly | 2/2 |
| Extracted parent links resolve | 5/5 |
| Fresh reader sessions with actual Memory tool use | 3/3 |
| Before audit: mode, owner, and production boundary | Correct |
| After audit: conditional mode, staging scope, owner, and production boundary | Correct |
| Historical October 10 question | Correct main mode and owner, but includes later policy without a clear date boundary |
| Native structured references in final response | 0/3; all are null |

All three answers contain the main requested facts. The historical caveat prevents a clean temporal acceptance claim. The plain-text memory IDs do not satisfy ELF's final source-verification contract. The shared-reader quote results in the fixed comparison must not be presented as native Agent citation results. These three questions are not added to the historical thirteen-question score.

See [frozen native protocol](native-agent-protocol.json), [native Agent summary](native-agent-summary.json), and [actual tool evidence](native-agent-tool-evidence.json).

## Runtime and setup

The tested stack used seven services: RAGFlow, Elasticsearch, MySQL, Silo object storage, Kvrocks, NATS, and ClickHouse. Only the RAGFlow API was published, on `127.0.0.1:19480`. The small fixture used about 2.4 GiB at idle. This is an observed test footprint, not a production capacity or maintenance-cost estimate.

The retained rc1 client needed one API compatibility change: nightly dataset creation requires `parse_type: 1` together with `parser_id: general`. The temporary client was updated before paid model calls. A separate initial launch issue came from a downloaded entrypoint losing its executable bit; restoring its mode fixed that setup issue without changing its contents. Neither correction changed source fixtures, retrieval settings, or scored answers. See [setup observations](setup-observations.json).

## Cost and cleanup

| Measure | Value |
| --- | ---: |
| New paid requests | 45; all completed |
| Chat requests | 31 |
| Fresh embedding requests | 14 |
| New prompt tokens | 34,190 |
| New completion tokens | 9,212 |
| Exact cached embedding vectors reused | 53 |
| New settled spend | $0.008500545 |
| Cumulative settled spend | $2.024643551 |
| Cumulative conservative exposure | $2.096824251 |
| Authorized cumulative ceiling | $10.00 |

The conservative total retains earlier unresolved reservations. This run added no uncertain requests. The cumulative budget gateway exited successfully before cleanup; its endpoint file and ledger lock are absent. See [budget summary](budget-summary.json) and [new request ledger](request-ledger.json).

Cleanup removed this task's seven containers, six volumes, one network, and seven downloaded images. No preexisting container ID is missing. Generated credentials, private Compose configuration, and private application logs were removed. Docker logical image use fell from 7.32 GB to 4.109 GB; logical volume use fell from 28.82 GB to 27.52 GB. These figures do not measure physical host filesystem reclamation. See [cleanup receipt](cleanup-receipt.json).

No tracked ELF source, model choice, or persistent deployment was changed. This report and its evidence remain local under ignored `tmp/`. The next useful work is a bounded check of current-decision retrieval and a source-verification layer around native Agent answers. A larger benchmark or custom memory engine expansion is not justified by this run.