refactor(memory): refactor TestMemoryStore to use the unified storage interface

[opieter-aws]

Description

TestMemoryStore hand-rolled its own filesystem persistence in both SDKs — path resolution, atomic temp-file writes, plus a records cache with load memoization to paper over cold-load races. The unified Storage interface exists precisely to delete this per-subsystem re-invention, and the storage design doc names the memory store as a consumer to migrate. This moves persistence onto a Storage backend internally.

The public config is unchanged: persist/path and the ~/.strands/memory/<name>.json default behave exactly as before. Internally the store now resolves a backend from that config — persist: false → InMemoryStorage, otherwise a LocalFileStorage rooted so an explicit path still writes to that exact file and the default still lands at the home-dir location. Dropping the bespoke cache also let the read-modify-write in add read fresh under a lock, closing the cold-load race by construction rather than guarding against it.

Related Issues

#3099

#3253

Documentation PR

N/A

Type of Change

Other (please describe): internal refactor — no public API or behavior change

Testing

Ran the TestMemoryStore unit suites in both SDKs (Python pytest, TS Node), plus type-check, lint, and format for each; all pass. Exercised each store end to end against both backends, confirming the default home-dir path, an explicit path, persist: false ephemerality, dedup, and cross-loop reuse from a synchronous agent all behave as before.

• I ran hatch run prepare

Checklist

• I have read the CONTRIBUTING document
• I have reviewed and understand every line of code in this PR, including any generated by AI tools, and I can explain why it works
• My change is focused and reasonably small; I have split unrelated work into separate PRs
• I have added any necessary tests that prove my fix is effective or my feature works
• I have updated the documentation accordingly
• I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
• My changes generate no new warnings
• Any dependent changes have been merged and published

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[github-actions]

Assessment: Comment

Clean, well-motivated refactor — moving TestMemoryStore onto the unified Storage interface deletes a good amount of bespoke node:fs code and eliminates the cold-load race by construction rather than papering over it. The PR description and tests are strong. My feedback is about process/consistency around the public-API change rather than the implementation itself.

Review themes

• API review process: This changes a public constructor contract and flips the default persistence behavior, yet it's labeled chore ("no user-facing impact") and carries no api/needs-review label. Per team/API_BAR_RAISING.md this is at least a moderate change to a customer-facing contract and should go through API review. (The description already contains the before/after material a reviewer needs — good.)
• Behavior change safety: Ephemeral-by-default silently drops persistence for existing new TestMemoryStore({ name }) callers on upgrade (inline comment). Needs loud migration/changelog notes.
• Composability: The memory/ prefix is hardcoded into the key rather than using the NAMESPACED/namespace() pattern the session subsystem uses (inline comment).
• Docs: Public API changed (path/persist removed) but docs/examples and a changelog entry aren't updated in this PR.

Nice work dropping the load-memoization complexity and improving browser-bundle safety with the static InMemoryStorage import.

[github-actions]

Re-review (latest push, merge commit 61e0ba8)

Updated assessment: Approve (non-blocking follow-ups below).

Both of my earlier inline concerns are addressed in the latest push:

• ✅ Namespacing — the constructor now uses this._storage = NAMESPACED in backend ? backend : namespace(backend, 'memory') and drops the memory/ prefix from _key, exactly matching the SessionManager convention. A caller-supplied pre-namespaced view is no longer double-prefixed.
• ✅ Ephemeral default — the class docstring now states plainly that the store "defaults to an ephemeral InMemoryStorage: entries are lost when the process exits", and the example shows the opt-in LocalFileStorage path.

Remaining follow-ups (both minor, non-blocking)

1. Test the new NAMESPACED in backend branch. The double-prefix guard added in response to my feedback is currently untested — no test passes a pre-namespaced view. Suggest adding a case asserting that new TestMemoryStore({ name: 'notes', storage: someStorage.namespace('memory') }) writes to memory/notes.json (not memory/memory/notes.json), i.e. it behaves identically to passing the raw backend. This locks in the exact edge case the branch exists for. The suite is otherwise strong (33 tests: ephemeral default, persistence/restart, concurrency, corruption paths, cross-SDK interop).
2. Release-note the breaking change. There's no TS CHANGELOG in the repo, so please make sure the "data no longer persists by default — pass a Storage backend" note lands in the release notes for upgraders. The ! in the title covers semver; this is about a human-readable heads-up.
3. API-review label. This changes a public constructor contract and default behavior; per team/API_BAR_RAISING.md it warrants the api/needs-review label rather than the chore classification.

⚠️ Housekeeping — duplicate inline threads

Apologies: a retry issue in my tooling caused the two inline threads to be posted three times each (lines 149 and 150, at 20:05 / 20:24 / 20:25). They're identical in content, and I don't have permission to delete or edit them. Please disregard the extra copies — each pair represents a single comment, both of which are now resolved by the changes above.

[opieter-aws]

Docs haven't yet been released, the TestMemoryStore was released last week. So breaking change seems acceptable to me.

[github-actions]

Re-review (updated after force-push)

Assessment: Comment — the two substantive concerns from my first pass are addressed in code. One follow-up (test coverage) and one process item remain, neither blocking.

Status of prior feedback

• ✅ Composability / namespacing — resolved. Constructor now uses NAMESPACED in backend ? backend : namespace(backend, 'memory') and drops the hardcoded prefix from _key, matching the SessionManager convention. A pre-namespaced view is no longer double-prefixed.
• ✅ Ephemeral-default clarity — the class docstring now states entries are ephemeral and lost on process exit, with the opt-in persistence path shown in the example. Remaining nit: make sure the "no longer persists by default" note lands in release notes (no TS CHANGELOG in-repo).
• 🆕 Test gap — the new "already-namespaced" branch of the guard is untested; all 33 tests pass a bare backend. Added an inline suggestion with a concrete test.
• ⏳ API review process — still labeled chore; this changes a public constructor contract and default behavior, so per team/API_BAR_RAISING.md it should carry api/needs-review. Please confirm with a designated API reviewer.

Housekeeping: my earlier tooling retried and created duplicate inline threads on lines 149/150. I lack permission to delete/edit them, so I've replied marking the extra copies as duplicates — please disregard those.

Nice, focused iteration on the namespacing fix.

[github-actions]

@mkmeral 👋 sorry for the wait — here's the consolidated public API change breakdown for bar raising. This PR now lands both SDKs (Python + TypeScript) and the surfaces are symmetric. Note: my earlier _read metadata-validation finding is now resolved (Python validates metadata shape), and the namespace() threads are resolved (TS uses the public helper).

TestMemoryStore — Public API Changes

1. Constructor config (BREAKING — removed + added params)

	Before	After
persist?: boolean / persist: bool	✅ (default true)	❌ removed
path?: string / path: str	✅	❌ removed
storage?: Storage	❌	✅ new (optional; defaults to InMemoryStorage)

persist=false → pass storage: new InMemoryStorage(); path=x → pass storage: new LocalFileStorage(x).

2. Behavior change: default persistence flipped (⚠️ silent data-loss on upgrade)

• Before: persisted to ~/.strands/memory/<name>.json by default — durable across restarts.
• After: ephemeral by default (InMemoryStorage) — entries lost on process exit, no error/warning.
• The most common call site TestMemoryStore({ name }) keeps compiling but silently stops persisting. This was flagged earlier and is now documented in the docstring; still worth a loud release-note callout.

3. Second, subtler location change (worth explicit bar-raiser attention)

Even after migrating to storage: new LocalFileStorage(), the default directory moves ~/.strands/ (home) → ./.strands/ (cwd) per the new docstrings. So a user doing the "obvious" 1:1 migration to keep persistence will not find their old data (different absolute path + memory/ namespace subdir). This is a second migration gotcha layered on top of #2 — recommend calling it out explicitly in migration notes.

4. Failure-mode / exception-contract changes

• Python add/search: previously raised OSError (with target path) on disk read/write failure → now raises StorageError from the backend. ValueError still covers malformed blobs, and now additionally validates metadata shape.
• TypeScript search/add: contract now documents @throws StorageError for backend read/write failures — a new exception type callers must account for.
• Constructor: the old path must not be empty validation is gone (param removed).

5. New composition behavior — auto-namespacing

• Backend is auto-scoped under a memory/ prefix unless the caller already passed a namespaced view (NAMESPACED in backend), so a pre-namespaced backend won't get double-prefixed (memory/memory/...). Consistent with SessionManager/offloader convention.
• Record blob key: memory/<sanitized-name>.json. On-disk record format unchanged (camelCase id/content/metadata/createdAt, Z-suffixed ms timestamps), so cross-SDK read compat holds.

6. Concurrency model change (Python, behavioral)

• Switched from threading.Lock → per-event-loop asyncio.Lock (rebinds per running loop to support a store reused across the fresh loops a sync Agent spins up).
• Dropped the in-memory cache + load memoization: every search/add reads fresh from the backend. Trade-off: eliminates the cold-load race by construction, at the cost of a backend read per operation. Prior cross-OS-thread guarantees are replaced by per-loop async serialization.

---

Open questions for bar raising

1. Backwards-compat shim vs. clean break. @opieter-aws proposed keeping persist/path as @deprecated aliases. Given (a) docs not yet released and (b) zero GitHub/internal usage, a clean break is defensible — but the deprecation shim is cheap insurance against the two silent behavior changes above (ci: update mypy requirement from <1.0.0,>=0.981 to >=0.981,<2.0.0 #2 + ci: update sphinx-rtd-theme requirement from <2.0.0,>=1.0.0 to >=1.0.0,<4.0.0 #3). Worth a deliberate decision rather than defaulting to either.
2. Is ephemeral-by-default the right "happy path"? The tenets caution against the obvious path silently doing the surprising thing. For a store literally named to persist "memory," a new user's mental model may expect durability. Confirm ephemeral-default was weighed against a persistent default (or a required explicit storage arg to force the choice).
3. Should storage be required rather than optional-with-ephemeral-default? Making it required would force callers to consciously choose durability vs. ephemerality and eliminate both silent-data-loss vectors — at the cost of ergonomics for the throwaway/prototype case.

Everything is internally consistent and well-tested across both SDKs; these are design-intent confirmations for the bar-raiser, not correctness gaps.

[pgrayy]

What if TestMemoryStore kept the same constructor arguments but initialized LocalFileStorage for you under the hood? Does TestMemoryStore need anything other than LocalFileStorage?

[github-actions]

@pgrayy that's the crux of the bar-raising decision, so let me lay out the tradeoff both ways rather than pick a side.

Your framing (keep path/persist, wrap LocalFileStorage internally):

• Zero breaking change — kills both silent-data-loss vectors (ci: update mypy requirement from <1.0.0,>=0.981 to >=0.981,<2.0.0 #2 and ci: update sphinx-rtd-theme requirement from <2.0.0,>=1.0.0 to >=1.0.0,<4.0.0 #3 above) outright.
• Still banks the core feat: add unified storage interface #3099 win: the hand-rolled node:fs/pathlib code is deleted and replaced by delegating to LocalFileStorage. You get the "stop re-inventing persistence per subsystem" benefit without exposing the abstraction in the constructor.
• Simpler happy path for the prototyping/testing audience this store is named for.

What that gives up — i.e., "does it need anything other than LocalFileStorage?": I think the honest answer is yes, in two concrete cases, not just speculative generality:

1. Browser. The TS store went out of its way to stay browser-bundle-safe (the old code used dynamic node:os/node:path imports precisely so it wouldn't break the bundle). LocalFileStorage doesn't exist in the browser. Accepting a Storage lets a browser user pass an in-memory or future IndexedDB backend cleanly; hardcoding LocalFileStorage under the hood reintroduces the exact node-only coupling this design was trying to shed.
2. Tests / ephemeral. The in-memory backend is a first-class need (it's the new default). Under a path/persist API you'd re-grow a persist=false branch, which is the bespoke conditional the unified interface is meant to collapse.

There's also a consistency dimension: SessionManager and the context offloader both accept a Storage. If TestMemoryStore instead hides it behind path/persist, it becomes the odd consumer out, and the "compose any backend" story gets an asterisk.

Possible middle ground worth weighing: keep an ergonomic persistence default (so the common call site doesn't silently lose data) and accept an optional storage for the browser/S3/in-memory cases — i.e., don't force everyone through new LocalFileStorage(), but don't hide the abstraction either. That costs a slightly wider config surface.

Net: if the team's read is that this store will only ever realistically be local-file or in-memory, your under-the-hood approach is the lower-risk call. If browser/community backends are in scope (the TS browser-safety work suggests they are), exposing Storage is the consistent choice — and then the open question collapses to just "persistent-or-ephemeral default?" (open Q2). This is squarely a bar-raiser call; flagging the inputs rather than deciding it.

[github-actions]

@pgrayy good question to pressure-test the whole approach — I think it splits into two parts:

"Does TestMemoryStore need anything other than LocalFileStorage?" — Yes, and the strongest driver is the browser bundle. LocalFileStorage pulls in node:fs/node:os/node:path; the old store went out of its way to await import('node:os') dynamically precisely so the module stayed browser-bundle-safe. A browser consumer can't use LocalFileStorage at all — they need InMemoryStorage today or an IndexedDB backend later. On top of that: in-memory for throwaway/test stores, and S3 for a store shared across instances. So the Storage seam earns its keep beyond just local files — this is the #3099 motivation.

"Keep the same constructor args but build LocalFileStorage under the hood?" — Viable and it does fix the two silent behavior changes I flagged (ephemeral-by-default data loss + the ~/.strands → ./.strands move), because persist=true default would keep persisting exactly where it did. The cost is that it re-couples the store to node:fs and gives up the browser/S3/in-memory flexibility above — unless you keep both.

The hybrid worth considering (close to @opieter-aws's earlier sketch, minus the deprecation): accept storage?: Storage for advanced backends and keep path/persist as node conveniences that construct a LocalFileStorage internally, with storage taking precedence.

new TestMemoryStore({ name })                          // persists to ~/.strands/memory (unchanged — no data loss)
new TestMemoryStore({ name, persist: false })          // ephemeral
new TestMemoryStore({ name, path: '/data/notes.json' })// custom file
new TestMemoryStore({ name, storage: new S3Storage() })// any backend

That gets backwards-compat + zero silent migration surprises + full extensibility. The price is a wider config surface and a precedence rule to document (what happens if both path and storage are passed — presumably storage wins, or throw).

Net: the pure "constructor-args-only, LocalFileStorage under the hood" option is the simplest but forfeits the browser story that partly motivated the migration; the hybrid keeps that story while erasing the breaking-change concerns. Given docs aren't released and usage is ~zero, the clean storage-only break is also still defensible — this is really a "how much do we value backwards-compat vs. a minimal surface" call for you all.

[opieter-aws]

@strandly-the-agent review latest implementation

[github-actions]

Re-review of the latest implementation

This pivot is a clean resolution — nicely done. The store now runs on the unified Storage interface internally while keeping the persist/path config and the exact prior on-disk locations, so it banks the #3099 win (hand-rolled node:fs/pathlib deleted) without the breaking change or the two silent-data-loss vectors I'd flagged. Verified across both SDKs:

• Backwards compatible ✅ — persist=True default → LocalFileStorage(~/.strands/memory) + key <name>.json (same path as before); persist=False → InMemoryStorage; explicit path → LocalFileStorage(dirname) + key basename. Public config surface is unchanged.
• Atomicity preserved ✅ — I checked LocalFileStorage on both SDKs: writes go to a temp sibling then rename/os.replace, so the "crash mid-write never leaves a partial file" guarantee the old _flush provided still holds. The new _write delegating atomicity to the backend is safe.
• Browser-bundle safety intact ✅ — the TS _resolve() keeps the dynamic node:os/node:path imports on the default/explicit-path branches, and persist=false uses InMemoryStorage with no node imports.
• Earlier findings all resolved ✅ — the private-symbol namespacing reach-in (_NAMESPACED/_NamespacedStorage) is gone entirely (rooting LocalFileStorage at the target dir sidesteps it), and the metadata-shape validation in _read is retained.

One actionable item (Important) — PR metadata is now stale

The code is no longer a breaking change, but the PR still advertises one:

• Title: refactor(memory)!: … — the ! marks a breaking change for semver/changelog automation.
• Type of Change: Breaking change.
• The Public API Changes section still documents the abandoned storage: Storage constructor arg and the "ephemeral by default" flip.

Please update the title (drop the !), the type-of-change, and the description to reflect the actual change: an internal refactor onto Storage, no public API or behavior change. Otherwise this will get mislabeled as breaking in release notes and confuse reviewers/consumers.

Minor / optional

• Explicit-path case uses basename(path) as the storage key un-sanitized (matching the old direct-write behavior). Since keys pass through normalizeKey/_normalize_key, a single-segment basename should round-trip identically — worth a quick confirmation there's a test for a path= with an unusual filename, but low risk.

Assessment: Approve (pending the metadata cleanup). The implementation is correct, backwards-compatible, well-tested, and preserves the guarantees of the original. This ended up in a much better place than a breaking change — good call incorporating the backwards-compatible route.

n