Serialize storage payload before truncating the backup file

[evnchn]

Motivation

User does app.storage.user['x'] = {1, 2, 3} (a set) or any non-JSON-serializable object after storage-user-.json already holds valid keys. on_change -> backup -> async_backup runs, opens the file 'w' (file now empty on disk), then dumps() raises TypeError. File is left as 0 bytes. On next server start initialize() loads {} -> the entire user's storage is permanently lost.

Minimal reproduction (nicegui/persistence/file_persistent_dict.py:52) — the exact test/script that was run to reproduce it:

import asyncio
from pathlib import Path

from nicegui import core
from nicegui.persistence.file_persistent_dict import FilePersistentDict


async def main() -> None:
    core.loop = asyncio.get_running_loop()  # make core.is_loop_running() -> True (async backup path)

    path = Path('storage-user-mre.json')
    d = FilePersistentDict(path)

    # 1) persist valid data
    d['a'] = 1
    d['b'] = 2
    await asyncio.sleep(0.3)
    before = path.read_text()
    assert before and '"a"' in before and '"b"' in before, f'setup failed: {before!r}'
    print(f'before: file = {before!r}  (valid data persisted)')

    # 2) assign ONE non-serializable value (a set) on top of the valid data
    d['bad'] = {1, 2, 3}
    await asyncio.sleep(0.5)  # let async_backup run and raise

    after = path.read_text()
    print(f'after:  file = {after!r}  (size={path.stat().st_size})')

    if after == '' and path.exists():
        print('BUG REPRODUCED: valid persisted data was truncated to 0 bytes; '
              'on restart initialize() would load {} and lose keys a, b.')
    else:
        print('NOT reproduced: file retained content.')
        raise SystemExit(1)


if __name__ == '__main__':
    asyncio.run(main())

→ before: file = '{"a":1,"b":2}' (valid data persisted)

Implementation

Serialize storage payload before opening/truncating the backup file in async_backup so a non-JSON-serializable value no longer wipes persisted data.

Verification

• FAILS with assert '' == '{"a":1,"b":2}' (file truncated to 0 bytes), fixed src PASSES.
• Local gates: pre-commit · mypy ./nicegui · pylint 10.00/10 · pytest (touched) — all green.

Progress

• The PR title is a short phrase starting with a verb.
• The implementation is complete.
• This PR does not address a security issue.
• Pytests have been added/updated.
• Documentation is not necessary.
• No breaking changes to the public API.

[falkoschindler]

@falkoschindler asked me to bring the open PRs up to date with main so the diffs are easier to read. This one couldn't be updated automatically — it conflicts.

The conflict is in nicegui/persistence/file_persistent_dict.py, against 936b071 ("Fix PermissionError when clearing storage on Windows", #6174). That commit touched the same write path this PR reorders, so the two changes need to be reconciled deliberately rather than merged mechanically — handing it back to you.

Assigning you to make the ownership visible.

---

Generated by Claude Code

[evnchn]

Closing this as subsumed by #6174 — and the remaining delta turns out to be harmful rather than neutral.

#6174 ("Fix PermissionError when clearing storage on Windows") introduced the temp-file + os.replace write, which already delivers what this PR was after: a failing dumps() can no longer destroy the existing storage file. I confirmed that on main — the real file survives a serialization failure with or without this change.

What was left here is the hoist of dumps() above aiofiles.open(). Reconciling the two changes made it clear that hoist is a regression: it takes the state snapshot before the coroutine's first await, so a write landing during the open() thread-pool round-trip is dropped at shutdown. That trades real data loss for temp-file tidiness, which inverts this PR's own purpose.

Probe: the mutation that gets lost

Write a, await asyncio.sleep(0) so the backup task parks on the open() await, write b, then await background_tasks.teardown():

tree	file after teardown
main (snapshot after the await)	{"a":1,"b":2}
this PR (snapshot hoisted above it)	{"a":1}

The mechanism is in nicegui/background_tasks.py: the queued follow-up backup is re-created from a plain coroutine, so it is no longer an _AwaitOnShutdown and is cancelled at teardown. On main that cancellation is benign, because the surviving task serializes after the await and picks up the late write. Hoisting moves the snapshot before that await, so the late write is carried only by the task that gets cancelled.

All 29 tests in tests/test_storage.py pass on the hoisted tree, including test_awaiting_backup_scheduled_during_teardown — the suite has no coverage for snapshot timing, which is why this needed a targeted probe rather than a test run.

If the leftover zero-byte .tmp on a serialization failure is still worth cleaning up, the safe shape is to unlink it in a failure path while leaving await f.write(dumps(...)) where main has it — I'm happy to open that separately if you'd like it.