Should the configuration be stored in plugin_metadata or etcd?

[renshunhang]

I currently have a business scenario where I need to store resource access permissions corresponding to different roles, and plan to implement authentication via custom plugins bound to all routes.
I intend to keep this data in plugin metadata. Should large datasets be migrated to etcd or Redis instead? What is the proper data size threshold for metadata storage?

[dosubot[bot]]

Plugin metadata is stored in etcd — it's APISIX's mechanism for global plugin configuration [1]. There's no explicit size limit enforced by APISIX itself, but etcd's default value size limit (~1.5 MB per entry) applies.

That said, plugin_metadata is designed for plugin configuration (log formats, API keys, etc.), not for application data like RBAC permission mappings [2]. Storing a large and frequently-changing permission dataset there has drawbacks:

• Every update requires an etcd write + config reload across all APISIX nodes
• It couples your authorization data tightly to the gateway's configuration lifecycle
• It doesn't scale well as roles/resources grow

Recommended approach: Use the forward-auth plugin to delegate authorization to an external service [3]. Your custom service holds the role→permission mappings (backed by Redis, a database, etc.) and APISIX simply forwards each request to it — returning 2xx for allow or 403 for deny. This gives you a single source of truth, independent scaling, and no gateway config changes when permissions update.

If latency from per-request external calls is a concern, you can layer in caching on the external service side or use short-lived JWTs that encode permissions.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[mg1986jp]

Plugin metadata is meant for small, shared configuration that applies to all instances of a plugin — stuff like connection strings, default timeouts, etc. It's not really designed for large or frequently changing datasets like RBAC permission tables.

For role-permission mappings, I'd keep them in etcd or Redis rather than plugin_metadata. Here's why:

• plugin_metadata gets loaded into every APISIX worker's memory. If the dataset grows large, you're multiplying that memory usage across all workers.
• Changes to plugin_metadata trigger a full reload across all workers, which isn't great if permissions change frequently.
• There's no built-in TTL or caching strategy for metadata — it's always fully loaded.

A practical threshold: if your data fits in a few KB and rarely changes, metadata is fine. Once you're looking at hundreds of entries or frequent updates, move it to Redis (for fast lookups) or etcd (if you want it to be part of the APISIX config ecosystem).

For an RBAC plugin specifically, Redis is probably the better fit — you can cache the permission lookups with a short TTL and handle invalidation cleanly.