Skip to content

Fix empty provider list from lazy langchain_community exports - #75

Open
enichols76work wants to merge 4 commits into
prompt-security:mainfrom
enichols76work:fix/empty-provider-list
Open

Fix empty provider list from lazy langchain_community exports#75
enichols76work wants to merge 4 commits into
prompt-security:mainfrom
enichols76work:fix/empty-provider-list

Conversation

@enichols76work

@enichols76work enichols76work commented Jul 23, 2026

Copy link
Copy Markdown

Note: #72 also resolves this via a broader dependency modernization — see comment for details on the overlap/tradeoffs.

User description

Summary

  • get_langchain_chat_models_info() read chat_models_module.__dict__.get(name) directly, which bypasses langchain_community.chat_models's lazy __getattr__-based exports (PEP 562) — every class always resolved to None, so provider discovery silently returned {}.
  • This broke --list-providers, the interactive TUI's Target/Attack LLM provider menus, and instantiating any real (non-test-mocked) LangChain backend via ClientLangChain.
  • Regression from 17f2502 ("Fix critical and high severity Dependabot vulnerabilities (Fix critical and high severity Dependabot vulnerabilities #70)"), which correctly bumped langchain-community to fix CVE-2025-68664 and moved the import from langchain.chat_models to langchain_community.chat_models, but didn't account for the newer package's lazy-loading mechanism. Uncaught for ~5 months because the existing test suite mocks chat_models_info entirely and never exercised the real discovery function.
  • Fix: use getattr(chat_models_module, model_cls_name, None) so the lazy loader actually fires, wrapped in try/except so one broken lazy import can't crash the whole CLI at startup, with a logger.warning if that ever happens.

Testing

  • Added tests/test_langchain_integration.py: a real-library regression test (now discovers 59 providers, was 0) and a synthetic test reproducing the exact lazy-__getattr__ mechanism, decoupled from langchain_community's actual internals.
  • Added a test to tests/test_chat_clients.py guarding the real (unmocked) chat_models_info module attribute, since every existing test there patches it with a fake dict.
  • Full suite passes: 96 passed.

Test plan

  • pytest — full suite green
  • python -m ps_fuzz.cli --list-providers — prints real provider names, not just the header
  • python3 -c "from ps_fuzz.langchain_integration import get_langchain_chat_models_info; print(len(get_langchain_chat_models_info()))" — returns 59, was 0

Generated description

Below is a concise technical summary of the changes proposed in this PR:
Fix provider discovery in ps_fuzz.langchain_integration so ClientLangChain and the CLI menus can enumerate real LangChain chat backends again. Add regression coverage for lazy langchain_community.chat_models exports and the chat_models_info wiring.

TopicDetails
Provider discovery Resolve chat model classes with getattr() so lazy langchain_community.chat_models exports are discovered instead of returning an empty provider list.
Modified files (1)
  • ps_fuzz/langchain_integration.py
Latest Contributors(0)
UserCommitDate
Regression tests Add regression tests that exercise real model discovery, lazy __getattr__ exports, and the unmocked chat_models_info cache used by ClientLangChain.
Modified files (2)
  • tests/test_chat_clients.py
  • tests/test_langchain_integration.py
Latest Contributors(0)
UserCommitDate
Review this PR on Baz | Customize your next review

Eric Nichols and others added 4 commits July 22, 2026 22:50
get_langchain_chat_models_info() read chat_models_module.__dict__ directly,
which bypasses langchain_community.chat_models' lazy __getattr__-based
exports and always resolved to None. This left --list-providers, the
interactive provider menus, and real ClientLangChain backend resolution
all returning zero providers. Use getattr() so the lazy loader fires.

This is a regression from 17f2502 ("Fix critical and high severity
Dependabot vulnerabilities (prompt-security#70)"), which bumped langchain-community
0.0.7 -> 0.3.x to fix CVE-2025-68664. That upgrade correctly moved the
import from langchain.chat_models to langchain_community.chat_models
(chat models relocated in the 0.1+ reorg) but kept the old
__dict__.get() lookup. langchain.chat_models used to import classes
eagerly, but langchain_community.chat_models 0.3.x resolves them lazily
via module-level __getattr__, so __dict__ no longer contains them.
Provider discovery has silently returned zero providers since that
commit, uncaught because tests/test_chat_clients.py mocks
chat_models_info entirely and never exercised the real discovery
function.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
If getattr() ever raises while lazily resolving a chat model class (e.g.
a broken submodule import), the class is skipped rather than crashing
the whole discovery pass. Log it at warning level so a silently missing
provider is visible in the console and prompt-security-fuzzer.log,
rather than swallowed entirely.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add comments covering the why behind the fake lazy-loading module setup
and the usage-site patch target, and assert len(models) == 1 (not just
membership) since the fake module's __all__ only ever lists one name.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every existing test in this file patches chat_models_info with a fake
dict, so none of them would catch the module-level attribute
(chat_clients.py:15) going empty again. Add an unmocked check that it's
populated and contains well-known backends.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@baz-reviewer

baz-reviewer Bot commented Jul 23, 2026

Copy link
Copy Markdown

Merger

Waiting for CI and review to complete.

Commit 6543fd2 · Updated 2026-07-23 07:32 UTC


Review this PR on Baz | Customize your next review

@enichols76work

Copy link
Copy Markdown
Author

Flagging for reviewers: #72 also fixes this same bug (empty provider list from langchain_community.chat_models's lazy exports), but via a different and much broader approach worth weighing against this PR:

  • This PR: a minimal, one-line fix (chat_models_module.__dict__.get()getattr()) that preserves all 59 dynamically-discovered langchain_community chat model backends, with no other behavior change.
  • Update Python dependencies and LangChain integrations #72: replaces the whole discovery mechanism with a hardcoded CHAT_MODEL_REGISTRY of just 2 backends (open_ai, ollama), imported directly from langchain-openai/langchain-ollama instead of langchain_community. It's bundled with a much larger dependency modernization — raises the Python floor from >=3.9 to >=3.11 (dropping 3.9/3.10 support), bumps most dependencies to new major versions, and drops chromadb to an optional extra. It also fixes the pkg_resources deprecation warning in base64_injection.py/harmful_behavior.py/custom_benchmark.py the same way this repo would eventually need to.

Both resolve the immediate bug; #72 is the bigger, more invasive change. Wanted to surface the overlap so we don't land both / duplicate review effort — happy to close this one in favor of #72 if that's the preferred direction, or keep this as the low-risk fix to land now with #72's broader changes considered separately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant