Allow using Zeta through an arbitrary OpenAI-compatible self-hosted API

[maxbrunsfeld]

Closes #49346

Release Notes:

• Added the ability to use a self-hosted OpenAI-compatible server for edit predictions.

[Muzosh]

@maxbrunsfeld Hello Max. Is there a way to set an API key for our OpenAI-compatible servers?

[saleh-mir]

Any example usage? 'cause when I set it in the settings from the UI, it doesn't work, but I'm hoping it would work if I set it in the JSON.

[vlabo]

For anyone else trying this out. This is what worked for me:
In Settings -> Edit Prediction -> bottom provider:
Set the api key
Set the url ending with /v1/completions (eg http://localhost:1234/v1/completions)
Prompt format, Infer did not work for me. Im using qwen model so I selected qwen as the format.

The json:

  "edit_predictions": {
    "mode": "subtle",
    "open_ai_compatible_api": {
      "model": "qwen-3.5",
      "prompt_format": "qwen",
      "max_output_tokens": 200,
      "api_url": "http://localhost:1234/v1/completions",
    },
    "provider": "open_ai_compatible_api",
  },

I guess you still need to set the api key in the settings if not local.

[maxbrunsfeld]

@Muzosh Yes, you can set the API key in the settings UI. Navigate to AI > Edit Predictions > Configure Providers

[sunyitao]

Invalid on macOS, even with various URL settings.

[zaheer-s]

Improving on vlabo's comment for Lmstudio. Add line "provider": "open_ai_compatible_api",
Final setting will look like
"edit_predictions": { "provider": "open_ai_compatible_api", "open_ai_compatible_api": { "model": "qwen3.6-35b-a3b", "prompt_format": "qwen", "api_url": "http://localhost:1234/v1/completions", }, "allow_data_collection": "no", },