Add declarative custom providers through native Pi AI

This commit is contained in:
amosblomqvist
2026-10-28 04:42:46 +02:00
parent 8f6fd7cd29
commit f84bef1f91
11 changed files with 811 additions and 3 deletions
+3
View File
@@ -111,6 +111,9 @@ export const GRAPH_FORMAT_FILE = path.join(PATHS.graph, "format.json");
*/
export const SETTINGS_FILE = path.join(DATA_ROOT, "settings.json");
/** Explicit custom-provider/model definitions; never read the user's Pi profile. */
export const CUSTOM_MODELS_FILE = path.join(DATA_ROOT, "models.json");
/**
* User prompt-override files (settings-storage wave, 2026-09-19): the
* per-segment system-prompt overrides mirroring the SOURCE layout
+159
View File
@@ -0,0 +1,159 @@
# Custom model providers
LearnHarness can register several independently named custom providers using its
existing Pi AI runtime. This is configuration wiring, not another agent runtime:
Pi owns the model protocols, request conversion, streaming, tools, thinking and
usage accounting. There is no additional dependency on Pi coding-agent.
## Configuration location
Create `models.json` in the LearnHarness app-data directory (the directory
containing `settings.json`; `LH_DATA_DIR` relocates it). LearnHarness does not read
`~/.pi/agent/models.json`, install a model server, or edit this file for you.
Custom-provider setup UI is not included yet. Once configured, models appear in
the existing model selectors when their credential/header references resolve.
The file uses Pi's `providers` / `models` structure. For example:
```json
{
"providers": {
"my-local-server": {
"baseUrl": "http://127.0.0.1:11434/v1",
"api": "openai-completions",
"apiKey": "unused-local-key",
"compat": {
"supportsDeveloperRole": false,
"supportsReasoningEffort": false,
"maxTokensField": "max_tokens"
},
"models": [
{
"id": "your-installed-model",
"name": "My local model",
"contextWindow": 8192,
"maxTokens": 1024,
"cost": {
"input": 0,
"output": 0,
"cacheRead": 0,
"cacheWrite": 0
}
}
]
},
"my-gateway": {
"baseUrl": "https://your-gateway.example/v1",
"api": "openai-completions",
"apiKey": "$MY_GATEWAY_API_KEY",
"headers": { "X-Tenant": "${MY_GATEWAY_TENANT}" },
"models": [
{
"id": "your-gateway-model-id",
"contextWindow": 32768,
"maxTokens": 4096,
"cost": {
"input": 1,
"output": 3,
"cacheRead": 0,
"cacheWrite": 0
}
}
]
}
}
}
```
**Replace all example IDs, limits, prices and compatibility flags with the actual
server/model configuration.** These numbers are examples, not recommendations or
automatically detected capabilities. Rates are USD per million tokens. Explicit
zero rates are allowed; LearnHarness does not assume that custom gateways are
free. Verify the endpoint's billing independently.
Use the exact API base URL your server documents. LearnHarness does not append
`/v1`, discover models, infer capabilities from names, or download models. Ollama,
vLLM, LM Studio, 9router and other gateways work only to the extent that their
configured endpoint implements the selected Pi protocol and compatibility flags.
For a keyless OpenAI-compatible local server, supply an explicit nonempty dummy
`apiKey` as above: the native Pi OpenAI client requires a key. Omitting `apiKey`
leaves a provider unavailable rather than implying unauthenticated access.
## Supported fields
Provider fields: `name`, `baseUrl`, `api`, `apiKey`, `headers`, `authHeader`,
`compat`, and the explicit `models` array. Each model requires `id`,
`contextWindow`, `maxTokens`, and `cost` with all four base rates. Unlike Pi
coding-agent's defaults, LearnHarness never invents limits or prices.
Optional Pi model fields are `name`, `api`, `baseUrl`, `reasoning`, `input`,
`thinkingLevelMap`, `inputLimits`, `promptCache`, `samplingParams`,
`samplingParamsByThinkingLevel`, `headers`, and `compat`. Cost tiers and Pi's
exported compatibility schemas are supported. A model's `api` and `baseUrl`
override its provider's values; model headers override provider headers. Names
default to model IDs, `reasoning` defaults to false, and `input` defaults to
`["text"]`. Declare image input only when the server/model supports it. Existing
onboarding and vision-slot requirements are unchanged.
The native chat transports available in the pinned Pi AI 1.4.0 dependency are:
`openai-completions`, `openai-responses`, `anthropic-messages`,
`google-generative-ai`, `azure-openai-responses`, `google-vertex`,
`bedrock-converse-stream`, `mistral-conversations`, `openai-codex-responses`, and
`pi-messages`. Protocol availability does **not** establish account access,
OAuth support, cloud credentials, subscription permission, or vendor approval.
This feature adds no login flows. In particular, an Anthropic Messages endpoint
is not a Claude Agent SDK or Claude Code runtime.
Use new provider IDs; built-in and subscription-route IDs cannot be replaced.
Provider IDs must be nonempty and cannot contain `:` (the provider/model
coordinate separator). Model IDs must be unique within a provider. Built-in
provider overrides, `modelOverrides`, OAuth definitions, executable plugins, and
automatic discovery are outside this phase.
## Credentials and trust
API keys and header values accept literals, `$NAME`, and `${NAME}`. References
use LearnHarness's existing key lookup: environment first, then the existing
flat `auth.json` file, respecting the fresh-user environment-ignore setting.
There is no auth migration or new secret file format. Prefer references over
putting secrets into `models.json`; protect any file containing literal secrets.
Unknown variable references keep the provider unavailable. All required header
references, including model-specific ones, must resolve before a provider becomes
available. An empty key does not configure a provider.
Pi's `$$` and `$!` literal escapes are supported. `!command` credential/header
values are rejected, never executed; resolved environment/file values are data,
not commands. Optional `authHeader: true` adds `Authorization: Bearer <key>` in
addition to the selected native transport's normal authentication behavior.
Only trusted people should edit this file: its endpoints receive model prompts,
attachments, tool results and configured credentials. Only HTTP(S) URLs are
accepted, without embedded URL credentials or fragments. This is not an
arbitrary-endpoint discovery or config-editing API and does not add a network
sandbox, endpoint reachability checks, or vendor policy exemptions.
## Reloading and failures
Changes take effect on the next catalog, settings-validation, or inference
operation without restarting. Credential/header reference changes are also
picked up through the existing key lookup. Every upstream model operation
revalidates the same saved provider/model ID against the current Pi catalog,
including later requests inside a long-running agent/tool loop. It uses the
current endpoint, model metadata and resolved credentials. A removed model,
invalid configuration or missing credential blocks the next operation, without
falling back. An HTTP operation already in flight retains its own native Pi
setup and is not implicitly cancelled by a file edit.
Deleting the file means no custom providers. Invalid JSON, invalid metadata,
reserved IDs, unsupported configuration, or an unreadable file disables the
custom routes, **not** built-in providers. The file accepts Pi's line comments
and trailing commas. `GET /api/models` includes an optional sanitized
`customProvidersError` diagnostic; rendering it in setup/settings UI is deferred.
Correcting the file restores valid providers on the next operation.
Saved selections and thinking preferences are not erased when a provider/model
is removed or becomes unavailable. New work using an unavailable selection fails
rather than falling back to another model. Select another configured model to
repair the slot. Custom-provider removal does not rewrite historical records;
like existing API-key models, historical cost estimates use the current catalog
and become unknown when the relevant model definition is unavailable.
+68
View File
@@ -0,0 +1,68 @@
/** Native Pi providers assembled from validated configuration. No IO or registry
* here: the server supplies key lookup, and Pi owns model dispatch/streaming. */
import { createProvider, type Provider, type ProviderStreams } from "@earendil-works/pi-ai";
import { anthropicMessagesApi } from "@earendil-works/pi-ai/api/anthropic-messages.lazy";
import { openAICompletionsApi } from "@earendil-works/pi-ai/api/openai-completions.lazy";
import { openAIResponsesApi } from "@earendil-works/pi-ai/api/openai-responses.lazy";
import { openAICodexResponsesApi } from "@earendil-works/pi-ai/api/openai-codex-responses.lazy";
import { azureOpenAIResponsesApi } from "@earendil-works/pi-ai/api/azure-openai-responses.lazy";
import { googleGenerativeAIApi } from "@earendil-works/pi-ai/api/google-generative-ai.lazy";
import { googleVertexApi } from "@earendil-works/pi-ai/api/google-vertex.lazy";
import { bedrockConverseStreamApi } from "@earendil-works/pi-ai/api/bedrock-converse-stream.lazy";
import { mistralConversationsApi } from "@earendil-works/pi-ai/api/mistral-conversations.lazy";
import { piMessagesApi } from "@earendil-works/pi-ai/api/pi-messages.lazy";
import { resolveCustomValue, type CustomProviderConfig } from "../core/custom-providers.ts";
const apis: Record<string, ProviderStreams> = {
"anthropic-messages": anthropicMessagesApi(),
"openai-completions": openAICompletionsApi(),
"openai-responses": openAIResponsesApi(),
"openai-codex-responses": openAICodexResponsesApi(),
"azure-openai-responses": azureOpenAIResponsesApi(),
"google-generative-ai": googleGenerativeAIApi(),
"google-vertex": googleVertexApi(),
"bedrock-converse-stream": bedrockConverseStreamApi(),
"mistral-conversations": mistralConversationsApi(),
"pi-messages": piMessagesApi(),
};
export const customProviderApis: ReadonlySet<string> = new Set(Object.keys(apis));
function resolveHeaders(headers: Record<string, string> | undefined, lookup: (name: string) => string | undefined): Record<string, string> | undefined {
const result: Record<string, string> = {};
for (const [name, value] of Object.entries(headers ?? {})) {
const resolved = resolveCustomValue(value, lookup);
if (resolved === undefined) return undefined;
result[name] = resolved;
}
return result;
}
export function customProviderAuth(config: CustomProviderConfig, lookup: (name: string) => string | undefined) {
const key = config.definition.apiKey === undefined ? undefined : resolveCustomValue(config.definition.apiKey, lookup);
const headers = resolveHeaders(config.definition.headers, lookup);
if (!key || !headers) return undefined;
// A model header is part of that model's request configuration. Unresolved
// references must not silently disappear from a request.
for (const model of config.models) if (!resolveHeaders(model.headers, lookup)) return undefined;
if (config.definition.authHeader) {
for (const name of Object.keys(headers)) if (name.toLowerCase() === "authorization") delete headers[name];
headers.Authorization = `Bearer ${key}`;
}
return { apiKey: key, headers };
}
export function createCustomProvider(config: CustomProviderConfig, lookup: (name: string) => string | undefined): Provider {
const auth = customProviderAuth(config, lookup);
return createProvider({
id: config.id,
name: config.definition.name ?? config.id,
baseUrl: config.definition.baseUrl,
models: config.models.map((model) => ({ ...model, headers: resolveHeaders(model.headers, lookup) })),
auth: { apiKey: {
name: "API key",
check: async () => auth ? { type: "api_key", source: "custom provider configuration" } : undefined,
resolve: async () => auth ? { auth, source: "custom provider configuration" } : undefined,
} },
// Direct native implementations: no copied stream or request conversion.
api: apis,
});
}
+35 -1
View File
@@ -5,9 +5,11 @@ import {
type Api,
type Model,
type Models,
type MutableModels,
type StreamFunction,
} from "@earendil-works/pi-ai";
import { builtinProviders } from "@earendil-works/pi-ai/providers/all";
import { lazyStream } from "@earendil-works/pi-ai/api/lazy";
import type { ThinkingLevel } from "@earendil-works/pi-agent-core";
import { type ModelRole } from "../../config.ts";
import { DEFAULT_THINKING_ORDER } from "../core/settings.ts";
@@ -49,6 +51,23 @@ export interface PiEngine {
export type { ThinkingLevel };
let collection: ReturnType<typeof createModels> | undefined;
interface ProviderSource {
sync(models: MutableModels): void;
isCustom(id: string): boolean;
}
let providerSource: ProviderSource | undefined;
/** Configuration IO is injected by the server; the agent never imports stores. */
export function configureProviderSource(source: ProviderSource): void {
// Server module re-evaluation can replace the source while this collection
// survives. Retire only the previous source's routes, not built-ins.
if (collection && providerSource) {
for (const provider of collection.getProviders()) {
if (providerSource.isCustom(provider.id)) collection.deleteProvider(provider.id);
}
}
providerSource = source;
}
/** The process-wide pi-ai Models collection — EVERY builtin provider's
* catalog (model slots wave, 2026-09-20): engines and the picker resolve
@@ -63,6 +82,7 @@ export function piModels() {
collection = createModels({ identity: APP_IDENTITY });
for (const provider of builtinProviders()) collection.setProvider(provider);
}
providerSource?.sync(collection);
return collection;
}
@@ -111,6 +131,9 @@ export function piEngineForSpec(spec: ResolvedModelSpec, background: boolean | (
const level = spec.thinkingLevel
? clampThinkingLevel(model, spec.thinkingLevel)
: defaultThinkingLevel(model);
// A lesson's agent loop can stay alive through many tool calls. Remember
// only that this is a custom route, never its configuration for that run.
const custom = providerSource?.isCustom(spec.provider) ?? false;
return {
model,
level,
@@ -118,9 +141,20 @@ export function piEngineForSpec(spec: ResolvedModelSpec, background: boolean | (
// `reasoning` (undefined = off on pi-ai's scale); the clamp above keeps
// the pin honest even when config or settings drift off the map.
streamFn: (requestModel, context, options) => {
const requestOptions = { ...options, reasoning: level === "off" ? undefined : level };
if (custom) {
// Revalidate before EVERY upstream model operation, including later
// tool-loop steps. Pi resolves auth and clamps the requested thinking
// against the current native model. Its in-flight operation retains
// its own setup; a subsequent operation uses the reloaded provider.
const current = piModels();
const currentModel = current.getModel(spec.provider, spec.id);
if (!currentModel) return lazyStream(requestModel, async () => { throw new Error("The selected custom model is unavailable."); });
return current.streamSimple(currentModel, context, requestOptions);
}
const purpose = typeof background === "function" ? background : () => background;
const stream = spec.provider === "chatgpt" ? chatGPTStream(purpose, lifecycle) : nativeId ? nativeSubscriptionStream(models.getProvider(nativeId)!, purpose, lifecycle) : models.streamSimple.bind(models);
return stream(requestModel, context, { ...options, reasoning: level === "off" ? undefined : level });
return stream(requestModel, context, requestOptions);
},
};
}
+4 -2
View File
@@ -73,13 +73,15 @@ export interface ModelCatalogResponse {
* are simply absent (the Keys page is the discovery surface, same
* discipline as the search engines). */
providers: CatalogProvider[];
/** Sanitized configuration diagnostic; built-in catalogs remain available. */
customProvidersError?: string;
}
/** One row of the providers surface. */
export interface CatalogProvider {
id: string;
/** The env var whose presence federates this provider in. */
/** The gating env var; empty for subscription and file-configured routes. */
envVar: string;
/** Whether that var is currently present (env or auth file). */
/** Whether credentials are present (and custom references resolve). */
keyPresent: boolean;
}
+146
View File
@@ -0,0 +1,146 @@
/** Declarative custom-provider configuration. The envelope/normalization follows
* Pi coding-agent 1.4.0's model-config/provider-composer (MIT, Mario Zechner).
* Pi AI owns the shared model/compatibility schemas and the runtime models.
* Deliberate differences: new provider IDs only, explicit limits/prices,
* and no OAuth, extensions, overrides, or executable credential commands.
*
* Adapted portions: Copyright (c) Mario Zechner. Permission is hereby granted,
* free of charge, to any person obtaining a copy of this software and associated
* documentation files (the "Software"), to deal in the Software without
* restriction, including without limitation the rights to use, copy, modify,
* merge, publish, distribute, sublicense, and/or sell copies of the Software,
* and to permit persons to whom the Software is furnished to do so, subject to
* the following conditions: The above copyright notice and this permission
* notice shall be included in all copies or substantial portions of the Software.
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE. */
import { Type, type Static } from "typebox";
import { Compile } from "typebox/compile";
import type { Api, Model } from "@earendil-works/pi-ai";
import { ProviderCompatSchema } from "@earendil-works/pi-ai/providers/compat-schema";
import { ModelCostSchema, ModelInputLimitsSchema, ModelInputModalitySchema, ModelPromptCacheSchema, ThinkingLevelMapSchema } from "@earendil-works/pi-ai/providers/model-schema";
const Headers = Type.Record(Type.String(), Type.String());
const Sampling = Type.Record(Type.String(), Type.Unknown());
const ModelSchema = Type.Object({
id: Type.String({ minLength: 1 }),
name: Type.Optional(Type.String({ minLength: 1 })),
api: Type.Optional(Type.String({ minLength: 1 })),
baseUrl: Type.Optional(Type.String({ minLength: 1 })),
reasoning: Type.Optional(Type.Boolean()),
thinkingLevelMap: Type.Optional(ThinkingLevelMapSchema),
input: Type.Optional(Type.Array(ModelInputModalitySchema)),
inputLimits: Type.Optional(ModelInputLimitsSchema),
cost: ModelCostSchema,
promptCache: Type.Optional(ModelPromptCacheSchema),
contextWindow: Type.Number({ exclusiveMinimum: 0 }),
maxTokens: Type.Number({ exclusiveMinimum: 0 }),
samplingParams: Type.Optional(Sampling),
samplingParamsByThinkingLevel: Type.Optional(Type.Object(Object.fromEntries(
["off", "minimal", "low", "medium", "high", "xhigh", "max"].map((level) => [level, Type.Optional(Sampling)]),
))),
headers: Type.Optional(Headers),
compat: Type.Optional(ProviderCompatSchema),
}, { additionalProperties: false });
const ProviderSchema = Type.Object({
name: Type.Optional(Type.String({ minLength: 1 })),
baseUrl: Type.Optional(Type.String({ minLength: 1 })),
api: Type.Optional(Type.String({ minLength: 1 })),
apiKey: Type.Optional(Type.String({ minLength: 1 })),
headers: Type.Optional(Headers),
compat: Type.Optional(ProviderCompatSchema),
authHeader: Type.Optional(Type.Boolean()),
models: Type.Array(ModelSchema),
}, { additionalProperties: false });
const ConfigSchema = Type.Object({
$schema: Type.Optional(Type.String()),
providers: Type.Record(Type.String(), ProviderSchema),
}, { additionalProperties: false });
const validator = Compile(ConfigSchema);
export type CustomProviderDefinition = Static<typeof ProviderSchema>;
export interface CustomProviderConfig {
id: string;
definition: CustomProviderDefinition;
models: Model<Api>[];
}
/** Error messages describe fields only: never include file contents or values. */
export class CustomProviderConfigError extends Error {}
function invalid(message: string): never { throw new CustomProviderConfigError(message); }
/** Pi-style values, but never execute commands. Escapes are processed once;
* a literal/variable's resolved value beginning with ! is still just data. */
export function resolveCustomValue(value: string, lookup: (name: string) => string | undefined): string | undefined {
if (value.startsWith("!")) invalid("Credential commands are not supported in models.json.");
let missing = false;
const resolved = value.replace(/\$\$|\$!|\$\{([A-Za-z_][A-Za-z0-9_]*)\}|\$([A-Za-z_][A-Za-z0-9_]*)/g, (match, braced: string | undefined, plain: string | undefined) => {
if (match === "$$") return "$";
if (match === "$!") return "!";
const found = lookup(braced ?? plain!);
if (found === undefined) missing = true;
return found ?? "";
});
return missing ? undefined : resolved;
}
function endpoint(value: string | undefined): string {
if (!value) invalid("Each custom model needs an api and baseUrl, at provider or model level.");
try {
const url = new URL(value);
if (!["http:", "https:"].includes(url.protocol) || url.username || url.password || url.hash) throw new Error();
} catch { invalid("Custom baseUrl must be an HTTP(S) URL without embedded credentials or a fragment."); }
return value;
}
function compat(base: Model<Api>["compat"], override: Model<Api>["compat"]): Model<Api>["compat"] {
if (!base && !override) return undefined;
const merged = { ...base, ...override } as Record<string, unknown>;
for (const name of ["openRouterRouting", "vercelGatewayRouting", "chatTemplateKwargs", "chatTemplateArgs"]) {
const a = (base as Record<string, unknown> | undefined)?.[name];
const b = (override as Record<string, unknown> | undefined)?.[name];
if (a || b) merged[name] = { ...(a as object | undefined), ...(b as object | undefined) };
}
return merged as Model<Api>["compat"];
}
export function parseCustomProviders(text: string, reserved: ReadonlySet<string>, supportedApis: ReadonlySet<string>): CustomProviderConfig[] {
let raw: unknown;
// Match Pi's line comments/trailing commas without touching JSON strings.
const json = text.replace(/^\uFEFF/, "")
.replace(/"(?:\\.|[^"\\])*"|\/\/[^\n]*/g, (match) => match[0] === '"' ? match : "")
.replace(/"(?:\\.|[^"\\])*"|,(\s*[}\]])/g, (match, tail: string | undefined) => tail ?? (match[0] === '"' ? match : ""));
try { raw = JSON.parse(json); }
catch { return invalid("models.json is not valid JSON."); }
if (!validator.Check(raw)) invalid("Invalid models.json structure: check provider/model fields and explicit contextWindow, maxTokens and cost.");
return Object.entries(raw.providers).map(([id, definition]) => {
if (!id.trim() || id.includes(":")) invalid("Custom provider IDs must be nonempty and must not contain a colon.");
if (reserved.has(id)) invalid("Custom provider IDs must not replace built-in or subscription providers.");
for (const value of [definition.apiKey, ...Object.values(definition.headers ?? {}), ...definition.models.flatMap((m) => Object.values(m.headers ?? {}))]) {
if (value !== undefined) resolveCustomValue(value, () => "validation");
}
const ids = new Set<string>();
const models = definition.models.map((model): Model<Api> => {
if (ids.has(model.id)) invalid("Duplicate model IDs are not supported within a custom provider.");
ids.add(model.id);
const api = model.api ?? definition.api;
if (!api || !supportedApis.has(api)) invalid("A custom model specifies a missing or unsupported Pi API.");
const rates = [model.cost, ...(model.cost.tiers ?? [])];
if (![model.contextWindow, model.maxTokens].every(Number.isFinite) || rates.some((rate) => Object.values(rate).some((n) => typeof n === "number" && (!Number.isFinite(n) || n < 0)))) invalid("Custom model limits and costs must be finite; limits must be positive and costs nonnegative.");
return {
...model,
provider: id,
api,
baseUrl: endpoint(model.baseUrl ?? definition.baseUrl),
name: model.name ?? model.id,
reasoning: model.reasoning ?? false,
input: model.input ?? ["text"],
compat: compat(definition.compat, model.compat),
};
});
return { id, definition, models };
});
}
+90
View File
@@ -0,0 +1,90 @@
/** Configuration composition only. Pi's Models collection is the one runtime
* registry. A failed reload removes custom routes, never alters built-ins. */
import type { MutableModels } from "@earendil-works/pi-ai";
import { builtinProviders } from "@earendil-works/pi-ai/providers/all";
import { createCustomProvider, customProviderApis, customProviderAuth } from "../agent/custom-providers.ts";
import { configureProviderSource } from "../agent/models.ts";
import { parseCustomProviders, resolveCustomValue, CustomProviderConfigError, type CustomProviderConfig } from "../core/custom-providers.ts";
import { NATIVE_SUBSCRIPTIONS } from "../core/contracts/subscriptions.ts";
import { readCustomProvidersFile } from "../store/custom-providers.ts";
const reserved = new Set([...builtinProviders().map((p) => p.id), "chatgpt", ...NATIVE_SUBSCRIPTIONS.map((p) => p.logicalId)]);
/** Instance-local so every Next server module graph registers into its own Pi
* collection. No process-global registry or personal Pi config is consulted. */
export class CustomProviderSource {
private text?: string;
private fingerprint?: string;
private configs: CustomProviderConfig[] = [];
private registered = new Set<string>();
private ready = new Map<string, boolean>();
private error?: string;
constructor(
private readonly read: typeof readCustomProvidersFile,
private readonly key: (name: string) => string | undefined,
) {}
sync(models: MutableModels): void {
try {
const record = this.read();
if (record.error) throw new CustomProviderConfigError(record.error);
if (record.text !== this.text || this.error) {
this.configs = parseCustomProviders(record.text!, reserved, customProviderApis);
this.text = record.text;
this.fingerprint = undefined;
}
// Resolve only referenced variables through the app's existing key seam.
// Changes in credentials/headers trigger a new immutable provider snapshot
// even when the configuration file itself has not changed.
const values = new Map<string, string | undefined>();
for (const config of this.configs) {
for (const template of [config.definition.apiKey, ...Object.values(config.definition.headers ?? {}), ...config.models.flatMap((m) => Object.values(m.headers ?? {}))]) {
if (template !== undefined) resolveCustomValue(template, (name) => {
const value = this.key(name);
values.set(name, value);
return value;
});
}
}
const fingerprint = JSON.stringify([...values]);
if (this.fingerprint === fingerprint) return;
const lookup = (name: string) => values.get(name);
const providers = this.configs.map((config) => createCustomProvider(config, lookup));
const ready = new Map(this.configs.map((config) => [config.id, customProviderAuth(config, lookup) !== undefined]));
// Everything validates/builds before the synchronous publication begins.
for (const id of this.registered) models.deleteProvider(id);
this.registered = new Set(providers.map((p) => p.id));
for (const provider of providers) models.setProvider(provider);
this.ready = ready;
this.fingerprint = fingerprint;
this.error = undefined;
} catch (error) {
for (const id of this.registered) models.deleteProvider(id);
this.registered.clear();
this.ready.clear();
this.fingerprint = undefined;
this.error = error instanceof CustomProviderConfigError ? error.message : "Unable to configure custom providers from models.json.";
}
}
configured(provider: string): boolean | undefined { return this.ready.get(provider); }
providers(): { id: string; envVar: string; keyPresent: boolean }[] {
return [...this.ready].map(([id, keyPresent]) => ({ id, envVar: "", keyPresent }));
}
getError(): string | undefined { return this.error; }
}
let source: CustomProviderSource | undefined;
export function installCustomProviderSource(key: (name: string) => string | undefined): void {
const next = new CustomProviderSource(readCustomProvidersFile, key);
configureProviderSource({
sync: (models) => next.sync(models),
isCustom: (id) => next.configured(id) !== undefined,
});
source = next;
}
/** Call piModels() before these reads to synchronize the snapshot. */
export function customProviderRows() { return source?.providers() ?? []; }
export function customProviderConfigured(provider: string): boolean | undefined { return source?.configured(provider); }
export function customProvidersError(): string | undefined { return source?.getError(); }
+6
View File
@@ -53,6 +53,8 @@ import {
} from "../core/auth-keys.ts";
import { authStore } from "../store/auth.ts";
import { isSubscriptionProvider, subscriptionConnected } from "../agent/subscription-connections.ts";
import { piModels } from "../agent/models.ts";
import { customProviderConfigured, installCustomProviderSource } from "./custom-providers.ts";
// The once-per-process boot record — globalThis anchor (HMR re-imports the
// module; the snapshot must not re-run and misread fill-gapped vars as real
@@ -201,6 +203,9 @@ export function keyValue(id: string): string | undefined {
* (gate by what would run). Providers absent from AUTH_PROVIDER_KEYS take
* no key at all: nothing to gate, present by definition. */
export function providerKeyPresent(provider: string): boolean {
piModels(); // Synchronize hand-edited custom configuration before admission.
const custom = customProviderConfigured(provider);
if (custom !== undefined) return custom;
if (isSubscriptionProvider(provider)) return subscriptionConnected(provider);
const info = AUTH_PROVIDER_KEYS.find((p) => p.id === provider);
return info ? keyValue(info.envVar) !== undefined : true;
@@ -244,3 +249,4 @@ export function deleteKey(id: string): AuthKeyStatus {
// Reflect the file into the env at first import (idempotent; the module is
// imported by every route/handler path that touches keys).
initAuthEnv();
installCustomProviderSource(keyValue);
+4
View File
@@ -20,6 +20,7 @@
import { clampThinkingLevel, getSupportedThinkingLevels, type Api, type Model } from "@earendil-works/pi-ai";
import type { ThinkingLevel } from "@earendil-works/pi-agent-core";
import { piModels } from "../agent/models.ts";
import { customProviderRows, customProvidersError } from "./custom-providers.ts";
import { chatGPTConnected, chatGPTModel, chatGPTModels, refreshChatGPTModels } from "../agent/chatgpt-bridge.ts";
import { NATIVE_SUBSCRIPTIONS } from "../core/contracts/subscriptions.ts";
import { nativeSubscriptionId, nativeSubscriptionConnected, nativeSubscriptionModel, nativeSubscriptionModels, refreshSubscriptionModels } from "../agent/subscription-bridge.ts";
@@ -135,6 +136,7 @@ export function catalogSlots(): SlotPins {
* with keyPresent read env-wins through the keys seam, fresh per call —
* saving a key immediately federates the provider into the picker. */
export function catalogProviders(): CatalogProvider[] {
piModels();
return AUTH_PROVIDER_KEYS.map((p) => ({
id: p.id,
envVar: p.envVar,
@@ -142,6 +144,7 @@ export function catalogProviders(): CatalogProvider[] {
})).concat([
{ id: "chatgpt", envVar: "", keyPresent: chatGPTConnected() },
...NATIVE_SUBSCRIPTIONS.map((p) => ({ id: p.logicalId, envVar: "", keyPresent: nativeSubscriptionConnected(p.providerId) })),
...customProviderRows(),
]).sort((a, b) => a.id.localeCompare(b.id));
}
@@ -211,6 +214,7 @@ export function modelCatalog(): ModelCatalogResponse {
rolePins: catalogRolePins(),
slots: catalogSlots(),
providers: catalogProviders(),
...(customProvidersError() ? { customProvidersError: customProvidersError() } : {}),
};
}
+12
View File
@@ -0,0 +1,12 @@
/** Read-only adapter. Configuration editing/UI is deliberately a later phase. */
import fs from "node:fs";
import { CUSTOM_MODELS_FILE } from "../../config.ts";
export function readCustomProvidersFile(): { text?: string; error?: string } {
try { return { text: fs.readFileSync(CUSTOM_MODELS_FILE, "utf8") }; }
catch (error) {
return (error as NodeJS.ErrnoException).code === "ENOENT"
? { text: '{"providers":{}}' }
: { error: "Unable to read models.json." };
}
}
+284
View File
@@ -0,0 +1,284 @@
import assert from "node:assert/strict";
import { after, describe, it } from "node:test";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { createModels, normalizeContext } from "@earendil-works/pi-ai";
import { parseCustomProviders, resolveCustomValue } from "../src/core/custom-providers.ts";
import { createCustomProvider, customProviderApis } from "../src/agent/custom-providers.ts";
const zero = { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 };
function model(id = "chat", extra = {}) {
return { id, contextWindow: 8192, maxTokens: 1024, cost: zero, ...extra };
}
function config(extra = {}, models = [model()]) {
return { providers: { gateway: { baseUrl: "http://example.invalid/custom/v1", api: "openai-completions", apiKey: "dummy", models, ...extra } } };
}
function parse(value: unknown) { return parseCustomProviders(JSON.stringify(value), new Set(["openai", "chatgpt"]), customProviderApis); }
const temp = fs.mkdtempSync(path.join(os.tmpdir(), "lh-custom-providers-"));
process.env.LH_DATA_DIR = temp;
process.env.LH_AUTH_FILE = path.join(temp, "auth.json");
process.env.LH_IGNORE_ENV_KEYS = "1";
fs.writeFileSync(process.env.LH_AUTH_FILE, JSON.stringify({ GW_KEY: "first-key", GW_HEADER: "first-header" }));
const { CUSTOM_MODELS_FILE, SETTINGS_FILE } = await import("../config.ts");
const { modelCatalog, validateModelsSettings, clampModelsLevels, catalogContextLimits } = await import("../src/server/model-catalog.ts");
const { piModels, piEngineForSpec } = await import("../src/agent/models.ts");
const { providerKeyPresent, keyValue } = await import("../src/server/keys.ts");
const { authStore } = await import("../src/store/auth.ts");
const { CustomProviderSource, installCustomProviderSource } = await import("../src/server/custom-providers.ts");
const { callCost } = await import("../src/server/pricing.ts");
after(() => fs.rmSync(temp, { recursive: true, force: true }));
function save(value: unknown) { fs.writeFileSync(CUSTOM_MODELS_FILE, JSON.stringify(value)); }
function sseText(text = "ok") {
const chunks = [
{ choices: [{ index: 0, delta: { role: "assistant", content: text }, finish_reason: null }] },
{ choices: [{ index: 0, delta: {}, finish_reason: "stop" }], usage: { prompt_tokens: 10, completion_tokens: 2, total_tokens: 12 } },
];
return new Response(chunks.map((chunk) => `data: ${JSON.stringify({ id: "test", object: "chat.completion.chunk", created: 1, model: "chat", ...chunk })}\n\n`).join("") + "data: [DONE]\n\n", { headers: { "content-type": "text/event-stream" } });
}
describe("declarative custom-provider configuration", () => {
it("uses Pi metadata and provider/model inheritance without rewriting base URLs", () => {
const [parsed] = parse(config({ compat: { supportsStore: false, chatTemplateKwargs: { a: true } } }, [model("named", {
name: "Named model", reasoning: true, input: ["text", "image"], thinkingLevelMap: { high: "high", xhigh: null },
compat: { maxTokensField: "max_tokens", chatTemplateKwargs: { b: false } },
samplingParams: { temperature: 0.5 }, samplingParamsByThinkingLevel: { high: { top_p: 0.9 } },
})]));
assert.equal(parsed!.models[0]!.baseUrl, "http://example.invalid/custom/v1");
assert.equal(parsed!.models[0]!.provider, "gateway");
assert.equal(parsed!.models[0]!.name, "Named model");
assert.deepEqual(parsed!.models[0]!.input, ["text", "image"]);
assert.deepEqual(parsed!.models[0]!.compat, { supportsStore: false, maxTokensField: "max_tokens", chatTemplateKwargs: { a: true, b: false } });
assert.deepEqual(parsed!.models[0]!.samplingParamsByThinkingLevel, { high: { top_p: 0.9 } });
});
it("accepts Pi line comments, trailing commas and BOM without corrupting strings", () => {
const text = `\uFEFF${JSON.stringify(config()).replace('"providers":', '// comment\n"providers":').replace(/}$/, ',}')}`;
assert.equal(parseCustomProviders(text, new Set(), customProviderApis)[0]!.models[0]!.baseUrl, "http://example.invalid/custom/v1");
});
it("registers multiple providers and mixed native APIs without fixed backend IDs", () => {
const value = config();
const raw = { providers: { ...value.providers, other: { ...value.providers.gateway, models: [model("chat"), model("anthropic", { api: "anthropic-messages", baseUrl: "https://example.invalid/anthropic" })] } } };
const models = createModels();
for (const entry of parse(raw)) models.setProvider(createCustomProvider(entry, () => undefined));
assert.equal(models.getProviders().length, 2);
assert.equal(models.getModel("other", "anthropic")!.api, "anthropic-messages");
assert.equal(models.getModel("other", "anthropic")!.baseUrl, "https://example.invalid/anthropic");
});
it("requires explicit finite limits and nonnegative costs, instead of guessing", () => {
for (const field of ["contextWindow", "maxTokens", "cost"]) {
const m: Record<string, unknown> = model(); delete m[field];
assert.throws(() => parse(config({}, [m as ReturnType<typeof model>])));
}
assert.throws(() => parse(config({}, [model("chat", { contextWindow: -1 })])));
assert.throws(() => parse(config({}, [model("chat", { cost: { ...zero, output: -1 } })])));
assert.throws(() => parseCustomProviders(JSON.stringify(config()).replace('"contextWindow":8192', '"contextWindow":1e999'), new Set(), customProviderApis));
});
it("rejects collisions, duplicate IDs, unsupported protocols, executable fields and invalid URLs", () => {
for (const id of ["openai", "chatgpt", "", "has:colon"]) assert.throws(() => parse({ providers: { [id]: config().providers.gateway } }));
assert.throws(() => parse(config({}, [model(), model()])));
assert.throws(() => parse(config({ api: "unknown-api" })));
assert.throws(() => parse(config({ oauth: "radius" })));
assert.throws(() => parse(config({ modelOverrides: {} })));
assert.throws(() => parse(config({ apiKey: "!touch /never-execute" })));
assert.throws(() => parse(config({ headers: { "x-key": "!secret-command" } })));
assert.throws(() => parse(config({}, [model("chat", { headers: { Authorization: "!command" } })])));
for (const baseUrl of ["file:///tmp/model", "https://secret@example.invalid", "http://example.invalid/#fragment"]) assert.throws(() => parse(config({ baseUrl })));
});
it("matches Pi value interpolation and escapes, without recursive expansion or commands", () => {
const lookup = (key: string) => ({ KEY: "secret", DATA: "!never-executed" })[key as "KEY"];
assert.equal(resolveCustomValue("Bearer ${KEY}/$KEY", lookup), "Bearer secret/secret");
assert.equal(resolveCustomValue("$$KEY $!literal", lookup), "$KEY !literal");
assert.equal(resolveCustomValue("$MISSING", lookup), undefined);
assert.equal(resolveCustomValue("$DATA", lookup), "!never-executed");
});
});
describe("custom providers in the existing application wiring", () => {
it("is empty before configuration and never creates a configuration file", () => {
assert.equal(fs.existsSync(CUSTOM_MODELS_FILE), false);
assert.equal(modelCatalog().customProvidersError, undefined);
assert.equal(fs.existsSync(CUSTOM_MODELS_FILE), false);
});
it("projects into existing pickers, validation, context limits and Pi pricing", async () => {
save(config({ apiKey: "$GW_KEY", headers: { "x-gateway": "$GW_HEADER" } }, [model("chat", { cost: { ...zero, input: 2, output: 4 } })]));
const catalog = modelCatalog();
assert.ok(catalog.providers.find((p) => p.id === "gateway")?.keyPresent);
assert.ok(catalog.models.some((m) => m.provider === "gateway" && m.id === "chat"));
assert.equal(JSON.stringify(catalog).includes("first-key"), false);
assert.equal(JSON.stringify(catalog).includes("first-header"), false);
assert.equal(validateModelsSettings({ strong: { model: { provider: "gateway", id: "chat" } } }).ok, true);
assert.equal(catalogContextLimits().get("gateway:chat"), 8192);
assert.ok(Math.abs(callCost({ inputTokens: 100, outputTokens: 50 }, "gateway:chat")! - 0.0004) < 1e-12);
assert.equal(providerKeyPresent("gateway"), true);
assert.equal((await piModels().getAuth("gateway"))!.auth.apiKey, "first-key");
});
it("uses current app key precedence and marks unresolved credentials/headers unavailable", () => {
process.env.LH_IGNORE_ENV_KEYS = "0";
process.env.GW_KEY = "real-env-key";
try { assert.equal(keyValue("GW_KEY"), "real-env-key"); }
finally { process.env.LH_IGNORE_ENV_KEYS = "1"; delete process.env.GW_KEY; }
save(config({ apiKey: "$MISSING_KEY" }));
assert.equal(providerKeyPresent("gateway"), false);
assert.equal(modelCatalog().models.some((m) => m.provider === "gateway"), false);
save(config({ headers: { "x-required": "$MISSING_HEADER" } }));
assert.equal(providerKeyPresent("gateway"), false);
save(config({ apiKey: undefined }));
assert.equal(providerKeyPresent("gateway"), false, "Pi-style keyless endpoints need an explicit dummy key");
});
it("revalidates every request of one engine while leaving an in-flight HTTP operation alone", async () => {
save(config({ apiKey: "$GW_KEY", headers: { "x-gateway": "$GW_HEADER" }, compat: { supportsStore: false, supportsDeveloperRole: false, supportsReasoningEffort: false, maxTokensField: "max_tokens" } }));
const engine = piEngineForSpec({ provider: "gateway", id: "chat" });
const context = normalizeContext({ messages: [{ role: "user", content: "hello", timestamp: 1 }] });
let started!: () => void, release!: (response: Response) => void;
const requestStarted = new Promise<void>((resolve) => { started = resolve; });
const firstResponse = new Promise<Response>((resolve) => { release = resolve; });
const requests: { url: string; headers: Headers; body: Record<string, unknown> }[] = [];
const original = globalThis.fetch;
globalThis.fetch = async (input, init) => {
const request = input instanceof Request ? input : new Request(input, init);
requests.push({ url: request.url, headers: request.headers, body: await request.json() as Record<string, unknown> });
if (requests.length === 1) { started(); return firstResponse; }
return sseText("héllo 🌍");
};
const pending = engine.streamFn(engine.model, context);
try {
await requestStarted;
authStore.setAuthVar("GW_KEY", "second-key");
authStore.setAuthVar("GW_HEADER", "second-header");
save(config({ baseUrl: "https://new.example.invalid/inference/v1", apiKey: "$GW_KEY", headers: { "x-gateway": "$GW_HEADER" }, compat: { supportsStore: false, maxTokensField: "max_tokens" } }, [model("chat", { maxTokens: 512 })]));
const second = await engine.streamFn(engine.model, context).result();
assert.equal(second.stopReason, "stop");
assert.deepEqual(second.content, [{ type: "text", text: "héllo 🌍" }]);
assert.equal(second.usage.input, 10);
assert.equal(second.usage.output, 2);
// Credential-only edits must also take effect on this same engine.
authStore.setAuthVar("GW_KEY", "third-key");
assert.equal((await engine.streamFn(engine.model, context).result()).stopReason, "stop");
// None of these cases may dispatch using the engine's old model/key.
save(config({ apiKey: "$MISSING_KEY" }));
assert.equal((await engine.streamFn(engine.model, context).result()).stopReason, "error");
save(config({}, [model("other-model")]));
assert.equal((await engine.streamFn(engine.model, context).result()).stopReason, "error");
fs.writeFileSync(CUSTOM_MODELS_FILE, "invalid configuration");
assert.equal((await engine.streamFn(engine.model, context).result()).stopReason, "error");
fs.rmSync(CUSTOM_MODELS_FILE);
assert.equal((await engine.streamFn(engine.model, context).result()).stopReason, "error");
release(sseText("original request finished"));
const first = await pending.result();
assert.equal(first.stopReason, "stop");
assert.deepEqual(first.content, [{ type: "text", text: "original request finished" }]);
} finally {
release(sseText());
await pending.result();
globalThis.fetch = original;
}
assert.equal(requests.length, 3, "unavailable configurations must not cause additional transport calls");
assert.equal(requests[0]!.url, "http://example.invalid/custom/v1/chat/completions");
assert.equal(requests[0]!.headers.get("authorization"), "Bearer first-key");
assert.equal(requests[0]!.headers.get("x-gateway"), "first-header");
assert.equal(requests[1]!.url, "https://new.example.invalid/inference/v1/chat/completions");
assert.equal(requests[1]!.headers.get("authorization"), "Bearer second-key");
assert.equal(requests[1]!.headers.get("x-gateway"), "second-header");
assert.equal(requests[2]!.headers.get("authorization"), "Bearer third-key");
assert.equal(requests[0]!.body.model, "chat");
assert.equal("store" in requests[0]!.body, false);
assert.equal(requests[0]!.body.max_tokens, 1024);
assert.equal(requests[1]!.body.max_tokens, 512, "current Pi model metadata, not the stale engine model, controls the request");
});
it("disables custom routes on malformed edits without removing built-ins or saved choices", () => {
fs.writeFileSync(SETTINGS_FILE, JSON.stringify({ version: 1, models: { strong: { model: { provider: "gateway", id: "chat" }, thinkingLevel: "high" } } }));
const before = fs.readFileSync(SETTINGS_FILE, "utf8");
fs.writeFileSync(CUSTOM_MODELS_FILE, '{ secret-token-do-not-log');
const catalog = modelCatalog();
assert.ok(catalog.customProvidersError);
assert.equal(catalog.customProvidersError!.includes("secret-token"), false);
assert.equal(piModels().getProvider("gateway"), undefined);
assert.ok(piModels().getProvider("openai"));
assert.deepEqual(catalog.slots.strong!.model, { provider: "gateway", id: "chat" });
const stored = { strong: { model: { provider: "gateway", id: "chat" }, thinkingLevel: "high" as const } };
const repaired = { strong: { ...stored.strong, thinkingLevel: "max" as const }, cheap: { model: { provider: "openai", id: piModels().getModels("openai")[0]!.id } } };
assert.equal(validateModelsSettings(repaired, stored).ok, true, "an unavailable custom pin must not prevent independent slot repair");
assert.equal(clampModelsLevels(repaired).strong!.thinkingLevel, "max", "preserve thinking while custom metadata is unavailable");
assert.equal(validateModelsSettings({ cheap: stored.strong }, stored).ok, false, "unchanged identity preservation is same-slot only");
assert.equal(fs.readFileSync(SETTINGS_FILE, "utf8"), before);
assert.throws(() => piEngineForSpec({ provider: "gateway", id: "chat" }), /catalog has no model/);
save(config());
assert.equal(modelCatalog().customProvidersError, undefined);
assert.ok(piModels().getProvider("gateway"));
fs.rmSync(CUSTOM_MODELS_FILE);
assert.equal(modelCatalog().models.some((m) => m.provider === "gateway"), false);
assert.equal(modelCatalog().customProvidersError, undefined);
assert.equal(fs.readFileSync(SETTINGS_FILE, "utf8"), before);
});
it("refreshes credentials without a file edit, preserving native header precedence and snapshots", async () => {
const values = new Map([["KEY", "one"], ["HEADER", "first"]]);
const source = new CustomProviderSource(() => ({ text: JSON.stringify(config({ apiKey: "$KEY", authHeader: true, headers: { "x-shared": "$HEADER" } }, [model("chat", { headers: { "x-shared": "model-${HEADER}" } })])) }), (name) => values.get(name));
const models = createModels();
source.sync(models);
const captured = createModels();
captured.setProvider(models.getProvider("gateway")!);
values.set("KEY", "two"); values.set("HEADER", "second");
source.sync(models);
const current = (await models.getAuth(models.getModel("gateway", "chat")!))!.auth;
const previous = (await captured.getAuth(captured.getModel("gateway", "chat")!))!.auth;
assert.equal(current.apiKey, "two");
assert.equal(current.headers!["x-shared"], "model-second");
assert.equal(current.headers!.Authorization, "Bearer two");
assert.equal(previous.apiKey, "one");
assert.equal(previous.headers!["x-shared"], "model-first");
});
it("rejects built-in and subscription-route collisions without replacing their providers", () => {
for (const id of ["openai", "github-copilot", "chatgpt", "github-copilot-subscription", "xai-subscription"]) {
const existing = piModels().getProvider("openai")!;
const models = createModels(); models.setProvider(existing);
const source = new CustomProviderSource(() => ({ text: JSON.stringify({ providers: { [id]: config().providers.gateway } }) }), () => undefined);
source.sync(models);
assert.ok(source.getError());
assert.equal(models.getProvider("openai"), existing);
assert.equal(models.getProviders().length, 1);
}
});
it("fails closed across partial publication and recovers without leaking internal errors", () => {
const text = JSON.stringify({ providers: { ...config().providers, second: config().providers.gateway } });
const source = new CustomProviderSource(() => ({ text }), () => undefined);
const models = createModels();
const builtin = piModels().getProvider("openai")!;
models.setProvider(builtin);
const publish = models.setProvider.bind(models);
models.setProvider = (provider) => {
if (provider.id === "second") throw new Error("private-value-do-not-log");
publish(provider);
};
source.sync(models);
assert.equal(models.getProvider("gateway"), undefined);
assert.equal(models.getProvider("second"), undefined);
assert.equal(models.getProvider("openai"), builtin);
assert.equal(source.getError()!.includes("private-value"), false);
models.setProvider = publish;
source.sync(models);
assert.ok(models.getProvider("gateway"));
assert.ok(models.getProvider("second"));
assert.equal(source.getError(), undefined);
});
it("retires old custom routes when server configuration bootstrap is reinstalled", () => {
save(config());
assert.ok(piModels().getProvider("gateway"));
fs.rmSync(CUSTOM_MODELS_FILE);
installCustomProviderSource(keyValue);
assert.equal(piModels().getProvider("gateway"), undefined);
assert.ok(piModels().getProvider("openai"));
});
it("treats unreadable config as unavailable and recovers without a restart", () => {
let unreadable = true;
const source = new CustomProviderSource(() => unreadable ? { error: "Unable to read models.json." } : { text: JSON.stringify(config()) }, () => undefined);
const models = createModels();
source.sync(models);
assert.equal(source.getError(), "Unable to read models.json.");
unreadable = false;
source.sync(models);
assert.ok(models.getProvider("gateway"));
assert.equal(source.getError(), undefined);
});
});