Here are a few things I wish I knew before I started.

1. "OpenAI-compatible" means different things to different providers

Most providers accept the same request shape. But behavior differs significantly:

• System prompts — some providers treat them as strict instructions, others as suggestions.

• Structured output — some enforce JSON Schema strictly, others silently fall back to plain text.

• Streaming — chunk formats vary. Finish reasons differ. Error handling is inconsistent.

The takeaway: "compatible" is a spectrum, not a binary.

2. Routing is not "choose a model"

Real users don't ask for "many models." They ask for a model that fits a constraint:

• Lower latency

• Better reasoning

• Lower cost

• Longer context

• Better at code

So routing becomes a policy engine, not a switch statement. You need to know what each model can do, how much it costs, how it behaves under load, and which tasks it's appropriate for.

3. Latency variance matters more than average speed

A model that usually responds in 1s but sometimes takes 12s feels unreliable, even if the average looks fine.

We started tracking p95 and p99 latency for every provider. That changed how we think about fallback and failover.

4. Streaming is where the abstraction breaks

Normal request-response is easy to normalize. Streaming is where you see the real differences:

• Delta semantics

• Finish reasons

• Heartbeat behavior

• Partial failures

• Connection endings

Building a stream normalization layer was the most humbling part of the project.

5. Usage accounting is closer to distributed state than billing

In a multi-model gateway, usage is part of runtime truth. You need to track:

• Received → admitted → forwarded → partially streamed → completed → failed → compensated

Not a single "token in / token out" record.

What I'm building

ModelBridge — one OpenAI-compatible API, multiple models, pay-as-you-go.

No vendor lock-in. No complex contracts. Just build.

https://zhuocore.com

What's next

I'm continuing to improve routing intelligence, cost optimization, and reliability. If you're building in this space too, I'd love to hear what challenges you've run into.