The short version
Hosted API models are the right default for most small and mid-sized companies. They are the most capable models available for agent work, they cost nothing when idle, and they require no infrastructure. OpenAI and Anthropic both offer business terms under which API data is not used for training.
Open-weight models earn their place in specific situations: strict data residency or air-gapped environments, very high-volume narrow tasks, deep fine-tuning, and offline or edge deployments. They are a real option, not a fallback, but they come with an operations job attached.
Where open-weight models win
Data that cannot leave. Some defense, healthcare, legal and financial environments do not allow data to be sent to any outside service. An open-weight model running inside your own cloud account or data center keeps every prompt and response there.
High-volume narrow tasks. If one step, such as tagging support tickets or extracting fields from a standard form, runs millions of times a month, a small fine-tuned model on dedicated hardware can be cheaper per call than an API model. The math only works when the hardware stays busy.
Control and stability. You choose when to upgrade. A model you host does not get retired or change behavior unless you change it, which helps in validated or audited processes.
Fine-tuning freedom. With the weights in hand, you can fine-tune for your domain, vocabulary or output format without provider limits. See RAG vs fine-tuning before assuming you need it, though.
Where API models win
Capability on hard work. For agents that plan, call tools, recover from errors and handle requests nobody anticipated, the leading API models are typically ahead. In our experience the gap is smallest on narrow, well-defined tasks and largest on open-ended ones.
No infrastructure. No GPUs to provision, no inference server to tune, no capacity planning for peak hours. That keeps a pilot to weeks rather than months. Our guide to connecting an LLM to internal data safely covers the controls that matter more than where the model runs.
Cost at normal volume. Per-token billing means a system used a few hundred times a day costs little. A dedicated GPU server costs the same whether it handles ten requests or ten thousand.
Speed of improvement. Providers ship stronger models regularly, and switching to a newer model is often a one-line change plus a test run.
What we recommend
Start with API models from OpenAI or Anthropic unless you have a hard data residency requirement. Build with the model behind a thin interface and keep a set of real test cases from day one.
Once the system is in production, look at your traffic. If one step is high-volume, narrow and stable, test an open-weight model on it against your cases. Move it only if accuracy holds and the cost difference covers the added operations work. For running-cost benchmarks, see what it costs to run an AI agent each month.
If your data cannot leave your environment, plan for open-weight models from the start, and budget engineering time for serving, monitoring and updates as part of the project rather than as an afterthought.