AIVory Smart Inference

Smart Inference is an LLM inference router and GPU marketplace from AIVory. It's a drop-in replacement for the OpenAI API that automatically routes every request to the cheapest healthy provider serving the model you asked for — in real time. Swap one URL, keep your SDK, and stop overpaying for inference. The problem Inference providers cut their prices constantly. Anthropic discounts a model, DeepSeek ships a new tier, smaller hosts undercut everyone on specific open-weight models. Your bill, meanwhile, keeps reflecting whatever you signed up for six months ago. Manually shopping for the cheapest endpoint per request isn't realistic — by the time you've benchmarked five providers, the prices have moved again. Smart Inference solves this with continuous, automated price discovery. Every request gets routed to the lowest-cost provider currently serving the model, with sub-second latency overhead. The result is typically 10–40% lower bills, with savings up to 89% on open-weight models like Llama and Qwen where the price spread between providers is widest. How it works Change one line: base_url = "https://api.aivory.net/v1" That's the entire migration. Keep your OpenAI Python SDK, your TypeScript client, your prompts, your retry logic, your streaming handlers, your tool-calling schemas. The response shape is identical. Only the bill changes. Under the hood, Smart Inference maintains a live catalogue of inference providers — DeepInfra, Together, Groq, Anthropic, OpenAI, Google, Mistral, Fireworks, and others — with per-model pricing refreshed continuously. Each request is matched to the cheapest healthy provider for that model at that moment. If a provider degrades, fails, or runs out of capacity, the router fails over before you notice. Features 50+ models through one endpoint: GPT-4, GPT-4o, Claude (Haiku, Sonnet, Opus), Gemini, Llama 3.x, Mistral, DeepSeek, Qwen, GLM, Gemma, and more OpenAI-compatible API: works with any OpenAI-compatible SDK in any language Live spot pricing: real-time provider rates, refreshed continuously, sortable catalogue Savings dashboard: every request logged with the price you paid and the price OpenAI, Azure, AWS, GCP, or the direct provider would have charged Custom inference endpoints: named endpoints with specific routing strategies, latency limits, and fallback providers Streaming, function calling, tool use: all OpenAI features supported transparently Per-key rate limiting and scoped permissions: granular API key management for teams and environments Observability: request volume, cost breakdown by provider and model, burn rate, runway, hot endpoints, recent activity Spot GPUs in one click When routing to commercial APIs isn't the right answer — you need a fine-tuned model, you have data residency requirements, or your workload is large enough that self-hosting beats per-token pricing — Smart Inference also aggregates spot GPU capacity from RunPod, CoreWeave, Vast.ai, Lambda Labs, and other providers in the same dashboard. Browse 38+ GPU types from H200 and B300 down to RTX 4090 and A40. See live spot pricing across providers. Deploy a vLLM instance with any HuggingFace model in one click. Set a price ceiling so you never overpay. And — this is the part that matters — route to your rented GPU through the same OpenAI-compatible endpoint. Same client code. Different backend. Pricing Pay-as-you-go from $10. No subscription, no seat fees, no minimums. Credits never expire. Auto-recharge available below a configurable threshold. Optional monthly spending cap. Full transaction history and cost attribution by provider, model, and time period. Who it's for Indie developers and small teams already paying an OpenAI, Anthropic, or Google bill they want to shrink AI app builders who want to stay model-agnostic without writing their own routing layer ML engineers running open-weight models who want one dashboard for API routing and GPU rental Teams who want OpenAI-compatible tooling without being locked into one provider How it compares Smart Inference overlaps with OpenRouter, LiteLLM, and Portkey on LLM routing. The differences: integrated spot GPU marketplace (route to a model you rent through the same endpoint), explicit savings dashboard that compares your bill against direct providers, and integration with the broader AIVory platform — Guard for policy-as-code on prompts and outputs, Architect for visual multi-model pipelines. Get started Sign up at aivory.net/smart-inference. Add $10 in credits, create an API key, swap your base_url. First request in under five minutes.

Discussion (0)