WebInfer

Back to Providers
🦙

llama.cpp

High-performance local LLM inference with llama.cpp server

Free
Local & Self-Hosted
Local

Configure llama.cpp

Enable llama.cpp in Settings

Go to Settings
About llama.cpp

llama.cpp provides efficient CPU and GPU inference for GGUF models. It supports a wide range of quantization formats and is optimized for running large language models on consumer hardware. Features an OpenAI-compatible API.

Key Features
  • Efficient CPU and GPU inference
  • GGUF model format support
  • Multiple quantization levels (Q4, Q5, Q8, etc.)
  • OpenAI-compatible API
  • Low memory usage with quantization
  • Multi-GPU support
Capabilities
text-generation
chat
local
Configuration Schema
Required and optional fields for configuring this provider
FieldTypeRequired
baseUrl
url
Optional
CLI Usage
For users running the daemon server locally or a self-hosted gateway

Add provider:

npx webinfer provider add llama-cpp

Test connection:

npx webinfer provider test llama-cpp
Programmatic Usage
Use llama.cpp in your code with WebInfer

Basic usage:

import { generateText } from "webinfer"

// WebInfer automatically routes to the best available provider
const result = await generateText({
  prompt: "Write a haiku about programming"
})

console.log(result.text)

Specify llama.cpp explicitly:

import { generateText } from "webinfer"

const result = await generateText({
  prompt: "Write a haiku about programming",
  provider: "llama-cpp"
})

console.log(result.text)
console.log("Provider:", result.provider)
console.log("Model:", result.model)

Streaming:

import { streamText } from "webinfer"

const { textStream } = await streamText({
  prompt: "Write a story about AI",
  provider: "llama-cpp"
})

for await (const chunk of textStream) {
  process.stdout.write(chunk)
}
Resources
llama.cpp WebsiteConfigure in Settings