host models -The Simplest Intro (SGLang and vLLM)

The Simplest Intro to Hosting Your Own Model

If you download model weights, you still need a serving engine that loads the model and exposes an HTTP API. The most beginner friendly choices are vLLM and SGLang. Both can present an OpenAI compatible endpoint so your app or curl can call them like OpenAI. You can get models from Hugging Face, ModelScope, or the official vendor portal for models like Llama. ModelScope is a popular hub similar to Hugging Face. Meta lets you request and download Llama weights from their site. 

Important note about file formats. vLLM and SGLang expect Transformers format weights such as safetensors. GGUF files are for llama.cpp and Ollama and do not load in vLLM or SGLang.

About endpoints. vLLM and SGLang both provide an OpenAI compatible HTTP server. If you need a different API shape, you can write a small wrapper that uses their native Python APIs. In practice you normally run one API style per server process. 

Host models locally on a MacBook or PC with Ollama

Llama and API Compatibility

Llama is just a model and its weights. It has no built in HTTP API and it is not inherently OpenAI compatible. Compatibility comes from the serving framework or cloud service you choose.

Meta hosted option

• Meta’s Llama API provides its own native API and also works with OpenAI style client libraries.

Self hosting compatibility

• vLLM ships an OpenAI compatible HTTP server. If you want a different API shape, wrap its Python engine with your own small service. (It does not include Meta’s native Llama API.)

• SGLang offers both an OpenAI compatible endpoint and a native SGLang program model. These can coexist in one deployment. (It is not Meta Llama’s native API.)

• Other choices include Hugging Face TGI (native HTTP and gRPC with an optional OpenAI compatible layer) and Ollama (its own REST API with some OpenAI style routes).

Can both endpoint styles run at the same time

• It depends on the framework. SGLang and TGI can serve native and OpenAI compatible endpoints together.

• vLLM usually exposes only the OpenAI compatible server. To have both, run a second process on another port or add a lightweight adapter service around its Python API.

Takeaway

• Llama itself is not OpenAI compatible. Meta’s hosted API supports both styles. With self hosting, tools like vLLM, SGLang, TGI and Ollama can give you OpenAI compatible endpoints, and some also offer their own native routes.

Example: serve Llama with vLLM or SGLang

# vLLM install and serve
pip install vllm
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --port 8000

# Call it with the OpenAI style Chat Completions route
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages":[{"role":"user","content":"write a poem"}],
    "temperature": 0.7
  }'

# SGLang install and serve
pip install sglang
sglang serve \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --port 30000 \
  --tp 1 \
  --trust-remote-code

# Call the SGLang server with the same route
curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages":[{"role":"user","content":"tell me a joke"}]
  }'

vLLM documents its OpenAI compatible server under the serve and api_server entrypoints. SGLang documents an OpenAI compatible server as well, and it also offers a native programming model for more complex templates. 

Summary and common gotchas

• Use Instruct variants for chat. Non instruct weights often need a manual chat template.

• Make sure the tokenizer matches the weights or loading will fail or text will look wrong.

• Transformers format for vLLM and SGLang. GGUF is for llama.cpp or Ollama.

• If memory is tight, lower max model length or use a supported quantization for your server.

• Multi GPU needs tensor parallel flags in each server and a fast interconnect.

• For production, add authentication and place a reverse proxy in front of the port.

• If you really need non OpenAI routes, build a small FastAPI or Flask wrapper that calls the vLLM or SGLang Python API. Keep it as a separate process if you want it to coexist with the OpenAI compatible server.

 

Open AI GPT-OSS

GPT-OSS is OpenAI’s first open-weight model family since GPT-2. It comes in two text-only variants, gpt-oss-120b and gpt-oss-20b, released under an Apache-2.0 style license for local deployment, fine-tuning, and commercial use. Both target strong reasoning and “agentic” workflows (tool use, browsing, function calling) with adjustable reasoning effort. 

What’s special about it (at a glance)

• Open weights + permissive use: downloadable checkpoints on Hugging Face; supported across popular runtimes (vLLM, Ollama, Fireworks, etc.).

• Runs on modest hardware: the 120B model uses Mixture-of-Experts (MoE) with ~5.1B “active” params per token and MXFP4 quantization so it fits on a single 80 GB GPU; the 20B model targets 16 GB devices.

• Reasoning first: competitive math/coding and tool-use results and smooth “test-time scaling” as you dial up chain-of-thought length.

• Safety work disclosed: extensive preparedness testing (cyber/biological) and jailbreak studies are summarized in the model card.

Comparison

Model	SWE-bench Verified (%)	AIME 2025 pass@1
GPT-OSS 20B (high)	60.7	98.7% (with tools)
GPT-OSS 120B (high)	62.4	97.9% (with tools)
GPT-4.1 (main)	54.6	—
OpenAI o3 (high)	69.1	98.4% (with tools)
OpenAI o4-mini (high)	68.1	99.5% (with tools)
GPT-5 (low)	—	—
GPT-5 (medium)	—	—
GPT-5 (high)	74.9	94.6% (no tools)

REF: 

https://openai.com/index/introducing-gpt-oss/

open weight

An open weight model is a type of AI model, particularly a LLM, where the final trained parameters, known as "weights," are made publicly available. These weights are the numerical values that have been learned by the model during its training on vast amounts of data and essentially represent the model's knowledge.

By releasing the weights, developers and researchers can download, examine, and use the pre-trained model to build their own applications, conduct further research, or fine-tune it for specific tasks.

Related Links

https://shiluqi.blogspot.com/2025/09/gpt-5.html

https://shiluqi.blogspot.com/2025/11/overview-of-gpt-51.html

https://shiluqi.blogspot.com/2025/10/open-ai-response-api.html

https://shiluqi.blogspot.com/2025/11/gemini-3-pro-in-context-how-it-compares.html