Captured at (local ISO): 2026-05-18 05:17:26

1. Introduction

GLM-4V-9B from Zhipu AI is a recent open vision-language model emphasizing image understanding + dialogue + reasoning. Self-hosting keeps data on-prem and slashes long‑term API invoices versus cloud SKUs.

This guide targets Linux boxes that already satisfy PyTorch + CUDA prerequisites and demonstrates end-to-end local inference.

2. Environment setup

Minimal checklist:

• OS: Ubuntu 20.04+ (recommended baseline)
• VRAM:
  • FP16 / BF16 path: aim for ≥ 24 GB (3090/4090, A10, A100 class)
  • Int4 quantized path: roughly ≥12 GB (3060/4070 tiers)
• Toolchain: Python 3.10+, CUDA 11.8+, PyTorch 2.x

2.1 Install dependencies

Assuming Torch is wired, pull media handling + tokenizer stacks:

pip install transformers>=4.45.0 accelerate tiktoken einops scipy pillow
# 强烈建议安装 flash-attn 以获得更快的推理速度（需支持 CUDA 11.6+）
pip install flash-attn --no-build-isolation

3. Download weights

Approx 18 GB on disk — ModelScope mirrors often saturate faster domestically versus raw Hugging Face pulls.

pip install modelscope
modelscope download --model ZhipuAI/glm-4v-9b --local_dir ./glm-4v-9b

Parallel downloader helper:

from modelscope import snapshot_download

model_id = 'ZhipuAI/glm-4v-9b'
local_dir = './glm-4v-9b'

# 增加 max_workers 参数（视你的服务器带宽而定，建议设置 4-8）
snapshot_download(
    model_id, 
    local_dir=local_dir, 
    cache_dir='./cache', # 临时缓存目录
    max_workers=16        # 开启8个线程同时下载
)

4. Core inference script

Save as inference.py adjacent to ./glm-4v-9b:

import os
import warnings
import torch

# 1. 屏蔽环境变量日志 (必须在 import transformers 之前执行)
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'  # 屏蔽 TensorFlow 日志
os.environ['TRANSFORMERS_VERBOSITY'] = 'error'  # 屏蔽 Transformers 自带的大部分警告
os.environ['HF_HUB_DISABLE_SYMLINKS_WARNING'] = '1' # 屏蔽 HF 软连接警告

# 2. 屏蔽 Python 警告
warnings.filterwarnings("ignore", category=FutureWarning)
warnings.filterwarnings("ignore", category=UserWarning)

# 3. 此时再导入剩下的库
from PIL import Image
import transformers
from transformers import AutoModelForCausalLM, AutoTokenizer

# 4. 屏蔽 transformers 内部库的输出
transformers.logging.set_verbosity_error()

def run_inference():
    model_path = "./glm-4v-9b"

    # 1. 加载分词器
    tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

    # 2. 加载模型
    print("[1/3] 正在加载模型至显存 (BF16)...")
    model = AutoModelForCausalLM.from_pretrained(
        model_path,
        trust_remote_code=True,
        torch_dtype=torch.bfloat16,
        device_map="auto"
    ).eval()

    # 兼容性小补丁
    if not hasattr(model.config, "num_hidden_layers"):
        model.config.num_hidden_layers = model.config.num_layers

    # 3. 准备输入
    image_path = "demo.jpg" 
    query = "请详细描述这张图片。"
    
    try:
        image = Image.open(image_path).convert("RGB")
    except Exception:
        print(f"错误：无法找到图片 {image_path}")
        return

    inputs = tokenizer.apply_chat_template(
        [{"role": "user", "image": image, "content": query}],
        add_generation_prompt=True,
        tokenize=True,
        return_tensors="pt",
        return_dict=True
    ).to("cuda")

    # 4. 执行推理
    print("[2/3] 模型正在思考...")
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=512,
            do_sample=False,
            use_cache=True,
            eos_token_id=tokenizer.eos_token_id
        )
        
        response_ids = outputs[0][len(inputs['input_ids'][0]):]
        response = tokenizer.decode(response_ids, skip_special_tokens=True)

    print("[3/3] 推理完成。")
    print("\n" + "="*30 + " 模型回答 " + "="*30)
    print(response)
    print("="*70 + "\n")

if __name__ == "__main__":
    run_inference()

5. Sample output

6. Advanced tuning

Plan A — VRAM constrained? Enable 4-bit

Sub‑20 GB boards benefit from BitsAndBytes 4‑bit quantization (~9–11 GB residency).

Install once: pip install bitsandbytes

Swap loader stanza:

model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    load_in_4bit=True,
    device_map="auto"
)

Plan B — High throughput APIs with vLLM

Serve OpenAI-compatible HTTP for frontends/backends needing batching throughput:

pip install vllm

python -m vllm.entrypoints.openai.api_server \
    --model ./glm-4v-9b \
    --trust-remote-code \
    --gpu-memory-utilization 0.9 \
    --max-model-len 4096 \
    --port 8000

7. FAQ troubleshooting

1. AttributeError: 'ChatGLMTokenizer' object has no attribute 'apply_chat_template'
   Bump transformers ≥ 4.44; confirm tokenizer_config.json landed beside weights.
2. CUDA OOM
   Shorten contexts, enable 4‑bit loader, reduce gpu_memory_utilization if using vLLM.
3. Weak visual answers
   Ensure Image.open(...).convert("RGB"); transparent / single‑channel PNGs degrade alignment.

8. Closing thoughts

GLM-4V-9B punches above its parameter class for multimodal workloads. Wired locally it slots into offline document QA, CV inspection benches, biomedical imaging tooling, etc. Ping the comment thread when you stall—community notes accumulate fast on fresh checkpoints.