Captured at (local ISO): 2026-05-18 05:17:31

0. Preface

In AI app development, the classic combo is RAG (retrieval-augmented generation). This post walks through deploying two core models:

1. GTE-Chinese-Large: A top-tier semantic embedding model—helps AI “understand” documents.
2. SeqGPT-560m: An ultra-light generative model—for summarization and dialogue.

Requirements: PyTorch 2.x, Python 3.10+, GPU recommended (CPU works).

1. Environment: Clear Dependency Landmines

ModelScope NLP models need many deps. Install once to avoid whack-a-mole errors:

# Enter your venv (e.g. torch29)
pip install modelscope -U
# Key pin: datasets < 3.0, stable transformers
pip install "datasets<3.0.0" "transformers>=4.40.0" -U
# Hidden ModelScope deps
pip install simplejson sortedcontainers oss2 addict pyyaml accelerate sentencepiece

2. Fast Weights: aria2 Multi-Thread Download

Weights are hundreds of MB to GB; default downloads are slow. Use aria2 with 16 threads to cut ~30 minutes to ~1 minute.

2.1 Install aria2

sudo apt-get update && sudo apt-get install aria2 -y

2.2 Download GTE embedding model (~620 MB)

mkdir -p /root/.cache/modelscope/hub/models/iic/nlp_gte_sentence-embedding_chinese-large
cd /root/.cache/modelscope/hub/models/iic/nlp_gte_sentence-embedding_chinese-large

aria2c -s 16 -x 16 -k 1M -o pytorch_model.bin "https://www.modelscope.cn/api/v1/models/iic/nlp_gte_sentence-embedding_chinese-large/repo?Revision=master&FilePath=pytorch_model.bin"

2.3 Download SeqGPT generative model (~1.04 GB)

mkdir -p /root/.cache/modelscope/hub/models/iic/nlp_seqgpt-560m
cd /root/.cache/modelscope/hub/models/iic/nlp_seqgpt-560m

aria2c -s 16 -x 16 -k 1M -o pytorch_model.bin "https://www.modelscope.cn/api/v1/models/iic/nlp_seqgpt-560m/repo?Revision=master&FilePath=pytorch_model.bin"

Note: After download, return to your project dir: cd ~/workspace/your_project.

3. Deployment: Avoid the is_decoder Error

ModelScope + newer Transformers often clash (is_decoder). Prefer native Transformers loading.

3.1 Deploy GTE semantic search

Full code

Create search.py for fuzzy semantic search—e.g. query “not enough VRAM” can match “upgrade hardware”.

import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel

model_path = "/root/.cache/modelscope/hub/models/iic/nlp_gte_sentence-embedding_chinese-large"

tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path)

def get_emb(text):
    inputs = tokenizer(text, padding=True, truncation=True, return_tensors='pt', max_length=512)
    with torch.no_grad():
        out = model(**inputs)
        return F.normalize(out.last_hidden_state[:, 0], p=2, dim=1)

# --- Mock knowledge base ---
knowledge_base = [
    "1. Upgrading server VRAM to 24GB can run larger models.",
    "2. Beijing tomorrow: sunny to cloudy; bring a light jacket.",
    "3. Fish-flavored shredded pork is a classic Sichuan dish with pork and wood ear.",
    "4. Python is widely used in AI.",
    "5. aria2c can greatly speed up large file downloads."
]

# Embed all entries
kb_embs = get_emb(knowledge_base)

def smart_search(user_query):
    query_emb = get_emb([user_query])
    scores = (query_emb @ kb_embs.T) * 100
    
    print(f"\n🔍 Query: 【{user_query}】")
    print("-" * 50)
    
    results = sorted(zip(scores[0].tolist(), knowledge_base), reverse=True)
    for score, text in results:
        star = "⭐" * int(score/20)
        print(f"[{score:.1f}%] {star} | {text}")

smart_search("I want to learn AI, which language?")
smart_search("VRAM is too small, what to do?")
smart_search("Do I need an umbrella tomorrow?")

Demo

3.2 Deploy SeqGPT copy assistant

For a 560M model, prompt format matters.

Create gen.py:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_path = "/root/.cache/modelscope/hub/models/iic/nlp_seqgpt-560m"
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True).to('cpu')

def ask_ai(task, content):
    prompt = f"任务：{task}\n输入：{content}\n输出："
    inputs = tokenizer(prompt, return_tensors="pt")
    
    with torch.no_grad():
        outputs = model.generate(
            **inputs, 
            max_new_tokens=64, 
            do_sample=True, 
            temperature=0.8,
            repetition_penalty=1.2
        )
    
    full_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return full_text.split("输出：")[-1].strip()

print(f"AI reply: {ask_ai('给这段话写一个标题', '今天我们成功在服务器上部署了两个AI模型')}")

4. Common pitfalls

1. ModuleNotFoundError:
   • Missing simplejson or sortedcontainers? pip install them.
   • cannot import name 'ALL_ALLOWED_EXTENSIONS'? datasets too new—pip install "datasets<3.0.0".
2. Download stalls:
   • Server throttling—switch to aria2c or export MODELSCOPE_PARALLEL_DOWNLOAD_COUNT=16.
3. Empty or repetitive output:
   • Set max_new_tokens.
   • Match SeqGPT prompt pattern (e.g. ending with “输出：”).

5. Closing

You now have semantic search and text generation—two LEGO bricks for a private enterprise knowledge-base assistant.

Logged 2026-01-25, PyTorch 2.9, ModelScope 1.20+.