Captured at (local ISO): 2026-05-18 05:17:29

Foreword

NewBie-image-Exp0.1 is a 3.5B anime image generation model built on the Next‑DiT architecture. It supports XML‑structured prompts and excels at multi‑character control and attribute binding. Deploying NewBie-image-Exp0.1 is challenging because it combines several top-tier components (Gemma 3, Jina CLIP, Flux VAE), and the reference code has hard bugs in dimension and dtype when adapting to Diffusers-style inference.
Below is a deployment guide meant to help you dodge the pitfalls in one pass.

This tutorial fixes core bugs in the reference code—floating-point indices, shape mismatches, dtype conflicts—and gets you to stable generation.

1. Hardware requirements and environment

• GPU VRAM: 16GB+ recommended (model + encoders use ~14–15GB).
• OS: Linux (recommended) / Windows.
• Basics: Python 3.10+, PyTorch 2.4+, CUDA 12.1+.

Core dependencies

pip install transformers accelerate safetensors diffusers timm torchdiffeq gradio
# Uninstall xformers if it causes version conflicts
pip uninstall xformers -y
# Install project-provided Flash-Attention wheel (pick one matching your env)
pip install flash_attn-2.8.3+cu12torch2.8cxx11abiTRUE-cp311-cp311-linux_x86_64.whl

In any deployment write-up, a section on how to wget payloads and pip install locally matters a lot—especially on servers where GitHub is flaky or blocked.

Here is a section you can drop straight into “environment prep”:

Extra: downloads and offline install in restricted networks

On many cloud GPUs (AutoDL, vendor AI clouds, etc.), pip install git+... or direct GitHub downloads often time out or fail TLS. Prefer a “local hop” workflow.

1. wget specific artifacts

If a direct fetch fails, use a mirror prefix (e.g. gh-proxy.com) and --no-check-certificate to skip cert checks (when you accept the risk).

Flash-Attention wheel (example):

# Pattern: wget [mirror][original GitHub URL]
wget --no-check-certificate https://mirror.ghproxy.com/https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.3/flash_attn-2.8.3+cu12torch2.8cxx11abiTRUE-cp311-cp311-linux_x86_64.whl

Model source zip:

wget --no-check-certificate https://mirror.ghproxy.com/https://github.com/NewBieAI-Lab/diffusers/archive/refs/heads/add-newbie-pipeline.zip

2. Local pip install

After .whl or .zip lands on disk, install by path to avoid mid-install network jitter.

• Install the .whl:

  pip install flash_attn-2.8.3+cu12torch2.8cxx11abiTRUE-cp311-cp311-linux_x86_64.whl
  

• Install from the downloaded source zip:

  # 1. Unzip
  unzip add-newbie-pipeline.zip
  # 2. Enter extracted folder
  cd diffusers-add-newbie-pipeline
  # 3. Editable install
  pip install -e .
  

Tip: After local installs, pip cache purge can reclaim disk space—worth mentioning to readers.

2. Fetching code and weights

1. Clone the repo:

   git clone https://github.com/NewBieAI-Lab/NewBie-image-Exp0.1.git
   cd NewBie-image-Exp0.1
   

2. Download weights from Hugging Face NewBie-image-Exp0.1; ensure the tree includes transformer, text_encoder, vae, clip_model.

3. Core step: patch reference bugs (auto script)

The reference models/model.py has logic holes for Diffusers-style inference (floats as indices, tensor ranks not aligned, etc.).
Run this Python patcher:

import os

path = 'models/model.py'
with open(path, 'r', encoding='utf-8') as f:
    content = f.read()

# Fix 1: slices must use integers
content = content.replace(':max_cap', ':int(max_cap)')
content = content.replace('torch.zeros(bsz, max_seq_len', 'torch.zeros(bsz, int(max_seq_len)')
content = content.replace('[:max_seq_len]', '[:int(max_seq_len)]')

# Fix 2: align text vs time features before concat (2D vs 1D)
old_cat = 'combined_features = torch.cat([t_emb, clip_emb], dim=-1)'
new_cat = """
            if clip_emb.ndim == 1:
                clip_emb = clip_emb.unsqueeze(0)
            if clip_emb.shape[0] != t_emb.shape[0]:
                clip_emb = clip_emb.expand(t_emb.shape[0], -1)
            combined_features = torch.cat([t_emb, clip_emb], dim=-1)
"""
content = content.replace(old_cat, new_cat)

with open(path, 'w', encoding='utf-8') as f:
    f.write(content)
print("✅ Patched models/model.py")

4. Inference script run_inference.py

This script wires components manually and avoids a custom Diffusers fork.

import torch
import os
import sys
from PIL import Image
from safetensors.torch import load_file
from torchvision.transforms.functional import to_pil_image

# Local models + transport
sys.path.append(os.getcwd())

from models import NextDiT_3B_GQA_patch2_Adaln_Refiner_WHIT_CLIP
from transport import Sampler, create_transport
from diffusers.models import AutoencoderKL
from transformers import AutoModel, AutoTokenizer

# --- Config ---
model_root = "./NewBie-image-Exp0.1"  # weights root
device = "cuda"
dtype = torch.bfloat16

print("1. Loading text encoders (Gemma 3 & Jina CLIP)...")
tokenizer = AutoTokenizer.from_pretrained(f"{model_root}/text_encoder")
text_encoder = AutoModel.from_pretrained(f"{model_root}/text_encoder", torch_dtype=dtype).to(device).eval()

clip_tokenizer = AutoTokenizer.from_pretrained(f"{model_root}/clip_model", trust_remote_code=True)
clip_model = AutoModel.from_pretrained(f"{model_root}/clip_model", torch_dtype=dtype, trust_remote_code=True).to(device).eval()

print("2. Loading VAE...")
vae = AutoencoderKL.from_pretrained(f"{model_root}/vae").to(device, dtype)

print("3. Initializing Transformer...")
model = NextDiT_3B_GQA_patch2_Adaln_Refiner_WHIT_CLIP(
    in_channels=16, qk_norm=True,
    cap_feat_dim=text_encoder.config.text_config.hidden_size,
)
ckpt_path = f"{model_root}/transformer/diffusion_pytorch_model.safetensors"
model.load_state_dict(load_file(ckpt_path), strict=True)
model.to(device, dtype).eval()

sampler = Sampler(create_transport("Linear", "velocity"))
sample_fn = sampler.sample_ode(sampling_method="midpoint", num_steps=28, time_shifting_factor=6.0)

@torch.no_grad()
def generate(user_prompt):
    system_prompt = "You are an assistant designed to generate high-quality images based on user prompts."
    prompts = [system_prompt + user_prompt, " "]  # CFG batch = 2

    txt_in = tokenizer(prompts, return_tensors="pt", padding=True).to(device)
    p_embeds = text_encoder(**txt_in, output_hidden_states=True).hidden_states[-2]

    clip_in = clip_tokenizer(prompts, return_tensors="pt", padding=True, truncation=True).to(device)
    c_res = clip_model.get_text_features(input_ids=clip_in.input_ids, attention_mask=clip_in.attention_mask)
    c_pooled = c_res[0].to(dtype)
    if c_pooled.ndim == 1: c_pooled = c_pooled.unsqueeze(0)
    if c_pooled.shape[0] == 1: c_pooled = c_pooled.repeat(2, 1)

    model_kwargs = dict(cap_feats=p_embeds, cap_mask=txt_in.attention_mask, cfg_scale=4.5,
                        clip_text_sequence=c_res[1].to(dtype), clip_text_pooled=c_pooled)

    z = torch.randn([2, 16, 128, 128], device=device, dtype=dtype)

    # ODE may run in float32; cast back for module weights
    def robust_forward(x, t, **kwargs):
        return model.forward_with_cfg(x.to(dtype), t.to(dtype), **kwargs)

    samples = sample_fn(z, robust_forward, **model_kwargs)[-1]

    samples = vae.decode(samples[:1].to(dtype) / 0.3611 + 0.1159).sample
    img = to_pil_image(((samples[0] + 1.0) / 2.0).clamp(0.0, 1.0).float().cpu())
    return img

if __name__ == "__main__":
    prompt = "<character_1><n>miku</n><gender>1girl</gender><appearance>blue_hair, long_twintails</appearance></character_1><general_tags><style>anime_style</style></general_tags>"
    result = generate(prompt)
    result.save("success_output.png")
    print("✨ Done. Saved success_output.png")

Run

python run_inference.py

Sample output

5. Advanced: interactive create.py

import torch
import os
import sys
import time
import builtins
from PIL import Image
from safetensors.torch import load_file
from torchvision.transforms.functional import to_pil_image

# Monkey-patch float index bugs if you didn't patch model.py yet
_orig_zeros = torch.zeros
def _safe_zeros(*args, **kwargs):
    new_args = list(args)
    if len(args) > 0:
        if isinstance(args[0], (list, tuple)):
            new_args[0] = tuple(int(s) for s in args[0])
        else:
            for i in range(len(new_args)):
                if isinstance(new_args[i], (int, float)):
                    new_args[i] = int(new_args[i])
                elif isinstance(new_args[i], torch.Tensor) and new_args[i].ndim == 0:
                    new_args[i] = int(new_args[i].item())
                else: break
    return _orig_zeros(*args, **kwargs)
torch.zeros = _safe_zeros

sys.path.append(os.getcwd())

from models import NextDiT_3B_GQA_patch2_Adaln_Refiner_WHIT_CLIP
from transport import Sampler, create_transport
from diffusers.models import AutoencoderKL
from transformers import AutoModel, AutoTokenizer

model_root = "./NewBie-image-Exp0.1"
device = "cuda"
dtype = torch.bfloat16

def load_all_models():
    print("🚀 Loading model stack...")
    tokenizer = AutoTokenizer.from_pretrained(f"{model_root}/text_encoder")
    text_encoder = AutoModel.from_pretrained(f"{model_root}/text_encoder", torch_dtype=dtype).to(device).eval()
    clip_tokenizer = AutoTokenizer.from_pretrained(f"{model_root}/clip_model", trust_remote_code=True)
    clip_model = AutoModel.from_pretrained(f"{model_root}/clip_model", torch_dtype=dtype, trust_remote_code=True).to(device).eval()
    vae = AutoencoderKL.from_pretrained(f"{model_root}/vae").to(device, dtype)

    model = NextDiT_3B_GQA_patch2_Adaln_Refiner_WHIT_CLIP(
        in_channels=16, qk_norm=True,
        cap_feat_dim=text_encoder.config.text_config.hidden_size,
    )
    ckpt_path = f"{model_root}/transformer/diffusion_pytorch_model.safetensors"
    model.load_state_dict(load_file(ckpt_path), strict=True)
    model.to(device, dtype).eval()
    sampler = Sampler(create_transport("Linear", "velocity"))
    return tokenizer, text_encoder, clip_tokenizer, clip_model, vae, model, sampler

@torch.no_grad()
def encode_prompts(user_input, tokenizer, text_encoder, clip_tokenizer, clip_model):
    system_prompt = "You are an assistant designed to generate high-quality images based on user prompts."
    prompts = [system_prompt + user_input, " "]
    txt_in = tokenizer(prompts, return_tensors="pt", padding=True).to(device)
    outputs = text_encoder(**txt_in, output_hidden_states=True)
    prompt_embeds = outputs.hidden_states[-2].to(dtype)
    clip_in = clip_tokenizer(prompts, return_tensors="pt", padding=True, truncation=True).to(device)
    clip_res = clip_model.get_text_features(input_ids=clip_in.input_ids, attention_mask=clip_in.attention_mask)
    c_pooled = clip_res[0].to(dtype)
    if c_pooled.ndim == 1: c_pooled = c_pooled.unsqueeze(0)
    if c_pooled.shape[0] == 1: c_pooled = c_pooled.repeat(2, 1)
    return prompt_embeds, txt_in.attention_mask, clip_res[1].to(dtype), c_pooled

def main():
    tokenizer, text_encoder, clip_tokenizer, clip_model, vae, model, sampler = load_all_models()
    print("\n✅ Ready. Type 'quit' to exit. English or XML tags recommended.")
    image_count = 1

    while True:
        try:
            print(f"\n[{image_count}] Prompt >> ", end='', flush=True)
            line = sys.stdin.buffer.readline()
            if not line: break
            user_input = line.decode('utf-8', errors='ignore').strip()

            if user_input.lower() in ['quit', 'exit']: break
            if not user_input: continue

            print(f"⏳ Generating...")
            p_embeds, p_masks, c_seq, c_pooled = encode_prompts(user_input, tokenizer, text_encoder, clip_tokenizer, clip_model)
            model_kwargs = dict(cap_feats=p_embeds, cap_mask=p_masks, cfg_scale=4.5, clip_text_sequence=c_seq, clip_text_pooled=c_pooled)
            z = torch.randn([2, 16, 128, 128], device=device, dtype=dtype)

            def robust_forward(x, t, **kwargs):
                t_input = t.to(dtype)
                if t_input.ndim == 0: t_input = t_input.expand(x.shape[0])
                return model.forward_with_cfg(x.to(dtype), t_input, **kwargs)

            sample_fn = sampler.sample_ode(sampling_method="midpoint", num_steps=28, time_shifting_factor=6.0)
            samples = sample_fn(z, robust_forward, **model_kwargs)[-1]

            samples = vae.decode(samples[:1].to(dtype) / 0.3611 + 0.1159).sample
            img = to_pil_image(((samples[0] + 1.0) / 2.0).clamp(0.0, 1.0).float().cpu())

            save_name = f"output_{int(time.time())}.png"
            img.save(save_name)
            print(f"✨ Saved: {save_name}")
            image_count += 1
        except Exception as e:
            print(f"❌ Error: {e}")

if __name__ == "__main__":
    main()

6. Pitfalls recap

1. Class choice: for the 3.5B build use NextDiT_3B_GQA_patch2_Adaln_Refiner_WHIT_CLIP; it bakes in width 2304—don’t pass hidden_size manually or you hit TypeError.
2. Dtype: torchdiffeq often runs the ODE in float32—cast back to bfloat16 at the module boundary or matmul dtypes disagree.
3. XML prompts: the model is sensitive to tags; follow the official XML pattern for multi-character and attributes.
4. Batch=2: keep CFG with batch 2 (pos + neg); use a single space " " for the negative text so CLIP never returns empty tensors.

After these steps, NewBie-image-Exp0.1 should run cleanly. Enjoy your anime generations!