Captured at (local ISO): 2026-05-18 05:16:58

---

Preface

In web scraping, IP bans are unavoidable. Take CCTV program data: APIs throttle hard per IP. If you hammer them from one address, within a minute you often see 403 (forbidden) or 429 (too many requests)—and the crawl dies.

A proxy is an IP-switching tool: requests go through an intermediary so the target sees the proxy’s IP, not yours—reducing ban risk. Using a runnable CCTV-style scraper I wrote as context, this post walks from how proxies work and where to get them to how they land in code, so you can build a more ban-resistant crawler.

I. What is a proxy?

A proxy is a server between your client and the origin. The flow looks like this:

That path hides your real IP—the site only sees the proxy egress.

Typical uses:

1. Avoid IP blocks: Rotate IPs under high QPS so one address is not burned.
2. Bypass region locks: Some content is geo-gated; use an egress in the right region.
3. Stability: Some providers load-balance across nodes, reducing single-point failures.

Proxies can be grouped along four axes:

Dimension	Type	Traits	When to use
Anonymity	Transparent	Origin can see both proxy and real IP	Testing only
	Anonymous	Origin sees proxy IP, not the real one	Everyday anti-scrape
	High-anonymity	Origin cannot tell the path is proxied	Strict sites (e.g., CCTV, e-commerce)
Protocol	HTTP	No TLS on the tunnel—easy to intercept or tamper	Legacy HTTP-only sites
	HTTPS	TLS to origin; often backward-compatible with HTTP; certificate checks; fits browsers and modern scrapers	Most HTTPS sites; sensitive data; browser-like traffic; strict HTTPS-only policies
	SOCKS5	TCP/UDP, no app-layer parsing—flexible	P2P, mail clients, firewall punch-through
Source	Free	$0, flaky, short-lived, leakage risk	Tests, low rate
	Paid	Stable, long sessions, support, sometimes dedicated tunnels	Production, high rate
Mode	Static	Fixed IP, manual rotation—predictable blocking	Small / infrequent jobs
	Dynamic (tunnel)	One endpoint, many egress IPs, auto rotation—lower ban odds, easier ops	High rate / scale (this article’s pattern)

II. How to get proxies

I used a free trial from a commercial provider—fine for a short, heavy burst; not for always-on jobs. Flow:

1. Open 快代理 (Kuaidaili).
   
2. Log in / register, then start a free trial from the homepage.
   
3. Pick tunnel proxy Pro one-day trial.
   
4. Choose one of three tunnel products—different rotation, concurrency, and bandwidth (all selectable; I picked the highest per-request concurrency for speed—the UI shown is already committed so buttons look disabled).

Product	IP rotation	Concurrency	Bandwidth	Default IP whitelist slots	Notes
Tunnel Pro	0–30 min configurable	10 req/s	7.5 Mbps	200	Flexible rotation; good for steady jobs; many whitelist slots for multi-machine setups.
Tunnel (per request)	New IP every request	20 req/s	10 Mbps	50	Strongest anti-ban; more latency churn.
Tunnel (15 s)	Auto rotate every 15 s	50 req/s	10 Mbps	50	Middle ground for medium/high rate; IPs a bit stickier than “every request.”

Pick based on the table and your workload.

III. How to configure a proxy

After checkout you’ll see a dashboard like this:

The small red box is username/password for programmatic access.
The large red box is IP whitelist: add your laptop or server’s public IP—the host that runs the crawler.

On a home PC, people often don’t know their public IP. From CMD:

 curl ipinfo.io

In the output, the highlighted block is your public IP.

After saving, the panel should look like the screenshot above.

IV. Example usage

First, the vendor’s minimal example:

#!/usr/bin/env Python
# -*- coding: utf-8 -*-
import requests
# 隧道域名:端口号
tunnel = "XXXX"
# 用户名密码方式
username = "XXXX"
password = "XXXX"
proxies = {
    "http": "http://%(user)s:%(pwd)s@%(proxy)s/" % {"user": username, "pwd": password, "proxy": tunnel},
    "https": "http://%(user)s:%(pwd)s@%(proxy)s/" % {"user": username, "pwd": password, "proxy": tunnel}
}
# 白名单方式（需提前设置白名单）
# proxies = {
#     "http": "http://%(proxy)s/" % {"proxy": tunnel},
#     "https": "http://%(proxy)s/" % {"proxy": tunnel}
# }
# 要访问的目标网页
target_url = "https://dev.kdlapi.com/testproxy"
# 使用隧道域名发送请求
response = requests.get(target_url, proxies=proxies)
# 获取页面内容
if response.status_code == 200:
    print(response.text)  # 请勿使用keep-alive复用连接(会导致隧道不能切换IP)

The proxies dict tells requests to route HTTP/HTTPS via your tunnel. Compared to normal requests, you only add proxies=—the library already integrates cleanly.

A quick health check before production (works for free or paid lists):

import requests

def check_proxy(proxy):
    """验证代理是否有效（测试能否访问目标网站）"""
    test_url = "https://api.cntv.cn/NewVideo/getVideoListByColumn?id=TOPC1451558496100826&bd=20240101"
    try:
        response = requests.get(
            test_url,
            proxies=proxy,
            verify=False,
            timeout=5  # 短超时，快速判断
        )
        # 若能正常返回200状态码，且响应包含目标字段（如"id"），则代理有效
        return response.status_code == 200 and "id" in response.text
    except Exception as e:
        return False

# 测试代理列表
proxy_list = [
    {"http": "http://123.123.123.123:8080", "https": "http://123.123.123.123:8080"},
    {"http": "http://124.124.124.124:8080", "https": "http://124.124.124.124:8080"}
]

# 筛选有效代理
valid_proxies = [p for p in proxy_list if check_proxy(p)]
print(f"有效代理数量：{len(valid_proxies)}/{len(proxy_list)}")
if valid_proxies:
    print("第一个有效代理：", valid_proxies[0])

V. Case study: proxies in a CCTV program crawler

I dug into proxies while collecting CCTV program IDs. Yangshipin’s layout forces one page hit per ID—brutal QPS—so my home IP got blocked fast. A follow-up post may deep-dive the Yangshipin scrape; here we focus on proxies.

Everything above applies. For speed I also used async I/O (another post—link TBD). Full source is sensitive, so it’s truncated here.

If you need the complete script, message me privately.

Runtime:

Correction: ZDCJ maps to 正点财经 (On the Hour: Finance), not “正点采集”; TTKX is 体坛快讯 (Sports Flash), not “天天快讯.”

Sample output files:

If you need the complete script, message me privately.

Conclusion

Centered on anti-IP-ban scraping, this article wired theory to practice: proxies as middleboxes that mask your IP; a taxonomy by anonymity, protocol, sourcing, and static vs dynamic; a Kuaidaili trial path from signup through whitelist + public IP lookup; and validation plus a CCTV-oriented case proving proxies + requests is enough to steer traffic away from your real address.