File tree Expand file tree Collapse file tree
Expand file tree Collapse file tree Original file line number Diff line number Diff line change @@ -10,6 +10,24 @@ Allow: /
1010User-agent: DuckDuckBot
1111Allow: /
1212
13+ # --- Allowed: 中文搜索引擎(目标读者在国内,显式放行)---
14+ # Bytespider 同时喂头条/抖音搜索和豆包,接受其抓取训练的代价换国内分发入口;
15+ # PetalBot 是华为花瓣搜索(上游模板把它当 SEO scraper 封了,对中文站是误伤)。
16+ User-agent: Baiduspider
17+ Allow: /
18+
19+ User-agent: Sogou web spider
20+ Allow: /
21+
22+ User-agent: 360 Spider
23+ Allow: /
24+
25+ User-agent: Bytespider
26+ Allow: /
27+
28+ User-agent: PetalBot
29+ Allow: /
30+
1331# --- Allowed: agent browsers (user-initiated, send AI users) ---
1432User-agent: ChatGPT-User
1533Allow: /
@@ -63,9 +81,6 @@ Disallow: /
6381User-agent: Applebot-Extended
6482Disallow: /
6583
66- User-agent: Bytespider
67- Disallow: /
68-
6984User-agent: Amazonbot
7085Disallow: /
7186
@@ -112,9 +127,6 @@ Disallow: /
112127User-agent: DotBot
113128Disallow: /
114129
115- User-agent: PetalBot
116- Disallow: /
117-
118130User-agent: BLEXBot
119131Disallow: /
120132
You can’t perform that action at this time.
0 commit comments