Ver código fonte

live 小红书图文帖图片真实下载到本地(绕防盗链)

read_one 图文 live 分支:CDN 直链 → _download_images 下载到 data/demo/xiaohongshu/<id>/image_N.webp,
media.images 用本地 /data 路径(与 fixture 一致)。复用 video_extract._default_download(iOS UA + xhs Referer),
幂等、单张失败退回 CDN。解决前端跨域加载被小红书防盗链挡掉的问题(Voodoo 16 图现可本地显示)。
data/ 已 gitignore(图不入库,存云端磁盘,/data 路由服务)。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SamLee 4 semanas atrás
pai
commit
4df659dc6b
3 arquivos alterados com 665 adições e 596 exclusões
  1. 19 1
      scripts/decompose.py
  2. 382 366
      web/frameworks.json
  3. 264 229
      web/payloads.json

+ 19 - 1
scripts/decompose.py

@@ -68,6 +68,24 @@ def load_post(src: dict, settings: Settings):
 
 
 # ---------- ① 读懂(图文/视频分流) ----------
+def _download_images(post) -> list[str]:
+    """live 图文帖:把 CDN 图下载到本地 /data(避开小红书防盗链),返回本地公开路径;
+    单张失败则退回 CDN 直链(至少有 url)。复用 video_extract 的下载器(iOS UA + 平台 Referer)。"""
+    base = DATA / "xiaohongshu" / post.id
+    base.mkdir(parents=True, exist_ok=True)
+    out = []
+    for n, url in enumerate(post.image_urls, 1):
+        dst = base / f"image_{n}.webp"
+        pub = f"/data/demo/xiaohongshu/{post.id}/image_{n}.webp"
+        try:
+            if not dst.exists():
+                dst.write_bytes(video_extract._default_download(url, post.platform))
+            out.append(pub)
+        except Exception:
+            out.append(url)
+    return out
+
+
 def read_one(src: dict, post, settings: Settings, extractor: GeminiExtractor):
     cid = src["cid"]
     if post.video_urls:  # 视频帖:下载 mp4 + 原生整段提炼
@@ -83,7 +101,7 @@ def read_one(src: dict, post, settings: Settings, extractor: GeminiExtractor):
         if src["from"] == "fixture":
             imgs = [f"/data/demo/xiaohongshu/{post.id}/image_{n}.webp" for n in range(1, len(post.image_urls) + 1)]
         else:
-            imgs = list(post.image_urls)  # 新帖未落盘 → 直接用 CDN url
+            imgs = _download_images(post)  # live 帖:下载 CDN 图到本地 /data,避开防盗链
         media = {"type": "image", "video_url": None, "images": imgs}
         cmap = {c.index: c.content for c in ec.cards}
         cards = [{"index": n, "content": cmap.get(n, ""), "image_url": imgs[n - 1] if n <= len(imgs) else None}

Diferenças do arquivo suprimidas por serem muito extensas
+ 382 - 366
web/frameworks.json


Diferenças do arquivo suprimidas por serem muito extensas
+ 264 - 229
web/payloads.json


Alguns arquivos não foram mostrados porque muitos arquivos mudaram nesse diff