|
|
@@ -119,9 +119,12 @@ body { src_url, src_type:"video"|"image" }
|
|
|
### `search.py`
|
|
|
```python
|
|
|
SEARCH_PATHS = {
|
|
|
- "xiaohongshu": "/crawler/xiao_hong_shu/keyword",
|
|
|
- "douyin": "/crawler/dou_yin/keyword", # TODO: 实测 code 10000,疑需 cookie/鉴权
|
|
|
+ "xiaohongshu": "/crawler/xiao_hong_shu/keyword", # host = settings.crawler_base_url (aiddit)
|
|
|
+ "weixin": "/crawler/wei_xin/keyword", # 微信公众号,搜索回包即带封面,无需 detail
|
|
|
+ "douyin": "/crawler/dou_yin/keyword", # 已解决 code 10000:抖音改走 piaoquantv 后端
|
|
|
}
|
|
|
+# 抖音独立后端:host=settings.douyin_base_url(crawapi.piaoquantv.com),body 必带 account_id + cookie_batch;
|
|
|
+# keyword 与 detail 都走此 host。小红书/微信仍走 aiddit(settings.crawler_base_url)。
|
|
|
class SearchError(RuntimeError): ...
|
|
|
|
|
|
def parse_search_response(response: dict, *, platform="xiaohongshu") -> list[dict]:
|
|
|
@@ -244,6 +247,6 @@ def extract_video(post, *, ..., oss_video_url: Optional[str] = None) -> Extracte
|
|
|
| 问题 | 现象 | 处置 |
|
|
|
|---|---|---|
|
|
|
| **dim_attributes 不一致** | 代码发 `["why"]`([`TYPE2ATTR` decompose.py:60](../scripts/decompose.py)),但 `format/ingest-payload-why.schema.json` 要 `const:"why原理"`(how/what 同类:`how工序`/`what构成`) | 本迭代**不改**,待入库 ingest 时统一对齐(确定以哪边为准) |
|
|
|
-| **抖音搜索鉴权** | `POST /crawler/dou_yin/keyword` 实测 `code:10000 未知错误` | 标 TODO;`search.py` 让抖音抛 `SearchError`,`expand_queries` 干净跳过;本迭代只上小红书 |
|
|
|
+| **抖音搜索鉴权** | aiddit `POST /crawler/dou_yin/keyword` 实测 `code:10000`(强限流) | **已解决**:抖音改走 piaoquantv 后端(`crawapi.piaoquantv.com`,body 带 `account_id=7450041106378522636` + `cookie_batch=default`),search/detail 实测 `code:0`,拿到视频直链。env:`CK_DOUYIN_BASE_URL/ACCOUNT_ID/COOKIE_BATCH` |
|
|
|
| **OSS 串行延迟** | 每张图一次 OSS POST(60s timeout),多图帖串行慢 | 暂接受;后续可并行/限流 |
|
|
|
| **B站视频** | `video_extract` 对 B站抛错(需 voice_data 音轨 + ffmpeg 合并) | 与本迭代无关,沿用现状 |
|