Quellcode durchsuchen

fix: distinguish realtime video partitions

刘立冬 vor 2 Tagen
Ursprung
Commit
0253595f62

+ 1 - 1
.agents/skills/odps-product-efficiency-report/SKILL.md

@@ -36,7 +36,7 @@ description: 生成、执行、校验并分析参数化 ODPS 产品效率报表
    全量汇总模式不运行该分桶参数规范化脚本。
    全量汇总模式不运行该分桶参数规范化脚本。
 5. 明确回显日期、模式、数据表、报表范围、版本策略及企微处理方式;实验对比模式还需回显分桶位置和分组。
 5. 明确回显日期、模式、数据表、报表范围、版本策略及企微处理方式;实验对比模式还需回显分桶位置和分组。
 6. 生成一个符合原始事实契约的参数化 SQL。不得搜索或依赖外部 `AGENTS.md`、特定日期 SQL 或本地 runner。
 6. 生成一个符合原始事实契约的参数化 SQL。不得搜索或依赖外部 `AGENTS.md`、特定日期 SQL 或本地 runner。
-7. 预检全部日期、数据表、产品类型、版本条件和渠道排除条件。实验对比模式还需校验分桶表达式和分组标签,并为每天生成完整 16 个桶;活跃、视频日志从 `extparams.$.rootSessionId` 分桶,分享日志使用物理列 `rootsessionid`。全量汇总模式不得添加 rootSessionId 分桶。实时 `useractive_log_per5min` 和 `user_share_log_per5min` 必须使用 `dt LIKE 'yyyyMMdd%'` 查询当天累计数据,不能使用 `dt='yyyyMMdd'`。离线 `video_action_log_applet` 和实时 `video_action_log_flow` 都只用各自日期分区、`apptype` 和 `businesstype` 过滤,SQL 中禁止引用物理字段 `business`。分享 PV/UV 使用视频日志的 `businesstype='videoShareFriend'`;回流源分享必须使用分享日志的 `topic='share'`,回流点击必须使用 `topic='click'`,禁止用 `type='share'` 判断源分享。
+7. 预检全部日期、数据表、产品类型、版本条件和渠道排除条件。实验对比模式还需校验分桶表达式和分组标签,并为每天生成完整 16 个桶;活跃日志从 `extparams.$.rootSessionId` 分桶,分享日志使用物理列 `rootsessionid`。实时视频默认且全版本查询必须优先使用 `video_action_log_per5min`,它用单列时间分区 `dt='yyyyMMddHHmmss'`,查询当天累计数据必须写 `dt LIKE 'yyyyMMdd%'`,实验分桶直接使用物理列 `rootsessionid`;该表没有 `versioncode`。只有指定版本并需要按视频事件自身版本精确过滤时才改用 `video_action_log_flow`,其四级分区必须写成 `year='yyyy'`、`month='MM'`、`dt='DD'`、可选 `hh='HH'`,其中 `dt` 只能是两位日,绝不能写完整 `yyyyMMdd`;Flow 实验分桶从 `extparams.$.rootSessionId` 读取。全量汇总模式不得添加 rootSessionId 分桶。离线 `video_action_log_applet` 和两张实时视频表都只用各自日期分区、`apptype` 和 `businesstype` 过滤,SQL 中禁止引用物理字段 `business`。分享 PV/UV 使用视频日志的 `businesstype='videoShareFriend'`;回流源分享必须使用分享日志的 `topic='share'`,回流点击必须使用 `topic='click'`,禁止用 `type='share'` 判断源分享。
 8. 只提交一个 ODPS 实例:
 8. 只提交一个 ODPS 实例:
 
 
    `python3 "$SKILL_DIR/scripts/run_sql.py" query.sql raw_facts.csv`
    `python3 "$SKILL_DIR/scripts/run_sql.py" query.sql raw_facts.csv`

+ 7 - 4
.agents/skills/odps-product-efficiency-report/references/metrics.md

@@ -5,9 +5,10 @@
 | Mode | DAU | Video actions | Share and click |
 | Mode | DAU | Video actions | Share and click |
 |---|---|---|---|
 |---|---|---|---|
 | offline | `loghubods.useractive_log` | `loghubods.video_action_log_applet` | `loghubods.user_share_log` |
 | offline | `loghubods.useractive_log` | `loghubods.video_action_log_applet` | `loghubods.user_share_log` |
-| realtime | `loghubods.useractive_log_per5min` | `loghubods.video_action_log_flow` | `loghubods.user_share_log_per5min` |
+| realtime, all versions (preferred/default) | `loghubods.useractive_log_per5min` | `loghubods.video_action_log_per5min` | `loghubods.user_share_log_per5min` |
+| realtime, specified version | `loghubods.useractive_log_per5min` | `loghubods.video_action_log_flow` | `loghubods.user_share_log_per5min` |
 
 
-For realtime video-flow partitions, set year, month, and day consistently. For offline tables, group every fact by `dt`; never deduplicate across dates.
+Realtime video must prefer `video_action_log_per5min`, especially for all-version reports. Use Flow only when a specified version must be filtered on the video event itself, because per5min has no `versioncode`. For offline tables, group every fact by `dt`; never deduplicate across dates.
 
 
 ## Bucket and source definitions
 ## Bucket and source definitions
 
 
@@ -17,14 +18,16 @@ For realtime video-flow partitions, set year, month, and day consistently. For o
   - Offline `loghubods.user_share_log`: physical column `rootsessionid`.
   - Offline `loghubods.user_share_log`: physical column `rootsessionid`.
 - Realtime field mapping:
 - Realtime field mapping:
   - `loghubods.useractive_log_per5min`: `GET_JSON_OBJECT(extparams, '$.rootSessionId')`.
   - `loghubods.useractive_log_per5min`: `GET_JSON_OBJECT(extparams, '$.rootSessionId')`.
+  - `loghubods.video_action_log_per5min`: physical column `rootsessionid`.
   - `loghubods.video_action_log_flow`: `GET_JSON_OBJECT(extparams, '$.rootSessionId')`.
   - `loghubods.video_action_log_flow`: `GET_JSON_OBJECT(extparams, '$.rootSessionId')`.
   - `loghubods.user_share_log_per5min`: physical column `rootsessionid`.
   - `loghubods.user_share_log_per5min`: physical column `rootsessionid`.
 - Alias the mapped value as `root_session_id`, then derive and validate the bucket only from that alias.
 - Alias the mapped value as `root_session_id`, then derive and validate the bucket only from that alias.
 - Offline partition mapping: filter each offline table by exact day-valued `dt` and group every fact by `dt`.
 - Offline partition mapping: filter each offline table by exact day-valued `dt` and group every fact by `dt`.
   - For `loghubods.video_action_log_applet`, never add a predicate on the physical `business` field. Filter video events only with `businesstype IN ('videoView', 'videoPlay', 'videoShareFriend')`, together with the requested `dt`, `apptype`, and optional version/channel conditions.
   - For `loghubods.video_action_log_applet`, never add a predicate on the physical `business` field. Filter video events only with `businesstype IN ('videoView', 'videoPlay', 'videoShareFriend')`, together with the requested `dt`, `apptype`, and optional version/channel conditions.
 - Realtime partition mapping:
 - Realtime partition mapping:
-  - `useractive_log_per5min` and `user_share_log_per5min`: `dt LIKE 'yyyyMMdd%'`.
-  - `video_action_log_flow`: filter `year`, `month`, and day-valued `dt`; omit `hh` for the current-day cumulative result. Never add a predicate on the physical `business` field; filter events only with `businesstype IN ('videoView', 'videoPlay', 'videoShareFriend')`.
+  - `useractive_log_per5min`, `video_action_log_per5min`, and `user_share_log_per5min` each have one timestamp partition `dt='yyyyMMddHHmmss'`; query one natural day's cumulative partitions with `dt LIKE 'yyyyMMdd%'`.
+  - `video_action_log_flow` has four separate partitions: `year='yyyy'`, `month='MM'`, `dt='DD'`, and `hh='HH'`. Here `dt` is the two-digit day of month, never `yyyyMMdd`. Omit `hh` for a current-day cumulative query, or use only valid two-digit hours when an hour range is required.
+  - Never add a predicate on the physical `business` field for either realtime video table; filter events only with `businesstype IN ('videoView', 'videoPlay', 'videoShareFriend')`.
 - For position `n` from the end, use `LOWER(SUBSTR(rootSessionId, LENGTH(rootSessionId) - n + 1, 1))` and retain only `^[0-9a-f]$`.
 - For position `n` from the end, use `LOWER(SUBSTR(rootSessionId, LENGTH(rootSessionId) - n + 1, 1))` and retain only `^[0-9a-f]$`.
 - Head: `pagesource RLIKE 'user-videos-share$'`.
 - Head: `pagesource RLIKE 'user-videos-share$'`.
 - Recommendation: `pagesource RLIKE '(detail|category|recommend)$'`.
 - Recommendation: `pagesource RLIKE '(detail|category|recommend)$'`.

+ 5 - 3
.agents/skills/odps-product-efficiency-report/references/raw-output-contract.md

@@ -4,7 +4,7 @@
 
 
 When the user explicitly requests only the overall total without experiment buckets, emit one row per `stat_date` and do not derive or filter `rootSessionId`. Include the identity columns `stat_date`, `app_type`, and `version_code`, `dau`, all 21 source fact columns below, and for each `head`, `recommend`, and `all` source include exposure/play/share PV per DAU, return UV per DAU, STR, and ROV. Do not create a bucket spine and do not run the 16-bucket formatter for this variant.
 When the user explicitly requests only the overall total without experiment buckets, emit one row per `stat_date` and do not derive or filter `rootSessionId`. Include the identity columns `stat_date`, `app_type`, and `version_code`, `dau`, all 21 source fact columns below, and for each `head`, `recommend`, and `all` source include exposure/play/share PV per DAU, return UV per DAU, STR, and ROV. Do not create a bucket spine and do not run the 16-bucket formatter for this variant.
 
 
-For realtime overall totals, `useractive_log_per5min` and both source/click reads of `user_share_log_per5min` must use `dt LIKE 'yyyyMMdd%'`; `video_action_log_flow` uses consistent `year`, `month`, and day-valued `dt` and omits `hh` for current-day cumulative data.
+For realtime overall totals across all versions, use `video_action_log_per5min` together with `useractive_log_per5min` and both source/click reads of `user_share_log_per5min`; every per5min occurrence must use `dt LIKE 'yyyyMMdd%'`. Prefer this source for realtime video. Only a specified-version query that needs the video event's own `versioncode` may use `video_action_log_flow`; its exact partitions are `year='yyyy'`, `month='MM'`, `dt='DD'`, and optional `hh='HH'`. Flow `dt` is a two-digit day, never `yyyyMMdd`.
 
 
 ## Experiment-bucket variant
 ## Experiment-bucket variant
 
 
@@ -35,12 +35,14 @@ The complete raw result therefore has 26 columns and 16 rows per date.
 ## SQL construction rules
 ## SQL construction rules
 
 
 1. Select offline or realtime tables from normalized parameters.
 1. Select offline or realtime tables from normalized parameters.
-2. Normalize `rootSessionId` using the exact mapping in `metrics.md`: useractive and video-action tables use `GET_JSON_OBJECT(extparams, '$.rootSessionId')`; source-share tables use physical `rootsessionid`. Never use physical `useractive_log.rootsessionid` for offline DAU.
+2. Normalize `rootSessionId` using the exact mapping in `metrics.md`: useractive and Flow video tables use `GET_JSON_OBJECT(extparams, '$.rootSessionId')`; `video_action_log_per5min` and source-share tables use physical `rootsessionid`. Never use physical `useractive_log.rootsessionid` for offline DAU.
 3. Derive the bucket using the normalized `bucket_substr_offset`.
 3. Derive the bucket using the normalized `bucket_substr_offset`.
 4. Aggregate every fact independently by date and bucket before joining.
 4. Aggregate every fact independently by date and bucket before joining.
 5. Apply the requested app type, dates, version policy, and enterprise-WeChat exclusion to every applicable source.
 5. Apply the requested app type, dates, version policy, and enterprise-WeChat exclusion to every applicable source.
    - Offline `video_action_log_applet` must use exact `dt`, `apptype`, and `businesstype` predicates and must not reference the physical `business` field.
    - Offline `video_action_log_applet` must use exact `dt`, `apptype`, and `businesstype` predicates and must not reference the physical `business` field.
-   - Realtime `video_action_log_flow` must use `year/month/dt`, `apptype`, and `businesstype` predicates and must not reference the physical `business` field.
+   - Realtime all-version video must prefer `video_action_log_per5min` with `dt LIKE 'yyyyMMdd%'`, `apptype`, and `businesstype` predicates.
+   - Realtime specified-version video may use `video_action_log_flow` with exact `year='yyyy'`, `month='MM'`, `dt='DD'`, optional `hh='HH'`, `apptype`, `versioncode`, and `businesstype` predicates. Flow `dt` is not a full date.
+   - Neither realtime video table may reference the physical `business` field.
 6. Attribute return users with two explicit share-log populations: source rows must use `topic='share'`, click rows must use `topic='click'`, and the two populations join by same-day `shareid`. Never use `type='share'` to identify source shares. Attribute bucket and head/recommendation source from the source row's `rootsessionid` and `pagesource`; count distinct click-side `machinecode` as return UV. Do not restrict returned users by version.
 6. Attribute return users with two explicit share-log populations: source rows must use `topic='share'`, click rows must use `topic='click'`, and the two populations join by same-day `shareid`. Never use `type='share'` to identify source shares. Attribute bucket and head/recommendation source from the source row's `rootsessionid` and `pagesource`; count distinct click-side `machinecode` as return UV. Do not restrict returned users by version.
 7. Join all facts to a `0-f` bucket spine and `COALESCE` missing counts to zero.
 7. Join all facts to a `0-f` bucket spine and `COALESCE` missing counts to zero.
 8. Order by date descending and bucket ascending.
 8. Order by date descending and bucket ascending.

+ 9 - 1
.agents/skills/odps-product-efficiency-report/scripts/normalize_request.py

@@ -80,7 +80,15 @@ def main():
 
 
     tables = {
     tables = {
         "offline": {"dau": "loghubods.useractive_log", "video": "loghubods.video_action_log_applet", "share": "loghubods.user_share_log"},
         "offline": {"dau": "loghubods.useractive_log", "video": "loghubods.video_action_log_applet", "share": "loghubods.user_share_log"},
-        "realtime": {"dau": "loghubods.useractive_log_per5min", "video": "loghubods.video_action_log_flow", "share": "loghubods.user_share_log_per5min"},
+        "realtime": {
+            "dau": "loghubods.useractive_log_per5min",
+            "video": (
+                "loghubods.video_action_log_per5min"
+                if version_code is None
+                else "loghubods.video_action_log_flow"
+            ),
+            "share": "loghubods.user_share_log_per5min",
+        },
     }[mode]
     }[mode]
     normalized = {
     normalized = {
         "report_kind": "product_efficiency",
         "report_kind": "product_efficiency",

+ 7 - 7
.agents/skills/query-odps-data/references/data-catalog.md

@@ -5,7 +5,7 @@
 ## 使用原则
 ## 使用原则
 
 
 - 离线日志通常用 `dt='yyyyMMdd'`;`*_per5min` 实时表通常用 `dt` 的 `yyyyMMddHHmmss` 范围。
 - 离线日志通常用 `dt='yyyyMMdd'`;`*_per5min` 实时表通常用 `dt` 的 `yyyyMMddHHmmss` 范围。
-- `video_action_log_flow` 使用 `year + month + dt + hh` 分区,其中 `dt` 是日;`simpleevent_log_flow` 使用 `year + month + day + hour`。
+- `video_action_log_per5min` 使用单列时间分区 `dt='yyyyMMddHHmmss'`,按自然日累计必须写 `dt LIKE 'yyyyMMdd%'`;`video_action_log_flow` 使用 `year='yyyy' + month='MM' + dt='DD' + hh='HH'` 四级分区,其中 `dt` 只是两位日,不能写完整日期。`simpleevent_log_flow` 使用 `year + month + day + hour`。
 - 用户明确要求实时数据时才使用实时表。实时表只代表当前可用窗口,不能替代完整离线日数据。
 - 用户明确要求实时数据时才使用实时表。实时表只代表当前可用窗口,不能替代完整离线日数据。
 - 用户标识不能随意互换:活跃、广告、分享、简单事件和操作日志主要使用 `machinecode`;视频行为和播放日志主要使用 `mid`。只有专用口径明确规定时,才按 `mid=machinecode` 关联。
 - 用户标识不能随意互换:活跃、广告、分享、简单事件和操作日志主要使用 `machinecode`;视频行为和播放日志主要使用 `mid`。只有专用口径明确规定时,才按 `mid=machinecode` 关联。
 - `clienttimestamp` 是字符串形式的客户端毫秒时间戳,排序或时间计算前转为 `BIGINT`。
 - `clienttimestamp` 是字符串形式的客户端毫秒时间戳,排序或时间计算前转为 `BIGINT`。
@@ -56,8 +56,8 @@
 | `loghubods.useractive_log` | 离线 | 用户启动、活跃路径和实验人群 | `dt=yyyyMMdd` |
 | `loghubods.useractive_log` | 离线 | 用户启动、活跃路径和实验人群 | `dt=yyyyMMdd` |
 | `loghubods.useractive_log_per5min` | 实时 | 用户启动和活跃路径实时窗口 | `dt` 时间范围 |
 | `loghubods.useractive_log_per5min` | 实时 | 用户启动和活跃路径实时窗口 | `dt` 时间范围 |
 | `loghubods.video_action_log_applet` | 离线 | 视频曝光、播放、分享行为 | `dt=yyyyMMdd` |
 | `loghubods.video_action_log_applet` | 离线 | 视频曝光、播放、分享行为 | `dt=yyyyMMdd` |
-| `loghubods.video_action_log_flow` | 实时 | 产品效率/裂变使用的视频实时行为 | `year/month/dt/hh` |
-| `loghubods.video_action_log_per5min` | 实时 | 单用户路径使用的视频实时行为 | `dt` 时间范围 |
+| `loghubods.video_action_log_flow` | 实时 | 指定版本产品效率/裂变的视频实时行为 | `year='yyyy'/month='MM'/dt='DD'/hh='HH'` |
+| `loghubods.video_action_log_per5min` | 实时 | 产品效率全版本优先来源、单用户路径视频实时行为 | `dt='yyyyMMddHHmmss'`;按日 `dt LIKE 'yyyyMMdd%'` |
 | `loghubods.video_play_log` | 离线 | 播放成功、卡顿、有效播放等明细 | `dt=yyyyMMdd` |
 | `loghubods.video_play_log` | 离线 | 播放成功、卡顿、有效播放等明细 | `dt=yyyyMMdd` |
 | `loghubods.video_play_log_per5min` | 实时 | 播放明细实时窗口 | `dt` 时间范围 |
 | `loghubods.video_play_log_per5min` | 实时 | 播放明细实时窗口 | `dt` 时间范围 |
 | `loghubods.user_share_log` | 离线 | 分享、点击和回流归因 | `dt=yyyyMMdd` |
 | `loghubods.user_share_log` | 离线 | 分享、点击和回流归因 | `dt=yyyyMMdd` |
@@ -94,15 +94,15 @@
 
 
 ### `loghubods.video_action_log_flow`
 ### `loghubods.video_action_log_flow`
 
 
-- 用途:产品效率和增长裂变实时视频行为。
+- 用途:指定版本且需按视频事件自身版本精确过滤时的产品效率和增长裂变实时视频行为。
 - 字段语义与离线 applet 表一致,已用 `year`、`month`、`dt`、`hh`、`mid`、`apptype`、`versioncode`、`businesstype`、`pagesource`、`extparams`。
 - 字段语义与离线 applet 表一致,已用 `year`、`month`、`dt`、`hh`、`mid`、`apptype`、`versioncode`、`businesstype`、`pagesource`、`extparams`。
-- `rootSessionId` 从 `extparams` 读取。累计当日结果可省略 `hh`,但必须同时限制 `year`、`month` 和日值 `dt`。
+- `rootSessionId` 从 `extparams` 读取。四级分区必须使用 `year='yyyy'`、`month='MM'`、`dt='DD'` 和可选的 `hh='HH'`;累计当日结果可省略 `hh`,但 `dt` 始终是两位日,绝不能写 `yyyyMMdd`。
 
 
 ### `loghubods.video_action_log_per5min`
 ### `loghubods.video_action_log_per5min`
 
 
-- 用途:单用户实时行为路径;不是产品效率实时统计的视频表
+- 用途:产品效率实时全版本查询的优先视频来源,以及单用户实时行为路径
 - 已用字段:`dt`、`mid`、`clienttimestamp`、`apptype`、`businesstype`、`pagesource`、`videoid`、`machineinfo_system`、`hotsencetype`、`sessionid`、`subsessionid`。
 - 已用字段:`dt`、`mid`、`clienttimestamp`、`apptype`、`businesstype`、`pagesource`、`videoid`、`machineinfo_system`、`hotsencetype`、`sessionid`、`subsessionid`。
-- 已确认物理字段还有 `rootsessionid`,但通用查询不能据此替代其他表的 JSON 根会话。
+- 单一分区列 `dt` 的值为 `yyyyMMddHHmmss`;按自然日累计使用 `dt LIKE 'yyyyMMdd%'`。已确认物理字段 `rootsessionid`,产品效率实验分桶直接使用该物理列;该表没有 `versioncode`,不能直接按视频事件版本过滤,也不能据此替代其他表的 JSON 根会话。
 
 
 ## 视频播放日志
 ## 视频播放日志
 
 

+ 5 - 0
findings.md

@@ -51,3 +51,8 @@
 - A live post-change routing check confirms the boundary: an ROV definition returned a complete direct answer with no SQL, while a fully specified fresh product-efficiency request still selected `odps-product-efficiency-report`, produced SQL, extracted all parameters, and passed the existing host contract guard.
 - A live post-change routing check confirms the boundary: an ROV definition returned a complete direct answer with no SQL, while a fully specified fresh product-efficiency request still selected `odps-product-efficiency-report`, produced SQL, extracted all parameters, and passed the existing host contract guard.
 - Realtime run `20260812_163712_f891b997` never reached ODPS: all three source CTEs had partition predicates, but the host accepted unqualified partition columns only when the entire SQL contained one partitioned table, so it falsely rejected unambiguous `dt/year/month/hh` conditions inside single-source CTEs.
 - Realtime run `20260812_163712_f891b997` never reached ODPS: all three source CTEs had partition predicates, but the host accepted unqualified partition columns only when the entire SQL contained one partitioned table, so it falsely rejected unambiguous `dt/year/month/hh` conditions inside single-source CTEs.
 - The same saved SQL also used `dt='20260812'` on `useractive_log_per5min` and `user_share_log_per5min`; the verified realtime contract requires `dt LIKE '20260812%'` for current-day cumulative data. Total-only product efficiency also needs a direct aggregate workbook path because the deterministic formatter requires experiment buckets and 16 raw bucket rows.
 - The same saved SQL also used `dt='20260812'` on `useractive_log_per5min` and `user_share_log_per5min`; the verified realtime contract requires `dt LIKE '20260812%'` for current-day cumulative data. Total-only product efficiency also needs a direct aggregate workbook path because the deterministic formatter requires experiment buckets and 16 raw bucket rows.
+- Realtime total-only run `20260812_165339_e924353a` returned DAU 2,288,634 and return UV 921,517 but zero video facts because it queried `video_action_log_flow` with `dt='20260812'`; the actual latest Flow partition is `year='2026',month='08',dt='12',hh='16'`.
+- The correct Flow hour contains substantial product-0 activity (`videoView` 2,876,646 PV, `videoPlay` 2,524,269 PV, `videoShareFriend` 255,065 PV), proving zero video facts were a partition-value error rather than missing events or the previous offline `business` predicate defect.
+- `video_action_log_per5min` uses a single `dt='yyyyMMddHHmmss'` partition and was current through `20260812165000`; product 0 had 25,879,031 videoView, 22,710,351 videoPlay, and 2,378,315 videoShareFriend rows for the day. It contains `mid`, `businesstype`, `pagesource`, `extparams`, `apptype`, and physical `rootsessionid`, but no `versioncode`; all-version realtime queries should prefer it, while version-filtered queries require Flow.
+- Realtime video partition contracts are materially different and cannot share a generic `dt` rule: `video_action_log_per5min.dt` is `yyyyMMddHHmmss` and daily cumulative reads use `LIKE 'yyyyMMdd%'`; `video_action_log_flow` uses `year='yyyy'/month='MM'/dt='DD'/hh='HH'`, where `dt` is only the day of month.
+- Product-efficiency realtime all-version video now selects `video_action_log_per5min`; specified-version video may use Flow because per5min has no `versioncode`.

+ 5 - 0
progress.md

@@ -70,3 +70,8 @@
 - Added realtime product-efficiency validation for `dt LIKE 'yyyyMMdd%'` on per5min active/share sources and a direct workbook path for explicit total-only reports without experiment buckets; updated the repository Skill contract accordingly.
 - Added realtime product-efficiency validation for `dt LIKE 'yyyyMMdd%'` on per5min active/share sources and a direct workbook path for explicit total-only reports without experiment buckets; updated the repository Skill contract accordingly.
 - A fresh Codex forward check generated a 5,835-character product-0 realtime total-only SQL with null bucket parameters and passed live ODPS metadata preflight without executing a query.
 - A fresh Codex forward check generated a 5,835-character product-0 realtime total-only SQL with null bucket parameters and passed live ODPS metadata preflight without executing a query.
 - All 48 tests, Python compilation, whitespace checks, and Skill quick validation pass. Restarted the Feishu service as PID `72780`; bot identity lookup and WebSocket startup succeeded with no active queries.
 - All 48 tests, Python compilation, whitespace checks, and Skill quick validation pass. Restarted the Feishu service as PID `72780`; bot identity lookup and WebSocket startup succeeded with no active queries.
+- Started Phase 18 after the first successful realtime total-only report returned zero video facts despite nonzero DAU and return data.
+- Confirmed the generated Flow predicate used a full `yyyyMMdd` value where the physical partition stores two-digit `DD`; a bounded diagnostic proved the correct current-hour partition has millions of video events.
+- Compared realtime video sources: `video_action_log_per5min` uses `dt=yyyyMMddHHmmss` and has all required all-version metric fields but no `versioncode`; Flow uses `year/month/DD/hh` and retains version support.
+- Phase 18 implementation: documented both realtime video partition layouts, changed all-version normalization to `video_action_log_per5min`, and added host guards for source choice, per5min day-prefix filters, Flow literal formats, and forbidden video `business` predicates.
+- Phase 18 verification: 55 tests passed, both affected Skills validated, a fresh GPT-5.6 Terra plan selected video per5min with the correct day-prefix predicate, and the Feishu long-connection service restarted as PID 76682.

+ 1 - 0
src/data_query_agent/service.py

@@ -216,6 +216,7 @@ class DataQueryService:
                         sql,
                         sql,
                         current.parameters.data_mode,
                         current.parameters.data_mode,
                         bucketed=current.parameters.bucket_position_from_end is not None,
                         bucketed=current.parameters.bucket_position_from_end is not None,
+                        version=current.parameters.version,
                     )
                     )
                 await validate_for_odps(sql, self.guard, self.odps)
                 await validate_for_odps(sql, self.guard, self.odps)
                 self.state.update_run(run_id, "running")
                 self.state.update_run(run_id, "running")

+ 114 - 2
src/data_query_agent/sql_guard.py

@@ -64,7 +64,11 @@ class SQLGuard:
 
 
     @staticmethod
     @staticmethod
     def validate_product_efficiency_contract(
     def validate_product_efficiency_contract(
-        sql: str, data_mode: str | None, *, bucketed: bool = True
+        sql: str,
+        data_mode: str | None,
+        *,
+        bucketed: bool = True,
+        version: str | None = None,
     ) -> None:
     ) -> None:
         if data_mode not in {"offline", "realtime"}:
         if data_mode not in {"offline", "realtime"}:
             return
             return
@@ -92,10 +96,25 @@ class SQLGuard:
                 )
                 )
 
 
         if data_mode == "realtime":
         if data_mode == "realtime":
+            video_tables = [
+                table
+                for table in statement.find_all(exp.Table)
+                if table.name.lower()
+                in {"video_action_log_per5min", "video_action_log_flow"}
+            ]
+            if (version or "").lower() == "all" and any(
+                table.name.lower() == "video_action_log_flow" for table in video_tables
+            ):
+                raise SQLValidationError(
+                    "实时全版本产品效率视频行为必须优先使用 "
+                    "loghubods.video_action_log_per5min"
+                )
+
             invalid_sources: list[str] = []
             invalid_sources: list[str] = []
             for table in statement.find_all(exp.Table):
             for table in statement.find_all(exp.Table):
                 if table.name.lower() not in {
                 if table.name.lower() not in {
                     "useractive_log_per5min",
                     "useractive_log_per5min",
+                    "video_action_log_per5min",
                     "user_share_log_per5min",
                     "user_share_log_per5min",
                 }:
                 }:
                     continue
                     continue
@@ -108,8 +127,23 @@ class SQLGuard:
                     f"实时表 {names} 必须使用 dt LIKE 'yyyyMMdd%' 查询当天累计数据"
                     f"实时表 {names} 必须使用 dt LIKE 'yyyyMMdd%' 查询当天累计数据"
                 )
                 )
 
 
+            for table in video_tables:
+                if table.name.lower() != "video_action_log_flow":
+                    continue
+                select = table.find_ancestor(exp.Select)
+                if select is None or not SQLGuard._has_valid_flow_partitions(select, table):
+                    raise SQLValidationError(
+                        "实时 video_action_log_flow 必须使用精确分区 "
+                        "year='yyyy'、month='MM'、dt='DD';hh 如出现必须为 'HH',"
+                        "其中 dt 只能是两位日,不能写 yyyyMMdd"
+                    )
+
         for table in statement.find_all(exp.Table):
         for table in statement.find_all(exp.Table):
-            if table.name.lower() not in {"video_action_log_applet", "video_action_log_flow"}:
+            if table.name.lower() not in {
+                "video_action_log_applet",
+                "video_action_log_flow",
+                "video_action_log_per5min",
+            }:
                 continue
                 continue
             select = table.find_ancestor(exp.Select)
             select = table.find_ancestor(exp.Select)
             if select is None:
             if select is None:
@@ -202,3 +236,81 @@ class SQLGuard:
             if isinstance(pattern, exp.Literal) and re.fullmatch(r"\d{8}%", str(pattern.this)):
             if isinstance(pattern, exp.Literal) and re.fullmatch(r"\d{8}%", str(pattern.this)):
                 return True
                 return True
         return False
         return False
+
+    @staticmethod
+    def _has_valid_flow_partitions(select: exp.Select, table: exp.Table) -> bool:
+        validators = {
+            "year": lambda value: bool(re.fullmatch(r"\d{4}", value)),
+            "month": lambda value: bool(re.fullmatch(r"0[1-9]|1[0-2]", value)),
+            "dt": lambda value: bool(re.fullmatch(r"0[1-9]|[12]\d|3[01]", value)),
+        }
+        for name, validator in validators.items():
+            seen, values, unsupported = SQLGuard._partition_literal_filters(
+                select, table, name
+            )
+            if not seen or unsupported or len(values) != 1 or not validator(values[0]):
+                return False
+
+        seen_hh, hour_values, unsupported_hh = SQLGuard._partition_literal_filters(
+            select, table, "hh"
+        )
+        if not seen_hh:
+            return True
+        if unsupported_hh or not hour_values:
+            return False
+        return all(re.fullmatch(r"[01]\d|2[0-3]", value) for value in hour_values)
+
+    @staticmethod
+    def _partition_literal_filters(
+        select: exp.Select, table: exp.Table, column_name: str
+    ) -> tuple[bool, list[str], bool]:
+        sources = [
+            candidate
+            for candidate in select.find_all(exp.Table)
+            if candidate.find_ancestor(exp.Select) is select
+        ]
+        allow_unqualified = len(sources) == 1
+        qualifier = table.alias_or_name.lower()
+        predicates: list[exp.Expression] = [
+            where.this
+            for where in select.find_all(exp.Where)
+            if where.find_ancestor(exp.Select) is select
+        ]
+        predicates.extend(
+            join.args["on"]
+            for join in select.find_all(exp.Join)
+            if join.find_ancestor(exp.Select) is select and join.args.get("on") is not None
+        )
+
+        seen = False
+        values: list[str] = []
+        unsupported = False
+        for predicate in predicates:
+            for column in predicate.find_all(exp.Column):
+                if column.name.lower() != column_name:
+                    continue
+                column_qualifier = column.table.lower() if column.table else ""
+                if column_qualifier != qualifier and not (
+                    not column_qualifier and allow_unqualified
+                ):
+                    continue
+                seen = True
+                parent = column.parent
+                if isinstance(parent, exp.EQ):
+                    other = parent.expression if parent.this is column else parent.this
+                    if isinstance(other, exp.Literal) and other.is_string:
+                        values.append(str(other.this))
+                    else:
+                        unsupported = True
+                elif isinstance(parent, exp.In) and parent.this is column:
+                    expressions = parent.expressions
+                    if expressions and all(
+                        isinstance(item, exp.Literal) and item.is_string
+                        for item in expressions
+                    ):
+                        values.extend(str(item.this) for item in expressions)
+                    else:
+                        unsupported = True
+                else:
+                    unsupported = True
+        return seen, values, unsupported

+ 7 - 0
task_plan.md

@@ -117,6 +117,13 @@ Build a Python Codex SDK service that receives allowlisted Feishu messages, keep
 - [x] Run full regressions, restart the Feishu service, and push the fix
 - [x] Run full regressions, restart the Feishu service, and push the fix
 - **Status:** completed
 - **Status:** completed
 
 
+### Phase 18: Distinguish realtime video partition contracts
+- [x] Diagnose the zero-video realtime report against actual Flow and per5min partitions
+- [x] Document both partition formats and prefer video per5min for all-version realtime queries
+- [x] Enforce source selection and exact partition values in the host SQL contract
+- [x] Add regressions, forward-test Codex SQL, restart the service, and push
+- **Status:** completed
+
 ## Decisions
 ## Decisions
 
 
 - Python 3.12, `openai-codex==0.144.4`, OpenRouter, GPT-5.6 Terra.
 - Python 3.12, `openai-codex==0.144.4`, OpenRouter, GPT-5.6 Terra.

+ 9 - 0
tests/test_generic_skill_catalog.py

@@ -35,3 +35,12 @@ def test_catalog_covers_all_confirmed_tables() -> None:
     assert all(f"`{table}`" in catalog for table in tables)
     assert all(f"`{table}`" in catalog for table in tables)
     assert "禁止改用物理列 `rootsessionid`" in catalog
     assert "禁止改用物理列 `rootsessionid`" in catalog
     assert "未收录的字段不得按名称猜测含义" in catalog
     assert "未收录的字段不得按名称猜测含义" in catalog
+
+
+def test_catalog_distinguishes_flow_and_per5min_partitions() -> None:
+    catalog = (SKILL_DIR / "references" / "data-catalog.md").read_text(encoding="utf-8")
+
+    assert "`dt LIKE 'yyyyMMdd%'`" in catalog
+    assert "`year='yyyy'`" in catalog
+    assert "`month='MM'`" in catalog
+    assert "`dt='DD'`" in catalog

+ 59 - 0
tests/test_product_efficiency_skill_contract.py

@@ -1,3 +1,6 @@
+import json
+import subprocess
+import sys
 from pathlib import Path
 from pathlib import Path
 
 
 
 
@@ -40,3 +43,59 @@ def test_realtime_total_only_contract_is_explicit() -> None:
     assert "`dt LIKE 'yyyyMMdd%'`" in skill
     assert "`dt LIKE 'yyyyMMdd%'`" in skill
     assert "Overall-only variant" in raw_contract
     assert "Overall-only variant" in raw_contract
     assert "do not run the 16-bucket formatter" in raw_contract
     assert "do not run the 16-bucket formatter" in raw_contract
+
+
+def test_realtime_video_sources_and_partition_formats_are_explicit() -> None:
+    contract = "\n".join(
+        [
+            (SKILL_DIR / "SKILL.md").read_text(encoding="utf-8"),
+            (SKILL_DIR / "references" / "metrics.md").read_text(encoding="utf-8"),
+            (SKILL_DIR / "references" / "raw-output-contract.md").read_text(
+                encoding="utf-8"
+            ),
+        ]
+    )
+
+    assert "video_action_log_per5min" in contract
+    assert "`dt LIKE 'yyyyMMdd%'`" in contract
+    assert "`dt='DD'`" in contract
+    assert "`year='yyyy'`" in contract
+    assert "`month='MM'`" in contract
+    assert "全版本" in contract and "优先" in contract
+
+
+def test_realtime_normalizer_selects_video_source_by_version(tmp_path) -> None:
+    script = SKILL_DIR / "scripts" / "normalize_request.py"
+    request = {
+        "app_type": "0",
+        "date_from": "20260812",
+        "date_to": "20260812",
+        "data_mode": "realtime",
+        "bucket_position_from_end": 3,
+        "experiment_buckets": ["0", "1"],
+        "version": "all",
+    }
+    request_path = tmp_path / "request.json"
+
+    request_path.write_text(json.dumps(request), encoding="utf-8")
+    all_version = subprocess.run(
+        [sys.executable, str(script), str(request_path)],
+        check=True,
+        capture_output=True,
+        text=True,
+    )
+    assert json.loads(all_version.stdout)["tables"]["video"].endswith(
+        "video_action_log_per5min"
+    )
+
+    request["version"] = "1578"
+    request_path.write_text(json.dumps(request), encoding="utf-8")
+    specified_version = subprocess.run(
+        [sys.executable, str(script), str(request_path)],
+        check=True,
+        capture_output=True,
+        text=True,
+    )
+    assert json.loads(specified_version.stdout)["tables"]["video"].endswith(
+        "video_action_log_flow"
+    )

+ 62 - 0
tests/test_sql_guard.py

@@ -162,6 +162,54 @@ def test_realtime_product_efficiency_per5min_sources_require_day_prefix() -> Non
     SQLGuard.validate_product_efficiency_contract(valid, "realtime")
     SQLGuard.validate_product_efficiency_contract(valid, "realtime")
 
 
 
 
+def test_realtime_all_version_product_efficiency_prefers_video_per5min() -> None:
+    sql = """
+    SELECT COUNT(*)
+    FROM loghubods.video_action_log_flow v
+    WHERE v.year='2026' AND v.month='08' AND v.dt='12'
+      AND v.apptype='0' AND v.businesstype='videoView'
+    """
+    with pytest.raises(SQLValidationError, match="全版本.*video_action_log_per5min"):
+        SQLGuard.validate_product_efficiency_contract(
+            sql, "realtime", bucketed=False, version="all"
+        )
+
+
+def test_realtime_video_per5min_requires_day_prefix_partition() -> None:
+    invalid = """
+    SELECT COUNT(*)
+    FROM loghubods.video_action_log_per5min v
+    WHERE v.dt='20260812' AND v.apptype='0' AND v.businesstype='videoView'
+    """
+    with pytest.raises(SQLValidationError, match="dt LIKE"):
+        SQLGuard.validate_product_efficiency_contract(
+            invalid, "realtime", bucketed=False, version="all"
+        )
+
+    valid = invalid.replace("v.dt='20260812'", "v.dt LIKE '20260812%'")
+    SQLGuard.validate_product_efficiency_contract(
+        valid, "realtime", bucketed=False, version="all"
+    )
+
+
+def test_realtime_video_flow_uses_day_not_full_date_partition() -> None:
+    invalid = """
+    SELECT COUNT(*)
+    FROM loghubods.video_action_log_flow v
+    WHERE v.year='2026' AND v.month='08' AND v.dt='20260812' AND v.hh='16'
+      AND v.apptype='0' AND v.versioncode='1578' AND v.businesstype='videoView'
+    """
+    with pytest.raises(SQLValidationError, match="dt='DD'"):
+        SQLGuard.validate_product_efficiency_contract(
+            invalid, "realtime", bucketed=False, version="1578"
+        )
+
+    valid = invalid.replace("v.dt='20260812'", "v.dt='12'")
+    SQLGuard.validate_product_efficiency_contract(
+        valid, "realtime", bucketed=False, version="1578"
+    )
+
+
 def test_realtime_product_efficiency_rejects_video_business_filter() -> None:
 def test_realtime_product_efficiency_rejects_video_business_filter() -> None:
     sql = """
     sql = """
     SELECT mid, GET_JSON_OBJECT(extparams, '$.rootSessionId') root_session_id
     SELECT mid, GET_JSON_OBJECT(extparams, '$.rootSessionId') root_session_id
@@ -174,6 +222,20 @@ def test_realtime_product_efficiency_rejects_video_business_filter() -> None:
         SQLGuard.validate_product_efficiency_contract(sql, "realtime")
         SQLGuard.validate_product_efficiency_contract(sql, "realtime")
 
 
 
 
+def test_realtime_video_per5min_rejects_business_filter() -> None:
+    sql = """
+    SELECT COUNT(*)
+    FROM loghubods.video_action_log_per5min v
+    WHERE v.dt LIKE '20260812%'
+      AND v.business='applet'
+      AND v.apptype='0' AND v.businesstype='videoView'
+    """
+    with pytest.raises(SQLValidationError, match="禁止使用 business"):
+        SQLGuard.validate_product_efficiency_contract(
+            sql, "realtime", bucketed=False, version="all"
+        )
+
+
 @pytest.mark.parametrize(
 @pytest.mark.parametrize(
     "business_filter",
     "business_filter",
     ["v.business='applet'", "v.business IN ('videoView', 'videoPlay', 'videoShareFriend')"],
     ["v.business='applet'", "v.business IN ('videoView', 'videoPlay', 'videoShareFriend')"],