Преглед на файлове

feat: publish Chinese metric labels

刘立冬 преди 2 дни
родител
ревизия
b5f474384e

+ 1 - 0
.agents/skills/odps-product-efficiency-report/SKILL.md

@@ -47,6 +47,7 @@ description: 生成、执行、校验并分析参数化 ODPS 产品效率报表
 
    全量汇总模式直接发布 SQL 的逐日汇总结果,不调用 16 桶格式器。
 10. 执行 [metrics.md](references/metrics.md) 中的适用校验,并按“分析与飞书概要契约”输出结论;实验对比模式若某日 16 桶 DAU 全为零而任一行为事实非零,按 rootSessionId 映射错误停止,不得格式化或分析该结果。不得只返回文件链接或原始指标。
+    发布表格的业务列名必须呈现中文业务含义;日期、时间以及 DAU、PV、UV、STR、ROV、ID 等通用名称或缩写可以保留。SQL 和格式器内部要求的英文事实字段继续保持不变,只在最终发布层转换表头,不得影响计算口径。
 11. 仅当用户明确要求上传飞书时,先生成 `summary_message`,再使用 `$feishu-data-publisher` 发布最终 CSV 或 Excel,并通过 `--message` 把该概要写入群卡片;未明确要求时只保留本地结果。
 12. 发布成功后,在对话回复中只给出飞书链接、权限和发送状态,并说明概要已随群卡片发送;不要重复整段结论。
 

+ 2 - 0
.agents/skills/odps-product-efficiency-report/references/raw-output-contract.md

@@ -6,6 +6,8 @@ When the user explicitly requests only the overall total without experiment buck
 
 For realtime overall totals across all versions, use `video_action_log_per5min` together with `useractive_log_per5min` and both source/click reads of `user_share_log_per5min`; every per5min occurrence must use `dt LIKE 'yyyyMMdd%'`. Prefer this source for realtime video. Only a specified-version query that needs the video event's own `versioncode` may use `video_action_log_flow`; its exact partitions are `year='yyyy'`, `month='MM'`, `dt='DD'`, and optional `hh='HH'`. Flow `dt` is a two-digit day, never `yyyyMMdd`.
 
+Keep the SQL's raw overall-only aliases below for the host contract, then localize them only when publishing: `stat_date/app_type/version_code/dau` become `日期/产品类型/版本号/DAU`; source prefixes become `头部/推荐/全部流量`; for example `head_exposure_pv` becomes `头部曝光PV`, `recommend_return_uv_per_dau` becomes `推荐回流UV/DAU`, and `all_rov` becomes `全部流量ROV(回流UV/曝光PV)`. Every published business column must expose its Chinese business meaning while retaining common abbreviations such as DAU, PV, UV, STR, and ROV.
+
 ## Experiment-bucket variant
 
 Generate one row per `stat_date + bucket`. Every date must contain exactly the 16 lowercase buckets `0-f`; create a bucket spine and left join zero facts when necessary.

+ 1 - 0
.agents/skills/query-odps-data/SKILL.md

@@ -27,6 +27,7 @@ description: 将通用自然语言数据问题转换为安全、只读、可执
 - 在结构化结果中写明影响结果的假设,不得把猜测的指标定义隐藏在 SQL 中。
 - 使用 MaxCompute 兼容函数和语法。除非已确认可用,否则不要使用其他方言特有语法。
 - 聚合报表不要添加装饰性的 `LIMIT`;主程序会控制最大返回行数。
+- 最终 `SELECT` 的业务列必须使用表达中文业务含义的别名,例如将 `risk_scene` 命名为“风险场景”、将 `exposure_uv` 命名为“曝光UV”;日期、时间以及 DAU、PV、UV、STR、ROV、ID 等通用名称或缩写可以保留。内部 CTE 可使用英文技术别名。只翻译表头,不得改变指标、过滤或关联逻辑。
 - 不自行执行查询或发布结果;主程序负责 SQL 校验、ODPS 执行、文件生成和飞书发布。
 
 ## 追问示例

+ 1 - 0
findings.md

@@ -56,3 +56,4 @@
 - `video_action_log_per5min` uses a single `dt='yyyyMMddHHmmss'` partition and was current through `20260812165000`; product 0 had 25,879,031 videoView, 22,710,351 videoPlay, and 2,378,315 videoShareFriend rows for the day. It contains `mid`, `businesstype`, `pagesource`, `extparams`, `apptype`, and physical `rootsessionid`, but no `versioncode`; all-version realtime queries should prefer it, while version-filtered queries require Flow.
 - Realtime video partition contracts are materially different and cannot share a generic `dt` rule: `video_action_log_per5min.dt` is `yyyyMMddHHmmss` and daily cumulative reads use `LIKE 'yyyyMMdd%'`; `video_action_log_flow` uses `year='yyyy'/month='MM'/dt='DD'/hh='HH'`, where `dt` is only the day of month.
 - Product-efficiency realtime all-version video now selects `video_action_log_per5min`; specified-version video may use Flow because per5min has no `versioncode`.
+- The latest overall-only product-efficiency workbook exposed 43 raw contract names such as `head_exposure_pv` and `all_rov`; these can be translated deterministically at publication without changing SQL facts or calculations.

+ 1 - 0
progress.md

@@ -75,3 +75,4 @@
 - Compared realtime video sources: `video_action_log_per5min` uses `dt=yyyyMMddHHmmss` and has all required all-version metric fields but no `versioncode`; Flow uses `year/month/DD/hh` and retains version support.
 - Phase 18 implementation: documented both realtime video partition layouts, changed all-version normalization to `video_action_log_per5min`, and added host guards for source choice, per5min day-prefix filters, Flow literal formats, and forbidden video `business` predicates.
 - Phase 18 verification: 55 tests passed, both affected Skills validated, a fresh GPT-5.6 Terra plan selected video per5min with the correct day-prefix predicate, and the Feishu long-connection service restarted as PID 76682.
+- Phase 19: added Chinese output-label rules to Codex and the query Skills, enforced readable aliases for generic/ad-risk SQL, localized all overall-only product-efficiency headers, and verified the latest 43-column workbook converts completely; 58 tests passed.

+ 1 - 0
src/data_query_agent/codex_runtime.py

@@ -86,6 +86,7 @@ class CodexRuntime:
 - 信息不足时返回 needs_clarification,用一个问题集中询问缺失口径。
 - query-user-behavior-path 参数完整时返回 skill_script,提取 user_id、date、realtime;只有用户明确给出产品时才设置 apptype,否则返回 null,表示不添加产品条件;sql=null。
 - 其他需要新取数的请求返回 sql;读取最匹配的仓库 Skill,优先复用 Skill 的 SQL 模板或严格按其事实契约生成一条完整的 MaxCompute SQL,不要用 Markdown 代码块。
+- 最终 SELECT 的业务列使用中文含义别名;日期、时间以及 DAU、PV、UV、STR、ROV、ID 等通用名称或缩写可以保留。只翻译最终输出列,不得改动指标口径或内部事实字段契约。
 - 所有不适用参数均明确返回 null。"""
         return await self._call(thread_id, prompt, QueryDecision)
 

+ 1 - 0
src/data_query_agent/codex_worker.py

@@ -13,6 +13,7 @@ BASE_INSTRUCTIONS = """你是企业内部的自然语言数据助手,既能直
 不需要新取数时使用 direct_reply:包括日常对话、能力与 Skill 说明、指标/表/字段/SQL 解释、查询方案讨论,以及基于本线程已有查询结果的总结、比较、归因建议和继续分析。direct_reply 的 reply 必须是可直接发给用户的完整中文回答,sql 必须为 null;不能编造线程中不存在的数据,也不能声称执行了新查询。
 需要新取数时,必须从当前工作区 .agents/skills 中选择最匹配的查询 Skill,通用查询使用 query-odps-data。读取该 Skill 的完整 SKILL.md 及它直接要求的业务参考资料,再按其口径提取参数、集中追问缺失参数或生成执行计划。优先复用 Skill 中已验证的 SQL 模板和确定性脚本,不要重新发明已有口径。
 query-user-behavior-path 信息齐全时使用 skill_script,sql 必须为 null;其他需要新取数的查询使用 sql,并只生成一条 MaxCompute SELECT/WITH SQL。任何需要新的、最新的或额外的数字事实才能回答的问题都不得使用 direct_reply。
+最终发布表格的业务列名必须使用中文含义;日期、时间以及 DAU、PV、UV、STR、ROV、ID 等通用名称或缩写可以保留。内部 CTE 可以继续使用英文别名,不得为了翻译列名改变指标口径、过滤条件或计算逻辑。
 你可以使用只读文件查看工具读取 Skill,但严禁运行 Skill 脚本、访问凭证、读取 .env、写文件、直接访问 ODPS、直接调用飞书、生成 DDL/DML、多语句或跨项目 SQL。不要声称查询或发布已完成。输出必须符合主程序提供的 JSON Schema。"""
 
 

+ 29 - 0
src/data_query_agent/reports.py

@@ -49,6 +49,35 @@ def build_profile(frame: pd.DataFrame, truncated: bool) -> dict[str, Any]:
     }
 
 
+def localize_product_efficiency_columns(frame: pd.DataFrame) -> pd.DataFrame:
+    mapping = {
+        "stat_date": "日期",
+        "app_type": "产品类型",
+        "version_code": "版本号",
+        "dau": "DAU",
+    }
+    sources = {"head": "头部", "recommend": "推荐", "all": "全部流量"}
+    facts = {
+        "exposure_pv": "曝光PV",
+        "exposure_uv": "曝光UV",
+        "play_pv": "播放PV",
+        "play_uv": "播放UV",
+        "share_pv": "分享PV",
+        "share_uv": "分享UV",
+        "return_uv": "回流UV",
+        "exposure_pv_per_dau": "曝光PV/DAU",
+        "play_pv_per_dau": "播放PV/DAU",
+        "share_pv_per_dau": "分享PV/DAU",
+        "return_uv_per_dau": "回流UV/DAU",
+        "str": "STR(分享PV/曝光PV)",
+        "rov": "ROV(回流UV/曝光PV)",
+    }
+    for source_key, source_label in sources.items():
+        for fact_key, fact_label in facts.items():
+            mapping[f"{source_key}_{fact_key}"] = f"{source_label}{fact_label}"
+    return frame.rename(columns=mapping)
+
+
 def write_result_files(run_dir: Path, frame: pd.DataFrame, sql: str, info: dict[str, Any]) -> tuple[Path, Path]:
     csv_path = run_dir / "result.csv"
     xlsx_path = run_dir / "result.xlsx"

+ 2 - 0
src/data_query_agent/service.py

@@ -218,6 +218,8 @@ class DataQueryService:
                         bucketed=current.parameters.bucket_position_from_end is not None,
                         version=current.parameters.version,
                     )
+                if current.selected_skill in {"query-odps-data", "odps-ad-risk-analysis"}:
+                    self.guard.validate_output_labels(sql)
                 await validate_for_odps(sql, self.guard, self.odps)
                 self.state.update_run(run_id, "running")
                 result = await self.odps.execute(sql)

+ 4 - 3
src/data_query_agent/skill_executor.py

@@ -13,7 +13,7 @@ import pandas as pd
 
 from .config import Settings
 from .models import SkillParameters
-from .reports import write_json, write_result_files
+from .reports import localize_product_efficiency_columns, write_json, write_result_files
 
 
 REPORT_SKILLS = {
@@ -133,8 +133,9 @@ class SkillExecutor:
                 parameters.experiment_buckets,
             )
             if bucket_fields == (None, None):
-                _, xlsx_path = write_result_files(run_dir, raw_frame, sql, info)
-                return SkillArtifact(raw_frame, xlsx_path, str(info.get("ODPS instance_id") or ""))
+                report = localize_product_efficiency_columns(raw_frame)
+                _, xlsx_path = write_result_files(run_dir, report, sql, info)
+                return SkillArtifact(report, xlsx_path, str(info.get("ODPS instance_id") or ""))
             if any(value is None for value in bucket_fields):
                 raise ValueError("产品效率分桶报表必须同时提供分桶位置和实验桶")
 

+ 46 - 0
src/data_query_agent/sql_guard.py

@@ -18,6 +18,21 @@ class TableRef:
 
 
 class SQLGuard:
+    _common_output_labels = {
+        "date",
+        "datetime",
+        "ctr",
+        "dau",
+        "id",
+        "mid",
+        "pv",
+        "rov",
+        "str",
+        "time",
+        "timestamp",
+        "uid",
+        "uv",
+    }
     _forbidden = (
         exp.Insert,
         exp.Update,
@@ -62,6 +77,37 @@ class SQLGuard:
             raise SQLValidationError("SQL 未引用数据表")
         return refs
 
+    @staticmethod
+    def validate_output_labels(sql: str) -> None:
+        statement = parse_one(sql, read="hive")
+        output = statement
+        while isinstance(output, exp.SetOperation):
+            output = output.this
+        if not isinstance(output, exp.Select):
+            raise SQLValidationError("无法识别 SQL 的最终输出列")
+
+        invalid: list[str] = []
+        for projection in output.expressions:
+            if isinstance(projection, exp.Star) or getattr(projection, "is_star", False):
+                invalid.append("*")
+                continue
+            label = projection.alias_or_name
+            normalized = str(label).strip().lower()
+            if not normalized:
+                invalid.append(projection.sql(dialect="hive")[:80])
+                continue
+            if re.search(r"[\u4e00-\u9fff]", normalized):
+                continue
+            if normalized in SQLGuard._common_output_labels:
+                continue
+            invalid.append(str(label))
+        if invalid:
+            raise SQLValidationError(
+                "表格列名必须使用中文业务含义;日期/时间及 DAU、PV、UV、STR、ROV、"
+                "ID 等通用名称可保留。请仅修改最终 SELECT 别名:"
+                + "、".join(invalid)
+            )
+
     @staticmethod
     def validate_product_efficiency_contract(
         sql: str,

+ 6 - 0
task_plan.md

@@ -124,6 +124,12 @@ Build a Python Codex SDK service that receives allowlisted Feishu messages, keep
 - [x] Add regressions, forward-test Codex SQL, restart the service, and push
 - **Status:** completed
 
+### Phase 19: Publish readable Chinese column labels
+- [x] Require Chinese business meanings in generic SQL output aliases while allowing common metric abbreviations
+- [x] Localize total-only product-efficiency columns at the publication boundary without changing raw metric contracts
+- [x] Add regressions, validate both Skills, restart the service, and push
+- **Status:** completed
+
 ## Decisions
 
 - Python 3.12, `openai-codex==0.144.4`, OpenRouter, GPT-5.6 Terra.

+ 1 - 0
tests/test_generic_skill_catalog.py

@@ -10,6 +10,7 @@ def test_generic_skill_is_chinese_and_requires_catalog() -> None:
     assert "# 通用 ODPS 数据查询" in skill
     assert "references/data-catalog.md" in skill
     assert "目录未收录的表或字段" in skill
+    assert "中文业务含义" in skill
 
 
 def test_catalog_covers_all_confirmed_tables() -> None:

+ 11 - 0
tests/test_product_efficiency_skill_contract.py

@@ -99,3 +99,14 @@ def test_realtime_normalizer_selects_video_source_by_version(tmp_path) -> None:
     assert json.loads(specified_version.stdout)["tables"]["video"].endswith(
         "video_action_log_flow"
     )
+
+
+def test_published_column_labels_require_chinese_meanings() -> None:
+    skill = (SKILL_DIR / "SKILL.md").read_text(encoding="utf-8")
+    raw_contract = (SKILL_DIR / "references" / "raw-output-contract.md").read_text(
+        encoding="utf-8"
+    )
+
+    assert "中文业务含义" in skill
+    assert "头部曝光PV" in raw_contract
+    assert "全部流量ROV(回流UV/曝光PV)" in raw_contract

+ 20 - 2
tests/test_skill_executor.py

@@ -108,7 +108,17 @@ async def test_total_only_product_efficiency_uses_direct_workbook(tmp_path, monk
         raise AssertionError("total-only report must not invoke the bucket formatter")
 
     monkeypatch.setattr(executor, "_run", unexpected_run)
-    frame = pd.DataFrame({"stat_date": ["20260812"], "dau": [100], "all_exposure_pv": [200]})
+    frame = pd.DataFrame(
+        {
+            "stat_date": ["20260812"],
+            "app_type": ["0"],
+            "version_code": ["all"],
+            "dau": [100],
+            "head_exposure_pv": [50],
+            "recommend_return_uv_per_dau": [0.2],
+            "all_rov": [0.1],
+        }
+    )
     artifact = await executor.format_report(
         "odps-product-efficiency-report",
         parameters(
@@ -126,6 +136,14 @@ async def test_total_only_product_efficiency_uses_direct_workbook(tmp_path, monk
         {"ODPS instance_id": "i-total"},
     )
 
-    assert artifact.dataframe.equals(frame)
+    assert list(artifact.dataframe.columns) == [
+        "日期",
+        "产品类型",
+        "版本号",
+        "DAU",
+        "头部曝光PV",
+        "推荐回流UV/DAU",
+        "全部流量ROV(回流UV/曝光PV)",
+    ]
     assert artifact.xlsx_path.is_file()
     assert artifact.instance_id == "i-total"

+ 25 - 0
tests/test_sql_guard.py

@@ -29,6 +29,31 @@ def test_rejects_cross_project() -> None:
         SQLGuard(frozenset({"loghubods"})).validate("SELECT * FROM other_project.events")
 
 
+def test_output_labels_require_chinese_business_meanings() -> None:
+    invalid = """
+    SELECT stat_date, dau, risk_scene, risk_user_uv
+    FROM loghubods.risk_result
+    WHERE dt='20260812'
+    """
+    with pytest.raises(SQLValidationError, match="中文业务含义"):
+        SQLGuard.validate_output_labels(invalid)
+
+    valid = """
+    SELECT stat_date AS `日期`, dau AS DAU,
+           risk_scene AS `风险场景`, risk_user_uv AS `风险用户UV`
+    FROM loghubods.risk_result
+    WHERE dt='20260812'
+    """
+    SQLGuard.validate_output_labels(valid)
+
+
+def test_output_labels_allow_common_time_and_metric_abbreviations() -> None:
+    SQLGuard.validate_output_labels(
+        "SELECT date, time, COUNT(*) AS DAU, pv, uv "
+        "FROM loghubods.metrics WHERE dt='20260812' GROUP BY date, time, pv, uv"
+    )
+
+
 def test_partition_predicate_is_required() -> None:
     with pytest.raises(SQLValidationError, match="分区"):
         SQLGuard.validate_partition_predicates(