소스 검색

posterior 单源也按归一化标题去重 (concatDedup 复用 tryEmit)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
刘立冬 10 시간 전
부모
커밋
338c7bfa17
1개의 변경된 파일8개의 추가작업 그리고 11개의 파일을 삭제
  1. 8 11
      api-module/src/main/java/com/tzld/piaoquan/api/service/contentplatform/impl/ContentPlatformPlanServiceImpl.java

+ 8 - 11
api-module/src/main/java/com/tzld/piaoquan/api/service/contentplatform/impl/ContentPlatformPlanServiceImpl.java

@@ -1030,22 +1030,19 @@ public class ContentPlatformPlanServiceImpl implements ContentPlatformPlanServic
         return buildDemandVideoContentItemVOList(out);
     }
 
+    /**
+     * 段间拼接 + video_id + 归一化标题 去重。标题归一化用 TitleNormalizer,
+     * 应对运营把同段内容重复上传成多个 video_id 的情况(单源 posterior 也能去掉同标题不同 id 的重复)。
+     */
     private List<VideoContentItemVO> concatDedup(List<VideoContentItemVO> a, List<VideoContentItemVO> b, int limit) {
-        Set<Long> seen = new HashSet<>();
+        Set<Long> seenIds = new HashSet<>();
+        Set<String> seenTitles = new HashSet<>();
         List<VideoContentItemVO> out = new ArrayList<>();
         for (VideoContentItemVO v : a) {
-            if (v.getVideoId() == null) continue;
-            if (seen.add(v.getVideoId())) {
-                out.add(v);
-                if (out.size() >= limit) return out;
-            }
+            if (tryEmit(v, seenIds, seenTitles, out) && out.size() >= limit) return out;
         }
         for (VideoContentItemVO v : b) {
-            if (v.getVideoId() == null) continue;
-            if (seen.add(v.getVideoId())) {
-                out.add(v);
-                if (out.size() >= limit) return out;
-            }
+            if (tryEmit(v, seenIds, seenTitles, out) && out.size() >= limit) return out;
         }
         return out;
     }