Home Explore Help
Sign In
Server
/
rag_server
36
1
Fork 0
Files Issues 0 Pull Requests 0 Wiki
Browse Source

性能优化，优化计算 token 的 cpu 开销

luojunhui 1 week ago
parent
2652cb85e8
commit
2972643ffa
1 changed files with 1 additions and 1 deletions
Split View Show Diff Stats
  1. 1 1
      applications/utils/chunks/topic_aware_chunking.py

+ 1 - 1
applications/utils/chunks/topic_aware_chunking.py
View File 

@@ -146,7 +146,7 @@ class TopicAwarePackerV2(TopicAwareChunker):
 
                     dataset_id=dataset_id,
 
                     text=text,
 
                     chunk_id=index,
 
-                    tokens=num_tokens(text),
 
+                    tokens=tokens,
 
                     text_type=text_type,
 
                     status=status,
 
                 )