Home Explore Help
Sign In
algorithm
/
recommend-emr-dataprocess
12
0
Fork 0
Files Issues 0 Pull Requests 1 Wiki
Tree: 1c1f4e15b4
Branches Tags
recommend-em...
/
src
/
main
/
scala
/
com
/
aliyun
/
odps
/
spark
/
examples
/
临时记录的脚本-广告

临时记录的脚本-广告 2.0 KB
History Raw

123456789101112131415161718192021222324252627282930313233343536373839404142434445 
  1. 

  2. 

  3. 

  4. nohup /opt/apps/SPARK2/spark-2.4.8-hadoop3.2-1.0.8/bin/spark-class2 org.apache.spark.deploy.SparkSubmit \
    5. 

  5. --class com.aliyun.odps.spark.examples.makedata.makedata_ad_31_originData_20240620 \
    6. 

  6. --master yarn --driver-memory 1G --executor-memory 2G --executor-cores 1 --num-executors 16 \
    7. 

  7. ./target/spark-examples-1.0.0-SNAPSHOT-shaded.jar \
    8. 

  8. tablePart:64 repartition:32 \
    9. 

  9. beginStr:2024062008 endStr:2024062008 \
    10. 

  10. savePath:/dw/recommend/model/31_ad_sample_data/ \
    11. 

  11. table:alg_recsys_ad_sample_all \
    12. 

  12. > p31_2024062008.log 2>&1 &
    13. 

  13. 

  14. 

  15. nohup /opt/apps/SPARK2/spark-2.4.8-hadoop3.2-1.0.8/bin/spark-class2 org.apache.spark.deploy.SparkSubmit \
    16. 

  16. --class com.aliyun.odps.spark.examples.makedata.makedata_14_valueData_20240608 \
    17. 

  17. --master yarn --driver-memory 1G --executor-memory 3G --executor-cores 1 --num-executors 32 \
    18. 

  18. ./target/spark-examples-1.0.0-SNAPSHOT-shaded.jar \
    19. 

  19. readPath:/dw/recommend/model/13_sample_data/ \
    20. 

  20. savePath:/dw/recommend/model/14_feature_data/ \
    21. 

  21. beginStr:20240615 endStr:20240615 repartition:1000 \
    22. 

  22. > p14_data_check.log 2>&1 &
    23. 

  23. 

  24. 

  25. nohup /opt/apps/SPARK2/spark-2.4.8-hadoop3.2-1.0.8/bin/spark-class2 org.apache.spark.deploy.SparkSubmit \
    26. 

  26. --class com.aliyun.odps.spark.examples.makedata.makedata_15_bucket_20240608 \
    27. 

  27. --master yarn --driver-memory 16G --executor-memory 1G --executor-cores 1 --num-executors 16 \
    28. 

  28. --conf spark.driver.maxResultSize=16G \
    29. 

  29. ./target/spark-examples-1.0.0-SNAPSHOT-shaded.jar \
    30. 

  30. readPath:/dw/recommend/model/14_feature_data/20240606/ fileName:20240606_200_v3 \
    31. 

  31. bucketNum:200 sampleRate:0.1 \
    32. 

  32. > p15_data2.log 2>&1 &
    33. 

  33. 

  34. 

  35. nohup /opt/apps/SPARK2/spark-2.4.8-hadoop3.2-1.0.8/bin/spark-class2 org.apache.spark.deploy.SparkSubmit \
    36. 

  36. --class com.aliyun.odps.spark.examples.makedata.makedata_16_bucketData_20240609 \
    37. 

  37. --master yarn --driver-memory 2G --executor-memory 4G --executor-cores 1 --num-executors 16 \
    38. 

  38. ./target/spark-examples-1.0.0-SNAPSHOT-shaded.jar \
    39. 

  39. beginStr:20240615 endStr:20240615 repartition:1000 \
    40. 

  40. > p16_data.log 2>&1 &
    41. 

  41. 

  42. 

  43. /dw/recommend/model/13_sample_data/
    44. 

  44. /dw/recommend/model/14_feature_data/
    45. 

  45. /dw/recommend/model/16_train_data/
    46.