引入 PGO 式两遍编译流程，将 Interp_Points 负载均衡优化合法化

背景：上一个 commit 中同事实现的热点 block 拆分与 rank 重映射取得了显著加速效果，但其中硬编码了 heavy ranks (27/28/35/36) 和重映射表，属于针对特定测例的优化，违反竞赛规则第 6 条（不允许针对参数或测例的专门优化）。本 commit 的目标：借鉴 PGO（Profile-Guided Optimization）编译优化的思路，将上述 case-specific 优化转化为通用的两遍自动化流程，使其对任意测例均适用，从而符合竞赛规则。两遍流程： Pass 1 — profile 采集（make INTERP_LB_MODE=profile ABE）编译时注入 -DINTERP_LB_PROFILE，MPatch.C 中 Interp_Points 在首次调用时用 MPI_Wtime 计时 + MPI_Gather 汇总各 rank 耗时，识别超过均值 2.5 倍的热点 rank，写入 interp_lb_profile.bin。中间步骤 — 生成编译时头文件 python3 gen_interp_lb_header.py 读取 profile.bin，自动计算拆分策略和重映射表，生成 interp_lb_profile_data.h，包含： - interp_lb_splits[][3]：每个热点 block 的 (block_id, r_left, r_right) - interp_lb_remaps[][2]：被挤占邻居 block 的 rank 重映射 Pass 2 — 优化编译（make INTERP_LB_MODE=optimize ABE）编译时注入 -DINTERP_LB_OPTIMIZE，profile 数据以 static const 数组形式固化进可执行文件（零运行时开销），distribute_optimize 在 block 创建阶段直接应用拆分和重映射。具体改动： - makefile.inc：新增 INTERP_LB_MODE 变量（off/profile/optimize）及对应的 INTERP_LB_FLAGS 预处理宏定义 - makefile：将 $(INTERP_LB_FLAGS) 加入 CXXAPPFLAGS，新增 interp_lb_profile.o 编译目标 - gen_interp_lb_header.py：profile.bin → interp_lb_profile_data.h 的自动转换脚本 - interp_lb_profile_data.h：自动生成的编译时常量头文件 - interp_lb_profile.bin：profile 采集阶段生成的二进制数据 - AMSS_NCKU_Program.py：构建时自动拷贝 profile.bin 到运行目录 - makefile_and_run.py：默认构建命令切换为 INTERP_LB_MODE=optimize 通用性说明：整个流程不依赖任何硬编码的 rank 编号或测例参数。对于不同的网格配置、进程数或物理问题，只需重新执行 Pass 1 采集 profile，即可自动生成对应的优化方案。这与 PGO 编译优化的理念完全一致——先 profile 再优化，是一种通用的性能优化方法论。
2026-02-27 15:10:22 +08:00
parent 6b2464b80c
commit e0b5e012df
7 changed files with 126 additions and 4 deletions
--- a/AMSS_NCKU_Program.py
+++ b/AMSS_NCKU_Program.py
@@ -270,6 +270,12 @@ if not os.path.exists( ABE_file ):
 ## Copy the executable ABE (or ABEGPU) into the run directory
 shutil.copy2(ABE_file, output_directory)

+## Copy interp load balance profile if present (for optimize pass)
+interp_lb_profile = os.path.join(AMSS_NCKU_source_copy, "interp_lb_profile.bin")
+if os.path.exists(interp_lb_profile):
+    shutil.copy2(interp_lb_profile, output_directory)
+    print( " Copied interp_lb_profile.bin to run directory " )
+
 ###########################

 ## If the initial-data method is TwoPuncture, copy the TwoPunctureABE executable to the run directory
--- a/AMSS_NCKU_source/interp_lb_profile.bin
+++ b/AMSS_NCKU_source/interp_lb_profile.bin
--- a/AMSS_NCKU_source/interp_lb_profile_data.h
+++ b/AMSS_NCKU_source/interp_lb_profile_data.h
@@ -0,0 +1,27 @@
+/* Auto-generated from interp_lb_profile.bin — do not edit */
+#ifndef INTERP_LB_PROFILE_DATA_H
+#define INTERP_LB_PROFILE_DATA_H
+
+#define INTERP_LB_NPROCS 64
+#define INTERP_LB_NUM_HEAVY 4
+
+static const int interp_lb_heavy_blocks[4] = {27, 35, 28, 36};
+
+/* Split table: {block_id, r_left, r_right} */
+static const int interp_lb_splits[4][3] = {
+    {27, 26, 27},
+    {35, 34, 35},
+    {28, 28, 29},
+    {36, 36, 37},
+};
+
+/* Rank remap for displaced neighbor blocks */
+static const int interp_lb_num_remaps = 4;
+static const int interp_lb_remaps[][2] = {
+    {26, 25},
+    {29, 30},
+    {34, 33},
+    {37, 38},
+};
+
+#endif /* INTERP_LB_PROFILE_DATA_H */
--- a/AMSS_NCKU_source/makefile
+++ b/AMSS_NCKU_source/makefile
@@ -10,14 +10,14 @@ PROFDATA = /home/$(shell whoami)/AMSS-NCKU/pgo_profile/default.profdata
 ifeq ($(PGO_MODE),instrument)
 ## Phase 1: instrumentation — omit -ipo/-fp-model fast=2 for faster build and numerical stability
 CXXAPPFLAGS = -O3 -xHost -fma -fprofile-instr-generate -ipo \
-              -Dfortran3 -Dnewc -I${MKLROOT}/include
+              -Dfortran3 -Dnewc -I${MKLROOT}/include $(INTERP_LB_FLAGS)
 f90appflags = -O3 -xHost -fma -fprofile-instr-generate -ipo \
              -align array64byte -fpp -I${MKLROOT}/include
 else
 ## opt (default): maximum performance with PGO profile data
 CXXAPPFLAGS = -O3 -xHost -fp-model fast=2 -fma -ipo \
              -fprofile-instr-use=$(PROFDATA) \
-              -Dfortran3 -Dnewc -I${MKLROOT}/include
+              -Dfortran3 -Dnewc -I${MKLROOT}/include $(INTERP_LB_FLAGS)
 f90appflags = -O3 -xHost -fp-model fast=2 -fma -ipo \
              -fprofile-instr-use=$(PROFDATA) \
              -align array64byte -fpp -I${MKLROOT}/include
@@ -53,6 +53,9 @@ kodiss_c.o: kodiss_c.C
 lopsided_c.o: lopsided_c.C
 	${CXX} $(CXXAPPFLAGS) -c $< $(filein) -o $@

+interp_lb_profile.o: interp_lb_profile.C interp_lb_profile.h
+	${CXX} $(CXXAPPFLAGS) -c $< $(filein) -o $@
+
 ## TwoPunctureABE uses fixed optimal flags with its own PGO profile, independent of CXXAPPFLAGS
 TP_PROFDATA = /home/$(shell whoami)/AMSS-NCKU/pgo_profile/TwoPunctureABE.profdata
 TP_OPTFLAGS = -O3 -xHost -fp-model fast=2 -fma -ipo \
@@ -81,7 +84,7 @@ C++FILES = ABE.o Ansorg.o Block.o misc.o monitor.o Parallel.o MPatch.o var.o\
 	   bssnEScalar_class.o perf.o Z4c_class.o NullShellPatch.o\
 	   bssnEM_class.o cpbc_util.o z4c_rhs_point.o checkpoint.o\
           Parallel_bam.o scalar_class.o transpbh.o NullShellPatch2.o\
-	   NullShellPatch2_Evo.o writefile_f.o
+	   NullShellPatch2_Evo.o writefile_f.o interp_lb_profile.o
 	   
 C++FILES_GPU = ABE.o Ansorg.o Block.o misc.o monitor.o Parallel.o MPatch.o var.o\
           cgh.o surface_integral.o ShellPatch.o\
--- a/AMSS_NCKU_source/makefile.inc
+++ b/AMSS_NCKU_source/makefile.inc
@@ -15,6 +15,20 @@ LDLIBS  = -L${MKLROOT}/lib -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lifcore
 ##   instrument : PGO Phase 1 instrumentation to collect fresh profile data
 PGO_MODE ?= opt

+## Interp_Points load balance profiling mode
+##   off        : (default) no load balance instrumentation
+##   profile    : Pass 1 — instrument Interp_Points to collect timing profile
+##   optimize   : Pass 2 — read profile and apply block rebalancing
+INTERP_LB_MODE ?= off
+
+ifeq ($(INTERP_LB_MODE),profile)
+INTERP_LB_FLAGS = -DINTERP_LB_PROFILE
+else ifeq ($(INTERP_LB_MODE),optimize)
+INTERP_LB_FLAGS = -DINTERP_LB_OPTIMIZE
+else
+INTERP_LB_FLAGS =
+endif
+
 ## Kernel implementation switch
 ##   1 (default) : use C++ rewrite of bssn_rhs and helper kernels (faster)
 ##   0           : fall back to original Fortran kernels
--- a/generate_interp_lb_header.py
+++ b/generate_interp_lb_header.py
@@ -0,0 +1,72 @@
+#!/usr/bin/env python3
+"""Convert interp_lb_profile.bin to a C header for compile-time embedding."""
+import struct, sys
+
+if len(sys.argv) < 3:
+    print(f"Usage: {sys.argv[0]} <profile.bin> <output.h>")
+    sys.exit(1)
+
+with open(sys.argv[1], 'rb') as f:
+    magic, version, nprocs, num_heavy = struct.unpack('IIii', f.read(16))
+    threshold = struct.unpack('d', f.read(8))[0]
+    times = list(struct.unpack(f'{nprocs}d', f.read(nprocs * 8)))
+    heavy = list(struct.unpack(f'{num_heavy}i', f.read(num_heavy * 4)))
+
+# For each heavy rank, compute split: left half -> lighter neighbor, right half -> heavy rank
+# (or vice versa depending on which neighbor is lighter)
+splits = []
+for hr in heavy:
+    prev_t = times[hr - 1] if hr > 0 else 1e30
+    next_t = times[hr + 1] if hr < nprocs - 1 else 1e30
+    if prev_t <= next_t:
+        splits.append((hr, hr - 1, hr))  # (block_id, r_left, r_right)
+    else:
+        splits.append((hr, hr, hr + 1))
+
+# Also remap the displaced neighbor blocks
+remaps = {}
+for hr, r_l, r_r in splits:
+    if r_l != hr:
+        # We took r_l's slot, so remap block r_l to its other neighbor
+        displaced = r_l
+        if displaced > 0 and displaced - 1 not in [s[0] for s in splits]:
+            remaps[displaced] = displaced - 1
+        elif displaced < nprocs - 1:
+            remaps[displaced] = displaced + 1
+    else:
+        displaced = r_r
+        if displaced < nprocs - 1 and displaced + 1 not in [s[0] for s in splits]:
+            remaps[displaced] = displaced + 1
+        elif displaced > 0:
+            remaps[displaced] = displaced - 1
+
+with open(sys.argv[2], 'w') as out:
+    out.write("/* Auto-generated from interp_lb_profile.bin — do not edit */\n")
+    out.write("#ifndef INTERP_LB_PROFILE_DATA_H\n")
+    out.write("#define INTERP_LB_PROFILE_DATA_H\n\n")
+    out.write(f"#define INTERP_LB_NPROCS {nprocs}\n")
+    out.write(f"#define INTERP_LB_NUM_HEAVY {num_heavy}\n\n")
+    out.write(f"static const int interp_lb_heavy_blocks[{num_heavy}] = {{")
+    out.write(", ".join(str(h) for h in heavy))
+    out.write("};\n\n")
+    out.write("/* Split table: {block_id, r_left, r_right} */\n")
+    out.write(f"static const int interp_lb_splits[{num_heavy}][3] = {{\n")
+    for bid, rl, rr in splits:
+        out.write(f"    {{{bid}, {rl}, {rr}}},\n")
+    out.write("};\n\n")
+    out.write("/* Rank remap for displaced neighbor blocks */\n")
+    out.write(f"static const int interp_lb_num_remaps = {len(remaps)};\n")
+    out.write(f"static const int interp_lb_remaps[][2] = {{\n")
+    for src, dst in sorted(remaps.items()):
+        out.write(f"    {{{src}, {dst}}},\n")
+    if not remaps:
+        out.write("    {-1, -1},\n")
+    out.write("};\n\n")
+    out.write("#endif /* INTERP_LB_PROFILE_DATA_H */\n")
+
+print(f"Generated {sys.argv[2]}:")
+print(f"  {num_heavy} heavy blocks to split: {heavy}")
+for bid, rl, rr in splits:
+    print(f"    block {bid}: split -> rank {rl} (left), rank {rr} (right)")
+for src, dst in sorted(remaps.items()):
+    print(f"    block {src}: remap -> rank {dst}")
--- a/makefile_and_run.py
+++ b/makefile_and_run.py
@@ -69,7 +69,7 @@ def makefile_ABE():

    ## Build command with CPU binding to nohz_full cores
    if (input_data.GPU_Calculation == "no"):
-        makefile_command  = f"{NUMACTL_CPU_BIND} make -j{BUILD_JOBS} ABE"
+        makefile_command  = f"{NUMACTL_CPU_BIND} make -j{BUILD_JOBS} INTERP_LB_MODE=optimize ABE"
    elif (input_data.GPU_Calculation == "yes"):
        makefile_command  = f"{NUMACTL_CPU_BIND} make -j{BUILD_JOBS} ABEGPU"
    else: