{ "date": "2026-10-22", "change": "Role suites (csrc/kernels/elastic/afdep_suites.hpp). Directional hybrid: the Attention dispatch launches the shared cubin without its forward warps; the FFN dispatch runs afdep_ffn_hybrid_dispatch.cuh (hybrid_dispatch.cuh without scale-out warps, generated); the Attention combine runs the one-CTA afdep_hybrid_collect.cuh. One NVLink domain: the FFN dispatch receive runs afdep_ffn_dispatch.cuh (dispatch.cuh in which only the notify warps trigger the copy epilogue, generated). Direct GIN keeps the shared kernels.", "native_source_sha256": { "final": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a", "pre_suites_8205d02b": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "hybrid_c7c4e90e": "c7c4e90e97048f899e3c099ca0276c6e877c99baf5bc7cb982ed25ae937e3915", "audit_comments_26cd2d09": "26cd2d09e3c0807925d9321eef4ac10d2f0bdc2fa3a0ff9dd0f65a05a2c97c69", "development_c8a74161": "c8a741613a6a9f890a15f8c24556f5035777257a02c14a2b1bec24fc874d975e", "ffn_arrival_trigger_fa0db53d": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "r10_candidate_a11fe415": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d" }, "device_diffs": { "development_c8a74161 -> hybrid_c7c4e90e": "identical device sources (git diff 2590e42 76cfd41 -- deep_ep is empty)", "hybrid_c7c4e90e -> audit_comments_26cd2d09": "afdep_hybrid_collect.cuh +5 comment lines only", "audit_comments_26cd2d09 -> final": "afdep_ffn_dispatch.cuh added (one-NVLink-domain FFN receive only); no other device file", "host": "c7c4e90e -> final also changes dispatch.hpp (direct template-args helper, FFN receive runtime), buffer.hpp (FFN receive planner) and afdep_suites.hpp (comments, AFDEP_SHARED_FFN_RECEIVE); the hybrid launch paths are unchanged" }, "afd_basis": "Masked ranks send and receive in every phase. On the role path the Attention never forwards (dispatch) or reduces/sends (combine) and the FFN never originates tokens, so those warp classes idle; with one CTA per SM they hold their SMs for the whole hybrid kernel, and in one NVLink domain the FFN's idle payload warps trigger the 132-CTA copy epilogue at receive entry.", "controls": [ "AFDEP_SHARED_ROLE_KERNELS=1", "AFDEP_SHARED_FFN_RECEIVE=1", "AFDEP_SHARED_HYBRID_SEND=1", "AFDEP_SHARED_HYBRID_FORWARD=1", "AFDEP_SHARED_HYBRID_COLLECT=1" ], "self_audit": { "review": "4 independent auditors and 1 adversarial verifier", "verdicts": [ { "id": "H1", "verdict": "confirmed" }, { "id": "H2", "verdict": "confirmed" }, { "id": "H3", "verdict": "confirmed" }, { "id": "H4", "verdict": "confirmed" }, { "id": "H5", "verdict": "confirmed" }, { "id": "H6", "verdict": "refuted" }, { "id": "DE-1", "verdict": "confirmed" }, { "id": "DE-2", "verdict": "confirmed" }, { "id": "DE-3", "verdict": "confirmed" }, { "id": "A", "verdict": "confirmed" }, { "id": "B", "verdict": "confirmed" }, { "id": "C", "verdict": "confirmed" }, { "id": "D", "verdict": "confirmed" }, { "id": "E", "verdict": "refuted" }, { "id": "F", "verdict": "confirmed" }, { "id": "G", "verdict": "refuted" }, { "id": "H", "verdict": "confirmed" }, { "id": "I", "verdict": "refuted" }, { "id": "J", "verdict": "refuted" }, { "id": "K", "verdict": "confirmed" }, { "id": "VG1", "verdict": "confirmed" }, { "id": "VG2", "verdict": "confirmed" }, { "id": "VG3", "verdict": "confirmed" }, { "id": "VG4", "verdict": "confirmed" }, { "id": "VG5", "verdict": "confirmed" }, { "id": "VG6", "verdict": "confirmed" }, { "id": "VG7", "verdict": "refuted" }, { "id": "VG8", "verdict": "refuted" }, { "id": "VG9", "verdict": "confirmed" } ], "addressed": "H1/VG6 screen output resolved; H5 --role-env limited to launch-shape controls; H2/H3/DE-1/DE-2/H4/K comments; A/B/F/H docs; C the screen driver checks every rank's digest per repeat; D/DE-3/VG4 fault markers, tree digests and negative controls; VG1/VG3/VG9 multi-domain layouts, hidden 7168/8192 and controls; VG2/VG5 suite records digest/env and cycles buckets" }, "baseline_profiles_pre_suites": { "native_source_sha256": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "one_domain_pokprod_4xH100": [ "## afd n4 F2 T1", "dev0: SM-us reserved per exchange (upper bound) 1967.8", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 10.91 us SM-us 261.9", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 11.62 us SM-us 1533.3", " 2:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 7.04 us SM-us 169.0", "dev1: SM-us reserved per exchange (upper bound) 1614.1", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 8.10 us SM-us 194.3", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 8.96 us SM-us 1182.7", " 2:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 9.76 us SM-us 234.2", "dev2: SM-us reserved per exchange (upper bound) 740.7", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 5.73 us SM-us 137.5", " 1:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 11.28 us SM-us 270.7", " 2:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 2.53 us SM-us 333.7", "dev3: SM-us reserved per exchange (upper bound) 735.7", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 5.70 us SM-us 136.7", " 1:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 11.23 us SM-us 269.6", " 2:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 2.50 us SM-us 329.5", "## masked n4 F2 T1", "dev0: SM-us reserved per exchange (upper bound) 1147.8", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 12.80 us SM-us 307.2", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 2.40 us SM-us 316.8", " 2:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 14.50 us SM-us 347.9", " 3:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 1.31 us SM-us 173.2", "dev1: SM-us reserved per exchange (upper bound) 1165.2", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 13.18 us SM-us 316.4", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 2.54 us SM-us 335.7", " 2:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 14.18 us SM-us 340.2", " 3:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 1.31 us SM-us 173.2", "dev2: SM-us reserved per exchange (upper bound) 1207.3", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 11.74 us SM-us 281.9", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 1.76 us SM-us 232.3", " 2:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 15.52 us SM-us 372.5", " 3:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 2.43 us SM-us 321.0", "dev3: SM-us reserved per exchange (upper bound) 1192.7", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 11.84 us SM-us 284.2", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 1.66 us SM-us 219.6", " 2:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 15.49 us SM-us 371.7", " 3:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 2.43 us SM-us 321.0", "## afd n4 F2 T512", "dev0: SM-us reserved per exchange (upper bound) 5294.3", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 21.82 us SM-us 523.8", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 29.02 us SM-us 3831.1", " 2:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 39.22 us SM-us 941.2", "dev1: SM-us reserved per exchange (upper bound) 5423.7", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 22.91 us SM-us 549.9", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 30.06 us SM-us 3968.4", " 2:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 37.60 us SM-us 902.4", "dev2: SM-us reserved per exchange (upper bound) 2039.8", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 17.94 us SM-us 430.5", " 1:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 47.53 us SM-us 1140.8", " 2:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 3.52 us SM-us 464.6", "dev3: SM-us reserved per exchange (upper bound) 2050.5", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 18.03 us SM-us 432.8", " 1:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 47.55 us SM-us 1141.2", " 2:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 3.58 us SM-us 473.1", "## masked n4 F2 T512", "dev0: SM-us reserved per exchange (upper bound) 3490.7", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 24.45 us SM-us 586.8", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 12.74 us SM-us 1681.2", " 2:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 43.23 us SM-us 1037.6", " 3:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 1.44 us SM-us 190.1", "dev1: SM-us reserved per exchange (upper bound) 3487.6", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 24.29 us SM-us 582.9", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 12.70 us SM-us 1676.9", " 2:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 43.47 us SM-us 1043.3", " 3:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 1.44 us SM-us 190.1", "dev2: SM-us reserved per exchange (upper bound) 2426.7", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 21.98 us SM-us 527.6", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 1.47 us SM-us 194.3", " 2:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 50.77 us SM-us 1218.4", " 3:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 3.62 us SM-us 477.3", "dev3: SM-us reserved per exchange (upper bound) 2412.7", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 21.95 us SM-us 526.8", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 1.34 us SM-us 177.4", " 2:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 50.91 us SM-us 1221.9", " 3:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 3.58 us SM-us 473.1", "## afd n3 F1 T1", "dev0: SM-us reserved per exchange (upper bound) 1616.4", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 8.10 us SM-us 194.3", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 9.09 us SM-us 1199.5", " 2:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 9.31 us SM-us 223.5", "dev1: SM-us reserved per exchange (upper bound) 733.8", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 5.73 us SM-us 137.5", " 1:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 10.97 us SM-us 263.4", " 2:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 2.53 us SM-us 333.7", "dev2: SM-us reserved per exchange (upper bound) 724.4", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 5.66 us SM-us 135.9", " 1:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 11.23 us SM-us 269.6", " 2:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 2.40 us SM-us 316.8", "## masked n3 F1 T1", "dev0: SM-us reserved per exchange (upper bound) 1252.2", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 13.95 us SM-us 334.8", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 2.59 us SM-us 342.1", " 2:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 16.66 us SM-us 399.7", " 3:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 1.31 us SM-us 173.2", "dev1: SM-us reserved per exchange (upper bound) 1271.0", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 12.70 us SM-us 304.9", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 1.60 us SM-us 211.2", " 2:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 18.14 us SM-us 435.5", " 3:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 2.43 us SM-us 321.0", "dev2: SM-us reserved per exchange (upper bound) 1280.4", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 12.51 us SM-us 300.3", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 1.73 us SM-us 228.1", " 2:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 18.11 us SM-us 434.7", " 3:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 2.43 us SM-us 321.0", "## afd n3 F1 T512", "dev0: SM-us reserved per exchange (upper bound) 5539.4", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 19.42 us SM-us 466.2", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 30.83 us SM-us 4069.7", " 2:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 41.79 us SM-us 1003.0", "dev1: SM-us reserved per exchange (upper bound) 2086.2", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 16.96 us SM-us 407.0", " 1:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 53.73 us SM-us 1289.4", " 2:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 2.94 us SM-us 388.6", "dev2: SM-us reserved per exchange (upper bound) 2086.4", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 16.99 us SM-us 407.8", " 1:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 53.62 us SM-us 1286.8", " 2:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 2.94 us SM-us 388.6", "## masked n3 F1 T512", "dev0: SM-us reserved per exchange (upper bound) 3909.3", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 24.80 us SM-us 595.2", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 13.28 us SM-us 1753.0", " 2:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 56.78 us SM-us 1362.8", " 3:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 1.50 us SM-us 198.5", "dev1: SM-us reserved per exchange (upper bound) 2961.0", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 22.94 us SM-us 550.6", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 2.62 us SM-us 346.4", " 2:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 68.93 us SM-us 1654.2", " 3:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 3.04 us SM-us 401.3", "dev2: SM-us reserved per exchange (upper bound) 2876.9", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 23.04 us SM-us 553.0", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 2.05 us SM-us 270.3", " 2:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 68.93 us SM-us 1654.2", " 3:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 3.01 us SM-us 397.1" ], "two_nodes_waldorf_hybrid": [ "## afd T1 node0 (afdep-v21-xa; node0 = FFN)", "dev0: SM-us reserved per exchange (upper bound) 1897.6", " 0:hybrid_dispatch grid 24 x 640 thr smem 227.0 KiB coop 0 dur 43.77 us SM-us 1050.6", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 2.34 us SM-us 308.4", " 2:hybrid_combine grid 24 x 512 thr smem 227.0 KiB coop 0 dur 22.21 us SM-us 533.0", " 3:afdep_ffn_complete grid 1 x 128 thr smem 0.0 KiB coop 0 dur 1.98 us SM-us 2.0", "dev1: SM-us reserved per exchange (upper bound) 2057.6", " 0:hybrid_dispatch grid 24 x 640 thr smem 227.0 KiB coop 0 dur 45.47 us SM-us 1091.4", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 3.49 us SM-us 460.4", " 2:hybrid_combine grid 24 x 512 thr smem 227.0 KiB coop 0 dur 20.99 us SM-us 503.8", " 3:afdep_ffn_complete grid 1 x 128 thr smem 0.0 KiB coop 0 dur 1.95 us SM-us 2.0", "anchor device 0 dispatch start = 0; period median 73.12 us", " dev0 0:dispatch_impl start 0.00 end 43.77 dur 43.77", " dev0 1:dispatch_copy_epilogue start 43.78 end 46.14 dur 2.34", " dev0 2:combine_impl start 46.43 end 68.69 dur 22.21", " dev0 3:afdep_ffn_complete_impl start 68.77 end 70.75 dur 1.98", " dev1 0:dispatch_impl start -1.37 end 44.08 dur 45.47", " dev1 1:dispatch_copy_epilogue start 42.64 end 46.17 dur 3.49", " dev1 2:combine_impl start 46.47 end 67.52 dur 20.99", " dev1 3:afdep_ffn_complete_impl start 67.59 end 69.56 dur 1.95", "## afd T1 node1 (afdep-v21-xb; node0 = FFN)", "dev0: SM-us reserved per exchange (upper bound) 1992.6", " 0:hybrid_dispatch grid 24 x 640 thr smem 227.0 KiB coop 0 dur 9.22 us SM-us 221.2", " 1:afdep_source_finalize grid 1 x 128 thr smem 0.0 KiB coop 0 dur 1.36 us SM-us 1.4", " 2:hybrid_combine grid 24 x 512 thr smem 227.0 KiB coop 0 dur 56.58 us SM-us 1357.8", " 3:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 3.14 us SM-us 414.0", "dev1: SM-us reserved per exchange (upper bound) 1993.3", " 0:hybrid_dispatch grid 24 x 640 thr smem 227.0 KiB coop 0 dur 9.09 us SM-us 218.1", " 1:afdep_source_finalize grid 1 x 128 thr smem 0.0 KiB coop 0 dur 1.34 us SM-us 1.3", " 2:hybrid_combine grid 24 x 512 thr smem 227.0 KiB coop 0 dur 56.64 us SM-us 1359.3", " 3:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 3.14 us SM-us 414.0", "anchor device 0 dispatch start = 0; period median 73.18 us", " dev0 0:dispatch_impl start 0.00 end 9.22 dur 9.22", " dev0 1:afdep_source_finalize_impl start 9.50 end 10.88 dur 1.36", " dev0 2:combine_impl start 11.17 end 67.74 dur 56.58", " dev0 3:combine_reduce_epilogue start 67.79 end 70.91 dur 3.14", " dev1 0:dispatch_impl start -0.83 end 8.29 dur 9.09", " dev1 1:afdep_source_finalize_impl start 8.56 end 9.90 dur 1.34", " dev1 2:combine_impl start 10.22 end 66.89 dur 56.64", " dev1 3:combine_reduce_epilogue start 66.94 end 70.08 dur 3.14", "## masked T1 node0 (afdep-v21-xa; node0 = FFN)", "dev0: SM-us reserved per exchange (upper bound) 2623.7", " 0:hybrid_dispatch grid 24 x 640 thr smem 227.0 KiB coop 0 dur 48.74 us SM-us 1169.7", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 2.30 us SM-us 304.1", " 2:hybrid_combine grid 24 x 512 thr smem 227.0 KiB coop 0 dur 35.23 us SM-us 845.6", " 3:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 2.24 us SM-us 295.7", "dev1: SM-us reserved per exchange (upper bound) 2737.7", " 0:hybrid_dispatch grid 24 x 640 thr smem 227.0 KiB coop 0 dur 49.44 us SM-us 1186.6", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 3.31 us SM-us 437.2", " 2:hybrid_combine grid 24 x 512 thr smem 227.0 KiB coop 0 dur 34.75 us SM-us 834.0", " 3:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 2.18 us SM-us 287.2", "anchor device 0 dispatch start = 0; period median 91.09 us", " dev0 0:dispatch_impl start 0.00 end 48.74 dur 48.74", " dev0 1:dispatch_copy_epilogue start 48.53 end 51.01 dur 2.30", " dev0 2:combine_impl start 51.33 end 86.51 dur 35.23", " dev0 3:combine_reduce_epilogue start 86.55 end 88.78 dur 2.24", " dev1 0:dispatch_impl start -0.39 end 48.94 dur 49.44", " dev1 1:dispatch_copy_epilogue start 47.29 end 51.05 dur 3.31", " dev1 2:combine_impl start 51.34 end 86.07 dur 34.75", " dev1 3:combine_reduce_epilogue start 86.12 end 88.28 dur 2.18", "## masked T1 node1 (afdep-v21-xb; node0 = FFN)", "dev0: SM-us reserved per exchange (upper bound) 2679.6", " 0:hybrid_dispatch grid 24 x 640 thr smem 227.0 KiB coop 0 dur 38.37 us SM-us 920.8", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 1.98 us SM-us 261.9", " 2:hybrid_combine grid 24 x 512 thr smem 227.0 KiB coop 0 dur 45.50 us SM-us 1092.1", " 3:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 3.10 us SM-us 409.7", "dev1: SM-us reserved per exchange (upper bound) 2647.1", " 0:hybrid_dispatch grid 24 x 640 thr smem 227.0 KiB coop 0 dur 39.14 us SM-us 939.3", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 1.66 us SM-us 219.6", " 2:hybrid_combine grid 24 x 512 thr smem 227.0 KiB coop 0 dur 44.58 us SM-us 1069.8", " 3:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 3.14 us SM-us 414.0", "anchor device 0 dispatch start = 0; period median 90.94 us", " dev0 0:dispatch_impl start 0.00 end 38.37 dur 38.37", " dev0 1:dispatch_copy_epilogue start 37.79 end 39.68 dur 1.98", " dev0 2:combine_impl start 39.97 end 85.38 dur 45.50", " dev0 3:combine_reduce_epilogue start 85.41 end 88.51 dur 3.10", " dev1 0:dispatch_impl start -1.07 end 38.12 dur 39.14", " dev1 1:dispatch_copy_epilogue start 38.11 end 39.77 dur 1.66", " dev1 2:combine_impl start 40.08 end 84.61 dur 44.58", " dev1 3:combine_reduce_epilogue start 84.64 end 87.79 dur 3.14", "## afd T512 node0 (afdep-v21-xa; node0 = FFN)", "dev0: SM-us reserved per exchange (upper bound) 5702.9", " 0:hybrid_dispatch grid 24 x 640 thr smem 227.0 KiB coop 0 dur 135.50 us SM-us 3252.1", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 8.50 us SM-us 1121.5", " 2:hybrid_combine grid 24 x 512 thr smem 227.0 KiB coop 0 dur 54.43 us SM-us 1306.4", " 3:afdep_ffn_complete grid 1 x 128 thr smem 0.0 KiB coop 0 dur 2.08 us SM-us 2.1", "dev1: SM-us reserved per exchange (upper bound) 5626.8", " 0:hybrid_dispatch grid 24 x 640 thr smem 227.0 KiB coop 0 dur 135.78 us SM-us 3258.6", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 7.97 us SM-us 1051.8", " 2:hybrid_combine grid 24 x 512 thr smem 227.0 KiB coop 0 dur 54.56 us SM-us 1309.4", " 3:afdep_ffn_complete grid 1 x 128 thr smem 0.0 KiB coop 0 dur 2.08 us SM-us 2.1", "anchor device 0 dispatch start = 0; period median 202.55 us", " dev0 0:dispatch_impl start 0.00 end 135.50 dur 135.50", " dev0 1:dispatch_copy_epilogue start 133.14 end 143.22 dur 8.50", " dev0 2:combine_impl start 143.50 end 198.02 dur 54.43", " dev0 3:afdep_ffn_complete_impl start 198.19 end 200.29 dur 2.08", " dev1 0:dispatch_impl start -0.24 end 135.44 dur 135.78", " dev1 1:dispatch_copy_epilogue start 134.18 end 143.07 dur 7.97", " dev1 2:combine_impl start 143.38 end 197.88 dur 54.56", " dev1 3:afdep_ffn_complete_impl start 198.06 end 200.19 dur 2.08", "## afd T512 node1 (afdep-v21-xb; node0 = FFN)", "dev0: SM-us reserved per exchange (upper bound) 5145.7", " 0:hybrid_dispatch grid 24 x 640 thr smem 227.0 KiB coop 0 dur 23.81 us SM-us 571.4", " 1:afdep_source_finalize grid 1 x 128 thr smem 0.0 KiB coop 0 dur 1.41 us SM-us 1.4", " 2:hybrid_combine grid 24 x 512 thr smem 227.0 KiB coop 0 dur 170.19 us SM-us 4084.6", " 3:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 3.65 us SM-us 481.5", "dev1: SM-us reserved per exchange (upper bound) 5149.1", " 0:hybrid_dispatch grid 24 x 640 thr smem 227.0 KiB coop 0 dur 23.26 us SM-us 558.3", " 1:afdep_source_finalize grid 1 x 128 thr smem 0.0 KiB coop 0 dur 1.38 us SM-us 1.4", " 2:hybrid_combine grid 24 x 512 thr smem 227.0 KiB coop 0 dur 171.26 us SM-us 4110.3", " 3:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 3.65 us SM-us 481.5", "anchor device 0 dispatch start = 0; period median 202.14 us", " dev0 0:dispatch_impl start 0.00 end 23.81 dur 23.81", " dev0 1:afdep_source_finalize_impl start 24.14 end 25.54 dur 1.41", " dev0 2:combine_impl start 25.89 end 196.13 dur 170.19", " dev0 3:combine_reduce_epilogue start 196.16 end 199.84 dur 3.65", " dev1 0:dispatch_impl start 0.52 end 23.61 dur 23.26", " dev1 1:afdep_source_finalize_impl start 23.96 end 25.36 dur 1.38", " dev1 2:combine_impl start 25.68 end 197.01 dur 171.26", " dev1 3:combine_reduce_epilogue start 197.06 end 200.69 dur 3.65", "## masked T512 node0 (afdep-v21-xa; node0 = FFN)", "dev0: SM-us reserved per exchange (upper bound) 6324.0", " 0:hybrid_dispatch grid 24 x 640 thr smem 227.0 KiB coop 0 dur 137.86 us SM-us 3308.5", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 8.10 us SM-us 1068.7", " 2:hybrid_combine grid 24 x 512 thr smem 227.0 KiB coop 0 dur 67.38 us SM-us 1617.0", " 3:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 2.24 us SM-us 295.7", "dev1: SM-us reserved per exchange (upper bound) 6362.3", " 0:hybrid_dispatch grid 24 x 640 thr smem 227.0 KiB coop 0 dur 138.42 us SM-us 3322.0", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 8.56 us SM-us 1129.9", " 2:hybrid_combine grid 24 x 512 thr smem 227.0 KiB coop 0 dur 67.39 us SM-us 1617.4", " 3:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 2.21 us SM-us 291.5", "anchor device 0 dispatch start = 0; period median 218.31 us", " dev0 0:dispatch_impl start 0.00 end 137.86 dur 137.86", " dev0 1:dispatch_copy_epilogue start 137.04 end 145.74 dur 8.10", " dev0 2:combine_impl start 146.03 end 213.58 dur 67.38", " dev0 3:combine_reduce_epilogue start 213.67 end 215.89 dur 2.24", " dev1 0:dispatch_impl start -0.20 end 137.87 dur 138.42", " dev1 1:dispatch_copy_epilogue start 136.26 end 145.46 dur 8.56", " dev1 2:combine_impl start 145.77 end 213.58 dur 67.39", " dev1 3:combine_reduce_epilogue start 213.64 end 215.89 dur 2.21", "## masked T512 node1 (afdep-v21-xb; node0 = FFN)", "dev0: SM-us reserved per exchange (upper bound) 5766.7", " 0:hybrid_dispatch grid 24 x 640 thr smem 227.0 KiB coop 0 dur 45.33 us SM-us 1087.9", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 1.54 us SM-us 202.8", " 2:hybrid_combine grid 24 x 512 thr smem 227.0 KiB coop 0 dur 165.14 us SM-us 3963.3", " 3:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 3.62 us SM-us 477.3", "dev1: SM-us reserved per exchange (upper bound) 5778.0", " 0:hybrid_dispatch grid 24 x 640 thr smem 227.0 KiB coop 0 dur 44.82 us SM-us 1075.6", " 1:dispatch_copy_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 1.57 us SM-us 207.0", " 2:hybrid_combine grid 24 x 512 thr smem 227.0 KiB coop 0 dur 165.17 us SM-us 3964.0", " 3:combine_reduce_epilogue grid 132 x 1024 thr smem 227.0 KiB coop 0 dur 3.71 us SM-us 490.0", "anchor device 0 dispatch start = 0; period median 218.24 us", " dev0 0:dispatch_impl start 0.00 end 45.33 dur 45.33", " dev0 1:dispatch_copy_epilogue start 44.75 end 46.59 dur 1.54", " dev0 2:combine_impl start 46.90 end 212.34 dur 165.14", " dev0 3:combine_reduce_epilogue start 212.35 end 215.95 dur 3.62", " dev1 0:dispatch_impl start 0.73 end 45.28 dur 44.82", " dev1 1:dispatch_copy_epilogue start 44.87 end 46.70 dur 1.57", " dev1 2:combine_impl start 46.98 end 212.53 dur 165.17", " dev1 3:combine_reduce_epilogue start 212.56 end 216.27 dur 3.71" ] }, "hybrid": { "resources_upper_bound_8a8f_piggy": { "native_source_sha256": "c8a741613a6a9f890a15f8c24556f5035777257a02c14a2b1bec24fc874d975e", "mean_over_8_gpus_per_role": { "masked T=1 ffn": { "mean": 2739, "min": 2516, "max": 2907, "devices": 8 }, "masked T=1 attention": { "mean": 2582, "min": 2503, "max": 2632, "devices": 8 }, "masked T=512 ffn": { "mean": 7235, "min": 6926, "max": 7455, "devices": 8 }, "masked T=512 attention": { "mean": 6426, "min": 6365, "max": 6495, "devices": 8 }, "shared T=1 ffn": { "mean": 1878, "min": 1702, "max": 2087, "devices": 8 }, "shared T=1 attention": { "mean": 1705, "min": 1693, "max": 1723, "devices": 8 }, "shared T=512 ffn": { "mean": 6553, "min": 6320, "max": 6820, "devices": 8 }, "shared T=512 attention": { "mean": 5419, "min": 5411, "max": 5427, "devices": 8 }, "split T=1 ffn": { "mean": 1928, "min": 1723, "max": 2136, "devices": 8 }, "split T=1 attention": { "mean": 699, "min": 683, "max": 714, "devices": 8 }, "split T=512 ffn": { "mean": 6557, "min": 6318, "max": 6741, "devices": 8 }, "split T=512 attention": { "mean": 1257, "min": 1242, "max": 1271, "devices": 8 } } }, "pre_suites_attention_combine_cta_lifetimes": { "native_source_sha256": "scratch pre-suites tree (8205d02b + per-CTA timer, see hlife-patch)", "result": [ "T=1: exchanges 32 ctas 24: kernel span median 59.2 us; CTA lifetime median 43.3 us; used SM-us median 1049 vs reserved 24 x span = 1420 (74% used)", "T=512: exchanges 32 ctas 24: kernel span median 196.2 us; CTA lifetime median 187.6 us; used SM-us median 4416 vs reserved 24 x span = 4710 (94% used)" ], "patch": "\"\"\"hlife-patch.py : scratch per-CTA lifetime stamps in hybrid_combine.cuh (Attention domains only).\"\"\"\nimport sys\np = sys.argv[1] + \"/afdep/third_party/deepep_v2/deep_ep/include/deep_ep/impls/hybrid_combine.cuh\"\ns = open(p).read()\nold = \" // Utils\\n const auto sm_idx = static_cast(blockIdx.x);\\n\"\nassert s.count(old) == 1\ns = s.replace(old, \" unsigned long long afd_t0; asm volatile(\\\"mov.u64 %0, %%globaltimer;\\\" : \\\"=l\\\"(afd_t0)); // SCRATCH\\n\" + old)\nold = \" // No barrier at epilogue\\n}\"\nassert s.count(old) == 1\ns = s.replace(old, \"\"\" // No barrier at epilogue\n // SCRATCH: CTA lifetime (last warp) on Attention domains.\n __syncthreads();\n if (threadIdx.x == 0 and kAfdFfnScaleoutRanks > 0 and scaleout_rank_idx >= kAfdFfnScaleoutRanks and scaleup_rank_idx == 0) {\n unsigned long long t1; asm volatile(\"mov.u64 %0, %%globaltimer;\" : \"=l\"(t1));\n printf(\"AFDLIFE cta %d start %llu end %llu\\\\n\", (int)blockIdx.x, afd_t0, t1);\n }\n}\"\"\")\nopen(p, \"w\").write(s)\nprint(\"patched\", p)\n" }, "overlap_8a8f_statler": { "native_source_sha256": { "afd": [ "c7c4e90e97048f899e3c099ca0276c6e877c99baf5bc7cb982ed25ae937e3915" ], "afd_shared": [ "c7c4e90e97048f899e3c099ca0276c6e877c99baf5bc7cb982ed25ae937e3915" ], "masked": [ "c7c4e90e97048f899e3c099ca0276c6e877c99baf5bc7cb982ed25ae937e3915" ] }, "per_repeat": { "afd": { "1": { "attention.gemm_us_per_gemm": [ 26.14, 26.51, 26.53 ], "attention.exchange_us": [ 127.65, 119.21, 118.8 ], "attention.makespan_us": [ 25529.66, 23841.49, 23759.02 ], "ffn.gemm_us_per_gemm": [ 34.67, 34.21, 33.91 ], "ffn.exchange_us": [ 127.53, 119.22, 118.76 ], "ffn.makespan_us": [ 25505.23, 23843.44, 23752.46 ] }, "32": { "attention.gemm_us_per_gemm": [ 26.41, 26.22, 26.04 ], "attention.exchange_us": [ 142.75, 142.06, 143.98 ], "attention.makespan_us": [ 28549.58, 28411.12, 28795.63 ], "ffn.gemm_us_per_gemm": [ 33.41, 33.33, 33.73 ], "ffn.exchange_us": [ 142.66, 142.03, 144.01 ], "ffn.makespan_us": [ 28532.24, 28405.34, 28801.44 ] }, "512": { "attention.gemm_us_per_gemm": [ 25.87, 26.08, 26.19 ], "attention.exchange_us": [ 365.55, 365.87, 367.61 ], "attention.makespan_us": [ 73110.8, 73174.27, 73521.47 ], "ffn.gemm_us_per_gemm": [ 36.78, 36.7, 36.88 ], "ffn.exchange_us": [ 365.41, 365.73, 367.49 ], "ffn.makespan_us": [ 73081.06, 73145.73, 73498.32 ] } }, "afd_shared": { "1": { "attention.gemm_us_per_gemm": [ 34.35, 34.06, 33.83 ], "attention.exchange_us": [ 130.93, 126.98, 123.72 ], "attention.makespan_us": [ 26186.43, 25396.7, 24744.29 ], "ffn.gemm_us_per_gemm": [ 35.04, 34.54, 34.56 ], "ffn.exchange_us": [ 130.72, 126.83, 123.43 ], "ffn.makespan_us": [ 26144.83, 25365.2, 24685.97 ] }, "32": { "attention.gemm_us_per_gemm": [ 32.92, 32.42, 32.55 ], "attention.exchange_us": [ 152.27, 145.92, 146.18 ], "attention.makespan_us": [ 30454.54, 29184.43, 29236.77 ], "ffn.gemm_us_per_gemm": [ 33.66, 33.43, 32.81 ], "ffn.exchange_us": [ 152.01, 145.85, 146.01 ], "ffn.makespan_us": [ 30401.49, 29169.82, 29201.71 ] }, "512": { "attention.gemm_us_per_gemm": [ 35.79, 35.68, 35.69 ], "attention.exchange_us": [ 370.55, 370.41, 368.24 ], "attention.makespan_us": [ 74110.7, 74081.36, 73648.8 ], "ffn.gemm_us_per_gemm": [ 37.25, 37.69, 37.22 ], "ffn.exchange_us": [ 370.12, 370.21, 368.0 ], "ffn.makespan_us": [ 74024.98, 74042.45, 73600.48 ] } }, "masked": { "1": { "attention.gemm_us_per_gemm": [ 34.03, 33.34, 34.53 ], "attention.exchange_us": [ 144.0, 138.46, 149.59 ], "attention.makespan_us": [ 28800.51, 27691.86, 29918.38 ], "ffn.gemm_us_per_gemm": [ 34.95, 34.9, 35.5 ], "ffn.exchange_us": [ 143.89, 138.43, 149.45 ], "ffn.makespan_us": [ 28778.42, 27686.93, 29890.91 ] }, "32": { "attention.gemm_us_per_gemm": [ 32.97, 32.32, 33.02 ], "attention.exchange_us": [ 176.7, 172.22, 177.34 ], "attention.makespan_us": [ 35340.24, 34443.81, 35467.73 ], "ffn.gemm_us_per_gemm": [ 35.31, 35.22, 35.42 ], "ffn.exchange_us": [ 176.45, 172.19, 177.23 ], "ffn.makespan_us": [ 35289.23, 34438.96, 35445.97 ] }, "512": { "attention.gemm_us_per_gemm": [ 35.76, 35.81, 36.05 ], "attention.exchange_us": [ 393.61, 382.81, 382.67 ], "attention.makespan_us": [ 78722.21, 76562.38, 76533.52 ], "ffn.gemm_us_per_gemm": [ 36.3, 36.56, 36.38 ], "ffn.exchange_us": [ 393.3, 382.61, 382.26 ], "ffn.makespan_us": [ 78659.58, 76521.07, 76452.82 ] } } } }, "screens": { "final_digest_8a8f_statler": { "arms": { "afd": { "_digest": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" }, "afd_shared": { "_env": { "AFDEP_SHARED_ROLE_KERNELS": "1" }, "_digest": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" }, "masked": { "role_aware": false } }, "passthrough": [ "--payload", "bf16", "--routing", "random", "--hidden", "2048", "--tokens", "1", "8", "32", "128", "512", "--capacity", "2048", "--hybrid", "on", "--ffn-ranks", "8", "--attention-ranks", "8" ], "processes": [ { "order": [ "afd", "afd_shared", "masked" ], "median_us": { "afd": { "1": 79.22784090042114, "8": 90.96863746643066, "32": 91.4625597000122, "128": 111.53167724609375, "512": 233.65896224975586 }, "afd_shared": { "1": 79.67039823532104, "8": 91.24871730804443, "32": 91.46664142608643, "128": 112.10736274719238, "512": 234.5895290374756 }, "masked": { "1": 97.55935668945312, "8": 107.33560085296631, "32": 112.71016120910645, "128": 129.72352027893066, "512": 249.0946388244629 } } }, { "order": [ "afd_shared", "masked", "afd" ], "median_us": { "afd_shared": { "1": 79.4214391708374, "8": 92.23504066467285, "32": 91.60183906555176, "128": 111.88440322875977, "512": 234.694242477417 }, "masked": { "1": 97.39168167114258, "8": 107.83535957336426, "32": 112.91312217712402, "128": 130.01880168914795, "512": 250.46095848083496 }, "afd": { "1": 79.18616056442261, "8": 90.83111763000488, "32": 91.0159969329834, "128": 110.9000015258789, "512": 234.18496131896973 } } }, { "order": [ "masked", "afd", "afd_shared" ], "median_us": { "masked": { "1": 97.71519660949707, "8": 107.0143175125122, "32": 112.90656089782715, "128": 129.53807830810547, "512": 249.74063873291016 }, "afd": { "1": 79.39976215362549, "8": 92.09367752075195, "32": 91.76623821258545, "128": 111.5450382232666, "512": 234.7661590576172 }, "afd_shared": { "1": 79.45336103439331, "8": 92.53576278686523, "32": 92.16719627380371, "128": 112.06023693084717, "512": 235.88128089904785 } } } ], "table": [ { "tokens": 1, "afd": { "median_us": 79.22784090042114, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 79.45336103439331, "paired_change_pct": [ 0.5585881552119254, 0.29712086649709146, 0.06750508983153214 ], "geomean_change_pct": 0.30753742477516166, "repeats_faster": 0 }, "masked": { "median_us": 97.55935668945312, "paired_change_pct": [ 23.13771974686558, 22.990786492178405, 23.06736690272979 ], "geomean_change_pct": 23.065276419063864, "repeats_faster": 0 } }, { "tokens": 8, "afd": { "median_us": 90.96863746643066, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 92.23504066467285, "paired_change_pct": [ 0.30788615660768404, 1.5456410438400292, 0.48003867150778756 ], "geomean_change_pct": 0.7763719270361902, "repeats_faster": 0 }, "masked": { "median_us": 107.33560085296631, "paired_change_pct": [ 17.99187482892155, 18.720723015459463, 16.201589939111848 ], "geomean_change_pct": 17.63328766293335, "repeats_faster": 0 } }, { "tokens": 32, "afd": { "median_us": 91.4625597000122, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 91.60183906555176, "paired_change_pct": [ 0.00446272888885435, 0.6436694123119091, 0.4369341808361016 ], "geomean_change_pct": 0.3613351576803092, "repeats_faster": 0 }, "masked": { "median_us": 112.90656089782715, "paired_change_pct": [ 23.23092812926315, 24.058545730443235, 23.037146446243195 ], "geomean_change_pct": 23.44141323208877, "repeats_faster": 0 } }, { "tokens": 128, "afd": { "median_us": 111.53167724609375, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 112.06023693084717, "paired_change_pct": [ 0.5161632240393788, 0.8876480517010066, 0.4618750558400997 ], "geomean_change_pct": 0.6217176808703595, "repeats_faster": 0 }, "masked": { "median_us": 129.72352027893066, "paired_change_pct": [ 16.31092034301318, 17.23967529324839, 16.130739987577325 ], "geomean_change_pct": 16.559434260010097, "repeats_faster": 0 } }, { "tokens": 512, "afd": { "median_us": 234.18496131896973, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 234.694242477417, "paired_change_pct": [ 0.3982585468838451, 0.217469625538258, 0.4749925823666068 ], "geomean_change_pct": 0.363515504630163, "repeats_faster": 0 }, "masked": { "median_us": 249.74063873291016, "paired_change_pct": [ 6.606070841917022, 6.950060785370682, 6.378466017164719 ], "geomean_change_pct": 6.6446071424763, "repeats_faster": 0 } } ], "native_source_sha256": { "afd": [ { "repeat": 0, "ranks": 16, "digest": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" }, { "repeat": 1, "ranks": 16, "digest": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" }, { "repeat": 2, "ranks": 16, "digest": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" } ], "afd_shared": [ { "repeat": 0, "ranks": 16, "digest": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" }, { "repeat": 1, "ranks": 16, "digest": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" }, { "repeat": 2, "ranks": 16, "digest": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" } ], "masked": [ { "repeat": 0, "ranks": 16, "digest": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" }, { "repeat": 1, "ranks": 16, "digest": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" }, { "repeat": 2, "ranks": 16, "digest": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" } ] }, "pods": [ "afdep-v21-s8b", "afdep-v21-s8c" ], "topology": "2 pods (soc) x 8 GPUs, InfiniBand, GIN" }, "c7c4e90e_two_nodes_waldorf": { "arms": { "afd": {}, "afd_shared": { "_env": { "AFDEP_SHARED_ROLE_KERNELS": "1" } }, "head": { "_label": "head" }, "masked": { "role_aware": false } }, "passthrough": [ "--payload bf16 --routing random --hidden 2048 --tokens 1 8 32 128 512 --capacity 2048 --hybrid on" ], "processes": [ { "order": [ "afd", "afd_shared", "head", "masked" ], "median_us": { "afd": { "1": 70.94352006912231, "8": 77.80752182006836, "32": 80.48456192016602, "128": 101.46391868591309, "512": 199.23888206481934 }, "afd_shared": { "1": 71.39031887054443, "8": 78.67271900177002, "32": 80.79008102416992, "128": 102.34111785888672, "512": 199.70383644104004 }, "head": { "1": 71.30543947219849, "8": 78.56096029281616, "32": 81.01480007171631, "128": 102.74159908294678, "512": 199.36039924621582 }, "masked": { "1": 88.2692003250122, "8": 98.28999996185303, "32": 104.2352819442749, "128": 121.03911876678467, "512": 216.05448722839355 } } }, { "order": [ "afd_shared", "head", "masked", "afd" ], "median_us": { "afd_shared": { "1": 71.30824089050293, "8": 78.69879961013794, "32": 80.98984241485596, "128": 102.14911937713623, "512": 199.4970417022705 }, "head": { "1": 71.632559299469, "8": 78.77648115158081, "32": 81.19431972503662, "128": 103.02703857421875, "512": 199.4103240966797 }, "masked": { "1": 88.17247867584229, "8": 98.24023723602295, "32": 104.22056198120117, "128": 120.76560020446777, "512": 215.49904823303223 }, "afd": { "1": 71.19752168655396, "8": 78.00559997558594, "32": 80.22216320037842, "128": 101.68263912200928, "512": 199.34463500976562 } } }, { "order": [ "head", "masked", "afd", "afd_shared" ], "median_us": { "head": { "1": 71.61759853363037, "8": 78.90640020370483, "32": 81.1346435546875, "128": 102.54424095153809, "512": 199.95967864990234 }, "masked": { "1": 88.46583843231201, "8": 98.4114408493042, "32": 104.37071800231934, "128": 121.21511936187744, "512": 216.51535987854004 }, "afd": { "1": 71.18040084838867, "8": 78.21184158325195, "32": 80.70511817932129, "128": 101.80967807769775, "512": 199.3656826019287 }, "afd_shared": { "1": 71.50184154510498, "8": 78.71263980865479, "32": 80.91248035430908, "128": 102.38783836364746, "512": 200.01416206359863 } } } ], "table": [ { "tokens": 1, "afd": { "median_us": 71.18040084838867, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 71.39031887054443, "paired_change_pct": [ 0.6297950834506016, 0.15550991288209026, 0.4515859602996164 ], "geomean_change_pct": 0.4121063837712491, "repeats_faster": 0 }, "head": { "median_us": 71.61759853363037, "paired_change_pct": [ 0.5101514595322376, 0.6110291518717226, 0.6142107659282692 ], "geomean_change_pct": 0.578452182096223, "repeats_faster": 0 }, "masked": { "median_us": 88.2692003250122, "paired_change_pct": [ 24.421793898877553, 23.842061615599942, 24.28398460517329 ], "geomean_change_pct": 24.182366972249515, "repeats_faster": 0 } }, { "tokens": 8, "afd": { "median_us": 78.00559997558594, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 78.69879961013794, "paired_change_pct": [ 1.111971132691325, 0.8886536796960165, 0.640309977703013 ], "geomean_change_pct": 0.8801276395820068, "repeats_faster": 0 }, "head": { "median_us": 78.77648115158081, "paired_change_pct": [ 0.9683362933601058, 0.9882382498642928, 0.8880479047582046 ], "geomean_change_pct": 0.9481981911518833, "repeats_faster": 0 }, "masked": { "median_us": 98.28999996185303, "paired_change_pct": [ 26.324547630691608, 25.939980292145705, 25.82677872960062 ], "geomean_change_pct": 26.030255601708376, "repeats_faster": 0 } }, { "tokens": 32, "afd": { "median_us": 80.48456192016602, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 80.91248035430908, "paired_change_pct": [ 0.37959963589906387, 0.9569415531217151, 0.2569380724120274 ], "geomean_change_pct": 0.5306970326392824, "repeats_faster": 0 }, "head": { "median_us": 81.1346435546875, "paired_change_pct": [ 0.6588072779426302, 1.2118303544494946, 0.5322157814227291 ], "geomean_change_pct": 0.8005197018534638, "repeats_faster": 0 }, "masked": { "median_us": 104.2352819442749, "paired_change_pct": [ 29.50965931536984, 29.91492353662877, 29.323542740393105 ], "geomean_change_pct": 29.582473451639157, "repeats_faster": 0 } }, { "tokens": 128, "afd": { "median_us": 101.68263912200928, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 102.34111785888672, "paired_change_pct": [ 0.8645429669329463, 0.4587609636756307, 0.5678834241165998 ], "geomean_change_pct": 0.6302498028267989, "repeats_faster": 0 }, "head": { "median_us": 102.74159908294678, "paired_change_pct": [ 1.2592460586790644, 1.322152398696419, 0.7215059390323741 ], "geomean_change_pct": 1.100608377074952, "repeats_faster": 0 }, "masked": { "median_us": 121.03911876678467, "paired_change_pct": [ 19.292769621354424, 18.767177216516572, 19.06050745919272 ], "geomean_change_pct": 19.03995714349702, "repeats_faster": 0 } }, { "tokens": 512, "afd": { "median_us": 199.34463500976562, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 199.70383644104004, "paired_change_pct": [ 0.2333652806129738, 0.07645387220851507, 0.3252713572399202 ], "geomean_change_pct": 0.21164417149133197, "repeats_faster": 0 }, "head": { "median_us": 199.4103240966797, "paired_change_pct": [ 0.060990696262264166, 0.032952523106954956, 0.2979429760535446 ], "geomean_change_pct": 0.13055822149392604, "repeats_faster": 0 }, "masked": { "median_us": 216.05448722839355, "paired_change_pct": [ 8.439921459759802, 8.10376121859273, 8.60212101340123 ], "geomean_change_pct": 8.381735744282581, "repeats_faster": 0 } } ], "native_source_sha256": { "afd": "c7c4e90e97048f899e3c099ca0276c6e877c99baf5bc7cb982ed25ae937e3915", "afd_shared": "c7c4e90e97048f899e3c099ca0276c6e877c99baf5bc7cb982ed25ae937e3915", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "c7c4e90e97048f899e3c099ca0276c6e877c99baf5bc7cb982ed25ae937e3915" }, "pods": [ "afdep-v21-xa", "afdep-v21-xb" ], "topology": "2 waldorf nodes x 2 H200, InfiniBand, GIN" }, "c7c4e90e_2F1A_waldorf": { "arms": { "afd": {}, "afd_shared": { "_env": { "AFDEP_SHARED_ROLE_KERNELS": "1" } }, "head": { "_label": "head" }, "masked": { "role_aware": false } }, "passthrough": [ "--payload bf16 --routing random --hidden 2048 --tokens 1 8 32 128 512 --capacity 2048 --hybrid on", "--ffn-ranks", "4", "--attention-ranks", "2" ], "processes": [ { "order": [ "afd", "afd_shared", "head", "masked" ], "median_us": { "afd": { "1": 77.537841796875, "8": 82.40288257598877, "32": 88.83768081665039, "128": 122.05495834350586, "512": 279.69568252563477 }, "afd_shared": { "1": 78.89328002929688, "8": 83.24616432189941, "32": 89.9681568145752, "128": 122.80704021453857, "512": 281.3162422180176 }, "head": { "1": 79.22319889068604, "8": 83.2643175125122, "32": 90.11312007904053, "128": 123.32640171051025, "512": 281.2943172454834 }, "masked": { "1": 106.94488048553467, "8": 112.85711765289307, "32": 116.78768157958984, "128": 143.41991901397705, "512": 295.37415504455566 } } }, { "order": [ "afd_shared", "head", "masked", "afd" ], "median_us": { "afd_shared": { "1": 78.79232168197632, "8": 83.34311962127686, "32": 90.42856216430664, "128": 123.24400424957275, "512": 280.95744132995605 }, "head": { "1": 78.97984027862549, "8": 83.32223892211914, "32": 89.650559425354, "128": 123.17008018493652, "512": 280.2951240539551 }, "masked": { "1": 107.32056140899658, "8": 112.48432159423828, "32": 115.94303607940674, "128": 143.36423873901367, "512": 295.51976203918457 }, "afd": { "1": 77.71008014678955, "8": 82.03296184539795, "32": 89.14167881011963, "128": 122.02415943145752, "512": 279.94959831237793 } } }, { "order": [ "head", "masked", "afd", "afd_shared" ], "median_us": { "head": { "1": 79.02503967285156, "8": 83.09576034545898, "32": 90.35784244537354, "128": 123.18024158477783, "512": 281.6156768798828 }, "masked": { "1": 106.88975811004639, "8": 113.00920009613037, "32": 116.45031929016113, "128": 143.45304012298584, "512": 297.84175872802734 }, "afd": { "1": 78.07575941085815, "8": 82.11848258972168, "32": 89.24776077270508, "128": 121.93535804748535, "512": 280.0317573547363 }, "afd_shared": { "1": 79.1104793548584, "8": 83.3627986907959, "32": 90.17840385437012, "128": 123.24671745300293, "512": 281.0409641265869 } } } ], "table": [ { "tokens": 1, "afd": { "median_us": 77.71008014678955, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 78.89328002929688, "paired_change_pct": [ 1.74809899400179, 1.3926655758718542, 1.3252768231881484 ], "geomean_change_pct": 1.488511089886635, "repeats_faster": 0 }, "head": { "median_us": 79.02503967285156, "paired_change_pct": [ 2.173592989892792, 1.6339709461596685, 1.215845057616427 ], "geomean_change_pct": 1.6737141129759436, "repeats_faster": 0 }, "masked": { "median_us": 106.94488048553467, "paired_change_pct": [ 37.92604747201624, 38.10378422757339, 36.90517891418808 ], "geomean_change_pct": 37.64398861052454, "repeats_faster": 0 } }, { "tokens": 8, "afd": { "median_us": 82.11848258972168, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 83.34311962127686, "paired_change_pct": [ 1.0233643770082956, 1.597111388405148, 1.515269232739036 ], "geomean_change_pct": 1.378264656323691, "repeats_faster": 0 }, "head": { "median_us": 83.2643175125122, "paired_change_pct": [ 1.0453941774780207, 1.5716573529930722, 1.19008257936275 ], "geomean_change_pct": 1.2688015822554188, "repeats_faster": 0 }, "masked": { "median_us": 112.85711765289307, "paired_change_pct": [ 36.95773026971547, 37.120882952184495, 37.617253183724955 ], "geomean_change_pct": 37.23166906737851, "repeats_faster": 0 } }, { "tokens": 32, "afd": { "median_us": 89.14167881011963, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 90.17840385437012, "paired_change_pct": [ 1.272518583930582, 1.4436382300227812, 1.0427635086948328 ], "geomean_change_pct": 1.2528402120876692, "repeats_faster": 0 }, "head": { "median_us": 90.11312007904053, "paired_change_pct": [ 1.435696261615016, 0.5708672105203982, 1.2438201956636252 ], "geomean_change_pct": 1.0827800811879618, "repeats_faster": 0 }, "masked": { "median_us": 116.45031929016113, "paired_change_pct": [ 31.46187575587962, 30.066022568832906, 30.479821882293656 ], "geomean_change_pct": 30.66793059731372, "repeats_faster": 0 } }, { "tokens": 128, "afd": { "median_us": 122.02415943145752, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 123.24400424957275, "paired_change_pct": [ 0.6161829730145785, 0.9996748379983211, 1.0754545904616997 ], "geomean_change_pct": 0.8969036846603329, "repeats_faster": 0 }, "head": { "median_us": 123.18024158477783, "paired_change_pct": [ 1.0416974322551509, 0.9390933392355638, 1.0209372877780787 ], "geomean_change_pct": 1.0005663057989, "repeats_faster": 0 }, "masked": { "median_us": 143.41991901397705, "paired_change_pct": [ 17.504377503732883, 17.488405088783377, 17.64679451477957 ], "geomean_change_pct": 17.546504144819487, "repeats_faster": 0 } }, { "tokens": 512, "afd": { "median_us": 279.94959831237793, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 281.0409641265869, "paired_change_pct": [ 0.5794010396403815, 0.36000873859212135, 0.3603901148155009 ], "geomean_change_pct": 0.433213498801055, "repeats_faster": 0 }, "head": { "median_us": 281.2943172454834, "paired_change_pct": [ 0.5715621726488873, 0.12342426767535386, 0.5656213924122966 ], "geomean_change_pct": 0.41998309314599247, "repeats_faster": 0 }, "masked": { "median_us": 295.51976203918457, "paired_change_pct": [ 5.60554684911303, 5.56177391240007, 6.359993431291366 ], "geomean_change_pct": 5.841804881647938, "repeats_faster": 0 } } ], "native_source_sha256": { "afd": "c7c4e90e97048f899e3c099ca0276c6e877c99baf5bc7cb982ed25ae937e3915", "afd_shared": "c7c4e90e97048f899e3c099ca0276c6e877c99baf5bc7cb982ed25ae937e3915", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "c7c4e90e97048f899e3c099ca0276c6e877c99baf5bc7cb982ed25ae937e3915" }, "pods": [ "afdep-v21-xb", "afdep-v21-xc", "afdep-v21-xa" ], "topology": "3 waldorf nodes x 2 H200, InfiniBand, GIN" }, "c7c4e90e_8a8f_piggy": { "arms": { "afd": {}, "afd_shared": { "_env": { "AFDEP_SHARED_ROLE_KERNELS": "1" } }, "head": { "_label": "head" }, "masked": { "role_aware": false } }, "passthrough": [ "--payload", "bf16", "--routing", "random", "--hidden", "2048", "--tokens", "1", "8", "32", "128", "512", "--capacity", "2048", "--hybrid", "on", "--ffn-ranks", "8", "--attention-ranks", "8" ], "processes": [ { "order": [ "afd", "afd_shared", "head", "masked" ], "median_us": { "afd": { "1": 62.52575874328613, "8": 71.87839984893799, "32": 73.80271911621094, "128": 96.61983966827393, "512": 214.2805576324463 }, "afd_shared": { "1": 63.26567888259888, "8": 72.20376014709473, "32": 74.31591987609863, "128": 97.35047817230225, "512": 214.9061679840088 }, "head": { "1": 63.115198612213135, "8": 72.14992046356201, "32": 74.49151992797852, "128": 97.56103992462158, "512": 215.61488151550293 }, "masked": { "1": 82.8168773651123, "8": 93.04047584533691, "32": 98.7395191192627, "128": 114.80152130126953, "512": 232.47967720031738 } } }, { "order": [ "afd_shared", "head", "masked", "afd" ], "median_us": { "afd_shared": { "1": 62.83071994781494, "8": 72.2484803199768, "32": 74.55703973770142, "128": 97.46256351470947, "512": 214.44687843322754 }, "head": { "1": 63.00287961959839, "8": 72.45800018310547, "32": 74.30567979812622, "128": 97.83848285675049, "512": 214.99552726745605 }, "masked": { "1": 82.22967624664307, "8": 93.55335712432861, "32": 98.88607978820801, "128": 115.18728256225586, "512": 232.69495964050293 }, "afd": { "1": 62.75560140609741, "8": 71.74783945083618, "32": 73.77568006515503, "128": 96.98671817779541, "512": 213.928804397583 } } }, { "order": [ "head", "masked", "afd", "afd_shared" ], "median_us": { "head": { "1": 62.739200592041016, "8": 72.67431974411011, "32": 74.66351985931396, "128": 96.9334363937378, "512": 214.19008255004883 }, "masked": { "1": 82.92928218841553, "8": 93.39992046356201, "32": 98.93327713012695, "128": 115.46895980834961, "512": 232.6253604888916 }, "afd": { "1": 62.725040912628174, "8": 72.09728002548218, "32": 74.19960021972656, "128": 97.39863872528076, "512": 214.05096054077148 }, "afd_shared": { "1": 63.364880084991455, "8": 72.40992069244385, "32": 74.34328079223633, "128": 97.69231796264648, "512": 214.88887786865234 } } } ], "table": [ { "tokens": 1, "afd": { "median_us": 62.725040912628174, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 63.26567888259888, "paired_change_pct": [ 1.183384502938467, 0.11970013836921645, 1.0200697569165973 ], "geomean_change_pct": 0.7732972767387825, "repeats_faster": 0 }, "head": { "median_us": 63.00287961959839, "paired_change_pct": [ 0.9427152597173416, 0.3940336925477128, 0.022574205144909598 ], "geomean_change_pct": 0.45239707329887136, "repeats_faster": 0 }, "masked": { "median_us": 82.8168773651123, "paired_change_pct": [ 32.45241486014751, 31.031612165625, 32.21080605420492 ], "geomean_change_pct": 31.896814267796337, "repeats_faster": 0 } }, { "tokens": 8, "afd": { "median_us": 71.87839984893799, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 72.2484803199768, "paired_change_pct": [ 0.4526537858946966, 0.6977783205355914, 0.4336372562892432 ], "geomean_change_pct": 0.5279511972438922, "repeats_faster": 0 }, "head": { "median_us": 72.45800018310547, "paired_change_pct": [ 0.3777499432300324, 0.9898008604926956, 0.8003626744642611 ], "geomean_change_pct": 0.7223126629293475, "repeats_faster": 0 }, "masked": { "median_us": 93.39992046356201, "paired_change_pct": [ 29.44149569394121, 30.391880564479212, 29.547079210964135 ], "geomean_change_pct": 29.792789316829094, "repeats_faster": 0 } }, { "tokens": 32, "afd": { "median_us": 73.80271911621094, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 74.34328079223633, "paired_change_pct": [ 0.6953683631623298, 1.0591019586079442, 0.19364062890403488 ], "geomean_change_pct": 0.6487446159921362, "repeats_faster": 0 }, "head": { "median_us": 74.49151992797852, "paired_change_pct": [ 0.9333000464156083, 0.7183935580168388, 0.6252319934522577 ], "geomean_change_pct": 0.7588926515990524, "repeats_faster": 0 }, "masked": { "median_us": 98.88607978820801, "paired_change_pct": [ 33.78845698596265, 34.03614809226661, 33.333975974475315 ], "geomean_change_pct": 33.719210713345895, "repeats_faster": 0 } }, { "tokens": 128, "afd": { "median_us": 96.98671817779541, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 97.46256351470947, "paired_change_pct": [ 0.7561992511443183, 0.4906293829241015, 0.3015229383175111 ], "geomean_change_pct": 0.5159442272087089, "repeats_faster": 0 }, "head": { "median_us": 97.56103992462158, "paired_change_pct": [ 0.9741273216547386, 0.8782281687206073, -0.4776271389737796 ], "geomean_change_pct": 0.45604876629175006, "repeats_faster": 1 }, "masked": { "median_us": 115.18728256225586, "paired_change_pct": [ 18.8177518151748, 18.766037996146334, 18.552950348759367 ], "geomean_change_pct": 18.712191382560594, "repeats_faster": 0 } }, { "tokens": 512, "afd": { "median_us": 214.05096054077148, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 214.88887786865234, "paired_change_pct": [ 0.29195852319723326, 0.24217123874619162, 0.39145693425712835 ], "geomean_change_pct": 0.308509702692894, "repeats_faster": 0 }, "head": { "median_us": 214.99552726745605, "paired_change_pct": [ 0.6226994636374883, 0.49863452136653663, 0.06499480727666374 ], "geomean_change_pct": 0.39515798279736725, "repeats_faster": 0 }, "masked": { "median_us": 232.6253604888916, "paired_change_pct": [ 8.493126846854615, 8.772149826090425, 8.677559727456652 ], "geomean_change_pct": 8.647550339152922, "repeats_faster": 0 } } ], "native_source_sha256": { "afd": "c7c4e90e97048f899e3c099ca0276c6e877c99baf5bc7cb982ed25ae937e3915", "afd_shared": "c7c4e90e97048f899e3c099ca0276c6e877c99baf5bc7cb982ed25ae937e3915", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "c7c4e90e97048f899e3c099ca0276c6e877c99baf5bc7cb982ed25ae937e3915" }, "pods": [ "afdep-v21-p8a", "afdep-v21-p8b" ], "topology": "2 pods (pgc) x 8 GPUs, InfiniBand, GIN" }, "development_1F2A_waldorf": { "arms": { "afd": {}, "afd_shared": { "_env": { "AFDEP_SHARED_ROLE_KERNELS": "1" } }, "head": { "_label": "head" }, "masked": { "role_aware": false } }, "passthrough": [ "--payload bf16 --routing random --hidden 2048 --tokens 1 8 32 128 512 --capacity 2048 --hybrid on", "--ffn-ranks", "2", "--attention-ranks", "4" ], "processes": [ { "order": [ "afd", "afd_shared", "head", "masked" ], "median_us": { "afd": { "1": 85.06639957427979, "8": 98.96503925323486, "32": 103.25536251068115, "128": 138.77551555633545, "512": 330.66287994384766 }, "afd_shared": { "1": 85.75072288513184, "8": 99.24615859985352, "32": 103.67656230926514, "128": 139.22536373138428, "512": 331.46751403808594 }, "head": { "1": 85.75352191925049, "8": 99.27167892456055, "32": 103.76936435699463, "128": 139.4931173324585, "512": 331.5760040283203 }, "masked": { "1": 107.34280109405518, "8": 119.72568035125732, "32": 126.46967887878418, "128": 158.6349582672119, "512": 344.14873123168945 } } }, { "order": [ "afd_shared", "head", "masked", "afd" ], "median_us": { "afd_shared": { "1": 85.93535900115967, "8": 99.31968212127686, "32": 103.985595703125, "128": 139.29935932159424, "512": 330.7863998413086 }, "head": { "1": 85.86872100830078, "8": 99.49512004852295, "32": 103.62584114074707, "128": 139.060959815979, "512": 331.41191482543945 }, "masked": { "1": 107.505202293396, "8": 119.53399658203125, "32": 126.04832172393799, "128": 158.73512268066406, "512": 343.9451026916504 }, "afd": { "1": 85.42143821716309, "8": 98.58848094940186, "32": 103.03455829620361, "128": 138.91200065612793, "512": 332.3315238952637 } } }, { "order": [ "head", "masked", "afd", "afd_shared" ], "median_us": { "head": { "1": 85.76079845428467, "8": 99.19928073883057, "32": 103.60687732696533, "128": 139.13527965545654, "512": 331.9744110107422 }, "masked": { "1": 107.40327835083008, "8": 119.83392238616943, "32": 125.64399719238281, "128": 158.28743934631348, "512": 343.98536682128906 }, "afd": { "1": 85.4193639755249, "8": 98.91520023345947, "32": 103.21135997772217, "128": 138.86807918548584, "512": 330.9681701660156 }, "afd_shared": { "1": 85.90104103088379, "8": 99.18928146362305, "32": 103.45632076263428, "128": 139.61919784545898, "512": 331.8182373046875 } } } ], "table": [ { "tokens": 1, "afd": { "median_us": 85.4193639755249, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 85.90104103088379, "paired_change_pct": [ 0.8044578285630877, 0.6016297486001854, 0.563896794521801 ], "geomean_change_pct": 0.656606050122388, "repeats_faster": 0 }, "head": { "median_us": 85.76079845428467, "paired_change_pct": [ 0.8077482395040247, 0.5236188952948639, 0.3997155479378067 ], "geomean_change_pct": 0.5768825961565804, "repeats_faster": 0 }, "masked": { "median_us": 107.40327835083008, "paired_change_pct": [ 26.187074604378523, 25.85271863497587, 25.73645289796842 ], "geomean_change_pct": 25.925270584106364, "repeats_faster": 0 } }, { "tokens": 8, "afd": { "median_us": 98.91520023345947, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 99.24615859985352, "paired_change_pct": [ 0.284059248336499, 0.7416699850059327, 0.27708707005262845 ], "geomean_change_pct": 0.4340370879077682, "repeats_faster": 0 }, "head": { "median_us": 99.27167892456055, "paired_change_pct": [ 0.3098464605677975, 0.9196197064709866, 0.2871960069843693 ], "geomean_change_pct": 0.5051277448551383, "repeats_faster": 0 }, "masked": { "median_us": 119.72568035125732, "paired_change_pct": [ 20.977752603017198, 21.24539847954363, 21.148137094539198 ], "geomean_change_pct": 21.123712205079915, "repeats_faster": 0 } }, { "tokens": 32, "afd": { "median_us": 103.21135997772217, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 103.67656230926514, "paired_change_pct": [ 0.40792050731546237, 0.9230275964180246, 0.23733897602451481 ], "geomean_change_pct": 0.5223402209525219, "repeats_faster": 0 }, "head": { "median_us": 103.62584114074707, "paired_change_pct": [ 0.4977967572970421, 0.5738684712401465, 0.383211062550215 ], "geomean_change_pct": 0.4849282041271108, "repeats_faster": 0 }, "masked": { "median_us": 126.04832172393799, "paired_change_pct": [ 22.4824317145772, 22.335965532626865, 21.734659071930306 ], "geomean_change_pct": 22.18392325212759, "repeats_faster": 0 } }, { "tokens": 128, "afd": { "median_us": 138.86807918548584, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 139.29935932159424, "paired_change_pct": [ 0.3241552901067868, 0.27885183687275106, 0.5408864761280885 ], "geomean_change_pct": 0.3812327652716885, "repeats_faster": 0 }, "head": { "median_us": 139.13527965545654, "paired_change_pct": [ 0.5170953775572418, 0.10723275105641861, 0.1924131676177332 ], "geomean_change_pct": 0.2720917056639349, "repeats_faster": 0 }, "masked": { "median_us": 158.6349582672119, "paired_change_pct": [ 14.310480225032629, 14.270273216788244, 13.984034541796486 ], "geomean_change_pct": 14.188170112444688, "repeats_faster": 0 } }, { "tokens": 512, "afd": { "median_us": 330.9681701660156, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 331.46751403808594, "paired_change_pct": [ 0.24333971033425605, -0.46493454362819264, 0.2568425653274975 ], "geomean_change_pct": 0.011180185310211144, "repeats_faster": 1 }, "head": { "median_us": 331.5760040283203, "paired_change_pct": [ 0.2761495589186591, -0.27671436613820877, 0.304029491483071 ], "geomean_change_pct": 0.10079719789664043, "repeats_faster": 1 }, "masked": { "median_us": 343.98536682128906, "paired_change_pct": [ 4.078429151204377, 3.494576337587163, 3.933066025274856 ], "geomean_change_pct": 3.8350603569055286, "repeats_faster": 0 } } ], "native_source_sha256": { "afd": "c8a741613a6a9f890a15f8c24556f5035777257a02c14a2b1bec24fc874d975e", "afd_shared": "c8a741613a6a9f890a15f8c24556f5035777257a02c14a2b1bec24fc874d975e", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "c8a741613a6a9f890a15f8c24556f5035777257a02c14a2b1bec24fc874d975e" }, "pods": [ "afdep-v21-xa", "afdep-v21-xb", "afdep-v21-xc" ], "topology": "3 waldorf nodes x 2 H200, InfiniBand, GIN" }, "development_two_nodes_waldorf": { "arms": { "afd": {}, "afd_shared": { "_env": { "AFDEP_SHARED_ROLE_KERNELS": "1" } }, "head": { "_label": "head" }, "masked": { "role_aware": false } }, "passthrough": [ "--payload bf16 --routing random --hidden 2048 --tokens 1 8 32 128 512 --capacity 2048 --hybrid on" ], "processes": [ { "order": [ "afd", "afd_shared", "head", "masked" ], "median_us": { "afd": { "1": 71.00791931152344, "8": 77.84519910812378, "32": 80.34080028533936, "128": 101.43464088439941, "512": 200.0050449371338 }, "afd_shared": { "1": 71.31736040115356, "8": 78.66144180297852, "32": 80.99823951721191, "128": 102.96112060546875, "512": 199.7590446472168 }, "head": { "1": 71.43376111984253, "8": 78.58776092529297, "32": 80.900559425354, "128": 102.91975975036621, "512": 199.5090389251709 }, "masked": { "1": 88.07447910308838, "8": 98.61167907714844, "32": 104.17304039001465, "128": 121.08471870422363, "512": 216.60632133483887 } } }, { "order": [ "afd_shared", "head", "masked", "afd" ], "median_us": { "afd_shared": { "1": 71.30784034729004, "8": 78.52824211120605, "32": 80.7484769821167, "128": 102.16584205627441, "512": 199.6869659423828 }, "head": { "1": 71.2609601020813, "8": 78.50831985473633, "32": 80.8951997756958, "128": 102.87335872650146, "512": 199.23599243164062 }, "masked": { "1": 88.29535961151123, "8": 98.47792148590088, "32": 104.65528011322021, "128": 121.15344047546387, "512": 216.7031955718994 }, "afd": { "1": 71.38136148452759, "8": 78.45864057540894, "32": 80.60199737548828, "128": 101.86039924621582, "512": 199.81040000915527 } } }, { "order": [ "head", "masked", "afd", "afd_shared" ], "median_us": { "head": { "1": 71.35967969894409, "8": 78.43151807785034, "32": 80.92887878417969, "128": 102.79672145843506, "512": 199.3336009979248 }, "masked": { "1": 88.52896213531494, "8": 98.83608341217041, "32": 104.71104145050049, "128": 121.56935691833496, "512": 216.45431518554688 }, "afd": { "1": 71.08495950698853, "8": 78.08511972427368, "32": 80.61216354370117, "128": 101.61064147949219, "512": 199.17752265930176 }, "afd_shared": { "1": 71.44407987594604, "8": 78.5539197921753, "32": 80.89799880981445, "128": 102.4015998840332, "512": 200.1631259918213 } } } ], "table": [ { "tokens": 1, "afd": { "median_us": 71.08495950698853, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 71.31736040115356, "paired_change_pct": [ 0.4357839134428998, -0.10299766733012916, 0.5051988092111381 ], "geomean_change_pct": 0.2789594833405795, "repeats_faster": 1 }, "head": { "median_us": 71.35967969894409, "paired_change_pct": [ 0.5997103033689166, -0.16867341830175686, 0.38646739600176616 ], "geomean_change_pct": 0.2719778223200242, "repeats_faster": 1 }, "masked": { "median_us": 88.29535961151123, "paired_change_pct": [ 24.034727333286774, 23.695258503369775, 24.539653323726608 ], "geomean_change_pct": 24.089394980594903, "repeats_faster": 0 } }, { "tokens": 8, "afd": { "median_us": 78.08511972427368, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 78.5539197921753, "paired_change_pct": [ 1.0485459658482021, 0.0887111161838483, 0.6003705565887474 ], "geomean_change_pct": 0.5784446188888959, "repeats_faster": 0 }, "head": { "median_us": 78.50831985473633, "paired_change_pct": [ 0.9538954562089419, 0.06331906717098601, 0.44361634431735464 ], "geomean_change_pct": 0.4862814949544436, "repeats_faster": 0 }, "masked": { "median_us": 98.61167907714844, "paired_change_pct": [ 26.67663543410155, 25.515712181185222, 26.574799092542143 ], "geomean_change_pct": 26.2546223192081, "repeats_faster": 0 } }, { "tokens": 32, "afd": { "median_us": 80.60199737548828, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 80.89799880981445, "paired_change_pct": [ 0.8183130234421299, 0.18173198108979438, 0.3545808145421159 ], "geomean_change_pct": 0.4511826791766982, "repeats_faster": 0 }, "head": { "median_us": 80.900559425354, "paired_change_pct": [ 0.6967308491160118, 0.363765675485217, 0.39288765684450944 ], "geomean_change_pct": 0.48434866509663443, "repeats_faster": 0 }, "masked": { "median_us": 104.65528011322021, "paired_change_pct": [ 29.66393167610033, 29.842043027393682, 29.89484073794264 ], "geomean_change_pct": 29.80023421109703, "repeats_faster": 0 } }, { "tokens": 128, "afd": { "median_us": 101.61064147949219, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 102.4015998840332, "paired_change_pct": [ 1.5048899545166305, 0.2998641398609392, 0.7784208356766076 ], "geomean_change_pct": 0.8598426048841423, "repeats_faster": 0 }, "head": { "median_us": 102.87335872650146, "paired_change_pct": [ 1.4641140866849645, 0.9944585803528305, 1.1672792944450228 ], "geomean_change_pct": 1.208431554188949, "repeats_faster": 0 }, "masked": { "median_us": 121.15344047546387, "paired_change_pct": [ 19.372156936227093, 18.94066916291297, 19.642347640203607 ], "geomean_change_pct": 19.318041169820965, "repeats_faster": 0 } }, { "tokens": 512, "afd": { "median_us": 199.81040000915527, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 199.7590446472168, "paired_change_pct": [ -0.12299704239676634, -0.06177559664902743, 0.49483662582019416 ], "geomean_change_pct": 0.10296927762907249, "repeats_faster": 2 }, "head": { "median_us": 199.3336009979248, "paired_change_pct": [ -0.2479967503413727, -0.2874763162920124, 0.07836142178052619 ], "geomean_change_pct": -0.1525050458508037, "repeats_faster": 2 }, "masked": { "median_us": 216.60632133483887, "paired_change_pct": [ 8.300428823144568, 8.454412564095826, 8.6740673824915 ], "geomean_change_pct": 8.476194592823717, "repeats_faster": 0 } } ], "native_source_sha256": { "afd": "c8a741613a6a9f890a15f8c24556f5035777257a02c14a2b1bec24fc874d975e", "afd_shared": "c8a741613a6a9f890a15f8c24556f5035777257a02c14a2b1bec24fc874d975e", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "c8a741613a6a9f890a15f8c24556f5035777257a02c14a2b1bec24fc874d975e" }, "pods": [ "afdep-v21-xa", "afdep-v21-xb" ], "topology": "2 waldorf nodes x 2 H200, InfiniBand, GIN" } }, "correctness": { "c7c4e90e_waldorf": { "result": [ "jit-split: afdep-v21-xa exit=0 afdep-v21-xb exit=0", "jit-control: afdep-v21-xb exit=0 afdep-v21-xa exit=0", "bf16-hyb-small: afdep-v21-xb exit=0 afdep-v21-xa exit=0 bf16-hyb-large: afdep-v21-xb exit=0 afdep-v21-xa exit=0 bf16-direct-small: afdep-v21-xa exit=0 afdep-v21-xb exit=0 bf16-direct-large: afdep-v21-xb exit=0 afdep-v21-xa exit=0 bf16-generic: afdep-v21-xb exit=0 afdep-v21-xa exit=0 fp8-hyb-small: afdep-v21-xa exit=0 afdep-v21-xb exit=0 fp8-hyb-large: afdep-v21-xa exit=0 afdep-v21-xb exit=0 fp8-direct-small: afdep-v21-xa exit=0 afdep-v21-xb exit=0 fp8-direct-large: afdep-v21-xb exit=0 afdep-v21-xa exit=0 fp8-generic: afdep-v21-xb exit=0 afdep-v21-xa exit=0 f1-hyb-small: afdep-v21-xb exit=0 afdep-v21-xa exit=0 f3-hyb-small: afdep-v21-xb exit=0 afdep-v21-xa exit=0 f1-hyb-large: afdep-v21-xa exit=0 afdep-v21-xb exit=0 f3-hyb-large: afdep-v21-xb exit=0 afdep-v21-xa exit=0 done", "f2dom-large-bf16: afdep-v21-xc exit=0 afdep-v21-xa exit=0 afdep-v21-xb exit=0", "a2dom-large-bf16: afdep-v21-xa exit=0 afdep-v21-xc exit=0 afdep-v21-xb exit=0", "f2dom-small-bf16: afdep-v21-xa exit=0 afdep-v21-xb exit=0 afdep-v21-xc exit=0", "a2dom-small-bf16: afdep-v21-xa exit=0 afdep-v21-xb exit=0 afdep-v21-xc exit=0", "a2dom-large-fp8: afdep-v21-xa exit=0 afdep-v21-xc exit=0 afdep-v21-xb exit=0", "f2dom-large-fp8: afdep-v21-xc exit=0 afdep-v21-xa exit=0 afdep-v21-xb exit=0", "f2dom-small-fp8: afdep-v21-xc exit=0 afdep-v21-xa exit=0 afdep-v21-xb exit=0", "a2dom-small-fp8: afdep-v21-xc exit=0 afdep-v21-xa exit=0 afdep-v21-xb exit=0", "ctl-AFDEP_SHARED_HYBRID_SEND: afdep-v21-xb exit=0 afdep-v21-xa exit=0", "ctl-AFDEP_SHARED_HYBRID_SEND-large: afdep-v21-xb exit=0 afdep-v21-xa exit=0", "ctl-AFDEP_SHARED_HYBRID_FORWARD: afdep-v21-xb exit=0 afdep-v21-xa exit=0", "ctl-AFDEP_SHARED_HYBRID_FORWARD-large: afdep-v21-xa exit=0 afdep-v21-xb exit=0", "ctl-AFDEP_SHARED_HYBRID_COLLECT: afdep-v21-xa exit=0 afdep-v21-xb exit=0", "ctl-AFDEP_SHARED_HYBRID_COLLECT-large: afdep-v21-xa exit=0 afdep-v21-xb exit=0", "ctl-AFDEP_SHARED_ROLE_KERNELS: afdep-v21-xb exit=0 afdep-v21-xa exit=0", "ctl-AFDEP_SHARED_ROLE_KERNELS-large: afdep-v21-xb exit=0 afdep-v21-xa exit=0", "ctl-AFDEP_UNION_HYBRID: afdep-v21-xb exit=0 afdep-v21-xa exit=0", "ctl-AFDEP_UNION_HYBRID-large: afdep-v21-xb exit=0 afdep-v21-xa exit=0", "ctl-AFDEP_HYBRID_ROUTE_GUARD: afdep-v21-xa exit=0 afdep-v21-xb exit=0", "ctl-AFDEP_HYBRID_ROUTE_GUARD-large: afdep-v21-xb exit=0 afdep-v21-xa exit=0", "ctl-AFDEP_ATTENTION_GRID_SYNC: afdep-v21-xb exit=0 afdep-v21-xa exit=0", "ctl-AFDEP_ATTENTION_GRID_SYNC-large: afdep-v21-xb exit=0 afdep-v21-xa exit=0", "mixed-ffn: afdep-v21-xa exit=0 afdep-v21-xb exit=0", "mixed-ffn-large: afdep-v21-xb exit=0 afdep-v21-xa exit=0", "mixed-ffn-f2dom: afdep-v21-xb exit=0 afdep-v21-xa exit=0 afdep-v21-xc exit=0", "mixed-ffn-a2dom: afdep-v21-xa exit=0 afdep-v21-xb exit=0 afdep-v21-xc exit=0", "mixed-attention: afdep-v21-xb exit=0 afdep-v21-xa exit=0", "mixed-attention-large: afdep-v21-xb exit=0 afdep-v21-xa exit=0", "mixed-attention-f2dom: afdep-v21-xa exit=0 afdep-v21-xb exit=0 afdep-v21-xc exit=0", "mixed-attention-a2dom: afdep-v21-xc exit=0 afdep-v21-xb exit=0 afdep-v21-xa exit=0", "inv-eager-dummy: afdep-v21-xb exit=1 afdep-v21-xa exit=1 attention_assertions=2", "inv-eager-duplicate: afdep-v21-xb exit=1 afdep-v21-xa exit=1 attention_assertions=2", "inv-graph-dummy: afdep-v21-xb exit=1 afdep-v21-xa exit=1 attention_assertions=2", "inv-graph-duplicate: afdep-v21-xb exit=1 afdep-v21-xa exit=1 attention_assertions=2" ], "logs": [ { "log": "fin-a2dom-large-bf16-n0.log", "log_sha256": "00d22b3c72356d8f9addd54251e2f852b69d033d03aef593de7e3660c060593b", "pod": "afdep-v21-xa" }, { "log": "fin-a2dom-large-fp8-n0.log", "log_sha256": "6de8eae5d803fcfa6e442148420ce0f1c802daaa0eda2d3b55627e0aa0e83e12", "pod": "afdep-v21-xa" }, { "log": "fin-a2dom-small-bf16-n0.log", "log_sha256": "b9862b1af903eb2b6f9ee3ce776cd777ad362a1ea122565a4123b8ba3239398b", "pod": "afdep-v21-xa" }, { "log": "fin-a2dom-small-fp8-n0.log", "log_sha256": "953ce2710f9d0c17bca2c3874e76d550c34b2ca518785e7fe3c6f5e341ca92d7", "pod": "afdep-v21-xa" }, { "log": "fin-ctl-AFDEP_ATTENTION_GRID_SYNC-large-n0.log", "log_sha256": "e246bc5259385dd4dc9ad01f0e57e636dc7889b948b6a872b4bf0da21ceff425", "pod": "afdep-v21-xa" }, { "log": "fin-ctl-AFDEP_ATTENTION_GRID_SYNC-n0.log", "log_sha256": "b47ac173de99f750028809fb7eaea4816b9c8c9251e718be4f36d524a1868058", "pod": "afdep-v21-xa" }, { "log": "fin-ctl-AFDEP_HYBRID_ROUTE_GUARD-large-n0.log", "log_sha256": "e246bc5259385dd4dc9ad01f0e57e636dc7889b948b6a872b4bf0da21ceff425", "pod": "afdep-v21-xa" }, { "log": "fin-ctl-AFDEP_HYBRID_ROUTE_GUARD-n0.log", "log_sha256": "b47ac173de99f750028809fb7eaea4816b9c8c9251e718be4f36d524a1868058", "pod": "afdep-v21-xa" }, { "log": "fin-ctl-AFDEP_SHARED_HYBRID_COLLECT-large-n0.log", "log_sha256": "e246bc5259385dd4dc9ad01f0e57e636dc7889b948b6a872b4bf0da21ceff425", "pod": "afdep-v21-xa" }, { "log": "fin-ctl-AFDEP_SHARED_HYBRID_COLLECT-n0.log", "log_sha256": "b47ac173de99f750028809fb7eaea4816b9c8c9251e718be4f36d524a1868058", "pod": "afdep-v21-xa" }, { "log": "fin-ctl-AFDEP_SHARED_HYBRID_FORWARD-large-n0.log", "log_sha256": "e246bc5259385dd4dc9ad01f0e57e636dc7889b948b6a872b4bf0da21ceff425", "pod": "afdep-v21-xa" }, { "log": "fin-ctl-AFDEP_SHARED_HYBRID_FORWARD-n0.log", "log_sha256": "b47ac173de99f750028809fb7eaea4816b9c8c9251e718be4f36d524a1868058", "pod": "afdep-v21-xa" }, { "log": "fin-ctl-AFDEP_SHARED_HYBRID_SEND-large-n0.log", "log_sha256": "e246bc5259385dd4dc9ad01f0e57e636dc7889b948b6a872b4bf0da21ceff425", "pod": "afdep-v21-xa" }, { "log": "fin-ctl-AFDEP_SHARED_HYBRID_SEND-n0.log", "log_sha256": "b47ac173de99f750028809fb7eaea4816b9c8c9251e718be4f36d524a1868058", "pod": "afdep-v21-xa" }, { "log": "fin-ctl-AFDEP_SHARED_ROLE_KERNELS-large-n0.log", "log_sha256": "e246bc5259385dd4dc9ad01f0e57e636dc7889b948b6a872b4bf0da21ceff425", "pod": "afdep-v21-xa" }, { "log": "fin-ctl-AFDEP_SHARED_ROLE_KERNELS-n0.log", "log_sha256": "b47ac173de99f750028809fb7eaea4816b9c8c9251e718be4f36d524a1868058", "pod": "afdep-v21-xa" }, { "log": "fin-ctl-AFDEP_UNION_HYBRID-large-n0.log", "log_sha256": "e246bc5259385dd4dc9ad01f0e57e636dc7889b948b6a872b4bf0da21ceff425", "pod": "afdep-v21-xa" }, { "log": "fin-ctl-AFDEP_UNION_HYBRID-n0.log", "log_sha256": "b47ac173de99f750028809fb7eaea4816b9c8c9251e718be4f36d524a1868058", "pod": "afdep-v21-xa" }, { "log": "fin-f2dom-large-bf16-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xa" }, { "log": "fin-f2dom-large-fp8-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xa" }, { "log": "fin-f2dom-small-bf16-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xa" }, { "log": "fin-f2dom-small-fp8-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xa" }, { "log": "fin-inv-eager-dummy-n0.log", "log_sha256": "fb001da9b57b0814db41eabd64e7d0c550ee0dbce105256cb888458b552bd3ad", "pod": "afdep-v21-xa" }, { "log": "fin-inv-eager-duplicate-n0.log", "log_sha256": "d75cbfb094a72478f05ae0724e9b4340d96aa37d959aa8dd22d413cbd1c86c54", "pod": "afdep-v21-xa" }, { "log": "fin-inv-graph-dummy-n0.log", "log_sha256": "aa0b0f52a3ed6219c4946b64d7ab69f47b51574e75c4cd6d5bf8247be7f1a1a8", "pod": "afdep-v21-xa" }, { "log": "fin-inv-graph-duplicate-n0.log", "log_sha256": "5521d545e9e8d6b80f8bc35f32e1b6f471dfcac85b9f96413d246574718678d1", "pod": "afdep-v21-xa" }, { "log": "fin-jit-control-n0.log", "log_sha256": "7422586b6d65cc7342969590937438e5ebbc68914209b4db8cca69c68c919640", "pod": "afdep-v21-xa" }, { "log": "fin-jit-split-n0.log", "log_sha256": "f4b9bbc67dbb245ec79711a6fc43f6db7e92b1f0aa568c85923b6174f747ff62", "pod": "afdep-v21-xa" }, { "log": "fin-m-bf16-direct-large-n0.log", "log_sha256": "83e5c2803c55429098a76057d4bb379ed3bf4756da24ce86bd904b011c9bf660", "pod": "afdep-v21-xa" }, { "log": "fin-m-bf16-direct-small-n0.log", "log_sha256": "e94f40ecd84cfd63153ac5c0219c1d864b259790222c0484941de02aaba55de1", "pod": "afdep-v21-xa" }, { "log": "fin-m-bf16-generic-n0.log", "log_sha256": "d0f6dd5e5d04965c9e0824b34f894261d24df458e306405dbcdbe295a7c22073", "pod": "afdep-v21-xa" }, { "log": "fin-m-bf16-hyb-large-n0.log", "log_sha256": "e246bc5259385dd4dc9ad01f0e57e636dc7889b948b6a872b4bf0da21ceff425", "pod": "afdep-v21-xa" }, { "log": "fin-m-bf16-hyb-small-n0.log", "log_sha256": "b47ac173de99f750028809fb7eaea4816b9c8c9251e718be4f36d524a1868058", "pod": "afdep-v21-xa" }, { "log": "fin-m-f1-hyb-large-n0.log", "log_sha256": "4ce1e7172d0ad4553e12c9af0b922602a2991b8a20a111e4d83595273d0b61f3", "pod": "afdep-v21-xa" }, { "log": "fin-m-f1-hyb-small-n0.log", "log_sha256": "6bcdf045265cc7043fd94b1e419a81e7c43160bdf9735236ecc9740f6be1d83e", "pod": "afdep-v21-xa" }, { "log": "fin-m-f3-hyb-large-n0.log", "log_sha256": "9777317877f3fecc8e57034edde65fba4568cb66a186d32d3e18979bfcab7f2f", "pod": "afdep-v21-xa" }, { "log": "fin-m-f3-hyb-small-n0.log", "log_sha256": "1af315839775f72be1d8f0bc6d856ce6a1636b2a3a6a9813e0ebcc3250a5d8fc", "pod": "afdep-v21-xa" }, { "log": "fin-m-fp8-direct-large-n0.log", "log_sha256": "70c9ef0ca8bdaa78df61c5cfa677e051fba96896f48c80a00e3d4b68962ef3c0", "pod": "afdep-v21-xa" }, { "log": "fin-m-fp8-direct-small-n0.log", "log_sha256": "b7ec1843e08efe1777a2ec3d6a773eb0ae5f211e6487aa75281cc200d847b858", "pod": "afdep-v21-xa" }, { "log": "fin-m-fp8-generic-n0.log", "log_sha256": "3770229dbea00cf7acd0b861e9ac31ac392a6d172d25d5ec5728da657bf99ec3", "pod": "afdep-v21-xa" }, { "log": "fin-m-fp8-hyb-large-n0.log", "log_sha256": "379fcc50d0b1e4d73c5acbcf816da45f688ea73cce14ce623402fc22f2e558f8", "pod": "afdep-v21-xa" }, { "log": "fin-m-fp8-hyb-small-n0.log", "log_sha256": "044225a4ac6aa9c88f1997adf1e146d4fff7eb33bcdcff4d4cde7da74594da08", "pod": "afdep-v21-xa" }, { "log": "fin-mixed-attention-a2dom-n0.log", "log_sha256": "00d22b3c72356d8f9addd54251e2f852b69d033d03aef593de7e3660c060593b", "pod": "afdep-v21-xa" }, { "log": "fin-mixed-attention-f2dom-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xa" }, { "log": "fin-mixed-attention-large-n0.log", "log_sha256": "e246bc5259385dd4dc9ad01f0e57e636dc7889b948b6a872b4bf0da21ceff425", "pod": "afdep-v21-xa" }, { "log": "fin-mixed-attention-n0.log", "log_sha256": "b47ac173de99f750028809fb7eaea4816b9c8c9251e718be4f36d524a1868058", "pod": "afdep-v21-xa" }, { "log": "fin-mixed-ffn-a2dom-n0.log", "log_sha256": "00d22b3c72356d8f9addd54251e2f852b69d033d03aef593de7e3660c060593b", "pod": "afdep-v21-xa" }, { "log": "fin-mixed-ffn-f2dom-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xa" }, { "log": "fin-mixed-ffn-large-n0.log", "log_sha256": "e246bc5259385dd4dc9ad01f0e57e636dc7889b948b6a872b4bf0da21ceff425", "pod": "afdep-v21-xa" }, { "log": "fin-mixed-ffn-n0.log", "log_sha256": "b47ac173de99f750028809fb7eaea4816b9c8c9251e718be4f36d524a1868058", "pod": "afdep-v21-xa" }, { "log": "fin-a2dom-large-bf16-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-a2dom-large-fp8-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-a2dom-small-bf16-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-a2dom-small-fp8-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-ctl-AFDEP_ATTENTION_GRID_SYNC-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-ctl-AFDEP_ATTENTION_GRID_SYNC-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-ctl-AFDEP_HYBRID_ROUTE_GUARD-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-ctl-AFDEP_HYBRID_ROUTE_GUARD-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-ctl-AFDEP_SHARED_HYBRID_COLLECT-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-ctl-AFDEP_SHARED_HYBRID_COLLECT-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-ctl-AFDEP_SHARED_HYBRID_FORWARD-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-ctl-AFDEP_SHARED_HYBRID_FORWARD-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-ctl-AFDEP_SHARED_HYBRID_SEND-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-ctl-AFDEP_SHARED_HYBRID_SEND-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-ctl-AFDEP_SHARED_ROLE_KERNELS-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-ctl-AFDEP_SHARED_ROLE_KERNELS-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-ctl-AFDEP_UNION_HYBRID-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-ctl-AFDEP_UNION_HYBRID-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-f2dom-large-bf16-n0.log", "log_sha256": "25b8382edbac8f2a512c7fb10a3a0a1b773af179fb22ce20ec024627a4345f78", "pod": "afdep-v21-xb" }, { "log": "fin-f2dom-large-fp8-n0.log", "log_sha256": "4824bc23535f5f394c8e7515d112c8cc8bab756bc0ec8cbc92c186cd2a46bab6", "pod": "afdep-v21-xb" }, { "log": "fin-f2dom-small-bf16-n0.log", "log_sha256": "04f8031315fe62717ca41789e2b3c9b8ff21fb222188adec566760631151c18e", "pod": "afdep-v21-xb" }, { "log": "fin-f2dom-small-fp8-n0.log", "log_sha256": "ea3c90e04914b7de4fdcfc7b5c5f562a47db5e258145b7798a98db32b953dd73", "pod": "afdep-v21-xb" }, { "log": "fin-inv-eager-dummy-n1.log", "log_sha256": "3d731d47e5227562342d6abaea691907975fb059075a3fed5bf17d07ffb5b4c9", "pod": "afdep-v21-xb" }, { "log": "fin-inv-eager-duplicate-n1.log", "log_sha256": "7ff43bdb5880a8c1d617d897b319a82c4cc79909762d78815b033d91426b651f", "pod": "afdep-v21-xb" }, { "log": "fin-inv-graph-dummy-n1.log", "log_sha256": "af1c3d00166ff300edf91e3972adea70d02d38f1c4d3e8b0097c3dc8f379aeed", "pod": "afdep-v21-xb" }, { "log": "fin-inv-graph-duplicate-n1.log", "log_sha256": "8efe486752a004f35a8e23b5d7799b4dbc940f6133b07011acf47af8724ba017", "pod": "afdep-v21-xb" }, { "log": "fin-jit-control-n1.log", "log_sha256": "841c6f7cc3b3df294fc3aa92d88f168caab465040d715d367fd9f48dab1c1260", "pod": "afdep-v21-xb" }, { "log": "fin-jit-split-n1.log", "log_sha256": "4085041f37a7d782774bc61f2a01c1e98b04178d23ad4bd1a30dea7e9c28ac25", "pod": "afdep-v21-xb" }, { "log": "fin-m-bf16-direct-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-m-bf16-direct-small-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-m-bf16-generic-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-m-bf16-hyb-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-m-bf16-hyb-small-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-m-f1-hyb-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-m-f1-hyb-small-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-m-f3-hyb-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-m-f3-hyb-small-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-m-fp8-direct-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-m-fp8-direct-small-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-m-fp8-generic-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-m-fp8-hyb-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-m-fp8-hyb-small-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-mixed-attention-a2dom-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-mixed-attention-f2dom-n0.log", "log_sha256": "25b8382edbac8f2a512c7fb10a3a0a1b773af179fb22ce20ec024627a4345f78", "pod": "afdep-v21-xb" }, { "log": "fin-mixed-attention-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-mixed-attention-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-mixed-ffn-a2dom-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-mixed-ffn-f2dom-n0.log", "log_sha256": "25b8382edbac8f2a512c7fb10a3a0a1b773af179fb22ce20ec024627a4345f78", "pod": "afdep-v21-xb" }, { "log": "fin-mixed-ffn-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-mixed-ffn-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xb" }, { "log": "fin-a2dom-large-bf16-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xc" }, { "log": "fin-a2dom-large-fp8-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xc" }, { "log": "fin-a2dom-small-bf16-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xc" }, { "log": "fin-a2dom-small-fp8-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xc" }, { "log": "fin-f2dom-large-bf16-n1.log", "log_sha256": "b9c6141fbf45e6d082bb85c881a1be65d8561d84fd5647eac22f61154d7818ec", "pod": "afdep-v21-xc" }, { "log": "fin-f2dom-large-fp8-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xc" }, { "log": "fin-f2dom-small-bf16-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xc" }, { "log": "fin-f2dom-small-fp8-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xc" }, { "log": "fin-mixed-attention-a2dom-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xc" }, { "log": "fin-mixed-attention-f2dom-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xc" }, { "log": "fin-mixed-ffn-a2dom-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xc" }, { "log": "fin-mixed-ffn-f2dom-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-xc" } ], "note": "a local DNS/API outage interrupted the first run; the cases it never started were rerun" }, "c7c4e90e_piggy_8a8f": { "result": [ "jit-split: afdep-v21-p8a exit=0 afdep-v21-p8b exit=0", "jit-control: afdep-v21-p8a exit=0 afdep-v21-p8b exit=0", "bf16-small: afdep-v21-p8a exit=0 afdep-v21-p8b exit=0", "bf16-large: afdep-v21-p8b exit=0 afdep-v21-p8a exit=0", "bf16-generic: afdep-v21-p8b exit=0 afdep-v21-p8a exit=0", "fp8-small: afdep-v21-p8a exit=0 afdep-v21-p8b exit=0", "fp8-large: afdep-v21-p8b exit=0 afdep-v21-p8a exit=0", "fp8-generic: afdep-v21-p8b exit=0 afdep-v21-p8a exit=0", "ctl-AFDEP_SHARED_HYBRID_SEND-large: afdep-v21-p8a exit=0 afdep-v21-p8b exit=0", "ctl-AFDEP_SHARED_HYBRID_FORWARD-large: afdep-v21-p8a exit=0 afdep-v21-p8b exit=0", "ctl-AFDEP_SHARED_HYBRID_COLLECT-large: afdep-v21-p8a exit=0 afdep-v21-p8b exit=0", "ctl-AFDEP_SHARED_ROLE_KERNELS-large: afdep-v21-p8a exit=0 afdep-v21-p8b exit=0", "ctl-AFDEP_HYBRID_ROUTE_GUARD-large: afdep-v21-p8b exit=0 afdep-v21-p8a exit=0", "mixed-ffn-large: afdep-v21-p8a exit=0 afdep-v21-p8b exit=0", "mixed-attention-large: afdep-v21-p8a exit=0 afdep-v21-p8b exit=0", "inv-eager: afdep-v21-p8b exit=1 afdep-v21-p8a exit=1 assertions=2", "inv-graph: afdep-v21-p8b exit=1 afdep-v21-p8a exit=1 assertions=2" ], "logs": [ { "log": "pgfin-bf16-generic-n0.log", "log_sha256": "1787c04a4e9120bdc3b3a07225cce046fe6296ee2ffd0822d1de0e93d83f510f", "pod": "afdep-v21-p8a" }, { "log": "pgfin-bf16-large-n0.log", "log_sha256": "2686ecb4c4d70af91f356c8dc79f313ad6788a459681c229945705c98cf781e3", "pod": "afdep-v21-p8a" }, { "log": "pgfin-bf16-small-n0.log", "log_sha256": "9aa9584c4055290a4c10dfd193f635e67b40dd188de8c40313174b055e9559fa", "pod": "afdep-v21-p8a" }, { "log": "pgfin-ctl-AFDEP_HYBRID_ROUTE_GUARD-large-n0.log", "log_sha256": "2686ecb4c4d70af91f356c8dc79f313ad6788a459681c229945705c98cf781e3", "pod": "afdep-v21-p8a" }, { "log": "pgfin-ctl-AFDEP_SHARED_HYBRID_COLLECT-large-n0.log", "log_sha256": "2686ecb4c4d70af91f356c8dc79f313ad6788a459681c229945705c98cf781e3", "pod": "afdep-v21-p8a" }, { "log": "pgfin-ctl-AFDEP_SHARED_HYBRID_FORWARD-large-n0.log", "log_sha256": "2686ecb4c4d70af91f356c8dc79f313ad6788a459681c229945705c98cf781e3", "pod": "afdep-v21-p8a" }, { "log": "pgfin-ctl-AFDEP_SHARED_HYBRID_SEND-large-n0.log", "log_sha256": "2686ecb4c4d70af91f356c8dc79f313ad6788a459681c229945705c98cf781e3", "pod": "afdep-v21-p8a" }, { "log": "pgfin-ctl-AFDEP_SHARED_ROLE_KERNELS-large-n0.log", "log_sha256": "2686ecb4c4d70af91f356c8dc79f313ad6788a459681c229945705c98cf781e3", "pod": "afdep-v21-p8a" }, { "log": "pgfin-fp8-generic-n0.log", "log_sha256": "f92d27de1619ee1fae8b6e70a674100b2f37f1e1785b2c7a8195970a28247ab6", "pod": "afdep-v21-p8a" }, { "log": "pgfin-fp8-large-n0.log", "log_sha256": "22cbe275b22ec4ad05edc6293cec263189b4b4bfe0bc81939f3b94182e0a264f", "pod": "afdep-v21-p8a" }, { "log": "pgfin-fp8-small-n0.log", "log_sha256": "c137e342c8c258c83e80d66bef521ee2db538efb79195127465f109f5c626bb0", "pod": "afdep-v21-p8a" }, { "log": "pgfin-inv-eager-n0.log", "log_sha256": "2ecd76c7dc887bb93ae23201b5fe277ad01ef166bde6e84af4b8cb5ca5be05ca", "pod": "afdep-v21-p8a" }, { "log": "pgfin-inv-graph-n0.log", "log_sha256": "dff73d20a4e54ca099b5c550bacb0ab9c2a29baea65c2b795ac6008327b0e029", "pod": "afdep-v21-p8a" }, { "log": "pgfin-jit-control-n0.log", "log_sha256": "8b87f25179f8b8ad3c239ead6fa5561d394c766607c7e8f2fc59e8087fffa4e5", "pod": "afdep-v21-p8a" }, { "log": "pgfin-jit-split-n0.log", "log_sha256": "0f1333ec34f2d2fc0c42a5f467ede3b92a2d5f3602e5287d661b5659a6d10071", "pod": "afdep-v21-p8a" }, { "log": "pgfin-mixed-attention-large-n0.log", "log_sha256": "2686ecb4c4d70af91f356c8dc79f313ad6788a459681c229945705c98cf781e3", "pod": "afdep-v21-p8a" }, { "log": "pgfin-mixed-ffn-large-n0.log", "log_sha256": "2686ecb4c4d70af91f356c8dc79f313ad6788a459681c229945705c98cf781e3", "pod": "afdep-v21-p8a" }, { "log": "pgfin-bf16-generic-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-p8b" }, { "log": "pgfin-bf16-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-p8b" }, { "log": "pgfin-bf16-small-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-p8b" }, { "log": "pgfin-ctl-AFDEP_HYBRID_ROUTE_GUARD-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-p8b" }, { "log": "pgfin-ctl-AFDEP_SHARED_HYBRID_COLLECT-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-p8b" }, { "log": "pgfin-ctl-AFDEP_SHARED_HYBRID_FORWARD-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-p8b" }, { "log": "pgfin-ctl-AFDEP_SHARED_HYBRID_SEND-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-p8b" }, { "log": "pgfin-ctl-AFDEP_SHARED_ROLE_KERNELS-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-p8b" }, { "log": "pgfin-fp8-generic-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-p8b" }, { "log": "pgfin-fp8-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-p8b" }, { "log": "pgfin-fp8-small-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-p8b" }, { "log": "pgfin-inv-eager-n1.log", "log_sha256": "db96e15159f4bebe55bb5823a03a96a31965a8e91f770bb940a9f7ce24120af8", "pod": "afdep-v21-p8b" }, { "log": "pgfin-inv-graph-n1.log", "log_sha256": "69c2994c97db6b18dcab31b801b69ffc007bdc48bc18d6b42c4a23d6da88630b", "pod": "afdep-v21-p8b" }, { "log": "pgfin-jit-control-n1.log", "log_sha256": "af14620a65e12fa28ef97f181cac322e898899b0f7d544ca4056586d4ca4ea35", "pod": "afdep-v21-p8b" }, { "log": "pgfin-jit-split-n1.log", "log_sha256": "e574a9c0a115e3149b462b0f0dcaaea1e22cee52d6111c50346957470419a648", "pod": "afdep-v21-p8b" }, { "log": "pgfin-mixed-attention-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-p8b" }, { "log": "pgfin-mixed-ffn-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-p8b" } ] }, "c7c4e90e_statler_8a8f": { "result": [ "s8b c7c4e90e97048f899e3c099ca0276c6e877c99baf5bc7cb982ed25ae937e3915", "s8c c7c4e90e97048f899e3c099ca0276c6e877c99baf5bc7cb982ed25ae937e3915", "jit-split: afdep-v21-s8b exit=0 afdep-v21-s8c exit=0", "jit-control: afdep-v21-s8b exit=0 afdep-v21-s8c exit=0", "bf16-small: afdep-v21-s8c exit=0 afdep-v21-s8b exit=0", "bf16-large: afdep-v21-s8b exit=0 afdep-v21-s8c exit=0", "bf16-generic: afdep-v21-s8b exit=0 afdep-v21-s8c exit=0", "fp8-small: afdep-v21-s8b exit=0 afdep-v21-s8c exit=0", "fp8-large: afdep-v21-s8c exit=0 afdep-v21-s8b exit=0", "fp8-generic: afdep-v21-s8c exit=0 afdep-v21-s8b exit=0", "ctl-AFDEP_SHARED_HYBRID_SEND-large: afdep-v21-s8c exit=0 afdep-v21-s8b exit=0", "ctl-AFDEP_SHARED_HYBRID_FORWARD-large: afdep-v21-s8c exit=0 afdep-v21-s8b exit=0", "ctl-AFDEP_SHARED_HYBRID_COLLECT-large: afdep-v21-s8c exit=0 afdep-v21-s8b exit=0", "ctl-AFDEP_SHARED_ROLE_KERNELS-large: afdep-v21-s8c exit=0 afdep-v21-s8b exit=0", "ctl-AFDEP_HYBRID_ROUTE_GUARD-large: afdep-v21-s8b exit=0 afdep-v21-s8c exit=0", "mixed-ffn-large: afdep-v21-s8c exit=0 afdep-v21-s8b exit=0", "mixed-attention-large: afdep-v21-s8c exit=0 afdep-v21-s8b exit=0", "inv-eager: afdep-v21-s8c exit=1 afdep-v21-s8b exit=1", "inv-graph: afdep-v21-s8c exit=1 afdep-v21-s8b exit=1", "afdep-v21-s8c exit=1", "afdep-v21-s8b exit=1" ], "logs": [ { "log": "sx-bf16-generic-n0.log", "log_sha256": "add3eb8c6a6b5d1f03e723ecac3e98fc01fc3cd3f1e047aa68efd046f44fbb63", "pod": "afdep-v21-s8b" }, { "log": "sx-bf16-large-n0.log", "log_sha256": "67f968b25fe3ca9447828449d1c28d569ea5880db960e7a88e9892ce21058b08", "pod": "afdep-v21-s8b" }, { "log": "sx-bf16-small-n0.log", "log_sha256": "dada05b1fa8bc00a8d31a0fa0dbb745da87fd01de2b7d62b33adde847325a60a", "pod": "afdep-v21-s8b" }, { "log": "sx-ctl-AFDEP_HYBRID_ROUTE_GUARD-large-n0.log", "log_sha256": "67f968b25fe3ca9447828449d1c28d569ea5880db960e7a88e9892ce21058b08", "pod": "afdep-v21-s8b" }, { "log": "sx-ctl-AFDEP_SHARED_HYBRID_COLLECT-large-n0.log", "log_sha256": "67f968b25fe3ca9447828449d1c28d569ea5880db960e7a88e9892ce21058b08", "pod": "afdep-v21-s8b" }, { "log": "sx-ctl-AFDEP_SHARED_HYBRID_FORWARD-large-n0.log", "log_sha256": "67f968b25fe3ca9447828449d1c28d569ea5880db960e7a88e9892ce21058b08", "pod": "afdep-v21-s8b" }, { "log": "sx-ctl-AFDEP_SHARED_HYBRID_SEND-large-n0.log", "log_sha256": "67f968b25fe3ca9447828449d1c28d569ea5880db960e7a88e9892ce21058b08", "pod": "afdep-v21-s8b" }, { "log": "sx-ctl-AFDEP_SHARED_ROLE_KERNELS-large-n0.log", "log_sha256": "67f968b25fe3ca9447828449d1c28d569ea5880db960e7a88e9892ce21058b08", "pod": "afdep-v21-s8b" }, { "log": "sx-fp8-generic-n0.log", "log_sha256": "e0aab481cee12d3b3f19d2256966ff19268ced76581b2622fefa8e82efdcff08", "pod": "afdep-v21-s8b" }, { "log": "sx-fp8-large-n0.log", "log_sha256": "bec80d829905b872552a9b0a9789e83f2773d16a681eaddc7beb680db8b8145f", "pod": "afdep-v21-s8b" }, { "log": "sx-fp8-small-n0.log", "log_sha256": "ad58549f603dbb63d916dfb2c04a9cd570dce5c88ca15625d4b2a05f7664372b", "pod": "afdep-v21-s8b" }, { "log": "sx-inv-eager-n0.log", "log_sha256": "3da90b1b93bac3142fa9d0f91fa6969d5f52b2f97d3df9bc61f46665b4c40847", "pod": "afdep-v21-s8b" }, { "log": "sx-inv-graph-n0.log", "log_sha256": "e5a3c44c5da99815c10c14ecb9a47106b6a6051a1c33dbd619c417001f0b2ea8", "pod": "afdep-v21-s8b" }, { "log": "sx-jit-control-n0.log", "log_sha256": "fea5f5b3a44ec0b4619e78a909bf8202c84252d075d9937f9f1977db68d58de1", "pod": "afdep-v21-s8b" }, { "log": "sx-jit-split-n0.log", "log_sha256": "153a7a349c52d8b175020a595c9feb9b12506c67a1a44b227f6679960ac4f537", "pod": "afdep-v21-s8b" }, { "log": "sx-mixed-attention-large-n0.log", "log_sha256": "67f968b25fe3ca9447828449d1c28d569ea5880db960e7a88e9892ce21058b08", "pod": "afdep-v21-s8b" }, { "log": "sx-mixed-ffn-large-n0.log", "log_sha256": "67f968b25fe3ca9447828449d1c28d569ea5880db960e7a88e9892ce21058b08", "pod": "afdep-v21-s8b" }, { "log": "sx-overlap-n0.log", "log_sha256": "e5916e17c797c53b7d8d705dc89e1b834b969c116b6887b4c78cfce3b8919538", "pod": "afdep-v21-s8b" }, { "log": "sx-bf16-generic-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "sx-bf16-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "sx-bf16-small-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "sx-ctl-AFDEP_HYBRID_ROUTE_GUARD-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "sx-ctl-AFDEP_SHARED_HYBRID_COLLECT-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "sx-ctl-AFDEP_SHARED_HYBRID_FORWARD-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "sx-ctl-AFDEP_SHARED_HYBRID_SEND-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "sx-ctl-AFDEP_SHARED_ROLE_KERNELS-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "sx-fp8-generic-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "sx-fp8-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "sx-fp8-small-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "sx-inv-eager-n1.log", "log_sha256": "2d4be86a1fe8a36697241993b8c64c127c53854d48cf1a26c8ecba1ecfdd8079", "pod": "afdep-v21-s8c" }, { "log": "sx-inv-graph-n1.log", "log_sha256": "883f97ea1bc214a0f445092b897e9af0a2f2efe54e290d54a791d9d3634a798d", "pod": "afdep-v21-s8c" }, { "log": "sx-jit-control-n1.log", "log_sha256": "52ebaa34f66716179a98c80e27136f592e2528ff26b46eee6e785ad639cac4c1", "pod": "afdep-v21-s8c" }, { "log": "sx-jit-split-n1.log", "log_sha256": "2e3cb3069ff68209da3733eacd4f8e2b2083e24dafc17606d9afb6abeaf4974f", "pod": "afdep-v21-s8c" }, { "log": "sx-mixed-attention-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "sx-mixed-ffn-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "sx-overlap-n1.log", "log_sha256": "f03ac15d9e16f06ac88b44e349d902152ba35b750a8ab366c47ecd456dcfa850", "pod": "afdep-v21-s8c" } ] }, "26cd2d09_statler_multi_domain": { "result": [ "L3-sup2-2f2a-bf16-small: afdep-v21-s8c exit=0 afdep-v21-s8b exit=0 afdep-v21-s8a exit=0 afdep-v21-s8d exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 8 \"native_source_sha256\": \"26cd2d09 ]", "L3-sup2-2f2a-bf16-large: afdep-v21-s8c exit=0 afdep-v21-s8a exit=0 afdep-v21-s8b exit=0 afdep-v21-s8d exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 8 \"native_source_sha256\": \"26cd2d09 ]", "L3-sup2-2f2a-fp8-large: afdep-v21-s8b exit=0 afdep-v21-s8c exit=0 afdep-v21-s8a exit=0 afdep-v21-s8d exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 8 \"native_source_sha256\": \"26cd2d09 ]", "L3-sup2-2f2a-h7168: afdep-v21-s8a exit=0 afdep-v21-s8d exit=0 afdep-v21-s8c exit=0 afdep-v21-s8b exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 8 \"native_source_sha256\": \"26cd2d09 ]", "L3-sup2-2f2a-mixed-ffn: afdep-v21-s8a exit=0 afdep-v21-s8c exit=0 afdep-v21-s8d exit=0 afdep-v21-s8b exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 8 \"native_source_sha256\": \"26cd2d09 ]", "L3-sup2-2f2a-mixed-att: afdep-v21-s8b exit=0 afdep-v21-s8d exit=0 afdep-v21-s8a exit=0 afdep-v21-s8c exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 8 \"native_source_sha256\": \"26cd2d09 ]", "L3-sup2-2f2a-inv-graph: afdep-v21-s8c exit=1 afdep-v21-s8b exit=1 afdep-v21-s8d exit=1 afdep-v21-s8a exit=1 mismatch=0 timeout=268 fault_markers=0 digests=[]", "L4-sup4-2f2a-bf16-small: afdep-v21-s8b exit=0 afdep-v21-s8d exit=0 afdep-v21-s8a exit=0 afdep-v21-s8c exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 16 \"native_source_sha256\": \"26cd2d09 ]", "L4-sup4-2f2a-bf16-large: afdep-v21-s8c exit=0 afdep-v21-s8b exit=0 afdep-v21-s8a exit=0 afdep-v21-s8d exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 16 \"native_source_sha256\": \"26cd2d09 ]", "L4-sup4-2f2a-fp8-large: afdep-v21-s8d exit=0 afdep-v21-s8a exit=0 afdep-v21-s8b exit=0 afdep-v21-s8c exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 16 \"native_source_sha256\": \"26cd2d09 ]", "L4-sup4-2f2a-h7168: afdep-v21-s8c exit=0 afdep-v21-s8b exit=0 afdep-v21-s8d exit=0 afdep-v21-s8a exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 16 \"native_source_sha256\": \"26cd2d09 ]", "L4-sup4-2f2a-mixed-ffn: afdep-v21-s8d exit=0 afdep-v21-s8c exit=0 afdep-v21-s8b exit=0 afdep-v21-s8a exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 16 \"native_source_sha256\": \"26cd2d09 ]", "L4-sup4-2f2a-mixed-att: afdep-v21-s8b exit=0 afdep-v21-s8a exit=0 afdep-v21-s8d exit=0 afdep-v21-s8c exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 16 \"native_source_sha256\": \"26cd2d09 ]", "L4-sup4-2f2a-inv-graph: afdep-v21-s8c exit=1 afdep-v21-s8b exit=1 afdep-v21-s8d exit=1 afdep-v21-s8a exit=1 mismatch=0 timeout=528 fault_markers=0 digests=[]", "L1-sup1-2f2a-bf16-small: afdep-v21-s8c exit=0 afdep-v21-s8d exit=0 afdep-v21-s8b exit=0 afdep-v21-s8a exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 4 \"native_source_sha256\": \"26cd2d09 ]", "L1-sup1-2f2a-bf16-large: afdep-v21-s8b exit=0 afdep-v21-s8a exit=0 afdep-v21-s8d exit=0 afdep-v21-s8c exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 4 \"native_source_sha256\": \"26cd2d09 ]", "L1-sup1-2f2a-fp8-large: afdep-v21-s8d exit=0 afdep-v21-s8c exit=0 afdep-v21-s8a exit=0 afdep-v21-s8b exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 4 \"native_source_sha256\": \"26cd2d09 ]", "L1-sup1-2f2a-h7168: afdep-v21-s8d exit=0 afdep-v21-s8a exit=0 afdep-v21-s8c exit=0 afdep-v21-s8b exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 4 \"native_source_sha256\": \"26cd2d09 ]", "L1-sup1-2f2a-mixed-ffn: afdep-v21-s8b exit=0 afdep-v21-s8d exit=0 afdep-v21-s8c exit=0 afdep-v21-s8a exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 4 \"native_source_sha256\": \"26cd2d09 ]", "L1-sup1-2f2a-mixed-att: afdep-v21-s8c exit=0 afdep-v21-s8a exit=0 afdep-v21-s8d exit=0 afdep-v21-s8b exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 4 \"native_source_sha256\": \"26cd2d09 ]", "L1-sup1-2f2a-inv-graph: afdep-v21-s8c exit=1 afdep-v21-s8d exit=1 afdep-v21-s8b exit=1 afdep-v21-s8a exit=1 mismatch=0 timeout=138 fault_markers=0 digests=[]", "L2-sup1-2f3a-bf16-small: afdep-v21-s4e exit=0 afdep-v21-s8c exit=0 afdep-v21-s8a exit=0 afdep-v21-s8d exit=0 afdep-v21-s8b exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 5 \"native_source_sha256\": \"26cd2d09 ]", "L2-sup1-2f3a-bf16-large: afdep-v21-s4e exit=0 afdep-v21-s8d exit=0 afdep-v21-s8c exit=0 afdep-v21-s8b exit=0 afdep-v21-s8a exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 5 \"native_source_sha256\": \"26cd2d09 ]", "L2-sup1-2f3a-fp8-large: afdep-v21-s4e exit=0 afdep-v21-s8b exit=0 afdep-v21-s8a exit=0 afdep-v21-s8c exit=0 afdep-v21-s8d exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 5 \"native_source_sha256\": \"26cd2d09 ]", "L2-sup1-2f3a-h7168: afdep-v21-s4e exit=0 afdep-v21-s8a exit=0 afdep-v21-s8c exit=0 afdep-v21-s8d exit=0 afdep-v21-s8b exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 5 \"native_source_sha256\": \"26cd2d09 ]", "L2-sup1-2f3a-mixed-ffn: afdep-v21-s4e exit=0 afdep-v21-s8b exit=0 afdep-v21-s8a exit=0 afdep-v21-s8d exit=0 afdep-v21-s8c exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 5 \"native_source_sha256\": \"26cd2d09 ]", "L2-sup1-2f3a-mixed-att: afdep-v21-s4e exit=0 afdep-v21-s8b exit=0 afdep-v21-s8a exit=0 afdep-v21-s8d exit=0 afdep-v21-s8c exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 5 \"native_source_sha256\": \"26cd2d09 ]", "L2-sup1-2f3a-inv-graph: afdep-v21-s8c exit=1 afdep-v21-s4e exit=1 afdep-v21-s8b exit=1 afdep-v21-s8d exit=1 afdep-v21-s8a exit=1 mismatch=0 timeout=268 fault_markers=0 digests=[]", "L5-sup4-2f3a-bf16-small: afdep-v21-s4e exit=0 afdep-v21-s8d exit=0 afdep-v21-s8c exit=0 afdep-v21-s8a exit=0 afdep-v21-s8b exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 20 \"native_source_sha256\": \"26cd2d09 ]", "L5-sup4-2f3a-bf16-large: afdep-v21-s4e exit=0 afdep-v21-s8c exit=0 afdep-v21-s8a exit=0 afdep-v21-s8d exit=0 afdep-v21-s8b exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 20 \"native_source_sha256\": \"26cd2d09 ]", "L5-sup4-2f3a-fp8-large: afdep-v21-s4e exit=0 afdep-v21-s8d exit=0 afdep-v21-s8b exit=0 afdep-v21-s8a exit=0 afdep-v21-s8c exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 20 \"native_source_sha256\": \"26cd2d09 ]", "L5-sup4-2f3a-h7168: afdep-v21-s4e exit=0 afdep-v21-s8d exit=0 afdep-v21-s8c exit=0 afdep-v21-s8a exit=0 afdep-v21-s8b exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 20 \"native_source_sha256\": \"26cd2d09 ]", "L5-sup4-2f3a-mixed-ffn: afdep-v21-s4e exit=0 afdep-v21-s8a exit=0 afdep-v21-s8c exit=0 afdep-v21-s8b exit=0 afdep-v21-s8d exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 20 \"native_source_sha256\": \"26cd2d09 ]", "L5-sup4-2f3a-mixed-att: afdep-v21-s4e exit=0 afdep-v21-s8a exit=0 afdep-v21-s8b exit=0 afdep-v21-s8d exit=0 afdep-v21-s8c exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 20 \"native_source_sha256\": \"26cd2d09 ]", "L5-sup4-2f3a-inv-graph: afdep-v21-s8c exit=1 afdep-v21-s8d exit=1 afdep-v21-s4e exit=1 afdep-v21-s8b exit=1 afdep-v21-s8a exit=1 mismatch=0 timeout=1035 fault_markers=0 digests=[]", "L6a-sup8-2f1a-bf16-small: afdep-v21-s8c exit=0 afdep-v21-s8a exit=0 afdep-v21-s8b exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 24 \"native_source_sha256\": \"26cd2d09 ]", "L6a-sup8-2f1a-bf16-large: afdep-v21-s8c exit=0 afdep-v21-s8a exit=0 afdep-v21-s8b exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 24 \"native_source_sha256\": \"26cd2d09 ]", "L6a-sup8-2f1a-fp8-large: afdep-v21-s8b exit=0 afdep-v21-s8c exit=0 afdep-v21-s8a exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 24 \"native_source_sha256\": \"26cd2d09 ]", "L6a-sup8-2f1a-h7168: afdep-v21-s8c exit=0 afdep-v21-s8a exit=0 afdep-v21-s8b exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 24 \"native_source_sha256\": \"26cd2d09 ]", "L6a-sup8-2f1a-mixed-ffn: afdep-v21-s8a exit=0 afdep-v21-s8b exit=0 afdep-v21-s8c exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 24 \"native_source_sha256\": \"26cd2d09 ]", "L6a-sup8-2f1a-mixed-att: afdep-v21-s8c exit=0 afdep-v21-s8a exit=0 afdep-v21-s8b exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 24 \"native_source_sha256\": \"26cd2d09 ]", "L6a-sup8-2f1a-inv-graph: afdep-v21-s8c exit=1 afdep-v21-s8b exit=1 afdep-v21-s8a exit=1 mismatch=0 timeout=22 fault_markers=0 digests=[]", "L6b-sup8-1f2a-bf16-small: afdep-v21-s8b exit=0 afdep-v21-s8c exit=0 afdep-v21-s8a exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 24 \"native_source_sha256\": \"26cd2d09 ]", "L6b-sup8-1f2a-bf16-large: afdep-v21-s8a exit=0 afdep-v21-s8b exit=0 afdep-v21-s8c exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 24 \"native_source_sha256\": \"26cd2d09 ]", "L6b-sup8-1f2a-fp8-large: afdep-v21-s8c exit=0 afdep-v21-s8a exit=0 afdep-v21-s8b exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 24 \"native_source_sha256\": \"26cd2d09 ]", "L6b-sup8-1f2a-h7168: afdep-v21-s8a exit=0 afdep-v21-s8b exit=0 afdep-v21-s8c exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 24 \"native_source_sha256\": \"26cd2d09 ]", "L6b-sup8-1f2a-mixed-ffn: afdep-v21-s8c exit=0 afdep-v21-s8b exit=0 afdep-v21-s8a exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 24 \"native_source_sha256\": \"26cd2d09 ]", "L6b-sup8-1f2a-mixed-att: afdep-v21-s8a exit=0 afdep-v21-s8c exit=0 afdep-v21-s8b exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 24 \"native_source_sha256\": \"26cd2d09 ]", "L6b-sup8-1f2a-inv-graph: afdep-v21-s8b exit=1 afdep-v21-s8c exit=1 afdep-v21-s8a exit=1 mismatch=0 timeout=523 fault_markers=0 digests=[]", "L3-h8192: afdep-v21-s8d exit=0 afdep-v21-s8c exit=0 afdep-v21-s8b exit=0 afdep-v21-s8a exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 8 \"native_source_sha256\": \"26cd2d09 ]", "L3-sms23: afdep-v21-s8c exit=0 afdep-v21-s8b exit=0 afdep-v21-s8d exit=0 afdep-v21-s8a exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 8 \"native_source_sha256\": \"26cd2d09 ]" ], "logs": [ { "log": "vg-F89-large-n0.log", "log_sha256": "de8003ae1396f7874189f1af334beb68664ac9e9e5a9384b77135f9f3c287881", "pod": "afdep-v21-s8a" }, { "log": "vg-F89-small-n0.log", "log_sha256": "0b98f40fa88c77236b560a80d67be66020e75d26f8487bb582f7cb183c0fe589", "pod": "afdep-v21-s8a" }, { "log": "vg-L1-sup1-2f2a-bf16-large-n0.log", "log_sha256": "9419bbebce5bda170d3e63d46ea5442277a8a0cf1310cc66419cd3c3beced4b6", "pod": "afdep-v21-s8a" }, { "log": "vg-L1-sup1-2f2a-bf16-small-n0.log", "log_sha256": "018c9d032cbd5fae077bb4b43a903a5cf840569e6972d1ab090e49cd537d52fb", "pod": "afdep-v21-s8a" }, { "log": "vg-L1-sup1-2f2a-fp8-large-n0.log", "log_sha256": "f11deac926dc6e132cbcf6775d9e6c8e49c130337e8c695fb295dfd0f7405198", "pod": "afdep-v21-s8a" }, { "log": "vg-L1-sup1-2f2a-h7168-n0.log", "log_sha256": "5336e8d5bca1ea447dcbd9c182e7da7ed70ae49b492766a9dcf3b3ea579713fe", "pod": "afdep-v21-s8a" }, { "log": "vg-L1-sup1-2f2a-inv-graph-n0.log", "log_sha256": "5ee819732e9fb7670c7ae312f8577dd1c48acf5b2aa790a1933382ff4e820f6a", "pod": "afdep-v21-s8a" }, { "log": "vg-L1-sup1-2f2a-mixed-att-n0.log", "log_sha256": "8b959cf8c842a77c132345acb314911bdabf547211231a5537e54becec4acbb0", "pod": "afdep-v21-s8a" }, { "log": "vg-L1-sup1-2f2a-mixed-ffn-n0.log", "log_sha256": "f40aac7e50f05d940a0eb24762cf38e605d1476da263caede38115e23efb2e44", "pod": "afdep-v21-s8a" }, { "log": "vg-L2-sup1-2f3a-bf16-large-n0.log", "log_sha256": "094ebc609987fc603956364773fee04c1a0cde078f50d5f092f0f1548595495f", "pod": "afdep-v21-s8a" }, { "log": "vg-L2-sup1-2f3a-bf16-small-n0.log", "log_sha256": "a9b61e42cad844c33500fd224e559c89367b688553429ef2b4b7301720b76b4a", "pod": "afdep-v21-s8a" }, { "log": "vg-L2-sup1-2f3a-fp8-large-n0.log", "log_sha256": "838fc97cc09f655709932a8f83615cd5db61618e18ca849b3df6f6e44f0d594f", "pod": "afdep-v21-s8a" }, { "log": "vg-L2-sup1-2f3a-h7168-n0.log", "log_sha256": "cac8ba8d2269e07f684510932eee5c259f74c775260ff41556341a1565b70cea", "pod": "afdep-v21-s8a" }, { "log": "vg-L2-sup1-2f3a-inv-graph-n0.log", "log_sha256": "536bbb870cf139862459ecca6386294934a653f50202e86a099f827426c9aa83", "pod": "afdep-v21-s8a" }, { "log": "vg-L2-sup1-2f3a-mixed-att-n0.log", "log_sha256": "b3cda1f15d29cede1e43f9dc067744b448ebb194b50febd4516ff6807be3d96f", "pod": "afdep-v21-s8a" }, { "log": "vg-L2-sup1-2f3a-mixed-ffn-n0.log", "log_sha256": "01c2c1915e3c32afd74db3ed287bfd450218997af83f442982d8906d1db3fb5a", "pod": "afdep-v21-s8a" }, { "log": "vg-L3-h8192-n0.log", "log_sha256": "3ec3c68468e9aefbeedf8050b58a3333eefc2c1f76c72e879302bc14ab2f100e", "pod": "afdep-v21-s8a" }, { "log": "vg-L3-sms23-n0.log", "log_sha256": "19619f926d19531c03a8f5d5132500bc1adb4c2986054feabf4aff3ff3173684", "pod": "afdep-v21-s8a" }, { "log": "vg-L3-sup2-2f2a-bf16-large-n0.log", "log_sha256": "d548a0f3215411b95264802f5df82e1608e8249fece04ff1f9e3abb41df2a761", "pod": "afdep-v21-s8a" }, { "log": "vg-L3-sup2-2f2a-bf16-small-n0.log", "log_sha256": "75a4e02b690b9baa4f11472674c3b83bcf0b90aff60816960f9a7f1f03a5e5a9", "pod": "afdep-v21-s8a" }, { "log": "vg-L3-sup2-2f2a-fp8-large-n0.log", "log_sha256": "0ffbbfaadecdecf43f54259c74d4dfddea0146e883c842975a427c043335b3d0", "pod": "afdep-v21-s8a" }, { "log": "vg-L3-sup2-2f2a-h7168-n0.log", "log_sha256": "3bff8c8e1faf69c548a44b7a56456cba346eb03aa173744ca88e2804948db5a0", "pod": "afdep-v21-s8a" }, { "log": "vg-L3-sup2-2f2a-inv-graph-n0.log", "log_sha256": "ac18e43b25d07b803e878d3a991513eae224e3b59cdfbca5ce518ca5bfc85f86", "pod": "afdep-v21-s8a" }, { "log": "vg-L3-sup2-2f2a-mixed-att-n0.log", "log_sha256": "3ad318f178374dc4c657727f493b98e416f70e3d37d33d455928b041024a2932", "pod": "afdep-v21-s8a" }, { "log": "vg-L3-sup2-2f2a-mixed-ffn-n0.log", "log_sha256": "66bdff0e34c7bccb78086f860cd239670801f04df156f6f1498610255590c8d7", "pod": "afdep-v21-s8a" }, { "log": "vg-L4-sup4-2f2a-bf16-large-n0.log", "log_sha256": "8c9d9a8c667a9e1ac5830b989507411105d375fbb140e02009d4332e3e49d540", "pod": "afdep-v21-s8a" }, { "log": "vg-L4-sup4-2f2a-bf16-small-n0.log", "log_sha256": "504755ef743936e3747445e4139f21ae8be60bde5c34a670c92e0596f7d79a9a", "pod": "afdep-v21-s8a" }, { "log": "vg-L4-sup4-2f2a-fp8-large-n0.log", "log_sha256": "615f37794391a61c4d346e21c330ef96b43f17cdf43020e3443d4f40a6f5f044", "pod": "afdep-v21-s8a" }, { "log": "vg-L4-sup4-2f2a-h7168-n0.log", "log_sha256": "402ab7851a7d1471fcd749b2eafeb2c99e0a7b255b892bc05b0d70cbf29142db", "pod": "afdep-v21-s8a" }, { "log": "vg-L4-sup4-2f2a-inv-graph-n0.log", "log_sha256": "28ed41d3ddb34c4c32673e47044c2913c2ff1bb529478540561dd6164db05ae8", "pod": "afdep-v21-s8a" }, { "log": "vg-L4-sup4-2f2a-mixed-att-n0.log", "log_sha256": "7ed07994c4f9f8a4f3db194904610bb916379aca4ac1e927deb1a6303dc93d9d", "pod": "afdep-v21-s8a" }, { "log": "vg-L4-sup4-2f2a-mixed-ffn-n0.log", "log_sha256": "329f24162dbf46b83da6cc8a8ce29ef11d459311eb2dd121aa6b93ca4c5bb03a", "pod": "afdep-v21-s8a" }, { "log": "vg-L5-sup4-2f3a-bf16-large-n0.log", "log_sha256": "779328adfa9971f7690fa5a0fb6769b422243c7cc6786f7210a537e4c1f451eb", "pod": "afdep-v21-s8a" }, { "log": "vg-L5-sup4-2f3a-bf16-small-n0.log", "log_sha256": "4bdce802efdad639e32932ff9f5d205bac6bfadfafff62b5fc971b366e91f83d", "pod": "afdep-v21-s8a" }, { "log": "vg-L5-sup4-2f3a-fp8-large-n0.log", "log_sha256": "680a7f90a3bdf244a051c2743f06d707520d6b445cccede3632449512a36c08c", "pod": "afdep-v21-s8a" }, { "log": "vg-L5-sup4-2f3a-h7168-n0.log", "log_sha256": "61f2794136ad5987735226bcd0c6cad597189699717b414fe6888770f15b33c4", "pod": "afdep-v21-s8a" }, { "log": "vg-L5-sup4-2f3a-inv-graph-n0.log", "log_sha256": "8d72458c2ba05f4dc60defb28549121dff0eb4826e1a575a7d31d4a27c81b009", "pod": "afdep-v21-s8a" }, { "log": "vg-L5-sup4-2f3a-mixed-att-n0.log", "log_sha256": "97df98cd7baf192460c7141e73e212aa2c6f1deca519bda08a3b3b7530ddebd8", "pod": "afdep-v21-s8a" }, { "log": "vg-L5-sup4-2f3a-mixed-ffn-n0.log", "log_sha256": "24a8eb51687630559902f12d4be8bcc123b45d4bab13c8b324e0beb0b1980823", "pod": "afdep-v21-s8a" }, { "log": "vg-L6a-sup8-2f1a-bf16-large-n0.log", "log_sha256": "40796c3b6efb73a9033dd5e0a6f934037f6cfb609c1defe30b47914e53680b5d", "pod": "afdep-v21-s8a" }, { "log": "vg-L6a-sup8-2f1a-bf16-small-n0.log", "log_sha256": "e94b1597fc6f7cb5ae42d0c7a0eb8492a18c66ba9e66ac985f1f1cb74f6ee65b", "pod": "afdep-v21-s8a" }, { "log": "vg-L6a-sup8-2f1a-fp8-large-n0.log", "log_sha256": "456a5018527a147934e129fb7a825105b4f0db24af6d91a00612f7dbb2340945", "pod": "afdep-v21-s8a" }, { "log": "vg-L6a-sup8-2f1a-h7168-n0.log", "log_sha256": "5efcf88faa76ba84b07b052955fcafe942565badd84be29ceaf982c7c9de006a", "pod": "afdep-v21-s8a" }, { "log": "vg-L6a-sup8-2f1a-inv-graph-n0.log", "log_sha256": "bbb08824448445ec691280a3c71a2234eb03233b7b6b33dd1bcbbec230c6e184", "pod": "afdep-v21-s8a" }, { "log": "vg-L6a-sup8-2f1a-mixed-att-n0.log", "log_sha256": "32e105d421399b5159106ff367c8fb9c4204d2f9726a47bed4499f9859a6319f", "pod": "afdep-v21-s8a" }, { "log": "vg-L6a-sup8-2f1a-mixed-ffn-n0.log", "log_sha256": "e9cd6af80f37bcd06fad500d678717173c5fce821013aad64fa60abb3e2b0ebe", "pod": "afdep-v21-s8a" }, { "log": "vg-L6b-sup8-1f2a-bf16-large-n0.log", "log_sha256": "c64a8146220582ef62f210e73f0a16e5f86d696d9725d6db674d0da4e63b2603", "pod": "afdep-v21-s8a" }, { "log": "vg-L6b-sup8-1f2a-bf16-small-n0.log", "log_sha256": "57c0d224ff11b0123fb7b029c614fb867ec27a9cae5c7374609044c3265d0191", "pod": "afdep-v21-s8a" }, { "log": "vg-L6b-sup8-1f2a-fp8-large-n0.log", "log_sha256": "fd6e643a944e3dc18fbe5aadd524997fa8f93a3ef27d735f2d086070128ba847", "pod": "afdep-v21-s8a" }, { "log": "vg-L6b-sup8-1f2a-h7168-n0.log", "log_sha256": "09326da65501b34fe1f5523e06dbbcc4b79d35a55b269d602e8a01250702ab0a", "pod": "afdep-v21-s8a" }, { "log": "vg-L6b-sup8-1f2a-inv-graph-n0.log", "log_sha256": "1040a79de2d1132a6693bab77147736f3ff23ed2479625b1545b59bdacca128e", "pod": "afdep-v21-s8a" }, { "log": "vg-L6b-sup8-1f2a-mixed-att-n0.log", "log_sha256": "43100585d99a7dec0fdec59f8a9ec0d04a20c8d3979eda8386446054acb0e454", "pod": "afdep-v21-s8a" }, { "log": "vg-L6b-sup8-1f2a-mixed-ffn-n0.log", "log_sha256": "6d1537a7f82fa104e76855bd98a001fae3a4652955d272ea7df5884f8d9e6cf1", "pod": "afdep-v21-s8a" }, { "log": "vg-NEGA-large-n0.log", "log_sha256": "b81b6d644ea97dbfb5881b8854bff1562302576f086748ac5e748cf5bfd99dd4", "pod": "afdep-v21-s8a" }, { "log": "vg-NEGB-large-n0.log", "log_sha256": "432402cda29a0701cb78f5f51f76b6e5cfd9db475c12fd8bdc9fa59365acc5fd", "pod": "afdep-v21-s8a" }, { "log": "vg-NEGC-large-n0.log", "log_sha256": "d61a24a54ecbae2638e55c8de63a97f9634de4f8b256e93835fbff4db9b9cd79", "pod": "afdep-v21-s8a" }, { "log": "vg-ctl-L3-AFDEP_ATTENTION_GRID_SYNC-n0.log", "log_sha256": "b1c514b035e8449e672c393c11751e0a017d0694b16841490ab8c9bc634d4933", "pod": "afdep-v21-s8a" }, { "log": "vg-ctl-L3-AFDEP_HYBRID_ROUTE_GUARD-n0.log", "log_sha256": "ffc298a445644aa0816dbfbcc8ff7cc09f2358583e61cee5ffff126b793d3547", "pod": "afdep-v21-s8a" }, { "log": "vg-ctl-L3-AFDEP_SHARED_HYBRID_COLLECT-n0.log", "log_sha256": "131833dc5eb4a7bce2a0d4e8d163f7ca2d0a18c2becfe79563ee265b15332ba4", "pod": "afdep-v21-s8a" }, { "log": "vg-ctl-L3-AFDEP_SHARED_HYBRID_FORWARD-n0.log", "log_sha256": "d4ff642e375b6de9bf5e807d7d4e1bdceaa384f7b429dfe555f0a9ce93988de6", "pod": "afdep-v21-s8a" }, { "log": "vg-ctl-L3-AFDEP_SHARED_HYBRID_SEND-n0.log", "log_sha256": "219fc1ec5fd80602c30c860ad9617910e5b0466a73d17a68bba051b47aa95b5a", "pod": "afdep-v21-s8a" }, { "log": "vg-ctl-L3-AFDEP_SHARED_ROLE_KERNELS-n0.log", "log_sha256": "aed732919bb039e88acf6cb180cfbe6f7e3eedc9068187e092767ed7179ffcf4", "pod": "afdep-v21-s8a" }, { "log": "vg-ctl-L3-AFDEP_UNION_HYBRID-n0.log", "log_sha256": "db01759252829b6ee6aaf0c90d177159b97ef93a033e063eada82eca74618e51", "pod": "afdep-v21-s8a" }, { "log": "vg-F89-large-n1.log", "log_sha256": "3faad4657002f636c2ff64ad19be47c5ab8ac5b507491b7eae16ab7e05ff194d", "pod": "afdep-v21-s8b" }, { "log": "vg-F89-small-n1.log", "log_sha256": "7e20993518e6d2d2f47aa45c55a1ed66e255a9186a1d2713cba78504164fb7d8", "pod": "afdep-v21-s8b" }, { "log": "vg-L1-sup1-2f2a-bf16-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L1-sup1-2f2a-bf16-small-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L1-sup1-2f2a-fp8-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L1-sup1-2f2a-h7168-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L1-sup1-2f2a-inv-graph-n1.log", "log_sha256": "1ad06994573aba2d50cd83571392659492750aefa6ecabfddeb6da3f30c3c88c", "pod": "afdep-v21-s8b" }, { "log": "vg-L1-sup1-2f2a-mixed-att-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L1-sup1-2f2a-mixed-ffn-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L2-sup1-2f3a-bf16-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L2-sup1-2f3a-bf16-small-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L2-sup1-2f3a-fp8-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L2-sup1-2f3a-h7168-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L2-sup1-2f3a-inv-graph-n1.log", "log_sha256": "340ecc317c35e6ea31b58a1f84ef64eab3eca3aedad71904ae4a31dac8b2d40f", "pod": "afdep-v21-s8b" }, { "log": "vg-L2-sup1-2f3a-mixed-att-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L2-sup1-2f3a-mixed-ffn-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L3-h8192-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L3-sms23-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L3-sup2-2f2a-bf16-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L3-sup2-2f2a-bf16-small-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L3-sup2-2f2a-fp8-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L3-sup2-2f2a-h7168-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L3-sup2-2f2a-inv-graph-n1.log", "log_sha256": "c91ec70afc934351952c5ab54777e7fce4c1309dccc61aaa7a9eefa287f42a14", "pod": "afdep-v21-s8b" }, { "log": "vg-L3-sup2-2f2a-mixed-att-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L3-sup2-2f2a-mixed-ffn-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L4-sup4-2f2a-bf16-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L4-sup4-2f2a-bf16-small-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L4-sup4-2f2a-fp8-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L4-sup4-2f2a-h7168-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L4-sup4-2f2a-inv-graph-n1.log", "log_sha256": "11e7927d3932f10048e99afc4e0420233316cd00cf2b0c745b2af3912a1ace38", "pod": "afdep-v21-s8b" }, { "log": "vg-L4-sup4-2f2a-mixed-att-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L4-sup4-2f2a-mixed-ffn-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L5-sup4-2f3a-bf16-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L5-sup4-2f3a-bf16-small-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L5-sup4-2f3a-fp8-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L5-sup4-2f3a-h7168-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L5-sup4-2f3a-inv-graph-n1.log", "log_sha256": "9b4469f2e107781c37402c848981be093b2b90a57c7a1e8636e30536f579f156", "pod": "afdep-v21-s8b" }, { "log": "vg-L5-sup4-2f3a-mixed-att-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L5-sup4-2f3a-mixed-ffn-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L6a-sup8-2f1a-bf16-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L6a-sup8-2f1a-bf16-small-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L6a-sup8-2f1a-fp8-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L6a-sup8-2f1a-h7168-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L6a-sup8-2f1a-inv-graph-n1.log", "log_sha256": "1085fc9153cb2f6f7402256b6d660b1829bd7b93610360b35e7cfee656f170a8", "pod": "afdep-v21-s8b" }, { "log": "vg-L6a-sup8-2f1a-mixed-att-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L6a-sup8-2f1a-mixed-ffn-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L6b-sup8-1f2a-bf16-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L6b-sup8-1f2a-bf16-small-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L6b-sup8-1f2a-fp8-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L6b-sup8-1f2a-h7168-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L6b-sup8-1f2a-inv-graph-n1.log", "log_sha256": "2569ca41d4773882bddaea5cd416b4f4a31823e6532b3020ab8cfae033f03cf7", "pod": "afdep-v21-s8b" }, { "log": "vg-L6b-sup8-1f2a-mixed-att-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-L6b-sup8-1f2a-mixed-ffn-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-NEGA-large-n1.log", "log_sha256": "ce409d6b7a875c28ae49a2f291949945cc542bec3c3e2f23cb6e09231739502e", "pod": "afdep-v21-s8b" }, { "log": "vg-NEGB-large-n1.log", "log_sha256": "501b7e6ae8950a9044acfbd9b1c424241998d7f7a192f6cda6689df57132d8de", "pod": "afdep-v21-s8b" }, { "log": "vg-NEGC-large-n1.log", "log_sha256": "73435ff55d87d62ee7191d0ca3267fb13b7a29e65196518c891ec9fffacf083a", "pod": "afdep-v21-s8b" }, { "log": "vg-ctl-L3-AFDEP_ATTENTION_GRID_SYNC-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-ctl-L3-AFDEP_HYBRID_ROUTE_GUARD-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-ctl-L3-AFDEP_SHARED_HYBRID_COLLECT-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-ctl-L3-AFDEP_SHARED_HYBRID_FORWARD-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-ctl-L3-AFDEP_SHARED_HYBRID_SEND-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-ctl-L3-AFDEP_SHARED_ROLE_KERNELS-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-ctl-L3-AFDEP_UNION_HYBRID-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "vg-F89-large-n2.log", "log_sha256": "ac4f87170e7dd5d7a65283d65015491e7f004fe6130c24dc0b378d99a1a3c8b9", "pod": "afdep-v21-s8c" }, { "log": "vg-F89-small-n2.log", "log_sha256": "a2cffb337453b2609fcc34053eeeb98fdd56312542ee69ce20279918716c4412", "pod": "afdep-v21-s8c" }, { "log": "vg-L1-sup1-2f2a-bf16-large-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L1-sup1-2f2a-bf16-small-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L1-sup1-2f2a-fp8-large-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L1-sup1-2f2a-h7168-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L1-sup1-2f2a-inv-graph-n2.log", "log_sha256": "79b763083f6bac6878bdaa000d3106ee17514bcb3416700dc8cb6396b89118d0", "pod": "afdep-v21-s8c" }, { "log": "vg-L1-sup1-2f2a-mixed-att-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L1-sup1-2f2a-mixed-ffn-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L2-sup1-2f3a-bf16-large-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L2-sup1-2f3a-bf16-small-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L2-sup1-2f3a-fp8-large-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L2-sup1-2f3a-h7168-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L2-sup1-2f3a-inv-graph-n2.log", "log_sha256": "da17781027ef292789598531a9d161ab4dc96609454e3c660d890ef48d93158f", "pod": "afdep-v21-s8c" }, { "log": "vg-L2-sup1-2f3a-mixed-att-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L2-sup1-2f3a-mixed-ffn-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L3-h8192-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L3-sms23-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L3-sup2-2f2a-bf16-large-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L3-sup2-2f2a-bf16-small-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L3-sup2-2f2a-fp8-large-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L3-sup2-2f2a-h7168-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L3-sup2-2f2a-inv-graph-n2.log", "log_sha256": "2a5604bda568f624b6091e99ec32b8a0c94026ad077d72933a5b58f12ef34126", "pod": "afdep-v21-s8c" }, { "log": "vg-L3-sup2-2f2a-mixed-att-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L3-sup2-2f2a-mixed-ffn-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L4-sup4-2f2a-bf16-large-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L4-sup4-2f2a-bf16-small-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L4-sup4-2f2a-fp8-large-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L4-sup4-2f2a-h7168-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L4-sup4-2f2a-inv-graph-n2.log", "log_sha256": "9f4c0d06d19a7972513f0a70fa1a9c7f7e4f96504a554c3d32e9b0966c75b792", "pod": "afdep-v21-s8c" }, { "log": "vg-L4-sup4-2f2a-mixed-att-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L4-sup4-2f2a-mixed-ffn-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L5-sup4-2f3a-bf16-large-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L5-sup4-2f3a-bf16-small-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L5-sup4-2f3a-fp8-large-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L5-sup4-2f3a-h7168-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L5-sup4-2f3a-inv-graph-n2.log", "log_sha256": "9d9b75fda423936c7c86235fccbc33dce29430f836257778444554528b1397d8", "pod": "afdep-v21-s8c" }, { "log": "vg-L5-sup4-2f3a-mixed-att-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L5-sup4-2f3a-mixed-ffn-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L6a-sup8-2f1a-bf16-large-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L6a-sup8-2f1a-bf16-small-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L6a-sup8-2f1a-fp8-large-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L6a-sup8-2f1a-h7168-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L6a-sup8-2f1a-inv-graph-n2.log", "log_sha256": "dceb2267548c2ad2fd4f4b3b957f23abc8fa621e9e7f7f44a9810343c63bf536", "pod": "afdep-v21-s8c" }, { "log": "vg-L6a-sup8-2f1a-mixed-att-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L6a-sup8-2f1a-mixed-ffn-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L6b-sup8-1f2a-bf16-large-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L6b-sup8-1f2a-bf16-small-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L6b-sup8-1f2a-fp8-large-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L6b-sup8-1f2a-h7168-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L6b-sup8-1f2a-inv-graph-n2.log", "log_sha256": "c3456fe20ff14a776732542af5f44ff4511bd04f40d3ef1e322c3eacd7944012", "pod": "afdep-v21-s8c" }, { "log": "vg-L6b-sup8-1f2a-mixed-att-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-L6b-sup8-1f2a-mixed-ffn-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-NEGA-large-n2.log", "log_sha256": "9bdf8196715d14097fa82e732ebdb61c1630dfae87d23923a169cd9ed31d03c5", "pod": "afdep-v21-s8c" }, { "log": "vg-NEGB-large-n2.log", "log_sha256": "4242fca884aa2d20f86137718b2b3cf42abee9f3436465df081f9b6b6932ad3c", "pod": "afdep-v21-s8c" }, { "log": "vg-NEGC-large-n2.log", "log_sha256": "22ab7c44a1cb4bc743ab57a74d5aada721179354ed1b1cf8ef044bb3c9ea896f", "pod": "afdep-v21-s8c" }, { "log": "vg-ctl-L3-AFDEP_ATTENTION_GRID_SYNC-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-ctl-L3-AFDEP_HYBRID_ROUTE_GUARD-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-ctl-L3-AFDEP_SHARED_HYBRID_COLLECT-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-ctl-L3-AFDEP_SHARED_HYBRID_FORWARD-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-ctl-L3-AFDEP_SHARED_HYBRID_SEND-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-ctl-L3-AFDEP_SHARED_ROLE_KERNELS-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-ctl-L3-AFDEP_UNION_HYBRID-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "vg-F89-large-n3.log", "log_sha256": "14358d3b15a669ed7ee47d99941b44b7ae4203cff6b47cb323d2a628014ed763", "pod": "afdep-v21-s8d" }, { "log": "vg-F89-small-n3.log", "log_sha256": "b40e196489045d1051d14404b37a1b769833e9d295cad5635727d29952318e22", "pod": "afdep-v21-s8d" }, { "log": "vg-L1-sup1-2f2a-bf16-large-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L1-sup1-2f2a-bf16-small-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L1-sup1-2f2a-fp8-large-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L1-sup1-2f2a-h7168-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L1-sup1-2f2a-inv-graph-n3.log", "log_sha256": "4e5be17b1e930cbea56d36dbb92bf97a320deb935a46da79adce2069ccbbbb14", "pod": "afdep-v21-s8d" }, { "log": "vg-L1-sup1-2f2a-mixed-att-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L1-sup1-2f2a-mixed-ffn-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L2-sup1-2f3a-bf16-large-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L2-sup1-2f3a-bf16-small-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L2-sup1-2f3a-fp8-large-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L2-sup1-2f3a-h7168-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L2-sup1-2f3a-inv-graph-n3.log", "log_sha256": "a40b1e729e798739fb00973cc4760778bee7f7fab6abf9d82ee8b4b6d32b5147", "pod": "afdep-v21-s8d" }, { "log": "vg-L2-sup1-2f3a-mixed-att-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L2-sup1-2f3a-mixed-ffn-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L3-h8192-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L3-sms23-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L3-sup2-2f2a-bf16-large-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L3-sup2-2f2a-bf16-small-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L3-sup2-2f2a-fp8-large-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L3-sup2-2f2a-h7168-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L3-sup2-2f2a-inv-graph-n3.log", "log_sha256": "9f0b7bec412069b399b0090cf98bcbedca6d9b8f93d6c9dbfc0e147f4e8dc5d3", "pod": "afdep-v21-s8d" }, { "log": "vg-L3-sup2-2f2a-mixed-att-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L3-sup2-2f2a-mixed-ffn-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L4-sup4-2f2a-bf16-large-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L4-sup4-2f2a-bf16-small-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L4-sup4-2f2a-fp8-large-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L4-sup4-2f2a-h7168-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L4-sup4-2f2a-inv-graph-n3.log", "log_sha256": "ec329d9cd7175e2e58813f863f3408d32948ac364a0b614394bf1367caf258b3", "pod": "afdep-v21-s8d" }, { "log": "vg-L4-sup4-2f2a-mixed-att-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L4-sup4-2f2a-mixed-ffn-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L5-sup4-2f3a-bf16-large-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L5-sup4-2f3a-bf16-small-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L5-sup4-2f3a-fp8-large-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L5-sup4-2f3a-h7168-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L5-sup4-2f3a-inv-graph-n3.log", "log_sha256": "fdd2ea29363a3890239fb402729c58be1ded52d032e6cff367ada07357fcfe69", "pod": "afdep-v21-s8d" }, { "log": "vg-L5-sup4-2f3a-mixed-att-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L5-sup4-2f3a-mixed-ffn-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-NEGA-large-n3.log", "log_sha256": "d854380bf9e254c8b548dd27f3ab95438226240201433208599c0095277ff41e", "pod": "afdep-v21-s8d" }, { "log": "vg-NEGB-large-n3.log", "log_sha256": "107d7cb3743d58ba9969777c01949594e65c1eb23fdc1aba03e3c32bdf55c43a", "pod": "afdep-v21-s8d" }, { "log": "vg-NEGC-large-n3.log", "log_sha256": "0757daa2a14e9d5ef4b3346697ec55fd8717908b8bd8ea259fa842823f4a8048", "pod": "afdep-v21-s8d" }, { "log": "vg-ctl-L3-AFDEP_ATTENTION_GRID_SYNC-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-ctl-L3-AFDEP_HYBRID_ROUTE_GUARD-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-ctl-L3-AFDEP_SHARED_HYBRID_COLLECT-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-ctl-L3-AFDEP_SHARED_HYBRID_FORWARD-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-ctl-L3-AFDEP_SHARED_HYBRID_SEND-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-ctl-L3-AFDEP_SHARED_ROLE_KERNELS-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-ctl-L3-AFDEP_UNION_HYBRID-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "vg-L2-sup1-2f3a-bf16-large-n4.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s4e" }, { "log": "vg-L2-sup1-2f3a-bf16-small-n4.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s4e" }, { "log": "vg-L2-sup1-2f3a-fp8-large-n4.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s4e" }, { "log": "vg-L2-sup1-2f3a-h7168-n4.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s4e" }, { "log": "vg-L2-sup1-2f3a-inv-graph-n4.log", "log_sha256": "0cebdcdc41aa57fc13c88a3af325298b52dc6e533db9457ce8611b3cd0bbd682", "pod": "afdep-v21-s4e" }, { "log": "vg-L2-sup1-2f3a-mixed-att-n4.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s4e" }, { "log": "vg-L2-sup1-2f3a-mixed-ffn-n4.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s4e" }, { "log": "vg-L5-sup4-2f3a-bf16-large-n4.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s4e" }, { "log": "vg-L5-sup4-2f3a-bf16-small-n4.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s4e" }, { "log": "vg-L5-sup4-2f3a-fp8-large-n4.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s4e" }, { "log": "vg-L5-sup4-2f3a-h7168-n4.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s4e" }, { "log": "vg-L5-sup4-2f3a-inv-graph-n4.log", "log_sha256": "4444274509a0c2599e565c63357a83db17e47ddb94b21151779c45c209301961", "pod": "afdep-v21-s4e" }, { "log": "vg-L5-sup4-2f3a-mixed-att-n4.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s4e" }, { "log": "vg-L5-sup4-2f3a-mixed-ffn-n4.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s4e" } ], "layouts": "pods s8a,s8b,s8c,s8d (8 x H100) and s4e (4 x H100), XNPROC GPUs per pod = GPUs per NVLink domain: L1 1/pod 2F:2A, L2 1/pod 2F:3A, L3 2/pod 2F:2A, L4 4/pod 2F:2A, L5 4/pod 2F:3A, L6a 8/pod 2F:1A, L6b 8/pod 1F:2A" }, "final_channels_per_sm": { "native_source_sha256": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a", "layout": "pods s8a-s8d, 2 GPUs per domain, 2F:2A, full GPU suite with EP_BUFFER_DEBUG=1", "result": [ "afdep-v21-s8a exit=0 afdep-v21-s8d exit=0 afdep-v21-s8b exit=0 afdep-v21-s8c exit=0", "h2048-bf16: 184 uses 8 channels per SM", "afdep-v21-s8a exit=0 afdep-v21-s8d exit=0 afdep-v21-s8b exit=0 afdep-v21-s8c exit=0", "h7168-bf16: 184 uses 7 channels per SM", "afdep-v21-s8d exit=0 afdep-v21-s8c exit=0 afdep-v21-s8b exit=0 afdep-v21-s8a exit=0", "h8192-fp8: 184 uses 7 channels per SM", "afdep-v21-s8d exit=0 afdep-v21-s8a exit=0 afdep-v21-s8c exit=0 afdep-v21-s8b exit=0", "h8192-bf16: 184 uses 6 channels per SM" ], "logs": [ { "log": "ch-h2048-bf16-n0.log", "log_sha256": "ba7b920086880bfddf2c1c8fd6d0878a03bfd0987572a6104604629203f4cc5d", "pod": "afdep-v21-s8a" }, { "log": "ch-h7168-bf16-n0.log", "log_sha256": "18e7f6a141dd6ba15ef42fc39b11d51d0c1ce8aee6a35ecc03eee38d686faa24", "pod": "afdep-v21-s8a" }, { "log": "ch-h8192-bf16-n0.log", "log_sha256": "8a8d0900d459c41aa4a520a849d8851325cee3c758fc6f7bfdda80aece078b7c", "pod": "afdep-v21-s8a" }, { "log": "ch-h8192-fp8-n0.log", "log_sha256": "44b84d0ee222da031ff6429c6c874b538b1631667b998b190a8e456bcfe183be", "pod": "afdep-v21-s8a" }, { "log": "ch-h2048-bf16-n1.log", "log_sha256": "5aa27a6a77301f4996280beb75eedae0b1a8e6e3ee44a718fda0c33705122a6d", "pod": "afdep-v21-s8b" }, { "log": "ch-h7168-bf16-n1.log", "log_sha256": "bb95b3262651bb3eae643ec35d85919369ebe05ab2c05127496b40acdc253eb8", "pod": "afdep-v21-s8b" }, { "log": "ch-h8192-bf16-n1.log", "log_sha256": "15ce1895e6525094b0aa9c52d697cba3bfa24a56cadcb6c16e276adccb1e522d", "pod": "afdep-v21-s8b" }, { "log": "ch-h8192-fp8-n1.log", "log_sha256": "463d6a7177341e3f96696a25814aaefe1b820804af2869b941599a034be19e91", "pod": "afdep-v21-s8b" }, { "log": "ch-h2048-bf16-n2.log", "log_sha256": "777794b913635688ebfdd054676f2fba5b31257029b1aa4bca30294cab2c8448", "pod": "afdep-v21-s8c" }, { "log": "ch-h7168-bf16-n2.log", "log_sha256": "12f5e91dad248a24603332a2dd512e3e73f698dcb7f3d967a162b22238d24b8e", "pod": "afdep-v21-s8c" }, { "log": "ch-h8192-bf16-n2.log", "log_sha256": "e4f433c006877d2b36950028e8c491bb51562e11d3fe6b4b30f9bc83da9e91ac", "pod": "afdep-v21-s8c" }, { "log": "ch-h8192-fp8-n2.log", "log_sha256": "12f5e91dad248a24603332a2dd512e3e73f698dcb7f3d967a162b22238d24b8e", "pod": "afdep-v21-s8c" }, { "log": "ch-h2048-bf16-n3.log", "log_sha256": "db5654949021243afaf504219492c8a23d845eb6caccb4b1d70b67ad78aa21d5", "pod": "afdep-v21-s8d" }, { "log": "ch-h7168-bf16-n3.log", "log_sha256": "1f5725affceacd2b4442727a85d791744da9e9859b97b5fdfd83f0d29be22571", "pod": "afdep-v21-s8d" }, { "log": "ch-h8192-bf16-n3.log", "log_sha256": "377e2e4bc6f9f7907084bb83cd275377e6b1e15f205be043c5ff6224a22ed00c", "pod": "afdep-v21-s8d" }, { "log": "ch-h8192-fp8-n3.log", "log_sha256": "a0287745cfccbc0888b4dd18d2cfd97dd545ee3d8ca1a90ad129d2ef356f6efb", "pod": "afdep-v21-s8d" } ] }, "final_statler": { "result": [ "8a8f-bf16-small: afdep-v21-s8b exit=0 afdep-v21-s8c exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 16 \"native_source_sha256\": \"9675783b ]", "8a8f-bf16-large: afdep-v21-s8c exit=0 afdep-v21-s8b exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 16 \"native_source_sha256\": \"9675783b ]", "8a8f-fp8-large: afdep-v21-s8c exit=0 afdep-v21-s8b exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 16 \"native_source_sha256\": \"9675783b ]", "8a8f-mixed-ffn: afdep-v21-s8b exit=0 afdep-v21-s8c exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 16 \"native_source_sha256\": \"9675783b ]", "8a8f-inv-graph: afdep-v21-s8c exit=1 afdep-v21-s8b exit=1 mismatch=0 timeout=10 fault_markers=0 digests=[]", "L3-bf16-large: afdep-v21-s8c exit=0 afdep-v21-s8a exit=0 afdep-v21-s8b exit=0 afdep-v21-s8d exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 8 \"native_source_sha256\": \"9675783b ]", "L3-fp8-large: afdep-v21-s8d exit=0 afdep-v21-s8c exit=0 afdep-v21-s8a exit=0 afdep-v21-s8b exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 8 \"native_source_sha256\": \"9675783b ]", "L1-bf16-large: afdep-v21-s8c exit=0 afdep-v21-s8a exit=0 afdep-v21-s8d exit=0 afdep-v21-s8b exit=0 mismatch=0 timeout=0 fault_markers=0 digests=[ 4 \"native_source_sha256\": \"9675783b ]", "F894-L3-small: afdep-v21-s8b exit=0 afdep-v21-s8d exit=0 afdep-v21-s8c exit=0 afdep-v21-s8a exit=0 mismatch=0 timeout=0 fault_markers=2533 digests=[ 8 \"native_source_sha256\": \"9c32697d ]" ], "logs": [ { "log": "f4-F894-L3-small-n0.log", "log_sha256": "f22398bdb86fd6637450755ea4177b51cc705de324c6bb8b6829dd614fb520e8", "pod": "afdep-v21-s8a" }, { "log": "f4-L1-bf16-large-n0.log", "log_sha256": "5825042d28fe1f459a09a0bbda2958447ee026c9051da0ac67c5d2156d30fac1", "pod": "afdep-v21-s8a" }, { "log": "f4-L3-bf16-large-n0.log", "log_sha256": "8aa1b50926dd3500bac89244c30c64c2ba895afe757ed18b578c3591bd036018", "pod": "afdep-v21-s8a" }, { "log": "f4-L3-fp8-large-n0.log", "log_sha256": "3b4eaf549ada8f62f321604ec89dd4027cc76d213982cb17eb7f9182581302da", "pod": "afdep-v21-s8a" }, { "log": "f4-8a8f-bf16-large-n0.log", "log_sha256": "2bbd1df8953222efed9302ea7794301c8b4cc02ce231667a960291740da10d4a", "pod": "afdep-v21-s8b" }, { "log": "f4-8a8f-bf16-small-n0.log", "log_sha256": "7732f9d995e216c67d961533b6a59cdc7b86cc8c1c25216a23d78c51469ea904", "pod": "afdep-v21-s8b" }, { "log": "f4-8a8f-fp8-large-n0.log", "log_sha256": "2762e56788886d57e83c0a66253e43611f3e1ee39a7de1ecfda30b4edd237fc5", "pod": "afdep-v21-s8b" }, { "log": "f4-8a8f-inv-graph-n0.log", "log_sha256": "45f6e02e5850b1133ed0f93380b17e7175d73175a9e4ab2308058b435f628478", "pod": "afdep-v21-s8b" }, { "log": "f4-8a8f-mixed-ffn-n0.log", "log_sha256": "10a3d29f46eb5ebdd7e3917461658e28dea8f5c3b327bad5cdadc80eed94c5d4", "pod": "afdep-v21-s8b" }, { "log": "f4-F894-L3-small-n1.log", "log_sha256": "329587b4476c2edb0edcb899ad9702b2c4bd5b4fb732bc7a24d2c388cd75e4c2", "pod": "afdep-v21-s8b" }, { "log": "f4-L1-bf16-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "f4-L3-bf16-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "f4-L3-fp8-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8b" }, { "log": "f4-8a8f-bf16-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "f4-8a8f-bf16-small-n1.log", "log_sha256": "eede120255254cad296bdddbd6a1aa14c082b1a66cbd920ef28fb7a3f67954b0", "pod": "afdep-v21-s8c" }, { "log": "f4-8a8f-fp8-large-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "f4-8a8f-inv-graph-n1.log", "log_sha256": "b633e4b4cc9e9261a9577a86bf995da72da1802cf45b8443418b1a8ef339b31c", "pod": "afdep-v21-s8c" }, { "log": "f4-8a8f-mixed-ffn-n1.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "f4-F894-L3-small-n2.log", "log_sha256": "b50551e26660a5718dd2bb6886457319b6fd27d3d04d94ac9028d91f9323d926", "pod": "afdep-v21-s8c" }, { "log": "f4-L1-bf16-large-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "f4-L3-bf16-large-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "f4-L3-fp8-large-n2.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8c" }, { "log": "f4-F894-L3-small-n3.log", "log_sha256": "5e463cb0970408b1cfd7d0e7578ba5da83d8aeb0248bae25426a7c6a599af423", "pod": "afdep-v21-s8d" }, { "log": "f4-L1-bf16-large-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" }, { "log": "f4-L3-bf16-large-n3.log", "log_sha256": "802b982b20c43cca4fe8ab55bab3d271e255b6917c6e12e253f5f930823b8b55", "pod": "afdep-v21-s8d" }, { "log": "f4-L3-fp8-large-n3.log", "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "pod": "afdep-v21-s8d" } ] } }, "faults_and_negative_controls": { "native_source_sha256": "26cd2d09e3c0807925d9321eef4ac10d2f0bdc2fa3a0ff9dd0f65a05a2c97c69", "trees": "F89 = both section 11 deferred-put patches + deferral markers; NEGA = F9 + collect that never waits; NEGB = F9 + collect waiting on FFN domain 0 only; NEGC = F8+F9 with the staging fix reverted", "tree_digests": { "F89": "7a99baa7", "NEGA": "059441a2", "NEGB": "fd4ff686", "NEGC": "24947671" }, "builder": "\"\"\"fin-faults.py : scratch fault trees for the role suites (never shipped).\n\nF89: both recorded deferred-put patches (section 11) plus markers printed when a deferral fires.\nNEGA: F9 + a collect that never waits; NEGB: F9 + a collect that waits on FFN domain 0 only;\nNEGC: F8 + F9 with the staging fix reverted. NEG trees must fail.\"\"\"\nimport subprocess, sys\nfrom pathlib import Path\n\ntree, kind = Path(sys.argv[1]), sys.argv[2]\nI = tree / \"afdep/third_party/deepep_v2/deep_ep/include/deep_ep/impls\"\nimport os\nE = Path(os.environ.get(\"AFD_FAULT_DIR\", \"/work\"))\n\n\ndef patch(name, old, new):\n p = I / name; s = p.read_text(); assert s.count(old) == 1, (name, old[:60]); p.write_text(s.replace(old, new))\n\n\ndef apply(which):\n subprocess.run([\"patch\", \"-s\", str(I / f\"hybrid_{which}.cuh\")], stdin=open(E / f\"fault_hybrid_{which}.cuh.patch\"), check=True)\n\n\nif kind in (\"F89\", \"NEGC\"):\n apply(\"dispatch\")\n patch(\"hybrid_dispatch.cuh\", \" fault_phase = 2;\\n update_scaleout_tail(true);\\n\",\n \" if (num_deferred > 0 and sm_idx == 0)\\n printf(\\\"AFDFAULT dispatch channel %d deferred %d\\\\n\\\", channel_idx, num_deferred);\\n\"\n \" fault_phase = 2;\\n update_scaleout_tail(true);\\n\")\napply(\"combine\")\npatch(\"hybrid_combine.cuh\", \" for (int k = 0; k < num_deferred; ++ k)\\n\",\n \" if (num_deferred > 0 and sm_idx == 0)\\n printf(\\\"AFDFAULT combine channel %d deferred %d\\\\n\\\", channel_idx, num_deferred);\\n\"\n \" for (int k = 0; k < num_deferred; ++ k)\\n\")\nif kind == \"NEGA\":\n patch(\"afdep_hybrid_collect.cuh\", \" for (int i = thread_idx; i < kNumChannels * kAfdFfnScaleoutRanks; i += kNumThreads) {\",\n \" for (int i = thread_idx; false and i < kNumChannels * kAfdFfnScaleoutRanks; i += kNumThreads) { // NEG: no wait\")\nif kind == \"NEGB\":\n patch(\"afdep_hybrid_collect.cuh\", \" for (int i = thread_idx; i < kNumChannels * kAfdFfnScaleoutRanks; i += kNumThreads) {\\n\"\n \" const int channel_idx = i / kAfdFfnScaleoutRanks, lane_idx = i % kAfdFfnScaleoutRanks;\",\n \" for (int i = thread_idx; i < kNumChannels; i += kNumThreads) { // NEG: FFN domain 0 only\\n\"\n \" const int channel_idx = i, lane_idx = 0;\")\nif kind == \"NEGC\":\n patch(\"hybrid_dispatch.cuh\", \" if (kAfdDirectional and kAfdFfnScaleoutRanks >= 2 and not afd_ffn_domain)\",\n \" if (false and kAfdDirectional and kAfdFfnScaleoutRanks >= 2 and not afd_ffn_domain) // NEG: staging fix reverted\")\n patch(\"hybrid_combine.cuh\", \" if (kAfdDirectional and kNumScaleoutRanks - kAfdFfnScaleoutRanks >= 2 and afd_ffn_domain) {\",\n \" if (false and kAfdDirectional and kNumScaleoutRanks - kAfdFfnScaleoutRanks >= 2 and afd_ffn_domain) { // NEG\")\nprint(\"built\", kind)\n", "result": [ "F89-small: afdep-v21-s8c exit=0 afdep-v21-s8d exit=0 afdep-v21-s8a exit=0 afdep-v21-s8b exit=0 mismatch=0 timeout=0 fault_markers=2370 digests=[ 8 \"native_source_sha256\": \"7a99baa7 ]", "F89-large: afdep-v21-s8c exit=0 afdep-v21-s8a exit=0 afdep-v21-s8b exit=0 afdep-v21-s8d exit=0 mismatch=0 timeout=0 fault_markers=1573 digests=[ 8 \"native_source_sha256\": \"7a99baa7 ]", "NEGA-large: afdep-v21-s8d exit=1 afdep-v21-s8c exit=1 afdep-v21-s8a exit=1 afdep-v21-s8b exit=1 mismatch=8 timeout=0 fault_markers=16 digests=[]", "NEGB-large: afdep-v21-s8d exit=1 afdep-v21-s8c exit=1 afdep-v21-s8a exit=1 afdep-v21-s8b exit=1 mismatch=6 timeout=0 fault_markers=796 digests=[]", "NEGC-large: afdep-v21-s8b exit=1 afdep-v21-s8a exit=1 afdep-v21-s8d exit=1 afdep-v21-s8c exit=1 mismatch=0 timeout=768 fault_markers=67 digests=[]" ] }, "jit_identity": { "one_domain_c7c4e90e_vs_head": [ "head run ok: 1", "fin run ok: 1", "cache keys and cubins IDENTICAL (7 kernels)", "fd1062e33be9ec12ebe3fcc95a77a427450ebd21225bea72baba2952a1743df1 /work/results/jitsame/head.log", "615ce568810be0e570570d0ee57f490443637ee1c2256480cb97462b3ee05e17 /work/results/jitsame/fin.log", "kernel.barrier.b93557a09a3551b512337c30ebf30458 3d9c72f432847274", "kernel.combine.9b2cce7bd2efeae084c368d60b5bf12a a76ab193ff0a233f", "kernel.combine_reduce_epilogue.1628db1effe728b024a8915db50ca9bd f3f8d42c7084770e", "kernel.dispatch.8e89d0d5858a90811fedd318955a0879 f0dc28c620910ac8", "kernel.dispatch.9b8e9031f1ad0b3a27f13abe89497f64 249f84b682e0693d", "kernel.dispatch_copy_epilogue.57bc467225be880e3d54c3478c20801d 7b98c3acb75390a7", "kernel.dispatch_copy_epilogue.7e83d2dca28e30793ad77e5946590c1b e9bda61de6d29d26" ], "hybrid_c7c4e90e_vs_head_piggy": { "p8a": [ "kernel.afdep_ffn_complete.9a6586415934814f78ff6b41c92d3502 222ab97e22623890", "kernel.barrier.2d97033d799411de45dd017cb4e2125c 0b082988ac728ccf", "kernel.combine.98ed994c3df640c709b853b1b12f8c65 e49c175b379ae7af", "kernel.combine.d69eff5ce5c1f101918a909e8da15257 2b9649d2dc0073e9", "kernel.combine_reduce_epilogue.5fe9ead2e9f0e69f940736757bbb78dc ffb5e7e6aff0503c", "kernel.dispatch.335372c5c2e2a2d9bfbd54a16af095b1 9bbd1e1842dabaf9", "kernel.dispatch.5f9e1356871f44bd59e1615e9777a439 14bb8accaf2d37c3", "kernel.dispatch.8342fd2e537029dc382560122731dc67 9aaabf8ac05b1ca3", "kernel.dispatch.ebd8b878c931b66cff84c0274446b8e1 2312723b44a0dd1a", "kernel.dispatch_copy_epilogue.1ef2362b9e5c105e1892a56fba5d2ef8 6e4d46898825a14e", "kernel.dispatch_copy_epilogue.6d4cd07232d437bf4567a6b21dfb1419 b839e573b8103b69" ], "p8b": [ "kernel.afdep_source_finalize.80e80f3e3e09c8c94226c490f78b4ebd 6dd8d747b13b052c", "kernel.barrier.2d97033d799411de45dd017cb4e2125c 0b082988ac728ccf", "kernel.combine.98ed994c3df640c709b853b1b12f8c65 e49c175b379ae7af", "kernel.combine.d69eff5ce5c1f101918a909e8da15257 2b9649d2dc0073e9", "kernel.combine_reduce_epilogue.5fe9ead2e9f0e69f940736757bbb78dc ffb5e7e6aff0503c", "kernel.dispatch.335372c5c2e2a2d9bfbd54a16af095b1 9bbd1e1842dabaf9", "kernel.dispatch.5f9e1356871f44bd59e1615e9777a439 14bb8accaf2d37c3", "kernel.dispatch.8342fd2e537029dc382560122731dc67 9aaabf8ac05b1ca3", "kernel.dispatch.ebd8b878c931b66cff84c0274446b8e1 2312723b44a0dd1a", "kernel.dispatch_copy_epilogue.1ef2362b9e5c105e1892a56fba5d2ef8 6e4d46898825a14e" ] }, "hybrid_identical": true } }, "one_domain_ffn_receive": { "final": { "native_source_sha256": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a", "validation": [ "role-kernel-built exit=0 \"native_source_sha256\": \"9675783b", "afdep_ffn_dispatch cache entries: before 270 after 276", "bf16-direct-small exit=0 bf16-hybrid-small exit=0 bf16-generic-control exit=0 bf16-off-large exit=0 bf16-on-large exit=0 fp8-direct-small exit=0 fp8-hybrid-small exit=0 fp8-generic-control exit=0 fp8-off-large exit=0 fp8-on-large exit=0 invalid-eager-dummy exit=1 assertion=yes invalid-eager-duplicate exit=1 assertion=yes invalid-eager-negative exit=1 assertion=yes invalid-eager-wide exit=1 assertion=yes invalid-graph-dummy exit=1 assertion=yes invalid-graph-duplicate exit=1 assertion=yes invalid-graph-negative exit=1 assertion=yes invalid-graph-wide exit=1 assertion=yes done", "f1-small exit=0 \"native_source_sha256\": \"9675783b", "f1-large exit=0 \"native_source_sha256\": \"9675783b", "f3-small exit=0 \"native_source_sha256\": \"9675783b", "f3-large exit=0 \"native_source_sha256\": \"9675783b", "view-n4 exit=0 \"native_source_sha256\": \"9675783b", "view-n3-large exit=0 \"native_source_sha256\": \"9675783b", "h7168 exit=0 \"native_source_sha256\": \"9675783b", "sms7 exit=0 \"native_source_sha256\": \"9675783b", "ctl-AFDEP_SHARED_FFN_RECEIVE exit=0 \"native_source_sha256\": \"9675783b", "ctl-AFDEP_SHARED_FFN_RECEIVE-large exit=0 \"native_source_sha256\": \"9675783b", "ctl-AFDEP_SHARED_ROLE_KERNELS exit=0 \"native_source_sha256\": \"9675783b", "ctl-AFDEP_SHARED_ROLE_KERNELS-large exit=0 \"native_source_sha256\": \"9675783b", "ctl-AFDEP_KEEP_ENTRY_BARRIERS exit=0 \"native_source_sha256\": \"9675783b", "ctl-AFDEP_KEEP_ENTRY_BARRIERS-large exit=0 \"native_source_sha256\": \"9675783b", "ctl-AFDEP_UNION_NOTIFY exit=0 \"native_source_sha256\": \"9675783b", "ctl-AFDEP_UNION_NOTIFY-large exit=0 \"native_source_sha256\": \"9675783b", "ctl-AFDEP_ALL_RANK_EXIT exit=0 \"native_source_sha256\": \"9675783b", "ctl-AFDEP_ALL_RANK_EXIT-large exit=0 \"native_source_sha256\": \"9675783b", "ctl-AFDEP_GRID_SYNC_EXIT exit=0 \"native_source_sha256\": \"9675783b", "ctl-AFDEP_GRID_SYNC_EXIT-large exit=0 \"native_source_sha256\": \"9675783b", "ctl-AFDEP_UNION_BUFFERS exit=0 \"native_source_sha256\": \"9675783b", "ctl-AFDEP_UNION_BUFFERS-large exit=0 \"native_source_sha256\": \"9675783b", "ctl-AFDEP_UNION_REDUCE_PREFIX exit=0 \"native_source_sha256\": \"9675783b", "mixed-ffn-f2 exit=0 \"native_source_sha256\": \"9675783b", "mixed-ffn-f1 exit=0 \"native_source_sha256\": \"9675783b", "mixed-ffn-f3 exit=0 \"native_source_sha256\": \"9675783b", "done" ], "faults": [ "applied F1", "F1 digest 8725ff48", "F1-small exit=0 mismatch_lines=0 timeout_lines=0", "F1-large exit=0 mismatch_lines=0 timeout_lines=0", "F1-f1-view exit=0 mismatch_lines=0 timeout_lines=0", "F1-f3-small exit=0 mismatch_lines=0 timeout_lines=0", "applied F2", "F2 digest f7cb54d6", "F2-small exit=1 mismatch_lines=6 timeout_lines=0", "applied F3", "F3 digest 9a1d160a", "F3-small exit=1 mismatch_lines=0 timeout_lines=4", "applied F5", "F5 digest 5dc478e9", "F5-small exit=0 mismatch_lines=0 timeout_lines=0", "F5-large exit=0 mismatch_lines=0 timeout_lines=0", "F5-f1-view exit=0 mismatch_lines=0 timeout_lines=0", "F5-f3-small exit=0 mismatch_lines=0 timeout_lines=0", "applied F6", "F6 digest 5d7e5150", "F6-small exit=0 mismatch_lines=0 timeout_lines=0", "F6-large exit=0 mismatch_lines=0 timeout_lines=0", "F6-f1-view exit=0 mismatch_lines=0 timeout_lines=0", "F6-f3-small exit=0 mismatch_lines=0 timeout_lines=0", "done" ], "screens": { "c-2a2f": { "arms": { "afd": { "_digest": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" }, "ffn_shared": { "_env": { "AFDEP_SHARED_FFN_RECEIVE": "1" }, "_digest": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" }, "head": { "_tree": "/work/src-head", "_digest": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879" }, "masked": { "role_aware": false, "_digest": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" } }, "passthrough": [ "--payload", "bf16", "--routing", "random", "--hidden", "2048", "--tokens", "1", "8", "32", "128", "512", "2048", "--capacity", "2048" ], "processes": [ { "order": [ "afd", "ffn_shared", "head", "masked" ], "median_us": { "afd": { "1": 19.833279848098755, "8": 23.311439752578735, "32": 24.417999982833862, "128": 32.50312089920044, "512": 66.0699200630188, "2048": 181.29615783691406 }, "ffn_shared": { "1": 20.174959897994995, "8": 23.31496000289917, "32": 24.574960470199585, "128": 32.23968029022217, "512": 66.45687818527222, "2048": 181.29952430725098 }, "head": { "1": 19.82711970806122, "8": 23.278239965438843, "32": 23.925119638442993, "128": 32.070560455322266, "512": 66.02272033691406, "2048": 180.6947135925293 }, "masked": { "1": 28.24895977973938, "8": 31.2774395942688, "32": 32.39192008972168, "128": 40.534560680389404, "512": 72.84456014633179, "2048": 187.72536277770996 } }, "native_source_sha256": { "afd": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a", "ffn_shared": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" } }, { "order": [ "ffn_shared", "head", "masked", "afd" ], "median_us": { "ffn_shared": { "1": 19.84120011329651, "8": 23.355759382247925, "32": 23.92799973487854, "128": 32.119200229644775, "512": 66.47575855255127, "2048": 181.07864379882812 }, "head": { "1": 19.74671959877014, "8": 23.311680555343628, "32": 24.526960849761963, "128": 32.427120208740234, "512": 66.41511917114258, "2048": 180.82703590393066 }, "masked": { "1": 28.2367205619812, "8": 31.386560201644897, "32": 32.7672803401947, "128": 40.138962268829346, "512": 73.00439834594727, "2048": 188.2754421234131 }, "afd": { "1": 19.759600162506104, "8": 23.657280206680298, "32": 24.04968023300171, "128": 32.590320110321045, "512": 66.24135971069336, "2048": 180.8735179901123 } }, "native_source_sha256": { "ffn_shared": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a", "afd": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" } }, { "order": [ "head", "masked", "afd", "ffn_shared" ], "median_us": { "head": { "1": 19.742720127105713, "8": 23.554080724716187, "32": 24.70080018043518, "128": 32.26288080215454, "512": 66.16136074066162, "2048": 180.84896087646484 }, "masked": { "1": 28.121920824050903, "8": 31.370561122894287, "32": 32.378801107406616, "128": 40.05231857299805, "512": 72.8331995010376, "2048": 188.2428741455078 }, "afd": { "1": 20.254160165786743, "8": 23.34656000137329, "32": 24.56552028656006, "128": 32.24056005477905, "512": 66.38335943222046, "2048": 181.31175994873047 }, "ffn_shared": { "1": 20.171600580215454, "8": 23.809279203414917, "32": 24.4213604927063, "128": 32.45248079299927, "512": 66.55695915222168, "2048": 181.3041591644287 } }, "native_source_sha256": { "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a", "afd": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a", "ffn_shared": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" } }, { "order": [ "masked", "afd", "ffn_shared", "head" ], "median_us": { "masked": { "1": 28.201040029525757, "8": 31.730879545211792, "32": 32.22432017326355, "128": 40.189199447631836, "512": 72.99400091171265, "2048": 187.89207458496094 }, "afd": { "1": 19.78040039539337, "8": 23.16920042037964, "32": 25.187519788742065, "128": 32.32111930847168, "512": 66.01112127304077, "2048": 180.83551406860352 }, "ffn_shared": { "1": 20.156320333480835, "8": 23.434720039367676, "32": 25.33039927482605, "128": 32.43384003639221, "512": 66.26168012619019, "2048": 181.12688064575195 }, "head": { "1": 19.72927987575531, "8": 23.56231927871704, "32": 25.268800258636475, "128": 32.27887988090515, "512": 66.1851978302002, "2048": 181.2156867980957 } }, "native_source_sha256": { "masked": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a", "afd": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a", "ffn_shared": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879" } }, { "order": [ "afd", "ffn_shared", "head", "masked" ], "median_us": { "afd": { "1": 20.183919668197632, "8": 23.2204806804657, "32": 24.762239456176758, "128": 32.33695983886719, "512": 66.1895203590393, "2048": 181.28863334655762 }, "ffn_shared": { "1": 19.759520292282104, "8": 23.777120113372803, "32": 24.738720655441284, "128": 32.3471200466156, "512": 66.06071949005127, "2048": 181.00607872009277 }, "head": { "1": 19.872400164604187, "8": 23.56112003326416, "32": 24.774880409240723, "128": 32.6256799697876, "512": 66.39392137527466, "2048": 181.0547161102295 }, "masked": { "1": 28.127520084381104, "8": 31.32680058479309, "32": 32.35335946083069, "128": 40.19631862640381, "512": 72.9911994934082, "2048": 188.02047729492188 } }, "native_source_sha256": { "afd": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a", "ffn_shared": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" } } ], "table": [ { "tokens": 1, "afd": { "median_us": 19.833279848098755, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "ffn_shared": { "median_us": 20.156320333480835, "paired_change_pct": [ 1.722761199928291, 0.4129635727409209, -0.40761791600102404, 1.9004667780891271, -2.1026608453273976 ], "geomean_change_pct": 0.29429301418542053, "repeats_faster": 2 }, "head": { "median_us": 19.74671959877014, "paired_change_pct": [ -0.031059613360551186, -0.06518635817541751, -2.5251110611090755, -0.2584402672150543, -1.5434043967400668 ], "geomean_change_pct": -0.889624171692871, "repeats_faster": 5 }, "masked": { "median_us": 28.201040029525757, "paired_change_pct": [ 42.432114083477536, 42.90127497397673, 38.845158692653946, 42.570622767036895, 39.35608418368628 ], "geomean_change_pct": 41.21022638380758, "repeats_faster": 0 } }, { "tokens": 8, "afd": { "median_us": 23.311439752578735, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "ffn_shared": { "median_us": 23.434720039367676, "paired_change_pct": [ 0.015100956259228582, -1.27453714796526, 1.9819588068409555, 1.1460025126913242, 2.3971916885225264 ], "geomean_change_pct": 0.8442115891339474, "repeats_faster": 1 }, "head": { "median_us": 23.554080724716187, "paired_change_pct": [ -0.1424184327191491, -1.4608596098848237, 0.8888706658740642, 1.6967303627431818, 1.4669780418672618 ], "geomean_change_pct": 0.48308450148650817, "repeats_faster": 2 }, "masked": { "median_us": 31.370561122894287, "paired_change_pct": [ 34.17206284227407, 32.67188758571673, 34.369093866715296, 36.95284675124695, 34.910215752539784 ], "geomean_change_pct": 34.60812004479812, "repeats_faster": 0 } }, { "tokens": 32, "afd": { "median_us": 24.56552028656006, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "ffn_shared": { "median_us": 24.574960470199585, "paired_change_pct": [ 0.6428064848720938, -0.5059547442805346, -0.5868379426615666, 0.5672630226492004, -0.09497848842426881 ], "geomean_change_pct": 0.0031160858466883212, "repeats_faster": 3 }, "head": { "median_us": 24.70080018043518, "paired_change_pct": [ -2.018512346373047, 1.984561175600641, 0.5506901229734407, 0.3227013639141241, 0.05104931275030644 ], "geomean_change_pct": 0.16981607895929507, "repeats_faster": 1 }, "masked": { "median_us": 32.378801107406616, "paired_change_pct": [ 32.655910035603156, 36.24829944819987, 31.80588373339379, 27.937647071017622, 30.65603181040388 ], "geomean_change_pct": 31.83297809243195, "repeats_faster": 0 } }, { "tokens": 128, "afd": { "median_us": 32.33695983886719, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "ffn_shared": { "median_us": 32.3471200466156, "paired_change_pct": [ -0.810508657907838, -1.4455822436891919, 0.6573109705915403, 0.3487525504445754, 0.031419798889698036 ], "geomean_change_pct": -0.24673968281627712, "repeats_faster": 2 }, "head": { "median_us": 32.27887988090515, "paired_change_pct": [ -1.3308274156799915, -0.5007618858248852, 0.06923188473637598, -0.13068677221044211, 0.8928487166359522 ], "geomean_change_pct": -0.20268633426898663, "repeats_faster": 3 }, "masked": { "median_us": 40.189199447631836, "paired_change_pct": [ 24.7097495840362, 23.16222158283654, 24.22959931510573, 24.343464296726424, 24.304569219553287 ], "geomean_change_pct": 24.14882464056167, "repeats_faster": 0 } }, { "tokens": 512, "afd": { "median_us": 66.1895203590393, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "ffn_shared": { "median_us": 66.45687818527222, "paired_change_pct": [ 0.5856797191283514, 0.35385572228836537, 0.2615108989451942, 0.3795706667563348, -0.19459405097569382 ], "geomean_change_pct": 0.2768706701908208, "repeats_faster": 1 }, "head": { "median_us": 66.1851978302002, "paired_change_pct": [ -0.07143905435289533, 0.26231264154013, -0.33441918796759396, 0.26370792345640215, 0.3088117501480525 ], "geomean_change_pct": 0.08548086001871624, "repeats_faster": 2 }, "masked": { "median_us": 72.9911994934082, "paired_change_pct": [ 10.253743423408412, 10.209691746653782, 9.716049509971892, 10.578338170910161, 10.276066509431981 ], "geomean_change_pct": 10.206427357303172, "repeats_faster": 0 } }, { "tokens": 2048, "afd": { "median_us": 181.28863334655762, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "ffn_shared": { "median_us": 181.12688064575195, "paired_change_pct": [ 0.0018568900615800743, 0.11340842539870888, -0.004192107728651084, 0.1611224314256754, -0.15585898643998286 ], "geomean_change_pct": 0.023206900270311692, "repeats_faster": 2 }, "head": { "median_us": 180.84896087646484, "paired_change_pct": [ -0.3317468232977072, -0.025698668715112838, -0.2552504439847114, 0.2102312322058486, -0.12903028282030826 ], "geomean_change_pct": -0.10647929069727002, "repeats_faster": 4 }, "masked": { "median_us": 188.02047729492188, "paired_change_pct": [ 3.546244453001224, 4.092320542858796, 3.8227604203595167, 3.9021983887967737, 3.713329304819357 ], "geomean_change_pct": 3.8152096361449006, "repeats_faster": 0 } } ] }, "c-3a1f": { "arms": { "afd": { "_digest": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" }, "ffn_shared": { "_env": { "AFDEP_SHARED_FFN_RECEIVE": "1" }, "_digest": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" }, "head": { "_tree": "/work/src-head", "_digest": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879" }, "masked": { "role_aware": false, "_digest": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" } }, "passthrough": [ "--payload", "bf16", "--routing", "random", "--hidden", "2048", "--tokens", "1", "8", "32", "128", "512", "2048", "--capacity", "2048", "--ffn-ranks", "1", "--attention-ranks", "3" ], "processes": [ { "order": [ "afd", "ffn_shared", "head", "masked" ], "median_us": { "afd": { "1": 18.629119992256165, "8": 20.391600131988525, "32": 23.248159885406494, "128": 35.40511965751648, "512": 96.19359970092773, "2048": 322.45704650878906 }, "ffn_shared": { "1": 18.535119891166687, "8": 20.953280925750732, "32": 24.23527956008911, "128": 35.528080463409424, "512": 96.37248039245605, "2048": 322.30751037597656 }, "head": { "1": 18.208799958229065, "8": 20.59015989303589, "32": 24.19848084449768, "128": 35.487200021743774, "512": 96.44296169281006, "2048": 322.7677536010742 }, "masked": { "1": 32.24272012710571, "8": 35.42639970779419, "32": 39.97727990150452, "128": 50.838561058044434, "512": 110.7535982131958, "2048": 343.11975479125977 } }, "native_source_sha256": { "afd": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a", "ffn_shared": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" } }, { "order": [ "ffn_shared", "head", "masked", "afd" ], "median_us": { "ffn_shared": { "1": 18.712159991264343, "8": 20.792800188064575, "32": 24.126880168914795, "128": 35.51184058189392, "512": 96.2877607345581, "2048": 322.5395965576172 }, "head": { "1": 18.27616035938263, "8": 20.488640069961548, "32": 24.131200313568115, "128": 35.54935932159424, "512": 96.47295951843262, "2048": 322.9169464111328 }, "masked": { "1": 32.489598989486694, "8": 35.889039039611816, "32": 39.31543946266174, "128": 50.75568199157715, "512": 110.5459213256836, "2048": 342.7280807495117 }, "afd": { "1": 18.51032018661499, "8": 20.769760608673096, "32": 23.592880964279175, "128": 35.38040041923523, "512": 96.22000217437744, "2048": 322.4046516418457 } }, "native_source_sha256": { "ffn_shared": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a", "afd": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" } }, { "order": [ "head", "masked", "afd", "ffn_shared" ], "median_us": { "head": { "1": 18.595199584960938, "8": 20.928879976272583, "32": 23.763760328292847, "128": 35.41839957237244, "512": 96.35776042938232, "2048": 322.55271911621094 }, "masked": { "1": 32.31127977371216, "8": 34.90455985069275, "32": 39.65000033378601, "128": 50.534160137176514, "512": 110.34848213195801, "2048": 342.60486602783203 }, "afd": { "1": 18.56607973575592, "8": 20.78271985054016, "32": 23.77503991127014, "128": 35.699360370635986, "512": 96.60367965698242, "2048": 322.4007987976074 }, "ffn_shared": { "1": 18.479999899864197, "8": 20.317280292510986, "32": 23.868800401687622, "128": 35.465439558029175, "512": 96.36352062225342, "2048": 322.5708770751953 } }, "native_source_sha256": { "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a", "afd": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a", "ffn_shared": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" } }, { "order": [ "masked", "afd", "ffn_shared", "head" ], "median_us": { "masked": { "1": 32.71736025810242, "8": 34.47535991668701, "32": 40.21440029144287, "128": 50.374159812927246, "512": 110.1754379272461, "2048": 342.1834373474121 }, "afd": { "1": 18.52952003479004, "8": 20.220400094985962, "32": 23.82991909980774, "128": 35.3930401802063, "512": 96.39096260070801, "2048": 322.6441764831543 }, "ffn_shared": { "1": 18.19599986076355, "8": 20.62615990638733, "32": 23.969680070877075, "128": 35.47600030899048, "512": 96.42303943634033, "2048": 322.92232513427734 }, "head": { "1": 18.644480109214783, "8": 20.76575994491577, "32": 24.011679887771606, "128": 35.52672028541565, "512": 96.19735717773438, "2048": 322.53368377685547 } }, "native_source_sha256": { "masked": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a", "afd": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a", "ffn_shared": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879" } }, { "order": [ "afd", "ffn_shared", "head", "masked" ], "median_us": { "afd": { "1": 18.49440038204193, "8": 20.80512046813965, "32": 23.791520595550537, "128": 35.44327974319458, "512": 96.33903980255127, "2048": 322.60223388671875 }, "ffn_shared": { "1": 18.68503987789154, "8": 20.62999963760376, "32": 23.56495976448059, "128": 35.53696036338806, "512": 96.35015964508057, "2048": 322.7783966064453 }, "head": { "1": 18.379679918289185, "8": 20.913519859313965, "32": 23.488720655441284, "128": 35.53112030029297, "512": 96.20376110076904, "2048": 322.498722076416 }, "masked": { "1": 32.68191933631897, "8": 35.28631925582886, "32": 40.2347207069397, "128": 50.302560329437256, "512": 110.57392120361328, "2048": 342.4502372741699 } }, "native_source_sha256": { "afd": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a", "ffn_shared": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" } } ], "table": [ { "tokens": 1, "afd": { "median_us": 18.52952003479004, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "ffn_shared": { "median_us": 18.535119891166687, "paired_change_pct": [ -0.5045869108608048, 1.090417683835132, -0.4636403436636405, -1.7999396282272317, 1.0307957647262889 ], "geomean_change_pct": -0.13528511345084482, "repeats_faster": 3 }, "head": { "median_us": 18.379679918289185, "paired_change_pct": [ -2.256252760204558, -1.2650231053360383, 0.15684436143477143, 0.6204158240952884, -0.6202983680624774 ], "geomean_change_pct": -0.6781298128834878, "repeats_faster": 3 }, "masked": { "median_us": 32.489598989486694, "paired_change_pct": [ 73.07698989811924, 75.52153967050374, 74.03393841665304, 76.56884904020205, 76.7125111450119 ], "geomean_change_pct": 75.17697285859859, "repeats_faster": 0 } }, { "tokens": 8, "afd": { "median_us": 20.769760608673096, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "ffn_shared": { "median_us": 20.62999963760376, "paired_change_pct": [ 2.754471400609182, 0.1109284783083142, -2.239550748777852, 2.0066853746478763, -0.8417198583592134 ], "geomean_change_pct": 0.34150386762548113, "repeats_faster": 2 }, "head": { "median_us": 20.76575994491577, "paired_change_pct": [ 0.973733104622232, -1.3535088054609345, 0.7032771782689684, 2.6970774434134093, 0.5210226556501718 ], "geomean_change_pct": 0.7000597854078583, "repeats_faster": 1 }, "masked": { "median_us": 35.28631925582886, "paired_change_pct": [ 73.73035700234436, 72.79466872923498, 67.94991272417863, 70.49791178581002, 69.60401315563298 ], "geomean_change_pct": 70.90241684625175, "repeats_faster": 0 } }, { "tokens": 32, "afd": { "median_us": 23.77503991127014, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "ffn_shared": { "median_us": 23.969680070877075, "paired_change_pct": [ 4.246012069549887, 2.263391255371161, 0.394365228270499, 0.5864936867136183, -0.9522755393462257 ], "geomean_change_pct": 1.2918796125228837, "repeats_faster": 1 }, "head": { "median_us": 24.011679887771606, "paired_change_pct": [ 4.0877254964498455, 2.281702476708891, -0.0474429612711158, 0.7627419430279669, -1.2727220981658571 ], "geomean_change_pct": 1.1452991241009292, "repeats_faster": 2 }, "masked": { "median_us": 39.97727990150452, "paired_change_pct": [ 71.95889953681602, 66.64111314844222, 66.77154058105543, 68.755924529208, 69.11369975429122 ], "geomean_change_pct": 68.63717372957348, "repeats_faster": 0 } }, { "tokens": 128, "afd": { "median_us": 35.40511965751648, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "ffn_shared": { "median_us": 35.51184058189392, "paired_change_pct": [ 0.3472966821814971, 0.3715055824728086, -0.6552521114614129, 0.23439672987057847, 0.2643113754490223 ], "geomean_change_pct": 0.111700143986182, "repeats_faster": 1 }, "head": { "median_us": 35.52672028541565, "paired_change_pct": [ 0.2318319074226638, 0.47754943515887494, -0.78701913800856, 0.37770167391302234, 0.24783416696998462 ], "geomean_change_pct": 0.10853117935794021, "repeats_faster": 1 }, "masked": { "median_us": 50.534160137176514, "paired_change_pct": [ 43.59098782837032, 43.457059247929976, 41.554805499379995, 42.3278688590849, 41.924112818864586 ], "geomean_change_pct": 42.56862852637404, "repeats_faster": 0 } }, { "tokens": 512, "afd": { "median_us": 96.33903980255127, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "ffn_shared": { "median_us": 96.36352062225342, "paired_change_pct": [ 0.18595903686364768, 0.07042045172465095, -0.24860236750996467, 0.03327784552291657, 0.011542405396691713 ], "geomean_change_pct": 0.010417376160898328, "repeats_faster": 1 }, "head": { "median_us": 96.35776042938232, "paired_change_pct": [ 0.2592292966035137, 0.2628947602773435, -0.25456507295923414, -0.20085433089368498, -0.14041940013049814 ], "geomean_change_pct": -0.015002946761222269, "repeats_faster": 3 }, "masked": { "median_us": 110.5459213256836, "paired_change_pct": [ 15.136140613862104, 14.88871214671519, 14.228032020912895, 14.300588929316117, 14.775818225131454 ], "geomean_change_pct": 14.665328036493609, "repeats_faster": 0 } }, { "tokens": 2048, "afd": { "median_us": 322.45704650878906, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "ffn_shared": { "median_us": 322.5708770751953, "paired_change_pct": [ -0.04637396962836604, 0.04185575955069076, 0.052753677479144834, 0.08620910321546216, 0.05460678855324641 ], "geomean_change_pct": 0.03780032120026533, "repeats_faster": 1 }, "head": { "median_us": 322.55271911621094, "paired_change_pct": [ 0.09635611801608324, 0.15889807007380963, 0.04712157016053542, -0.0342459943034501, -0.03208651380234295 ], "geomean_change_pct": 0.04718085370458702, "repeats_faster": 2 }, "masked": { "median_us": 342.60486602783203, "paired_change_pct": [ 6.407894789766222, 6.3037021966553475, 6.2667547058120165, 6.055978160596975, 6.152469295801821 ], "geomean_change_pct": 6.237289739986629, "repeats_faster": 0 } } ] } }, "overlap": { "o-2a2f": { "native_source_sha256": { "afd": [ "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" ], "ffn_shared": [ "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" ], "masked": [ "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" ] }, "per_repeat": { "afd": { "1": { "attention.gemm_us_per_gemm": [ 26.02, 26.08, 25.99 ], "attention.exchange_us": [ 34.62, 34.64, 34.31 ], "attention.makespan_us": [ 6924.38, 6928.77, 6861.9 ], "ffn.gemm_us_per_gemm": [ 25.92, 25.83, 25.92 ], "ffn.exchange_us": [ 34.61, 34.66, 34.32 ], "ffn.makespan_us": [ 6922.11, 6932.37, 6864.16 ] }, "32": { "attention.gemm_us_per_gemm": [ 26.22, 26.21, 26.21 ], "attention.exchange_us": [ 45.01, 44.83, 44.12 ], "attention.makespan_us": [ 9002.93, 8965.04, 8823.92 ], "ffn.gemm_us_per_gemm": [ 29.06, 27.97, 29.16 ], "ffn.exchange_us": [ 45.01, 44.82, 44.1 ], "ffn.makespan_us": [ 9002.56, 8963.12, 8819.07 ] }, "512": { "attention.gemm_us_per_gemm": [ 27.32, 27.35, 27.36 ], "attention.exchange_us": [ 112.19, 112.14, 111.95 ], "attention.makespan_us": [ 22437.39, 22428.83, 22389.58 ], "ffn.gemm_us_per_gemm": [ 30.22, 30.15, 30.72 ], "ffn.exchange_us": [ 112.15, 112.14, 111.93 ], "ffn.makespan_us": [ 22430.43, 22427.84, 22385.14 ] } }, "ffn_shared": { "1": { "attention.gemm_us_per_gemm": [ 26.43, 26.24, 26.34 ], "attention.exchange_us": [ 36.57, 35.96, 36.98 ], "attention.makespan_us": [ 7313.89, 7192.53, 7395.31 ], "ffn.gemm_us_per_gemm": [ 29.1, 28.15, 29.51 ], "ffn.exchange_us": [ 36.56, 35.93, 36.98 ], "ffn.makespan_us": [ 7311.38, 7186.75, 7395.68 ] }, "32": { "attention.gemm_us_per_gemm": [ 26.3, 26.05, 26.22 ], "attention.exchange_us": [ 46.8, 47.58, 46.69 ], "attention.makespan_us": [ 9360.96, 9516.1, 9338.61 ], "ffn.gemm_us_per_gemm": [ 34.4, 34.91, 33.81 ], "ffn.exchange_us": [ 46.82, 47.6, 46.7 ], "ffn.makespan_us": [ 9364.45, 9519.31, 9339.17 ] }, "512": { "attention.gemm_us_per_gemm": [ 27.69, 27.39, 27.49 ], "attention.exchange_us": [ 120.62, 121.08, 120.48 ], "attention.makespan_us": [ 24124.35, 24215.65, 24095.78 ], "ffn.gemm_us_per_gemm": [ 46.02, 46.08, 45.8 ], "ffn.exchange_us": [ 120.61, 121.09, 120.4 ], "ffn.makespan_us": [ 24121.54, 24217.41, 24080.42 ] } }, "masked": { "1": { "attention.gemm_us_per_gemm": [ 29.35, 29.17, 29.22 ], "attention.exchange_us": [ 43.59, 43.46, 43.39 ], "attention.makespan_us": [ 8718.67, 8691.65, 8677.3 ], "ffn.gemm_us_per_gemm": [ 29.48, 29.42, 29.15 ], "ffn.exchange_us": [ 43.57, 43.4, 43.39 ], "ffn.makespan_us": [ 8714.96, 8679.73, 8678.38 ] }, "32": { "attention.gemm_us_per_gemm": [ 29.98, 28.57, 29.45 ], "attention.exchange_us": [ 55.77, 55.29, 54.69 ], "attention.makespan_us": [ 11154.54, 11058.94, 10937.26 ], "ffn.gemm_us_per_gemm": [ 35.49, 35.67, 35.78 ], "ffn.exchange_us": [ 55.76, 55.26, 54.67 ], "ffn.makespan_us": [ 11151.52, 11052.62, 10934.06 ] }, "512": { "attention.gemm_us_per_gemm": [ 30.06, 30.06, 30.14 ], "attention.exchange_us": [ 120.34, 119.38, 121.06 ], "attention.makespan_us": [ 24068.54, 23876.14, 24211.89 ], "ffn.gemm_us_per_gemm": [ 35.49, 36.04, 36.15 ], "ffn.exchange_us": [ 120.33, 119.37, 121.07 ], "ffn.makespan_us": [ 24066.46, 23873.9, 24213.12 ] } } } }, "o-3a1f": { "native_source_sha256": { "afd": [ "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" ], "ffn_shared": [ "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" ], "masked": [ "9675783bee5e9a41c3239cb0e8f2342dc5cbac8feb459f72d75f6d6f2700425a" ] }, "per_repeat": { "afd": { "1": { "attention.gemm_us_per_gemm": [ 25.97, 26.12, 26.06 ], "attention.exchange_us": [ 34.74, 34.72, 34.79 ], "attention.makespan_us": [ 6947.01, 6943.3, 6958.21 ], "ffn.gemm_us_per_gemm": [ 25.96, 25.23, 25.96 ], "ffn.exchange_us": [ 34.63, 34.67, 34.68 ], "ffn.makespan_us": [ 6925.3, 6934.16, 6936.78 ] }, "32": { "attention.gemm_us_per_gemm": [ 26.15, 26.18, 26.2 ], "attention.exchange_us": [ 45.67, 45.98, 45.85 ], "attention.makespan_us": [ 9133.81, 9195.87, 9169.74 ], "ffn.gemm_us_per_gemm": [ 29.71, 27.86, 29.81 ], "ffn.exchange_us": [ 45.62, 45.95, 45.81 ], "ffn.makespan_us": [ 9123.68, 9190.61, 9161.33 ] }, "512": { "attention.gemm_us_per_gemm": [ 26.65, 26.34, 26.63 ], "attention.exchange_us": [ 149.07, 148.0, 149.53 ], "attention.makespan_us": [ 29814.67, 29599.28, 29905.84 ], "ffn.gemm_us_per_gemm": [ 31.62, 31.22, 31.54 ], "ffn.exchange_us": [ 149.03, 147.97, 149.41 ], "ffn.makespan_us": [ 29805.63, 29594.5, 29882.18 ] } }, "ffn_shared": { "1": { "attention.gemm_us_per_gemm": [ 26.1, 26.18, 26.14 ], "attention.exchange_us": [ 38.62, 38.35, 37.94 ], "attention.makespan_us": [ 7724.64, 7670.16, 7588.13 ], "ffn.gemm_us_per_gemm": [ 31.62, 31.41, 31.23 ], "ffn.exchange_us": [ 38.56, 38.26, 37.88 ], "ffn.makespan_us": [ 7711.62, 7652.66, 7576.72 ] }, "32": { "attention.gemm_us_per_gemm": [ 26.06, 26.14, 25.9 ], "attention.exchange_us": [ 47.26, 47.11, 47.0 ], "attention.makespan_us": [ 9452.11, 9421.23, 9399.9 ], "ffn.gemm_us_per_gemm": [ 35.6, 35.07, 35.46 ], "ffn.exchange_us": [ 47.18, 47.09, 46.97 ], "ffn.makespan_us": [ 9435.71, 9418.61, 9393.9 ] }, "512": { "attention.gemm_us_per_gemm": [ 27.1, 26.94, 26.65 ], "attention.exchange_us": [ 150.66, 150.34, 151.84 ], "attention.makespan_us": [ 30132.8, 30068.37, 30367.02 ], "ffn.gemm_us_per_gemm": [ 40.4, 40.35, 41.2 ], "ffn.exchange_us": [ 150.66, 150.33, 151.72 ], "ffn.makespan_us": [ 30131.17, 30066.56, 30344.54 ] } }, "masked": { "1": { "attention.gemm_us_per_gemm": [ 30.81, 30.73, 30.77 ], "attention.exchange_us": [ 47.82, 47.79, 47.67 ], "attention.makespan_us": [ 9563.95, 9557.31, 9533.97 ], "ffn.gemm_us_per_gemm": [ 30.89, 30.82, 30.78 ], "ffn.exchange_us": [ 47.79, 47.76, 47.42 ], "ffn.makespan_us": [ 9558.62, 9552.5, 9483.31 ] }, "32": { "attention.gemm_us_per_gemm": [ 29.66, 27.79, 27.71 ], "attention.exchange_us": [ 61.74, 60.56, 60.41 ], "attention.makespan_us": [ 12348.38, 12111.68, 12081.39 ], "ffn.gemm_us_per_gemm": [ 33.03, 36.58, 36.51 ], "ffn.exchange_us": [ 61.52, 60.56, 60.33 ], "ffn.makespan_us": [ 12303.63, 12111.12, 12065.14 ] }, "512": { "attention.gemm_us_per_gemm": [ 30.9, 30.88, 31.01 ], "attention.exchange_us": [ 165.91, 163.9, 163.59 ], "attention.makespan_us": [ 33181.46, 32780.69, 32717.44 ], "ffn.gemm_us_per_gemm": [ 35.98, 36.55, 36.48 ], "ffn.exchange_us": [ 165.9, 163.92, 163.61 ], "ffn.makespan_us": [ 33180.29, 32784.89, 32721.62 ] } } } } } }, "trigger_variants": { "arrival_fa0db53d": { "validation": [ "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "role-kernel-built exit=0 \"native_source_sha256\": \"fa0db53d", "afdep_ffn_dispatch cache entries: before 0 after 6", "role-kernel-built exit=0 \"native_source_sha256\": \"fa0db53d", "bf16-direct-small exit=0 bf16-hybrid-small exit=0 bf16-generic-control exit=0 bf16-off-large exit=0 bf16-on-large exit=0 fp8-direct-small exit=0 fp8-hybrid-small exit=0 fp8-generic-control exit=0 fp8-off-large exit=0 fp8-on-large exit=0 invalid-eager-dummy exit=1 assertion=yes invalid-eager-duplicate exit=1 assertion=yes invalid-eager-negative exit=1 assertion=yes invalid-eager-wide exit=1 assertion=yes invalid-graph-dummy exit=1 assertion=yes invalid-graph-duplicate exit=1 assertion=yes invalid-graph-negative exit=1 assertion=yes invalid-graph-wide exit=1 assertion=yes done", "f1-small exit=0 \"native_source_sha256\": \"fa0db53d", "f1-large exit=0 \"native_source_sha256\": \"fa0db53d", "f3-small exit=0 \"native_source_sha256\": \"fa0db53d" ], "screens": { "c-2a2f": { "arms": { "afd": {}, "ffn_shared": { "_env": { "AFDEP_SHARED_FFN_RECEIVE": "1" } }, "head": { "_tree": "/work/src-head", "_digest": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879" }, "masked": { "role_aware": false } }, "passthrough": [ "--payload", "bf16", "--routing", "random", "--hidden", "2048", "--tokens", "1", "8", "32", "128", "512", "2048", "--capacity", "2048" ], "processes": [ { "order": [ "afd", "ffn_shared", "head", "masked" ], "median_us": { "afd": { "1": 20.604159832000732, "8": 24.227120876312256, "32": 24.587680101394653, "128": 33.069679737091064, "512": 67.06767797470093, "2048": 182.06151962280273 }, "ffn_shared": { "1": 19.802799820899963, "8": 23.641040325164795, "32": 24.18536067008972, "128": 32.365520000457764, "512": 66.2117600440979, "2048": 181.3599967956543 }, "head": { "1": 19.88160014152527, "8": 23.51240038871765, "32": 24.152560234069824, "128": 32.3472797870636, "512": 66.12256050109863, "2048": 181.31792068481445 }, "masked": { "1": 28.26927900314331, "8": 31.363999843597412, "32": 32.13752031326294, "128": 39.98728156089783, "512": 73.09632062911987, "2048": 188.31175804138184 } }, "native_source_sha256": { "afd": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "ffn_shared": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" } }, { "order": [ "ffn_shared", "head", "masked", "afd" ], "median_us": { "ffn_shared": { "1": 19.802240133285522, "8": 23.499759435653687, "32": 25.06480097770691, "128": 32.54743933677673, "512": 66.49487972259521, "2048": 180.92791557312012 }, "head": { "1": 19.78328049182892, "8": 23.37183952331543, "32": 24.088879823684692, "128": 32.3143994808197, "512": 66.11248016357422, "2048": 180.90871810913086 }, "masked": { "1": 28.21384072303772, "8": 31.301839351654053, "32": 32.92191982269287, "128": 40.225441455841064, "512": 73.06519985198975, "2048": 188.557767868042 }, "afd": { "1": 20.621519088745117, "8": 24.068158864974976, "32": 25.08911967277527, "128": 33.318480253219604, "512": 67.00663805007935, "2048": 182.7239990234375 } }, "native_source_sha256": { "ffn_shared": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "afd": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" } }, { "order": [ "head", "masked", "afd", "ffn_shared" ], "median_us": { "head": { "1": 20.271360874176025, "8": 23.300639390945435, "32": 23.701759576797485, "128": 32.53720045089722, "512": 66.33080005645752, "2048": 181.0094451904297 }, "masked": { "1": 28.219040632247925, "8": 31.37503981590271, "32": 33.02911996841431, "128": 40.099520683288574, "512": 73.17239999771118, "2048": 188.1972885131836 }, "afd": { "1": 20.25544047355652, "8": 23.993759155273438, "32": 24.842160940170288, "128": 33.27440023422241, "512": 66.84639930725098, "2048": 181.6320037841797 }, "ffn_shared": { "1": 19.777039885520935, "8": 23.25055956840515, "32": 24.7489595413208, "128": 32.38944053649902, "512": 66.00783824920654, "2048": 180.9117603302002 } }, "native_source_sha256": { "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "afd": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "ffn_shared": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" } }, { "order": [ "masked", "afd", "ffn_shared", "head" ], "median_us": { "masked": { "1": 28.351839780807495, "8": 31.434240341186523, "32": 33.188719749450684, "128": 40.55016040802002, "512": 73.33175897598267, "2048": 188.47159385681152 }, "afd": { "1": 20.181440114974976, "8": 24.015759229660034, "32": 25.9661602973938, "128": 32.81535983085632, "512": 66.76664113998413, "2048": 182.04815864562988 }, "ffn_shared": { "1": 19.843440055847168, "8": 23.38383913040161, "32": 24.590801000595093, "128": 32.291918992996216, "512": 66.05112075805664, "2048": 180.8572006225586 }, "head": { "1": 19.76199984550476, "8": 23.596320152282715, "32": 25.101680755615234, "128": 32.51255989074707, "512": 66.22992038726807, "2048": 180.89143753051758 } }, "native_source_sha256": { "masked": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "afd": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "ffn_shared": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879" } } ], "table": [ { "tokens": 1, "afd": { "median_us": 20.429800152778625, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "ffn_shared": { "median_us": 19.802519977092743, "paired_change_pct": [ -3.8893117585710124, -3.9729321197619405, -2.3618374957589117, -1.6748064419694453 ], "geomean_change_pct": -2.979738519641695, "repeats_faster": 4 }, "head": { "median_us": 19.832440316677094, "paired_change_pct": [ -3.5068631595122968, -4.064873171121985, 0.0785981457193774, -2.0783465752723163 ], "geomean_change_pct": -2.405912551780076, "repeats_faster": 3 }, "masked": { "median_us": 28.244159817695618, "paired_change_pct": [ 37.20180407083491, 36.81747014668946, 39.3158577276455, 40.484720710143684 ], "geomean_change_pct": 38.44674698851562, "repeats_faster": 0 } }, { "tokens": 8, "afd": { "median_us": 24.041959047317505, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "ffn_shared": { "median_us": 23.44179928302765, "paired_change_pct": [ -2.419109369782746, -2.3616240548771206, -3.0974703966091255, -2.6312726290076527 ], "geomean_change_pct": -2.627799837006428, "repeats_faster": 4 }, "head": { "median_us": 23.44211995601654, "paired_change_pct": [ -2.950084292902555, -2.8931142825089906, -2.8887501947591487, -1.7465159996245427 ], "geomean_change_pct": -2.6209187345179075, "repeats_faster": 4 }, "masked": { "median_us": 31.36951982975006, "paired_change_pct": [ 29.458221650526962, 30.05498063753369, 30.763335635995936, 30.890054487074003 ], "geomean_change_pct": 30.290369624819192, "repeats_faster": 0 } }, { "tokens": 32, "afd": { "median_us": 24.96564030647278, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "ffn_shared": { "median_us": 24.669880270957947, "paired_change_pct": [ -1.636264298404111, -0.09692924815830617, -0.3751742816333614, -5.296737295951881 ], "geomean_change_pct": -1.873435975012916, "repeats_faster": 4 }, "head": { "median_us": 24.12072002887726, "paired_change_pct": [ -1.7696662130403662, -3.9867474910885714, -4.590588419901708, -3.3292544291399606 ], "geomean_change_pct": -3.424765398646734, "repeats_faster": 4 }, "masked": { "median_us": 32.97551989555359, "paired_change_pct": [ 30.705785095357776, 31.2199082792735, 32.955905277167474, 27.815277150475758 ], "geomean_change_pct": 30.661077218680898, "repeats_faster": 0 } }, { "tokens": 128, "afd": { "median_us": 33.17203998565674, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "ffn_shared": { "median_us": 32.377480268478394, "paired_change_pct": [ -2.1293213065003203, -2.314153918735129, -2.6595812140686292, -1.5951092432267555 ], "geomean_change_pct": -2.175297893023398, "repeats_faster": 4 }, "head": { "median_us": 32.429919838905334, "paired_change_pct": [ -2.1844782162109055, -3.0135851478486186, -2.2155163673453493, -0.9227384422112239 ], "geomean_change_pct": -2.0869327257266312, "repeats_faster": 4 }, "masked": { "median_us": 40.16248106956482, "paired_change_pct": [ 20.91826071133056, 20.730120792211192, 20.511625757409103, 23.570671225401753 ], "geomean_change_pct": 21.426358072777994, "repeats_faster": 0 } }, { "tokens": 512, "afd": { "median_us": 66.92651867866516, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "ffn_shared": { "median_us": 66.13144040107727, "paired_change_pct": [ -1.2762003344232276, -0.7637427311330769, -1.254459577082223, -1.071673473025725 ], "geomean_change_pct": -1.0917317541768723, "repeats_faster": 4 }, "head": { "median_us": 66.17624044418335, "paired_change_pct": [ -1.4091996355663494, -1.3344318003790212, -0.7713194070836482, -0.8038756234430955 ], "geomean_change_pct": -1.0801421316164905, "repeats_faster": 4 }, "masked": { "median_us": 73.13436031341553, "paired_change_pct": [ 8.988894257965896, 9.041733741935175, 9.463487571534767, 9.832931122345956 ], "geomean_change_pct": 9.331224532885285, "repeats_faster": 0 } }, { "tokens": 2048, "afd": { "median_us": 182.0548391342163, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "ffn_shared": { "median_us": 180.91983795166016, "paired_change_pct": [ -0.38532185637133454, -0.9829488517745344, -0.3965399483426446, -0.6541994337825652 ], "geomean_change_pct": -0.6050508427595358, "repeats_faster": 4 }, "head": { "median_us": 180.95908164978027, "paired_change_pct": [ -0.4084327866365589, -0.9934551148225501, -0.34275820382939637, -0.6353929222453369 ], "geomean_change_pct": -0.5953355779343861, "repeats_faster": 4 }, "masked": { "median_us": 188.39167594909668, "paired_change_pct": [ 3.433036498612352, 3.1926670146137814, 3.614607884194765, 3.5284263565034557 ], "geomean_change_pct": 3.442064113703802, "repeats_faster": 0 } } ] }, "c-3a1f": { "arms": { "afd": {}, "ffn_shared": { "_env": { "AFDEP_SHARED_FFN_RECEIVE": "1" } }, "head": { "_tree": "/work/src-head", "_digest": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879" }, "masked": { "role_aware": false } }, "passthrough": [ "--payload", "bf16", "--routing", "random", "--hidden", "2048", "--tokens", "1", "8", "32", "128", "512", "2048", "--capacity", "2048", "--ffn-ranks", "1", "--attention-ranks", "3" ], "processes": [ { "order": [ "afd", "ffn_shared", "head", "masked" ], "median_us": { "afd": { "1": 18.629519939422607, "8": 20.907840728759766, "32": 24.286799430847168, "128": 36.025519371032715, "512": 97.10279941558838, "2048": 324.13936614990234 }, "ffn_shared": { "1": 18.233360052108765, "8": 20.34983992576599, "32": 23.712159395217896, "128": 35.48200011253357, "512": 96.32215976715088, "2048": 322.89785385131836 }, "head": { "1": 18.21079969406128, "8": 20.137521028518677, "32": 23.609360456466675, "128": 35.69040060043335, "512": 96.28223896026611, "2048": 322.9583930969238 }, "masked": { "1": 32.532719373703, "8": 34.92184042930603, "32": 39.252800941467285, "128": 50.64847946166992, "512": 110.81984043121338, "2048": 343.2080841064453 } }, "native_source_sha256": { "afd": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "ffn_shared": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" } }, { "order": [ "ffn_shared", "head", "masked", "afd" ], "median_us": { "ffn_shared": { "1": 18.120480179786682, "8": 20.81272006034851, "32": 23.746320009231567, "128": 35.70320010185242, "512": 96.31887912750244, "2048": 322.7515983581543 }, "head": { "1": 18.609039783477783, "8": 20.252079963684082, "32": 23.68280053138733, "128": 35.53439974784851, "512": 96.4567232131958, "2048": 322.55584716796875 }, "masked": { "1": 32.64583945274353, "8": 35.17216086387634, "32": 39.5851194858551, "128": 50.27320146560669, "512": 110.30600070953369, "2048": 342.4392890930176 }, "afd": { "1": 18.629679679870605, "8": 21.108320951461792, "32": 24.393279552459717, "128": 36.146479845047, "512": 97.19768047332764, "2048": 323.8924789428711 } }, "native_source_sha256": { "ffn_shared": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "afd": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" } }, { "order": [ "head", "masked", "afd", "ffn_shared" ], "median_us": { "head": { "1": 18.134559988975525, "8": 20.686079263687134, "32": 24.0722393989563, "128": 35.68400025367737, "512": 96.48560047149658, "2048": 322.6656150817871 }, "masked": { "1": 32.528719902038574, "8": 34.78079915046692, "32": 39.55191969871521, "128": 50.41152000427246, "512": 110.3656005859375, "2048": 342.3296928405762 }, "afd": { "1": 18.832799792289734, "8": 20.96895933151245, "32": 23.940240144729614, "128": 35.99799990653992, "512": 97.24256038665771, "2048": 323.3282279968262 }, "ffn_shared": { "1": 18.561279773712158, "8": 20.53712010383606, "32": 23.762240409851074, "128": 35.486000776290894, "512": 96.2554407119751, "2048": 322.6350402832031 } }, "native_source_sha256": { "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "afd": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "ffn_shared": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" } }, { "order": [ "masked", "afd", "ffn_shared", "head" ], "median_us": { "masked": { "1": 32.53023982048035, "8": 35.477439165115356, "32": 39.397679567337036, "128": 50.820560455322266, "512": 110.58512210845947, "2048": 343.1437683105469 }, "afd": { "1": 18.753679990768433, "8": 21.043280363082886, "32": 24.340720176696777, "128": 36.03056073188782, "512": 97.29360103607178, "2048": 323.6454391479492 }, "ffn_shared": { "1": 18.23863983154297, "8": 20.930880308151245, "32": 24.243760108947754, "128": 35.83864092826843, "512": 96.77023887634277, "2048": 322.9202461242676 }, "head": { "1": 18.590720295906067, "8": 20.271599292755127, "32": 23.86423945426941, "128": 35.57415962219238, "512": 96.287522315979, "2048": 322.457275390625 } }, "native_source_sha256": { "masked": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "afd": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "ffn_shared": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879" } } ], "table": [ { "tokens": 1, "afd": { "median_us": 18.69167983531952, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "ffn_shared": { "median_us": 18.235999941825867, "paired_change_pct": [ -2.126516886114249, -2.7332702914592444, -1.4417400576240302, -2.746341835197119 ], "geomean_change_pct": -2.263432400287424, "repeats_faster": 4 }, "head": { "median_us": 18.400759994983673, "paired_change_pct": [ -2.2476169365763354, -0.11079039869442697, -3.707573016307819, -0.868947827533495 ], "geomean_change_pct": -1.743351873339316, "repeats_faster": 4 }, "masked": { "median_us": 32.531479597091675, "paired_change_pct": [ 74.62993936230922, 75.23564555979674, 72.72375993375151, 73.46056793383207 ], "geomean_change_pct": 74.00971705689139, "repeats_faster": 0 } }, { "tokens": 8, "afd": { "median_us": 21.00611984729767, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "ffn_shared": { "median_us": 20.674920082092285, "paired_change_pct": [ -2.6688590669538437, -1.4003998318625643, -2.0594213611136003, -0.5341375155977546 ], "geomean_change_pct": -1.668894541695165, "repeats_faster": 4 }, "head": { "median_us": 20.261839628219604, "paired_change_pct": [ -3.6843579891130296, -4.056414481031534, -1.3490419975215495, -3.667113952830048 ], "geomean_change_pct": -3.1951398824176858, "repeats_faster": 4 }, "masked": { "median_us": 35.04700064659119, "paired_change_pct": [ 67.02748448465708, 66.62699484603299, 65.86802711853153, 68.5927220137927 ], "geomean_change_pct": 67.02585400687369, "repeats_faster": 0 } }, { "tokens": 32, "afd": { "median_us": 24.313759803771973, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "ffn_shared": { "median_us": 23.75428020954132, "paired_change_pct": [ -2.366059131280218, -2.6522040295435034, -0.7435169146276399, -0.39834510665732736 ], "geomean_change_pct": -1.5449278048727866, "repeats_faster": 4 }, "head": { "median_us": 23.77351999282837, "paired_change_pct": [ -2.789329966302867, -2.912601479208421, 0.5513698000884304, -1.9575457051740797 ], "geomean_change_pct": -1.7868271166723737, "repeats_faster": 3 }, "masked": { "median_us": 39.47479963302612, "paired_change_pct": [ 61.62195868267224, 62.27879240560543, 65.21103990438655, 61.859136793558925 ], "geomean_change_pct": 62.73636251592332, "repeats_faster": 0 } }, { "tokens": 128, "afd": { "median_us": 36.028040051460266, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "ffn_shared": { "median_us": 35.594600439071655, "paired_change_pct": [ -1.5087062393226103, -1.226342772781297, -1.4222988265412129, -0.5326583869940449 ], "geomean_change_pct": -1.1732431682054623, "repeats_faster": 4 }, "head": { "median_us": 35.629079937934875, "paired_change_pct": [ -0.9302260632190307, -1.6933325176403224, -0.8722697196449025, -1.2667055422523887 ], "geomean_change_pct": -1.1911751437502116, "repeats_faster": 4 }, "masked": { "median_us": 50.52999973297119, "paired_change_pct": [ 40.590560097227055, 39.081873756775856, 40.03978036322506, 41.048486126792305 ], "geomean_change_pct": 40.18825700168418, "repeats_faster": 0 } }, { "tokens": 512, "afd": { "median_us": 97.22012042999268, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "ffn_shared": { "median_us": 96.32051944732666, "paired_change_pct": [ -0.8039311463065579, -0.904138186781478, -1.0151107403564952, -0.5379204327476406 ], "geomean_change_pct": -0.8154324094016951, "repeats_faster": 4 }, "head": { "median_us": 96.3721227645874, "paired_change_pct": [ -0.84504304743096, -0.7623198995321379, -0.7784245007034896, -1.0340646346513216 ], "geomean_change_pct": -0.8550218321389802, "repeats_faster": 4 }, "masked": { "median_us": 110.47536134719849, "paired_change_pct": [ 14.126308508282758, 13.486247997248402, 13.495161117827115, 13.661248973054096 ], "geomean_change_pct": 13.691944501959231, "repeats_faster": 0 } }, { "tokens": 2048, "afd": { "median_us": 323.76895904541016, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "ffn_shared": { "median_us": 322.8247261047363, "paired_change_pct": [ -0.3830180558845875, -0.3522405300797504, -0.21439133784194597, -0.2240702126347971 ], "geomean_change_pct": -0.29345829607362317, "repeats_faster": 4 }, "head": { "median_us": 322.61073112487793, "paired_change_pct": [ -0.36434113727252804, -0.41267762044517875, -0.20493506525683935, -0.3671189559946386 ], "geomean_change_pct": -0.3372993126197654, "repeats_faster": 4 }, "masked": { "median_us": 342.7915287017822, "paired_change_pct": [ 5.882876301955986, 5.726224397269131, 5.876834497709416, 6.0245956853062 ], "geomean_change_pct": 5.877580121364012, "repeats_faster": 0 } } ] } }, "overlap": { "o-2a2f": { "native_source_sha256": { "afd": [ "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" ], "ffn_shared": [ "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" ], "masked": [ "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" ] }, "per_repeat": { "afd": { "1": { "attention.gemm_us_per_gemm": [ 25.91, 26.1, 25.87 ], "attention.exchange_us": [ 34.91, 35.15, 35.27 ], "attention.makespan_us": [ 6981.71, 7030.42, 7054.56 ], "ffn.gemm_us_per_gemm": [ 25.94, 25.86, 25.98 ], "ffn.exchange_us": [ 34.92, 35.13, 35.21 ], "ffn.makespan_us": [ 6984.96, 7026.34, 7042.35 ] }, "32": { "attention.gemm_us_per_gemm": [ 26.11, 26.23, 26.0 ], "attention.exchange_us": [ 44.86, 45.2, 45.61 ], "attention.makespan_us": [ 8972.88, 9039.22, 9122.69 ], "ffn.gemm_us_per_gemm": [ 28.91, 29.03, 28.93 ], "ffn.exchange_us": [ 44.84, 45.15, 45.62 ], "ffn.makespan_us": [ 8968.9, 9029.65, 9123.62 ] }, "512": { "attention.gemm_us_per_gemm": [ 27.16, 27.22, 27.12 ], "attention.exchange_us": [ 112.2, 112.46, 112.96 ], "attention.makespan_us": [ 22439.6, 22491.49, 22591.81 ], "ffn.gemm_us_per_gemm": [ 29.75, 29.7, 29.51 ], "ffn.exchange_us": [ 112.21, 112.4, 112.9 ], "ffn.makespan_us": [ 22442.62, 22480.54, 22579.09 ] } }, "ffn_shared": { "1": { "attention.gemm_us_per_gemm": [ 26.31, 26.45, 26.2 ], "attention.exchange_us": [ 36.45, 35.51, 35.9 ], "attention.makespan_us": [ 7290.53, 7101.65, 7180.85 ], "ffn.gemm_us_per_gemm": [ 28.9, 27.62, 28.11 ], "ffn.exchange_us": [ 36.46, 35.47, 35.88 ], "ffn.makespan_us": [ 7292.75, 7094.53, 7175.18 ] }, "32": { "attention.gemm_us_per_gemm": [ 26.0, 26.12, 25.96 ], "attention.exchange_us": [ 47.25, 47.16, 46.52 ], "attention.makespan_us": [ 9450.59, 9432.88, 9304.96 ], "ffn.gemm_us_per_gemm": [ 34.93, 34.28, 33.88 ], "ffn.exchange_us": [ 47.3, 47.14, 46.48 ], "ffn.makespan_us": [ 9459.46, 9427.89, 9295.31 ] }, "512": { "attention.gemm_us_per_gemm": [ 27.38, 27.6, 27.41 ], "attention.exchange_us": [ 121.67, 120.76, 119.87 ], "attention.makespan_us": [ 24333.28, 24151.17, 23973.12 ], "ffn.gemm_us_per_gemm": [ 46.19, 45.83, 45.52 ], "ffn.exchange_us": [ 121.61, 120.78, 119.85 ], "ffn.makespan_us": [ 24321.79, 24156.72, 23970.82 ] } }, "masked": { "1": { "attention.gemm_us_per_gemm": [ 29.13, 29.34, 29.24 ], "attention.exchange_us": [ 43.33, 43.55, 43.53 ], "attention.makespan_us": [ 8666.24, 8709.57, 8706.29 ], "ffn.gemm_us_per_gemm": [ 29.2, 29.42, 29.5 ], "ffn.exchange_us": [ 43.32, 43.53, 43.49 ], "ffn.makespan_us": [ 8664.72, 8706.46, 8698.75 ] }, "32": { "attention.gemm_us_per_gemm": [ 27.42, 27.94, 27.82 ], "attention.exchange_us": [ 55.16, 54.41, 54.82 ], "attention.makespan_us": [ 11032.37, 10882.99, 10964.02 ], "ffn.gemm_us_per_gemm": [ 36.18, 35.47, 35.73 ], "ffn.exchange_us": [ 55.15, 54.38, 54.8 ], "ffn.makespan_us": [ 11030.86, 10875.86, 10959.04 ] }, "512": { "attention.gemm_us_per_gemm": [ 29.99, 29.91, 29.8 ], "attention.exchange_us": [ 119.52, 119.66, 119.2 ], "attention.makespan_us": [ 23903.31, 23931.07, 23839.65 ], "ffn.gemm_us_per_gemm": [ 35.68, 35.76, 35.84 ], "ffn.exchange_us": [ 119.45, 119.66, 119.3 ], "ffn.makespan_us": [ 23889.04, 23932.35, 23860.21 ] } } } }, "o-3a1f": { "native_source_sha256": { "afd": [ "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" ], "ffn_shared": [ "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" ], "masked": [ "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" ] }, "per_repeat": { "afd": { "1": { "attention.gemm_us_per_gemm": [ 25.88, 25.9, 25.92 ], "attention.exchange_us": [ 34.89, 34.71, 35.76 ], "attention.makespan_us": [ 6978.0, 6941.34, 7152.83 ], "ffn.gemm_us_per_gemm": [ 25.92, 25.9, 25.18 ], "ffn.exchange_us": [ 34.75, 34.68, 35.57 ], "ffn.makespan_us": [ 6950.29, 6935.41, 7113.55 ] }, "32": { "attention.gemm_us_per_gemm": [ 26.12, 26.16, 26.01 ], "attention.exchange_us": [ 46.41, 46.03, 46.8 ], "attention.makespan_us": [ 9281.55, 9205.71, 9359.39 ], "ffn.gemm_us_per_gemm": [ 29.84, 29.88, 27.47 ], "ffn.exchange_us": [ 46.4, 45.98, 46.7 ], "ffn.makespan_us": [ 9279.82, 9196.98, 9339.34 ] }, "512": { "attention.gemm_us_per_gemm": [ 26.61, 26.55, 26.51 ], "attention.exchange_us": [ 149.49, 149.5, 148.86 ], "attention.makespan_us": [ 29898.58, 29900.42, 29771.02 ], "ffn.gemm_us_per_gemm": [ 31.51, 31.45, 30.93 ], "ffn.exchange_us": [ 149.47, 149.53, 148.7 ], "ffn.makespan_us": [ 29894.78, 29905.12, 29740.14 ] } }, "ffn_shared": { "1": { "attention.gemm_us_per_gemm": [ 26.27, 26.18, 26.23 ], "attention.exchange_us": [ 37.83, 37.55, 37.68 ], "attention.makespan_us": [ 7565.58, 7510.5, 7536.45 ], "ffn.gemm_us_per_gemm": [ 30.71, 29.89, 30.53 ], "ffn.exchange_us": [ 37.8, 37.4, 37.64 ], "ffn.makespan_us": [ 7560.27, 7479.14, 7528.02 ] }, "32": { "attention.gemm_us_per_gemm": [ 26.12, 26.0, 26.07 ], "attention.exchange_us": [ 46.55, 47.31, 46.69 ], "attention.makespan_us": [ 9310.05, 9462.66, 9337.17 ], "ffn.gemm_us_per_gemm": [ 34.26, 35.99, 33.8 ], "ffn.exchange_us": [ 46.51, 47.26, 46.66 ], "ffn.makespan_us": [ 9302.88, 9452.54, 9331.31 ] }, "512": { "attention.gemm_us_per_gemm": [ 26.7, 26.66, 26.62 ], "attention.exchange_us": [ 152.72, 152.58, 149.91 ], "attention.makespan_us": [ 30544.24, 30515.42, 29981.28 ], "ffn.gemm_us_per_gemm": [ 40.84, 41.48, 39.34 ], "ffn.exchange_us": [ 152.44, 152.52, 149.74 ], "ffn.makespan_us": [ 30487.7, 30504.78, 29948.1 ] } }, "masked": { "1": { "attention.gemm_us_per_gemm": [ 30.76, 30.64, 30.67 ], "attention.exchange_us": [ 47.81, 47.66, 47.51 ], "attention.makespan_us": [ 9561.17, 9531.97, 9502.96 ], "ffn.gemm_us_per_gemm": [ 30.5, 30.65, 30.71 ], "ffn.exchange_us": [ 47.73, 47.65, 47.6 ], "ffn.makespan_us": [ 9546.61, 9529.49, 9519.18 ] }, "32": { "attention.gemm_us_per_gemm": [ 27.8, 27.72, 27.56 ], "attention.exchange_us": [ 60.44, 59.87, 59.92 ], "attention.makespan_us": [ 12087.01, 11974.58, 11984.38 ], "ffn.gemm_us_per_gemm": [ 36.84, 36.28, 36.55 ], "ffn.exchange_us": [ 60.4, 59.89, 59.92 ], "ffn.makespan_us": [ 12080.4, 11977.34, 11983.09 ] }, "512": { "attention.gemm_us_per_gemm": [ 31.21, 30.9, 32.01 ], "attention.exchange_us": [ 164.77, 165.04, 173.41 ], "attention.makespan_us": [ 32953.79, 33007.41, 34682.43 ], "ffn.gemm_us_per_gemm": [ 36.14, 36.18, 36.84 ], "ffn.exchange_us": [ 164.74, 164.95, 173.47 ], "ffn.makespan_us": [ 32947.07, 32989.04, 34694.45 ] } } } } } }, "counts_acc5a690_vs_arrival": { "screens": { "c-2a2f": { "arms": { "arrival": { "_digest": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" }, "counts": { "_tree": "/work/src-fin3b", "_digest": "acc5a6907f9221b71da93428f8f5754d9ab6ee4b705c78842d080935ca007d34" }, "v21_shape": { "_env": { "AFDEP_SHARED_FFN_RECEIVE": "1" }, "_digest": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" } }, "passthrough": [ "--payload", "bf16", "--routing", "random", "--hidden", "2048", "--tokens", "1", "8", "32", "128", "512", "2048", "--capacity", "2048" ], "processes": [ { "order": [ "arrival", "counts", "v21_shape" ], "median_us": { "arrival": { "1": 20.208719968795776, "8": 23.860799074172974, "32": 25.444079637527466, "128": 33.03816080093384, "512": 66.9918417930603, "2048": 181.88600540161133 }, "counts": { "1": 20.16103982925415, "8": 23.273359537124634, "32": 24.47975993156433, "128": 32.21943974494934, "512": 66.36160135269165, "2048": 181.41047477722168 }, "v21_shape": { "1": 20.175440311431885, "8": 23.37056040763855, "32": 24.2631995677948, "128": 32.51240015029907, "512": 66.33872032165527, "2048": 181.35696411132812 } }, "native_source_sha256": { "arrival": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "counts": "acc5a6907f9221b71da93428f8f5754d9ab6ee4b705c78842d080935ca007d34", "v21_shape": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" } }, { "order": [ "counts", "v21_shape", "arrival" ], "median_us": { "counts": { "1": 19.815840125083923, "8": 23.459599018096924, "32": 24.611600637435913, "128": 32.45296001434326, "512": 66.28759860992432, "2048": 180.85944175720215 }, "v21_shape": { "1": 20.335919857025146, "8": 23.476799726486206, "32": 24.33784008026123, "128": 32.55928039550781, "512": 66.52776002883911, "2048": 181.4132785797119 }, "arrival": { "1": 20.597280263900757, "8": 24.008560180664062, "32": 25.078399181365967, "128": 33.001999855041504, "512": 67.01807975769043, "2048": 182.55752563476562 } }, "native_source_sha256": { "counts": "acc5a6907f9221b71da93428f8f5754d9ab6ee4b705c78842d080935ca007d34", "v21_shape": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "arrival": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" } }, { "order": [ "v21_shape", "arrival", "counts" ], "median_us": { "v21_shape": { "1": 19.79368031024933, "8": 23.393919467926025, "32": 24.025280475616455, "128": 32.18135952949524, "512": 66.16631984710693, "2048": 180.99231719970703 }, "arrival": { "1": 20.789999961853027, "8": 23.976240158081055, "32": 24.602320194244385, "128": 32.73352026939392, "512": 66.78175926208496, "2048": 181.83863639831543 }, "counts": { "1": 19.772719740867615, "8": 23.117520809173584, "32": 23.653520345687866, "128": 32.13688015937805, "512": 66.20712041854858, "2048": 180.9040069580078 } }, "native_source_sha256": { "v21_shape": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "arrival": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "counts": "acc5a6907f9221b71da93428f8f5754d9ab6ee4b705c78842d080935ca007d34" } }, { "order": [ "arrival", "counts", "v21_shape" ], "median_us": { "arrival": { "1": 20.58295965194702, "8": 23.855199813842773, "32": 25.054080486297607, "128": 33.041839599609375, "512": 67.07759857177734, "2048": 182.38944053649902 }, "counts": { "1": 20.15760064125061, "8": 23.424720764160156, "32": 23.715519905090332, "128": 32.131919860839844, "512": 66.25256061553955, "2048": 180.72951316833496 }, "v21_shape": { "1": 20.288079977035522, "8": 23.427040576934814, "32": 25.193519592285156, "128": 32.46496081352234, "512": 66.457679271698, "2048": 181.3541603088379 } }, "native_source_sha256": { "arrival": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "counts": "acc5a6907f9221b71da93428f8f5754d9ab6ee4b705c78842d080935ca007d34", "v21_shape": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" } } ], "table": [ { "tokens": 1, "arrival": { "median_us": 20.59011995792389, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "counts": { "median_us": 19.986720383167267, "paired_change_pct": [ -0.2359384444697521, -3.793899625604469, -4.893122765040836, -2.066559026928738 ], "geomean_change_pct": -2.7633658537819628, "repeats_faster": 4 }, "v21_shape": { "median_us": 20.231760144233704, "paired_change_pct": [ -0.1646796898332914, -1.2689073679969076, -4.792302325309361, -1.4326398141853436 ], "geomean_change_pct": -1.9300764184041252, "repeats_faster": 4 } }, { "tokens": 8, "arrival": { "median_us": 23.918519616127014, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "counts": { "median_us": 23.349040150642395, "paired_change_pct": [ -2.4619441085030003, -2.2865226337448585, -3.5815429910850516, -1.8045501737227854 ], "geomean_change_pct": -2.5358215679496032, "repeats_faster": 4 }, "v21_shape": { "median_us": 23.41048002243042, "paired_change_pct": [ -2.054577740713892, -2.214878569045231, -2.4287406462216388, -1.7948256155855158 ], "geomean_change_pct": -2.123529272065172, "repeats_faster": 4 } }, { "tokens": 32, "arrival": { "median_us": 25.066239833831787, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "counts": { "median_us": 24.09763991832733, "paired_change_pct": [ -3.7899571126198683, -1.8613570210530006, -3.856546216232426, -5.342684924874219 ], "geomean_change_pct": -3.720552496651164, "repeats_faster": 4 }, "v21_shape": { "median_us": 24.300519824028015, "paired_change_pct": [ -4.641079915466806, -2.9529759684780665, -2.3454686959278215, 0.5565524788020415 ], "geomean_change_pct": -2.3636399113499684, "repeats_faster": 3 } }, { "tokens": 128, "arrival": { "median_us": 33.02008032798767, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "counts": { "median_us": 32.178159952163696, "paired_change_pct": [ -2.4781072436736773, -1.663656272680003, -1.8227190510081814, -2.7538410385004375 ], "geomean_change_pct": -2.180619352770796, "repeats_faster": 4 }, "v21_shape": { "median_us": 32.488680481910706, "paired_change_pct": [ -1.5913738473598782, -1.3414928230964795, -1.6868358042595122, -1.7459039601834259 ], "geomean_change_pct": -1.5915227342918281, "repeats_faster": 4 } }, { "tokens": 512, "arrival": { "median_us": 67.00496077537537, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "counts": { "median_us": 66.27007961273193, "paired_change_pct": [ -0.940771926103301, -1.0899762428395854, -0.8604727546652535, -1.229975392388194 ], "geomean_change_pct": -1.0304005057534815, "repeats_faster": 4 }, "v21_shape": { "median_us": 66.39819979667664, "paired_change_pct": [ -0.9749268775480768, -0.731623064438891, -0.9215681374351581, -0.9241823101582725 ], "geomean_change_pct": -0.8881185145067971, "repeats_faster": 4 } }, { "tokens": 2048, "arrival": { "median_us": 182.13772296905518, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "counts": { "median_us": 180.88172435760498, "paired_change_pct": [ -0.2614443169168812, -0.9301637232752347, -0.5139883683797142, -0.9101005865698064 ], "geomean_change_pct": -0.6543210792442578, "repeats_faster": 4 }, "v21_shape": { "median_us": 181.355562210083, "paired_change_pct": [ -0.2908642086646873, -0.6267871187862983, -0.46542319903595253, -0.5676207046942272 ], "geomean_change_pct": -0.4877553850164196, "repeats_faster": 4 } } ] }, "c-3a1f": { "arms": { "arrival": { "_digest": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" }, "counts": { "_tree": "/work/src-fin3b", "_digest": "acc5a6907f9221b71da93428f8f5754d9ab6ee4b705c78842d080935ca007d34" }, "v21_shape": { "_env": { "AFDEP_SHARED_FFN_RECEIVE": "1" }, "_digest": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" } }, "passthrough": [ "--payload", "bf16", "--routing", "random", "--hidden", "2048", "--tokens", "1", "8", "32", "128", "512", "2048", "--capacity", "2048", "--ffn-ranks", "1", "--attention-ranks", "3" ], "processes": [ { "order": [ "arrival", "counts", "v21_shape" ], "median_us": { "arrival": { "1": 18.670960068702698, "8": 21.51368021965027, "32": 24.5580792427063, "128": 36.1190402507782, "512": 97.40240097045898, "2048": 323.7892723083496 }, "counts": { "1": 18.574800491333008, "8": 19.99080002307892, "32": 23.30183982849121, "128": 35.49247980117798, "512": 96.35200023651123, "2048": 322.5119209289551 }, "v21_shape": { "1": 18.128719925880432, "8": 20.33128023147583, "32": 24.092799425125122, "128": 35.63423991203308, "512": 96.28007888793945, "2048": 322.9381561279297 } }, "native_source_sha256": { "arrival": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "counts": "acc5a6907f9221b71da93428f8f5754d9ab6ee4b705c78842d080935ca007d34", "v21_shape": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" } }, { "order": [ "counts", "v21_shape", "arrival" ], "median_us": { "counts": { "1": 18.118239641189575, "8": 20.784000158309937, "32": 23.807519674301147, "128": 35.41311979293823, "512": 96.36919975280762, "2048": 322.841854095459 }, "v21_shape": { "1": 18.106080293655396, "8": 20.461599826812744, "32": 23.70368003845215, "128": 35.51535964012146, "512": 96.09895706176758, "2048": 322.7220916748047 }, "arrival": { "1": 19.002079963684082, "8": 20.994079113006592, "32": 24.25960063934326, "128": 35.90975999832153, "512": 97.22239971160889, "2048": 323.697452545166 } }, "native_source_sha256": { "counts": "acc5a6907f9221b71da93428f8f5754d9ab6ee4b705c78842d080935ca007d34", "v21_shape": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "arrival": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" } }, { "order": [ "v21_shape", "arrival", "counts" ], "median_us": { "v21_shape": { "1": 18.16767990589142, "8": 20.340880155563354, "32": 23.665599822998047, "128": 35.44656038284302, "512": 96.15295886993408, "2048": 322.8445625305176 }, "arrival": { "1": 19.35167968273163, "8": 21.493359804153442, "32": 24.01576042175293, "128": 36.00376009941101, "512": 97.20736026763916, "2048": 323.5822296142578 }, "counts": { "1": 18.598079681396484, "8": 20.322880744934082, "32": 23.50559949874878, "128": 35.4912006855011, "512": 96.38351917266846, "2048": 322.52471923828125 } }, "native_source_sha256": { "v21_shape": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "arrival": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "counts": "acc5a6907f9221b71da93428f8f5754d9ab6ee4b705c78842d080935ca007d34" } }, { "order": [ "arrival", "counts", "v21_shape" ], "median_us": { "arrival": { "1": 18.659120202064514, "8": 21.54103994369507, "32": 24.07472014427185, "128": 36.26319885253906, "512": 97.55216121673584, "2048": 323.45510482788086 }, "counts": { "1": 18.602800369262695, "8": 20.372880697250366, "32": 23.455519676208496, "128": 35.35399913787842, "512": 96.29240036010742, "2048": 322.7233695983887 }, "v21_shape": { "1": 18.132479786872864, "8": 20.387200117111206, "32": 23.712799549102783, "128": 35.48647999763489, "512": 96.28024101257324, "2048": 322.8378486633301 } }, "native_source_sha256": { "arrival": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "counts": "acc5a6907f9221b71da93428f8f5754d9ab6ee4b705c78842d080935ca007d34", "v21_shape": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" } } ], "table": [ { "tokens": 1, "arrival": { "median_us": 18.83652001619339, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "counts": { "median_us": 18.586440086364746, "paired_change_pct": [ -0.5150221360650797, -4.651281986938605, -3.894235609984875, -0.30183541448854934 ], "geomean_change_pct": -2.3601189702115333, "repeats_faster": 4 }, "v21_shape": { "median_us": 18.130599856376648, "paired_change_pct": [ -2.9041899336028187, -4.7152715478572915, -6.118330792219274, -2.8224289756886844 ], "geomean_change_pct": -4.1498806299266615, "repeats_faster": 4 } }, { "tokens": 8, "arrival": { "median_us": 21.503520011901855, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "counts": { "median_us": 20.347880721092224, "paired_change_pct": [ -7.07865962970099, -1.0006581072970366, -5.445770553718521, -5.422947311263004 ], "geomean_change_pct": -4.76345595895693, "repeats_faster": 4 }, "v21_shape": { "median_us": 20.36404013633728, "paired_change_pct": [ -5.4960377587766285, -2.5363307593899487, -5.362026500702688, -5.356472248321442 ], "geomean_change_pct": -4.695746999233386, "repeats_faster": 4 } }, { "tokens": 32, "arrival": { "median_us": 24.167160391807556, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "counts": { "median_us": 23.480559587478638, "paired_change_pct": [ -5.115381385489215, -1.863513632243996, -2.1242755342531616, -2.571994458721394 ], "geomean_change_pct": -2.9274852185046774, "repeats_faster": 4 }, "v21_shape": { "median_us": 23.708239793777466, "paired_change_pct": [ -1.8946099692196516, -2.291548855876646, -1.4580450196268413, -1.5033221279424969 ], "geomean_change_pct": -1.7874610004287805, "repeats_faster": 4 } }, { "tokens": 128, "arrival": { "median_us": 36.061400175094604, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "counts": { "median_us": 35.452160239219666, "paired_change_pct": [ -1.7347095749221109, -1.3830229035407537, -1.4236274558398043, -2.5072242478051066 ], "geomean_change_pct": -1.7631848644449222, "repeats_faster": 4 }, "v21_shape": { "median_us": 35.500919818878174, "paired_change_pct": [ -1.342229293411723, -1.0983096467882536, -1.5476153463679698, -2.1418928265612425 ], "geomean_change_pct": -1.533269960344008, "repeats_faster": 4 } }, { "tokens": 512, "arrival": { "median_us": 97.31240034103394, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "counts": { "median_us": 96.36059999465942, "paired_change_pct": [ -1.078413595026606, -0.8775754983749828, -0.8475089671218683, -1.2913715502720136 ], "geomean_change_pct": -1.0238779334501569, "repeats_faster": 4 }, "v21_shape": { "median_us": 96.21651887893677, "paired_change_pct": [ -1.1522529951391247, -1.155538901707609, -1.0846929644031222, -1.303836007627468 ], "geomean_change_pct": -1.1741126690724912, "repeats_faster": 4 } }, { "tokens": 2048, "arrival": { "median_us": 323.6398410797119, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "counts": { "median_us": 322.62404441833496, "paired_change_pct": [ -0.394500833918332, -0.2643204149367362, -0.32681348949143985, -0.22622466567085997 ], "geomean_change_pct": -0.3029853278637362, "repeats_faster": 4 }, "v21_shape": { "median_us": 322.8412055969238, "paired_change_pct": [ -0.26286114248077475, -0.30131867356145703, -0.2279689724060563, -0.19083209859347416 ], "geomean_change_pct": -0.2457536334899424, "repeats_faster": 4 } } ] } }, "overlap": { "o-2a2f": { "native_source_sha256": { "arrival": [ "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" ], "counts": [ "acc5a6907f9221b71da93428f8f5754d9ab6ee4b705c78842d080935ca007d34" ], "v21_shape": [ "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" ] }, "per_repeat": { "arrival": { "1": { "attention.gemm_us_per_gemm": [ 25.89, 25.95, 25.87 ], "attention.exchange_us": [ 35.24, 35.0, 35.15 ], "attention.makespan_us": [ 7047.25, 7000.96, 7030.64 ], "ffn.gemm_us_per_gemm": [ 25.82, 25.98, 25.9 ], "ffn.exchange_us": [ 35.22, 34.94, 35.07 ], "ffn.makespan_us": [ 7043.1, 6987.58, 7014.0 ] }, "32": { "attention.gemm_us_per_gemm": [ 26.17, 26.2, 26.06 ], "attention.exchange_us": [ 45.2, 45.51, 44.94 ], "attention.makespan_us": [ 9039.09, 9101.01, 8988.77 ], "ffn.gemm_us_per_gemm": [ 28.67, 29.31, 29.03 ], "ffn.exchange_us": [ 45.23, 45.45, 44.92 ], "ffn.makespan_us": [ 9046.18, 9090.51, 8984.64 ] }, "512": { "attention.gemm_us_per_gemm": [ 27.09, 26.41, 27.07 ], "attention.exchange_us": [ 113.22, 116.57, 112.55 ], "attention.makespan_us": [ 22644.11, 23313.82, 22509.94 ], "ffn.gemm_us_per_gemm": [ 29.37, 28.71, 29.72 ], "ffn.exchange_us": [ 113.2, 116.62, 112.56 ], "ffn.makespan_us": [ 22639.44, 23324.27, 22511.7 ] } }, "counts": { "1": { "attention.gemm_us_per_gemm": [ 26.01, 26.14, 26.15 ], "attention.exchange_us": [ 34.9, 34.48, 34.72 ], "attention.makespan_us": [ 6980.66, 6896.64, 6944.67 ], "ffn.gemm_us_per_gemm": [ 26.01, 26.03, 26.08 ], "ffn.exchange_us": [ 34.79, 34.51, 34.7 ], "ffn.makespan_us": [ 6958.75, 6901.94, 6939.49 ] }, "32": { "attention.gemm_us_per_gemm": [ 26.11, 26.33, 26.19 ], "attention.exchange_us": [ 45.0, 44.18, 44.79 ], "attention.makespan_us": [ 8999.74, 8836.61, 8958.13 ], "ffn.gemm_us_per_gemm": [ 29.19, 29.02, 29.01 ], "ffn.exchange_us": [ 44.99, 44.17, 44.85 ], "ffn.makespan_us": [ 8998.13, 8833.3, 8970.1 ] }, "512": { "attention.gemm_us_per_gemm": [ 27.22, 27.34, 27.31 ], "attention.exchange_us": [ 112.35, 111.65, 111.6 ], "attention.makespan_us": [ 22469.36, 22329.66, 22319.78 ], "ffn.gemm_us_per_gemm": [ 30.58, 30.56, 30.49 ], "ffn.exchange_us": [ 112.22, 111.64, 111.64 ], "ffn.makespan_us": [ 22444.94, 22328.1, 22327.44 ] } }, "v21_shape": { "1": { "attention.gemm_us_per_gemm": [ 26.15, 26.45, 26.42 ], "attention.exchange_us": [ 37.43, 36.14, 37.02 ], "attention.makespan_us": [ 7486.82, 7227.9, 7403.46 ], "ffn.gemm_us_per_gemm": [ 30.09, 28.31, 29.49 ], "ffn.exchange_us": [ 37.36, 36.14, 37.04 ], "ffn.makespan_us": [ 7471.31, 7228.82, 7407.31 ] }, "32": { "attention.gemm_us_per_gemm": [ 26.01, 26.26, 26.07 ], "attention.exchange_us": [ 47.69, 47.07, 46.99 ], "attention.makespan_us": [ 9537.25, 9413.94, 9398.4 ], "ffn.gemm_us_per_gemm": [ 34.85, 34.48, 34.57 ], "ffn.exchange_us": [ 47.6, 47.08, 46.94 ], "ffn.makespan_us": [ 9520.14, 9415.74, 9388.43 ] }, "512": { "attention.gemm_us_per_gemm": [ 27.08, 27.64, 27.3 ], "attention.exchange_us": [ 125.01, 120.43, 122.98 ], "attention.makespan_us": [ 25002.61, 24085.3, 24595.78 ], "ffn.gemm_us_per_gemm": [ 47.56, 45.67, 46.69 ], "ffn.exchange_us": [ 124.97, 120.41, 122.97 ], "ffn.makespan_us": [ 24993.47, 24081.97, 24594.45 ] } } } }, "o-3a1f": { "native_source_sha256": { "arrival": [ "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" ], "counts": [ "acc5a6907f9221b71da93428f8f5754d9ab6ee4b705c78842d080935ca007d34" ], "v21_shape": [ "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" ] }, "per_repeat": { "arrival": { "1": { "attention.gemm_us_per_gemm": [ 25.96, 26.08, 26.09 ], "attention.exchange_us": [ 35.29, 34.9, 35.05 ], "attention.makespan_us": [ 7058.06, 6979.09, 7009.47 ], "ffn.gemm_us_per_gemm": [ 25.93, 26.0, 26.08 ], "ffn.exchange_us": [ 35.22, 34.82, 34.92 ], "ffn.makespan_us": [ 7044.51, 6964.45, 6984.93 ] }, "32": { "attention.gemm_us_per_gemm": [ 26.2, 26.32, 26.21 ], "attention.exchange_us": [ 45.84, 46.17, 46.03 ], "attention.makespan_us": [ 9167.74, 9233.82, 9205.5 ], "ffn.gemm_us_per_gemm": [ 30.14, 30.06, 29.78 ], "ffn.exchange_us": [ 45.84, 46.13, 45.92 ], "ffn.makespan_us": [ 9168.61, 9225.6, 9184.77 ] }, "512": { "attention.gemm_us_per_gemm": [ 26.64, 26.76, 26.75 ], "attention.exchange_us": [ 149.55, 149.99, 150.05 ], "attention.makespan_us": [ 29910.64, 29997.31, 30010.05 ], "ffn.gemm_us_per_gemm": [ 31.57, 31.57, 31.49 ], "ffn.exchange_us": [ 149.51, 149.91, 149.97 ], "ffn.makespan_us": [ 29902.43, 29981.42, 29993.18 ] } }, "counts": { "1": { "attention.gemm_us_per_gemm": [ 26.03, 26.02, 26.05 ], "attention.exchange_us": [ 34.78, 34.14, 34.38 ], "attention.makespan_us": [ 6956.4, 6828.42, 6875.22 ], "ffn.gemm_us_per_gemm": [ 26.13, 25.9, 26.14 ], "ffn.exchange_us": [ 34.61, 34.06, 34.34 ], "ffn.makespan_us": [ 6922.93, 6811.78, 6868.88 ] }, "32": { "attention.gemm_us_per_gemm": [ 26.45, 26.16, 26.22 ], "attention.exchange_us": [ 45.5, 45.22, 45.59 ], "attention.makespan_us": [ 9100.82, 9043.66, 9117.55 ], "ffn.gemm_us_per_gemm": [ 30.48, 30.36, 30.26 ], "ffn.exchange_us": [ 45.52, 45.22, 45.56 ], "ffn.makespan_us": [ 9104.16, 9044.34, 9112.54 ] }, "512": { "attention.gemm_us_per_gemm": [ 26.73, 26.65, 26.72 ], "attention.exchange_us": [ 148.42, 148.9, 148.45 ], "attention.makespan_us": [ 29683.09, 29780.11, 29689.22 ], "ffn.gemm_us_per_gemm": [ 32.43, 32.05, 31.71 ], "ffn.exchange_us": [ 148.37, 148.74, 148.57 ], "ffn.makespan_us": [ 29673.87, 29747.2, 29714.9 ] } }, "v21_shape": { "1": { "attention.gemm_us_per_gemm": [ 26.25, 26.18, 26.29 ], "attention.exchange_us": [ 37.81, 37.81, 37.37 ], "attention.makespan_us": [ 7562.35, 7562.83, 7474.54 ], "ffn.gemm_us_per_gemm": [ 30.66, 30.63, 30.11 ], "ffn.exchange_us": [ 37.75, 37.68, 37.41 ], "ffn.makespan_us": [ 7550.46, 7536.93, 7481.65 ] }, "32": { "attention.gemm_us_per_gemm": [ 26.16, 26.07, 26.14 ], "attention.exchange_us": [ 46.72, 46.58, 46.74 ], "attention.makespan_us": [ 9343.31, 9315.33, 9347.23 ], "ffn.gemm_us_per_gemm": [ 34.58, 34.32, 34.39 ], "ffn.exchange_us": [ 46.61, 46.48, 46.61 ], "ffn.makespan_us": [ 9322.13, 9296.72, 9321.74 ] }, "512": { "attention.gemm_us_per_gemm": [ 27.14, 26.68, 26.85 ], "attention.exchange_us": [ 149.38, 149.51, 149.2 ], "attention.makespan_us": [ 29875.01, 29901.84, 29839.2 ], "ffn.gemm_us_per_gemm": [ 40.26, 39.96, 40.0 ], "ffn.exchange_us": [ 149.33, 149.44, 149.16 ], "ffn.makespan_us": [ 29865.39, 29887.2, 29832.54 ] } } } } } }, "first_count_8d46de0b": { "screens": { "c-2a2f": { "arms": { "v21_shape": { "_env": { "AFDEP_SHARED_FFN_RECEIVE": "1" }, "_digest": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" }, "first_count": { "_tree": "/work/src-fin3c", "_digest": "8d46de0bb554e89f1bf8b61d4f4157b5fafa5e1409a65d6ef0916298f3728b47" }, "counts": { "_tree": "/work/src-fin3b", "_digest": "acc5a6907f9221b71da93428f8f5754d9ab6ee4b705c78842d080935ca007d34" } }, "passthrough": [ "--payload", "bf16", "--routing", "random", "--hidden", "2048", "--tokens", "1", "8", "32", "128", "512", "2048", "--capacity", "2048" ], "processes": [ { "order": [ "v21_shape", "first_count", "counts" ], "median_us": { "v21_shape": { "1": 20.22879958152771, "8": 23.5263192653656, "32": 24.757519960403442, "128": 32.29135990142822, "512": 66.44160032272339, "2048": 181.0788059234619 }, "first_count": { "1": 20.25063991546631, "8": 23.279680013656616, "32": 23.97439956665039, "128": 32.337599992752075, "512": 66.28167867660522, "2048": 181.19159698486328 }, "counts": { "1": 20.145119428634644, "8": 23.559759855270386, "32": 23.921760320663452, "128": 32.22800016403198, "512": 66.0153603553772, "2048": 180.65991401672363 } }, "native_source_sha256": { "v21_shape": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "first_count": "8d46de0bb554e89f1bf8b61d4f4157b5fafa5e1409a65d6ef0916298f3728b47", "counts": "acc5a6907f9221b71da93428f8f5754d9ab6ee4b705c78842d080935ca007d34" } }, { "order": [ "first_count", "counts", "v21_shape" ], "median_us": { "first_count": { "1": 20.277520418167114, "8": 23.525919914245605, "32": 23.81824016571045, "128": 32.5490403175354, "512": 66.1625599861145, "2048": 180.67079544067383 }, "counts": { "1": 19.893440008163452, "8": 23.28727960586548, "32": 24.113680124282837, "128": 32.44447946548462, "512": 66.19855880737305, "2048": 180.95840454101562 }, "v21_shape": { "1": 20.24232029914856, "8": 23.22775959968567, "32": 24.613999128341675, "128": 32.539520263671875, "512": 66.5613603591919, "2048": 181.33960723876953 } }, "native_source_sha256": { "first_count": "8d46de0bb554e89f1bf8b61d4f4157b5fafa5e1409a65d6ef0916298f3728b47", "counts": "acc5a6907f9221b71da93428f8f5754d9ab6ee4b705c78842d080935ca007d34", "v21_shape": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" } }, { "order": [ "counts", "v21_shape", "first_count" ], "median_us": { "counts": { "1": 19.791600108146667, "8": 23.37936043739319, "32": 24.651600122451782, "128": 32.34679937362671, "512": 66.04480028152466, "2048": 180.67631721496582 }, "v21_shape": { "1": 20.259199142456055, "8": 23.662480115890503, "32": 23.71384024620056, "128": 32.26920008659363, "512": 66.30199909210205, "2048": 180.79895973205566 }, "first_count": { "1": 19.763200283050537, "8": 23.36351990699768, "32": 24.848880767822266, "128": 32.68239974975586, "512": 66.16016149520874, "2048": 181.12360000610352 } }, "native_source_sha256": { "counts": "acc5a6907f9221b71da93428f8f5754d9ab6ee4b705c78842d080935ca007d34", "v21_shape": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "first_count": "8d46de0bb554e89f1bf8b61d4f4157b5fafa5e1409a65d6ef0916298f3728b47" } }, { "order": [ "v21_shape", "first_count", "counts" ], "median_us": { "v21_shape": { "1": 19.79696035385132, "8": 23.576319217681885, "32": 23.873839378356934, "128": 32.3088002204895, "512": 66.21111869812012, "2048": 180.70591926574707 }, "first_count": { "1": 20.196640491485596, "8": 23.479599952697754, "32": 23.9517605304718, "128": 32.343279123306274, "512": 66.20543956756592, "2048": 180.9342384338379 }, "counts": { "1": 19.78007972240448, "8": 23.232719898223877, "32": 25.51535964012146, "128": 32.54599928855896, "512": 66.11711978912354, "2048": 181.08975410461426 } }, "native_source_sha256": { "v21_shape": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "first_count": "8d46de0bb554e89f1bf8b61d4f4157b5fafa5e1409a65d6ef0916298f3728b47", "counts": "acc5a6907f9221b71da93428f8f5754d9ab6ee4b705c78842d080935ca007d34" } }, { "order": [ "first_count", "counts", "v21_shape" ], "median_us": { "first_count": { "1": 20.235038995742798, "8": 23.228639364242554, "32": 25.087920427322388, "128": 32.452160120010376, "512": 66.49352073669434, "2048": 181.8346405029297 }, "counts": { "1": 20.258159637451172, "8": 23.24512004852295, "32": 23.88232111930847, "128": 32.201600074768066, "512": 66.3260817527771, "2048": 180.71864128112793 }, "v21_shape": { "1": 20.288320779800415, "8": 23.280320167541504, "32": 25.005760192871094, "128": 32.810399532318115, "512": 66.3595986366272, "2048": 181.6699981689453 } }, "native_source_sha256": { "first_count": "8d46de0bb554e89f1bf8b61d4f4157b5fafa5e1409a65d6ef0916298f3728b47", "counts": "acc5a6907f9221b71da93428f8f5754d9ab6ee4b705c78842d080935ca007d34", "v21_shape": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" } } ], "table": [ { "tokens": 1, "v21_shape": { "median_us": 20.24232029914856, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "first_count": { "median_us": 20.235038995742798, "paired_change_pct": [ 0.10796653479399065, 0.1738936964653881, -2.4482648890403635, 2.0188964896144856, -0.2626229377774125 ], "geomean_change_pct": -0.09221019273003561, "repeats_faster": 2 }, "counts": { "median_us": 19.893440008163452, "paired_change_pct": [ -0.4136684065498386, -1.7235192696747448, -2.3080825210383904, -0.08526880463017861, -0.14866258610851668 ], "geomean_change_pct": -0.9400110817505469, "repeats_faster": 5 } }, { "tokens": 8, "v21_shape": { "median_us": 23.5263192653656, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "first_count": { "median_us": 23.36351990699768, "paired_change_pct": [ -1.048354606290136, 1.2836378527181447, -1.263435647610145, -0.4102390372776821, -0.22199352468960099 ], "geomean_change_pct": -0.33607948203464577, "repeats_faster": 4 }, "counts": { "median_us": 23.28727960586548, "paired_change_pct": [ 0.14214118888549176, 0.25624514462692094, -1.1964919869375201, -1.4573917000594072, -0.15120118093406854 ], "geomean_change_pct": -0.48386189601331564, "repeats_faster": 3 } }, { "tokens": 32, "v21_shape": { "median_us": 24.613999128341675, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "first_count": { "median_us": 23.97439956665039, "paired_change_pct": [ -3.163161718158991, -3.232952753764229, 4.786405364283253, 0.3263871842310806, 0.328565233840461 ], "geomean_change_pct": -0.2339996525706156, "repeats_faster": 2 }, "counts": { "median_us": 24.113680124282837, "paired_change_pct": [ -3.375780938788231, -2.0326603631132367, 3.9544834008969465, 6.875811786070174, -4.4927211366399655 ], "geomean_change_pct": 0.08875493498308185, "repeats_faster": 3 } }, { "tokens": 128, "v21_shape": { "median_us": 32.3088002204895, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "first_count": { "median_us": 32.452160120010376, "paired_change_pct": [ 0.1431964818607856, 0.029256896802354326, 1.280476931728769, 0.10671675389204438, -1.091847150336811 ], "geomean_change_pct": 0.09074176943828505, "repeats_faster": 1 }, "counts": { "median_us": 32.34679937362671, "paired_change_pct": [ -0.19621266366498524, -0.29207805590595326, 0.24047477726389932, 0.7341624153503368, -1.8555076019430516 ], "geomean_change_pct": -0.2776502937476222, "repeats_faster": 3 } }, { "tokens": 512, "v21_shape": { "median_us": 66.3595986366272, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "first_count": { "median_us": 66.20543956756592, "paired_change_pct": [ -0.24069505451612683, -0.5991469689401008, -0.21392657662746561, -0.008577306449231159, 0.20181270354040493 ], "geomean_change_pct": -0.1724627742002216, "repeats_faster": 4 }, "counts": { "median_us": 66.11711978912354, "paired_change_pct": [ -0.6415257388079687, -0.5450633067909383, -0.38792014433849387, -0.14196846518355066, -0.05050796650176714 ], "geomean_change_pct": -0.35365538567480037, "repeats_faster": 5 } }, { "tokens": 2048, "v21_shape": { "median_us": 181.0788059234619, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "first_count": { "median_us": 181.12360000610352, "paired_change_pct": [ 0.06228838368254053, -0.36881727510035267, 0.179558706824956, 0.12634847215771128, 0.0906271457278729 ], "geomean_change_pct": 0.017806070340409264, "repeats_faster": 1 }, "counts": { "median_us": 180.71864128112793, "paired_change_pct": [ -0.2313312729239736, -0.21021480279924676, -0.06783364089683053, 0.21240855884898657, -0.5236730871393869 ], "geomean_change_pct": -0.16441623803374883, "repeats_faster": 4 } } ] }, "c-3a1f": { "arms": { "v21_shape": { "_env": { "AFDEP_SHARED_FFN_RECEIVE": "1" }, "_digest": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" }, "first_count": { "_tree": "/work/src-fin3c", "_digest": "8d46de0bb554e89f1bf8b61d4f4157b5fafa5e1409a65d6ef0916298f3728b47" }, "counts": { "_tree": "/work/src-fin3b", "_digest": "acc5a6907f9221b71da93428f8f5754d9ab6ee4b705c78842d080935ca007d34" } }, "passthrough": [ "--payload", "bf16", "--routing", "random", "--hidden", "2048", "--tokens", "1", "8", "32", "128", "512", "2048", "--capacity", "2048", "--ffn-ranks", "1", "--attention-ranks", "3" ], "processes": [ { "order": [ "v21_shape", "first_count", "counts" ], "median_us": { "v21_shape": { "1": 18.19495975971222, "8": 20.809839963912964, "32": 23.762799501419067, "128": 35.51048040390015, "512": 96.17800235748291, "2048": 323.4022521972656 }, "first_count": { "1": 18.875679969787598, "8": 20.45904040336609, "32": 23.297600746154785, "128": 35.64311981201172, "512": 96.25239849090576, "2048": 322.3235893249512 }, "counts": { "1": 18.60743999481201, "8": 20.357120037078857, "32": 23.71824026107788, "128": 35.42975902557373, "512": 96.1892032623291, "2048": 322.80839920043945 } }, "native_source_sha256": { "v21_shape": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "first_count": "8d46de0bb554e89f1bf8b61d4f4157b5fafa5e1409a65d6ef0916298f3728b47", "counts": "acc5a6907f9221b71da93428f8f5754d9ab6ee4b705c78842d080935ca007d34" } }, { "order": [ "first_count", "counts", "v21_shape" ], "median_us": { "first_count": { "1": 18.72600018978119, "8": 20.421199798583984, "32": 24.10088062286377, "128": 35.539679527282715, "512": 96.41272068023682, "2048": 322.3094367980957 }, "counts": { "1": 18.148080110549927, "8": 20.656639337539673, "32": 24.072959423065186, "128": 35.42240023612976, "512": 96.15823745727539, "2048": 322.8958320617676 }, "v21_shape": { "1": 18.206400275230408, "8": 20.56663990020752, "32": 23.800079822540283, "128": 35.45223951339722, "512": 96.33032321929932, "2048": 323.2089614868164 } }, "native_source_sha256": { "first_count": "8d46de0bb554e89f1bf8b61d4f4157b5fafa5e1409a65d6ef0916298f3728b47", "counts": "acc5a6907f9221b71da93428f8f5754d9ab6ee4b705c78842d080935ca007d34", "v21_shape": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" } }, { "order": [ "counts", "v21_shape", "first_count" ], "median_us": { "counts": { "1": 18.55712056159973, "8": 20.56439995765686, "32": 23.43559980392456, "128": 35.393840074539185, "512": 96.33592128753662, "2048": 322.6156806945801 }, "v21_shape": { "1": 18.179120421409607, "8": 20.63807964324951, "32": 23.765119314193726, "128": 35.536561012268066, "512": 96.2289571762085, "2048": 322.9270362854004 }, "first_count": { "1": 18.741679787635803, "8": 20.921759605407715, "32": 23.86863946914673, "128": 35.59191942214966, "512": 96.51280403137207, "2048": 322.6112747192383 } }, "native_source_sha256": { "counts": "acc5a6907f9221b71da93428f8f5754d9ab6ee4b705c78842d080935ca007d34", "v21_shape": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "first_count": "8d46de0bb554e89f1bf8b61d4f4157b5fafa5e1409a65d6ef0916298f3728b47" } }, { "order": [ "v21_shape", "first_count", "counts" ], "median_us": { "v21_shape": { "1": 18.68560016155243, "8": 20.806400775909424, "32": 23.608319759368896, "128": 35.48784017562866, "512": 96.39151573181152, "2048": 322.6603317260742 }, "first_count": { "1": 18.24127972126007, "8": 20.39512038230896, "32": 24.219759702682495, "128": 35.56455969810486, "512": 96.216721534729, "2048": 322.91887283325195 }, "counts": { "1": 18.171200156211853, "8": 20.74023962020874, "32": 23.7717604637146, "128": 35.489360094070435, "512": 96.2876033782959, "2048": 322.48952865600586 } }, "native_source_sha256": { "v21_shape": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432", "first_count": "8d46de0bb554e89f1bf8b61d4f4157b5fafa5e1409a65d6ef0916298f3728b47", "counts": "acc5a6907f9221b71da93428f8f5754d9ab6ee4b705c78842d080935ca007d34" } }, { "order": [ "first_count", "counts", "v21_shape" ], "median_us": { "first_count": { "1": 18.23032021522522, "8": 20.535519123077393, "32": 23.969439268112183, "128": 35.542399883270264, "512": 96.2181568145752, "2048": 322.91568756103516 }, "counts": { "1": 18.261359930038452, "8": 20.5457603931427, "32": 23.83623957633972, "128": 35.55432081222534, "512": 96.39392375946045, "2048": 323.02160263061523 }, "v21_shape": { "1": 18.19432020187378, "8": 20.43063998222351, "32": 23.814640045166016, "128": 35.52199959754944, "512": 96.0976791381836, "2048": 322.8580093383789 } }, "native_source_sha256": { "first_count": "8d46de0bb554e89f1bf8b61d4f4157b5fafa5e1409a65d6ef0916298f3728b47", "counts": "acc5a6907f9221b71da93428f8f5754d9ab6ee4b705c78842d080935ca007d34", "v21_shape": "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" } } ], "table": [ { "tokens": 1, "v21_shape": { "median_us": 18.19495975971222, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "first_count": { "median_us": 18.72600018978119, "paired_change_pct": [ 3.7412570242812304, 2.853940958651169, 3.0945356716140626, -2.3778762065485792, 0.19786401993591163 ], "geomean_change_pct": 1.4759114877453072, "repeats_faster": 1 }, "counts": { "median_us": 18.261359930038452, "paired_change_pct": [ 2.267002733433454, -0.32032781768411844, 2.079309292351428, -2.7529220410003608, 0.36846514418147347 ], "geomean_change_pct": 0.3115033206708917, "repeats_faster": 2 } }, { "tokens": 8, "v21_shape": { "median_us": 20.63807964324951, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "first_count": { "median_us": 20.45904040336609, "paired_change_pct": [ -1.6857388675511564, -0.7071651097565468, 1.3745463098403743, -1.9767012950969498, 0.5133424158280775 ], "geomean_change_pct": -0.5045372136768145, "repeats_faster": 3 }, "counts": { "median_us": 20.56439995765686, "paired_change_pct": [ -2.1755089304828057, 0.43759913028498243, -0.3570084371524951, -0.3179846260449204, 0.5634694313019883 ], "geomean_change_pct": -0.3747218012048692, "repeats_faster": 3 } }, { "tokens": 32, "v21_shape": { "median_us": 23.765119314193726, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "first_count": { "median_us": 23.969439268112183, "paired_change_pct": [ -1.9576765575811117, 1.2638646700613432, 0.4355970343947524, 2.58993418229585, 0.6500170594750942 ], "geomean_change_pct": 0.5853826601916357, "repeats_faster": 1 }, "counts": { "median_us": 23.7717604637146, "paired_change_pct": [ -0.18751679632076357, 1.146549097984395, -1.3865678766963319, 0.6923012989132449, 0.0906985414549144 ], "geomean_change_pct": 0.06734987314387375, "repeats_faster": 2 } }, { "tokens": 128, "v21_shape": { "median_us": 35.51048040390015, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "first_count": { "median_us": 35.56455969810486, "paired_change_pct": [ 0.37352186341303995, 0.24664172161099618, 0.15577874815315695, 0.21618538095447004, 0.05743000380595831 ], "geomean_change_pct": 0.20985732439444682, "repeats_faster": 0 }, "counts": { "median_us": 35.42975902557373, "paired_change_pct": [ -0.22731705515747214, -0.08416753829100232, -0.401617189912129, 0.004282927431620287, 0.09098928844684107 ], "geomean_change_pct": -0.1237181809215171, "repeats_faster": 3 } }, { "tokens": 512, "v21_shape": { "median_us": 96.2289571762085, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "first_count": { "median_us": 96.25239849090576, "paired_change_pct": [ 0.07735254590370833, 0.08553636921773133, 0.29497031194447576, -0.18133774093650468, 0.1253700167080707 ], "geomean_change_pct": 0.08026173597546205, "repeats_faster": 1 }, "counts": { "median_us": 96.2876033782959, "paired_change_pct": [ 0.011646015275457167, -0.1786413211052662, 0.11115584587730876, -0.10780238564225275, 0.3082744806467863 ], "geomean_change_pct": 0.028779766708808552, "repeats_faster": 2 } }, { "tokens": 2048, "v21_shape": { "median_us": 322.9270362854004, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "first_count": { "median_us": 322.6112747192383, "paired_change_pct": [ -0.333535980342059, -0.2783105655804641, -0.09778108695830401, 0.0801279493499063, 0.017864888275331836 ], "geomean_change_pct": -0.12245731738541599, "repeats_faster": 3 }, "counts": { "median_us": 322.80839920043945, "paired_change_pct": [ -0.18362673506180371, -0.09688141801773575, -0.0964166996984206, -0.05293587505927899, 0.050670352757098414 ], "geomean_change_pct": -0.07586710552518472, "repeats_faster": 4 } } ] } }, "overlap": { "o-2a2f": { "native_source_sha256": { "counts": [ "acc5a6907f9221b71da93428f8f5754d9ab6ee4b705c78842d080935ca007d34" ], "first_count": [ "8d46de0bb554e89f1bf8b61d4f4157b5fafa5e1409a65d6ef0916298f3728b47" ], "v21_shape": [ "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" ] }, "per_repeat": { "counts": { "1": { "attention.gemm_us_per_gemm": [ 25.92, 25.96, 26.09 ], "attention.exchange_us": [ 34.6, 35.91, 34.68 ], "attention.makespan_us": [ 6919.94, 7181.39, 6935.66 ], "ffn.gemm_us_per_gemm": [ 25.89, 25.73, 26.04 ], "ffn.exchange_us": [ 34.49, 35.9, 34.67 ], "ffn.makespan_us": [ 6897.41, 7179.84, 6934.54 ] }, "32": { "attention.gemm_us_per_gemm": [ 26.21, 26.09, 26.32 ], "attention.exchange_us": [ 45.16, 45.11, 44.51 ], "attention.makespan_us": [ 9032.62, 9021.12, 8901.2 ], "ffn.gemm_us_per_gemm": [ 28.91, 28.7, 29.11 ], "ffn.exchange_us": [ 45.15, 45.07, 44.45 ], "ffn.makespan_us": [ 9030.98, 9013.34, 8889.65 ] }, "512": { "attention.gemm_us_per_gemm": [ 26.45, 26.73, 27.42 ], "attention.exchange_us": [ 116.26, 115.74, 111.98 ], "attention.makespan_us": [ 23252.37, 23148.4, 22396.56 ], "ffn.gemm_us_per_gemm": [ 28.97, 28.84, 30.11 ], "ffn.exchange_us": [ 116.32, 115.71, 111.99 ], "ffn.makespan_us": [ 23263.47, 23141.71, 22398.34 ] } }, "first_count": { "1": { "attention.gemm_us_per_gemm": [ 26.02, 26.26, 26.05 ], "attention.exchange_us": [ 35.3, 34.99, 35.27 ], "attention.makespan_us": [ 7059.79, 6997.57, 7054.38 ], "ffn.gemm_us_per_gemm": [ 26.7, 26.84, 26.99 ], "ffn.exchange_us": [ 35.23, 35.01, 35.21 ], "ffn.makespan_us": [ 7046.3, 7002.3, 7042.26 ] }, "32": { "attention.gemm_us_per_gemm": [ 26.32, 26.25, 26.25 ], "attention.exchange_us": [ 44.36, 44.32, 44.96 ], "attention.makespan_us": [ 8871.65, 8863.54, 8992.74 ], "ffn.gemm_us_per_gemm": [ 29.73, 29.62, 29.37 ], "ffn.exchange_us": [ 44.34, 44.35, 44.91 ], "ffn.makespan_us": [ 8868.3, 8870.61, 8981.31 ] }, "512": { "attention.gemm_us_per_gemm": [ 27.45, 27.51, 27.31 ], "attention.exchange_us": [ 112.85, 113.77, 111.8 ], "attention.makespan_us": [ 22569.12, 22753.97, 22359.54 ], "ffn.gemm_us_per_gemm": [ 32.32, 33.03, 31.89 ], "ffn.exchange_us": [ 112.83, 113.78, 111.77 ], "ffn.makespan_us": [ 22565.54, 22755.23, 22353.47 ] } }, "v21_shape": { "1": { "attention.gemm_us_per_gemm": [ 26.45, 26.34, 26.41 ], "attention.exchange_us": [ 35.63, 36.77, 35.95 ], "attention.makespan_us": [ 7126.54, 7354.26, 7189.87 ], "ffn.gemm_us_per_gemm": [ 27.6, 29.2, 28.05 ], "ffn.exchange_us": [ 35.57, 36.8, 35.86 ], "ffn.makespan_us": [ 7113.22, 7359.63, 7172.83 ] }, "32": { "attention.gemm_us_per_gemm": [ 26.21, 26.19, 26.15 ], "attention.exchange_us": [ 47.49, 47.81, 47.36 ], "attention.makespan_us": [ 9498.77, 9562.16, 9471.09 ], "ffn.gemm_us_per_gemm": [ 34.18, 34.74, 34.29 ], "ffn.exchange_us": [ 47.5, 47.81, 47.32 ], "ffn.makespan_us": [ 9499.95, 9561.81, 9464.88 ] }, "512": { "attention.gemm_us_per_gemm": [ 27.58, 27.59, 27.49 ], "attention.exchange_us": [ 120.92, 119.97, 121.09 ], "attention.makespan_us": [ 24183.62, 23994.35, 24218.7 ], "ffn.gemm_us_per_gemm": [ 45.98, 45.62, 46.14 ], "ffn.exchange_us": [ 120.89, 119.99, 121.02 ], "ffn.makespan_us": [ 24177.1, 23997.38, 24203.82 ] } } } }, "o-3a1f": { "native_source_sha256": { "counts": [ "acc5a6907f9221b71da93428f8f5754d9ab6ee4b705c78842d080935ca007d34" ], "first_count": [ "8d46de0bb554e89f1bf8b61d4f4157b5fafa5e1409a65d6ef0916298f3728b47" ], "v21_shape": [ "fa0db53d7b00e1539646f606e5ab4e03bb74728ea965b609e3a61c8d8539c432" ] }, "per_repeat": { "counts": { "1": { "attention.gemm_us_per_gemm": [ 25.89, 25.99, 26.07 ], "attention.exchange_us": [ 34.43, 34.53, 34.52 ], "attention.makespan_us": [ 6886.7, 6906.18, 6903.47 ], "ffn.gemm_us_per_gemm": [ 25.89, 26.07, 26.05 ], "ffn.exchange_us": [ 34.37, 34.49, 34.54 ], "ffn.makespan_us": [ 6874.69, 6897.49, 6907.5 ] }, "32": { "attention.gemm_us_per_gemm": [ 26.15, 26.32, 26.24 ], "attention.exchange_us": [ 45.73, 45.44, 45.63 ], "attention.makespan_us": [ 9145.07, 9088.05, 9126.13 ], "ffn.gemm_us_per_gemm": [ 29.92, 30.12, 29.9 ], "ffn.exchange_us": [ 45.71, 45.46, 45.59 ], "ffn.makespan_us": [ 9141.52, 9092.3, 9118.83 ] }, "512": { "attention.gemm_us_per_gemm": [ 26.66, 26.67, 26.7 ], "attention.exchange_us": [ 149.34, 149.42, 148.61 ], "attention.makespan_us": [ 29868.75, 29883.63, 29721.22 ], "ffn.gemm_us_per_gemm": [ 31.34, 31.7, 31.87 ], "ffn.exchange_us": [ 149.33, 149.33, 148.58 ], "ffn.makespan_us": [ 29865.36, 29866.78, 29716.48 ] } }, "first_count": { "1": { "attention.gemm_us_per_gemm": [ 26.16, 26.08, 26.24 ], "attention.exchange_us": [ 34.81, 34.8, 34.82 ], "attention.makespan_us": [ 6962.83, 6960.98, 6963.25 ], "ffn.gemm_us_per_gemm": [ 26.06, 26.02, 26.15 ], "ffn.exchange_us": [ 34.68, 34.76, 34.71 ], "ffn.makespan_us": [ 6935.33, 6952.06, 6941.55 ] }, "32": { "attention.gemm_us_per_gemm": [ 26.14, 26.17, 25.93 ], "attention.exchange_us": [ 45.64, 45.5, 46.79 ], "attention.makespan_us": [ 9127.23, 9099.76, 9357.54 ], "ffn.gemm_us_per_gemm": [ 30.04, 30.1, 28.65 ], "ffn.exchange_us": [ 45.53, 45.48, 46.85 ], "ffn.makespan_us": [ 9106.99, 9096.86, 9369.39 ] }, "512": { "attention.gemm_us_per_gemm": [ 26.75, 26.59, 26.58 ], "attention.exchange_us": [ 147.9, 147.85, 148.49 ], "attention.makespan_us": [ 29580.82, 29569.62, 29698.26 ], "ffn.gemm_us_per_gemm": [ 32.61, 32.69, 32.48 ], "ffn.exchange_us": [ 147.87, 147.75, 148.46 ], "ffn.makespan_us": [ 29574.26, 29550.29, 29692.45 ] } }, "v21_shape": { "1": { "attention.gemm_us_per_gemm": [ 26.11, 26.14, 26.12 ], "attention.exchange_us": [ 37.64, 37.89, 38.12 ], "attention.makespan_us": [ 7528.03, 7577.68, 7623.33 ], "ffn.gemm_us_per_gemm": [ 30.26, 30.56, 31.23 ], "ffn.exchange_us": [ 37.56, 37.82, 38.09 ], "ffn.makespan_us": [ 7511.57, 7564.02, 7618.61 ] }, "32": { "attention.gemm_us_per_gemm": [ 26.08, 26.12, 26.07 ], "attention.exchange_us": [ 46.72, 46.54, 47.22 ], "attention.makespan_us": [ 9344.54, 9307.06, 9443.52 ], "ffn.gemm_us_per_gemm": [ 34.46, 34.36, 35.53 ], "ffn.exchange_us": [ 46.66, 46.5, 47.16 ], "ffn.makespan_us": [ 9332.08, 9299.33, 9431.65 ] }, "512": { "attention.gemm_us_per_gemm": [ 26.87, 27.02, 26.79 ], "attention.exchange_us": [ 149.21, 149.23, 150.83 ], "attention.makespan_us": [ 29842.8, 29846.42, 30166.8 ], "ffn.gemm_us_per_gemm": [ 40.09, 40.0, 40.6 ], "ffn.exchange_us": [ 149.1, 149.19, 150.75 ], "ffn.makespan_us": [ 29820.78, 29837.52, 30150.24 ] } } } } } } }, "pre_suites_epilogue_trace_2a2f_t1": { "native_source_sha256": "development tree, one-domain kernels identical to the pre-suites tree (split = one-CTA candidate, shared = the pre-suites shape)", "trace": [ "## split", "anchor device 2 dispatch start = 0; period median 22.18 us", " dev0 0:dispatch_impl start -5.47 end 5.52 dur 11.04", " dev0 1:dispatch_copy_epilogue start -4.32 end 7.59 dur 11.94", " dev0 2:combine_impl start 7.78 end 14.94 dur 7.14", " dev1 0:dispatch_impl start -2.67 end 5.49 dur 8.16", " dev1 1:dispatch_copy_epilogue start -1.49 end 7.61 dur 9.12", " dev1 2:combine_impl start 7.80 end 17.68 dur 9.86", " dev2 0:dispatch_impl start 0.00 end 5.66 dur 5.66", " dev2 1:combine_impl start 6.08 end 17.60 dur 11.49", " dev2 2:combine_reduce_epilogue start 17.57 end 20.10 dur 2.53", " dev3 0:dispatch_impl start -0.05 end 5.55 dur 5.60", " dev3 1:combine_impl start 5.96 end 17.58 dur 11.58", " dev3 2:combine_reduce_epilogue start 17.58 end 20.11 dur 2.53", "dev0: SM-us reserved per exchange (upper bound) 1757.9", " 0:dispatch_impl grid 1 x 160 thr smem 227.0 KiB coop 0 dur 11.04 us SM-us 11.0", " 2:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 7.14 us SM-us 171.3", "dev1: SM-us reserved per exchange (upper bound) 1449.3", " 0:dispatch_impl grid 1 x 160 thr smem 227.0 KiB coop 0 dur 8.16 us SM-us 8.2", " 2:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 9.86 us SM-us 236.5", "dev2: SM-us reserved per exchange (upper bound) 480.3", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 5.66 us SM-us 135.9", " 1:combine_impl grid 1 x 32 thr smem 227.0 KiB coop 0 dur 11.49 us SM-us 11.5", "dev3: SM-us reserved per exchange (upper bound) 478.6", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 5.60 us SM-us 134.4", " 1:combine_impl grid 1 x 32 thr smem 227.0 KiB coop 0 dur 11.58 us SM-us 11.6", "## shared", "anchor device 2 dispatch start = 0; period median 21.73 us", " dev0 0:dispatch_impl start -4.98 end 5.52 dur 10.50", " dev0 1:dispatch_copy_epilogue start -3.73 end 7.57 dur 11.30", " dev0 2:combine_impl start 7.74 end 14.77 dur 7.04", " dev1 0:dispatch_impl start -2.48 end 5.52 dur 8.00", " dev1 1:dispatch_copy_epilogue start -1.24 end 7.56 dur 8.80", " dev1 2:combine_impl start 7.72 end 17.36 dur 9.63", " dev2 0:dispatch_impl start 0.00 end 5.70 dur 5.70", " dev2 1:combine_impl start 6.21 end 17.25 dur 11.13", " dev2 2:combine_reduce_epilogue start 17.22 end 19.74 dur 2.53", " dev3 0:dispatch_impl start -0.04 end 5.71 dur 5.76", " dev3 1:combine_impl start 6.20 end 17.27 dur 11.04", " dev3 2:combine_reduce_epilogue start 17.25 end 19.74 dur 2.50", "dev0: SM-us reserved per exchange (upper bound) 1911.6", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 10.50 us SM-us 251.9", " 2:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 7.04 us SM-us 169.0", "dev1: SM-us reserved per exchange (upper bound) 1585.2", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 8.00 us SM-us 192.0", " 2:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 9.63 us SM-us 231.2", "dev2: SM-us reserved per exchange (upper bound) 734.6", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 5.70 us SM-us 136.7", " 1:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 11.13 us SM-us 267.2", "dev3: SM-us reserved per exchange (upper bound) 735.3", " 0:dispatch_impl grid 24 x 832 thr smem 227.0 KiB coop 0 dur 5.76 us SM-us 138.2", " 1:combine_impl grid 24 x 1024 thr smem 227.0 KiB coop 0 dur 11.04 us SM-us 265.0" ] }, "pre_suites_starvation_under_load_statler": { "native_source_sha256": { "afd": [ "c7c4e90e97048f899e3c099ca0276c6e877c99baf5bc7cb982ed25ae937e3915" ], "afd_shared": [ "c7c4e90e97048f899e3c099ca0276c6e877c99baf5bc7cb982ed25ae937e3915" ], "masked": [ "c7c4e90e97048f899e3c099ca0276c6e877c99baf5bc7cb982ed25ae937e3915" ] }, "per_repeat": { "afd": { "1": { "attention.gemm_us_per_gemm": [ 26.81, 26.63, 26.83 ], "attention.exchange_us": [ 38.64, 38.69, 39.22 ], "attention.makespan_us": [ 7727.49, 7738.34, 7843.38 ], "ffn.gemm_us_per_gemm": [ 30.66, 30.66, 31.46 ], "ffn.exchange_us": [ 38.62, 38.68, 39.17 ], "ffn.makespan_us": [ 7723.87, 7735.23, 7834.74 ] }, "32": { "attention.gemm_us_per_gemm": [ 26.47, 26.26, 26.46 ], "attention.exchange_us": [ 48.11, 47.85, 48.72 ], "attention.makespan_us": [ 9621.17, 9570.99, 9744.27 ], "ffn.gemm_us_per_gemm": [ 34.86, 35.37, 35.8 ], "ffn.exchange_us": [ 48.07, 47.88, 48.75 ], "ffn.makespan_us": [ 9613.63, 9576.34, 9749.78 ] }, "512": { "attention.gemm_us_per_gemm": [ 27.86, 27.4, 27.65 ], "attention.exchange_us": [ 127.31, 126.57, 126.99 ], "attention.makespan_us": [ 25462.48, 25314.42, 25398.24 ], "ffn.gemm_us_per_gemm": [ 47.93, 47.24, 47.4 ], "ffn.exchange_us": [ 127.28, 126.59, 126.99 ], "ffn.makespan_us": [ 25455.81, 25317.97, 25398.58 ] } }, "afd_shared": { "1": { "attention.gemm_us_per_gemm": [ 26.78, 26.7, 26.78 ], "attention.exchange_us": [ 39.46, 38.94, 38.82 ], "attention.makespan_us": [ 7891.54, 7788.85, 7764.91 ], "ffn.gemm_us_per_gemm": [ 31.45, 30.99, 30.78 ], "ffn.exchange_us": [ 39.5, 38.9, 38.82 ], "ffn.makespan_us": [ 7900.02, 7779.63, 7764.42 ] }, "32": { "attention.gemm_us_per_gemm": [ 26.59, 26.55, 26.55 ], "attention.exchange_us": [ 49.81, 48.17, 48.37 ], "attention.makespan_us": [ 9961.81, 9633.38, 9674.32 ], "ffn.gemm_us_per_gemm": [ 36.6, 35.21, 35.49 ], "ffn.exchange_us": [ 49.79, 48.15, 48.36 ], "ffn.makespan_us": [ 9957.79, 9630.5, 9672.9 ] }, "512": { "attention.gemm_us_per_gemm": [ 27.66, 27.84, 27.78 ], "attention.exchange_us": [ 127.46, 126.8, 126.56 ], "attention.makespan_us": [ 25491.2, 25360.72, 25312.03 ], "ffn.gemm_us_per_gemm": [ 47.93, 47.52, 47.7 ], "ffn.exchange_us": [ 127.42, 126.77, 126.51 ], "ffn.makespan_us": [ 25483.33, 25353.07, 25301.44 ] } }, "masked": { "1": { "attention.gemm_us_per_gemm": [ 30.79, 31.15, 31.05 ], "attention.exchange_us": [ 47.56, 47.69, 47.2 ], "attention.makespan_us": [ 9511.82, 9538.18, 9439.74 ], "ffn.gemm_us_per_gemm": [ 31.01, 31.19, 30.98 ], "ffn.exchange_us": [ 47.52, 47.69, 47.14 ], "ffn.makespan_us": [ 9503.87, 9538.51, 9427.15 ] }, "32": { "attention.gemm_us_per_gemm": [ 27.64, 27.62, 27.98 ], "attention.exchange_us": [ 59.11, 58.78, 57.71 ], "attention.makespan_us": [ 11822.4, 11756.56, 11541.74 ], "ffn.gemm_us_per_gemm": [ 36.06, 35.76, 35.94 ], "ffn.exchange_us": [ 59.1, 58.75, 57.62 ], "ffn.makespan_us": [ 11820.62, 11750.8, 11524.74 ] }, "512": { "attention.gemm_us_per_gemm": [ 29.43, 29.03, 29.26 ], "attention.exchange_us": [ 122.43, 121.5, 120.79 ], "attention.makespan_us": [ 24485.94, 24299.42, 24158.4 ], "ffn.gemm_us_per_gemm": [ 35.25, 35.06, 35.3 ], "ffn.exchange_us": [ 122.49, 121.48, 120.75 ], "ffn.makespan_us": [ 24498.16, 24296.35, 24149.73 ] } } } } }, "r10_one_cta_receive": { "native_source_sha256": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "correctness_pokprod": { "result": [ "jit-split exit=0", "jit-control exit=0", "jit-generic exit=0", "bf16-direct-small exit=0 bf16-hybrid-small exit=0 bf16-generic-control exit=0 bf16-off-large exit=0 bf16-on-large exit=0 fp8-direct-small exit=0 fp8-hybrid-small exit=0 fp8-generic-control exit=0 fp8-off-large exit=0 fp8-on-large exit=0 invalid-eager-dummy exit=1 assertion=yes invalid-eager-duplicate exit=1 assertion=yes invalid-eager-negative exit=1 assertion=yes invalid-eager-wide exit=1 assertion=yes invalid-graph-dummy exit=1 assertion=yes invalid-graph-duplicate exit=1 assertion=yes invalid-graph-negative exit=1 assertion=yes invalid-graph-wide exit=1 assertion=yes done", "f1-small exit=0", "f1-large exit=0", "f3-small exit=0", "f3-large exit=0", "view-n4-bf16-small exit=0", "view-n4-bf16-large exit=0", "view-n4-fp8-small exit=0", "view-n4-fp8-large exit=0", "view-n3-bf16-small exit=0", "view-n3-bf16-large exit=0", "view-n3-fp8-small exit=0", "view-n3-fp8-large exit=0", "sms7 exit=0", "sms23 exit=0", "ctl-AFDEP_SHARED_RECEIVE_GRID exit=0", "ctl-AFDEP_SHARED_RECEIVE_GRID-large exit=0", "ctl-AFDEP_SHARED_ROLE_KERNELS exit=0", "ctl-AFDEP_SHARED_ROLE_KERNELS-large exit=0", "ctl-AFDEP_KEEP_ENTRY_BARRIERS exit=0", "ctl-AFDEP_KEEP_ENTRY_BARRIERS-large exit=0", "ctl-AFDEP_UNION_NOTIFY exit=0", "ctl-AFDEP_UNION_NOTIFY-large exit=0", "ctl-AFDEP_ALL_RANK_EXIT exit=0", "ctl-AFDEP_ALL_RANK_EXIT-large exit=0", "ctl-AFDEP_GRID_SYNC_EXIT exit=0", "ctl-AFDEP_GRID_SYNC_EXIT-large exit=0", "ctl-AFDEP_UNION_BUFFERS exit=0", "ctl-AFDEP_UNION_BUFFERS-large exit=0", "ctl-AFDEP_UNION_REDUCE_PREFIX exit=0", "mixed-ffn-f2 exit=0", "mixed-ffn-f2-large exit=0", "mixed-ffn-f1 exit=0", "mixed-ffn-f1-large exit=0", "mixed-ffn-f3 exit=0", "mixed-ffn-f3-large exit=0", "mixed-attention-f2 exit=0", "mixed-attention-f2-large exit=0", "mixed-attention-f1 exit=0", "mixed-attention-f1-large exit=0", "mixed-attention-f3 exit=0", "mixed-attention-f3-large exit=0" ], "logs": [ { "log": "ctl-AFDEP_ALL_RANK_EXIT-large.log", "log_sha256": "08bf9b987f8fd5e9069c0b466ceb9ba4fd688b760027b9d6ace8121a5a7fb029" }, { "log": "ctl-AFDEP_ALL_RANK_EXIT.log", "log_sha256": "c41fb44683b7042b3076f8960ff03046f2ac668a12bac504d5cd2ab4df36b924" }, { "log": "ctl-AFDEP_GRID_SYNC_EXIT-large.log", "log_sha256": "08bf9b987f8fd5e9069c0b466ceb9ba4fd688b760027b9d6ace8121a5a7fb029" }, { "log": "ctl-AFDEP_GRID_SYNC_EXIT.log", "log_sha256": "c41fb44683b7042b3076f8960ff03046f2ac668a12bac504d5cd2ab4df36b924" }, { "log": "ctl-AFDEP_KEEP_ENTRY_BARRIERS-large.log", "log_sha256": "08bf9b987f8fd5e9069c0b466ceb9ba4fd688b760027b9d6ace8121a5a7fb029" }, { "log": "ctl-AFDEP_KEEP_ENTRY_BARRIERS.log", "log_sha256": "c41fb44683b7042b3076f8960ff03046f2ac668a12bac504d5cd2ab4df36b924" }, { "log": "ctl-AFDEP_SHARED_RECEIVE_GRID-large.log", "log_sha256": "08bf9b987f8fd5e9069c0b466ceb9ba4fd688b760027b9d6ace8121a5a7fb029" }, { "log": "ctl-AFDEP_SHARED_RECEIVE_GRID.log", "log_sha256": "c41fb44683b7042b3076f8960ff03046f2ac668a12bac504d5cd2ab4df36b924" }, { "log": "ctl-AFDEP_SHARED_ROLE_KERNELS-large.log", "log_sha256": "08bf9b987f8fd5e9069c0b466ceb9ba4fd688b760027b9d6ace8121a5a7fb029" }, { "log": "ctl-AFDEP_SHARED_ROLE_KERNELS.log", "log_sha256": "c41fb44683b7042b3076f8960ff03046f2ac668a12bac504d5cd2ab4df36b924" }, { "log": "ctl-AFDEP_UNION_BUFFERS-large.log", "log_sha256": "44829fa5ba9c42a2ba250f3d366c8aeabf1f75c025632afa1a00e2fdf92cb846" }, { "log": "ctl-AFDEP_UNION_BUFFERS.log", "log_sha256": "96e5277ac3c370fb346e890598002becd29b851e7b22eeebe838da1b2442818f" }, { "log": "ctl-AFDEP_UNION_NOTIFY-large.log", "log_sha256": "08bf9b987f8fd5e9069c0b466ceb9ba4fd688b760027b9d6ace8121a5a7fb029" }, { "log": "ctl-AFDEP_UNION_NOTIFY.log", "log_sha256": "c41fb44683b7042b3076f8960ff03046f2ac668a12bac504d5cd2ab4df36b924" }, { "log": "ctl-AFDEP_UNION_REDUCE_PREFIX.log", "log_sha256": "c8681009edbe0ccefe83fb76c819ad9d4175e9e8858032c345f15107b0a876e7" }, { "log": "f1-large.log", "log_sha256": "cf3f60bc462b20f4e4f28f344ce96457c71444da37b8d2aca92f113448475c7b" }, { "log": "f1-small.log", "log_sha256": "c8681009edbe0ccefe83fb76c819ad9d4175e9e8858032c345f15107b0a876e7" }, { "log": "f3-large.log", "log_sha256": "d12f554c5f6f0262a23486a0cca949c738b5287eb4eb237bc457657ef8336ada" }, { "log": "f3-small.log", "log_sha256": "1668b97d179f92febe712e2e708b12088a309ee82c5153db35015324f89acf7c" }, { "log": "jit-control.log", "log_sha256": "3391e6bcc6fdce82911f78f2a07a731997b0f56bf35938c7b4df00513d3e5b82" }, { "log": "jit-generic.log", "log_sha256": "32e7c537248c5f4e22891cb8159026e74e529b0bc66596b5a1138e60d405972a" }, { "log": "jit-split.log", "log_sha256": "30b81f554de08a60016fa964075716265fa0f7645bd8af85cfa7fa51d41c7b29" }, { "log": "mixed-attention-f1-large.log", "log_sha256": "cf3f60bc462b20f4e4f28f344ce96457c71444da37b8d2aca92f113448475c7b" }, { "log": "mixed-attention-f1.log", "log_sha256": "c8681009edbe0ccefe83fb76c819ad9d4175e9e8858032c345f15107b0a876e7" }, { "log": "mixed-attention-f2-large.log", "log_sha256": "34de5a39da2d2b8b4cf6597706c393b0f3c883a530938f09082fba823cfb46d7" }, { "log": "mixed-attention-f2.log", "log_sha256": "c41fb44683b7042b3076f8960ff03046f2ac668a12bac504d5cd2ab4df36b924" }, { "log": "mixed-attention-f3-large.log", "log_sha256": "d12f554c5f6f0262a23486a0cca949c738b5287eb4eb237bc457657ef8336ada" }, { "log": "mixed-attention-f3.log", "log_sha256": "1668b97d179f92febe712e2e708b12088a309ee82c5153db35015324f89acf7c" }, { "log": "mixed-ffn-f1-large.log", "log_sha256": "cf3f60bc462b20f4e4f28f344ce96457c71444da37b8d2aca92f113448475c7b" }, { "log": "mixed-ffn-f1.log", "log_sha256": "c8681009edbe0ccefe83fb76c819ad9d4175e9e8858032c345f15107b0a876e7" }, { "log": "mixed-ffn-f2-large.log", "log_sha256": "34de5a39da2d2b8b4cf6597706c393b0f3c883a530938f09082fba823cfb46d7" }, { "log": "mixed-ffn-f2.log", "log_sha256": "c41fb44683b7042b3076f8960ff03046f2ac668a12bac504d5cd2ab4df36b924" }, { "log": "mixed-ffn-f3-large.log", "log_sha256": "d12f554c5f6f0262a23486a0cca949c738b5287eb4eb237bc457657ef8336ada" }, { "log": "mixed-ffn-f3.log", "log_sha256": "1668b97d179f92febe712e2e708b12088a309ee82c5153db35015324f89acf7c" }, { "log": "sms23.log", "log_sha256": "7c597a382b73dd3a4f8e26898f5dd37db323803367b58aa3d81879523a2bb799" }, { "log": "sms7.log", "log_sha256": "55abd24f25bfa2995a49d372b31dfdbf1c0e6799bb8f188fa9dcd9b918c5d274" }, { "log": "view-n3-bf16-large.log", "log_sha256": "b9655d9cebe2e73a6c3a55cb89fd607169508a5fe4f73644f6ede39291280e35" }, { "log": "view-n3-bf16-small.log", "log_sha256": "29c078be938bf1b766f2842088939435d8a25f1899f562be817b9a95716a66b1" }, { "log": "view-n3-fp8-large.log", "log_sha256": "f8ea1f04a19a43b09a10f83d94e5bfb7bbcd4a0d8ec3a9c573f516a8e8fe1efc" }, { "log": "view-n3-fp8-small.log", "log_sha256": "8966a01dd421c8f2dbbf67595f92c205a5fd8c695440d3e2f23fea91ae94cd5d" }, { "log": "view-n4-bf16-large.log", "log_sha256": "cf3f60bc462b20f4e4f28f344ce96457c71444da37b8d2aca92f113448475c7b" }, { "log": "view-n4-bf16-small.log", "log_sha256": "c8681009edbe0ccefe83fb76c819ad9d4175e9e8858032c345f15107b0a876e7" }, { "log": "view-n4-fp8-large.log", "log_sha256": "c2ecbe40a08bcb2ae365ad6f579faec199dc4c686d7367cba45b8a3fc3104426" }, { "log": "view-n4-fp8-small.log", "log_sha256": "89719f31a5ed4c1e700388d7173098e6771fca84a48418f462319cb6976d8ff4" }, { "log": "bf16-direct-small.log", "log_sha256": "c41fb44683b7042b3076f8960ff03046f2ac668a12bac504d5cd2ab4df36b924" }, { "log": "bf16-generic-control.log", "log_sha256": "fa2e1fd2168f0b2a01cdbac2b0ddf22bf4f754f5a4d1b99e92b96efaa8eb95aa" }, { "log": "bf16-hybrid-small.log", "log_sha256": "4dca56574a4f30a5902525f27f4ba47410ad59ab3da0ee9123e418fadefec108" }, { "log": "bf16-off-large.log", "log_sha256": "08bf9b987f8fd5e9069c0b466ceb9ba4fd688b760027b9d6ace8121a5a7fb029" }, { "log": "bf16-on-large.log", "log_sha256": "2c1e9a2bec82923411743d77b4c0c4c57aff0b75010eaf4d4da6bc6b0cfd26e4" }, { "log": "fp8-direct-small.log", "log_sha256": "aab1b759ccece5d6d1b3b8ea2fb47c83a2fca4ceafeaedea75b9acf3e369c826" }, { "log": "fp8-generic-control.log", "log_sha256": "0ec3d7596a0e1bd27db48a41dd032678bf165c1287a8e160590ac626f4b66296" }, { "log": "fp8-hybrid-small.log", "log_sha256": "82f8cfc6dcb36473e80f81fe99fee9ea39ad4727e12440ea099bb7fbf4e391f2" }, { "log": "fp8-off-large.log", "log_sha256": "e140990c819d2e8c6ecfdad712b237c628823f834399ad5c5dc548bd06e1d44c" }, { "log": "fp8-on-large.log", "log_sha256": "e9929119f584cfb1f2b750396650886957e0b8672bf379b05fd3f808f3c93f40" }, { "log": "invalid-eager-dummy.log", "log_sha256": "8c4ae7df81309d67a13c66568fe746a015e71e6600eb282b9a2cd04acc2c5bf6" }, { "log": "invalid-eager-duplicate.log", "log_sha256": "25e4e0ab392796cda1a0f22f8e0b964f3d90de4c1fe41bdf8ab4347b46e2fd98" }, { "log": "invalid-eager-negative.log", "log_sha256": "fb7f11794c60e3e1aa82cad10802b92ece855f0b7ee3374837df9abe80c8fa7d" }, { "log": "invalid-eager-wide.log", "log_sha256": "56b969dd8dfb986255174c2c4a91a231ab63be964f9f95807d8664bc00428161" }, { "log": "invalid-graph-dummy.log", "log_sha256": "b230d59367a126e6cdb364fd229027f06645eb5f6302a20872f6c771d75aa0ae" }, { "log": "invalid-graph-duplicate.log", "log_sha256": "148dbac7c5ab57cbb4e81e6cd32b9680e29af492bcd8727a3852272521803026" }, { "log": "invalid-graph-negative.log", "log_sha256": "a87daae9c53b8a1f669c38c68f26515742b1c871f15a4fd6c2a4e416ecf02a24" }, { "log": "invalid-graph-wide.log", "log_sha256": "2db309179702c0c82f3e0bb0c19fa2dd28e23466f0e33d27add86ba1d217e683" } ] }, "correctness_piggy_8gpu": { "result": [ "f2-bf16-small exit=0", "f2-bf16-large exit=0", "f2-fp8-small exit=0", "f2-fp8-large exit=0", "f1-bf16-small exit=0", "f1-bf16-large exit=0", "f1-fp8-small exit=0", "f1-fp8-large exit=0", "f4-bf16-small exit=0", "f4-bf16-large exit=0", "f4-fp8-small exit=0", "f4-fp8-large exit=0", "f1-view-small exit=0", "f1-view-large exit=0", "ctl-AFDEP_SHARED_RECEIVE_GRID-f2 exit=0", "ctl-AFDEP_GRID_SYNC_EXIT-f2 exit=0", "mixed-ffn-f2 exit=0", "mixed-attention-f2 exit=0", "inv-eager exit=1 assertion=yes", "inv-graph exit=1 assertion=yes" ], "logs": [ { "log": "ctl-AFDEP_GRID_SYNC_EXIT-f2.log", "log_sha256": "2e352c42466205c4a2ae443d96d53cb1a8696f4e368cff7c9585e8d1f15c5ce9" }, { "log": "ctl-AFDEP_SHARED_RECEIVE_GRID-f2.log", "log_sha256": "2e352c42466205c4a2ae443d96d53cb1a8696f4e368cff7c9585e8d1f15c5ce9" }, { "log": "f1-bf16-large.log", "log_sha256": "62042c0b1df589266cc704ae9e50c1a94b772bcb1568a370536a52d63b44c999" }, { "log": "f1-bf16-small.log", "log_sha256": "afa146b2b37ff52cf58c0cd3f86917f49af287210b2cbc117e9aa868aa1f7293" }, { "log": "f1-fp8-large.log", "log_sha256": "f9416a7642cfd00f5b33b63ac5882aed1d8e782c16d688f52de3b5aa9f8a7797" }, { "log": "f1-fp8-small.log", "log_sha256": "cf7275426203a9c5182d5677c0356ace98e67b0550daa8784a3cb1692a168e60" }, { "log": "f1-view-large.log", "log_sha256": "62042c0b1df589266cc704ae9e50c1a94b772bcb1568a370536a52d63b44c999" }, { "log": "f1-view-small.log", "log_sha256": "afa146b2b37ff52cf58c0cd3f86917f49af287210b2cbc117e9aa868aa1f7293" }, { "log": "f2-bf16-large.log", "log_sha256": "2e352c42466205c4a2ae443d96d53cb1a8696f4e368cff7c9585e8d1f15c5ce9" }, { "log": "f2-bf16-small.log", "log_sha256": "c1c841f11d689e834fbe7137d477f66b07e14c879e5df6a0c74c9428ddc3d38f" }, { "log": "f2-fp8-large.log", "log_sha256": "4975bf74ab93ae4222127e3f7d5241ec725b6af95ed8a266fe9fd217668589ac" }, { "log": "f2-fp8-small.log", "log_sha256": "26f02f20b1d535914cebb5c1deaa7822d6976a478196d4affb6329329bf0062b" }, { "log": "f4-bf16-large.log", "log_sha256": "c33217b4dd1605f436914c444e5bb937f60f1432e356a3ce0e1297b432ebbdd3" }, { "log": "f4-bf16-small.log", "log_sha256": "8facdfa23a30dd164aad4caf043ef332ef218b95b40208165e9c2be12cba709c" }, { "log": "f4-fp8-large.log", "log_sha256": "049dbbddbdac6e54713327c5f9fcbeeb3bf46e6588da814df2fbe0ba20d18165" }, { "log": "f4-fp8-small.log", "log_sha256": "0124b6b40fafd034df5c2ef70996a506af46e0c1dba45b658dc578b01841abd0" }, { "log": "inv-eager.log", "log_sha256": "8ad89c5c2e982b9566c64ec19c8a21d16ad5a4ebb33a14c0a1eb6825480bbfd3" }, { "log": "inv-graph.log", "log_sha256": "a0b90c8dd8607edd0cbae2a35b93322daf7b6495bf4a1ef86127011b71647ecc" }, { "log": "mixed-attention-f2.log", "log_sha256": "2e352c42466205c4a2ae443d96d53cb1a8696f4e368cff7c9585e8d1f15c5ce9" }, { "log": "mixed-ffn-f2.log", "log_sha256": "2e352c42466205c4a2ae443d96d53cb1a8696f4e368cff7c9585e8d1f15c5ce9" } ] }, "faults": { "script_sha256": "63e419ddaed769dc6f56f8df5516e83394821dfa2e45e6b8308a0d7210d4c2e6", "script": "\"\"\"p1-faults.py : apply one scratch fault to a copied tree (never shipped).\"\"\"\nimport sys\nfrom pathlib import Path\n\ntree, fault = Path(sys.argv[1]), sys.argv[2]\nimpls = tree / \"afdep/third_party/deepep_v2/deep_ep/include/deep_ep/impls\"\n\n\ndef patch(name, old, new):\n path = impls / name\n text = path.read_text()\n assert text.count(old) == 1, (name, old[:60])\n path.write_text(text.replace(old, new))\n\n\nSPIN = \"\"\"// FAULT INJECTION (scratch only)\n__device__ __forceinline__ void afd_fault_spin(const unsigned long long ns) {\n unsigned long long t0, t;\n asm volatile(\"mov.u64 %0, %%globaltimer;\" : \"=l\"(t0));\n do { asm volatile(\"mov.u64 %0, %%globaltimer;\" : \"=l\"(t)); } while (t - t0 < ns);\n}\n\nconstexpr int kAfdepDispatchPhase = 0;\"\"\"\npatch(\"afdep_roles.cuh\", \"constexpr int kAfdepDispatchPhase = 0;\", SPIN)\nLATE_CTA = \"\"\" const int sm_idx, const int thread_idx) {\n // FAULT: the last sender CTA releases its arrival 50 us late.\n if (blockIdx.x == gridDim.x - 1 and threadIdx.x == 0)\n afd_fault_spin(50000);\n // Every CTA's TMA stores complete before the arrival release.\"\"\"\nif fault in (\"F1\", \"F2\"):\n patch(\"afdep_roles.cuh\", \"\"\" const int sm_idx, const int thread_idx) {\n // Every CTA's TMA stores complete before the arrival release.\"\"\", LATE_CTA)\nif fault == \"F2\":\n # Receiver counts one arrival per sender per round: must release early and read stale rows.\n patch(\"afdep_roles.cuh\", \"* kNumSenderCtas;\", \"* 1;\")\nif fault == \"F3\":\n # Receiver expects one arrival more than senders release: must time out.\n patch(\"afdep_roles.cuh\", \"* kNumSenderCtas;\", \"* (kNumSenderCtas + 1);\")\nif fault == \"F5\":\n # The FFN receive publishes its expert prefix sum 100 us late.\n patch(\"dispatch.cuh\", \"\"\" } else if (warp_idx == 1) {\n // Exclusive prefix sum for later expanding\"\"\", \"\"\" } else if (warp_idx == 1) {\n // FAULT: late expert prefix sum.\n if (lane_idx == 0)\n afd_fault_spin(100000);\n __syncwarp();\n // Exclusive prefix sum for later expanding\"\"\")\nif fault == \"F6\":\n # The receiver's waiting thread starts its wait 100 us late.\n patch(\"afdep_roles.cuh\", \"\"\" if (sm_idx != 0 or thread_idx != 0)\n return;\n\"\"\", \"\"\" if (sm_idx != 0 or thread_idx != 0)\n return;\n afd_fault_spin(100000); // FAULT: late receiver\n\"\"\")\nprint(\"applied\", fault)\n", "result": [ "F1-small exit=0 mismatch_lines=0 timeout_lines=0", "F1-large exit=0 mismatch_lines=0 timeout_lines=0", "F1-f1-small exit=0 mismatch_lines=0 timeout_lines=0", "F1-f3-small exit=0 mismatch_lines=0 timeout_lines=0", "F2-small exit=1 mismatch_lines=4 timeout_lines=0", "F3-small exit=1 mismatch_lines=0 timeout_lines=4", "F5-small exit=0 mismatch_lines=0 timeout_lines=0", "F5-large exit=0 mismatch_lines=0 timeout_lines=0", "F5-f1-small exit=0 mismatch_lines=0 timeout_lines=0", "F5-f3-small exit=0 mismatch_lines=0 timeout_lines=0", "F6-small exit=0 mismatch_lines=0 timeout_lines=0", "F6-large exit=0 mismatch_lines=0 timeout_lines=0", "F6-f1-small exit=0 mismatch_lines=0 timeout_lines=0", "F6-f3-small exit=0 mismatch_lines=0 timeout_lines=0" ] }, "screens": { "pokprod_2a2f": { "arms": { "afd": {}, "afd_shared": { "_env": { "AFDEP_SHARED_RECEIVE_GRID": "1" } }, "head": { "_tree": "/work/src-final", "_digest": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879" }, "masked": { "role_aware": false } }, "passthrough": [ "--payload", "bf16", "--routing", "random", "--hidden", "2048", "--tokens", "1", "8", "32", "128", "512", "2048", "--capacity", "2048" ], "processes": [ { "order": [ "afd", "afd_shared", "head", "masked" ], "median_us": { "afd": { "1": 20.850640535354614, "8": 24.655040502548218, "32": 26.144559383392334, "128": 35.074719190597534, "512": 70.76527833938599, "2048": 192.69231796264648 }, "afd_shared": { "1": 20.626718997955322, "8": 24.55072045326233, "32": 26.203839778900146, "128": 35.37608027458191, "512": 70.38848161697388, "2048": 192.41616249084473 }, "head": { "1": 20.589120388031006, "8": 24.416799545288086, "32": 25.976558923721313, "128": 35.3327202796936, "512": 70.37607908248901, "2048": 192.32895851135254 }, "masked": { "1": 31.46239995956421, "8": 35.32920002937317, "32": 35.525439977645874, "128": 43.82200002670288, "512": 78.69312047958374, "2048": 201.39880180358887 } }, "native_source_sha256": { "afd": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "afd_shared": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d" } }, { "order": [ "afd_shared", "head", "masked", "afd" ], "median_us": { "afd_shared": { "1": 20.609359741210938, "8": 24.966639280319214, "32": 26.185200214385986, "128": 35.34416079521179, "512": 70.5517578125, "2048": 192.84791946411133 }, "head": { "1": 20.685120820999146, "8": 24.71648097038269, "32": 25.95088005065918, "128": 35.35383939743042, "512": 70.4259204864502, "2048": 192.62520790100098 }, "masked": { "1": 31.70408010482788, "8": 34.53552007675171, "32": 35.604000091552734, "128": 44.17703866958618, "512": 79.05695915222168, "2048": 200.59231758117676 }, "afd": { "1": 20.81968069076538, "8": 24.73167896270752, "32": 26.02039933204651, "128": 35.198320150375366, "512": 70.44856071472168, "2048": 192.4955177307129 } }, "native_source_sha256": { "afd_shared": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "afd": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d" } }, { "order": [ "head", "masked", "afd", "afd_shared" ], "median_us": { "head": { "1": 20.702239274978638, "8": 25.326720476150513, "32": 26.369600296020508, "128": 35.45159935951233, "512": 70.86375951766968, "2048": 192.67640113830566 }, "masked": { "1": 31.4682400226593, "8": 35.892640352249146, "32": 35.65079927444458, "128": 43.65544080734253, "512": 78.97783994674683, "2048": 201.1725616455078 }, "afd": { "1": 20.947439670562744, "8": 24.62048053741455, "32": 26.6211998462677, "128": 35.39135932922363, "512": 70.62095880508423, "2048": 193.00600051879883 }, "afd_shared": { "1": 20.53712010383606, "8": 25.17807960510254, "32": 26.25815987586975, "128": 35.24671912193298, "512": 70.49576044082642, "2048": 192.8061580657959 } }, "native_source_sha256": { "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "afd": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "afd_shared": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d" } } ], "table": [ { "tokens": 1, "afd": { "median_us": 20.850640535354614, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 20.609359741210938, "paired_change_pct": [ -1.0739312157801995, -1.010202570723051, -1.958805339362324 ], "geomean_change_pct": -1.3485982928103102, "repeats_faster": 3 }, "head": { "median_us": 20.685120820999146, "paired_change_pct": [ -1.2542547404247428, -0.6463109197727523, -1.1705506708234337 ], "geomean_change_pct": -1.0240706586469361, "repeats_faster": 3 }, "masked": { "median_us": 31.4682400226593, "paired_change_pct": [ 50.894165127522875, 52.27937726677192, 50.224755471578455 ], "geomean_change_pct": 51.13034768573894, "repeats_faster": 0 } }, { "tokens": 8, "afd": { "median_us": 24.655040502548218, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 24.966639280319214, "paired_change_pct": [ -0.42311854760532785, 0.9500378763851236, 2.26477735412447 ], "geomean_change_pct": 0.9245979740952759, "repeats_faster": 1 }, "head": { "median_us": 24.71648097038269, "paired_change_pct": [ -0.9662971644094798, -0.06145151870904364, 2.868505907765373 ], "geomean_change_pct": 0.6003509186207623, "repeats_faster": 2 }, "masked": { "median_us": 35.32920002937317, "paired_change_pct": [ 43.29402551872394, 39.64082312740367, 45.78367102829224 ], "geomean_change_pct": 42.88384280582176, "repeats_faster": 0 } }, { "tokens": 32, "afd": { "median_us": 26.144559383392334, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 26.203839778900146, "paired_change_pct": [ 0.2267408474493804, 0.6333526255168254, -1.3637250480610708 ], "geomean_change_pct": -0.17160912851742482, "repeats_faster": 1 }, "head": { "median_us": 25.976558923721313, "paired_change_pct": [ -0.6425828686091317, -0.267172230910806, -0.9451097309667866 ], "geomean_change_pct": -0.6186750511446126, "repeats_faster": 3 }, "masked": { "median_us": 35.604000091552734, "paired_change_pct": [ 35.88081350574417, 36.83110561528986, 33.91882965576711 ], "geomean_change_pct": 35.53814574581568, "repeats_faster": 0 } }, { "tokens": 128, "afd": { "median_us": 35.198320150375366, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 35.34416079521179, "paired_change_pct": [ 0.8591974246372924, 0.4143397872778065, -0.40868791149034633 ], "geomean_change_pct": 0.28690604707715117, "repeats_faster": 1 }, "head": { "median_us": 35.35383939743042, "paired_change_pct": [ 0.7355756369540112, 0.4418371285636269, 0.17021112336579325 ], "geomean_change_pct": 0.44894267348360994, "repeats_faster": 0 }, "masked": { "median_us": 43.82200002670288, "paired_change_pct": [ 24.938990355338973, 25.508940429121772, 23.35056249533489 ], "geomean_change_pct": 24.596142425931312, "repeats_faster": 0 } }, { "tokens": 512, "afd": { "median_us": 70.62095880508423, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 70.49576044082642, "paired_change_pct": [ -0.5324598888808318, 0.14648574325912822, -0.17728216435486388 ], "geomean_change_pct": -0.1881372816332938, "repeats_faster": 2 }, "head": { "median_us": 70.4259204864502, "paired_change_pct": [ -0.549986188184548, -0.032137247435282124, 0.343808292458303 ], "geomean_change_pct": -0.08011055110004328, "repeats_faster": 2 }, "masked": { "median_us": 78.97783994674683, "paired_change_pct": [ 11.203011316052924, 12.219409949848847, 11.833429173239951 ], "geomean_change_pct": 11.751164374513712, "repeats_faster": 0 } }, { "tokens": 2048, "afd": { "median_us": 192.69231796264648, "paired_change_pct": [ 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 192.8061580657959, "paired_change_pct": [ -0.14331420926457783, 0.18307009823024245, -0.10354209323324426 ], "geomean_change_pct": -0.021367719519904682, "repeats_faster": 2 }, "head": { "median_us": 192.62520790100098, "paired_change_pct": [ -0.18856976507199974, 0.06737308578244061, -0.17077157166471846 ], "geomean_change_pct": -0.09739085559340444, "repeats_faster": 2 }, "masked": { "median_us": 201.1725616455078, "paired_change_pct": [ 4.518334686611714, 4.206227732424761, 4.231247269389193 ], "geomean_change_pct": 4.318507186465337, "repeats_faster": 0 } } ] }, "statler_isolation": { "arms": { "split": {}, "ffn_only": { "_env": { "SCRATCH_SHARED_COLLECT": "1" } }, "att_only": { "_env": { "SCRATCH_SHARED_FFN_RECEIVE": "1" } }, "shared": { "_env": { "AFDEP_SHARED_RECEIVE_GRID": "1" } } }, "passthrough": [ "--payload", "bf16", "--routing", "random", "--hidden", "2048", "--tokens", "1", "8", "--capacity", "2048" ], "processes": [ { "order": [ "split", "ffn_only", "att_only", "shared" ], "median_us": { "split": { "1": 21.111600399017334, "8": 24.43735957145691 }, "ffn_only": { "1": 20.967520475387573, "8": 24.726639986038208 }, "att_only": { "1": 21.211040019989014, "8": 24.23632025718689 }, "shared": { "1": 21.064400672912598, "8": 24.760559797286987 } }, "native_source_sha256": { "split": "26853291ff54efc8d03f321fce45016ead65d4a0c644c5e3cd3121dd61710d86", "ffn_only": "26853291ff54efc8d03f321fce45016ead65d4a0c644c5e3cd3121dd61710d86", "att_only": "26853291ff54efc8d03f321fce45016ead65d4a0c644c5e3cd3121dd61710d86", "shared": "26853291ff54efc8d03f321fce45016ead65d4a0c644c5e3cd3121dd61710d86" } }, { "order": [ "ffn_only", "att_only", "shared", "split" ], "median_us": { "ffn_only": { "1": 21.080880165100098, "8": 24.87768054008484 }, "att_only": { "1": 21.05824112892151, "8": 24.32255983352661 }, "shared": { "1": 21.163039207458496, "8": 25.17311930656433 }, "split": { "1": 21.51584029197693, "8": 24.45383906364441 } }, "native_source_sha256": { "ffn_only": "26853291ff54efc8d03f321fce45016ead65d4a0c644c5e3cd3121dd61710d86", "att_only": "26853291ff54efc8d03f321fce45016ead65d4a0c644c5e3cd3121dd61710d86", "shared": "26853291ff54efc8d03f321fce45016ead65d4a0c644c5e3cd3121dd61710d86", "split": "26853291ff54efc8d03f321fce45016ead65d4a0c644c5e3cd3121dd61710d86" } }, { "order": [ "att_only", "shared", "split", "ffn_only" ], "median_us": { "att_only": { "1": 21.006799936294556, "8": 24.876080751419067 }, "shared": { "1": 20.727039575576782, "8": 24.387840032577515 }, "split": { "1": 21.3644802570343, "8": 24.329919815063477 }, "ffn_only": { "1": 21.253520250320435, "8": 24.93023991584778 } }, "native_source_sha256": { "att_only": "26853291ff54efc8d03f321fce45016ead65d4a0c644c5e3cd3121dd61710d86", "shared": "26853291ff54efc8d03f321fce45016ead65d4a0c644c5e3cd3121dd61710d86", "split": "26853291ff54efc8d03f321fce45016ead65d4a0c644c5e3cd3121dd61710d86", "ffn_only": "26853291ff54efc8d03f321fce45016ead65d4a0c644c5e3cd3121dd61710d86" } }, { "order": [ "shared", "split", "ffn_only", "att_only" ], "median_us": { "shared": { "1": 20.963200330734253, "8": 24.747120141983032 }, "split": { "1": 21.171040534973145, "8": 24.25800085067749 }, "ffn_only": { "1": 21.397359371185303, "8": 24.5578396320343 }, "att_only": { "1": 21.234639883041382, "8": 25.160160064697266 } }, "native_source_sha256": { "shared": "26853291ff54efc8d03f321fce45016ead65d4a0c644c5e3cd3121dd61710d86", "split": "26853291ff54efc8d03f321fce45016ead65d4a0c644c5e3cd3121dd61710d86", "ffn_only": "26853291ff54efc8d03f321fce45016ead65d4a0c644c5e3cd3121dd61710d86", "att_only": "26853291ff54efc8d03f321fce45016ead65d4a0c644c5e3cd3121dd61710d86" } }, { "order": [ "split", "ffn_only", "att_only", "shared" ], "median_us": { "split": { "1": 21.44760012626648, "8": 24.80631947517395 }, "ffn_only": { "1": 21.03800058364868, "8": 24.68608021736145 }, "att_only": { "1": 20.85576057434082, "8": 24.25047993659973 }, "shared": { "1": 20.843520164489746, "8": 24.78343963623047 } }, "native_source_sha256": { "split": "26853291ff54efc8d03f321fce45016ead65d4a0c644c5e3cd3121dd61710d86", "ffn_only": "26853291ff54efc8d03f321fce45016ead65d4a0c644c5e3cd3121dd61710d86", "att_only": "26853291ff54efc8d03f321fce45016ead65d4a0c644c5e3cd3121dd61710d86", "shared": "26853291ff54efc8d03f321fce45016ead65d4a0c644c5e3cd3121dd61710d86" } } ], "table": [ { "tokens": 1, "split": { "median_us": 21.3644802570343, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "ffn_only": { "median_us": 21.080880165100098, "paired_change_pct": [ -0.6824680313505138, -2.021580942107215, -0.5193667497590204, 1.0690019502739734, -1.9097686464051966 ], "geomean_change_pct": -0.8191731261335544, "repeats_faster": 4 }, "att_only": { "median_us": 21.05824112892151, "paired_change_pct": [ 0.4710188668420878, -2.1268012629097943, -1.6741821773173182, 0.30040728495690505, -2.7594674855992096 ], "geomean_change_pct": -1.1664406307675623, "repeats_faster": 3 }, "shared": { "median_us": 20.963200330734253, "paired_change_pct": [ -0.22357246827641175, -1.6397271950842174, -2.983647033714476, -0.98171936280389, -2.816538718646322 ], "geomean_change_pct": -1.7347252882435882, "repeats_faster": 5 } }, { "tokens": 8, "split": { "median_us": 24.43735957145691, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "ffn_only": { "median_us": 24.726639986038208, "paired_change_pct": [ 1.1837629746185074, 1.7332308245642913, 2.4674150401951644, 1.2360407735266365, -0.48471220381094415 ], "geomean_change_pct": 1.222458165223217, "repeats_faster": 1 }, "att_only": { "median_us": 24.32255983352661, "paired_change_pct": [ -0.8226719981025865, -0.5368450727762042, 2.2448119044660597, 3.7190171588050713, -2.2407174878582903 ], "geomean_change_pct": 0.4491954387836783, "repeats_faster": 3 }, "shared": { "median_us": 24.760559797286987, "paired_change_pct": [ 1.322566068911879, 2.941379637969721, 0.23806168682141138, 2.016321519305575, -0.09223391227537459 ], "geomean_change_pct": 1.2790331047966452, "repeats_faster": 1 } } ] }, "statler_launch_variants": { "arms": { "split": {}, "coop1": { "_env": { "SCRATCH_RECV_COOP": "1" } }, "c2": { "_env": { "SCRATCH_RECV_COOP": "1", "SCRATCH_RECV_CTAS": "2", "SCRATCH_RECV_CLUSTER": "2" } }, "shared": { "_env": { "AFDEP_SHARED_RECEIVE_GRID": "1" } } }, "passthrough": [ "--payload", "bf16", "--routing", "random", "--hidden", "2048", "--tokens", "1", "8", "32", "--capacity", "2048" ], "processes": [ { "order": [ "split", "coop1", "c2", "shared" ], "median_us": { "split": { "1": 21.075119972229004, "8": 24.528321027755737, "32": 25.866960287094116 }, "coop1": { "1": 21.269360780715942, "8": 24.71911907196045, "32": 25.674320459365845 }, "c2": { "1": 21.15407943725586, "8": 24.484400749206543, "32": 25.754319429397583 }, "shared": { "1": 20.909440517425537, "8": 24.817839860916138, "32": 26.215040683746338 } }, "native_source_sha256": { "split": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "coop1": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "c2": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "shared": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4" } }, { "order": [ "coop1", "c2", "shared", "split" ], "median_us": { "coop1": { "1": 21.345280408859253, "8": 24.3504798412323, "32": 25.834879875183105 }, "c2": { "1": 21.186400651931763, "8": 24.553200006484985, "32": 25.73912024497986 }, "shared": { "1": 20.943520069122314, "8": 24.799519777297974, "32": 25.517919063568115 }, "split": { "1": 21.39159917831421, "8": 24.623680114746094, "32": 25.76591968536377 } }, "native_source_sha256": { "coop1": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "c2": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "shared": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "split": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4" } }, { "order": [ "c2", "shared", "split", "coop1" ], "median_us": { "c2": { "1": 21.173200607299805, "8": 24.232879877090454, "32": 26.11311912536621 }, "shared": { "1": 21.124800443649292, "8": 25.089759826660156, "32": 25.881839990615845 }, "split": { "1": 21.386719942092896, "8": 24.574880599975586, "32": 25.927599668502808 }, "coop1": { "1": 20.979679822921753, "8": 24.2576801776886, "32": 25.78463912010193 } }, "native_source_sha256": { "c2": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "shared": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "split": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "coop1": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4" } }, { "order": [ "shared", "split", "coop1", "c2" ], "median_us": { "shared": { "1": 21.043280363082886, "8": 25.179039239883423, "32": 25.987520217895508 }, "split": { "1": 21.104320287704468, "8": 24.400559663772583, "32": 25.747519731521606 }, "coop1": { "1": 21.234800815582275, "8": 24.479680061340332, "32": 26.013439893722534 }, "c2": { "1": 20.92079997062683, "8": 24.249279499053955, "32": 25.846000909805298 } }, "native_source_sha256": { "shared": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "split": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "coop1": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "c2": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4" } }, { "order": [ "split", "coop1", "c2", "shared" ], "median_us": { "split": { "1": 21.23319983482361, "8": 24.64784026145935, "32": 26.21623992919922 }, "coop1": { "1": 20.959359407424927, "8": 24.544719457626343, "32": 26.009119749069214 }, "c2": { "1": 21.184239387512207, "8": 24.446159601211548, "32": 25.67255973815918 }, "shared": { "1": 20.780720710754395, "8": 24.78767991065979, "32": 25.578880310058594 } }, "native_source_sha256": { "split": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "coop1": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "c2": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "shared": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4" } } ], "table": [ { "tokens": 1, "split": { "median_us": 21.23319983482361, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "coop1": { "median_us": 21.234800815582275, "paired_change_pct": [ 0.9216593250377381, -0.21652784847385753, -1.9032377114080679, 0.6182645358819183, -1.2896804510338922 ], "geomean_change_pct": -0.37979992276657715, "repeats_faster": 3 }, "c2": { "median_us": 21.173200607299805, "paired_change_pct": [ 0.37465725049679044, -0.9592481827654376, -0.9983734549815049, -0.8695864854958435, -0.23058440410429482 ], "geomean_change_pct": -0.5380576922009772, "repeats_faster": 4 }, "shared": { "median_us": 20.943520069122314, "paired_change_pct": [ -0.7861376591060232, -2.094649892496747, -1.2246828833630485, -0.2892295216782914, -2.130998283768437 ], "geomean_change_pct": -1.307786639450692, "repeats_faster": 5 } }, { "tokens": 8, "split": { "median_us": 24.574880599975586, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "coop1": { "median_us": 24.479680061340332, "paired_change_pct": [ 0.7778683424308097, -1.1095022037351132, -1.2907506142158076, 0.3242564869740283, -0.41837663153900806 ], "geomean_change_pct": -0.34650230873857213, "repeats_faster": 3 }, "c2": { "median_us": 24.446159601211548, "paired_change_pct": [ -0.17905945743084128, -0.28622897931045443, -1.3916678923171277, -0.6199864544223277, -0.8182488125061504 ], "geomean_change_pct": -0.6599790946926753, "repeats_faster": 5 }, "shared": { "median_us": 24.817839860916138, "paired_change_pct": [ 1.1803450910186086, 0.7141079714017895, 2.0951443673955517, 3.1904168873087224, 0.5673505171935878 ], "geomean_change_pct": 1.544776022755845, "repeats_faster": 0 } }, { "tokens": 32, "split": { "median_us": 25.866960287094116, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "coop1": { "median_us": 25.834879875183105, "paired_change_pct": [ -0.7447331483490416, 0.2676410959183029, -0.5513836615371304, 1.0327991393880742, -0.7900453333100521 ], "geomean_change_pct": -0.15963773652578617, "repeats_faster": 3 }, "c2": { "median_us": 25.754319429397583, "paired_change_pct": [ -0.43546229029752803, -0.1040111927350873, 0.7155288543303717, 0.3824880194697844, -2.0738297807325834 ], "geomean_change_pct": -0.30780025253761556, "repeats_faster": 3 }, "shared": { "median_us": 25.881839990615845, "paired_change_pct": [ 1.3456563615860562, -0.9625141459108533, -0.17649022073783494, 0.9321305076235298, -2.431163358520927 ], "geomean_change_pct": -0.26774704214190814, "repeats_faster": 3 } } ] }, "statler_2a2f": { "arms": { "afd": {}, "afd_shared": { "_env": { "AFDEP_SHARED_RECEIVE_GRID": "1" } }, "head": { "_tree": "/work/src-head", "_digest": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879" }, "masked": { "role_aware": false } }, "passthrough": [ "--payload", "bf16", "--routing", "random", "--hidden", "2048", "--tokens", "1", "8", "32", "128", "512", "2048", "--capacity", "2048" ], "processes": [ { "order": [ "afd", "afd_shared", "head", "masked" ], "median_us": { "afd": { "1": 21.0316801071167, "8": 24.326640367507935, "32": 25.923279523849487, "128": 34.70744013786316, "512": 70.30104160308838, "2048": 192.36687660217285 }, "afd_shared": { "1": 21.18167996406555, "8": 25.32863974571228, "32": 26.162559986114502, "128": 34.73615884780884, "512": 70.56407928466797, "2048": 192.50791549682617 }, "head": { "1": 21.129679679870605, "8": 24.403600692749023, "32": 26.11199975013733, "128": 35.881839990615845, "512": 70.69008111953735, "2048": 192.4014377593994 }, "masked": { "1": 31.363999843597412, "8": 34.45288062095642, "32": 35.236960649490356, "128": 43.525118827819824, "512": 78.61151933670044, "2048": 201.18864059448242 } }, "native_source_sha256": { "afd": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "afd_shared": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d" } }, { "order": [ "afd_shared", "head", "masked", "afd" ], "median_us": { "afd_shared": { "1": 20.808640718460083, "8": 24.8471200466156, "32": 25.998400449752808, "128": 34.61336016654968, "512": 70.52328109741211, "2048": 192.21927642822266 }, "head": { "1": 20.822800397872925, "8": 24.353840351104736, "32": 25.7260799407959, "128": 35.945680141448975, "512": 70.38072109222412, "2048": 192.41808891296387 }, "masked": { "1": 31.63367986679077, "8": 34.340800046920776, "32": 35.95263957977295, "128": 43.65216016769409, "512": 78.72143983840942, "2048": 201.28960609436035 }, "afd": { "1": 21.301040649414062, "8": 24.592000246047974, "32": 25.931599140167236, "128": 34.415600299835205, "512": 70.45968055725098, "2048": 192.1613597869873 } }, "native_source_sha256": { "afd_shared": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "afd": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d" } }, { "order": [ "head", "masked", "afd", "afd_shared" ], "median_us": { "head": { "1": 21.07055902481079, "8": 24.997600317001343, "32": 25.849119424819946, "128": 34.58696007728577, "512": 70.37440061569214, "2048": 192.02207565307617 }, "masked": { "1": 31.315280199050903, "8": 34.061199426651, "32": 35.37840008735657, "128": 43.69192123413086, "512": 78.49855899810791, "2048": 200.72375297546387 }, "afd": { "1": 20.892640352249146, "8": 24.210959672927856, "32": 26.225759983062744, "128": 35.91464042663574, "512": 70.15503883361816, "2048": 192.21519470214844 }, "afd_shared": { "1": 20.962400436401367, "8": 24.76591944694519, "32": 26.077120304107666, "128": 36.33008003234863, "512": 70.3713607788086, "2048": 192.0024013519287 } }, "native_source_sha256": { "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "afd": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "afd_shared": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d" } }, { "order": [ "masked", "afd", "afd_shared", "head" ], "median_us": { "masked": { "1": 31.387919187545776, "8": 34.086079597473145, "32": 35.477439165115356, "128": 43.37759971618652, "512": 78.34488153457642, "2048": 200.8184814453125 }, "afd": { "1": 21.244159936904907, "8": 24.677600860595703, "32": 25.899440050125122, "128": 34.70471978187561, "512": 70.24879932403564, "2048": 191.7409610748291 }, "afd_shared": { "1": 20.82479953765869, "8": 24.39679980278015, "32": 25.957599878311157, "128": 35.718079805374146, "512": 70.0984001159668, "2048": 191.93143844604492 }, "head": { "1": 20.971519947052002, "8": 24.595279693603516, "32": 25.864399671554565, "128": 34.434800148010254, "512": 70.26560068130493, "2048": 192.3795986175537 } }, "native_source_sha256": { "masked": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "afd": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "afd_shared": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879" } }, { "order": [ "afd", "afd_shared", "head", "masked" ], "median_us": { "afd": { "1": 21.060640811920166, "8": 24.510319232940674, "32": 26.13968014717102, "128": 34.47503924369812, "512": 70.20888090133667, "2048": 192.48215675354004 }, "afd_shared": { "1": 21.023919582366943, "8": 24.707599878311157, "32": 26.165119409561157, "128": 34.43743944168091, "512": 70.51232099533081, "2048": 192.2480869293213 }, "head": { "1": 20.68719983100891, "8": 24.32960033416748, "32": 25.47231912612915, "128": 34.35248017311096, "512": 70.09024143218994, "2048": 192.13272094726562 }, "masked": { "1": 31.67151927947998, "8": 34.4756805896759, "32": 35.741679668426514, "128": 43.548641204833984, "512": 78.68880033493042, "2048": 201.38479232788086 } }, "native_source_sha256": { "afd": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "afd_shared": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d" } } ], "table": [ { "tokens": 1, "afd": { "median_us": 21.060640811920166, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 20.962400436401367, "paired_change_pct": [ 0.7132091025771059, -2.311623826545417, 0.33389788449937896, -1.97400321072575, -0.17435950729685068 ], "geomean_change_pct": -0.6902070530728888, "repeats_faster": 3 }, "head": { "median_us": 20.971519947052002, "paired_change_pct": [ 0.4659616932873867, -2.2451497061214853, 0.8515853887394842, -1.2833644195046845, -1.773170077046704 ], "geomean_change_pct": -0.8044721339423022, "repeats_faster": 3 }, "masked": { "median_us": 31.387919187545776, "paired_change_pct": [ 49.127410096849374, 48.50767334534396, 49.88665707673341, 47.748460192202494, 50.38250527284116 ], "geomean_change_pct": 49.12756425572369, "repeats_faster": 0 } }, { "tokens": 8, "afd": { "median_us": 24.510319232940674, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 24.76591944694519, "paired_change_pct": [ 4.118938591876731, 1.03740971866908, 2.292184124522234, -1.1378782702654244, 0.8048881105773065 ], "geomean_change_pct": 1.4082250391423523, "repeats_faster": 1 }, "head": { "median_us": 24.403600692749023, "paired_change_pct": [ 0.31636232574014933, -0.9684445858832103, 3.2491097201450048, -0.33358658913895667, -0.7373176051102437 ], "geomean_change_pct": 0.29361968330621213, "repeats_faster": 3 }, "masked": { "median_us": 34.340800046920776, "paired_change_pct": [ 41.626135382729125, 39.642158845698084, 40.68504465247389, 38.12558112932509, 40.65781951685994 ], "geomean_change_pct": 40.14227976401929, "repeats_faster": 0 } }, { "tokens": 32, "afd": { "median_us": 25.931599140167236, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 26.077120304107666, "paired_change_pct": [ 0.9230331449571194, 0.2576058237846901, -0.5667697677820382, 0.22456017610217316, 0.09732048076682709 ], "geomean_change_pct": 0.1860274940124862, "repeats_faster": 1 }, "head": { "median_us": 25.849119424819946, "paired_change_pct": [ 0.7279951833031761, -0.792543484342989, -1.4361473546850134, -0.13529396196496624, -2.5530573338484186 ], "geomean_change_pct": -0.844117438666403, "repeats_faster": 4 }, "masked": { "median_us": 35.477439165115356, "paired_change_pct": [ 35.927865982667264, 38.64412829089061, 34.89942754835258, 36.98149109190476, 36.73342392559717 ], "geomean_change_pct": 36.63165630678478, "repeats_faster": 0 } }, { "tokens": 128, "afd": { "median_us": 34.70471978187561, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 34.73615884780884, "paired_change_pct": [ 0.08274511122572381, 0.5746227437312124, 1.1567416540380737, 2.9199487270540114, -0.10906384109217182 ], "geomean_change_pct": 0.9191582501367312, "repeats_faster": 1 }, "head": { "median_us": 34.58696007728577, "paired_change_pct": [ 3.3837121034792306, 4.445890318005286, -3.696766370422333, -0.7777605915329167, -0.355500887818605 ], "geomean_change_pct": 0.5563127964550496, "repeats_faster": 3 }, "masked": { "median_us": 43.548641204833984, "paired_change_pct": [ 25.405730457018795, 26.838293644127177, 21.654903724797347, 24.99049117474876, 26.31933758507461 ], "geomean_change_pct": 25.02844831546749, "repeats_faster": 0 } }, { "tokens": 512, "afd": { "median_us": 70.24879932403564, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 70.51232099533081, "paired_change_pct": [ 0.37415901042359145, 0.09026515541672442, 0.30834840773656946, -0.21409505858612787, 0.4321961696278187 ], "geomean_change_pct": 0.1978954593735871, "repeats_faster": 1 }, "head": { "median_us": 70.37440061569214, "paired_change_pct": [ 0.5533908283257727, -0.11206333097507626, 0.3126814348919593, 0.023916931578837897, -0.1689807152936207 ], "geomean_change_pct": 0.1214182732586222, "repeats_faster": 2 }, "masked": { "median_us": 78.61151933670044, "paired_change_pct": [ 11.821272550315909, 11.725513394068642, 11.892973481601942, 11.524869162811013, 12.078129325990016 ], "geomean_change_pct": 11.80840190604886, "repeats_faster": 0 } }, { "tokens": 2048, "afd": { "median_us": 192.21519470214844, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 192.21927642822266, "paired_change_pct": [ 0.07331766109868187, 0.03013958753181889, -0.11070579022093296, 0.09934099117270279, -0.12160598580494009 ], "geomean_change_pct": -0.0059457384860461815, "repeats_faster": 2 }, "head": { "median_us": 192.3795986175537, "paired_change_pct": [ 0.017966272487779378, 0.1336008062500893, -0.10047023044744696, 0.33307308941430414, -0.18154192168671202 ], "geomean_change_pct": 0.04036179993929245, "repeats_faster": 2 }, "masked": { "median_us": 201.18864059448242, "paired_change_pct": [ 4.5859059252459256, 4.750302723446476, 4.4265794317145835, 4.734262475580686, 4.6251744704523645 ], "geomean_change_pct": 4.6243794706315455, "repeats_faster": 0 } } ] }, "statler_3a1f_view": { "arms": { "afd": {}, "afd_shared": { "_env": { "AFDEP_SHARED_RECEIVE_GRID": "1" } }, "head": { "_tree": "/work/src-head", "_digest": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879" }, "masked": { "role_aware": false } }, "passthrough": [ "--payload", "bf16", "--routing", "random", "--hidden", "2048", "--tokens", "1", "8", "32", "128", "512", "2048", "--capacity", "2048", "--ffn-ranks", "1", "--attention-ranks", "3" ], "processes": [ { "order": [ "afd", "afd_shared", "head", "masked" ], "median_us": { "afd": { "1": 19.95407998561859, "8": 21.547839641571045, "32": 25.043920278549194, "128": 37.35743999481201, "512": 102.16479778289795, "2048": 348.70288848876953 }, "afd_shared": { "1": 20.23784041404724, "8": 21.645359992980957, "32": 25.22871971130371, "128": 37.214800119400024, "512": 102.35663890838623, "2048": 349.5975112915039 }, "head": { "1": 20.318639278411865, "8": 21.544159650802612, "32": 25.013760328292847, "128": 37.17151999473572, "512": 102.54807949066162, "2048": 348.6800003051758 }, "masked": { "1": 35.75991988182068, "8": 37.962000370025635, "32": 43.14351797103882, "128": 54.93528127670288, "512": 118.66999626159668, "2048": 371.39808654785156 } }, "native_source_sha256": { "afd": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "afd_shared": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d" } }, { "order": [ "afd_shared", "head", "masked", "afd" ], "median_us": { "afd_shared": { "1": 20.385841131210327, "8": 21.686960458755493, "32": 24.923839569091797, "128": 37.30263948440552, "512": 102.4400806427002, "2048": 349.1216850280762 }, "head": { "1": 20.132960081100464, "8": 21.826239824295044, "32": 25.26368021965027, "128": 37.54840016365051, "512": 102.1947193145752, "2048": 348.5844039916992 }, "masked": { "1": 36.01263999938965, "8": 39.330079555511475, "32": 43.92087936401367, "128": 55.292160511016846, "512": 119.15224075317383, "2048": 371.6655158996582 }, "afd": { "1": 20.41432023048401, "8": 21.73840045928955, "32": 25.13487935066223, "128": 37.23536014556885, "512": 102.50159740447998, "2048": 350.32360076904297 } }, "native_source_sha256": { "afd_shared": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "afd": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d" } }, { "order": [ "head", "masked", "afd", "afd_shared" ], "median_us": { "head": { "1": 20.368800163269043, "8": 21.698240041732788, "32": 24.94879961013794, "128": 37.31703996658325, "512": 102.53376007080078, "2048": 349.2176818847656 }, "masked": { "1": 35.7044792175293, "8": 38.92224073410034, "32": 43.34503889083862, "128": 54.787437915802, "512": 118.58751773834229, "2048": 371.1387252807617 }, "afd": { "1": 20.06432056427002, "8": 21.801199913024902, "32": 25.77015995979309, "128": 37.552000284194946, "512": 102.25711822509766, "2048": 348.3852005004883 }, "afd_shared": { "1": 20.006799697875977, "8": 22.13088035583496, "32": 25.25607943534851, "128": 37.725040912628174, "512": 102.40679740905762, "2048": 348.9699172973633 } }, "native_source_sha256": { "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "afd": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "afd_shared": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d" } }, { "order": [ "masked", "afd", "afd_shared", "head" ], "median_us": { "masked": { "1": 35.748640298843384, "8": 38.472479581832886, "32": 42.9475212097168, "128": 54.78208065032959, "512": 118.58392238616943, "2048": 371.2455368041992 }, "afd": { "1": 20.391039848327637, "8": 21.713839769363403, "32": 24.78927969932556, "128": 37.193440198898315, "512": 102.64135837554932, "2048": 349.96912002563477 }, "afd_shared": { "1": 20.43503999710083, "8": 21.98016047477722, "32": 25.22592067718506, "128": 37.64400005340576, "512": 102.3963212966919, "2048": 348.75240325927734 }, "head": { "1": 20.28704047203064, "8": 21.790320873260498, "32": 24.843519926071167, "128": 37.292640209198, "512": 102.56983757019043, "2048": 349.49817657470703 } }, "native_source_sha256": { "masked": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "afd": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "afd_shared": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879" } }, { "order": [ "afd", "afd_shared", "head", "masked" ], "median_us": { "afd": { "1": 20.480480194091797, "8": 21.725120544433594, "32": 24.758000373840332, "128": 37.016719579696655, "512": 102.56487846374512, "2048": 349.6345520019531 }, "afd_shared": { "1": 20.243279933929443, "8": 22.115600109100342, "32": 25.44167995452881, "128": 37.822641134262085, "512": 102.21695899963379, "2048": 348.92967224121094 }, "head": { "1": 20.411840677261353, "8": 21.84544086456299, "32": 24.916319847106934, "128": 37.263439893722534, "512": 102.58416175842285, "2048": 350.1553726196289 }, "masked": { "1": 36.04351997375488, "8": 39.021679162979126, "32": 43.48320007324219, "128": 55.39535999298096, "512": 119.28567886352539, "2048": 372.187442779541 } }, "native_source_sha256": { "afd": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "afd_shared": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "masked": "a11fe415dbe92ebae1f08d3f79418245c5b7cab08285f552af0ac6499bdacb1d" } } ], "table": [ { "tokens": 1, "afd": { "median_us": 20.391039848327637, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 20.243279933929443, "paired_change_pct": [ 1.4220672094787723, -0.1395054988466149, -0.2866823534332652, 0.21578178013712535, -1.1581772395687362 ], "geomean_change_pct": 0.007193591710930036, "repeats_faster": 3 }, "head": { "median_us": 20.318639278411865, "paired_change_pct": [ 1.8269912371606356, -1.3782489262777364, 1.517517615529096, -0.5100248789201678, -0.3351460326122879 ], "geomean_change_pct": 0.21658359823417683, "repeats_faster": 3 }, "masked": { "median_us": 35.75991988182068, "paired_change_pct": [ 79.21106815044219, 76.40871502355098, 77.95010353408544, 75.31543543021075, 75.98962344717243 ], "geomean_change_pct": 76.9693539066128, "repeats_faster": 0 } }, { "tokens": 8, "afd": { "median_us": 21.725120544433594, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 21.98016047477722, "paired_change_pct": [ 0.4525760031264259, -0.23663194829073309, 1.5122123742055793, 1.2265021214238558, 1.79736431780948 ], "geomean_change_pct": 0.9476579578852062, "repeats_faster": 1 }, "head": { "median_us": 21.790320873260498, "paired_change_pct": [ -0.017078235357448346, 0.40407464739640186, -0.4722669931144563, 0.3522228436308339, 0.553830391335719 ], "geomean_change_pct": 0.16347376268626146, "repeats_faster": 2 }, "masked": { "median_us": 38.92224073410034, "paired_change_pct": [ 76.1754356886324, 80.92444119412848, 78.53256192034938, 77.17953153598684, 79.61547823483654 ], "geomean_change_pct": 78.47749747534756, "repeats_faster": 0 } }, { "tokens": 32, "afd": { "median_us": 25.043920278549194, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 25.22871971130371, "paired_change_pct": [ 0.737901377656125, -0.8396291807339584, -1.994867417379853, 1.7614105095251187, 2.76144910883378 ], "geomean_change_pct": 0.47051767649539666, "repeats_faster": 2 }, "head": { "median_us": 24.94879961013794, "paired_change_pct": [ -0.12042823136672043, 0.5124387795585106, -3.187253594609607, 0.21880517467025573, 0.6394679330964159 ], "geomean_change_pct": -0.3977119490105929, "repeats_faster": 2 }, "masked": { "median_us": 43.34503889083862, "paired_change_pct": [ 72.27142352785889, 74.74076064286533, 68.19856360405238, 73.25037972315616, 75.63292437456775 ], "geomean_change_pct": 72.79929844638053, "repeats_faster": 0 } }, { "tokens": 128, "afd": { "median_us": 37.23536014556885, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 37.64400005340576, "paired_change_pct": [ -0.38182454534303467, 0.18068668752939665, 0.46080269259600737, 1.211396020636979, 2.177182537286404 ], "geomean_change_pct": 0.7257509551201746, "repeats_faster": 1 }, "head": { "median_us": 37.292640209198, "paired_change_pct": [ -0.4976786420646473, 0.8407062986845348, -0.6256932142988503, 0.2667137263162367, 0.666510476420501 ], "geomean_change_pct": 0.12833623530874583, "repeats_faster": 2 }, "masked": { "median_us": 54.93528127670288, "paired_change_pct": [ 47.05312056803672, 48.493690660856494, 45.89752210579627, 47.28963052993485, 49.64956544492081 ], "geomean_change_pct": 47.67111668754755, "repeats_faster": 0 } }, { "tokens": 512, "afd": { "median_us": 102.50159740447998, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 102.3963212966919, "paired_change_pct": [ 0.18777615152330007, -0.06001541764957263, 0.14637531993662023, -0.23873132890629423, -0.33921891131017956 ], "geomean_change_pct": -0.06097687664452511, "repeats_faster": 3 }, "head": { "median_us": 102.54807949066162, "paired_change_pct": [ 0.3751602470531523, -0.29938859264194706, 0.27053553875258096, -0.06968029894655059, 0.018801070080298743 ], "geomean_change_pct": 0.058794394193650845, "repeats_faster": 2 }, "masked": { "median_us": 118.66999626159668, "paired_change_pct": [ 16.155465323557515, 16.24427693842565, 15.969939107120812, 15.532300295840473, 16.30265705983436 ], "geomean_change_pct": 16.040593971769024, "repeats_faster": 0 } }, { "tokens": 2048, "afd": { "median_us": 349.6345520019531, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "afd_shared": { "median_us": 348.9699172973633, "paired_change_pct": [ 0.256557324951201, -0.3430872879612745, 0.16783629041503545, -0.3476640356921523, -0.20160472033046162 ], "geomean_change_pct": -0.09392202733640165, "repeats_faster": 3 }, "head": { "median_us": 349.2176818847656, "paired_change_pct": [ -0.00656380671033574, -0.4964543563510393, 0.23895429056153272, -0.1345671443506946, 0.14896142692237824 ], "geomean_change_pct": -0.05026590966400457, "repeats_faster": 3 }, "masked": { "median_us": 371.39808654785156, "paired_change_pct": [ 6.508462880086796, 6.092057481644031, 6.531139884124193, 6.079512608714155, 6.450418200504937 ], "geomean_change_pct": 6.332124257615712, "repeats_faster": 0 } } ] }, "statler_24cta_trim": { "arms": { "split": {}, "g24": { "_env": { "SCRATCH_RECV_COOP": "1", "SCRATCH_RECV_CTAS": "24", "SCRATCH_RECV_CLUSTER": "2" } }, "shared": { "_env": { "AFDEP_SHARED_RECEIVE_GRID": "1" } } }, "passthrough": [ "--payload", "bf16", "--routing", "random", "--hidden", "2048", "--tokens", "1", "8", "32", "128", "--capacity", "2048" ], "processes": [ { "order": [ "split", "g24", "shared" ], "median_us": { "split": { "1": 21.17192029953003, "8": 24.47119951248169, "32": 26.10487937927246, "128": 34.54200029373169 }, "g24": { "1": 21.016000509262085, "8": 24.854079484939575, "32": 26.155120134353638, "128": 35.94767928123474 }, "shared": { "1": 20.612080097198486, "8": 24.342719316482544, "32": 25.780080556869507, "128": 34.55943942070007 } }, "native_source_sha256": { "split": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "g24": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "shared": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4" } }, { "order": [ "g24", "shared", "split" ], "median_us": { "g24": { "1": 21.12175941467285, "8": 25.01031994819641, "32": 26.07927918434143, "128": 34.65384006500244 }, "shared": { "1": 21.049920320510864, "8": 24.627439975738525, "32": 25.934799909591675, "128": 34.52855944633484 }, "split": { "1": 21.053121089935303, "8": 24.445760250091553, "32": 26.387200355529785, "128": 36.16424083709717 } }, "native_source_sha256": { "g24": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "shared": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "split": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4" } }, { "order": [ "shared", "split", "g24" ], "median_us": { "shared": { "1": 20.76792001724243, "8": 24.63919997215271, "32": 26.126400232315063, "128": 34.80967998504639 }, "split": { "1": 21.213600635528564, "8": 24.323760271072388, "32": 26.14807963371277, "128": 34.5854389667511 }, "g24": { "1": 20.9935200214386, "8": 25.0325608253479, "32": 26.0480797290802, "128": 34.42800045013428 } }, "native_source_sha256": { "shared": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "split": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "g24": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4" } }, { "order": [ "split", "g24", "shared" ], "median_us": { "split": { "1": 21.397600173950195, "8": 24.61359977722168, "32": 25.792720317840576, "128": 35.80272078514099 }, "g24": { "1": 20.80031991004944, "8": 24.47856068611145, "32": 25.892560482025146, "128": 36.18384003639221 }, "shared": { "1": 20.670559406280518, "8": 24.62696075439453, "32": 25.757360458374023, "128": 36.00135922431946 } }, "native_source_sha256": { "split": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "g24": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "shared": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4" } }, { "order": [ "g24", "shared", "split" ], "median_us": { "g24": { "1": 20.977840423583984, "8": 24.83031988143921, "32": 26.111119985580444, "128": 34.97048020362854 }, "shared": { "1": 21.128239631652832, "8": 24.820079803466797, "32": 26.36712074279785, "128": 34.6016800403595 }, "split": { "1": 21.25607967376709, "8": 24.4977605342865, "32": 25.670959949493408, "128": 34.368239641189575 } }, "native_source_sha256": { "g24": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "shared": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "split": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4" } }, { "order": [ "shared", "split", "g24" ], "median_us": { "shared": { "1": 20.74392080307007, "8": 24.734880924224854, "32": 26.05504035949707, "128": 34.653199911117554 }, "split": { "1": 21.149519681930542, "8": 24.421679973602295, "32": 25.890159606933594, "128": 34.67719912528992 }, "g24": { "1": 20.730080604553223, "8": 24.747679233551025, "32": 25.6496798992157, "128": 35.70751905441284 } }, "native_source_sha256": { "shared": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "split": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "g24": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4" } } ], "table": [ { "tokens": 1, "split": { "median_us": 21.192760467529297, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "g24": { "median_us": 20.98568022251129, "paired_change_pct": [ -0.7364461421640933, 0.3260244618569219, -1.0374505387896082, -2.791342295608912, -1.3089866732410194, -1.9832085252303622 ], "geomean_change_pct": -1.2600531567051432, "repeats_faster": 5 }, "shared": { "median_us": 20.75592041015625, "paired_change_pct": [ -2.644258028611468, -0.015203301262389779, -2.1009192448909753, -3.3977677952632734, -0.6014281282170297, -1.9177687482283767 ], "geomean_change_pct": -1.7863240771946565, "repeats_faster": 6 } }, { "tokens": 8, "split": { "median_us": 24.45847988128662, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "g24": { "median_us": 24.842199683189392, "paired_change_pct": [ 1.5646146494069368, 2.309438087951232, 2.91402540715906, -0.5486360887171027, 1.3575091759398417, 1.3348764716477612 ], "geomean_change_pct": 1.482942454952596, "repeats_faster": 1 }, "shared": { "median_us": 24.633319973945618, "paired_change_pct": [ -0.5250261473027207, 0.7431952362630723, 1.2968377321801894, 0.054282905766656064, 1.315709118509778, 1.2824709477853302 ], "geomean_change_pct": 0.6920927420357215, "repeats_faster": 1 } }, { "tokens": 32, "split": { "median_us": 25.997519493103027, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "g24": { "median_us": 26.063679456710815, "paired_change_pct": [ 0.1924573347045211, -1.1669338430737475, -0.3824368979802206, 0.387086600227704, 1.7146224253126174, -0.9288459838366236 ], "geomean_change_pct": -0.03523583655093976, "repeats_faster": 3 }, "shared": { "median_us": 25.994920134544373, "paired_change_pct": [ -1.2442073287679989, -1.7144692875434409, -0.08291010927530262, -0.1370924006107832, 2.7118611640317036, 0.6368471846705859 ], "geomean_change_pct": 0.01818328658460544, "repeats_faster": 4 } }, { "tokens": 128, "split": { "median_us": 34.63131904602051, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "g24": { "median_us": 35.33899962902069, "paired_change_pct": [ 4.069477666463173, -4.176503466223358, -0.45521618727515367, 1.0644980127024084, 1.7523171647034097, 2.971174013796052 ], "geomean_change_pct": 0.8351809560884105, "repeats_faster": 2 }, "shared": { "median_us": 34.627439975738525, "paired_change_pct": [ 0.0504867315734181, -4.522924725920008, 0.6483682873329011, 0.5548138097395849, 0.6792329243716866, -0.06920747574120645 ], "geomean_change_pct": -0.4607132431393546, "repeats_faster": 2 } } ] }, "statler_24cta_trim_vs_head": { "arms": { "g24": { "_env": { "SCRATCH_RECV_COOP": "1", "SCRATCH_RECV_CTAS": "24", "SCRATCH_RECV_CLUSTER": "2" } }, "shared": { "_env": { "AFDEP_SHARED_RECEIVE_GRID": "1" } }, "head": { "_tree": "/work/src-head", "_digest": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879" } }, "passthrough": [ "--payload", "bf16", "--routing", "random", "--hidden", "2048", "--tokens", "1", "8", "32", "128", "512", "2048", "--capacity", "2048" ], "processes": [ { "order": [ "g24", "shared", "head" ], "median_us": { "g24": { "1": 20.67184090614319, "8": 24.49015974998474, "32": 25.415918827056885, "128": 35.764559507369995, "512": 70.07375955581665, "2048": 192.32416152954102 }, "shared": { "1": 20.904159545898438, "8": 24.43552017211914, "32": 25.86151957511902, "128": 34.58127975463867, "512": 70.3875207901001, "2048": 192.1724033355713 }, "head": { "1": 20.966880321502686, "8": 24.582080841064453, "32": 25.87007999420166, "128": 34.679919481277466, "512": 70.30303955078125, "2048": 192.10335731506348 } }, "native_source_sha256": { "g24": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "shared": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879" } }, { "order": [ "shared", "head", "g24" ], "median_us": { "shared": { "1": 21.166080236434937, "8": 24.990800619125366, "32": 26.03551983833313, "128": 34.500160217285156, "512": 70.39687871932983, "2048": 192.19207763671875 }, "head": { "1": 20.918959379196167, "8": 24.23375964164734, "32": 26.013920307159424, "128": 34.385279417037964, "512": 70.33087968826294, "2048": 191.97815895080566 }, "g24": { "1": 21.007919311523438, "8": 24.55456018447876, "32": 25.82919955253601, "128": 34.44928050041199, "512": 70.35184144973755, "2048": 192.20736503601074 } }, "native_source_sha256": { "shared": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "g24": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4" } }, { "order": [ "head", "g24", "shared" ], "median_us": { "head": { "1": 20.934799909591675, "8": 24.676800966262817, "32": 25.980160236358643, "128": 34.51095938682556, "512": 70.46279907226562, "2048": 192.27727890014648 }, "g24": { "1": 21.04688048362732, "8": 24.844720363616943, "32": 26.046639680862427, "128": 35.90991973876953, "512": 70.33472061157227, "2048": 192.4605655670166 }, "shared": { "1": 21.109360456466675, "8": 24.48016047477722, "32": 26.13487958908081, "128": 36.029919385910034, "512": 70.16736030578613, "2048": 191.96343421936035 } }, "native_source_sha256": { "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "g24": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "shared": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4" } }, { "order": [ "g24", "shared", "head" ], "median_us": { "g24": { "1": 20.72391986846924, "8": 24.561599493026733, "32": 25.652559995651245, "128": 36.06023907661438, "512": 70.21503925323486, "2048": 192.12623596191406 }, "shared": { "1": 20.762319564819336, "8": 24.213680028915405, "32": 25.739200115203857, "128": 34.60911989212036, "512": 70.30783891677856, "2048": 192.16768264770508 }, "head": { "1": 21.075359582901, "8": 24.782960414886475, "32": 25.937520265579224, "128": 35.947760343551636, "512": 70.4313588142395, "2048": 192.11824417114258 } }, "native_source_sha256": { "g24": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "shared": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879" } }, { "order": [ "shared", "head", "g24" ], "median_us": { "shared": { "1": 20.886000394821167, "8": 24.479119777679443, "32": 25.70191979408264, "128": 34.60000038146973, "512": 70.278799533844, "2048": 192.1361541748047 }, "head": { "1": 20.72711944580078, "8": 24.72920060157776, "32": 25.58527946472168, "128": 34.52311992645264, "512": 69.98167991638184, "2048": 191.79831504821777 }, "g24": { "1": 20.998560190200806, "8": 24.957040548324585, "32": 26.058720350265503, "128": 34.53816056251526, "512": 70.4641604423523, "2048": 192.28520393371582 } }, "native_source_sha256": { "shared": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "g24": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4" } }, { "order": [ "head", "g24", "shared" ], "median_us": { "head": { "1": 21.165759563446045, "8": 24.82792019844055, "32": 26.04896068572998, "128": 34.4215202331543, "512": 70.32464027404785, "2048": 192.1343231201172 }, "g24": { "1": 20.94040036201477, "8": 24.41223978996277, "32": 25.819040536880493, "128": 34.54535961151123, "512": 70.42448043823242, "2048": 192.34911918640137 }, "shared": { "1": 21.019439697265625, "8": 24.34167981147766, "32": 25.886399745941162, "128": 34.38239932060242, "512": 70.19927978515625, "2048": 191.93632125854492 } }, "native_source_sha256": { "head": "8205d02bfc622ba98c680c977d8cfef4e47f9394afea26fc5828d29e3589b879", "g24": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4", "shared": "0455d7b6417034273fc7d9dafbe0ca483b609355934edf64550b4dea0f62e9e4" } } ], "table": [ { "tokens": 1, "g24": { "median_us": 20.969480276107788, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "shared": { "median_us": 20.96179962158203, "paired_change_pct": [ 1.1238410783541264, 0.7528633491311165, 0.2968609665834343, 0.1852916658325876, -0.5360357775013869, 0.37744901665885955 ], "geomean_change_pct": 0.36540241602733836, "repeats_faster": 1 }, "head": { "median_us": 20.95084011554718, "paired_change_pct": [ 1.4272527381526956, -0.42345903470065727, -0.5325282011404697, 1.6958167984738592, -1.292663601415367, 1.0761933751757047 ], "geomean_change_pct": 0.3188079145503675, "repeats_faster": 3 } }, { "tokens": 8, "g24": { "median_us": 24.558079838752747, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "shared": { "median_us": 24.457319974899292, "paired_change_pct": [ -0.2231082950189145, 1.776616772481887, -1.4673535604513788, -1.4165179438338504, -1.9149737314395843, -0.28903525072745406 ], "geomean_change_pct": -0.5965818016100299, "repeats_faster": 5 }, "head": { "median_us": 24.703000783920288, "paired_change_pct": [ 0.3753388790359802, -1.3064805087985354, -0.6758755779760373, 0.901247990476306, -0.9129285433729928, 1.7027540776848094 ], "geomean_change_pct": 0.00831462395536331, "repeats_faster": 3 } }, { "tokens": 32, "g24": { "median_us": 25.824120044708252, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "shared": { "median_us": 25.87395966053009, "paired_change_pct": [ 1.7532348568400469, 0.7987869905819789, 0.33877655351917113, 0.33774453531070225, -1.369217487992369, 0.2608896677026129 ], "geomean_change_pct": 0.34909262291904497, "repeats_faster": 1 }, "head": { "median_us": 25.958840250968933, "paired_change_pct": [ 1.7869161852267723, 0.7151625208040135, -0.2552323267735357, 1.1108453502351567, -1.8168232329911715, 0.8905061693561533 ], "geomean_change_pct": 0.3984504564273239, "repeats_faster": 2 } }, { "tokens": 128, "g24": { "median_us": 35.15495955944061, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "shared": { "median_us": 34.5906400680542, "paired_change_pct": [ -3.30852600739423, 0.1476945704934618, 0.3341685194883537, -4.0241529774967315, 0.17904780667903353, -0.4717284542451572 ], "geomean_change_pct": -1.206762432295616, "repeats_faster": 3 }, "head": { "median_us": 34.5170396566391, "paired_change_pct": [ -3.032723011362737, -0.18578351258529135, -3.895749035700591, -0.31191898873373924, -0.04354787810832983, -0.35848339617708946 ], "geomean_change_pct": -1.316979712733768, "repeats_faster": 6 } }, { "tokens": 512, "g24": { "median_us": 70.34328103065491, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "shared": { "median_us": 70.29331922531128, "paired_change_pct": [ 0.44775852797440674, 0.06401718656428024, -0.23794834802911558, 0.13216493863801304, -0.2630570028006596, -0.31977609444160393 ], "geomean_change_pct": -0.029843965424514174, "repeats_faster": 3 }, "head": { "median_us": 70.3277599811554, "paired_change_pct": [ 0.3271980787358242, -0.029795611660832222, 0.1820984850436469, 0.3080815211460086, -0.6847176251609244, -0.1417691171639368 ], "geomean_change_pct": -0.0070893434453211945, "repeats_faster": 3 } }, { "tokens": 2048, "g24": { "median_us": 192.30468273162842, "paired_change_pct": [ 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 ], "geomean_change_pct": 0.0, "repeats_faster": 0 }, "shared": { "median_us": 192.15191841125488, "paired_change_pct": [ -0.07890750322934181, -0.007953597037824167, -0.25830296517711515, 0.02157263196433945, -0.0775149391954888, -0.21460869153053785 ], "geomean_change_pct": -0.10267120800258045, "repeats_faster": 5 }, "head": { "median_us": 192.11080074310303, "paired_change_pct": [ -0.11480835934575273, -0.11924937692274762, -0.095233361873448, -0.0041596561403900445, -0.25321183093519606, -0.11166989856399301 ], "geomean_change_pct": -0.11641528154857639, "repeats_faster": 6 } } ] } }, "cta_lifetimes_pre_suites": [ "f2ac01def4ccd175b474df9531487f5a4da79d11866224c3b6e0e6bed2b3e23c life.log", "attention_combine CTAs 1..23 n 1529 median us 0.61 p90 0.64", "attention_combine CTA 0 n 71 median us 1074.3 p90 1203.04", "ffn_dispatch CTAs 1..23 n 1633 median us 0.58 p90 0.61", "ffn_dispatch CTA 0 n 70 median us 405.46 p90 1407.39", "scratch digest cb5846aed1168a8e04e0c4f1ee082bfb4b259d875f0c115ca9b4266f1b1d50ff", "--- /work/src-head/afdep/third_party/deepep_v2/deep_ep/include/deep_ep/impls/dispatch.cuh\t2026-10-21 11:55:52.000000000 +0000", "+++ /work/src-life/afdep/third_party/deepep_v2/deep_ep/include/deep_ep/impls/dispatch.cuh\t2026-10-21 14:30:05.832996731 +0000", "@@ -76,6 +76,7 @@", " constexpr int kNumNotifyDstRanks = kAfdDirectionalNotify ? kAfdFfnRanks : kNumRanks;", " constexpr int kNumNotifyExperts = kAfdDirectionalNotify ? kAfdFfnRanks * kNumExpertsPerRank : kNumExperts;", "+ unsigned long long afd_t0; asm volatile(\"mov.u64 %0, %%globaltimer;\" : \"=l\"(afd_t0)); // SCRATCH lifetime", " // Utils", " const auto sm_idx = static_cast(blockIdx.x), thread_idx = static_cast(threadIdx.x);", " const auto warp_idx = ptx::get_warp_idx(), lane_idx = ptx::get_lane_idx();", "@@ -544,6 +545,7 @@", " }", " if (not kReuseSlotIndices and clean_counters and thread_idx < kNumRanks)", " workspace_layout.get_scaleup_atomic_sender_counter()[thread_idx] = 0;", "+ if (threadIdx.x == 0 and (kAfdFfnRanks > 0 and rank_idx == 0)) { unsigned long long t1; asm volatile(\"mov.u64 %0, %%globaltimer;\" : \"=l\"(t1)); printf(\"AFDLIFE ffn_dispatch cta %d ns %llu\\n\", (int)blockIdx.x, t1 - afd_t0); } // SCRATCH", " }", " } // namespace deep_ep::elastic", "--- /work/src-head/afdep/third_party/deepep_v2/deep_ep/include/deep_ep/impls/combine.cuh\t2026-10-21 11:55:52.000000000 +0000", "+++ /work/src-life/afdep/third_party/deepep_v2/deep_ep/include/deep_ep/impls/combine.cuh\t2026-10-21 14:30:05.832996731 +0000", "@@ -42,6 +42,7 @@", " void* buffer, void* workspace,", " const int rank_idx,", " int num_reduced_tokens) {", "+ unsigned long long afd_t0; asm volatile(\"mov.u64 %0, %%globaltimer;\" : \"=l\"(afd_t0)); // SCRATCH lifetime", " // Utils", " const auto sm_idx = static_cast(blockIdx.x);", " const auto thread_idx = static_cast(threadIdx.x);", "@@ -263,6 +264,7 @@", " kNumSMs, kNumThreads, kNumQPs, kNumTimeoutCycles, comm::kCombineTag1, true, true, false>(", " gin, workspace_layout, 0, rank_idx, sm_idx, thread_idx);", " }", "+ if (threadIdx.x == 0 and (kAfdDirectedFfnRanks > 0 and rank_idx == kNumRanks - 1)) { unsigned long long t1; asm volatile(\"mov.u64 %0, %%globaltimer;\" : \"=l\"(t1)); printf(\"AFDLIFE attention_combine cta %d ns %llu\\n\", (int)blockIdx.x, t1 - afd_t0); } // SCRATCH", " }", " } // deep_ep::elastic", "command terminated with exit code 1" ], "reading": "The pre-suites tree's receive CTAs 1..23 live ~0.6 us. On the candidate digest a11fe415 a one-CTA receive was slower at one token in 6 of 8 pokprod repeats (median +0.22 us) and 3 of 5 on statler (+0.04 us), faster at 8; scratch builds with extra host switches (same device code) were slower at one token in 16 of 16 (+0.3-0.4 us). The earlier '22 of 24' pooled those builds. Cooperative and two-CTA clustered launches recovered little. No mechanism found; the one-domain suites keep the shared shapes." }, "benchmark_limits": "Two Buffers in one process fault with an illegal address on this DeepEP build (the pre-suites head and Masked too), and two graphs captured on one Buffer corrupt each other, so every A/B is paired across fresh processes.", "scope": "communication only; synthetic GEMM load; no serving claim" }