Hansung Kim
8dadbdd42d
tensor: Do DMA mvin for next m/n loop at the last k iter
...
This increases util by pulling the DMA wait time out of the K-loop
wraparound (next N) and overlapping it with the last K iter.
2024-10-29 19:43:22 -07:00
Hansung Kim
367fa927f8
sgemm_impl: Fix default FP_SIZE to 16
2024-10-29 14:56:18 -07:00
Hansung Kim
24064dc7a2
sgemm_impl: Do proper addr gen and store for wgmma
2024-10-29 01:31:55 -07:00
Hansung Kim
bd7a8e39b9
sgemm_impl: Split out smem addr gen to functions
...
so that the addr gen code can also be used for wgmma.
2024-10-29 01:30:48 -07:00
Hansung Kim
e55c8b480e
sgemm_impl: Comment out GEMMINI_DMA code in single_tile
...
This is already done in the higher-level thread_block_gemm function, and
flash also has explicit DMA sync code. Also having this executed twice
sometimes triggers vx_bar movement into a branch which we really want to
avoid.
2024-10-28 12:47:49 -07:00
Hansung Kim
36eb50060f
sgemm_impl: Add skeleton wgmma routine for single_tile
2024-10-28 12:47:20 -07:00
Hansung Kim
6417a625b1
sgemm_impl: Add tiling params for hopper tensor core
2024-10-23 19:50:18 -07:00
Hansung Kim
68cd6455fe
sgemm_impl: Add mmio reconverge barrier to avoid slip-off; switch to FP32
2024-10-02 15:17:44 -07:00
Hansung Kim
ba66d2c2bd
sgemm_impl: barrier dumb dumb
2024-09-11 00:01:56 -07:00
Hansung Kim
2152c80ffd
sgemm_impl: Add missing reconvergence barrier after mmio
2024-09-10 18:05:01 -07:00
Hansung Kim
ccddd0bcc9
sgemm_impl: Remove unused FLEXIBLE_LAYOUT
2024-09-10 15:54:17 -07:00
Hansung Kim
d31c8ffd7d
flash: Fix grid size to hw cluster size
...
Verified fast config, minus the barrier stall at the end.
2024-09-09 15:44:03 -07:00
Hansung Kim
1f51f7f9d4
sgemm_impl: Mark threadblock_barrier convergent
...
Thank you Chris Lattner
2024-09-08 22:49:38 -07:00
Hansung Kim
8efa6868ea
flash: Restructure for full software pipelining
...
Verified up to P and O before PV; need to fix iteration for V load.
2024-09-08 18:45:32 -07:00
Hansung Kim
30a4d15cde
sgemm_impl: Parameterize BM on NUM_CORES
2024-09-08 15:29:15 -07:00
Hansung Kim
443a37be6c
sgemm_impl: Add DMA_FAST option; fix dbuf offset for dma
2024-09-08 14:58:41 -07:00
Hansung Kim
42913c00c4
sgemm_impl: Use 12-bit cmd interface, allow DIM=16
2024-09-08 14:28:27 -07:00
Hansung Kim
adcd0a9d49
sgemm_impl: Fix wrong smem address for fp16
...
Verified results for fp16 256x256.
2024-09-08 02:23:51 -07:00
Hansung Kim
33bc084c37
flash: Fix DMA layout for GEMM II
2024-09-07 19:51:05 -07:00
Hansung Kim
a967c262b1
sgemm_impl: Add new block-row-major layout for DMA
2024-09-07 16:38:22 -07:00
Hansung Kim
d2f086344d
flash: Fix DMA addr stride, stop at S=Q*K
2024-09-07 15:48:37 -07:00
Hansung Kim
9f067acdb9
sgemm_impl: Remove #if 0, FP_SIZE 16
2024-09-05 19:55:36 -07:00
Hansung Kim
a832fa7b84
sgemm_impl: 128x64 tile; fix unrolled asm, comment out actual gemm
2024-09-05 16:23:32 -07:00
Hansung Kim
81924b601a
sgemm_impl: Rewrite tile param constraint
2024-09-05 16:23:32 -07:00
Hansung Kim
ced98a6ff4
sgemm_impl: Refactor DMA layout remap logic into constexpr func
2024-09-03 16:20:31 -07:00
Hansung Kim
58fa2a3e91
sgemm_impl: Switch for allowing MN-major with DMA
2024-09-03 15:12:58 -07:00
Hansung Kim
f028a97f75
sgemm_tcore: Verify wo DMA; warn untested against K-major A + DMA
2024-09-03 14:42:19 -07:00
Hansung Kim
7aa0e6cbe4
sgemm_tcore: Fix correctness for GEMMINI_DMA
...
Remap the logical SMEM row/col coordinates to the DMA's two-level
block-row-major layout.
2024-09-02 23:46:50 -07:00
Hansung Kim
dd1b408f56
sgemm_tcore: Add debug mode with tile copy-out
2024-09-02 21:55:55 -07:00
Hansung Kim
9d71fa44a7
sgemm_tcore: Fix invocation with compile time threadblock size
2024-09-02 17:03:46 -07:00
Hansung Kim
bdd955836d
sgemm_impl: Specify leading dimension to wmma load
...
This is necessary for when loading a subtile from a full tile in SMEM
into RF, but that subtile is split by non-major dimension.
2024-09-02 00:14:35 -07:00
Hansung Kim
ee0295cbef
sgemm_impl: Accept threads_per_threadblock in load_tile_to_smem
...
Needed for warp-specialized kernels.
2024-08-29 21:43:57 -07:00
Hansung Kim
091f40c365
sgemm_impl: Parameterize BM/BN/BK in single_tile
2024-08-20 19:41:34 -07:00
Hansung Kim
526c2bd334
sgemm_impl: load_tile: accept k_index for consistency + fix gmem addr gen
2024-08-20 17:46:35 -07:00
Hansung Kim
03c61d72ff
sgemm_impl: Add param to load accumulation tile in single_tile
2024-08-19 18:08:58 -07:00
Hansung Kim
134ba825de
sgemm_impl: Fix typo bug for BK_adjusted
2024-08-19 18:02:00 -07:00
Hansung Kim
7ac038fadf
sgemm_impl: Rename initialize_C
2024-08-19 16:12:35 -07:00
Hansung Kim
4aba018733
sgemm_impl: Fix wrong barrier count; add barrier for write_to_smem
2024-08-19 15:33:23 -07:00
Hansung Kim
e93e54cdec
sgemm_impl: Drop volatile quanitifier
...
doesn't seem to do much & creates excessive type errors.
2024-08-19 15:19:53 -07:00
Hansung Kim
42ddb9a48e
sgemm_impl: Accept layout template param at gemm_single_tile and wmma_load
2024-08-19 13:16:51 -07:00
Hansung Kim
1b133e7b5c
sgemm_impl: Rename dmem load function
2024-08-18 22:26:49 -07:00
Hansung Kim
46b5047775
sgemm_impl: Remove GMEM_COALESCED_A option
...
Uncoalesced GMEM accesses is verified to yield slow performance and the
relevant code is not used anymore; remove the cruft
2024-08-18 22:26:02 -07:00
Hansung Kim
04643fa64d
sgemm_impl: Refactor dmem_load into one unified logic
...
Replace the confusing logic that had slightly different use of BM/BN/BK
for A and B, into one logic that accepts matrix memory layout as a
proper argument & does compile-time logic to determine the right
dimensions.
TODO: !GMEM_COALESCED_A is not updated yet
2024-08-18 22:05:22 -07:00
Hansung Kim
b44b202a21
sgemm_impl: Rename to wmma
2024-08-18 16:21:22 -07:00
Hansung Kim
b978bf8757
sgemm_impl: Split tile offset addr gen from wmma store
...
& add an option to write to smem in gemm_single_tile.
2024-08-18 16:10:29 -07:00
Hansung Kim
d0809d292a
sgemm: Specify A/B tile SMEM address via template args
...
& split single-time GEMM into a separate function.
2024-08-16 18:01:57 -07:00
Hansung Kim
a1858e0c80
sgemm_impl: Parameterize BK/TCK by FP_SIZE
2024-08-15 20:33:33 -07:00
Hansung Kim
014f7cd06f
sgemm_tcore: Unpack arg params, remove threadblock_dim_y
...
thread_block_gemm is meant to be reusable, so it shouldn't assume what
the kernel arg struct looks like.
threadblock_dim_y was ambiguous and didn't match the literal name either
(it was used as # of warps that participate in a barrier).
2024-08-14 20:34:49 -07:00
Hansung Kim
1b1264207b
sgemm_tcore: Add compile-time write_to_gmem param to thread_block_gemm
2024-08-14 17:48:31 -07:00
Hansung Kim
ee6339a35f
sgemm_tcore: Split all impl code into sgemm_impl.hpp
...
This is to make thread_block_gemm a re-usable library function for GEMM
operations for use in other kernels.
2024-08-14 16:24:48 -07:00
