Hansung Kim
03c61d72ff
sgemm_impl: Add param to load accumulation tile in single_tile
2024-08-19 18:08:58 -07:00
Hansung Kim
134ba825de
sgemm_impl: Fix typo bug for BK_adjusted
2024-08-19 18:02:00 -07:00
Hansung Kim
7ac038fadf
sgemm_impl: Rename initialize_C
2024-08-19 16:12:35 -07:00
Hansung Kim
4aba018733
sgemm_impl: Fix wrong barrier count; add barrier for write_to_smem
2024-08-19 15:33:23 -07:00
Hansung Kim
e93e54cdec
sgemm_impl: Drop volatile quanitifier
...
doesn't seem to do much & creates excessive type errors.
2024-08-19 15:19:53 -07:00
Hansung Kim
42ddb9a48e
sgemm_impl: Accept layout template param at gemm_single_tile and wmma_load
2024-08-19 13:16:51 -07:00
Hansung Kim
1b133e7b5c
sgemm_impl: Rename dmem load function
2024-08-18 22:26:49 -07:00
Hansung Kim
46b5047775
sgemm_impl: Remove GMEM_COALESCED_A option
...
Uncoalesced GMEM accesses is verified to yield slow performance and the
relevant code is not used anymore; remove the cruft
2024-08-18 22:26:02 -07:00
Hansung Kim
04643fa64d
sgemm_impl: Refactor dmem_load into one unified logic
...
Replace the confusing logic that had slightly different use of BM/BN/BK
for A and B, into one logic that accepts matrix memory layout as a
proper argument & does compile-time logic to determine the right
dimensions.
TODO: !GMEM_COALESCED_A is not updated yet
2024-08-18 22:05:22 -07:00
Hansung Kim
b44b202a21
sgemm_impl: Rename to wmma
2024-08-18 16:21:22 -07:00
Hansung Kim
b978bf8757
sgemm_impl: Split tile offset addr gen from wmma store
...
& add an option to write to smem in gemm_single_tile.
2024-08-18 16:10:29 -07:00
Hansung Kim
d0809d292a
sgemm: Specify A/B tile SMEM address via template args
...
& split single-time GEMM into a separate function.
2024-08-16 18:01:57 -07:00
Hansung Kim
64b9717064
sgemm_tcore: Remove duplicate float_type decl
2024-08-16 16:26:18 -07:00
Hansung Kim
a1858e0c80
sgemm_impl: Parameterize BK/TCK by FP_SIZE
2024-08-15 20:33:33 -07:00
Hansung Kim
014f7cd06f
sgemm_tcore: Unpack arg params, remove threadblock_dim_y
...
thread_block_gemm is meant to be reusable, so it shouldn't assume what
the kernel arg struct looks like.
threadblock_dim_y was ambiguous and didn't match the literal name either
(it was used as # of warps that participate in a barrier).
2024-08-14 20:34:49 -07:00
Hansung Kim
70919c39c9
Encode dependency to sgemm header in makefile
2024-08-14 20:03:07 -07:00
Hansung Kim
1b1264207b
sgemm_tcore: Add compile-time write_to_gmem param to thread_block_gemm
2024-08-14 17:48:31 -07:00
Hansung Kim
ee6339a35f
sgemm_tcore: Split all impl code into sgemm_impl.hpp
...
This is to make thread_block_gemm a re-usable library function for GEMM
operations for use in other kernels.
2024-08-14 16:24:48 -07:00
Hansung Kim
0534e5d1f6
sgemm_tcore: Fix addr gen for GMEM->SMEM for M-major A
...
This fixes correctness for TRANSPOSE_AT_PRODUCE/COLUMN=0/0, provided the
matrices are already stored in the correct layout in GMEM.
2024-08-14 15:35:35 -07:00
Hansung Kim
409424b032
sgemm_tcore: Fix fp16 addr gen in vx_wmma_load
2024-08-14 13:48:03 -07:00
Hansung Kim
e69fbea83a
sgemm_tcore: Fix casting error
2024-08-12 17:57:50 -07:00
Hansung Kim
95e3e96c6c
tensor: Change B in-memory layout to column-major
2024-08-12 15:22:07 -07:00
Hansung Kim
88cddc2b66
sgemm_tcore: Support data move for fp16-packed elements
...
Since core does not support memory accesses to non-word-aligned
addresses, pack fp16 elements in pairs into fp32 values, and do regular
tile movement with conditionally compressed column dimensions.
Perf seems to stay the same for fp32 256x256.
2024-07-30 21:43:10 -07:00
Hansung Kim
7f26548724
sgemm_tcore: Fix mem addr stride to 4
...
Otherwise incurs misaligned accesses not supported in lsu.
2024-07-30 14:06:46 -07:00
Hansung Kim
5d5a6fbad2
sgemm_tcore: Template-ize kernel code
2024-07-29 20:11:51 -07:00
Hansung Kim
5f342914bd
sgemm_tcore: Support fp16 input generation in host code
2024-07-29 17:18:35 -07:00
Hansung Kim
bca53a9c76
sgemm_tcore: Skip load at last k-iter; do DMA by default
2024-07-19 16:37:51 -07:00
Hansung Kim
1f844fa9e9
Set BM==BN==64, update doc
2024-07-19 16:37:15 -07:00
Hansung Kim
bebdd3353e
Use SWISH in activate_block for tcore and gemmini
2024-06-19 15:41:50 -07:00
Hansung Kim
ae9e707280
sgemm_{gemmini_dma,tcore}: Separate activate_block
2024-06-19 14:50:22 -07:00
Hansung Kim
50b843d8c4
sgemm_tcore: Fix address overlap for DMA
...
Enforce square shapes of tiles in smem. TODO need to configure loop
bounds correctly.
2024-06-18 15:06:07 -07:00
Hansung Kim
36b02ad595
sgemm_tcore: Add warp-specialized kernel with activations
...
FIXME; only tested with WARP_SPECIALIZED == 0.
2024-06-17 19:14:33 -07:00
Hansung Kim
85cace9524
sgemm_tcore: Fix smem allocation for non-dma
2024-06-15 01:28:27 -07:00
Hansung Kim
cfb6ae4a91
sgemm_tcore: Fix wrong double-buf addr for wmma_load
2024-06-15 00:51:35 -07:00
Hansung Kim
9d6ff196b3
sgemm_tcore: Use old opcodes to match frozen rtl
2024-06-15 00:26:57 -07:00
Hansung Kim
ce4f3a24e3
sgemm_tcore: Replace hardcoded NUM_LANES with NUM_THREADS
2024-06-12 21:01:37 -07:00
Hansung Kim
21452661f2
sgemm_tcore: Fix double-buffered addr for GEMMINI_DMA
2024-06-12 13:36:29 -07:00
Hansung Kim
635da96154
sgemm_tcore: Constify smem pointer for wmma_load
2024-06-12 13:36:29 -07:00
Hansung Kim
32e31c51a4
sgemm_tcore: Blocksize 64; Fix kernel launch on larger dim
...
& fix addrgen assembly too large offset error
2024-06-11 22:27:12 -07:00
Hansung Kim
03d1df8f53
sgemm_tcore: Separate transpose control on AS read/write
...
Make separate control flags on transposed AS read/write to make it easy
to model bank-conflict-free GMEM _and_ SMEM access.
2024-06-11 21:16:23 -07:00
Hansung Kim
34eaab4c87
sgemm_tcore: Fix warp-specialized kernel for larger dim
2024-06-11 20:50:32 -07:00
Hansung Kim
9febfb9bdc
sgemm_tcore: Move global_dmem_load back to kernel.cpp
2024-06-11 20:12:30 -07:00
Hansung Kim
ca7fd84a83
sgemm_tcore: Split util functions to a header file
2024-06-11 19:06:22 -07:00
Hansung Kim
dab9d7c6fc
sgemm_tcore: Fix kernel launch for smaller TBs than cluster threads
...
E.g. bm32bn32bk32wm16wn8
2024-06-11 14:09:31 -07:00
Hansung Kim
e3c4a4d2f5
sgemm_tcore: Improve agen for !transpose_as smem load
2024-06-10 22:08:37 -07:00
Hansung Kim
dc7bd6b248
sgemm_tcore: Fix warp_row/col calculation bug
2024-06-10 19:52:37 -07:00
Hansung Kim
3b2f5a31de
sgemm_tcore: Improve write_result addr gen
2024-06-10 19:34:00 -07:00
Hansung Kim
a22762db94
sgemm_tcore: Add GEMMINI_DMA to non-warp-specialized mode
...
~63% util for 128x128; ~83% for the k-loop.
FIXME: result is not correct currently. Need to fix the transpose
2024-06-10 19:32:39 -07:00
Hansung Kim
39449ece37
Add warp-specialized{, and dma enabled} kernel
...
NOTE: warpspecial_dma is hacked, need to get rid of dma invocation in
the consmer code.
2024-06-10 16:39:49 -07:00
Hansung Kim
90662484b1
sgemm_tcore: inline left out
2024-06-07 20:34:29 -07:00
