-
1e3d476e70
Switch header configs to flash
Hansung Kim
2024-11-08 21:56:26 -08:00
-
c114a7a4ab
new gemm kernel
Richard Yan
2024-11-08 20:55:27 -08:00
-
4e087a8aab
flash: Fix loop iteration for gemmini
Hansung Kim
2024-11-08 16:43:08 -08:00
-
4055255018
flash: Fix tcore kernel for CISC arg field changes
Hansung Kim
2024-11-08 16:40:16 -08:00
-
c001618fb9
sgemm_impl: Fix wrong next block_m logic for DMA
Hansung Kim
2024-10-29 22:35:56 -07:00
-
21b6655c10
sgemm_impl: Implement fast coalesced wmma_store
Hansung Kim
2024-10-29 22:34:22 -07:00
-
6b39a6fe70
Add convenience script for switching input/args binaries
Hansung Kim
2024-10-29 20:14:33 -07:00
-
8dadbdd42d
tensor: Do DMA mvin for next m/n loop at the last k iter
Hansung Kim
2024-10-29 19:43:22 -07:00
-
-
367fa927f8
sgemm_impl: Fix default FP_SIZE to 16
Hansung Kim
2024-10-29 14:56:18 -07:00
-
e1b0fc3944
generate_matrix.py: Rand [0,1); also save non-swizzled row-major B
Hansung Kim
2024-10-29 14:55:32 -07:00
-
24064dc7a2
sgemm_impl: Do proper addr gen and store for wgmma
Hansung Kim
2024-10-29 01:31:55 -07:00
-
bd7a8e39b9
sgemm_impl: Split out smem addr gen to functions
Hansung Kim
2024-10-29 01:30:48 -07:00
-
ae98ae6e93
sgemm_tcore: Fix DMA smem addresses, add markers
Hansung Kim
2024-10-28 17:26:07 -07:00
-
b4dadfaf61
Merge remote-tracking branch 'origin/kernels' into kernels-hopper
Hansung Kim
2024-10-28 14:25:18 -07:00
-
-
fd1c9f4729
update gemmini dma kernel
Richard Yan
2024-10-28 13:47:13 -07:00
-
d0421426be
sgemm_tcore: Hardcode CISC spadQuartile addresses
Hansung Kim
2024-10-28 12:49:22 -07:00
-
e55c8b480e
sgemm_impl: Comment out GEMMINI_DMA code in single_tile
Hansung Kim
2024-10-28 12:47:49 -07:00
-
36eb50060f
sgemm_impl: Add skeleton wgmma routine for single_tile
Hansung Kim
2024-10-28 12:47:20 -07:00
-
e8a943e893
Push golden sgemm_tcore kernels
Hansung Kim
2024-10-27 19:40:08 -07:00
-
379d863456
Merge branch 'kernels' of https://github.com/hansungk/vortex-private into kernels
Richard Yan
2024-10-24 17:31:01 -07:00
-
-
fb928b5cda
new unaligned access kernel, update idle kernel
Richard Yan
2024-10-24 17:13:13 -07:00
-
8cc0c3bae4
fp16 no dma kernel
Richard Yan
2024-10-24 17:12:29 -07:00
-
04a64dee7a
fp16 dma kernel
Richard Yan
2024-10-24 17:12:02 -07:00
-
6417a625b1
sgemm_impl: Add tiling params for hopper tensor core
Hansung Kim
2024-10-23 19:50:18 -07:00
-
-
-
68cd6455fe
sgemm_impl: Add mmio reconverge barrier to avoid slip-off; switch to FP32
Hansung Kim
2024-10-02 15:17:44 -07:00
-
34d0956cd5
tensor: Attempt row-major mapping for C store (WIP)
Hansung Kim
2024-10-02 15:14:55 -07:00
-
3490294626
generate_matrix.py: switch to fp16 rand, generate row-major A
Hansung Kim
2024-10-02 11:01:23 -07:00
-
db2789bf23
Add asm label for cisc compute
Hansung Kim
2024-10-02 10:59:14 -07:00
-
6f6ee5616f
Add convergent attribute to vx_barrier
Hansung Kim
2024-10-02 10:57:45 -07:00
-
221d5f75c2
flash: Optimize smem alloc for tcore for 8banks
Hansung Kim
2024-09-19 21:31:11 -07:00
-
d0ef06cec1
flash: Complete Q_IS_K_MAJOR code for GEMM II
Hansung Kim
2024-09-19 20:36:03 -07:00
-
b9cafd6372
idle: unused const
Hansung Kim
2024-09-18 18:10:29 -07:00
-
be15cffbf3
flash: Revert to gemmini config, remove DEBUG and unnecessary checks
Hansung Kim
2024-09-12 14:25:33 -07:00
-
b5916f3f07
flash: Fix hardcoded barrier for tcore; move tcore-specific flags
Hansung Kim
2024-09-11 22:08:06 -07:00
-
d69707f686
flash: Enable GEMM II fence; Pull 1st KV move out of the loop
Hansung Kim
2024-09-11 19:24:06 -07:00
-
18cf0e73cd
flash: Add early return for warp-indivisible row iter
Hansung Kim
2024-09-11 00:56:09 -07:00
-
068d48534e
flash: Swap S1/S0 to avoid GEMM II - softmax bank conflict
Hansung Kim
2024-09-11 00:55:36 -07:00
-
ba66d2c2bd
sgemm_impl: barrier dumb dumb
Hansung Kim
2024-09-11 00:01:56 -07:00
-
dc746272fb
flash: Conditionally enable GEMM II fence code, fix tile_k for DEBUG
Hansung Kim
2024-09-10 22:53:35 -07:00
-
28b2eaec8f
sgemm_gemmini_dma: Fix tile size to (128,64,128)
Hansung Kim
2024-09-10 18:29:26 -07:00
-
2152c80ffd
sgemm_impl: Add missing reconvergence barrier after mmio
Hansung Kim
2024-09-10 18:05:01 -07:00
-
ccddd0bcc9
sgemm_impl: Remove unused FLEXIBLE_LAYOUT
Hansung Kim
2024-09-10 15:54:17 -07:00
-
90e03894fc
flash: Add flag in SMEM for dependency check on O
Hansung Kim
2024-09-10 13:37:32 -07:00
-
88760596cb
flash: Remove bogus mvout to SMEM code
Hansung Kim
2024-09-09 17:18:59 -07:00
-
a17edac875
flash: Fix barrier stall with DEBUG
Hansung Kim
2024-09-09 17:02:05 -07:00
-
b652e25945
flash: Warp-specialize between warp 0 and 1-7
Hansung Kim
2024-09-09 16:42:30 -07:00
-
d31c8ffd7d
flash: Fix grid size to hw cluster size
Hansung Kim
2024-09-09 15:43:31 -07:00
-
829af5d429
flash: Comment out mvout to smem
Hansung Kim
2024-09-09 15:21:49 -07:00
-
ecc800964a
flash: Change smem alloc for less bank conflicts; noskip stc
Hansung Kim
2024-09-09 13:47:18 -07:00
-
1f51f7f9d4
sgemm_impl: Mark threadblock_barrier convergent
Hansung Kim
2024-09-08 22:49:38 -07:00
-
714b9f501e
flash: Restructure to do delayed fence for better concurrency
Hansung Kim
2024-09-08 22:06:49 -07:00
-
6911843a82
flash: Remove unnecessary dmem preload, fix rowmax/rowsum dependency
Hansung Kim
2024-09-08 21:11:59 -07:00
-
a4dd45bc1b
flash: Replace CISC with RISC
Hansung Kim
2024-09-08 20:52:28 -07:00
-
6547e92757
flash: Load Q to both quartiles; preload O for acc
Hansung Kim
2024-09-08 19:47:55 -07:00
-
8efa6868ea
flash: Restructure for full software pipelining
Hansung Kim
2024-09-08 18:45:32 -07:00
-
cdb8377b62
flash: Do GEMM II in Gemmini; verify 1st iteration
Hansung Kim
2024-09-08 16:09:06 -07:00
-
3f50ac57ee
flash: use 12bit dma interface
Hansung Kim
2024-09-08 15:29:46 -07:00
-
30a4d15cde
sgemm_impl: Parameterize BM on NUM_CORES
Hansung Kim
2024-09-08 15:29:15 -07:00
-
443a37be6c
sgemm_impl: Add DMA_FAST option; fix dbuf offset for dma
Hansung Kim
2024-09-08 14:56:48 -07:00
-
42913c00c4
sgemm_impl: Use 12-bit cmd interface, allow DIM=16
Hansung Kim
2024-09-08 14:28:27 -07:00
-
adcd0a9d49
sgemm_impl: Fix wrong smem address for fp16
Hansung Kim
2024-09-08 02:23:51 -07:00
-
c51dc4902d
flash: Fix online softmax for DMA layout
Hansung Kim
2024-09-07 23:21:28 -07:00
-
2e1485877d
flash: Add Gemmini-accelerated kernel
Hansung Kim
2024-09-07 22:40:50 -07:00
-
b3be271b88
flash: Split impl to header file
Hansung Kim
2024-09-07 21:16:35 -07:00
-
03308f8033
flash: Write fast config for DMA
Hansung Kim
2024-09-07 20:46:58 -07:00
-
8d32a03d09
flash: Write DMA code for warp-specialized
Hansung Kim
2024-09-07 20:32:08 -07:00
-
33bc084c37
flash: Fix DMA layout for GEMM II
Hansung Kim
2024-09-07 19:50:04 -07:00
-
e02892ab7d
flash: Fix DMA for up to GEMM II
Hansung Kim
2024-09-07 17:49:37 -07:00
-
4d6cdeb00b
Fallback to 4 cores for flash
Hansung Kim
2024-09-07 17:40:49 -07:00
-
863e92a85e
generate_matrix.py: Default to range, fp32
Hansung Kim
2024-09-07 17:40:21 -07:00
-
a967c262b1
sgemm_impl: Add new block-row-major layout for DMA
Hansung Kim
2024-09-07 16:38:22 -07:00
-
ed9bf6f73e
common.mk: Switch to -Os to prevent branch code duplication
Hansung Kim
2024-09-07 15:49:19 -07:00
-
d2f086344d
flash: Fix DMA addr stride, stop at S=Q*K
Hansung Kim
2024-09-07 15:48:37 -07:00
-
9f067acdb9
sgemm_impl: Remove #if 0, FP_SIZE 16
Hansung Kim
2024-09-05 19:55:36 -07:00
-
-
a832fa7b84
sgemm_impl: 128x64 tile; fix unrolled asm, comment out actual gemm
Hansung Kim
2024-09-05 16:22:19 -07:00
-
137df9bee2
WIP: flash: Use Gemmini DMA
Hansung Kim
2024-09-04 15:58:18 -07:00
-
87a1c2bbfc
Cores per cluster 4 to 8
Hansung Kim
2024-09-05 14:35:47 -07:00
-
bde6f0ea2e
py: Write P_expected, don't rewrite vars
Hansung Kim
2024-09-04 23:35:52 -07:00
-
dcd69ea304
Increase SMEM size to 256KB
Hansung Kim
2024-09-04 23:34:48 -07:00
-
81924b601a
sgemm_impl: Rewrite tile param constraint
Hansung Kim
2024-09-04 23:27:31 -07:00
-
bfb414c4eb
flash: Add DMA config logic
Hansung Kim
2024-09-03 16:21:28 -07:00
-
741bb80fe8
Merge branch 'kernels' of https://github.com/hansungk/vortex-private into kernels
Richard Yan
2024-09-05 16:22:43 -07:00
-
-
dd3244fba0
large fp16 kernel
Richard Yan
2024-09-05 16:22:38 -07:00
-
ced98a6ff4
sgemm_impl: Refactor DMA layout remap logic into constexpr func
Hansung Kim
2024-09-03 16:20:31 -07:00
-
58fa2a3e91
sgemm_impl: Switch for allowing MN-major with DMA
Hansung Kim
2024-09-03 15:12:58 -07:00
-
f028a97f75
sgemm_tcore: Verify wo DMA; warn untested against K-major A + DMA
Hansung Kim
2024-09-03 14:42:19 -07:00
-
7aa0e6cbe4
sgemm_tcore: Fix correctness for GEMMINI_DMA
Hansung Kim
2024-09-02 23:46:50 -07:00
-
dd1b408f56
sgemm_tcore: Add debug mode with tile copy-out
Hansung Kim
2024-09-02 21:55:55 -07:00
-
9d71fa44a7
sgemm_tcore: Fix invocation with compile time threadblock size
Hansung Kim
2024-09-02 17:03:46 -07:00
-
70273fd00d
flash: Cleanup debug code
Hansung Kim
2024-09-02 00:40:05 -07:00
-
8125192846
flash: Specify leading_dim for split QK GEMM; fix uninit'd RF before GEMM
Hansung Kim
2024-09-02 00:15:57 -07:00
-
bdd955836d
sgemm_impl: Specify leading dimension to wmma load
Hansung Kim
2024-09-02 00:14:35 -07:00
-
602fe4a400
flash: Change timing for QKV move
Hansung Kim
2024-09-01 22:06:46 -07:00
-
aea257349a
flash: Correct schedule with inter-warpgroup barriers
Hansung Kim
2024-09-01 20:40:26 -07:00
-
e5e65312d2
flash: Restructure to inter-warpgroup parallelism
Hansung Kim
2024-09-01 19:58:33 -07:00
-
f7603b18d3
flash.py: Write V to file
Hansung Kim
2024-09-01 18:17:05 -07:00
-
6cc1b5ca37
flash: Reduce smem_scratchpad alloc size
Hansung Kim
2024-09-01 16:02:06 -07:00
-
817cc9a5a5
flash: Fix overlap in smem alloc for P tile
Hansung Kim
2024-08-31 15:18:14 -07:00
-
bdd6e6a9ce
flash: Double-buffer between online softmax and GEMM II
Hansung Kim
2024-08-30 22:47:55 -07:00
-
042b47ff19
flash: Restructure for warp-specialization
Hansung Kim
2024-08-30 21:49:53 -07:00