80 Commits

Author	SHA1	Message	Date
Hansung Kim	1c9b022156	flash: Rename nowarpspec to default	2024-11-09 19:58:45 -08:00
Hansung Kim	8fe6d918f2	flash: Update tcore kernel to use new CISC	2024-11-09 19:49:20 -08:00
Hansung Kim	76a6aaf085	flash: doc update	2024-11-09 19:09:09 -08:00
Hansung Kim	673e07ed43	flash: Add non-warp-specialized gemmini flash kernel	2024-11-09 19:08:39 -08:00
Hansung Kim	ad75561efe	flash: Reduce fence calls to improve util	2024-11-09 16:44:17 -08:00
Hansung Kim	952b8debbb	flash: Update to use new CISC interface	2024-11-09 16:21:34 -08:00
Hansung Kim	365b1d8e67	flash: Add begin end markers	2024-11-09 10:19:16 -08:00
Hansung Kim	1e3d476e70	Switch header configs to flash	2024-11-08 21:56:42 -08:00
Hansung Kim	4e087a8aab	flash: Fix loop iteration for gemmini ... Kernel is software-pipelined around 2 GEMMs and softmax; it requires two iterations to fully complete a tile.	2024-11-08 16:43:08 -08:00
Hansung Kim	4055255018	flash: Fix tcore kernel for CISC arg field changes	2024-11-08 16:40:16 -08:00
Hansung Kim	221d5f75c2	flash: Optimize smem alloc for tcore for 8banks ... Divide into first half & last half for warpgroup 0 & 1, and allocate Q/K and P/V in different banks for parallel acccess.	2024-09-19 21:31:39 -07:00
Hansung Kim	d0ef06cec1	flash: Complete Q_IS_K_MAJOR code for GEMM II	2024-09-19 20:36:03 -07:00
Hansung Kim	be15cffbf3	flash: Revert to gemmini config, remove DEBUG and unnecessary checks	2024-09-12 14:25:33 -07:00
Hansung Kim	b5916f3f07	flash: Fix hardcoded barrier for tcore; move tcore-specific flags	2024-09-11 22:08:44 -07:00
Hansung Kim	d69707f686	flash: Enable GEMM II fence; Pull 1st KV move out of the loop	2024-09-11 19:24:06 -07:00
Hansung Kim	18cf0e73cd	flash: Add early return for warp-indivisible row iter	2024-09-11 00:56:09 -07:00
Hansung Kim	068d48534e	flash: Swap S1/S0 to avoid GEMM II - softmax bank conflict ... + remove spurrious fences to better overlap GEMM I and DMA	2024-09-11 00:55:36 -07:00
Hansung Kim	dc746272fb	flash: Conditionally enable GEMM II fence code, fix tile_k for DEBUG	2024-09-10 22:53:35 -07:00
Hansung Kim	90e03894fc	flash: Add flag in SMEM for dependency check on O ... TODO: results unverified. Stalls O rescale until GEMM II finishes.	2024-09-10 13:42:47 -07:00
Hansung Kim	88760596cb	flash: Remove bogus mvout to SMEM code	2024-09-09 17:18:59 -07:00
Hansung Kim	a17edac875	flash: Fix barrier stall with DEBUG ... Verified for up to P_expected on 2nd iter; O_before_PV is partially correct	2024-09-09 17:02:05 -07:00
Hansung Kim	b652e25945	flash: Warp-specialize between warp 0 and 1-7 ... Finishes without stalls; No dependency check between O rescale and GEMM-II.	2024-09-09 16:42:30 -07:00
Hansung Kim	d31c8ffd7d	flash: Fix grid size to hw cluster size ... Verified fast config, minus the barrier stall at the end.	2024-09-09 15:44:03 -07:00
Hansung Kim	829af5d429	flash: Comment out mvout to smem ... Verified up to O_before_PV; still stalls without DEBUG	2024-09-09 15:21:49 -07:00
Hansung Kim	ecc800964a	flash: Change smem alloc for less bank conflicts; noskip stc	2024-09-09 13:47:18 -07:00
Hansung Kim	714b9f501e	flash: Restructure to do delayed fence for better concurrency ... Verified up to O_before_PV of 2nd iteration; O_after_PV needs preload fix. FIXME: Stalls at barrier without DEBUG set.	2024-09-08 22:16:17 -07:00
Hansung Kim	6911843a82	flash: Remove unnecessary dmem preload, fix rowmax/rowsum dependency	2024-09-08 21:11:59 -07:00
Hansung Kim	a4dd45bc1b	flash: Replace CISC with RISC ... spadQuartile in hw does not match spad addresses in kernel; match them later for optimization.	2024-09-08 20:52:28 -07:00
Hansung Kim	6547e92757	flash: Load Q to both quartiles; preload O for acc	2024-09-08 19:47:55 -07:00
Hansung Kim	8efa6868ea	flash: Restructure for full software pipelining ... Verified up to P and O before PV; need to fix iteration for V load.	2024-09-08 18:45:32 -07:00
Hansung Kim	cdb8377b62	flash: Do GEMM II in Gemmini; verify 1st iteration	2024-09-08 16:09:06 -07:00
Hansung Kim	3f50ac57ee	flash: use 12bit dma interface	2024-09-08 15:29:56 -07:00
Hansung Kim	c51dc4902d	flash: Fix online softmax for DMA layout	2024-09-07 23:21:28 -07:00
Hansung Kim	2e1485877d	flash: Add Gemmini-accelerated kernel	2024-09-07 22:40:58 -07:00
Hansung Kim	b3be271b88	flash: Split impl to header file	2024-09-07 21:16:35 -07:00
Hansung Kim	03308f8033	flash: Write fast config for DMA ... MAC utilization is 20-25% for the loop.	2024-09-07 20:47:24 -07:00
Hansung Kim	8d32a03d09	flash: Write DMA code for warp-specialized ... TODO: result unverified	2024-09-07 20:32:08 -07:00
Hansung Kim	33bc084c37	flash: Fix DMA layout for GEMM II	2024-09-07 19:51:05 -07:00
Hansung Kim	e02892ab7d	flash: Fix DMA for up to GEMM II ... yeah	2024-09-07 19:31:41 -07:00
Hansung Kim	d2f086344d	flash: Fix DMA addr stride, stop at S=Q*K	2024-09-07 15:48:37 -07:00
Hansung Kim	137df9bee2	WIP: flash: Use Gemmini DMA ... Paused by the barrier bug in warp-divergent branches (tmask not being considered)	2024-09-05 16:23:32 -07:00
Hansung Kim	bfb414c4eb	flash: Add DMA config logic	2024-09-05 16:23:32 -07:00
Hansung Kim	f028a97f75	sgemm_tcore: Verify wo DMA; warn untested against K-major A + DMA	2024-09-03 14:42:19 -07:00
Hansung Kim	70273fd00d	flash: Cleanup debug code	2024-09-02 00:40:05 -07:00
Hansung Kim	8125192846	flash: Specify leading_dim for split QK GEMM; fix uninit'd RF before GEMM	2024-09-02 00:15:57 -07:00
Hansung Kim	602fe4a400	flash: Change timing for QKV move ... Verified with warp_specialized false; true remains to be fixed.	2024-09-01 22:06:46 -07:00
Hansung Kim	aea257349a	flash: Correct schedule with inter-warpgroup barriers	2024-09-01 20:40:51 -07:00
Hansung Kim	e5e65312d2	flash: Restructure to inter-warpgroup parallelism ... This is similar to https://tridao.me/blog/2024/flash3/#inter-warpgroup-overlapping-with-pingpong-scheduling	2024-09-01 19:58:33 -07:00
Hansung Kim	6cc1b5ca37	flash: Reduce smem_scratchpad alloc size	2024-09-01 16:02:06 -07:00
Hansung Kim	817cc9a5a5	flash: Fix overlap in smem alloc for P tile	2024-08-31 16:24:28 -07:00