extern/whisper.cpp
1
0
Fork 1
mirror of https://github.com/ggerganov/whisper.cpp.git synced 2025-06-01 07:25:49 +02:00
Code Issues Packages Projects Releases Wiki Activity

metal : optimize multi-sequence FA vec kernel (llama/13493)

Browse Source 
* batched-bench : fix pp batch contents

* metal : optimize multi-sequence FA vec kernel

ggml-ci
This commit is contained in:
Georgi Gerganov 2025-05-13 18:04:00 +03:00
parent a77a924b20
commit e11fc21e6c
1 changed files with 5 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
ggml/src/ggml-metal/ggml-metal.metal
View File

@ -3887,6 +3887,11 @@ kernel void kernel_flash_attn_ext_vec(
sm[tiisg] = pm[ic + tiisg]; sm[tiisg] = pm[ic + tiisg];
} }
// skip -INF blocks
if (simd_max(sm[tiisg]) == -INFINITY) {
continue;
}
// Q*K^T // Q*K^T
{ {
// each simdgroup processes 1 query and NE (NW/NL) head elements // each simdgroup processes 1 query and NE (NW/NL) head elements