extern/whisper.cpp
1
0
Fork 1
mirror of https://github.com/ggerganov/whisper.cpp.git synced 2025-08-14 08:38:43 +02:00
Code Issues Packages Projects Releases Wiki Activity

talk-llama : only copy used KV cache in get / set state (#890)

Browse Source 
---------

Co-authored-by: ejones <evan.q.jones@gmail.com>
This commit is contained in:
Luis Herrera
2023-05-08 12:59:21 -05:00
committed by GitHub
parent 9931d66400
commit 4e4d00c67a
2 changed files with 81 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
examples/talk-llama/llama.h
View File

@ -23,7 +23,7 @@
#define LLAMA_FILE_MAGIC 'ggjt'
#define LLAMA_FILE_MAGIC_UNVERSIONED 'ggml'
#define LLAMA_SESSION_MAGIC 'ggsn'
#define LLAMA_SESSION_VERSION 0
#define LLAMA_SESSION_VERSION 1
#ifdef __cplusplus
extern "C" {
@ -127,7 +127,8 @@ extern "C" {
// Sets the current rng seed.
LLAMA_API void llama_set_rng_seed(struct llama_context * ctx, int seed);
// Returns the size in bytes of the state (rng, logits, embedding and kv_cache)
// Returns the maximum size in bytes of the state (rng, logits, embedding
// and kv_cache) - will often be smaller after compacting tokens
LLAMA_API size_t llama_get_state_size(struct llama_context * ctx);
// Copies the state to the specified destination address.