harald/nixcfg
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

fix(rag): send explicit encoding_format to avoid llama.cpp null error

Browse source 
When encoding_format is unset, LiteLLM forwards it to the backend as JSON
null, and llama.cpp's embeddings endpoint rejects it with a 500
("type must be string, but is null"). Pin encoding_format="float" so the
gateway always relays a string.
This commit is contained in:
Harald Hoyer 2026-05-22 08:34:42 +02:00
parent f0fe1d5b27
commit 6fd6060dd7
1 changed files with 3 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

4
packages/rag/default.nix
View file

@ -34,7 +34,9 @@ writers.writePython3Bin "rag"
def embed(texts): def embed(texts):
resp = client.embeddings.create(model=EMBED_MODEL, input=texts) # encoding_format is explicit: llama.cpp rejects a null value, and
# LiteLLM forwards an unset one as JSON null.
resp = client.embeddings.create(model=EMBED_MODEL, input=texts, encoding_format="float")
return [d.embedding for d in resp.data] return [d.embedding for d in resp.data]