Instructions to use ChaoticNeutrals/Kunocchini-7b-128k-test with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ChaoticNeutrals/Kunocchini-7b-128k-test with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ChaoticNeutrals/Kunocchini-7b-128k-test", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ChaoticNeutrals/Kunocchini-7b-128k-test", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("ChaoticNeutrals/Kunocchini-7b-128k-test", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ChaoticNeutrals/Kunocchini-7b-128k-test with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ChaoticNeutrals/Kunocchini-7b-128k-test"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ChaoticNeutrals/Kunocchini-7b-128k-test",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ChaoticNeutrals/Kunocchini-7b-128k-test

SGLang

How to use ChaoticNeutrals/Kunocchini-7b-128k-test with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ChaoticNeutrals/Kunocchini-7b-128k-test" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ChaoticNeutrals/Kunocchini-7b-128k-test",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ChaoticNeutrals/Kunocchini-7b-128k-test" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ChaoticNeutrals/Kunocchini-7b-128k-test",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ChaoticNeutrals/Kunocchini-7b-128k-test with Docker Model Runner:
```
docker model run hf.co/ChaoticNeutrals/Kunocchini-7b-128k-test
```

Kunoichi with long legs!

by Spacellary - opened Feb 17, 2024

Discussion

Spacellary

Feb 17, 2024

•

edited Feb 17, 2024

Keep up the good work, I feel like it does longer contexts better than the original, not sure about what I'm sacrificing for it yet, but it's a positive first impression.

Looking forward especially if you're cooking with Kunoichi again.

Nitral-AI

The Chaotic Neutrals org Feb 18, 2024

•

edited Feb 18, 2024

Honestly i need to graph out perplexity between the original and extended. But thank you for the feedback! it was bit weird to get these models to merge at all, however i think it can be improved further.

Spacellary

Feb 18, 2024

@Test157t Absolutely, that'd be nice.

Lewdiculous

Feb 23, 2024

•

edited Feb 23, 2024

@Test157t - Honestly there's something about this model that just jives with my character cards so well. It's a special good boy. Couple other people I recommended it to also were positive on it. It just feels "consistent".

Nitral-AI

The Chaotic Neutrals org Feb 23, 2024

At this rate i might have to do a v2 for this one.

Lewdiculous

Feb 25, 2024

@Test157t -- Booyah! Not sure it's worth testing so late but I'm uploading the GGUF-Imatrix quants from the F16 for this model here, including the new IQ3_S that should be compatible with the next version of KCPP.

Lewdiculous

Feb 28, 2024

•

edited Feb 28, 2024

@Test157t -- Already uploading v2 quants from the new config to the same repo, if no major issues are found than this model should be good on Quants, the new ones are tagged with v2 in the file names for easy distinction. If needed I'll do a v3 or whatever as necessary. Love me some longer context :')

Nitral-AI

The Chaotic Neutrals org Feb 28, 2024

@Lewdiculous Thank you! the long text version (1.2) is still "regarded" so im going have to go back to the drawing board with that one. Been testing lelantacles v5 for comparison and it kicks ass at 16k still.

Lewdiculous

Feb 28, 2024

Been testing lelantacles v5 for comparison and it kicks ass at 16k still

In a way it was progress :)

Will hold off until 1.2 or more realistically the next version entirely is cleared for testing then. It's all about experimenting, cheers!

Lewdiculous

Feb 28, 2024

•

edited Feb 28, 2024

Fun times, I look outside and there's already IQ4_NL and now a new IQ4_XS merged 11 hours ago. GGUF moving fast xD

Nitral-AI

The Chaotic Neutrals org Feb 28, 2024

•

edited Feb 28, 2024

Might redo 1.2 without normalizing and in slerp instead of dare-ties. but i will test it locally along with the quants before i clear it for use.

Nitral-AI

The Chaotic Neutrals org Feb 28, 2024

Oh it would appear that ive missed that, im falling behind it would seem.

Lewdiculous

Feb 28, 2024

•

edited Feb 28, 2024

@Test157t, yeah, things are moving quite fast in GGUF land now.

IQ4_
NL: https://github.com/ggerganov/llama.cpp/pull/5590
XS: https://github.com/ggerganov/llama.cpp/pull/5747

Honestly with the very small loss in quality it's much more interesting for me to just push for higher context, I'm staying around the same VRAM usage, but with 4K more context.

Need to see how it goes, maybe you can do real Bbenchmarks.

I've uploaded these 2 news quant-types for Kunocchini-7b-128k-test-GGUF-Imatrix, they are in the v2 list.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment