| | --- |
| | license: apache-2.0 |
| | language: |
| | - zh |
| | - en |
| | tags: |
| | - dream-coder |
| | - diffusion |
| | - dlm |
| | --- |
| | |
| | # Dream-Coder GGUF Q8_0 Quantization Guide |
| | |
| | This guide is specifically designed for GGUF Q8_0 quantization of the Dream-Coder v0-Instruct-7B model. |
| |
|
| | ## Quick Start |
| |
|
| | ### 1. Environment Setup |
| |
|
| | ```bash |
| | # 1. Clone and compile llama.cpp |
| | git clone https://github.com/ggerganov/llama.cpp |
| | cd llama.cpp |
| | make -j$(nproc) |
| | |
| | # 2. Install Python dependencies |
| | pip install transformers>=4.46.2 torch safetensors numpy |
| | ``` |
| |
|
| | ### 2. Execute Quantization |
| |
|
| | #### Method 1: Use the provided script |
| |
|
| | ```bash |
| | # Set llama.cpp path |
| | export LLAMA_CPP_PATH=/path/to/llama.cpp |
| | |
| | # Run quantization script |
| | ./quantize_example.sh |
| | ``` |
| |
|
| | #### Method 2: Manual execution |
| |
|
| | ```bash |
| | python quantize_dream_q8_0.py \ |
| | --model_path /path/to/Dream-Coder-v0-Instruct-7B \ |
| | --llama_cpp_path /path/to/llama.cpp \ |
| | --output_dir ./gguf_output \ |
| | --keep_f16 |
| | ``` |
| |
|
| | ### 3. Parameter Description |
| |
|
| | - `--model_path`: Dream-Coder model path (default: current directory) |
| | - `--llama_cpp_path`: llama.cpp project path (required) |
| | - `--output_dir`: Output directory (default: ./gguf_output) |
| | - `--keep_f16`: Keep F16 intermediate files |
| |
|
| | ## Architecture Adaptation |
| |
|
| | ### Dream-Coder Special Configuration Handling |
| |
|
| | This quantization script specifically handles the following special configurations of Dream-Coder: |
| |
|
| | 1. **Architecture Mapping**: DreamModel → LlamaForCausalLM (compatibility) |
| | 2. **Special Token IDs**: |
| | - `mask_token_id`: 151666 (critical diffusion token) |
| | - `bos_token_id`: 151665 |
| | - `eos_token_id`: 151643 |
| | - `pad_token_id`: 151643 |
| |
|
| | 3. **Model Parameters**: |
| | - Vocabulary size: 152,064 |
| | - Hidden dimension: 3,584 |
| | - Attention heads: 28 (4 key-value heads) |
| | - Layers: 28 |
| | - Context length: 32,768 |
| |
|
| | 4. **Diffusion Features**: |
| | - Preserve `mask_token_id` metadata |
| | - RoPE theta: 1,000,000.0 |
| | - Activation function: SiLU |
| |
|
| | ## Output Description |
| |
|
| | ### File Structure |
| | ``` |
| | gguf_output/ |
| | ├── dream-coder-7b-f16.gguf # F16 intermediate file (optionally kept) |
| | └── dream-coder-7b-q8_0.gguf # Final Q8_0 quantized file |
| | ``` |
| |
|
| | ### Performance Expectations |
| |
|
| | | Metric | Original (BF16) | Q8_0 | |
| | |--------|-----------------|------| |
| | | Memory Usage | ~14GB | ~6.7GB | |
| | | Inference Speed | 1.0x | 1.2-1.5x | |
| | | Precision Loss | 0% | <0.1% | |
| | |
| | ## Usage |
| | |
| | ### llama.cpp Command Line |
| | |
| | Since Dream-Coder is a diffusion-based model, you need to use the dedicated `llama-diffusion-cli` tool: |
| | |
| | ```bash |
| | # Basic usage |
| | ./llama.cpp/build/bin/llama-diffusion-cli \ |
| | -m gguf_output/dream-coder-7b-q8_0.gguf \ |
| | -p "def quicksort(arr):" \ |
| | -n 512 \ |
| | -c 2048 \ |
| | --diffusion-steps 128 |
| | |
| | # Advanced parameters |
| | ./llama.cpp/build/bin/llama-diffusion-cli \ |
| | -m gguf_output/dream-coder-7b-q8_0.gguf \ |
| | -p "Write a binary search function" \ |
| | -n 256 \ |
| | -c 2048 \ |
| | --temp 0.1 \ |
| | --top-p 0.95 \ |
| | --repeat-penalty 1.1 \ |
| | --diffusion-steps 128 \ |
| | --diffusion-algorithm 4 \ |
| | --diffusion-alg-temp 0.0 \ |
| | -t 8 |
| | |
| | # Visualize generation process |
| | ./llama.cpp/build/bin/llama-diffusion-cli \ |
| | -m gguf_output/dream-coder-7b-q8_0.gguf \ |
| | -p "def fibonacci(n):" \ |
| | -n 256 \ |
| | --diffusion-steps 64 \ |
| | --diffusion-visual |
| | ``` |
| | |
| | #### Diffusion Parameter Description |
| | |
| | - `--diffusion-steps N`: Diffusion denoising steps (default: 128) |
| | - `--diffusion-algorithm N`: Algorithm selection: |
| | - 0 = ORIGIN (original algorithm) |
| | - 1 = ENTROPY_BASED (entropy-based) |
| | - 2 = MARGIN_BASED (margin-based) |
| | - 3 = RANDOM (random) |
| | - 4 = LOW_CONFIDENCE (low confidence, default) |
| | - `--diffusion-alg-temp F`: Algorithm temperature (default: 0.0) |
| | - `--diffusion-visual`: Enable visualization mode, show generation progress |
| | - `--diffusion-eps F`: Time step epsilon value |
| |
|
| | ### Python (llama-cpp-python) |
| |
|
| | ```bash |
| | pip install llama-cpp-python |
| | ``` |
| |
|
| | ```python |
| | from llama_cpp import Llama |
| | |
| | # Load model |
| | llm = Llama( |
| | model_path="gguf_output/dream-coder-7b-q8_0.gguf", |
| | n_ctx=2048, |
| | n_threads=8, |
| | n_gpu_layers=0 # CPU inference, set >0 to enable GPU acceleration |
| | ) |
| | |
| | # Generate code |
| | output = llm( |
| | "def fibonacci(n):", |
| | max_tokens=512, |
| | temperature=0.1, |
| | top_p=0.95, |
| | repeat_penalty=1.1 |
| | ) |
| | |
| | print(output['choices'][0]['text']) |
| | ``` |
| |
|
| | ### With GPU Acceleration |
| |
|
| | If compiled with CUDA support: |
| |
|
| | ```bash |
| | # Compile CUDA version |
| | cd llama.cpp |
| | make clean |
| | make LLAMA_CUBLAS=1 -j$(nproc) |
| | |
| | # Use GPU acceleration (partial layers) |
| | ./build/bin/llama-diffusion-cli \ |
| | -m gguf_output/dream-coder-7b-q8_0.gguf \ |
| | -p "def quicksort(arr):" \ |
| | -n 512 \ |
| | --diffusion-steps 128 \ |
| | -ngl 20 # Number of GPU layers |
| | ``` |
| |
|
| | ## Troubleshooting |
| |
|
| | ### Common Issues |
| |
|
| | 1. **Conversion Failure**: |
| | - Ensure llama.cpp is compiled correctly |
| | - Check Python dependency versions |
| | - Verify model file integrity |
| |
|
| | 2. **Quantization Failure**: |
| | - Check disk space (~20GB temporary space needed) |
| | - Ensure sufficient memory (32GB+ recommended) |
| |
|
| | 3. **Inference Errors**: |
| | - Verify GGUF file integrity |
| | - Check context length settings |
| | - Try reducing `n_gpu_layers` |
| |
|
| | ### Model Validation |
| |
|
| | ```bash |
| | # File integrity check |
| | ls -lh gguf_output/dream-coder-7b-q8_0.gguf |
| | |
| | # Simple inference test |
| | echo "def hello():" | ./llama.cpp/build/bin/llama-diffusion-cli -m gguf_output/dream-coder-7b-q8_0.gguf -n 20 --diffusion-steps 64 |
| | ``` |
| |
|
| | ## Performance Optimization |
| |
|
| | ### CPU Optimization |
| | - Use `-t` parameter to set thread count |
| | - Enable AVX2/AVX512 compilation options |
| | - Adjust batch size (`-b` parameter) |
| |
|
| | ### GPU Optimization |
| | - Use CUDA/OpenCL compilation |
| | - Adjust GPU layer count (`-ngl`) |
| | - Monitor GPU memory usage |
| |
|
| | ### Memory Optimization |
| | - Use `--mmap` to enable memory mapping |
| | - Adjust `--mlock` parameter |
| | - Set appropriate context length |
| |
|
| | ## Important Notes |
| |
|
| | 1. **Diffusion Features**: Dream-Coder uses diffusion generation, different from traditional autoregressive models |
| | 2. **Dedicated Tool**: Must use `llama-diffusion-cli` instead of the regular `main` tool |
| | 3. **Special Tokens**: Maintain correct handling of `mask_token_id` (151666) |
| | 4. **Context Length**: Supports maximum 32K tokens, but 2K-4K recommended for optimal performance |
| | 5. **Generation Parameters**: Recommend using lower temperature (0.1-0.3) and appropriate top_p (0.9-0.95) |
| | 6. **Diffusion Steps**: Recommend 64-128 steps, more steps may improve quality but increase inference time |
| | |
| | ## Technical Support |
| | |
| | If you encounter issues, please check: |
| | 1. llama.cpp version and compilation status |
| | 2. Python dependency version compatibility |
| | 3. Model file integrity |
| | 4. System resources (memory/disk) |
| | |
| | For more information, refer to: |
| | - [llama.cpp GitHub](https://github.com/ggerganov/llama.cpp) |
| | - [GGUF Format Documentation](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md) |
