I've just finished a thorough evaluation (multiple hour-long chats with 274 messages total over both TheBloke/Nous-Hermes-Llama2-GGML (q5_K_M) and TheBloke/Redmond-Puffin-13B-GGML (q5_K_M)) so I'd like to give my feedback. Supporting model backends: tranformers, bitsandbytes(8-bit inference),. EXL2 (and AWQ)What is GPTQ GPTQ is a novel method for quantizing large language models like GPT-3,LLama etc which aims to reduce the model’s memory footprint and computational requirements without. 33B you can only fit on 24GB VRAM, even 16Gb are not enough. It needs to run on a GPU. However, llama. 1, 1. Model Description. ExLlamaV2 is a library designed to squeeze even more performance out of GPTQ. When comparing llama. Once the quantization is completed, the weights can be stored and reused. It is integrated in various libraries in 🤗 ecosystem, to quantize a model, use/serve already quantized model or further. Click the Refresh icon next to Model in the top left. The model will automatically load, and is now. 24 # GPU version!pip install ctransformers[gptq] On you computer: We also outperform a recent Triton implementation for GPTQ by 2. Downloaded Robin 33B GPTQ and noticed the new model interface, switched over to EXllama and read I needed to put in a split for the cards. Lots of people have asked if I will make 13B, 30B, quantized, and ggml flavors. GGML_TYPE_Q3_K - "type-0" 3-bit quantization in super-blocks containing 16 blocks, each block having 16 weights. Pygmalion 7B SuperHOT 8K GPTQ. The Exllama_HF model loader seems to load GPTQ models. Text Generation Transformers English gptj text generation conversational gptq 4bit. 2k 3. github. Download: GGML (Free) Download: GPTQ (Free) Now that you know what iteration of Llama 2 you need,. First, we explore and expand various areas in the same topic using the 7K conversations created by WizardLM. It is now able to fully offload all inference to the GPU. 3. GPTQ clearly outperforms here. It's true that GGML is slower. Open comment sort options. Interact privately with your documents using the power of GPT, 100% privately, no data leaks (by imartinez) Suggest topics Source Code. ) Test 3 TheBloke_Wizard-Vicuna-13B-Uncensored-GPTQ GPTQ-for-LLaMa The first one is to be installed when you want to load and interact with GPTQ models; the second one is to be ued with GGUF/GGML files, that can run on CPU only. According to open leaderboard on HF, Vicuna 7B 1. GPTQ dataset: The dataset used for quantisation. GGUF boasts extensibility and future-proofing through enhanced metadata storage. In both cases I'm pushing everything I can to the GPU; with a 4090 and 24gb of ram, that's between 50 and 100 tokens per second (GPTQ has a much more variable. Click Download. Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4. It's the reason there's no GGML k-quants for Open Llama 3B yet, and it also causes this GPTQ issue. Please specify it manually using --model_type argument Press any key to continue . and some compatibility enhancements. Last week, Hugging Face announced that Transformers and TRL now natively support AutoGPTQ. These aren't the old GGML quants, this was done with the last version before the change to GGUF, and the GGUF is the latest version. 01 is default, but 0. LLMs are so large it can take a few hours to quantize some these models. In this paper, we address this challenge, and propose GPTQ, a new one-shot weight quantization method based on approximate second-order information, that is both highly-accurate and highly-efficient. i understand that GGML is a file format for saving model parameters in a single file, that its an old problematic format, and. This is the pattern that we should follow and try to apply to LLM inference. Locked post. Some time back I created llamacpp-for-kobold, a lightweight program that combines KoboldAI (a full featured text writing client for autoregressive LLMs) with llama. GGML_TYPE_Q4_K - "type-1" 4-bit quantization in super-blocks containing 8 blocks, each block having 32 weights. Under Download custom model or LoRA, enter TheBloke/airoboros-33b-gpt4-GPTQ. github. These files will not work in llama. 2023. 0, 0. I have suffered a lot with out of memory errors and trying to stuff torch. Open the text-generation-webui UI as normal. Block scales and mins are quantized with 4 bits. GPTQ model: anon8231489123/vicuna-13b-GPTQ-4bit-128g on huggingfaceoriginal model: lm-. I am in the middle of some comprehensive GPTQ perplexity analysis - using a method that is 100% comparable to the perplexity scores of llama. GGML_TYPE_Q3_K - "type-0" 3-bit quantization in super-blocks containing 16 blocks, each block having 16 weights. whisper. Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4. Finding a way to try GPTQ to. • 6 mo. Oobabooga’s Text Generation WebUI [15]: A very versatile Web UI for running LLMs, compatible with both GPTQ and GGML models with many configuration options. We’re on a journey to advance and democratize artificial intelligence through open source and open science. I'm also still a bit curious of GGML is competitive with GPTQ/exllama when running on Nvidia GPU. 1 results in slightly better accuracy. bin. This video explains difference between GGML and GPTQ in AI models in very easy terms. GGML is a C library for machine learning (ML) — the “GG” refers to the initials of its originator (Georgi Gerganov). Repeat the process by entering in the 7B model, TheBloke/WizardLM-7B-V1. 0. GPTQ-for-LLaMa vs text-generation-webui. safetensors along with all of the . The weights in a GGML file are encoded as a list of layers, the length of which is. The team is also working on a full benchmark, similar to what was done for GPT4-x-Vicuna. So the end. , only utilizes 4 bits and represents a significant advancement in the field of weight quantization. As far as I'm aware, GPTQ 4-bit w/ Exllama is still the best option. Update to include TheBloke_Wizard-Vicuna-13B-Uncensored-GPTQ GPTQ-for-LLaMa VS Auto GPTQ VS ExLlama (This does not change GGML test results. There's also a half-context 3 epoch version that you can get here. bin. cpp, and adds a versatile Kobold API endpoint, additional format support, backward compatibility, as well as a fancy UI with persistent stories, editing tools, save formats, memory, world info,. In the top left, click the refresh icon next to. However, we made it in a continuous conversation format instead of the instruction format. Before you can download the model weights and tokenizer you have to read and agree to the License Agreement and submit your request by giving your email address. 2t/s. Have ‘char a’ perform an action on ‘char b’ and also have ‘char b’ perform and action on ‘user’ and have ‘user perform an action on either ‘char’ and see how well it keeps up with who is doing. They collaborated with LAION and Ontocord to create the training dataset. In the Model dropdown, choose the model you just. We dive deep into the world of GPTQ 4-bit quantization for large language models like LLaMa. 2023. Using a dataset more appropriate to the model's training can improve quantisation accuracy. Updated to the latest fine-tune by Open Assistant oasst-sft-7-llama-30b-xor. In order for their Accuracy or perplexity whatever you want to call it. cpp team have done a ton of work on 4bit quantisation and their new methods q4_2 and q4_3 now beat 4bit GPTQ in this benchmark. A standalone Python/C++/CUDA implementation of Llama for use with 4-bit GPTQ weights, designed to be fast and memory-efficient on modern GPUs. Currently I am unable to get GGML to work with my Geforce 3090 GPU. txt","contentType":"file. At a higher level, the process involves the following steps: Some GPTQ clients have had issues with models that use Act Order plus Group Size, but this is generally resolved now. Model Architecture Llama 2 is an auto-regressive language model that uses an optimized transformer architecture. This is an example to launch koboldcpp in streaming mode, load a 8k SuperHOT variant of a 4 bit quantized ggml model and split it between the GPU and CPU. I plan to make 13B and 30B, but I don't have plans to make quantized models and ggml, so I will rely on the community for that. GPTQ is better, when you can fit your whole model into memory. Once the quantization is completed, the weights can be stored and reused. That is, it starts with WizardLM's instruction, and then expands into various areas in one conversation using. cpp) can. . New comments cannot be posted. sponsored. GPTQ is an alternative method to quantize LLM (vs llama. Using Llama. 50 tokens/s, 511 tokens, context 44,. 5B tokens high-quality programming-related data, achieving 73. cpp and libraries and UIs which support this format, such as: text-generation-webui; KoboldCpp; ParisNeo/GPT4All-UI; llama-cpp-python; ctransformers; Repositories available 4-bit. This script duplicates the addend and scale to match ggml's expectations, at the cost of wasting some memory. cpp. safetensors along with all of the . You can now start fine-tuning the model with the following command: accelerate launch scripts/finetune. 5 if they can get it to be cheaper overall. GGML files are for CPU + GPU inference using llama. This should just work. as today's master, you don't need to run migrate script. For the first time ever, this means GGML can now outperform AutoGPTQ and GPTQ-for-LLaMa inference (though it still loses to exllama) Note: if you test this, be aware that you should now use --threads 1 as it's no longer beneficial to use. 13B is parameter count, meaning it was trained on 13 billion parameters. 0 model and it seems it was trained on the following template: ### Human: <your prompt here> ### Assistant:With this option you use the GGML format model and LLaMA interface called llama. Note that the GPTQ dataset is not the same as the dataset. 5B parameter Language Model trained on English and 80+ programming languages. There are already bleeding edge 4-bit quantization efforts such as GPTQ for LLaMA. A simple one-file way to run various GGML and GGUF models with KoboldAI's UI llama. This adds full GPU acceleration to llama. Output Models generate text only. Especially good for story telling. GPTQ. GGUF) Thus far, we have explored sharding and quantization techniques. Quantize Llama models with GGML and llama. cpp and libraries and UIs which support this format, such as: KoboldCpp, a powerful GGML web UI with full GPU acceleration out of the box. cpp is a way to use 4-bit quantization to reduce the memory requirements and speed up the inference. Context sizes: (512 | 1024 | 2048) ⨯ (7B | 13B | 30B | 65B) ⨯ (llama | alpaca[-lora] | vicuna-GPTQ) models, first 406 lines of wiki. Reply reply MrTopHatMan90 • Yeah that seems to of worked. Further, we show that our model can also provide robust results in the extreme quantization regime,WizardLM-7B-uncensored-GGML is the uncensored version of a 7B model with 13B-like quality, according to benchmarks and my own findings. This might help get a 33B model to load on your setup but you can expect shuffling between VRAM and system RAM. So it seems that GPTQ has a similar latency problem. Hi all, looking for a guide/some advice on how to do this. com. /bin/gpt-2 -h usage: . This format is good for people that does not have a GPU, or they have a really weak one. cpp and GPTQ-for-LLaMa you can also consider the following projects: gpt4all - gpt4all: open-source LLM chatbots that you can run anywhere. OpenLLaMA is an openly licensed reproduction of Meta's original LLaMA model. Note that some additional quantization schemes are also supported in the 🤗 optimum library, but this is out of scope for this blogpost. txt","path":"examples/whisper/CMakeLists. GGML files are for CPU + GPU inference using llama. 1 results in slightly better accuracy. For ref, 13900k is 2x the single core performance vs 1950x. However, there are two differences which I accommodated changing the output format (and adding corresponding support to main. jsons and . Hacker NewsDamp %: A GPTQ parameter that affects how samples are processed for quantisation. cpp is another framework/library that does the more of the same but specialized in models that runs on CPU and quanitized and run much faster. GPU Installation (GPTQ Quantised) First, let’s create a virtual environment: conda create -n vicuna python=3. In the Model drop-down: choose the model you just downloaded, vicuna-13B-1. OpenLLaMA is an openly licensed reproduction of Meta's original LLaMA model. NousResearch's Nous-Hermes-13B GPTQ. cpp team have done a ton of work on 4bit quantisation and their new methods q4_2 and q4_3 now beat 4bit GPTQ in this. 4bit means how it's quantized/compressed. GGML vs. cpp users to enjoy the GPTQ quantized models. Download the 3B, 7B, or 13B model from Hugging Face. 4bit and 5bit GGML models for GPU inference. are other backends with their own quantized format, but they're only useful if you have a recent graphics card (GPU). Under Download custom model or LoRA, enter this repo name: TheBloke/stable-vicuna-13B-GPTQ. Bitsandbytes can perform integer quantization but also supports many other formats. That's it. empty_cache() everywhere to prevent memory leaks. The GGML_TYPE_Q5_K is a type-1 5-bit quantization, while the GGML_TYPE_Q2_K is a type-1 2-bit quantization. To download from a specific branch, enter for example TheBloke/Wizard-Vicuna-30B. 0-16k-GPTQ:gptq-4bit-32g-actorder_True. GGML is a weight quantization method that can be applied to any model. The model will start downloading. I've used these with koboldcpp, but CPU-based inference is too slow for regular usage on my laptop. One option to download the model weights and tokenizer of Llama 2 is the Meta AI website. Instead, these models have often already been sharded and quantized for us to use. llama. Text Generation • Updated Sep 27 • 15. cpp. Input Models input text only. Click the Model tab. Click Download. cpp you can also consider the following projects: gpt4all - gpt4all: open-source LLM chatbots that you can run anywhere. cpp library, also created by Georgi Gerganov. < llama-30b FP32 2nd load INFO:Loaded the model in 68. 0-GPTQ. 🌙 GGML vs GPTQ vs bitsandbytes Abstract: This article compares GGML, GPTQ, and bitsandbytes in the context of software development. Here's some more info on the model, from their model card: Model Description. GGML files consists of binary-encoded data that is laid out according to a specified. This causes various problems. Check the first 4 bytes of the generated file. alpaca-lora - Instruct-tune LLaMA on consumer hardware. 5. I don't have enough VRAM to run the GPTQ one, I just grabbed the. GGML, GPTQ, and bitsandbytes all offer unique features and capabilities that cater to different needs. 0 license, with full access to source code, model weights, and training datasets. cpp team on August 21st 2023. Models by stock have 16bit precision, and each time you go lower, (8 bit, 4bit, etc) you sacrifice some. I noticed SSD activities (likely due to low system RAM) on the first text generation. Supports transformers, GPTQ, AWQ, EXL2, llama. In the top left, click the refresh icon next to Model. To recap, every Spark. Oobabooga: If you require further instruction, see here and hereBaku. Under Download custom model or LoRA, enter TheBloke/falcon-7B-instruct-GPTQ. For Kobold CCP you use GGML files insted of the normal gptq or f16 formats. GPTQ-for-LLaMa vs bitsandbytes. Llama, GPTQ 4bit, AutoGPTQ: WizardLM 7B: 43. 4. The gpu is waiting for more work while cpu is maxed out. This is probably stupid and maybe ggml already works this way, but I am wondering, since the main bottleneck seems to be memory bandwidth, could the batches be processed in. GGML speed strongly depends on the performance and the positioning of RAM slots Reply. Text Generation • Updated Sep 27 • 23. If your cpu (the core that is running python inference) is at 100% and gpu is 25%, the bottleneck is cpu. What would take me 2-3 minutes of wait time for a GGML 30B model takes 6-8 seconds pause followed by super fast text from the model - 6-8 tokens a second at least. 2023年8月28日 13:33. GGML to GGUF is the transition from prototype technology demonstrator to a mature and user-friendy solution. Under Download custom model or LoRA, enter TheBloke/falcon-40B-instruct-GPTQ. GGML is designed for CPU and Apple M series but can also offload some layers on the GPU. It was designed to be good at. smspillaz/ggml-gobject: GObject-introspectable wrapper for use of GGML on the GNOME platform. py oasst-sft-7-llama-30b/ oasst-sft-7-llama-30b-xor/ llama30b_hf/. 01 is default, but 0. after prompt ingestion). Good inference speed in AutoGPTQ and GPTQ-for-LLaMa. The model will start downloading. One of the most popular is GPTQ – introduced in March 2023 which uses 4 bits (16 distinct values!) to represent a floating point. TheBloke/MythoMax-L2-13B-GPTQ differs from other language models in several key ways: 1. GPTQ is a one-shot weight quantization method based on approximate second-order information, allowing for highly accurate and efficient quantization of GPT models with 175 billion parameters. Note i compared orca-mini-7b vs wizard-vicuna-uncensored-7b (both the q4_1 quantizations) in llama. Scales are quantized with 6 bits. cpp GGML models, so we can compare to figures people have been doing there for a. c) T4 GPU. Supports transformers, GPTQ, AWQ, EXL2, llama. Start text-generation-webui normally. nf4 without double quantization significantly uses more memory than GPTQ. If you’re looking for an approach that is more CPU-friendly, GGML is currently your best option. bat to activate env, then from that browse to the AutoGPTQ and run the command - it should work. 8% pass@1 on HumanEval. GPTQ dataset: The dataset used for quantisation. GPTQ can lower the weight precision to 4-bit or 3-bit. 1 results in slightly better accuracy. Vicuna v1. Unique Merging Technique. Use both exllama and GPTQ. cuda. To download from a specific branch, enter for example TheBloke/Wizard-Vicuna-7B. once the GPTQ version is shared. OpenChatKit is an open-source large language model for creating chatbots, developed by Together. The model will automatically load, and is now ready for use!GGML vs. 1 results in slightly better accuracy. Supports transformers, GPTQ, AWQ, EXL2, llama. Block scales and mins are quantized with 4 bits. cpp. bitsandbytes: VRAM Usage. Open comment sort options. Note that the GPTQ dataset is not the same as the dataset. So here it is, after exllama, GPTQ and SuperHOT stole GGML the show for a while, finally there's a new koboldcpp version with: full support for GPU acceleration using CUDA and OpenCL. 9 min read. ローカルLLMの量子化フォーマットとしては、llama. 0更新【6. ggmlv3. And the wildcard is GGML - I wouldn't bet against that becoming the performance champion before long. ggml is a library that provides operations for running machine learning models. The change is not actually specific to Alpaca, but the alpaca-native-GPTQ weights published online were apparently produced with a later version of GPTQ-for-LLaMa. Since the original full-precision Llama2 model requires a lot of VRAM or multiple GPUs to load, I have modified my code so that quantized GPTQ and GGML model variants (also known as llama. Testing the new BnB 4-bit or "qlora" vs GPTQ Cuda upvotes. To use with your GPU using GPTQ pick one of the . Currently 4-bit (RtN) with 32 bin-size is supported by GGML implementations. cpp. All reactions. auto-gptq: 4-bit quantization with exllama kernels. 4 Python text-generation-webui VS GPTQ-for-LLaMa 4 bits quantization of LLaMA using GPTQ InfluxDB. For GPTQ tests, I used models with groupsize 128 and no desc_act, which are the ones that are widely used. 4bit and 5bit GGML models for GPU inference. 4bit quantised GPTQ models for GPU inference - TheBloke/stable-vicuna-13B-GPTQ. Use in Transformers. GPTQ is a specific format for GPU only. This ends up effectively using 2. GPTQ dataset: The dataset used for quantisation. GGML: 3 quantized versions. Under Download custom model or LoRA, enter TheBloke/WizardCoder-15B-1. wv, attention. 4375 bpw. Learning Resources:TheBloke Quantized Models - from Hugging Face (Optimum) -. 01 is default, but 0. One quantized using q4_1, another one was quantized using q5_0, and the last one was quantized using q5_1. Using a dataset more appropriate to the model's training can improve quantisation accuracy. GGML vs GPTQ — Source:1littlecoder 2. The lower bit quantization can reduce the file size and memory bandwidth requirements, but also introduce more errors and noise that can affect the accuracy of the model. We propose SmoothQuant, a training-free, accuracy-preserving, and. GGML 30B model VS GPTQ 30B model 7900xtx FULL VRAM Scenario 2. Build whisper. Damp %: A GPTQ parameter that affects how samples are processed for quantisation. 其实有一个感想是目前. Maybe now we can do a vs perplexity test to confirm. cpp (GGUF), Llama models. Once it's finished it will say "Done". Damp %: A GPTQ parameter that affects how samples are processed for quantisation. • 5 mo. In the Model dropdown, choose the model you just downloaded: WizardCoder-Python-34B-V1. In the Model drop-down: choose the model you just downloaded, falcon-40B-instruct-GPTQ. Context is hugely important for my setting - the characters require about 1,000 tokens apiece, then there is stuff like the setting and creatures. Now, I've expanded it to support more models and formats. Scales and mins are quantized with 6 bits. Uses GGML_TYPE_Q5_K for the attention. Llama 2. The current release includes the following features: An efficient implementation of the GPTQ algorithm: gptq. Pros: GGML was an early attempt to create a file format for storing GPT models. For reference, I'm used to 13B models generating at 2T/s, and 7B models at 4 T/s. This end up using 3. Python 27. Llama 2. There are 2 main formats for quantized models: GGML and GPTQ. GPTQ means it will run on your graphics card at 4bit (vs GGML which runs on CPU, or the non-GPTQ version which runs at 8bit). Quantization can reduce memory and accelerate inference. But GGML allows to run them on a medium gaming PC at a speed that is good enough for chatting. Tensor library for. . Gptq-triton runs faster. GGML/GGUF is a C library for machine learning (ML) — the “GG” refers to. . In addition to defining low-level machine learning primitives (like a tensor type), GGML defines a binary format for distributing LLMs. Supports transformers, GPTQ, AWQ, EXL2, llama. Untick Autoload model. So, in this article, we will. KoboldCpp, a powerful GGML web UI with GPU acceleration on all platforms (CUDA and OpenCL). en-encoder-openvino. GPU/GPTQ Usage. GGML_TYPE_Q2_K - "type-1" 2-bit quantization in super-blocks containing 16 blocks, each block having 16 weight. py does work on the QLORA, but when trying to apply it to a GGML model it refuses and claims it's lacking a dtype. GitHub Copilot's extension generates a multitude of requests as you type, which can pose challenges, given that language models typically process one. AWQ vs. GPTQ vs. Repositories available 4bit GPTQ models for GPU inference. To use with your GPU using GPTQ pick one of the . Along with most 13B models ran in 4bit with around Pre-layers set to 40 in Oobabooga. Let’s break down the. The older GGML format revisions are unsupported and probably wouldn't work with anything other than KoboldCCP since the Devs put some effort to offer backwards compatibility, and contemporary legacy versions of llamaCPP. The 8bit models are higher quality than 4 bit, but again more memory etc. Click Download. 58 seconds. Pre-Quantization (GPTQ vs. 256 70 2,931 contributions in the last year Contribution Graph; Day of Week: November Nov: December Dec: January Jan: February Feb: March Mar: April Apr: May May: June Jun:. ggml's distinguishing feature is efficient operation on CPU. For my box with AMD 3700X, the 3090 only gets to 60-75% GPU. However, I was curious to see the trade-off in perplexity for the chat. And switching to GPTQ-for-Llama to load. cpp - convert-lora-to-ggml. GGML_TYPE_Q3_K - "type-0" 3-bit quantization in super-blocks containing 16 blocks, each block having 16 weights. Did not test GGUF yet, but is pretty much GGML V2. Results. AWQ, on the other hand, is an activation-aware weight quantization approach that protects salient weights by. cpp. The metrics obtained include execution time, memory usage, and. A discussion thread on GitHub that compares the performance of GGML, a generative model for text generation, with and without GPU acceleration and three different GPTQ. GGML_TYPE_Q4_K - "type-1" 4-bit quantization in super-blocks containing 8 blocks, each block having 32 weights. Share Sort by: Best. It is now able to fully offload all inference to the GPU. That was it's main purpose, to let the llama. My understanding was training quantisation was the big breakthrough with qlora, so in terms of comparison it’s apples vs oranges. , 2023) was first applied to models ready to deploy. Uses that GPT doesn’t allow but are legal (for example, NSFW content) Enterprises using it as an alternative to GPT-3. GGUF is a new format introduced by the llama. panchovix. GGML vs. text-generation-webui - A Gradio web UI for Large Language Models. For the first time ever, this means GGML can now outperform AutoGPTQ and GPTQ-for-LLaMa inference (though it still loses to exllama) Note: if you test this, be aware that you should now use --threads 1 as it's no longer beneficial to use. Once it's finished it will say "Done". It was discovered and developed by kaiokendev. txt input file containing some technical blog posts and papers that I collected. That's like 50% of the whole job. model-specific. Click Download. koboldcpp. GPTQ is a one-shot weight quantization method based on approximate second-order information, allowing for highly accurate and efficient quantization of GPT models with 175 billion parameters. That is, it starts with WizardLM's instruction, and then expands into various areas in one conversation using. *Its technically not compression. GPTQ tries to solve an optimization problem for each. GGUF / GGML versions run on most computers, mostly thanks to quantization. 16 tokens per second (30b), also requiring autotune. For more general-purpose projects that require complex data manipulation, GPTQ's flexibility and extensive capabilities. . Just anecdotally, switching from a Q4 GPTQ model to Q6_K GGML for MythoMax-L2-13B produced palpable improvements. Try 4bit 32G and you will more than likely be happy with the result!GGML vs. Due to the massive size of Large Language Models (LLMs), quantization has become an essential technique to run them efficiently. (2) And does the mean we'd do well to download new GPTQ quants of our favorite models in light of the new information? (3) I'm also still a bit curious of GGML is competitive with GPTQ/exllama when running on Nvidia GPU. Deploy.