Discussions

• 1 You must be logged in to vote

  🙏

  Stuck on Sentencepiece

  frenzybiscuit asked Feb 17, 2025 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  How to JSON serialise image and video embeddings from Vision models?

  Thireus asked Apr 26, 2025 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  💬

  Recommended/Best practice for chat implementation. Extend input_ids or _gen_begin_reuse()/_gen_feed_tokens()?

  karlsolomon started Apr 14, 2025 in General

  

  

  0
• 3 You must be logged in to vote

  💬

  Prompt Caching & System Prompts

  nonnull-ca started Nov 2, 2023 in General

  

  

  1
• 1 You must be logged in to vote

  🙏

  Stuck on missing prompt_template

  draker42 asked Mar 20, 2025 in Q&A · Unanswered

  

  

  5
• 1 You must be logged in to vote

  💬

  Layer repeats via a config file.

  dnhkng started Jan 9, 2024 in General

  

  

  

  

  6
• 1 You must be logged in to vote

  💬

  Are the aya vision models supported?

  GoudaCouda started Mar 9, 2025 in General

  

  

  0
• 1 You must be logged in to vote

  🙏

  GPU benchmarks for exllamav2

  volodymyr-barannik asked Mar 3, 2025 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  💬

  What is the easiest way to quantize only part of the model?

  Aure20 started Feb 18, 2025 in General

  

  

  0
• 1 You must be logged in to vote

  🙏

  Can we re-enqueue a finished job?

  nvlas asked Feb 2, 2025 in Q&A · Answered

  

  

  3
• 2 You must be logged in to vote

  🙏

  Greedy Sampling

  luisfrentzen-cc asked Mar 4, 2024 in Q&A · Answered

  

  

  

  

  6
• 1 You must be logged in to vote

  🙏

  Which cache type to use?

  Mithadon asked Jan 29, 2025 in Q&A · Answered

  

  

  1
• 1 You must be logged in to vote

  💬

  There is not enough said about how much better the quality of inference is with Exl2 and exllamav2 over GGUF.

  timrohrbaugh started Nov 11, 2024 in General

  

  

  1
• 1 You must be logged in to vote

  🙏

  Model loading from Huggingface cache

  mpetruc asked Nov 29, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Mirostat and EXL2?

  timrohrbaugh asked Nov 23, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  💬

  Priority for MllamaForConditionalGeneration Support?

  gros87 started Sep 30, 2024 in General

  

  

  1
• 1 You must be logged in to vote

  🙏

  Question on understanding how model.forward factors in temperature?

  efawe asked Oct 31, 2024 in Q&A · Unanswered

  

  

  1
• 3 You must be logged in to vote

  💡

  Future plan?

  Katehuuh started Apr 22, 2024 in Ideas

  

  

  

  

  9
• 1 You must be logged in to vote

  🙏

  Question about dequantization

  bug Something isn't working

  HaoWeiWang asked Sep 25, 2024 in Q&A · Unanswered

  

  

  1
• 2 You must be logged in to vote

  💬

  Phi 3.5 MOE and supporting MOE in general

  niceblue88 started Sep 24, 2024 in General

  

  

  1
• 1 You must be logged in to vote

  💡

  Injecting noise in hidden state inputs, query, key/value or attention head outputs

  xonfour started Jun 10, 2024 in Ideas

  

  

  

  

  31
• 1 You must be logged in to vote

  💬

  Optimal Quantizing

  GHBigD started Aug 16, 2024 in General

  

  

  2
• 1 You must be logged in to vote

  💡

  Liger kernels and exllamav2?

  jim-plus started Aug 24, 2024 in Ideas

  

  

  0
• 1 You must be logged in to vote

  💬

  Addressing a skill issue getting CUDA 12.1 to support the latest MSVC under Windows when building current exllamav2

  jim-plus started Aug 13, 2024 in General

  

  

  2
• 1 You must be logged in to vote

  💬

  Measurement/inference error (3): hidden_states

  Katehuuh started Jul 30, 2024 in General

  

  

  0