3/7/2026 at 4:37:03 PM
by fatihturker
3/8/2026 at 5:02:45 PM
Author here.I'm currently working on further speed improvements — it's already around 8× faster in some cases, but there’s still potential for more optimization.
Since this is an open-source project, community support is very important. I believe AI shouldn’t be controlled or driven by only a few companies, so contributions, feedback, and ideas are always very welcome. Feel free to open an issue or PR if you'd like to help.
by fatihturker
3/7/2026 at 4:38:55 PM
Author here.The architecture page explains how ternary quantization, dynamic sparsity, and mmap layer streaming work together to push models far beyond normal RAM limits.
Happy to answer questions about the implementation or benchmarks.
by fatihturker
3/7/2026 at 4:41:54 PM
This is cool projectby MrLey
3/8/2026 at 9:24:58 AM
Had to fix hardware detection myself only to find engine.generate()'s not implemented and yields "".Maybe the author could get a large param model to help him get this done though.
by LukeB42
3/8/2026 at 4:48:23 PM
Happy to help if needed. The project is already tested and benchmarked with several models and everything is working as expected. If you run into any specific issues, feel free to open an issue or PR.by fatihturker