<![CDATA[Going into the rabbithole of testing local LLMs right now.]]>Going into the rabbithole of testing local LLMs right now. I don't have a dedicated GPU, but 32 GiB of RAM should be enough for anyone.

]]>https://board.circlewithadot.net/topic/fab6c79a-2d1c-4def-ad61-47946437277a/going-into-the-rabbithole-of-testing-local-llms-right-now.RSS for NodeThu, 16 Apr 2026 20:24:54 GMTThu, 26 Feb 2026 09:52:19 GMT60<![CDATA[Reply to Going into the rabbithole of testing local LLMs right now. on Fri, 27 Feb 2026 18:22:38 GMT]]>@tomgag
Good question! Why is not part of the fediverse?!

]]>https://board.circlewithadot.net/post/https://dresden.network/users/blingblingmk/statuses/116143984398848269https://board.circlewithadot.net/post/https://dresden.network/users/blingblingmk/statuses/116143984398848269Fri, 27 Feb 2026 18:22:38 GMT<![CDATA[Reply to Going into the rabbithole of testing local LLMs right now. on Thu, 26 Feb 2026 11:01:10 GMT]]>Interesting, it seems that Qwen 2.5 Coder is actually less aggressive than Qwen 3.5 in rejecting sensitive topics.

Link Preview Image
]]>https://board.circlewithadot.net/post/https://infosec.exchange/users/tomgag/statuses/116136586147803776https://board.circlewithadot.net/post/https://infosec.exchange/users/tomgag/statuses/116136586147803776Thu, 26 Feb 2026 11:01:10 GMT<![CDATA[Reply to Going into the rabbithole of testing local LLMs right now. on Thu, 26 Feb 2026 10:57:55 GMT]]>@1ad6e959c292f74de615d4c6e5ec43d0b7ec4908a55de93aa2527c46a8bd1d5b I'm not sure, I don't have any beefy GPU 😅 you shoulkd ask this in the Ollama Reddit community (or similar).

]]>https://board.circlewithadot.net/post/https://infosec.exchange/users/tomgag/statuses/116136573383282289https://board.circlewithadot.net/post/https://infosec.exchange/users/tomgag/statuses/116136573383282289Thu, 26 Feb 2026 10:57:55 GMT<![CDATA[Reply to Going into the rabbithole of testing local LLMs right now. on Thu, 26 Feb 2026 10:51:57 GMT]]>@tomgag maybe I’ll check I’m running on renewable energy before I leave a machine running over the weekend then 🤣

]]>https://board.circlewithadot.net/post/https://fosstodon.org/users/sealjay/statuses/116136549924150944https://board.circlewithadot.net/post/https://fosstodon.org/users/sealjay/statuses/116136549924150944Thu, 26 Feb 2026 10:51:57 GMT<![CDATA[Reply to Going into the rabbithole of testing local LLMs right now. on Thu, 26 Feb 2026 10:50:55 GMT]]>@sealjay well, I'm running on local CPU with 32 GiB of RAM, so I wouldn't call it "fast". 3-5 tokens per second maybe? I guess it's OK if you give it a task and then go to grab a coffee 😅

]]>https://board.circlewithadot.net/post/https://infosec.exchange/users/tomgag/statuses/116136545845564113https://board.circlewithadot.net/post/https://infosec.exchange/users/tomgag/statuses/116136545845564113Thu, 26 Feb 2026 10:50:55 GMT<![CDATA[Reply to Going into the rabbithole of testing local LLMs right now. on Thu, 26 Feb 2026 10:44:08 GMT]]>@tomgag how fast does it feel? I tried using foundry local and ollama but at the time I felt slowed down. I’d be keen to swap back to a local model given how the large providers are slowly catching down the subscription token limits.

]]>https://board.circlewithadot.net/post/https://fosstodon.org/users/sealjay/statuses/116136519182812009https://board.circlewithadot.net/post/https://fosstodon.org/users/sealjay/statuses/116136519182812009Thu, 26 Feb 2026 10:44:08 GMT<![CDATA[Reply to Going into the rabbithole of testing local LLMs right now. on Thu, 26 Feb 2026 10:18:17 GMT]]>First impressions of Mistral Small 3.2: seems pretty solid, it answers "uncomfortable" political question quite neutrally.

I don't understand why and by are not based on this.

]]>https://board.circlewithadot.net/post/https://infosec.exchange/users/tomgag/statuses/116136417545167228https://board.circlewithadot.net/post/https://infosec.exchange/users/tomgag/statuses/116136417545167228Thu, 26 Feb 2026 10:18:17 GMT<![CDATA[Reply to Going into the rabbithole of testing local LLMs right now. on Thu, 26 Feb 2026 09:53:49 GMT]]>Heretic quantized versions of Qwen 3.5 have just been released but even the base Qwen 3.5 model seems to have issue with ollama currently, and I don't have bandwidth to do a manual patch now. Trying Mistral 3.2.

]]>https://board.circlewithadot.net/post/https://infosec.exchange/users/tomgag/statuses/116136321299578974https://board.circlewithadot.net/post/https://infosec.exchange/users/tomgag/statuses/116136321299578974Thu, 26 Feb 2026 09:53:49 GMT