Build choice - am4 with more ram or am5 with less?

Elf_Boy

Elf_Boy

Quasi-regular
Joined
Nov 9, 2019
Messages
463
Reaction score
211
Just upgraded mu Mobo/CPU.

Now I am looking to build a second system from older parts to run an LLM and maybe even be that Linux box I've wanted to learn with.

I'll need to pick up an NVME drive and maybe a PSU - have everything else.

The question and for which I am asking advise: I have an Asus AM4 board with a 5950 and 128gb of ddr4, from 2 upgrades ago, -or- I can use my am5 670e board with the 7950x3d, but which has no ram - I would need to buy ddr5 and realistically getting more than 32 or 64gb at this point is going to be costly (blew my discretionary budget for some time with the mobo/cpu upgrade).

For an LLM is more memory (though ddr4 and slower) better or less but faster memory with a faster cpu better?

Both Mobo's are Asus TUF/Gaming models.

I know Linux can run on low ram just fine - or so I have read, but an LLM?

If it matters the old GPU is a Sapphire rx 6800 with a mild factory overclock.
 
For running LLMs locally it's going to be the size of the LLM you want to run as opposed to the speed of the LLM itself.

Speed won't matter if you need to run an 80gb LLM and only have 64 gigs of ram. You don't want your LLM caching to disk or running in virtual memory. And really most dont give that option.
 
In my experience - memory was always a binary thing (no pun intended): either you had enough, or you didn't. Whatever that magic number is - having more didn't really help (unless you had ~a lot~ more and could start RAM caching ridiculous amounts or something), and if you are under that magic number, you are just under water... it's like you can drown in 3" of water as easily as 300'.

Compute scales though - not always linearly, but scales none the less.
 
Lets hope 128gb is enough - ddr5 is too expensive these days. Even 4*16 is $500+

How important is memory speed?
 
Elf_Boy said:
Lets hope 128gb is enough - ddr5 is too expensive these days. Even 4*16 is $500+

How important is memory speed?
Click to expand...
For memory you want capacity, Stability, then speed. In that order. That should be your priorities.

Speed isn't just the MTS. You can overcome that with more memory channels. Honestly I'd look to DDR4, see if you can find a 4 channel TR4 chip to run on it. The 4 memory channels will serve your better, then you can also be running ECC ram. All win's.
 
Elf_Boy said:
How important is memory speed?
Click to expand...
If you got only two channels, the faster the better. But it only helps on Intel platforms. Ryzen is limited by its fabric so I doubt you would see a major difference with 8000+ RAM.

Arrow Lake is pretty decent with LLM.


1770090890759.png

Token generation speed is what you want to be concerned about. Prompt processing speed only matters if you are doing RAG (feeding LLM very large documents).

Arrow Lake is churning out some impressive numbers even with only 6400 RAM and it can easily do 9200 Gear 2 if you get a decent mobo like the 1DPC Z890 OCF.

Having a GPU or multiple GPUs with 16GB RAM will help matters too. In LM Studio, you can offload some layers of the LLM model onto the GPU and that should make the speed better or help with low system RAM issues.

Your LLM issues will really depend on what your demands are. If you want "accurate" coding related tasks from an LLM, you better be ready to use 70B parameter models, though gpt-OSS-20B looks good but I haven't tried compiling its code yet so not sure how accurate it is.

The two utilities in my sig (rudi_float_bench and memory latency) were created with Deepseek and required over 300GB RAM. At that time, I was lucky enough to have access to a Cascade Lake 6248R server with six channels of 384GB DDR4-2933 RAM. Even then, it was slow as heck. I got maybe 6 tokens per second max out of it. Average would be 3 to 4.5 tokens/sec.

The higher the billions of parameters a model has, the slower it's gonna be in token generation.
 
You must log in or register to reply here.
Become a Patron!
Back
Top