Primegrid PPSE Benchmarks

Skillz · Friday at 3:23 PM

This sub project is normally best to run single threaded. Current FFT length is around 1.5MB in size so it easily fits in the L3 cache domain with as many physical cores as a CPU has, in most cases.

Their is an up-coming Primegrid challenge series challenge in 4 days running this project so I am benchmarking a few of my hosts to get an idea of what I'll be able to bring to the table.

The current number I am using is a number I pulled off Primegrid's site. This number was completed on July 21st, 20206

Date Completed	Number	Points per task
July 21, 2026	5235*2^2378137+1	144.87

Will update the number for future challenges/benchmarks since this particular subproject's candidates progress very quickly.

Preferences notes about this sub project pulled from Primegrid.com

Proth Prime Search Extended LLR (PPSE)
k·2n+1 for 1200<k<10000
NOTE: PPSE tasks have smaller 'n' values and therefore are shorter than PPS tasks.

Supported platforms:

Windows: 32 bit, 64 bit
Linux: 32 bit, 64 bit
Mac: 64 bit
Multi-threading is supported but is NOT recommended. Click here to set the maximum number of threads.
Uses fast proof tasks so no double check tasks are needed. Everyone is "first"!

Deadline: 4 days (up to 30 days)
Recent average CPU time: 0:38:53
FFT sizes: 160K to 192K (uses up to 1536K cache per task)

Skillz · Friday at 3:27 PM

This is a Minisforum BD795i SE, I only ran these tests once. This is not an average of multiple tests.

This is a 16 core, 32 thread processor. I ran 16 tasks simultaneously on physical cores only and 32 tasks on all threads simultaneously.

Physical cores only is a bit faster on this host.

Code:

──────────────────────────────────────────────────────────────────────────────────────
                                   BENCHMARK SUMMARY
                                    Ryzen 9 7945HX
 Work Unit:                                               Proth Prime Search Extended
 Number Tested:                                                      5235*2^2378137+1
 Points per Task:                                                              144.87
 Benchmark Duration:                                                          10m 39s
 L3 Cache:                                                                    1.50 MB
 Affinity:                                                                  ascending
──────────────────────────────────────────────────────────────────────────────────────
       CPU Config         │    Task Duration    │   Tasks per Day  │    Estimated PPD
──────────────────────────────────────────────────────────────────────────────────────
  16 Tasks x  1 Threads   │       0:15:54       │     1447.736     │          209,733★
  32 Tasks x  1 Threads   │       0:34:07       │     1350.520     │          195,649
──────────────────────────────────────────────────────────────────────────────────────

Skillz · Friday at 4:07 PM

This is a Epyc 9654 CPU. Only one test, not an average.

This is a 96 core, 192 thread CPU. Running 96 tasks on all physical cores and then 192 tasks on 192 threads.

Again physical cores are faster, but a lot faster in this case. A whole lot faster. Kind of weird that the 7945HX was so much closer.

Code:

──────────────────────────────────────────────────────────────────────────────────────
                                   BENCHMARK SUMMARY
                                     EPYC 9654 96
 Work Unit:                                               Proth Prime Search Extended
 Number Tested:                                                      5235*2^2378137+1
 Points per Task:                                                              144.87
 Benchmark Duration:                                                          12m 24s
 L3 Cache:                                                                    1.50 MB
 Affinity:                                                                  ascending
──────────────────────────────────────────────────────────────────────────────────────
       CPU Config         │    Task Duration    │   Tasks per Day  │    Estimated PPD
──────────────────────────────────────────────────────────────────────────────────────
  96 Tasks x  1 Threads   │       0:26:27       │     5225.872     │          757,072★
 192 Tasks x  1 Threads   │       3:19:27       │     1386.128     │          200,808
──────────────────────────────────────────────────────────────────────────────────────

Skillz · Friday at 4:22 PM

This is a dual EPYC 9B14 host. Again I only ran the test once so no averages.

Same as above, but its two 96-core CPUs with a combined 384 threads. I again ran tasks on all physical cores, 192 of them, then again on all the threads of which their are 384 of them.

Code:

──────────────────────────────────────────────────────────────────────────────────────
                                   BENCHMARK SUMMARY
                             AMD EPYC 9B14 96-Core Pro...
 Work Unit:                                               Proth Prime Search Extended
 Number Tested:                                                      5235*2^2378137+1
 Points per Task:                                                              144.87
 Benchmark Duration:                                                          12m 27s
 L3 Cache:                                                                    1.50 MB
 Affinity:                                                                  ascending
──────────────────────────────────────────────────────────────────────────────────────
       CPU Config         │    Task Duration    │   Tasks per Day  │    Estimated PPD
──────────────────────────────────────────────────────────────────────────────────────
 192 Tasks x  1 Threads   │       0:21:48       │    12682.180     │        1,837,267★
 384 Tasks x  1 Threads   │       1:09:46       │     7925.336     │        1,148,143
──────────────────────────────────────────────────────────────────────────────────────

Here the 384 tasks run is much closer to the physical cores only run. While also being over 5x more than the single socket setup despite only having twice as many CPUs running. Something is a bit off here. I am going to rerun the single socket system. Maybe something was running while the threaded test run was going that skewed the results.

Skillz · Saturday at 11:04 AM

Single socket AMD 7H12 (Rome) host. Only one test run, no averages here either.

This is a 64 core part with 128 threads. Two tests one physical cores only and one all threads.

Code:

──────────────────────────────────────────────────────────────────────────────────────
                                   BENCHMARK SUMMARY
                             AMD EPYC 7H12 64-Core Pro...
 Work Unit:                                               Proth Prime Search Extended
 Number Tested:                                                      5235*2^2378137+1
 Points per Task:                                                              144.87
 Benchmark Duration:                                                          10m 28s
 L3 Cache:                                                                    1.50 MB
 Affinity:                                                                  ascending
──────────────────────────────────────────────────────────────────────────────────────
       CPU Config         │    Task Duration    │   Tasks per Day  │    Estimated PPD
──────────────────────────────────────────────────────────────────────────────────────
  64 Tasks x  1 Threads   │       0:45:18       │     2034.040     │          294,671★
 128 Tasks x  1 Threads   │       1:34:54       │     1942.142     │          281,358
──────────────────────────────────────────────────────────────────────────────────────

This thing isn't much faster than the AMD 7945HX system despite having significantly more cores. All previous benchmarks were done with CPUs that have AVX512 instructions while this one does not. This is probably the main reason why it's not as impressive as the others.

StefanR5R · Sunday at 5:55 AM

Do the 9654 and 9B14 systems have different RAM setups? Socket power limit might differ too.

A core running PrimeGrid LLR/PRST/Genefer (if the data are retained in the CPU caches) pulls all the per-core Wattage that it can get. Therefore the power efficiency of the manufacturing node is of primary importance to the per-core performance here. Zen: GloFo 14nm, Zen 2/3: TSMC 7nm, Zen 4: TSMC 5nm, Zen 5: TSMC 4nm (a tweaked 5nm-generation process), Zen 6: TSMC 2nm.

AVX2/FMA3 --> AVX-512 uplift can be tested in isolation on Zen 4/5 by forcing the application into its respective code path, either by a switch in the application itself, or if it doesn't have one, by a BIOS option or by a kernel boot parameter.
Genefer:

StefanR5R said:
The Genefer application for CPUs has got a command line switch which toggles between different instruction sets:
-x <implementation> set a specific implementation (i32, sse2, sse4, avx, fma, 512)
I tried the default AVX512 and also the Zen-3-style FMA3 (-x fma) on EPYC 9554P @ 400W with genefer -n 20 -b 2615062. This workunit gets 34,066.53 credit.
FMA3:

Code:

tasks x threads, affinity | avg. task duration | tasks/day | points/day | power | efficiency --------------------------+-----------------------+-----------+------------+-------+------------ 8x8, ascending | 5:54:07 = 21247 s | 32.5 | 1,108,218 | 475 W | 2,330 PPD/W

AVX512:

Code:

tasks x threads, affinity | avg. task duration | tasks/day | points/day | power | efficiency --------------------------+-----------------------+-----------+------------+-------+------------ 8x8, ascending | 5:09:43 = 18583 s | 37.1 | 1,267,104 | 474 W | 2,670 PPD/W

So in this specific case, AVX512 gives +14.3 % throughput and +14.6 % power efficiency over AVX2 FMA3.

(Same workunit on RTX 4090 with Kaby Lake and Z270 PC: 1,771,460 PPD; 380 W; 4,700 PPD/W)

LLR2:

StefanR5R said:
Here are the tests of llrESP on Zen 4, forced to use FMA3 like Zen 3 would, or using the new AVX-512 features, as Zen 4 does per default.

Software
sllr2_1.3.0_linux64_220821 from https://www.primegrid.com/download/
"LLR2 Program - Version 1.3.0, using Gwnum Library Version 30.9"

Task
Proth prime test of 238411*2^24201228+1 (would give 31,512.33 credits at PrimeGrid)

Host
Epyc 9554P, all BIOS settings at default, OpenSUSE Leap 15.5, graphical desktop shut off

Tests with FMA3
The BIOS has got an AVX512 option which can be set to auto/disabled/enabled, but this is doing nothing at all. Must be a bug of the BIOS. Instead, I forced LLR2 to use FMA3 (256 bits wide AVX2) instead of AVX-512 by means of booting with the kernel command line option clearcpuid=304. I stole this trick from Phoronix' report AMD Zen 4 AVX-512 Performance Analysis On The Ryzen 9 7950X.

Code:

# | task duration | tasks/day | points/day | avg clock | Tctl | system power | efficiency --+-----------------------+-----------+------------+-----------+-------+--------------+----------- 1 | 15:41:41 = 56501 s | 12.2 | 385,490 | 3.67 GHz | 47 °C | 405...435 W | 920 PPD/W 2 | 15:40:07 = 56407 s | 12.2 | 386,120 | 3.45 GHz | 49 °C | 405...425 W | 930 PPD/W 3 | 10:21:40 = 37300 s | 18.5 | 583,923 | 3.18 GHz | 51 °C | 439...442 W | 1330 PPD/W 4 | 8:46:59 = 31619 s | 21.8 | 688,859 | 3.01 GHz | 52 °C | 425...426 W | 1620 PPD/W 5 | 22:44:17 = 81857 s | 8.443 | 266,058 | 3.73 GHz | 43 °C | 375...405 W | 680 PPD/W

Code:

>>>> Sun 5 Nov 14:58:12 CET 2023, starting test 1 of 5: 238411*2^24201228+1 (31,512.33 credits), 8 tasks x 8 threads Starting Proth prime test of 238411*2^24201228+1 Using all-complex FMA3 FFT length 2560K, Pass1=512, Pass2=5K, clm=2, 8 threads, a = 3, L2 = 468*404, M = 189072 238411*2^24201228+1, bit: 250000 / 24201227 [1.03%], 189072 checked. Time per bit: 2.373 ms. elapsed: 0:10:01 (601 s), remaining: 15:59:38 (57578 s), total: 16:09:39 (58179 s) min = 55942 s, median = 55942 s, max = 58179 s average task duration: 15:41:41 (56501 s), 12.2 tasks/day, 385,490 points/day <<<< Sun 5 Nov 15:08:13 CET 2023, finished test 1 of 5: 238411*2^24201228+1 (31,512.33 credits), 8 tasks x 8 threads

Tests with AVX-512
That is, after reboot without the mentioned kernel command line parameter.

Code:

# | task duration | tasks/day | points/day | avg clock | Tctl | system power | efficiency --+-----------------------+-----------+------------+-----------+-------+--------------+----------- 1 | 15:40:48 = 56448 s | 12.2 | 385,836 | 3.71 GHz | 46 °C | 380...405 W | 980 PPD/W 2 | 15:15:06 = 54906 s | 12.5 | 396,677 | 3.71 GHz | 47 °C | 385...422 W | 980 PPD/W 3 | 10:38:09 = 38289 s | 18.0 | 568,860 | 3.65 GHz | 50 °C | 426...431 W | 1330 PPD/W 4 | 8:04:01 = 29041 s | 23.8 | 749,993 | 3.43 GHz | 51 °C | 418...420 W | 1790 PPD/W 5 | 22:36:39 = 81399 s | 8.491 | 267,571 | 3.65 GHz | 43 °C | 340...352 W | 770 PPD/W

Code:

>>>> Sun 5 Nov 16:12:06 CET 2023, starting test 1 of 5: 238411*2^24201228+1 (31,512.33 credits), 8 tasks x 8 threads Starting Proth prime test of 238411*2^24201228+1 Using all-complex AVX-512 FFT length 2520K, Pass1=1344, Pass2=1920, clm=1, 8 threads, a = 3, L2 = 468*404, M = 189072 238411*2^24201228+1, bit: 250000 / 24201227 [1.03%], 189072 checked. Time per bit: 2.334 ms. elapsed: 0:10:00 (600 s), remaining: 15:58:02 (57482 s), total: 16:08:02 (58082 s) min = 53780 s, median = 58082 s, max = 58082 s average task duration: 15:40:48 (56448 s), 12.2 tasks/day, 385,836 points/day <<<< Sun 5 Nov 16:22:06 CET 2023, finished test 1 of 5: 238411*2^24201228+1 (31,512.33 credits), 8 tasks x 8 threads

Explanation of the table contents

Each test ran only ten minutes. After that, progress was checked, and total task duration was extrapolated from this. Task duration is trivially converted to tasks/day (in all tests, 8 tasks were running in parallel) and points/day (PPD).

"avg clock" is the average of core clocks across all logical CPUs, sampled at circa 5 minutes into a test. If you dig into the results, you will notice that high clocks indicate that the CPUs didn't have much to do (waited a lot for RAM accesses to complete), whereas low clocks indicate that the vector arithmetic units were hard at work.

"Tctl" is the reading of the "controlling" CPU temperature at the same time. Tccd1…Tccd8 are always a little lower than Tctl. Ambient temperature was at a constant 24 °C during all tests, and the liquid cooling system operated with a fixed pump speed and fixed fan speeds at all times. Thus, Tctl and power draw correlate closely in these tests.

"system power" was also obtained at ~5 minutes into each test. It is the reading on a kill-a-watt, that is, includes not only CPU power draw but RAM, SSD, cooling system (which was constant during the tests), and PSU conversion losses (it's a platinum PSU).

"efficiency" is "points/day" divided by "system power".

Now the important bits, about the first column:
All of the 5 tests were made with the mentioned test candidate (spoiler: it's composite, not prime), and with 8 simultaneous tasks, and 8 threads per task (so, 64 software threads in total on this 64c/128t hardware, which consists of 8 CCXs with 8c/16t/32MB L3$ each). Each task has got ~20 MB cache footprint.
What differed between the 5 tests was CPU affinity.

No affinity; the software threads were randomly spread by the OS kernel over the 128 hardware threads.

8x affinity masks of 0-63. This informed the OS to leave the SMT threads alone and use only real cores. As you can see, this makes only little difference because the Linux kernel isn't totally dumb.

Pairs of tasks were assigned to pairs of CCXs. A similar situation would arise if you would operate a 7950X ( = 2 8c/16t CCXs) without affinity. Again only the lower half of the SMT sibling threads was admitted.

Each task was assigned to one CCX exclusively. This is the only one of the 5 tests which guaranteed that each task had enough level 3 cache available for itself, and that inter-thread communications could happen within cache. Only the lower half of the SMT sibling threads was admitted here as well.

This test was set up just for the LULZ: Each task was bound to 8 logical CPUs which came from 8 different CCXs. In other words, this test forced 8 concurrent tasks onto each CCX. Only the lower half of the SMT sibling threads was used.

Code:

1) 0-127 0-127 0-127 0-127 0-127 0-127 0-127 0-127 2) 0-63 0-63 0-63 0-63 0-63 0-63 0-63 0-63 3) 0-15 0-15 16-31 16-31 32-47 32-47 48-63 48-63 4) 0-7 8-15 16-23 24-31 32-39 40-47 48-55 56-63 5) 0,8,16,24,32,40,48,56 1,9,17,25,33,41,49,57 2,10,18,26,34,42,50,58 3,11,19,27,35,43,51,59 4,12,20,28,36,44,52,60 5,13,21,29,37,45,53,61 6,14,22,30,38,46,54,62 7,15,23,31,39,47,55,63

Notes:
– Linux numbers logical CPUs differently from Windows.
– The masks were applied to each task as a whole. All individual threads of the task inherited the same mask. Thus, how the threads were scheduled within the constraints of these masks was up to the operating system.

Conclusion
With the exception of test #3, AVX-512 provided superior performance and power efficiency over FMA3. Note however that all tests except #4 were heavily influenced by randomness in the thread scheduling, and by GMI/ IMC/ RAM performance. Vice versa, these latter influences were eliminated in test #4 and the vector units used to their fullest, highlighting the FAM3 -> AVX-512 improvements.

The reason for the AVX2/FMA3 --> AVX-512 uplift in Zen 4 running PrimeGrid LLR2/PRST/Genefer specifically is probably not due to new instructions but due to fewer instructions doing the same work:

StefanR5R said:
Other posters in the CPU subforum have linked to it before, but anyway:
mersenneforum.org > Great Internet Mersenne Prime Search > Hardware > Zen4's AVX512 Teardown
An analysis by the author of y-cruncher. One of the points to take home: Although theoretical peak throughput of Zen3/Zen4 AVX-256 and Zen4 AVX-512 is the same clock-for-clock, moving an application to AVX-512 on Zen4 can reduce bottlenecks of the CPU's frontend ( = utilize the execution units better), and also reduce energy spent in the CPU's frontend ( = spend respectively more of the overall power budget in the execution units and elsewhere).

The Zen 4 --> Zen 5 uplift in PrimeGrid is less due to the change from 5 to 4nm, more due to widening of the vector pipelines' datapath from 256 to 512 bits.
________
(All of the above is for classic chiplet based AMD CPUs. Their monolithic APUs and the dense CPUs differ in some regards.)

Skillz · Sunday at 8:29 AM

StefanR5R said:
Do the 9654 and 9B14 systems have different RAM setups? Socket power limit might differ too.

They do have similar RAM setups, but the socket power. That's something I forgot about. The 9654 is 360W while the 9B14 is 400W.

Skillz · 2026-08-12T19:19:53-0400

I'm seeing between 781k and 833k PPD on the 9654 single socket setup running the sub project for ~3 days now. Ignoring the first day since it's a partial update + pendings haven't settled out.

The dual 9B14 is seeing between 1.73m and 1.84m points per day across 3 days, ignoring the first day update as well.

Primegrid PPSE Benchmarks

Skillz

FPS Regular

Skillz

FPS Regular

Skillz

FPS Regular

Skillz

FPS Regular

Skillz

FPS Regular

StefanR5R

Sort-of-Regular

Skillz

FPS Regular

Skillz

FPS Regular