Content creation benchmarks

keljian

n00b
Joined
Jun 6, 2019
Messages
28
Points
3
Thus far the only somewhat reliable benchmark/review house for content creation is puget systems. That said, they have a vested interest in selling systems so...

While this is "The FPS review", one way you can boost your audience is to look at common content creation packages Adobe CC, Autocad(fusion, inventor, 3dsmax), solidworks, Davinci, excel, visual studio and create benchmarks for them.

No one else is doing this
 

Dan_D

Administrator
Staff member
Joined
May 6, 2019
Messages
597
Points
93
The short answer: We are considering many things for evolving our format and trying to bring more value to our readers in everything we do or plan to do in the future. Right now, we have a predominantly gaming oriented format and I don't see that changing. However, never say never. We really don't know where this path will lead us.

However, the real answer to your query is a much longer topic. I can't speak for Brent, David or the others, but I'll comment on my areas specifically, that being motherboards and CPU's. The full answer may seem to jump around a bit, but some comments need additional context for clarity. I hope this answer is sufficient. (However, I do not promise, nor think you will like it.)

As it is, benchmarking done for something like a motherboard review is extremely time consuming. The focus is obviously primarily slanted towards gaming just like HardOCP's was. So while I think its important to include some benchmarks of that nature, (the nature being content creation and productivity) that will never be our focus. You also have to consider the reason why those applications and benchmarks for those applications you mentioned weren't used by HardOCP and aren't being used for motherboard reviews by other sites. Those reasons are: 1.) Cost of the software. 2.) The fact that reviewers don't necessarily know how to use some of those applications on any level what so ever.

If a pre-existing benchmark is available for those applications, then no real learning is required and that works fine. If pre-existing benchmarks for those applications aren't available, creating them becomes a bigger issue. Not only is it time consuming, but we may not have the prerequisite skills to do so for those applications. Adequate tests for some applications could take quite a while to run. Gooseberry can take an hour to run on some configurations. Cinebench's single thread test isn't exactly quick either. The time required to produce a single motherboard review can take anywhere from 20 to 30 hours to write once its all said and done. When I wrote our original motherboard review for this site, (which you will see very soon), I had to come up with the format, benchmark comparison systems etc. We don't own everything Kyle did, so we couldn't just move on with the same benchmarks we used before. I personally believed that we needed to update our benchmarks and do things a bit differently.

When choosing a benchmark, cost of the software comes into play. All the games had to be purchased as I don't think I owned a single one that I used for the testing. With productivity and synthetic benchmarks, they needed to be low cost if not no cost. Buying software is part of the game, but that's only something you want to do when there is a definite value add for doing so. Benchmarks used for a motherboard or CPU review need to be relatively quick when possible, but more importantly they need to be as automated and consistent as possible. Synthetic benchmarks are fantastic for this purpose as they fit the bill nicely, even though their meaning doesn't necessarily correlate to anything in the real world. Sandra Memory bandwidth is interesting, but many applications don't benefit from memory bandwidth, so its academic at best. It still has value as there are some tuning elements to the BIOS configuration, even if we are talking about automatic settings. So variance between one brand or one model and another can exist even though that's almost never the case. So that's why it gets tested.

In a motherboard review, we don't really care about the CPU. For CPU's, ideally you want the highest end chips because they are more demanding on the motherboard VRM's. They generate the most heat and use the most power. A Core i7 7740X is a poor test of an X299 motherboard because it only has 4c/8t, is limited to two memory channels and 16 PCIe lanes. It doesn't even fully enable the platform. A 9980XE on the other hand is worth using. It enables the platform and punishes the VRM's when overclocked. The point being that aside from stressing the motherboard, the CPU is irrelevant. Largely, whatever data you get out of a benchmark is irrelevant by itself. The motherboard does nothing to determine system performance. It's all about your CPU, RAM, graphics card and sometimes storage.

When it comes to motherboard reviews, it simply makes no sense to invest a bunch of time and money into deep benchmarking of content creation software for one simple reason: Motherboards all perform the same relative to other motherboards with the same chipset, CPU and RAM. The variance between one motherboard or another is within an acceptable margin for error. The primary reason why we run benchmarks in the first place is to stress the motherboard and its subsystems in various ways to determine how reliable the motherboard is expected to be and to see if any anomalies pop up during testing which would indicate driver or BIOS issues. The actual numbers you get are essentially meaningless unless you get a result on one extreme or the other which we've only seen a few times out of a couple of hundred reviews since AMD and Intel went to integrated memory controllers.

For CPU reviews, that's different and there will be some experimentation on format. Determining what benchmarks to use is definitely part of the decision making process. Obviously, we will always have a gaming slant, but HEDT systems are in a crossover realm and both productivity and gaming data is valuable for readers to make an informed decision about what to buy. CPU performance is something that needs to be evaluated and application testing is absolutely an essential part of that. But again, its a matter of procuring the software, whether or not existing benchmarks are available and whether or not we can learn the software in a short enough time to create a standardized test we can use out of it. For Adobe products, this is something we can potentially do as they are available to me. Some of the others you mentioned, not necessarily. I wouldn't have a clue what the hell to do with Visual Studio.

When you get down to it, we are all veteran reviewers of a review site that had a slant towards gaming as a niche in the review industry. We literally created this site to pick up where HardOCP left off and continue the work that we all did for years because we not only enjoyed doing it, but felt that there was a place for that specific type of content. There are other sites out there that get more into the content creation and prosumer side than we ever did at HardOCP.
 

keljian

n00b
Joined
Jun 6, 2019
Messages
28
Points
3
For motherboard reviews, there is simply no point, as you said. But for CPU and GPU reviews there definitely is. I respectfully disagree that they should be confined to the HEDT platform, as many streamers and youtubers use them these days.

Well let’s look at the suggestions:
1. Davinci - free to download and the key features (including rendering) are available with the free version, if you can use premiere, it’s a very short learning curve. This is increasingly used both in the industry and by streamers/youtubers

2. Adobe you covered, premiere and after effects are the two that load up cpus and gpus. The latter you can buy animations for from libraries(eg envato market) for less than $10.

3. AutoCAD inventor/autoCAD/maya/3ds max/fusion- if you’re serious about investigating these, contact the company, explain the use case and it is possible they will offer you a license for free. As you mentioned however, there is the learning curve. These are definitely more niche at the moment but I can see in the near future that these will be used more. As mentioned in the other thread, high schoolers are learning to use them now...

4. Visual studio - again a free product for the kind of use you’re talking, and chromium is open source/used as an example for some. This has a learning curve, but there are detailed instructions for doing the compile....
 
Last edited:

Dan_D

Administrator
Staff member
Joined
May 6, 2019
Messages
597
Points
93
For motherboard reviews, there is simply no point, as you said. But for CPU and GPU reviews there definitely is. I respectfully disagree that they should be confined to the HEDT platform, as many streamers and youtubers use them these days.
I never meant to suggest that they should be confined to the HEDT platform. I merely stated this is one reason why we might include such benchmarks. I think those are of more value to people using HEDT hardware, but I realize that there are people who use productivity applications like that on more mainstream hardware. I'm not speaking about use in GPU reviews. That's not my area. That's entirely up to Brent.

Well let’s look at the suggestions:
1. Davinci - free to download and the key features (including rendering) are available with the free version, if you can use premiere, it’s a very short learning curve. This is increasingly used both in the industry and by streamers/youtubers
I haven't ever heard of this software. It would depend on what kind of test we could come up with and how quickly it could be done. Again, the point is to stress the hardware, not really provide performance metrics for it. Regarding CPU reviews, this is a software package I'm willing to take a look at.

2. Adobe you covered, premiere and after effects are the two that load up cpus and gpus. The latter you can buy animations for from libraries(eg envato market) for less than $10.
I actually have these two software packages as part of my Adobe Cloud subscription. However, I do not really use them myself. One point your missing is that I need some sort of benchmark test for them, especially since I don't really know this software. There is another point I didn't bring up which is installation time of all the related software packages on the test box. The testing procedures we have now are quite long. I am looking at doing something using one of these software packages, but exactly which application and what that test will look like is still up in the air.

3. AutoCAD inventor/autoCAD/maya/3ds max/fusion- if you’re serious about investigating these, contact the company, explain the use case and it is possible they will offer you a license for free. As you mentioned however, there is the learning curve. These are definitely more niche at the moment but I can see in the near future that these will be used more. As mentioned in the other thread, high schoolers are learning to use them now...
Yes there is a learning curve here. I used to support these some ten years ago, but I only learned to do very basic things in them. Admittedly, I haven't learned to do enough to make a worth while test. Again, there are costs involved here, a learning curve and again, this continues to add time and complexity to the review procedures. These are already longer and more complex than you probably realize.

The value for our readers is something certainly debatable, but again we have a gaming slant. We can't be the alpha and the omega for everyone. There is a reason why we maintain the gaming slant we had at HardOCP.

4. Visual studio - again a free product for the kind of use you’re talking, and chromium is open source/used as an example for some. This has a learning curve, but there are detailed instructions for doing the compile....
Again, this is beyond the scope of what we normally cover. We all have day jobs and only so much time to learn and continue to get work done. At the end of the day, we have to choose what kind of content we want to do, and chose to focus primarily on gaming. We may branch out some, but that's something we'll sort of feel out as time goes. The benchmarks aren't necessarily set in stone and will continually be evaluated and adjusted.
 

Dan_D

Administrator
Staff member
Joined
May 6, 2019
Messages
597
Points
93
I appreciate the link. I will definitely look into adding this one to the lineup.
 
Become a Patron!
Top