Welcome Stranger to OCC!Login | Register

Serious Statistics Review



I am actually not sure how much this needs a conclusion, but I will give it one just in case. Prior to doing all of these tests I completed the game, so I could have access to every map, and some familiarity with them all. I played using the Vulkan API in Borderless Fullscreen/Window without v-sync, and after having done all of these tests, that is also what I find to be the best combination, for my system at least. The screen tearing from the lack of v-sync was not that bad and I enjoyed the high framerates, which neither DirectX 11 or OpenGL could quite match. If you are not able to run with Vulkan for some reason, I hope you can use DirectX 11 because it was able to keep up with Vulkan at times, and OpenGL was clearly behind both.

After some more experimenting with R, and remembering that box plots exist (as I said, I am not a statistics guy), I put together the following graphs. They are boxplots of the frame time data for each of the runs on each of the graphs. For those of you not familiar, boxplots show the quartiles of the data, letting us see how the data is distributed. The lower line of the box represents the first quartile (25th percentile), the middle line is the median (50th percentile), and the top line is the third quartile (75th percentile). The two lines sticking out from the box are the whiskers, and show where the data outside of the first and third quartile fall, though the outliers are still shown as dots. These percentiles might not line up with those I was finding earlier in the data, but editing the boxplots to reach to the 1st and 99th does not look as good. (This also removes the outlier dots from the graph, and I think showing them outside the boxplot is valuable.) I should also mention that I did impose a limit on the y-axis of 50 ms, and that there were some runs that had outliers reaching past that, but allowing the graphs to stretch enough to show those points scales down the plots too much to be of any value. Also this ensures the vertical scale for all of the graphs is the same.


Something else we learned is to not worry about the double buffering imposed on DirectX 11 by the Desktop Window Manager, when not in Exclusive Fullscreen. Frames will be dropped to ensure v-sync, without cutting into performance. This cannot be said of the v-sync options in Serious Sam Fusion 2017, which may or may not be good because reduced framerates also mean reduced power and reduced temperatures. It really is up to the player which they want.

As far as OCAT and R are concerned, I will say I do intend to continue using them for future reviews and analyses, though I am not sure how deep into the data I may go. In the future my procedure will likely include having OCAT on in the background and on occasion starting a 300 second run while I play, but without adhering to any specific course. For just reviewing a game I feel that random samples throughout are going to be as or more valuable than repeatable courses, but that is for game reviews. Performance analyses would be a different situation when repetition will be of greater value.

Once I have the data, it is easy enough to process it, and hopefully I will remember to write down whatever happened during the test, to provide context to those numbers, if needed.

I am not sure I will do the videos with frame time overlays for reviews, but they are doable, if the game does not mind the multiple hooks or present other issues. They may be cool, but they can become more trouble than they are worth and that is on a game-by-game basis.

I hope you found this interesting, even if it did get rather repetitive.

  1. Serious Statistics - Introduction
  2. Serious Statistics - Software/Tools
  3. Serious Statistics - Testing and Processing
  4. Serious Statistics - Vulkan - Hatsheput
  5. Serious Statistics - Vulkan - Dunes
  6. Serious Statistics - Vulkan - Thebes - Karnak
  7. Serious Statistics - OpenGL - Hatsheput
  8. Serious Statistics - OpenGL - Dunes
  9. Serious Statistics - OpenGL - Thebes - Karnak
  10. Serious Statistics - DirectX 11 - Hatsheput
  11. Serious Statistics - DirectX 11 - Dunes
  12. Serious Statistics - DirectX 11 - Thebes - Karnak
  13. Serious Statistics - V-Sync Tests
  14. Serious Statistics - Conclusion
Related Products
Random Pic
© 2001-2018 Overclockers Club ® Privacy Policy
Elapsed: 0.2521281242   (xlweb1)