Frame Capture and Analysis Tools Reviewccokeman - April 1, 2013
Category: Video Cards
» Discuss this article (5)
Over the past years, evaluations of video card performance have centered around actual game play or the use of time demos. The premise is to measure how a particular video card plays through a selected game sequence, determining the performance results measured either by an in-game frame rate capture or the use of FRAPS, a popular frame rate measuring tool. The problem is that many times the actual frame rates presented just do not match up with the gaming experience reported by the masses or for that matter us as editors, especially when it comes to multiple GPU configurations. Lately there has been a tremendous amount of buzz around looking deeper into this phenomena by again using FRAPS to measure in conjunction with the reported frames per second; the length of time needed to generate each frame sent to the display.
However, you are still using a software tool that measures the FPS data output from the game engine to the DirectX layer, giving you a false sense of what is going on, which we will explain later. Measuring what actually gets to the viewer is going to give a better representation of the displayed FPS instead of taking the reported software value as the truth, especially when your lying eyes tell you so much different. It's not an entirely new thought stream but is now getting traction as a way of determining the real performance for the end user.
Recently I was introduced to a set of hardware and software tools that NVIDIA has developed and have been using in-house for a little over two years now to improve gaming performance for the end user. Up front the tools have been provided by NVIDIA with the look going forward that these tools will be adopted and expanded upon by the open source community to really dig a bit deeper into what drives gaming performance metrics. NVIDIA calls this tool FCAT, or Frame Capture Analysis Tool. Let's start with a look at the hardware requirements for capturing the data needed for the analysis.
Why do we need to look at a hardware-based capture solution when we have software solutions to do this for us? Software solutions are counting frames that are generated by the game engine rather than what actually gets delivered to the end user's display or monitor. To measure the data sent out of the DVI port requires a set of hardware tools that can capture and store the data. What we have is a multi-part solution that includes a specially built capture system, preferably Intel Z77-based; a gaming system that has no special requirements. The capture system needs to be configured with a three to four disk RAID 0 storage drive that can handle the massive data stream delivered by the capture card. A configuration setup on the Intel controller is preferred to handle data rates of up to 650MB/sec when capturing outputs up to 2560x1440. Above that the capture card cannot handle the throughput. PCIe-based disk drive is another option to handle the data. At the heart of the capture system is the DataPath Limited VisionDVI-DL capture card that captures the full resolution data at 60Hz. Splitting the display signal to feed the DataPath Limited VisionDVI-DL capture card is the job of a Gefen DVI DL Splitter that sends the display signal to the display and the capture card.
Software: The software portion of the FCAT toolbox includes proprietary and programs readily available for the end user.
- Overlay: DXFrameOverlay.dll + Enable Overlay.exe Overlays a fixed color sequence over the game while it’s running
- Extractor: Extractor.exe Output = CSV of overlay colors and scanlines
- Analysis Tools: run_doall.bat, run_nv.bat, run_amd.bat These are batch files that expedite the use of the PERL script files below.
- fcat.pl: PERL script that combines bars and identifies runts and drops , Output = CSV of HW frametimes (similar to Fraps), and CSV of Original FPS, New FPS, Runts, drops
- gen_percentiles.pl PERL script that generates the 95 and 99th frametime percentile calculations.
- pivot.pl PERL script that generates a summary CSV file (good for a pivot table).
- doall.pl PERL helper script to generate a big batch file.
In addition to the batch files and perl scripts we will be using Virtual Dub as the capture medium and utility used to play back the recorded benchmark runs to easily identify both runt and dropped frames. Datapath's Vision software is used to to verify the display output is what is being passed through to the capture card. Extractor is used to analyze the overlay data, which is output to a .CSV file for further analysis and incorporation into the output charts and graphs. Overlay is the software that provides the visual indicator for each frame during the benchmark sequence again making it easy to identify runt and dropped frames.
So why go through all the trouble of setting up this kind of test regimen and investing in the hardware and software to visualize the actual performance into such low level detail? It's about digging deeper into the root causes for the performance issues driving gamers to write tomes on the subjects in forums across the Web. Especially when confronted with the stuttering and unrealistic FPS numbers that just do not add up to the game play experience. In no way is this article going to be as comprehensive as I would like but it gives a good start to how and why we will begin incorporating these metrics into our upcoming video card reviews.