Skip to main content

Methodology

Assessing CI stability​

During performance measurements we measure React component render times with microsecond precision using React.Profiler. This means that the same code will run faster or slower depending on the machine. For this reason, baseline & current measurements need to be run on the same machine. Optimally, they should be run one after another.

Moreover, in order to achieve meaningful results your CI agent needs to have stable performance. It does not matter really if your agent is fast or slow as long as it is consistent in its performance. That's why during the performance tests the agent should not be used for any other work that might impact measuring render times.

In order to help you assess your machine stability, you can use reassure check-stability command. It runs performance measurements twice for the current code, so baseline and current measurements refer to the same code. In such case the expected changes are 0% (no change). The degree of random performance changes will reflect the stability of your machine. This command can be run both on CI and local machines.

Normally, the random changes should be below 5%. Results of 10% and more considered too high and mean that you should work on tweaking your machine stability.

Note: As a trick of last resort you can increase the run option, from the default value of 10 to 20, 50 or even 100, for all or some of your tests, based on the assumption that more test runs will even out measurement fluctuations. That will however make your tests run even longer.

You can refer to our example GitHub workflow.

Analyzing results​

Markdown report

Looking at the example you can notice that test scenarios can be assigned to certain categories:

  • Significant Changes To Duration shows test scenario where the performance change is statistically significant and should be looked into as it marks a potential performance loss/improvement
  • Meaningless Changes To Duration shows test scenarios where the performance change is not stastatistically significant
  • Changes To Count shows test scenarios where the render or execution count did change
  • Added Scenarios shows test scenarios which do not exist in the baseline measurements
  • Removed Scenarios shows test scenarios which do not exist in the current measurements