tests: Don't treat new stats as a cause for failures

We currently fail the stat diff stage of tests if there are new
stats. This is usually undesirable since this would require any change
that adds a stat to also update the regressions.

Change-Id: Ieadebac6fd17534e1b49b6b9a1d56f037a423325
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/3962
Reviewed-by: Anthony Gutierrez <anthony.gutierrez@amd.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Maintainer: Jason Lowe-Power <jason@lowepower.com>

1 files changed, 6 insertions, 2 deletions

diff --git a/tests/diff-out b/tests/diff-out
index c00d4f3de..1eeac8137 100755
--- a/tests/diff-out
+++ b/tests/diff-out
@@ -361,8 +361,12 @@ if ($added_stats)
 }
 
 cleanup();
-# Exit code is 0 if all stats are found (with no extras) & no stats error, 1 otherwise
-$status = ($missing_stats == 0 && $added_stats == 0 && $max_err_mag == 0.0) ? 0 : 1;
+# Exit codes:
+# 0 if all stats are found (with no extras) & no stats error
+# 1 if there are additional stats, but no stat errors
+# 2 otherwise
+$no_hard_errors = $missing_stats == 0 && $max_err_mag == 0.0;
+$status = $no_hard_errors ? ($added_stats == 0 ? 0 : 1) : 2;
 exit $status;
 
 sub cleanup