stats: Add beta support for HDF5 stat dumps

This changeset add support for stat dumps in the HDF5 file
format. HDF5 is a binary data format that represents data in a
file-system-like balanced tree. It has native support for
N-dimensional arrays and binary data (e.g., frame buffers).

It has the following benefits over traditional text stat files:

  * Efficient storage of time series (multiple stat dumps)

  * Fast lookup of stats

  * Plenty of existing tooling (e.g., Python libraries and graphical
    viewers)

  * File format can be used to store frame buffers together with
    normal stats.

Drawbacks:

  * Large startup cost (single stat dump larger than text equivalent)

  * Stat dumps are slower than text

Known limitations:

  * Distributions and histograms aren't supported.

HDF5 stat output can be enabled using the 'h5' URL scheme when
overriding the stat file name on gem5's command line. The following
parameters are supported:

  * chunking (unsigned): Number of time steps to pre-allocate
    (default: 10)

  * desc (bool): Output stat descriptions (default: True)

  * formulas (bool): Output derived stats (default: True)

Example gem5 command line:

./build/ARM/gem5.opt \
  --stats-file="h5://stats.h5?desc=False;formulas=False" \
  configs/example/fs.py

Example Python stat consumer that computes IPC:
  import h5py

  f = h5py.File('stats.h5', 'r')
  group = f['/system/cpu']
  for i, c in zip(group['committedInsts'], group['numCycles']):
      print i, c, i / c

Change-Id: I351c6cbff2fb7bef9012f47876ba227ed288975b
Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/8121
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jason Lowe-Power <jason@lowepower.com>
Reviewed-by: Daniel Carvalho <odanrc@yahoo.com.br>
Reviewed-by: Ciro Santilli <ciro.santilli@arm.com>

1 files changed, 43 insertions, 3 deletions

diff --git a/SConstruct b/SConstruct
index 9e27b484c..39e7ccc06 100755
--- a/SConstruct
+++ b/SConstruct
@@ -1,6 +1,6 @@
 # -*- mode:python -*-
 
-# Copyright (c) 2013, 2015-2017, 2019 ARM Limited
+# Copyright (c) 2013, 2015-2019 ARM Limited
 # All rights reserved.
 #
 # The license below extends only to copyright in the software and shall
@@ -591,6 +591,9 @@ if sys.platform == 'cygwin':
     # cygwin has some header file issues...
     main.Append(CCFLAGS=["-Wno-uninitialized"])
 
+
+have_pkg_config = readCommand(['pkg-config', '--version'], exception='')
+
 # Check for the protobuf compiler
 protoc_version = readCommand([main['PROTOC'], '--version'],
                              exception='').split()
@@ -620,7 +623,7 @@ else:
         # protobuf without the involvement of pkg-config. Later on we
         # check go a library config check and at that point the test
         # will fail if libprotobuf cannot be found.
-        if readCommand(['pkg-config', '--version'], exception=''):
+        if have_pkg_config:
             try:
                 # Attempt to establish what linking flags to add for protobuf
                 # using pkg-config
@@ -913,6 +916,42 @@ def is_isa_kvm_compatible(isa):
 main['HAVE_PERF_ATTR_EXCLUDE_HOST'] = conf.CheckMember(
     'linux/perf_event.h', 'struct perf_event_attr', 'exclude_host')
 
+def check_hdf5():
+    return \
+        conf.CheckLibWithHeader('hdf5', 'hdf5.h', 'C',
+                                'H5Fcreate("", 0, 0, 0);') and \
+        conf.CheckLibWithHeader('hdf5_cpp', 'H5Cpp.h', 'C++',
+                                'H5::H5File("", 0);')
+
+def check_hdf5_pkg(name):
+    print("Checking for %s using pkg-config..." % name, end="")
+    if not have_pkg_config:
+        print(" pkg-config not found")
+        return False
+
+    try:
+        main.ParseConfig('pkg-config --cflags-only-I --libs-only-L %s' % name)
+        print(" yes")
+        return True
+    except:
+        print(" no")
+        return False
+
+# Check if there is a pkg-config configuration for hdf5. If we find
+# it, setup the environment to enable linking and header inclusion. We
+# don't actually try to include any headers or link with hdf5 at this
+# stage.
+if not check_hdf5_pkg('hdf5-serial'):
+    check_hdf5_pkg('hdf5')
+
+# Check if the HDF5 libraries can be found. This check respects the
+# include path and library path provided by pkg-config. We perform
+# this check even if there isn't a pkg-config configuration for hdf5
+# since some installations don't use pkg-config.
+have_hdf5 = check_hdf5()
+if not have_hdf5:
+    print("Warning: Couldn't find any HDF5 C++ libraries. Disabling")
+    print("         HDF5 support.")
 
 ######################################################################
 #
@@ -1018,6 +1057,7 @@ sticky_vars.AddVariables(
                  backtrace_impls[-1], backtrace_impls),
     ('NUMBER_BITS_PER_SET', 'Max elements in set (default 64)',
                  64),
+    BoolVariable('USE_HDF5', 'Enable the HDF5 support', have_hdf5),
     )
 
 # These variables get exported to #defines in config/*.hh (see src/SConscript).
@@ -1025,7 +1065,7 @@ export_vars += ['USE_FENV', 'SS_COMPATIBLE_FP', 'TARGET_ISA', 'TARGET_GPU_ISA',
                 'CP_ANNOTATE', 'USE_POSIX_CLOCK', 'USE_KVM', 'USE_TUNTAP',
                 'PROTOCOL', 'HAVE_PROTOBUF', 'HAVE_VALGRIND',
                 'HAVE_PERF_ATTR_EXCLUDE_HOST', 'USE_PNG',
-                'NUMBER_BITS_PER_SET']
+                'NUMBER_BITS_PER_SET', 'USE_HDF5']
 
 ###################################################
 #