config: Add the ability to read a config file using C++ and Python

This patch adds the ability to load in config.ini files generated from
gem5 into another instance of gem5 built without Python configuration
support. The intended use case is for configuring gem5 when it is a
library embedded in another simulation system.

A parallel config file reader is also provided purely in Python to
demonstrate the approach taken and to provided similar functionality
for as-yet-unknown use models. The Python configuration file reader
can read both .ini and .json files.

C++ configuration file reading:

A command line option has been added for scons to enable C++ configuration
file reading: --with-cxx-config

There is an example in util/cxx_config that shows C++ configuration in action.
util/cxx_config/README explains how to build the example.

Configuration is achieved by the object CxxConfigManager. It handles
reading object descriptions from a CxxConfigFileBase object which
wraps a config file reader. The wrapper class CxxIniFile is provided
which wraps an IniFile for reading .ini files. Reading .json files
from C++ would be possible with a similar wrapper and a JSON parser.

After reading object descriptions, CxxConfigManager creates
SimObjectParam-derived objects from the classes in the (generated with this
patch) directory build/ARCH/cxx_config

CxxConfigManager can then build SimObjects from those SimObjectParams (in an
order dictated by the SimObject-value parameters on other objects) and bind
ports of the produced SimObjects.

A minimal set of instantiate-replacing member functions are provided by
CxxConfigManager and few of the member functions of SimObject (such as drain)
are extended onto CxxConfigManager.

Python configuration file reading (configs/example/read_config.py):

A Python version of the reader is also supplied with a similar interface to
CxxConfigFileBase (In Python: ConfigFile) to config file readers.

The Python config file reading will handle both .ini and .json files.

The object construction strategy is slightly different in Python from the C++
reader as you need to avoid objects prematurely becoming the children of other
objects when setting parameters.

Port binding also needs to be strictly in the same port-index order as the
original instantiation.

2 files changed, 428 insertions, 0 deletions

diff --git a/src/python/m5/SimObject.py b/src/python/m5/SimObject.py
index 9f4c2c155..40203a307 100644
--- a/src/python/m5/SimObject.py
+++ b/src/python/m5/SimObject.py
@@ -114,6 +114,279 @@ def public_value(key, value):
                isinstance(value, (FunctionType, MethodType, ModuleType,
                                   classmethod, type))
 
+def createCxxConfigDirectoryEntryFile(code, name, simobj, is_header):
+    entry_class = 'CxxConfigDirectoryEntry_%s' % name
+    param_class = '%sCxxConfigParams' % name
+
+    code('#include "params/%s.hh"' % name)
+
+    if not is_header:
+        for param in simobj._params.values():
+            if isSimObjectClass(param.ptype):
+                code('#include "%s"' % param.ptype._value_dict['cxx_header'])
+                code('#include "params/%s.hh"' % param.ptype.__name__)
+            else:
+                param.ptype.cxx_ini_predecls(code)
+
+    if is_header:
+        member_prefix = ''
+        end_of_decl = ';'
+        code('#include "sim/cxx_config.hh"')
+        code()
+        code('class ${param_class} : public CxxConfigParams,'
+            ' public ${name}Params')
+        code('{')
+        code('  private:')
+        code.indent()
+        code('class DirectoryEntry : public CxxConfigDirectoryEntry')
+        code('{')
+        code('  public:')
+        code.indent()
+        code('DirectoryEntry();');
+        code()
+        code('CxxConfigParams *makeParamsObject() const')
+        code('{ return new ${param_class}; }')
+        code.dedent()
+        code('};')
+        code()
+        code.dedent()
+        code('  public:')
+        code.indent()
+    else:
+        member_prefix = '%s::' % param_class
+        end_of_decl = ''
+        code('#include "%s"' % simobj._value_dict['cxx_header'])
+        code('#include "base/str.hh"')
+        code('#include "cxx_config/${name}.hh"')
+
+        if simobj._ports.values() != []:
+            code('#include "mem/mem_object.hh"')
+            code('#include "mem/port.hh"')
+
+        code()
+        code('${member_prefix}DirectoryEntry::DirectoryEntry()');
+        code('{')
+
+        def cxx_bool(b):
+            return 'true' if b else 'false'
+
+        code.indent()
+        for param in simobj._params.values():
+            is_vector = isinstance(param, m5.params.VectorParamDesc)
+            is_simobj = issubclass(param.ptype, m5.SimObject.SimObject)
+
+            code('parameters["%s"] = new ParamDesc("%s", %s, %s);' %
+                (param.name, param.name, cxx_bool(is_vector),
+                cxx_bool(is_simobj)));
+
+        for port in simobj._ports.values():
+            is_vector = isinstance(port, m5.params.VectorPort)
+            is_master = port.role == 'MASTER'
+
+            code('ports["%s"] = new PortDesc("%s", %s, %s);' %
+                (port.name, port.name, cxx_bool(is_vector),
+                cxx_bool(is_master)))
+
+        code.dedent()
+        code('}')
+        code()
+
+    code('bool ${member_prefix}setSimObject(const std::string &name,')
+    code('    SimObject *simObject)${end_of_decl}')
+
+    if not is_header:
+        code('{')
+        code.indent()
+        code('bool ret = true;')
+        code()
+        code('if (false) {')
+        for param in simobj._params.values():
+            is_vector = isinstance(param, m5.params.VectorParamDesc)
+            is_simobj = issubclass(param.ptype, m5.SimObject.SimObject)
+
+            if is_simobj and not is_vector:
+                code('} else if (name == "${{param.name}}") {')
+                code.indent()
+                code('this->${{param.name}} = '
+                    'dynamic_cast<${{param.ptype.cxx_type}}>(simObject);')
+                code('if (simObject && !this->${{param.name}})')
+                code('   ret = false;')
+                code.dedent()
+        code('} else {')
+        code('    ret = false;')
+        code('}')
+        code()
+        code('return ret;')
+        code.dedent()
+        code('}')
+
+    code()
+    code('bool ${member_prefix}setSimObjectVector('
+        'const std::string &name,')
+    code('    const std::vector<SimObject *> &simObjects)${end_of_decl}')
+
+    if not is_header:
+        code('{')
+        code.indent()
+        code('bool ret = true;')
+        code()
+        code('if (false) {')
+        for param in simobj._params.values():
+            is_vector = isinstance(param, m5.params.VectorParamDesc)
+            is_simobj = issubclass(param.ptype, m5.SimObject.SimObject)
+
+            if is_simobj and is_vector:
+                code('} else if (name == "${{param.name}}") {')
+                code.indent()
+                code('this->${{param.name}}.clear();')
+                code('for (auto i = simObjects.begin(); '
+                    'ret && i != simObjects.end(); i ++)')
+                code('{')
+                code.indent()
+                code('${{param.ptype.cxx_type}} object = '
+                    'dynamic_cast<${{param.ptype.cxx_type}}>(*i);')
+                code('if (*i && !object)')
+                code('    ret = false;')
+                code('else')
+                code('    this->${{param.name}}.push_back(object);')
+                code.dedent()
+                code('}')
+                code.dedent()
+        code('} else {')
+        code('    ret = false;')
+        code('}')
+        code()
+        code('return ret;')
+        code.dedent()
+        code('}')
+
+    code()
+    code('void ${member_prefix}setName(const std::string &name_)'
+        '${end_of_decl}')
+
+    if not is_header:
+        code('{')
+        code.indent()
+        code('this->name = name_;')
+        code('this->pyobj = NULL;')
+        code.dedent()
+        code('}')
+
+    if is_header:
+        code('const std::string &${member_prefix}getName()')
+        code('{ return this->name; }')
+
+    code()
+    code('bool ${member_prefix}setParam(const std::string &name,')
+    code('    const std::string &value, const Flags flags)${end_of_decl}')
+
+    if not is_header:
+        code('{')
+        code.indent()
+        code('bool ret = true;')
+        code()
+        code('if (false) {')
+        for param in simobj._params.values():
+            is_vector = isinstance(param, m5.params.VectorParamDesc)
+            is_simobj = issubclass(param.ptype, m5.SimObject.SimObject)
+
+            if not is_simobj and not is_vector:
+                code('} else if (name == "${{param.name}}") {')
+                code.indent()
+                param.ptype.cxx_ini_parse(code,
+                    'value', 'this->%s' % param.name, 'ret =')
+                code.dedent()
+        code('} else {')
+        code('    ret = false;')
+        code('}')
+        code()
+        code('return ret;')
+        code.dedent()
+        code('}')
+
+    code()
+    code('bool ${member_prefix}setParamVector('
+        'const std::string &name,')
+    code('    const std::vector<std::string> &values,')
+    code('    const Flags flags)${end_of_decl}')
+
+    if not is_header:
+        code('{')
+        code.indent()
+        code('bool ret = true;')
+        code()
+        code('if (false) {')
+        for param in simobj._params.values():
+            is_vector = isinstance(param, m5.params.VectorParamDesc)
+            is_simobj = issubclass(param.ptype, m5.SimObject.SimObject)
+
+            if not is_simobj and is_vector:
+                code('} else if (name == "${{param.name}}") {')
+                code.indent()
+                code('${{param.name}}.clear();')
+                code('for (auto i = values.begin(); '
+                    'ret && i != values.end(); i ++)')
+                code('{')
+                code.indent()
+                code('${{param.ptype.cxx_type}} elem;')
+                param.ptype.cxx_ini_parse(code,
+                    '*i', 'elem', 'ret =')
+                code('if (ret)')
+                code('    this->${{param.name}}.push_back(elem);')
+                code.dedent()
+                code('}')
+                code.dedent()
+        code('} else {')
+        code('    ret = false;')
+        code('}')
+        code()
+        code('return ret;')
+        code.dedent()
+        code('}')
+
+    code()
+    code('bool ${member_prefix}setPortConnectionCount('
+        'const std::string &name,')
+    code('    unsigned int count)${end_of_decl}')
+
+    if not is_header:
+        code('{')
+        code.indent()
+        code('bool ret = true;')
+        code()
+        code('if (false)')
+        code('    ;')
+        for port in simobj._ports.values():
+            code('else if (name == "${{port.name}}")')
+            code('    this->port_${{port.name}}_connection_count = count;')
+        code('else')
+        code('    ret = false;')
+        code()
+        code('return ret;')
+        code.dedent()
+        code('}')
+
+    code()
+    code('SimObject *${member_prefix}simObjectCreate()${end_of_decl}')
+
+    if not is_header:
+        code('{')
+        if hasattr(simobj, 'abstract') and simobj.abstract:
+            code('    return NULL;')
+        else:
+            code('    return this->create();')
+        code('}')
+
+    if is_header:
+        code()
+        code('static CxxConfigDirectoryEntry'
+            ' *${member_prefix}makeDirectoryEntry()')
+        code('{ return new DirectoryEntry; }')
+
+    if is_header:
+        code.dedent()
+        code('};')
+
 # The metaclass for SimObject.  This class controls how new classes
 # that derive from SimObject are instantiated, and provides inherited
 # class behavior (just like a class controls how instances of that
@@ -583,6 +856,11 @@ struct PyObject;
         code('#endif // __PARAMS__${cls}__')
         return code
 
+    # Generate the C++ declaration/definition files for this SimObject's
+    # param struct to allow C++ initialisation
+    def cxx_config_param_file(cls, code, is_header):
+        createCxxConfigDirectoryEntryFile(code, cls.__name__, cls, is_header)
+        return code
 
 # This *temporary* definition is required to support calls from the
 # SimObject class definition to the MetaSimObject methods (in
diff --git a/src/python/m5/params.py b/src/python/m5/params.py
index f16cabaff..b7df7c660 100644
--- a/src/python/m5/params.py
+++ b/src/python/m5/params.py
@@ -120,6 +120,18 @@ class ParamValue(object):
     def config_value(self):
         return str(self)
 
+    # Prerequisites for .ini parsing with cxx_ini_parse
+    @classmethod
+    def cxx_ini_predecls(cls, code):
+        pass
+
+    # parse a .ini file entry for this param from string expression
+    # src into lvalue dest (of the param's C++ type)
+    @classmethod
+    def cxx_ini_parse(cls, code, src, dest, ret):
+        code('// Unhandled param type: %s' % cls.__name__)
+        code('%s false;' % ret)
+
     # allows us to blithely call unproxy() on things without checking
     # if they're really proxies or not
     def unproxy(self, base):
@@ -454,6 +466,11 @@ class String(ParamValue,str):
         self = value
         return value
 
+    @classmethod
+    def cxx_ini_parse(self, code, src, dest, ret):
+        code('%s = %s;' % (dest, src))
+        code('%s true;' % ret)
+
     def getValue(self):
         return self
 
@@ -500,6 +517,19 @@ class NumericParamValue(ParamValue):
     def config_value(self):
         return self.value
 
+    @classmethod
+    def cxx_ini_predecls(cls, code):
+        # Assume that base/str.hh will be included anyway
+        # code('#include "base/str.hh"')
+        pass
+
+    # The default for parsing PODs from an .ini entry is to extract from an
+    # istringstream and let overloading choose the right type according to
+    # the dest type.
+    @classmethod
+    def cxx_ini_parse(self, code, src, dest, ret):
+        code('%s to_number(%s, %s);' % (ret, src, dest))
+
 # Metaclass for bounds-checked integer parameters.  See CheckedInt.
 class CheckedIntType(MetaParamValue):
     def __init__(cls, name, bases, dict):
@@ -592,6 +622,20 @@ class Cycles(CheckedInt):
         from m5.internal.core import Cycles
         return Cycles(self.value)
 
+    @classmethod
+    def cxx_ini_predecls(cls, code):
+        # Assume that base/str.hh will be included anyway
+        # code('#include "base/str.hh"')
+        pass
+
+    @classmethod
+    def cxx_ini_parse(cls, code, src, dest, ret):
+        code('uint64_t _temp;')
+        code('bool _ret = to_number(%s, _temp);' % src)
+        code('if (_ret)')
+        code('    %s = Cycles(_temp);' % dest)
+        code('%s _ret;' % ret)
+
 class Float(ParamValue, float):
     cxx_type = 'double'
     cmdLineSettable = True
@@ -613,6 +657,14 @@ class Float(ParamValue, float):
     def config_value(self):
         return self
 
+    @classmethod
+    def cxx_ini_predecls(cls, code):
+        code('#include <sstream>')
+
+    @classmethod
+    def cxx_ini_parse(self, code, src, dest, ret):
+        code('%s (std::istringstream(%s) >> %s).eof();' % (ret, src, dest))
+
 class MemorySize(CheckedInt):
     cxx_type = 'uint64_t'
     ex_str = '512MB'
@@ -738,6 +790,24 @@ class AddrRange(ParamValue):
     def swig_predecls(cls, code):
         Addr.swig_predecls(code)
 
+    @classmethod
+    def cxx_ini_predecls(cls, code):
+        code('#include <sstream>')
+
+    @classmethod
+    def cxx_ini_parse(cls, code, src, dest, ret):
+        code('uint64_t _start, _end;')
+        code('char _sep;')
+        code('std::istringstream _stream(${src});')
+        code('_stream >> _start;')
+        code('_stream.get(_sep);')
+        code('_stream >> _end;')
+        code('bool _ret = !_stream.fail() &&'
+            '_stream.eof() && _sep == \':\';')
+        code('if (_ret)')
+        code('   ${dest} = AddrRange(_start, _end);')
+        code('${ret} _ret;')
+
     def getValue(self):
         # Go from the Python class to the wrapped C++ class generated
         # by swig
@@ -783,6 +853,16 @@ class Bool(ParamValue):
     def config_value(self):
         return self.value
 
+    @classmethod
+    def cxx_ini_predecls(cls, code):
+        # Assume that base/str.hh will be included anyway
+        # code('#include "base/str.hh"')
+        pass
+
+    @classmethod
+    def cxx_ini_parse(cls, code, src, dest, ret):
+        code('%s to_bool(%s, %s);' % (ret, src, dest))
+
 def IncEthernetAddr(addr, val = 1):
     bytes = map(lambda x: int(x, 16), addr.split(':'))
     bytes[5] += val
@@ -850,6 +930,11 @@ class EthernetAddr(ParamValue):
     def ini_str(self):
         return self.value
 
+    @classmethod
+    def cxx_ini_parse(self, code, src, dest, ret):
+        code('%s = Net::EthAddr(%s);' % (dest, src))
+        code('%s true;' % ret)
+
 # When initializing an IpAddress, pass in an existing IpAddress, a string of
 # the form "a.b.c.d", or an integer representing an IP.
 class IpAddress(ParamValue):
@@ -1154,6 +1239,16 @@ class Time(ParamValue):
         assert false
         return str(self)
 
+    @classmethod
+    def cxx_ini_predecls(cls, code):
+        code('#include <time.h>')
+
+    @classmethod
+    def cxx_ini_parse(cls, code, src, dest, ret):
+        code('char *_parse_ret = strptime((${src}).c_str(),')
+        code('    "%a %b %d %H:%M:%S %Y", &(${dest}));')
+        code('${ret} _parse_ret && *_parse_ret == \'\\0\';');
+
 # Enumerated types are a little more complex.  The user specifies the
 # type as Enum(foo) where foo is either a list or dictionary of
 # alternatives (typically strings, but not necessarily so).  (In the
@@ -1306,6 +1401,19 @@ class Enum(ParamValue):
     def swig_predecls(cls, code):
         code('%import "python/m5/internal/enum_$0.i"', cls.__name__)
 
+    @classmethod
+    def cxx_ini_parse(cls, code, src, dest, ret):
+        code('if (false) {')
+        for elem_name in cls.map.iterkeys():
+            code('} else if (%s == "%s") {' % (src, elem_name))
+            code.indent()
+            code('%s = Enums::%s;' % (dest, elem_name))
+            code('%s true;' % ret)
+            code.dedent()
+        code('} else {')
+        code('    %s false;' % ret)
+        code('}')
+
     def getValue(self):
         return int(self.map[self.value])
 
@@ -1336,6 +1444,16 @@ class TickParamValue(NumericParamValue):
     def getValue(self):
         return long(self.value)
 
+    @classmethod
+    def cxx_ini_predecls(cls, code):
+        code('#include <sstream>')
+
+    # Ticks are expressed in seconds in JSON files and in plain
+    # Ticks in .ini files.  Switch based on a config flag
+    @classmethod
+    def cxx_ini_parse(self, code, src, dest, ret):
+        code('${ret} to_number(${src}, ${dest});')
+
 class Latency(TickParamValue):
     ex_str = "100ns"
 
@@ -1485,6 +1603,14 @@ class Voltage(float,ParamValue):
     def ini_str(self):
         return '%f' % self.getValue()
 
+    @classmethod
+    def cxx_ini_predecls(cls, code):
+        code('#include <sstream>')
+
+    @classmethod
+    def cxx_ini_parse(self, code, src, dest, ret):
+        code('%s (std::istringstream(%s) >> %s).eof();' % (ret, src, dest))
+
 class Current(float, ParamValue):
     cxx_type = 'double'
     ex_str = "1mA"
@@ -1510,6 +1636,14 @@ class Current(float, ParamValue):
     def ini_str(self):
         return '%f' % self.getValue()
 
+    @classmethod
+    def cxx_ini_predecls(cls, code):
+        code('#include <sstream>')
+
+    @classmethod
+    def cxx_ini_parse(self, code, src, dest, ret):
+        code('%s (std::istringstream(%s) >> %s).eof();' % (ret, src, dest))
+
 class NetworkBandwidth(float,ParamValue):
     cxx_type = 'float'
     ex_str = "1Gbps"
@@ -1541,6 +1675,14 @@ class NetworkBandwidth(float,ParamValue):
     def config_value(self):
         return '%f' % self.getValue()
 
+    @classmethod
+    def cxx_ini_predecls(cls, code):
+        code('#include <sstream>')
+
+    @classmethod
+    def cxx_ini_parse(self, code, src, dest, ret):
+        code('%s (std::istringstream(%s) >> %s).eof();' % (ret, src, dest))
+
 class MemoryBandwidth(float,ParamValue):
     cxx_type = 'float'
     ex_str = "1GB/s"
@@ -1571,6 +1713,14 @@ class MemoryBandwidth(float,ParamValue):
     def config_value(self):
         return '%f' % self.getValue()
 
+    @classmethod
+    def cxx_ini_predecls(cls, code):
+        code('#include <sstream>')
+
+    @classmethod
+    def cxx_ini_parse(self, code, src, dest, ret):
+        code('%s (std::istringstream(%s) >> %s).eof();' % (ret, src, dest))
+
 #
 # "Constants"... handy aliases for various values.
 #