summaryrefslogtreecommitdiff
path: root/tests
AgeCommit message (Collapse)Author
2016-12-19stats: update referencesCurtis Dunham
2016-12-05stats: Update stats to reflect cache changesAndreas Hansson
2016-11-30tests: Regression stats updated for recent patchesJason Lowe-Power
2016-11-30riscv: [Patch 8/5] Added some regression tests to RISC-VAlec Roelke
This patch is the eighth patch in a series adding RISC-V to gem5, and third of the bonus patches to the original series of five. It adds some regression tests to RISC-V. Regression tests included: - se/00.hello - se/02.insttest (split into several binaries which are not included due to large size) The tests added to 00.insttest will need to be build manually; to facilitate this, a Makefile is included. The required toolchain and compiler (riscv64-unknown-elf-gcc) can be built from the riscv-tools GitHub repository at https://github.com/riscv/riscv-tools. Note that because EBREAK only makes sense when gdb is running or while in FS mode, it is not included in the linux-rv64i insttest. ERET is not included because it does not make sense in SE mode and, in fact, causes a panic by design. Note also that not every system call is tested in linux-rv64i; of the ones defined in linux/process.hh, some have been given numbers but not definitions for the toolchain, or are merely stubs that always return 0. Of the ones that do work properly, only a subset are tested due to similar functionality. Signed-off by: Alec Roelke Signed-off by: Jason Lowe-Power <jason@lowepower.com>
2016-11-30tests: Check for TrafficGen as part of memcheck regressionAndreas Hansson
Since protobuf is still considered optional we do not always have the TrafficGen. Check before running the memcheck regression.
2016-11-17stats, alpha: Update ALPHA statsAndreas Hansson
Reflect the removal of the syscall tracking.
2016-11-17tests, ruby: Move rubytests from ALPHA (linux) to NULL (none)Andreas Hansson
This patch avoids compiling ALPHA six times as part of running 'util/regress', and instead relis on NULL with different protocols to run the rubytest. All we need is the memory system, so there is really no need to compile the ISA over and over again. The one downside is the removal of running 'hello' for the variuos ALPHA and protocol combinations, but if this is a concern we should rather beef up the synthetic tests for the variuos protocols. --HG-- rename : build_opts/NULL => build_opts/NULL_MESI_Two_Level rename : build_opts/NULL => build_opts/NULL_MOESI_CMP_directory rename : build_opts/NULL => build_opts/NULL_MOESI_CMP_token rename : build_opts/NULL => build_opts/NULL_MOESI_hammer rename : tests/quick/se/60.rubytest/ref/alpha/linux/rubytest-ruby-MESI_Two_Level/config.ini => tests/quick/se/60.rubytest/ref/null/none/rubytest-ruby-MESI_Two_Level/config.ini rename : tests/quick/se/60.rubytest/ref/alpha/linux/rubytest-ruby-MESI_Two_Level/simerr => tests/quick/se/60.rubytest/ref/null/none/rubytest-ruby-MESI_Two_Level/simerr rename : tests/quick/se/60.rubytest/ref/alpha/linux/rubytest-ruby-MESI_Two_Level/simout => tests/quick/se/60.rubytest/ref/null/none/rubytest-ruby-MESI_Two_Level/simout rename : tests/quick/se/60.rubytest/ref/alpha/linux/rubytest-ruby-MESI_Two_Level/stats.txt => tests/quick/se/60.rubytest/ref/null/none/rubytest-ruby-MESI_Two_Level/stats.txt rename : tests/quick/se/60.rubytest/ref/alpha/linux/rubytest-ruby-MOESI_CMP_directory/config.ini => tests/quick/se/60.rubytest/ref/null/none/rubytest-ruby-MOESI_CMP_directory/config.ini rename : tests/quick/se/60.rubytest/ref/alpha/linux/rubytest-ruby-MOESI_CMP_directory/simerr => tests/quick/se/60.rubytest/ref/null/none/rubytest-ruby-MOESI_CMP_directory/simerr rename : tests/quick/se/60.rubytest/ref/alpha/linux/rubytest-ruby-MOESI_CMP_directory/simout => tests/quick/se/60.rubytest/ref/null/none/rubytest-ruby-MOESI_CMP_directory/simout rename : tests/quick/se/60.rubytest/ref/alpha/linux/rubytest-ruby-MOESI_CMP_directory/stats.txt => tests/quick/se/60.rubytest/ref/null/none/rubytest-ruby-MOESI_CMP_directory/stats.txt rename : tests/quick/se/60.rubytest/ref/alpha/linux/rubytest-ruby-MOESI_CMP_token/config.ini => tests/quick/se/60.rubytest/ref/null/none/rubytest-ruby-MOESI_CMP_token/config.ini rename : tests/quick/se/60.rubytest/ref/alpha/linux/rubytest-ruby-MOESI_CMP_token/simerr => tests/quick/se/60.rubytest/ref/null/none/rubytest-ruby-MOESI_CMP_token/simerr rename : tests/quick/se/60.rubytest/ref/alpha/linux/rubytest-ruby-MOESI_CMP_token/simout => tests/quick/se/60.rubytest/ref/null/none/rubytest-ruby-MOESI_CMP_token/simout rename : tests/quick/se/60.rubytest/ref/alpha/linux/rubytest-ruby-MOESI_CMP_token/stats.txt => tests/quick/se/60.rubytest/ref/null/none/rubytest-ruby-MOESI_CMP_token/stats.txt rename : tests/quick/se/60.rubytest/ref/alpha/linux/rubytest-ruby-MOESI_hammer/config.ini => tests/quick/se/60.rubytest/ref/null/none/rubytest-ruby-MOESI_hammer/config.ini rename : tests/quick/se/60.rubytest/ref/alpha/linux/rubytest-ruby-MOESI_hammer/simerr => tests/quick/se/60.rubytest/ref/null/none/rubytest-ruby-MOESI_hammer/simerr rename : tests/quick/se/60.rubytest/ref/alpha/linux/rubytest-ruby-MOESI_hammer/simout => tests/quick/se/60.rubytest/ref/null/none/rubytest-ruby-MOESI_hammer/simout rename : tests/quick/se/60.rubytest/ref/alpha/linux/rubytest-ruby-MOESI_hammer/stats.txt => tests/quick/se/60.rubytest/ref/null/none/rubytest-ruby-MOESI_hammer/stats.txt rename : tests/quick/se/60.rubytest/ref/alpha/linux/rubytest-ruby/config.ini => tests/quick/se/60.rubytest/ref/null/none/rubytest-ruby/config.ini rename : tests/quick/se/60.rubytest/ref/alpha/linux/rubytest-ruby/simerr => tests/quick/se/60.rubytest/ref/null/none/rubytest-ruby/simerr rename : tests/quick/se/60.rubytest/ref/alpha/linux/rubytest-ruby/simout => tests/quick/se/60.rubytest/ref/null/none/rubytest-ruby/simout rename : tests/quick/se/60.rubytest/ref/alpha/linux/rubytest-ruby/stats.txt => tests/quick/se/60.rubytest/ref/null/none/rubytest-ruby/stats.txt
2016-11-17alpha: Remove ALPHA tru64 support and associated testsAndreas Hansson
No one appears to be using it, and it is causing build issues and increases the development and maintenance effort.
2016-10-19stats: Update stats to reflect recent changes to floatsAndreas Hansson
Mostly just splitting out the floats ops and corresponding reads/writes.
2016-10-14config: Make configs/common a Python packageAndreas Hansson
Continue along the same line as the recent patch that made the Ruby-related config scripts Python packages and make also the configs/common directory a package. All affected config scripts are updated (hopefully). Note that this change makes it apparent that the current organisation and naming of the config directory and its subdirectories is rather chaotic. We mix scripts that are directly invoked with scripts that merely contain convenience functions. While it is not addressed in this patch we should follow up with a re-organisation of the config structure, and renaming of some of the packages.
2016-10-13stats: update referencesCurtis Dunham
2016-10-13ruby: Fix regressions and make Ruby configs Python packagesAndreas Hansson
This patch moves the addition of network options into the Ruby module to avoid the regressions all having to add it explicitly. Doing this exposes an issue in our current config system though, namely the fact that addtoPath is relative to the Python script being executed. Since both example and regression scripts use the Ruby module we would end up with two different (relative) paths being added. Instead we take a first step at turning the config modules into Python packages, simply by adding a __init__.py in the configs/ruby, configs/topologies and configs/network subdirectories. As a result, we can now add the top-level configs directory to the Python search path, and then use the package names in the various modules. The example scripts are also updated, and the messy path-deducing variations in the scripts are unified.
2016-09-22tests, arm: Reinstate accidentally removed switcheroo testsAndreas Sandberg
Two of the switcheroo tests were accidentally removed due to unexpected Mercurial behavior. Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
2016-09-22test: Make the memtest and memcheck tests functional onlyAndreas Sandberg
The memtest and memcheck are not designed to test timing. Make them functional only to make ref diffs less noisy in the future. Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
2016-09-22tests: Make remaining switcheroo tests functional onlyAndreas Sandberg
The switcheroo tests only really serve to check functional correctness. Checking for stat differences in them just increases the size of reference diffs. Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
2016-09-16tests, arm: Make switcheroo and checkpoint tests functionalAndreas Sandberg
Switcheroo and checkpoint tests should generally be considered to be successful if they run to completion. Remove all reference output files from the switcheroo and checkopint tests to make them purely functional. Change-Id: I70b47853bd662b7a33716d9e0d2154b16077f9dc Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com> Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
2016-09-16tests: Add support for functional only testsAndreas Sandberg
Modify the ClassicTest class to only emit a stat verification test unit if there is a reference stat file. This makes it possible to design tests that don't care about stat changes. To generate purely functional tests, we need to be able to create empty test reference directories. This does not work well with many revision control systems. As a workaround, add a file named EMPTY to the list of ignored files in the test harness. This file can be used as a placeholder in otherwise empty test directories. Change-Id: I583c8c4e55479f0d48fa99d0b0d1eac9221e6652 Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com> Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
2016-08-22tests: Add example of using KVM acceleration with an appDavid Hashe
Add #ifdef's to gpu-hello.cpp demonstrating how to annotate an application for KVM acceleration.
2016-08-13tests: remove EIO testsSteve Reinhardt
An email sent to gem5-users and gem5-dev asking if anyone was still using EIO traces got no responses, so it seems like it's not worth maintaining this any longer.
2016-08-12stats: Update to match classic memory changesAndreas Sandberg
2016-08-12mem: Add snoop filter to SystemXBar by defaultAndreas Hansson
This patch changes the default behaviour of the SystemXBar, adding a snoop filter. With the recent updates to the snoop filter allocation behaviour this change no longer causes problems for the regressions without caches. Change-Id: Ibe0cd437b71b2ede9002384126553679acc69cc1 Reviewed-by: Nikos Nikoleris <nikos.nikoleris@arm.com> Reviewed-by: Jason Lowe-Power <jason@lowepower.com> Reviewed-by: Tony Gutierrez <anthony.gutierrez@amd.com>
2016-08-02stats: update referencesCurtis Dunham
2016-07-22tests: Add regex-based ignore rules for ref filesAndreas Sandberg
There are cases where we need to ignore files with specific extensions (e.g., when Mercurial litters the file system with patch rejects). Implement this functionality using a helper class (FileIgnoreList) that supports both regular expressions and basic string comparisons. Change-Id: I34549754bd2e10ed230ffb2dc057403349f8fa78 Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
2016-07-21stats: update referencesCurtis Dunham
2016-07-01mem: tester for new HMC configurationAbdul Mutaal Ahmad
This patch provides the example test script to configure different HMC architecture and run traffic through traffic generator. Committed by Jason Lowe-Power <jason@lowepower.com>
2016-06-21stats: Update stats to reflect ARM changesAndreas Sandberg
2016-06-20tests: Add a test command to get test status as an exit codeAndreas Sandberg
Add a "test" command to tests.py that queries a test pickle file and returns different exit codes depending on the outcome of the tests in the file. The following exit codes can currently be returned: * 0: All tests were successful or skipped. * 1: General fault in the script such as incorrect parameters or failing to parse a pickle file. * 2: At least one test failed to run. This is what the summary formatter usually shows as a 'FAILED'. * 3: All tests ran correctly, but at least one failed to verify its output. When displaying test output using the summary formatter, such a test would show up as 'CHANGED'. The command can be invoked like this: ./tests/tests.py test `find build/ARM/tests/opt/ -name status.pickle` Change-Id: I7e6bc661516f38ff08dfda7c4359a1e10bf97864 Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com> Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
2016-06-20tests: Split test results into running and verificationAndreas Sandberg
The test base class already assumes that test cases consists of a run stage and a verification stage. Reflect this in the results class to make it possible to detect cases where a run was successful, but didn't verify. Change-Id: I31ef393e496671221c5408aca41649cd8dda74ca Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com> Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
2016-06-20mem: Resolve TrafficGen trace relative to the configAndreas Sandberg
The traffic generator currently resolves relative trace paths relative to gem5's current working directory. This can lead to surprising results for relative paths where the expectation would normally be that they are resolved relative to the configuration file. This changeset implements config-relative trace file lookups. The old behavior is kept as a fallback for configs that expect that behavior. Change-Id: I1bda4e16725842666ffc37dcb6838c23a6ff138c Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com> Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
2016-06-12stats: update EIO statsSteve Reinhardt
2016-06-06stats: Add power stats to test referencesAndreas Sandberg
Change-Id: Ic827213134b199446822f128b81d4a480e777fee
2016-06-06stats: update EIO statsSteve Reinhardt
2016-06-02stats: Update ref stats to match ARM TLB changesAndreas Sandberg
2016-06-02tests: Remove working dir assumption in tgen testsAndreas Sandberg
The traffic generator tests currently assume that they are run from the root of the source directory. This sometimes breaks tests when they are run using the new test framework. Change-Id: I6538a7902694c5d2c980295e076ea1c09acc4291 Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com> Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
2016-06-02stats: Update to match ARM ISA changesAndreas Sandberg
2016-06-02tests: add 'CHANGED' output to pickle viewerCurtis Dunham
Change-Id: I64c69fde8657c273adea69122877c5348a4f867a Reviewed-by: Andreas Sandberg <andreas.sandberg@arm.com>
2016-06-02tests: Only run Ruby tests when testing Ruby targetsAndreas Sandberg
Limit the test configs to Ruby-only configs when testing a Ruby target that isn't MI_example. This avoids re-running configs that has already been tested by the generic (non-Ruby) ISA target. This behavior was the expected behavior prior to switching to the new test framework. Change-Id: I3f138dbf9c7071ce862d1073aaec57c59afbc921 Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com> Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
2016-05-31stats: update and fix e273e86a873dCurtis Dunham
2016-05-31scons: Use the new test framework from sconsAndreas Sandberg
Rewrite the SCons script responsible for launching tests to use the new test framework. The biggest visible change after this changeset is that SCons no longer produces a "status" file in test build directories. Instead, it creates a status.pickle file. That file can be inspected using the new tests.py script. Another visible change is that timed out tests are now reported as failed rather than a using a separate summary message. Since the pickle file will remain in the build directory after running tests, it's now possible to convert the test results into other formats. For example: ./tests/tests.py show --format junit -o junit.xml \ `find build/ -name status.pickle` To better facilitate running outside of scons, there is now a set of targets that generate test listings that can be used by the test script. There are currently three targets, quick.list, long.list, and all.list. For example: scons build/ARM/tests/opt/all.list for F in `cat build/ARM/tests/opt/all.list`; do ./tests/tests.py run build/ARM/gem5.opt $F done Change-Id: I2c0f8ca0080d7af737362e198eda4cb3a72e6c36 Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com> Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
2016-05-31tests: Fix incorrect stat.txt ignore when updating refsAndreas Sandberg
ClassicTest was incorrectly ignoring stats.txt when updating reference statistics. This was caused by ignore rules being applied too aggressively when listing reference files. This changeset splits the ignore rules into two different lists: 1) diff_ignore_files that lists the files that shouldn't be diff:ed using the normal diff tool, and 2) ref_ignore_files which lists files that should be ignored by the test system. Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
2016-05-31stats: update for snoop filter tweakCurtis Dunham
--HG-- extra : source : 2323557eb4f4866fa1ea1575a9f5969e0022adc1
2016-05-27mem, config: Selective use of snoop filterStephan Diestelhorst
Disable the default snoop filter in the SystemXBar so that the typical membus does not have a snoop filter by default. Instead, add the snoop filter only when there are caches added to the system (with the caches / l2cache options). The underlying problem is that the snoop filter grows without bounds (for now) if there are no caches to tell it that lines have been evicted. This causes slow regression runs for all the atomic regressions. This patch fixes this behaviour. --HG-- extra : source : f97c20511828209757440839ed48d741d02d428f
2016-05-26tests: Add test infrastructure as a Python moduleAndreas Sandberg
Implement gem5's test infrastructure as a Python module and a run script that can be used without scons. The new implementation has several features that were lacking from the previous test infrastructure such as support for multiple output formats, automatic runtime tracking, and better support for being run in a cluster environment. Tests consist of one or more steps (TestUnit). Units are run in two stages, the first a run stage and then a verify stage. Units in the verify stage are automatically skipped if any unit run stage wasn't run. The library currently contains TestUnit implementations that run gem5, diff stat files, and diff output files. Existing tests are implemented by the ClassicTest class and "just work". New tests can that don't rely on the old "run gem5 once and diff output" strategy can be implemented by subclassing the Test base class or ClassicTest. Test results can be output in multiple formats. The module currently supports JUnit, text (short and verbose), and Python's pickle format. JUnit output allows CI systems to automatically get more information about test failures. The pickled output contains all state necessary to reconstruct a tests results object and is mainly intended for the build system and CI systems. Since many JUnit parsers parsers assume that test suite names look like Java package names. We currently output path-like names with slashes separating components. Test names are translated according to these rules: * '.' -> '-" * '/' -> '.' The test tool, tests.py, supports the following features: * Test listing. Example: ./tests.py list arm/quick * Running tests. Example: ./tests.py run -o output.pickle --format pickle \ ../build/ARM/gem5.opt \ quick/se/00.hello/arm/linux/simple-timing * Displaying pickled results. Example: ./tests.py show --format summary *.pickle Change-Id: I527164bd791237aacfc65e7d7c0b67b695c5d17c Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com> Reviewed-by: Curtis Dunham <curtis.dunham@arm.com> Reviewed-by: Joel Hestness <jthestness@gmail.com>
2016-05-09tests: Enable test running outside of gem5's source treeAndreas Sandberg
The learning gem5 scripts currently assumes that the current working directory is the root of gem5's source tree. This isn't necessarily the case when running the tests using gem5's new test runner. Change-Id: Ief569bbe77b1b3e2b0fb0e6c575fb0705bbba9b3 Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com> Reviewed-by: Curtis Dunham <curtis.dunham@arm.com>
2016-05-07tests: update EIO ref stats for removed cache statsSteve Reinhardt
Complaints about changes in EIO tests were due to reference files that still have removed cache stats from cset 11454:e55afadc4e19.
2016-04-28tests: Remove stale reference output filesAndreas Sandberg
Remove test reference files that are not generated any more: * chair.cook.ppm: This file should be generated by eon and not mcf, so it shouldn't be included as an output from mcf. * system.pc.terminal: The terminal device has been renamed so this file is no longer generated. Change-Id: I3962efe1ff25479ca276115f7564eccb5fac8cf9 Signed-off-by: Andreas Sandberg <andreas.sandberg@arm.com>
2016-04-25tests: Add a basic memcheck regressionAndreas Hansson
This patch adds a simple regression that calls the existing memcheck.py script. --HG-- rename : tests/configs/learning-gem5-p1-simple.py => tests/configs/memcheck.py rename : tests/quick/se/70.tgen/test.py => tests/quick/se/51.memcheck/test.py
2016-04-21tests: Update learning gem5 tests scripts with copyrightJason Power
2016-04-21stats: Update stats to reflect cache changesAndreas Hansson
Removed unused stats, now counting WriteLineReq, and changed how uncacheable writes are handled while responses are outstanding.
2016-04-09stats: Match current behaviourAndreas Hansson
Small changes to the branch predictor and BTB caused stats changes throughout.