| The GDB Performance Testsuite |
| ============================= |
| |
| This README contains notes on hacking on GDB's performance testsuite. |
| For notes on GDB's regular testsuite or how to run the performance testsuite, |
| see ../README. |
| |
| Generated tests |
| *************** |
| |
| The testcase generator lets us easily test GDB on large programs. |
| The "monster" tests are mocks of real programs where GDB's |
| performance has been a problem. Often it is difficult to build |
| these monster programs, but when measuring performance one doesn't |
| need the "real" program, all one needs is something that looks like |
| the real program along the axis one is measuring; for example, the |
| number of CUs (compilation units). |
| |
| Structure of generated tests |
| **************************** |
| |
| Generated tests consist of a binary and potentially any number of |
| shared libraries. One of these shared libraries, called "tail", is |
| special. It is used to provide mocks of system provided code, and |
| contains no generated code. Typically system-provided libraries |
| are searched last which can have significant performance consequences, |
| so we provide a means to exercise that. |
| |
| The binary and the generated shared libraries can have a mix of |
| manually written and generated code. Manually written code is |
| specified with the {binary,gen_shlib}_extra_sources config parameters, |
| which are lists of source files in testsuite/gdb.perf. Generated |
| files are controlled with various configuration knobs. |
| |
| Once a large test program is built, it makes sense to use it as much |
| as possible (i.e., with multiple tests). Therefore perf data collection |
| for generated tests is split into two passes: the first pass builds |
| all the generated tests, and the second pass runs all the performance |
| tests. The first pass is called "build-perf" and the second pass is |
| called "check-perf". See ../README for instructions on running the tests. |
| |
| Generated test directory layout |
| ******************************* |
| |
| All output lives under testsuite/gdb.perf in the build directory. |
| |
| Because some of the tests can get really large (and take potentially |
| minutes to compile), parallelism is built into their compilation. |
| Note however that we don't run the tests in parallel as it can skew |
| the results. |
| |
| To keep things simple and stay consistent, we use the same |
| mechanism used by "make check-parallel". There is one catch: we need |
| one .exp for each "worker" but the .exp file must come from the source |
| tree. To avoid generating .exp files for each worker we invoke |
| lib/build-piece.exp for each worker with different arguments. |
| The file build.piece.exp lives in "lib" to prevent dejagnu from finding |
| it when it goes to look for .exp scripts to run. |
| |
| Another catch is that each parallel build worker needs its own directory |
| so that their gdb.{log,sum} files don't collide. On the other hand |
| its easier if their output (all the object files and shared libraries) |
| are in the same directory. |
| |
| The above considerations yield the resulting layout: |
| |
| $objdir/testsuite/gdb.perf/ |
| |
| gdb.log, gdb.sum: result of doing final link and running tests |
| |
| workers/ |
| |
| gdb.log, gdb.sum: result of gen-workers step |
| |
| $program_name/ |
| |
| ${program_name}-0.worker |
| ... |
| ${program_name}-N.worker: input to build-pieces step |
| |
| outputs/ |
| |
| ${program_name}/ |
| |
| ${program_name}-0/ |
| ... |
| ${program_name}-N/ |
| |
| gdb.log, gdb.sum: for each build-piece worker |
| |
| pieces/ |
| |
| generated sources, object files, shlibs |
| |
| ${run_name_1}: binary for test config #1 |
| ... |
| ${run_name_N}: binary for test config #N |
| |
| Generated test configuration knobs |
| ********************************** |
| |
| The monster program generator provides various knobs for building various |
| kinds of monster programs. For a list of the knobs see function |
| GenPerfTest::init_testcase in testsuite/lib/perftest.exp. |
| Most knobs are self-explanatory. |
| Here is a description of the less obvious ones. |
| |
| binary_extra_sources |
| |
| This is the list of non-machine generated sources that go |
| into the test binary. There must be at least one: the one |
| with main. |
| |
| class_specs |
| |
| List of pairs of keys and values. |
| Supported keys are: |
| count: number of classes |
| Default: 1 |
| name: list of namespaces and class name prefix |
| E.g., { ns0 ns1 foo } -> ns0::ns1::foo_<cu#>_{0,1,...} |
| There is no default, this value must be specified. |
| nr_members: number of members |
| Default: 0 |
| nr_static_members: number of static members |
| Default: 0 |
| nr_methods: number of methods |
| Default: 0 |
| nr_inline_methods: number of inline methods |
| Default: 0 |
| nr_static_methods: number of static methods |
| Default: 0 |
| nr_static_inline_methods: number of static inline methods |
| Default: 0 |
| |
| E.g., |
| class foo {}; |
| namespace ns1 { class bar {}; } |
| would be represented as: |
| { |
| { count 1 name { foo } } |
| { count 1 name { ns1 bar } } |
| } |
| |
| The naming of each class is "class_<cu_nr>_<class_nr>", |
| where <cu_nr> is the number of the compilation unit the |
| class is defined in. |
| |
| There's currently no support for nesting classes in classes, |
| or for specifying baseclasses or templates. |
| |
| Misc. configuration knobs |
| ************************* |
| |
| These knobs control building or running of the test and are specified |
| like any global Tcl variable. |
| |
| CAT_PROGRAM |
| |
| Default is /bin/cat, you shouldn't need to change this. |
| |
| SHA1SUM_PROGRAM |
| |
| Default is /usr/bin/sha1sum. |
| |
| PERF_TEST_COMPILE_PARALLELISM |
| |
| An integer, specifies the amount of parallelism in the builds. |
| Akin to make's -j flag. The default is 10. |
| |
| Writing a generated test program |
| ******************************** |
| |
| The best way to write a generated test program is to take an existing |
| one as boilerplate. Two good examples are gmonster1.exp and gmonster2.exp. |
| gmonster1.exp builds a big binary with various custom manually written |
| code, and gmonster2 is (essentially) the equivalent binary split up over |
| several shared libraries. |
| |
| Writing a performance test that uses a generated program |
| ******************************************************** |
| |
| The best way to write a test is to take an existing one as boilerplate. |
| Good examples are gmonster1-*.exp and gmonster2-*.exp. |
| |
| The naming used thus far is that "foo.exp" builds the test program |
| and there is one "foo-bar.exp" file for each performance test |
| that uses test program "foo". |
| |
| In addition to writing the test driver .exp script, one must also |
| write a python script that is used to run the test. |
| This contents of this script is defined by the performance testsuite |
| harness. It defines a class, which is a subclass of one of the |
| classes in gdb.perf/lib/perftest/perftest.py. |
| See gmonster-null-lookup.py for an example. |
| |
| Note: Since gmonster1 and gmonster2 are treated as being variations of |
| the same program, each test shares the same python script. |
| E.g., gmonster1-null-lookup.exp and gmonster2-null-lookup.exp |
| both use gmonster-null-lookup.py. |
| |
| Running performance tests for generated programs |
| ************************************************ |
| |
| There are two steps: build and run. |
| |
| Example: |
| |
| bash$ make -j10 build-perf RUNTESTFLAGS="gmonster1.exp" |
| bash$ make -j10 check-perf RUNTESTFLAGS="gmonster1-null-lookup.exp" \ |
| GDB_PERFTEST_MODE=run |