diff options
author | Eric Sandeen <sandeen@sandeen.net> | 2017-01-13 08:56:15 -0600 |
---|---|---|
committer | Eryu Guan <eguan@redhat.com> | 2017-01-18 15:59:33 +0800 |
commit | 2ad51674753978db5c1a036eaf08b00994a218eb (patch) | |
tree | 00d5a14269bee65fbfe81ffcc8326ae20ecab11a | |
parent | acb8acd2367ac8b9d3f9a471b9bd18e0e6d83eda (diff) | |
download | xfstests-dev-2ad51674753978db5c1a036eaf08b00994a218eb.tar.gz |
tools: compare-failures script
This is a simple script to compare failures across runs.
Given files containing stdout from several runs, each of which contains
a Failures: line, it will print a table of all failures for each run.
Test subdir names are abbreviated for compactness, i.e. generic->g.
For 7 results files named test 1 through test 7:
Failures:
g/075 g/082 g/209 g/233 g/270 g/388 x/004 x/073 x/076
-----------------------------------------------------
g/082 g/233 x/004 x/073 test1
g/082 g/233 x/004 x/073 x/076 test2
g/082 x/004 x/073 x/076 test3
g/082 g/388 x/004 x/073 test4
g/082 g/270 x/004 x/073 test5
g/082 x/004 x/073 test6
g/075 g/082 g/209 g/233 x/004 x/073 test7
This lets us easily spot unique failures and outliers.
This could be enhanced to output CSV etc, but for now I think it's
helpful to visualize changes in failures across multiple runs.
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Reviewed-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>
-rw-r--r-- | README | 3 | ||||
-rwxr-xr-x | tools/compare-failures | 67 |
2 files changed, 70 insertions, 0 deletions
@@ -272,6 +272,9 @@ Pass/failure: The elapsed time for the most recent pass for each test is kept in "check.time". + The compare-failures script in tools/ may be used to compare failures + across multiple runs, given files containing stdout from those runs. + __________________ SUBMITTING PATCHES __________________ diff --git a/tools/compare-failures b/tools/compare-failures new file mode 100755 index 0000000000..9604a63855 --- /dev/null +++ b/tools/compare-failures @@ -0,0 +1,67 @@ +#!/bin/bash + +# Compare test failures across runs +# +# Takes multiple "results" files as arguments, comprised of the +# stdout from a ./check run, each containing a Failures: line. +# +# Outputs a table of failures for comparison across runs +# +#----------------------------------------------------------------------- +# Copyright (c) 2017 Red Hat, Inc. All Rights Reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +#----------------------------------------------------------------------- +# + +filter_names() { + sed -e s/btrfs/b/ \ + -e s/cifs/c/g \ + -e s/f2fs/f/g \ + -e s/generic/g/g \ + -e s/overlay/o/g \ + -e s/shared/s/g \ + -e s/udf/u/g \ + -e s/xfs/x/g +} + +# ALLFAILURES: A B C D E F G +# THESEFAILURES: A C D G + +# We want to print the header (ALLFAILURES) and then +# if a run didn't fail a particular test, print spaces instead + +# All tests that failed in any run, all on one line, unique +ALLFAILURES=`grep -h ^Failures: $* \ + | tr " " "\n" \ + | sort | uniq \ + | filter_names \ + | tr "\n" " " \ + | sed -e "s/^Failures: //g"` + +# Header +echo "Failures:" +echo $ALLFAILURES +echo $ALLFAILURES | sed -e "s/./-/g" + +# Per-file failures +for FILE in $*; do + THESEFAILURES=`grep ^Failures: $FILE | filter_names` + for FAILURE in $ALLFAILURES; do + CELL=`echo $THESEFAILURES \ + | grep -wo "$FAILURE" || echo $FAILURE | sed -e "s/./ /g"` + echo -n "$CELL " + done + echo $FILE +done |