diff options
| author | Jacob Keller <jacob.e.keller@intel.com> | 2014-05-27 11:02:59 -0700 |
|---|---|---|
| committer | Jacob Keller <jacob.e.keller@intel.com> | 2014-05-27 11:02:59 -0700 |
| commit | 2e9d51089b54460699953bd9d6f828262c7c6dea (patch) | |
| tree | f6de7c934ccc24d06a96b945e45262a4cff4776d | |
| parent | 5c0229250b1f4c1e34405a508ee58ed91fffef35 (diff) | |
| parent | 423f40362657a7c6d1c4d8284af4d280d35fff0e (diff) | |
| download | aiaiai-2e9d51089b54460699953bd9d6f828262c7c6dea.tar.gz | |
aiaiai: merge development branch to master
Finally merge the development changes after some fairly serious use in
the email environment. I have appended the shortlog below, as well as a
helpful short summary of the large changes in no particular order.
* Moved as much as possible for configuring email setup to the cfgfile
* Added several options previously not configurable at all
* Add support for "hooks" framework, including a simple dispatcher hook
* Add hook which automatically detects project based on git-index info
* Add systemd scripts for the email setup
* Allow default values for project configuration
* Add script to help keep project repos up to date
These changes hopefully make configuration and setup of the email
workflow easier. In addition, the hook enabling automatic project
detection makes handling multiple project kernel trees much easier.
Artem Bityutskiy (6):
test-pattchet: add a couple quotes
aiaiai-test-patchset: move --targets option down
aiaiai-diff-log-helper: fix logs diffing for gcc-4.8
aiaiai-checker: use --no-data for smatch
aiaiai-checker: ignore debugging output from spatch
TODO: remove a completed item
Jacob Keller (51):
Add support for choosing build targets
aiaiai-sh-functions: remove extra quotes in call to ini_config_get
aiaiai-test-patchset: use merge_addresses only once
aiaiai-sh-functions: remove [email] section from cfgfile
aiaiai-sh-functions: re-order options to match example config
aiaiai-email: add [defaults] section to config
aiaiai-email: move test-patchset options into the configuration file
aiaiai: don't allow arbitrary validator command
aiaiai-email: move -J bigjobs into configuration file
aiaiai-test-patchset: add missing colon on targets option
aiaiai-email-dispatcher-helper: fix typo on aiaiai
aiaiai-email-test-patchset: fix parameter substitution
aiaiai-email-test-patchset: move -- to end of all options
email: add "debug" section to configuration file
email-test-patchset: parse general configuration settings earlier
sh-function: add insert_header function
doc: add CONFIGURATION file describing the cfgfile
aiaiai: add initial support for dispatcher hook
hook: add support for X-Aiaiai-Project
aiaiai: fixup srcdir in email and gerrit scripts
dispatcher: add aiaiai-email-sh-functions to source list
doc: rename CONFIGURATION to configuration.txt
email: don't allow pcfg_branch to default
email: add canonical_url variable for list_projects
aiaiai: split sed project listing from list_projects
email: rename error_test_patchset_failed
aiaiai: change dispatcher hook to email hook
aiaiai: add git-find-base utility for finding base commit from mbox
email: example hook for aiaiai-email-test-patchset
aiaiai: use fail_usage on incorrect number of arguments
apply-patch: prefix diff output with '> '
autodetect-project: fix change of function name
systemd: add scripts for running the aiaiai processes
email-hook: remove options that can't be used
git-find-base: Add authorship, copyright, and license notice
git-find-base: add detection for possible duplicate patches
git-find-base: add help text to the program
send-mail-on-failure: generalize email address
aiaiai-project-update: fix srcdir line from location change
aiaiai-project-update: fix typo of printf in usage statement
aiaiai-project-update: correct function to upstream change
git-find-base: rewritten to use newer design
aiaiai-email-test-patchset: correct hook calling to actually grab error
aiaiai: extract patches from email prior to using git-am
aiaiai-diff-log-helper: explicitely call out python2
aiaiai: add checks to configuration file
aiaiai-email-lda: move reap options to cfgfile
email: fix incorrect variable name in email for hook rejection
aiaiai: don't display log output from git-find-base
aiaiai: don't cat the $mbox file at begining of email-testpatchset
aiaiai: don't hardcode KCFLAGS or W=1
Signed-off-by: Jacob Keller <jacob.e.keller@intel.com>
29 files changed, 1459 insertions, 276 deletions
diff --git a/aiaiai-test-patchset b/aiaiai-test-patchset index 577e987..45527f9 100755 --- a/aiaiai-test-patchset +++ b/aiaiai-test-patchset @@ -86,6 +86,7 @@ Options: an mbox file with quick fixes which will be applied first and the user patch-set will be tested on top of the fixes; + --targets list of make targets to build (defaults to all) -K --keywords=FILE match keywords from FILE against the patch; -M, --kmake-opts additional options to append to the final kernel compilation 'make' command @@ -189,7 +190,7 @@ test_configuration() aiaiai-make-kernel $verbose $sparse $smatch $cppcheck $coccinelle -o "$obj1" \ --check-only "$mod_files" -j "$jobs" $arch_opt -D "$defconfig" \ -O "$log1.stdout.log" -E "$log1.stderr.log" --keep-going \ - ${kmake_opts:+-M "$kmake_opts"} -- "$cloned_kernel1" all + ${kmake_opts:+-M "$kmake_opts"} -- "$cloned_kernel1" "$targets" verbose "Done with non-patched kernel" @@ -211,7 +212,7 @@ test_configuration() aiaiai-make-kernel $verbose $sparse $smatch $cppcheck $coccinelle -o "$obj2" \ --check-only "$mod_files" -j "$jobs" $arch_opt -D "$defconfig" \ -O "$log2.stdout.log" -E "$log2.stderr.log" \ - ${kmake_opts:+-M "$kmake_opts"} -- "$cloned_kernel2" all + ${kmake_opts:+-M "$kmake_opts"} -- "$cloned_kernel2" "$targets" verbose "Done with patched kernel" @@ -238,7 +239,7 @@ cleanup_handler() } set_cleanup_handler cleanup_handler -TEMP=`getopt -n $PROG -o j:,c:,i:,w:,C:,p,Q:,K:,M:,v,h --long jobs:,commit-id:,input:,workdir:,logdir:,confdir:,preserve,bisectability,sparse,smatch,cppcheck,coccinelle,quick-fixes:,keywords:,kmake-opts:,verbose,help -- "$@"` || +TEMP=`getopt -n $PROG -o j:,c:,i:,w:,C:,p,Q:,K:,M:,v,h --long jobs:,commit-id:,input:,workdir:,logdir:,confdir:,preserve,bisectability,sparse,smatch,cppcheck,coccinelle,quick-fixes:,targets:,keywords:,kmake-opts:,verbose,help -- "$@"` || fail_usage "" eval set -- "$TEMP" @@ -253,6 +254,7 @@ smatch= cppcheck= coccinelle= quick_fixes= +targets="all" keywords= kmake_opts= verbose= @@ -288,10 +290,6 @@ while true; do -p|--preserve) preserve="--preserve" ;; - -Q|--quick-fixes) - quick_fixes="$(opt_check_read "$1" "$2")" - shift - ;; --bisectability) bisectability="--bisectability" ;; @@ -311,6 +309,14 @@ while true; do coccinelle="yes" program_required "spatch" "Usually Linux distribution provide a 'spatch' or 'coccinelle' package" ;; + -Q|--quick-fixes) + quick_fixes="$(opt_check_read "$1" "$2")" + shift + ;; + --targets) + targets="$2" + shift + ;; -K|--keywords) keywords="$(opt_check_read "$1" "$2")" shift @@ -335,7 +341,7 @@ while true; do shift done -[ "$#" -ge 2 ] || die "Insufficient arguments" +[ "$#" -ge 2 ] || fail_usage "Insufficient arguments" kernel_tree="$(readlink -ev -- "$1")"; shift defconfigs="$@" @@ -105,8 +105,13 @@ The 'aiaiai-test-patchset' script does all the testing work and outputs the test results to stdout. 'aiaiai-email-test-patchset' captures the output and sends it back to the patch submitter. +1.2 Email Configuration +~~~~~~~~~~~~~~~~~~~~~~~ -1.2 Non-e-mail scripts +aiaiai-email scripts use a common configuration file, which is documented in +doc/email/example-aiaiai.cfg + +1.3 Non-e-mail scripts ~~~~~~~~~~~~~~~~~~~~~~ * aiaiai-test-patchset diff --git a/doc/TODO.txt b/doc/TODO.txt index adc9a97..f5a879d 100644 --- a/doc/TODO.txt +++ b/doc/TODO.txt @@ -10,23 +10,11 @@ implementing them. containing the text for end-users to refer instead. Just like email.preamble refers the file, which contains the preamble. This is more flexible and consistent. - * Move command-line options of 'aiaiai-email-*' scripts to the configuration - file. This should make it simpler for users to configure the whole thing - when all the knobs are in one place. Just make each script to parse the - config file and pick own configuration from there. Namely, most of - the 'aiaiai-email-dispatcher', 'aiaiai-email-testpatchset', and - 'aiaiai-email-lda' options have to go away. * Eventually make non-email scripts (aiaiai-test-patchset) also support the config file. This could be an alternative to specifying everythig via options. E.g., would could run 'aiaia-test-patchset --config <path>/aiaiai.cfg' and have all the details defined in 'aiaiai.cfg'. - * Additional item for the previous one: aiaiai-email-dispatcher is too - difficutl to execute because it requires the "<validator>" argument. - But there is only one validator which makes sense in Aiaiai context - - "aiaiai-email-test-patchset". So remove the silly argument and - just run the validator, without exposing unnecessary complexity to - the user. * Start assigning versions to the scripts and do official releases, with signed tags, and tarballs. * Provide packages for various distributions. Including packages for the @@ -37,16 +25,3 @@ implementing them. patch-sets. * Describe some review policy, e.g., how many days I promise to wait for review/reply before applying patches. - -* Functional improvements - * Gcc 4.8 change the format of warnings and now they look like this: - - arch/x86/kernel/e820.c: In function ‘e820_setup_gap’: - arch/x86/kernel/e820.c:630:6: warning: variable ‘found’ set but not used [-Wunused-but-set-variable] - int found; - ^ - Aiaiai does not process them correctly which leads to ugly diffs. Fix this. - * We hard-code 'KCFLAGS='-Wno-missing-field-initializers - -Wno-sign-compare' and W=1 in 'aiaiai-make-kernel'. This is not that - nice. It is better to make these things to be the default value of - the -M parameter. The users could disable them or re-define easily. diff --git a/doc/email/configuration.txt b/doc/email/configuration.txt new file mode 100644 index 0000000..3ba16a4 --- /dev/null +++ b/doc/email/configuration.txt @@ -0,0 +1,208 @@ +1 Overview +~~~~~~~~~~ + +aiaiai-email scripts can be configured using the configuration file, which +includes many options for customizing the behavior of the aiaiai testing +process. Most of these values are specified "per-project" which allows the +ability for a single aiaiai-email setup to handle multiple different +repositories or kernels. This document describes and explains each +configuration option, and also includes some sections specific to complex +settings. + +The configuration file is in INI format, and generally is designed to be human +editable. Each section will be described below. + +2.1 Global Settings +~~~~~~~~~~~~~~~~~~~ + +The global configuration settings are recognized by various email scripts, +though mostly have to deal with aiaiai-email-test-patchset. + +* ownname +* ownmail + These settings change the name and email address which the scripts will + reply as. +* adminname +* adminmail + The address to reply to when attempting to contact a real person about the + aiaiai-email setup. It will also modify the reply-to email header so that + replies by default go to the administrator. +* workdir + Temporary location used to store kernel trees for compiling, as well as + other various temporary files generated by the aiaiai-email scripts +* max_validators + The maximum number of validators to run in parallel. This can be used to + allow concurrent validation of different patches or patch series at the + same time. Be careful to ensure your test machine has enough CPU to handle + extra validators. +* jobs + Passed directly into the make commands, which allows make to parallelize + its proccess. Not to be confused with the max_validators parameter. Be + careful to ensure you have enough CPU for this. +* preamble + Location of a file which contains text. This text is used to supply the + start of every email sent by the aiaiai-email-scripts. Useful to customize + the header of each email, or to provide information for users about what + the system is for. +* signature + A line of text to end every email sent by aiaiai. Don't end with + punctuation, as the email script will add suitable punctuation for you. +* built_preamble + A line of text that is inserted if the patch or series passes the + validation without error. Don't end with puncuation as aiaiai does this for + you. + +2.2 Debug Settings +~~~~~~~~~~~~~~~~~~ + +These settings are generally not useful, but can be helpful for debugging +issues with aiaiai email scripts. These have replaced options which used to +perform the same behavior. + +* disable_notifications + This option disables all emails sent by the email scripts. Their contents + will still be displayed in the stdout of the respective process, but no + actual replies will be sent. This replaces the old "--test-mode" command line + option. This is useful to prevent spam emails from being generated when + testing changes to aiaiai +* preserve_files + This option modifies the behavior so that no files will be cleaned up on + exit. Very useful if you need to inspect individual results or generated + files. This replaces the old "-p" option. + +2.3 Default Project Settings +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Settings described here can either appear in the [defaults] section or in the +[prj_<project>] sections, where <project> specifies a project name. A project +is specified when sending the email to a special address. For example, if the +aiaiai address is "aiaiai@aiaiai.net" then you can send the email to +"aiaiai+project@aiaiai.net" which aiaiai can extract the +project to determine +which project to use for that patch series. See the README for more information +about projects. + +The per-project settings always override the default settings, and are not +additive. + +* configs + A space seperated list of defconfig,arch,cross-compiler (no spaces). The + arch and cross-compiler are optional. Defconfig name should reside inside + the kernel, or possibly inside the defconfigdir as specified by other + option. Multiple configs are supported by seperating with spaces. Each + config will be tested. See the example-aiaiai.cfg for details and examples. +* always_cc + A comma seperated list of addresses to always Cc when creating replies. For + example, it could be a mailing list to which all aiaiai replies go. +* reply_to_all + A boolean (0 or 1) indicating whether to reply to all addresses or only the + original sender. +* accept_notify + A boolean (0 or 1) indicating whether to send an acceptance mail which + tells the submitter that aiaiai has begun work on the patch or series. +* unwanted_keywords + Path to a file containing unwanted keywords (one keyword per line). Aiaiai + will check the patch against this list and notify if any keywords were + found. This might be useful to help prevent some internal confidential + names don't leak through commit messages or commentaries. +* kmake_opts + A list of additional kernel build options which are appended to the make + command used to compile the kernel. Useful to change the level of warnings, + for example. +* targets + A list of kernel build targets, space seperated. Note that all is only + implicit if the target lits is empty. If you assign a value, such as + namespacecheck, then all is not included by default and you will have to + add it to the list. +* defconfigdir + Path to a directory containing the defconfigs. Useful if your + configurations are not included in the kernel tree. If you specify a + defconfigdir, then all configurations must reside in it, as aiaiai will not + check the kernel directory for configs. +* bisectability + A boolean (0 or 1) indicating whether to check if patch series can be + compiled inbetween each patch. Useful to ensure that patch series don't + break git-bisect. +* sparse + A boolean (0 or 1) indicating whether to use the sparse tool during static + analysis. +* smatch + A boolean (0 or 1) indicating whether to use the smatch tool during static + analysis. +* cppcheck + A boolean (0 or 1) indicating whether to use the cppcheck tool during static + analysis. +* coccinelle + A boolean (0 or 1) indicating whether to use the coccinelle scripts during + static analysis. + +The following options do not have default settings per project and must be +specified for each project. + +* name + A human readable name for the project +* description + A short one line projet description, that should start with a lowercase letter. +* path + Path to the kernel source tree of the project. Note aiaiai does not treat + this directory as read-only, and never changes anything htere. This does + mean that aiaiai does not automatically update the tree for you. You could + use something like a cronjob to update the tree, or any other method. It + also does not depend on the tree being (or not) bare, so you can feel free + to make this a bare remote or a local tree. Generally, aiaiai works best if + the tree is locally stored on the same machine as the aiaiai process. +* branch + What git refspec to validate against. Usually this should be a remote + branch ie: origin/master, but can actually be any refspec. Note that the + defconfigs specified in the config option must actually be available from + the refspec. +* canonical_url + The url to use when displaying a url for user consumption. Intended to be a + patch-submitter visible mirror of the local path used internally. If not + supplied, aiaiai will display the path instead. + +2.4 Hooks +~~~~~~~~~ + +The [hooks] section contains settings for various points where a hook can be +run to customize the behavior of aiaiai. Generally, hooks should output +mbox-style headers, which the aiaiai program will parse for required +information. Only certain headers are supported, outlined below. + +A hook is expected to return zero for successful runs, 127 for patch rejection, +and any other non zero value at run-time error. In addition, for a zero return +status, the output should be a list of headers outlined below, (one per line) +which will be parsed by aiaiai for customizing behavior. For 127 exit status, +the stdout will be used for creating a subsequent response email, and it should +contain information explaining to the user why the hook rejected the patch. For +other non-zero exit status, the output is ignored. + +Hooks are passed the configuration file, and the mbox file for processing and +therefor have full access to the configuration file. Discretion should be used +if adding custom variables to the configuration file such that the hook won't +have a namespace collission with regular aiaiai configuration options. + +The only currently supported hook is the email hook configured by setting +hooks.email to the path of an executable script. + +2.4.1 Email Hook +~~~~~~~~~~~~~~~~ + +This hook is run from within aiaiai-email-test-patchset, and is run just prior +to determining the project. This hook can be used to customize the behavior and +settings of aiaiai-email-test-patchset. Primarily it can be used to implement +any kind of patch-rejection scheme desired, as well as use the following +headers to customize various aspects of aiaiai. + +* X-Aiaiai-Project + This header indicates to email-test-patchset which project the patch series + belongs to. This overrides the default +prj format from the cfg_ownmail + address. This can be used to implement any elaborate scheme desired for + determining the project, instead of relying on users mailing patches to the + correct email address. + +* X-Aiaiai-Commit + This header indicates what commit to test against instead of the default + refspec supplied by the project. It can be used to implement some form of + patch-version detection, or other setup. It likely should be used in tandem + with the X-Aiaiai-Project setting. + diff --git a/doc/email/example-aiaiai.cfg b/doc/email/example-aiaiai.cfg index 4636e28..e9f7585 100644 --- a/doc/email/example-aiaiai.cfg +++ b/doc/email/example-aiaiai.cfg @@ -20,46 +20,59 @@ # compiling the kernel, storing temporary files, etc. workdir = /home/aiaiai/work/aiaiai/workdir + # The maximum number of validators to run in parallel. This is + # different than the jobs variable, which controls how many threads to + # run for the make program. This can be used to allow concurrent + # validation of different patch series. + max_validators = 1 + # How many parallel processes should be used for validating a patch # or patch-set. This parameter simply translates to the "-j" option # of the "make" program. Note, this parameter is not about how many # patches or patch-sets can be validated at parallel. jobs = 14 -# This section allows configuring contents of the replies which the Aiaiai -# e-mail front-end will be sending to the patch submitter. -# Do not put dots at the end of the sentences, the scripts will do it. -[email] # Every e-mail the front-end sends will start with the preamble preamble = /home/aiaiai/work/aiaiai/preamble.txt - # .. and end with this signature. + # .. and end with this signature. Do not end with puncuation, as aiaiai + # will insert it for you. signature = Regards, Aiaiai - # When the patch/patch-set passes the validation, this phrase - # is put after the preamble text. + # When the patch/patch-set passes the validation, this phrase is put + # after the preamble text. As above, do not end with punctuation, as + # aiaiai will insert it for you. built_preamble = I have tested your changes -# The e-mail front-end may operate on several project. Each project has its -# own kernel tree, kernel configuration, and some other settings. These are -# configured in per-project sections. Section names have to start with "prj_" -# and follow with the name of the project. - -# This is the "android" project section. -[prj_android] - # Human-readable project name - name = Android - - # Short, one line project description. Start it with a small letter. - description = the Android x86 kernel - - # Path to the kernel tree of the project. Note, Aiaiai does treats this - # directory as read-only and never changes anything there. This means - # that you have to keep it up-to-date with other means. For example, you - # may setup a cron job which will periodically execute 'git fetch -f' in - # this directory in order to get the newest contents from the server. - path = /home/aiaiai/work/git/android-x86 - +# These options control the LDA email program +[lda] + # Reap all archived mail older than number of minutes specified. + reap_archive = + + # Reap all incomplete series older than number of minutes specified. + reap_incomplete = 1440 + +[hooks] + # A hook called by aiaiai-email-test-patchset, which should output a + # zero exit status as well as the supported custom headers. If the hook + # wants to reject a patch, exit with return code 127. Any other return + # code indicates an internal error in the hook. + email = + +# These options are probably not useful, but may help debug issues with aiaiai +[debug] + # Set to (1) to disable sending reply emails when a patch is tested. + # This can help prevent spam while testing or debugging aiaiai. + disable_notifications = 0 + + # Set to (1) to preserve generated files instead of allowing aiaiai to + # cleanup after itself. + preserve_files = 0 + +# The "defaults" section specifies various default settings for projects. These +# values are overridden by the same values in the prj_* sections, so you may +# customize the settings per project if desired. +[defaults] # The kernel configurations to test. You may specify more than one # configuration using blanks as a delimiter. The configuration consists # of the following comma-separated components: the defconfig file to @@ -71,26 +84,19 @@ # # means: use the "omap2_defconfig" defconfig, use ARCH="arm", use # CROSS_COMPILE="arm-eabi-". The last component is optional. - configs = android_defconfig,i386 android_debug_defconfig,i386 - - # Git refspec to validate the patches against. Usually this a the - # remote branch name. But may actually be any refspec. - # - # Note, the defconfig files specified in the "configs" option must be - # part of this git refspec. - branch = origin/3.0-mid + configs = x86_64_defconfig,x86_64 i386_defconfig,i386 # Comma-separated list of e-mail addresses to always CC when replying. # These are maintainers or other project supervisors. This may be a # separate mailing list for replies from Aiaiai. - always_cc = "Artem Bityutskiy" <artem.bityutskiy@fake.domain.com>, "Aiaiai archives" <aiaiai-archives@fake.comain.com> + always_cc = "Aiaiai archives" <aiaiai-archives@fake.domain.com> # Reply only to the sender (0) or to everyone in To: and Cc: (1) reply_to_all = 0 # Notify the sender that his/her e-mail has been accepted by the Aiaiai # front-end and put to the queue for validation. - accept_notify = 1 + accept_notify = 0 # Path to the file containing unwanted keywords (one keyword per line). # Aiaiai will check the patch against this list and notify if any of @@ -101,6 +107,82 @@ # Additional kernel build options which are appended to the end of the # "make" command. For example, W=2 KALLSYMS_EXTRA_PASS=1. - kmake_opts = + kmake_opts = KCFLAGS='-Wno-missing-field-initializers -Wno-sign-compare' W=1 + + # List of targets (space seperated) for which to run a kernel make. + # Leaving this empty will default to "all" as the only target. However, + # the "all" target is not implicit so you should explicitly have it in + # the list if you want it to run. + targets = all namespacecheck + + # Path to the directory containing the deconfig files (those specified + # for each project). Leave empty to use the kernel tree default + # configs. + defconfigdir = + + # Whether to test bisectability between patches in a series (1), or + # only to check results of the squished patch-set (0) + bisectability = 0 + + # Set to (1) to enable sparse during kernel builds + sparse = 1 + + # Set to (1) to enable smatch during kernel builds + smatch = 0 + + # Set to (1) to enable cppcheck during kernel builds + cppcheck = 0 + + # Set to (1) to enable coccinelle scripts during kernel builds + coccinelle = 1 + + +# The e-mail front-end may operate on several project. Each project has its +# own kernel tree, kernel configuration, and some other settings. These are +# configured in per-project sections. Section names have to start with "prj_" +# and follow with the name of the project. The 'name', 'description', 'path' +# and 'branch' settings are required and are not provided by the [defaults] +# section, so must be set for each project. + +# This is the "android" project section. +[prj_android] + # Human-readable project name. + name = Android + + # Short, one line project description. Start it with a small letter. + description = the Android x86 kernel + + # Path to the kernel tree of the project. Note, Aiaiai does treats this + # directory as read-only and never changes anything there. This means + # that you have to keep it up-to-date with other means. For example, + # you may setup a cron job which will periodically execute 'git fetch + # -f' in this directory in order to get the newest contents from the + # server. No default value. + path = /home/aiaiai/work/git/android-x86 + + # Remote url used for display only. Aiaiai does not directly use this, + # and does not attempt to keep the path up to date itself. This is used + # only for display to the user. + canonical_url = git://aiaiai/git/android-x86 + + # Git refspec to validate the patches against. Usually this a the + # remote branch name. But may actually be any refspec. + # + # Note, the defconfig files specified in the "configs" option must be + # part of this git refspec. + branch = origin/3.0-mid + + # See the [defaults] section above for detailed explanation of the + # options below, and their formats. Be aware that any value set here + # will override the default value, ignoring anything set there. For + # example, setting the value to empty here will ignore the default + # value and actually set the option to the empty value. To use the + # default, simply leave the option out of the per-project section. + configs = android_defconfig,i386 android_debug_defconfig,i386 + always_cc = + reply_to_all = 1 + accept_notify = 1 + unwanted_keywords = + kmake_opts = W=12 KALLSYMS_EXTRA_PASS=1 KCFLAGS='-Wno-missing-initializers -Wno-sign-compare' # More "prj_xyz" sections may be added diff --git a/email/aiaiai-email-autodetect-project b/email/aiaiai-email-autodetect-project new file mode 100755 index 0000000..aee5a7c --- /dev/null +++ b/email/aiaiai-email-autodetect-project @@ -0,0 +1,134 @@ +#!/bin/sh -efu + +# Copyright 2014 Intel Corporation +# Author: Jacob Keller +# License: GPLv2 + +srcdir="$(readlink -ev -- ${0%/*})" +PATH="$srcdir:$srcdir/..:$srcdir/../helpers:$srcdir/../helpers/libshell:$PATH" + +. shell-error +. shell-args +. shell-signal +. aiaiai-sh-functions +. aiaiai-email-sh-functions + +PROG="${0##*/}" +export message_time="yes" + +# This is a small trick to make sure the script is portable - check if 'dash' +# is present, and if yes - use it. +if can_switch_to_dash; then + exec dash -euf -- "$srcdir/$PROG" "$@" + exit $? +fi + +show_usage() +{ + cat <<-EOF +Usage: $PROG [options] <cfgfile.ini> <mbox> + +This is an example hook used by aiaiai to automatically determine the project +that the commit applies to, by using the index information from the first patch +in the series, and checking each project for information about where patches +apply. It depends on git-find-base which is included in the helpers folder. + +<cfgfile.ini> - the configuration file. +<mbox> - the mbox file containing the patch series +EOF +} + +fail_usage() +{ + [ -z "$1" ] || printf "%s\n" "$1" + show_usage + exit 1 +} + +verbose=-v +tmpdir= +cleanup_handler() +{ + if [ "$cfg_preserve_files" = "1" ]; then + verbose "Preserved tmpdir: $tmpdir" + else + [ -z "$tmpdir" ] || verbose "Removing $tmpdir"; + rm -rf -- "$tmpdir" >&2 + fi +} +set_cleanup_handler cleanup_handler + +[ "$#" -eq 2 ] || fail_usage "Insufficient or too many arguments" + +cfgfile="$(readlink -fv -- "$1")"; shift +mbox="$(readlink -fv -- "$1")"; shift +commit= + +# Parse the configuration file early +parse_config "$cfgfile" + +# Create a temporary directory for storage of any files we might need +tmpdir="$(mktemp --tmpdir="$cfg_workdir" -dt "$PROG.XXXX")" +tmp_find_base="$tmpdir/git-find-base.out" + +# Get the expected project from the patch email +to="$(fetch_header "To" < "$mbox")" +cc="$(fetch_header "Cc" < "$mbox")" +list="$(merge_addresses "$to" "$cc")" +expected_prj="$(fetch_project_name "$list" "cfg_ownmail")" + +# Loop through every project and check if we can find a base commit. If we're +# given an expected project in the email address, try it first, so that we +# don't accidentally land on the wrong one. +for prj in $expected_prj $(get_cfgfile_projects_list "$cfgfile"); do + # First to be safe, unset the project config variables that we use + unset pcfg_branch pcfg_path pcfg_name branch_base + + # Now, parse the (new) project for pcfg variables + parse_prj_config "$cfgfile" "$prj" + + # Skip non-existent projects + if [ -z "$pcfg_name" ]; then + verbose "Could not find $prj project" + continue + fi + + # We set our own project variable called branch_base + branch_base="$(ini_config_get "$cfgfile" "prj_$prj" "branch_base")" + + # Use git-find-base with the mbox file as input, and check to see if we + # can find a commit in this project. Use the branch_base if it's + # supplied, otherwise use the parent(s) of pcfg_branch as the limiter. + # This enables only checking a small range instead of having to check + # the full history. + commit="$(git --git-dir="$(git_dir "$pcfg_path")" find-base "$pcfg_branch" "${branch_base:-$pcfg_branch}^!" 2>>"$tmp_find_base" < "$mbox" \ + || verbose "No matching base commit in project $prj" )" + + # Break out of loop once we find a base + [ -z "$commit" ] || break; +done + +if [ -n "$commit" ]; then + if [ -n "$expected_prj" ] && [ "$expected_prj" != "$prj" ]; then + verbose "Expected project $expected_prj but got $prj" + printf "%s\n" "X-Aiaiai-Expected-Project: $expected_prj" + fi + # We found a commit, so insert proper headers + printf "%s\n" "X-Aiaiai-Project: $prj" + printf "%s\n" "X-Aiaiai-Commit: $commit" +else + if [ -z "$expected_prj" ]; then + + verbose "No project found. Output of git-find-base:" + cat "$tmp_find_base" 1>&2 + + cat <<EOF +Aiaiai was unable to correctly determine the project for your patch submission. +Normally the diff index information provided by git's diff mechanism is enough. +The most probably cause is that your patch is based against an older reference +to the project, and you need to rebase your submission against the current +tree. +EOF + exit 127 + fi +fi diff --git a/email/aiaiai-email-dispatcher b/email/aiaiai-email-dispatcher index 9057fd3..e80aea3 100755 --- a/email/aiaiai-email-dispatcher +++ b/email/aiaiai-email-dispatcher @@ -4,14 +4,15 @@ # Author: Artem Bityutskiy # License: GPLv2 -srcdir="$(readlink -ev -- ${0%/*}/..)" -PATH="$srcdir/helpers:$srcdir/email:$srcdir/helpers/libshell:$PATH" +srcdir="$(readlink -ev -- ${0%/*})" +PATH="$srcdir:$srcdir/..:$srcdir/../helpers:$srcdir/../helpers/libshell:$PATH" echo $PATH . shell-error . shell-args . shell-signal . aiaiai-sh-functions +. aiaiai-email-sh-functions PROG="${0##*/}" message_time="yes" @@ -26,17 +27,17 @@ fi show_usage() { cat <<-EOF -Usage: $PROG [options] <queuedir> <validator> +Usage: $PROG [options] <queuedir> <cfgfile.ini> This program dispatches the incoming queue of patches in the <queuedir> -directory and runs the <validator> program for each patch (or patch series). -Patches are supposed to be in mbox format. +directory and runs the aiaiai-email-test-patchset program for each patch (or +patch series). Patches are supposed to be in mbox format. <cfgfile.ini> is used +for general configuration of aiaiai-email-test-patchset. <queuedir> - the directory containing the queue of patches -<validator> - the validation command. +<cfgfile.ini> - aiaiai-email-test-patchset's configuration file Options: - -J, --bigjobs=N how many validators can be run at the same time (default 1); -v, --verbose be verbose; -h, --help show this text and exit. EOF @@ -95,18 +96,12 @@ cleanup_handler() } set_cleanup_handler cleanup_handler -TEMP=`getopt -n $PROG -o J:,v,h --long bigjobs:,verbose,help -- "$@"` || +TEMP=`getopt -n $PROG -o J:,v,h --long verbose,help -- "$@"` || fail_usage "" eval set -- "$TEMP" -bigjobs=1 - while true; do case "$1" in - -J|--bigjobs) - bigjobs="$(opt_check_number "$1" "$2")" - shift - ;; -v|--verbose) verbose="--verbose" ;; -h|--help) @@ -121,7 +116,7 @@ while true; do shift done -[ "$#" -eq 2 ] || die "Insufficient or too many arguments" +[ "$#" -eq 2 ] || fail_usage "Insufficient or too many arguments" program_required "inotifywait" "" @@ -129,7 +124,9 @@ program_required "inotifywait" "" mkdir $verbose -p -- "$1" 1>&2 queuedir="$(readlink -fv -- "$1")"; shift -validator="$1"; shift +cfgfile="$1"; shift + +parse_config "$cfgfile" in_fifo="$(mktemp -dt "$PROG.in_fifo.XXXX")" tmpdir="$(mktemp -dt "$PROG.tmpdir.XXXX")" @@ -142,10 +139,10 @@ mkfifo -- "$fifo" # -P option of xargs provides us the parallelism. # # We execute the below command as a separate process - it will run the -# validator for every file name read from the fifo. -tail -f -- "$fifo" | xargs -I{} -P"$bigjobs" -- \ +# aiaiai-email-test-patchset for every file name read from the fifo. +tail -f -- "$fifo" | xargs -I{} -P"$cfg_max_validators" -- \ aiaiai-email-dispatcher-helper $verbose -- "$in_fifo/{}" "$tmpdir/STOP" \ - "$queuedir" "$validator" >&2 & + "$queuedir" "$cfgfile" >&2 & # Loop forever, wait for new files in the queue directory using inotify and # validate them diff --git a/email/aiaiai-email-dispatcher-helper b/email/aiaiai-email-dispatcher-helper index 63c56e1..36d03a3 100755 --- a/email/aiaiai-email-dispatcher-helper +++ b/email/aiaiai-email-dispatcher-helper @@ -4,12 +4,13 @@ # Author: Artem Bityutskiy # License: GPLv2 -srcdir="$(readlink -ev -- ${0%/*}/..)" -PATH="$srcdir/helpers:$srcdir/email:$srcdir/libshell:$PATH" +srcdir="$(readlink -ev -- ${0%/*})" +PATH="$srcdir:$srcdir/..:$srcdir/../helpers:$srcdir/../helpers/libshell:$PATH" . shell-error . shell-signal . aiaiai-sh-functions +. aiaiai-email-sh-functions PROG="${0##*/}" message_time="yes" @@ -24,22 +25,22 @@ fi show_usage() { cat <<-EOF -Usage: $PROG [options] <mbox> <fail_file> <queuedir> <validator> +Usage: $PROG [options] <mbox> <fail_file> <queuedir> <cfgfile.ini> -This is a helper program which runs the <validator> program for the -<mbox> file. If the validator returns success - just exit. Otherwise -it moves the <mbox> file back to the <queuedir> directory and create +This is a helper program which runs the aiaiai-email-test-patchset program for +the <mbox> file. If aiaiai-email-test-patchset returns success - just exit. +Otherwise it moves the <mbox> file back to the <queuedir> directory and create file <fail_file>. -The validator program should return failure only in case of internal -bug or error, and not if any test on the patch <mbox> failed. So the -reason we move <mbox> back to the queue is to make sure we do not -lose the patch in case of an internal bug. +aiaiai-email-test-patchset will return failure only in case of internal bug or +error, and not if any test on the patch <mbox> failed. So the reason we move +<mbox> back to the queue is to make sure we do not lose the patch in case of an +internal bug. -<mbox> - the mbox file to feed to the <validator> +<mbox> - the mbox file to feed to aiaiai-email-test-patchset <fail_file> - the file to create in case of failure <queuedir> - the directory containing the queue of patches -<validator> - the validation command. +<cfgfile.ini> - aiaiai-email-test-patchset's configuration Options: -v, --verbose be verbose; @@ -83,22 +84,25 @@ while true; do shift done -[ "$#" -eq 4 ] || die "Insufficient or too many arguments" +[ "$#" -eq 4 ] || fail_usage "Insufficient or too many arguments" mbox="$(readlink -fv -- "$1")"; shift fail_file="$1"; shift queuedir="$(readlink -fv -- "$1")"; shift -validator="$1"; shift +cfgfile="$1"; shift + +# Parse the configuration file +parse_config "$cfgfile" tmpdir="$(mktemp -dt "$PROG.XXXX")" verbose "Validating \"$mbox\", queue directory is: $queuedir" -verbose "Validator command: $validator" +verbose "Configuration file: $cfgfile" mv $verbose -- "$mbox" "$tmpdir" >&2 file="$tmpdir/${mbox##*/}" -if ! cat -- "$file" | eval "$validator"; then +if ! cat -- "$file" | aiaiai-email-test-patchset "$verbose" "$cfgfile"; then verbose "Validation failed" touch -- "$fail_file" mv $verbose -- "$file" "$queuedir" >&2 diff --git a/email/aiaiai-email-lda b/email/aiaiai-email-lda index ded1641..365d7cf 100755 --- a/email/aiaiai-email-lda +++ b/email/aiaiai-email-lda @@ -4,8 +4,8 @@ # Author: Artem Bityutskiy # License: GPLv2 -srcdir="$(readlink -ev -- ${0%/*}/..)" -PATH="$srcdir/helpers:$srcdir/email:$srcdir/helpers/libshell:$PATH" +srcdir="$(readlink -ev -- ${0%/*})" +PATH="$srcdir:$srcdir/..:$srcdir/../helpers:$srcdir/../helpers/libshell:$PATH" . shell-error . shell-args @@ -47,12 +47,6 @@ e-mail address, etc). <config.ini> - the configuration file Options: - --reap-archive=MIN remove all files created earlier than MIN - minutes ago from the mail archive directory; - --reap-incomplete=MIN remove all incomplete patch series older than - current time minus MIN minutes; - --test-mode test mode - work as usually, but do not send - replies; -v, --verbose be verbose; -h, --help show this text and exit. EOF @@ -111,10 +105,10 @@ reject_and_reply() [ -z "$verbose" ] || cat -- "$lda_tmp/mail" >&2 - if [ -z "$test_mode" ]; then + if [ "$cfg_disable_notifications" != "1" ]; then mutt -x -H "$lda_tmp/mail" </dev/null else - verbose "Do not actually send the email - we are in test mode" + verbose "Email notifications have been disabled in the configuration file" fi } @@ -219,24 +213,11 @@ queue_series() # later. local subj="$(fetch_header "Subject" < "$dir/0")" subj="X-Aiaiai-Cover-Letter-Subject: $subj" + insert_header "$mbox" "$subj" local id="$(fetch_header "Message-Id" < "$dir/0")" id="X-Aiaiai-Cover-Letter-Message-Id: $id" - - # The below trick allows us to avoid creating a separate - # temporary file: open the "$mbox" file, unlink, use the open - # file descriptor for reading and redirect the output to the - # new version of the "$mbox" file. We could instead use the - # "sponge" tool, though. - exec 3<$mbox - rm $verbose "$mbox" >&2 - verbose "Adding \"$subj\"" - formail -s formail -I "$subj" <&3 > "$mbox" - - exec 3<$mbox - rm $verbose "$mbox" >&2 - verbose "Adding \"$id\"" - formail -s formail -I "$id" <&3 > "$mbox" + insert_header "$mbox" "$id" fi queue_mboxfile "$mbox" "$n" rm $verbose -rf -- "$dir" >&2 @@ -470,28 +451,14 @@ cleanup_handler() } set_cleanup_handler cleanup_handler -TEMP=`getopt -n $PROG -o v,h --long test-mode,reap-archive:,reap-incomplete:,verbose,help -- "$@"` || +TEMP=`getopt -n $PROG -o v,h --long reap-archive:,reap-incomplete:,verbose,help -- "$@"` || fail_usage "" eval set -- "$TEMP" -test_mode= -archive_min= -incomplete_min= verbose= while true; do case "$1" in - --test-mode) - test_mode=y - ;; - --reap-archive) - archive_min="$(opt_check_number "$1" "$2")" - shift - ;; - --reap-incomplete) - incomplete_min="$(opt_check_number "$1" "$2")" - shift - ;; -v|--verbose) verbose=-v ;; -h|--help) @@ -506,7 +473,7 @@ while true; do shift done -[ "$#" -eq 2 ] || die "Insufficient or too many arguments" +[ "$#" -eq 2 ] || fail_usage "Insufficient or too many arguments" program_required "mutt" "" program_required "find" "" @@ -582,8 +549,8 @@ process_mbox "$mbox" process_all_series "$mbox" -[ -z "$incomplete_min" ] || reap_old "$lda_tmp" "$incomplete_min" -[ -z "$archive_min" ] || reap_old "$mail" "$archive_min" -[ -z "$archive_min" ] || reap_old "$queue_saved" "$archive_min" +[ -z "$cfg_lda_reap_incomplete" ] || reap_old "$lda_tmp" "$cfg_lda_reap_incomplete" +[ -z "$cfg_lda_reap_archive" ] || reap_old "$mail" "$cfg_lda_reap_archive" +[ -z "$cfg_lda_reap_archive" ] || reap_old "$queue_saved" "$cfg_lda_reap_archive" rm $verbose -f -- "$lda_tmp_lock" >&2 diff --git a/email/aiaiai-email-sh-functions b/email/aiaiai-email-sh-functions index bb77b4d..ad65260 100644 --- a/email/aiaiai-email-sh-functions +++ b/email/aiaiai-email-sh-functions @@ -4,6 +4,7 @@ # Author: Artem Bityutskiy # License: GPLv2 +. shell-args . shell-error . shell-ini-config . shell-quote @@ -134,33 +135,114 @@ ini_config_get_or_die() eval "$var=\"\$result\"" } -# Parse the "global" and "email" sections of the config file. The result is a -# set of per-option variables and their values are exactly as in the -# configuration file: +# Get the list of projects from config file "$1". +get_cfgfile_projects_list() +{ + local cfgfile="$1" + + LC_ALL=C sed -n -e "s/^\[prj_\(.*\)\]$/\1/p" "$cfgfile" +} + +# Like opt_check_number, except doesn't fail on empty number. +# Arguments: $1 is config name, $2 is config value. +# If $2 is a positive decimal, outputs it, otherwise fails. +config_check_number() +{ + if [ -n "$2" ]; then + [ -n "${2##0*}" -a -n "${2##*![0-9]*}" ] && + [ "$2" -gt 0 ] 2>/dev/null || + fatal "$1: $2: invalid number." + fi + printf %s "$2" +} + + +# Like opt_check_file, except requires executable permissions +# Arguments: $1 is config name, $2 is config value. +# If $2 is an executable file, outputs canonicalized file name, +# otherwise fails. +config_check_exec() +{ + local value + [ -x "$2" ] && + value="$(readlink -ev "$2")" || + fatal "$1: $2: not an executable file." + printf %s "$value" +} + +# Check whether config value is boolean. Empty string is considered false. +# Arguments: $1 is config name, $2 is config value. +# If $2 is a boolean value, output canonicalized value, +# otherwise, fails. +config_check_boolean() +{ + local value + value="$(echo "$2" | tr '[:upper:]' '[:lower:]')" + + # Prefixing with _ makes it easier to catch empty string as false + case "_$value" in + _|_0|_false|no) + printf %s "0" + ;; + _1|_true|yes) + printf %s "1" + ;; + *) + fatal "$1: $2: not a boolean value." + ;; + esac +} + +# Parse the "global" section of the config file. The result is a set of +# per-option variables and their values are exactly as in the configuration +# file: # # cfg_ownname, cfg_ownmail, cfg_adminname, cfg_adminmail, cfg_workdir, -# cfg_jobs, cfg_signature, cfg_built_preamble. +# cfg_max_validators, cfg_jobs, cfg_preamble, cfg_signature, +# cfg_built_preamble, cfg_disable_notifications, cfg_preserve_files, +# and cfg_email_hook # -# Additinally, the following variables are defined: -# o cfg_preamble - contains the email preamble read from the file definded -# in email.preamble variable +# Additionally, the following variables are set: +# o cfg_ownmail_local - the local portion of the ownmail address +# o cfg_ownmail_domain - the domain portion of the ownmail address +# o cfg_preamble - the contents of the file pointed to by the preamble file # # Usage: parse_config <cfgfile> parse_config() { local cfgfile="$1" - ini_config_get_or_die cfg_ownname "$cfgfile" "global" "ownname" - ini_config_get_or_die cfg_ownmail "$cfgfile" "global" "ownmail" - ini_config_get_or_die cfg_adminmail "$cfgfile" "global" "adminmail" - ini_config_get_or_die cfg_adminname "$cfgfile" "global" "adminname" - ini_config_get_or_die cfg_workdir "$cfgfile" "global" "workdir" - ini_config_get_or_die cfg_jobs "$cfgfile" "global" "jobs" - - ini_config_get_or_die cfg_signature "$cfgfile" "email" "signature" - ini_config_get_or_die cfg_preamble "$cfgfile" "email" "preamble" - ini_config_get_or_die cfg_built_preamble "$cfgfile" "email" "built_preamble" - + ini_config_get_or_die cfg_ownname "$cfgfile" "global" "ownname" + ini_config_get_or_die cfg_ownmail "$cfgfile" "global" "ownmail" + ini_config_get_or_die cfg_adminmail "$cfgfile" "global" "adminmail" + ini_config_get_or_die cfg_adminname "$cfgfile" "global" "adminname" + ini_config_get_or_die cfg_workdir "$cfgfile" "global" "workdir" + cfg_workdir="$(opt_check_dir "workdir" "$cfg_workdir")" + ini_config_get_or_die cfg_max_validators "$cfgfile" "global" "max_validators" + cfg_max_validators="$(config_check_number "max_validators" "$cfg_max_validators")" + ini_config_get_or_die cfg_jobs "$cfgfile" "global" "jobs" + cfg_jobs="$(config_check_number "jobs" "$cfg_jobs")" + ini_config_get_or_die cfg_preamble "$cfgfile" "global" "preamble" + ini_config_get_or_die cfg_signature "$cfgfile" "global" "signature" + ini_config_get_or_die cfg_built_preamble "$cfgfile" "global" "built_preamble" + + # Get Email LDA settings + cfg_lda_reap_archive="$(ini_config_get "$cfgfile" "lda" "reap_archive")" + cfg_lda_reap_archive="$(config_check_number "reap_archive" "$cfg_lda_reap_archive")" + cfg_lda_reap_incomplete="$(ini_config_get "$cfgfile" "lda" "reap_incomplete")" + cfg_lda_reap_incomplete="$(config_check_number "reap_incomplete" "$cfg_lda_reap_incomplete")" + + # Get the location of email hook(s) + cfg_email_hook="$(ini_config_get "$cfgfile" "hooks" "email")" + cfg_email_hook="$(config_check_exec "email" "$cfg_email_hook")" + + # Debug options + cfg_disable_notifications="$(ini_config_get "$cfgfile" "debug" "disable_notifications")" + cfg_disable_notifications="$(config_check_boolean "disable_notifications" "$cfg_disable_notifications")" + cfg_preserve_files="$(ini_config_get "$cfgfile" "debug" "preserve_files")" + cfg_preserve_files="$(config_check_boolean "preserve_files" "$cfg_preserve_files")" + + # Get the contents of the preamble file cfg_preamble="$(cat "$cfg_preamble")" # Get the local and domain parts of own e-mail address @@ -168,16 +250,57 @@ parse_config() cfg_ownmail_domain="$(printf "%s" "$cfg_ownmail" | LC_ALL=C sed "s/.*@//g")" } +# Parse the "defaults" section of the config file. The result is a set of +# per-option variables and their values are exactly as in the configuration +# file: +# +# __dcfg_configs, __dcfg_always_cc, __dcfg_reply_to_all, __dcfg_accept_notify, +# __dcfg_unwanted_keywords, __dcfg_kmake_opts, __dcfg_targets, +# __dcfg_defconfigdir, __dcfg_bisectability, __dcfg_sparse, __dcfg_smatch, +# __dcfg_cppcheck, __dcfg_coccinelle +# +# It is expected that this is used internally by the parse_prj_config and +# should not normally be called outside of this file. +# +# Usage: _parse_defaults_config <cfgfile> +__parse_default_config() +{ + local cfgfile="$1"; shift + + __dcfg_configs="$(ini_config_get "$cfgfile" "defaults" "configs")" + __dcfg_always_cc="$(ini_config_get "$cfgfile" "defaults" "always_cc")" + __dcfg_reply_to_all="$(ini_config_get "$cfgfile" "defaults" "reply_to_all")" + __dcfg_accept_notify="$(ini_config_get "$cfgfile" "defaults" "accept_notify")" + __dcfg_unwanted_keywords="$(ini_config_get "$cfgfile" "defaults" "unwanted_keywords")" + __dcfg_kmake_opts="$(ini_config_get "$cfgfile" "defaults" "kmake_opts")" + __dcfg_targets="$(ini_config_get "$cfgfile" "defaults" "targets")" + __dcfg_defconfigdir="$(ini_config_get "$cfgfile" "defaults" "defconfigdir")" + __dcfg_bisectability="$(ini_config_get "$cfgfile" "defaults" "bisectability")" + __dcfg_sparse="$(ini_config_get "$cfgfile" "defaults" "sparse")" + __dcfg_smatch="$(ini_config_get "$cfgfile" "defaults" "smatch")" + __dcfg_cppcheck="$(ini_config_get "$cfgfile" "defaults" "cppcheck")" + __dcfg_coccinelle="$(ini_config_get "$cfgfile" "defaults" "coccinelle")" +} + # Similar to "parse_config", but parses a project configuration section. If the # project is found, the following variables are defined: # -# cfg_name. pcfg_description, pcfg_path, pcfg_configs, pcfg_branch, -# pcfg_reply_to_all, pcfg_accept_notify, pcfg_always_cc, -# pcfg_unwanted_keywords, and pcfg_kmake_opts. +# pcfg_name, pcfg_description, pcfg_path, pcfg_branch and pcfg_canonical_url. +# +# The following variables are defined, but receive default values from the +# [defaults] section, if they are not specified in the project section: +# +# pcfg_configs, pcfg_always_cc, pcfg_reply_to_all, pcfg_accept_notify, +# pcfg_unwanted_keywords, pcfg_kmake_opts, pcfg_targets, pcfg_defconfigdir, +# pcfg_bisectability, pcfg_sparse, pcfg_smatch, pcfg_cppcheck, pcfg_coccinelle # # If the project is not found, this function only defined an empty "pcfg_name" # variable. # +# Note, this function implicitly uses _parse_defaults_config to grab the +# default configurations, and if the variable is not defined in the project +# section, it will use the default value. +# # Usage: parse_prj_config <cfgfile> <prj> parse_prj_config() { @@ -189,14 +312,63 @@ parse_prj_config() ini_config_get_or_die pcfg_description "$cfgfile" "prj_$prj" "description" ini_config_get_or_die pcfg_path "$cfgfile" "prj_$prj" "path" - ini_config_get_or_die pcfg_configs "$cfgfile" "prj_$prj" "configs" ini_config_get_or_die pcfg_branch "$cfgfile" "prj_$prj" "branch" - ini_config_get_or_die pcfg_reply_to_all "$cfgfile" "prj_$prj" "reply_to_all" - ini_config_get_or_die pcfg_accept_notify "$cfgfile" "prj_$prj" "accept_notify" + pcfg_canonical_url="$(ini_config_get "$cfgfile" "prj_$prj" "canonical_url")" + + # The following options all take default value from the "defaults" + # section, and hence "override" those settings. First we need to populate those. + __parse_default_config "$cfgfile" + + # ini_config_is_set is important here, so that defining a value as + # empty in the project section actually does define it as empty, rather + # than using the default. This allows defaults to only be used if the + # project config does not specify anything. + pcfg_configs="$(ini_config_get "$cfgfile" "prj_$prj" "configs")" + ini_config_is_set "$cfgfile" "prj_$prj" "configs" || pcfg_configs="$__dcfg_configs" + + pcfg_reply_to_all="$(ini_config_get "$cfgfile" "prj_$prj" "reply_to_all")" + ini_config_is_set "$cfgfile" "prj_$prj" "reply_to_all" || pcfg_reply_to_all="$__dcfg_reply_to_all" + pcfg_reply_to_all="$(config_check_boolean "reply_to_all" "$pcfg_reply_to_all")" + + pcfg_accept_notify="$(ini_config_get "$cfgfile" "prj_$prj" "accept_notify")" + ini_config_is_set "$cfgfile" "prj_$prj" "accept_notify" || pcfg_accept_notify="$__dcfg_accept_notify" + pcfg_accept_notify="$(config_check_boolean "accept_notify" "$pcfg_accept_notify")" pcfg_always_cc="$(ini_config_get "$cfgfile" "prj_$prj" "always_cc")" - pcfg_unwanted_keywords="$(ini_config_get "$cfgfile" ""prj_$prj"" "unwanted_keywords")" + ini_config_is_set "$cfgfile" "prj_$prj" "always_cc" || pcfg_always_cc="$__dcfg_always_cc" + + pcfg_unwanted_keywords="$(ini_config_get "$cfgfile" "prj_$prj" "unwanted_keywords")" + ini_config_is_set "$cfgfile" "prj_$prj" "unwanted_keywords" || pcfg_unwanted_keywords="$__dcfg_unwanted_keywords" + pcfg_kmake_opts="$(ini_config_get "$cfgfile" "prj_$prj" "kmake_opts")" + ini_config_is_set "$cfgfile" "prj_$prj" "kmake_opts" || pcfg_kmake_opts="$__dcfg_kmake_opts" + + pcfg_targets="$(ini_config_get "$cfgfile" "prj_$prj" "targets")" + ini_config_is_set "$cfgfile" "prj_$prj" "targets" || pcfg_targets="$__dcfg_targets" + + pcfg_defconfigdir="$(ini_config_get "$cfgfile" "prj_$prj" "defconfigdir")" + ini_config_is_set "$cfgfile" "prj_$prj" "defconfigdir" || pcfg_defconfigdir="$__dcfg_defconfigdir" + [ -z "$pcfg_defconfigdir" ] || pcfg_defconfigdir="$(opt_check_dir "defconfigdir" "$pcfg_defconfigdir")" + + pcfg_bisectability="$(ini_config_get "$cfgfile" "prj_$prj" "bisectability")" + ini_config_is_set "$cfgfile" "prj_$prj" "bisectability" || pcfg_bisectability="$__dcfg_bisectability" + pcfg_bisectability="$(config_check_boolean "bisectability" "$pcfg_bisectability")" + + pcfg_sparse="$(ini_config_get "$cfgfile" "prj_$prj" "sparse")" + ini_config_is_set "$cfgfile" "prj_$prj" "sparse" || pcfg_sparse="$__dcfg_sparse" + pcfg_sparse="$(config_check_boolean "sparse" "$pcfg_sparse")" + + pcfg_smatch="$(ini_config_get "$cfgfile" "prj_$prj" "smatch")" + ini_config_is_set "$cfgfile" "prj_$prj" "smatch" || pcfg_smatch="$__dcfg_smatch" + pcfg_smatch="$(config_check_boolean "smatch" "$pcfg_smatch")" + + pcfg_cppcheck="$(ini_config_get "$cfgfile" "prj_$prj" "cppcheck")" + ini_config_is_set "$cfgfile" "prj_$prj" "cppcheck" || pcfg_cppcheck="$__dcfg_cppcheck" + pcfg_cppcheck="$(config_check_boolean "cppcheck" "$pcfg_cppcheck")" + + pcfg_coccinelle="$(ini_config_get "$cfgfile" "prj_$prj" "coccinelle")" + ini_config_is_set "$cfgfile" "prj_$prj" "coccinelle" || pcfg_coccinelle="$__dcfg_coccinelle" + pcfg_coccinelle="$(config_check_boolean "coccinelle" "$pcfg_coccinelle")" } # Compose (but not send) e-mail reply. This function assumes that the following diff --git a/email/aiaiai-email-test-patchset b/email/aiaiai-email-test-patchset index 0028d3e..5a1cebf 100755 --- a/email/aiaiai-email-test-patchset +++ b/email/aiaiai-email-test-patchset @@ -4,8 +4,8 @@ # Author: Artem Bityutskiy # License: GPLv2 -srcdir="$(readlink -ev -- ${0%/*}/..)" -PATH="$srcdir:$srcdir/helpers:$srcdir/email:$srcdir/helpers/libshell:$PATH" +srcdir="$(readlink -ev -- ${0%/*})" +PATH="$srcdir:$srcdir/..:$srcdir/../helpers:$srcdir/../helpers/libshell:$PATH" . shell-error . shell-args @@ -35,19 +35,6 @@ The mbox file containing the patches to test is expected to come from stdin Options: -i, --input=MBOX use the MBOX file instead of stdin; - -C, --confdir=CDIR path to the directory containing the defconfig files - (those you specify in the config file); by default the - defconfig files are assumed to be part of the - <kernel-tree>; this option makes it possible to use - stand-alone defconfig files instead; - -p, --preserve preserve all the temporary files - do not clean up; - --test-mode test mode - work as usually, but do not actually - send replies; - --bisectability test bisectability; - --sparse check with sparse while building; - --smatch check with smatch while building; - --cppcheck check with cppcheck; - --coccinelle check with coccinelle; -v, --verbose be verbose; -h, --help show this text and exit. EOF @@ -60,7 +47,6 @@ fail_usage() exit 1 } -preserve= verbose= mbox= tmpdir= @@ -68,7 +54,7 @@ cleanup_handler() { rm $verbose -rf -- "$mbox" >&2 - if [ -n "$preserve" ]; then + if [ "$cfg_preserve_files" = "1" ]; then verbose "Preserved tmpdir: $tmpdir" else [ -z "$tmpdir" ] || verbose "Removing $tmpdir"; @@ -82,14 +68,22 @@ list_projects() { local prj - LC_ALL=C sed -n -e "s/^\[prj_\(.*\)\]$/\1/p" "$cfgfile" | \ - while read -r prj; do + get_cfgfile_projects_list "$cfgfile" | \ + while read -r prj; do # Get project description local descr ini_config_get_or_die descr "$cfgfile" "prj_$prj" "description" - local email="$cfg_ownmail_local+$prj@$cfg_ownmail_domain" - printf "* %s\n" "$prj ($email): $descr" + + # If given a url, display a sample clone line in the project + url="$(ini_config_get "$cfgfile" "prj_$prj" "canonical_url")" + branch="$(ini_config_get "$cfgfile" "prj_$prj" "branch")" + + if [ -n "$url" ]; then + printf "* %s\n" "$prj ($email) [git clone -b $branch $url]: $descr" + else + printf "* %s\n" "$prj ($email): $descr" + fi done } @@ -101,10 +95,10 @@ send_email() [ -z "$verbose" ] || cat -- "$tmpdir/mail" >&2 - if [ -z "$test_mode" ]; then + if [ "$cfg_disable_notifications" != "1" ]; then mutt -x -H "$tmpdir/mail" </dev/null else - verbose "Do not actually send the email - we are in test mode" + verbose "Email nofications have been disabled in the configuration file" fi } @@ -128,6 +122,23 @@ EOF exit 0 } +# This function is called when a hook has requested discarding a patch, and +# should be passed the main body content from the hook output as stdin. +error_hook_rejected_patch() +{ + send_email <<EOF +$(cat) + +List of projects $cfg_ownname supports: + +$(list_projects) + +Please, contact "$cfg_adminname" <$cfg_adminmail> +if you have any questions. +EOF + exit 0 +} + # This function is called when the patch submitter specifies a non-existing # project. It sends a sensible reply back, without carbon-copying anyone else. error_project_not_found() @@ -144,10 +155,10 @@ EOF exit 0 } -# This function is called when the 'aiaiai-test-patchest' fails. This most -# probably means a bug or configuration issues. This function sends -# corresponding e-mail notification. -error_test_patchset_failed() +# This function is called when an internal error occurs, such as when +# aiaiai-test-patchset fails. This most probably means a bug or configuration +# issue occurred. This function sends a corresponding email notification. +error_internal_error_occurred() { send_email <<EOF Sorry, but an internal $cfg_ownname error happened. Please, contact @@ -186,18 +197,11 @@ $(cat -- $tmpdir/test-patchset.log) EOF } -TEMP=`getopt -n $PROG -o i:,C:,p,v,h --long test-mode,input:,confdir:,preserve,bisectability,sparse,smatch,cppcheck,coccinelle,verbose,help -- "$@"` || +TEMP=`getopt -n $PROG -o i:,C:,p,v,h --long input:,verbose,help -- "$@"` || fail_usage "" eval set -- "$TEMP" mbox= -confdir= -test_mode= -bisectability= -sparse= -smatch= -cppcheck= -coccinelle= while true; do case "$1" in @@ -205,31 +209,6 @@ while true; do mbox="$(opt_check_read "$1" "$2")" shift ;; - -C|--confdir) - confdir="$(opt_check_dir "$1" "$2")" - shift - ;; - -p|--preserve) - preserve="--preserve" - ;; - --test-mode) - test_mode=y - ;; - --bisectability) - bisectability="--bisectability" - ;; - --sparse) - sparse="--sparse" - ;; - --smatch) - smatch="--smatch" - ;; - --cppcheck) - cppcheck="--cppcheck" - ;; - --coccinelle) - coccinelle="--coccinelle" - ;; -v|--verbose) verbose=-v ;; -h|--help) @@ -244,14 +223,15 @@ while true; do shift done -[ "$#" -eq 1 ] || die "Insufficient or too many arguments" +[ "$#" -eq 1 ] || fail_usage "Insufficient or too many arguments" program_required "mutt" "" program_required "grep" "" program_required "sed" "" program_required "formail" "" -cfgfile="$1"; shift +cfgfile="$(readlink -fv -- "$1")"; shift +parse_config "$cfgfile" # Save the mbox to a temporary file if it comes from stdin if [ -z "$mbox" ]; then @@ -259,9 +239,8 @@ if [ -z "$mbox" ]; then cat > "$mbox" fi -cat "$mbox" # Use the cover letter subject and message ID if possible. If the cover letter -# was present, 'aiaiai-email-lda' would save them in special privat headers. +# was present, 'aiaiai-email-lda' would save them in special private headers. subj="$(fetch_header "X-Aiaiai-Cover-Letter-Subject" < "$mbox")" [ -n "$subj" ] || fetch_header_or_die subj "Subject" < "$mbox" @@ -286,17 +265,13 @@ printf "\n" verbose "Testing mbox: \"$from: $subj (Message-Id: $id)\"" verbose "parsing config file \"$cfgfile\"" -parse_config "$cfgfile" - mkdir $verbose -p -- "$cfg_workdir" >&2 tmpdir="$(mktemp --tmpdir="$cfg_workdir" -dt "$PROG.XXXX")" mv $verbose -- "$mbox" "$tmpdir/mbox" >&2 mbox="$tmpdir/mbox" - -# Find out the project name -prj="$(fetch_project_name "$to" "$cfg_ownmail")" -verbose "Project \"$prj\"" +hookoutput="$tmpdir/hook" +touch -- "$hookoutput" # Replies will refer the first patch of the patch-set under test reply_subj="$subj" @@ -306,6 +281,37 @@ reply_to="$from" # And do not Cc anyone thus far reply_cc= +# Run the aiaiai email hook. Output is stored in $hookoutput and we can parse +# this via fetch_header similar to how we parse the mbox. +hookscript="$(readlink -ev -- $cfg_email_hook)" +if [ -f "$hookscript" ] && [ -x "$hookscript" ]; then + # Hook points to an executable file, so we run it + verbose "Executing \"$hookscript\"" + + # Grab the error code here, using an || section to prevent exit on + # command failure. Otherwise, the non-zero exit code from the hook + # script would crash aiaiai-email-test-patchset + hookret="0" + "$hookscript" "$cfgfile" "$mbox" > "$hookoutput" || hookret="$?" + + # Error code 127 is an expected output of the hook, and + # indicates that we should reject this patch. The reply email + # will be sent to the user, and the hook is expected to have + # outputted the rejection indication. As a precaution, the + # rejection email will include a list of projects supported. + if [ "$hookret" -eq "127" ]; then + error_hook_rejected_patch < "$hookoutput" + elif [ "$hookret" -ne "0" ]; then + verbose "Hook exited with error code \"$hookret\"..." + error_internal_error_occurred + fi +fi + +# Find out the project name +prj="$(fetch_header "X-Aiaiai-Project" < "$hookoutput")" +[ -n "$prj" ] || prj="$(fetch_project_name "$to" "$cfg_ownmail")" +verbose "Project \"$prj\"" + # Reject the e-mail if the project has not been specified if [ -z "$prj" ]; then error_no_project_specified @@ -319,11 +325,21 @@ if [ -z "$pcfg_name" ]; then error_project_not_found fi +bisectability= +sparse= +smatch= +cppcheck= +coccinelle= +[ "$pcfg_bisectability" != "1" ] || bisectablity="--bisectability" +[ "$pcfg_sparse" != "1" ] || sparse="--sparse" +[ "$pcfg_smatch" != "1" ] || smatch="--smatch" +[ "$pcfg_cppcheck" != "1" ] || cppcheck="--cppcheck" +[ "$pcfg_coccinelle" != "1" ] || coccinelle="--coccinelle" + # Create the Cc list for replies that we'll be sending if [ "$pcfg_reply_to_all" = "1" ]; then # All the patch recipients will be CCed - reply_cc="$(merge_addresses "$to" "$cc")" - reply_cc="$(merge_addresses "$reply_cc" "$pcfg_always_cc")" + reply_cc="$(merge_addresses "$to" "$cc" "$pcfg_always_cc")" reply_cc=$(strip_address "$reply_cc" "$cfg_ownmail") else reply_cc="$pcfg_always_cc" @@ -335,18 +351,22 @@ if [ "$pcfg_accept_notify" = "1" ]; then send_accepted_email fi +# Use the supplied commit from the hook or default to branch head +commit="$(fetch_header "X-Aiaiai-Commit" < "$hookoutput")" +[ -n "$commit" ] || commit="$pcfg_branch" + # Test the path (or patch-set) verbose "Test configs \"$pcfg_configs\" branch \"$pcfg_branch\" of \"$pcfg_path\"" -aiaiai-test-patchset $verbose $preserve \ - $bisectability $sparse $smatch $cppcheck $coccinelle \ - -i "$mbox" -j "$cfg_jobs" -c "$pcfg_branch" -w "$tmpdir" \ - ${confdir:+-C "$confdir"} \ - ${pcfg_unwanted_keywords:+-K "$pcfg_unwanted_keywords"} -- \ - ${pcfg_kmake_opts:+-M "$pcfg_kmake_opts"} \ +aiaiai-test-patchset $verbose ${cfg_preserve_files:+--preserve} \ + ${pcfg_targets:+--targets "$pcfg_targets"} $bisectability $sparse $smatch $cppcheck $coccinelle \ + -i "$mbox" -j "$cfg_jobs" -c "$commit" -w "$tmpdir" \ + ${pcfg_defconfigdir:+-C "$pcfg_defconfigdir"} \ + ${pcfg_unwanted_keywords:+-K "$pcfg_unwanted_keywords"} \ + ${pcfg_kmake_opts:+-M "$pcfg_kmake_opts"} -- \ "$pcfg_path" "$pcfg_configs" > "$tmpdir/test-patchset.log" || { verbose "aiaiai-test-patchset failed" - error_test_patchset_failed + error_internal_error_occurred } # Mail the results of testing diff --git a/email/systemd/aiaiai-project-update b/email/systemd/aiaiai-project-update new file mode 100755 index 0000000..e8b369d --- /dev/null +++ b/email/systemd/aiaiai-project-update @@ -0,0 +1,93 @@ +#!/bin/sh -efu + +# Copyright 2014 Intel Corporation +# Author: Jacob Keller +# License: GPLv2 + +# Run git-remote-update for each unique repository in the aiaiai cfg file + +srcdir="$(readlink -ev -- ${0%/*})" +PATH="$srcdir/../..:$srcdir/../../email:$srcdir/../../helpers:$srcdir/../../helpers/libshell:$PATH" + +. shell-error +. shell-args +. shell-signal +. aiaiai-sh-functions +. aiaiai-email-sh-functions + +PROG="${0##*/}" +export message_time="yes" + +if can_switch_to_dash; then + exec dash -euf -- "$srcdir/$PROG" "$@" + exit $? +fi + +# Find all unique repository locations. This helps prevent updating a +# repository multiple times. +find_unique_repositories() +{ + local cfgfile="$1"; shift + + get_cfgfile_projects_list "$cfgfile" | \ + while read -r prj; do + ini_config_get_or_die path "$cfgfile" "prj_$prj" "path" + printf "%s\n" "$path" + done | sort -u +} + +show_usage() +{ + cat <<-EOF +Usage: $PROG [options] <cfgfile.ini> + +The configuration file passed by the argument is parsed for projects. Then, +git-remote-update is run on each project. + +<cfgfile.ini> - the configuration file. + +Options: + -v, --verbose be verbose; + -h, --help show this text and exit. +EOF +} + +fail_usage() +{ + [ -z "$1" ] || printf "%s\n" "$1" + show_usage + exit 1 +} + +verbose= +TEMP=`getopt -n $PROG -o v,h --long verbose,help -- "$@"` || + fail_usage "" +eval set -- "$TEMP" + +while true; do + case "$1" in + -v|--verbose) + verbose=-v + ;; + -h|--help) + show_usage + exit 0 + ;; + --) shift; break + ;; + *) fail_usage "Unrecognized option: $1" + ;; + esac + shift +done + +[ "$#" -eq 1 ] || fail_usage "Insufficient or too many arguments" +cfgfile="$(readlink -fv -- "$1")"; shift +parse_config "$cfgfile" + +find_unique_repositories "$cfgfile" | \ +while read -r repo; do + verbose "Updating $repo" + + git --git-dir="$(git_dir "$repo")" remote update +done diff --git a/email/systemd/aiaiai-project-update.service b/email/systemd/aiaiai-project-update.service new file mode 100644 index 0000000..07356e7 --- /dev/null +++ b/email/systemd/aiaiai-project-update.service @@ -0,0 +1,13 @@ +# Copyright 2014 Intel Corporation +# Author: Jacob Keller +# License: GPLv2 +[Unit] +Description=Aiaiai Project Updater +OnFailure=send-mail-on-failure@%n.service + +[Service] +User=aiaiai +ExecStart=/home/aiaiai/git/aiaiai/email/systemd/aiaiai-project-update -v /home/aiaiai/work/aiaiai.cfg + +[Install] +WantedBy=multi-user.target diff --git a/email/systemd/aiaiai-project-update.timer b/email/systemd/aiaiai-project-update.timer new file mode 100644 index 0000000..b262fac --- /dev/null +++ b/email/systemd/aiaiai-project-update.timer @@ -0,0 +1,15 @@ +# Copyright 2014 Intel Corporation +# Author: Jacob Keller +# License: GPLv2 +[Unit] +Description=Update aiaiai project repositories + +[Timer] +# Wait for boot to finish +OnBootSec=5min +# Run every minute, so that we keep our trees updated +OnUnitInactiveSec=1min +Unit=aiaiai-project-update.service + +[Install] +WantedBy=multi-user.target diff --git a/email/systemd/aiaiai.conf b/email/systemd/aiaiai.conf new file mode 100644 index 0000000..456f7d8 --- /dev/null +++ b/email/systemd/aiaiai.conf @@ -0,0 +1,17 @@ +# Copyright 2014 Intel Corporation +# Author: Jacob Keller +# License: GPLv2 + +# Default Configuration for aiaiai systemd service +# NOTE: The settings below must exist here, else the systemd service will not be +# created + +# Extra PATH components +PATH=/usr/local/programs/smatch/bin:/usr/bin:/usr/local/bin:/usr/sbin:/usr/local/sbin + +# Queue directory to watch +AIAIAI_QUEUE_DIR=/home/aiaiai/work/email/queue + +# Validator command +AIAIAI_CONFIG_FILE=/home/aiaiai/work/aiaiai.cfg + diff --git a/email/systemd/aiaiai.service b/email/systemd/aiaiai.service new file mode 100644 index 0000000..939382a --- /dev/null +++ b/email/systemd/aiaiai.service @@ -0,0 +1,14 @@ +# Copyright 2014 Intel Corporation +# Author: Jacob Keller +# License: GPLv2 +[Unit] +Description=Aiaiai Email Dispatcher +OnFailure=send-mail-on-failure@%n.service + +[Service] +User=aiaiai +EnvironmentFile=/home/aiaiai/git/aiaiai/email/systemd/aiaiai.conf +ExecStart=/home/aiaiai/git/aiaiai/email/aiaiai-email-dispatcher -v ${AIAIAI_QUEUE_DIR} ${AIAIAI_CONFIG_FILE} + +[Install] +WantedBy=multi-user.target diff --git a/email/systemd/send-mail-on-failure.sh b/email/systemd/send-mail-on-failure.sh new file mode 100755 index 0000000..e8fbb0d --- /dev/null +++ b/email/systemd/send-mail-on-failure.sh @@ -0,0 +1,14 @@ +#!/bin/sh +# +# Copyright 2014 Intel Corporation +# Author: Jacob Keller +# License: GPLv2 + +service=$1 +email=$2 + +( +echo "$service has crashed, and you need to restart it!" +echo "Here is the systemctl status output:" +systemctl status -n 100 "$service" +) | mutt -s "$service crashed!" $email diff --git a/email/systemd/send-mail-on-failure@.service b/email/systemd/send-mail-on-failure@.service new file mode 100644 index 0000000..8a707d1 --- /dev/null +++ b/email/systemd/send-mail-on-failure@.service @@ -0,0 +1,10 @@ +# Copyright 2014 Intel Corporation +# Author: Jacob Keller +# License: GPLv2 +[Unit] +Description=Send mail on failure of %i + +[Service] +User=aiaiai +# You should customize the email address here +ExecStart=/home/aiaiai/git/aiaiai/systemd/send-mail-on-failure.sh %i root@localhost diff --git a/gerrit/aiaiai-jenkins-test-patchset b/gerrit/aiaiai-jenkins-test-patchset index 53da26e..e94eacb 100755 --- a/gerrit/aiaiai-jenkins-test-patchset +++ b/gerrit/aiaiai-jenkins-test-patchset @@ -4,8 +4,8 @@ # Author: Artem Bityutskiy # License: GPLv2 -srcdir="$(readlink -ev -- ${0%/*}/..)" -PATH="$srcdir:$srcdir/helpers/libshell:$PATH" +srcdir="$(readlink -ev -- ${0%/*})" +PATH="$srcdir:$srcdir/..:$srcdir/../helpers:$srcdir/../helpers/libshell:$PATH" . shell-error . shell-args @@ -181,7 +181,7 @@ while true; do shift done -[ "$#" -ge 4 ] || die "Insufficient arguments" +[ "$#" -ge 4 ] || fail_usage "Insufficient arguments" [ -n "$tmpdir" ] || tmpdir="$(mktemp -dt "$PROG.XXXX")" diff --git a/helpers/aiaiai-checker b/helpers/aiaiai-checker index b3f7274..b265b45 100755 --- a/helpers/aiaiai-checker +++ b/helpers/aiaiai-checker @@ -75,13 +75,17 @@ run_coccinelle() # Coccinelle is not stable enough so far and dies because of # internal issues sometimes or just never stops. So we specify # a timeout as well as ignore its error code. - flags="-D report -no_show_diff -very_quiet -no_includes -include_headers -timeout 60" + flags="-D report --no-show-diff --very-quiet --no-includes --include-headers --timeout 60" # Run coccinelle for each semantic patch in parallel. This may load the # system too heavily, though. We use aiaiai-locker to make sure # we have non-scrambled output. + # + # Also redirect stderr to /dev/null since it crashes sometimes + # and prints different kind of uninteresting debugging + # information to stderr aiaiai-locker -s -l "$tmpdir/lockfile" -c \ - "spatch $flags -sp_file $spatch $file_to_check" ||: & + "spatch $flags --sp-file $spatch $file_to_check" 2> /dev/null ||: & pids="$pids $!" done @@ -185,7 +189,7 @@ fi if [ -n "$run_smatch" ]; then # Smatch uses stderr for reporting about internal issues and stdout for # check results - smatch --project=kernel "$@" > "$tmpdir/smatch" 2>/dev/null & + smatch --no-data --project=kernel "$@" > "$tmpdir/smatch" 2>/dev/null & pid_smatch="$!" fi if [ -n "$run_cppcheck" ]; then diff --git a/helpers/aiaiai-diff-log b/helpers/aiaiai-diff-log index c7a572a..e5d346f 100755 --- a/helpers/aiaiai-diff-log +++ b/helpers/aiaiai-diff-log @@ -165,7 +165,7 @@ while true; do shift done -[ "$#" = 3 ] || die "Insufficient or too many arguments" +[ "$#" = 3 ] || fail_usage "Insufficient or too many arguments" compile_helpers "$srcdir" diff --git a/helpers/aiaiai-diff-log-helper b/helpers/aiaiai-diff-log-helper index 2372391..93623e2 100755 --- a/helpers/aiaiai-diff-log-helper +++ b/helpers/aiaiai-diff-log-helper @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python2 # -*- coding: utf-8 -*- """ @@ -46,7 +46,15 @@ Licence: GPLv2 # drivers/i.c: In function ‘gluebi_erase’: # drivers/i.c:177:2: warning: comparison of unsigned expression < 0 is always false [-Wtype-limits] # -# 4. Any other line comprises an independent block +# 4. All lines starting with a white-space after any of the above blocks are +# also part of the block, e.g., +# +# drivers/char/agp/isoch.c: In function ‘agp_3_5_enable’: +# drivers/char/agp/isoch.c:320:13: warning: variable ‘arqsz’ set but not used [-Wunused-but-set-variable] +# u32 isoch, arqsz; +# ^ +# +# 5. Any other line comprises an independent block import sys import os @@ -57,10 +65,12 @@ def gen_blocks(stream): """Parses input stream. Yields found blocks.""" btype, prefix, block = "", "", [] for line in stream: - # append line to the current block if prefix matches - if prefix and line.startswith(prefix): + # Append line to the current block if it starts with a white-space + if line.startswith(" "): + block.append(line) + # Append line to the current block if prefix matches + elif prefix and line.startswith(prefix): block.append(line) - # Define prefix for cases 2 and 3 for further processing elif not prefix and btype in ("ifi", "infunc"): block.append(line) diff --git a/helpers/aiaiai-extract-patches b/helpers/aiaiai-extract-patches new file mode 100755 index 0000000..fe7b7a1 --- /dev/null +++ b/helpers/aiaiai-extract-patches @@ -0,0 +1,76 @@ +#!/usr/bin/env python2 +# -*- coding: utf-8 -*- + +""" +Extract attached patches from MIME messaged passed to standard input. Will +output an mbox where each message contains a single patch. Assumes attachments +will begin with '>From' to indicate they are a seperate email to be extracted. + +Copyright 2014 Intel Corporation +Author: Jacob Keller <jacob.e.keller@intel.com> +License: GPLv2 +""" + +import argparse +import email +import sys + +def main(): + """Extract attached patches from stdin as a MIME email.""" + headers_to_copy = ["From", "Date", "Subject", "To"] + + description = """\ +Extract git patches out of an email message, including if the patch is embedded +in the email as an attachment from git-format-patch. %(prog)s only works on a +single email message at a time, so another program for splitting mbox files +such as formail is suggested.""" + + parser = argparse.ArgumentParser(description=description) + group = parser.add_mutually_exclusive_group() + group.add_argument("--discard-inline", + dest="inline", + action="store_false", + help=("discard inlined segment, " + "keeping only git-format-patch attachments")) + group.add_argument("--discard-attachments", + dest="attachments", + action="store_false", + help=("discard any git-format-patch attachments, " + "keeping only inlined email")) + + args = parser.parse_args() + + # Now, grab standard input and parse it as an email message + msg = email.message_from_file(sys.stdin) + patches = [] + + if msg.is_multipart(): + segments = msg.get_payload() + for segment in segments[:]: + if segment.get_payload().startswith(">From "): + segments.remove(segment) + patches.append(email.message_from_string( + segment.get_payload()[1:])) + + inline = email.message.Message() + inline.set_unixfrom(msg.get_unixfrom()) + for header in headers_to_copy: + inline[header] = msg[header] + + if msg.is_multipart(): + inline.set_payload("".join([x.get_payload() + for x in msg.get_payload()])) + else: + inline.set_payload(msg.get_payload()) + + if args.inline: + print inline.as_string(unixfrom=True) + + if args.attachments: + for patch in patches: + print patch.as_string(unixfrom=True) + +if __name__ == "__main__": + sys.exit(main()) + +# vim: ts=4 et sw=4 sts=4 ai sta: diff --git a/helpers/aiaiai-make-kernel b/helpers/aiaiai-make-kernel index 4f43890..4338a27 100755 --- a/helpers/aiaiai-make-kernel +++ b/helpers/aiaiai-make-kernel @@ -58,7 +58,9 @@ Options: --check-only=FILE check only files listed in FILE; -M, --kmake-opts additional options to append to the final kernel compilation 'make' command - (e.g., W=2 KALLSYMS_EXTRA_PASS=1) + (e.g., W=12 KALLSYMS_EXTRA_PASS=1) + defaults to "KCFLAGS='-Wno-missing-field-initializers + -Wno-sign-compare' W=1" -v, --verbose be verbose; -h, --help show this text and exit. EOF @@ -118,8 +120,7 @@ make_target() "make $keep_going -j $jobs -C $kernel_tree ${arch:+ARCH="$arch"} \ ${cross:+CROSS_COMPILE="$cross"} ${objdir:+O="$objdir"} \ CHECK=\"aiaiai-checker $sparse $smatch $cppcheck $coccinelle $check_only --\" \ - KCFLAGS='-Wno-missing-field-initializers -Wno-sign-compare' \ - C=$check ${check:+CF="-D__CHECK_ENDIAN__"} W=1 SHELL=\"aiaiai-locker $split -l $lockfile\" \ + C=$check ${check:+CF="-D__CHECK_ENDIAN__"} SHELL=\"aiaiai-locker $split -l $lockfile\" \ $kmake_opts $target" } @@ -155,7 +156,7 @@ coccinelle= cocci_path= check_only= check=0 -kmake_opts= +kmake_opts="KCFLAGS='-Wno-missing-field-initializers -Wno-sign-compare' W=1" verbose= while true; do @@ -233,7 +234,7 @@ while true; do shift done -[ "$#" -ge 2 ] || die "Insufficient arguments" +[ "$#" -ge 2 ] || fail_usage "Insufficient arguments" compile_helpers "$srcdir" diff --git a/helpers/aiaiai-match-keywords b/helpers/aiaiai-match-keywords index f316792..54af7c5 100755 --- a/helpers/aiaiai-match-keywords +++ b/helpers/aiaiai-match-keywords @@ -85,7 +85,7 @@ while true; do shift done -[ "$#" -eq 1 ] || die "Please, specify exactly one argument - the file with keywords" +[ "$#" -eq 1 ] || fail_usage "Please, specify exactly one argument - the file with keywords" file="$(readlink -ev -- "$1")" while IFS= read -r kw; do diff --git a/helpers/aiaiai-sh-functions b/helpers/aiaiai-sh-functions index 8d15570..799672c 100644 --- a/helpers/aiaiai-sh-functions +++ b/helpers/aiaiai-sh-functions @@ -132,6 +132,24 @@ fetch_header_per_patch() formail -s sh -c "formail -z -c -x \"$hdr:\" | head -n1" | aiaiai-decode-rfc-2047 } +# Insert a header into the given mbox file +# Usage: insert_header <mbox_file> <header> +insert_header() +{ + local mbox="$1"; shift + local header="$1"; shift + + # The below trick allows us to avoid creating a separate temporary + # file; open the "$mbox" file, unlink, use the open file descriptor for + # reading and redirect the output to the new version of the "$mbox" + # file. We could instead use the "sponge" tool, however. + exec 3<$mbox + rm $verbose "$mbox" >&2 + verbose "Adding \"$header\"" + formail -s formail -I "$header" <&3 > "$mbox" + exec 3<&- +} + git_dir() { local path="$1" @@ -157,7 +175,7 @@ apply_patch() cmt="$(git rev-parse "HEAD^{commit}")" - am="$(git am 2>&1)" || { + am="$(formail -s aiaiai-extract-patches | git am --3way 2>&1)" || { cat <<EOF Failed to apply patch(es) with git am on top of: $(git log -1 --oneline "$cmt") @@ -170,7 +188,7 @@ $(patch --merge=diff3 -p1 < .git/rebase-apply/patch 2>&1) $(print_separator) -$(git diff --no-color) +$(git diff --no-color | sed 's/^/> /') EOF return 1 } diff --git a/helpers/aiaiai-test-bisectability b/helpers/aiaiai-test-bisectability index 948a807..66d5cc5 100755 --- a/helpers/aiaiai-test-bisectability +++ b/helpers/aiaiai-test-bisectability @@ -190,7 +190,7 @@ while true; do shift done -[ "$#" -eq 2 ] || die "Please, specify exactly 2 arguments - the kernel tree and the configuration" +[ "$#" -eq 2 ] || fail_usage "Please, specify exactly 2 arguments - the kernel tree and the configuration" kernel_tree="$(readlink -ev -- "$1")"; shift defconfig_orig="$1" diff --git a/helpers/git-find-base b/helpers/git-find-base new file mode 100755 index 0000000..db7900f --- /dev/null +++ b/helpers/git-find-base @@ -0,0 +1,295 @@ +#!/usr/bin/perl +# +# Copyright 2014 Intel Corporation +# Author: Jacob Keller +# License: GPLv2 +# +# Find the most recent commit that a given mbox (*with* index information!) can +# apply, assuming that our tree has the blobs available. Passes all options +# directly to git-log, so you can shorten or filter what commits to check as +# you desire, such as limiting how far back the check will run, and so on. +use 5.016; +use warnings; +use strict; +use Getopt::Long; +use File::Basename; + +my $PROG = basename($0); + +sub show_usage { + print STDERR <<END; +Usage: $PROG [options] -- [options and arguments to git-log] ... + +This is a specialized extension of git, which can be used to help find a commit +to which a given mbox is applicable. The mbox is expected to be passed in via +standard input. + +This works by parsing diff and index information from the patch +or series of patches, in order to determine what initial index it should check +for. By default it will search the entire history (each commit, going backwards +from HEAD). You may pass arguments to git-log which limit this search. Detailed +explanation of the various git-log options which may be useful here, is beyond +the scope of this usage output, however a few examples are provided below. + +Examples: +; check only the most recent commit, and stop if it fails. + git find-base -- -1 HEAD < "mbox-file" +; check the most recent commit of a branch + git find-base -- -1 branch < "mbox-file" +; check commits between two branches + git find-base -- master..devel < "mbox-file" + +Essentially, the arguments are passed to generate a list of commit objects to +check, and you can use the powerful options in git-log to craft this list to +what you want to check against. + +The tool works by checking index information, and will return the first commit +from git-log for which the mbox passed has matching initial index information. +This means that the mbox *will* apply cleanly to that patch, because it has +exact initial index as it expected. It does *not* require that the patch be +based exactly on the commit that was supplied, but only that the files it +modified are exactly what it thought. + +Warnings and errors are printed to the standard error, and the only output to +standard out will be a single commit id. If nothing was found, no standard +output will be generated, and this utility will exit with a non-zero exit code. + +Options: + -k, --keep Keep duplicate diff chunks. + -?, -h Show this text and exit. +END +} + +sub match_index { + my ($x, $y) = @_; + + return ( index $x,$y ) == 0 or ( index $y,$x ) == 0; +} + +sub hash_comp(\%\%) { + my %x = %{ shift @_ }; + my %y = %{ shift @_ }; + + ( grep { not ( ( exists $y{$_} ) and $x{$_} eq $y{$_} ) } keys %x ) == 0; +} + +sub path_exists(\%$) { + my %tree = %{ shift @_ }; + my $path = shift @_; + + return exists $tree{$path} and $tree{$path}->{status} eq ""; +} + +my $duplicates = ''; + +Getopt::Long::Configure("pass_through"); +GetOptions('h|?' => sub { show_usage; exit 0; }, + 'keep!' => \$duplicates ); + +# Slurp the contents into $mbox for processing +my $mbox = do { local $/; <STDIN> }; + +# Array of hrefs to chunk contexts +my @chunks = (); + +# The possible list of extended headers supported by git-diff output +my $extended_headers = qr/(old mode|new mode|deleted file mode|new file mode|copy from|copy to|rename from|rename to|similarity index|dissimilarity index|index)/; + +# Split mbox apart by diff header chunks, finding a diff line followed by any number of extended header lines +while ($mbox =~ /^(?<chunk>diff (?s:.*?))(?=^(?!$extended_headers))/gm) { + + # Capture the block + my $rawchunk = $+{chunk}; + + print STDERR "Found a diff chunk\n"; + print STDERR $rawchunk; + + # Check whether it has expected format + if ( $rawchunk =~ /^diff --git [iwcoab]\/(?<oldpath>\S+) [iwcoab]\/(?<newpath>\S+)$/m ) { + # We have a standard git diff chunk. Now, we need to parse the extended + # headers from the section. + + my %chunk = (); + $chunk{oldpath} = $+{oldpath}; + $chunk{newpath} = $+{newpath}; + $chunk{oldindex} = ""; + $chunk{newindex} = ""; + $chunk{action} = "none"; + + if ( $rawchunk =~ /^index (?<oldindex>[0-9a-fA-F]+)[.]{2}(?<newindex>[0-9a-fA-F]+)( (?<mode>[0-7]{6}))?$/m ) { + $chunk{oldindex} = $+{oldindex}; + $chunk{newindex} = $+{newindex}; + $chunk{oldmode} = $+{mode}; + $chunk{newmode} = $+{mode}; + } + + + if ( $rawchunk =~ /^old mode (?<mode>[0-7]{6})$/m ) { + $chunk{oldmode} = $+{mode}; + } + + if ( $rawchunk =~ /^new mode (?<mode>[0-7]{6})$/m ) { + $chunk{newmode} = $+{mode}; + } + + if ( $rawchunk =~ /^deleted file mode (?<mode>[0-7]{6})$/m ) { + $chunk{oldmode} = $+{mode}; + $chunk{action} = "delete"; + } + + if ( $rawchunk =~ /^new file mode (?<mode>[0-7]{6})$/m ) { + $chunk{newmode} = $+{mode}; + $chunk{action} = "create"; + } + + if ( $rawchunk =~ /^rename from \Q$chunk{oldpath}\E$/m ) { + $chunk{action} = "rename"; + } + + if ( $rawchunk =~ /^rename to \Q$chunk{newpath}\E$/m ) { + $chunk{action} = "rename"; + } + + if ( $rawchunk =~ /^similarity index (?<similarity>[0-9]{1,3}%)$/m ) { + $chunk{similarity} = $+{similarity}; + } + + if ( $rawchunk =~ /^dissimilarity index (?<dissimilarity>[0-9]{1,3}%)$/m ) { + $chunk{similarity} = 100 - $+{dissimilarity}; + } + + if ( not $duplicates and ( grep { hash_comp ( %$_, %chunk ) } @chunks ) > 0 ) { + print STDERR "Skipping duplicate diff chunk. Disable this behavior with --keep.\n"; + } else { + push (@chunks, \%chunk); + } + + } elsif ( $rawchunk =~ /^diff --(combined|cc) (?<newfile>\S+)$/m ) { + # We can't use combined diff formats, since these are used for multiple + # parents, and are not suitable for this process + print STDERR "Found a combined diff format, indicating a merge. We can't find a base commit for a merge!\n"; + exit 1; + } else { + # Non git-formats are not supported, as we need the index information + print STDERR "Found a diff chunk, but it does not have a recognized format.\n"; + exit 1; + } +} + +# We have collated all the chunks. Now we need to loop over a series of commits +# based on user input. For each commit, we will try to build up the list of +# changes and see if it is applicable. +sub check_commit { + my ( $commit ) = @_; + + # Our current view of the tree + my %tree = (); + + # For each chunk, we need to build up the tree. looking up from git-ls-tree + # for the first time we find a path. We want to see if our patch could cleanly apply to the given commit. + for my $chunk ( @chunks ) { + + # If the path doesn't exist yet, just fill in some information about it + # from the real tree + if ( not exists $tree{$chunk->{oldpath}} ) { + open my $ls_tree, '-|', 'git', => 'ls-tree' => '--full-tree' => $commit => '--' => $chunk->{oldpath} + or die "Couldn't open pipe to git-ls-tree: ", $!; + + my $ls_tree_output = <$ls_tree>; + close $ls_tree or do { + print STDERR "git-ls-tree failed: ", $? >> 8; + return 0; + }; + + # Only add the tree object if we actually have output + if ( defined $ls_tree_output ) { + chomp $ls_tree_output; + $ls_tree_output =~ /\A([0-7]{6}) (blob|tree|commit) (\S+)/ or do { + print STDERR "Unexpected git-ls-tree output.\n"; + return 0; + }; + + $tree{$chunk->{oldpath}} = { + mode => $1, + index => $3, + status => "", + }; + } + } + + # We have now added any known information about this path to the tree. + # We will now attempt to modify the tree based on the contents of the + # chunk. + + if ( $chunk->{action} eq "create" ) { + if ( path_exists( %tree, $chunk->{oldpath} ) ) { + # This path already exists, so we can't add it! + print STDERR "$chunk->{oldpath} already exists.\n"; + return 0; + } else { + # We found a patch that either doesn't exist, or is already + # been renamed or deleted. We can simply add it here now. + $tree{$chunk->{oldpath}}->{mode} = $chunk->{mode}; + $tree{$chunk->{oldpath}}->{index} = $chunk->{newindex}; + $tree{$chunk->{oldpath}}->{status} = ""; + } + } else { + if ( not path_exists( %tree, $chunk->{oldpath} ) ) { + # This path no longer exists, we can't modify it. + print STDERR "$chunk->{oldpath} does not exist.\n"; + return 0; + } else { + if ( not match_index( $tree{$chunk->{oldpath}}->{index}, $chunk->{oldindex} ) ) { + print STDERR "$chunk->{oldpath} does not have matching index.\n"; + return 0; + } + + if ( $chunk->{newindex} ) { + $tree{$chunk->{oldpath}}->{index} = $chunk->{newindex}; + } + + if ( $chunk->{newmode} ) { + $tree{$chunk->{oldpath}}->{mode} = $chunk->{newmode}; + } + + # Handle special case here for rename and delete actions + if ( $chunk->{action} eq "rename" ) { + if ( path_exists( %tree, $chunk->{newpath} ) ) { + print STDERR "$chunk->{newpath} already exists.\n"; + return 0; + } + + $tree{$chunk->{newpath}} = $tree{$chunk->{oldpath}}; + $tree{$chunk->{oldpath}}->{status} = "renamed"; + } elsif ( $chunk->{action} eq "delete" ) { + $tree{$chunk->{oldpath}}->{status} = "deleted"; + } + } + } + } + + # If we get here, that means we had no issues verifying each chunk, and we + # can exit true. + return 1; +} + +# Open the git-log pipe. Pass all of our ARGV directly to the rev-list command. +open my $log, '-|', 'git' => 'log' => @ARGV => '--pretty=%H' + or die "Couldn't open pipe to git-log: ", $!; + +# Loop through each commit in the list, checking if the diff chunks can apply +# cleanly to the commit. Easily allow modifying which commits are checked via +# options to the git-log command, which allows limiting what can be checked. +while ( <$log> ) { + chomp; + + if (check_commit $_) { + # Print the commit hash we found, and exit with a good return status. + print "$_\n"; + exit 0; + } +} + +# We failed to find a commit, so exit 1 +print STDERR "Failed to find matching base commit.\n"; +exit 1; diff --git a/helpers/libshell/shell-ini-config b/helpers/libshell/shell-ini-config index 5ed42f1..ba0bfbd 100644 --- a/helpers/libshell/shell-ini-config +++ b/helpers/libshell/shell-ini-config @@ -41,6 +41,39 @@ ini_config_get() { done < "$fn" } +# Usage: ini_config_is_set file section var +ini_config_is_set() +{ + local fn section var sect= str eof= n v + fn="$1" section="$2" var="$3" + + while [ -z "$eof" ]; do + read -r str || eof=1 + + case "$str" in + "["*"]") + [ "$str" != "[$section]" ] || + sect=1 + ;; + "$shell_ini_config_comment"*|'') + ;; + *) + if [ -n "$sect" ]; then + shell_var_trim n "${str%%=*}" + + if [ "$n" = "$var" ]; then + # Return success, since we found match. + return 0 + fi + fi + ;; + esac + done < "$fn" + + # We did not find match, so return failure. + return 1 +} + # Usage: ini_config_set file section var value ini_config_set() { local fn fn_tmp section var value sect= don= str eof= n v |