mbox/am: add --stdin-line-sep option

For all the good things it has going for it, mutt has some questionable
design choices, like how it handles piping multiple messages. Instead of
supporting a sensible option like "pipe as a valid mbox", it will
instead simply concatenate all messages together.

The only option is to tell mutt to use a $pipe_sep variable, but then
the receiving end has to know which string was used. This change
supports passing --stdin-line-sep to mbox/am/shazam commands. See docs
for details.

Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>

7 files changed, 63 insertions, 4 deletions

diff --git a/b4/__init__.py b/b4/__init__.py
index acabaf3..16ca88b 100644
--- a/b4/__init__.py
+++ b/b4/__init__.py
@@ -2532,11 +2532,21 @@ def get_strict_thread(msgs, msgid, noparent=False):
     return strict
 
 
-def mailsplit_bytes(bmbox: bytes, outdir: str) -> list:
+def mailsplit_bytes(bmbox: bytes, outdir: str, pipesep: Optional[str] = None) -> List[email.message.Message]:
+    msgs = list()
+    if pipesep:
+        logger.debug('Mailsplitting using pipesep=%s', pipesep)
+        if '\\' in pipesep:
+            import codecs
+            pipesep = codecs.decode(pipesep.encode(), 'unicode_escape')
+        for chunk in bmbox.split(pipesep.encode()):
+            if chunk.strip():
+                msgs.append(email.message_from_bytes(chunk))
+        return msgs
+
     logger.debug('Mailsplitting the mbox into %s', outdir)
     args = ['mailsplit', '--mboxrd', '-o%s' % outdir]
     ecode, out = git_run_command(None, args, stdin=bmbox)
-    msgs = list()
     if ecode > 0:
         logger.critical('Unable to parse mbox received from the server')
         return msgs
diff --git a/b4/command.py b/b4/command.py
index a2da445..245bfd0 100644
--- a/b4/command.py
+++ b/b4/command.py
@@ -20,6 +20,8 @@ def cmd_retrieval_common_opts(sp):
                     help='Use a specific project instead of default (linux-mm, linux-hardening, etc)')
     sp.add_argument('-m', '--use-local-mbox', dest='localmbox', default=None,
                     help='Instead of grabbing a thread from lore, process this mbox file (or - for stdin)')
+    sp.add_argument('--stdin-pipe-sep',
+                    help='When accepting messages on stdin, split using this pipe separator string')
     sp.add_argument('-C', '--no-cache', dest='nocache', action='store_true', default=False,
                     help='Do not use local cache')
 
diff --git a/b4/mbox.py b/b4/mbox.py
index c2e1555..ff96a11 100644
--- a/b4/mbox.py
+++ b/b4/mbox.py
@@ -698,7 +698,7 @@ def get_msgs(cmdargs: argparse.Namespace) -> Tuple[Optional[str], Optional[list]
         if cmdargs.localmbox == '-':
             # The entire mbox is passed via stdin, so mailsplit it and use the first message for our msgid
             with tempfile.TemporaryDirectory() as tfd:
-                msgs = b4.mailsplit_bytes(sys.stdin.buffer.read(), tfd)
+                msgs = b4.mailsplit_bytes(sys.stdin.buffer.read(), tfd, pipesep=cmdargs.stdin_pipe_sep)
             if not len(msgs):
                 logger.critical('Stdin did not contain any messages')
                 sys.exit(1)
diff --git a/docs/maintainer/am-shazam.rst b/docs/maintainer/am-shazam.rst
index 75864ca..afda6f1 100644
--- a/docs/maintainer/am-shazam.rst
+++ b/docs/maintainer/am-shazam.rst
@@ -64,7 +64,20 @@ The following flags are common to both commands:
   By default, b4 will retrieve threads from remote public-inbox servers,
   but it can also use a local mailbox/maildir. This is useful if you
   have a tool like ``mbsync`` or ``lei`` copying remote messages locally
-  and you need to do some work while offline.
+  and you need to do some work while offline. You can pass ``-`` to read
+  messages from stdin.
+
+``--stdin-pipe-sep STDIN_PIPE_SEP`` **(0.11+)**
+  When reading input from stdin, split messages using the string passed
+  as parameter. Otherwise, b4 expects stdin to be a single message or a
+  valid mbox.
+
+  This is most useful when piping threads directly from mutt. In your
+  ``.muttrc`` add the following configuration parameter::
+
+      set pipe_sep = '\n---randomstr---\n'
+
+  Then invoke b4 with ``-m - --stdin-pipe-sep='\n---randomstr---\n'``
 
 ``-C, --no-cache``
   By default, b4 will cache the retrieved threads for about 10 minutes.
diff --git a/docs/maintainer/mbox.rst b/docs/maintainer/mbox.rst
index dc56a51..bede66e 100644
--- a/docs/maintainer/mbox.rst
+++ b/docs/maintainer/mbox.rst
@@ -31,6 +31,25 @@ Option flags
   public-inbox supported collating and retrieving threads from across
   multiple lists. This flag will probably go away in the future.
 
+``-m LOCALMBOX, --use-local-mbox LOCALMBOX``
+  By default, b4 will retrieve threads from remote public-inbox servers,
+  but it can also use a local mailbox/maildir. This is useful if you
+  have a tool like ``mbsync`` or ``lei`` copying remote messages locally
+  and you need to do some work while offline. You can pass ``-`` to read
+  messages from stdin.
+
+``--stdin-pipe-sep STDIN_PIPE_SEP`` **(0.11+)**
+  When reading input from stdin, split messages using the string passed
+  as parameter. Otherwise, b4 expects stdin to be a single message or a
+  valid mbox.
+
+  This is most useful when piping threads directly from mutt. In your
+  ``.muttrc`` add the following configuration parameter::
+
+      set pipe_sep = '\n---randomstr---\n'
+
+  Then invoke b4 with ``-m - --stdin-pipe-sep='\n---randomstr---\n'``
+
 ``-C, --no-cache``
   By default, b4 will cache the retrieved threads for about 10 minutes.
   This lets you force b4 to ignore cache and retrieve the latest
diff --git a/man/b4.5 b/man/b4.5
index a645880..56c13ac 100644
--- a/man/b4.5
+++ b/man/b4.5
@@ -115,6 +115,9 @@ Use a specific project instead of default (linux\-mm, linux\-hardening, etc)
 .BI \-m \ LOCALMBOX\fR,\fB \ \-\-use\-local\-mbox \ LOCALMBOX
 Instead of grabbing a thread from lore, process this mbox file (or \- for stdin)
 .TP
+.BI \-\-stdin\-pipe\-sep \ STDIN_PIPE_SEP
+When accepting messages on stdin, split using this pipe separator string
+.TP
 .B  \-C\fP,\fB  \-\-no\-cache
 Do not use local cache
 .TP
@@ -184,6 +187,9 @@ Use a specific project instead of default (linux\-mm, linux\-hardening, etc)
 .BI \-m \ LOCALMBOX\fR,\fB \ \-\-use\-local\-mbox \ LOCALMBOX
 Instead of grabbing a thread from lore, process this mbox file (or \- for stdin)
 .TP
+.BI \-\-stdin\-pipe\-sep \ STDIN_PIPE_SEP
+When accepting messages on stdin, split using this pipe separator string
+.TP
 .B  \-C\fP,\fB  \-\-no\-cache
 Do not use local cache
 .TP
@@ -287,6 +293,9 @@ Use a specific project instead of default (linux\-mm, linux\-hardening, etc)
 .BI \-m \ LOCALMBOX\fR,\fB \ \-\-use\-local\-mbox \ LOCALMBOX
 Instead of grabbing a thread from lore, process this mbox file (or \- for stdin)
 .TP
+.BI \-\-stdin\-pipe\-sep \ STDIN_PIPE_SEP
+When accepting messages on stdin, split using this pipe separator string
+.TP
 .B  \-C\fP,\fB  \-\-no\-cache
 Do not use local cache
 .TP
diff --git a/man/b4.5.rst b/man/b4.5.rst
index 0813883..072c107 100644
--- a/man/b4.5.rst
+++ b/man/b4.5.rst
@@ -75,6 +75,8 @@ options:
                         Use a specific project instead of default (linux-mm, linux-hardening, etc)
   -m LOCALMBOX, --use-local-mbox LOCALMBOX
                         Instead of grabbing a thread from lore, process this mbox file (or - for stdin)
+  --stdin-pipe-sep STDIN_PIPE_SEP
+                        When accepting messages on stdin, split using this pipe separator string
   -C, --no-cache
                         Do not use local cache
   -o OUTDIR, --outdir OUTDIR
@@ -124,6 +126,8 @@ options:
                         Use a specific project instead of default (linux-mm, linux-hardening, etc)
   -m LOCALMBOX, --use-local-mbox LOCALMBOX
                         Instead of grabbing a thread from lore, process this mbox file (or - for stdin)
+  --stdin-pipe-sep STDIN_PIPE_SEP
+                        When accepting messages on stdin, split using this pipe separator string
   -C, --no-cache
                         Do not use local cache
   -o OUTDIR, --outdir OUTDIR
@@ -197,6 +201,8 @@ options:
                         Use a specific project instead of default (linux-mm, linux-hardening, etc)
   -m LOCALMBOX, --use-local-mbox LOCALMBOX
                         Instead of grabbing a thread from lore, process this mbox file (or - for stdin)
+  --stdin-pipe-sep STDIN_PIPE_SEP
+                        When accepting messages on stdin, split using this pipe separator string
   -C, --no-cache        Do not use local cache
   -v WANTVER, --use-version WANTVER
                         Get a specific version of the patch/series