auto-t: end process_io on HUP signal, detect process crash

When HUP is received the IO read callback was never completing which
caused it to block indefinitely until waited for. This didn't matter
for most transient processes but for IWD, hostapd, wpa_supplicant
it would cause test-runner to hang if the process crashed.

Detecting a crash is somewhat hacky because we have no process
management like systemd and the return code isn't reliable as some
processes return non-zero under normal circumstances. So to detect
a crash the process output is being checked for the string:
"++++++++ backtrace ++++++++". This isn't 100% reliable obviously
since its dependent on how the binary is compiled, but even if the
crash itself isn't detected any test should still fail if written
correctly.

Doing this allows auto-tests to handle IWD crashes gracefully by
failing the test, printing the exception (event without debugging)
and continue with other tests.

1 files changed, 7 insertions, 3 deletions

diff --git a/tools/utils.py b/tools/utils.py
index 5984fc697..d5445ea76 100644
--- a/tools/utils.py
+++ b/tools/utils.py
@@ -175,23 +175,27 @@ class Process(subprocess.Popen):
 	def process_io(self, source, condition):
 		if condition & GLib.IO_HUP:
 			self.hup = True
+			self.wait()
+			bt = self.out.partition("++++++++ backtrace ++++++++")
+			if bt[1]:
+				raise Exception(f"Process {self.args[0]} crashed!\n{bt[1] + bt[2]}")
 
 		data = source.read()
 
 		if not data:
-			return True
+			return not self.hup
 
 		try:
 			data = data.decode('utf-8')
 		except:
-			return True
+			return not self.hup
 
 		# Save data away in case the caller needs it (e.g. list_sta)
 		self.out += data
 
 		self._write_io(self, data)
 
-		return True
+		return not self.hup
 
 	def _append_outfile(self, file, append=True):
 		gid = int(os.environ.get('SUDO_GID', os.getgid()))