shutdown better

After a frustrating search for a suitable channel to use for shutdown
(tokio::sync::watch::Receiver and
futures::future::Shared<tokio::sync::oneshot::Receiver> didn't look
quite right) in which I rethought my life decisions, I finally just made
my own (server/base/shutdown.rs). We can easily poll it or wait for it
in async or sync contexts. Most importantly, it's convenient; not that
it really matters here, but it's also efficient.

We now do a slightly better job of propagating a "graceful" shutdown
signal, and this channel will give us tools to improve it over time.

* Shut down even when writer or syncer operations are stuck. Fixes #117
* Not done yet: streamers should instantly shut down without waiting for
  a connection attempt or frame or something. I'll probably
  implement that when removing --rtsp-library=ffmpeg. The code should be
  cleaner then.
* Not done yet: fix a couple places that sleep for up to a second when
  they could shut down immediately. I just need to do the plumbing for
  mock clocks to work.

I also implemented an immediate shutdown mode, activated by a second
signal. I think this will mitigate the streamer wait situation.

server/base/clock.rs

@@ -13,6 +13,8 @@ use std::thread;
 use std::time::Duration as StdDuration;
 use time::{Duration, Timespec};
 
+use crate::shutdown::ShutdownError;
+
 /// Abstract interface to the system clocks. This is for testability.
 pub trait Clocks: Send + Sync + 'static {
     /// Gets the current time from `CLOCK_REALTIME`.
@@ -35,16 +37,21 @@ pub trait Clocks: Send + Sync + 'static {
     ) -> Result<T, mpsc::RecvTimeoutError>;
 }
 
-pub fn retry_forever<C, T, E>(clocks: &C, f: &mut dyn FnMut() -> Result<T, E>) -> T
+pub fn retry<C, T, E>(
+    clocks: &C,
+    shutdown_rx: &crate::shutdown::Receiver,
+    f: &mut dyn FnMut() -> Result<T, E>,
+) -> Result<T, ShutdownError>
 where
     C: Clocks,
     E: Into<Error>,
 {
     loop {
         let e = match f() {
-            Ok(t) => return t,
+            Ok(t) => return Ok(t),
             Err(e) => e.into(),
         };
+        shutdown_rx.check()?;
         let sleep_time = Duration::seconds(1);
         warn!(
             "sleeping for {} after error: {}",