contractcourt/chain_watcher: don't delete syncDispatch clients on Cancel()

This commit makes clients subscribing to channel events that are marked
"sync dispatch" _not_ being deleted from the list of clients when they
call Cancel(). Instead a go routine will be launched that will send an
error on every read of the ProcessACK channel.

This fixes a race in handing off the breach info while lnd was shutting
down. The breach arbiter could end up being shut down (and calling
Cancel()) before while the ChainWatcher was in the process of
dispatching a breach. Since the breach arbiter no longer was among the
registered clients at this point, the ChainWatcher would assume the
breach was handed off successfully, and mark the channel as pending
closed. When lnd now was restarted, the breach arbiter would not know
about the breach, and the ChainWatcher wouldn't attempt to re-dispatch,
as it was already marked as pending closed.

contractcourt/chain_watcher.go

@@ -251,6 +251,30 @@ func (c *chainWatcher) SubscribeChannelEvents(syncDispatch bool) *ChainEventSubs
 
 	if syncDispatch {
 		sub.ProcessACK = make(chan error, 1)
+
+		// If this client is syncDispatch, we cannot safely delete it
+		// from our list of clients. This is because of a potential
+		// race at shutdown, where the client shuts down and calls
+		// Cancel(). In this case we must make sure the ChainWatcher
+		// doesn't think it has successfully handed off a contract
+		// breach to the client. We start a goroutine that will send an
+		// error on the ProcessACK channel until the ChainWatcher is
+		// shutdown.
+		sub.Cancel = func() {
+			c.wg.Add(1)
+			go func() {
+				defer c.wg.Done()
+
+				err := fmt.Errorf("cancelled")
+				for {
+					select {
+					case sub.ProcessACK <- err:
+					case <-c.quit:
+						return
+					}
+				}
+			}()
+		}
 	}
 
 	c.Lock()