In this commit, we modify the logic within RegisterSpendNtfn for the
BitcoindNotifier to account for the recent changes made to the
TxNotifier. Since it is now able to handle spend notification
registration and dispatch, we can bypass all the current logic within
the BitcoindNotifier and interact directly with the TxNotifier instead.
The most notable changes include the following:
1. We'll only attempt a historical rescan if the TxNotifier tells us
so.
2. We'll dispatch the historical rescan within the main goroutine to
prevent WaitGroup panics, due to the asynchronous nature of the
notifier.
In this commit, we address an issue w.r.t. updating the confirm hints
for transactions and spend hints for outpoints on chain updates.
Previously, upon a block being disconnected, we'd attempt to commit a
new height hint for all outstanding confirmation notifications. This is
not correct because we'll end up modifying the height hint for things
that have confirmed at a previous height than the one being
disconnected. This would cause issues on restart when attempting a
historical dispatch, as we would start scanning at a height above which
the transaction actually confirmed in.
This has been addressed by only updating the hints for outstanding
notifications that are still unconfirmed/unspent and for notifications
that were confirmed/spent within the block being connected/disconnected.
In this commit, we introduce support to the TxNotifier to detect
spending transactions of registered outpoints being reorged out of the
chain. In the event that a reorg does occur, we'll consume the Spend
notification if it hasn't been consumed yet, and dispatch a Reorg
notification instead.
In this commit, we add support to allow the TxNotifier to properly
determine whether a new block extending the chain contains a transaction
that spends a registered outpoint. In the event that it does, spend
notifications will be dispatched to all active registered clients for
such outpoint.
In this commit, we introduce the registration logic for spend
notifications to the TxNotifier. Most of this logic was taken from the
different existing ChainNotifier implementations, however, it features
some useful additions in order to make the ChainNotifier a bit more robust.
Some of these additions include the following:
1. RegisterSpend will now return a HistoricalSpendDispatch struct,
which includes the details for successfully determining if an outpoint
was spent in the past. A HistoricalSpendDispatch will only be returned
upon the first registration of an outpoint. This is done as,
previously, if multiple clients registered for the same outpoint, then
multiple historical rescans would also be dispatched, incurring a toll
on the backend itself.
2. UpdateSpendDetails will now be used to determine when a historical
rescan has completed, no matter if a spending transaction was found or
not. This is needed in order to responsibly update the spend hints for
outpoints at tip, otherwise we'd attempt to update them even though we
haven't yet determined if they have been spent or not. This will
dispatch notifications to all currently registered clients for the
same outpoint. In the event that another client registers later on,
then the spending details are cached in memory in order to prevent
further historical rescans.
In this commit, we introduce the required fields for the TxNotifier to
properly carry its duties in notifying its registered clients about the
spend of an outpoint. These are not yet used, but will be throughout the
some of the following commits.
In this commit, we add a new channel within the SpendEvent struct that
will be sent upon whenever the spending transaction of the registered
outpoint gets reorged out of the chain. This will pave the road for
successfully handling a funding transaction getting reorged out of the
chain among other things.
In this commit, we modify our height hint cache to no longer start a
database transaction if no outpoints/txids are provided to update the
height hints for.
In this commit, we modify the set of tests that start the different
backend notifiers with UnsafeStart to stop them within the tests
themselves. This prevents us from running into a panic when attempting
to run the package-level tests with a filter (using test.run).
Removes details field from conf notifications, in favor
of using the details on the confSet. We also bound the
requested conf depth to the reorg saftey limit, as the
behavior of state tracking within the notifier is
undefined otherwise.
In this commit, we address a small bug where it's possible to deliver a
confirmation notification with stale confirmation details upon
registration. This can happen if a transaction has confirmed but was
reorged out of the chain later on, and a subsequent notification is
registered.
In this commit, we'll attempt to consume a reorg notification for a
transaction that was previously reorged out of the chain upon block
inclusion to ensure that it is not lingering due to a client not
handling it the first time.
In this commit, we mark the rescan status for a transaction as complete
if we happen to detect it has confirmed within a new block that extends
the chain. We do this as otherwise, it's possible for us to not
immediately dispatch the notification upon a subsequent registration due
to the rescan state machine.
This commit ensures that a confSet's details
are assigned in the confNotifications index
after discovering the transaction at tip. The
recent changes allow a later notification to
be dispatched on registration if an earlier one
has already discovered the confirmation details.
Before this change, it was observed that a later
registration would attempt an immediate delivery,
but fail to do so because the confset's details
were nil. This commit remedies that dispatch path,
allowing the integration tests to pass again.
This commit removes shadowing of the currentHeight
variable when registering for neutrino spend
notifications. Currently, a locally scoped variable
is used when determining if the backend is fully
synced before attempting to call GetUtxo, which
means that the variable won't be updated after
breaking out of the loop. As a result, this could
cause us to scan unnecessarily if the backend is
catching up, e.g. after being offline for some time.
After joining the two forked chains, it is necessary to ensure they both agree on the same best hash before proceeding to UnsafeStart the notifier.
This is because when the BitcoindClient starts, it retrieves its best known block then calls GetBlockHeaderVerbose on the hash of the retrieved block. This block could be a reorged block if JoinNodes has not completed sync. If it is the case that the best block retrieved has been reorged out of the chain, GetBlockHeaderVerbose errors because bitcoind sets the number of confirmations to -1 on reorged blocks, and the btcd rpc client panics when parsing a block whose number of confirmations is negative.
This parsing error is expected to be fixed, and as a more permanent solution chain backends should ensure that the `best block` they retrieve during startup has not been reorged out of the chain.
In this commit, we address a bug where it's possible that we still
attempt to manually scan for a transaction to determine whether it's
been included in the chain even after successfully checking the txindex
and not finding it there. Now, we'll short-circuit this process by
exiting early if the txindex lookup was successful but the transaction
in question was not found. Otherwise, we'll fall back to the manual
scan.
In this commit, we extract some of the helper test variables and
functions into their own file and guard them under a build flag. This is
needed as some unit tests will be introduced in a future commit where
most of the same functions within the interface tests are reused. In
order to prevent these variables and functions from being exportable, we
guard them by the "debug" build tag.
rescanning
In this commit, we modify the rescan options Neutrino uses when
performing a rescan for historical chain events to disable disconnected
block notifications. This is needed as the Neutrino backend will mutate
its internal state while rewinding, which causes disconnected block
notifications to be sent. Since the notifier acts upon these
notifications, they would cause it to also rewind unnecessarily.
In this commit, we extend the different ChainNotifier implementations to
cache height hints for our spend events. Each outpoint we've requested a
spend notification for will have its initial height hint cached. We then
increment this height hint at every new block for unspent outpoints.
This allows us to retrieve the *exact* height at which the outpoint has
been spent. By doing this, we optimize the different ChainNotifier
implementations since they will no longer have to rescan forward (and
possibly fetch blocks in the neutrino/pruned node case) from the initial
height hint.
In this commit, we alter the different chain notifiers to query their
height hint cache before registering a confimation notification. We do
this as it's possible that the cache has a higher height hint, which
can potentially reduce the amount of blocked fetched when attempting
historical dispatches.
In this commit, we extend our TxConfNotifier to cache height hints for
our confirmation events. Each transaction we've requested a confirmation
notification for will have its initial height hint cached. We increment
this height hint at every new block for unconfirmed transactions. This
allows us to retrieve the *exact* height at which the transaction has
been included in a block. By doing this, we optimize the different
ChainNotifier implementations since they will no longer have to scan
forward (and possibly fetch blocks in the neutrino/pruned node case)
from the initial height hint looking for the confirmation.
In this commit, we introduce two new interfaces: SpendHintCache and
ConfirmHintCache for a height hint cache. The SpendHintCache is
responsible for maintaining the earliest height at which an outpoint
could have been spent within the chain, while the ConfirmHintCache is
responsible for maintaining the earliest height at which a transaction
confirms within the chain. We also add an implementation of these
interfaces with a single struct HeightHintCache, backed by a channeldb
instance, which will cary the duties of the interfaces above.
Tests for the case where a chain backend skips a series of blocks, such that the notifier's best block is out of date. Also tests the case where a notifier's best block has been reorged out of the chain.
This tests the case where a client registers for block notifications with an outdated best block, to ensure that the client is properly caught up on the blocks that it has missed.
Switches all ChainNotifier parameters to be TestChainNotifiers. This allows access to the extra testing methods provided by the TestChainNotifier interface.
TestChainNotifier wraps the ChainNotifier interface to allow adding additional testing methods with access to private fields in the notifiers. These testing methods are only compiled when the build tag "debug" is set. UnsafeStart allows starting a notifier with a specified best block.
UnsafeStart is useful for the purpose of testing cases where a notifier's best block is out of date when it receives a new block.
This resolves the situation where a notifier's chain backend skips a series of blocks, causing the notifier to need to dispatch historical block notifications to clients.
Additionally, if the current notifier's best block has been reorged out, this logic enables the notifier to rewind to the common ancestor between the current chain and the outdated best block and dispatches notifications from the ancestor.
This prevents the situation where we notify clients about a newly connected block, and then the block connection itself fails. We also want to set our best block in between connecting the block and notifying clients, in case a client makes queries about the new block they have received.
If the chain backend misses telling the notifier about a series of disconnected blocks, the notifier is now able to disconnect the tip to its new best block.
If a client passes in their best known block when registering for block notifications, check to see if it's behind our best block. If so, dispatch the missed block notifications to the client.
This is necessary because clients that persist their best known block can miss new blocks while registering for notifications.
Clients can optionally pass their best block known into RegisterBlockEpochNtfn. This enables the notifiers to catch up clients on blocks they may have missed.
In this commit, we introduce a nice optimization with regards to lnd's
interaction with a bitcoind backend. Within lnd, we currently have three
different subsystems responsible for watching the chain: chainntnfs,
lnwallet, and routing/chainview. Each of these subsystems has an active
RPC and ZMQ connection to the underlying bitcoind node. This would incur
a toll on the underlying bitcoind node and would cause us to miss ZMQ
events, which are crucial to lnd. We remedy this issue by sharing the
same connection to a bitcoind node between the different clients within
lnd.
