Commit Graph

364 Commits

Author SHA1 Message Date
Wilmer Paulino
9d51114546
chainntnfs/txnotifier: define ReorgSafetyLimit as 144 blocks 2019-01-11 16:58:15 -08:00
Olaoluwa Osuntokun
28eb8474f4
chainntnfs: modify all historical rescans to scan backwards
In this commit, we modify all existing historical rescans for
ChainNotifier backends to scan backwards rather than forwards. If we
know that a transaction has been confirmed, or outpoint spent, the it's
likely that the event has recently transpired assuming we've been
offline for a short period of time. Therefore, if we scan backwards
rather than forwards, then we can save potentially hundreds or thousands
of block fetches if the event recently happened close to the tip of the
chain.

We bound this search at the genesis block, to ensure we don't underflow
the uint32 used throughout the package in the main loop.
2018-11-24 15:50:12 -06:00
Wilmer Paulino
e402a4e146
chainntnfs: dispatch conf/spend notifications after blocks
In this commit, we alter the different ChainNotifier implementations to
dispatch confirmation and spend notifications after blocks. We do this
to ensure the external consistency of our registered clients.
2018-10-31 09:20:22 -07:00
Wilmer Paulino
770e005943
chainntnfs/txnotifer: prevent dispatching notifications within ConnectTip
In this commit, we modify the TxNotifier's ConnectTip method to no
longer dispatch notifications to any clients who had a request fulfilled
within the height connected. Instead, it will queue the notifications
for dispatch and we add a new method NotifyHeight, which will actually
dispatch them. We do this to allow the users of the TxNotifier to be
more flexible when dispatching notifications.
2018-10-31 09:20:22 -07:00
Wilmer Paulino
9a025867d0
Revert "chainntnfs/btcdnotify: disable height hint cache in testing"
This reverts commit 98e7c968d4bf5900a5d7be7f557eab8f623633c0.
2018-10-31 09:20:22 -07:00
Wilmer Paulino
5127f2aa82
Revert "chainntnfs/bitcoindnotify: disable height hints in testing"
This reverts commit ab28db5b0dc780a09b7474e187cefff809ac2714.
2018-10-31 09:20:22 -07:00
Wilmer Paulino
35bfb79099
Revert "chainntnfs/interface_test: run tests w/ disabled cache"
This reverts commit 12761a4f435a033e8425dbcb3b407bb094fcae9f.
2018-10-31 09:20:22 -07:00
Wilmer Paulino
88ac985c95
Revert "chainntnfs/height_hint_cache_test: add tests for disabled cache"
This reverts commit 45a2c9aca8f10f341848845e1f51261bd09fcb68.
2018-10-31 09:20:22 -07:00
Wilmer Paulino
716c20b18d
Revert "chainntnfs/height_hint_cache: add disable flag to hint cache"
This reverts commit 7df9ae026665de75175853e486143ab80122ce9d.
2018-10-31 09:20:22 -07:00
Wilmer Paulino
e6b1a27cd7
chainntnfs/neutrinonotify: make filter update synchronous
In this commit, we modify the notifier to handle filter updates
synchronously. We do this to prevent race conditions between new block
notifications and filter updates. Otherwise, it's possible for a new
block to come in that should match our filter, but doesn't due to the
filter being updated after.

We also modify their order so that the filter is updated first. We do
this so we can immediately start watching for the event at tip while the
rescan is ongoing.
2018-10-30 17:59:31 -07:00
Wilmer Paulino
60a1d73e08
chainntnfs/txnotifier: commit height hint after rescan is complete
In this commit, we'll now commit the current height of the TxNotifier as
the height hint for an outpoint/transaction after a rescan has been
completed and has determined that the outpoint is unspent/transaction is
unconfirmed. We do this to prevent another potentially long rescan if
the daemon is restarted before a new block comes in (which is when the
hints will be updated again).
2018-10-30 17:59:31 -07:00
Wilmer Paulino
6458868ba0
chainntnfs/interface_test: add spend reorg test 2018-10-30 17:59:31 -07:00
Wilmer Paulino
deca4cfe44
chainntnfs/neutrinonotify: remove old spend notification handling logic
In this commit, we remove the old spend notification logic within the
NeutrinoNotifier as it's been phased out by the TxNotifier.
2018-10-30 17:59:31 -07:00
Wilmer Paulino
bfd11a251e
chainntnfs/neutrinonotify: handle spend notification registration w/ TxNotifier
In this commit, we modify the logic within RegisterSpendNtfn for the
NeutrinoNotifier to account for the recent changes made to the
TxNotifier. Since it is now able to handle spend notification
registration and dispatch, we can bypass all the current logic within
the NeutrinoNotifier and interact directly with the TxNotifier instead.

The most notable change is that now we'll only attempt a historical
rescan if the TxNotifier tells us so.
2018-10-30 17:59:31 -07:00
Wilmer Paulino
74139c9a3f
chainntnfs/btcdnotify: remove old spend notification handling logic
In this commit, we remove the old spend notification logic within the
BtcdNotifier as it's been phased out by the TxNotifier.
2018-10-30 17:59:31 -07:00
Wilmer Paulino
88edd320d5
chainntnfs/btcdnotify: handle spend notification registration w/ TxNotifier
In this commit, we modify the logic within RegisterSpendNtfn for the
BtcdNotifier to account for the recent changes made to the TxNotifier.
Since it is now able to handle spend notification registration and
dispatch, we can bypass all the current logic within the
BtcdNotifier and interact directly with the TxNotifier instead.

The most notable change is that now we'll only attempt a historical
rescan if the TxNotifier tells us so.
2018-10-30 17:59:31 -07:00
Wilmer Paulino
0927f35dc1
chainntnfs/bitcoindnotify: remove old spend notification handling logic
In this commit, we remove the old spend notification logic within the
BitcoindNotifier as it's been phased out by the TxNotifier.
2018-10-30 17:59:31 -07:00
Wilmer Paulino
180dffd154
chainntnfs/bitcoindnotify: handle spend notification registration w/ TxNotifier
In this commit, we modify the logic within RegisterSpendNtfn for the
BitcoindNotifier to account for the recent changes made to the
TxNotifier. Since it is now able to handle spend notification
registration and dispatch, we can bypass all the current logic within
the BitcoindNotifier and interact directly with the TxNotifier instead.

The most notable changes include the following:

  1. We'll only attempt a historical rescan if the TxNotifier tells us
  so.

  2. We'll dispatch the historical rescan within the main goroutine to
  prevent WaitGroup panics, due to the asynchronous nature of the
  notifier.
2018-10-30 17:59:31 -07:00
Wilmer Paulino
1fe3d59836
chainntnfs/txnotifier_test: extend tests to handle spend notifications 2018-10-30 17:59:31 -07:00
Wilmer Paulino
2935392f51
chainntnfs/txnotifier: correctly update confirm/spend hints on chain updates
In this commit, we address an issue w.r.t. updating the confirm hints
for transactions and spend hints for outpoints on chain updates.
Previously, upon a block being disconnected, we'd attempt to commit a
new height hint for all outstanding confirmation notifications. This is
not correct because we'll end up modifying the height hint for things
that have confirmed at a previous height than the one being
disconnected. This would cause issues on restart when attempting a
historical dispatch, as we would start scanning at a height above which
the transaction actually confirmed in.

This has been addressed by only updating the hints for outstanding
notifications that are still unconfirmed/unspent and for notifications
that were confirmed/spent within the block being connected/disconnected.
2018-10-30 17:59:31 -07:00
Wilmer Paulino
f27e73fcb8
chainntnfs/txnotifier: detect reorgs for spending transactions of registered outpoints
In this commit, we introduce support to the TxNotifier to detect
spending transactions of registered outpoints being reorged out of the
chain. In the event that a reorg does occur, we'll consume the Spend
notification if it hasn't been consumed yet, and dispatch a Reorg
notification instead.
2018-10-30 17:59:31 -07:00
Wilmer Paulino
e6b2755a57
chainntnfs/txnotifier: watch for spends at tip
In this commit, we add support to allow the TxNotifier to properly
determine whether a new block extending the chain contains a transaction
that spends a registered outpoint. In the event that it does, spend
notifications will be dispatched to all active registered clients for
such outpoint.
2018-10-30 17:59:31 -07:00
Wilmer Paulino
4d7fa9ecc4
chainntnfs/txnotifier: allow registration of spend notifications
In this commit, we introduce the registration logic for spend
notifications to the TxNotifier. Most of this logic was taken from the
different existing ChainNotifier implementations, however, it features
some useful additions in order to make the ChainNotifier a bit more robust.

Some of these additions include the following:

  1. RegisterSpend will now return a HistoricalSpendDispatch struct,
  which includes the details for successfully determining if an outpoint
  was spent in the past. A HistoricalSpendDispatch will only be returned
  upon the first registration of an outpoint. This is done as,
  previously, if multiple clients registered for the same outpoint, then
  multiple historical rescans would also be dispatched, incurring a toll
  on the backend itself.

  2. UpdateSpendDetails will now be used to determine when a historical
  rescan has completed, no matter if a spending transaction was found or
  not. This is needed in order to responsibly update the spend hints for
  outpoints at tip, otherwise we'd attempt to update them even though we
  haven't yet determined if they have been spent or not. This will
  dispatch notifications to all currently registered clients for the
  same outpoint. In the event that another client registers later on,
  then the spending details are cached in memory in order to prevent
  further historical rescans.
2018-10-30 17:59:31 -07:00
Wilmer Paulino
fc7a33b64f
chainntnfs/txnotifier: add fields/structs to track spend notifications
In this commit, we introduce the required fields for the TxNotifier to
properly carry its duties in notifying its registered clients about the
spend of an outpoint. These are not yet used, but will be throughout the
some of the following commits.
2018-10-30 17:59:31 -07:00
Wilmer Paulino
87123d5e23
chainntnfs: extend SpendEvent with reorg channel
In this commit, we add a new channel within the SpendEvent struct that
will be sent upon whenever the spending transaction of the registered
outpoint gets reorged out of the chain. This will pave the road for
successfully handling a funding transaction getting reorged out of the
chain among other things.
2018-10-30 17:59:31 -07:00
Wilmer Paulino
f4128c9afb
chainntnfs/txnotifier: move rescanState & confNtfnSet decl to top 2018-10-30 17:59:31 -07:00
Wilmer Paulino
405e8f0fad
chainntnfs/txnotifier: rename hintCache -> confirmHintCache 2018-10-30 17:59:31 -07:00
Wilmer Paulino
f65401b439
chainntnfs/txnotifier: rename Register -> RegisterConf 2018-10-30 17:59:31 -07:00
Wilmer Paulino
82f6fd7a91
chainntnfs: rename TxConfNotifier -> TxNotifier 2018-10-30 17:59:31 -07:00
Wilmer Paulino
f8789e9db0
chainntnfs: rename txconfnotifier.go -> txnotifier.go 2018-10-30 17:59:31 -07:00
Wilmer Paulino
f4cf1073d4
chainntnfs/height_hint_cache: prevent db transactions with no updates
In this commit, we modify our height hint cache to no longer start a
database transaction if no outpoints/txids are provided to update the
height hints for.
2018-10-30 17:59:31 -07:00
Wilmer Paulino
39d86d5731
chainntnfs/interface_test: stop UnsafeStart notifiers within test
In this commit, we modify the set of tests that start the different
backend notifiers with UnsafeStart to stop them within the tests
themselves. This prevents us from running into a panic when attempting
to run the package-level tests with a filter (using test.run).
2018-10-30 17:59:31 -07:00
Conner Fromknecht
ba28ec3be0
chainntnfs/neutrinonotify/neutrino: fix debug logs 2018-10-26 18:32:32 -07:00
Conner Fromknecht
e03c818aa7
chainntnfs/txconfnotifier: remove ntfn details, bound conf depth
Removes details field from conf notifications, in favor
of using the details on the confSet. We also bound the
requested conf depth to the reorg saftey limit, as the
behavior of state tracking within the notifier is
undefined otherwise.
2018-10-26 18:32:31 -07:00
Wilmer Paulino
589dc96d88
chainntnfs/tx_notifier: extract conf reorg dispatch into method 2018-10-26 18:32:31 -07:00
Wilmer Paulino
5ae8243d0d
chainntnfs/tx_notifier: remove cached conf details on reorg
In this commit, we address a small bug where it's possible to deliver a
confirmation notification with stale confirmation details upon
registration. This can happen if a transaction has confirmed but was
reorged out of the chain later on, and a subsequent notification is
registered.
2018-10-26 18:32:31 -07:00
Wilmer Paulino
b28145b69e
chainntnfs/tx_notifier: consume reorg notification for transactions on block inclusion
In this commit, we'll attempt to consume a reorg notification for a
transaction that was previously reorged out of the chain upon block
inclusion to ensure that it is not lingering due to a client not
handling it the first time.
2018-10-26 18:32:31 -07:00
Wilmer Paulino
a4dee14b20
chainntnfs/tx_notifier: mark rescan as complete for transactions confirmed at tip
In this commit, we mark the rescan status for a transaction as complete
if we happen to detect it has confirmed within a new block that extends
the chain. We do this as otherwise, it's possible for us to not
immediately dispatch the notification upon a subsequent registration due
to the rescan state machine.
2018-10-26 18:32:31 -07:00
Conner Fromknecht
eee5311557
chainntnfs/txconnotifier: add debug logs, log errs/warnings, godocs 2018-10-26 18:32:31 -07:00
Conner Fromknecht
2dcb86bced
chainntnfs/txconfnotifier: set confset details at tip
This commit ensures that a confSet's details
are assigned in the confNotifications index
after discovering the transaction at tip. The
recent changes allow a later notification to
be dispatched on registration if an earlier one
has already discovered the confirmation details.

Before this change, it was observed that a later
registration would attempt an immediate delivery,
but fail to do so because the confset's details
were nil. This commit remedies that dispatch path,
allowing the integration tests to pass again.
2018-10-26 18:32:30 -07:00
Conner Fromknecht
df9bb56068
chainntnfs/neutrino: use HistoricalConfDispatch in ntfn registry 2018-10-26 18:32:30 -07:00
Conner Fromknecht
6cd0f867ad
chainntnfs/btcd: use HistoricalConfDispatch in ntfn registry 2018-10-26 18:32:30 -07:00
Conner Fromknecht
a4c9f62c6b
chainntnfs/bitcoind: use HistoricalConfDispatch in ntfn registry 2018-10-26 18:32:30 -07:00
Conner Fromknecht
11c231d814
chainntnfs/txconfnotifier_test: remove clientID argument...
to UpdateConfDetails
2018-10-26 18:32:30 -07:00
Conner Fromknecht
32e7368e1e
chainntnfs/txconfnotifier: remove clientID from UpdateConfDetails signature 2018-10-26 18:32:30 -07:00
Conner Fromknecht
9ae6d43916
chainntnfs/txconfnotifier: split out ntfn dispatch into helper 2018-10-26 18:32:30 -07:00
Conner Fromknecht
217b1fc0ef
chainntnfs/txconfnotifier: return HistoricalConfDispatch from Register 2018-10-26 18:32:29 -07:00
Conner Fromknecht
f94de2308f
chainntnfs/txconfnotifier: return HistoricalConfDispatch from Register 2018-10-26 18:32:29 -07:00
Conner Fromknecht
74122e00f5
chainntnfs/txconfnotifier: add PkScript to ConfNtfn 2018-10-26 18:32:29 -07:00
Conner Fromknecht
37c864d6f6
chainntnfs/txconfnotifier: add HistoricalConfDispatch struct 2018-10-26 18:32:29 -07:00
Conner Fromknecht
e804b30669
chainntnfs/txconfnotifier_test: update to use multi-value Register 2018-10-26 18:32:29 -07:00
Conner Fromknecht
cf7700e6cb
chainntnfs/bitcoind+btcd+neutrino: let tcn query for height hint 2018-10-26 18:32:29 -07:00
Conner Fromknecht
8b8007bb5a
chainntnfs/txconfnotifier: query conf hint in Register 2018-10-26 18:32:28 -07:00
Conner Fromknecht
a1756b0b1b
chainntnfs/bitcoind+btcd+neutrino: pass nil conf details 2018-10-26 18:32:28 -07:00
Conner Fromknecht
1babec971f
chainntnfs/txconfnotifier: isolate scanning ntfns 2018-10-26 18:32:28 -07:00
Conner Fromknecht
2f0b5596da
chainntnfs/txconfnotifier: add rescanStates 2018-10-26 18:32:28 -07:00
Conner Fromknecht
012d17efaa
chainntnfs/txnotifier_test: update nil spend details to restore tests 2018-10-26 18:32:28 -07:00
Conner Fromknecht
7661d00d5a
chainntnfs/txconfnotifier_test: update height hint cache test 2018-10-26 18:32:28 -07:00
Conner Fromknecht
d3bde428ea
chainntnfs/txconfnotifier_test: use tcn instead of txConfNotifier 2018-10-26 18:32:28 -07:00
Johan T. Halseth
dbf9b4ea4c
chainntnfs+queue: move ConcurrentQueue to own package 'queue' 2018-10-18 12:38:10 -07:00
Johan T. Halseth
6db0bc4b05
chainntnfs/neutrino_debug: new neutrino API 2018-10-16 19:27:07 -07:00
Johan T. Halseth
e0fd163096
chainntnfs/neutrino: new neutrino API 2018-10-16 19:26:41 -07:00
Conner Fromknecht
59b459674d
multi: init subsystem loggers via build pkg 2018-10-05 13:04:45 +09:00
Conner Fromknecht
3403794e81
chainntnfs: switch debug tag to dev
This avoids collisions with the logging level tags
2018-10-05 12:59:34 +09:00
Conner Fromknecht
91b0e9cae5
chainntnfs/txconfnotifier: commit conf hints before notifying 2018-10-02 01:09:26 -07:00
Conner Fromknecht
7dceac92d2
chainntnfs/neutrino: commit spend hints before notifying 2018-10-02 01:09:26 -07:00
Conner Fromknecht
e498a837a3
chainntnfs/btcd: commit spend hints before notifying 2018-10-02 01:09:26 -07:00
Conner Fromknecht
fb7b12a7db
chainntnfs/bitcoind: commit spend hints before notifying 2018-10-01 16:47:01 -07:00
Olaoluwa Osuntokun
6afee3d099
Merge pull request #1856 from maurycy/typos
multi: fix various typos in comments
2018-09-27 20:38:10 -07:00
Conner Fromknecht
b5b8602820
chainntnfs/neutrinonotify/neutrino: remove height shadowing
This commit removes shadowing of the currentHeight
variable when registering for neutrino spend
notifications. Currently, a locally scoped variable
is used when determining if the backend is fully
synced before attempting to call GetUtxo, which
means that the variable won't be updated after
breaking out of the loop. As a result, this could
cause us to scan unnecessarily if the backend is
catching up, e.g. after being offline for some time.
2018-09-20 18:47:04 -07:00
Wilmer Paulino
2fa64dd6ff multi: update to new BitcoindClient API 2018-09-12 21:05:19 -07:00
maurycy
ac24b12bf2
multi: fix various typos in comments 2018-09-07 06:51:49 +02:00
Valentine Wallace
98d1482942
chainntnfs/interface_test: fix unreliable historical block ntfns test
After joining the two forked chains, it is necessary to ensure they both agree on the same best hash before proceeding to UnsafeStart the notifier.
This is because when the BitcoindClient starts, it retrieves its best known block then calls GetBlockHeaderVerbose on the hash of the retrieved block. This block could be a reorged block if JoinNodes has not completed sync. If it is the case that the best block retrieved has been reorged out of the chain, GetBlockHeaderVerbose errors because bitcoind sets the number of confirmations to -1 on reorged blocks, and the btcd rpc client panics when parsing a block whose number of confirmations is negative.

This parsing error is expected to be fixed, and as a more permanent solution chain backends should ensure that the `best block` they retrieve during startup has not been reorged out of the chain.
2018-08-31 16:18:25 -07:00
Conner Fromknecht
98e7c968d4
chainntnfs/btcdnotify: disable height hint cache in testing 2018-08-26 15:34:20 -07:00
Conner Fromknecht
ab28db5b0d
chainntnfs/bitcoindnotify: disable height hints in testing 2018-08-26 15:34:05 -07:00
Conner Fromknecht
12761a4f43
chainntnfs/interface_test: run tests w/ disabled cache 2018-08-26 15:32:10 -07:00
Conner Fromknecht
45a2c9aca8
chainntnfs/height_hint_cache_test: add tests for disabled cache 2018-08-26 15:32:09 -07:00
Conner Fromknecht
7df9ae0266
chainntnfs/height_hint_cache: add disable flag to hint cache 2018-08-26 15:32:06 -07:00
Conner Fromknecht
20ba489b4a
chainntnfs/interface: fix TxConfStatus godocs for linter 2018-08-24 11:29:09 -07:00
Johan T. Halseth
f82fd5fe86
chainntnfs/bitcoind: fallback to scan manually only in case of err != nil 2018-08-24 14:46:20 +02:00
Johan T. Halseth
01d8953737
chainntnfs/btcd: fallback to scan manually only in case of err != nil 2018-08-24 14:46:20 +02:00
Johan T. Halseth
31dc387e0f
chainntnfs/btcd+bitcoind tests: use common TxConfStatus 2018-08-24 14:46:20 +02:00
Johan T. Halseth
79b2ee7da5
chainntnfs/btcd+bitcoind: use common TxConfStatus 2018-08-24 14:46:20 +02:00
Johan T. Halseth
b552984fcd
chainntnfs: define common TxConfStatus type 2018-08-24 14:46:20 +02:00
Johan T. Halseth
89c33731fe
chainntnfs/bitcoind: remove unnecessary check for tx==nil 2018-08-24 14:46:20 +02:00
Johan T. Halseth
4a60887974
chainntnfs/btcd: remove unnecessary check for tx==nil 2018-08-24 14:46:19 +02:00
Johan T. Halseth
a91466e9fa
chainntnfs/bitcoind tests: check found mempool case 2018-08-24 13:51:54 +02:00
Johan T. Halseth
cb3ec07685
chainntnfs/test_util: set 10ms trickleInterval for miner 2018-08-24 13:50:47 +02:00
Conner Fromknecht
ecc91305ac
chainntnfs/btcdnotify: initialize with height hint cache 2018-08-24 04:05:46 -07:00
Conner Fromknecht
16a321f6c5
chainntnfs/bitcoind: initialize with height hint cache 2018-08-24 04:05:18 -07:00
Olaoluwa Osuntokun
9b6b78a932
chainntnfs/neutrinonotify: update to latest API changes 2018-08-24 03:36:25 -07:00
Wilmer Paulino
191c4f3e28
multi: enable txindex on miner harness 2018-08-24 03:36:25 -07:00
Wilmer Paulino
cb688de7ad
chainntnfs: guard test chain notifier under debug flag 2018-08-24 03:36:25 -07:00
Wilmer Paulino
7895f01e2e
chainntnfs: ensure proper fallback to scanning tx manually
In this commit, we address a bug where it's possible that we still
attempt to manually scan for a transaction to determine whether it's
been included in the chain even after successfully checking the txindex
and not finding it there. Now, we'll short-circuit this process by
exiting early if the txindex lookup was successful but the transaction
in question was not found. Otherwise, we'll fall back to the manual
scan.
2018-08-24 03:36:24 -07:00
Wilmer Paulino
f88f6f68d4
chainntnfs: extract helper test vars and functions
In this commit, we extract some of the helper test variables and
functions into their own file and guard them under a build flag. This is
needed as some unit tests will be introduced in a future commit where
most of the same functions within the interface tests are reused. In
order to prevent these variables and functions from being exportable, we
guard them by the "debug" build tag.
2018-08-24 03:36:24 -07:00
Wilmer Paulino
2f644d9b62
chainntnfs/neutrinonotify: disable disconnected block ntfns while
rescanning

In this commit, we modify the rescan options Neutrino uses when
performing a rescan for historical chain events to disable disconnected
block notifications. This is needed as the Neutrino backend will mutate
its internal state while rewinding, which causes disconnected block
notifications to be sent. Since the notifier acts upon these
notifications, they would cause it to also rewind unnecessarily.
2018-08-21 13:02:17 -04:00
Wilmer Paulino
94beabf34b
chainntnfs: cache spend hints within the different chain notifiers
In this commit, we extend the different ChainNotifier implementations to
cache height hints for our spend events. Each outpoint we've requested a
spend notification for will have its initial height hint cached. We then
increment this height hint at every new block for unspent outpoints.
This allows us to retrieve the *exact* height at which the outpoint has
been spent. By doing this, we optimize the different ChainNotifier
implementations since they will no longer have to rescan forward (and
possibly fetch blocks in the neutrino/pruned node case) from the initial
height hint.
2018-08-21 12:55:53 -04:00
Wilmer Paulino
7e872566c4
chainntnfs: query the hint cache before registering a conf ntfn
In this commit, we alter the different chain notifiers to query their
height hint cache before registering a confimation notification. We do
this as it's possible that the cache has a higher height hint, which
can potentially reduce the amount of blocked fetched when attempting
historical dispatches.
2018-08-21 12:55:52 -04:00
Wilmer Paulino
30fd219b1c
chainntnfs: add height hint caches to chain notifiers 2018-08-21 12:55:51 -04:00
Wilmer Paulino
6be642a033
chainntnfs: cache confirm hints within TxConfNotifier
In this commit, we extend our TxConfNotifier to cache height hints for
our confirmation events. Each transaction we've requested a confirmation
notification for will have its initial height hint cached. We increment
this height hint at every new block for unconfirmed transactions. This
allows us to retrieve the *exact* height at which the transaction has
been included in a block. By doing this, we optimize the different
ChainNotifier implementations since they will no longer have to scan
forward (and possibly fetch blocks in the neutrino/pruned node case)
from the initial height hint looking for the confirmation.
2018-08-21 12:55:50 -04:00
Wilmer Paulino
b78cd533f2
chainntnfs: add height hint cache interfaces and channeldb implementation
In this commit, we introduce two new interfaces: SpendHintCache and
ConfirmHintCache for a height hint cache. The SpendHintCache is
responsible for maintaining the earliest height at which an outpoint
could have been spent within the chain, while the ConfirmHintCache is
responsible for maintaining the earliest height at which a transaction
confirms within the chain. We also add an implementation of these
interfaces with a single struct HeightHintCache, backed by a channeldb
instance, which will cary the duties of the interfaces above.
2018-08-21 12:55:49 -04:00
Valentine Wallace
fb7deac898
chaintnfs/interface_test: test notifiers handling missed blocks
Tests for the case where a chain backend skips a series of blocks, such that the notifier's best block is out of date. Also tests the case where a notifier's best block has been reorged out of the chain.
2018-08-11 22:45:25 -07:00
Valentine Wallace
dea22eb874
chainntnfs/interface_test: add test for client catchup on block ntfns
This tests the case where a client registers for block notifications with an outdated best block, to ensure that the client is properly caught up on the blocks that it has missed.
2018-08-11 22:45:17 -07:00
Valentine Wallace
f6ba64056a
chainntnfs/interface_test: utilize TestChainNotifier interface in tests
Switches all ChainNotifier parameters to be TestChainNotifiers. This allows access to the extra testing methods provided by the TestChainNotifier interface.
2018-08-11 22:45:17 -07:00
Valentine Wallace
b3a5b3b576
chainntnfs: add ChainNotifier test interface for test methods
TestChainNotifier wraps the ChainNotifier interface to allow adding additional testing methods with access to private fields in the notifiers. These testing methods are only compiled when the build tag "debug" is set. UnsafeStart allows starting a notifier with a specified best block.

UnsafeStart is useful for the purpose of testing cases where a notifier's best block is out of date when it receives a new block.
2018-08-11 22:45:03 -07:00
Valentine Wallace
79cbea1c9c
chainntnfs: enable notifiers to catch up on missed blocks
This resolves the situation where a notifier's chain backend skips a series of blocks, causing the notifier to need to dispatch historical block notifications to clients.

Additionally, if the current notifier's best block has been reorged out, this logic enables the notifier to rewind to the common ancestor between the current chain and the outdated best block and dispatches notifications from the ancestor.
2018-08-10 01:08:58 -07:00
Valentine Wallace
3df5b26699
chainntnfs: notify clients after block connect has succeeded
This prevents the situation where we notify clients about a newly connected block, and then the block connection itself fails. We also want to set our best block in between connecting the block and notifying clients, in case a client makes queries about the new block they have received.
2018-08-10 01:08:58 -07:00
Valentine Wallace
cbf1799c40
chainntnfs: rewind chain on missed disconnected blocks
If the chain backend misses telling the notifier about a series of disconnected blocks, the notifier is now able to disconnect the tip to its new best block.
2018-08-10 01:08:57 -07:00
Valentine Wallace
a5e1cf9c97
chainntnfs: dispatch historical block ntfns to clients
If a client passes in their best known block when registering for block notifications, check to see if it's behind our best block. If so, dispatch the missed block notifications to the client.

This is necessary because clients that persist their best known block can miss new blocks while registering for notifications.
2018-08-10 01:08:57 -07:00
Valentine Wallace
02ee5650c8
chainntnfs/neutrino: add neutrino ChainConn implementation 2018-08-10 01:08:57 -07:00
Valentine Wallace
f4005175d8
chainntnfs/interface: add ChainConn interface
This allows notifiers to pass their chain backend into interface functions to retrieve information from the chain.
2018-08-10 01:08:57 -07:00
Valentine Wallace
d4cf271526
chainntnfs: track best block in btcd and bitcoind 2018-08-10 01:08:57 -07:00
Valentine Wallace
1ffc3bb82e
multi: update to latest RegisterBlockEpochNtfn interface 2018-08-10 01:08:57 -07:00
Valentine Wallace
71a81f59a9
chainntnfs: allow clients to pass in best block
Clients can optionally pass their best block known into RegisterBlockEpochNtfn. This enables the notifiers to catch up clients on blocks they may have missed.
2018-08-10 01:08:54 -07:00
Wilmer Paulino
936fcc1f16
multi: update to latest BitcoindClient interface
In this commit, we introduce a nice optimization with regards to lnd's
interaction with a bitcoind backend. Within lnd, we currently have three
different subsystems responsible for watching the chain: chainntnfs,
lnwallet, and routing/chainview. Each of these subsystems has an active
RPC and ZMQ connection to the underlying bitcoind node. This would incur
a toll on the underlying bitcoind node and would cause us to miss ZMQ
events, which are crucial to lnd. We remedy this issue by sharing the
same connection to a bitcoind node between the different clients within
lnd.
2018-08-08 20:26:30 -07:00
Conner Fromknecht
1ded697e8d
multi: sort import paths with gofmt 2018-08-02 18:20:49 -07:00
Olaoluwa Osuntokun
b1a775b29e
chainntnfs/neutrinonotify: update async conf dispatch to use scripts 2018-07-31 21:29:04 -07:00
Olaoluwa Osuntokun
1a5b1d9792
chainntnfs: update interface tests to pass the prev script into RegisterSpendNtfn 2018-07-31 21:28:53 -07:00
Olaoluwa Osuntokun
f87b1a94c5
chainntnfs/neutrinonotify: update neutrino backend to use new API
In this commit, we update the neutrino backend for the ChainNotifier to
use the new API which requires that callers pass the outpoint along with
the pkScript to be notified of any spends.
2018-07-31 21:28:53 -07:00
Olaoluwa Osuntokun
6781b17056
chainntnfs: update bitcoind and btcd backends to match new spend ntfn API 2018-07-31 21:28:53 -07:00
Olaoluwa Osuntokun
7a0b7d2742
chainntnfs: update RegisterSpendNtfn to take the prev output script
In this commit, we update the RegisterSpendNtfn method to also take the
prev output script of the item that we're attempting to watch for. This
change is required due to the recent modifications in the neutrino
protocol (BIP 158 + 157). With the new protocol, we'll match on the
script, but then dispatch notifications based on the precise outpoint
that matches.
2018-07-31 21:28:52 -07:00
Olaoluwa Osuntokun
21847dc691
chainntnfs/neutrinonotify: update conf notifications to use pkScript
In this commit, we update the implementation of conf notifications for
neutrino to use the output script rather than the txid when matching
blocks for relevant items. The change itself is rather minor as we just
pass in the script, yet match based on the txid as normal when we go to
dispatch notifications.
2018-07-31 21:28:49 -07:00
Olaoluwa Osuntokun
e93149e576
chainntnfs/btcdnotify: update RegisterConfirmationsNtfn to take pkScript 2018-07-31 21:28:49 -07:00
Olaoluwa Osuntokun
f45e41afc1
chainntnfs/bitcoindnotify: update RegisterConfirmationsNtfn to take pkScript 2018-07-31 21:28:48 -07:00
Olaoluwa Osuntokun
6387c9e085 chainntnfs: update interface to use pkScript for conf notifications 2018-07-31 21:14:41 -07:00
Olaoluwa Osuntokun
0f98030186 chainntnfs: update RegisterConfirmationsNtfn to also pass in pkScript of output
In this commit, we prep for an upcoming final change to BIP 158. The
change results in the txid no longer being included in the regular
filter. As a result, neutrino will now need to match based on the output
script of the transaction that we wish to receive confirmation
notifications for.
2018-07-31 21:13:43 -07:00
Wilmer Paulino
8f5f3bcab2 chainntnfs/interface_test: remove code found within helper functions 2018-07-31 18:23:25 -07:00
Wilmer Paulino
65b6257e1e chainntnfs/bitcoindnotify: make historical spend rescans async 2018-07-31 18:23:25 -07:00
Wilmer Paulino
12816a910d chainntnfs: make historical confirmation rescans async 2018-07-31 18:23:25 -07:00
Wilmer Paulino
867d8524bf chainntnfs/txconfnotifier: make confirmation notifcation registration async
In this commit, we modify our TxConfNotifier struct to allow handling
notification registrations asynchronously. The Register method has been
refactored into two: Register and UpdateConfDetails. In the case that a
transaction we registered for notifications on has already confirmed,
we'll need to determine its confirmation details on our own. Once done,
this can be provided within UpdateConfDetails.

This change will pave down the road for our different chain notifiers to
handle potentially long rescans asynchronously to prevent blocking the
caller.
2018-07-31 18:23:25 -07:00
Wilmer Paulino
c43506dee9 chainntnfs: add unique ID field to track conf ntfns within notifier 2018-07-31 18:23:25 -07:00
Wilmer Paulino
ad904ebe8c chainntnfs/txconfnotifier: use concrete ErrTxConfNotifierExiting error 2018-07-31 18:23:25 -07:00
Johan T. Halseth
181014c363
chainntnfs/bitcoindnotify: remove all mempool spend clients 2018-07-22 23:09:09 +02:00
Johan T. Halseth
3808105dcd
chainntnfs/btcdnotify: remove all mempool spend clients 2018-07-22 23:09:08 +02:00
Johan T. Halseth
02f7f29b97
chainntnfs/interface_test: remove mempool spend tests 2018-07-22 23:09:08 +02:00
Johan T. Halseth
44d7b84df0
chainntnfs: remove mempool option from RegisterSpendNtfn 2018-07-22 23:09:08 +02:00
Olaoluwa Osuntokun
274687471a
chainntnfs/btcdnotify: switch to async rescan call for historical spend ntfns
In this commit, we modify the way to handle historical spend dispatches
to ensure that we don't block the client for very old rescans. Rather
than blocking and waiting for the rescan to finish (which may take
minutes in the worst case), we'll now instead launch a goroutine to
handle the async response of the rescan.
2018-07-18 01:31:49 -07:00
Johan T. Halseth
07618ca0c6
chainntnfs/interface_test: increase wait time for mempool spends
This commit increases the time we wait for a spend client to notify a
mempool spend from 50ms to 10s. This is done to catch the case where
bitcoind would use up to 7 seconds before notifying about a mempool
spend, which wasn't caught by the test.
2018-07-16 15:08:59 +02:00
Johan T. Halseth
4e47e4e7f1
chainntnfs/bitcoindnotify: filter out mempool spends from relevant txs
This commit fix a bug within the bitcoind notifier logic, which would
ignore the passed mempool argument, and notify spentness whether the
spending transaction was confirmed or not. The logic used to fix this is
similar to what is already done for the btcd backend.
2018-07-16 12:27:05 +02:00
Olaoluwa Osuntokun
80ff2c8d9f
lnwallet+chainntnfs: update neutrino API usage due to recent changes 2018-07-13 17:15:12 -07:00
Olaoluwa Osuntokun
6f60f139f4 multi: switch over import paths from roasbeef/* to btcsuite/* 2018-07-13 17:05:39 -07:00
valentinewallace
8fcd6b56cb chainntnfs: expand test for mempool spend notifications
Make sure new clients get notified about txs that are already in the mempool.

Fixes #1074.
2018-05-01 19:09:56 -07:00
Olaoluwa Osuntokun
005510b54f
chainntnfs/bitcoindnotify: fix possible panic with lack of txindex
In this commit, we fix a recently introduced bug which can result in a
panic when bitcoind nodes without a txindex active are started. The
issue was that we would still defence the transaction's blockhash, which
would be nil if we detected that the backend didn't have the txindex
active.
2018-04-17 20:08:58 -07:00
practicalswift
663c396235 multi: fix a-vs-an typos 2018-04-17 19:02:04 -07:00
Wilmer Paulino
13fb866574
chainntnfs: remove txindex requirement when registering notfications
Before this commit, we relied on the need of full nodes to enable the
transaction index. This allowed us to fetch historical details about
transactions in order to register and dispatch confirmation and spend
notifications.

This commit allows us to drop that requirement by providing a fallback
method to use when the transaction index is not enabled. This fallback
method relies on manually scanning blocks for the transactions
requested, starting from the earliest height the transactions could have
been included in, to the current height in the chain.
2018-04-17 14:59:51 -04:00
Johan T. Halseth
53ec1c6fd2
chainntnfs test: Test historical dispatch for mempool and non-mempool clients
This commit extends the test to exercise a scanario that wasn't properly
covered, by registering for a confirmed spend notification for a
historical spend. We also extend the test to make sure it handles buried
spends properly.
2018-04-16 20:09:17 +02:00
Johan T. Halseth
5d6dd90d18
chainntnfs/btcdnotify: correctly notify on confirmed rescan spends
This commit fixes a recently introduced bug in the btcdnotifier, where
we would skip all spend clients waiting for a confirmed spend in
txUpdates. The regular case where a spend is included in a new block was
correctly handled in onBlockConnected, but the txUpdates queue is also
used for confirmed spends during rescans, which we would miss. This
commit fixes that by checking if the tx update is confirmed or
unconfirmed, and acts accordingly.
2018-04-16 20:09:08 +02:00
Olaoluwa Osuntokun
86ad6d318e
Merge pull request #889 from wpaulino/chainnotifier-updates
chainntnfs: add incremental update notifications within ChainNotifier
2018-04-12 19:21:26 -07:00
Alex
89e2ba41c9 chainntnfs/bitcoindnotify: rescan blocks manually instead of rewinding 2018-04-06 20:35:27 -06:00
Wilmer Paulino
19a2bd9c7d
chainntnfs: send incremental update notifications for tx confirmations
In this commit, we introduce the ability for the different ChainNotifier
implements to send incremental updates to the subscribers of transaction
confirmations. These incremental updates represent how many
confirmations are left for the transaction to be confirmed. They are
sent to the subscriber at every new height of the chain.
2018-04-06 00:30:25 -04:00