Commit Graph

325 Commits

Author SHA1 Message Date
Olaoluwa Osuntokun
7e298f1434
Merge pull request #3367 from cfromknecht/batched-graph-updates
Batched graph updates
2020-11-25 18:40:40 -08:00
Wilmer Paulino
791ba3eb50
discovery: rate limit incoming channel updates
This change was largely motivated by an increase in high disk usage as a
result of channel update spam. With an in memory graph, this would've
gone mostly undetected except for the increased bandwidth usage, which
this doesn't aim to solve yet. To minimize the effects to disks, we
begin to rate limit channel updates in two ways. Keep alive updates,
those which only increase their timestamps to signal liveliness, are now
limited to one per lnd's rebroadcast interval (current default of 24H).
Non keep alive updates are now limited to one per block per direction.
2020-11-25 15:38:08 -08:00
Conner Fromknecht
f8154c65c5
discovery/gossiper: increase validation barrier size to 1000
This allows for a 1000 different validation operations to proceed
concurrently. Now that we are batching operations at the db level, the
average number of outstanding requests will be higher since the commit
latency has increased. To compensate, we allow for more outstanding
requests to keep the gossiper busy while batches are constructed.
2020-11-24 16:39:47 -08:00
Conner Fromknecht
fb9218d100
discovery/gossiper: channel announcements can't be outdated 2020-11-24 16:38:14 -08:00
Andras Banki-Horvath
d89f51d1d0
multi: add reset closure to kvdb.Update
Similarly as with kvdb.View this commits adds a reset closure to the
kvdb.Update call in order to be able to reset external state if the
underlying db backend needs to retry the transaction.
2020-11-05 17:57:12 +01:00
Andras Banki-Horvath
2a358327f4
multi: add reset closure to kvdb.View
This commit adds a reset() closure to the kvdb.View function which will
be called before each retry (including the first) of the view
transaction. The reset() closure can be used to reset external state
(eg slices or maps) where the view closure puts intermediate results.
2020-11-05 17:57:12 +01:00
yyforyongyu
ef38b12fda
multi: use timeout field in dialer 2020-09-16 11:50:04 +08:00
eugene
49d8f04197 multi: migrate instances of mockSigner to the mock package
This commit moves all localized instances of mock implementations of
the Signer interface to the lntest/mock package. This allows us to
remove a lot of code and have it housed under a single interface in
many cases.
2020-08-28 15:43:51 -04:00
Conner Fromknecht
cff52f7622
Merge pull request #4352 from matheusdtech/discovery-lock-premature
discovery: correctly lock premature messages
2020-06-26 22:50:09 -07:00
Brian Mancini
28931390ff discovery: prevent endBlock overflow in replyChanRangeQuery
Modifies syncer.replyChanRangeQuery method to use the LastBlockHeight
method on the query. LastBlockHeight safely calculates the ending
block height and prevents an overflow of start_block + num_blocks.

Prior to this change, query messages that had a start_block +
num_blocks that overflows uint32_max would return zero results in the
reply message.

Tests are added to fix the bug and ensure proper start and end values
are supplied to the channel graph filter.
2020-06-18 16:48:09 -04:00
Matheus Degiovani
44f83731bc discovery: Correctly lock premature annoucements
This reworks the locking behavior of the Gossiper so that a race
condition on channel updates and block notifications doesn't cause any
loss of messages.

This fixes an issue that manifested mostly as flakes on itests during
WaitForNetworkChannelOpen calls.

The previous behavior allowed ChannelUpdates to be missed if they
happened concurrently to block notifications. The
processNetworkAnnoucement call would check for the current block height,
then lock the gossiper and add the msg to the prematureAnnoucements
list. New blocks would trigger an update to the current block height
then a lock and check of the aforementioned list.

However, specially during itests it could happen that the missing lock
before checking the height could case a race condition if the following
sequence of events happened:

- A new ChannelUpdate message was received and started processing on a
  separate goroutine
- The isPremature() call was made and verified that the ChannelUpdate
  was in fact premature
- The goroutine was scheduled out
- A new block started processing in the gossiper. It updated the block
  height, asked and was granted the lock for the gossiper and verified
  there was zero premature announcements. The lock was released.
- The goroutine processing the ChannelUpdate asked for the gossiper lock
  and was granted it. It added the ChannelUpdate in the
  prematureAnnoucements list. This can never be processed now.

The way to fix this behavior is to ensure that both isPremature checks
done inside processNetworkAnnoucement and best block updates are made
inside the same critical section (i.e. while holding the same lock) so
that they can't both check and update the prematureAnnoucements list
concurrently.
2020-06-05 15:58:33 -03:00
Matheus Degiovani
ccc8f8e48f discovery: Log new blocks
This should help debug some flaky itests.
2020-06-05 13:31:40 -03:00
Conner Fromknecht
d0d2ca403d
multi: rename ReadTx to RTx 2020-05-26 18:20:37 -07:00
Roei Erez
ae2c37e043 Ensure chain notifier is started before accessed.
The use case comes from the RPC layer that is ready before the
chain notifier which is used in the sub server.
2020-04-30 12:54:33 +03:00
Conner Fromknecht
0f94b8dc62
multi: return input.Signature from SignOutputRaw 2020-04-10 14:27:35 -07:00
Conner Fromknecht
ec784db511
multi: remove returned error from WipeChannel
The linter complains about not checking the return value from
WipeChannel in certain places. Instead of checking we simply remove the
returned error because the in-memory modifications cannot fail.
2020-04-02 17:39:29 -07:00
Conner Fromknecht
4e793497c8
Merge pull request #2669 from cfromknecht/use-netann-in-discovery
netann+discovery+server: consolidate network announcements to netann pkg
2020-03-23 13:38:06 -07:00
Conner Fromknecht
92456d063d
discovery: remove unused updateChanPolicies struct 2020-03-19 13:43:57 -07:00
Conner Fromknecht
5c2fc4a2d6
discovery/gossiper: use netann pkg for signing channel updates 2020-03-19 13:43:39 -07:00
Olaoluwa Osuntokun
ace7a78494
discovery: covert to use new kvdb abstraction 2020-03-18 19:35:07 -07:00
Conner Fromknecht
089ac647d8
discovery/chan_series: use netann.ChannelUpdateFromEdge helper 2020-03-17 16:24:25 -07:00
Conner Fromknecht
7b0d564692
discovery: move remotePubFromChanInfo to gossiper, remove utils 2020-03-17 16:24:10 -07:00
Conner Fromknecht
6a813e3433
discovery/multi: move CreateChanAnnouncement to netann 2020-03-17 16:23:54 -07:00
Conner Fromknecht
d82aacbdc5
discovery/utils: use netann.ChannelUpdateFromEdge 2020-03-17 16:23:37 -07:00
Conner Fromknecht
df44d19936
discovery/multi: move SignAnnouncement to netann 2020-03-17 16:23:01 -07:00
Wilmer Paulino
57b69e3b1a
discovery: check ChainHash in QueryChannelRange messages
If the provided ChainHash in a QueryChannelRange message does not match
that of our current chain, then we should send a blank response, rather
than reply with channels for the wrong chain.
2020-01-17 11:51:09 -08:00
Wilmer Paulino
1bacdfb41e
discovery: interpret block range from ReplyChannelRange messages
We move from our legacy way of interpreting ReplyChannelRange messages
which was incorrect. Previously, we'd rely on the Complete field of the
ReplyChannelRange message to determine when our peer had sent all of
their replies. Now, we properly adhere to the specification by
interpreting the block ranges of these messages as intended.

Due to the large number of nodes deployed with the previous method, we
still maintain and detect when we are communicating with them, such that
we are still able to sync with them for backwards compatibility.
2020-01-06 14:03:13 -08:00
Wilmer Paulino
d688e13d35
discovery: remove unnecessary test check
It's not possible to send another reply once all replies have been sent
without another request. The purpose of the check is also done within
another test, TestGossipSyncerReplyChanRangeQueryNoNewChans, so it can
be removed from here.
2020-01-06 14:02:31 -08:00
Wilmer Paulino
c7c0853531
discovery: cover requested range in ReplyChannelRange messages
In order to properly adhere to the spec, when handling a
QueryChannelRange message, we must reply with a series of
ReplyChannelRange messages, that when consumed together cover the
entirety of the block range requested.
2020-01-06 14:00:15 -08:00
Wilmer Paulino
1f781ea431
discovery: use inclusive range in FilterChannelRange
FilterChannelRange takes an inclusive range, so it was possible for us
to return channels for an additional block that was not requested.
2020-01-06 14:00:14 -08:00
Johan T. Halseth
c04ef68cc3
Merge pull request #3826 from arik-so/wrong_chain_error_fix
fix order in wrong chain error message
2019-12-12 09:46:31 +01:00
Arik Sosman
e83df875ad
fix wrong chain error message 2019-12-11 17:43:24 -08:00
Olaoluwa Osuntokun
6a9b96122d
discovery: properly set FirstBlockHeight and NumBlocks in responses
In this commit we fix in a bug in `lnd` that could cause other
implementations which implement a strict version of the spec to
disconnect when trying to sync their channel graph using the gossip
query feature. Before this commit, we would embed the request to a
`QueryChannelRange` in the response, causing some clients to reject the
response as the `FirstBlockHeight` and `NumBlocks` field would be
identical for each chunk of the response.

In order to remedy this, we now properly set these two fields with each
returned chunk. Note that even after this commit, we keep our existing
behavior surrounding the `Complete` field as is. Otherwise, current
`lnd` clients which rely on this field (rather than the two
aforementioned fields) wouldn't be able to properly detect when a set of
responses to their query was "complete".

Partially fixes #3728.
2019-12-10 17:05:58 -08:00
Conner Fromknecht
5e27b5022c
multi: remove LocalFeatures and GlobalFeatures 2019-11-08 05:32:00 -08:00
Conner Fromknecht
16318c5a41
multi: merge local+global features from remote peer 2019-11-08 05:31:47 -08:00
cryptagoras
0ad6c4748f
discovery/gossiper: fix minor typo
It was missing a space
"addingto waiting batch" -> "adding to waiting batch"
2019-10-15 14:16:03 +03:00
Olaoluwa Osuntokun
3f8526a0ca
peer+lnpeer: add new methods to expose local+global features for lnpeer interface 2019-09-25 18:26:01 -07:00
Wilmer Paulino
04a7cda3d5
Merge pull request #3534 from alrs/discovery-test-improvements
discovery: Goroutine Test Fixes and Linting
2019-09-25 16:12:30 -07:00
Lars Lehtonen
0cae1e69ab discovery: error string lint fixes
discovery: lint fix to remove append loop
2019-09-25 18:42:38 +00:00
Lars Lehtonen
58c23074d1 discovery: use error channels with test goroutines 2019-09-25 18:41:42 +00:00
Joost Jager
c80feeb4b3
routing+discovery: extract local channel manager
The policy update logic that resided part in the gossiper and
part in the rpc server is extracted into its own object.

This prepares for additional validation logic to be added for policy
updates that would otherwise make the gossiper heavier.

It is also a small first step towards separation of our own channel data
from the rest of the graph.
2019-09-23 13:07:08 +02:00
Joost Jager
4b2eb9cb81
discovery: push max htlc migration further up the call tree
As a preparation for making the gossiper less responsible for validating
and supplementing local channel policy updates, this commits moves the
on-the-fly max htlc migration up the call tree. The plan for a follow up
commit is to move it out of the gossiper completely for local channel
updates, so that we don't need to return a list of final applied policies
anymore.
2019-09-23 13:07:06 +02:00
Joost Jager
339ff357d1
channeldb: invalidate channel signature cache on update 2019-09-23 13:07:04 +02:00
Joost Jager
5090bb27ad
discovery: remove redundant signature setting
The signature is retrieved, not used and overwritten with a
new signature.
2019-09-23 13:07:02 +02:00
Conner Fromknecht
1d41d4d666
multi: move WaitPredicate, WaitNoError, WaitInvariant to lntest/wait 2019-09-19 12:46:29 -07:00
Johan T. Halseth
92123c603d
gossiper: retransmit self NodeAnnouncement 2019-09-16 10:54:42 +02:00
Johan T. Halseth
24004fcb37
gossiper+server: define SelfNodeAnnouncement 2019-09-16 10:54:42 +02:00
Johan T. Halseth
e36d15582c
discovery/gossiper test: add TestRetransmit
This commit adds a test that ensures outdated announcements are
retransmitted when the RetransmitTicker ticks.
2019-09-16 10:54:38 +02:00
Johan T. Halseth
70d63abe9f
discovery/test: set global test timestamp 2019-09-16 10:23:01 +02:00
Johan T. Halseth
8b9fd039ec
discovery/gossiper test: remove mockGraphSource.SelfEdges 2019-09-16 10:23:01 +02:00