Commit Graph

350 Commits

Author SHA1 Message Date
Olaoluwa Osuntokun
355b6a260f
Merge pull request #4964 from halseth/gossip-local-no-batch
channeldb+routing+gossiper: add local updates to graph immediately
2021-02-10 18:00:47 -08:00
Conner Fromknecht
1ee5eb97d5
Merge pull request #5006 from wpaulino/new-rate-limit-chan-updates
discovery: use token bucket based rate limiting to throttle gossip
2021-02-10 17:31:41 -08:00
Conner Fromknecht
5afc6b9284
Merge pull request #4945 from cfromknecht/no-graph-sync
discovery: no graph sync
2021-02-10 17:07:24 -08:00
Wilmer Paulino
83a0d03c0b
discovery: use token bucket based rate limiting to throttle gossip
The recently added gossip throttling was shown to be too aggressive,
especially with our auto channel enable/disable signaling. We switch to
a token bucket based system instead as it's based on time, rather than a
block height which isn't constantly updated at a given rate.
2021-02-10 16:21:13 -08:00
Wilmer Paulino
bfc8523873
Revert "discovery: add new option to toggle gossip rate limiting"
This reverts commit 13a2598ded5944c1bc91ecff2c2b1b156486bda5.
2021-02-10 15:45:48 -08:00
Johan T. Halseth
c9afc93151
discovery/gossiper: add local updates to graph immediately
Since the batch interval can potentially be long, adding local updates
to the graph could be slow. This would slow down operations like adding
our own channel update and announcements during the funding process, and
updating edge policies for local channels.

Now we instead check whether the update is remote or not, and only for
remote updates use the SchedulerOption to lazily add them to the graph.
2021-02-10 23:54:03 +01:00
Johan T. Halseth
7e34132c53
routing: let graph methods take scheduler option 2021-02-10 23:54:03 +01:00
Wilmer Paulino
904003fbcb discovery: use source of ann upon confirmed channel ann batch
We do this instead of using the source of the AnnounceSignatures
message, as we filter out the source when broadcasting any
announcements, leading to the remote node not receiving our channel
update. Note that this is done more for the sake of correctness and to
address a flake within the integration tests, as channel updates are
sent directly and reliably to channel counterparts.
2021-02-10 13:22:28 -08:00
Conner Fromknecht
58e924ad1c
discovery: don't historical sync when NumActiveSyncers == 0
Currently when numgraphsyncpeers=0, lnd will still attempt to perform
an initial historical sync. We change this behavior here to forgoe
historical sync entirely when numgraphsyncpeers is zero, since the
routing table isn't being updated anyway while the node is active.

This permits a no-graph lnd mode where no syncing occurs at all.
2021-02-10 09:35:45 -08:00
Olaoluwa Osuntokun
555de44d9f Revert "Merge pull request #4895 from wpaulino/disallow-premature-chan-updates"
This reverts commit 6e6384114c890cdfd486ace5885118150940df86, reversing
changes made to 98ea4332716f953c039308c4e28cb4e55f8f89bc.
2021-02-09 19:55:45 -08:00
Conner Fromknecht
b1fee734ec
discovery/sync_manager: remove unneeded markGraphSyncing
AFAICT it's not possible to flip back from bein synced_to_chain, so we
remove the underlying call that could reflect this. The method is moved
into the test file since it's still used to test correctness of other
portions of the flow.
2021-01-29 00:19:48 -08:00
Conner Fromknecht
e42301dee2
lntest: call markGraphSynced from gossipSyncer
Rather than performing this call in the SyncManager, we give each
gossipSyncer the ability to mark the first sync completed. This permits
pinned syncers to contribute towards the rpc-level synced_to_graph
value, allowing the value to be true after the first pinned syncer or
regular syncer complets. Unlinke regular syncers, pinned syncers can
proceed in parallel possibly decreasing the waiting time if consumers
rely on this field before proceeding to load their application.
2021-01-29 00:19:48 -08:00
Conner Fromknecht
fcd5cb625a
config: expose gossip.pinned-syncers for conf
The pinned syncer set is exposed as a comma-separated list of pubkeys.
2021-01-29 00:19:47 -08:00
Conner Fromknecht
340414356d
discovery: perform initial historical sync for pinned peers 2021-01-29 00:19:47 -08:00
Conner Fromknecht
2f0d56d539
discovery: add support for PinnedSyncers
A pinned syncer is an ActiveSyncer that is configured to always remain
active for the lifetime of the connection. Pinned syncers do not count
towards the total NumActiveSyncer count, which are rotated periodically.

This features allows nodes to more tightly synchronize their routing
tables by ensuring they are always receiving gossip from distinguished
subset of peers.
2021-01-29 00:19:47 -08:00
Conner Fromknecht
9e932f2a64
discovery/sync_manager: Pause/Resume HistoricalSyncTicker
This gives each initial historical syncer an equal amount of time before
being rotated, even if some fail.
2021-01-29 00:19:47 -08:00
Conner Fromknecht
ef0cd82c1f
discovery/sync_manager: make setHistoricalSyncer closure 2021-01-29 00:19:46 -08:00
Conner Fromknecht
72fbd1283b
discovery/sync_manager: break out IsGraphSynced check 2021-01-29 00:19:46 -08:00
Conner Fromknecht
7c6aa20bd8
discovery: handle err for linter 2021-01-29 00:19:46 -08:00
Wilmer Paulino
7ef1f3f636
discovery: use source of ann upon confirmed channel ann batch
We do this instead of using the source of the AnnounceSignatures
message, as we filter out the source when broadcasting any
announcements, leading to the remote node not receiving our channel
update. Note that this is done more for the sake of correctness and to
address a flake within the integration tests, as channel updates are
sent directly and reliably to channel counterparts.
2021-01-06 13:16:44 -08:00
Wilmer Paulino
00d4e92362
discovery: prevent rebroadcast of premature channel updates
As similarly done with premature channel announcements, we'll no longer
allow premature channel updates to be rebroadcast once mature. This is
no longer necessary as channel announcements that we're not aware of are
usually broadcast to us with their accompanying channel updates.
2021-01-06 12:52:41 -08:00
Wilmer Paulino
871a6f1690
discovery: prevent rebroadcast of previously premature announcements 2020-12-08 15:18:08 -08:00
Wilmer Paulino
a4f33ae63c
discovery: adhere to proper channel chunk splitting for ReplyChannelRange 2020-12-08 15:18:07 -08:00
Wilmer Paulino
c5fc7334a4
discovery: limit NumBlocks to best known height for outgoing QueryChannelRange
This is done to ensure we don't receive replies for channels in blocks
not currently known to us, which we wouldn't be able to process.
2020-12-08 15:18:06 -08:00
Olaoluwa Osuntokun
13a2598ded
discovery: add new option to toggle gossip rate limiting
In this commit, we add a new option to toggle gossip rate limiting. This
new option can be useful in contexts that require near instant
propagation of gossip messages like integration tests.
2020-11-30 16:38:56 -08:00
Olaoluwa Osuntokun
7e298f1434
Merge pull request #3367 from cfromknecht/batched-graph-updates
Batched graph updates
2020-11-25 18:40:40 -08:00
Wilmer Paulino
791ba3eb50
discovery: rate limit incoming channel updates
This change was largely motivated by an increase in high disk usage as a
result of channel update spam. With an in memory graph, this would've
gone mostly undetected except for the increased bandwidth usage, which
this doesn't aim to solve yet. To minimize the effects to disks, we
begin to rate limit channel updates in two ways. Keep alive updates,
those which only increase their timestamps to signal liveliness, are now
limited to one per lnd's rebroadcast interval (current default of 24H).
Non keep alive updates are now limited to one per block per direction.
2020-11-25 15:38:08 -08:00
Conner Fromknecht
f8154c65c5
discovery/gossiper: increase validation barrier size to 1000
This allows for a 1000 different validation operations to proceed
concurrently. Now that we are batching operations at the db level, the
average number of outstanding requests will be higher since the commit
latency has increased. To compensate, we allow for more outstanding
requests to keep the gossiper busy while batches are constructed.
2020-11-24 16:39:47 -08:00
Conner Fromknecht
fb9218d100
discovery/gossiper: channel announcements can't be outdated 2020-11-24 16:38:14 -08:00
Andras Banki-Horvath
d89f51d1d0
multi: add reset closure to kvdb.Update
Similarly as with kvdb.View this commits adds a reset closure to the
kvdb.Update call in order to be able to reset external state if the
underlying db backend needs to retry the transaction.
2020-11-05 17:57:12 +01:00
Andras Banki-Horvath
2a358327f4
multi: add reset closure to kvdb.View
This commit adds a reset() closure to the kvdb.View function which will
be called before each retry (including the first) of the view
transaction. The reset() closure can be used to reset external state
(eg slices or maps) where the view closure puts intermediate results.
2020-11-05 17:57:12 +01:00
yyforyongyu
ef38b12fda
multi: use timeout field in dialer 2020-09-16 11:50:04 +08:00
eugene
49d8f04197 multi: migrate instances of mockSigner to the mock package
This commit moves all localized instances of mock implementations of
the Signer interface to the lntest/mock package. This allows us to
remove a lot of code and have it housed under a single interface in
many cases.
2020-08-28 15:43:51 -04:00
Conner Fromknecht
cff52f7622
Merge pull request #4352 from matheusdtech/discovery-lock-premature
discovery: correctly lock premature messages
2020-06-26 22:50:09 -07:00
Brian Mancini
28931390ff discovery: prevent endBlock overflow in replyChanRangeQuery
Modifies syncer.replyChanRangeQuery method to use the LastBlockHeight
method on the query. LastBlockHeight safely calculates the ending
block height and prevents an overflow of start_block + num_blocks.

Prior to this change, query messages that had a start_block +
num_blocks that overflows uint32_max would return zero results in the
reply message.

Tests are added to fix the bug and ensure proper start and end values
are supplied to the channel graph filter.
2020-06-18 16:48:09 -04:00
Matheus Degiovani
44f83731bc discovery: Correctly lock premature annoucements
This reworks the locking behavior of the Gossiper so that a race
condition on channel updates and block notifications doesn't cause any
loss of messages.

This fixes an issue that manifested mostly as flakes on itests during
WaitForNetworkChannelOpen calls.

The previous behavior allowed ChannelUpdates to be missed if they
happened concurrently to block notifications. The
processNetworkAnnoucement call would check for the current block height,
then lock the gossiper and add the msg to the prematureAnnoucements
list. New blocks would trigger an update to the current block height
then a lock and check of the aforementioned list.

However, specially during itests it could happen that the missing lock
before checking the height could case a race condition if the following
sequence of events happened:

- A new ChannelUpdate message was received and started processing on a
  separate goroutine
- The isPremature() call was made and verified that the ChannelUpdate
  was in fact premature
- The goroutine was scheduled out
- A new block started processing in the gossiper. It updated the block
  height, asked and was granted the lock for the gossiper and verified
  there was zero premature announcements. The lock was released.
- The goroutine processing the ChannelUpdate asked for the gossiper lock
  and was granted it. It added the ChannelUpdate in the
  prematureAnnoucements list. This can never be processed now.

The way to fix this behavior is to ensure that both isPremature checks
done inside processNetworkAnnoucement and best block updates are made
inside the same critical section (i.e. while holding the same lock) so
that they can't both check and update the prematureAnnoucements list
concurrently.
2020-06-05 15:58:33 -03:00
Matheus Degiovani
ccc8f8e48f discovery: Log new blocks
This should help debug some flaky itests.
2020-06-05 13:31:40 -03:00
Conner Fromknecht
d0d2ca403d
multi: rename ReadTx to RTx 2020-05-26 18:20:37 -07:00
Roei Erez
ae2c37e043 Ensure chain notifier is started before accessed.
The use case comes from the RPC layer that is ready before the
chain notifier which is used in the sub server.
2020-04-30 12:54:33 +03:00
Conner Fromknecht
0f94b8dc62
multi: return input.Signature from SignOutputRaw 2020-04-10 14:27:35 -07:00
Conner Fromknecht
ec784db511
multi: remove returned error from WipeChannel
The linter complains about not checking the return value from
WipeChannel in certain places. Instead of checking we simply remove the
returned error because the in-memory modifications cannot fail.
2020-04-02 17:39:29 -07:00
Conner Fromknecht
4e793497c8
Merge pull request #2669 from cfromknecht/use-netann-in-discovery
netann+discovery+server: consolidate network announcements to netann pkg
2020-03-23 13:38:06 -07:00
Conner Fromknecht
92456d063d
discovery: remove unused updateChanPolicies struct 2020-03-19 13:43:57 -07:00
Conner Fromknecht
5c2fc4a2d6
discovery/gossiper: use netann pkg for signing channel updates 2020-03-19 13:43:39 -07:00
Olaoluwa Osuntokun
ace7a78494
discovery: covert to use new kvdb abstraction 2020-03-18 19:35:07 -07:00
Conner Fromknecht
089ac647d8
discovery/chan_series: use netann.ChannelUpdateFromEdge helper 2020-03-17 16:24:25 -07:00
Conner Fromknecht
7b0d564692
discovery: move remotePubFromChanInfo to gossiper, remove utils 2020-03-17 16:24:10 -07:00
Conner Fromknecht
6a813e3433
discovery/multi: move CreateChanAnnouncement to netann 2020-03-17 16:23:54 -07:00
Conner Fromknecht
d82aacbdc5
discovery/utils: use netann.ChannelUpdateFromEdge 2020-03-17 16:23:37 -07:00
Conner Fromknecht
df44d19936
discovery/multi: move SignAnnouncement to netann 2020-03-17 16:23:01 -07:00