Commit Graph

37 Commits

Author SHA1 Message Date
Wilmer Paulino
1bacdfb41e
discovery: interpret block range from ReplyChannelRange messages
We move from our legacy way of interpreting ReplyChannelRange messages
which was incorrect. Previously, we'd rely on the Complete field of the
ReplyChannelRange message to determine when our peer had sent all of
their replies. Now, we properly adhere to the specification by
interpreting the block ranges of these messages as intended.

Due to the large number of nodes deployed with the previous method, we
still maintain and detect when we are communicating with them, such that
we are still able to sync with them for backwards compatibility.
2020-01-06 14:03:13 -08:00
Wilmer Paulino
c7c0853531
discovery: cover requested range in ReplyChannelRange messages
In order to properly adhere to the spec, when handling a
QueryChannelRange message, we must reply with a series of
ReplyChannelRange messages, that when consumed together cover the
entirety of the block range requested.
2020-01-06 14:00:15 -08:00
Wilmer Paulino
1f781ea431
discovery: use inclusive range in FilterChannelRange
FilterChannelRange takes an inclusive range, so it was possible for us
to return channels for an additional block that was not requested.
2020-01-06 14:00:14 -08:00
Johan T. Halseth
c04ef68cc3
Merge pull request #3826 from arik-so/wrong_chain_error_fix
fix order in wrong chain error message
2019-12-12 09:46:31 +01:00
Arik Sosman
e83df875ad
fix wrong chain error message 2019-12-11 17:43:24 -08:00
Olaoluwa Osuntokun
6a9b96122d
discovery: properly set FirstBlockHeight and NumBlocks in responses
In this commit we fix in a bug in `lnd` that could cause other
implementations which implement a strict version of the spec to
disconnect when trying to sync their channel graph using the gossip
query feature. Before this commit, we would embed the request to a
`QueryChannelRange` in the response, causing some clients to reject the
response as the `FirstBlockHeight` and `NumBlocks` field would be
identical for each chunk of the response.

In order to remedy this, we now properly set these two fields with each
returned chunk. Note that even after this commit, we keep our existing
behavior surrounding the `Complete` field as is. Otherwise, current
`lnd` clients which rely on this field (rather than the two
aforementioned fields) wouldn't be able to properly detect when a set of
responses to their query was "complete".

Partially fixes #3728.
2019-12-10 17:05:58 -08:00
Joost Jager
3d7de2ad39
multi: remove dead code 2019-09-10 17:21:59 +02:00
Conner Fromknecht
35a2de23a3
discovery/syncer: add flag to prevent historical gossip filter dump 2019-07-30 17:25:15 -07:00
Johan T. Halseth
c4415f0400
Merge pull request #3044 from cfromknecht/spelling-fixes
multi: fix spelling mistakes
2019-05-07 08:50:36 +02:00
Conner Fromknecht
17b2140cb5
multi: fix spelling mistakes 2019-05-04 15:35:37 -07:00
Conner Fromknecht
bf4543e2bd
discovery/syncer: make gossip sends synchronous
This commit makes all replies in the gossip syncer synchronous, meaning
that they will wait for each message to be successfully written to the
remote peer before attempting to send the next. This helps throttle
messages the remote peer has requested, preventing unintended
disconnects when the remote peer is slow to process messages. This
changes also helps out congestion in the peer by forcing the syncer to
buffer the messages instead of dumping them into the peer's queue.
2019-04-26 20:05:10 -07:00
Conner Fromknecht
23d10336c2
discovery/syncer: separate query + reply handlers
This commit creates a distinct replyHandler, completely isolating the
requesting state machine from the processing of queries from the remote
peer. Before the two were interlaced, and the syncer could only reply to
messages in certain states. Now the two will be complete separated,
which is preliminary step to make the replies synchronous (as otherwise
we would be blocking our own requesting state machine).

With this changes, the channelGraphSyncer of each peer will drive the
replyHanlder of the other. The two can now operate independently, or
even spun up conditionally depending on advertised support for gossip
queries, as shown below:

          A                                 B
 channelGraphSyncer ---control-msg--->
                                        replyHandler
 channelGraphSyncer <--control-msg----
           gossiper <--gossip-msgs----

                    <--control-msg---- channelGraphSyncer
       replyHandler
                    ---control-msg---> channelGraphSyncer
                    ---gossip-msgs---> gossiper
2019-04-26 20:03:14 -07:00
Wilmer Paulino
227e492ccf
discovery: make historicalSync transition synchronous
We do this to ensure that the state transition from chansSynced to
syncingChans has occurred by the time we return back to the caller.
2019-04-24 13:20:18 -07:00
Wilmer Paulino
9a6e8ecb9e
discovery: remove channel synchronization from ActiveSync GossipSyncers
In this commit, we remove the ability for ActiveSync GossipSyncers to
synchronize our graph with our remote peers. This serves as a starting
point towards allowing the daemon to only synchronize our graph through
historical syncs, which will be routinely done by the SyncManager.
2019-04-24 13:19:06 -07:00
Conner Fromknecht
a4b4fe666a
discovery: make batch size distinct from chunk size, reduce to 500
This commit reduces the number of channels a syncer will request from
the remote node in a single QueryShortChanIDs message. The current size
is derived from the chunkSize, which is meant to signal the maximum
number of short chan ids that can fit in a single ReplyChannelRange
message. For EncodingSortedPlain, this number is 8000, and we use the
same number to dictate the size of the batch from the remote peer.

We modify this by introducing a separately configurable batchSize, so
that both can be tuned independently. The value is chosen to reduce the
amount of buffering the remote party will perform, only requiring them
queue 500 responses, as opposed to 8000. In turn, this reduces larges
spikes in allocation on the remote node at the expense of a few extra
round trips for the control messages. However, will be negligible since
the control messages are much smaller than the messages being returned.
2019-04-06 15:27:26 -07:00
Wilmer Paulino
8b6a9bb5d3
discovery: make timestamp range check inclusive within FilterGossipMsgs
As required by the spec:

> SHOULD send all gossip messages whose timestamp is greater or equal to
first_timestamp, and less than first_timestamp plus timestamp_range.
2019-04-03 15:44:46 -07:00
Wilmer Paulino
70be812747
discovery+server: use new gossiper's SyncManager subsystem 2019-04-03 15:44:43 -07:00
Wilmer Paulino
e075817e44
discovery: introduce GossipSyncer signal delivery of chansSynced state
In this commit, we introduce another feature to the GossipSyncer in
which it can deliver a signal to an external caller once it reaches its
terminal chansSynced state. This is yet to be used, but will serve
useful with a round-robin sync mechanism, where we wait for to finish
syncing with a specific peer before moving on to the next.
2019-04-03 15:08:31 -07:00
Wilmer Paulino
042241dc48
discovery: allow gossip syncer to perform historical syncs
In this commit, we introduce the ability for gossip syncers to perform
historical syncs. This allows us to reconcile any channels we're missing
that the remote peer has starting from the genesis block of the chain.
This commit serves as a prerequisite to the SyncManager, introduced in a
later commit, where we'll be able to make spot checks by performing
historical syncs with peers to ensure we have as much of the graph as
possible.
2019-04-03 15:08:30 -07:00
Wilmer Paulino
ca4fbd598c
discovery: introduce GossipSyncer sync transitions
In this commit, we introduce the ability for GossipSyncer's to
transition their sync type. This allows us to be more flexible with our
gossip syncers, as we can now prevent them from receiving new graph
updates at any time. It's now possible to transition between the
different sync types, as long as the GossipSyncer has reached its
terminal chansSynced sync state. Certain transitions require some
additional wire messages to be sent, like in the case of an ActiveSync
GossipSyncer transitioning to a PassiveSync type.
2019-04-03 15:08:29 -07:00
Wilmer Paulino
acc42c1b68
discovery: set GossipSyncer update horizon to current time
With the introduction of the gossip sync manager in a later commit,
retrieving the backlog of updates within the last hour is no longer
necessary as we'll be forcing full syncs periodically.
2019-04-03 15:08:28 -07:00
Wilmer Paulino
8d7c0a9899
discovery: replace GossipSyncer syncChanUpdates flag with SyncerType
In this commit, we introduce a new type: SyncerType. This type denotes
the type of sync a GossipSyncer is currently under. We only introduce
the two possible entry states, ActiveSync and PassiveSync. An ActiveSync
GossipSyncer will exchange channels with the remote peer and receive new
graph updates from them, while a PassiveSync GossipSyncer will not and
will only response to the remote peer's queries.

This commit does not modify the behavior and is only meant to be a
refactor.
2019-04-03 15:08:27 -07:00
Wilmer Paulino
7e92b9a4e2
discovery: export gossipSyncer 2019-04-03 15:08:26 -07:00
Wilmer Paulino
d954cfc4ba
discovery: include peerPub in gossipSyncerCfg 2019-04-03 15:08:25 -07:00
Conner Fromknecht
5fc0a192ad
discovery/syncer: move CGTimeSerires to chan_series 2018-11-30 15:33:34 -08:00
Conner Fromknecht
ef309f43d2
discovery: pass peer quit signal to ProcessQueryMsg
This commit passes the peer's quit signal to the
gossipSyncer when attempt to hand off gossip query
messages. This allows a rate-limited peer's read
handler to break out immediately, which would
otherwise remain stuck until the rate-limited
gossip syncer pulled the message.
2018-11-01 17:35:55 -07:00
Olaoluwa Osuntokun
c5979c7d80
discovery: fix logging for gossip syncer rate limiting 2018-09-13 15:55:44 -07:00
Conner Fromknecht
a56e7122d6
discovery/syncer: use rate limiter for gossip queries
This commit replaces the simplistic rate limiting
technique added in 557cb6e2, to use the
golang.org/x/time's rate limiter. This has the
benefit of performing traffic shaping to meet a
target maximum rate, and yet tolerate bursts. Bursts
are expected during initial sync, though should become
more rare afterwards. Performing traffic shaping with
this mechanism should improve the ability of the gossip
syncer to detect sustained bursts from the remote peer,
and penalize them appropriately.

This commit also modifies the default parameters to
accept bursts of 10 queries, with a target rate of 1
reply every 5 seconds.
2018-09-11 00:12:35 -07:00
Conner Fromknecht
557cb6e253
discovery/syncer: delay replies after initial sync to prevent DOS 2018-09-05 22:08:16 -07:00
Wilmer Paulino
c61b037628 discovery/syncer: store chunk size within gossiperSyncerCfg
In this commit, we allow the gossiper syncer to store the chunk size for
its respective encoding type. We do this to prevent a race condition
that would arise within the unit tests by modifying the values of the
encodingTypeToChunkSize map to allow for easier testing.
2018-07-30 13:49:36 -07:00
Olaoluwa Osuntokun
6f60f139f4 multi: switch over import paths from roasbeef/* to btcsuite/* 2018-07-13 17:05:39 -07:00
Olaoluwa Osuntokun
f2273d9503
discovery: log failed attempts in replyShortChanIDs 2018-06-13 16:44:59 -07:00
Olaoluwa Osuntokun
e6d46f681b
discovery: if gossipSyncer is shutting down, don't filter messages 2018-06-04 17:18:54 -07:00
Olaoluwa Osuntokun
a9340d22c4 discovery: ensure stopping/starting a gossipSyncer is idempotent 2018-06-04 17:18:07 -07:00
Olaoluwa Osuntokun
994d9cf7e4
discovery: attempt to request the full chan ann for stray chan updates
In this commit, we extend the AuthenticatedGossiper to take advantage of
the new query features in the case that it gets a channel update w/o
first receiving the full channel announcement. If this happens, we'll
attempt to find a syncer that's fully synced, and request the channel
announcement from it.
2018-05-31 16:31:01 -07:00
Olaoluwa Osuntokun
6a6700e629
discovery: add new SyncState method to gossipSyncer
This new method allows outside callers to sample the current state of
the gossipSyncer in a concurrent-safe manner. In order to achieve this,
we now only modify the g.state variable atomically.
2018-05-31 16:31:00 -07:00
Olaoluwa Osuntokun
5789ef7c10
discovery: add new gossipSyncer struct to manage sync state for each peer
In this commit, introduce a new struct, the gossipSyncer. The role of
this struct is to encapsulate the state machine required to implement
the new gossip query range feature recently added to the spec. With this
change, each peer that knows of this new feature will have a new
goroutine that will be managed by the gossiper.

Once created and started, the gossipSyncer will start to progress
through each possible state, finally ending at the chansSynced stage. In
this stage, it has synchronized state with the remote peer, and is
simply awaiting any new messages from the gossiper to send directly to
the peer. Each message will only be sent if the remote peer actually has
a set update horizon, and the message isn't before or after that
horizon.

A set of unit tests has been added to ensure that two state machines
properly terminate and synchronize channel state.
2018-05-31 16:30:53 -07:00