Commit Graph

109 Commits

Author SHA1 Message Date
Olaoluwa Osuntokun
55dfc17c05
discovery: de-duplicate node announcements with legacy chan graph sync
Fixes #1072.
2018-05-31 16:30:58 -07:00
Olaoluwa Osuntokun
0016b3f458
discovery+lnd: create new chanSeries impl of the ChannelGraphTimeSeries interface
In this commit, we create a new concrete implementation for the new
discovery.ChannelGraphTimeSeries interface. We also export the
createChannelAnnouncement method to allow the chanSeries struct to
re-use the existing code for creating wire messages from the database
structs.
2018-05-31 16:30:55 -07:00
Olaoluwa Osuntokun
45bd544f11
discovery: update AuthenticatedGossiper to be aware of new gossipSyncers
In this commit, we update the logic in the AuthenticatedGossiper to
ensure that can properly create, manage, and dispatch messages to any
gossipSyncer instances created by the server.

With this set of changes, the gossip now has complete knowledge of the
current set of peers we're conneted to that support the new range
queries. Upon initial connect, InitSyncState will be called by the
server if the new peer understands the set of gossip queries. This will
then create a new spot in the peerSyncers map for the new syncer. For
each new gossip query message, we'll then attempt to dispatch the
message directly to the gossip syncer. When the peer has disconnected,
we then expect the server to call the PruneSyncState method which will
allow us to free up the resources.

Finally, when we go to broadcast messages, we'll send the messages
directly to the peers that have gossipSyncer instances active, so they
can properly be filtered out. For those that don't we'll broadcast
directly, ensuring we skip *all* peers that have an active gossip
syncer.
2018-05-31 16:30:53 -07:00
Olaoluwa Osuntokun
5789ef7c10
discovery: add new gossipSyncer struct to manage sync state for each peer
In this commit, introduce a new struct, the gossipSyncer. The role of
this struct is to encapsulate the state machine required to implement
the new gossip query range feature recently added to the spec. With this
change, each peer that knows of this new feature will have a new
goroutine that will be managed by the gossiper.

Once created and started, the gossipSyncer will start to progress
through each possible state, finally ending at the chansSynced stage. In
this stage, it has synchronized state with the remote peer, and is
simply awaiting any new messages from the gossiper to send directly to
the peer. Each message will only be sent if the remote peer actually has
a set update horizon, and the message isn't before or after that
horizon.

A set of unit tests has been added to ensure that two state machines
properly terminate and synchronize channel state.
2018-05-31 16:30:53 -07:00
Conner Fromknecht
3854c1ed68
discovery/gossiper: exit early on validation barrier quit 2018-05-08 16:32:08 -07:00
practicalswift
663c396235 multi: fix a-vs-an typos 2018-04-17 19:02:04 -07:00
Jim Posen
4b2cb68fe6 discovery: Fix formatting issue in log line. 2018-04-09 20:49:23 -07:00
Olaoluwa Osuntokun
3fa2e08665
test: update testUpdateChannelPolicy to ensure Bob's link uses the proper policies
In this commit, we update the testUpdateChannelPolicy to exercise the
recent set of changes within the switch. If one applies this test to a
fresh branch (without those new changes) it should fail. This is due to
the fact that before, Bob would attempt to apply the constraints of the
incoming link (which we updated) instead of the outgoing link. With the
recent set of changes, the test now properly passes.
2018-04-06 14:52:01 -07:00
Olaoluwa Osuntokun
9d4cea93f0
discovery: fix deadlock in processChanPolicyUpdate
In this commit, we fix an existing deadlock in the
processChanPolicyUpdate method. Before this commit, within
processChanPolicyUpdate, we would directly call updateChannel *within*
the ForEachChannel closure. This would at times result in a deadlock, as
updateChannel will itself attempt to create a write transaction in order
to persist the newly updated channel.

We fix this deadlock by simply performing another loop once we know the
set of channels that we wish to update. This second loop will actually
update the channels on disk.
2018-04-06 14:51:57 -07:00
Olaoluwa Osuntokun
2326d4afeb
discovery: bump rejected edges up to the trace logging level 2018-03-23 15:50:56 -07:00
Matt Drollette
1b7b35a8bc discovery: format log message with Errorf 2018-03-16 16:25:18 -07:00
Olaoluwa Osuntokun
800eea931f
build+multi: switch from bolt to bbolt
In this commit, we switch from boltbd/bolt to coreos/bbolt as the
former is no longer being actively maintained.
2018-03-10 19:01:13 -08:00
Olaoluwa Osuntokun
e7e4cdcf49
discovery: avoid always validating ECDSA sigs by asking router if item is fresh
In this commit, we reduce the amount of unnecessary work that the
gossiper can carry out. When CPU profiling some nodes, I noticed that
we’d spend a lot of time validating the signatures for an announcement,
only to realize that the router already had it.

To remedy this, we’ll use the new methods added to the channel router
in order to avoid unnecessarily validating an announcement that is
actually stale. This should reduce memory usage (since it uses big
int’s under the scenes), and also idle CPU usage.
2018-03-06 13:34:51 -05:00
practicalswift
b8e1351cf3 multi: fix some recently introduced typos 2018-02-18 15:27:29 -08:00
Olaoluwa Osuntokun
bf05e47780
discovery: add additional gossiper level reject cache
In this commit, we’ll add a new reject cache to ensure that we don’t
attempt to re-process any announcements already rejected by the
ChannelRouter.
2018-02-06 20:14:35 -08:00
Olaoluwa Osuntokun
cd9d2d7e6f
discovery: update graph API usage to match recent API changes 2018-02-06 20:14:32 -08:00
Daniel McNally
8543497dcc multi: fixing it's/its typos in comments 2018-02-06 19:13:07 -08:00
practicalswift
a93736d21e multi: comprehensive typo fixes across all packages 2018-02-06 19:11:11 -08:00
Olaoluwa Osuntokun
0a4de859a2
discovery+routing: reduce number of active validation barrier jobs
In order to reduce high CPU utilization during the initial network view
sync, we slash down the total number of active in-flight jobs that can
be launched.
2018-01-28 14:55:32 -08:00
Johan T. Halseth
a55c74f80c
discovery/gossiper: restrict database access per channel ID
This commit uses the multimutex.Mutex to esure database
state stays consistent when handling an announcement, by
restricting access to one goroutine per channel ID.

This fixes a bug where the goroutine would read the
database, make some decisions based on what was read,
then write its data to the database, but the read data
would be outdated at this point. For instance, an
AuthProof could have been added between reading the
database and when the decision whether to announce
the channel is made, making it not announce it.
Similarly, when receiving the AuthProof, the edge
policy could be added between reading the edge state
and adding the proof to the database, also resulting
in the edge not being announced.
2018-01-24 10:26:40 +01:00
Johan T. Halseth
3e01529de4
discovery/gossiper: don't reset batch after broadcast
This commit fixes a bug that could cause annoucements
to get lost, and resultet in flaky integration tests.
After a set of announcements was broadcastet, we would
reset (clear) the announcement batch, making any
annoucement that was added between the call to Emit()
and Reset() to be deleted, without ever being broadcast.

We can just remove the Reset() call, as the batch will
actually be reset within the call to Emit(), making
the previous call only delete those messages we hadn't
sent yet.
2018-01-24 10:26:39 +01:00
Olaoluwa Osuntokun
69e6ec9954
peer+funding: remove unneeded channel handoff code with the breach arbiter
We no longer need to hand off new channels that come online as the
chainWatcher will be persistent, and always have an active signal for
the entire lifetime of the channel.
2018-01-22 19:19:50 -08:00
Olaoluwa Osuntokun
8807d1d752
fundingmgr: add new function closure to send new channels to the ChainArbitrator 2018-01-22 19:19:38 -08:00
Olaoluwa Osuntokun
ebd97b83f3
discovery: ensure we only send node announcements that have active channels 2018-01-16 17:18:21 -08:00
Johan T. Halseth
483abbee5b
discovery: make sure ChannelUpdates always have incremented timestamp
This commit ensures that we always increment the timestamp of
ChannelUpdates we send telling the network about changes to
our channel policy. We do this because it could happen
(especially during tests) that we issued an update, but the
ChannelUpdate would have the same timestamp as our last
ChannelUpdate, and would be ignored by the network.
2018-01-12 22:57:03 +01:00
Johan T. Halseth
8370fa2cde
discovery/gossiper: apply TimeLockDelta to edge when processing policy update
Also rename various instances of "FeeUpdate" to "PolicyUpdate"
2018-01-12 22:56:32 +01:00
Johan T. Halseth
3f0a525331
discovery test: update gossiper test to account for timestamp deduping 2018-01-09 13:09:28 +01:00
Johan T. Halseth
ea95f37277
discovery: keep newest messages when deduping in gossiper
This commit makes the gossiper aware of the timestamps
of ChannelUpdates and NodeAnnouncements, such that it
only keeps the newest message when deduping. Earlier
it would always keep the last received message, which
in some cases could be outdated.
2018-01-09 13:09:25 +01:00
Johan T. Halseth
2ddd139084 discovery: check proof for nilness before creating chanAnn
This commit makes sure we are not attempting to create a
channel announcement with a nil ChannelAuthProof, as that
could cause a crash at startup whe the gossiper would
attempt to reprocess an edge coming from the fundingmanager.

It also makes sure we check the correct error returned from
processRejectedEdge.
2018-01-02 19:51:38 +01:00
Olaoluwa Osuntokun
38715ca768
discovery: refactor re-processing rejected announcements into new method 2017-12-26 16:28:23 +01:00
Olaoluwa Osuntokun
68dabe3594
discovery: update usage of d.cfg.Broadcast to reflect recent API change
When broadcasting a set of messages, we must now call broadcast for
each (msg, …senders) tuple, rather than just calling it once for the
entire batch.
2017-12-26 16:26:34 +01:00
Olaoluwa Osuntokun
f45502f890
discovery: update message processing to emit networkMsg rather than lnwire.Message
In this commit, we make an incremental step towards page of the new
feature of deDupedAnnoucnements to return the set of senders for each
message. All methods the process new channel announcements, will now
return an instance of networkMsg rather than lnwire.Message. This will
allow passing the returned announcement directly into
deDupedAnnoucnements.AddMsg().
2017-12-26 16:23:12 +01:00
Olaoluwa Osuntokun
f1b40e0b4d
discovery: modify deDupedAnnouncements to keep track of senders for each msg
In this commit, we modify the deDupedAnnouncements struct slightly. The
element of this struct will now keep track of the set of senders that
sent a particular message. Each time a message is added, we’ll replace
the new message with the old (as normal), but we’ll also add the new
sender to the set of known senders.

With this new feature, we’ll be able to avoid re-sending a message to
the peer that sent it to us in the first place.
2017-12-26 16:19:03 +01:00
Johan T. Halseth
e3ae204fcb discovery: make gossiper able to resend failed AnnounceSignatures
This commit makes the gossiper track the state of a local
AnnounceSignature message, such that it can retry sending
it to the remote peer if needed. It will also persist this
state in the WaitingProofStore, such that it can resume
from this state at startup.
2017-12-19 13:01:59 -06:00
Johan T. Halseth
db829cd0c5 discovery: reprocess premature ChannelUpdates
This commit makes the gossiper store received ChannelUpdates
that is not for any known channel in a map, such that they
can be reprocessed when the ChannelAnnouncement arrives.

This is done to handle the case where we receive a ChannelUpdate
from our channel counterparty before we have been able to process
our own local ChannelAnnouncement.
2017-12-17 18:35:34 -08:00
Johan T. Halseth
10b838e9b4 gossiper: private channel cleanup + comments
This commit adds some comments and does some cleanup
of the logic that makes sure non-public channels
(channels with no AuthProof) are not broadcasted
to the network.
2017-12-17 18:35:34 -08:00
nsa
dab7f31e44 gossiper: Only send channel updates for edges with AuthProof. 2017-12-17 18:35:34 -08:00
Olaoluwa Osuntokun
37c54153fa
discovery: properly set node's color when syncing graph state
This commit fixes an existing bug wherein we would blank out a node’s
color instead of properly setting the field when syncing graph state
with another node This would cause the node to reject the node
announcement and we would generate an we would invalidate the signature
of the node announcement. We fix this simply by properly setting the
node announcement.
2017-12-02 18:33:21 -08:00
Olaoluwa Osuntokun
ee7c3ff01c
discovery: properly set node's color field when adding NodeAnnouncement to router
In this commit, we fix an existing bug when processing new node
announcement. Before this commit, we wouldn’t also copy over the color
field of a node’s announcement. As a result, when went to synchronize
our graph state with that of a connecting peer, we would generate an
invalid node announcement. We fix this by properly setting the color
field of a node.
2017-12-02 18:28:41 -08:00
Olaoluwa Osuntokun
3802cb90df
discovery: publicly export announcement validation related functions 2017-12-01 19:31:54 -08:00
Olaoluwa Osuntokun
12a0488bc6
discovery: use lnwire.ChanUpdateFlag masks when examinig ChannelUpdate msgs
In this commit, we now properly examine the Flag field within the
ChannelUpdate message as a bitfield. Before this commit we would
manually check the flags for zero or one. This was incorrect as a their
bit has now been defined. To properly dispatch the messages, we’ll now
treat it properly as a bitmask.
2017-11-30 22:37:21 -08:00
Olaoluwa Osuntokun
c0b8c29382
discovery: continue in loop after processing announcement signatures 2017-11-29 18:35:19 -08:00
Olaoluwa Osuntokun
4ab16b5c91
discovery: process AnnouncementSignatures msgs serially 2017-11-29 17:53:21 -08:00
Olaoluwa Osuntokun
3067d05ae8
discovery: validate incoming announcements in parallel 2017-11-29 16:45:14 -08:00
Olaoluwa Osuntokun
2dcd2b8a6d
discovery: add a mutex in order to make deDupedAnnouncements thread-safe 2017-11-29 16:21:16 -08:00
Laura Cressman
ed6ad22e85 config+server+networktest: make trickleDelay configurable
Add option to set trickleDelay for AuthenticatedGossiper in
command line, with default value of 300 milliseconds. Pass this
value to newServer, which uses it when creating a new instance of
AuthenticatedGossiper. Also set this value to 300 milliseconds when
creating nodes in integration tests.
2017-11-15 16:50:19 -08:00
Laura Cressman
39d38da732 discovery: create deDupedAnnouncements struct in gosspier.go
For Part 1 of Issue #275. Create isolated private struct in
networkHandler goroutine that will de-duplicate
announcements added to the batch. The struct contains maps
for each of channel announcements, channel updates, and
node announcements to keep track of unique announcements.

The struct has a Reset method to reset stored announcements, an
AddMsg(lnwire.Message) method to add a new message to the current
batch, and a Batch method to return the set of de-duplicated
announcements.

Also fix a few minor typos.
2017-11-15 16:50:19 -08:00
Olaoluwa Osuntokun
8c181b6f4e
discovery: return an error if ChannelUpdate has unknown flags
In this commit, we fix an existing bug that could result in a panic if
we received a ChannelUpdate message with an unknown set of flags. If
the flag wasn’t set to zero or one, then the pubKey parameter would be
still nil when we attempted to validate it, causing an error to occur.

We remedy this by instead returning an error if the flags are unknown.
In a future commit, we will properly handle the set of flags that
indicates the channel should be disabled.
2017-11-11 14:59:16 -08:00
Conner Fromknecht
3b2e4caa93 discovery/gossiper: isolate sync with node from query
This commit refactors the SynchronizeNode logic such that
  it can be called without interacting with the gossiper's
  main execution loop. This method does not require access
  to any of the gossiper's internal state, making the change
  fairly straightforward. The primary motivation behind
  this change is to minimize the possibility of introducing
  deadlock scenarios between the gossiper and server.
2017-10-27 15:48:37 -07:00
Jim Posen
9fd77a6e40 multi: Update lnd to use new feature vector API. 2017-10-17 22:47:20 -07:00
Olaoluwa Osuntokun
03f9479eb4
discovery: only interpret remote announcements as pre-mature
This commit fixes a slight bug in the announcement processing logic
within the AuthenticatedGossiper. Before this commit, it was possible
for us to ignore one of our now announcements due to it being
pre-mature, rendering (atm) the channel unusable by the ChannelRouter
itself. To fix this, we know only check for a pre mature announcement
iff the message is coming from a remote node.
2017-10-17 19:44:13 -07:00
Olaoluwa Osuntokun
490e0d0f46
discovery: broadcast OG chan ann in retransmitStaleChannels 2017-10-04 20:46:12 -07:00
Olaoluwa Osuntokun
0d28bb14cb
discovery: also return latest ChannelAnnouncement from updateChannel
This commit adds an additional return value to the updateChannel
method. We also now return the original ChannelAnnouncement, this can
be useful as it let’s the caller ensure that the channel announcement
will be broadcast along side the new channel update.
2017-10-04 20:46:10 -07:00
Olaoluwa Osuntokun
4be2e95a78
discovery: fix boltdb deadlock bug in ann retrans, retrans at start
This commit does two things. First we fix a deadlock bug within boltdb
that could arise when the channel router attempted to open a
transaction, while the retransmission loop was attempting to send a
message to the channel router (circular wait). Second, we’ve refactored
out the retransmission into its own function. This allows us to kick
off retransmission as soon as the AuthenticatedGossiper is created.
2017-10-04 20:46:10 -07:00
Olaoluwa Osuntokun
81cd954cfc
discovery: don't prune *our* channels during retransmission tick
This commit modifies the recently modified logic for self-channel
retransmission to exclude pruning *our* channels which haven’t been
updated since the broadcastInterval. Instead, we only re-broadcast
channels of ours that haven’t been updated in 24 hours.
2017-10-04 20:46:08 -07:00
Brandon
3907ae65c2 routing+discovery: implement 2-week network view pruning 2017-10-04 20:40:21 -07:00
Olaoluwa Osuntokun
c529597ff9
discovery: add more detail to error logging in processNetworkAnnouncement 2017-09-12 17:12:22 +02:00
Olaoluwa Osuntokun
0b4ff2aabb
discovery: ignore any incoming channel ann's for an unknown chain
This commit fixes an existing bug in the announcement processing of the
authenticated gossiper. We will now only examine an incoming
announcement for validity if it targets the chain that we’re currently
active on.
2017-09-03 16:41:09 -07:00
Olaoluwa Osuntokun
1e513875e5
discovery: rename service.go to gossiper.go 2017-09-03 16:21:10 -07:00