Commit Graph

321 Commits

Author SHA1 Message Date
Wilmer Paulino
70be812747
discovery+server: use new gossiper's SyncManager subsystem 2019-04-03 15:44:43 -07:00
Wilmer Paulino
a188657b2f
discovery: introduce gossiper SyncManager subsystem
In this commit, we introduce a new subsystem for the gossiper: the
SyncManager. This subsystem is a major overhaul on the way the daemon
performs the graph query sync state machine with peers.

Along with this subsystem, we also introduce the concept of an active
syncer. An active syncer is simply a GossipSyncer currently operating
under an ActiveSync sync type. Before this commit, all GossipSyncer's
would act as active syncers, which means that we were receiving new
graph updates from all of them. This isn't necessary, as it greatly
increases bandwidth usage as the network grows. The SyncManager changes
this by requiring a specific number of active syncers. Once we reach
this specified number, any future peers will have a GossipSyncer with a
PassiveSync sync type.

It is responsible for three main things:

1. Choosing different peers randomly to receive graph updates from to
ensure we don't only receive them from the same set of peers.

2. Choosing different peers to force a historical sync with to ensure we
have as much of the public network as possible. The first syncer
registered with the manager will also attempt a historical sync.

3. Managing an in-order queue of active syncers where the next cannot be
started until the current one has completed its state machine to ensure
they don't overlap and request the same set of channels, which
significantly reduces bandwidth usage and addresses a number of issues.
2019-04-03 15:08:32 -07:00
Wilmer Paulino
e075817e44
discovery: introduce GossipSyncer signal delivery of chansSynced state
In this commit, we introduce another feature to the GossipSyncer in
which it can deliver a signal to an external caller once it reaches its
terminal chansSynced state. This is yet to be used, but will serve
useful with a round-robin sync mechanism, where we wait for to finish
syncing with a specific peer before moving on to the next.
2019-04-03 15:08:31 -07:00
Wilmer Paulino
042241dc48
discovery: allow gossip syncer to perform historical syncs
In this commit, we introduce the ability for gossip syncers to perform
historical syncs. This allows us to reconcile any channels we're missing
that the remote peer has starting from the genesis block of the chain.
This commit serves as a prerequisite to the SyncManager, introduced in a
later commit, where we'll be able to make spot checks by performing
historical syncs with peers to ensure we have as much of the graph as
possible.
2019-04-03 15:08:30 -07:00
Wilmer Paulino
ca4fbd598c
discovery: introduce GossipSyncer sync transitions
In this commit, we introduce the ability for GossipSyncer's to
transition their sync type. This allows us to be more flexible with our
gossip syncers, as we can now prevent them from receiving new graph
updates at any time. It's now possible to transition between the
different sync types, as long as the GossipSyncer has reached its
terminal chansSynced sync state. Certain transitions require some
additional wire messages to be sent, like in the case of an ActiveSync
GossipSyncer transitioning to a PassiveSync type.
2019-04-03 15:08:29 -07:00
Wilmer Paulino
acc42c1b68
discovery: set GossipSyncer update horizon to current time
With the introduction of the gossip sync manager in a later commit,
retrieving the backlog of updates within the last hour is no longer
necessary as we'll be forcing full syncs periodically.
2019-04-03 15:08:28 -07:00
Wilmer Paulino
8d7c0a9899
discovery: replace GossipSyncer syncChanUpdates flag with SyncerType
In this commit, we introduce a new type: SyncerType. This type denotes
the type of sync a GossipSyncer is currently under. We only introduce
the two possible entry states, ActiveSync and PassiveSync. An ActiveSync
GossipSyncer will exchange channels with the remote peer and receive new
graph updates from them, while a PassiveSync GossipSyncer will not and
will only response to the remote peer's queries.

This commit does not modify the behavior and is only meant to be a
refactor.
2019-04-03 15:08:27 -07:00
Wilmer Paulino
7e92b9a4e2
discovery: export gossipSyncer 2019-04-03 15:08:26 -07:00
Wilmer Paulino
d954cfc4ba
discovery: include peerPub in gossipSyncerCfg 2019-04-03 15:08:25 -07:00
Wilmer Paulino
c72db902f0
discovery: replace waitPredicate with lntest version 2019-04-03 15:08:12 -07:00
Wilmer Paulino
0ab97957ea discovery: check if stale within isMsgStale for ChannelUpdate messages
In this commit, we address an assumption of the gossiper's recently
introduce reliable sender. The reliable sender is currently only used
for messages of unannounced channels. This makes sense as peers should
be able to retrieve messages from the network if they've previously
announced. However, within isMsgStale, we assumed that the reliable
sender would be used for every ChannelUpdate being sent, even if the
channel is already announced. Due to this, checking if the policy is
stale was unnecessary. But since this isn't the case, we should actually
be checking whether it is stale to prevent sending it later on.
2019-03-28 17:22:23 -07:00
Wilmer Paulino
93414bd27a discovery: clean up TestSignatureAnnouncementRetryAtStartup
In this commit, we remove code from
TestSignatureAnnouncementRetryAtStartup that is not crucial to the
assumptions the test is exercising.
2019-03-28 17:22:23 -07:00
Wilmer Paulino
95ed11b01b discovery: properly store and retrieve edge policies within mockGraphSource
In this commit, we address an issue with our router mock in which it was
not properly storing and retrieving edge policies. Previously, they were
being appended to a slice of policies, but this doesn't always work like
when you attempt to update the same edge twice. Instead, the slice can
only contain up to two entries, each one being the latest version of
each direction.
2019-03-28 17:22:22 -07:00
Wilmer Paulino
3a2c4ec594 discovery: add missing offline peer check before sending message reliably 2019-03-28 17:21:28 -07:00
Wilmer Paulino
3e81e89062 discovery: add log when attempting to send msg reliably and peer is offline 2019-03-28 17:21:28 -07:00
Wilmer Paulino
5cec4513de
discovery: reject announcements for known zombie edges
In this commit, we leverage the recently introduced zombie edge index to
quickly reject announcements for edges we've previously deemed as
zombies. Care has been taken to ensure we don't reject fresh updates for
edges we've considered zombies.
2019-03-27 13:08:03 -07:00
Wilmer Paulino
23796d3247
routing+discovery: extend ChannelGraphSource with zombie index methods 2019-03-27 13:07:30 -07:00
Conner Fromknecht
0ae06c8189
discovery+server: send lazy gossip msgs 2019-03-05 17:08:48 -08:00
Conner Fromknecht
f39edd8000
peer: add SendMessageLazy 2019-03-05 17:08:22 -08:00
Wilmer Paulino
12168f022e
server+discovery: send channel updates to remote peers reliably
In this commit, we also allow channel updates for our channels to be
sent reliably to our channel counterparty. This is especially crucial
for private channels, since they're not announced, in order to ensure
each party can receive funds from the other side.
2019-02-14 18:33:27 -08:00
Wilmer Paulino
4996d49118
server+discovery: use reliableSender to replace existing resend logic 2019-02-14 18:33:27 -08:00
Wilmer Paulino
2f679f6015
discovery/reliable_sender: implement message-agnostic reliable sender
In this commit, we implement a new subsystem for the gossiper that
uses some of the existing logic for resending channel announcement
signatures and implements it in a way to make it message-agnostic,
meaning that any type of message can be resent. Along the way we also
modify the way this works to prevent multiple goroutines per peer _and_
message.

A peerHandler will be spawned for each peer for which we attempt to send
a message reliably to. This handler is responsible for managing requests
to reliably send messages to a peer while also taking the peer's
connection lifecycle into account by requesting notifications for when
the peer connects/disconnects. A peer connection notification is first
requested to determine when we should attempt to send any pending
messages. After the messages are sent, a peer disconnection notification
is requested to ensure we don't continue to request connection
notifications while the peer remains connected. Once there are no more
pending messages left to be sent for a given peer, the peerHandler can
be torn down.
2019-02-14 18:33:27 -08:00
Wilmer Paulino
6e556aa897
discovery/gossiper_test: prevent race conditions within mockGraphSource 2019-02-14 18:33:27 -08:00
Wilmer Paulino
73b4bc4b68
server+discovery: remove channeldb.DB reference within the gossiper
Now that we've replaced the built-in messageStore with the
channeldb.GossipMessageStore, the reference to channeldb.DB is no longer
needed.
2019-02-14 18:29:39 -08:00
Wilmer Paulino
2277535e6b
server+discovery: replace gossiper message store with MessageStore 2019-02-14 18:29:39 -08:00
Wilmer Paulino
847b064461
discovery/message_store: add gossip message store
In this commit, we add a new store within the database that'll be
responsible for storing gossip messages which we need to reliably send
to peers. This aims to replace the current messageStore that exists
within the gossiper, so much of this logic is borrowed from there.
One of the main differences between the two is that we now index
messages with a new key format in which we take into account the
message's type. This allows us to store different messages for a
specific channel with a peer. The old key format is still supported in
order to prevent a database migration.
2019-02-14 18:29:39 -08:00
Johan T. Halseth
7d34ce9d08
lnwire+multi: define HasMaxHtlc helper on msgFlags 2019-01-22 08:42:30 +01:00
Valentine Wallace
207c4f030a
discovery/gossiper: include max HTLC when rebroadcasting stale channel updates
Co-authored-by: Johan T. Halseth <johanth@gmail.com>
2019-01-22 08:42:29 +01:00
Valentine Wallace
19c403711e
discovery/chan_series+utils: include max htlc when syncing with peers
In this commit, we ensure that max HTLC is included when we're
synchronizing ChannelUpdates with remote peers.
2019-01-22 08:42:28 +01:00
Valentine Wallace
513ac23479
discovery/gossiper: persist remote channel policy updates' max htlc 2019-01-22 08:42:28 +01:00
Valentine Wallace
15168c391e
discovery+routing: validate msg flags and max htlc in ChannelUpdates
In this commit, we alter the ValidateChannelUpdateAnn function in
ann_validation to validate a remote ChannelUpdate's message flags
and max HTLC field. If the message flag is set but the max HTLC
field is not set or vice versa, the ChannelUpdate fails validation.

Co-authored-by: Johan T. Halseth <johanth@gmail.com>
2019-01-22 08:42:27 +01:00
Valentine Wallace
f316cc6c7e
discovery/gossiper_test: set ChannelUpdate max htlc
In this commit, we alter the gossiper test's helper method
that creates channel updates to include the max htlc field
in the ChannelUpdates it creates.

Co-authored-by: Johan T. Halseth <johanth@gmail.com>
2019-01-22 08:42:26 +01:00
Valentine Wallace
7ab8900eb6
discovery/gossiper_test: mock AddEdge: set capacity
In this commit, we modify the mockGraphSource's `AddEdge`
method to set the capacity of the edge it's adding to be a large
capacity.

This will enable us to test the validation of each ChannelUpdate's
max HTLC, since future validation checks will ensure the specified
max HTLC is less than total channel capacity.
2019-01-22 08:42:26 +01:00
Valentine Wallace
0fd6004958
multi: partition lnwire.ChanUpdateFlag into ChannelFlags and MessageFlags
In this commit:

* we partition lnwire.ChanUpdateFlag into two (ChanUpdateChanFlags and
ChanUpdateMsgFlags), from a uint16 to a pair of uint8's

* we rename the ChannelUpdate.Flags to ChannelFlags and add an
additional MessageFlags field, which will be used to indicate the
presence of the optional field HtlcMaximumMsat within the ChannelUpdate.

* we partition ChannelEdgePolicy.Flags into message and channel flags.
This change corresponds to the partitioning of the ChannelUpdate's Flags
field into MessageFlags and ChannelFlags.

Co-authored-by: Johan T. Halseth <johanth@gmail.com>
2019-01-22 08:42:26 +01:00
Johan T. Halseth
5dcd2a4530
gossiper: validate our own node announement 2018-12-05 09:31:25 +01:00
Olaoluwa Osuntokun
640fe2558b
discovery: update TestReceiveRemoteChannelUpdateFirst due to stricter stale node checks
A recent commit modified the `IsNodeStale` method in the mocks to mirror
the actual implementation in the gossiper. As a result, we now expect
one less node announcement to be broadcast.
2018-12-03 21:04:28 -08:00
Olaoluwa Osuntokun
5ad2592891
Merge pull request #2123 from halseth/node-announcement-no-forward
[Tests] No forwarding of stale node announcements
2018-12-03 20:11:43 -08:00
Conner Fromknecht
5fc0a192ad
discovery/syncer: move CGTimeSerires to chan_series 2018-11-30 15:33:34 -08:00
Conner Fromknecht
1c81a8fad6
discovery/chan_series: copy chan_series to discovery 2018-11-30 15:33:34 -08:00
Olaoluwa Osuntokun
1fd3aac925
multi: switch from bolt packge to bbolt package for all imports 2018-11-29 20:33:49 -08:00
Wilmer Paulino
55094f1470
discovery/gossiper: send node anns when constructing full chan proof
In this commit, we allow the gossiper to also broadcast the
corresponding node announcements, if we know of them, of a channel when
constructing its full proof. We do this to ensure peers (other than our
remote peer) receive all the relevant announcements for a channel.

The tests changes were made to ensure the new behavior introduced works
as intended. Previously, the node announcements for each test channel
announcement were not processed, so they never existed from the
gossiper's point of view.

This also addresses an existing flake in the integration test
`testNodeAnnouncement`. This problem arose due to the node announcement
being sent before the connection between Dave (node announcement sender)
and Alice (node announcement receiver) was initiated and the full
channel proof was constructed.
2018-11-11 17:48:07 -08:00
Wilmer Paulino
c9e79527af
discovery/gossiper: add trace log when skipping unadvertised node ann 2018-11-11 17:48:06 -08:00
Wilmer Paulino
4f45e339f1
discovery+routing: add FetchLightningNode to ChannelGraphSource interface 2018-11-11 17:48:05 -08:00
Conner Fromknecht
ef309f43d2
discovery: pass peer quit signal to ProcessQueryMsg
This commit passes the peer's quit signal to the
gossipSyncer when attempt to hand off gossip query
messages. This allows a rate-limited peer's read
handler to break out immediately, which would
otherwise remain stuck until the rate-limited
gossip syncer pulled the message.
2018-11-01 17:35:55 -07:00
Conner Fromknecht
96c47f7de4
discovery/gossiper: bypass main event loop for queries
This commit restructures the delivery of gossip
query related messages, such that they are delivered
directly to the gossip syncers. Gossip query rate
limiting was introduced in #1824 on a per-peer basis.
However, since all gossip query messages were being
delivered in the main event loop, the end result is
that one rate-limited peer could stall all other
peers.

In addition, since no other peers would be able to
submit gossip-related messages through the blocked
event loop, the back pressure would eventually rate
limit the read handlers of all peers as well.
The end result would be lengthy delays in reading
messages related to htlc forwarding.

The fix is to lift the delivery of gossip query
messages outside of the main event loop. With
this change, the rate limiting backpressure is
delivered only to the intended peer.
2018-11-01 17:28:20 -07:00
Johan T. Halseth
13b33d79b3
discover/gossiper test: add TestNodeAnnouncementNoChannels
This commit adds a new test TestNodeAnnouncementNoChannels that ensures
a node announcement for a unknown node is not forwarded by the gossiper.
2018-10-30 10:01:51 +01:00
Johan T. Halseth
2911944e5a
discovery/gossiper test: define unknown node announcement as stal
To mimic the current behaviour of the router's IsStaleNode, we make the
mockGraphSource consider a unknown node with no channels in the graph as
stale.
2018-10-30 10:01:33 +01:00
Wilmer Paulino
748da2f50a
discovery/gossiper_test: modify TestProcessAnnouncement to process node
ann last

In this commit, we modify TestProcessAnnouncement to process the node
announcement last. We do this due to the recent change in the gossiper
where we'll only forward node announcements of nodes who intend to
advertised themselves within the network.

This change was needed in order to allow the node announcement to be
broadcast to the greater network, as otherwise the gossiper would assume
the node intends to stay private due to not having any advertised edges.
2018-10-24 17:28:56 -07:00
Wilmer Paulino
80196eb20f
discovery: ensure we only broadcast NodeAnnouncements of public nodes
In this commit, we modify the gossiper to no longer broadcast
NodeAnnouncements of nodes who intend to remain private. We do this to
prevent leaking their information to the greater network.
2018-10-24 17:28:55 -07:00
Olaoluwa Osuntokun
d67c1fdcaa
Merge pull request #1954 from Roasbeef/tor-node-ann-fixes
server: ensure each time we update a node ann we also update the on-disk version
2018-10-24 17:10:42 -07:00
Olaoluwa Osuntokun
eaa613f4f5
discovery+chan_series: replace usage of makeNodeAnn w/ new NodeAnnouncement method 2018-10-23 18:42:28 -07:00
sevastos
98275178c7 discovery/gossiper: fix logging on failed channel update 2018-10-22 15:37:38 +03:00
Conner Fromknecht
59b459674d
multi: init subsystem loggers via build pkg 2018-10-05 13:04:45 +09:00
Johan T. Halseth
54a608d09d
htlcswitch+discovery mock: adhere to new lnpeer interface 2018-09-28 09:05:18 +02:00
Olaoluwa Osuntokun
6afee3d099
Merge pull request #1856 from maurycy/typos
multi: fix various typos in comments
2018-09-27 20:38:10 -07:00
Olaoluwa Osuntokun
68774e3ae4
Merge pull request #1807 from wpaulino/retransmit-zero-stale-channels
discovery/gossiper: check ErrNoGraphEdgesFound for restransmitStaleChannels
2018-09-25 21:14:23 -07:00
Olaoluwa Osuntokun
25145acc46
Merge pull request #1738 from joostjager/validation
payment failure message with attached channel_update not validated
2018-09-17 21:11:02 -07:00
Olaoluwa Osuntokun
c5979c7d80
discovery: fix logging for gossip syncer rate limiting 2018-09-13 15:55:44 -07:00
Joost Jager
b5fd32ff77
discovery+routing: move validation logic to routing package
Previously, gossiper was the only object that validated channel
updates. Because updates can also be received as part of a
failed payment session in the routing package, validation logic
needs to be available there too. Gossiper already depends on
routing and having routing call the validation logic inside
gossiper would be a circular dependency. Therefore the validation
was moved to routing.
2018-09-11 20:12:07 +02:00
Conner Fromknecht
a56e7122d6
discovery/syncer: use rate limiter for gossip queries
This commit replaces the simplistic rate limiting
technique added in 557cb6e2, to use the
golang.org/x/time's rate limiter. This has the
benefit of performing traffic shaping to meet a
target maximum rate, and yet tolerate bursts. Bursts
are expected during initial sync, though should become
more rare afterwards. Performing traffic shaping with
this mechanism should improve the ability of the gossip
syncer to detect sustained bursts from the remote peer,
and penalize them appropriately.

This commit also modifies the default parameters to
accept bursts of 10 queries, with a target rate of 1
reply every 5 seconds.
2018-09-11 00:12:35 -07:00
maurycy
ac24b12bf2
multi: fix various typos in comments 2018-09-07 06:51:49 +02:00
Conner Fromknecht
dc0030f11d
discovery/gossiper: remove go-errors pkg 2018-09-05 22:08:16 -07:00
Conner Fromknecht
045a2c7fff
discovery/syncer_test: add DOS delayed replies test 2018-09-05 22:08:16 -07:00
Conner Fromknecht
557cb6e253
discovery/syncer: delay replies after initial sync to prevent DOS 2018-09-05 22:08:16 -07:00
Olaoluwa Osuntokun
d2a7d910b7
Merge pull request #1825 from Roasbeef/extra-gossip-message-data
channeldb+discovery: ensure we store, validate and propagate announcements with opaque data
2018-09-05 17:53:02 -07:00
Olaoluwa Osuntokun
88aa74be57
discovery: add new set of tests to ensure we validate w/ extra data 2018-09-04 20:52:44 -07:00
Olaoluwa Osuntokun
841e24399c
discovery: ensure we populate ExtraOpaqueData when writing/reading from disk 2018-09-04 20:52:44 -07:00
Olaoluwa Osuntokun
0fc20cee64
discovery: update utils to properly include opaque data in gossip msgs 2018-09-04 20:52:44 -07:00
Conner Fromknecht
8e94e55839
discovery/gossiper: require explict gossip syncer init
This commit removes the fallback in fetchGossipSyncer
that creates a gossip syncer if one is not registered
w/in the gossiper. Now that we register gossip syncers
explicitly before reading any gossip query messages,
this should not longer be required. The fallback also
did not honor the cfg.NoChanUpdates flag, which may
have led to inconsistencies between configuration and
actual behavior.
2018-09-04 17:32:25 -07:00
Wilmer Paulino
6f0fad7946
discovery/gossiper: check ErrNoGraphEdgesFound for
restransmitStaleChannels

In this commit, we add an additional error check for
ErrNoGraphEdgesFound when restransmitting stale channels during the
gossiper's startup. We do this to prevent benign log messages as we'll
log that we were unable to retransmit stale channels when we didn't have
any channels in our graph to begin with.
2018-09-04 16:22:06 -07:00
Olaoluwa Osuntokun
4f43c1c943
Merge pull request #1811 from Roasbeef/autopilot-cpu-usage-fix
autopilot: modify the graph interface to return raw bytes for node pubkeys, not entire key
2018-09-03 19:56:51 -07:00
Olaoluwa Osuntokun
8f843c5eaa
discovery: update autopilot.Node usage to match recent API changes 2018-08-29 15:45:39 -07:00
Wilmer Paulino
a1b1b06b9e
discovery/gossiper: remove optimistic channel announcement request
In this commit, we aim to resolve an issue with nodes requesting for
channel announcements when receiving a channel update for a channel
they're not aware of. This can happen if a node is not caught up with
the chain or if they receive updates for zombie channels. This would
lead to a spam issue, as if a node is not caught up with the chain,
every new update they receive is premature, causing them to manually
request the backing channel announcement. Ideally, we should be able to
detect this as a potential DoS vector and ban the node responsible, but
for now we'll simply remove this functionality.
2018-08-27 15:54:21 -07:00
Olaoluwa Osuntokun
13a6d413ac
discovery: select on peer's QuitSignal to allow caller to unblock if disconnecting
In this commit, we select on the peer's QuitSignal to allow the caller
to unblock if the peer itself is disconnecting. With this change, we now
ensure that it isn't possible for a peer to block on this method and
prevent a graceful exit.
2018-08-25 17:30:16 -07:00
Olaoluwa Osuntokun
19552b0dbf
htlcswitch+funding+discovery: update mock peers to add new QuitSignal method 2018-08-25 17:30:15 -07:00
Johan T. Halseth
c56082006f
discovery/gossiper: formatting 2018-08-23 07:40:40 +02:00
Johan T. Halseth
0bc415c683
discovery/gossiper: don't return on errChan for ChannelUpdate not yet processed
Previosuly we would immediately return nil on the error channel for
premature ChannelUpdates, which would break the expection that a a
returned non-error meant the update was successfully added to the
database. This meant that the caller would believe the update was added
to the database, while it is actually still in volatile memory and can
be lost during restarts.

This change makes us handle premature ChannelUpdates as we handle other
premature announcements within the gossiper, by deferring sending on the
error channel until we have reprocessed the update.
2018-08-23 07:40:40 +02:00
Johan T. Halseth
7bbb2bbc80
discovery/gossiper: return on errChan on ValidationBarrier error 2018-08-23 07:40:40 +02:00
Johan T. Halseth
c8ce2aa2d2
discovery/gossiper: reaturn early of no premature to process 2018-08-23 07:40:39 +02:00
Johan T. Halseth
d0e8aeece1
discovery/gossiper: return on errChan in case of ChanAnn/Upd on wrong chain
Previously we wouldn't return anything in the case where the
announcement were meant for a chain we didn't recognize. After this
change we should return an error on the error channel in all flows
within the gossiper.
2018-08-23 07:40:39 +02:00
Johan T. Halseth
54f67f72b1
discovery/gossiper: define ErrGossiperShuttingDown 2018-08-23 07:40:39 +02:00
Valentine Wallace
1ffc3bb82e
multi: update to latest RegisterBlockEpochNtfn interface 2018-08-10 01:08:57 -07:00
Conner Fromknecht
cf2c371042
multi: fix linting errors 2018-08-02 18:20:50 -07:00
Conner Fromknecht
1ded697e8d
multi: sort import paths with gofmt 2018-08-02 18:20:49 -07:00
Olaoluwa Osuntokun
88bedec65b
discovery: update unit tests to account for new ChainNotifier API 2018-07-31 21:28:57 -07:00
Olaoluwa Osuntokun
c41d1a0a44
Merge pull request #1596 from wpaulino/syncer-cfg-chunk-size
discovery/syncer: store chunk size within gossiperSyncerCfg
2018-07-31 20:55:10 -07:00
Conner Fromknecht
54c4b09f87
discovery/gossiper: copy bolt key to prevent panic
Corrects an instance that holds a reference to a boltdb
byte slice after returning from the transaction. This
can cause panics under certain conditions, which is
avoided by creating a copy of the key.
2018-07-30 18:30:19 -07:00
Wilmer Paulino
c61b037628 discovery/syncer: store chunk size within gossiperSyncerCfg
In this commit, we allow the gossiper syncer to store the chunk size for
its respective encoding type. We do this to prevent a race condition
that would arise within the unit tests by modifying the values of the
encodingTypeToChunkSize map to allow for easier testing.
2018-07-30 13:49:36 -07:00
Johan T. Halseth
d7b2977e8c
multi test: remove mempool bool from RegisterSpendNtfn 2018-07-22 23:09:08 +02:00
Wilmer Paulino
6d4da72156
discovery/gossiper: update to latest NotifyWhenOnline changes 2018-07-19 12:34:08 -07:00
Wilmer Paulino
3ab17063ff
multi: satisfy new lnpeer interface 2018-07-19 12:32:29 -07:00
Olaoluwa Osuntokun
daf4a25fd9
discovery: fix missing formatting argument
Fixes #1576.
2018-07-17 16:43:06 -07:00
Olaoluwa Osuntokun
6f60f139f4 multi: switch over import paths from roasbeef/* to btcsuite/* 2018-07-13 17:05:39 -07:00
Wilmer Paulino
9ecfdb3c32
discovery/gossiper: hold ref to block epoch stream and cancel when stopped 2018-06-13 19:20:53 -07:00
Olaoluwa Osuntokun
dddbfa7efe
Merge pull request #1205 from wpaulino/initial-peer-bootstrap
server: improve initial peer bootstrapping
2018-06-13 16:51:41 -07:00
Olaoluwa Osuntokun
f2273d9503
discovery: log failed attempts in replyShortChanIDs 2018-06-13 16:44:59 -07:00
Wilmer Paulino
800b1363ab
discovery: randomize order of different bootstrappers
In this commit, we randomize the order of the different bootstrappers in
order to prevent from always querying potentially unreliable
bootstrappers first.
2018-06-13 08:59:37 -07:00
Olaoluwa Osuntokun
1a15924d65
discovery: fix log for adding new gossip syncers
In this commit, we fix the logging when adding new gossip syncers. The
old log would log the byte array, rather than the byte slice. We fix
this by slicing before logging.
2018-06-11 22:58:30 -07:00
Conner Fromknecht
bf515a2341
discovery/gossiper: mostly deprecate SendToPeer
This commit changes the gossiper to direct messages to
peer objects, instead of sending them through the
server every time. The primary motivation is to reduce
contention on the server's mutex and, more importantly,
avoid deadlocks in the Triangle of Death.
2018-06-08 16:29:48 -07:00
Conner Fromknecht
769f0f0a94
discovery/gossiper_test: mock out new lnpeer.Peer 2018-06-08 13:47:57 -07:00
Olaoluwa Osuntokun
d98d4523e1
Merge pull request #1159 from wpaulino/onion-service-support
multi: add onion services support
2018-06-04 21:46:51 -07:00
Wilmer Paulino
2e0484be19
multi: ensure addresses are no longer assumed to be TCP addresses only
In this commit, we go through the codebase looking for TCP address
assumptions and modifying them to include the recently introduced onion
addresses. This enables us to fully support onion addresses within the
daemon.
2018-06-04 20:41:49 -07:00
Wilmer Paulino
3bb1733fa2
discovery+server: use network-specific functions for fallback SRV lookup
In this commit, we fix a bug where a fallback SRV lookup would leak
information if `lnd` was set to route connections over Tor. We solve
this by using the network-specific functions rather than the standard
ones found in the `net` package.
2018-06-04 20:41:43 -07:00
maurycy
3be08e69cf multi: 64bit aligment of atomic vars on arm/x86-32 2018-06-04 20:02:34 -07:00
Olaoluwa Osuntokun
e6d46f681b
discovery: if gossipSyncer is shutting down, don't filter messages 2018-06-04 17:18:54 -07:00
Olaoluwa Osuntokun
0ec4a06e6b
discovery: fix deadlock by ensure we don't hold sync mutex during send
In this commit, we fix an existing deadlock in the
gossiper->server->peer pipeline by ensuring that we're not holding the
syncer mutex while we attempt to have a syncer filter out the rest of
gossip messages.
2018-06-04 17:18:17 -07:00
Olaoluwa Osuntokun
a9340d22c4 discovery: ensure stopping/starting a gossipSyncer is idempotent 2018-06-04 17:18:07 -07:00
Olaoluwa Osuntokun
b3ac3492a0
discovery: if unable to find gossipSyncer for peer, create one
In this commit we fix an existing bug caused by a scheduling race
condition. We'll now ensure that if we get a gossip message from a peer
before we create an instance for it, then we create one on the spot so
we can service the message. Before this commit, we would drop the first
message, and therefore never sync up with the peer at all, causing them
to miss channel announcements.
2018-05-31 17:54:45 -07:00
Olaoluwa Osuntokun
994d9cf7e4
discovery: attempt to request the full chan ann for stray chan updates
In this commit, we extend the AuthenticatedGossiper to take advantage of
the new query features in the case that it gets a channel update w/o
first receiving the full channel announcement. If this happens, we'll
attempt to find a syncer that's fully synced, and request the channel
announcement from it.
2018-05-31 16:31:01 -07:00
Olaoluwa Osuntokun
6a6700e629
discovery: add new SyncState method to gossipSyncer
This new method allows outside callers to sample the current state of
the gossipSyncer in a concurrent-safe manner. In order to achieve this,
we now only modify the g.state variable atomically.
2018-05-31 16:31:00 -07:00
Olaoluwa Osuntokun
55dfc17c05
discovery: de-duplicate node announcements with legacy chan graph sync
Fixes #1072.
2018-05-31 16:30:58 -07:00
Olaoluwa Osuntokun
0016b3f458
discovery+lnd: create new chanSeries impl of the ChannelGraphTimeSeries interface
In this commit, we create a new concrete implementation for the new
discovery.ChannelGraphTimeSeries interface. We also export the
createChannelAnnouncement method to allow the chanSeries struct to
re-use the existing code for creating wire messages from the database
structs.
2018-05-31 16:30:55 -07:00
Olaoluwa Osuntokun
45bd544f11
discovery: update AuthenticatedGossiper to be aware of new gossipSyncers
In this commit, we update the logic in the AuthenticatedGossiper to
ensure that can properly create, manage, and dispatch messages to any
gossipSyncer instances created by the server.

With this set of changes, the gossip now has complete knowledge of the
current set of peers we're conneted to that support the new range
queries. Upon initial connect, InitSyncState will be called by the
server if the new peer understands the set of gossip queries. This will
then create a new spot in the peerSyncers map for the new syncer. For
each new gossip query message, we'll then attempt to dispatch the
message directly to the gossip syncer. When the peer has disconnected,
we then expect the server to call the PruneSyncState method which will
allow us to free up the resources.

Finally, when we go to broadcast messages, we'll send the messages
directly to the peers that have gossipSyncer instances active, so they
can properly be filtered out. For those that don't we'll broadcast
directly, ensuring we skip *all* peers that have an active gossip
syncer.
2018-05-31 16:30:53 -07:00
Olaoluwa Osuntokun
5789ef7c10
discovery: add new gossipSyncer struct to manage sync state for each peer
In this commit, introduce a new struct, the gossipSyncer. The role of
this struct is to encapsulate the state machine required to implement
the new gossip query range feature recently added to the spec. With this
change, each peer that knows of this new feature will have a new
goroutine that will be managed by the gossiper.

Once created and started, the gossipSyncer will start to progress
through each possible state, finally ending at the chansSynced stage. In
this stage, it has synchronized state with the remote peer, and is
simply awaiting any new messages from the gossiper to send directly to
the peer. Each message will only be sent if the remote peer actually has
a set update horizon, and the message isn't before or after that
horizon.

A set of unit tests has been added to ensure that two state machines
properly terminate and synchronize channel state.
2018-05-31 16:30:53 -07:00
Conner Fromknecht
3854c1ed68
discovery/gossiper: exit early on validation barrier quit 2018-05-08 16:32:08 -07:00
Dimitris Tsapakidis
4009f7f874 multi: fix typos in comments 2018-04-17 19:03:27 -07:00
practicalswift
663c396235 multi: fix a-vs-an typos 2018-04-17 19:02:04 -07:00
Jim Posen
4b2cb68fe6 discovery: Fix formatting issue in log line. 2018-04-09 20:49:23 -07:00
Olaoluwa Osuntokun
3fa2e08665
test: update testUpdateChannelPolicy to ensure Bob's link uses the proper policies
In this commit, we update the testUpdateChannelPolicy to exercise the
recent set of changes within the switch. If one applies this test to a
fresh branch (without those new changes) it should fail. This is due to
the fact that before, Bob would attempt to apply the constraints of the
incoming link (which we updated) instead of the outgoing link. With the
recent set of changes, the test now properly passes.
2018-04-06 14:52:01 -07:00
Olaoluwa Osuntokun
9d4cea93f0
discovery: fix deadlock in processChanPolicyUpdate
In this commit, we fix an existing deadlock in the
processChanPolicyUpdate method. Before this commit, within
processChanPolicyUpdate, we would directly call updateChannel *within*
the ForEachChannel closure. This would at times result in a deadlock, as
updateChannel will itself attempt to create a write transaction in order
to persist the newly updated channel.

We fix this deadlock by simply performing another loop once we know the
set of channels that we wish to update. This second loop will actually
update the channels on disk.
2018-04-06 14:51:57 -07:00
Johan T. Halseth
ed6682ea4a
multi test: make mock adhere to api change 2018-04-03 22:04:02 +02:00
Olaoluwa Osuntokun
2326d4afeb
discovery: bump rejected edges up to the trace logging level 2018-03-23 15:50:56 -07:00
Olaoluwa Osuntokun
ce85632390
discovery: ensure target host for DNS fallback is fully qualified
In this commit, fix the inability of some users to connect to the DNS
seed using our direct TCP fallback. We do this as some resolvers filter
out our large SRV requests due to their size (they also include public
keys). Instead, we’ll use a direct TCP resolution in this case.
However, after a recent change, we forgot the period at the end of the
target DNS host. This is an issue as the domain needs to be fully
qualified.

The fix is easy, add a period within our string formatting to target
the proper sub-domain and SRV target.

Fixes #854.
2018-03-18 16:05:10 -07:00
Matt Drollette
1b7b35a8bc discovery: format log message with Errorf 2018-03-16 16:25:18 -07:00
Olaoluwa Osuntokun
30019538c4
discovery: update DNS bootstrapper to adhere to new query schema
In this commit, we update the DNS bootstrapper to match the new query
semantics expected by the new DNS server. We no longer hard code the
target DNS host, and instead, we’ll re-use the same target endpoint as
we only need the soaShim in order to establish a direct TCP connection
for the queries.
2018-03-14 18:27:29 -07:00
Olaoluwa Osuntokun
800eea931f
build+multi: switch from bolt to bbolt
In this commit, we switch from boltbd/bolt to coreos/bbolt as the
former is no longer being actively maintained.
2018-03-10 19:01:13 -08:00
Olaoluwa Osuntokun
e7e4cdcf49
discovery: avoid always validating ECDSA sigs by asking router if item is fresh
In this commit, we reduce the amount of unnecessary work that the
gossiper can carry out. When CPU profiling some nodes, I noticed that
we’d spend a lot of time validating the signatures for an announcement,
only to realize that the router already had it.

To remedy this, we’ll use the new methods added to the channel router
in order to avoid unnecessarily validating an announcement that is
actually stale. This should reduce memory usage (since it uses big
int’s under the scenes), and also idle CPU usage.
2018-03-06 13:34:51 -05:00
practicalswift
b8e1351cf3 multi: fix some recently introduced typos 2018-02-18 15:27:29 -08:00
Olaoluwa Osuntokun
bf05e47780
discovery: add additional gossiper level reject cache
In this commit, we’ll add a new reject cache to ensure that we don’t
attempt to re-process any announcements already rejected by the
ChannelRouter.
2018-02-06 20:14:35 -08:00
Olaoluwa Osuntokun
cd9d2d7e6f
discovery: update graph API usage to match recent API changes 2018-02-06 20:14:32 -08:00
Daniel McNally
8543497dcc multi: fixing it's/its typos in comments 2018-02-06 19:13:07 -08:00
practicalswift
a93736d21e multi: comprehensive typo fixes across all packages 2018-02-06 19:11:11 -08:00
nsa
e132ad8433 torsvc: Added new module to house Tor functions
This commit adds a new module named 'torsvc' which houses all Tor
functionality in an attempt to isolate it and make it reusable in
other projecs. Some additional tweaks were made to config.go and
to the bootstrapper.
2018-02-05 17:37:46 -08:00
nsa
e2142c778f multi: Added Tor support
This commit adds Tor support. Users can set the --TorSocks flag
to specify which port Tor's SOCKS5 proxy is listening on so that
lnd can connect to it. When this flag is set, ALL traffic gets
routed over Tor including DNS traffic. Special functions for
DNS lookups were added, and since Tor doesn't natively support
SRV requests, the proxySRV function routes connects us to
a DNS server via Tor and SRV requests can be issued directly
to the DNS server.

Co-authored-by: MeshCollider <dobsonsa68@gmail.com>
2018-02-05 17:37:46 -08:00
Olaoluwa Osuntokun
0a4de859a2
discovery+routing: reduce number of active validation barrier jobs
In order to reduce high CPU utilization during the initial network view
sync, we slash down the total number of active in-flight jobs that can
be launched.
2018-01-28 14:55:32 -08:00
Johan T. Halseth
a55c74f80c
discovery/gossiper: restrict database access per channel ID
This commit uses the multimutex.Mutex to esure database
state stays consistent when handling an announcement, by
restricting access to one goroutine per channel ID.

This fixes a bug where the goroutine would read the
database, make some decisions based on what was read,
then write its data to the database, but the read data
would be outdated at this point. For instance, an
AuthProof could have been added between reading the
database and when the decision whether to announce
the channel is made, making it not announce it.
Similarly, when receiving the AuthProof, the edge
policy could be added between reading the edge state
and adding the proof to the database, also resulting
in the edge not being announced.
2018-01-24 10:26:40 +01:00
Johan T. Halseth
3e01529de4
discovery/gossiper: don't reset batch after broadcast
This commit fixes a bug that could cause annoucements
to get lost, and resultet in flaky integration tests.
After a set of announcements was broadcastet, we would
reset (clear) the announcement batch, making any
annoucement that was added between the call to Emit()
and Reset() to be deleted, without ever being broadcast.

We can just remove the Reset() call, as the batch will
actually be reset within the call to Emit(), making
the previous call only delete those messages we hadn't
sent yet.
2018-01-24 10:26:39 +01:00
Olaoluwa Osuntokun
69e6ec9954
peer+funding: remove unneeded channel handoff code with the breach arbiter
We no longer need to hand off new channels that come online as the
chainWatcher will be persistent, and always have an active signal for
the entire lifetime of the channel.
2018-01-22 19:19:50 -08:00
Olaoluwa Osuntokun
8807d1d752
fundingmgr: add new function closure to send new channels to the ChainArbitrator 2018-01-22 19:19:38 -08:00
Olaoluwa Osuntokun
ebd97b83f3
discovery: ensure we only send node announcements that have active channels 2018-01-16 17:18:21 -08:00
Johan T. Halseth
483abbee5b
discovery: make sure ChannelUpdates always have incremented timestamp
This commit ensures that we always increment the timestamp of
ChannelUpdates we send telling the network about changes to
our channel policy. We do this because it could happen
(especially during tests) that we issued an update, but the
ChannelUpdate would have the same timestamp as our last
ChannelUpdate, and would be ignored by the network.
2018-01-12 22:57:03 +01:00
Johan T. Halseth
8370fa2cde
discovery/gossiper: apply TimeLockDelta to edge when processing policy update
Also rename various instances of "FeeUpdate" to "PolicyUpdate"
2018-01-12 22:56:32 +01:00
Johan T. Halseth
3f0a525331
discovery test: update gossiper test to account for timestamp deduping 2018-01-09 13:09:28 +01:00
Johan T. Halseth
ea95f37277
discovery: keep newest messages when deduping in gossiper
This commit makes the gossiper aware of the timestamps
of ChannelUpdates and NodeAnnouncements, such that it
only keeps the newest message when deduping. Earlier
it would always keep the last received message, which
in some cases could be outdated.
2018-01-09 13:09:25 +01:00
Johan T. Halseth
2ddd139084 discovery: check proof for nilness before creating chanAnn
This commit makes sure we are not attempting to create a
channel announcement with a nil ChannelAuthProof, as that
could cause a crash at startup whe the gossiper would
attempt to reprocess an edge coming from the fundingmanager.

It also makes sure we check the correct error returned from
processRejectedEdge.
2018-01-02 19:51:38 +01:00
Olaoluwa Osuntokun
6b0f984e31
discovery: update tests to reflect new broadcast related changes 2017-12-26 16:29:37 +01:00
Olaoluwa Osuntokun
38715ca768
discovery: refactor re-processing rejected announcements into new method 2017-12-26 16:28:23 +01:00
Olaoluwa Osuntokun
68dabe3594
discovery: update usage of d.cfg.Broadcast to reflect recent API change
When broadcasting a set of messages, we must now call broadcast for
each (msg, …senders) tuple, rather than just calling it once for the
entire batch.
2017-12-26 16:26:34 +01:00
Olaoluwa Osuntokun
f45502f890
discovery: update message processing to emit networkMsg rather than lnwire.Message
In this commit, we make an incremental step towards page of the new
feature of deDupedAnnoucnements to return the set of senders for each
message. All methods the process new channel announcements, will now
return an instance of networkMsg rather than lnwire.Message. This will
allow passing the returned announcement directly into
deDupedAnnoucnements.AddMsg().
2017-12-26 16:23:12 +01:00
Olaoluwa Osuntokun
f1b40e0b4d
discovery: modify deDupedAnnouncements to keep track of senders for each msg
In this commit, we modify the deDupedAnnouncements struct slightly. The
element of this struct will now keep track of the set of senders that
sent a particular message. Each time a message is added, we’ll replace
the new message with the old (as normal), but we’ll also add the new
sender to the set of known senders.

With this new feature, we’ll be able to avoid re-sending a message to
the peer that sent it to us in the first place.
2017-12-26 16:19:03 +01:00