Commit Graph

5161 Commits

Author SHA1 Message Date
Conner Fromknecht
e702a6a266
autopilot/agent_test: remove Fatalf calls from goroutines 2018-09-05 20:10:36 -07:00
Conner Fromknecht
2b578e06fc
autopilot/agent_test: ensure directives use unique keys
This commit ensures that the mock attachment
directives use unique keys, ensuring that they
aren't skipped due to already having pending
connection requests. The tests fail when
they're all the same since they collide
in the pendingConns map.
2018-09-05 20:10:35 -07:00
Conner Fromknecht
7d9483c20d
autopilot/agent_test: adds TestAgentSkipPendingConns
Adds a test asserting that the agent prevents
itself from making duplicate outstanding
connection requests to the same peer.
2018-09-05 20:10:35 -07:00
Conner Fromknecht
c214bea9db
autopilot/prefattach: set NodeID for selection candidates 2018-09-05 20:10:34 -07:00
Conner Fromknecht
08b6bf54fb
autopilot/interface: expose NodeID in AttachmentDirective 2018-09-05 20:10:34 -07:00
Conner Fromknecht
4599b0eac3
autopilot/agent: track pending connections
This commit modifies the autopilot agent to track
all pending connection requests, and forgo further
attempts if a connection is already present.
Previously, the agent would try and establish
hundreds of requests to a node, especially if the
connections were timing out and not returning.

This resulted in an OOM OMM when cranking up
maxchannels to 200, since there would be close
to 10k pending connections before the program was
terminated. The issue was compounded by periodic
batch timeouts, causing autopilot to try and
process thousands of triggers for failing
connections to the same peer.

With these fixes, autopilot will skip nodes that we
are trying to connect to during heuristic selection.
The CPU and memory utilization have been significantly
reduced as a result.
2018-09-05 20:10:34 -07:00
Olaoluwa Osuntokun
4100dfcf6b
Merge pull request #1801 from cfromknecht/remove-initial-graph-sync-request
server: Stop requesting initial graph sync
2018-09-05 19:16:56 -07:00
Olaoluwa Osuntokun
1ac1092aaa
lnd: ensure that we set the wallet birthday for --noencryptwallet
In this commit, we ensure that we always set the wallet birthday. If the
user has provided a seed, or is creating a new one, then it will be
overwritten below. However, before this commit, if a user started with
the --noencryptwallet flag, then we would _always_ start to rescan from
genesis with the recent bug fix to ensure that we always start after the
birthday.
2018-09-05 18:58:48 -07:00
Conner Fromknecht
055dc233a1 lnwire/features: add required gq to local features
This commit adds the required feature name to our
set of local known features. This will allow other
peers connecting to us to set the required gossip
queries feature bit. This is required for the
subsequent commits, which instruct the server to
set the bit depending on user configured preferences.
2018-09-05 18:51:04 -07:00
Olaoluwa Osuntokun
aec1e84e3e
Merge pull request #1844 from wpaulino/prune-peer-conn-disk-chans
peer: prune persistent peer connection on zero on-disk channels
2018-09-05 18:45:54 -07:00
Wilmer Paulino
ea51ec34b1
peer: prune persistent peer connection on zero on-disk channels
In this commit, we fix a small bug with regards to the persistent peer
connection pruning logic. Before this commit, it'd be the case that we'd
prune a persistent connection to a peer if all links happen to be
inactive. This isn't ideal, as the channels are still open, so we should
always be atttempting to connect to them. We fix this by looking at the
set of channels on-disk instead and prune the persistent connection if
there aren't any.
2018-09-05 18:30:45 -07:00
Olaoluwa Osuntokun
b0347879db
Merge pull request #1817 from halseth/autopilot-node-update
Reduce goroutine overhead on autopilot updates
2018-09-05 18:08:37 -07:00
Olaoluwa Osuntokun
b2efbce1ac
Merge pull request #1849 from cfromknecht/resync-golock
dep: constrain siphash, x/sys, and errgo.v1 + upd neutrino
2018-09-05 17:58:49 -07:00
Olaoluwa Osuntokun
d2a7d910b7
Merge pull request #1825 from Roasbeef/extra-gossip-message-data
channeldb+discovery: ensure we store, validate and propagate announcements with opaque data
2018-09-05 17:53:02 -07:00
Olaoluwa Osuntokun
146e0efe71
Merge pull request #1802 from wpaulino/equal-zmq-conf-error
config: ensure ZMQ options are not equal and defer creating lnddir until all flag parsing is done
2018-09-05 17:33:40 -07:00
Conner Fromknecht
322e53ec5f
dep: constrain siphash, x/sys, and errgo.v1, upd neutrino 2018-09-05 17:08:59 -07:00
Olaoluwa Osuntokun
785efcfaa2
channeldb: also ignore the EOF error when trying to read ExtraOpaqueBytes
In this commit, we account for the additional case wherein the
announcement hasn't yet been written with the extra zero byte to
indicate that there aren't any remaining bytes to be read. Before this
commit, we accounted for the case where the announcement was written
with the extra byte, but now we ensure that legacy nodes that upgrade
will be able to boot properly.
2018-09-05 16:42:23 -07:00
Olaoluwa Osuntokun
d1a80e35ef
Merge pull request #1841 from cfromknecht/sync-gq-start
peer: Synchronous registration of gossip queries
2018-09-05 16:37:03 -07:00
Johan T. Halseth
baee07ef48
Merge pull request #1846 from roeierez/invoice_amt_paid
rpcserver: amount paid should be in satoshis as other fields used for amount
2018-09-05 12:58:59 +02:00
Roei Erez
f0201fbb32 amount paid should be in satoshies as other fields used for amount 2018-09-05 11:41:48 +03:00
Olaoluwa Osuntokun
16b5a67c3a
Merge pull request #1842 from wpaulino/dedup-chan-updates-horizon
channeldb: dedup channel edges returned from ChanUpdatesInHorizon
2018-09-04 21:24:40 -07:00
Olaoluwa Osuntokun
48072f8e82
channeldb: add limit on the max number of opaque bytes per announcement
In this commit, we add a new limit on the largest number of extra opaque
bytes that we'll allow to be written per vertex/edge. We do this in
order to limit the amount of disk space that we expose, as it's possible
that nodes may start to pad their announcements adding an additional
externalized cost as nodes may need to continue to store and relay these
large announcements.
2018-09-04 20:52:45 -07:00
Olaoluwa Osuntokun
7808288249
chanseries+server: populate ExtraOpaqueData field when reading from disk 2018-09-04 20:52:45 -07:00
Olaoluwa Osuntokun
88aa74be57
discovery: add new set of tests to ensure we validate w/ extra data 2018-09-04 20:52:44 -07:00
Olaoluwa Osuntokun
841e24399c
discovery: ensure we populate ExtraOpaqueData when writing/reading from disk 2018-09-04 20:52:44 -07:00
Olaoluwa Osuntokun
0fc20cee64
discovery: update utils to properly include opaque data in gossip msgs 2018-09-04 20:52:44 -07:00
Olaoluwa Osuntokun
a35bdd4233
channeldb: extend set of graph tests to include opque data where pertinent 2018-09-04 20:52:43 -07:00
Olaoluwa Osuntokun
f5c582d8d7
channeldb: add new ExtraOpaqueData to edge policy+update and node ann
In this commit, we add a mirror set of fields to the ones we recently
added to the set of gossip wire messages. With these set of fields in
place, we ensure that we'll be able to properly store and re-validate
gossip messages that contain a set of extra/optional fields.
2018-09-04 20:52:43 -07:00
Olaoluwa Osuntokun
adde6037c1
lnwire: extend quickcheck parser tests to add extra data for gossip messages 2018-09-04 20:52:42 -07:00
Olaoluwa Osuntokun
941a123ab0 lnwire: add new ExtraOpaqueData field to gossip messages
In this commit, we add a new field to all the existing gossip messages:
ExtraOpqueData. We do this, as before this commit, if we came across a
ChannelUpdate message with a set of optional fields, then we wouldn't be
able to properly parse the signatures related to the message. If we
never corrected this behavior, then we would violate the forwards
compatible principle we use when parsing existing messages.

As these messages can now be padded out to the max message size, we've
increased the MaxPayloadLength value for all of these messages.

Fixes #1814.
2018-09-04 20:52:39 -07:00
Olaoluwa Osuntokun
309e656a97
Merge pull request #1839 from cfromknecht/assume-channel-valid
routing: Assume channel valid
2018-09-04 20:50:06 -07:00
Olaoluwa Osuntokun
554bc314fb
Merge pull request #1821 from wpaulino/prune-edge-update-index
channeldb: properly prune stale entries within edge update index
2018-09-04 20:49:44 -07:00
Olaoluwa Osuntokun
7435fba78b
Merge pull request #1836 from cfromknecht/reconnect-to-all-known-addrs
server: Reconnect to all known addrs for persistent peers
2018-09-04 19:44:35 -07:00
Olaoluwa Osuntokun
de087d73ee
Merge pull request #1774 from cfromknecht/onchain-close-race
contractcourt/channel_arbitrator: handle onchain close race on restart
2018-09-04 19:16:14 -07:00
Olaoluwa Osuntokun
45c607c537
Merge pull request #1840 from cfromknecht/backoff-short-conns
server: always backoff for unstable peers
2018-09-04 19:12:54 -07:00
Olaoluwa Osuntokun
2c8914e7bc
lnwallet: ensure that each backend uses a distinct seed in integration tests
In this commit, we add an additional degree of isolation to the set of
integration tests. A bug was recently fixed to ensure that the wallet
always starts rescans from _after_ it's birthday. In the past it would
miss some funds that were deposited _right_ before the birthday of the
wallet. Fixing this bug exposed a test flake wherein the btcd node would
itself rescan back and collect some of the funds that were last sent to
the bitcoind node.

In order to fix this, we now ensure that each backend will use a unique
HD seed such that the tests are still deterministic for each backend and
role.
2018-09-04 18:52:30 -07:00
Wilmer Paulino
3f58c2dea4
channeldb/graph: dedup channel edges returned from ChanUpdatesInHorizon
In this commit, we ensure that we de-duplicate the set of channel edges
returned from ChanUpdatesInHorizon. Other subsystems within lnd use this
method to retrieve and send all the channels with updates within a time
series to network peers. However, since the method looks at the edge
update index, which can include up to two entries per edge for each
policy, it's possible that we'd send channel announcements and updates
twice, causing extra bandwidth.
2018-09-04 18:48:21 -07:00
Wilmer Paulino
d3cf3168d2
channeldb/graph_test: ensure policies for an edge have different
timestamps

In this commit, we ensure policies for edges we create in
TestChanUpdatesInHorizon have different update timestamps. This ensures
that there are two entries per edge in the edge update index. Because of
this, the test will fail because ChanUpdatesInHorizon will return
duplicate channel edges due to looking at all the entries within the
edge update index. This will be addressed in a future commit to allow
the set of tests to pass once again.
2018-09-04 18:36:25 -07:00
Wilmer Paulino
85ea08fd17
channeldb: add migration to properly prune edge update index
In this commit, we introduce a migration to fix some of the recent
issues found w.r.t. the edge update index. The migration attempts to fix
two things:

1) Edge policies include an extra byte at the end due to reading an
extra byte for the node's public key from the serialized node info.

2) Properly prune all stale entries within the edge update index.

As a result of this migration, nodes will have a slightly smaller in
size channeldb. We will also no longer send stale edges to our peers in
response to their gossip queries, which should also fix the fetching
channel announcement for closed channels issue.
2018-09-04 18:33:43 -07:00
Wilmer Paulino
c1633da252
channeldb/graph_test: extend prune edge update index test to update edges
In this commit, we extend TestChannelEdgePruningUpdateIndexDeletion test
to include one more update for each edge. By doing this, we can
correctly determine whether old entries were properly pruned from the
index once a new update has arrived.
2018-09-04 18:33:41 -07:00
Wilmer Paulino
8dec659e10
channeldb/graph_test: properly check entries within edge update index
Due to entries within the edge update index having a nil value, the
tests need to be modified to account for this. Previously, we'd assume
that if we were unable to retrieve a value for a certain key that the
entry was non-existent, which is why the improper pruning bug was not
caught. Instead, we'll assert the number of entries to be the expected
value and populate a lookup map to determine whether the correct entries
exist within it.
2018-09-04 18:33:40 -07:00
Wilmer Paulino
2f22e6c35f
channeldb/graph: properly determine old update timestamp for an edge
In this commit, we fix a lingering issue within the edge update index
where entries were not being properly pruned due to an incorrect
calculation of the offset of an edge's last update time. Since the
offset is being determined from the end to the start, we need to
subtract all the fields after an edge policy's last update time from the
total amount of bytes of the serialized edge policy to determine the
correct offset. This was also slightly off as the edge policy included
an extra byte, which has been fixed in the previous commit.

Instead of continuing the slicing approach however, we'll switch to
deserializing the raw bytes of an edge's policy to ensure this doesn't
happen in the future when/if the serialization methods change or extra
data is included.
2018-09-04 18:33:38 -07:00
Wilmer Paulino
492d581df6
channeldb/graph: fix off-by-one public key slice
In this commit, we fix an off-by-one error when slicing the public key
from the serialized node info byte slice. This would cause us to write
an extra byte to all edge policies. Even though the values were read
correctly, when attempting to calculate the offset of an edge's update
time going backwards, we'd always be incorrect, causing us to not
properly prune the edge update index.
2018-09-04 18:31:38 -07:00
Wilmer Paulino
06344da62e
channeldb/graph: refactor UpdateEdgePolicy to use existing db transaction 2018-09-04 18:31:37 -07:00
Wilmer Paulino
aa3e2b6ba4
channeldb/graph: identify edge chan id on failure 2018-09-04 18:31:36 -07:00
Conner Fromknecht
8e94e55839
discovery/gossiper: require explict gossip syncer init
This commit removes the fallback in fetchGossipSyncer
that creates a gossip syncer if one is not registered
w/in the gossiper. Now that we register gossip syncers
explicitly before reading any gossip query messages,
this should not longer be required. The fallback also
did not honor the cfg.NoChanUpdates flag, which may
have led to inconsistencies between configuration and
actual behavior.
2018-09-04 17:32:25 -07:00
Conner Fromknecht
51090a41b5
peer: log disconnect to info, remove go-errors pkg 2018-09-04 17:28:51 -07:00
Conner Fromknecht
adf6b8619f
peer: dispatch gossip sync in peer start
This commit moves the gossip sync dispatch
such that it is more tightly coupled to the
life cycle of the peer. In testing, I noticed
that the gossip syncer needs to be dispatched
before the first gossip messages come across
the wire.

The prior spawn location in the server happens
after starting all of the peer's goroutines,
which could permit an ordering where the
gossip syncer has not yet been registered.
The new location registers the gossip syncer
within the read handler such that the call is
blocks before any messages are read.
2018-09-04 17:28:50 -07:00
Conner Fromknecht
2415675c3d
server: move gossip dispatch to peer
See next commit msg for more detail.
2018-09-04 17:28:50 -07:00
Olaoluwa Osuntokun
1217992d9d
autopilot: optimize heavy loaded agent by fetching raw bytes for ChannelEdge
In this commit, we implement an optimization to the autopilot agent to
ensure that we don't spin and waste CPU when we either have a large
graph, or a high max channel target for the agent. Before this commit,
each time we went to read the state of a channel from disk, we would
decompress the EC Point each time. However, for the case of the instal
ChannlEdge struct to feed to the agent, we only actually need to obtain
the pubkey, and can save the potentially expensive point decompression
for each directional channel in the graph.
2018-09-04 16:43:07 -07:00