Commit Graph

142 Commits

Author SHA1 Message Date
Johan T. Halseth
fbfc9a53af
autopilot: TestConstrainedPrefAttachmentNeedMoreChan->TestConstraintsChannelBudget
Since the constraints are no longer something the heuristic needs to be
aware of, we move the test.
2019-01-08 10:10:58 +01:00
Johan T. Halseth
0e1713956b
autopilot/agent: call ChannelBudget on constrainsts
We let the agent call ChannelBudget on its constraints directly, and
not go through the heuristic. This is needed since when we want to have
multiple active heuristics concurrently, it won't make sense anymore to
ask each of the heuristics.

The mockConstraints are also updated to act as the mockHeuristic did
before, by making it possible to control the responses it gives by
sending them on the contained channels.
2019-01-08 10:10:58 +01:00
Johan T. Halseth
1d82e12fcf
autopilot: define AgentConstraints
To decouple the autopilot heuristic from the constraints, we start by
abstracting them behind an interface to make them easier to mock. We
also rename them HeuristicConstraints->AgentConstraints to make it clear
that they are now constraints the agent must adhere to.
2019-01-08 10:10:58 +01:00
Olaoluwa Osuntokun
21460c9e67
Merge pull request #2293 from halseth/unit-tests-weighted-choice
[autopilot] weighted choice unit tests and optimizations
2019-01-07 19:42:29 -08:00
Johan T. Halseth
dbf7b380a9
autopilot/manager: only set m.pilot if started successfully
This commit fixes a subtle bug within the autopilot manager, that would
cause the active pilot to not be reset in case it wasn't started
successfully.

We also make sure the associated goroutines close over the started
pilot, and not the active pilot.
2018-12-19 19:48:44 +01:00
Johan T. Halseth
902d6edad2
autopilot/choice: add weightedChoice tests 2018-12-19 08:27:35 +01:00
Johan T. Halseth
4ac3c171ec
autopilot/choice: avoid costly map allocations
This commit makes the weightedChoice algorithm take a slice of weights
instead of a map of node scores. This let us avoid costly map allocation
and iteration.

In addition we make the chooseN algorithm keep track of the remaining
nodes by keeping a slice of weights through its entire run, similarly
avoiding costly map allocation and iteration.

In total this brings the runtime of the TestChooseNSample testcase down
from ~73s to ~3.6s.
2018-12-19 08:27:34 +01:00
Johan T. Halseth
3d2a39a18c
autopilot/choice_test: add chooseN tests 2018-12-19 08:27:34 +01:00
Johan T. Halseth
a202860ce1
autopilot: make n uint32 in chooseN
To avoid negative values being input, as it doesn't really make sense.
2018-12-19 08:27:34 +01:00
Johan T. Halseth
6310ed0f1c
autopilot: add autopilot.Manager
This commit adds a new type Manager responsible for managing an
autopilot agent, making it possible to start and stop it at runtime.
2018-12-13 12:26:29 +01:00
Olaoluwa Osuntokun
d24a6d2a3d
Merge pull request #2304 from halseth/weigthed-choice-errnopositive
[autopilot] Don't fail on no more positive nodes
2018-12-12 12:21:06 -08:00
Johan T. Halseth
8e57ed7025
autopilot/agent_test: let agent self-trigger retry
This ensures that it is done updating its pending connections map before
re-querying the heuristic.
2018-12-11 10:21:40 +01:00
Johan T. Halseth
dda7355ad6
autopilot/agent_test: ensure mochHeuristics exits cleanly 2018-12-11 10:20:38 +01:00
Johan T. Halseth
0255db5120
autopilot/prefattach_test: ensure non-connected nodes get 0-score
This addition to the unit tests makes sure nodes that have no channels
in the graph are left out od the scored nodes, implicitly giving them a
score of 0.
2018-12-10 15:05:24 +01:00
Johan T. Halseth
b9c33da3f8
autopilot/prefattach: skip nodes having no channels 2018-12-10 15:01:43 +01:00
Johan T. Halseth
aeeff4714d
autopilot/graph: define addRandNode 2018-12-10 15:01:43 +01:00
Johan T. Halseth
5224c94aaf
autopilot/prefattach_test: assert scored nodes having addresses
This addtion to the test makes sure the scored nodes have their
addresses populated. This would fail without the previous commit.
2018-12-10 15:01:05 +01:00
Johan T. Halseth
d9eef4625e
autopilot/prefattach: populate Addrs field 2018-12-10 15:00:41 +01:00
Johan T. Halseth
55ba5fe654
autopilot/graph: add addresses to memNode 2018-12-10 14:56:54 +01:00
Johan T. Halseth
40db2dd5a5
autopilot/choice: return ErrNoPositive in case no choice can be made 2018-12-10 13:53:36 +01:00
Johan T. Halseth
f1e8c8d5b5
autopilot/agent: move choice algorithms to new file choice.go 2018-12-10 13:53:31 +01:00
Johan T. Halseth
6130189d95
autopilot/interface+agent: remove Select 2018-12-06 14:26:28 +01:00
Johan T. Halseth
b3d315298c
autopilot/agent: use NodeScores to select channel candidates
This commit makes the autopilot agent use the new NodeScores heuristic
API to select channel candiates, instead of the Select API. The result
will be similar, but instead of selecting a set of nodes to open
channels to, we get a score based results which can later be used
together with other heuristics to choose nodes to open channels to.

This commit also makes the existing autopilot agent tests compatible
with the new NodeScores API.
2018-12-06 14:26:26 +01:00
Johan T. Halseth
e84bd29836
autopilot/agent: add weightedChoice and chooseN algorithm
The algorithms will be used to select nodes at random from the weighted
distribution set by the node's scores given by the heuristic.
2018-12-06 14:26:05 +01:00
Johan T. Halseth
be45697c6d
autopilot/prefattach_test: use NodeScores API for prefAttach tests
This commit converts the existing unit tests of Select into tests of
NodeScores.
2018-12-06 14:26:05 +01:00
Johan T. Halseth
5e8e54083f
autopilot/prefattach+interface: add API NodeScores
This commit adds a new method NodeScores to the AttachementHeuristic
interface. Its intended use is to score a set of nodes according to
their preference as channel counterparties.

The PrefAttach heuristic gets a NodeScores method that will score the
ndoes according to their number of already existing channels, similar to
what is done already in Select.
2018-12-06 14:26:05 +01:00
Johan T. Halseth
5ecc209c41
autopilot/interface+agent: remove NodeKey from AttachmentDirective
Instead parse the pubkey bytes only when needed.
2018-12-06 14:26:05 +01:00
Johan T. Halseth
89c3c5319f
autopilot/agent: split opening logic into own method
This commit takes the logic after the autopilot agent has decided that it
needs to open more channels, and moves it into a new method openChan.
2018-12-06 14:26:03 +01:00
Johan T. Halseth
26810fe928
autopilot/agent: split attachement directive attempts into method 2018-12-06 14:25:22 +01:00
Johan T. Halseth
fb10175ea5
autopilot/agent: add maps to agent struct
This commit moves the maps used by the controller loop to the Agent
struct, in preparation for breaking it up into smaller parts.
2018-12-06 14:24:18 +01:00
Johan T. Halseth
86e6d230f2
autopilot/agent: add attachment directive goroutine to wait group 2018-12-06 14:24:15 +01:00
Johan T. Halseth
35f4ec84d1
autopilot/prefattach: use HeuristicConstraints
This commit makes the pref attach heuristic and the agent use the
HeuristicConstraints internally.
2018-12-06 11:25:30 +01:00
Johan T. Halseth
e98d859882
autopilot: define HeuristicConstraints
This commit defines a new struct HeuristicConstraints that will be used
to keep track of the initial constraints the autopilot agent needs to
adhere to. This is currently done in the ConstrainedPrefAttachement
heuristic itself, but this lets us share these constraints and common
method netween several heuristics.
2018-12-06 11:20:09 +01:00
Olaoluwa Osuntokun
1fd3aac925
multi: switch from bolt packge to bbolt package for all imports 2018-11-29 20:33:49 -08:00
ErikEk
c5f61b3c06 Typo fix autopilot 2018-10-09 18:28:34 +02:00
Olaoluwa Osuntokun
3fda1029c0
Merge pull request #1786 from Roasbeef/golang-1.1
build+docs: bump to golang 1.11
2018-10-08 13:26:07 +09:00
Olaoluwa Osuntokun
d8e6085c17
multi: update to go 1.11 gofmt 2018-10-07 14:37:38 +09:00
Conner Fromknecht
59b459674d
multi: init subsystem loggers via build pkg 2018-10-05 13:04:45 +09:00
Conner Fromknecht
d5f97f7bdc
autopilot/multi: replace PubKey -> NodeKey on directive 2018-09-05 20:10:37 -07:00
Conner Fromknecht
e702a6a266
autopilot/agent_test: remove Fatalf calls from goroutines 2018-09-05 20:10:36 -07:00
Conner Fromknecht
2b578e06fc
autopilot/agent_test: ensure directives use unique keys
This commit ensures that the mock attachment
directives use unique keys, ensuring that they
aren't skipped due to already having pending
connection requests. The tests fail when
they're all the same since they collide
in the pendingConns map.
2018-09-05 20:10:35 -07:00
Conner Fromknecht
7d9483c20d
autopilot/agent_test: adds TestAgentSkipPendingConns
Adds a test asserting that the agent prevents
itself from making duplicate outstanding
connection requests to the same peer.
2018-09-05 20:10:35 -07:00
Conner Fromknecht
c214bea9db
autopilot/prefattach: set NodeID for selection candidates 2018-09-05 20:10:34 -07:00
Conner Fromknecht
08b6bf54fb
autopilot/interface: expose NodeID in AttachmentDirective 2018-09-05 20:10:34 -07:00
Conner Fromknecht
4599b0eac3
autopilot/agent: track pending connections
This commit modifies the autopilot agent to track
all pending connection requests, and forgo further
attempts if a connection is already present.
Previously, the agent would try and establish
hundreds of requests to a node, especially if the
connections were timing out and not returning.

This resulted in an OOM OMM when cranking up
maxchannels to 200, since there would be close
to 10k pending connections before the program was
terminated. The issue was compounded by periodic
batch timeouts, causing autopilot to try and
process thousands of triggers for failing
connections to the same peer.

With these fixes, autopilot will skip nodes that we
are trying to connect to during heuristic selection.
The CPU and memory utilization have been significantly
reduced as a result.
2018-09-05 20:10:34 -07:00
Olaoluwa Osuntokun
b0347879db
Merge pull request #1817 from halseth/autopilot-node-update
Reduce goroutine overhead on autopilot updates
2018-09-05 18:08:37 -07:00
Olaoluwa Osuntokun
1217992d9d
autopilot: optimize heavy loaded agent by fetching raw bytes for ChannelEdge
In this commit, we implement an optimization to the autopilot agent to
ensure that we don't spin and waste CPU when we either have a large
graph, or a high max channel target for the agent. Before this commit,
each time we went to read the state of a channel from disk, we would
decompress the EC Point each time. However, for the case of the instal
ChannlEdge struct to feed to the agent, we only actually need to obtain
the pubkey, and can save the potentially expensive point decompression
for each directional channel in the graph.
2018-09-04 16:43:07 -07:00
Johan T. Halseth
0d4df54118
autopilot/agent: signal balanceUpdates on own channel 2018-09-04 10:18:15 +02:00
Johan T. Halseth
a9a9c9aeb4
autopilot/agent: signal chanPendingOpenUpdates on own channel 2018-09-04 10:17:59 +02:00
Johan T. Halseth
186e6d4da4
autopilot/agent: signal chanOpenFailureUpdates on own channel
We do this to avoid a huge amount of goroutines piling up when autopilot
is trying to open many channels, as they will all block trying to send
the update on the stateUpdates channel. Now we instead send them on a
buffered channel, similar to what is done with the nodeUpdates.
2018-09-04 10:17:33 +02:00
Johan T. Halseth
4a88c61a90
autopilot/agent: signal nodeUpdates on own channel
We do this to avoid a huge amount of goroutines piling up on initial
graph sync, as they will all block trying to send the node update on the
stateUpdates channel. Now we instead make a new buffered channel
nodeUpdates, and just return immediately if there is already a signal in
the channel waiting to be processed.
2018-09-04 10:17:09 +02:00
Johan T. Halseth
3e992f094d
autopilot/agent: move signal processing out of select 2018-09-04 10:06:15 +02:00
Olaoluwa Osuntokun
4f43c1c943
Merge pull request #1811 from Roasbeef/autopilot-cpu-usage-fix
autopilot: modify the graph interface to return raw bytes for node pubkeys, not entire key
2018-09-03 19:56:51 -07:00
Olaoluwa Osuntokun
a429c56c10
autopilot: update the Node interface to return a raw bytes, not the key
In this commit, we modify the Node interface to return a set of raw
bytes, rather than the full pubkey struct. We do this as within the
package, commonly we only require the pubkey bytes for fingerprinting
purposes. Before this commit, we were forced to _always_ decompress the
pubkey which can be expensive done thousands of times a second.
2018-08-29 15:44:47 -07:00
Wilmer Paulino
e1a376d9f8
autopilot: use updateBalance rather than tracking balance explicitly
In this commit, we modify the balanceUpdate autopilot signal to update
the balance according to what's returned to the WalletBalance callback
rather than explicitly tracking the balance. This gives the agent a
better sense of what the wallet's balance actually is.
2018-08-29 02:06:02 -07:00
Conner Fromknecht
1898e57807
autopilot/agent_test: test that agent recovers from initial empty graph 2018-08-23 18:58:12 -07:00
Conner Fromknecht
602856750b
autopilot/agent: add OnNodeUpdates signal
Adds a new external signal alerting autopilot that
new nodes have been added to the channel graph or
an existing node has modified its channel
announcment. This allows autopilot to examine its
current state, and attempt to open channels if our
target state is not yet met.
2018-08-23 18:58:12 -07:00
Wilmer Paulino
aa7c2e6d47
autopilot: add OnChannelPendingOpen state update 2018-08-16 20:22:40 -07:00
Wilmer Paulino
956acdfa14
autopilot/agent_test: pass in agent's quit chan to heuristic
In this commit, we alter our mock heuristic to also take in a quit chan.
It's possible that at the end of a test the agent is blocked on a
NeedMoreChans/Select call as their mock implementations use channels. To
prevent this, we use the agent's quit chan so that the heuristic can
safely exit once the agent does.
2018-08-16 20:19:33 -07:00
Wilmer Paulino
dd5b6394d9
autopilot: read quit chan when sending state updates 2018-08-16 20:19:32 -07:00
Wilmer Paulino
454b549c7e
autopilot+pilot: refactor connection logic out of OpenChannel
In this commit, we refactor the existing connection logic outside of the
ChanController's OpenChannel method. We do this as previously it was
possible for peers to stall us while attempting to connect to them. In
order to remedy this, we now attempt to connect the peer before tracking
them in our set of pending opens.
2018-08-16 20:17:03 -07:00
Wilmer Paulino
2fbe95ebba
autopilot: modify OpenChannel to no longer take in set of addrs 2018-08-16 19:19:30 -07:00
Wilmer Paulino
89325a13b7
autopilot+config: support opening private channels with autopilot agent 2018-08-10 20:45:59 -07:00
Joost Jager
6c918a1806 channeldb: store unknown policies in database
The commit ensures that for every channel, there will always
be two entries in the edges bucket. If the policy from one or
both ends of the channel is unknown, it is marked as such.

This allows efficient lookup of incoming edges. This is
required for backwards payment path finding.
2018-08-09 20:23:40 -07:00
Conner Fromknecht
1ded697e8d
multi: sort import paths with gofmt 2018-08-02 18:20:49 -07:00
Olaoluwa Osuntokun
6f60f139f4 multi: switch over import paths from roasbeef/* to btcsuite/* 2018-07-13 17:05:39 -07:00
maurycy
3be08e69cf multi: 64bit aligment of atomic vars on arm/x86-32 2018-06-04 20:02:34 -07:00
practicalswift
663c396235 multi: fix a-vs-an typos 2018-04-17 19:02:04 -07:00
Olaoluwa Osuntokun
800eea931f
build+multi: switch from bolt to bbolt
In this commit, we switch from boltbd/bolt to coreos/bbolt as the
former is no longer being actively maintained.
2018-03-10 19:01:13 -08:00
Olaoluwa Osuntokun
00ab66469d
autopilot: refresh available funds on channel open/close events 2018-03-10 16:53:02 -08:00
Olaoluwa Osuntokun
ce2d5a2156
autopilot: limit the number of outstanding channel open goroutines
In this commit, we fix an existing bug that would at times cause us to
spiral out of control and potentially created thousands of outbound
connections. Our fix is simple: limit the total number of outstanding
channel establishment attempts. Without this limit, we have no way to
bound the number of active goroutines.

Fixes #772.
2018-03-10 16:53:02 -08:00
Olaoluwa Osuntokun
9f52372cd2
autopilot: modify interfaces to specify *exactly* how many chans to open
In this commit, we fix a regression introduced by a recent change which
would allow the agent to detect a channel as failed, and blacklist the
node, promising faster convergence with the ideal state of the
heuristic.

The source of this bug is that we would use the set of blacklisted
nodes in order to compute how many additional channels we should open.
If 10 failures happened, then we would think that we had opened up 10
channels, and stop much earlier than we actually should.

To fix this, while ensuring we don’t retry to failed peers, the
NeedMoreChans method will now return *how* anymore channels to open,
and the Select method will take in how many channels it should try to
open *exactly*.
2018-02-08 20:08:18 -08:00
MeshCollider
61414df669 multi: Fix two minor typos 2018-02-08 15:21:04 -08:00
Olaoluwa Osuntokun
cb48a5827a
autopilot: update API usage to account for recent channeldb changes 2018-02-06 20:14:32 -08:00
Daniel McNally
8543497dcc multi: fixing it's/its typos in comments 2018-02-06 19:13:07 -08:00
practicalswift
a93736d21e multi: comprehensive typo fixes across all packages 2018-02-06 19:11:11 -08:00
Simon Horlick
73468cf06e autopilot: Exclude nodes that previously failed 2018-01-27 18:35:31 -08:00
Simon Horlick
a43d7d3532 autopilot: Make autopilot aware of channel open failures 2018-01-27 18:35:31 -08:00
Matt Drollette
adf0d98194 multi: fix several typos in godoc comments 2017-12-17 18:40:05 -08:00
Olaoluwa Osuntokun
2e2b262999
autopilot: ensure access to pendingOpens in thread-safe when logging 2017-12-10 15:46:49 -08:00
Olaoluwa Osuntokun
9bcb139bcc
autopilot: shuffle set of candidates using Fisher–Yates before selecting
In this commit we modify the ConstrainedPrefAttachment.Select method to
first shuffle the set of potential candidates before selecting them.
This serves to remove the existing grouping between candidates which
may have influenced the selection.
2017-10-30 18:55:55 -07:00
Jim Posen
9fd77a6e40 multi: Update lnd to use new feature vector API. 2017-10-17 22:47:20 -07:00
Olaoluwa Osuntokun
0769cae565
autopilot: remove spammy trace log 2017-09-03 16:46:56 -07:00
Olaoluwa Osuntokun
8d88f98297
autopilot: update graph.go to match recent API changes 2017-08-22 00:52:31 -07:00
Olaoluwa Osuntokun
77e3e74ea2
autopilot: for ConstrainedPrefAttachment.Select factor in skipNodes
This commit modifies the Select method for the
ConstrainedPrefAttachment attachment heuristic slightly. Previously, it
was possible for an autopilot.Agent to go over the allotted number of
channels as it would unconditionally attempt to establish channel with
all returned Attachment Directives. To remedy this, we now assume that
we already have active, or pending channels to each of the nodes in the
set of skipNodes. Therefore, we now use the size of the skipNodes map
as an  upper limit within the primary selection loop.
2017-08-22 00:51:43 -07:00
Olaoluwa Osuntokun
a8576eaa69
autopilot: fix bug in skip nodes test, use new instance for each run 2017-08-22 00:51:39 -07:00
Olaoluwa Osuntokun
804edb8f02
autopilot: ensure all access to pendingOpens is thread-safe
In this commit, we ensure that we grab the mutex for the pending open
channel state when we attempt to merge the pending state with the
committed state.
2017-08-22 00:51:33 -07:00
Olaoluwa Osuntokun
09521b9c2a
autopilot: fix linter error 2017-08-15 18:26:59 -07:00
Olaoluwa Osuntokun
711b695a2f
autopilot: add tracking of pending channel state
This commit adds tracking of the pending channels state within the
autopilot.Agent. This fixes a class of bugs which was discovered during
the latest test net block storm wherein the Agent would attempt to
repeatedly attach to the same node due to rapid closure of other
channels.

In this commit we fix this issue by ensuring that we always factor in
the pending channel state when querying the heuristic w.r.t if we need
more channels, and if so to which nodes should be attached to.
2017-08-15 18:23:58 -07:00
Olaoluwa Osuntokun
8034780ca4
autopilot: fix chanID collision test flake by instead using atomic counter
This commit fixes a prior occasional test flake caused by the collision
of the randomly selected 64-bit integers. In order to get around this,
we now instead have a atomic monotonically increasing counter for each
channel ID used within the tests.
2017-08-15 17:56:24 -07:00
Olaoluwa Osuntokun
48be9261ef
autopilot: fix linter error 2017-08-10 22:07:22 -07:00
Olaoluwa Osuntokun
306c4aef8e
autopilot: "Look ma no hands!", introducing autopilot mode
This commit introduces the initial implementation of the autopilot
mode. Autopilot is new mode within lnd that enables automatic channel
management. This means that if enabled lnd will attempt to
automatically manage channels according to a set of heuristic defined
within the main configuration for autopilot.Agent instance.

The autopilot.Agent implements a simple closed control loop. It takes
in external signals such as wallet balance updates, new open channel,
and channels that are now closed the updates its internal state. With
each external trigger it will consult the registered
AttachmentHeuristic to decide: if it needs to open any more channels,
and if so how much it should use to open the channels, ultimately
returning a set of recommended AttachmentDirectives. The
autopilot.Agent loop will then take those attempt to establish
connection, and go back in waiting for a new external signal.

With this first implementation the default heuristic is the
ConstrainedPrefAttachment implementation of AttachmentHeuristic. Given
a min and max channel size, a limit on the number of channels, and the
percentage of wallet funds to allocate to channels, it will attempt to
execute a heuristic drive by the Barabási–Albert model model in order
to attempt to drive the global graph towards a scale free topology.

This is commit implements a foundational layer for future simulations,
optimization, and additional heuristics.
2017-08-10 21:14:55 -07:00