Commit Graph

549 Commits

Author SHA1 Message Date
Olaoluwa Osuntokun
85c0dd3de0
routing/chainview: update to latest API changes 2018-08-24 03:36:25 -07:00
Wilmer Paulino
191c4f3e28
multi: enable txindex on miner harness 2018-08-24 03:36:25 -07:00
Johan T. Halseth
4b04e1afd1
routing: return ErrRejected if found in reject cache
We make sure to return an error other than ErrIgnored, as ErrIgnored is
expected to only be returned for updates where we already have the
necessary information in the database.

In case of a channel ID found in the rejectCache, there was a
possibility that we had rejected an invalid update for this channel
earlier, and when attempting to add the current update we wouldn't
distinguish the failure to add from an outdated/ignored update.
2018-08-23 07:40:40 +02:00
Johan T. Halseth
18e680f192
router: simplify edge outdated logic, return ErrOutdated if old. 2018-08-23 07:40:39 +02:00
Wilmer Paulino
6c903393e5 router+server: use first hop channel id rather than node key 2018-08-20 21:04:52 -07:00
Olaoluwa Osuntokun
518b0d2cf7
Merge pull request #1679 from cfromknecht/move-linter-v2
Makefile: move linter v2
2018-08-09 21:03:05 -07:00
Olaoluwa Osuntokun
3438baaf0c
routing: eliminate nested db transactions by using new FetchOtherNode method 2018-08-09 20:48:06 -07:00
Joost Jager
29b6bae4ed routing: convert fee limit test to table-driven 2018-08-09 20:23:40 -07:00
Joost Jager
3e7473f4f0 routing: backward searching 2018-08-09 20:23:40 -07:00
Joost Jager
e52d829168 routing: path finding test refactored 2018-08-09 20:23:40 -07:00
Joost Jager
6c918a1806 channeldb: store unknown policies in database
The commit ensures that for every channel, there will always
be two entries in the edges bucket. If the policy from one or
both ends of the channel is unknown, it is marked as such.

This allows efficient lookup of incoming edges. This is
required for backwards payment path finding.
2018-08-09 20:23:40 -07:00
Wilmer Paulino
9d2eeb6304
multi: update to latest fee estimation interface 2018-08-09 17:29:52 -07:00
Wilmer Paulino
936fcc1f16
multi: update to latest BitcoindClient interface
In this commit, we introduce a nice optimization with regards to lnd's
interaction with a bitcoind backend. Within lnd, we currently have three
different subsystems responsible for watching the chain: chainntnfs,
lnwallet, and routing/chainview. Each of these subsystems has an active
RPC and ZMQ connection to the underlying bitcoind node. This would incur
a toll on the underlying bitcoind node and would cause us to miss ZMQ
events, which are crucial to lnd. We remedy this issue by sharing the
same connection to a bitcoind node between the different clients within
lnd.
2018-08-08 20:26:30 -07:00
Conner Fromknecht
a76b9a86cd
routing/chainview/interface_test: use EdgePoint field names 2018-08-07 16:57:40 -07:00
Conner Fromknecht
9eff876f52
routing/pathfind_test: fmt whitespace 2018-08-02 18:20:49 -07:00
Conner Fromknecht
614b3b3862
routing/pathfind: fmt whitespace 2018-08-02 18:20:49 -07:00
Conner Fromknecht
1ded697e8d
multi: sort import paths with gofmt 2018-08-02 18:20:49 -07:00
Olaoluwa Osuntokun
3c60861005
routing/chainview: give neutrino backend more time to catch up for re-org test
In this commit, we modify the test to explitlcy give the neutrino
backend more time to catch up compared to the RPC backends. We do this
as a recent change has been made in the neutrino backend to wait for the
filter headers to finish syncing before proceeding with the rescan
itself. As a result, we'll need to account for this in the test and
sleep enough to give the backend a chance to catch up.
2018-07-31 21:29:03 -07:00
Olaoluwa Osuntokun
8c0a151cec
routing: update UTXO verification to use new interface, graph pruning too 2018-07-31 21:28:55 -07:00
Olaoluwa Osuntokun
fb98f59407
routing/chainview: ensure bitcoind impl meets new interface requirements 2018-07-31 21:28:55 -07:00
Olaoluwa Osuntokun
9b9029cf4b
routing/chainview: ensure btcd impl meets new interface requirements 2018-07-31 21:28:55 -07:00
Olaoluwa Osuntokun
7e6997d6d3
routing/chainview: update neutrino API due to recent changes
In this commit, we ensure that the neutrino backend meets the target
interface, and also we update the API usage for the internal neutrino
rescan struct to use the new InputWithScript struct.
2018-07-31 21:28:54 -07:00
Olaoluwa Osuntokun
ec0fb9c537
routing/chainview: modify the UpdateFilter method to take a channeldb.EdgePoint
In this commit, we update the existing UpdateFilter method to take the
new channeldb.EdgePoint struct in place of the prior wire.OutPoint. We
must do this as the caller now typically has this type due to the
preparation to enable lnd to be able to be compatible with the new
neutrino protocol.
2018-07-31 21:28:54 -07:00
Olaoluwa Osuntokun
a6c814010c
routing: exit gracefully if generateSphinxPacket is passed a nil set of hops 2018-07-30 13:41:06 -07:00
Olaoluwa Osuntokun
c903a9a711
routing: add new TestEmptyRoutesGenerateSphinxPacket test 2018-07-30 13:40:25 -07:00
Olaoluwa Osuntokun
e0baa49690
Merge pull request #1387 from wpaulino/send-disable-chan-update
multi: send a channel update with disabled flag set on channel close
2018-07-23 17:46:37 -07:00
Olaoluwa Osuntokun
ef56f76200
routing: demote channel update logging to trace 2018-07-23 17:01:39 -07:00
Olaoluwa Osuntokun
2e75499787
channeldb+routing: move adding shell nodes into db txn of AddChannelEdge
In this commit, we fix a slight race condition that can occur when we go
to add a shell node for a node announcement, but then right afterwards,
a new block arrives that causes us to prune an unconnected node. To
ensure this doesn't happen, we now add shell nodes within the same db
transaction as AddChannelEdge. This ensures that the state is fully
consistent and shell nodes will be added atomically along with the new
channel edge.

As a result of this change, we no longer need to add shell nodes within
the ChannelRouter, as the database will take care of this operation as
it should.
2018-07-22 21:02:53 -07:00
Olaoluwa Osuntokun
f8cbe34e93
routing: prune nodes from the channel graph on start up 2018-07-21 19:52:26 -07:00
Olaoluwa Osuntokun
505544e1b3
routing/chainview: update API usage for neutrino due to recent changes 2018-07-13 17:15:12 -07:00
Olaoluwa Osuntokun
6f60f139f4 multi: switch over import paths from roasbeef/* to btcsuite/* 2018-07-13 17:05:39 -07:00
Wilmer Paulino
191e111661
routing: include disabled field in topology updates 2018-07-11 17:07:02 -07:00
Olaoluwa Osuntokun
16df90fc23
Merge pull request #1504 from cfromknecht/chainview-filter-locking
[routing/chainview] improve filter locking granularity in btcd/bitcoind
2018-07-10 17:19:36 -07:00
maurycy
3915c6a569 routing: do not reacquire the lock for topology notifications 2018-07-10 17:09:52 -07:00
Olaoluwa Osuntokun
e13f96e64b
routing: ensure we create a pruneViewSnapshot for route sessions
In this commit, we fix an existing bug that could at times lead to a
panic if a user manually crafts a route via SendToRoute, and that route
results in a payment error. The fix is simple: create the map even
though it won't be used in the sessions since the user is feeding the
router manual routes.
2018-07-08 16:34:25 -07:00
Conner Fromknecht
daa28e0d0f
routing/chainview/bitcoind: improve filterMtx locking
In this commit, we modify the granularity of the locking
around the filterMtx in the bitcoind chainview, such that
we only lock once per block connected or filter update.
Currently, we acquire and release the lock for every
update to the map.

We also fix a bug that would cause us to not fully remove
all previous outpoints spent by a txn when doing manual
filter, as we previously would only remove the first output
detected.
2018-07-06 19:08:17 -07:00
Conner Fromknecht
0f6bc3538d
routing/chainview/btcd: improve filterMtx locking
In this commit, we modify the granularity of the locking
around the filterMtx in the btcd chainview, such that we
only lock once per block connected or filter update.
Currently, we acquire and release the lock for every
update to the map.

We also fix a bug that would cause us to not fully remove
all previous outpoints spent by a txn when doing manual
filter, as we previously would only remove the first output
detected.
2018-07-06 19:08:17 -07:00
Olaoluwa Osuntokun
39c1f46f85
routing: new newLogClosure to delay spew evaluation until log print time
In this commit, we update the generateSphinxPacket to use newLogClosure
to delay the spew evaluation until log print time. Before this commit,
even if we weren't on the trace logging level, the spew call would
always be evaluated.
2018-06-30 17:14:28 -05:00
Olaoluwa Osuntokun
c045defd0f
Merge pull request #1358 from joostjager/newfee
routing: routing may come up with suboptimal routes (weight function)
2018-06-28 19:18:13 -07:00
Joost Jager
b52796749f routing: new weight function
In this commit, a new weight function is introduced. This will create a
meaningful effect of time lock on route selection. Also, removes the
squaring of the fee term. This led to suboptimal routes.

Unit test added that covers the weight function and asserts that the
lowest fee route is indeed returned.
2018-06-28 14:52:39 +02:00
Joost Jager
416f368979 routing: add time lock asserts to NewRoute test
This comment extends the unit tests for NewRoute with checks
on the total time lock for a route as well as the expected time
lock values for every hop along the route.
2018-06-28 14:04:02 +02:00
Joost Jager
d4471878df routing: fix route fee calculation and channel capacity check
This commit fixes the logic inside the newRoute function to
address the following problems:

- Fee calculation for a hop does not include the fee that needs
  to be paid to the next hop.

- The incoming channel capacity "sanity" check does not include
  the fee to be paid to the current hop.
2018-06-28 13:53:45 +02:00
Olaoluwa Osuntokun
6a6c29cd00
routing: fix incorrect expiry values in spec_example.json
In this commit, we fix the incorrect expiry values in the
spec_example.json test file. Many of the time locks were incorrect which
allowed bugs within the path finding logic related to CLTV deltas to go
un-detected.
2018-06-26 17:29:42 -07:00
Olaoluwa Osuntokun
6c986864ff
routing: fix bug in newRoute, use time lock delta of prior hop, not current
In this commit, we fix an existing bug in the newRoute method. Before
this commit we would use the time lock delta of the current hop to
compute the outgoing time lock for the current hop. This is incorrect as
the time lock delta of the _outgoing_ hop should be used, as this is
what we're paying for "transit" on. This is a bug left over from when we
switched the meaning of the CLTV delta on the ChannelUpdate message
sometime last year.

The fix is simple: use the CLTV delta of the prior (later in the route)
hop.
2018-06-26 17:29:42 -07:00
Wilmer Paulino
ddf8f2cb01
routing: allow specifying a fee limit during route construction 2018-06-12 18:14:48 -07:00
Wilmer Paulino
6a6de812ba
routing: denote a payment's fee limit in millisatoshis instead 2018-06-12 18:14:47 -07:00
Sebastian Delgado
6746609ec6
Add constant and percentage-based fee limits to payments
- Extend SendRequest and QueryRoutesRequest protos
- newRoute function takes fee limit and cuts off routes that exceed it
- queryRoutes, payInvoice and sendPayment commands take the feeLimit inputs and pass them down to newRoute
- When no feeLimit is included, don't enforce any feeLimits at all (by setting feeLimit to maxValue)
2018-06-12 18:14:43 -07:00
Olaoluwa Osuntokun
eb65b0cd5a
routing: check to see if pubkey is nil before logging in sendPayment
In this commit, we now need to check to see if the pubkey is nil as if
the user specified a set of manual routes, then the dest pubkey would be
empty.
2018-06-06 20:45:56 -07:00
Olaoluwa Osuntokun
0113035f6d
routing: add new NewPaymentSessionFromRoutes function instead of interface
In this commit, we modify the recent refactoring of the mission control
sub-system to overload the existing payment session, rather than create
a brand new one. This allows us to re-use more of the existing logic, and
also feedback into mission control the failures incurred by any user
selected routes.
2018-06-06 20:45:56 -07:00
t4sk
53eb396ae9 routing/router: add SendToRoute 2018-06-06 20:45:00 -07:00
maurycy
3be08e69cf multi: 64bit aligment of atomic vars on arm/x86-32 2018-06-04 20:02:34 -07:00
Olaoluwa Osuntokun
f494433cbf
routing+server: add new QueryBandwidth method to reduce outbound failures
In this commit, we introduce a new method to the channel router's config
struct: QueryBandwidth. This method allows the channel router to query
for the up-to-date available bandwidth of a particular link. In the case
that this link emanates from/to us, then we can query the switch to see
if the link is active (if not bandwidth is zero), and return the current
best estimate for the available bandwidth of the link. If the link,
isn't one of ours, then we can thread through the total maximal
capacity of the link.

In order to implement this, the missionControl struct will now query the
switch upon creation to obtain a fresh bandwidth snapshot. We take care
to do this in a distinct db transaction in order to now introduced a
circular waiting condition between the mutexes in bolt, and the channel
state machine.

The aim of this change is to reduce the number of unnecessary failures
during HTLC payment routing as we'll now skip any links that are
inactive, or just don't have enough bandwidth for the payment. Nodes
that have several hundred channels (all of which in various states of
activity and available bandwidth) should see a nice gain from this w.r.t
payment latency.
2018-05-14 16:23:54 -07:00
Conner Fromknecht
1b1f5f197a
routing/chainview/neutrino: cache filter entries
This commit alters the neutrino chainview such that it
caches the filter entries corresponding to watched
outpoints at the moment they are added to the filter.
Previously, we would rederive each filter entry when
reconstructing the relevant filter entries, which
would lead to unnecessary work on the gc. Now, each is
created at most once, and reused across subsequent
reconstructions.
2018-05-08 20:46:46 -07:00
Conner Fromknecht
99e7ec0895
routing/validation_barrier_test: test semaphore and quit
Adds test checking the basic functionality of the barrier's
semaphore and behavior during shutdown.
2018-05-08 16:32:08 -07:00
Conner Fromknecht
995e3fa85f
routing/validation_barrier: adds ErrVBarrierShuttingDown
Adds a new error ErrVBarrierShuttingDown that is returned
from WaitForDependants if the validation barrier's quit
chan is closed. This allows any blocked goroutines to
distinguish whether the dependent task has been completed,
or if validation should be aborted entirely.
2018-05-08 16:32:08 -07:00
Conner Fromknecht
eaa8cdf916
routing/router: improve validation barrier shutdown
This commit improves the shutdown of the router's
pending validation tasks, by ensuring the pending
tasks exit early if the validation barrier
receives a shutdown request.

Currently, any goroutines blocked by WaitForDependants
will continue execution after a shutdown is signaled.
This may lead to unnexpected behavior as the relation
between updates is no longer upheld. It also has the
side effect of slowing down shutdown, since we
continue to process the remaining updates.

To remedy this, WaitForDependants now returns an error
that signals if a shutdown was requested. The blocked
goroutines can exit early upon seeing this error,
without also signaling completion of their task to
the dependent tasks, which should will now properly
wait to read the validation barrier's quit signal.
2018-05-08 16:32:08 -07:00
Kenneth Perry
f7c5a7a19e routing/chainview/neutrino: error if we get a nil cfilter back (#1176) 2018-05-03 15:21:56 -07:00
Olaoluwa Osuntokun
76b9501e04
routing: update TestSendPaymentErrorPathPruning due to recent UnknownPeer change
In this commit, we update the TestSendPaymentErrorPathPruning test to
reflect the new behavior w.r.t how we respond to UnknownPeer errors. In
this new test, we expect that we'll find alternative route in light of
us getting an UnknownPeer error "pointing" to our destination node.
2018-04-23 17:53:46 -07:00
Olaoluwa Osuntokun
bd9f1b597e
routing: prune edges instead of vertexes in response to an FailUnknownNextPeer error
In this commit we fix an lingering bug in the Mission Control logic we
execute in response to the FailUnknownNextPeer error. Historically, we
would treat this as the _next_ node not being online. As a result, we
would then prune away the vertex from the current reachable graph all
together. It was recently realized, that this would at times be a bit
_tooo_ aggressive if the channel we attempt to route over was faulty,
down, or the incoming node had connectivity issues with the outgoing
node.

In light of this realization, we'll now instead only prune the _edge_
that we attempted to route over. This ensures that we'll continue to
explore the possible edges. Additionally, this guards us against failure
modes where nodes report FailUnknownNextPeer to other nodes in an
attempt to more closely control our retry logic.

This change is a stop gap on the path to a more intelligent set of
autopilot heuristics.

Fixes #1114.
2018-04-23 17:51:02 -07:00
Wilmer Paulino
beb08b3887
routing: add test for finding paths with additional edges 2018-04-20 04:01:33 -04:00
Wilmer Paulino
4598df054e
routing: modify findPath to take into account additional edges
In this commit, we modify our path finding algorithm to take an
additional set of edges that are currently not known to us that are
used to temporarily extend our graph with during a payment session.
These edges should assist the sender of a payment in successfully
constructing a path to the destination.

These edges should usually represent private channels, as they are not
publicly advertised to the network for routing.
2018-04-20 04:01:32 -04:00
Wilmer Paulino
7965247db1
routing: add additionalEdges field to a missionControl's paymentSession
In this commit, we introduce the ability for payment sessions to store
an additional set of edges that can be used to assist a payment in
successfully reaching its destination.
2018-04-20 04:01:31 -04:00
Wilmer Paulino
5ddee85479
routing: prevent logging the public key params within the routing hints 2018-04-20 04:01:30 -04:00
Wilmer Paulino
1e5659e5f4
routing: introduce routing hints to lightning payments
In this commit, we add a new field of routing hints to payments over the
Lightning Network. These routing hints can later be used within the path
finding algorithm in order to craft a path that will reach the
destination succesfully.
2018-04-20 04:01:29 -04:00
Dimitris Tsapakidis
4009f7f874 multi: fix typos in comments 2018-04-17 19:03:27 -07:00
practicalswift
663c396235 multi: fix a-vs-an typos 2018-04-17 19:02:04 -07:00
Olaoluwa Osuntokun
a6ffe999c6
routing: prune vertex, not ege after repeated FeeInsufficientErrors
In this commit, we modify the way we handle FeeInsufficientErrors to
more aggressively route around nodes that repeatedly return the same
error to us. This will ensure we skip older nodes on the network which
are running a buggier older version of lnd. Eventually most nodes will
upgrade to this new version, making this change less needed.

We also update the existing test to properly use a multi-hop route to
ensure that we route around the offending node.
2018-04-06 14:52:02 -07:00
Olaoluwa Osuntokun
1a05b48080
routing: add new test case to ensure we route around time lock failures 2018-03-23 17:55:16 -07:00
Olaoluwa Osuntokun
62b8ddb839
routing: update TestAddEdgeUnknownVertexes due to additional node in test graph 2018-03-23 17:55:15 -07:00
Olaoluwa Osuntokun
d5721165de
routing: fix TestPathInsufficientCapacity to actually send 1 BTC 2018-03-23 17:55:15 -07:00
Olaoluwa Osuntokun
de1150b78b
routing: add new vertex and to new channels to basic_graph.json
In this commit, we add a new node to the current default test graph
that we use for our path finding tests. This new node connects roasbeef
to sophon via a new route with very high fees. With this new node and
the two channels it adds, we can properly test that we’ll route around
failures that we run into during payment routing.
2018-03-23 17:55:15 -07:00
Olaoluwa Osuntokun
1365392d2b
routing: don't traverse links that advertise a timelock delta of zero 2018-03-23 17:55:14 -07:00
Olaoluwa Osuntokun
f935bd0baf
routing: add vertex pruning for non-final CLTV related errors
In this commit, we add vertex pruning for any non-final CLTV error.
Before this commit, we assumed that any source of this error was due to
the local node setting the incorrect time lock. However, it’s been
recently noticed on main net that there’re a set of nodes that seem to
not be properly scanned to the chain. Without this patch, users aren’t
able to route successfully as atm, we’ll stop all path finding attempts
if we encounter this.
2018-03-23 17:55:14 -07:00
Olaoluwa Osuntokun
635780858e
routing: add rejected spent outputs to the reject cache 2018-03-23 15:50:27 -07:00
Olaoluwa Osuntokun
93b04b39fe
routing: account for remaining routing onion errors in SendPayment
In this commit, we address a number of edge cases that were unaccounted
for when responding to errors that can be sent back due to an HTLC
routing failure. Namely:

   * We’ll no longer stop payment attempts if we’re unable to apply a
channel update, instead, we’ll log the error, prune the channel and
continue.
    * We’ll no remember which channels were pruned due to insufficient
fee errors. If we ever get a repeat fee error from a channel, then we
prune it. This ensure that we don’t get stuck in a loop due to a node
continually advertising the same fees.
   * We also correct an error in which node we’d prune due to a
temporary or permanent node failure. Before this commit, we would prune
the next node, when we should actually be pruning the node that sent us
the error.

Finally, we also add a new test to exercise the fee insufficient error
handling and channel pruning.

Fixes #865.
2018-03-21 15:23:48 -07:00
Olaoluwa Osuntokun
363cec7012
routing: add new PayAttemptTimeout field to LightningPayment
In this commit, we add a new field to the LightningPayment struct:
PayAttemptTimeout. This new field allows the caller to control exactly
how much time should be spent attempting to route a payment to the
destination. The default value we’ll use is 60 seconds, but callers are
able to specify a diff value. Once the timeout has passed, we’ll
abandon th e payment attempt, and return an error back to the original
caller.
2018-03-21 15:23:45 -07:00
Alex
948cb09c14 routing/chainview: bitcoind back-end now requires explicit NotifyBlocks() 2018-03-14 12:16:40 -06:00
Olaoluwa Osuntokun
800eea931f
build+multi: switch from bolt to bbolt
In this commit, we switch from boltbd/bolt to coreos/bbolt as the
former is no longer being actively maintained.
2018-03-10 19:01:13 -08:00
Olaoluwa Osuntokun
aa0410c90a
routing: add new methods to check the freshness of an edge/node
In this commit, we add a set of new methods to check the freshness of
an edge/node. This will allow callers to skip expensive validation in
the case that the router already knows of an item, or knows of a
fresher version of that time.

A set of tests have been added to ensure basic correctness of these new
methods.
2018-03-06 13:34:48 -05:00
t4sk
55f751fd37 routing/router Fix FindRoutes in router test
In router_test FindRoutes is passing DefaultFinalCLTVDelta in place
where numPaths is expected. This commit passes a default numPaths for
function calls to FindRoutes so that final cltv delta are correctly
passed.
2018-02-28 19:25:19 -08:00
Olaoluwa Osuntokun
7abdd30a87
routing: correct recent type change in heap_test.go 2018-02-12 16:44:42 -08:00
Olaoluwa Osuntokun
528aa67df7
routing: weight findPath more heavily towards fees during edge relaxation
In this commit, we modify the edgeWeight function that’s used within
the findPath method to weight fees more heavily than the time lock
value at an edge. We do this in order to greedily prefer lower fees
during path finding. This is a simple stop gap in place of more complex
weighting parameters that will be investigated later.

We also modify the edge distance to use an int64 rather than a float.
Finally an additional test has been added in order to excessive this
new change. Before the commit, the test was failing as we preferred the
route with lower total time lock.
2018-02-12 16:27:38 -08:00
Olaoluwa Osuntokun
ad0f5b31f6
routing: return cached route response in FindRoutes if enough routes
In this commit, we modify the caching structure to return a set of
cached routes for a request if the number of routes requested is less
than or equal to the number of cached of routes.
2018-02-12 16:22:28 -08:00
Olaoluwa Osuntokun
8a29c01354
routing: modify FindRoutes to take a max number of routes to return 2018-02-12 16:19:52 -08:00
Olaoluwa Osuntokun
33ba60ce66
routing: modify findPaths to take an upper limit on the number of routes
In this commit, we modify the findPaths method to take the max number
of routes to return. With this change, FindRoutes can eventually itself
also take a max number of routes in order to make the function useable
again.
2018-02-12 16:18:49 -08:00
Olaoluwa Osuntokun
b54b8dd7f1
routing: reject any new announcements which were pruned as zombies 2018-02-06 20:14:34 -08:00
Olaoluwa Osuntokun
6d05cb5aae
routing: extract zombie pruning to distinct method 2018-02-06 20:14:34 -08:00
Olaoluwa Osuntokun
1f3124f48a
routing: use [33]byte instead of *btcutil.Publickey for SendToSwitch
With this change, we can avoid unnecessarily serializing a public key.
2018-02-06 20:14:33 -08:00
Olaoluwa Osuntokun
6751cd8b9f
routing: update package to account for recent channeldb API changes 2018-02-06 20:14:32 -08:00
Daniel McNally
8543497dcc multi: fixing it's/its typos in comments 2018-02-06 19:13:07 -08:00
Alex Bosworth
bbc2ecd580 routing: typo fix
"we're closed" should be "were closed"
2018-02-06 19:12:10 -08:00
practicalswift
a93736d21e multi: comprehensive typo fixes across all packages 2018-02-06 19:11:11 -08:00
Olaoluwa Osuntokun
0a4de859a2
discovery+routing: reduce number of active validation barrier jobs
In order to reduce high CPU utilization during the initial network view
sync, we slash down the total number of active in-flight jobs that can
be launched.
2018-01-28 14:55:32 -08:00
Johan T. Halseth
b07f242dc2
routing: use multimutex.Mutex instead of package internal mutex 2018-01-24 10:26:40 +01:00
Olaoluwa Osuntokun
eba6ba0760
routing: properly prune edges that return a FailPermanentChannelFailure
In this commit, we now account for a case where a node sends us a
FailPermanentChannelFailure during a payment attempt. Before this
commit, we wouldn’t properly prune the edge to avoid re-using it. We
remedy this by properly attempting to prune the edge if possible.
Future changes well send a FailPermanentChannelFailure in the case that
we ned to go on-chain for an outgoing HTLC, and cancel back the
incoming HTLC.
2018-01-22 19:19:34 -08:00
Alex
187f59556a multi: add bitcoind drivers and tests 2018-01-15 13:59:34 -08:00
Johan T. Halseth
0b8e7ff836
routing: embed FeeSchema in ChannelPolicy
This commit embeds the FeeSchema within the new
struct ChannelPolicy, which also contains the
TimeLockDelta for a channel.
2018-01-12 22:56:31 +01:00
Olaoluwa Osuntokun
d70e4bb0a0
routing: account for case where final destination send TemporaryChannelFailure
In this commit, we fix an existing bug that could cause lnd to crash if
we sent a payment, and the *destination* sent a temp channel failure
error message. When handling such a message, we’ll look in the nextHop
map to see which channel was *after* the node that sent the payment.
However, if the destination sends this error, then there’ll be no entry
in this map.

To address this case, we now add a prevHop map. If we attempt to lookup
a node in the nextHop map, and they don’t have an entry, then we’ll
consult the prevHop map.

We also update the set of tests to ensure that we’re properly setting
both the prevHop map and the nextHop map.
2018-01-10 15:15:55 -08:00
Johan T. Halseth
0571918d3d
routing: process only single ChannelEdgePolicy per channelID concurrently
This commit adds synchronization around the processing
of multiple ChannelEdgePolicy updates for the same
channel ID at the same time.

This fixes a bug that could cause the database access
HasChannelEdge to be out of date when the goroutine
came to the point where it was calling UpdateEdgePolicy.
This happened because a second goroutine would have
called UpdateEdgePolicy in the meantime.

This bug was quite benign, as if this happened at
runtime, we would eventually get the ChannelEdgePolicy
we had lost again, either from a peer sending it to
us, or if we would fail a payment since we were using
outdated information. However, it would cause some of
the tests to flake, since losing routing information
made payments we expected to go through fail if this
happened.

This is fixed by introducing a new mutex type, that
when locking and unlocking takes an additional
(id uint64) parameter, keeping an internal map
tracking what ID's are currently locked and the
count of goroutines waiting for the mutex. This
ensure we can still process updates concurrently,
only avoiding updates with the same channel ID from
being run concurrently.
2018-01-09 13:09:29 +01:00
Olaoluwa Osuntokun
98f63cdce1
routing: add new paymentSession companion struct to missionControl
In this commit, we modify the pruning semantics of the missionControl
struct. Before this commit, on each payment attempt, we would fetch a
new graph pruned view each time. This served to instantly propagate any
detected failures to all outstanding payment attempts. However, this
meant that we could at times get stuck in a retry loop if sends take a
few second, then we may prune an edge, try another, then the original
edge is now unpruned.

To remedy this, we now introduce the concept of a paymentSession. The
session will start out as a snapshot of the latest graph prune view.
Any payment failures are now reported directly to the paymentSession
rather than missionControl. The rationale for this is that
edges/vertexes pruned as result of failures will never decay for a
local payment session, only for the global prune view. With this in
place, we ensure that our set of prune view only grows for a session.

Fixes #536.
2018-01-08 19:50:24 -08:00
Matt Drollette
adf0d98194 multi: fix several typos in godoc comments 2017-12-17 18:40:05 -08:00
Olaoluwa Osuntokun
7a3116f8a9
routing: ensure we set the TotalFees value on routes properly
Before this commit, we wouldn’t properly set the TotalFees attribute.
As a result, our sorting algorithm at the end to select candidate
routes would simply maintain the time-lock order rather than also sort
by total fees. This commit fixes this issue and also allows the test
added in the prior commit to pass.
2017-12-06 18:45:52 -08:00
Olaoluwa Osuntokun
cc3b32c0d4
routing: extend TestFindRoutesFeeSorting to assert proper fee attributes 2017-12-06 18:39:56 -08:00
Olaoluwa Osuntokun
455568279b
routing: ensure that new blocks connected monotonically increase in height
This commit fixes an existing bug within the ChannelRouter. Prior to
this commit, if the chain view skipped blocks or for some reason we had
a gap in blocks delivered, then we would simply accept them. This had
the potential to cause us to miss on-chain channel closure events. To
remedy this, we won’t process any blocks whose heights aren’t
*strictly* increasing.

A longer term fix would be to have the ChainView take a block height,
and re-dispatch any notifications from that height to the current
height.
2017-12-05 17:46:21 -08:00
Olaoluwa Osuntokun
ffd6b65ec1
routing: ensure access to r.bestHeight is thread-safe 2017-12-05 17:46:20 -08:00
Olaoluwa Osuntokun
a78ff8ba04
routing: during path finding skip edges that are marked disabled
In this commit, we implement adherence of the disabled bit within a
ChannelUpdate during path finding. If a channel is marked as disabled,
then we won’t attempt to route through it. A test has been added to
exercise this new check.
2017-11-30 22:31:27 -08:00
Olaoluwa Osuntokun
b61670fe23
routing: during path finding skip edge if amount < edge.MinHTLC
In this commit, we update path finding to skip an edge if the amount
we’re trying to route through it is below the MinHTLC (in mSAT) value
for that node. We also add a new test to exercise this behavior. In
order for out test to work properly, we’ve modified the JSON to make
the edge to Goku have a higher min HTLC value.
2017-11-30 22:29:45 -08:00
Olaoluwa Osuntokun
73ccb76853
routing: update ChannelUpdate handling to recognize lnwire.ChanUpdateFlag 2017-11-30 22:24:35 -08:00
Olaoluwa Osuntokun
7757721a92
routing: don't dispatch notification if error is non nil after processing ann 2017-11-29 17:53:09 -08:00
Olaoluwa Osuntokun
5bc9f07d12
routing: validate incoming announcements in parallel 2017-11-29 16:44:14 -08:00
Olaoluwa Osuntokun
fcd5e4aa41
routing: when updating chain view filter on restart, use best height as height hint
In this commit, we modify the high value passed into UpdateFilter upon
restart. Before this commit, we would pass in the prune height, which
would cause a full rescan within the FilteredChainView if the best
height as > than the prune height. This was redundant as we would
shortly carry out a manual rescan in the method below. To fix this, we
now pass in the bestHeight, this isn’t an issue as the
syncGraphWithChain method will manually scan up to that best height.
2017-11-29 16:26:39 -08:00
Olaoluwa Osuntokun
978023ab1d
routing: ensure notifyTopologyChange is theadsafe 2017-11-29 16:24:30 -08:00
Olaoluwa Osuntokun
33ce4e5689
routing: add new ValidationBarrier to allow for safe parallel validation of announcements
In this commit, we add a new abstraction, the ValidationBarrier. This
struct will be used to allow parallel validation of announcements
within notes AuthenticatedGossiper as well as the ChannelRouter.
Naively validating the announcement in parallel would run into issues
as it would be possible for validate an update announcement, before
validating the channel announcement itself. We solve this by creating a
waiting dependance using the ValidationBarrier to ensure that the
defendant jobs wait until their parents have been full validated.
2017-11-29 16:24:20 -08:00
Olaoluwa Osuntokun
d2c8f01768
routing/chainview: remove possibility of deadlock in shutdown of blockEventQueue 2017-11-16 16:01:21 -08:00
Olaoluwa Osuntokun
f189e2395a
routing: if the graph has never been pruned, prune with current height
In this commit we ensure that if this is the first time that the
ChannelRouter is starting, then we set the pruned height+hash to the
current best height. Otherwise, it’s possible that we attempt to update
the filter with a 0 prune height, which will restart a historical
rescan unnecessarily.
2017-11-15 18:19:39 -08:00
Olaoluwa Osuntokun
2b052cc889
routing: only update the filter if we have a non-zero chain view
In this commit we ensure that we only update the filter, if we have a
non-zero chain view. Otherwise, a mini rescan may be kicked off
unnecessarily  if we don’t yet know of any channels yet in the greater
graph.
2017-11-15 18:17:51 -08:00
Laura Cressman
7408aa6c8d config+pathfind_test+notifications_test: format and fix newVertex import
Run go fmt so config file is formatted correctly. Also rename
newVertex to NewVertex in pathfind_test and notifications_test
as it is now exported from the routing package.
2017-11-15 16:50:19 -08:00
Laura Cressman
39d38da732 discovery: create deDupedAnnouncements struct in gosspier.go
For Part 1 of Issue #275. Create isolated private struct in
networkHandler goroutine that will de-duplicate
announcements added to the batch. The struct contains maps
for each of channel announcements, channel updates, and
node announcements to keep track of unique announcements.

The struct has a Reset method to reset stored announcements, an
AddMsg(lnwire.Message) method to add a new message to the current
batch, and a Batch method to return the set of de-duplicated
announcements.

Also fix a few minor typos.
2017-11-15 16:50:19 -08:00
Conner Fromknecht
e62bb0319d
routing/chainview/btcd: makes rescan logging statement print to trace 2017-11-11 15:53:42 -08:00
Conner Fromknecht
153794adb0
routing/router: sets bestHeight before filtering chain
This commit alters the behavior of the router's logic on
 startup, ensuring that the chain view is filtered using
 the router's latest prune height. Before, the chain was
 filtered using the bestHeight variable, which was
 uninitialized, benignly forcing a rescan from genesis.
 In tracking down this, we realized that we should
 actually be using the prune height, as this is
 representative of the channel view loaded from disk.
 The best height/hash are now only used during
 startup to determine if we are out of sync.
2017-11-11 15:53:38 -08:00
Olaoluwa Osuntokun
5dbf5084e7
routing: fix linter error 2017-11-06 16:49:36 -08:00
Olaoluwa Osuntokun
9294358b5b
routing: populate the chain filter before calling syncGraphWithChain
In this commit we fix an existing bug within the ChannelRouter. Before
this commit, we would sync our graph prune state, *then* update the
cain filter. This is incorrect as the blocks we manually pruned may
have included channel closing transactions. As a result, we would miss
the pruning of a set of channels, and assume that they were still
active.

In this commit, we fix this by reversing the order: we first update the
chain filter and THEN sync the channel graph.
2017-11-06 16:10:24 -08:00
Olaoluwa Osuntokun
ccf94e9457
routing: add new TestRouterChansClosedOfflinePruneGraph test
In this commit we add a new test to the set of unit tests for the
ChannelRouter: TestRouterChansClosedOfflinePruneGraph. This tests that
if channels are closed while the ChannelRouter is down, then upon
restart the channels are properly recognized as being closed.
2017-11-06 16:06:15 -08:00
Olaoluwa Osuntokun
2824f45aad
routing: update tests to adhere to new newMockChainView API 2017-11-06 16:00:57 -08:00
Olaoluwa Osuntokun
3be905109c
routing: add RestartRouter method to testCtx 2017-11-06 16:00:22 -08:00
Olaoluwa Osuntokun
cdf2f43432
routing: add Reset() method to mockChainView
In this commit, we add a Reset() method to the mockChainView struct.
With this new method tests are able to fully simulate a restart of the
ChannelRouter. This is necessary as the FilteredChainView instances are
assumed to be stateless, and don’t write their state to disk before a
restart.
2017-11-06 15:58:57 -08:00
Olaoluwa Osuntokun
fea8cbf920
routing: add quit channel to mockChainView 2017-11-06 15:57:29 -08:00
Olaoluwa Osuntokun
4f32ee94ea
routing: implement FilterBlock for mockChainView 2017-11-06 15:56:31 -08:00
Olaoluwa Osuntokun
5f915280bc
routing/chainview: if column wrapping on comments in neutrino.go 2017-11-06 15:55:21 -08:00
Alex
1c408adcab routing/chainview: update neutrino driver to comply with BIP 2017-11-02 20:04:07 -07:00
Johan T. Halseth
256db86b02
routing: make channel router aware of stale blocks
This commit make the channel router handle the case where
it wakes up on a stale branch, and also the case where a
reorg happens while it is active.
2017-11-03 00:05:32 +01:00
Johan T. Halseth
8eb994c14b
routing/chainView test: add testFilterBlockDisconnected
This commit adds a test for the FilteredChainView interfaces,
making sure they notify about disconnected/connected blocks
in the correct order during a reorg.
2017-11-03 00:05:31 +01:00
Johan T. Halseth
6d15be5b79
routing/chainview: use blockEventQueue for neutrino block events
This commit makes use of the blockEventQueue within the neutrino
implementation of FilteredChainView to ensure connected and
disconnected blocks are consumed in order by the reader.

It also specifies that neutrino is not to send disconnected blocks
notifications during rescans, making it consistent with the btcd
implementation.
2017-11-03 00:05:30 +01:00
Johan T. Halseth
082f012fcf
routing/chainview: implement staleBlocks for btcd view.
This commit moves btcd view away from using the deprecated
callbacks onBlockConnected/Disconnected, and instead use
onFilteredBlockConnected/disconnected.

This commit also implements the sending of disconnected blocks
over the staleBlocks channel. To send these blocks, the
blockEventQueue is used to ensure the ordering of blocks are
correctly kept.

It also changes the way filter updates are handled. Since we
now load the tx filter to the rpc server itself, we can call
RescanBlocks instead of manually filtering blocks. These
rescanned blocks are also added to the blockEventQueue,
ensuring the ordering is kept.
2017-11-03 00:05:30 +01:00
Johan T. Halseth
e9c2f703d9
routing/chainview: additional comments to FilteredChainView godoc 2017-11-03 00:05:29 +01:00
Johan T. Halseth
a219aca0d3
routing/chainView: add blockEventQueue data structure
blockEventQueue is an ordered queue for block events sent from a
FilteredChainView. The two types of possible block events are
connected/new blocks, and disconencted/stale blocks. The
blockEventQueue keeps the order of these events intact, while
still being non-blocking. This is important in order for the
chainView's call to onFilteredBlockConnected/Disconnected to not
get blocked, and for the consumer of the block events to always
get the events in the correct order.
2017-11-03 00:05:28 +01:00
Olaoluwa Osuntokun
57b9b78ce3
routing: modify path finding tests to accommodate for flip in fee calc
Before this commit, we would expect that structurally we don’t pay any
fee for the first hop, but do for the final hop. After the latest
commit, this is now flipped as when we say fee, we mean the fee that we
need to pay to transit a link. For the final hop, there’s no additional
distance to be traveled, so the fee is nothing.
2017-10-24 18:29:23 -07:00
Olaoluwa Osuntokun
a7fe7ae941
routing: correct miscalculation multi-hop onion payload fees
In this commit we fix an existing miscalculation in the fees that we
prescribe within the onion payloads for multi-hop routes. Before this
commit, if a route had more than 3 hops, then we would erroneously give
the second to last hop zero fees.

In this commit we correct this behavior, and also re-write the fee
calculation code fragment within newRoute for readability and clarity.
There are now only two cases: this is the last hop, and this is any
other hop. In the case of the last hop, simply send the exact amount
with no additional fee. In the case of an intermediate hop, we use the
_prior_ (closer to the destination) hop to calculate the amount of fees
we need, which allows us to compute the incoming flow. Using that
incoming flow, we then can compute the amount that the hop should
forward out.

Partially fixes #391.
2017-10-24 18:27:35 -07:00
Olaoluwa Osuntokun
176fde4ec0
routing: modify applyChannelUpdate to not error out if update is stale
In this commit we fix a slight bug within the existing SendPayment loop
which would cause the wrong error to be returned to users. Prior to
this commit, if we received an update identical to what we were already
aware of, then that error would be returned rather than the
ForwardingError that encapsulated this update.

In this commit with remedy this by properly returning the exact error.

Partially fixes #391.
2017-10-24 18:23:04 -07:00
Olaoluwa Osuntokun
25614f67f8
routing: restore in memory selfNode within the ChannelRouter
In this commit we restore the in memory ChannelRouter as we’ll no
dynamically set the ChannelRouter’s pointer within he spec path finding
test example.
2017-10-22 18:36:59 -07:00
Olaoluwa Osuntokun
76ec41103e
routing: extend missionControl.RequestRoute to take final CLTV delta 2017-10-22 18:36:58 -07:00
Olaoluwa Osuntokun
828057889e
routing/testdata: modify spec example to set bob as initial source node 2017-10-22 18:36:58 -07:00
Olaoluwa Osuntokun
6e00abf3f1
routing: update tests in router_test.go to pass in new param to FindRoutes 2017-10-22 18:36:52 -07:00
Olaoluwa Osuntokun
69a3783d55
routing: add new test case from spec to assert proper route calc 2017-10-22 18:36:52 -07:00
Olaoluwa Osuntokun
acd160a419
routing: update path finding tests to account for change in fee calculation 2017-10-22 18:36:51 -07:00
Olaoluwa Osuntokun
aee1619488
routing: add new FinalCLTVDelta attribute to SendPayment
In this commit, we’ll now optionally allow the user to pass in the CLTV
delta value specified by the recipient a payment. If the value isn’t
specified, then we’ll use the current global default for the payment.
2017-10-22 18:36:51 -07:00
Olaoluwa Osuntokun
c8f45e3a04
routing: add new param to FindRoutes for CLTV expiry for final hop
In this commit, we modify the FindRoutes method to pass in the CLTV
expiry for the final hop. If the value isn’t passed in, then we’ll use
the current global default value in place.
2017-10-22 18:36:50 -07:00
Olaoluwa Osuntokun
4b82e2ec43
routing: correct fee calculation when converting from path to route
In this commit, we correct the fee calculation when converting from a
path to route. Previously we would apply the “no fee” case at the
_first_ hop, rather than the last hop. As a result, we needed to swap
the edges during path finding, otherwise, if the incoming and outgoing
edges had different fee rates, then we would create an invalid onion
payload.

In this commit we now properly switch fee calculation into three cases:
  * a single hop route, so there’s no fee
  * we’re at the first hop in a multi hop route, and we apply the fee
for the _next_ hop
  * we’re at an intermediate hop and the fee calculation proceeds as
normal
2017-10-22 18:36:50 -07:00
Olaoluwa Osuntokun
8df69a83a7
routing: revert the swapping of in/out edges during path finding
In this commit we revert a commit which was added in the past as way to
allow the path -> route conversion code to remain the same, while
properly respecting the necessary time locks and fees. In an upcoming
change, this swap is no longer necessary as we’ll always use: the time
lock of the outgoing node and the fee of the incoming node.
2017-10-22 18:36:49 -07:00
Olaoluwa Osuntokun
23b6d84493
routing: modify newPath to take the preferred final CLTV delta of last hop
In this commit, rather than reading the final CLTV delta from the
channel graph itself (which would require _both_ edges to be advertised
in order to route over), we now instead have moved to allowing the
receiving node to choose their own final CLTV delta.
2017-10-22 18:36:48 -07:00
Olaoluwa Osuntokun
65482faa79
routing: remove the selfNode attribute from memory
In this commit, we’ve removed the selfNode attribute from memory, as
the set of new tests we’ll write, will depend on us being able to
switch the source node dynamically from the database itself.
2017-10-22 18:36:48 -07:00
Olaoluwa Osuntokun
f5881ad3df
routing: only write a single edge as edges in test data are now directional 2017-10-22 18:36:47 -07:00
Olaoluwa Osuntokun
86283b0d06
routing: make setting source node optional when parsing testdata 2017-10-22 18:36:47 -07:00
Olaoluwa Osuntokun
e70031da7b
routing: properly assign flags and write both edges when parsing test data 2017-10-22 18:36:46 -07:00
Olaoluwa Osuntokun
4ff66efa2b
routing: convert testChan stuct to take an int64 for FeeRate 2017-10-22 18:36:46 -07:00
Olaoluwa Osuntokun
fae3dacc9b
routing: modify basic_graph.json to include both directional edges 2017-10-22 18:36:45 -07:00
Olaoluwa Osuntokun
3c01d5bab9
routing: convert basic_graph.json testdata to use fixed point fee rate 2017-10-22 18:36:44 -07:00
Jim Posen
9fd77a6e40 multi: Update lnd to use new feature vector API. 2017-10-17 22:47:20 -07:00
Olaoluwa Osuntokun
b4273d1eaa
routing: abstract path finding behind mission control
In this commit, from the PoV of the SendPayment method we now delegate
all path finding+verification to missionControl. This change doesn’t
materially affect anything, it simply expands the abstraction to make
way for future features that more heavily utilize mission control.
2017-10-17 19:44:17 -07:00
Olaoluwa Osuntokun
209fb98d0f
routing: properly use vertexDecay for vertexes in missionControl 2017-10-16 20:31:32 -07:00
Olaoluwa Osuntokun
bbb34cebe0
routing: modify the TestSendPaymentRouteFailureFallback to clear missionControl between attempts
In order to maintain the original essence of the test, we need to clear
the state of missionControl with each attempt, essentially advancing
time between each payment attempt.
2017-10-16 19:07:40 -07:00
Olaoluwa Osuntokun
8ef829ed80
routing: modify SendPayment loop to be lazy, iterative, and use missionControl
In this commit we modify the SendPayment loop to optimize for
time-to-first-payment-success-or-failure. The prior logic would first
attempt to find at least 100 routes to the destination, then
iteratively prune them away as errors were encountered. In this commit,
we modify this approach to instead take a lazy approach: we first find
the current “best” path, attempt to send to that, and if an error
occurs we prune a section of the graph by reporting to missionControl,
then continue.

With this new approach, if the first known path has sufficient
capacity, and is available, then the payment speed is greatly improved
from the PoV of users. Additionally, we avoid the excessive computation
of crawling most of the graph in the k-shortest paths loop. With the
decay on missionControl, all routes will now feed information into the
central knowledge hung, allowing all payments to iteratively find out
the inactive portions of the payment graph.
2017-10-16 19:05:47 -07:00
Olaoluwa Osuntokun
276f2e467b
routing: end path finding on an additional set of critical-ish errors 2017-10-16 18:58:35 -07:00
Olaoluwa Osuntokun
e06177e55c
routing: introduce new missionControl system within ChannelRouter
This commit adds a new system within the ChannelRouter: missionControl.
The purpose of this system to is to act as a shared memory of sorts
between payment sending attempts, recording which edges/vertexes word
or didn’t work. Allowing execution attempts to pass on their iterative
knowledge of the graph to later attempts will reduce the number of
failures encountered, and generally lead to a better UX when sending
payments.

The current capabilities of missionControl are rather limited just to
introduce the new abstraction. Later follow up commits will also add
preferential treatment for reliable nodes, knowledge the impact that
target payments have on unbalancing the payment graph, etc.
2017-10-16 18:57:36 -07:00
Olaoluwa Osuntokun
ae6bde2d77
routing: avoid internal bolt db deadlock by reusing transaction in findPath
This commit fixes a bug that could lead to a deadlock inside bolt db
itself. In a recent commit we allowed a db transaction to be passed
directly into findPath, however, the initial call to graph.ForEachNode
instead passed a _nil_ transaction causing the method itself to create
a _new_ transaction, leading to a deadlock.

We fix this issue by instead re-using the transaction pointer.
2017-10-16 18:48:27 -07:00
Olaoluwa Osuntokun
b29a73a0dd
routing: don't prune our own channels during zombie channel collection
This commit is a precautionary commit which ensure that we don’t
attempt to prune our _own_ channels during zombie channel collection.
2017-10-16 18:14:01 -07:00
Olaoluwa Osuntokun
51b072c4b5
routing: return proper error if encounter non ForwardingError in SendPayment 2017-10-10 22:19:28 -07:00
Olaoluwa Osuntokun
646f79f566
routing: perform path finding inside a single DB transaction
This commit modifies the path finding logic such that all path finding
is done inside a _single_ database transaction. With this change, we
ensure that we don’t end up possibly creating hundreds of database
transactions slowing down the path finding and payment sending process
all together.
2017-10-10 22:19:27 -07:00
Olaoluwa Osuntokun
eb7b5b342e
routing: add a basic test to exercise route pruning in response to errors 2017-10-10 22:19:25 -07:00
Olaoluwa Osuntokun
ce7179a468
routing: add basic route pruning in response to HTLC onion errors
This commit adds basic route pruning in response to HTLC onion errors.
With this new change, the router will now prune routes in response to
HTLC errors, which will reduce the time to payment success, and also
avoid a bunch of unnecessary network traffic.

We now respond to two errors lnwire.FailTemporaryChannelFailure and
lnwire.FailUnknownNextPeer. In response to the first error, we’ll prune
all routes that contain the channel which was unable to be routed over.
In response to the second error we’ll prune all routes that contain the
node which couldn’t be found.
2017-10-10 22:19:25 -07:00
Olaoluwa Osuntokun
8d7f3943bb
routing: add two new methods to filter routes based on node/channel 2017-10-10 22:19:24 -07:00
Olaoluwa Osuntokun
f6ac31281b
routing: also include the source node in the nextHopMap index
In this commit we modify the newRoute function to also add the source
node to the nextHopMap index. With this addition the indexes will now
allow the router to react based on failures that occur during the
_first_ hop, meaning the channel directly attached to the source node.
2017-10-10 22:19:23 -07:00
Olaoluwa Osuntokun
70e114fa6f
routing: add additional indexes to the Route struct to allow for querying
This commit adds three new indexes to the Route struct. These indexes
allow a caller to check if a channel is in the route, check if a node
is in the route, query the next node after a target node, and query the
next channel after a target node. The combination of these new indexes
will allow the ChannelRouter to prune away routes from the available
set in response to any received errors.
2017-10-10 22:19:22 -07:00
Olaoluwa Osuntokun
7ba6b7fa09
routing: add a String() method to the vertex type 2017-10-10 22:19:22 -07:00
Olaoluwa Osuntokun
3b7855e449
routing: implement 2-week zombie channel pruning
This commit implements 2-week zombie channel pruning. This means that
every GraphPruneInterval (currently set to one hour), we’ll scan the
channel graph, marking any channels which haven’t had *both* edges
updated in 2 weeks as a “zombie”. During the second pass, all “zombie”
channel are removed from the channel graph all together.

Adding this functionality means we’ll ensure that we maintain a
“healthy” network view, which will cut down on the number of failed
HTLC routing attempts, and also reflect an active portion of the graph.
2017-10-04 20:46:09 -07:00
Olaoluwa Osuntokun
e81689057a
discovery+routing: remove DeleteEdge from ChannelGraphSource interface
This commit removes the recently added DeleteEdge method from the
ChannelGraphSource interface as it’s no longer needed.
2017-10-04 20:46:08 -07:00
Brandon
3907ae65c2 routing+discovery: implement 2-week network view pruning 2017-10-04 20:40:21 -07:00
Laura Cressman
156772d04a channeldb: use binary.Read/Write with bools in channel.go
Use binary.Read/Write in functions to serialize and deserialize
channel close summary and HTLC boolean data, as well as in
methods to put and fetch channel funding info. Remove lnd
implementations of readBool and writeBool as they are no
longer needed. Also fix a few minor typos.
2017-10-02 23:13:47 -07:00
Laura Cressman
29687f49eb routing: replace sort.Sort with sort.Slice in router.go
Use sort.Slice in FindRoutes function in routing/router.go, as part
of the move to use new language features. Remove sortableRoutes type
wrapper for slice of Routes since it is no longer needed to sort routes.
2017-10-02 23:13:47 -07:00
Laura Cressman
8822bb11ce routing: replace sort.Sort with sort.Slice in heap_test.go
Use sort.Slice in TestHeapOrdering function in routing/heap_test.go,
as part of the move to use new language features.
2017-10-02 23:13:47 -07:00
Olaoluwa Osuntokun
7eb0e56406
routing: modify TestSendPaymentRouteFailureFallback to use non-critical error
In this commit we modify the existing
TestSendPaymentRouteFailureFallback to use a non-critical error aside
from FailChannelDisabled. This is necessary as the behavior of the
current error handling can fail due to us sending in a nil error.
2017-10-02 22:14:14 -07:00
Olaoluwa Osuntokun
3ba70fe6ec
routing: add preliminary version of more intelligent payment error handling
This commit modifies the way we currently interpret errors when sending
payments via the SendToSwitch method. We split the errors into two
broad sections: critical errors which cause us to abandon the payment
dispatch all together, and errors which are transient meaning we should
continue trying to remainder of the returned routes.

Note that we haven’t yet properly implemented all the necessary
measures such as filtering edges that are detected as being temporarily
inactive, etc.

This change should correct erroneous behavior such as continuing to try
all available routes in the face of an invalid payment hash error and
the like.
2017-10-02 22:14:13 -07:00
Olaoluwa Osuntokun
486b464e1c
routing: move path caching into FindRoutes
This commit modifies the way we do path caching. Rather than only
caching within SendPayment, we now cache routes within FindRoutes. This
is more natural as SendPayment eventually calls FindRoute. As a result
of this commit, queries to FindRoute are now properly cached, speeding
up applications which are focused on graph visualization or querying
rather than sending payments.
2017-10-02 22:14:13 -07:00
John Griffith
1057a1a7c3 routing: handle onion errors in ChannelRouter 2017-10-02 22:13:05 -07:00
Jim Posen
d8a2ed27b8 routing/chainview: Fix data race in block disconnected callback. 2017-09-29 13:53:02 -07:00
Olaoluwa Osuntokun
0e626ce42c
routing: add a select+quit case when receiving error to ensure graceful shutdown 2017-09-25 20:55:09 -07:00
Olaoluwa Osuntokun
e5f3ee0fb6
chainntnfs+routing/chainview: reduce neutrino.WaitForMoreCFHeaders value
This commit reduces the neutrino.WaitForMoreCFHeaders parameter when
instantiating a neutrino instance as a lower value will allow the tests
to complete more quickly.
2017-09-13 16:46:11 +02:00
Olaoluwa Osuntokun
b07e7fb7cc
routing: hop-payload for last hop should be the absolute timeout, not delta
This commit fixes an oversight in the path finding code when converting
a path into a route. Currently, for the last hop, we’d emplace the
expiry delta of the last hop within the per-hop payload. This was left
over from a prior version of the specification.

To fix this, we’ll now emplace the _absolute_ final HTLC expiry with
the payload, such that, the final hop that verify that the HTLC has not
been tampered with in flight.
2017-09-12 21:27:47 +02:00
Olaoluwa Osuntokun
9f0efddc20
multi: switch from btcrpcclient to rpcclient 2017-08-24 18:54:24 -07:00
Olaoluwa Osuntokun
321cc28cd8
routing: in findPath skip edge if incoming edge isn't advertised 2017-08-22 00:54:15 -07:00
Olaoluwa Osuntokun
f5d221012d
routing: update ChannelGraphSource due to latest API changes 2017-08-22 00:53:36 -07:00
Olaoluwa Osuntokun
bc4ad34190
routing: don't re-validate a channel update's edge if it already exists
By avoiding re-validating the channel edge, we avoid wasted network
bandwidth and queries.
2017-08-22 00:53:33 -07:00
Olaoluwa Osuntokun
5301da790c
routing: fix path finding, bug use the proper policy during path finding
This commit fixes an lingering bug within the path finding logic of the
router. Previously we used the edge policy directly attached to the
outgoing channel of the node we were traversing to calculate the fees
and time lock information. This is incorrect, as we instead should be
using the policy of the *connecting* node as we’ll need to pay for
transit as they dictate.

To remedy this, we now grab the incoming+outgoing edges and use those
accordingly when building the initial path.
2017-08-22 00:53:15 -07:00
Olaoluwa Osuntokun
6467fdd829
routing: update path finding and notifications to use mSAT 2017-08-22 00:53:12 -07:00
Olaoluwa Osuntokun
5ef077e5c8
routing: cap number of yen's algorithm iterations at 100
This commit makes a precautionary change in order to ensure that the
upper bound on the number of iteration’s within our version of Yen’s
algorithm is fixed.
2017-08-15 19:56:41 -07:00
Olaoluwa Osuntokun
8c3441b30f
routing: update test to account for proper time locks 2017-08-02 21:07:35 -07:00
Olaoluwa Osuntokun
67f17b319a
routing: invalidate routing cache on each new block
This commit makes the routing cache invalidation a bit more aggressive.
We now invalidate the cache on each new block as the routes in the
cache are based on the current block height. Using the cached items may
cause our routes to fail due to them having time locks which have
already expired.
2017-08-02 21:07:06 -07:00
Olaoluwa Osuntokun
f61d977176
routing: obtain current height when creating a new route 2017-08-02 21:02:24 -07:00
Olaoluwa Osuntokun
d331ddd2f4
routing: when creating a route, base time locks off current height
This commit implements some missing functionality, namely before all
time locks were calculated off of a base height of 0 essentially.
That’s incorrect as all time locks within HTLC’s would then be already
expired. We remedy this requesting the latest height when creating a
route to ensure that our time locks are set properly.
2017-08-02 21:01:54 -07:00
Johan T. Halseth
39a59bbe6f routing: Require adding edge to node before adding node.
This commit introduces the requirement specified in BOLT#7,
where we ignore any node announcements for a specific node
if we yet haven't seen any channel announcements where this
node takes part. This is to prevent someone DoS-ing the
network with cheap node announcements. In the router this
is enforced by requiring a call to AddNode(node_id) to
be preceded by an AddEdge(edge_id) call, where node_id is
one of the nodes in edge_id.
2017-08-02 15:58:58 -07:00
Conner Fromknecht
14a06526b8 routing/notifs: order invariant testing of ntfn delivery (#238)
Modifies the test cases in `TestEdgeUpdateNotification` and
`TestNodeUpdateNotification` to check for the possibility of notifications
being delivered out of order.  This addresses some sporadic failures that
were observed when running the test suite. 

I looked through some of the open issues but didn't see any addressing this
issue in particular, but if someone could point me to any relevant issues
that would be much appreciated!

Issue
-----
Currently the test suite validates notifications received in the order they
are submitted. The check fails because the verification of each
notification is statically linked to the order in which they are delivered,
seen
[here](1be4d67ce4/routing/notifications_test.go (L403))
and
[here](1be4d67ce4/routing/notifications_test.go (L499))
in `routing/notifications_test.go`.  The notifications are typically
delivered in this order, but causes the test to fail otherwise.

Proposed Changes
-------------------
Construct an index that maps a public key to its corresponding edges and/or
nodes.  When a notification is received, use its identifying public key and
the index to look up the edge/node to use for validation. Entries are
removed from the index after they are verified to ensure that the same
entry is validated twice. The logic to dynamically handle the verification
of incoming notifications rests can be found here
[here](https://github.com/cfromknecht/lnd/blob/order-invariant-ntfns/routing/notifications_test.go#L420)
and
[here](https://github.com/cfromknecht/lnd/blob/order-invariant-ntfns/routing/notifications_test.go#L539).

Encountered Errors
--------------------
 * `TestEdgeUpdateNotification`: notifications_test.go:379: min HTLC of
   edge doesn't match: expected 16.7401473 BTC, got 19.4852751 BTC
 * `TestNodeUpdateNotification`: notifications_test.go:485: node identity
   keys don't match: expected
   027b139b2153ac5f3c83c2022e58b3219297d0fb3170739ee6391cddf2e06fe3e7, got
   03921deafb61ee13d18e9d96c3ecd9e572e59c8dbd0bb922b5b6ac609d10fe4ee4


Recreating Failing Behavior
---------------------------
The failures can be somewhat difficult to recreate, I was able to reproduce
them by running the unit tests repeatedly until they showed up.  I used the
following commands to bring them out of hiding:

```
./gotest.sh -i
go test -test.v ./routing && while [ $? -eq 0 ]; do go test -test.v ./routing; done
```

I was unable to recreate these errors, or any others in this package, after
making the proposed changes and leaving the script running continuously for
~30 minutes. Previously, I could consistently generate an error after ~20
seconds had elapsed on the latest commit in master at the time of writing:
78f6caf5d2e570fea0e5c05cc440cb7395a99c1d. Moar stability ftw!
2017-07-31 21:38:03 -07:00
Andrey Samokhvalov
2d378b3280 htlcswitch+router: add onion error obfuscation
Within the network, it's important that when an HTLC forwarding failure
occurs, the recipient is notified in a timely manner in order to ensure
that errors are graceful and not unknown. For that reason with
accordance to BOLT №4 onion failure obfuscation have been added.
2017-07-14 19:08:04 -07:00
Olaoluwa Osuntokun
9daa659bb3
routing/chainview: convert chainview integration tests to use sub-tests 2017-07-04 15:53:58 -07:00
Andrey Samokhvalov
8fa2b95c12 lnd: remove seelog logger
The btclog package has been changed to defining its own logging
interface (rather than seelog's) and provides a default implementation
for callers to use.

There are two primary advantages to the new logger implementation.

First, all log messages are created before the call returns.  Compared
to seelog, this prevents data races when mutable variables are logged.

Second, the new logger does not implement any kind of artifical rate
limiting (what seelog refers to as "adaptive logging").  Log messages
are outputted as soon as possible and the application will appear to
perform much better when watching standard output.

Because log rotation is not a feature of the btclog logging
implementation, it is handled by the main package by importing a file
rotation package that provides an io.Reader interface for creating
output to a rotating file output.  The rotator has been configured
with the same defaults that btcd previously used in the seelog config
(10MB file limits with maximum of 3 rolls) but now compresses newly
created roll files.  Due to the high compressibility of log text, the
compressed files typically reduce to around 15-30% of the original
10MB file.
2017-06-25 14:19:56 +01:00
Olaoluwa Osuntokun
9b9f419e51
routing: fix vet error, ensure wait group in topologyClient isn't copied 2017-06-25 13:48:52 +01:00
Olaoluwa Osuntokun
5c45d52ab6
routing: wait for topology clients to fully exit before closing ntfn chan
This commit fixes a send on closed channel panic by adding additional
synchronization when cancelling the notifications for a particular
topology client. We now ensure that all goroutines belonging to a
particular topology client exit fully before we close the notification
channel in order to avoid a panic.
2017-06-25 13:31:55 +01:00
Olaoluwa Osuntokun
1be4d67ce4
multi: run all test instances in parallel 2017-06-17 01:00:07 +02:00
Olaoluwa Osuntokun
2452f2ed82
routing: update test to assert correctness of per-hop payloads 2017-06-16 22:45:30 +02:00
Olaoluwa Osuntokun
fca51d6165
routing: use converted hop payloads for sphinx packet when creating onion 2017-06-16 22:43:00 +02:00
Olaoluwa Osuntokun
4d8bb21d9d
routing: add ToHopPayloads method to routing.Route
This commit adds a new method to the routing.Route struct:
ToHopPayloads. This function will converts a complete route into the
series of per-hop payloads that is to be encoded within each HTLC using
an opaque Sphinx packet.

We can now use this function when creating the sphinx packet to
properly encoded the hop payload for each hop in the route.
2017-06-16 22:37:47 +02:00
Olaoluwa Osuntokun
dfd37df0ed
routing: set expiry on initial outgoing HTLC using total time lock 2017-06-16 22:35:14 +02:00
Olaoluwa Osuntokun
62cd6ee046
routing: populate the OutgoingTimeLock field within route hops
This commit inches towards fully validation+adherance of the per-hop
payloads within an HTLC’s route by properly calculating the outgoing
time lock value for each hop according to the current draft
specification.
2017-06-16 22:33:59 +02:00
Olaoluwa Osuntokun
e4563ca13b
routing/chainview: make filter updates synchronous for neutrino
This commit fixes a possible race condition wherein a call to
FilterBlock after a call to UpdateFilter would result in the call to
FilterBlock not yet using the updated filter. We fix this by ensuring
the internal chain filter is updated by the time the call to
FilterBlock returns.
2017-06-09 12:18:33 -07:00
Olaoluwa Osuntokun
0c134a8cb3
routing/chainview: if the chain filter is empty, don't fetch the cfilter 2017-06-08 23:11:24 -07:00
Olaoluwa Osuntokun
38beeebe3d
routing/chainview: optimize neutrino implementation of FilterBlock
This commit optimizes the neutrino implementation of FilterBlock method
of the ChainView interface. The old implementation would _always_ fetch
the entire block and manually scan through it. Instead, we can just
fetch the filter, and then if the items match, fetch the block itself.
This will save bandwidth during a lnd node’s pruning of the channel
graph after a period of dormancy.
2017-06-08 22:24:03 -07:00
Olaoluwa Osuntokun
7d30634757
routing/chainview: fix race condition in neutrino implementation 2017-06-06 12:01:24 -07:00
Olaoluwa Osuntokun
9fd70958f4
multi: fix linter errors 2017-06-05 19:45:01 -07:00
Olaoluwa Osuntokun
cbf7cd48f5
multi: update to latest neutrino API changes 2017-06-05 19:41:07 -07:00
Olaoluwa Osuntokun
8f81133d6c
chainntnfs/neutrinonotify: add ChainNotifier implementation for neutrino
This commit adds an initial rough implementation father ChainNotifier
interface for neutrino, our new light client implementation. This
implementation largely borrows from the existing BtcdNotifier
implementation. As a result, a follow up commit will perform two
refactoring in order to further consolidate code.
2017-06-05 19:04:56 -07:00
Olaoluwa Osuntokun
48ea8a3e70
routing/chainview: add neutrino light client to set of interface tests 2017-06-05 18:58:53 -07:00
Olaoluwa Osuntokun
cc25049684
routing/chainview: add new light client p2p version of FilteredChainView
This commit adds a new implementation of the FilteredChainView
interface. This implementation speaks purely to the p2p network and is
backed by a new experimental light client implementation.
2017-06-05 18:57:36 -07:00
Olaoluwa Osuntokun
461d84dd75
routing/chainview: report line # in test failures, bump up timeouts 2017-06-05 18:54:58 -07:00
bryanvu
4ac7cc719f lnwallet: replace hard-coded fees and adjust tests accordingly
This commit replaces the hard-coded 5000 satoshi fees with calls to the
FeeEstimator interface. This should provide a way to cleanly plug in
additional fee calculation algorithms in the future. This change
affected quite a few tests. When possible, the tests were changed to
assert amounts sent rather than balances so that fees wouldn't need to
be taken into account. There were several tests for which this wasn't
possible, so calls to the static fee calculator were made.
2017-05-15 20:26:11 -07:00
Olaoluwa Osuntokun
a4ba72c08c
routing: close exit chan before ntfnChan to cancel topology clients
This commit fixes a panic due to a send on a closed channel that could
possibly occur depending on the order of channel closes when a client
goes to cancel a topology notification client.

Previously we closed the ntfnChan first, this would possible result in
a panic as the goroutine may have succeeded on a send at the same time
the channel was closed. Instead, we now close the `exit` channel first
which is meant to be a signal to the goroutine that the client has been
canceled.
2017-05-15 18:47:26 -07:00
Olaoluwa Osuntokun
bfc869739d
routing: fix linter error 2017-05-15 18:21:13 -07:00
Olaoluwa Osuntokun
40a7523b8f
routing: fix incorrect logging fmt directive, log chanPoint for undo fetch failure 2017-05-15 18:20:49 -07:00
Olaoluwa Osuntokun
cc19695dad
routing: reject channel announcement if connected vertexes are unknown
This commit modifies the processing in the routing package eo new
announcements. Previously,  if we cgot a cnew channel announcement but
didn’t yet know of the verses that the chanell connected, the
cnnounacment would be accepted. This behavior was eronoues as if the
channel were to be queried for, the DB query would fail as we would be
unable to retrieve the two nodes involved int he channel.

To avoid such an error case, we will now _reject_ any channel
announcements in which we don’t yet have a valid node announcement for
the connected nodes. This case has been inserted into the handling of
channel announcement, a new test has been added, and finally older
tests have also been updated to ensure that nodes are added to the
database _before_ the edge is.
2017-05-14 19:27:24 -07:00
Olaoluwa Osuntokun
31cb1cb65b
routing/chainview: linter fixes 2017-05-11 15:36:47 -07:00
Olaoluwa Osuntokun
828d28581a
routing: abandon ChainNotifier for FilteredChainView
This commit modifies the routing package to no longer use the
ChainNotifier for pruning the channel graph. Instead, we now use the
FilteredChainView interface to more (from the ChannelRouter’s PoV)
efficiently maintain the channel graph.

Rather than scanning the _entire_ block manually, we now rely on the
FilteredChainView to provide us with FilteredBlocks which include
_only_ the relevant transactions that we care about.
2017-05-11 15:20:48 -07:00
Olaoluwa Osuntokun
2ec0f5788e
routing/chainview: add behavioral interface level tests
This commit adds a new set of behavioral interface level tests to the
chain view package. This set of tests can now be used in order to check
proper conformity to this “specification” for all future
implementations of the chain view package.
2017-05-11 15:20:45 -07:00
Olaoluwa Osuntokun
7a42e31a44
routing/chainview: add btcd-websockets impl of FilteredChainView
This commit adds the first concrete implementation of the
chainview.FilteredChainView interface. The implementation of this
interface, BtcdFilteredChainView is backed by a web sockets connection
to an active btcd instance.
2017-05-11 15:20:43 -07:00
Olaoluwa Osuntokun
7bdd7023f4
routing/chainview: add new chainview package for watching subsets of the UTXO set
This commit creates a new package as sub-package within the routing
package: chainview. This package is centered around a single interface
definition: the FilteredChainView. This interface is to be used to
allow the routing package to watch a _subset_ of the UTXO set for any
modifications. In the case of LN, the subset of the UTXO set that we
care about is the set of currently opened channels.

In a future commit the routing package will be modified to remove the
current full block scanning with processing of FilteredBlock
notification, and proper updates to the filter as observed by the
FilteredChainView.
2017-05-11 15:20:41 -07:00
Olaoluwa Osuntokun
a0c2278a69
routing: add TODO in newRoute for more sanity checks 2017-04-16 15:24:17 -07:00
Olaoluwa Osuntokun
1bb225a9bd
routing: print proper error msg on payment send fail 2017-04-14 17:12:04 -07:00
Olaoluwa Osuntokun
79807022a5
routing: update graph traversal to use latest API 2017-04-14 13:15:00 -07:00
Olaoluwa Osuntokun
c8bf521c75
routing: display channel capacity in info logging message 2017-04-14 11:48:09 -07:00
Olaoluwa Osuntokun
5442e42cc1
routing: fix slice mutation bug that could result in an infinite loop
This commit fixes a pretty nasty unnoticed bug within the main
k-shortest paths algorithm loop. After a new candidate path is found,
the rootPath (the path up to the pivot node) and the spurPath (the
_new_ path after the pivot node) are to be combined into a new candiate
shortest path. The prior logic simply appended the spurPath onto the
end of the rootPath to create a slice. However, if the case that the
currnet rootPath is really a sub-path in a larger slice, then this will
mutate the underlying slice.

This bug would manifest when doing path finding and cause an infinite
loop as the slice kept growing with new spurPaths, causing the loop to
never terminate. We remedy this bug by properly create a new backing
slice, and adding the elements to them rather than incorrectly mutating
an underlying slice.
2017-04-13 14:48:17 -07:00
Olaoluwa Osuntokun
a4e26eaa4a
routing: fix bug in path finding when len(rootPath) > len(shortestPath)
This commit fixes a bug within the k-shortest paths routine which could
result in a daemon panic when traversing a graph with particular
characteristics. Before referencing the path to create a sub-slice, we
we’re properly asserting that the length of the path was at least as
long as the current rootPath in question. We fix this by simply
ensuring the length of the slice is adequate before proceeding with the
operation.
2017-04-13 14:44:59 -07:00
Olaoluwa Osuntokun
2d10d83f07
routing: assert that paths have same length in isSamePath 2017-04-13 14:42:35 -07:00
Olaoluwa Osuntokun
41a54145a7
routing: capitalize first letter of new error messages 2017-04-11 22:02:17 -07:00
Olaoluwa Osuntokun
ca053e5273
multi: minor coding style and comment clean ups post-discovery merge
This commit implements some minor coding style, commenting and naming
clean up after the recent major discovery service was merged into the
codebase.

Highlights of the naming changes:
* fundingManager.SendToDiscovery -> SendAnnouncement
* discovery.Discovery -> discovery.AuthenticatedGossiper

The rest of the changes consist primary of grammar fixes and proper
column wrapping.
2017-04-01 20:14:05 +02:00
Andrey Samokhvalov
07076cce21 routing+channeldb: add AddProof method
Originally we adding the edge without proof in order to able to use it
for payments path constrcution. This method will allow us to populate
the announcement proof after the exchange of the half proofs and
constrcutrion of full proof finished.
2017-03-29 19:49:05 -07:00
Andrey Samokhvalov
4bca54e30c routing: add validation of utxo
Add check that the edge that was received really exist in the blockchain
and that the announced funding keys and capcity corresponds to reality.
2017-03-29 19:49:05 -07:00
Andrey Samokhvalov
19174ebdfd channeldb: add storing of node signature and add edge signature
In order to properly announce the channel the announcements proofs
should be persistent in boltdb.
2017-03-29 19:49:05 -07:00
Andrey Samokhvalov
c3b2854428 lnwire: converge discovery part of messages with specification
Change the name of fields of messages which are belong to the discovery
subsystem in a such way so they were the same with the names that are
defined in the specification.
2017-03-29 19:49:05 -07:00
Andrey Samokhvalov
4c52b6e6a4 lnd: replace 'routing' with 'discovery' package
Add usage of the 'discovery' package in the lnd, now discovery service
will be handle all lnwire announcement messages and send them to the
remote party.
2017-03-29 19:49:05 -07:00
Andrey Samokhvalov
b4ac7071ff discovery+routing: split 'routing' package on 'routing' and 'discovery'
In this commit the routing package was divided on two separete one,
this was done because 'routing' package start take too much responsibily
on themself, so with following commit:

Routing pacakge:
Enitites:
* channeldb.ChannelEdge
* channeldb.ChannelPolicy
* channeldb.NodeLightning

Responsibilities:
* send topology notification
* find payment paths
* send payment
* apply topology changes to the graph
* prune graph
* validate that funding point exist and corresponds to given one
* to be the source of topology data

Discovery package:
Entities:
* lnwire.AnnounceSignature
* lnwire.ChannelAnnouncement
* lnwire.NodeAnnouncement
* lnwire.ChannelUpdateAnnouncement

Responsibilities:
* validate announcement signatures
* sync topology with newly connected peers
* handle the premature annoucement
* redirect topology changes to the router susbsystem
* broadcast announcement to the rest of the network
* exchange channel announcement proofs

Before that moment all that was in the 'routing' which is quite big for
one subsystem.

split
2017-03-29 19:49:05 -07:00
bryanvu
085b7333cb lnwire: add support for Features in NodeAnnouncement
Add support for Features in NodeAnnouncment according to spec.
2017-03-29 12:03:43 -07:00
bryanvu
9ffac9eae1 lnwire: update NodeAnnouncement to handle multiple addresses
This commit modifies address handling in the NodeAnnouncement struct,
switching from net.TCPAddr to []net.Addr. This enables more flexible
address handling with multiple types and multiple addresses for each
node. This commit addresses the first part of issue #131 .
2017-03-29 12:03:43 -07:00
Olaoluwa Osuntokun
6b3a258e86
multi: fix formatting issues in packge README's 2017-03-27 16:25:25 -07:00
Olaoluwa Osuntokun
69b257154f
routing: fix build on go 1.7.5 by using sort.Interface instead of sort.Lice
This commit fixes the issue of broken builds in versions other than go
1.7.5 by sorting according to the sort.Interface interface rather than
the newly available sort.Slice function.
2017-03-21 12:21:00 -07:00
Olaoluwa Osuntokun
3da8cd7551
routing: add a caching layer in front of the KSP algorithm
This commit adds caching to our route finding. Caching is done on a
tuple-basis mapping a (dest, amt) pair to a previously calculated set
of shortest paths. The cache invalidated on two occasions: when a block
closes a set of transactions, or we received a new channel update or
channel announcement message.

With this change, payments are now snappier from the PoV of an
application developer as we no longer need to do a series of disk-seeks
before we dispatch each payment.
2017-03-21 12:20:55 -07:00
Olaoluwa Osuntokun
b126298b2b
routing: implement route failure fallback in SendPayment
This commit adds payment route failure fallback to SendPayment. By
this, we mean that we now take all the possible routes found during
path finding and try them in series. Either a route fails and we move
onto the next one, or the route is successful and we terminate early.

With this commit, sending payments using lnd is now much more robust as
if there exists an eligible route with sufficient capacity, it will be
utilized.
2017-03-21 12:20:45 -07:00
Olaoluwa Osuntokun
9818f662cf
routing: exit early in FindRoutes if unable to convert any paths to a route 2017-03-21 12:20:42 -07:00
Olaoluwa Osuntokun
1df5bece85
routing: FindRoute is now FindRoutes, uses our KSP algo, ranks by fee
This commit modifies the existing FindRoute method on the ChannelRouter
to now use the KSP implementation added in a prior commit.

This new method FindRoutes, is able to find all the possible paths
between a source and destination. The method takes all paths reported
by findPaths, and attempt to turn each of them into a route. A route
differs from a path in that is has complete time-lock and fee
information. Some paths may not be able to be turned into routes as
once fees are accounted for the have an insufficient flow. We then take
the routes, sort them by total fee (with time-lock being a
time-breaker), then return them in sorted order.
2017-03-21 12:20:40 -07:00
Olaoluwa Osuntokun
aaa04bb2e5
routing: add findPaths function whcih implements Yen's Algorithm
With this commit we make our routing more robust by looking for the
k-shortest paths rather than a single shortest path and using that
unconditionally. In a nut shell Yen’s algorithm does the following:
   * Find the shortest path from the source to the destination
   * From k=1…K, walk the kth shortest path and find possible
divergence from each node in the path

Our version of Yen’s implemented is slightly modified, rather than
actually increasing the edge weights or remove vertexes from the graph,
we instead use two black-lists: one for edges and the other for
vertexes. Our modified version of Djikstra’s algorithm is then made
aware of these black lists in order to more efficiently implement the
path iteration via spur and root node.
2017-03-21 12:20:37 -07:00
Olaoluwa Osuntokun
56d27f2d58
routing: rename findRoute to findPath, return path in forwards order
This commit modifies the findRoute method by first calling it findPath,
but also making the following modifications.

First, two new black-listing maps are now passed in. These two maps
contain vertexes but also edges to ignore while performing path
finding. These maps will be used in order to ensure that we don’t
duplicate paths or back-track when executing our KSP algorithm.

Next, we now ensure that the path returned from the findPath function
is ordered properly in the direction of source to target. Such a change
is required for our KSP algorithm to function properly.
2017-03-21 12:20:35 -07:00
Olaoluwa Osuntokun
fd18c2d036
routing: newRoute now expects the path in forwards order 2017-03-21 12:20:33 -07:00
Olaoluwa Osuntokun
9f96ceb1e0
routing: introduce new heap for storing candidate shortest paths
This commit adds a new heap structure to heap.go which will be used for
storing candidate paths during the iterations of the k-shortest paths
algorithm.
2017-03-21 12:20:30 -07:00
Olaoluwa Osuntokun
e0ef63e4e0
routing: findRoute now returns a slice of selected hops in reverse order
This commit modifies the findRoute function to decouple the
validation+creation of a route, from the path finding algorithm itself.
When we say “route”, we mean the full payment route complete with
time-lock and fee information. When we say “path” we simple mean an
ordered set of channel edges from one node to another target node.

With this commit we can now perform path finding independent of route
creation which will be needed in the up coming refactor to implement a
new modified k-shortest paths algorithm.
2017-03-21 12:20:28 -07:00
Olaoluwa Osuntokun
c6c56173a8
routing: modify findRoute to accept starting node as a paramter
This commit slightly modified findRoute to accept the node which should
be used as the starting point in our path finding algorithm. With this
change, as we move to a k-shortest paths algorithm this modification
will be needed as all of our path finding attempts won’t always
originate from a the same starting point.
2017-03-21 12:20:25 -07:00
Olaoluwa Osuntokun
b6199f27da
routing: replace linear vertex scan with distanceHeap usage
In this commit we now utilize the node distance heap that was added in
a prior commit into our core path finding logic. With this new data
structure, we no longer linearly scan the distance of all vertexes from
the source node when deciding which one to greedily explore.

Instead, we now start with the source added to our distance heap, then
new vertexes are progressively added to our heap as their edges are
explored. With this change, we move the computational complexity of our
path finding algorithm closer to the theoretical limit.
2017-03-21 12:20:23 -07:00
Olaoluwa Osuntokun
20aba8060f
routing: add sufficient link capacity to our relaxation condition
This commit modifies our modified version of Dijkstra's to include
sufficient link capacity within the algorithm’s relaxation condition.
With this change, we’ll now avoid exploring the graph in the direction
of a link that has insufficient capacity, allowing us to terminate path
finding more quickly.
2017-03-21 12:20:21 -07:00
Olaoluwa Osuntokun
606b23df43
routing: introduce a heap to keep track of closest nodes during pathfinding
This commit introduces a new heap struct that will be used to keep
track of the next closest node to the source during path finding within
our modified Dijkstra's algorithm.
2017-03-21 12:20:18 -07:00
Olaoluwa Osuntokun
d0509955a7
routing: only log topology client sends if clients are active 2017-03-16 12:16:32 -07:00
Olaoluwa Osuntokun
8ed79ae497
routing: fix bug when sending topology ntfns to multiple clients
This commit fixes a bug which was originally introduced when the
topology notifications were added to the channel router. The issue was
that a pointer to the loop-scope range variable was being passed into
the goroutine which dispatches the notification rather than the value
itself. It seems that the memory location is re-used between range
iterations causing the same client to receive _all_ the notifications.

This bug is fixed by passing a copy of the client struct rather than a
pointer to the range variable.

In the process, we also add some additional debug logging messages, and
remove the Curve parameter from any public keys involved in a
notification so the pretty print properly.
2017-03-14 20:07:55 -07:00
Olaoluwa Osuntokun
f6ab1390ed
routing: include the channel point of chan within edge update ntfn
This commit modifies the `ChannelEdgeUpdate` struct to include the
channel point itself within the notifications. Such a change improves
the notificaiton experience for callers as it allows them to filter out
update notifications based on a familiar object within the codebase: a
channel point.
2017-03-14 20:07:33 -07:00
Andrey Samokhvalov
61991a1c89 lnd: fix latest goclean.sh lint warning 2017-03-13 16:30:23 -07:00
Andrey Samokhvalov
fd97a4bd19 lnd: partially fix golint warnings 2017-03-13 16:30:23 -07:00
Andrey Samokhvalov
f2843dd4c9 lnd: fix gofmt warnings 2017-03-13 16:30:23 -07:00
Olaoluwa Osuntokun
addab3273f
routing: add a set of test cases for client topology notifications 2017-03-08 14:46:18 -08:00
Olaoluwa Osuntokun
1a78c73f7e
routing: add new notificaiton client for topology changes
This commit adds some new functionality to the channel router: the
ability to dispatch notification to registered clients upon either a
channel being closed, a new node appearing, or an exiting client being
updated or opened for the first time.

With this change, the integration tests will now be able to eliminate
most of the sleep as we gain a new syntonization point into the
propagation of information within the test network. Additionally, this
also paves the way for client side software to dynamically visualize
the channel graph in real-time as nodes+channels are updated.
2017-03-08 14:46:14 -08:00
Olaoluwa Osuntokun
7bdf02bc9e
routing: modify path finding routines use new EdgeInfo/EdgePolicy
This commit modifies the path finding routines to properly use the new
channel edge related API exposed by the database. Additionally, a new
type `ChannelHop` has been introduced which couples an edges routing
policy with the capacity and origin chain of the channel.
2017-03-08 14:25:05 -08:00
Olaoluwa Osuntokun
2dfab8c6d7
routing+lnd: provide payment premiere as response to SendPayment 2017-02-21 01:43:48 -08:00
Olaoluwa Osuntokun
78561c3e35
routing: fix compile error in definition due to chainntfns API change 2017-02-21 01:43:01 -08:00
Olaoluwa Osuntokun
12f69692aa
multi: update sub-systems to use latest iteration of wire messages 2017-02-21 01:42:37 -08:00
Olaoluwa Osuntokun
5560f032f6
rpc+routing: fix panic when channel edge but no edge info in db 2017-02-16 19:35:32 +08:00
Olaoluwa Osuntokun
384fe61e73
multi: fix go vet warnings throughout code base 2017-02-16 19:33:19 +08:00
Olaoluwa Osuntokun
a094681dae
routing: only log pre-mature announcements if we have any 2017-02-07 18:38:48 -08:00
Olaoluwa Osuntokun
bf78122dc7
routing: ensure advertised transaction index is within bounds
This commit fixes bug that could result in the panicking or crashing of
nodes in the case of an at-funding-time reorganization within the
network. In order to avoid such a case, we now ensure that the
advertised transaction index is within the bounds of the block before
attempting to access it.

Note that this is a temporary patch commit until full advertisement
validation which is implemented in the discovery PR lands in master.
Additionally, better reorg handling during the funding process is being
specified within the spec and will properly be implemented within lnd
at a later date.
2017-02-06 14:54:57 -08:00
Olaoluwa Osuntokun
9cef2f8657
routing: add support for detection+processing of premature announcements
This commit fixes a prior block propagation race-condition by detecting
and properly processing “premature” announcements. A premature
announcement is one that’s received with an anchored block height which
is beyond our chain tip. Once received, we now store these
announcements in a special map that’s caches them in memory. Once a new
block arrives, we check the map for the existence of any entries,
processing them as normal if so.
2017-02-02 17:44:25 -08:00
Christopher Jämthagen
24a569b047 routing: add proper test for excessive number of hops 2017-02-02 00:24:16 -08:00
Christopher Jämthagen
4200d8a7e0 routing: check route length after edges for route is collected
This fixes the bug reproduced in #114. The prevHop map in newRoute may
include many more edges than what is used to produce the final route, and
thus the check prior to building the route could result with incorrect
errors being reported. We move this check to after the number of edges
to be used for the route is deduced.
2017-02-02 00:24:16 -08:00
Olaoluwa Osuntokun
08f0d0fbea
routing+rpcserver: move route querying+sending to SendPayment
This commit moves much of the logic for querying for a potential route,
constructing the HTLC including the Sphinx packet, and sending the
ultimate payment from the rpcServer to the ChannelRouter.

This movement paves the way for muilt-path path finding as well as
adding automatic retry logic to the ChannelRouter. Additionally, by
having the ChannelRouter construct the Sphinx packet, we’ll be able to
also include the proper time-lock and general per-hop-payload
information properly in the future.
2017-02-01 18:29:55 -08:00
Olaoluwa Osuntokun
db40b4322e
routing: rebroadcast our outgoing channels with a long period
This commit adds new behavior to the ChannelRouter struct: we know
rebroadcast our outgoing channels every 30 minutes. This new behavior
should ensure that both directions of an advertised channel edge are
always propagated though the network, fixing the issue of “ghost” edges
which exist but aren’t advertised.
2017-01-22 14:40:09 -08:00
Olaoluwa Osuntokun
8fd4d7ea6b
routing: fix bug in channel capacity display by ensuring properly typed arithmetic
This commit fixes a slight bug in the storage of the capacity of a
channel. Previously, we were subtracting a the hard coded fee amount
without first casting the integer to a btcutil.Amount which results in
a display/rounding error when the amount is converted to BTC.
2017-01-22 14:21:58 -08:00
Trevin Hofmann
40c7bac3aa multi: fix a variety of typos throughout the repo 2017-01-17 17:02:56 -08:00
Olaoluwa Osuntokun
7312565644
routing: allow full syncing graph state with partially advertised edges
This commit is similar to the prior commit to channeldb: we no longer
assume that _both_ edges of a channel will always be advertised. Such
an assumption resulted in the inability for a node to sync graph state
since we were previously returning an error when _both_ edges weren’t
found within the graph database.

To remedy this bug, we now carefully ensure that if one edge doesn’t
exist, then we still sync the other.
2017-01-17 13:18:05 -08:00
Olaoluwa Osuntokun
0c7fcb1755
routing: fix nil pointer panic when node has no outgoing channels 2017-01-17 13:07:05 -08:00
Trevin Hofmann
a13ac90d46 multi: add link to LICENSE in README license badges (#100) 2017-01-12 16:31:08 -08:00
Olaoluwa Osuntokun
4ccdad0d66
multi: add README's for all sub-packages 2017-01-10 15:02:37 -08:00
Olaoluwa Osuntokun
f1e23748c9
routing/testdata: ensure test channels have unique chan ID's
This commit fixes a bug in the test data that was uncovered due to the
recent bug fix within the AddChannelEdge method within the ChannelGraph
struct of channeldb.

The storage and assertion of unique channel ID’s wasn’t correct due to
bug in the channeldb which caused the defect in the test data to go
unnoticed.
2017-01-09 19:47:52 -08:00
Olaoluwa Osuntokun
ec0c7c5989
routing: fix panic in inner loop of path finding
This commit seems to fix a sporadic error within the integration tests
which would at times cause a panic when a payment as initiated.

This issue was with the way were deleting from the middle of the slice
of unvisited nodes within the graph. Assigning the last element to the
middle would at times cause a panic the last element may be nil. To fix
this, we now manually copy every item over by one, preserving the order
of the slice, and possibly fixing the panic once and for all.
2017-01-07 21:22:23 -08:00
Olaoluwa Osuntokun
5affed38fc
multi: update btcsuite API's to latest upstream changes
This commit makes a large number of minor changes concerning API usage
within the deamon to match the latest version on the upstream btcsuite
libraries.

The major changes are the switch from wire.ShaHash to chainhash.Hash,
and that wire.NewMsgTx() now takes a paramter indicating the version of
the transaction to be created.
2017-01-05 13:56:34 -08:00
Olaoluwa Osuntokun
82815b703e
rpcserver: refactor logic for ListPayments/DeleteAllPayments
This commit slightly refactors the logic for the new outgoing payment
related RPC’s to more closely match the style of the rest of the
codebase. Additionally the tests have been updated to reflect the
changes to the protos of the new RPC’s.
2016-12-30 16:42:10 -08:00
Olaoluwa Osuntokun
e327ffe954
routing: rewrite package to conform to BOLT07 and factor in fees+timelocks
This commit overhauls the routing package significantly to simplify the
code, conform to the rest of the coding style within the package, and
observe the new authenticated gossiping scheme outlined in BOLT07.

As a major step towards a more realistic path finding algorithm, fees
are properly calculated and observed during path finding. If a path has
sufficient capacity _before_ fees are applied, but afterwards the
finalized route would exceed the capacity of a single link, the route
is marked as invalid.

Currently a naive weighting algorithm is used which only factors in the
time-lock delta at each hop, thereby optimizing for the lowest time
lock. Fee calculation also isn’t finalized since we aren’t yet using
milli-satoshi throughout the daemon. The final TODO item within the PR
is to properly perform a multi-path search and rank the results based
on a summation heuristic rather than just return the first (out of
many) route found.

On the server side, once nodes are initially connected to the daemon,
our routing table will be synced with the peer’s using a naive “just
send everything scheme” to hold us over until I spec out some a
efficient graph reconciliation protocol. Additionally, the routing
table is now pruned by the channel router itself once new blocks arrive
rather than depending on peers to tell us when a channel flaps or is
closed.

Finally, the validation of peer announcements aren’t yet fully
implemented as they’ll be implemented within the pending discovery
package that was blocking on the completion of this package. Most off
the routing message processing will be moved out of this package and
into the discovery package where full validation will be carried out.
2016-12-27 16:44:22 -08:00
Olaoluwa Osuntokun
b49d2827a2
lnwallet/btcwallet: add compile-time interface assertions for Signer+BlockChainIO 2016-12-27 16:43:31 -08:00
Olaoluwa Osuntokun
ddc1eb4c8a
lnwallet: correct comment for logging object 2016-12-27 16:43:07 -08:00
BitfuryLightning
327768f4ad routing: Move tools inside lnd. Refactor and delete unneeded stuff
Use [33]byte for graph vertex representation.
Delete unneeded stuff:
1. DeepEqual for graph comparison
2. EdgePath
3. 2-thread BFS
4. Table transfer messages and neighborhood radius
5. Beacons

Refactor:
1. Change ID to Vertex
2. Test use table driven approach
3. Add comments
4. Make graph internal representation private
5. Use wire.OutPoint as  EdgeId
6. Decouple routing messages from routing implementation
7. Delete Async methods
8. Delete unneeded channels and priority buffer from manager
9. Delete unneeded interfaces in internal graph realisation
10. Renamed ID to Vertex
2016-11-23 20:37:43 -06:00