Commit Graph

678 Commits

Author SHA1 Message Date
Joost Jager
559d3c0910
routing: also fail pairs for node-level failures
This commit modifies the interpretation of node-level failures.
Previously only the failing node was marked. With this commit, also the
incoming and outgoing connections involved in the route are marked as
failed.

The change prepares for the removal of node-level failures in mission
control probability estimation.
2019-10-22 15:52:34 +02:00
Joost Jager
843c87a203
routing: create mc getWeight method 2019-10-22 15:52:32 +02:00
Joost Jager
a3f7dbc633
routing: change representation of pair results in mc
This commit changes the in-memory structure of the mission control
state. It prepares for calculation of a node probability. For this we
need to be able to efficiently look up the last results for all channels
of a node.
2019-10-22 15:52:30 +02:00
Joost Jager
aefbee78d6
routing: use pairResult constructors
To make it explicit whether a failure or a success result is
instantiated.
2019-10-22 15:52:28 +02:00
Joost Jager
37ef46bc48
routing/test: extract mission control test parameters 2019-10-22 15:52:26 +02:00
Johan T. Halseth
610a93e2ff
routing/chainview: check GetCFilter error
Since the filter will never be nil in non-error cases now, remove the
nil-check.
2019-10-15 10:31:25 +02:00
Wilmer Paulino
0fc401de19
routing+routerrpc: take max cltv limit into account within path finding
With the introduction of the max CLTV limit parameter, nodes are able to
reject HTLCs that exceed it. This should also be applied to path
finding, otherwise HTLCs crafted by the same node that exceed it never
left the switch. This wasn't a big deal since the previous max CLTV
limit was ~5000 blocks. Once it was lowered to 1008, the issue became
more apparent. Therefore, all of our path finding attempts now have a
restriction of said limit in in order to properly carry out HTLCs to the
network.
2019-10-11 18:04:49 -04:00
Olaoluwa Osuntokun
5ef1f85998
Merge pull request #3578 from wpaulino/no-freelist-sync-option
config: add sync freelist cli flag
2019-10-04 15:16:30 -07:00
Olaoluwa Osuntokun
c03fb8aca0
Merge pull request #3561 from Roasbeef/chan-validation-cleanup
chanvalidate: create new channel validation package
2019-10-04 12:32:07 -07:00
Wilmer Paulino
194a9dea81
multi: support sync freelist option within btcwallet 2019-10-04 12:19:40 -04:00
Olaoluwa Osuntokun
6ebada112f
routing: update 3d party channel verification to use new chanvalidate package
In the process of moving to use the new package, we no longer need to
fetch the outpoint directly, and instead only need to pass the funding
transaction into the new verification logic.
2019-10-03 16:23:17 -07:00
Wilmer Paulino
5ed1084130
Merge pull request #3572 from joostjager/canceled-spelling
multi: fix canceled spelling
2019-10-03 16:50:06 -04:00
Joost Jager
b58dbb2d70
multi: fix canceled spelling 2019-10-03 17:27:36 +02:00
Joost Jager
9f3656efeb
routing: log performance metrics 2019-10-03 15:07:11 +02:00
Olaoluwa Osuntokun
18f88cbd8d
Merge pull request #3440 from joostjager/buildroute
routing: add build route functionality
2019-09-24 20:24:24 -07:00
Joost Jager
299821152a
routing+routerrpc: add BuildRoute function 2019-09-24 10:00:44 +02:00
Valentine Wallace
9a52cb6dab
multi: update internals to support updating max htlc
In this commit, we update the router and link to support users
updating the max HTLC policy for their channels. By updating these internal
systems before updating the RPC server and lncli, we protect users from
being shown an option that doesn't actually work.
2019-09-23 13:07:10 +02:00
Joost Jager
c80feeb4b3
routing+discovery: extract local channel manager
The policy update logic that resided part in the gossiper and
part in the rpc server is extracted into its own object.

This prepares for additional validation logic to be added for policy
updates that would otherwise make the gossiper heavier.

It is also a small first step towards separation of our own channel data
from the rest of the graph.
2019-09-23 13:07:08 +02:00
Joost Jager
fd63ed13ff
routing/route: add vertex constructor for hex string 2019-09-17 11:11:00 +02:00
Joost Jager
8c44cf4a22
channeldb: add ComputeFee function 2019-09-17 11:10:58 +02:00
Joost Jager
f60e4b1e14
lnwire+htlcswitch: report height for invalid payment details failure
Extends the invalid payment details failure with the new accept height
field. This allows sender to distinguish between a genuine invalid
details situation and a delay caused by intermediate nodes.
2019-09-16 10:10:20 +02:00
Lars Lehtonen
3b29ecb921 routing: Fix dropped errors in tests.
lnd: Fix dropped errors in tests.

contractcourt: Fix dropped errors in tests.

htlcswitch: Fix dropped errors in tests.

invoices: Fix dropped error in tests.

lnwallet: Fix dropped errors in tests.

macaroons: Fix dropped error in tests.
2019-09-13 06:28:05 +00:00
Joost Jager
3d7de2ad39
multi: remove dead code 2019-09-10 17:21:59 +02:00
Conner Fromknecht
947cc50c13
routing/route/route: prevent modification of hop.TLVRecords
Currently the underlying array backing the hop's TLVRecords is modified
when combining custom records with the primitive forwarding info. This
commit uses a fresh slice to prevent modifications from mutating the
hop itself.
2019-09-04 14:14:34 -07:00
Conner Fromknecht
278e10a2fd
routing/route/route: omit next_hop_id for final hop
BOLT04 says to omit this field for final hops, but must be present on
all other hops.
2019-09-04 12:05:10 -07:00
Conner Fromknecht
dc0f429271
htlcswitch+routing: use TLV hop constructors 2019-09-04 12:05:09 -07:00
Joost Jager
ff0c5a0d5e
routing: process successes in mission control
This commit modifies paymentLifecycle so that it not only feeds
failures into mission control, but successes as well.
This allows for more accurate probability estimates. Previously,
the success probability for a successful pair and a pair with
no history was equal. There was no force that pushed towards
previously successful routes.
2019-08-23 09:15:41 +02:00
Joost Jager
2594abfba1
routing/test: remove unused parameter in mission control test 2019-08-23 09:15:39 +02:00
Olaoluwa Osuntokun
4697cfde30
routing: extend path finding to be TLV-EOB aware, allow dest TLV records
In this commit, we extend the path finding to be able to recognize when
a node needs the new TLV format, or the legacy format based on the
feature bits they expose. We also extend the `LightningPayment` struct
to allow the caller to specify an arbitrary set of TLV records which can
be used for a number of use-cases including various variants of
spontaneous payments.
2019-08-22 18:53:01 -07:00
Olaoluwa Osuntokun
5b4c8ac232
routing/route+tlv: add new TLV-EOB awareness to Hop+Route
In this commit, we extend the Hop struct to carry an arbitrary set of
TLV values, and add a new field that allows us to distinguish between
the modern and legacy TLV payload.

We add a new `PackPayload` method that will be used to encode the
combined required routing TLV fields along any set of TLV fields that
were specified as part of path finding.

Finally, the `ToSphinxPath` has been extended to be able to recognize if
a hop needs the modern, or legacy payload.
2019-08-22 18:53:00 -07:00
Joost Jager
d9ec158412
routing: stricter payment result interpretation
This commit overhauls the interpretation of failed payments. It changes
the interpretation rules so that we always apply the strongest possible
set of penalties, without making assumptions that would hurt good nodes.

Main changes are:

- Apply different rule sets for intermediate and final nodes. Both types
of nodes have different sets of failures that we expect. Penalize nodes
that send unexpected failure messages.

- Distinguish between direct payments and multi-hop payments. For direct
payments, we can infer more about the performance of our peer because we
trust ourselves.

- In many cases it is impossible for the sender to determine which of
the two nodes in a pair is responsible for the failure. In this
situation, we now penalize bidirectionally. This does not hurt the good
node of the pair, because only its connection to a bad node is
penalized.

- Previously we always penalized the outgoing connection of the
reporting node. This is incorrect for policy related failures. For
policy related failures, it could also be that the reporting node
received a wrongly crafted htlc from its predecessor. By penalizing the
incoming channel, we surely hit the responsible node.

- FailExpiryTooSoon is a failure that could have been caused by any node
up to the reporting node by delaying forwarding of the htlc. We don't
know which node is responsible, therefore we now penalize all node pairs
in the route.
2019-08-17 10:24:09 +02:00
Joost Jager
e7a457f1ce
routing: query bandwidth hints before each payment attempt
Previously the bandwidth hints were only queried once per payment. This
did not allow for concurrent payments changing channel balances.
2019-08-17 10:24:07 +02:00
Joost Jager
e135cf7326
routing: penalize all node pairs for unknown outcomes
When an undecryptable failure comes back for a payment attempt, we
previously only penalized our own outgoing connection. However,
any node could have caused this failure. It is therefore better to
penalize all node connections along the route. Then at least we know for
sure that we will hit the responsible node.
2019-08-17 10:24:03 +02:00
Joost Jager
c39d7a29cd
routing/test: do not test local channel mission control
This commit updates existing tests to not rely on mission control for
pruning of local channels. Information about local channels should
already be up to date before path finding starts. If not, the problem
should be fixed where bandwidth hints are set up.
2019-08-17 10:24:01 +02:00
Joost Jager
45dacd0df1
routing: isolate failure interpretation from mission control
This commit moves the payment outcome interpretation logic into a
separate file. Also, mission control isn't updated directly anymore, but
results are stored in an interpretedResult struct. This allows the
mission control state to be locked for a minimum amount of time and
makes it easier to unit test the result interpretation.
2019-08-17 10:23:59 +02:00
Joost Jager
e7af6a077a
routing: convert to nillable failure reason
This commit converts several functions from returning a bool and a
failure reason to a nillable failure reason as return parameter. This
will take away confusion about the interpretation of the two separate
values.
2019-08-17 10:23:57 +02:00
Joost Jager
f1769c8c8c
routing: convert to node pair based
Previously mission control tracked failures on a per node, per channel basis.
This commit changes this to tracking on the level of directed node pairs. The goal
of moving to this coarser-grained level is to reduce the number of required
payment attempts without compromising payment reliability.
2019-08-13 19:21:37 +02:00
Joost Jager
a332990d2c
routing/route: add vertex constructor from bytes 2019-08-13 18:45:08 +02:00
Joost Jager
b610f417d2
routing: extended DirectedNodePair functionality 2019-08-13 18:45:06 +02:00
Joost Jager
2b4debf42b
routing/test: remove unused methods from mock 2019-08-13 18:45:02 +02:00
Joost Jager
aabd68ebcd
multi: rename FailUnknownPaymentHash to FailIncorrectDetails
Align naming better with the lightning spec. Not the full name of the
failure (FailIncorrectOrUnknownPaymentDetails) is used, because this
would cause too many long lines in the code.
2019-08-08 11:04:17 +02:00
Joost Jager
7e7b620355
routing: persist mission control data 2019-07-31 08:44:00 +02:00
Joost Jager
9e26f0d079
routing: define payment result
This commit groups together all payment result data. It is a preparation
for historical payment results being retrieved from the database.
2019-07-29 09:38:36 +02:00
Joost Jager
1507ff6590
routing: extract timestamp from applying payment result
This commit prepares for timestamps being supplied from the database for
processing of historical payment results.
2019-07-29 09:38:34 +02:00
Joost Jager
b4a7665bae
routing: provide payment id to mission control 2019-07-29 09:38:32 +02:00
Joost Jager
334b6a3bfe
routing: move unreadable failure handling into mission control 2019-07-29 09:38:30 +02:00
Joost Jager
934ea8e78d
routing: move failure interpretation into mission control 2019-07-29 09:38:26 +02:00
nsa
7762d55e80 itest+routing: updating tests to account for BlockPadding 2019-07-23 21:03:30 -04:00
nsa
3dde7bc4e2 routing: adding the BlockPadding value to sendpayment
This commit adds the BlockPadding value (currently 3) to sendpayment
calls so that if some blocks are mined while the htlc is in-flight, the
exit hop won't reject it.
2019-07-23 21:03:11 -04:00
Olaoluwa Osuntokun
8c389d13f9
Merge pull request #3197 from breez/optimize_prune_zombie_channels
Optimize prune zombie channels
2019-07-18 20:56:56 -07:00
Roei Erez
da9edc876a router: only prune disabled channels when AssumeChannelValid=true. 2019-07-18 17:05:20 +03:00
Joost Jager
34b264a3d8
routing: create PathFindingConfig for global parameters 2019-07-18 15:49:25 +02:00
Olaoluwa Osuntokun
92e14af72a
Merge pull request #3298 from joostjager/globalize-mc
routing: globalize mc
2019-07-16 17:39:36 -07:00
Roei Erez
9781ea0082 router: Optimize pruneZombieChannels.
The current approach iterates all channels in the graph in order to
filter those in need. This approach is time consuming, several seconds
on my mobile device for ~40,000 channels, while during this time the
db is locked in a transaction.

The proposed change is to use an existing functionality that utilize the
fact that channel update are saved indexed by date. This method enables
us to go over only a small subset of the channels, only those that
were updated before the "channel expiry" time and further filter
them for our need.
The same graph that took several seconds to prune was pruned, after
the change, in several milliseconds.

In addition for testing purposes I added Initiator field to the
testChannel structure to reflect the channeldEdgePolicy direction.
2019-07-16 17:48:07 +03:00
Joost Jager
8055bcf2e0
routing: report failures to mission control directly
As there is no more state kept in the payment session, failure reporting
can go straight to mission control.
2019-07-13 22:38:25 +02:00
Joost Jager
dc13da5abb
routing: move second chance logic into mission control
If nodes return a channel policy related failure, they may get a second
chance. Our graph may not be up to date. Previously this logic was
contained in the payment session.

This commit moves that into global mission control and thereby removes
the last mission control state that was kept on the payment level.

Because mission control is not aware of the relation between payment
attempts and payments, the second chance logic is no longer based
tracking second chances given per payment.

Instead a time based approach is used. If a node reports a policy
failure that prevents forwarding to its peer, it will get a second
chance. But it will get it only if the previous second chance was
long enough ago.

Also those second chances are no longer dependent on whether an
associated channel update is valid. It will get the second chance
regardless, to prevent creating a dependency between mission control and
the graph. This would interfer with (future) replay of history, because
the graph may not be the same anymore at that point.
2019-07-13 22:38:23 +02:00
Joost Jager
d31efddf1b
routing: move generateBandwidthHints 2019-07-13 22:38:21 +02:00
Joost Jager
37e2751695
routing+routerrpc: isolate payment session source from mission control 2019-07-13 22:38:19 +02:00
Joost Jager
cf3dd3fb94
routing/test: create mission control test context 2019-07-13 22:38:17 +02:00
nsa
7e6a34532e routing: add index map to distanceHeap
This commit adds the pubkeyIndices map to the distanceHeap to avoid
duplicate entries on the heap. This happened in the earlier iteration
of the findPath algorithm and would cause the driving loop to
evaluate already evaluated entries when there was no need.
2019-07-12 06:08:01 -04:00
nsa
61a1ab08fb routing+channeldb: findPath uses Vertex instead of LightningNode
This commit modifies the nodeWithDist struct to use a route.Vertex
instead of a *channeldb.LightningNode. This change, coupled with
the new ForEachNodeChannel function, allows the findPath Djikstra's
algorithm to cut down on database lookups since we no longer need
to call the FetchOtherNode function.
2019-07-12 06:08:01 -04:00
Joost Jager
2726f50d7c
htlcswitch: do not consider unknown failures an error 2019-07-11 19:49:49 +02:00
Joost Jager
418fe364de
htlcswitch: use distinct error for unreadable failures 2019-07-11 19:49:45 +02:00
Joost Jager
2b47632b26
htlcswitch+routing+routerrpc: return error source as index 2019-07-11 10:37:26 +02:00
Joost Jager
569f821142
routing/test: allow arbitrary graph source nodes 2019-07-11 10:37:22 +02:00
Olaoluwa Osuntokun
ba5fbb3c27
Merge pull request #3156 from joostjager/extended-fail
routerrpc: add more failure reasons and route hints
2019-07-08 19:12:03 -07:00
Joost Jager
5ce04091d8
routing+routerrpc+channeldb: return route on invalid payment details 2019-07-04 09:27:16 +02:00
Joost Jager
7bc56f5a7b
routing: add subscriber event constructors 2019-07-04 09:27:14 +02:00
Joost Jager
ae46fb00cb
routing+channeldb: add more failure reasons 2019-07-04 09:27:12 +02:00
nsa
678fb674eb routing: move call to PruneGraph outside of the collating loop
This commit moves the call to PruneGraph outside of the loop
that collates all of the spentOutputs. With this change, if
a node has been offline for a long period of time, resyncing
with the chain no longer takes up as much memory (1MB vs 200MB
in some cases) or time. Previously, PruneGraph was called
for every block and allocated a very large map further down
in the pruneGraphNodes function. Now, pruneGraphNodes is only
called once.
2019-07-02 21:21:28 -04:00
Johan T. Halseth
8b83f03308
rpcserver+router: log terminal sendPayment error 2019-07-01 16:22:57 -07:00
Conner Fromknecht
3a4aba463b
routing+watchtower/wtclient: prints stats once per minute 2019-06-19 20:07:11 -07:00
Olaoluwa Osuntokun
5f4accd39f
Merge pull request #3195 from halseth/routeing-nil-copy-pubkey
routing+zpay32: copy pubkeys before nilling Curve and spewing
2019-06-18 17:19:25 -07:00
Conner Fromknecht
529db69ec9
routing/router: log aggregate graph processing stats 2019-06-13 18:18:30 -07:00
Johan T. Halseth
386f8ece54
routing+zpay32: copy pubkeys before nilling Curve and spewing
Since nilling the pubkey curve will lead to a nil-pointer exception if
the key is later used for signature verification, we make sure to make a
copy before nilling and spewing.
2019-06-12 12:19:43 +02:00
Wilmer Paulino
c4546081c3
Merge pull request #3172 from joostjager/sendtoroute-timeout
routing+lnrpc: move default payment timeout out of router
2019-06-11 16:27:19 -07:00
Johan T. Halseth
5485101f9f
Merge pull request #3170 from cfromknecht/remove-router-reject-cache
routing/router: remove router-level reject cache
2019-06-11 21:52:27 +02:00
Joost Jager
2e920de292
routing+lnrpc: move default payment timeout out of router
This commit moves the default timeout out of router and thereby fixes a
bug that caused SendToRoute to not return the actual error, but a
timeout result instead. SendToRoute only tries a single route, so a
timeout should never happen.
2019-06-11 08:36:17 +02:00
Johan T. Halseth
2dea790b55
multi: make GetPaymentResult take payment hash
Used for logging in the switch, and when we remove the pending payments,
only the router will have the hash stored across restarts.
2019-06-07 16:53:32 +02:00
Conner Fromknecht
f8db49e7e4
routing/router: remove router-level reject cache
This is now redundant since we have one at the db-level
2019-06-06 13:02:16 +02:00
Joost Jager
07d289c14e
routing: add SendPaymentAsync to router 2019-06-05 12:41:51 +02:00
Joost Jager
eb2647e8fc
channeldb: add subscription to control tower
Allows other sub-systems to subscribe to payment success and fail
events.
2019-06-05 12:41:49 +02:00
Joost Jager
87d3207baf
channeldb+routing: move control tower interface to routing
This commit creates an empty shall for control tower in the routing
package. It is a preparation for adding event notification.
2019-06-05 12:41:47 +02:00
Joost Jager
19d5f8f82c
routing: move default cltv assignment out of router
This commit lifts default setting up to the rpc server level, in line
with other payment defaults.
2019-06-05 12:41:43 +02:00
Olaoluwa Osuntokun
7453dbeacc
Merge pull request #2802 from joostjager/probability
routing: probability based path finding
2019-06-05 11:30:55 +02:00
Joost Jager
054e42f680
routing+routerrpc: expose mission control parameters in lnd config
This commit exposes the three main parameters that influence mission
control and path finding to the user as command line or config file
flags. It allows for fine-tuning for optimal results.
2019-06-04 13:22:50 +02:00
Johan T. Halseth
6d52128da0
routing/router test: assert SendToRoute init values
This commit adds an assertion to the SendToRoute test that the payment
value stored to the DB during SendToRoute execution is the correct one.

This assertion would fail before the previous commit that fixed a
missing value initialization.
2019-06-04 10:34:59 +02:00
Johan T. Halseth
1161d87eec
routing/router: correct SendToRoute's amount record in DB
Previously we would mistakenly use the payment value from the dummy
LightningPayment struct, which would obviously be 0 always. Now we
instead calculate the value from the given route.
2019-06-04 10:13:33 +02:00
Joost Jager
f7982f0f4c
routing+routerrpc: expose mission control state over rpc
This commit exposes mission control state for rpc for development
purposes.
2019-06-04 10:00:27 +02:00
Joost Jager
7133f37bb8
routing: global probability based mission control
Previously every payment had its own local mission control state which
was in effect only for that payment. In this commit most of the local
state is removed and payments all tap into the global mission control
probability estimator.

Furthermore the decay time of pruned edges and nodes is extended, so
that observations about the network can better benefit future payment
processes.

Last, the probability function is transformed from a binary output to a
gradual curve, allowing for a better trade off between candidate routes.
2019-06-04 10:00:25 +02:00
Joost Jager
3349d517aa
routing: add min probability to path finding
This commit adds a new restriction to pathfinding that allows returning
only routes with a minimum success probability.
2019-06-04 10:00:23 +02:00
Joost Jager
6b70791c2d
routing: use probability source in path finding
This PR replaces the previously used edge and node ignore lists in path
finding by a probability based system. It modifies path finding so that
it not only compares routes on fee and time lock, but also takes route
success probability into account.

Allowing routes to be compared based on success probability is achieved
by introducing a 'virtual' cost of a payment attempt and using that to
translate probability into another cost factor.
2019-06-04 10:00:21 +02:00
Joost Jager
68f2a04f42
routerrpc: expose mission control reset rpc 2019-06-04 10:00:17 +02:00
Joost Jager
3db48dc53f
routing: Route String() method 2019-06-04 10:00:15 +02:00
Joost Jager
35003bd87c
routing/test: sort nodes for channel graph
This commit ensures that channel endpoints in EdgeInfo are ordered
node1 < node2 to not violate assumptions being made by dependent code.
2019-06-04 10:00:13 +02:00
Olaoluwa Osuntokun
4068e78af6
Merge pull request #2717 from cfromknecht/send-payment-payreq
lnrpc+rpcserver: display payreq via listpayments
2019-05-31 17:09:04 -07:00
Joost Jager
1b5655f7fa
Merge pull request #3113 from joostjager/sendtoroute-error-fix
routing: return structured error for send to route
2019-05-30 23:20:14 +02:00
Conner Fromknecht
078734b511
routing/router: set payreq before persisting PaymentCreationInfo 2019-05-29 16:31:12 -07:00
Joost Jager
cbfc7dd42d
routing: check findPath error 2019-05-29 14:07:00 +02:00
Joost Jager
0e273a5731
routing: return structured error for send to route 2019-05-29 14:06:58 +02:00
Johan T. Halseth
5882e54bf3
route test: add a simple unit test for the TotalFees method
Checks fees on 0, 1, and 2-hop routes.
2019-05-28 11:16:34 +02:00
Johan T. Halseth
f829d33d6f
route: return 0 fee from TotalFees for empty route.
Otherwise we would crash calling the method on empty routes.
2019-05-28 11:03:14 +02:00
Johan T. Halseth
649ee875fd
routing/router_test: add test case for ForwardingError on SendHTLC 2019-05-27 20:19:00 +02:00
Johan T. Halseth
1b788904f0
channeldb+router: record payment failure reason in db 2019-05-27 20:18:59 +02:00
Johan T. Halseth
60e2367973
routing/router_test: add TestRouterPaymentStateMachine
TestRouterPaymentStateMachine tests that the router interacts as
expected with the ControlTower during a payment lifecycle, such that it
payment attempts are not sent twice to the switch, and results are
handled after a restart.
2019-05-27 20:18:59 +02:00
Johan T. Halseth
3f76bc0629
routing+server: define PaymentSessionSource 2019-05-27 20:18:59 +02:00
Johan T. Halseth
f4306b1178
routing/payment_session+router: make PaymentSession interface 2019-05-27 20:18:58 +02:00
Johan T. Halseth
adc4640f4f
routing: export MissionControl 2019-05-27 20:18:58 +02:00
Johan T. Halseth
3323800e02
routing/router: resume payment state machine at startup
On startup the router will fetch the in-flight payments from the control
tower, and resume their execution.
2019-05-27 20:18:58 +02:00
Johan T. Halseth
677cb018c9
routing/control_tower: add FetchInFlightPayments 2019-05-27 20:18:58 +02:00
Johan T. Halseth
de1bf8a518
routing/router: persist payment state machine
This commit makes the router use the ControlTower to drive the payment
life cycle state machine, to keep track of active payments across
restarts.  This lets the router resume payments on startup, such that
their final results can be handled and stored when ready.
2019-05-27 20:18:58 +02:00
Johan T. Halseth
dd73c51a34
routing/payment_lifecycle: extract create attempt into method 2019-05-27 20:18:58 +02:00
Johan T. Halseth
e9b2182cdc
routing/payment_lifecycle: extract error handling into method 2019-05-27 20:18:58 +02:00
Johan T. Halseth
83bfaa4fb4
routing: extract payment flow into method on paymentLifecycle
This encapsulates all state needed to resume a payment from any point of
the payment flow, and that must be shared between the different stages
of the execution. This is done to prepare for breaking the send loop
into smaller parts, and being able to resume the payment from any point
from persistent state.
2019-05-27 20:18:57 +02:00
Johan T. Halseth
ae7bf2cb7b
routing/router: move sending and receiving payment result into loop 2019-05-27 20:18:57 +02:00
Wilmer Paulino
6e0084fec9
Merge pull request #3075 from cfromknecht/crtr-nil-path-curves
routing/router: nil path pubkeys before spewing
2019-05-23 20:01:54 -07:00
Xavi Soler
28021361d1
routing: add color to node update 2019-05-23 10:52:12 +02:00
Johan T. Halseth
2c1c29e060
Merge pull request #3081 from halseth/route-remove-totalfees
routing/route+multi: remove redundant TotalFees field
2019-05-20 15:00:01 +02:00
Johan T. Halseth
15bed506b1
routing/route+multi: remove redundant TotalFees field
Instead get it on demand using method TotalFees().
2019-05-16 23:58:32 +02:00
Johan T. Halseth
cd02c22977
htlcswitch+router: move deobfuscator creation to GetPaymentResult call
In this commit we move handing the deobfuscator from the router to the
switch from when the payment is initiated, to when the result is
queried.

We do this because only the router can recreate the deobfuscator after a
restart, and we are preparing for being able to handle results across
restarts.

Since the deobfuscator cannot be nil anymore, we can also get rid of
that special case.
2019-05-16 23:56:12 +02:00
Johan T. Halseth
5a8b892bb6
routing/router: extract sessionKey generation
We extract the session key such that we can later store it across
restarts.
2019-05-16 23:56:12 +02:00
Johan T. Halseth
ec087a9f73
htlcswitch+router: define PaymentResult, GetPaymentResult
This lets us distinguish an critical error from a actual payment result
(success or failure). This is important since we know that we can only
attempt another payment when a final result from the previous payment
attempt is received.
2019-05-16 23:56:12 +02:00
Johan T. Halseth
ed81cec0fa
routing/router: remove sendToSwitch method 2019-05-16 23:56:11 +02:00
Johan T. Halseth
c9e8ff6a34
switch+router+server: move NextPaymentID to router
This commit moves the responsibility of generating a unique payment ID
from the switch to the router. This will make it easier for the router
to keep track of which HTLCs were successfully forwarded onto the
network, as it can query the switch for existing HTLCs as long as the
paymentIDs are kept.

The router is expected to maintain a map from paymentHash->paymentID,
such that they can be replayed on restart. This also lets the router
check the status of a sent payment after a restart, by querying the
switch for the paymentID in question.
2019-05-16 23:56:06 +02:00
Johan T. Halseth
f1cb54f943
routing/router: define PaymentAttemptDispatcher interface
The switch satisfies this interface, and makes it easy to mock the send
method from the router.
2019-05-16 23:53:08 +02:00
Joost Jager
ba3fa94268
lnrpc+routing: Only accept a single route for SendToRoute 2019-05-15 11:54:46 +02:00
Conner Fromknecht
4d26e90ef3
routing/router: nil path pubkeys before spewing 2019-05-14 20:32:58 -07:00
Olaoluwa Osuntokun
f8c824fb1d
Merge pull request #3054 from joostjager/remove-k-shortest
lnrpc+routing: remove k shortest path finding
2019-05-10 15:51:01 -07:00
Johan T. Halseth
10070ecab7
lnwallet: make BlockChainIO.GetUTXO take cancel chan
Use quit channels as cancel chan for call to GetUTXO.
2019-05-09 14:44:47 +02:00
Joost Jager
7a5bd29a69
lnrpc+routing: remove k shortest path finding 2019-05-07 19:54:06 +02:00
Johan T. Halseth
c4415f0400
Merge pull request #3044 from cfromknecht/spelling-fixes
multi: fix spelling mistakes
2019-05-07 08:50:36 +02:00
Conner Fromknecht
3f59e31248
routing/router: break rescans on router quit
This commit reevaluates the router's quit channel between each block
during the initial call to syncGraphWithChain, which, in the worst case,
may have to scan several thousand blocks on startup if the node has not
been active for some time. Without this, attempting to stop the daemon
will not exit until the rescan has completed, which for certain backends
could be several hours.
2019-05-06 15:04:40 -07:00
Conner Fromknecht
17b2140cb5
multi: fix spelling mistakes 2019-05-04 15:35:37 -07:00
Olaoluwa Osuntokun
bab957382f
router: convert Route.ToHopPayloads() to Route.ToSphinxPath()
In this commit, we update the process that we use to generate a sphinx
packet to send our onion routed HTLC. Due to recent changes in the
`sphinx` package we use, we now need to use a new PaymentPath struct. As
a result, it no longer makes sense to split up the nodes in a route and
their per hop paylods as they're now in the same struct. All tests have
been updated accordingly.
2019-04-30 20:13:32 -07:00
Johan T. Halseth
ee257fd0eb
multi: move Route to sub-pkg routing/route 2019-04-29 14:52:33 +02:00
Wilmer Paulino
998680ad59
routing: fall back to edge MaxHTLC within findPath instead of Capacity
In this commit, we make our findPath function use an edge's MaxHTLC as
its available bandwidth instead of its Capacity. We do this as it's
possible for the capacity of an edge to not exist when operating as a
light client. For channels that do not support the MaxHTLC optional
field, we'll fall back to using the edge's Capacity.
2019-04-18 21:57:42 -07:00
Wilmer Paulino
0b0a9f4172
channeldb+routing: refactor DeleteChannelEdge to use ChannelID
In this commit, we refactor DeleteChannelEdge to use ChannelIDs rather
than ChannelPoints. We do this as the only use of DeleteChannelEdge is
when we are pruning zombie channels from our graph. When running under a
light client, we are unable to obtain the ChannelPoint of each edge due
to the expensive operations required to do so. As a stop-gap, we'll
resort towards using an edge's ChannelID instead, which is already
gossiped between nodes.
2019-04-18 21:57:41 -07:00
Wilmer Paulino
5d3621cc83
routing: skip max htlc validation when capacity is 0
Since light clients no longer have access to an edge's capacity, they
are unable to validate whether the max HTLC value for an updated edge
policy respects the capacity limit. As a stop-gap, we'll skip this
check.
2019-04-18 21:57:38 -07:00
Wilmer Paulino
f2637d63ba
routing: prevent fetching blocks for graph edges with AssumeChannelValid
This serves as a stop-gap for light clients as blocks need to be
downloaded from the P2P network, and even with caches, would be too
costly for them to verify. Doing this has two side effects however:
we'll no longer know of the channel capacity and outpoint, which are
essential for some of lnd's responsibilities.
2019-04-18 21:57:37 -07:00
Wilmer Paulino
fd1aa478a9
routing: disable FilteredChainView when AssumeChannelValid is active
In this commit, we disable attempting to determine when a channel has
been closed out on-chain whenever AssumeChannelValid is active. Since
the flag indicates that performing this operation is expensive, we do
this as a temporary optimization until we can include proofs of channels
being closed in the gossip protocol.

With this change, the only way for channels being removed from the graph
will be once they're considered zombies: which can happen when both
edges of a channel have their disabled bits set or when both edges
haven't had an update within the past two weeks.
2019-04-18 21:57:37 -07:00
Wilmer Paulino
7e7b8a1940
routing: prune graph nodes after pruning zombie channels
We do this to ensure we don't leave any stray nodes in our graph that
were part of the zombie channels that we've pruned.
2019-04-18 21:57:36 -07:00
Wilmer Paulino
7eb720e535
routing: mark policy as stale if edge remains double disabled
To ensure we don't mark an edge as live again just because an update
with a fresh timestamp was received, we'll ensure that we reject any
new updates for zombie channels if they remain disabled when running
with AssumeChannelValid.
2019-04-18 21:57:35 -07:00
Wilmer Paulino
f23c3b488e
routing: prune channels with disabled bit set on both edges
In this commit, we add an additional heuristic when running with
AssumeChannelValid. Since AssumeChannelValid being present assumes that
we're not able to quickly determine whether channels are valid, we also
assume that any channels with the disabled bit set on both sides are
considered zombie. This should be relatively safe to do, since the
disabled bits are usually set when the channel is closed on-chain. In
the case that they aren't, we'll have to wait until both edges haven't
had a new update within two weeks to prune them.
2019-04-18 21:57:34 -07:00
Wilmer Paulino
78bdcbb115
routing: prune channels only if both edges are present
We do this to ensure we don't prune too aggressively, as it's possible
that we've only received the channel announcement for a channel, but not
its accompanying channel updates.
2019-04-18 21:57:31 -07:00
Wilmer Paulino
29664c9704
routing: make test channel policies optional 2019-04-18 20:47:32 -07:00
Olaoluwa Osuntokun
373927fb3e
Merge pull request #2950 from joostjager/remove-route-cache
routing: remove route cache
2019-04-17 19:34:19 -07:00
Olaoluwa Osuntokun
a3c980b1e8
Merge pull request #2871 from halseth/router-subscribetopology-check-started
routing/notifications: check router started on SubscribeTopology
2019-04-16 21:32:10 -07:00
Johan T. Halseth
2b9efcbaeb
routing/router: define ErrRouterShuttingDown 2019-04-15 16:55:10 +02:00
Joost Jager
3b94627c12
routing: remove route cache
This commit removes the QueryRoutes route cache. It is causing wrong
routes to be returned because not all of the request parameters are
stored.

The cache allowed high frequency QueryRoutes calls to the same
destination and with the same amount to be returned fast. This behaviour
can also be achieved by caching the request on the client side. In case
a route is invalidated because of for example a channel update,
the subsequent SendToRoute call will fail. This is a trigger to call
QueryRoutes again for a fresh route.
2019-04-13 08:10:32 +02:00
Johan T. Halseth
c2af637d3c
routing/notifications: check router startd on SubscribeTopology
If the router is not yet started, return an error to avoid a deadlock
waiting for it to handle the subscription request.
2019-04-02 10:21:08 +02:00
Wilmer Paulino
44a01db0ef
routing: expose VerifyChannelUpdateSignature function 2019-03-27 13:07:47 -07:00
Wilmer Paulino
23796d3247
routing+discovery: extend ChannelGraphSource with zombie index methods 2019-03-27 13:07:30 -07:00
Wilmer Paulino
174645fcba
routing+server: expose DefaultChannelPruneExpiry 2019-03-27 13:07:13 -07:00
Wilmer Paulino
c82d73a826
channeldb+routing: extend edge lookup methods with zombie index check
In this commit, we extend the graph's FetchChannelEdgesByID and
HasChannelEdge methods to also check the zombie index whenever the edge
to be looked up doesn't exist within the edge index. We do this to
signal to callers that the edge is known, but only as a zombie, and the
only information that we have about the edge are the node public keys of
the two parties involved in the edge.

In the event that an edge does exist within the zombie index, we make
an additional check on edge policies to ensure they are not within the
router's pruning window, indicating that it is a fresh update.
2019-03-27 13:06:57 -07:00
Joost Jager
06cd64cbbc
routing: add cltv limit to payment session 2019-03-19 10:36:51 +01:00
Joost Jager
e3bb3d46a2
routing: abstract path finding interface 2019-03-19 10:36:48 +01:00
Joost Jager
6006549ed5
routing: enforce cltv limit in path finding 2019-03-19 10:36:45 +01:00
Joost Jager
c5961d4904
routing: only warn for zero cltv delta edges
This condition may be caused by a bug somewhere else in the system.
Expose it here as a warn log line.
2019-03-19 10:36:43 +01:00
Joost Jager
2be1051fb6
zpay32: move HopHint and DefaultFinalCLTVDelta 2019-03-15 10:08:32 +01:00
Olaoluwa Osuntokun
d86fe393cd
multi: update build to latest version of neutrino+btcwallet
In this commit, we update the build to point to the latest version of
neutrino and btcwallet. The latest version of neutrino includes a number
of bug fixes, and new features like reliably transaction broadcast. The
latest version of btcwallet contains a number of bug fixes related to
properly remove invalid transactions from its database.
2019-03-13 15:03:39 -07:00
Joost Jager
6cc82b4a34
routing: add todo describing route cache bug 2019-03-06 15:31:09 +01:00
Joost Jager
c62c9d64da
routing: add source parameter to query routes
This commit allows execution of QueryRoutes from any source node.
Previously this was restricted to only the self node.
2019-03-06 15:31:03 +01:00
Joost Jager
7719bc432f
routing: take Vertex types as path finding source and target nodes
Currently public keys are represented either as a 33-byte array (Vertex) or as a
btcec.PublicKey struct. The latter isn't useable as index into maps and
cannot be used easily in compares. Therefore the 33-byte array
representation is used predominantly throughout the code base.

This commit converts the argument types of source and target nodes for
path finding to Vertex. Path finding executes no crypto operations and
using Vertex simplifies the code.

Additionally, it prepares for the path finding source parameter to be
exposed over rpc in a follow up commit without requiring conversion back
and forth between Vertex and btcec.PublicKey.
2019-03-06 15:31:00 +01:00
Joost Jager
b09adc3219
lnrpc+routing: add edges and nodes restrictions to query routes
This commit allows the execution of QueryRoutes to be controlled using
lists of black-listed edges and nodes. Any path returned will not pass
through the edges and/or nodes on the list.
2019-03-06 15:30:57 +01:00
Joost Jager
4376f3e1bd
routing: allow nil maps for ignored edges and nodes
This allows removing a lot of empty map initialization code and makes
the code more readable.
2019-03-06 15:30:54 +01:00
Joost Jager
b2b28b49b1
routing: export RestrictParams and EdgeLocator 2019-03-06 15:30:52 +01:00
Joost Jager
4937304732
routing: remove redundant fee limit check in newRoute
This check was a left over from when the fee limit wasn't checked yet in
the path finding algorithm.
2019-03-06 15:30:49 +01:00
Joost Jager
f4cc2e235a
routing: add todo describing fee limit bug 2019-03-06 15:30:46 +01:00
Johan T. Halseth
cee18892b5
Merge pull request #2565 from joostjager/sendpayment-refactor
routing: sendPayment broken down into multiple functions
2019-03-06 10:20:20 +01:00
Valentine Wallace
648adaea69 routing/pathfind: ensure max htlc is considered during path finding
In this commit, we update the path finding logic to
ignore a channel if the HTLC value (including the fees
at the point) exceeds the max HTLC value (if set) of the
link.
2019-03-01 08:05:27 -08:00
Valentine Wallace
348a66ed13 routing: update test edge policy fields to include max htlc + new flags
Since the MaxHTLC field was recently added to the ChannelEdgePolicy struct,
and the Flags field was broken into ChannelFlags and MessageFlags, the
test edge policies should be updated accordingly.
2019-03-01 08:05:27 -08:00
Valentine Wallace
0c6c1040d8 routing/ntfns+rpcserver: include max htlc in topology notifications 2019-02-21 18:39:32 -08:00
Johan T. Halseth
01ea797d78
routing/router test: add TestIgnoreChannelEdgePolicyForUnknownChannel 2019-02-14 14:21:18 +01:00
Johan T. Halseth
23ce82f4e3
routing/router: don't apply channel policy updates for unknown channels 2019-02-14 14:21:18 +01:00
Johan T. Halseth
1259bacd49
routing: remove unused code, nodeIndex and chanIndex 2019-02-13 12:23:45 +01:00
Joost Jager
f8721ffbca
routing: sendPayment broken down into multiple functions 2019-02-11 08:42:41 +01:00
Alex Bosworth
b3ae5bc16e
trivial: typo fix 2019-02-10 21:18:34 -08:00
Joost Jager
7c30a8c493
routing: add outgoing channel restriction 2019-02-08 23:05:56 +01:00
Joost Jager
7b94d9adb0
routing: move payment session to separate file 2019-02-01 09:14:54 +01:00
Joost Jager
9e012ecc93
multi: move Input interface and related code
This commit is a step to split the lnwallet package. It puts the Input
interface and implementations in a separate package along with all their
dependencies from lnwallet.
2019-01-31 13:25:33 +01:00
orbitalturtle
08750f180b multi: deprecate IncorrectHtlcAmount onion error
In this commit, we deprecate the `IncorrectHtlcAmount` onion error.
We'll still decode this error to use when retrying paths, but we'll no
longer send this ourselves. The `UnknownPaymentHash` error has been
amended to also include the value of the payment as well. This allows us
to worry about one less error.
2019-01-29 18:21:13 -08:00
Johan T. Halseth
7d34ce9d08
lnwire+multi: define HasMaxHtlc helper on msgFlags 2019-01-22 08:42:30 +01:00
Valentine Wallace
cced754659
routing/router: when applying onion failure channel update, set max htlc
In this commit, we ensure that when we update an edge
as a result of a ChannelUpdate being returned from an
onion failure, the max htlc portion of the channel update
is included in the edge update.
2019-01-22 08:42:29 +01:00
Valentine Wallace
15168c391e
discovery+routing: validate msg flags and max htlc in ChannelUpdates
In this commit, we alter the ValidateChannelUpdateAnn function in
ann_validation to validate a remote ChannelUpdate's message flags
and max HTLC field. If the message flag is set but the max HTLC
field is not set or vice versa, the ChannelUpdate fails validation.

Co-authored-by: Johan T. Halseth <johanth@gmail.com>
2019-01-22 08:42:27 +01:00
Valentine Wallace
0fd6004958
multi: partition lnwire.ChanUpdateFlag into ChannelFlags and MessageFlags
In this commit:

* we partition lnwire.ChanUpdateFlag into two (ChanUpdateChanFlags and
ChanUpdateMsgFlags), from a uint16 to a pair of uint8's

* we rename the ChannelUpdate.Flags to ChannelFlags and add an
additional MessageFlags field, which will be used to indicate the
presence of the optional field HtlcMaximumMsat within the ChannelUpdate.

* we partition ChannelEdgePolicy.Flags into message and channel flags.
This change corresponds to the partitioning of the ChannelUpdate's Flags
field into MessageFlags and ChannelFlags.

Co-authored-by: Johan T. Halseth <johanth@gmail.com>
2019-01-22 08:42:26 +01:00
Conner Fromknecht
a5b22788bf
routing/router: avoid naked spew of channel policies 2019-01-09 23:54:41 -08:00
Olaoluwa Osuntokun
11c24d3e0f
Merge pull request #1745 from halseth/channeldb-avoid-create-buckets
channeldb: avoid creating empty buckets
2018-12-17 20:01:29 -08:00
Joost Jager
b2cb760c65
routing: check for empty hops list
This commit fixes a crash that could be triggered by sending an empty
hop list to the SendToRoute rpc.
2018-12-10 11:27:07 +01:00
Johan T. Halseth
81fe6e73ed
router+graph: return ErrGraphNodesNotFound if no nodes to prune
Avoids creating a bucket unneccessarily.
2018-12-06 10:22:15 +01:00
Johan T. Halseth
87df6b71c5
router test: only cleanup if context creation succeeds 2018-12-06 10:22:15 +01:00
Joost Jager
3ed16837e4
routing: prune channels unidirectional
In this commit we introduce pruning of channel edges instead of channels.
Channel failures apply to a single direction and it is unnecessarily
restricting to prune both directions.
2018-12-05 14:39:23 +01:00
Joost Jager
8b8e82a171
routing: remove route hop maps
Hop maps were used in a test to verify the population of the hop map
itself and further only in a single function (getFailedChannelID).
Rewrote that function and removed the hop maps completely.
2018-12-05 14:39:23 +01:00
Joost Jager
42debc6c12
routing: fix node ordering in test graph
There is the general assumption that channel edge policy nodes are
ordered such that the node1 pubkey is smaller than the key of node 2. In
the test graph, this assumption didn't hold. This commit fixes the test
graph and also adds a check to prevent this from happening again.
2018-12-05 09:15:27 +01:00
Johan T. Halseth
5f8425cb03
routing/pathfind_test: check error before attempting cleanup 2018-12-04 12:20:45 +01:00
Johan T. Halseth
67b018476c
routing/pathfind test: add TestPathSourceEdgesBandwidth
This commit adds a new test that checks that the bandwidth hints are
considered correclty for local channels, and that disable flags are
ignored in this case.
2018-12-04 12:20:45 +01:00
Johan T. Halseth
0b6a19866b
routing/pathfind: pass ignore disable flags for local channels
To decouple our own path finding from the graph state, we don't consider
the disable bit when attempting to use local channels. Instead the
bandwidth hints will be zero for local inactive channels.

We alos modify the unit test to check that the disable flag is ignored
for local edges.
2018-12-04 12:20:45 +01:00
Johan T. Halseth
20b9d5d000
routing: define graphParams and restrictParams
To avoid the findPath parameter list getting out of hand, we define new
structs that wraps the mandatory and optional parameters to findPath.
2018-12-04 12:20:44 +01:00
Olaoluwa Osuntokun
5075394617
Merge pull request #1706 from joostjager/errorprocessing
routing: make routing retry behaviour consistent
2018-12-03 18:29:07 -08:00
Joost Jager
b6ce03e569
routing: make routing retry behaviour consistent
Fixes the following issues:
- If the channel update of FailFeeInsufficient contains an invalid channel
  update, it is not possible to properly add to the failed channels set.
- FailAmountBelowMinimum may apply a channel update, but does not retry.
- FailIncorrectCltvExpiry immediately prunes the vertex without
  trying one more time.

In this commit, the logic for all three policy related errors is
aligned.
2018-12-03 13:23:27 +01:00
Olaoluwa Osuntokun
1fd3aac925
multi: switch from bolt packge to bbolt package for all imports 2018-11-29 20:33:49 -08:00