Commit Graph

587 Commits

Author SHA1 Message Date
Joost Jager
6b70791c2d
routing: use probability source in path finding
This PR replaces the previously used edge and node ignore lists in path
finding by a probability based system. It modifies path finding so that
it not only compares routes on fee and time lock, but also takes route
success probability into account.

Allowing routes to be compared based on success probability is achieved
by introducing a 'virtual' cost of a payment attempt and using that to
translate probability into another cost factor.
2019-06-04 10:00:21 +02:00
Joost Jager
68f2a04f42
routerrpc: expose mission control reset rpc 2019-06-04 10:00:17 +02:00
Joost Jager
3db48dc53f
routing: Route String() method 2019-06-04 10:00:15 +02:00
Joost Jager
35003bd87c
routing/test: sort nodes for channel graph
This commit ensures that channel endpoints in EdgeInfo are ordered
node1 < node2 to not violate assumptions being made by dependent code.
2019-06-04 10:00:13 +02:00
Olaoluwa Osuntokun
4068e78af6
Merge pull request #2717 from cfromknecht/send-payment-payreq
lnrpc+rpcserver: display payreq via listpayments
2019-05-31 17:09:04 -07:00
Joost Jager
1b5655f7fa
Merge pull request #3113 from joostjager/sendtoroute-error-fix
routing: return structured error for send to route
2019-05-30 23:20:14 +02:00
Conner Fromknecht
078734b511
routing/router: set payreq before persisting PaymentCreationInfo 2019-05-29 16:31:12 -07:00
Joost Jager
cbfc7dd42d
routing: check findPath error 2019-05-29 14:07:00 +02:00
Joost Jager
0e273a5731
routing: return structured error for send to route 2019-05-29 14:06:58 +02:00
Johan T. Halseth
5882e54bf3
route test: add a simple unit test for the TotalFees method
Checks fees on 0, 1, and 2-hop routes.
2019-05-28 11:16:34 +02:00
Johan T. Halseth
f829d33d6f
route: return 0 fee from TotalFees for empty route.
Otherwise we would crash calling the method on empty routes.
2019-05-28 11:03:14 +02:00
Johan T. Halseth
649ee875fd
routing/router_test: add test case for ForwardingError on SendHTLC 2019-05-27 20:19:00 +02:00
Johan T. Halseth
1b788904f0
channeldb+router: record payment failure reason in db 2019-05-27 20:18:59 +02:00
Johan T. Halseth
60e2367973
routing/router_test: add TestRouterPaymentStateMachine
TestRouterPaymentStateMachine tests that the router interacts as
expected with the ControlTower during a payment lifecycle, such that it
payment attempts are not sent twice to the switch, and results are
handled after a restart.
2019-05-27 20:18:59 +02:00
Johan T. Halseth
3f76bc0629
routing+server: define PaymentSessionSource 2019-05-27 20:18:59 +02:00
Johan T. Halseth
f4306b1178
routing/payment_session+router: make PaymentSession interface 2019-05-27 20:18:58 +02:00
Johan T. Halseth
adc4640f4f
routing: export MissionControl 2019-05-27 20:18:58 +02:00
Johan T. Halseth
3323800e02
routing/router: resume payment state machine at startup
On startup the router will fetch the in-flight payments from the control
tower, and resume their execution.
2019-05-27 20:18:58 +02:00
Johan T. Halseth
677cb018c9
routing/control_tower: add FetchInFlightPayments 2019-05-27 20:18:58 +02:00
Johan T. Halseth
de1bf8a518
routing/router: persist payment state machine
This commit makes the router use the ControlTower to drive the payment
life cycle state machine, to keep track of active payments across
restarts.  This lets the router resume payments on startup, such that
their final results can be handled and stored when ready.
2019-05-27 20:18:58 +02:00
Johan T. Halseth
dd73c51a34
routing/payment_lifecycle: extract create attempt into method 2019-05-27 20:18:58 +02:00
Johan T. Halseth
e9b2182cdc
routing/payment_lifecycle: extract error handling into method 2019-05-27 20:18:58 +02:00
Johan T. Halseth
83bfaa4fb4
routing: extract payment flow into method on paymentLifecycle
This encapsulates all state needed to resume a payment from any point of
the payment flow, and that must be shared between the different stages
of the execution. This is done to prepare for breaking the send loop
into smaller parts, and being able to resume the payment from any point
from persistent state.
2019-05-27 20:18:57 +02:00
Johan T. Halseth
ae7bf2cb7b
routing/router: move sending and receiving payment result into loop 2019-05-27 20:18:57 +02:00
Wilmer Paulino
6e0084fec9
Merge pull request #3075 from cfromknecht/crtr-nil-path-curves
routing/router: nil path pubkeys before spewing
2019-05-23 20:01:54 -07:00
Xavi Soler
28021361d1
routing: add color to node update 2019-05-23 10:52:12 +02:00
Johan T. Halseth
2c1c29e060
Merge pull request #3081 from halseth/route-remove-totalfees
routing/route+multi: remove redundant TotalFees field
2019-05-20 15:00:01 +02:00
Johan T. Halseth
15bed506b1
routing/route+multi: remove redundant TotalFees field
Instead get it on demand using method TotalFees().
2019-05-16 23:58:32 +02:00
Johan T. Halseth
cd02c22977
htlcswitch+router: move deobfuscator creation to GetPaymentResult call
In this commit we move handing the deobfuscator from the router to the
switch from when the payment is initiated, to when the result is
queried.

We do this because only the router can recreate the deobfuscator after a
restart, and we are preparing for being able to handle results across
restarts.

Since the deobfuscator cannot be nil anymore, we can also get rid of
that special case.
2019-05-16 23:56:12 +02:00
Johan T. Halseth
5a8b892bb6
routing/router: extract sessionKey generation
We extract the session key such that we can later store it across
restarts.
2019-05-16 23:56:12 +02:00
Johan T. Halseth
ec087a9f73
htlcswitch+router: define PaymentResult, GetPaymentResult
This lets us distinguish an critical error from a actual payment result
(success or failure). This is important since we know that we can only
attempt another payment when a final result from the previous payment
attempt is received.
2019-05-16 23:56:12 +02:00
Johan T. Halseth
ed81cec0fa
routing/router: remove sendToSwitch method 2019-05-16 23:56:11 +02:00
Johan T. Halseth
c9e8ff6a34
switch+router+server: move NextPaymentID to router
This commit moves the responsibility of generating a unique payment ID
from the switch to the router. This will make it easier for the router
to keep track of which HTLCs were successfully forwarded onto the
network, as it can query the switch for existing HTLCs as long as the
paymentIDs are kept.

The router is expected to maintain a map from paymentHash->paymentID,
such that they can be replayed on restart. This also lets the router
check the status of a sent payment after a restart, by querying the
switch for the paymentID in question.
2019-05-16 23:56:06 +02:00
Johan T. Halseth
f1cb54f943
routing/router: define PaymentAttemptDispatcher interface
The switch satisfies this interface, and makes it easy to mock the send
method from the router.
2019-05-16 23:53:08 +02:00
Joost Jager
ba3fa94268
lnrpc+routing: Only accept a single route for SendToRoute 2019-05-15 11:54:46 +02:00
Conner Fromknecht
4d26e90ef3
routing/router: nil path pubkeys before spewing 2019-05-14 20:32:58 -07:00
Olaoluwa Osuntokun
f8c824fb1d
Merge pull request #3054 from joostjager/remove-k-shortest
lnrpc+routing: remove k shortest path finding
2019-05-10 15:51:01 -07:00
Johan T. Halseth
10070ecab7
lnwallet: make BlockChainIO.GetUTXO take cancel chan
Use quit channels as cancel chan for call to GetUTXO.
2019-05-09 14:44:47 +02:00
Joost Jager
7a5bd29a69
lnrpc+routing: remove k shortest path finding 2019-05-07 19:54:06 +02:00
Johan T. Halseth
c4415f0400
Merge pull request #3044 from cfromknecht/spelling-fixes
multi: fix spelling mistakes
2019-05-07 08:50:36 +02:00
Conner Fromknecht
3f59e31248
routing/router: break rescans on router quit
This commit reevaluates the router's quit channel between each block
during the initial call to syncGraphWithChain, which, in the worst case,
may have to scan several thousand blocks on startup if the node has not
been active for some time. Without this, attempting to stop the daemon
will not exit until the rescan has completed, which for certain backends
could be several hours.
2019-05-06 15:04:40 -07:00
Conner Fromknecht
17b2140cb5
multi: fix spelling mistakes 2019-05-04 15:35:37 -07:00
Olaoluwa Osuntokun
bab957382f
router: convert Route.ToHopPayloads() to Route.ToSphinxPath()
In this commit, we update the process that we use to generate a sphinx
packet to send our onion routed HTLC. Due to recent changes in the
`sphinx` package we use, we now need to use a new PaymentPath struct. As
a result, it no longer makes sense to split up the nodes in a route and
their per hop paylods as they're now in the same struct. All tests have
been updated accordingly.
2019-04-30 20:13:32 -07:00
Johan T. Halseth
ee257fd0eb
multi: move Route to sub-pkg routing/route 2019-04-29 14:52:33 +02:00
Wilmer Paulino
998680ad59
routing: fall back to edge MaxHTLC within findPath instead of Capacity
In this commit, we make our findPath function use an edge's MaxHTLC as
its available bandwidth instead of its Capacity. We do this as it's
possible for the capacity of an edge to not exist when operating as a
light client. For channels that do not support the MaxHTLC optional
field, we'll fall back to using the edge's Capacity.
2019-04-18 21:57:42 -07:00
Wilmer Paulino
0b0a9f4172
channeldb+routing: refactor DeleteChannelEdge to use ChannelID
In this commit, we refactor DeleteChannelEdge to use ChannelIDs rather
than ChannelPoints. We do this as the only use of DeleteChannelEdge is
when we are pruning zombie channels from our graph. When running under a
light client, we are unable to obtain the ChannelPoint of each edge due
to the expensive operations required to do so. As a stop-gap, we'll
resort towards using an edge's ChannelID instead, which is already
gossiped between nodes.
2019-04-18 21:57:41 -07:00
Wilmer Paulino
5d3621cc83
routing: skip max htlc validation when capacity is 0
Since light clients no longer have access to an edge's capacity, they
are unable to validate whether the max HTLC value for an updated edge
policy respects the capacity limit. As a stop-gap, we'll skip this
check.
2019-04-18 21:57:38 -07:00
Wilmer Paulino
f2637d63ba
routing: prevent fetching blocks for graph edges with AssumeChannelValid
This serves as a stop-gap for light clients as blocks need to be
downloaded from the P2P network, and even with caches, would be too
costly for them to verify. Doing this has two side effects however:
we'll no longer know of the channel capacity and outpoint, which are
essential for some of lnd's responsibilities.
2019-04-18 21:57:37 -07:00
Wilmer Paulino
fd1aa478a9
routing: disable FilteredChainView when AssumeChannelValid is active
In this commit, we disable attempting to determine when a channel has
been closed out on-chain whenever AssumeChannelValid is active. Since
the flag indicates that performing this operation is expensive, we do
this as a temporary optimization until we can include proofs of channels
being closed in the gossip protocol.

With this change, the only way for channels being removed from the graph
will be once they're considered zombies: which can happen when both
edges of a channel have their disabled bits set or when both edges
haven't had an update within the past two weeks.
2019-04-18 21:57:37 -07:00
Wilmer Paulino
7e7b8a1940
routing: prune graph nodes after pruning zombie channels
We do this to ensure we don't leave any stray nodes in our graph that
were part of the zombie channels that we've pruned.
2019-04-18 21:57:36 -07:00
Wilmer Paulino
7eb720e535
routing: mark policy as stale if edge remains double disabled
To ensure we don't mark an edge as live again just because an update
with a fresh timestamp was received, we'll ensure that we reject any
new updates for zombie channels if they remain disabled when running
with AssumeChannelValid.
2019-04-18 21:57:35 -07:00
Wilmer Paulino
f23c3b488e
routing: prune channels with disabled bit set on both edges
In this commit, we add an additional heuristic when running with
AssumeChannelValid. Since AssumeChannelValid being present assumes that
we're not able to quickly determine whether channels are valid, we also
assume that any channels with the disabled bit set on both sides are
considered zombie. This should be relatively safe to do, since the
disabled bits are usually set when the channel is closed on-chain. In
the case that they aren't, we'll have to wait until both edges haven't
had a new update within two weeks to prune them.
2019-04-18 21:57:34 -07:00
Wilmer Paulino
78bdcbb115
routing: prune channels only if both edges are present
We do this to ensure we don't prune too aggressively, as it's possible
that we've only received the channel announcement for a channel, but not
its accompanying channel updates.
2019-04-18 21:57:31 -07:00
Wilmer Paulino
29664c9704
routing: make test channel policies optional 2019-04-18 20:47:32 -07:00
Olaoluwa Osuntokun
373927fb3e
Merge pull request #2950 from joostjager/remove-route-cache
routing: remove route cache
2019-04-17 19:34:19 -07:00
Olaoluwa Osuntokun
a3c980b1e8
Merge pull request #2871 from halseth/router-subscribetopology-check-started
routing/notifications: check router started on SubscribeTopology
2019-04-16 21:32:10 -07:00
Johan T. Halseth
2b9efcbaeb
routing/router: define ErrRouterShuttingDown 2019-04-15 16:55:10 +02:00
Joost Jager
3b94627c12
routing: remove route cache
This commit removes the QueryRoutes route cache. It is causing wrong
routes to be returned because not all of the request parameters are
stored.

The cache allowed high frequency QueryRoutes calls to the same
destination and with the same amount to be returned fast. This behaviour
can also be achieved by caching the request on the client side. In case
a route is invalidated because of for example a channel update,
the subsequent SendToRoute call will fail. This is a trigger to call
QueryRoutes again for a fresh route.
2019-04-13 08:10:32 +02:00
Johan T. Halseth
c2af637d3c
routing/notifications: check router startd on SubscribeTopology
If the router is not yet started, return an error to avoid a deadlock
waiting for it to handle the subscription request.
2019-04-02 10:21:08 +02:00
Wilmer Paulino
44a01db0ef
routing: expose VerifyChannelUpdateSignature function 2019-03-27 13:07:47 -07:00
Wilmer Paulino
23796d3247
routing+discovery: extend ChannelGraphSource with zombie index methods 2019-03-27 13:07:30 -07:00
Wilmer Paulino
174645fcba
routing+server: expose DefaultChannelPruneExpiry 2019-03-27 13:07:13 -07:00
Wilmer Paulino
c82d73a826
channeldb+routing: extend edge lookup methods with zombie index check
In this commit, we extend the graph's FetchChannelEdgesByID and
HasChannelEdge methods to also check the zombie index whenever the edge
to be looked up doesn't exist within the edge index. We do this to
signal to callers that the edge is known, but only as a zombie, and the
only information that we have about the edge are the node public keys of
the two parties involved in the edge.

In the event that an edge does exist within the zombie index, we make
an additional check on edge policies to ensure they are not within the
router's pruning window, indicating that it is a fresh update.
2019-03-27 13:06:57 -07:00
Joost Jager
06cd64cbbc
routing: add cltv limit to payment session 2019-03-19 10:36:51 +01:00
Joost Jager
e3bb3d46a2
routing: abstract path finding interface 2019-03-19 10:36:48 +01:00
Joost Jager
6006549ed5
routing: enforce cltv limit in path finding 2019-03-19 10:36:45 +01:00
Joost Jager
c5961d4904
routing: only warn for zero cltv delta edges
This condition may be caused by a bug somewhere else in the system.
Expose it here as a warn log line.
2019-03-19 10:36:43 +01:00
Joost Jager
2be1051fb6
zpay32: move HopHint and DefaultFinalCLTVDelta 2019-03-15 10:08:32 +01:00
Olaoluwa Osuntokun
d86fe393cd
multi: update build to latest version of neutrino+btcwallet
In this commit, we update the build to point to the latest version of
neutrino and btcwallet. The latest version of neutrino includes a number
of bug fixes, and new features like reliably transaction broadcast. The
latest version of btcwallet contains a number of bug fixes related to
properly remove invalid transactions from its database.
2019-03-13 15:03:39 -07:00
Joost Jager
6cc82b4a34
routing: add todo describing route cache bug 2019-03-06 15:31:09 +01:00
Joost Jager
c62c9d64da
routing: add source parameter to query routes
This commit allows execution of QueryRoutes from any source node.
Previously this was restricted to only the self node.
2019-03-06 15:31:03 +01:00
Joost Jager
7719bc432f
routing: take Vertex types as path finding source and target nodes
Currently public keys are represented either as a 33-byte array (Vertex) or as a
btcec.PublicKey struct. The latter isn't useable as index into maps and
cannot be used easily in compares. Therefore the 33-byte array
representation is used predominantly throughout the code base.

This commit converts the argument types of source and target nodes for
path finding to Vertex. Path finding executes no crypto operations and
using Vertex simplifies the code.

Additionally, it prepares for the path finding source parameter to be
exposed over rpc in a follow up commit without requiring conversion back
and forth between Vertex and btcec.PublicKey.
2019-03-06 15:31:00 +01:00
Joost Jager
b09adc3219
lnrpc+routing: add edges and nodes restrictions to query routes
This commit allows the execution of QueryRoutes to be controlled using
lists of black-listed edges and nodes. Any path returned will not pass
through the edges and/or nodes on the list.
2019-03-06 15:30:57 +01:00
Joost Jager
4376f3e1bd
routing: allow nil maps for ignored edges and nodes
This allows removing a lot of empty map initialization code and makes
the code more readable.
2019-03-06 15:30:54 +01:00
Joost Jager
b2b28b49b1
routing: export RestrictParams and EdgeLocator 2019-03-06 15:30:52 +01:00
Joost Jager
4937304732
routing: remove redundant fee limit check in newRoute
This check was a left over from when the fee limit wasn't checked yet in
the path finding algorithm.
2019-03-06 15:30:49 +01:00
Joost Jager
f4cc2e235a
routing: add todo describing fee limit bug 2019-03-06 15:30:46 +01:00
Johan T. Halseth
cee18892b5
Merge pull request #2565 from joostjager/sendpayment-refactor
routing: sendPayment broken down into multiple functions
2019-03-06 10:20:20 +01:00
Valentine Wallace
648adaea69 routing/pathfind: ensure max htlc is considered during path finding
In this commit, we update the path finding logic to
ignore a channel if the HTLC value (including the fees
at the point) exceeds the max HTLC value (if set) of the
link.
2019-03-01 08:05:27 -08:00
Valentine Wallace
348a66ed13 routing: update test edge policy fields to include max htlc + new flags
Since the MaxHTLC field was recently added to the ChannelEdgePolicy struct,
and the Flags field was broken into ChannelFlags and MessageFlags, the
test edge policies should be updated accordingly.
2019-03-01 08:05:27 -08:00
Valentine Wallace
0c6c1040d8 routing/ntfns+rpcserver: include max htlc in topology notifications 2019-02-21 18:39:32 -08:00
Johan T. Halseth
01ea797d78
routing/router test: add TestIgnoreChannelEdgePolicyForUnknownChannel 2019-02-14 14:21:18 +01:00
Johan T. Halseth
23ce82f4e3
routing/router: don't apply channel policy updates for unknown channels 2019-02-14 14:21:18 +01:00
Johan T. Halseth
1259bacd49
routing: remove unused code, nodeIndex and chanIndex 2019-02-13 12:23:45 +01:00
Joost Jager
f8721ffbca
routing: sendPayment broken down into multiple functions 2019-02-11 08:42:41 +01:00
Alex Bosworth
b3ae5bc16e
trivial: typo fix 2019-02-10 21:18:34 -08:00
Joost Jager
7c30a8c493
routing: add outgoing channel restriction 2019-02-08 23:05:56 +01:00
Joost Jager
7b94d9adb0
routing: move payment session to separate file 2019-02-01 09:14:54 +01:00
Joost Jager
9e012ecc93
multi: move Input interface and related code
This commit is a step to split the lnwallet package. It puts the Input
interface and implementations in a separate package along with all their
dependencies from lnwallet.
2019-01-31 13:25:33 +01:00
orbitalturtle
08750f180b multi: deprecate IncorrectHtlcAmount onion error
In this commit, we deprecate the `IncorrectHtlcAmount` onion error.
We'll still decode this error to use when retrying paths, but we'll no
longer send this ourselves. The `UnknownPaymentHash` error has been
amended to also include the value of the payment as well. This allows us
to worry about one less error.
2019-01-29 18:21:13 -08:00
Johan T. Halseth
7d34ce9d08
lnwire+multi: define HasMaxHtlc helper on msgFlags 2019-01-22 08:42:30 +01:00
Valentine Wallace
cced754659
routing/router: when applying onion failure channel update, set max htlc
In this commit, we ensure that when we update an edge
as a result of a ChannelUpdate being returned from an
onion failure, the max htlc portion of the channel update
is included in the edge update.
2019-01-22 08:42:29 +01:00
Valentine Wallace
15168c391e
discovery+routing: validate msg flags and max htlc in ChannelUpdates
In this commit, we alter the ValidateChannelUpdateAnn function in
ann_validation to validate a remote ChannelUpdate's message flags
and max HTLC field. If the message flag is set but the max HTLC
field is not set or vice versa, the ChannelUpdate fails validation.

Co-authored-by: Johan T. Halseth <johanth@gmail.com>
2019-01-22 08:42:27 +01:00
Valentine Wallace
0fd6004958
multi: partition lnwire.ChanUpdateFlag into ChannelFlags and MessageFlags
In this commit:

* we partition lnwire.ChanUpdateFlag into two (ChanUpdateChanFlags and
ChanUpdateMsgFlags), from a uint16 to a pair of uint8's

* we rename the ChannelUpdate.Flags to ChannelFlags and add an
additional MessageFlags field, which will be used to indicate the
presence of the optional field HtlcMaximumMsat within the ChannelUpdate.

* we partition ChannelEdgePolicy.Flags into message and channel flags.
This change corresponds to the partitioning of the ChannelUpdate's Flags
field into MessageFlags and ChannelFlags.

Co-authored-by: Johan T. Halseth <johanth@gmail.com>
2019-01-22 08:42:26 +01:00
Conner Fromknecht
a5b22788bf
routing/router: avoid naked spew of channel policies 2019-01-09 23:54:41 -08:00
Olaoluwa Osuntokun
11c24d3e0f
Merge pull request #1745 from halseth/channeldb-avoid-create-buckets
channeldb: avoid creating empty buckets
2018-12-17 20:01:29 -08:00
Joost Jager
b2cb760c65
routing: check for empty hops list
This commit fixes a crash that could be triggered by sending an empty
hop list to the SendToRoute rpc.
2018-12-10 11:27:07 +01:00
Johan T. Halseth
81fe6e73ed
router+graph: return ErrGraphNodesNotFound if no nodes to prune
Avoids creating a bucket unneccessarily.
2018-12-06 10:22:15 +01:00
Johan T. Halseth
87df6b71c5
router test: only cleanup if context creation succeeds 2018-12-06 10:22:15 +01:00
Joost Jager
3ed16837e4
routing: prune channels unidirectional
In this commit we introduce pruning of channel edges instead of channels.
Channel failures apply to a single direction and it is unnecessarily
restricting to prune both directions.
2018-12-05 14:39:23 +01:00
Joost Jager
8b8e82a171
routing: remove route hop maps
Hop maps were used in a test to verify the population of the hop map
itself and further only in a single function (getFailedChannelID).
Rewrote that function and removed the hop maps completely.
2018-12-05 14:39:23 +01:00
Joost Jager
42debc6c12
routing: fix node ordering in test graph
There is the general assumption that channel edge policy nodes are
ordered such that the node1 pubkey is smaller than the key of node 2. In
the test graph, this assumption didn't hold. This commit fixes the test
graph and also adds a check to prevent this from happening again.
2018-12-05 09:15:27 +01:00
Johan T. Halseth
5f8425cb03
routing/pathfind_test: check error before attempting cleanup 2018-12-04 12:20:45 +01:00
Johan T. Halseth
67b018476c
routing/pathfind test: add TestPathSourceEdgesBandwidth
This commit adds a new test that checks that the bandwidth hints are
considered correclty for local channels, and that disable flags are
ignored in this case.
2018-12-04 12:20:45 +01:00
Johan T. Halseth
0b6a19866b
routing/pathfind: pass ignore disable flags for local channels
To decouple our own path finding from the graph state, we don't consider
the disable bit when attempting to use local channels. Instead the
bandwidth hints will be zero for local inactive channels.

We alos modify the unit test to check that the disable flag is ignored
for local edges.
2018-12-04 12:20:45 +01:00
Johan T. Halseth
20b9d5d000
routing: define graphParams and restrictParams
To avoid the findPath parameter list getting out of hand, we define new
structs that wraps the mandatory and optional parameters to findPath.
2018-12-04 12:20:44 +01:00
Olaoluwa Osuntokun
5075394617
Merge pull request #1706 from joostjager/errorprocessing
routing: make routing retry behaviour consistent
2018-12-03 18:29:07 -08:00
Joost Jager
b6ce03e569
routing: make routing retry behaviour consistent
Fixes the following issues:
- If the channel update of FailFeeInsufficient contains an invalid channel
  update, it is not possible to properly add to the failed channels set.
- FailAmountBelowMinimum may apply a channel update, but does not retry.
- FailIncorrectCltvExpiry immediately prunes the vertex without
  trying one more time.

In this commit, the logic for all three policy related errors is
aligned.
2018-12-03 13:23:27 +01:00
Olaoluwa Osuntokun
1fd3aac925
multi: switch from bolt packge to bbolt package for all imports 2018-11-29 20:33:49 -08:00
Joost Jager
6ba1144528
routing: move failed channels map into payment session
This is a small preparatory step towards moving mission control logic
out of router and reusing the acquired routing result data.
2018-11-29 10:31:38 +01:00
Joost Jager
ac04729cff
routing: use complete route in test
Previously not all route fields were properly populated. Example: prev
and next hop maps.
2018-11-29 10:31:38 +01:00
Joost Jager
71037969b7
routing: remove pruneVertexFailure function 2018-11-29 10:31:38 +01:00
Joost Jager
aca136a91c
routing: remove unused pruneVertexFailure parameters 2018-11-29 10:31:38 +01:00
Joost Jager
dd7e2e9e04
routing: move logging into applyChannelUpdate
To remove code duplicated at all call sites to check err and log.
2018-11-29 10:31:38 +01:00
Wilmer Paulino
4f45e339f1
discovery+routing: add FetchLightningNode to ChannelGraphSource interface 2018-11-11 17:48:05 -08:00
Wilmer Paulino
18ecb31983
routing/router: extend ChannelGraphSource interface with IsPublicNode
method
2018-10-24 17:28:54 -07:00
Olaoluwa Osuntokun
ccc1dffac4
routing: minor style clean ups 2018-10-23 17:16:39 -07:00
Olaoluwa Osuntokun
fbd91feace
Merge pull request #1888 from joostjager/routestruct
lnrpc+routing: fix unmarshallRoute and simplify route structure
2018-10-23 17:07:23 -07:00
Johan T. Halseth
29acb9f1f8
routing/chainview: new neutrino API 2018-10-16 19:28:29 -07:00
Joost Jager
1d97cf1229
htlcswitch+routing: implement expiry_too_far failure
In this commit we add a check to HtlcSatifiesPolicy to verify that the
time lock for the outgoing htlc that is requested in the onion packet
isn't too far in the future.

Without this check, anyone could force an unreasonably long time lock on
the forwarding node.
2018-10-15 08:51:08 +02:00
Joost Jager
a779004a18
lnrpc+routing: fix issues with missing data in unmarshallRoute
In this commit the dependency of unmarshallRoute on edge policies being
available is removed. Edge policies may be unknown and reported as nil.
SendToRoute does not need the policies, but it does need pubkeys of the
route hops. In this commit, unmarshallRoute is modified so that it
takes the pubkeys from edgeInfo instead of channelEdgePolicy.

In addition to this, the route structure is simplified. No more connection
to the database at that point. Fees are determined based on incoming and
outgoing amounts.
2018-10-13 22:59:23 +02:00
Johan T. Halseth
d8863bea8c
routing/chainview: demote logging of new utxos for filter to Trace
It was previously Debug, which would spam the logs at startup, and not
provide very useful (human-readable) information.
2018-10-10 15:00:50 +02:00
Olaoluwa Osuntokun
3fda1029c0
Merge pull request #1786 from Roasbeef/golang-1.1
build+docs: bump to golang 1.11
2018-10-08 13:26:07 +09:00
Olaoluwa Osuntokun
d8e6085c17
multi: update to go 1.11 gofmt 2018-10-07 14:37:38 +09:00
Conner Fromknecht
59b459674d
multi: init subsystem loggers via build pkg 2018-10-05 13:04:45 +09:00
Olaoluwa Osuntokun
6afee3d099
Merge pull request #1856 from maurycy/typos
multi: fix various typos in comments
2018-09-27 20:38:10 -07:00
Olaoluwa Osuntokun
25145acc46
Merge pull request #1738 from joostjager/validation
payment failure message with attached channel_update not validated
2018-09-17 21:11:02 -07:00
Wilmer Paulino
2fa64dd6ff multi: update to new BitcoindClient API 2018-09-12 21:05:19 -07:00
Joost Jager
a7fec827cc
routing: validate channel update in failure message 2018-09-11 20:12:09 +02:00
Joost Jager
b5fd32ff77
discovery+routing: move validation logic to routing package
Previously, gossiper was the only object that validated channel
updates. Because updates can also be received as part of a
failed payment session in the routing package, validation logic
needs to be available there too. Gossiper already depends on
routing and having routing call the validation logic inside
gossiper would be a circular dependency. Therefore the validation
was moved to routing.
2018-09-11 20:12:07 +02:00
Joost Jager
ac8a3af84c
routing: add channel update validation test 2018-09-11 20:10:15 +02:00
Joost Jager
47d2e1e024
routing: add option to specify explicit id for test channels 2018-09-11 20:10:15 +02:00
Joost Jager
c5c580ab54
routing: add private key map to test graph instance 2018-09-11 20:10:14 +02:00
Joost Jager
5daf75b264
routing: restructure test context creation 2018-09-11 20:10:14 +02:00
Joost Jager
2d255e3bc3
routing: use fixed time value in tests 2018-09-11 20:10:14 +02:00
maurycy
ac24b12bf2
multi: fix various typos in comments 2018-09-07 06:51:49 +02:00
Conner Fromknecht
89113654fe
routing/conf: add experimental assume valid conf 2018-09-03 20:15:18 -07:00
Conner Fromknecht
1e473b2364
routing/router: add assume chan valid 2018-09-03 20:15:12 -07:00
Olaoluwa Osuntokun
85c0dd3de0
routing/chainview: update to latest API changes 2018-08-24 03:36:25 -07:00
Wilmer Paulino
191c4f3e28
multi: enable txindex on miner harness 2018-08-24 03:36:25 -07:00
Johan T. Halseth
4b04e1afd1
routing: return ErrRejected if found in reject cache
We make sure to return an error other than ErrIgnored, as ErrIgnored is
expected to only be returned for updates where we already have the
necessary information in the database.

In case of a channel ID found in the rejectCache, there was a
possibility that we had rejected an invalid update for this channel
earlier, and when attempting to add the current update we wouldn't
distinguish the failure to add from an outdated/ignored update.
2018-08-23 07:40:40 +02:00
Johan T. Halseth
18e680f192
router: simplify edge outdated logic, return ErrOutdated if old. 2018-08-23 07:40:39 +02:00
Wilmer Paulino
6c903393e5 router+server: use first hop channel id rather than node key 2018-08-20 21:04:52 -07:00
Olaoluwa Osuntokun
518b0d2cf7
Merge pull request #1679 from cfromknecht/move-linter-v2
Makefile: move linter v2
2018-08-09 21:03:05 -07:00
Olaoluwa Osuntokun
3438baaf0c
routing: eliminate nested db transactions by using new FetchOtherNode method 2018-08-09 20:48:06 -07:00
Joost Jager
29b6bae4ed routing: convert fee limit test to table-driven 2018-08-09 20:23:40 -07:00
Joost Jager
3e7473f4f0 routing: backward searching 2018-08-09 20:23:40 -07:00
Joost Jager
e52d829168 routing: path finding test refactored 2018-08-09 20:23:40 -07:00
Joost Jager
6c918a1806 channeldb: store unknown policies in database
The commit ensures that for every channel, there will always
be two entries in the edges bucket. If the policy from one or
both ends of the channel is unknown, it is marked as such.

This allows efficient lookup of incoming edges. This is
required for backwards payment path finding.
2018-08-09 20:23:40 -07:00
Wilmer Paulino
9d2eeb6304
multi: update to latest fee estimation interface 2018-08-09 17:29:52 -07:00
Wilmer Paulino
936fcc1f16
multi: update to latest BitcoindClient interface
In this commit, we introduce a nice optimization with regards to lnd's
interaction with a bitcoind backend. Within lnd, we currently have three
different subsystems responsible for watching the chain: chainntnfs,
lnwallet, and routing/chainview. Each of these subsystems has an active
RPC and ZMQ connection to the underlying bitcoind node. This would incur
a toll on the underlying bitcoind node and would cause us to miss ZMQ
events, which are crucial to lnd. We remedy this issue by sharing the
same connection to a bitcoind node between the different clients within
lnd.
2018-08-08 20:26:30 -07:00
Conner Fromknecht
a76b9a86cd
routing/chainview/interface_test: use EdgePoint field names 2018-08-07 16:57:40 -07:00
Conner Fromknecht
9eff876f52
routing/pathfind_test: fmt whitespace 2018-08-02 18:20:49 -07:00
Conner Fromknecht
614b3b3862
routing/pathfind: fmt whitespace 2018-08-02 18:20:49 -07:00
Conner Fromknecht
1ded697e8d
multi: sort import paths with gofmt 2018-08-02 18:20:49 -07:00
Olaoluwa Osuntokun
3c60861005
routing/chainview: give neutrino backend more time to catch up for re-org test
In this commit, we modify the test to explitlcy give the neutrino
backend more time to catch up compared to the RPC backends. We do this
as a recent change has been made in the neutrino backend to wait for the
filter headers to finish syncing before proceeding with the rescan
itself. As a result, we'll need to account for this in the test and
sleep enough to give the backend a chance to catch up.
2018-07-31 21:29:03 -07:00
Olaoluwa Osuntokun
8c0a151cec
routing: update UTXO verification to use new interface, graph pruning too 2018-07-31 21:28:55 -07:00
Olaoluwa Osuntokun
fb98f59407
routing/chainview: ensure bitcoind impl meets new interface requirements 2018-07-31 21:28:55 -07:00
Olaoluwa Osuntokun
9b9029cf4b
routing/chainview: ensure btcd impl meets new interface requirements 2018-07-31 21:28:55 -07:00
Olaoluwa Osuntokun
7e6997d6d3
routing/chainview: update neutrino API due to recent changes
In this commit, we ensure that the neutrino backend meets the target
interface, and also we update the API usage for the internal neutrino
rescan struct to use the new InputWithScript struct.
2018-07-31 21:28:54 -07:00
Olaoluwa Osuntokun
ec0fb9c537
routing/chainview: modify the UpdateFilter method to take a channeldb.EdgePoint
In this commit, we update the existing UpdateFilter method to take the
new channeldb.EdgePoint struct in place of the prior wire.OutPoint. We
must do this as the caller now typically has this type due to the
preparation to enable lnd to be able to be compatible with the new
neutrino protocol.
2018-07-31 21:28:54 -07:00
Olaoluwa Osuntokun
a6c814010c
routing: exit gracefully if generateSphinxPacket is passed a nil set of hops 2018-07-30 13:41:06 -07:00
Olaoluwa Osuntokun
c903a9a711
routing: add new TestEmptyRoutesGenerateSphinxPacket test 2018-07-30 13:40:25 -07:00
Olaoluwa Osuntokun
e0baa49690
Merge pull request #1387 from wpaulino/send-disable-chan-update
multi: send a channel update with disabled flag set on channel close
2018-07-23 17:46:37 -07:00
Olaoluwa Osuntokun
ef56f76200
routing: demote channel update logging to trace 2018-07-23 17:01:39 -07:00
Olaoluwa Osuntokun
2e75499787
channeldb+routing: move adding shell nodes into db txn of AddChannelEdge
In this commit, we fix a slight race condition that can occur when we go
to add a shell node for a node announcement, but then right afterwards,
a new block arrives that causes us to prune an unconnected node. To
ensure this doesn't happen, we now add shell nodes within the same db
transaction as AddChannelEdge. This ensures that the state is fully
consistent and shell nodes will be added atomically along with the new
channel edge.

As a result of this change, we no longer need to add shell nodes within
the ChannelRouter, as the database will take care of this operation as
it should.
2018-07-22 21:02:53 -07:00
Olaoluwa Osuntokun
f8cbe34e93
routing: prune nodes from the channel graph on start up 2018-07-21 19:52:26 -07:00
Olaoluwa Osuntokun
505544e1b3
routing/chainview: update API usage for neutrino due to recent changes 2018-07-13 17:15:12 -07:00
Olaoluwa Osuntokun
6f60f139f4 multi: switch over import paths from roasbeef/* to btcsuite/* 2018-07-13 17:05:39 -07:00
Wilmer Paulino
191e111661
routing: include disabled field in topology updates 2018-07-11 17:07:02 -07:00
Olaoluwa Osuntokun
16df90fc23
Merge pull request #1504 from cfromknecht/chainview-filter-locking
[routing/chainview] improve filter locking granularity in btcd/bitcoind
2018-07-10 17:19:36 -07:00
maurycy
3915c6a569 routing: do not reacquire the lock for topology notifications 2018-07-10 17:09:52 -07:00
Olaoluwa Osuntokun
e13f96e64b
routing: ensure we create a pruneViewSnapshot for route sessions
In this commit, we fix an existing bug that could at times lead to a
panic if a user manually crafts a route via SendToRoute, and that route
results in a payment error. The fix is simple: create the map even
though it won't be used in the sessions since the user is feeding the
router manual routes.
2018-07-08 16:34:25 -07:00
Conner Fromknecht
daa28e0d0f
routing/chainview/bitcoind: improve filterMtx locking
In this commit, we modify the granularity of the locking
around the filterMtx in the bitcoind chainview, such that
we only lock once per block connected or filter update.
Currently, we acquire and release the lock for every
update to the map.

We also fix a bug that would cause us to not fully remove
all previous outpoints spent by a txn when doing manual
filter, as we previously would only remove the first output
detected.
2018-07-06 19:08:17 -07:00
Conner Fromknecht
0f6bc3538d
routing/chainview/btcd: improve filterMtx locking
In this commit, we modify the granularity of the locking
around the filterMtx in the btcd chainview, such that we
only lock once per block connected or filter update.
Currently, we acquire and release the lock for every
update to the map.

We also fix a bug that would cause us to not fully remove
all previous outpoints spent by a txn when doing manual
filter, as we previously would only remove the first output
detected.
2018-07-06 19:08:17 -07:00
Olaoluwa Osuntokun
39c1f46f85
routing: new newLogClosure to delay spew evaluation until log print time
In this commit, we update the generateSphinxPacket to use newLogClosure
to delay the spew evaluation until log print time. Before this commit,
even if we weren't on the trace logging level, the spew call would
always be evaluated.
2018-06-30 17:14:28 -05:00
Olaoluwa Osuntokun
c045defd0f
Merge pull request #1358 from joostjager/newfee
routing: routing may come up with suboptimal routes (weight function)
2018-06-28 19:18:13 -07:00
Joost Jager
b52796749f routing: new weight function
In this commit, a new weight function is introduced. This will create a
meaningful effect of time lock on route selection. Also, removes the
squaring of the fee term. This led to suboptimal routes.

Unit test added that covers the weight function and asserts that the
lowest fee route is indeed returned.
2018-06-28 14:52:39 +02:00
Joost Jager
416f368979 routing: add time lock asserts to NewRoute test
This comment extends the unit tests for NewRoute with checks
on the total time lock for a route as well as the expected time
lock values for every hop along the route.
2018-06-28 14:04:02 +02:00
Joost Jager
d4471878df routing: fix route fee calculation and channel capacity check
This commit fixes the logic inside the newRoute function to
address the following problems:

- Fee calculation for a hop does not include the fee that needs
  to be paid to the next hop.

- The incoming channel capacity "sanity" check does not include
  the fee to be paid to the current hop.
2018-06-28 13:53:45 +02:00
Olaoluwa Osuntokun
6a6c29cd00
routing: fix incorrect expiry values in spec_example.json
In this commit, we fix the incorrect expiry values in the
spec_example.json test file. Many of the time locks were incorrect which
allowed bugs within the path finding logic related to CLTV deltas to go
un-detected.
2018-06-26 17:29:42 -07:00
Olaoluwa Osuntokun
6c986864ff
routing: fix bug in newRoute, use time lock delta of prior hop, not current
In this commit, we fix an existing bug in the newRoute method. Before
this commit we would use the time lock delta of the current hop to
compute the outgoing time lock for the current hop. This is incorrect as
the time lock delta of the _outgoing_ hop should be used, as this is
what we're paying for "transit" on. This is a bug left over from when we
switched the meaning of the CLTV delta on the ChannelUpdate message
sometime last year.

The fix is simple: use the CLTV delta of the prior (later in the route)
hop.
2018-06-26 17:29:42 -07:00
Wilmer Paulino
ddf8f2cb01
routing: allow specifying a fee limit during route construction 2018-06-12 18:14:48 -07:00
Wilmer Paulino
6a6de812ba
routing: denote a payment's fee limit in millisatoshis instead 2018-06-12 18:14:47 -07:00
Sebastian Delgado
6746609ec6
Add constant and percentage-based fee limits to payments
- Extend SendRequest and QueryRoutesRequest protos
- newRoute function takes fee limit and cuts off routes that exceed it
- queryRoutes, payInvoice and sendPayment commands take the feeLimit inputs and pass them down to newRoute
- When no feeLimit is included, don't enforce any feeLimits at all (by setting feeLimit to maxValue)
2018-06-12 18:14:43 -07:00
Olaoluwa Osuntokun
eb65b0cd5a
routing: check to see if pubkey is nil before logging in sendPayment
In this commit, we now need to check to see if the pubkey is nil as if
the user specified a set of manual routes, then the dest pubkey would be
empty.
2018-06-06 20:45:56 -07:00
Olaoluwa Osuntokun
0113035f6d
routing: add new NewPaymentSessionFromRoutes function instead of interface
In this commit, we modify the recent refactoring of the mission control
sub-system to overload the existing payment session, rather than create
a brand new one. This allows us to re-use more of the existing logic, and
also feedback into mission control the failures incurred by any user
selected routes.
2018-06-06 20:45:56 -07:00
t4sk
53eb396ae9 routing/router: add SendToRoute 2018-06-06 20:45:00 -07:00
maurycy
3be08e69cf multi: 64bit aligment of atomic vars on arm/x86-32 2018-06-04 20:02:34 -07:00
Olaoluwa Osuntokun
f494433cbf
routing+server: add new QueryBandwidth method to reduce outbound failures
In this commit, we introduce a new method to the channel router's config
struct: QueryBandwidth. This method allows the channel router to query
for the up-to-date available bandwidth of a particular link. In the case
that this link emanates from/to us, then we can query the switch to see
if the link is active (if not bandwidth is zero), and return the current
best estimate for the available bandwidth of the link. If the link,
isn't one of ours, then we can thread through the total maximal
capacity of the link.

In order to implement this, the missionControl struct will now query the
switch upon creation to obtain a fresh bandwidth snapshot. We take care
to do this in a distinct db transaction in order to now introduced a
circular waiting condition between the mutexes in bolt, and the channel
state machine.

The aim of this change is to reduce the number of unnecessary failures
during HTLC payment routing as we'll now skip any links that are
inactive, or just don't have enough bandwidth for the payment. Nodes
that have several hundred channels (all of which in various states of
activity and available bandwidth) should see a nice gain from this w.r.t
payment latency.
2018-05-14 16:23:54 -07:00
Conner Fromknecht
1b1f5f197a
routing/chainview/neutrino: cache filter entries
This commit alters the neutrino chainview such that it
caches the filter entries corresponding to watched
outpoints at the moment they are added to the filter.
Previously, we would rederive each filter entry when
reconstructing the relevant filter entries, which
would lead to unnecessary work on the gc. Now, each is
created at most once, and reused across subsequent
reconstructions.
2018-05-08 20:46:46 -07:00
Conner Fromknecht
99e7ec0895
routing/validation_barrier_test: test semaphore and quit
Adds test checking the basic functionality of the barrier's
semaphore and behavior during shutdown.
2018-05-08 16:32:08 -07:00
Conner Fromknecht
995e3fa85f
routing/validation_barrier: adds ErrVBarrierShuttingDown
Adds a new error ErrVBarrierShuttingDown that is returned
from WaitForDependants if the validation barrier's quit
chan is closed. This allows any blocked goroutines to
distinguish whether the dependent task has been completed,
or if validation should be aborted entirely.
2018-05-08 16:32:08 -07:00
Conner Fromknecht
eaa8cdf916
routing/router: improve validation barrier shutdown
This commit improves the shutdown of the router's
pending validation tasks, by ensuring the pending
tasks exit early if the validation barrier
receives a shutdown request.

Currently, any goroutines blocked by WaitForDependants
will continue execution after a shutdown is signaled.
This may lead to unnexpected behavior as the relation
between updates is no longer upheld. It also has the
side effect of slowing down shutdown, since we
continue to process the remaining updates.

To remedy this, WaitForDependants now returns an error
that signals if a shutdown was requested. The blocked
goroutines can exit early upon seeing this error,
without also signaling completion of their task to
the dependent tasks, which should will now properly
wait to read the validation barrier's quit signal.
2018-05-08 16:32:08 -07:00
Kenneth Perry
f7c5a7a19e routing/chainview/neutrino: error if we get a nil cfilter back (#1176) 2018-05-03 15:21:56 -07:00
Olaoluwa Osuntokun
76b9501e04
routing: update TestSendPaymentErrorPathPruning due to recent UnknownPeer change
In this commit, we update the TestSendPaymentErrorPathPruning test to
reflect the new behavior w.r.t how we respond to UnknownPeer errors. In
this new test, we expect that we'll find alternative route in light of
us getting an UnknownPeer error "pointing" to our destination node.
2018-04-23 17:53:46 -07:00
Olaoluwa Osuntokun
bd9f1b597e
routing: prune edges instead of vertexes in response to an FailUnknownNextPeer error
In this commit we fix an lingering bug in the Mission Control logic we
execute in response to the FailUnknownNextPeer error. Historically, we
would treat this as the _next_ node not being online. As a result, we
would then prune away the vertex from the current reachable graph all
together. It was recently realized, that this would at times be a bit
_tooo_ aggressive if the channel we attempt to route over was faulty,
down, or the incoming node had connectivity issues with the outgoing
node.

In light of this realization, we'll now instead only prune the _edge_
that we attempted to route over. This ensures that we'll continue to
explore the possible edges. Additionally, this guards us against failure
modes where nodes report FailUnknownNextPeer to other nodes in an
attempt to more closely control our retry logic.

This change is a stop gap on the path to a more intelligent set of
autopilot heuristics.

Fixes #1114.
2018-04-23 17:51:02 -07:00
Wilmer Paulino
beb08b3887
routing: add test for finding paths with additional edges 2018-04-20 04:01:33 -04:00
Wilmer Paulino
4598df054e
routing: modify findPath to take into account additional edges
In this commit, we modify our path finding algorithm to take an
additional set of edges that are currently not known to us that are
used to temporarily extend our graph with during a payment session.
These edges should assist the sender of a payment in successfully
constructing a path to the destination.

These edges should usually represent private channels, as they are not
publicly advertised to the network for routing.
2018-04-20 04:01:32 -04:00
Wilmer Paulino
7965247db1
routing: add additionalEdges field to a missionControl's paymentSession
In this commit, we introduce the ability for payment sessions to store
an additional set of edges that can be used to assist a payment in
successfully reaching its destination.
2018-04-20 04:01:31 -04:00