Commit Graph

60 Commits

Author SHA1 Message Date
carla
ab594ea57b
channeldb: update QueryPayments to use sequence nr index and paginator
Use the new paginatior strcut for payments. Add some tests which will
specifically test cases on and around the missing index we force in our
test to ensure that we properly handle this case. We also add a sanity
check in the test that checks that we can query when we have no
payments.
2020-06-10 12:55:05 +02:00
carla
38624e8612
channeldb: add fetchPaymentWithSequenceNumber lookup and test
With our new index of sequence number to index, it is possible for
more than one sequence number to point to the same hash because legacy
lnd allowed duplicate payments under the same hash. We now store these
payments in a nested bucket within the payments database. To allow
lookup of the correct payment from an index, we require matching of the
payment hash and sequence number.
2020-06-10 12:55:03 +02:00
carla
c8d11285f3
channeldb: index payments by sequence number
Add an entry to a payments index bucket which maps sequence number
to payment hash when we initiate payments. This allows for more
efficient paginated queries. We create the top level bucket in its
own migration so that we do not need to create it on the fly.

When we retry payments and provide them with a new sequence number, we
delete the index for their existing payment so that we do not have an
index that points to a non-existent payment.

If we delete a payment, we also delete its index entry. This prevents
us from looking up entries from indexes to payments that do not exist.
2020-06-10 12:49:06 +02:00
Conner Fromknecht
455ddfebdb
multi: rename: ReadBucket to RBucket 2020-05-26 18:21:08 -07:00
Conner Fromknecht
d0d2ca403d
multi: rename ReadTx to RTx 2020-05-26 18:20:37 -07:00
Olaoluwa Osuntokun
cc1dbb5677
channeldb: eliminate extra copy in QueryPayments
In this commit, we eliminate an extraneous copy in the `QueryPayments`
method. Before this commit, we would copy each payment from the initial
FetchPayments call into a new slice. However, pointers to payments are
return from `FetchPayments`, so we can just maintain that same reference
rather than copying again when we want to limit our response.
2020-05-04 12:13:47 -07:00
Joost Jager
7fc1938f10
routing: payment splitting pre-check
This commit reverts cb4cd49dc8d3b0255afe9ff29af9c46c2dbb2c98 to bring
back the insufficient local balance failure.

Distinguishing betweeen this failure and a regular "no route" failure
prevents meaningless htlcs from being sent out.
2020-04-18 08:30:26 +02:00
Joost Jager
cb4cd49dc8
routing: map insufficient local bandwidth error to no path
With mpp it isn't possible anymore for findPath to determine that there
isn't enough local bandwidth. The full payment amount isn't known at
that point.

In a follow-up, this payment outcome can be reintroduced on a higher
level (payment lifecycle).
2020-04-09 08:20:35 +02:00
bitromortac
4c5e8aec41 channeldb: add payments query
Adds a PaymentsQuery struct, which contains parameters to restrict the
response of QueryPayments, returning a PaymentsQuerySlice with the
payments query result. The behavior of this api is the same as
the QueryInvoices one.
2020-04-07 07:03:21 +02:00
bitromortac
4800a84122 channeldb: export sequenceNum in MPPayment
Exports sequenceNum in MPPayment for later use
in the rpcserver.
2020-04-07 07:03:21 +02:00
Johan T. Halseth
9a1ec950bd
channeldb/payments: extract common info fetch into fetchCreationInfo 2020-04-02 19:29:15 +02:00
Johan T. Halseth
70202be580
channeldb: make database logic MPP compatible
This commit redefines how the control tower handles shard and payment
level settles and failures. We now consider the payment in flight as
long it has active shards, or it has no active shards but has not
reached a terminal condition (settle of one of the shards, or a payment
level failure has been encountered).

We also make it possible to settle/fail shards regardless of the payment
level status (since we must allow late shards recording their status
even though we have already settled/failed the payment).

Finally, we make it possible to Fail the payment when it is already
failed. This is to allow multiple concurrent shards that reach terminal
errors to mark the payment failed, without havinng to synchronize.
2020-04-02 19:29:14 +02:00
Johan T. Halseth
9bcf41d401
channeldb: make FailureReason Error() 2020-04-02 10:24:34 +02:00
Olaoluwa Osuntokun
f0911765af
channeldb: convert to uniformly use new kvdb abstractions
In this commit, we migrate all the code in `channeldb` to only reference
the new `kvdb` package rather than `bbolt` directly.

In many instances, we need to add two version to fetch a bucket as both
read and write when needed. As an example, we add a new
`fetchChanBucketRw` function. This function is identical to
`fetchChanBucket`, but it will be used to fetch the main channel bucket
for all _write_ transactions. We need a new method as you can pass a
write transaction where a read is accepted, but not the other way around
due to the stronger typing of the new `kvdb` package.
2020-03-18 19:34:49 -07:00
Joost Jager
48c0e42c26
channeldb+routing: store all payment htlcs
This commit converts the database structure of a payment so that it can
not just store the last htlc attempt, but all attempts that have been
made. This is a preparation for mpp sending.

In addition to that, we now also persist the fail time of an htlc. In a
later commit, the full failure reason will be added as well.

A key change is made to the control tower interface. Previously the
control tower wasn't aware of individual htlc outcomes. The payment
remained in-flight with the latest attempt recorded, but an outcome was
only set when the payment finished. With this commit, the outcome of
every htlc is expected by the control tower and recorded in the
database.

Co-authored-by: Johan T. Halseth <johanth@gmail.com>
2020-03-09 18:31:33 +01:00
Joost Jager
fa3a762a2c
channeldb: add error return value to fetchPaymentStatus
Preparation for when we need to return errors in a next commit.
2020-03-09 11:43:38 +01:00
Joost Jager
cc5e18c487
channeldb: isolate duplicate payments
Duplicate payments is legacy that we keep alive for accounting purposes.
This commit isolates the deserialization logic for duplicate payments in
its own file, so that regular payment logic and db structure can evolve
without needing to handle/migrate the legacy data.
2020-03-09 11:43:30 +01:00
Johan T. Halseth
bee2380441
channeldb: rename PaymentAttemptInfo to HTLCAttemptInfo
To better distinguish payments from HTLCs, we rename the attempt info
struct to HTLCAttemptInfo. We also embed it into the HTLCAttempt struct,
to avoid having to duplicate this information.

The paymentID term is renamed to attemptID.
2020-03-09 11:43:26 +01:00
Joost Jager
967b4b2dc3
channeldb: remove redundant MPPaymentCreationInfo struct
MPPaymentCreationInfo and PaymentCreationInfo are identical except for
the naming of CreationTime/CreationDate.
2020-03-09 11:43:24 +01:00
Johan T. Halseth
cdf3a49c54
channeldb: remove unused payments status serialization 2020-02-18 13:01:57 +01:00
Joost Jager
de2b7b78de
channeldb: custom records sanity check 2019-12-12 17:49:36 +01:00
Joost Jager
d02de70d20
multi: do not use tlv.Record outside wire format handling
This commit prepares for more manipulation of custom records. A list of
tlv.Record types is more difficult to use than the more basic
map[uint64][]byte.

Furthermore fields and variables are renamed to make them more
consistent.
2019-12-12 00:14:58 +01:00
Joost Jager
5d4ceca038
tlv: remove unused error return value
This commit also modifies a previous migration. Because the change is so
limited, we don't consider this a significant risk.
2019-12-04 13:27:29 +01:00
Joost Jager
97344af8f3
routing: local balance check 2019-12-04 09:45:07 +01:00
Conner Fromknecht
063f24f2ed
channeldb+routing: expose HTLCs in payment subscriptions
This commit modifies the FetchPayment method to return MPPayment structs
converted from the legacy on-disk format. This allows us to attach the
HTLCs to the events given to clients subscribing to the outcome of an
HTLC.

This commit also bubbles up to the routerrpc/router_server, by
populating HTLCAttempts in the response and extracting the legacy route
field from the HTLCAttempts.
2019-11-19 20:43:38 -08:00
Conner Fromknecht
01cecb1f27
multi: rename to FailureReasonPaymentDetails
Prior name is too long XD
2019-11-19 20:42:21 -08:00
Conner Fromknecht
77602451b8
channeldb: add MPPayment from FetchPayments 2019-11-19 20:41:08 -08:00
Conner Fromknecht
fbd599a2cb
multi: rename Payment.PaymentPreimage to Payment.Preimage
Preliminary step to exposing PaymentPreimage() as a method that can be
shared between legacy payments and mutli-path payments.
2019-11-19 20:40:29 -08:00
Conner Fromknecht
6d971e5113
routing/route/route: add optional MPP field to Hop
This commit also modifies the Router serialization to persist the MPP
struct when present, and properly restore it when loading from disk.
2019-11-04 14:11:28 -08:00
Olaoluwa Osuntokun
c78e3aaa9d
channeldb: update route.Hop serialization to include new EOB related fields
We also include a migration for the existing routes stored on disk.
2019-08-22 18:53:04 -07:00
Joost Jager
eb4e65e54f
channeldb: export route serialization 2019-07-29 13:42:57 +02:00
Joost Jager
ae46fb00cb
routing+channeldb: add more failure reasons 2019-07-04 09:27:12 +02:00
Joost Jager
2644759924
channeldb: add todo for more detailed failure reasons 2019-06-05 12:41:55 +02:00
Joost Jager
f03533c67a
routerrpc: convert sendpayment to async
Modify the routerrpc SendPayment api to asynchronous. This allows
callers to pick up a payment after the rpc connection was lost or lnd
was restarted.
2019-06-05 12:41:53 +02:00
Johan T. Halseth
95b4828780
channeldb: derive PaymentStatus implicitly from stored info 2019-05-27 20:18:59 +02:00
Johan T. Halseth
d705b8a013
channeldb+multi: rename Grounded->Unknown, Completed->Succeeded 2019-05-27 20:18:59 +02:00
Johan T. Halseth
d78d3f50b9
channeldb: move deprecated code to migration_09_legacy_serialization.go
And unexport deprecated code.
2019-05-27 20:18:59 +02:00
Johan T. Halseth
1b788904f0
channeldb+router: record payment failure reason in db 2019-05-27 20:18:59 +02:00
Johan T. Halseth
de1bf8a518
routing/router: persist payment state machine
This commit makes the router use the ControlTower to drive the payment
life cycle state machine, to keep track of active payments across
restarts.  This lets the router resume payments on startup, such that
their final results can be handled and stored when ready.
2019-05-27 20:18:58 +02:00
Johan T. Halseth
6d80661bbb
channeldb/payments: add StatusFailed 2019-05-27 20:18:57 +02:00
Johan T. Halseth
693807cf6e
channeldb/control_tower: add FetchSentPayments 2019-05-27 20:18:56 +02:00
Johan T. Halseth
178996f0d3
channeldb/payments+control_tower: split OutgoingPayments
This commit changes the format used to store payments within the
DB. Previously this was serialized as one continuous struct
OutgoingPayment, which also contained an Invoice struct we where only
using a few fields of. We now split it up into two simpler sub-structs
CreationInfo, AttemptInfo and PaymentPreimage.

We also want to associate the payments more closely with payment
statuses, so we move to this hierarchy:

There's one top-level bucket "sentPaymentsBucket" which contains a set
of sub-buckets indexed by a payment's payment hash. Each such sub-bucket
contains several fields:
paymentStatusKey -> the payment's status
paymentCreationInfoKey -> the payment's CreationInfo.
paymentAttemptInfoKey -> the payment's AttemptInfo.
paymentSettleInfoKey -> the payment's preimage (or zeroes for
non-settled payments)

The CreationInfo is information that is static during the whole payment
lifcycle. The attempt info is set each time a new payment attempt
(route+paymentID) is sent on the network. The preimage is information
only known when a payment succeeds.  It therefore makes sense to split
them.

We keep legacy serialization code for migration puproses.
2019-05-27 20:18:56 +02:00
Johan T. Halseth
2417f40532
channeldb: put payment status in new bucket
We move the payment status to a new bucket hierarchy. Old buckets and
fetch methods are kept around for migration purposes.
2019-05-27 20:18:56 +02:00
Johan T. Halseth
15bed506b1
routing/route+multi: remove redundant TotalFees field
Instead get it on demand using method TotalFees().
2019-05-16 23:58:32 +02:00
Johan T. Halseth
89fd43ebcb
channeldb: add Route (de)serialization + test
We will store the routes used during payment attempts, so we need
serialization code.
2019-05-09 09:59:43 +02:00
Olaoluwa Osuntokun
1fd3aac925
multi: switch from bolt packge to bbolt package for all imports 2018-11-29 20:33:49 -08:00
Conner Fromknecht
86b347c996
channeldb/payments: make payment status helper methods
This commit splits FetchPaymentStatus and
UpdatePaymentStatus, such that they each invoke
helper methods that can be composed into different
db txns. This enables us to improve performance on
send/receive, as we can remove the exclusive lock
from the control tower, and allow concurrent calls
to utilize Batch more effectively.
2018-08-21 19:23:25 -07:00
Conner Fromknecht
e7c0f4c5dc
channeldb/payments: touch up docs 2018-08-21 19:23:23 -07:00
Vadym Popov
7296cfb425
channeldb: add payment statuses: ground, in flight, completed 2018-08-21 19:23:18 -07:00
Olaoluwa Osuntokun
dbf7e4525a
lntest: ensure we always cancel active streaming notifications 2018-07-06 12:21:54 -07:00