Commit Graph

641 Commits

Author SHA1 Message Date
Conner Fromknecht
fbd599a2cb
multi: rename Payment.PaymentPreimage to Payment.Preimage
Preliminary step to exposing PaymentPreimage() as a method that can be
shared between legacy payments and mutli-path payments.
2019-11-19 20:40:29 -08:00
Conner Fromknecht
5e27b5022c
multi: remove LocalFeatures and GlobalFeatures 2019-11-08 05:32:00 -08:00
Conner Fromknecht
6d971e5113
routing/route/route: add optional MPP field to Hop
This commit also modifies the Router serialization to persist the MPP
struct when present, and properly restore it when loading from disk.
2019-11-04 14:11:28 -08:00
Olaoluwa Osuntokun
38e313a869
Merge pull request #3633 from joostjager/safe-migrations
channeldb: isolate migrations
2019-11-01 18:28:56 -07:00
Olaoluwa Osuntokun
a73ee28e46
multi: convert the existing channeldb.ChannelType uint8 into a bit field
In this commit, we convert the existing `channeldb.ChannelType` type
into a _bit field_. This doesn't require us to change the current
serialization or interpretation or the type as it is, since all the
current defined values us a distinct bit. This PR lays the ground work
for any future changes that may introduce new channel types (like anchor
outputs), and also any changes that may modify the existing invariants
around channels (if we're the initiator, we always have the funding
transaction).
2019-10-31 16:34:37 -07:00
Joost Jager
4486a06b1a
migration_01_to_11: remove version checking for migration tests 2019-10-31 11:25:07 +01:00
Joost Jager
60503d6c44
channeldb/migration_01_to_11: remove unused code 2019-10-31 07:18:36 +01:00
Joost Jager
f5191440c5
channeldb: initialize migrations logger 2019-10-30 08:56:18 +01:00
Joost Jager
43449ca7a7
channeldb/migration_01_to_11: add references to untested migrations 2019-10-29 11:49:33 +01:00
Joost Jager
6913cd64b6
channeldb: reference migrations in package
This commit removes the migrations from channeldb and references those
in the migrations_01_to_11 package. This creates a one-way dependency on
the migrations. Future changes to channeldb won't be able to break
migrations anymore.
2019-10-24 12:49:40 +02:00
Joost Jager
6e463c1634
channeldb: make copy for migrations
This commit is a direct copy of the complete channeldb package. It only
changes the package declaration at the top of every file. We make this
full copy so that review can be focused on the actual changes made.
Otherwise changes may drown in all the file moves.

Linting for the new package is disabled, as it contains lots of
pre-existing issues.
2019-10-24 12:49:38 +02:00
Olaoluwa Osuntokun
e9d2ad5d73 Revert "channeldb: require minimum db upgrade version"
This reverts commit f1942a4c33dbea7144f756d9cd0139ab1fe8197f.
2019-10-12 14:30:26 -07:00
Olaoluwa Osuntokun
0386e0c297 Revert "channeldb: remove unsupported migrations"
This reverts commit 3ab4c749c6da4c2693b3978c0a0863787c9c3495.
2019-10-12 00:47:45 -07:00
Wilmer Paulino
fa96450db8
channeldb: add freelist sync option modifier 2019-10-04 12:19:41 -04:00
Joost Jager
b58dbb2d70
multi: fix canceled spelling 2019-10-03 17:27:36 +02:00
Joost Jager
ac7c93f544
channeldb: fail migration for accepted hodl invoices 2019-09-28 08:30:33 +02:00
Joost Jager
31f72f6c7d
channeldb: extract before migration func 2019-09-28 08:30:31 +02:00
Joost Jager
f47de09850
channeldb: log error for migration test failures 2019-09-28 08:23:49 +02:00
Olaoluwa Osuntokun
b399203e71
lnwallet: update channel state machine to be aware of tweakless commits
In this commit, we update the channel state machine to be aware of
tweakless commits. In several areas, we'll now check the channel's type
to see if it's `SingleFunderTweakless`. If so, then we'll opt to use the
remote party's non-delay based point directly in the script, skipping
any additional cryptographic operations. Along the way we move the
`validateCommitmentSanity` method to be defined _before_ it's used as is
cutomary within the codebase.

Notably, within the `NewUnilateralCloseSummary` method, we'll now _blank
out_ the `SingleTweak` value if the commitment is tweakless. This
indicates to callers the witness type they should map to, as the value
isn't needed at all any longer when sweeping a non-delay output.

We also update the signing+verification tests to also test that we're
able to properly generate a valid witness for the new tweakless
commitment format.
2019-09-25 18:25:43 -07:00
Olaoluwa Osuntokun
21133e1856
channeldb: define new channel type, SingleFunderTweakless
In this commit, we define a new channel type: SingleFunderTweakless.
We'll use this channel type to denote channels with commitments that
don't tweak the remote party's key in their non-delay output.
2019-09-25 18:25:40 -07:00
Johan T. Halseth
2a6ad6e634
channeldb+lnwallet: don't pass isRestoredChan to ChanSyncMsg
Since we have access to the internal state of the channel, we can
instead get it directly instead of passing it in as a parameter.
2019-09-25 14:04:44 +02:00
Johan T. Halseth
ac0e9b6016
channeldb/channel: add BroadcastedCommitment 2019-09-25 14:04:43 +02:00
Johan T. Halseth
02b2787e44
multi: make MarkCommitmentBroadcasted take closeTx 2019-09-25 14:04:43 +02:00
Johan T. Halseth
a810092e53
channeldb/channel: make putChanStatus take optional extra closures 2019-09-25 14:04:43 +02:00
Johan T. Halseth
eb1b84c0b4
channeldb+lnwallet: make ChanSyncMsg method on OpenChannel 2019-09-25 14:04:42 +02:00
Johan T. Halseth
0122dda88a
channeldb/channel: remove unused FullSync method
The exported FullSync method is only used by test code, so we remove it
and instead use SyncPending.
2019-09-25 14:04:42 +02:00
Olaoluwa Osuntokun
18f88cbd8d
Merge pull request #3440 from joostjager/buildroute
routing: add build route functionality
2019-09-24 20:24:24 -07:00
Joost Jager
339ff357d1
channeldb: invalidate channel signature cache on update 2019-09-23 13:07:04 +02:00
Joost Jager
a0e91b5d41
channeldb: add forward fee calculation function 2019-09-17 11:11:02 +02:00
Joost Jager
8c44cf4a22
channeldb: add ComputeFee function 2019-09-17 11:10:58 +02:00
Joost Jager
95502da7e8
Merge pull request #3414 from joostjager/report-accept-height
lnwire+htlcswitch: report htlc accept height
2019-09-16 13:49:29 +02:00
Joost Jager
d3e206ef95
invoices: return accept height in hodl event
This is a preparation for passing back the accept height in the
incorrect payment details failure message to the sender.
2019-09-16 10:10:16 +02:00
Joost Jager
49a20a87a2
channeldb+invoices: make htlc cancelation stricter
Previously it was possible to cancel a canceled htlc. This would
subtract the htlc amount from the invoice amount again.
2019-09-16 10:10:13 +02:00
Joost Jager
3ab4c749c6
channeldb: remove unsupported migrations
To reduce maintenance on migration code, we move to only supporting
upgrades from previous major releases.
2019-09-13 10:13:00 +02:00
Joost Jager
f1942a4c33
channeldb: require minimum db upgrade version 2019-09-13 10:12:58 +02:00
Lars Lehtonen
2f51ccd10b channeldb: Fix dropped error in migrations test
channeldb: Fix dropped error and wrap with context

channeldb: Fix empty error condition in waitingproof test

channeldb: Fix empty error condition in codec

channeldb: Wrap error in context
2019-09-12 15:49:21 +00:00
Joost Jager
93b40440c5
Merge pull request #3488 from joostjager/fix-channeldb-test
channeldb/test: make route comparison a pure function
2019-09-12 00:11:17 +02:00
Joost Jager
3d7de2ad39
multi: remove dead code 2019-09-10 17:21:59 +02:00
Joost Jager
62a9c2c3ac
channeldb/test: make route comparison a pure function
Previously the route to compare was modified in order for DeepEqual to
function properly. This created problems when tests were ran in
parallel.
2019-09-10 15:46:49 +02:00
Joost Jager
7f4af8f9ae
channeldb/test: make hop record comparison stricter 2019-09-10 15:46:48 +02:00
Olaoluwa Osuntokun
62dcc4b7d5
channeldb: ensure test for migration 9 is fully enclosed
In this commit, we ensure that the test for migration 9 uses the same
encoding/decoding functions as was present in the repo when the
migration was first added. Otherwise, the test will fail as it'll try to
use the decoding functions of master (migration 10 and onwards) rather
than the decoding function of migration 9.
2019-09-09 03:57:52 -07:00
chokoboko
ac617a0d01 channeldb: return errors in various missed places 2019-09-09 02:41:43 -07:00
Olaoluwa Osuntokun
754809406d
channeldb: fix migration bug due to interplay between migration #9 and #10
In this commit, we fix an issue that was recently introduced as a result
of migration #10. The new TLV format ended up modifying the
serialization functions called in `serializePaymentAttemptInfo`.
Migration #9, also used this `serializePaymentAttemptInfo` method to
serialize the _new_ (pre TLV, but new payment attempt structure) routes
into the database during its migration. However, migration #10 failed to
copy over the existing unmodified `serializePaymentAttemptInfo` method
into the legacy serialization for migration #9. As a result, once
migration #9 was run, the routes/payments were serialized using the
_new_ format, rather than the format used for v0.7.1. This then lead to
de-serialization either failing, or causing partial payment corruption
as migration #10 was expecting the "legacy" format (no TLV info).

We fix this issue by adding a new fully enclosed
`serializePaymentAttemptInfoMigration9`method that will be used for
migration #9. Note that our tests didn't catch this, as they test the
migration in isolation, rather than in series which is how users will
encounter the migrations.

Fixes #3463.
2019-09-09 02:16:13 -07:00
Joost Jager
d6d9ec6aa5
invoices: replay awareness
Previously the invoice registry wasn't aware of replayed htlcs. This was
dealt with by keeping the invoice accept/settle logic idempotent, so
that a replay wouldn't have an effect.

This mechanism has two limitations:

1. No accurate tracking of the total amount paid to an invoice. The total
amount couldn't just be increased with every htlc received, because it
could be a replay which would lead to counting the htlc amount multiple
times. Therefore the total amount was set to the amount of the first
htlc that was received, even though there may have been multiple htlcs
paying to the invoice.

2. Impossible to check htlc expiry consistently for hodl invoices. When
an htlc is new, its expiry needs to be checked against the invoice cltv
delta. But for a replay, that check must be skipped. The htlc was
accepted in time, the invoice was moved to the accepted state and a
replay some blocks later shouldn't lead to that htlc being cancelled.
Because the invoice registry couldn't recognize replays, it stopped
checking htlc expiry heights when the invoice reached the accepted
state. This prevents hold htlcs from being cancelled after a restart.
But unfortunately this also caused additional htlcs to be accepted on an
already accepted invoice without their expiry being checked.

In this commit, the invoice registry starts to persistently track htlcs
so that replays can be recognized. For replays, an htlc resolution
action is returned early. This fixes both limitations mentioned above.
2019-09-04 19:20:31 +02:00
Joost Jager
53eea09b63
channeldb: add now function
Needed for time control in unit tests.
2019-09-04 19:20:29 +02:00
Joost Jager
144856757d
channeldb+invoices: move invoice cancel logic into registry
This commit is a continuation of the centralization of invoice state
transition logic in the invoice registry.
2019-09-04 19:20:25 +02:00
Joost Jager
416bc8c68c
channeldb+invoices: move hold invoice settle logic into registry
This commit is a continuation of the centralization of invoice state
transition logic in the invoice registry.
2019-09-04 19:20:23 +02:00
Joost Jager
ad3522f1a6
channeldb+invoices: move invoice accept or settle logic into registry
As the logic around invoice mutations gets more complex, the friction
caused by having this logic split between invoice registry and channeldb
becomes more apparent. This commit brings a clearer separation of
concerns by centralizing the accept/settle logic in the invoice
registry.

The original AcceptOrSettle method is renamed to UpdateInvoice because
the update to perform is controlled by the callback.
2019-09-04 19:20:21 +02:00
Joost Jager
4105142c96
channeldb+invoices: add invoice htlcs
This commit adds a set of htlcs to the Invoice struct and
serializes/deserializes this set to/from disk. It is a preparation for
accurate invoice accounting across restarts of lnd.

A migration is added for the invoice htlcs.

In addition to these changes, separate final cltv delta and expiry
invoice fields are created and populated. Previously it was required
to decode this from the stored payment request. The reason to create
a combined commit is to prevent multiple migrations.
2019-09-04 19:20:17 +02:00
Joost Jager
061b34b924
channeldb: add int64 to codec 2019-09-04 19:20:15 +02:00
Joost Jager
762609a169
channeldb: fix suppressed error 2019-09-04 19:20:05 +02:00
Olaoluwa Osuntokun
441b15055b
channeldb: properly compare payment attempts in serialization test 2019-08-23 12:50:25 -07:00
Olaoluwa Osuntokun
c78e3aaa9d
channeldb: update route.Hop serialization to include new EOB related fields
We also include a migration for the existing routes stored on disk.
2019-08-22 18:53:04 -07:00
Johan T. Halseth
21baa7bf18
multi: fix linter errors 2019-08-07 10:53:10 +02:00
Olaoluwa Osuntokun
7767eecbb8 Merge pull request #3164 from joostjager/persistent-mc
routing: persistent mission control
2019-08-02 15:45:49 -07:00
Olaoluwa Osuntokun
69c9e2b732
Merge pull request #3278 from Crypt-iQ/bbolt_options_0707
channeldb: specify freelist bbolt options by default
2019-07-30 16:52:28 -07:00
Joost Jager
eb4e65e54f
channeldb: export route serialization 2019-07-29 13:42:57 +02:00
Olaoluwa Osuntokun
8c389d13f9
Merge pull request #3197 from breez/optimize_prune_zombie_channels
Optimize prune zombie channels
2019-07-18 20:56:56 -07:00
Conner Fromknecht
0223039fb4
channeldb/graph: add NumZombies method and tests 2019-07-16 16:32:27 -07:00
Roei Erez
7300f33fe2 graph: implement index for disabled channels.
This commit adds an index bucket, disabledEdgePolicyBucket, for those
ChannelEdgePolicy with disabled bit on.
The main purpose is to be able to iterate over these fast when prune is
needed without the need for iterating the whole graph.

The entry points for accessing this index are:
1. When updating ChannelEdgePolicy - insert an entry.
2. When deleting ChannelEdge - delete the associated entries.
3. When querying for disabled channels - implemented DisabledChannelIDs
function
2019-07-16 17:48:07 +03:00
nsa
61a1ab08fb routing+channeldb: findPath uses Vertex instead of LightningNode
This commit modifies the nodeWithDist struct to use a route.Vertex
instead of a *channeldb.LightningNode. This change, coupled with
the new ForEachNodeChannel function, allows the findPath Djikstra's
algorithm to cut down on database lookups since we no longer need
to call the FetchOtherNode function.
2019-07-12 06:08:01 -04:00
Olaoluwa Osuntokun
ba5fbb3c27
Merge pull request #3156 from joostjager/extended-fail
routerrpc: add more failure reasons and route hints
2019-07-08 19:12:03 -07:00
nsa
66d15c8e76 channeldb + wtdb: specify freelist bbolt options by default
This commit specifies two bbolt options when opening the underlying
channel and watchtower databases so that there is reduced heap
pressure in case the bbolt database has a lot of free pages in the
B+ tree.
2019-07-08 18:41:25 -04:00
Joost Jager
5ce04091d8
routing+routerrpc+channeldb: return route on invalid payment details 2019-07-04 09:27:16 +02:00
Joost Jager
ae46fb00cb
routing+channeldb: add more failure reasons 2019-07-04 09:27:12 +02:00
Johan T. Halseth
97dda4f8c8
channeldb/migrations: skip migration for empty outgoing payments bucket
Previously the migration would fail if the source node was not set in
the database. Since we know that the source node must have been set
before making any payments, we check whether there actually are any
payments to migrate, and return early if not.
2019-06-18 21:14:16 +02:00
Johan T. Halseth
f98f452528
channeldb: avoid modifying bucket during cursor traversal
This is not safe according to bbolt documentation.
2019-06-14 21:09:03 +02:00
Johan T. Halseth
b7a37728c0
channeldb/migrations: avoid modifying bucket during ForEach loop
bbolt documenation state that this is not safe and leads to undefined
behavior.
2019-06-14 21:09:02 +02:00
Joost Jager
18c025151a
invoices+channeldb: move invoice state check to invoiceregistry 2019-06-10 16:12:08 +02:00
Joost Jager
2644759924
channeldb: add todo for more detailed failure reasons 2019-06-05 12:41:55 +02:00
Joost Jager
f03533c67a
routerrpc: convert sendpayment to async
Modify the routerrpc SendPayment api to asynchronous. This allows
callers to pick up a payment after the rpc connection was lost or lnd
was restarted.
2019-06-05 12:41:53 +02:00
Joost Jager
eb2647e8fc
channeldb: add subscription to control tower
Allows other sub-systems to subscribe to payment success and fail
events.
2019-06-05 12:41:49 +02:00
Joost Jager
87d3207baf
channeldb+routing: move control tower interface to routing
This commit creates an empty shall for control tower in the routing
package. It is a preparation for adding event notification.
2019-06-05 12:41:47 +02:00
Johan T. Halseth
7cb25a5afb
channeldb/control_tower test: add TestPaymentControlDeleteNonInFlight
TestPaymentControlDeleteNonInFlight checks that calling DeletaPayments
only deletes payments from the database that are not in-flight.
2019-05-27 20:19:00 +02:00
Johan T. Halseth
589f0fcc5d
control_tower: make initial bucket bucket creation stricter 2019-05-27 20:18:59 +02:00
Johan T. Halseth
95b4828780
channeldb: derive PaymentStatus implicitly from stored info 2019-05-27 20:18:59 +02:00
Johan T. Halseth
d705b8a013
channeldb+multi: rename Grounded->Unknown, Completed->Succeeded 2019-05-27 20:18:59 +02:00
Johan T. Halseth
d78d3f50b9
channeldb: move deprecated code to migration_09_legacy_serialization.go
And unexport deprecated code.
2019-05-27 20:18:59 +02:00
Johan T. Halseth
1b788904f0
channeldb+router: record payment failure reason in db 2019-05-27 20:18:59 +02:00
Johan T. Halseth
677cb018c9
routing/control_tower: add FetchInFlightPayments 2019-05-27 20:18:58 +02:00
Johan T. Halseth
de1bf8a518
routing/router: persist payment state machine
This commit makes the router use the ControlTower to drive the payment
life cycle state machine, to keep track of active payments across
restarts.  This lets the router resume payments on startup, such that
their final results can be handled and stored when ready.
2019-05-27 20:18:58 +02:00
Johan T. Halseth
59c2557cc9
channeldb/control_tower test: test new payment behavior 2019-05-27 20:18:57 +02:00
Johan T. Halseth
1af1832ff7
channeldb/control_tower: add payment information during state changes
This commit gives a new responsibility to the control tower, letting it
populate the payment bucket structure as the payment goes through
its different stages.

The payment will transition states Grounded->InFlight->Success/Failed,
where the CreationInfo/AttemptInfo/Preimage must be set accordingly.

This will be the main driver for the router state machine.
2019-05-27 20:18:57 +02:00
Johan T. Halseth
6d80661bbb
channeldb/payments: add StatusFailed 2019-05-27 20:18:57 +02:00
Johan T. Halseth
bb4aadd16c
channeldb/control_tower: remove non-strict option
Since we have performed a migration, the db should be in a consistent
state, and we can remove the non-strict option.
2019-05-27 20:18:57 +02:00
Johan T. Halseth
b7189ba028
channeldb test: add TestOutgoingPaymentsMigration 2019-05-27 20:18:57 +02:00
Johan T. Halseth
693807cf6e
channeldb/control_tower: add FetchSentPayments 2019-05-27 20:18:56 +02:00
Johan T. Halseth
801521ed2d
channeldb/migration: add migration for new payment bucket structure
migrateOutgoingPayments moves the OutgoingPayments into a new bucket format
where they all reside in a top-level bucket indexed by the payment hash. In
this sub-bucket we store information relevant to this payment, such as the
payment status.

To avoid that the router resend payments that have the status InFlight (we
cannot resume these payments for pre-migration payments) we delete those
statuses, so only Completed payments remain in the new bucket structure.
2019-05-27 20:18:56 +02:00
Johan T. Halseth
178996f0d3
channeldb/payments+control_tower: split OutgoingPayments
This commit changes the format used to store payments within the
DB. Previously this was serialized as one continuous struct
OutgoingPayment, which also contained an Invoice struct we where only
using a few fields of. We now split it up into two simpler sub-structs
CreationInfo, AttemptInfo and PaymentPreimage.

We also want to associate the payments more closely with payment
statuses, so we move to this hierarchy:

There's one top-level bucket "sentPaymentsBucket" which contains a set
of sub-buckets indexed by a payment's payment hash. Each such sub-bucket
contains several fields:
paymentStatusKey -> the payment's status
paymentCreationInfoKey -> the payment's CreationInfo.
paymentAttemptInfoKey -> the payment's AttemptInfo.
paymentSettleInfoKey -> the payment's preimage (or zeroes for
non-settled payments)

The CreationInfo is information that is static during the whole payment
lifcycle. The attempt info is set each time a new payment attempt
(route+paymentID) is sent on the network. The preimage is information
only known when a payment succeeds.  It therefore makes sense to split
them.

We keep legacy serialization code for migration puproses.
2019-05-27 20:18:56 +02:00
Johan T. Halseth
f022810f8b
channeldb/codec: add privateKey serialization 2019-05-27 20:18:56 +02:00
Johan T. Halseth
2417f40532
channeldb: put payment status in new bucket
We move the payment status to a new bucket hierarchy. Old buckets and
fetch methods are kept around for migration purposes.
2019-05-27 20:18:56 +02:00
Johan T. Halseth
d027e10201
htlcswitch+channeldb: move control tower to channeldb 2019-05-27 20:18:56 +02:00
Conner Fromknecht
28bf49807e
watchtower/wtdb: add CSessionStatus field to ClientSession
This commit adds persisted status bit-field to ClientSessions, that can
be used to modify behavior of their handling in the client. Currently,
only a default CSessionActive status is defined. However, the intention
is that this could later be used to signal that a session is abandoned
without needing to perform a db migration to add the field. As we move
forward with testing, this will likely be useful if a session gets
borked and we need a simple method of the client to temporarily ignore
certain sessions.

The field may be useful in signaling other types of status changes,
though this was the primary motivation that warranted the addition.
2019-05-24 18:24:41 -07:00
Conner Fromknecht
2a904cb69f
watchtower/wtdb: add Encode/Decode methods to wtclient structs 2019-05-23 20:48:08 -07:00
Johan T. Halseth
15bed506b1
routing/route+multi: remove redundant TotalFees field
Instead get it on demand using method TotalFees().
2019-05-16 23:58:32 +02:00
Joost Jager
064e8492de
cnct+htlcswitch+invoices: move invoice parameter check out of link
This commit is the final step in making the link unaware of invoices. It
now purely offers the htlc to the invoice registry and follows
instructions from the invoice registry about how and when to respond to
the htlc.

The change also fixes a bug where upon restart, hodl htlcs were
subjected to the invoice minimum cltv delta requirement again. If the
block height has increased in the mean while, the htlc would be canceled
back.

Furthermore the invoice registry interaction is aligned between link and
contract resolvers.
2019-05-15 14:42:12 +02:00
AdamISZ
6e054886d8 Typo corrections in various comments 2019-05-10 17:15:54 +02:00
Olaoluwa Osuntokun
fb1819bf7f
Merge pull request #3064 from halseth/route-serialization
[Reliable payments] add routing.Route (de)serialization + test
2019-05-09 16:46:14 -07:00
Johan T. Halseth
21c989ffcc
channeldb/graph: don't nil curve of returned PubKey
Nilling the key would cause signature verification to crash if the
returned key was used.

A test to exercise the behavior is added.
2019-05-09 10:49:58 +02:00
Johan T. Halseth
89fd43ebcb
channeldb: add Route (de)serialization + test
We will store the routes used during payment attempts, so we need
serialization code.
2019-05-09 09:59:43 +02:00