Commit Graph

770 Commits

Author SHA1 Message Date
Joost Jager
91d716be1f
Merge pull request #3770 from joostjager/mpp-prep-registry
channeldb+invoices: prepare invoice db for mpp
2019-12-05 11:38:56 +01:00
Joost Jager
00ef493aa0
channeldb: keep open channel data in historical channel bucket 2019-12-05 09:19:39 +01:00
Olaoluwa Osuntokun
183797f102
Merge pull request #3655 from carlaKC/fundingmgr-optionupfrontshutdown
Add Option Upfront Shutdown
2019-12-04 21:05:24 -08:00
Conner Fromknecht
46990c412c
channeldb/channel: allow storing empty closes
This is preparation for the subsequent commit, allowing us to fix a race
condition in the integration test assertions.
2019-12-04 14:25:29 -08:00
Conner Fromknecht
1c0dc98a7c
channeldb: differentiate force vs coop close 2019-12-04 14:25:28 -08:00
Joost Jager
6c07902a63
channeldb/test: add single htlc cancel test 2019-12-04 14:51:44 +01:00
Joost Jager
00d93ed87b
channeldb: stricter validation of invoice updates 2019-12-04 14:51:42 +01:00
Joost Jager
a4a3c41924
channeldb: split cancel and add htlc updates
Previously the cancel and add actions were combined in a single map.
Nil values implictly signaled cancel actions. This wasn't very obvious.
Furthermore this split prepares for processing the adds and cancels
separately, which is more efficient if there are already two maps.
2019-12-04 14:51:40 +01:00
Joost Jager
915867e90f
invoiceregistry: promote update closure to method
This commit moves the update code into its own function as a preparation
for extending the logic further for mpp.

In order to make this change cleanly, structured result codes are
introduced. This also prepares for a future htlc notifier rpc hook that
reports htlc settle decisions to external applications.

Furthermore the awkward use of errNoUpdate as a way to signal no update
is removed.
2019-12-04 14:51:36 +01:00
Joost Jager
e234f88b82
channeldb: export now function
To allow mocking when testing other packages.
2019-12-04 14:51:31 +01:00
Joost Jager
5d4ceca038
tlv: remove unused error return value
This commit also modifies a previous migration. Because the change is so
limited, we don't consider this a significant risk.
2019-12-04 13:27:29 +01:00
Joost Jager
883f9e5f9a
Merge pull request #3749 from joostjager/extended-routing-failures
routing: local balance check before path finding
2019-12-04 12:17:17 +01:00
Joost Jager
97344af8f3
routing: local balance check 2019-12-04 09:45:07 +01:00
carla
77222d8b69
channeldb: Add upfront shutdown script to OpenChannel
This commit adds fields for upfront shutdown scripts set
by the local and remote peer to the OpenChannel struct.
These values are optional, so they are added with their
own keys in the chanBucket in the DB.
2019-12-03 11:38:29 +02:00
Olaoluwa Osuntokun
6753a02439
channeldb: add new NoFundingTxBit modifier to ChannelType
In this commit, we add a new bit to the existing ChannelType bitfield.
If this bit is set, then it signals that we have the funding transaction
stored on disk. A future change will enable lnd to have the funding
transaction be constructed externally, allowing for things like funding
from a hardware wallet, or a channel created as a sub-branch within an
existing channel factory.
2019-12-02 17:11:16 -06:00
Olaoluwa Osuntokun
d59aba35a0 Merge branch 'refresh-chan-id' 2019-11-27 15:21:42 -06:00
Olaoluwa Osuntokun
5bdb0d3d66
channeldb+lntest: code style fixes 2019-11-27 15:21:28 -06:00
Roei Erez
8b3dd9415e channeldb: refresh channel state within RefreshShortChanID
Refresh channel memory state whenever the short channel id is refreshed.
This is to make the in-memory channel consistent with the disk data.

Fixes #3765.
2019-11-27 15:10:00 -06:00
Conner Fromknecht
3b253e05f6
multi: restructure invoice Terms field
This commit restructures an invoice's ContractTerms to better encompass
the restrictions placed on settling. For instance, the final ctlv delta
and invoice expiry are moved from the main invoice body (where
additional metadata is stored). Additionally, it moves the State field
outside of the terms since it is rather metadata about the invoice
instead of any terms offered to the sender in the payment request.
2019-11-22 02:25:02 -08:00
Conner Fromknecht
6cabea563e
channeldb/invoices: optimize deserializeHtlcs
Instead of allocating a byte slice to read in each htlc's raw TLV
stream, use a LimitReader to truncate the stream to the proper length.
2019-11-22 02:24:44 -08:00
Conner Fromknecht
4c872c438b
channeldb: complete migration 12 for TLV invoices 2019-11-22 02:24:28 -08:00
Conner Fromknecht
76682ad820
channeldb/migration12: add tlv invoice migration tests 2019-11-22 02:24:11 -08:00
Conner Fromknecht
4b358aa2ad
channeldb/migration12: write out invoices using TLV encoding 2019-11-22 02:23:59 -08:00
Conner Fromknecht
0f8048d336
channeldb/migration12: copy invoice deserialization 2019-11-22 02:23:47 -08:00
Olaoluwa Osuntokun
6f9fcfaccc
Merge pull request #3499 from cfromknecht/mpp-payments-rpc
channeldb+rpcserver: expose legacy payments as multi-path payments
2019-11-21 18:38:53 -08:00
Olaoluwa Osuntokun
f6614dd435
channeldb: add new AbandonChannel method 2019-11-20 17:42:49 -08:00
Conner Fromknecht
063f24f2ed
channeldb+routing: expose HTLCs in payment subscriptions
This commit modifies the FetchPayment method to return MPPayment structs
converted from the legacy on-disk format. This allows us to attach the
HTLCs to the events given to clients subscribing to the outcome of an
HTLC.

This commit also bubbles up to the routerrpc/router_server, by
populating HTLCAttempts in the response and extracting the legacy route
field from the HTLCAttempts.
2019-11-19 20:43:38 -08:00
Conner Fromknecht
01cecb1f27
multi: rename to FailureReasonPaymentDetails
Prior name is too long XD
2019-11-19 20:42:21 -08:00
Conner Fromknecht
77602451b8
channeldb: add MPPayment from FetchPayments 2019-11-19 20:41:08 -08:00
Conner Fromknecht
fbd599a2cb
multi: rename Payment.PaymentPreimage to Payment.Preimage
Preliminary step to exposing PaymentPreimage() as a method that can be
shared between legacy payments and mutli-path payments.
2019-11-19 20:40:29 -08:00
Conner Fromknecht
5e27b5022c
multi: remove LocalFeatures and GlobalFeatures 2019-11-08 05:32:00 -08:00
Conner Fromknecht
6d971e5113
routing/route/route: add optional MPP field to Hop
This commit also modifies the Router serialization to persist the MPP
struct when present, and properly restore it when loading from disk.
2019-11-04 14:11:28 -08:00
Olaoluwa Osuntokun
38e313a869
Merge pull request #3633 from joostjager/safe-migrations
channeldb: isolate migrations
2019-11-01 18:28:56 -07:00
Olaoluwa Osuntokun
a73ee28e46
multi: convert the existing channeldb.ChannelType uint8 into a bit field
In this commit, we convert the existing `channeldb.ChannelType` type
into a _bit field_. This doesn't require us to change the current
serialization or interpretation or the type as it is, since all the
current defined values us a distinct bit. This PR lays the ground work
for any future changes that may introduce new channel types (like anchor
outputs), and also any changes that may modify the existing invariants
around channels (if we're the initiator, we always have the funding
transaction).
2019-10-31 16:34:37 -07:00
Joost Jager
4486a06b1a
migration_01_to_11: remove version checking for migration tests 2019-10-31 11:25:07 +01:00
Joost Jager
60503d6c44
channeldb/migration_01_to_11: remove unused code 2019-10-31 07:18:36 +01:00
Joost Jager
f5191440c5
channeldb: initialize migrations logger 2019-10-30 08:56:18 +01:00
Joost Jager
43449ca7a7
channeldb/migration_01_to_11: add references to untested migrations 2019-10-29 11:49:33 +01:00
Joost Jager
6913cd64b6
channeldb: reference migrations in package
This commit removes the migrations from channeldb and references those
in the migrations_01_to_11 package. This creates a one-way dependency on
the migrations. Future changes to channeldb won't be able to break
migrations anymore.
2019-10-24 12:49:40 +02:00
Joost Jager
6e463c1634
channeldb: make copy for migrations
This commit is a direct copy of the complete channeldb package. It only
changes the package declaration at the top of every file. We make this
full copy so that review can be focused on the actual changes made.
Otherwise changes may drown in all the file moves.

Linting for the new package is disabled, as it contains lots of
pre-existing issues.
2019-10-24 12:49:38 +02:00
Olaoluwa Osuntokun
e9d2ad5d73 Revert "channeldb: require minimum db upgrade version"
This reverts commit f1942a4c33dbea7144f756d9cd0139ab1fe8197f.
2019-10-12 14:30:26 -07:00
Olaoluwa Osuntokun
0386e0c297 Revert "channeldb: remove unsupported migrations"
This reverts commit 3ab4c749c6da4c2693b3978c0a0863787c9c3495.
2019-10-12 00:47:45 -07:00
Wilmer Paulino
fa96450db8
channeldb: add freelist sync option modifier 2019-10-04 12:19:41 -04:00
Joost Jager
b58dbb2d70
multi: fix canceled spelling 2019-10-03 17:27:36 +02:00
Joost Jager
ac7c93f544
channeldb: fail migration for accepted hodl invoices 2019-09-28 08:30:33 +02:00
Joost Jager
31f72f6c7d
channeldb: extract before migration func 2019-09-28 08:30:31 +02:00
Joost Jager
f47de09850
channeldb: log error for migration test failures 2019-09-28 08:23:49 +02:00
Olaoluwa Osuntokun
b399203e71
lnwallet: update channel state machine to be aware of tweakless commits
In this commit, we update the channel state machine to be aware of
tweakless commits. In several areas, we'll now check the channel's type
to see if it's `SingleFunderTweakless`. If so, then we'll opt to use the
remote party's non-delay based point directly in the script, skipping
any additional cryptographic operations. Along the way we move the
`validateCommitmentSanity` method to be defined _before_ it's used as is
cutomary within the codebase.

Notably, within the `NewUnilateralCloseSummary` method, we'll now _blank
out_ the `SingleTweak` value if the commitment is tweakless. This
indicates to callers the witness type they should map to, as the value
isn't needed at all any longer when sweeping a non-delay output.

We also update the signing+verification tests to also test that we're
able to properly generate a valid witness for the new tweakless
commitment format.
2019-09-25 18:25:43 -07:00
Olaoluwa Osuntokun
21133e1856
channeldb: define new channel type, SingleFunderTweakless
In this commit, we define a new channel type: SingleFunderTweakless.
We'll use this channel type to denote channels with commitments that
don't tweak the remote party's key in their non-delay output.
2019-09-25 18:25:40 -07:00
Johan T. Halseth
2a6ad6e634
channeldb+lnwallet: don't pass isRestoredChan to ChanSyncMsg
Since we have access to the internal state of the channel, we can
instead get it directly instead of passing it in as a parameter.
2019-09-25 14:04:44 +02:00
Johan T. Halseth
ac0e9b6016
channeldb/channel: add BroadcastedCommitment 2019-09-25 14:04:43 +02:00
Johan T. Halseth
02b2787e44
multi: make MarkCommitmentBroadcasted take closeTx 2019-09-25 14:04:43 +02:00
Johan T. Halseth
a810092e53
channeldb/channel: make putChanStatus take optional extra closures 2019-09-25 14:04:43 +02:00
Johan T. Halseth
eb1b84c0b4
channeldb+lnwallet: make ChanSyncMsg method on OpenChannel 2019-09-25 14:04:42 +02:00
Johan T. Halseth
0122dda88a
channeldb/channel: remove unused FullSync method
The exported FullSync method is only used by test code, so we remove it
and instead use SyncPending.
2019-09-25 14:04:42 +02:00
Olaoluwa Osuntokun
18f88cbd8d
Merge pull request #3440 from joostjager/buildroute
routing: add build route functionality
2019-09-24 20:24:24 -07:00
Joost Jager
339ff357d1
channeldb: invalidate channel signature cache on update 2019-09-23 13:07:04 +02:00
Joost Jager
a0e91b5d41
channeldb: add forward fee calculation function 2019-09-17 11:11:02 +02:00
Joost Jager
8c44cf4a22
channeldb: add ComputeFee function 2019-09-17 11:10:58 +02:00
Joost Jager
95502da7e8
Merge pull request #3414 from joostjager/report-accept-height
lnwire+htlcswitch: report htlc accept height
2019-09-16 13:49:29 +02:00
Joost Jager
d3e206ef95
invoices: return accept height in hodl event
This is a preparation for passing back the accept height in the
incorrect payment details failure message to the sender.
2019-09-16 10:10:16 +02:00
Joost Jager
49a20a87a2
channeldb+invoices: make htlc cancelation stricter
Previously it was possible to cancel a canceled htlc. This would
subtract the htlc amount from the invoice amount again.
2019-09-16 10:10:13 +02:00
Joost Jager
3ab4c749c6
channeldb: remove unsupported migrations
To reduce maintenance on migration code, we move to only supporting
upgrades from previous major releases.
2019-09-13 10:13:00 +02:00
Joost Jager
f1942a4c33
channeldb: require minimum db upgrade version 2019-09-13 10:12:58 +02:00
Lars Lehtonen
2f51ccd10b channeldb: Fix dropped error in migrations test
channeldb: Fix dropped error and wrap with context

channeldb: Fix empty error condition in waitingproof test

channeldb: Fix empty error condition in codec

channeldb: Wrap error in context
2019-09-12 15:49:21 +00:00
Joost Jager
93b40440c5
Merge pull request #3488 from joostjager/fix-channeldb-test
channeldb/test: make route comparison a pure function
2019-09-12 00:11:17 +02:00
Joost Jager
3d7de2ad39
multi: remove dead code 2019-09-10 17:21:59 +02:00
Joost Jager
62a9c2c3ac
channeldb/test: make route comparison a pure function
Previously the route to compare was modified in order for DeepEqual to
function properly. This created problems when tests were ran in
parallel.
2019-09-10 15:46:49 +02:00
Joost Jager
7f4af8f9ae
channeldb/test: make hop record comparison stricter 2019-09-10 15:46:48 +02:00
Olaoluwa Osuntokun
62dcc4b7d5
channeldb: ensure test for migration 9 is fully enclosed
In this commit, we ensure that the test for migration 9 uses the same
encoding/decoding functions as was present in the repo when the
migration was first added. Otherwise, the test will fail as it'll try to
use the decoding functions of master (migration 10 and onwards) rather
than the decoding function of migration 9.
2019-09-09 03:57:52 -07:00
chokoboko
ac617a0d01 channeldb: return errors in various missed places 2019-09-09 02:41:43 -07:00
Olaoluwa Osuntokun
754809406d
channeldb: fix migration bug due to interplay between migration #9 and #10
In this commit, we fix an issue that was recently introduced as a result
of migration #10. The new TLV format ended up modifying the
serialization functions called in `serializePaymentAttemptInfo`.
Migration #9, also used this `serializePaymentAttemptInfo` method to
serialize the _new_ (pre TLV, but new payment attempt structure) routes
into the database during its migration. However, migration #10 failed to
copy over the existing unmodified `serializePaymentAttemptInfo` method
into the legacy serialization for migration #9. As a result, once
migration #9 was run, the routes/payments were serialized using the
_new_ format, rather than the format used for v0.7.1. This then lead to
de-serialization either failing, or causing partial payment corruption
as migration #10 was expecting the "legacy" format (no TLV info).

We fix this issue by adding a new fully enclosed
`serializePaymentAttemptInfoMigration9`method that will be used for
migration #9. Note that our tests didn't catch this, as they test the
migration in isolation, rather than in series which is how users will
encounter the migrations.

Fixes #3463.
2019-09-09 02:16:13 -07:00
Joost Jager
d6d9ec6aa5
invoices: replay awareness
Previously the invoice registry wasn't aware of replayed htlcs. This was
dealt with by keeping the invoice accept/settle logic idempotent, so
that a replay wouldn't have an effect.

This mechanism has two limitations:

1. No accurate tracking of the total amount paid to an invoice. The total
amount couldn't just be increased with every htlc received, because it
could be a replay which would lead to counting the htlc amount multiple
times. Therefore the total amount was set to the amount of the first
htlc that was received, even though there may have been multiple htlcs
paying to the invoice.

2. Impossible to check htlc expiry consistently for hodl invoices. When
an htlc is new, its expiry needs to be checked against the invoice cltv
delta. But for a replay, that check must be skipped. The htlc was
accepted in time, the invoice was moved to the accepted state and a
replay some blocks later shouldn't lead to that htlc being cancelled.
Because the invoice registry couldn't recognize replays, it stopped
checking htlc expiry heights when the invoice reached the accepted
state. This prevents hold htlcs from being cancelled after a restart.
But unfortunately this also caused additional htlcs to be accepted on an
already accepted invoice without their expiry being checked.

In this commit, the invoice registry starts to persistently track htlcs
so that replays can be recognized. For replays, an htlc resolution
action is returned early. This fixes both limitations mentioned above.
2019-09-04 19:20:31 +02:00
Joost Jager
53eea09b63
channeldb: add now function
Needed for time control in unit tests.
2019-09-04 19:20:29 +02:00
Joost Jager
144856757d
channeldb+invoices: move invoice cancel logic into registry
This commit is a continuation of the centralization of invoice state
transition logic in the invoice registry.
2019-09-04 19:20:25 +02:00
Joost Jager
416bc8c68c
channeldb+invoices: move hold invoice settle logic into registry
This commit is a continuation of the centralization of invoice state
transition logic in the invoice registry.
2019-09-04 19:20:23 +02:00
Joost Jager
ad3522f1a6
channeldb+invoices: move invoice accept or settle logic into registry
As the logic around invoice mutations gets more complex, the friction
caused by having this logic split between invoice registry and channeldb
becomes more apparent. This commit brings a clearer separation of
concerns by centralizing the accept/settle logic in the invoice
registry.

The original AcceptOrSettle method is renamed to UpdateInvoice because
the update to perform is controlled by the callback.
2019-09-04 19:20:21 +02:00
Joost Jager
4105142c96
channeldb+invoices: add invoice htlcs
This commit adds a set of htlcs to the Invoice struct and
serializes/deserializes this set to/from disk. It is a preparation for
accurate invoice accounting across restarts of lnd.

A migration is added for the invoice htlcs.

In addition to these changes, separate final cltv delta and expiry
invoice fields are created and populated. Previously it was required
to decode this from the stored payment request. The reason to create
a combined commit is to prevent multiple migrations.
2019-09-04 19:20:17 +02:00
Joost Jager
061b34b924
channeldb: add int64 to codec 2019-09-04 19:20:15 +02:00
Joost Jager
762609a169
channeldb: fix suppressed error 2019-09-04 19:20:05 +02:00
Olaoluwa Osuntokun
441b15055b
channeldb: properly compare payment attempts in serialization test 2019-08-23 12:50:25 -07:00
Olaoluwa Osuntokun
c78e3aaa9d
channeldb: update route.Hop serialization to include new EOB related fields
We also include a migration for the existing routes stored on disk.
2019-08-22 18:53:04 -07:00
Johan T. Halseth
21baa7bf18
multi: fix linter errors 2019-08-07 10:53:10 +02:00
Olaoluwa Osuntokun
7767eecbb8 Merge pull request #3164 from joostjager/persistent-mc
routing: persistent mission control
2019-08-02 15:45:49 -07:00
Olaoluwa Osuntokun
69c9e2b732
Merge pull request #3278 from Crypt-iQ/bbolt_options_0707
channeldb: specify freelist bbolt options by default
2019-07-30 16:52:28 -07:00
Joost Jager
eb4e65e54f
channeldb: export route serialization 2019-07-29 13:42:57 +02:00
Olaoluwa Osuntokun
8c389d13f9
Merge pull request #3197 from breez/optimize_prune_zombie_channels
Optimize prune zombie channels
2019-07-18 20:56:56 -07:00
Conner Fromknecht
0223039fb4
channeldb/graph: add NumZombies method and tests 2019-07-16 16:32:27 -07:00
Roei Erez
7300f33fe2 graph: implement index for disabled channels.
This commit adds an index bucket, disabledEdgePolicyBucket, for those
ChannelEdgePolicy with disabled bit on.
The main purpose is to be able to iterate over these fast when prune is
needed without the need for iterating the whole graph.

The entry points for accessing this index are:
1. When updating ChannelEdgePolicy - insert an entry.
2. When deleting ChannelEdge - delete the associated entries.
3. When querying for disabled channels - implemented DisabledChannelIDs
function
2019-07-16 17:48:07 +03:00
nsa
61a1ab08fb routing+channeldb: findPath uses Vertex instead of LightningNode
This commit modifies the nodeWithDist struct to use a route.Vertex
instead of a *channeldb.LightningNode. This change, coupled with
the new ForEachNodeChannel function, allows the findPath Djikstra's
algorithm to cut down on database lookups since we no longer need
to call the FetchOtherNode function.
2019-07-12 06:08:01 -04:00
Olaoluwa Osuntokun
ba5fbb3c27
Merge pull request #3156 from joostjager/extended-fail
routerrpc: add more failure reasons and route hints
2019-07-08 19:12:03 -07:00
nsa
66d15c8e76 channeldb + wtdb: specify freelist bbolt options by default
This commit specifies two bbolt options when opening the underlying
channel and watchtower databases so that there is reduced heap
pressure in case the bbolt database has a lot of free pages in the
B+ tree.
2019-07-08 18:41:25 -04:00
Joost Jager
5ce04091d8
routing+routerrpc+channeldb: return route on invalid payment details 2019-07-04 09:27:16 +02:00
Joost Jager
ae46fb00cb
routing+channeldb: add more failure reasons 2019-07-04 09:27:12 +02:00
Johan T. Halseth
97dda4f8c8
channeldb/migrations: skip migration for empty outgoing payments bucket
Previously the migration would fail if the source node was not set in
the database. Since we know that the source node must have been set
before making any payments, we check whether there actually are any
payments to migrate, and return early if not.
2019-06-18 21:14:16 +02:00
Johan T. Halseth
f98f452528
channeldb: avoid modifying bucket during cursor traversal
This is not safe according to bbolt documentation.
2019-06-14 21:09:03 +02:00
Johan T. Halseth
b7a37728c0
channeldb/migrations: avoid modifying bucket during ForEach loop
bbolt documenation state that this is not safe and leads to undefined
behavior.
2019-06-14 21:09:02 +02:00
Joost Jager
18c025151a
invoices+channeldb: move invoice state check to invoiceregistry 2019-06-10 16:12:08 +02:00
Joost Jager
2644759924
channeldb: add todo for more detailed failure reasons 2019-06-05 12:41:55 +02:00
Joost Jager
f03533c67a
routerrpc: convert sendpayment to async
Modify the routerrpc SendPayment api to asynchronous. This allows
callers to pick up a payment after the rpc connection was lost or lnd
was restarted.
2019-06-05 12:41:53 +02:00
Joost Jager
eb2647e8fc
channeldb: add subscription to control tower
Allows other sub-systems to subscribe to payment success and fail
events.
2019-06-05 12:41:49 +02:00
Joost Jager
87d3207baf
channeldb+routing: move control tower interface to routing
This commit creates an empty shall for control tower in the routing
package. It is a preparation for adding event notification.
2019-06-05 12:41:47 +02:00
Johan T. Halseth
7cb25a5afb
channeldb/control_tower test: add TestPaymentControlDeleteNonInFlight
TestPaymentControlDeleteNonInFlight checks that calling DeletaPayments
only deletes payments from the database that are not in-flight.
2019-05-27 20:19:00 +02:00
Johan T. Halseth
589f0fcc5d
control_tower: make initial bucket bucket creation stricter 2019-05-27 20:18:59 +02:00
Johan T. Halseth
95b4828780
channeldb: derive PaymentStatus implicitly from stored info 2019-05-27 20:18:59 +02:00
Johan T. Halseth
d705b8a013
channeldb+multi: rename Grounded->Unknown, Completed->Succeeded 2019-05-27 20:18:59 +02:00
Johan T. Halseth
d78d3f50b9
channeldb: move deprecated code to migration_09_legacy_serialization.go
And unexport deprecated code.
2019-05-27 20:18:59 +02:00
Johan T. Halseth
1b788904f0
channeldb+router: record payment failure reason in db 2019-05-27 20:18:59 +02:00
Johan T. Halseth
677cb018c9
routing/control_tower: add FetchInFlightPayments 2019-05-27 20:18:58 +02:00
Johan T. Halseth
de1bf8a518
routing/router: persist payment state machine
This commit makes the router use the ControlTower to drive the payment
life cycle state machine, to keep track of active payments across
restarts.  This lets the router resume payments on startup, such that
their final results can be handled and stored when ready.
2019-05-27 20:18:58 +02:00
Johan T. Halseth
59c2557cc9
channeldb/control_tower test: test new payment behavior 2019-05-27 20:18:57 +02:00
Johan T. Halseth
1af1832ff7
channeldb/control_tower: add payment information during state changes
This commit gives a new responsibility to the control tower, letting it
populate the payment bucket structure as the payment goes through
its different stages.

The payment will transition states Grounded->InFlight->Success/Failed,
where the CreationInfo/AttemptInfo/Preimage must be set accordingly.

This will be the main driver for the router state machine.
2019-05-27 20:18:57 +02:00
Johan T. Halseth
6d80661bbb
channeldb/payments: add StatusFailed 2019-05-27 20:18:57 +02:00
Johan T. Halseth
bb4aadd16c
channeldb/control_tower: remove non-strict option
Since we have performed a migration, the db should be in a consistent
state, and we can remove the non-strict option.
2019-05-27 20:18:57 +02:00
Johan T. Halseth
b7189ba028
channeldb test: add TestOutgoingPaymentsMigration 2019-05-27 20:18:57 +02:00
Johan T. Halseth
693807cf6e
channeldb/control_tower: add FetchSentPayments 2019-05-27 20:18:56 +02:00
Johan T. Halseth
801521ed2d
channeldb/migration: add migration for new payment bucket structure
migrateOutgoingPayments moves the OutgoingPayments into a new bucket format
where they all reside in a top-level bucket indexed by the payment hash. In
this sub-bucket we store information relevant to this payment, such as the
payment status.

To avoid that the router resend payments that have the status InFlight (we
cannot resume these payments for pre-migration payments) we delete those
statuses, so only Completed payments remain in the new bucket structure.
2019-05-27 20:18:56 +02:00
Johan T. Halseth
178996f0d3
channeldb/payments+control_tower: split OutgoingPayments
This commit changes the format used to store payments within the
DB. Previously this was serialized as one continuous struct
OutgoingPayment, which also contained an Invoice struct we where only
using a few fields of. We now split it up into two simpler sub-structs
CreationInfo, AttemptInfo and PaymentPreimage.

We also want to associate the payments more closely with payment
statuses, so we move to this hierarchy:

There's one top-level bucket "sentPaymentsBucket" which contains a set
of sub-buckets indexed by a payment's payment hash. Each such sub-bucket
contains several fields:
paymentStatusKey -> the payment's status
paymentCreationInfoKey -> the payment's CreationInfo.
paymentAttemptInfoKey -> the payment's AttemptInfo.
paymentSettleInfoKey -> the payment's preimage (or zeroes for
non-settled payments)

The CreationInfo is information that is static during the whole payment
lifcycle. The attempt info is set each time a new payment attempt
(route+paymentID) is sent on the network. The preimage is information
only known when a payment succeeds.  It therefore makes sense to split
them.

We keep legacy serialization code for migration puproses.
2019-05-27 20:18:56 +02:00
Johan T. Halseth
f022810f8b
channeldb/codec: add privateKey serialization 2019-05-27 20:18:56 +02:00
Johan T. Halseth
2417f40532
channeldb: put payment status in new bucket
We move the payment status to a new bucket hierarchy. Old buckets and
fetch methods are kept around for migration purposes.
2019-05-27 20:18:56 +02:00
Johan T. Halseth
d027e10201
htlcswitch+channeldb: move control tower to channeldb 2019-05-27 20:18:56 +02:00
Conner Fromknecht
28bf49807e
watchtower/wtdb: add CSessionStatus field to ClientSession
This commit adds persisted status bit-field to ClientSessions, that can
be used to modify behavior of their handling in the client. Currently,
only a default CSessionActive status is defined. However, the intention
is that this could later be used to signal that a session is abandoned
without needing to perform a db migration to add the field. As we move
forward with testing, this will likely be useful if a session gets
borked and we need a simple method of the client to temporarily ignore
certain sessions.

The field may be useful in signaling other types of status changes,
though this was the primary motivation that warranted the addition.
2019-05-24 18:24:41 -07:00
Conner Fromknecht
2a904cb69f
watchtower/wtdb: add Encode/Decode methods to wtclient structs 2019-05-23 20:48:08 -07:00
Johan T. Halseth
15bed506b1
routing/route+multi: remove redundant TotalFees field
Instead get it on demand using method TotalFees().
2019-05-16 23:58:32 +02:00
Joost Jager
064e8492de
cnct+htlcswitch+invoices: move invoice parameter check out of link
This commit is the final step in making the link unaware of invoices. It
now purely offers the htlc to the invoice registry and follows
instructions from the invoice registry about how and when to respond to
the htlc.

The change also fixes a bug where upon restart, hodl htlcs were
subjected to the invoice minimum cltv delta requirement again. If the
block height has increased in the mean while, the htlc would be canceled
back.

Furthermore the invoice registry interaction is aligned between link and
contract resolvers.
2019-05-15 14:42:12 +02:00
AdamISZ
6e054886d8 Typo corrections in various comments 2019-05-10 17:15:54 +02:00
Olaoluwa Osuntokun
fb1819bf7f
Merge pull request #3064 from halseth/route-serialization
[Reliable payments] add routing.Route (de)serialization + test
2019-05-09 16:46:14 -07:00
Johan T. Halseth
21c989ffcc
channeldb/graph: don't nil curve of returned PubKey
Nilling the key would cause signature verification to crash if the
returned key was used.

A test to exercise the behavior is added.
2019-05-09 10:49:58 +02:00
Johan T. Halseth
89fd43ebcb
channeldb: add Route (de)serialization + test
We will store the routes used during payment attempts, so we need
serialization code.
2019-05-09 09:59:43 +02:00
Conner Fromknecht
678ca9feff
channeldb/codec: add NewUnknownElementType constructor 2019-04-26 17:20:23 -07:00
Wilmer Paulino
0b0a9f4172
channeldb+routing: refactor DeleteChannelEdge to use ChannelID
In this commit, we refactor DeleteChannelEdge to use ChannelIDs rather
than ChannelPoints. We do this as the only use of DeleteChannelEdge is
when we are pruning zombie channels from our graph. When running under a
light client, we are unable to obtain the ChannelPoint of each edge due
to the expensive operations required to do so. As a stop-gap, we'll
resort towards using an edge's ChannelID instead, which is already
gossiped between nodes.
2019-04-18 21:57:41 -07:00
Wilmer Paulino
292defd6ba
channeldb: add IsDisabled method to ChannelEdgePolicy 2019-04-18 21:57:32 -07:00
Conner Fromknecht
f82b7c9bac
channeldb/graph: remove MarkEdgeZombie
This commit removes the MarkEdgeZombie method from channeldb. This
method is currently not used in any live code paths in production, and
is only used in unit tests. However, incorrect usage of this method
could result in an edge being present in both the zombie and channel
indexes, which deviates from any state we would expect to see in
production. Removing the method will help mitigate the potential for
writing incorrect unit tests in the future, by forcing zombie edges to
be created via the relevant, production APIs, e.g. DeleteChannelEdge.

The existing unit tests that use this method have been modified to use
the DeleteChannelEdge instead. No regressions were discovered in the
process.
2019-04-09 22:12:02 -07:00
Conner Fromknecht
4a755435e6
channeldb/graph: skip unknown edges in FetchChanInfos
This commit modifies FetchChanInfos to skip any channels that are not in
the graph at the time of the call. Currently the entire call will fail
if the edge is not found, which stalls a gossip sync in the following
scenario:

 1. Remote peer queries for a channel range
 2. We return the set of channel ids in that range
 3. A channel from that set is removed from the graph, e.g. via close.
 4. Remote peer queries for removed edge, causing the query to fail.

To remedy this, we will now skip any edges that are not known in the
database at the time of the query. This prevents the syncer state
machines from halting, which otherwise could only be resolved by
disconnecting and reconnecting.
2019-04-09 17:35:58 -07:00
Conner Fromknecht
e91bacd1bc
channeldb/graph: filter zombie channels in FilterKnownChanIDs
This commit modifies FilterKnownChanIDs to skip edges that
we ourselves have deemed zombies. This prevents us from requesting
the updates from them, as this wastes bandwidth and cpu cycles.
2019-04-05 13:01:08 -07:00
Conner Fromknecht
5d98a94d60
channeldb: write chan updates through reject+channel cache 2019-04-01 16:34:51 -07:00
Conner Fromknecht
6d3e081f7b
channeldb: accept cache sizes in ChannelGraph 2019-04-01 16:33:36 -07:00
Conner Fromknecht
baa968b1ff
channeldb/options: add Options w/ functional modifiers 2019-04-01 16:33:13 -07:00
Conner Fromknecht
ae3a00a5da
channeldb/graph+db: integrate reject and channel caches 2019-04-01 16:32:52 -07:00
Conner Fromknecht
b20a254faa
channeldb/channel_cache: add channelCache w/ randomized eviction 2019-04-01 16:25:31 -07:00
Conner Fromknecht
af0ea3590b
channeldb/reject_cache: add rejectCache w/ randomized eviction 2019-04-01 16:25:31 -07:00
Conner Fromknecht
3c46ceec1d
channeldb/db: init one ChannelGraph per channeldb.DB 2019-04-01 16:25:30 -07:00
Conner Fromknecht
ecbb78651a
channeldb/graph: add newChannelGraph constructor 2019-04-01 16:25:30 -07:00
Conner Fromknecht
ac315fd051
channeldb/db: remove unused buffer pool 2019-04-01 16:25:30 -07:00
Conner Fromknecht
2f927493c8
Revert "channeldb: convert invoice settle/cancel calls to use Batch"
This reverts commit da76c34418a40761fdc99e11200755661699f7eb.
2019-04-01 16:25:30 -07:00
Conner Fromknecht
e35f676b4c
Revert "channeldb: convert concurrent channel state machine calls to use Batch"
This reverts commit e8da6dd0b48843a48cecdcff894f48e8020d940e.
2019-04-01 16:25:30 -07:00
Conner Fromknecht
f6d93c8e5d
Revert "channeldb: convert all Update calls to use Batch"
This reverts commit 3555d3dbd431fe3ddf2660d4dc270b5a595fe517.
2019-04-01 16:25:26 -07:00
Olaoluwa Osuntokun
b8cbe3a1f8
channeldb: in RestoreChannelShells don't exit if edge already exists
During the restore process, it may be possible that we have already
heard about our prior edge from a node on the network (or our channel
peers). As a result, we shouldn't exit if this happens, and instead
should continue with the rest of the restoration process.
2019-03-28 17:54:06 -07:00
Olaoluwa Osuntokun
64b8facf0e
channeldb: don't read/write funding transactions if a restore channel 2019-03-28 17:53:43 -07:00
Olaoluwa Osuntokun
19ef4bbcb9
channeldb: within AddrsForNode don't fail if no graph node is found
In this commit, we modify the `AddrsForNode` method to not fail if no
graph node is found. We do this as when the backup is being
restored/created, it's possible that we don't yet have a
NodeAnnouncement for that node.
2019-03-28 17:53:42 -07:00
Olaoluwa Osuntokun
b93ff26265
channeldb: set restored chan status within RestoreChannelShells
In this commit, we move the location where we restore the channel status
to within the `RestoreChannelShells` method itself. Before this commit,
we attempted to use `ApplyChanStatus` which creates a DB transaction and
relies on a fully populated channel state, which in the restoration
case, we don't yet have.
2019-03-28 17:53:41 -07:00
Olaoluwa Osuntokun
13e7244d14
channeldb: also restore channel capacity in RestoreChannelShells 2019-03-28 17:53:40 -07:00
Wilmer Paulino
c82d73a826
channeldb+routing: extend edge lookup methods with zombie index check
In this commit, we extend the graph's FetchChannelEdgesByID and
HasChannelEdge methods to also check the zombie index whenever the edge
to be looked up doesn't exist within the edge index. We do this to
signal to callers that the edge is known, but only as a zombie, and the
only information that we have about the edge are the node public keys of
the two parties involved in the edge.

In the event that an edge does exist within the zombie index, we make
an additional check on edge policies to ensure they are not within the
router's pruning window, indicating that it is a fresh update.
2019-03-27 13:06:57 -07:00
Wilmer Paulino
e98f4d6d9d
channeldb: extend DeleteChannelEdge to mark edge as zombie
We mark the edges as zombies when pruning them to ensure we don't
attempt to reprocess them later on. This also applies to channels that
have been removed from the graph due to being stale.
2019-03-27 13:06:34 -07:00
Wilmer Paulino
b780dfacdb
channeldb: add zombie edge index
In this commit, we add a zombie edge index to the database. This allows
us to quickly determine across restarts whether we're attempting to
process an edge we've previously deemed as zombie.
2019-03-27 13:06:12 -07:00
Wilmer Paulino
a26a643273
channeldb: remove unused buckets 2019-03-27 13:05:48 -07:00
Olaoluwa Osuntokun
da76c34418
channeldb: convert invoice settle/cancel calls to use Batch 2019-03-19 17:11:17 -07:00
Olaoluwa Osuntokun
e8da6dd0b4
channeldb: convert concurrent channel state machine calls to use Batch 2019-03-19 17:11:14 -07:00
Olaoluwa Osuntokun
3555d3dbd4 channeldb: convert all Update calls to use Batch
In this commit, we convert all the `Update` calls which are serial, to
use `Batch` calls which are optimistically batched together for
concurrent writers. This should increase performance slightly during the
initial graph sync, and also updates at tip as we can coalesce more of
these individual transactions into a single transaction.
2019-03-19 17:10:46 -07:00
Joost Jager
32f2b047e8
htlcswitch: hodl invoice
This commit modifies the invoice registry to handle invoices for which
the preimage is not known yet (hodl invoices). In that case, the
resolution channel passed in from links and resolvers is stored until we
either learn the preimage or want to cancel the htlc.
2019-03-15 10:09:17 +01:00
Joost Jager
19f79613df
channeldb: store hold invoice 2019-03-15 10:08:55 +01:00
Joost Jager
8392f6d28f
lnrpc/invoicesrpc: remove lnrpc type from add invoice 2019-03-15 10:08:52 +01:00
Olaoluwa Osuntokun
33ad645f8c
lnwallet: update TestChanSyncFailure to pass with new borked update restriction
In this commit, we update the `TestChanSyncFailure` method to pass given
the new behavior around updating borked channel states. In order to do
this, we add a new method to allow the test to clear an existing channel
state. This method may be of independent use in other areas in the
codebase in the future as well.
2019-03-08 19:15:10 -08:00
Olaoluwa Osuntokun
032eacb796
channeldb: prevent mutating on-disk commitment state if channel is borked 2019-03-08 19:15:04 -08:00
Olaoluwa Osuntokun
cbe0bf6a22
Merge pull request #2501 from cfromknecht/batch-preimage-writes
htlcswitch: batch preimage writes/consistency fix
2019-02-21 17:00:00 -08:00
Conner Fromknecht
0a3e1cfbe5
channeldb+witness_beacon: use sha256 lookup+delete witness 2019-02-19 17:06:42 -08:00
Conner Fromknecht
e8b7f1fca3
channeldb/witness_cache: create AddSha256Witnesses helper + test 2019-02-19 17:05:30 -08:00
Conner Fromknecht
56b6becc48
channeldb/witness_cache_test: test batch preimage insertion 2019-02-19 17:05:17 -08:00
Conner Fromknecht
30f61b7630
multi: make AddPreimage variadic, optimistically compute key
In this commit, we modify the WitnessCache's
AddPreimage method to accept a variadic number
of preimages. This enables callers to batch
preimage writes in performance critical areas
of the codebase, e.g. the htlcswitch.

Additionally, we lift the computation of the
witnesses' keys outside of the db transaction.
This saves us from having to do hashing inside
and blocking other callers, and limits extraneous
blocking at the call site.
2019-02-19 17:05:04 -08:00
Wilmer Paulino
9febc9cc04
channeldb: add gossiper message store key migration
In this commit, we introduce a migration for the message store
sub-bucket that will migrate all keys within it to a new key format.
This new key format is composed of the peer's public key, followed by
the short channel ID, followed by the message type. This migration is
needed in order to provide backwards-compatibility with messages that
were previously stored before the introduction of the new key format.
2019-02-14 18:29:39 -08:00
Johan T. Halseth
2b012b06a0
channeldb/graph test: add test for update policy for unknown edge 2019-02-14 14:21:18 +01:00
Joost Jager
1b87fbfab2
invoices+htlcswitch+lnrpc: cancel invoice 2019-02-06 07:29:18 +01:00
Joost Jager
436dd41c77
channeldb: move idempotency up the call tree
As a preparation for subscribing to single invoices, InvoiceRegistry
needs to become aware of settling a settled invoice.
2019-02-01 09:42:56 +01:00
Joost Jager
bacd92418a
invoices: use lntypes.Hash and lntypes.Preimage
Previously chainhash.Hash was used, which converts to/from string in
reversed format. Payment hashes and preimages are supposed to be
non-reversed.
2019-02-01 09:42:29 +01:00
Olaoluwa Osuntokun
9a66ac396b
Merge pull request #2493 from halseth/tor-addr-sanity-check
Validate addresses before writing them to DB
2019-01-31 17:15:22 -08:00
Olaoluwa Osuntokun
480ec3bbca
channeldb: ensure restored channels can't be mutated 2019-01-28 20:24:48 -08:00
Olaoluwa Osuntokun
b2b57314fa
channeldb: add new RestoreChannelShells method
In this commit, we add a new type (ChannelShell) along with a new
method, RestoreChannelShells which allows a caller to insert a series of
channel shells into the database. These channel shells will allow a
restored node to initiate the DLP protocol and recover their set of
existing channels.

When we insert a channel shell, we re-create the original link node, and
also add the outgoing edge to the channel graph. This way we can be sure
that upon start up, we attempt to connect to the remote peers, and that
the normal graph query commands will operate as expected.
2019-01-28 20:24:41 -08:00
Olaoluwa Osuntokun
fa30af0475
channeldb: when fetching/inserting commits check for ChanStatusRestored
If the ChanStatusRestored flag is set, then we don't need to write or
read the set of commits for a channel as they won't exist. This will be
the case when we restore a channel from an SCB.
2019-01-28 20:24:40 -08:00
Olaoluwa Osuntokun
c656788b0b
channeldb: add new methods to allow adding a new edge+policy w/ existing db transaction 2019-01-28 20:24:38 -08:00
Olaoluwa Osuntokun
11c6887ffa
channeldb: refactor syncPending to expose new syncNewChannel function
The new syncNewChannel function will allow callers to insert a new
channel given the OpenChannel struct, and set of addresses for the
channel peer. This new method will also create a new LinkNode for the
peer if one doesn't already exist.
2019-01-28 20:24:37 -08:00
Olaoluwa Osuntokun
7a88f580ea
channeldb: add new FetchChannelMethod
In this commit, we add a new method, FetchChannel which is required in
order to implement the new chanbackup.LiveChannelSource interface.
2019-01-28 20:24:36 -08:00
Olaoluwa Osuntokun
aaf6456e12
channeldb: add HasChanStatus and ApplyChanState methods to OpenChannel
These methods allow callers to properly query if a channel is in a
particular flag, and also modify the channel status in a thread safe
manner.
2019-01-28 20:24:35 -08:00
Olaoluwa Osuntokun
a410262dda
channeldb: modify the String() method of ChannelStatus reflect all flags
In this commit, we modify the String() method of the ChannelStatus type
to reflect the fact that it's a flag set. With these new changes, we'll
now print the variable name of each assigned bit with a bar delimiting
them all.
2019-01-28 20:24:33 -08:00
Olaoluwa Osuntokun
f57f40e767
channeldb: add prefix naming to ChannelStatus enum variables
In this commit, we add a prefix naming scheme to the ChannelStatus enum
variables. We do this as it enables outside callers to more easily
identify each individual enum variable as a part of the greater
enum-like type.
2019-01-28 20:24:32 -08:00
Olaoluwa Osuntokun
f4054d2a66 channeldb: add new AddrsForNode method
In this commit, we add a new AddrsForNode method. This method will allow
a wrapper sturct to implement the new chanbackup.LiveChannelSource
method which is required to implement the full SCB feature set.
2019-01-28 20:24:22 -08:00
Johan T. Halseth
7d34ce9d08
lnwire+multi: define HasMaxHtlc helper on msgFlags 2019-01-22 08:42:30 +01:00
Johan T. Halseth
01e679786d
channeldb/graph_test: add TestEdgePolicyMissingMaxHtcl 2019-01-22 08:42:28 +01:00
Johan T. Halseth
6fcc76fc68
channeldb/channel: ignore ChannelEdgePolicy with missing max_htlc
If the max_htlc field is not found when fetching a ChannelEdgePolicy
from the DB, we treat this as an unknown policy.

This is done to ensure we won't propagate invalid data further. The data
will be overwritten with a valid one when we receive an update for this
channel.

It shouldn't be very common, but old data could be lingering in the DB
added before this field was validated.
2019-01-22 08:42:28 +01:00
Johan T. Halseth
8dd074ee57
channeldb/graph_test: assert MaxHTLC field correctness
Co-authored-by: Valentine Wallace <valentine.m.wallace@gmail.com>
2019-01-22 08:42:27 +01:00
Valentine Wallace
69d4bf051f
channeldb/graph: add max HTLC to ChannelEdgePolicy
Adding this field will allow us to persist an edge's
max HTLC to disk, thus preserving it between restarts.

Co-authored-by: Johan T. Halseth <johanth@gmail.com>
2019-01-22 08:42:27 +01:00
Johan T. Halseth
b9c5248915
channeldb/graph: extract ChannelEdgePolicy serialization 2019-01-22 08:42:27 +01:00
Johan T. Halseth
91c9e45031
channeldb/graph: check correct bucket for nilness 2019-01-22 08:42:27 +01:00
Valentine Wallace
0fd6004958
multi: partition lnwire.ChanUpdateFlag into ChannelFlags and MessageFlags
In this commit:

* we partition lnwire.ChanUpdateFlag into two (ChanUpdateChanFlags and
ChanUpdateMsgFlags), from a uint16 to a pair of uint8's

* we rename the ChannelUpdate.Flags to ChannelFlags and add an
additional MessageFlags field, which will be used to indicate the
presence of the optional field HtlcMaximumMsat within the ChannelUpdate.

* we partition ChannelEdgePolicy.Flags into message and channel flags.
This change corresponds to the partitioning of the ChannelUpdate's Flags
field into MessageFlags and ChannelFlags.

Co-authored-by: Johan T. Halseth <johanth@gmail.com>
2019-01-22 08:42:26 +01:00
Johan T. Halseth
8af2473644
channeldb/addr_test: add tests for invalid onion addresses 2019-01-17 14:16:15 +01:00
Johan T. Halseth
6a3e1423d2
channeldb/addr: sanity check onion address length before writing to db 2019-01-17 14:16:15 +01:00
Johan T. Halseth
e2e00f9dd2
channeldb/addr_test: add invalid TCP cases
These are invalid length IP addresses that earlier would not fail to
serialize.
2019-01-17 14:16:15 +01:00
Johan T. Halseth
02b7bb356a
channeldb/addr: validate ip before writing 2019-01-17 14:16:12 +01:00
Johan T. Halseth
6f605b2de2
channeldb/addr_test: inspect error string
Lets us match on more than static errors.
2019-01-17 14:14:24 +01:00
Wilmer Paulino
b951f06456
multi: move CsvDelay into ChannelConstraints 2019-01-11 16:58:15 -08:00
Olaoluwa Osuntokun
2e2d5fcf54
Merge pull request #2354 from joostjager/invoice-state
channeldb+lnrpc: invoice state
2019-01-04 14:48:17 -08:00