Commit Graph

833 Commits

Author SHA1 Message Date
Olaoluwa Osuntokun
0a1d592cf9
channeldb: return more detailed errors in DeleteInvoice
This PR was created in order to help to debug the failures in:
https://github.com/lightningnetwork/lnd/issues/5113

We add some more contextual errors so we can figure out where the
assumptions of the current scan to delete function in the invoice
registry is incorrect.
2021-03-17 14:22:46 -07:00
eugene
bb84f0ebc8
multi: store KeyLocator in OpenChannel, use ECDH
This commit adds a RevocationKeyLocator field to the OpenChannel
struct so that the SCB derivation doesn't have to brute-force the
sha chain root key and match the public key. ECDH derivation is now
used to derive the key instead of regular private key derivation a
la DerivePrivKey. The legacy can still be used to recover old
channels.
2021-03-05 12:49:18 -05:00
eugene
986e69c81b
channeldb: add KeyLocator Record 2021-03-05 12:49:16 -05:00
Conner Fromknecht
5f34880040
channeldb/invoices: continue cancel loop early 2021-03-03 16:28:02 -08:00
Conner Fromknecht
fc9af92dd9
channeldb: only accept/settle non-empty HTLC sets
Prior to AMP, there could only be one HTLC set. Now that there can be
multiple, we introduce the inherent risk that we might be persisting a
settle/accept of the wrong HTLC set. To migitigate this risk, we add a
check to assert that an invoice can only be transitioned into an
Accepted or Settled state if the specified HTLC set is non-empty, i.e.
has accepted HTLCs.

To do this check, we unroll the loops in UpdateInvoice to first process
any added or canceled HTLCs. This ensures that the set of accepted HTLCs
is up-to-date when we got to transition the invoice into a new state, at
which point we can accurately check that the HTLC set is non-empty.
Previously this sort of check wasn't possible because a newly added HTLC
wouldn't be populated on the invoice when going to call
updateInvoiceState.

Once the invoice-level transition checks have completed, we complete a
final pass to move the HTLCs into their final states and recompute the
amount paid.

With this refactor, it is now possible to make stronger assertions
about the invoice and the state transitions being made by the
invoiceregistry before those changes are ultimately persisted.
2021-03-03 16:28:02 -08:00
Conner Fromknecht
e1b0fe5e98
channeldb/invoices: update AMP invoice htlcs 2021-03-03 16:28:02 -08:00
Conner Fromknecht
fad25f3f26
channeldb/invoices: add invoices to set id index on update 2021-03-03 16:28:01 -08:00
Conner Fromknecht
7c0d1e0093
channeldb/invoice: map zero-value timestamps to 0
Mainly affects ResolveTime which can be 0 before its settled.
2021-03-03 16:26:43 -08:00
Conner Fromknecht
da049ebcf7
channeldb: ensure invoices can't be added with HTLCs 2021-03-03 16:26:43 -08:00
Conner Fromknecht
b2234703f6
invoices+channeldb: add HTLCSet method to invoice 2021-03-03 16:26:43 -08:00
Conner Fromknecht
7f05c9d3bb
channeldb/invoice: persist optional AMP fields on InvoiceHTLC 2021-03-03 16:26:43 -08:00
Olaoluwa Osuntokun
9a6bb19770
lnwire: prep wire messages for TLV extensions
Messages:
- UpdateFulfillHTLC
- UpdateFee
- UpdateFailMalformedHTLC
- UpdateFailHTLC
- UpdateAddHTLC
- Shutdown
- RevokeAndAck
- ReplyShortChanIDsEnd
- ReplyChannelRange
- QueryShortChanIDs
- QueryChannelRange
- NodeAnnouncement
- Init
- GossipTimestampRange
- FundingSigned
- FundingLocked
- FundingCreated
- CommitSig
- ClosingSigned
- ChannelUpdate
- ChannelReestablish
- ChannelAnnouncement
- AnnounceSignatures

lnwire: update quickcheck tests, use constant for Error

multi: update unit tests to pass deep equal assertions with messages

In this commit, we update a series of unit tests in the code base to now
pass due to the new wire message encode/decode logic. In many instances,
we'll now manually set the extra bytes to an empty byte slice to avoid
comparisons that fail due to one message having an empty byte slice and
the other having a nil pointer.
2021-02-24 17:31:55 +01:00
Johan T. Halseth
466c079bbe
channeldb/migration21: add body and tests for migration 21
In this commit, we migrate all wire messages in the database from the
`legacy` to the `current` encoding.

This affects the way we write the `CommitDiff` and `LogUpdates` struct
to disk. We also need to migrate the network results bucket in the
switch as it includes a wire message without a length prefix.
2021-02-24 17:31:55 +01:00
Johan T. Halseth
d4e90eae83
channeldb/migration21: add new serialization to package 'current'
We copy the new serialization togic to a new package 'current' and
export the methods we will need from the migration. This ensures that we
have isolation between the legacy and current serialization methods,
cleanly inidcating what type we are using during the migration.
2021-02-24 17:31:49 +01:00
Johan T. Halseth
db66fef6cc
channeldb+htlcswitch: write wire messages using length prefix
In this commit, we modify the way we write wire messages across the
entire database. We'll now ensure that we always write wire messages
with a length prefix. We update the `codec.go` file to always write a 2
byte length prefix, this affects the way we write the `CommitDiff` and
`LogUpdates` struct to disk, and the network results bucket in the
switch as it includes a wire message.
2021-02-24 17:15:27 +01:00
Johan T. Halseth
4133b4d04e
channeldb/migration21: copy over legacy types+codec for migration21
The legacy encoding depends on the lnwire21 version of lnwire, so it will
let us change lnwire after the migration. To make sure it is separated
from the new encoding, we add it to a new package 'legacy'.

We also put common types in a new package 'common', which will house
types that won't change during the migration, and can be used by both
legacy and current serialization code.
2021-02-24 17:15:21 +01:00
Johan T. Halseth
90d36dbdd4
channeldb: rename Encode/Decode to serializeLogUpdate/deserializeLogUpdate
To more easily use different version of it post-/pre-migration, we
rename the method and make it take the LogUpdate as an argument.
2021-02-24 14:34:57 +01:00
Johan T. Halseth
7569cca19b
channeldb/migration: copy current lnwire to migration dir
To avoid code changing underneath the static migrations, we copy the
lnwire dependency in its current form into the migration directory.

Ideally the migrations doesn't depend on any code that might change,
this takes us a step closer.
2021-02-24 14:34:57 +01:00
Olaoluwa Osuntokun
c6a8816043
channeldb: mark ApplyMigration as a test helper
With this change, errors from migrations will have the proper local line
number.
2021-02-24 14:34:57 +01:00
Johan T. Halseth
59f41e518d
channeldb: add failedHtlcsOnly option to DeletePayments 2021-02-22 12:50:25 +01:00
Johan T. Halseth
6eb42aec0a
channeldb/payments: use only Read bucket when iterating 2021-02-22 12:50:25 +01:00
Johan T. Halseth
7e3738d773
rpcserver+channeldb: delete only failed payments if requested 2021-02-22 12:50:25 +01:00
Johan T. Halseth
82fe6f5f59
channeldb/payments_test: handle that dupe payments will always have StatusSuccess
Legacy duplicate payments would only be migrated over if they had
succeeded. Alter the test to reflect this.
2021-02-22 12:50:24 +01:00
Johan T. Halseth
b5d6d7b4fd
channeldb: add SchedulerOp arg to graph update methods 2021-02-10 23:54:03 +01:00
eugene
9b09895bde
channeldb+lnwallet: lastWasRevokeKey to store last sent rev/sig 2021-02-03 16:43:04 -05:00
Conner Fromknecht
814b81bd25
Merge pull request #4902 from Crypt-iQ/lntestchannels_01072021
lntest/channels: introduce subpackage to deduplicate static structs
2021-02-01 14:12:08 -08:00
eugene
f8b98a1d3b
lntest/channels: introduce subpackage to deduplicate static structs 2021-01-25 14:04:39 -05:00
Oliver Gugger
02267565fe
multi: unify code blocks in READMEs 2021-01-22 09:14:11 +01:00
Andras Banki-Horvath
f1831f0581
etcd: add support for user specified ports for testing 2021-01-08 15:39:11 +01:00
Andras Banki-Horvath
ce4ca86ca6
kvdb: fix kvdb.Batch to use state reset when using etcd backend 2021-01-08 15:39:11 +01:00
Andras Banki-Horvath
4a6af67241
etcd: add support for namespaces
This commit adds support for etcd namespaces. This is useful when
using a shared etcd database where separate users have access to
separate namespaces.
2020-12-16 18:25:38 +01:00
Johan T. Halseth
d5cd9861d2
channeldb+lnwallet: define zero-fee channel type 2020-12-15 10:13:07 +01:00
Conner Fromknecht
d870bb5002
Merge pull request #4804 from wpaulino/proper-gossip-query-reply-chunk-splitting
discovery: adhere to proper channel chunk splitting for ReplyChannelRange
2020-12-10 18:10:07 -08:00
eugene
f4593f95f4
channeldb/migration20: TestMigrateOutpointIndex 2020-12-10 17:37:59 -08:00
eugene
204b6c51cf
channeldb: MigrateOutpointIndex, store indexStatus in outpoint index
Adds an outpoint index that stores a tlv stream. Currently the stream
only contains the outpoint's indexStatus. This should cut down on
big bbolt transactions in several places throughout the codebase.
2020-12-10 17:37:59 -08:00
Conner Fromknecht
08ee754a6d
channeldb/db: properly reinit wallet during Wipe
Previously we wouldn't recreate some of the top level buckets that are
now considered expected with our migration logic. This bug was
preexisting, but never surfaced because the other TLB buckets were not
touched by this unit test.
2020-12-10 17:37:58 -08:00
Oliver Gugger
c9ab713f59
etcd: increment port for embedded etcd server
Instead of letting the OS pick a random port for us in a way that wasn't
concurrency safe, we use an atomically incremented port to avoid
collisions.
2020-12-09 09:54:23 +01:00
Wilmer Paulino
a4f33ae63c
discovery: adhere to proper channel chunk splitting for ReplyChannelRange 2020-12-08 15:18:07 -08:00
Yong
582b164c46
kvdb: add timeout options for bbolt (#4787)
* mod: bump btcwallet version to accept db timeout

* btcwallet: add DBTimeOut in config

* kvdb: add database timeout option for bbolt

This commit adds a DBTimeout option in bbolt config. The relevant
functions walletdb.Open/Create are updated to use this config. In
addition, the bolt compacter also applies the new timeout option.

* channeldb: add DBTimeout in db options

This commit adds the DBTimeout option for channeldb. A new unit
test file is created to test the default options. In addition,
the params used in kvdb.Create inside channeldb_test is updated
with a DefaultDBTimeout value.

* contractcourt+routing: use DBTimeout in kvdb

This commit touches multiple test files in contractcourt and routing.
The call of function kvdb.Create and kvdb.Open are now updated with
the new param DBTimeout, using the default value kvdb.DefaultDBTimeout.

* lncfg: add DBTimeout option in db config

The DBTimeout option is added to db config. A new unit test is
added to check the default DB config is created as expected.

* migration: add DBTimeout param in kvdb.Create/kvdb.Open

* keychain: update tests to use DBTimeout param

* htlcswitch+chainreg: add DBTimeout option

* macaroons: support DBTimeout config in creation

This commit adds the DBTimeout during the creation of macaroons.db.
The usage of kvdb.Create and kvdb.Open in its tests are updated with
a timeout value using kvdb.DefaultDBTimeout.

* walletunlocker: add dbTimeout option in UnlockerService

This commit adds a new param, dbTimeout, during the creation of
UnlockerService. This param is then passed to wallet.NewLoader
inside various service calls, specifying a timeout value to be
used when opening the bbolt. In addition, the macaroonService
is also called with this dbTimeout param.

* watchtower/wtdb: add dbTimeout param during creation

This commit adds the dbTimeout param for the creation of both
watchtower.db and wtclient.db.

* multi: add db timeout param for walletdb.Create

This commit adds the db timeout param for the function call
walletdb.Create. It touches only the test files found in chainntnfs,
lnwallet, and routing.

* lnd: pass DBTimeout config to relevant services

This commit enables lnd to pass the DBTimeout config to the following
services/config/functions,
  - chainControlConfig
  - walletunlocker
  - wallet.NewLoader
  - macaroons
  - watchtower
In addition, the usage of wallet.Create is updated too.

* sample-config: add dbtimeout option
2020-12-07 15:31:49 -08:00
Johan T. Halseth
743ea7be74
contractcourt/chain_watcher: handleKnownRemoteState
Similar to what we did for the local state handling, we extract handling
all known remote states we can act on (breach, current, pending state)
into its own method.

Since we want to handle the case where we lost state (both in case of
local and remote close) last, we don't rely on the remote state number
to check which commit we are looking at, but match on TXIDs directly.
2020-12-03 08:01:08 +01:00
Olaoluwa Osuntokun
7e298f1434
Merge pull request #3367 from cfromknecht/batched-graph-updates
Batched graph updates
2020-11-25 18:40:40 -08:00
Conner Fromknecht
82a238317c
lncfg+itest: expose configurable batch-commit-interval
This will permit a greater degree of tuning or customization depending
on various hardware/environmental factors.
2020-11-25 16:45:25 -08:00
Wilmer Paulino
791ba3eb50
discovery: rate limit incoming channel updates
This change was largely motivated by an increase in high disk usage as a
result of channel update spam. With an in memory graph, this would've
gone mostly undetected except for the increased bandwidth usage, which
this doesn't aim to solve yet. To minimize the effects to disks, we
begin to rate limit channel updates in two ways. Keep alive updates,
those which only increase their timestamps to signal liveliness, are now
limited to one per lnd's rebroadcast interval (current default of 24H).
Non keep alive updates are now limited to one per block per direction.
2020-11-25 15:38:08 -08:00
Conner Fromknecht
edaa5d4308
channeldb/graph: add batched APIs for graph ingestion 2020-11-24 16:39:28 -08:00
Conner Fromknecht
9cfe08c879
channeldb/graph: split out cache updates for policy changes 2020-11-24 16:38:40 -08:00
Andras Banki-Horvath
511e817624
lncfg+channeldb: add config to be able to run lnd on embedded etcd 2020-11-18 15:34:43 +01:00
Oliver Gugger
a8ef4fc158
channeldb: store and evaluate last compaction time 2020-11-13 10:19:51 +01:00
Oliver Gugger
505be0d8bc
channeldb: implement compaction
This commit adds the compaction feature of the bbolt compact to our bolt
backend. This will try to open the DB in read-only mode and create a
compacted copy in a temporary file. Once the compaction was successful,
the temporary file and the old source file are swapped atomically.
2020-11-13 10:19:50 +01:00
Oliver Gugger
f8907fdb47
multi: add AutoCompact option to bolt backend
With this commit we thread the new AutoCompact flags all the way through
to the bolt backend.
2020-11-13 10:19:49 +01:00
Oliver Gugger
6131a53eb4
lncfg+channeldb: add autocompact flags to BoltConfig 2020-11-13 10:19:49 +01:00