Commit Graph

142 Commits

Author SHA1 Message Date
Johan T. Halseth
466c079bbe
channeldb/migration21: add body and tests for migration 21
In this commit, we migrate all wire messages in the database from the
`legacy` to the `current` encoding.

This affects the way we write the `CommitDiff` and `LogUpdates` struct
to disk. We also need to migrate the network results bucket in the
switch as it includes a wire message without a length prefix.
2021-02-24 17:31:55 +01:00
eugene
204b6c51cf
channeldb: MigrateOutpointIndex, store indexStatus in outpoint index
Adds an outpoint index that stores a tlv stream. Currently the stream
only contains the outpoint's indexStatus. This should cut down on
big bbolt transactions in several places throughout the codebase.
2020-12-10 17:37:59 -08:00
Conner Fromknecht
08ee754a6d
channeldb/db: properly reinit wallet during Wipe
Previously we wouldn't recreate some of the top level buckets that are
now considered expected with our migration logic. This bug was
preexisting, but never surfaced because the other TLB buckets were not
touched by this unit test.
2020-12-10 17:37:58 -08:00
Yong
582b164c46
kvdb: add timeout options for bbolt (#4787)
* mod: bump btcwallet version to accept db timeout

* btcwallet: add DBTimeOut in config

* kvdb: add database timeout option for bbolt

This commit adds a DBTimeout option in bbolt config. The relevant
functions walletdb.Open/Create are updated to use this config. In
addition, the bolt compacter also applies the new timeout option.

* channeldb: add DBTimeout in db options

This commit adds the DBTimeout option for channeldb. A new unit
test file is created to test the default options. In addition,
the params used in kvdb.Create inside channeldb_test is updated
with a DefaultDBTimeout value.

* contractcourt+routing: use DBTimeout in kvdb

This commit touches multiple test files in contractcourt and routing.
The call of function kvdb.Create and kvdb.Open are now updated with
the new param DBTimeout, using the default value kvdb.DefaultDBTimeout.

* lncfg: add DBTimeout option in db config

The DBTimeout option is added to db config. A new unit test is
added to check the default DB config is created as expected.

* migration: add DBTimeout param in kvdb.Create/kvdb.Open

* keychain: update tests to use DBTimeout param

* htlcswitch+chainreg: add DBTimeout option

* macaroons: support DBTimeout config in creation

This commit adds the DBTimeout during the creation of macaroons.db.
The usage of kvdb.Create and kvdb.Open in its tests are updated with
a timeout value using kvdb.DefaultDBTimeout.

* walletunlocker: add dbTimeout option in UnlockerService

This commit adds a new param, dbTimeout, during the creation of
UnlockerService. This param is then passed to wallet.NewLoader
inside various service calls, specifying a timeout value to be
used when opening the bbolt. In addition, the macaroonService
is also called with this dbTimeout param.

* watchtower/wtdb: add dbTimeout param during creation

This commit adds the dbTimeout param for the creation of both
watchtower.db and wtclient.db.

* multi: add db timeout param for walletdb.Create

This commit adds the db timeout param for the function call
walletdb.Create. It touches only the test files found in chainntnfs,
lnwallet, and routing.

* lnd: pass DBTimeout config to relevant services

This commit enables lnd to pass the DBTimeout config to the following
services/config/functions,
  - chainControlConfig
  - walletunlocker
  - wallet.NewLoader
  - macaroons
  - watchtower
In addition, the usage of wallet.Create is updated too.

* sample-config: add dbtimeout option
2020-12-07 15:31:49 -08:00
Conner Fromknecht
82a238317c
lncfg+itest: expose configurable batch-commit-interval
This will permit a greater degree of tuning or customization depending
on various hardware/environmental factors.
2020-11-25 16:45:25 -08:00
Oliver Gugger
f8907fdb47
multi: add AutoCompact option to bolt backend
With this commit we thread the new AutoCompact flags all the way through
to the bolt backend.
2020-11-13 10:19:49 +01:00
Andras Banki-Horvath
d89f51d1d0
multi: add reset closure to kvdb.Update
Similarly as with kvdb.View this commits adds a reset closure to the
kvdb.Update call in order to be able to reset external state if the
underlying db backend needs to retry the transaction.
2020-11-05 17:57:12 +01:00
Andras Banki-Horvath
2a358327f4
multi: add reset closure to kvdb.View
This commit adds a reset() closure to the kvdb.View function which will
be called before each retry (including the first) of the view
transaction. The reset() closure can be used to reset external state
(eg slices or maps) where the view closure puts intermediate results.
2020-11-05 17:57:12 +01:00
carla
a550ca3d64
multi: store peer flap rate on disk on best effort basis
Since we will use peer flap rate to determine how we rate limit, we
store this value on disk per peer per channel. This allows us to
restart with memory of our peers past behaviour, so we don't give badly
behaving peers have a fresh start on restart. Last flap timestamp is
stored with our flap count so that we can degrade this all time flap
count over time for peers that have not recently flapped.
2020-09-08 13:49:46 +02:00
Andras Banki-Horvath
cbce8e8872
channeldb: move makeTestDB out of test to make it available for other tests
This commit moves makeTestDB to db.go and exports it so that we'll be
able to use this function in other unit tests to make them testable with
etcd if needed.
2020-07-28 17:57:29 +02:00
carla
578af99618
channeldb: save resolver outcomes in close summaries top level bucket
Add a new top level bucket which holds closed channels nested by chain
hash which contains additional information about channel closes. We add
resolver resolutions under their own key so that we can extend the
bucket with additional information if required.
2020-07-07 19:49:36 +02:00
Wilmer Paulino
d23be59f53
channeldb: remove graph insertion of restored channels
This was initially done as there were a few assertions throughout the
codebase requiring a channel's policy to be known. Now that these have
been addressed, we no longer need to store restored channels in the
graph, as their policies where incomplete anyway.
2020-06-18 14:51:44 -07:00
carla
c8d11285f3
channeldb: index payments by sequence number
Add an entry to a payments index bucket which maps sequence number
to payment hash when we initiate payments. This allows for more
efficient paginated queries. We create the top level bucket in its
own migration so that we do not need to create it on the fly.

When we retry payments and provide them with a new sequence number, we
delete the index for their existing payment so that we do not have an
index that points to a non-existent payment.

If we delete a payment, we also delete its index entry. This prevents
us from looking up entries from indexes to payments that do not exist.
2020-06-10 12:49:06 +02:00
Conner Fromknecht
66652ec4bc
channeldb/db: lower-case ErrDryRunMigrationOk message 2020-05-27 22:42:22 -07:00
Conner Fromknecht
0f3ab775c7
Merge pull request #4285 from cfromknecht/pay-addr-index
channeldb: index payments by payment addr, use payment hash as fallback
2020-05-27 17:36:30 -07:00
Conner Fromknecht
cbf71b5452
channeldb+invoices: use payment addr as primary index 2020-05-26 18:55:27 -07:00
Conner Fromknecht
24cce7a6ec
channeldb: consolidate top-level bucket create/wipe 2020-05-26 18:55:27 -07:00
Conner Fromknecht
455ddfebdb
multi: rename: ReadBucket to RBucket 2020-05-26 18:21:08 -07:00
Conner Fromknecht
d0d2ca403d
multi: rename ReadTx to RTx 2020-05-26 18:20:37 -07:00
Andras Banki-Horvath
bce0597643 channeldb: silence linter 2020-05-22 11:26:25 +02:00
Andras Banki-Horvath
b54f8a1b97 kvdb+channeldb: create ExtendedBackend interface to allow rich tx logic
This commit adds the ExtendedBackend interface which is an extension to
the walletdb.DB interface. This paves the way to using etcd.db.View and
etcd.db.Update in the global View and Update functions without much code
rewrite.
2020-05-22 11:26:25 +02:00
Andras Banki-Horvath
248a00f211 kvdb+etcd: enable channeldb testing with both bdb/etcd 2020-05-22 11:26:25 +02:00
Conner Fromknecht
c775819372
channeldb/db: adds optional dry-run abort for migrations 2020-05-11 15:38:45 -07:00
Olaoluwa Osuntokun
f0911765af
channeldb: convert to uniformly use new kvdb abstractions
In this commit, we migrate all the code in `channeldb` to only reference
the new `kvdb` package rather than `bbolt` directly.

In many instances, we need to add two version to fetch a bucket as both
read and write when needed. As an example, we add a new
`fetchChanBucketRw` function. This function is identical to
`fetchChanBucket`, but it will be used to fetch the main channel bucket
for all _write_ transactions. We need a new method as you can pass a
write transaction where a read is accepted, but not the other way around
due to the stronger typing of the new `kvdb` package.
2020-03-18 19:34:49 -07:00
Olaoluwa Osuntokun
fc808ac538
channeldb/kvdb: create new package for an abstract kv store
In this commit, we create a new package `kvdb`, which is meant to serve
as the basis for any future database abstractions within `lnd`. Rather
than directly use the `walletdb` package (which we base off of), we
instead use a series of type-aliases to re-type the fundamental
types/interfaces of the `walletdb` package. This lets us type
`kvdb.RwTx` instead of `walletdb.ReadWriteTransaction` everywhere.
Additionally, our usage of type-aliases is also intended to create an
easy pathway in the future wherein we can gradually re-defined or
re-implement these types to wean off of the `walletdb` package.
2020-03-18 19:34:45 -07:00
Joost Jager
866623e84b
channeldb/migration13: migrate to mpp structure
This commit migrates the payments in the database to a new structure
that allows for multiple htlcs per payments. The migration introduces a
new sub-bucket that contains a list of htlcs and moves the old single
htlc into that.
2020-03-09 18:31:39 +01:00
carla
11d975bd13
channeldb: save channel status on channel close
Add an optional channel status CloseChannel which will be stored on the
hitsorical channel which is persisted at channel close. This status is
used to set the close initiator for channels that do not complete the
funding flow or we abandon. In follow up commits, this status will be
used to record force and breach closes. The value is written to the
historical channel bucket for diplay over rpc.
2020-02-21 13:33:53 +02:00
carla
8d632b8022
channeldb: add historical channel bucket lookup 2020-02-21 13:23:30 +02:00
carla
ed81c88239
channeldb: replace fetch channels booleans with optional filters
This changes replaces the pending an waiting booleans in fetchChannels
with optional filters which can be more flexibly used. This change
allows filtering of channels without having to reason about the matrix
of possible boolean combinations. A test is added to ensure that the
combinations of these filters act as expected.
2020-02-06 21:39:54 +02:00
Joost Jager
b600ecda86
channeldb: inject clock into database
Use our standard clock mock for database time queries.
2020-01-20 11:08:30 +01:00
Joost Jager
e234f88b82
channeldb: export now function
To allow mocking when testing other packages.
2019-12-04 14:51:31 +01:00
Conner Fromknecht
4c872c438b
channeldb: complete migration 12 for TLV invoices 2019-11-22 02:24:28 -08:00
Olaoluwa Osuntokun
f6614dd435
channeldb: add new AbandonChannel method 2019-11-20 17:42:49 -08:00
Joost Jager
6913cd64b6
channeldb: reference migrations in package
This commit removes the migrations from channeldb and references those
in the migrations_01_to_11 package. This creates a one-way dependency on
the migrations. Future changes to channeldb won't be able to break
migrations anymore.
2019-10-24 12:49:40 +02:00
Olaoluwa Osuntokun
e9d2ad5d73 Revert "channeldb: require minimum db upgrade version"
This reverts commit f1942a4c33dbea7144f756d9cd0139ab1fe8197f.
2019-10-12 14:30:26 -07:00
Olaoluwa Osuntokun
0386e0c297 Revert "channeldb: remove unsupported migrations"
This reverts commit 3ab4c749c6da4c2693b3978c0a0863787c9c3495.
2019-10-12 00:47:45 -07:00
Wilmer Paulino
fa96450db8
channeldb: add freelist sync option modifier 2019-10-04 12:19:41 -04:00
Joost Jager
3ab4c749c6
channeldb: remove unsupported migrations
To reduce maintenance on migration code, we move to only supporting
upgrades from previous major releases.
2019-09-13 10:13:00 +02:00
Joost Jager
f1942a4c33
channeldb: require minimum db upgrade version 2019-09-13 10:12:58 +02:00
Joost Jager
53eea09b63
channeldb: add now function
Needed for time control in unit tests.
2019-09-04 19:20:29 +02:00
Joost Jager
4105142c96
channeldb+invoices: add invoice htlcs
This commit adds a set of htlcs to the Invoice struct and
serializes/deserializes this set to/from disk. It is a preparation for
accurate invoice accounting across restarts of lnd.

A migration is added for the invoice htlcs.

In addition to these changes, separate final cltv delta and expiry
invoice fields are created and populated. Previously it was required
to decode this from the stored payment request. The reason to create
a combined commit is to prevent multiple migrations.
2019-09-04 19:20:17 +02:00
Olaoluwa Osuntokun
c78e3aaa9d
channeldb: update route.Hop serialization to include new EOB related fields
We also include a migration for the existing routes stored on disk.
2019-08-22 18:53:04 -07:00
nsa
66d15c8e76 channeldb + wtdb: specify freelist bbolt options by default
This commit specifies two bbolt options when opening the underlying
channel and watchtower databases so that there is reduced heap
pressure in case the bbolt database has a lot of free pages in the
B+ tree.
2019-07-08 18:41:25 -04:00
Johan T. Halseth
801521ed2d
channeldb/migration: add migration for new payment bucket structure
migrateOutgoingPayments moves the OutgoingPayments into a new bucket format
where they all reside in a top-level bucket indexed by the payment hash. In
this sub-bucket we store information relevant to this payment, such as the
payment status.

To avoid that the router resend payments that have the status InFlight (we
cannot resume these payments for pre-migration payments) we delete those
statuses, so only Completed payments remain in the new bucket structure.
2019-05-27 20:18:56 +02:00
Conner Fromknecht
5d98a94d60
channeldb: write chan updates through reject+channel cache 2019-04-01 16:34:51 -07:00
Conner Fromknecht
6d3e081f7b
channeldb: accept cache sizes in ChannelGraph 2019-04-01 16:33:36 -07:00
Conner Fromknecht
ae3a00a5da
channeldb/graph+db: integrate reject and channel caches 2019-04-01 16:32:52 -07:00
Conner Fromknecht
3c46ceec1d
channeldb/db: init one ChannelGraph per channeldb.DB 2019-04-01 16:25:30 -07:00
Conner Fromknecht
ac315fd051
channeldb/db: remove unused buffer pool 2019-04-01 16:25:30 -07:00
Olaoluwa Osuntokun
b8cbe3a1f8
channeldb: in RestoreChannelShells don't exit if edge already exists
During the restore process, it may be possible that we have already
heard about our prior edge from a node on the network (or our channel
peers). As a result, we shouldn't exit if this happens, and instead
should continue with the rest of the restoration process.
2019-03-28 17:54:06 -07:00