Commit Graph
134 Commits
Author SHA1 Message Date
carla a550ca3d64 multi: store peer flap rate on disk on best effort basis
Since we will use peer flap rate to determine how we rate limit, we
store this value on disk per peer per channel. This allows us to
restart with memory of our peers past behaviour, so we don't give badly
behaving peers have a fresh start on restart. Last flap timestamp is
stored with our flap count so that we can degrade this all time flap
count over time for peers that have not recently flapped.
2020-09-08 13:49:46 +02:00
Andras Banki-Horvath cbce8e8872 channeldb: move makeTestDB out of test to make it available for other tests
This commit moves makeTestDB to db.go and exports it so that we'll be
able to use this function in other unit tests to make them testable with
etcd if needed.
2020-07-28 17:57:29 +02:00
carla 578af99618 channeldb: save resolver outcomes in close summaries top level bucket
Add a new top level bucket which holds closed channels nested by chain
hash which contains additional information about channel closes. We add
resolver resolutions under their own key so that we can extend the
bucket with additional information if required.
2020-07-07 19:49:36 +02:00
Wilmer Paulino d23be59f53 channeldb: remove graph insertion of restored channels
This was initially done as there were a few assertions throughout the
codebase requiring a channel's policy to be known. Now that these have
been addressed, we no longer need to store restored channels in the
graph, as their policies where incomplete anyway.
2020-06-18 14:51:44 -07:00
carla c8d11285f3 channeldb: index payments by sequence number
Add an entry to a payments index bucket which maps sequence number
to payment hash when we initiate payments. This allows for more
efficient paginated queries. We create the top level bucket in its
own migration so that we do not need to create it on the fly.

When we retry payments and provide them with a new sequence number, we
delete the index for their existing payment so that we do not have an
index that points to a non-existent payment.

If we delete a payment, we also delete its index entry. This prevents
us from looking up entries from indexes to payments that do not exist.
2020-06-10 12:49:06 +02:00
Conner Fromknecht 66652ec4bc channeldb/db: lower-case ErrDryRunMigrationOk message 2020-05-27 22:42:22 -07:00
Conner Fromknecht 0f3ab775c7 Merge pull request #4285 from cfromknecht/pay-addr-index
channeldb: index payments by payment addr, use payment hash as fallback
2020-05-27 17:36:30 -07:00
Conner Fromknecht cbf71b5452 channeldb+invoices: use payment addr as primary index 2020-05-26 18:55:27 -07:00
Conner Fromknecht 24cce7a6ec channeldb: consolidate top-level bucket create/wipe 2020-05-26 18:55:27 -07:00
Conner Fromknecht 455ddfebdb multi: rename: ReadBucket to RBucket 2020-05-26 18:21:08 -07:00
Conner Fromknecht d0d2ca403d multi: rename ReadTx to RTx 2020-05-26 18:20:37 -07:00
Andras Banki-Horvath bce0597643 channeldb: silence linter 2020-05-22 11:26:25 +02:00
Andras Banki-Horvath b54f8a1b97 kvdb+channeldb: create ExtendedBackend interface to allow rich tx logic
This commit adds the ExtendedBackend interface which is an extension to
the walletdb.DB interface. This paves the way to using etcd.db.View and
etcd.db.Update in the global View and Update functions without much code
rewrite.
2020-05-22 11:26:25 +02:00
Andras Banki-Horvath 248a00f211 kvdb+etcd: enable channeldb testing with both bdb/etcd 2020-05-22 11:26:25 +02:00
Conner Fromknecht c775819372 channeldb/db: adds optional dry-run abort for migrations 2020-05-11 15:38:45 -07:00
Olaoluwa Osuntokun f0911765af channeldb: convert to uniformly use new kvdb abstractions
In this commit, we migrate all the code in `channeldb` to only reference
the new `kvdb` package rather than `bbolt` directly.

In many instances, we need to add two version to fetch a bucket as both
read and write when needed. As an example, we add a new
`fetchChanBucketRw` function. This function is identical to
`fetchChanBucket`, but it will be used to fetch the main channel bucket
for all _write_ transactions. We need a new method as you can pass a
write transaction where a read is accepted, but not the other way around
due to the stronger typing of the new `kvdb` package.
2020-03-18 19:34:49 -07:00
Olaoluwa Osuntokun fc808ac538 channeldb/kvdb: create new package for an abstract kv store
In this commit, we create a new package `kvdb`, which is meant to serve
as the basis for any future database abstractions within `lnd`. Rather
than directly use the `walletdb` package (which we base off of), we
instead use a series of type-aliases to re-type the fundamental
types/interfaces of the `walletdb` package. This lets us type
`kvdb.RwTx` instead of `walletdb.ReadWriteTransaction` everywhere.
Additionally, our usage of type-aliases is also intended to create an
easy pathway in the future wherein we can gradually re-defined or
re-implement these types to wean off of the `walletdb` package.
2020-03-18 19:34:45 -07:00
Joost Jager 866623e84b channeldb/migration13: migrate to mpp structure
This commit migrates the payments in the database to a new structure
that allows for multiple htlcs per payments. The migration introduces a
new sub-bucket that contains a list of htlcs and moves the old single
htlc into that.
2020-03-09 18:31:39 +01:00
carla 11d975bd13 channeldb: save channel status on channel close
Add an optional channel status CloseChannel which will be stored on the
hitsorical channel which is persisted at channel close. This status is
used to set the close initiator for channels that do not complete the
funding flow or we abandon. In follow up commits, this status will be
used to record force and breach closes. The value is written to the
historical channel bucket for diplay over rpc.
2020-02-21 13:33:53 +02:00
carla 8d632b8022 channeldb: add historical channel bucket lookup 2020-02-21 13:23:30 +02:00
carla ed81c88239 channeldb: replace fetch channels booleans with optional filters
This changes replaces the pending an waiting booleans in fetchChannels
with optional filters which can be more flexibly used. This change
allows filtering of channels without having to reason about the matrix
of possible boolean combinations. A test is added to ensure that the
combinations of these filters act as expected.
2020-02-06 21:39:54 +02:00
Joost Jager b600ecda86 channeldb: inject clock into database
Use our standard clock mock for database time queries.
2020-01-20 11:08:30 +01:00
Joost Jager e234f88b82 channeldb: export now function
To allow mocking when testing other packages.
2019-12-04 14:51:31 +01:00
Conner Fromknecht 4c872c438b channeldb: complete migration 12 for TLV invoices 2019-11-22 02:24:28 -08:00
Olaoluwa Osuntokun f6614dd435 channeldb: add new AbandonChannel method 2019-11-20 17:42:49 -08:00
Joost Jager 6913cd64b6 channeldb: reference migrations in package
This commit removes the migrations from channeldb and references those
in the migrations_01_to_11 package. This creates a one-way dependency on
the migrations. Future changes to channeldb won't be able to break
migrations anymore.
2019-10-24 12:49:40 +02:00
Olaoluwa Osuntokun e9d2ad5d73 Revert "channeldb: require minimum db upgrade version"
This reverts commit f1942a4c33.
2019-10-12 14:30:26 -07:00
Olaoluwa Osuntokun 0386e0c297 Revert "channeldb: remove unsupported migrations"
This reverts commit 3ab4c749c6.
2019-10-12 00:47:45 -07:00
Wilmer Paulino fa96450db8 channeldb: add freelist sync option modifier 2019-10-04 12:19:41 -04:00
Joost Jager 3ab4c749c6 channeldb: remove unsupported migrations
To reduce maintenance on migration code, we move to only supporting
upgrades from previous major releases.
2019-09-13 10:13:00 +02:00
Joost Jager f1942a4c33 channeldb: require minimum db upgrade version 2019-09-13 10:12:58 +02:00
Joost Jager 53eea09b63 channeldb: add now function
Needed for time control in unit tests.
2019-09-04 19:20:29 +02:00
Joost Jager 4105142c96 channeldb+invoices: add invoice htlcs
This commit adds a set of htlcs to the Invoice struct and
serializes/deserializes this set to/from disk. It is a preparation for
accurate invoice accounting across restarts of lnd.

A migration is added for the invoice htlcs.

In addition to these changes, separate final cltv delta and expiry
invoice fields are created and populated. Previously it was required
to decode this from the stored payment request. The reason to create
a combined commit is to prevent multiple migrations.
2019-09-04 19:20:17 +02:00
Olaoluwa Osuntokun c78e3aaa9d channeldb: update route.Hop serialization to include new EOB related fields
We also include a migration for the existing routes stored on disk.
2019-08-22 18:53:04 -07:00
nsa 66d15c8e76 channeldb + wtdb: specify freelist bbolt options by default
This commit specifies two bbolt options when opening the underlying
channel and watchtower databases so that there is reduced heap
pressure in case the bbolt database has a lot of free pages in the
B+ tree.
2019-07-08 18:41:25 -04:00
Johan T. Halseth 801521ed2d channeldb/migration: add migration for new payment bucket structure
migrateOutgoingPayments moves the OutgoingPayments into a new bucket format
where they all reside in a top-level bucket indexed by the payment hash. In
this sub-bucket we store information relevant to this payment, such as the
payment status.

To avoid that the router resend payments that have the status InFlight (we
cannot resume these payments for pre-migration payments) we delete those
statuses, so only Completed payments remain in the new bucket structure.
2019-05-27 20:18:56 +02:00
Conner Fromknecht 5d98a94d60 channeldb: write chan updates through reject+channel cache 2019-04-01 16:34:51 -07:00
Conner Fromknecht 6d3e081f7b channeldb: accept cache sizes in ChannelGraph 2019-04-01 16:33:36 -07:00
Conner Fromknecht ae3a00a5da channeldb/graph+db: integrate reject and channel caches 2019-04-01 16:32:52 -07:00
Conner Fromknecht 3c46ceec1d channeldb/db: init one ChannelGraph per channeldb.DB 2019-04-01 16:25:30 -07:00
Conner Fromknecht ac315fd051 channeldb/db: remove unused buffer pool 2019-04-01 16:25:30 -07:00
Olaoluwa Osuntokun b8cbe3a1f8 channeldb: in RestoreChannelShells don't exit if edge already exists
During the restore process, it may be possible that we have already
heard about our prior edge from a node on the network (or our channel
peers). As a result, we shouldn't exit if this happens, and instead
should continue with the rest of the restoration process.
2019-03-28 17:54:06 -07:00
Olaoluwa Osuntokun 19ef4bbcb9 channeldb: within AddrsForNode don't fail if no graph node is found
In this commit, we modify the `AddrsForNode` method to not fail if no
graph node is found. We do this as when the backup is being
restored/created, it's possible that we don't yet have a
NodeAnnouncement for that node.
2019-03-28 17:53:42 -07:00
Olaoluwa Osuntokun b93ff26265 channeldb: set restored chan status within RestoreChannelShells
In this commit, we move the location where we restore the channel status
to within the `RestoreChannelShells` method itself. Before this commit,
we attempted to use `ApplyChanStatus` which creates a DB transaction and
relies on a fully populated channel state, which in the restoration
case, we don't yet have.
2019-03-28 17:53:41 -07:00
Olaoluwa Osuntokun 13e7244d14 channeldb: also restore channel capacity in RestoreChannelShells 2019-03-28 17:53:40 -07:00
Wilmer Paulino b780dfacdb channeldb: add zombie edge index
In this commit, we add a zombie edge index to the database. This allows
us to quickly determine across restarts whether we're attempting to
process an edge we've previously deemed as zombie.
2019-03-27 13:06:12 -07:00
Wilmer Paulino 9febc9cc04 channeldb: add gossiper message store key migration
In this commit, we introduce a migration for the message store
sub-bucket that will migrate all keys within it to a new key format.
This new key format is composed of the peer's public key, followed by
the short channel ID, followed by the message type. This migration is
needed in order to provide backwards-compatibility with messages that
were previously stored before the introduction of the new key format.
2019-02-14 18:29:39 -08:00
Olaoluwa Osuntokun b2b57314fa channeldb: add new RestoreChannelShells method
In this commit, we add a new type (ChannelShell) along with a new
method, RestoreChannelShells which allows a caller to insert a series of
channel shells into the database. These channel shells will allow a
restored node to initiate the DLP protocol and recover their set of
existing channels.

When we insert a channel shell, we re-create the original link node, and
also add the outgoing edge to the channel graph. This way we can be sure
that upon start up, we attempt to connect to the remote peers, and that
the normal graph query commands will operate as expected.
2019-01-28 20:24:41 -08:00
Olaoluwa Osuntokun 7a88f580ea channeldb: add new FetchChannelMethod
In this commit, we add a new method, FetchChannel which is required in
order to implement the new chanbackup.LiveChannelSource interface.
2019-01-28 20:24:36 -08:00
Olaoluwa Osuntokun f57f40e767 channeldb: add prefix naming to ChannelStatus enum variables
In this commit, we add a prefix naming scheme to the ChannelStatus enum
variables. We do this as it enables outside callers to more easily
identify each individual enum variable as a part of the greater
enum-like type.
2019-01-28 20:24:32 -08:00