Commit Graph

805 Commits

Author SHA1 Message Date
Andras Banki-Horvath
f1831f0581
etcd: add support for user specified ports for testing 2021-01-08 15:39:11 +01:00
Andras Banki-Horvath
ce4ca86ca6
kvdb: fix kvdb.Batch to use state reset when using etcd backend 2021-01-08 15:39:11 +01:00
Andras Banki-Horvath
4a6af67241
etcd: add support for namespaces
This commit adds support for etcd namespaces. This is useful when
using a shared etcd database where separate users have access to
separate namespaces.
2020-12-16 18:25:38 +01:00
Johan T. Halseth
d5cd9861d2
channeldb+lnwallet: define zero-fee channel type 2020-12-15 10:13:07 +01:00
Conner Fromknecht
d870bb5002
Merge pull request #4804 from wpaulino/proper-gossip-query-reply-chunk-splitting
discovery: adhere to proper channel chunk splitting for ReplyChannelRange
2020-12-10 18:10:07 -08:00
eugene
f4593f95f4
channeldb/migration20: TestMigrateOutpointIndex 2020-12-10 17:37:59 -08:00
eugene
204b6c51cf
channeldb: MigrateOutpointIndex, store indexStatus in outpoint index
Adds an outpoint index that stores a tlv stream. Currently the stream
only contains the outpoint's indexStatus. This should cut down on
big bbolt transactions in several places throughout the codebase.
2020-12-10 17:37:59 -08:00
Conner Fromknecht
08ee754a6d
channeldb/db: properly reinit wallet during Wipe
Previously we wouldn't recreate some of the top level buckets that are
now considered expected with our migration logic. This bug was
preexisting, but never surfaced because the other TLB buckets were not
touched by this unit test.
2020-12-10 17:37:58 -08:00
Oliver Gugger
c9ab713f59
etcd: increment port for embedded etcd server
Instead of letting the OS pick a random port for us in a way that wasn't
concurrency safe, we use an atomically incremented port to avoid
collisions.
2020-12-09 09:54:23 +01:00
Wilmer Paulino
a4f33ae63c
discovery: adhere to proper channel chunk splitting for ReplyChannelRange 2020-12-08 15:18:07 -08:00
Yong
582b164c46
kvdb: add timeout options for bbolt (#4787)
* mod: bump btcwallet version to accept db timeout

* btcwallet: add DBTimeOut in config

* kvdb: add database timeout option for bbolt

This commit adds a DBTimeout option in bbolt config. The relevant
functions walletdb.Open/Create are updated to use this config. In
addition, the bolt compacter also applies the new timeout option.

* channeldb: add DBTimeout in db options

This commit adds the DBTimeout option for channeldb. A new unit
test file is created to test the default options. In addition,
the params used in kvdb.Create inside channeldb_test is updated
with a DefaultDBTimeout value.

* contractcourt+routing: use DBTimeout in kvdb

This commit touches multiple test files in contractcourt and routing.
The call of function kvdb.Create and kvdb.Open are now updated with
the new param DBTimeout, using the default value kvdb.DefaultDBTimeout.

* lncfg: add DBTimeout option in db config

The DBTimeout option is added to db config. A new unit test is
added to check the default DB config is created as expected.

* migration: add DBTimeout param in kvdb.Create/kvdb.Open

* keychain: update tests to use DBTimeout param

* htlcswitch+chainreg: add DBTimeout option

* macaroons: support DBTimeout config in creation

This commit adds the DBTimeout during the creation of macaroons.db.
The usage of kvdb.Create and kvdb.Open in its tests are updated with
a timeout value using kvdb.DefaultDBTimeout.

* walletunlocker: add dbTimeout option in UnlockerService

This commit adds a new param, dbTimeout, during the creation of
UnlockerService. This param is then passed to wallet.NewLoader
inside various service calls, specifying a timeout value to be
used when opening the bbolt. In addition, the macaroonService
is also called with this dbTimeout param.

* watchtower/wtdb: add dbTimeout param during creation

This commit adds the dbTimeout param for the creation of both
watchtower.db and wtclient.db.

* multi: add db timeout param for walletdb.Create

This commit adds the db timeout param for the function call
walletdb.Create. It touches only the test files found in chainntnfs,
lnwallet, and routing.

* lnd: pass DBTimeout config to relevant services

This commit enables lnd to pass the DBTimeout config to the following
services/config/functions,
  - chainControlConfig
  - walletunlocker
  - wallet.NewLoader
  - macaroons
  - watchtower
In addition, the usage of wallet.Create is updated too.

* sample-config: add dbtimeout option
2020-12-07 15:31:49 -08:00
Johan T. Halseth
743ea7be74
contractcourt/chain_watcher: handleKnownRemoteState
Similar to what we did for the local state handling, we extract handling
all known remote states we can act on (breach, current, pending state)
into its own method.

Since we want to handle the case where we lost state (both in case of
local and remote close) last, we don't rely on the remote state number
to check which commit we are looking at, but match on TXIDs directly.
2020-12-03 08:01:08 +01:00
Olaoluwa Osuntokun
7e298f1434
Merge pull request #3367 from cfromknecht/batched-graph-updates
Batched graph updates
2020-11-25 18:40:40 -08:00
Conner Fromknecht
82a238317c
lncfg+itest: expose configurable batch-commit-interval
This will permit a greater degree of tuning or customization depending
on various hardware/environmental factors.
2020-11-25 16:45:25 -08:00
Wilmer Paulino
791ba3eb50
discovery: rate limit incoming channel updates
This change was largely motivated by an increase in high disk usage as a
result of channel update spam. With an in memory graph, this would've
gone mostly undetected except for the increased bandwidth usage, which
this doesn't aim to solve yet. To minimize the effects to disks, we
begin to rate limit channel updates in two ways. Keep alive updates,
those which only increase their timestamps to signal liveliness, are now
limited to one per lnd's rebroadcast interval (current default of 24H).
Non keep alive updates are now limited to one per block per direction.
2020-11-25 15:38:08 -08:00
Conner Fromknecht
edaa5d4308
channeldb/graph: add batched APIs for graph ingestion 2020-11-24 16:39:28 -08:00
Conner Fromknecht
9cfe08c879
channeldb/graph: split out cache updates for policy changes 2020-11-24 16:38:40 -08:00
Andras Banki-Horvath
511e817624
lncfg+channeldb: add config to be able to run lnd on embedded etcd 2020-11-18 15:34:43 +01:00
Oliver Gugger
a8ef4fc158
channeldb: store and evaluate last compaction time 2020-11-13 10:19:51 +01:00
Oliver Gugger
505be0d8bc
channeldb: implement compaction
This commit adds the compaction feature of the bbolt compact to our bolt
backend. This will try to open the DB in read-only mode and create a
compacted copy in a temporary file. Once the compaction was successful,
the temporary file and the old source file are swapped atomically.
2020-11-13 10:19:50 +01:00
Oliver Gugger
f8907fdb47
multi: add AutoCompact option to bolt backend
With this commit we thread the new AutoCompact flags all the way through
to the bolt backend.
2020-11-13 10:19:49 +01:00
Oliver Gugger
6131a53eb4
lncfg+channeldb: add autocompact flags to BoltConfig 2020-11-13 10:19:49 +01:00
Andras Banki-Horvath
adfd99ec08
etcd: allow readwrite bucket Delete with nil key (bbolt compatibility) 2020-11-05 17:57:13 +01:00
Andras Banki-Horvath
d89f51d1d0
multi: add reset closure to kvdb.Update
Similarly as with kvdb.View this commits adds a reset closure to the
kvdb.Update call in order to be able to reset external state if the
underlying db backend needs to retry the transaction.
2020-11-05 17:57:12 +01:00
Andras Banki-Horvath
2a358327f4
multi: add reset closure to kvdb.View
This commit adds a reset() closure to the kvdb.View function which will
be called before each retry (including the first) of the view
transaction. The reset() closure can be used to reset external state
(eg slices or maps) where the view closure puts intermediate results.
2020-11-05 17:57:12 +01:00
Andras Banki-Horvath
ffb27284df
channeldb: change walletdb dependency to kvdb 2020-11-05 17:57:08 +01:00
Johan T. Halseth
d1a118388d
payment_control: remove unused Attempts field 2020-10-06 10:41:25 +02:00
András Bánki-Horváth
111db801e7
Merge pull request #4457 from bhandras/etcd_tx_queue
etcd: STM transaction queue to effectively reduce retries for conflicting transactions
2020-09-17 17:35:59 +02:00
Andras Banki-Horvath
9c47392dfa
etcd: integrate the commitQueue to the STM commit loop
This commit integrates an externally passed commitQueue instance with
the STM to reduce retries for conflicting transactions.
2020-09-17 14:50:19 +02:00
Andras Banki-Horvath
6f3a45b75f
etcd: make embedded etcd context cancelable 2020-09-17 14:50:18 +02:00
Andras Banki-Horvath
281c0b9d92
etcd: add commit queue to effectively reduce transaction retries
This commit adds commitQueue which is a lightweight contention manager
for STM transactions. The queue attempts to queue up transactions that
conflict for sequential execution, while leaving all "unblocked"
transactons to run freely in parallel.
2020-09-17 14:50:18 +02:00
Andras Banki-Horvath
b4b5a9d7de
etcd: increase message and transaction limits for embedded etcd 2020-09-17 14:50:14 +02:00
Carla Kirk-Cohen
db0802bc8d
Merge pull request #4440 from carlaKC/4165-ratelimit
chanfitness: Rate limit in memory events based on peer flap rate
2020-09-14 13:11:07 +02:00
Johan T. Halseth
517fcd691a
channeldb: correct anchor comment 2020-09-11 11:10:58 +02:00
carla
a550ca3d64
multi: store peer flap rate on disk on best effort basis
Since we will use peer flap rate to determine how we rate limit, we
store this value on disk per peer per channel. This allows us to
restart with memory of our peers past behaviour, so we don't give badly
behaving peers have a fresh start on restart. Last flap timestamp is
stored with our flap count so that we can degrade this all time flap
count over time for peers that have not recently flapped.
2020-09-08 13:49:46 +02:00
eugene
62e19185f1 htlcswitch+channeldb: single tx for removing fwdpkgs
This commit changes RemoveFwdPkg to RemoveFwdPkgs so that a single
tx is used instead of N where N is the number of fwd pkgs to remove.
2020-08-31 20:52:51 -04:00
András Bánki-Horváth
c3821e5ad1
Merge pull request #4493 from bhandras/invoice_gc
invoices: garbage collect settled/canceled invoices
2020-08-24 08:34:58 +02:00
Johan T. Halseth
85b53c357f
htlcswitch: use kvdb.View for loading fwdpkgs
Not necessary to open a write tx.
2020-08-13 14:36:48 +02:00
Olaoluwa Osuntokun
fcb41dae37
Merge pull request #4475 from guggero/travis-windows
travis: add integration test on Windows
2020-08-06 18:15:02 -07:00
Andras Banki-Horvath
a5778c4673
channeldb: add DeleteInvoices call
This commit extends channeldb with the DeleteInvoices call which is when
passed a slice of delete references will attempt to delete the invoices
pointed to by the references and also clean up all our invoice indexes.
2020-08-05 12:24:15 +02:00
Andras Banki-Horvath
92f3b0a30c
channeldb+invoices: add ScanInvoices and integrate with InvoiceRegistry
This commit adds channeldb.ScanInvoices to scan through all invoices in
the database. The new call will also replace the already existing
channeldb.FetchAllInvoicesWithPaymentHash call in preparation to collect
invoices we'd like to delete and watch for expiry in one scan in later
commits.
2020-08-05 12:24:14 +02:00
Oliver Gugger
97c73706b5
channeldb: fix for Windows clock resolution
We use the event timestamp of a forwarding event as its primary storage
key. On systems with a bad clock resolution this can lead to collisions
of the events if some of the timestamps are identical. We fix this
problem by shifting the timestamps on the nanosecond level until only
unique values remain.
2020-08-05 09:15:03 +02:00
Olaoluwa Osuntokun
e616903d4f
config: unify old and new config values for sync-freelist
In this commit, unify the old and new values for `sync-freelist`, and
also ensure that we don't break behavior for any users that're using the
_old_ value.

To do this, we first rename what was `--db.bolt.no-sync-freelist`, to
`--db.bolt.sync-freelist`. This gets rid of the negation on the config
level, and lets us override that value if the user is specifying the
legacy config option.

In the future, we'll deprecate the old config option, in favor of the
new DB scoped option.
2020-08-04 18:55:00 -07:00
Conner Fromknecht
f144dea3da
Merge pull request #4411 from bhandras/test_channeldb
kvdb+etcd+tests: change etcd flattened bucket key derivation to make it compatible with bbolt
2020-07-31 17:37:13 -07:00
Andras Banki-Horvath
5346ed8a5c
kvdb+etcd: assert on bucket/value key when putting value/bucket
This commit extends compatibility with the bbolt kvdb implementation,
which returns ErrIncompatibleValue in case of a bucket/value key
collision. Furthermore the commit also adds an extra precondition to the
transaction when a key doesn't exist. This is needed as we fix reads to
a snapshot revision and other writers may commit the key otherwise.
2020-08-01 00:29:01 +02:00
nsa
e4188ba9c2 channeldb+lnwallet: store updates the peer should sign under new key
This fixes a long-standing force close bug. When we receive a
revocation, store the updates that the remote should sign next under
a new database key. Previously, these were not persisted which would
lead to force closure.
2020-07-29 19:03:44 -04:00
Andras Banki-Horvath
12a341ba59
etcd: remove the lock set concept
This commit removes the lock set which was used to only add bucket keys
to the tx predicate while also bumping their mod version.
This was useful to reduce the size of the compare set but wasn't useful
to reduce contention as top level buckets were always in the lock set.
2020-07-28 17:57:30 +02:00
Andras Banki-Horvath
cbce8e8872
channeldb: move makeTestDB out of test to make it available for other tests
This commit moves makeTestDB to db.go and exports it so that we'll be
able to use this function in other unit tests to make them testable with
etcd if needed.
2020-07-28 17:57:29 +02:00
Andras Banki-Horvath
63e9d6102f
kvdb+etcd: change flattened bucket key derivation algorithm
This commit changes the key derivation algo we use to emulate buckets
similar to bbolt. The issue with prefixing keys with either a bucket or
a value prefix is that the cursor couldn't effectively iterate trough
all keys in a bucket, as it skipped the bucket keys.
While there are multiple ways to fix that issue (eg. two pointers,
iterating value keys then bucket keys, etc), the cleanest is to instead
of prefixes in keys we use a postfix indicating whether a key is a
bucket or a value. This also simplifies all operations where we
(recursively) iterate a bucket and is equivalent with the prefixing key
derivation with the addition that bucket and value keys are now
continous.
2020-07-28 17:57:29 +02:00
Andras Banki-Horvath
9173958f9c
kvdb: s/hu/en/g 2020-07-28 17:57:28 +02:00