Commit Graph

30 Commits

Author SHA1 Message Date
Andras Banki-Horvath
d89f51d1d0
multi: add reset closure to kvdb.Update
Similarly as with kvdb.View this commits adds a reset closure to the
kvdb.Update call in order to be able to reset external state if the
underlying db backend needs to retry the transaction.
2020-11-05 17:57:12 +01:00
Andras Banki-Horvath
2a358327f4
multi: add reset closure to kvdb.View
This commit adds a reset() closure to the kvdb.View function which will
be called before each retry (including the first) of the view
transaction. The reset() closure can be used to reset external state
(eg slices or maps) where the view closure puts intermediate results.
2020-11-05 17:57:12 +01:00
Andras Banki-Horvath
9c47392dfa
etcd: integrate the commitQueue to the STM commit loop
This commit integrates an externally passed commitQueue instance with
the STM to reduce retries for conflicting transactions.
2020-09-17 14:50:19 +02:00
Andras Banki-Horvath
6f3a45b75f
etcd: make embedded etcd context cancelable 2020-09-17 14:50:18 +02:00
Andras Banki-Horvath
281c0b9d92
etcd: add commit queue to effectively reduce transaction retries
This commit adds commitQueue which is a lightweight contention manager
for STM transactions. The queue attempts to queue up transactions that
conflict for sequential execution, while leaving all "unblocked"
transactons to run freely in parallel.
2020-09-17 14:50:18 +02:00
Andras Banki-Horvath
b4b5a9d7de
etcd: increase message and transaction limits for embedded etcd 2020-09-17 14:50:14 +02:00
Olaoluwa Osuntokun
e616903d4f
config: unify old and new config values for sync-freelist
In this commit, unify the old and new values for `sync-freelist`, and
also ensure that we don't break behavior for any users that're using the
_old_ value.

To do this, we first rename what was `--db.bolt.no-sync-freelist`, to
`--db.bolt.sync-freelist`. This gets rid of the negation on the config
level, and lets us override that value if the user is specifying the
legacy config option.

In the future, we'll deprecate the old config option, in favor of the
new DB scoped option.
2020-08-04 18:55:00 -07:00
Andras Banki-Horvath
5346ed8a5c
kvdb+etcd: assert on bucket/value key when putting value/bucket
This commit extends compatibility with the bbolt kvdb implementation,
which returns ErrIncompatibleValue in case of a bucket/value key
collision. Furthermore the commit also adds an extra precondition to the
transaction when a key doesn't exist. This is needed as we fix reads to
a snapshot revision and other writers may commit the key otherwise.
2020-08-01 00:29:01 +02:00
Andras Banki-Horvath
12a341ba59
etcd: remove the lock set concept
This commit removes the lock set which was used to only add bucket keys
to the tx predicate while also bumping their mod version.
This was useful to reduce the size of the compare set but wasn't useful
to reduce contention as top level buckets were always in the lock set.
2020-07-28 17:57:30 +02:00
Andras Banki-Horvath
63e9d6102f
kvdb+etcd: change flattened bucket key derivation algorithm
This commit changes the key derivation algo we use to emulate buckets
similar to bbolt. The issue with prefixing keys with either a bucket or
a value prefix is that the cursor couldn't effectively iterate trough
all keys in a bucket, as it skipped the bucket keys.
While there are multiple ways to fix that issue (eg. two pointers,
iterating value keys then bucket keys, etc), the cleanest is to instead
of prefixes in keys we use a postfix indicating whether a key is a
bucket or a value. This also simplifies all operations where we
(recursively) iterate a bucket and is equivalent with the prefixing key
derivation with the addition that bucket and value keys are now
continous.
2020-07-28 17:57:29 +02:00
Andras Banki-Horvath
9173958f9c
kvdb: s/hu/en/g 2020-07-28 17:57:28 +02:00
Andras Banki-Horvath
70a69ce990
kvdb: make etcd tests use testify require instead of assert 2020-07-28 17:57:24 +02:00
yyforyongyu
b01947b7dc lnd: fix go fmt 2020-07-08 16:46:51 +08:00
Andras Banki-Horvath
3e5e60cac4 lnd+kvdb: pass LND abort context to etcd 2020-06-03 17:44:44 +02:00
Andras Banki-Horvath
d3545830c9 kvdb+etcd: integrate the abort context to the STM retry loop
This commit extends the etcd.BackendConfig to also provide an abort
context and integrates it with the STM retry loop in order to be able
stop LND when conflicting transactions keep the loop running.
2020-06-03 17:44:44 +02:00
Andras Banki-Horvath
415de2f0c7 kvdb+etcd: remove retry goroutine from the STM
This commit removes the retry goroutine from the STM as the retry loop
is only running when the STM transaction is encapsulated in Update/View
whereas for self-standing transactions we use a different approach.
By removing the goroutine we won't catch panics thrown that are supposed
to be catched outside of the STM.
2020-06-03 17:44:44 +02:00
Conner Fromknecht
ef9068319b
multi: rename ReadCursor to RCursor 2020-05-26 18:21:08 -07:00
Conner Fromknecht
455ddfebdb
multi: rename: ReadBucket to RBucket 2020-05-26 18:21:08 -07:00
Conner Fromknecht
d0d2ca403d
multi: rename ReadTx to RTx 2020-05-26 18:20:37 -07:00
Andras Banki-Horvath
c3fcfd1530 etcd: add namespace support to separate key spaces
This commit extends etcd db with namespaces without additional storage
space requirements. This is simply done by instead of using an all zero
root bucket id, we use the sha256 hash of the name space as our root
bucket id.
2020-05-22 11:26:25 +02:00
Andras Banki-Horvath
85aee9b064 kvdb+lncfg: fully move etcd behind build tag
This commit separates all etcd related sources (sans a few stubs and
config) from the rest of the source tree and makes compilation conditional
depending on whether the kvdb_etcd build tag is specified.
2020-05-22 11:26:25 +02:00
Andras Banki-Horvath
3ef331e016 lncfg+etcd: add TLS config
This commit extends lncfg with etcd TLS config and passes these
parameters to the etcd client upon construction.
2020-05-22 11:26:25 +02:00
Andras Banki-Horvath
b54f8a1b97 kvdb+channeldb: create ExtendedBackend interface to allow rich tx logic
This commit adds the ExtendedBackend interface which is an extension to
the walletdb.DB interface. This paves the way to using etcd.db.View and
etcd.db.Update in the global View and Update functions without much code
rewrite.
2020-05-22 11:26:25 +02:00
Andras Banki-Horvath
02f9d91e49 kvdb+etcd: only lock on bucket path mod revision for reads and writes
This commit reduces the compare set size the STM will submit in
transactions by adding only the bucket keys along the bucket path to a
specific lock set. This lock set then used to filter the read set,
effectively removing all read only keys from the transaction predicate
that are not bucket keys.
By tracking if a read-write tx actually changes something, we can also
"bump" the mod revision of the bucket keys.
With this trick we essentially implement a read-write lock for our
bucket structure greatly reducing transaction processing time.
2020-05-22 11:26:25 +02:00
Andras Banki-Horvath
859a457e48 kvdb+etcd: extend kvdb and STM with stats
The commit itslef adds stats to the transactions such that we can see
how LND behaves with and etcd backend.
2020-05-22 11:26:25 +02:00
Andras Banki-Horvath
248a00f211 kvdb+etcd: enable channeldb testing with both bdb/etcd 2020-05-22 11:26:25 +02:00
Andras Banki-Horvath
3b7525659c channeldb+kvdb: add walletdb interface test
This commit adds walletdb interface test suite to the existing test set.
2020-05-22 11:26:25 +02:00
Andras Banki-Horvath
6a24a03cec channeldb+kvdb: walletdb/kvdb interface etcd implementation
This commit adds a full interface implementation of the walletdb/kvdb
interface with detailed tests.
2020-05-22 11:26:25 +02:00
Andras Banki-Horvath
137dee04e8 channeldb+kvdb: an extended STM on top of etcd clientv3
This commit adds an extended STM, similar to what available in etcd's
clientv3 module. This incarnation of said STM supports additional
features, like positioning in key intervals while taking into account
deletes and writes as well. This is a preliminary work to support all
features of the kvdb interface.
2020-05-22 11:26:24 +02:00
Olaoluwa Osuntokun
fc808ac538
channeldb/kvdb: create new package for an abstract kv store
In this commit, we create a new package `kvdb`, which is meant to serve
as the basis for any future database abstractions within `lnd`. Rather
than directly use the `walletdb` package (which we base off of), we
instead use a series of type-aliases to re-type the fundamental
types/interfaces of the `walletdb` package. This lets us type
`kvdb.RwTx` instead of `walletdb.ReadWriteTransaction` everywhere.
Additionally, our usage of type-aliases is also intended to create an
easy pathway in the future wherein we can gradually re-defined or
re-implement these types to wean off of the `walletdb` package.
2020-03-18 19:34:45 -07:00