Commit Graph

27 Commits

Author SHA1 Message Date
Andras Banki-Horvath
f1831f0581
etcd: add support for user specified ports for testing 2021-01-08 15:39:11 +01:00
Andras Banki-Horvath
ce4ca86ca6
kvdb: fix kvdb.Batch to use state reset when using etcd backend 2021-01-08 15:39:11 +01:00
Andras Banki-Horvath
4a6af67241
etcd: add support for namespaces
This commit adds support for etcd namespaces. This is useful when
using a shared etcd database where separate users have access to
separate namespaces.
2020-12-16 18:25:38 +01:00
Oliver Gugger
c9ab713f59
etcd: increment port for embedded etcd server
Instead of letting the OS pick a random port for us in a way that wasn't
concurrency safe, we use an atomically incremented port to avoid
collisions.
2020-12-09 09:54:23 +01:00
Andras Banki-Horvath
adfd99ec08
etcd: allow readwrite bucket Delete with nil key (bbolt compatibility) 2020-11-05 17:57:13 +01:00
Andras Banki-Horvath
d89f51d1d0
multi: add reset closure to kvdb.Update
Similarly as with kvdb.View this commits adds a reset closure to the
kvdb.Update call in order to be able to reset external state if the
underlying db backend needs to retry the transaction.
2020-11-05 17:57:12 +01:00
Andras Banki-Horvath
2a358327f4
multi: add reset closure to kvdb.View
This commit adds a reset() closure to the kvdb.View function which will
be called before each retry (including the first) of the view
transaction. The reset() closure can be used to reset external state
(eg slices or maps) where the view closure puts intermediate results.
2020-11-05 17:57:12 +01:00
Andras Banki-Horvath
9c47392dfa
etcd: integrate the commitQueue to the STM commit loop
This commit integrates an externally passed commitQueue instance with
the STM to reduce retries for conflicting transactions.
2020-09-17 14:50:19 +02:00
Andras Banki-Horvath
6f3a45b75f
etcd: make embedded etcd context cancelable 2020-09-17 14:50:18 +02:00
Andras Banki-Horvath
281c0b9d92
etcd: add commit queue to effectively reduce transaction retries
This commit adds commitQueue which is a lightweight contention manager
for STM transactions. The queue attempts to queue up transactions that
conflict for sequential execution, while leaving all "unblocked"
transactons to run freely in parallel.
2020-09-17 14:50:18 +02:00
Andras Banki-Horvath
b4b5a9d7de
etcd: increase message and transaction limits for embedded etcd 2020-09-17 14:50:14 +02:00
Andras Banki-Horvath
5346ed8a5c
kvdb+etcd: assert on bucket/value key when putting value/bucket
This commit extends compatibility with the bbolt kvdb implementation,
which returns ErrIncompatibleValue in case of a bucket/value key
collision. Furthermore the commit also adds an extra precondition to the
transaction when a key doesn't exist. This is needed as we fix reads to
a snapshot revision and other writers may commit the key otherwise.
2020-08-01 00:29:01 +02:00
Andras Banki-Horvath
12a341ba59
etcd: remove the lock set concept
This commit removes the lock set which was used to only add bucket keys
to the tx predicate while also bumping their mod version.
This was useful to reduce the size of the compare set but wasn't useful
to reduce contention as top level buckets were always in the lock set.
2020-07-28 17:57:30 +02:00
Andras Banki-Horvath
63e9d6102f
kvdb+etcd: change flattened bucket key derivation algorithm
This commit changes the key derivation algo we use to emulate buckets
similar to bbolt. The issue with prefixing keys with either a bucket or
a value prefix is that the cursor couldn't effectively iterate trough
all keys in a bucket, as it skipped the bucket keys.
While there are multiple ways to fix that issue (eg. two pointers,
iterating value keys then bucket keys, etc), the cleanest is to instead
of prefixes in keys we use a postfix indicating whether a key is a
bucket or a value. This also simplifies all operations where we
(recursively) iterate a bucket and is equivalent with the prefixing key
derivation with the addition that bucket and value keys are now
continous.
2020-07-28 17:57:29 +02:00
Andras Banki-Horvath
9173958f9c
kvdb: s/hu/en/g 2020-07-28 17:57:28 +02:00
Andras Banki-Horvath
70a69ce990
kvdb: make etcd tests use testify require instead of assert 2020-07-28 17:57:24 +02:00
yyforyongyu
b01947b7dc lnd: fix go fmt 2020-07-08 16:46:51 +08:00
Andras Banki-Horvath
d3545830c9 kvdb+etcd: integrate the abort context to the STM retry loop
This commit extends the etcd.BackendConfig to also provide an abort
context and integrates it with the STM retry loop in order to be able
stop LND when conflicting transactions keep the loop running.
2020-06-03 17:44:44 +02:00
Andras Banki-Horvath
415de2f0c7 kvdb+etcd: remove retry goroutine from the STM
This commit removes the retry goroutine from the STM as the retry loop
is only running when the STM transaction is encapsulated in Update/View
whereas for self-standing transactions we use a different approach.
By removing the goroutine we won't catch panics thrown that are supposed
to be catched outside of the STM.
2020-06-03 17:44:44 +02:00
Andras Banki-Horvath
c3fcfd1530 etcd: add namespace support to separate key spaces
This commit extends etcd db with namespaces without additional storage
space requirements. This is simply done by instead of using an all zero
root bucket id, we use the sha256 hash of the name space as our root
bucket id.
2020-05-22 11:26:25 +02:00
Andras Banki-Horvath
85aee9b064 kvdb+lncfg: fully move etcd behind build tag
This commit separates all etcd related sources (sans a few stubs and
config) from the rest of the source tree and makes compilation conditional
depending on whether the kvdb_etcd build tag is specified.
2020-05-22 11:26:25 +02:00
Andras Banki-Horvath
3ef331e016 lncfg+etcd: add TLS config
This commit extends lncfg with etcd TLS config and passes these
parameters to the etcd client upon construction.
2020-05-22 11:26:25 +02:00
Andras Banki-Horvath
02f9d91e49 kvdb+etcd: only lock on bucket path mod revision for reads and writes
This commit reduces the compare set size the STM will submit in
transactions by adding only the bucket keys along the bucket path to a
specific lock set. This lock set then used to filter the read set,
effectively removing all read only keys from the transaction predicate
that are not bucket keys.
By tracking if a read-write tx actually changes something, we can also
"bump" the mod revision of the bucket keys.
With this trick we essentially implement a read-write lock for our
bucket structure greatly reducing transaction processing time.
2020-05-22 11:26:25 +02:00
Andras Banki-Horvath
859a457e48 kvdb+etcd: extend kvdb and STM with stats
The commit itslef adds stats to the transactions such that we can see
how LND behaves with and etcd backend.
2020-05-22 11:26:25 +02:00
Andras Banki-Horvath
3b7525659c channeldb+kvdb: add walletdb interface test
This commit adds walletdb interface test suite to the existing test set.
2020-05-22 11:26:25 +02:00
Andras Banki-Horvath
6a24a03cec channeldb+kvdb: walletdb/kvdb interface etcd implementation
This commit adds a full interface implementation of the walletdb/kvdb
interface with detailed tests.
2020-05-22 11:26:25 +02:00
Andras Banki-Horvath
137dee04e8 channeldb+kvdb: an extended STM on top of etcd clientv3
This commit adds an extended STM, similar to what available in etcd's
clientv3 module. This incarnation of said STM supports additional
features, like positioning in key intervals while taking into account
deletes and writes as well. This is a preliminary work to support all
features of the kvdb interface.
2020-05-22 11:26:24 +02:00