Commit Graph
27 Commits
Author SHA1 Message Date
Andras Banki-Horvath f1831f0581 etcd: add support for user specified ports for testing 2021-01-08 15:39:11 +01:00
Andras Banki-Horvath ce4ca86ca6 kvdb: fix kvdb.Batch to use state reset when using etcd backend 2021-01-08 15:39:11 +01:00
Andras Banki-Horvath 4a6af67241 etcd: add support for namespaces
This commit adds support for etcd namespaces. This is useful when
using a shared etcd database where separate users have access to
separate namespaces.
2020-12-16 18:25:38 +01:00
Oliver Gugger c9ab713f59 etcd: increment port for embedded etcd server
Instead of letting the OS pick a random port for us in a way that wasn't
concurrency safe, we use an atomically incremented port to avoid
collisions.
2020-12-09 09:54:23 +01:00
Andras Banki-Horvath adfd99ec08 etcd: allow readwrite bucket Delete with nil key (bbolt compatibility) 2020-11-05 17:57:13 +01:00
Andras Banki-Horvath d89f51d1d0 multi: add reset closure to kvdb.Update
Similarly as with kvdb.View this commits adds a reset closure to the
kvdb.Update call in order to be able to reset external state if the
underlying db backend needs to retry the transaction.
2020-11-05 17:57:12 +01:00
Andras Banki-Horvath 2a358327f4 multi: add reset closure to kvdb.View
This commit adds a reset() closure to the kvdb.View function which will
be called before each retry (including the first) of the view
transaction. The reset() closure can be used to reset external state
(eg slices or maps) where the view closure puts intermediate results.
2020-11-05 17:57:12 +01:00
Andras Banki-Horvath 9c47392dfa etcd: integrate the commitQueue to the STM commit loop
This commit integrates an externally passed commitQueue instance with
the STM to reduce retries for conflicting transactions.
2020-09-17 14:50:19 +02:00
Andras Banki-Horvath 6f3a45b75f etcd: make embedded etcd context cancelable 2020-09-17 14:50:18 +02:00
Andras Banki-Horvath 281c0b9d92 etcd: add commit queue to effectively reduce transaction retries
This commit adds commitQueue which is a lightweight contention manager
for STM transactions. The queue attempts to queue up transactions that
conflict for sequential execution, while leaving all "unblocked"
transactons to run freely in parallel.
2020-09-17 14:50:18 +02:00
Andras Banki-Horvath b4b5a9d7de etcd: increase message and transaction limits for embedded etcd 2020-09-17 14:50:14 +02:00
Andras Banki-Horvath 5346ed8a5c kvdb+etcd: assert on bucket/value key when putting value/bucket
This commit extends compatibility with the bbolt kvdb implementation,
which returns ErrIncompatibleValue in case of a bucket/value key
collision. Furthermore the commit also adds an extra precondition to the
transaction when a key doesn't exist. This is needed as we fix reads to
a snapshot revision and other writers may commit the key otherwise.
2020-08-01 00:29:01 +02:00
Andras Banki-Horvath 12a341ba59 etcd: remove the lock set concept
This commit removes the lock set which was used to only add bucket keys
to the tx predicate while also bumping their mod version.
This was useful to reduce the size of the compare set but wasn't useful
to reduce contention as top level buckets were always in the lock set.
2020-07-28 17:57:30 +02:00
Andras Banki-Horvath 63e9d6102f kvdb+etcd: change flattened bucket key derivation algorithm
This commit changes the key derivation algo we use to emulate buckets
similar to bbolt. The issue with prefixing keys with either a bucket or
a value prefix is that the cursor couldn't effectively iterate trough
all keys in a bucket, as it skipped the bucket keys.
While there are multiple ways to fix that issue (eg. two pointers,
iterating value keys then bucket keys, etc), the cleanest is to instead
of prefixes in keys we use a postfix indicating whether a key is a
bucket or a value. This also simplifies all operations where we
(recursively) iterate a bucket and is equivalent with the prefixing key
derivation with the addition that bucket and value keys are now
continous.
2020-07-28 17:57:29 +02:00
Andras Banki-Horvath 9173958f9c kvdb: s/hu/en/g 2020-07-28 17:57:28 +02:00
Andras Banki-Horvath 70a69ce990 kvdb: make etcd tests use testify require instead of assert 2020-07-28 17:57:24 +02:00
yyforyongyu b01947b7dc lnd: fix go fmt 2020-07-08 16:46:51 +08:00
Andras Banki-Horvath d3545830c9 kvdb+etcd: integrate the abort context to the STM retry loop
This commit extends the etcd.BackendConfig to also provide an abort
context and integrates it with the STM retry loop in order to be able
stop LND when conflicting transactions keep the loop running.
2020-06-03 17:44:44 +02:00
Andras Banki-Horvath 415de2f0c7 kvdb+etcd: remove retry goroutine from the STM
This commit removes the retry goroutine from the STM as the retry loop
is only running when the STM transaction is encapsulated in Update/View
whereas for self-standing transactions we use a different approach.
By removing the goroutine we won't catch panics thrown that are supposed
to be catched outside of the STM.
2020-06-03 17:44:44 +02:00
Andras Banki-Horvath c3fcfd1530 etcd: add namespace support to separate key spaces
This commit extends etcd db with namespaces without additional storage
space requirements. This is simply done by instead of using an all zero
root bucket id, we use the sha256 hash of the name space as our root
bucket id.
2020-05-22 11:26:25 +02:00
Andras Banki-Horvath 85aee9b064 kvdb+lncfg: fully move etcd behind build tag
This commit separates all etcd related sources (sans a few stubs and
config) from the rest of the source tree and makes compilation conditional
depending on whether the kvdb_etcd build tag is specified.
2020-05-22 11:26:25 +02:00
Andras Banki-Horvath 3ef331e016 lncfg+etcd: add TLS config
This commit extends lncfg with etcd TLS config and passes these
parameters to the etcd client upon construction.
2020-05-22 11:26:25 +02:00
Andras Banki-Horvath 02f9d91e49 kvdb+etcd: only lock on bucket path mod revision for reads and writes
This commit reduces the compare set size the STM will submit in
transactions by adding only the bucket keys along the bucket path to a
specific lock set. This lock set then used to filter the read set,
effectively removing all read only keys from the transaction predicate
that are not bucket keys.
By tracking if a read-write tx actually changes something, we can also
"bump" the mod revision of the bucket keys.
With this trick we essentially implement a read-write lock for our
bucket structure greatly reducing transaction processing time.
2020-05-22 11:26:25 +02:00
Andras Banki-Horvath 859a457e48 kvdb+etcd: extend kvdb and STM with stats
The commit itslef adds stats to the transactions such that we can see
how LND behaves with and etcd backend.
2020-05-22 11:26:25 +02:00
Andras Banki-Horvath 3b7525659c channeldb+kvdb: add walletdb interface test
This commit adds walletdb interface test suite to the existing test set.
2020-05-22 11:26:25 +02:00
Andras Banki-Horvath 6a24a03cec channeldb+kvdb: walletdb/kvdb interface etcd implementation
This commit adds a full interface implementation of the walletdb/kvdb
interface with detailed tests.
2020-05-22 11:26:25 +02:00
Andras Banki-Horvath 137dee04e8 channeldb+kvdb: an extended STM on top of etcd clientv3
This commit adds an extended STM, similar to what available in etcd's
clientv3 module. This incarnation of said STM supports additional
features, like positioning in key intervals while taking into account
deletes and writes as well. This is a preliminary work to support all
features of the kvdb interface.
2020-05-22 11:26:24 +02:00