In our current invoice pagination logic, we would not return any
invoices if our offset index was more than 1 off our last index and we
were paginating backwards. This commit adds a test case for this
behaviour before fixing it in the next commit.
Add an entry to a payments index bucket which maps sequence number
to payment hash when we initiate payments. This allows for more
efficient paginated queries. We create the top level bucket in its
own migration so that we do not need to create it on the fly.
When we retry payments and provide them with a new sequence number, we
delete the index for their existing payment so that we do not have an
index that points to a non-existent payment.
If we delete a payment, we also delete its index entry. This prevents
us from looking up entries from indexes to payments that do not exist.
Update our current tests to include lookup of duplicate payments. We
do so in preparation for changing our lookup to be based on a new
payments index. We add an append duplicate function which will add a
duplicate payment with the minimum information required to successfully
read it from disk in tests.
This commit extends the etcd.BackendConfig to also provide an abort
context and integrates it with the STM retry loop in order to be able
stop LND when conflicting transactions keep the loop running.
This commit removes the retry goroutine from the STM as the retry loop
is only running when the STM transaction is encapsulated in Update/View
whereas for self-standing transactions we use a different approach.
By removing the goroutine we won't catch panics thrown that are supposed
to be catched outside of the STM.
Previously it wasn't possible to store a preimage in the invoice
database and signal that a payment should not be settled right away. The
only way to hold a payment was to insert the magic UnknownPreimage value
in the invoice database. This commit introduces a distinct flag to
signal that an invoice is a hold invoice and thereby allows the preimage
to be present in the database already.
Preparation for (key send) hodl invoices for which we already know the
preimage.
This line was incorrectly moved when the migtest package was created for
migration 12. This PR introduces a negative test for CreateTLB which
surfaced this.
This commit extends etcd db with namespaces without additional storage
space requirements. This is simply done by instead of using an all zero
root bucket id, we use the sha256 hash of the name space as our root
bucket id.
This commit separates all etcd related sources (sans a few stubs and
config) from the rest of the source tree and makes compilation conditional
depending on whether the kvdb_etcd build tag is specified.
This commit adds the ExtendedBackend interface which is an extension to
the walletdb.DB interface. This paves the way to using etcd.db.View and
etcd.db.Update in the global View and Update functions without much code
rewrite.
This commit reduces the compare set size the STM will submit in
transactions by adding only the bucket keys along the bucket path to a
specific lock set. This lock set then used to filter the read set,
effectively removing all read only keys from the transaction predicate
that are not bucket keys.
By tracking if a read-write tx actually changes something, we can also
"bump" the mod revision of the bucket keys.
With this trick we essentially implement a read-write lock for our
bucket structure greatly reducing transaction processing time.
This commit adds an extended STM, similar to what available in etcd's
clientv3 module. This incarnation of said STM supports additional
features, like positioning in key intervals while taking into account
deletes and writes as well. This is a preliminary work to support all
features of the kvdb interface.
The btcsuite/fastsha256 registers itself in the crypto package of
golang as a replacement for sha256. This causes problems in TLS1.3
connections that require the hash implementations to be serializable
and results in the "tls: internal error: failed to clone hash" error.
By removing all uses of the library we fix that error.
In this commit, we eliminate an extraneous copy in the `QueryPayments`
method. Before this commit, we would copy each payment from the initial
FetchPayments call into a new slice. However, pointers to payments are
return from `FetchPayments`, so we can just maintain that same reference
rather than copying again when we want to limit our response.
This commit reverts cb4cd49dc8d3b0255afe9ff29af9c46c2dbb2c98 to bring
back the insufficient local balance failure.
Distinguishing betweeen this failure and a regular "no route" failure
prevents meaningless htlcs from being sent out.
This commit removes ChanStatusDefault from the list of
orderedChanStatusFlags since it is not flag. As with the prior commit,
the logic around these flags assumes everything in the list is a flag,
but ChanStatusDefault is not.
It turns out we properly special case that if the channel is in
ChanStatusDefault that we only return ChanStatusDefault. However, if any
of the bits are set we would always report ChanStatusDefault since
status&0 == 0. This fixed simply by removing ChanStatusDefault from the
list since we only need the list to express non-default status flags.
This commit resovles a lingering issue w/in the codebase wrt how the
ChannelStatus flags are defined. Currently ChannelStatus is improperly
used to define a bit field and the individual flags themselves. As a
result, HasChanStatus accepts queries on particular status (combinations
of flags) and individual flags themselves.
This is an issue because the way HasChanStatus computes whether the
channel has a particular status assumes the provided inputs are all
flags (or at least combinations of flags).
However, ChanStatusDefault is simply the absence of any other flag.
Hence, HasChanStatus will always return true when querying for
ChanStatusDefault because status&0 == 0 is always true.
Longer term we should should consider splitting these definitions into
flags and particular states, and change the way construct or operate on
them, but for now I've just special-cased this one value. Fortunately,
we don't query HasChannelStatus w/ ChanStatusDefault anywhere in the
codebase so we dodge a bullet here, but it'd be nice to have some
greater assurances moving forward.
With mpp it isn't possible anymore for findPath to determine that there
isn't enough local bandwidth. The full payment amount isn't known at
that point.
In a follow-up, this payment outcome can be reintroduced on a higher
level (payment lifecycle).
Adds a PaymentsQuery struct, which contains parameters to restrict the
response of QueryPayments, returning a PaymentsQuerySlice with the
payments query result. The behavior of this api is the same as
the QueryInvoices one.
We add validation making sure we are not trying to register MPP shards
for non-MPP payments, and vice versa. We also add validtion of total
sent amount against payment value, and matching MPP options.
We also add methods for copying Route/Hop, since it is useful to use
for modifying the route amount in the test.
This commit finally enables MP payments within the payment lifecycle
(used for SendPayment). This is done by letting the loop launch shards
as long as there is value remaining to send, inspecting the outcomes for
the sent shards when the full payment amount has been filled.
The method channeldb.MPPayment.SentAmt() is added to easily look up how
much value we have sent for the payment.
This commit redefines how the control tower handles shard and payment
level settles and failures. We now consider the payment in flight as
long it has active shards, or it has no active shards but has not
reached a terminal condition (settle of one of the shards, or a payment
level failure has been encountered).
We also make it possible to settle/fail shards regardless of the payment
level status (since we must allow late shards recording their status
even though we have already settled/failed the payment).
Finally, we make it possible to Fail the payment when it is already
failed. This is to allow multiple concurrent shards that reach terminal
errors to mark the payment failed, without havinng to synchronize.
To move towards how we will handle existing attempt in case of MPP
(collecting their outcome will be done in separate goroutines separate
from the payment loop), we move to collect their outcome first.
To easily fetch HTLCs that are still not resolved, we add the utility
method InFlightHTLCs to channeldb.MPPayment.
This commit changes the fallback in NextLocalHtlcIndex to
RemoteCommitment since the LocalHtlcIndex field lags behind
on the LocalCommitment. Without this bug fix, open circuits
would get prematurely trimmed, resulting in more erroneous
logs. A test case is included to check that the fix works.
In this commit, we add a new channel type bit: a frozen channel. A
frozen channel is one that can only be cooperatively closed by the
responder, but not the initiator. This channel type is useful for
certain classes of channel factory like protocols. We then add a new key
on the channel bucket level to store the height after which this
restriction no longer applies.
Add a balance at height lookup function which can be used to
obtain local/remote balance at a given height. The current in memory
commits and revocation log are used to source this information.
The explicit `bbolt` dep is gone, as we depend on `kvdb`, which is
actually `walletdb`, which has its own module that defines the proper
`bbolt` version.
In this commit, we migrate all the code in `channeldb` to only reference
the new `kvdb` package rather than `bbolt` directly.
In many instances, we need to add two version to fetch a bucket as both
read and write when needed. As an example, we add a new
`fetchChanBucketRw` function. This function is identical to
`fetchChanBucket`, but it will be used to fetch the main channel bucket
for all _write_ transactions. We need a new method as you can pass a
write transaction where a read is accepted, but not the other way around
due to the stronger typing of the new `kvdb` package.
In this commit, we create a new package `kvdb`, which is meant to serve
as the basis for any future database abstractions within `lnd`. Rather
than directly use the `walletdb` package (which we base off of), we
instead use a series of type-aliases to re-type the fundamental
types/interfaces of the `walletdb` package. This lets us type
`kvdb.RwTx` instead of `walletdb.ReadWriteTransaction` everywhere.
Additionally, our usage of type-aliases is also intended to create an
easy pathway in the future wherein we can gradually re-defined or
re-implement these types to wean off of the `walletdb` package.
This commit migrates the payments in the database to a new structure
that allows for multiple htlcs per payments. The migration introduces a
new sub-bucket that contains a list of htlcs and moves the old single
htlc into that.
This commit extends the htlc fail info with the full failure reason that
was received over the wire. In a later commit, this info will also be
exposed on the rpc interface. Furthermore it serves as a building block
to make SendToRoute reliable across restarts.
This commit converts the database structure of a payment so that it can
not just store the last htlc attempt, but all attempts that have been
made. This is a preparation for mpp sending.
In addition to that, we now also persist the fail time of an htlc. In a
later commit, the full failure reason will be added as well.
A key change is made to the control tower interface. Previously the
control tower wasn't aware of individual htlc outcomes. The payment
remained in-flight with the latest attempt recorded, but an outcome was
only set when the payment finished. With this commit, the outcome of
every htlc is expected by the control tower and recorded in the
database.
Co-authored-by: Johan T. Halseth <johanth@gmail.com>
Duplicate payments is legacy that we keep alive for accounting purposes.
This commit isolates the deserialization logic for duplicate payments in
its own file, so that regular payment logic and db structure can evolve
without needing to handle/migrate the legacy data.
Previously this was tested as a white box. Database access methods were
duplicated as test code and compared to the return value of the code
under test. This approaches leads to brittle test because it relies
heavily on implementation details. This commit changes this and prepares
for additional test coverage being added in later commits.
To better distinguish payments from HTLCs, we rename the attempt info
struct to HTLCAttemptInfo. We also embed it into the HTLCAttempt struct,
to avoid having to duplicate this information.
The paymentID term is renamed to attemptID.
Add an optional channel status CloseChannel which will be stored on the
hitsorical channel which is persisted at channel close. This status is
used to set the close initiator for channels that do not complete the
funding flow or we abandon. In follow up commits, this status will be
used to record force and breach closes. The value is written to the
historical channel bucket for diplay over rpc.
This commit adds two new channel statuses which indicate the party that
initatited closing the channel. These statuses are set in conjunction
with the existing commit broadcast status so that we do not need to
migrate existing logic to handle multiple types of closes. This status
is set for locally initiated force closes in this commit because they
follow a similar pattern to cooparative closes, marking the commitment
broadcast then proceeding with tx broadcast. Remote force closes are
added in the following commit, as they are handled differently.
This changes replaces the pending an waiting booleans in fetchChannels
with optional filters which can be more flexibly used. This change
allows filtering of channels without having to reason about the matrix
of possible boolean combinations. A test is added to ensure that the
combinations of these filters act as expected.
This commit updates the channel state machine to
persistently store remote updates that we have received a
signature for, but that we haven't yet included in a commit
signature of our own.
Previously those updates were only stored in memory and
dropped across restarts. This lead to the production of
an invalid signature and channel force closure. The remote
party expects us to include those updates.
This commit removes channeldb.FetchAllInvoices and changes tests such
that expectation sets are prepared in the test case instead of selected
from the DB.
This commit adds handling code for the key send custom record. If this
record is present and its hash matches the payment hash, invoice
registry will insert a new invoice into the database "just in time". The
subsequent settle flow is unchanged. The newly inserted invoice is
picked up and settled. Notifications will be broadcast as usual.
This field isn't optional. It was introduced in
5ed31b1030176374d4184fee0b80ae7679e0e355 which was part of release
0.4.0. This release contained breaking database changes, so it is safe
to assume that from there on the field is always populated.
If there would still be empty payment request fields in the wild, users
would also experience issues with ListInvoices. ListInvoices decodes the
payment request.
This commits builds on top of PR #3694 to further clarify invoice
state by defining pending invoices as the ones which are not
settled or canceled. Automatic cancellation of expired invoices
makes this possbile. While this change only directly affects
ChannelDB, users of the listinvoices RPC will receive actual
pending invoices when pending_only flag is set.
Previously we would return nil features when we didn't have a node
announcement or a given node. With this change, we can always assume the
feature vector is populated during pathfinding.
This commit changes how FetchAllInvoicesWithPaymentHash behaves
when the DB is empty and also adds a unit test to test that
case as well as normal expected behavior.
This commit adds InvoiceExpryWatcher which is a separate class that
receives new invoices (and existing ones upon restart) from InvoiceRegistry
and actively watches their expiry. When an invoice is expired
InvoiceExpiryWatcher will call into InvoiceRegistry to cancel the
invoice and by that notify all subscribers about the state change.
This commit prepares for more manipulation of custom records. A list of
tlv.Record types is more difficult to use than the more basic
map[uint64][]byte.
Furthermore fields and variables are renamed to make them more
consistent.
Before we change the Hop struct, isolate the code that is used in older
migrations to prevent breaking them.
route.go was taken from commit 6e463c1634061d595953f20813860207e5d485ce
Previously the cancel and add actions were combined in a single map.
Nil values implictly signaled cancel actions. This wasn't very obvious.
Furthermore this split prepares for processing the adds and cancels
separately, which is more efficient if there are already two maps.
This commit moves the update code into its own function as a preparation
for extending the logic further for mpp.
In order to make this change cleanly, structured result codes are
introduced. This also prepares for a future htlc notifier rpc hook that
reports htlc settle decisions to external applications.
Furthermore the awkward use of errNoUpdate as a way to signal no update
is removed.
This commit adds fields for upfront shutdown scripts set
by the local and remote peer to the OpenChannel struct.
These values are optional, so they are added with their
own keys in the chanBucket in the DB.
In this commit, we add a new bit to the existing ChannelType bitfield.
If this bit is set, then it signals that we have the funding transaction
stored on disk. A future change will enable lnd to have the funding
transaction be constructed externally, allowing for things like funding
from a hardware wallet, or a channel created as a sub-branch within an
existing channel factory.
Refresh channel memory state whenever the short channel id is refreshed.
This is to make the in-memory channel consistent with the disk data.
Fixes#3765.
This commit restructures an invoice's ContractTerms to better encompass
the restrictions placed on settling. For instance, the final ctlv delta
and invoice expiry are moved from the main invoice body (where
additional metadata is stored). Additionally, it moves the State field
outside of the terms since it is rather metadata about the invoice
instead of any terms offered to the sender in the payment request.
This commit modifies the FetchPayment method to return MPPayment structs
converted from the legacy on-disk format. This allows us to attach the
HTLCs to the events given to clients subscribing to the outcome of an
HTLC.
This commit also bubbles up to the routerrpc/router_server, by
populating HTLCAttempts in the response and extracting the legacy route
field from the HTLCAttempts.
