Add start and end height parameters to the rpc and cli GetTransactions
endpoints. Default to returning all transactions from genesis to tip,
including unconfirmed transactions to maintain backwards compatibility.
This enforces the _actualized_ fee rate of the commitment transaction,
rather than the fee floor used for estimation. The new value of 250
sat/kw corresponds to 1 sat/byte, rather than 253 which is only rounded
up during estimation to account for the fact that BOLT 3 rounds down to
the nearest satoshi and that the vbyte fee estimation is lossy.
Previously we would incorrectly fail to sign the next commitment even
though the fee was technically high enough. Restarting with this commit
should solve the issue as long as the channel hasn't already gone to
chain.
This commit introduces the Signature interface which will be used by our
witness construction methods instead of passing in raw byte slices. This
will be used later to inject various kinds of mock signatures, e.g.
73-byte signatures for simulating worst-case witness weight.
This commit adds an additional santity check that rejects zero-value
HTLCs, preventing them from being added to the channel state even if the
channel config's minhtlc value is zero.
This commit adds a test to exercise that HTLC signatures are sent in the
correct order, i.e. they match the sorting of the HTLC outputs on the
commitment after applying BOLT 3's BIP69+CLTV sort.
This commit fixes#4118 by properly sorting the HTLC signatures sent
over the wire to match the BOLT3 BIP69+CLTV sorting of the commitment
outputs.
To do so, we expose the slice of cltv deltas for HTLCs on the unsigned
commitment after applying the commitment sorting. This will be used to
locate the proper output index, as the CLTV serves as a tie breaker
between HTLCs that otherwise have the same payment hash and amount.
Note that #3412 fixed the issue partially by ensuring the commitment was
constructed properly (and the second-level prev outpoint's txid was
correct), but failed to address that the HTLC signatures were still sent
out in the incorrect order. With this, we pass the test case introduce
in the next commit.
We currently write each HTLCs OutputIndex to disk, but we don't use it
when restoring. The restoration is modified to use these directly, since
we will have lost access to the sorting of CLTVs after the initial
signing process.
In case the funding manager detects that a funding flow is requested
to be executed with the help of a PsbtIntent, the normal channel
negotiation with the remote peer is interrupted, as soon as the
accept_channel message was received. With the remote peer's funding
multisig key and our local key, we can derive the funding output
script and its address. This is enough to start the PSBT funding
and signing process which the user will do externally to the daemon.
We add a new funding assembler and intent type that handle channel
funding through the use of a PSBT. The PsbtIntent is in itself a
simple state machine that can be stepped through the process of
assembling the required information for the funding output, verifying
a user supplied PSBT for correctness, accepting a fully signed PSBT
and then assembling the funding wire message.
In this commit, we make the internal channel funding flow aware of
frozen channels. We also update the testSingleFunderReservationWorkflow
method to ensure that the created channels have the proper type bit set.
As frozen channels can only be created via the non-default channel
assembler, we extend both the ShimIntent and CannedAssembler to also
accept and expose this new channel status along with the thaw height.
If we are the initiator, we check that our starting balance after
subtracting fees are not less than two times the default dust limit.
This commit adds a similar check for the non-initiator case, checking
that the remote party has a starting balance of reasonable size.
We also increase the witness size for these types to account for the 3
extra bytes. The size won't be correct in all cases, but it is just an
upper bound in any case.
This commit modifies the NP2WKH and NP2WSH input tests to ensure the
input count is properly incremented and accounted for in the size
estimate. 253 is chosen because it is the lowest value that, when
serialized, occupies more than one byte on the wire.
This commit adds two new channel statuses which indicate the party that
initatited closing the channel. These statuses are set in conjunction
with the existing commit broadcast status so that we do not need to
migrate existing logic to handle multiple types of closes. This status
is set for locally initiated force closes in this commit because they
follow a similar pattern to cooparative closes, marking the commitment
broadcast then proceeding with tx broadcast. Remote force closes are
added in the following commit, as they are handled differently.
Since our HTLC must also be added to the remote commitment, we do the
balance caluclation also from the remote chain perspective and report
our minimum balance from the two commit views as our available balance.
When we send non-dust HTLCs as the non-initiator, the remote party will
have to pay the extra commitment fee. To account for this we figure out
if they can afford paying this fee, if not we report that we only have
balance available for dust HTLCs, since these HTLCs won't increase the
commitment fee.
Since we want to handle the edge case where paying the HTLC fee would
take the initiator below the reserve, we move the subtraction of the
reserve into availableBalance where this calculation will be performed.
This commit adds an extra validation step when adding HTLCs. Previously
we would only validate the remote commitment resulting from adding an
HTLC, which in most cases is enough. However, there are situations where
the dustlimits are different, which could lead to the resulting remote
commitment from adding the HTLC being valid, but not the local
commitment.
Now we also validate the local commitment. A test to trigger the case is
added.
add
To ba able to validate the commitment sanity both for remote and local
commitments, and at the same time predict both our and their add, we let
validateCommitmentSanity take an extra payment descriptor to make this
possible.
This commit fixes the TestMaxAcceptedHTLCs, TestMaxPendingAmount,
TestMinHTLC, & TestChanReserve unit tests to pass with the new
ReceiveHTLC logic. Instead of asserting specific failures upon
receiving a new commitment signature, the various assertions were
moved to assert on the error returned from ReceiveHTLC.
This commit checks the commitment sanity when receiving an HTLC so
that if a commitment transaction will overflow from an ADD, it is
caught earlier rather than in ReceiveNewCommitment.
The unit test TestNewBreachRetributionSkipsDustHtlcs triggered a state
transition from Bob, even though it was Alice that had added the HTLCs.
This is wrong since it will lead to Bob still owing Alice a commitment,
which is not accounted for in the unit tests.
We add a sanity check that the add heights has been set for all entries
found in the logs, and return an error otherwise. This won't happen
during normal operation, but it does reveal the mistake in the unit
test, which is fixed by making Alice trigger the transition.
In addition we resolve a long standing TODO by removing a (purposeful)
panic in the channel state machine. Old version of lnd had a bug that
could lead to the parent entries being lost during channel restore. A
panic was added to get to the bottom of if.
This is now fixed, so new nodes shouldn't encounter it. However, to be
on the safe side, instead of panicking we return an error back to
gracefully exit the channel state machine.
Updates were always restored with the same log index. This could cause a
crash when the logs were compacted and possibly other problems
elsewhere.
Extended unit test to cover the crash scenario.
This commit updates the channel state machine to
persistently store remote updates that we have received a
signature for, but that we haven't yet included in a commit
signature of our own.
Previously those updates were only stored in memory and
dropped across restarts. This lead to the production of
an invalid signature and channel force closure. The remote
party expects us to include those updates.
This test asserts that remote updates that are locked-in on the local
commitment, but haven't been signed for on the remote commitment, are
properly restored after a restart.
When creating the keyring, the tweak is already calculated in the remote
commitment case. We add the calculation also for our own commitment, so
we can use it in all cases without deriving the tweak.
Based on the current channel type, we derive the script used for the
to_remote output. Currently only the unencumbered p2wkh type is used,
but that will change with upcoming channel types.
To make the channel state machine less concerned about the type of
commitment, we nil the local tweak when creating the keyring, depending
on the commitment type.
We abstract away how keys are generated for the different channel types
types (currently tweak(less)).
Intention is that more of the logic that is unique for each commitment
type lives in commitment.go, making the channel state machine oblivious
to the keys and outputs being created on the commitment tx for a given
channel state.
createCommitmentTx would earlier mutate the passed commitment struct
after evaluating the htlc view and calculating the final balances, which
was confusing since the balances are supposed to only be *after*
subtracting fees.
Instead we take the needed parameters as arguments, and return the final
balances, tx and fee to populate the commitment struct in a proper way.
PURE CODE MOVE:
Moving createCommitmentTx, CreateCommitTx, createStateHintObfuscator,
CommitmentKeyRing, DeriveCommitmentKeys, addHTLC, genHtlcScripts
We move the methods and structs to a new file commitment.go in
preparation for defining all the logic that is dependent on the channel
type in this new file.
Instead of passing delays and dustlimits separately, we pass the correct
channel config to CreateCommitTx from the POV of the local party that
owns the commit tx.
To make it more clear which commitment we are actually creating, we
rename variables to denote local and remote, to prepare for the case
when both outputs might be delayed.
In this commit, we implement the currently defined transition methods
for the new `FundingStateStep` method. At this point, we're now able to
serve the "responder" of the externally initiated channel funding flow
by being able to register and cancel a funding flow according to its
expected pending channel ID.
This commit gets upfront shutdown scripts from openchannel and
acceptchannel wire messages sent from our peer and sets upfront
shutdown scripts in our open and accept channel messages when
the remote peer supports option upfront shutdown and we have
the feature enabled.
This commit adds fields for upfront shutdown scripts set
by the local and remote peer to the OpenChannel struct.
These values are optional, so they are added with their
own keys in the chanBucket in the DB.
This commit sets our close addresss to the address
specified by option upfront shutdown, if specified,
and disconnects from peers that fail to provide
their upfront shutdown address for coopertaive closes
of channels that were opened with the option set.
In this commit, we fix a long standing bug within the newly created
`verifyFundingInputs` method. Before this commit, the method would
attempt to derive the pkScript by looking at the last items on the
witness stack, and making a p2wsh output script from that. This is
incorrect as typically non of these scripts will actually be p2wsh, and
instead will be p2wkh. We fix this by using the newly available
`txscript.ComputePkScript` method to derive the proper pkScript.
This resolves an issue w.r.t passing incorrect arguments for all
backends, but an issue still stands for the neutrino backend. As is, we
pass a height hint of zero into the `GetUtxo` method call. With the way
the current utxo scanner is set up for neutrino, this'll cause it to
never find the UTXO, as it takes the height hint as a UTXO birth height,
rather than a lower bound of the birth of the UTXO.
In this commit, we add a new method `RegisterFundingIntent` that allows
a caller to "inject" a pre-populated chanfunding.Intent into a funding
workflow. As an example, if we've already agreed upon the "shape" of the
funding output _outside_ the protocol, then we can use this to pass down
the details of the output, then leverage the normal wire protocol to
carry out the remainder of the funding flow.
In this commit, we start to thread the pending channel ID from wire
protocol all the way down into the reservation context. This change will
allow negotiation to take place _outside_ the protocol that may result
in a particular chanfunding.Assembler being dispatched.
In this commit, we make the wallet aware of the second type of funding
intent: the ShimIntent. If we have one of these, then we don't need to
construct the funding transaction, and can instead just obtain the
outpoint directly from it.
In this commit, we begin to integrate the new channel funding package
into the existing codebase. With this set of changes, we'll no longer
construct and sign the funding transaction within this package, instead
delegating it to the new chanfunding package. We use the new
chanfunding.WalletAssembler to carry out all channel funding, providing
it with an implementation of all its interfaces backed by the wallet.
In this commit, we introduce a series of new abstractions for channel
funding. The end goal is to enable uses cases that construct the funding
transaction externally, eventually handing the funding outpoint to lnd.
An example of such a use case includes channel factories and external
channel funding using a hardware wallet.
We also add a new chanfunding.Assembler meant to allow external channel
funding in contexts similar to how channel factories
can be constructed. With this channel funder, we'll only obtain the
channel point and funding output from it, as this alone is enough to
carry out a funding flow as normal.
In this commit, we make an incremental change to move the existing coin
selection code into a new chanfunding package. In later commits, this
package will grow to serve all the lower level channel funding needs in
the daemon.
Instead of tracking local updates in a separate link variable, query
this state from the channel itself.
This commit also fixes the issue where the commit tx was not updated
anymore after a failed first attempt because the revocation window was
closed. Also those pending updates will be taken into account when the
remote party revokes.
Previously the channel method FullySynced was used to decide whether to
send a new commit sig message. However, it could happen that FullySynced
was false, but that we didn't owe a commitment signature. Instead we
were waiting on the other party to send us a signature. If that
happened, we'd send out an empty commit sig. This commit modifies the
condition that triggers a new commit sig and fixes this deviation from
the spec.
To facilitate the logging, this commit adds a new OweCommitment method.
For the logging, we only need to consider the remote perspective. In a
later commit, we'll also start using the local perspective to support
the decision to send another signature.
In this commit, we create a new chainfee package, that houses all fee
related functionality used within the codebase. The creation of this new
package furthers our long-term goal of extracting functionality from the
bloated `lnwallet` package into new distinct packages. Additionally,
this new packages resolves a class of import cycle that could arise if a
new package that was imported by something in `lnwallet` wanted to use
the existing fee related functions in the prior `lnwallet` package.
In this commit, we convert the existing `channeldb.ChannelType` type
into a _bit field_. This doesn't require us to change the current
serialization or interpretation or the type as it is, since all the
current defined values us a distinct bit. This PR lays the ground work
for any future changes that may introduce new channel types (like anchor
outputs), and also any changes that may modify the existing invariants
around channels (if we're the initiator, we always have the funding
transaction).
Without this, it was possible for a combination of our balance and max
fee allocation to result in a fee rate below the fee floor causing the
remote party to reject the update and close the channel.
In this commit, we use the recently added `chanvalidate` package to
verify channels once they have been confirmed in the funding manager. We
expose a new method on the `LightningWallet` struct: `ValidateChannels`
which calls the new shared 1st party verification code.
After the channel is fully confirmed in the funding manager, we'll now
use this newly exposed method to handle all validation. As a result, we
can remove the existing validation code in the funding manager, and rely
on the new code in isolation.
In this commit, we create a new `chanvalidate` package which it to house
all logic required for 1st and 3rd party channel verification. 1st party
verification occurs when we find a channel in the chain that is
allegedly ours, while 3rd party verification will occur when a peer
sends us a channel proof of a new channel.
In the scope of the recent CVE, we actually fully verified 3rd party
channels, but failed to also include those checks in our 1st party
verification code. In order to unify this logic, and prevent future
issues, in this PR we move to concentrate all validation logic into a
single function. Both 1st and 3rd party validation will then use this
function. Additionally, having all the logic in a single place makes it
easier to audit, and also write tests against.
In this commit, we move to make a full deep copy of the commitment
transaction in `getSignedCommitTx` to ensure that we don't mutate the
commitment on disk, possibly resulting in a "hot commitment".
In this commit, we consolidate the number of areas where we derive our
commitment keys. Before this commit, the `isOurCommitment` function in
the chain watcher used a custom routine to derive the expected
scripts/keys for our commitment at that height. With the recent changes,
we now have additional logic in `DeriveCommitmentKeys` that wasn't
copied over to this area. As a result, the prior logic would erroneously
detect if it was our commitment that had hit the chain or not.
In this commit, we remove the old custom code, and use
`DeriveCommitmentKeys` wihtin the chain watcher as well. This ensures
that we only need to maintain the key derivation code in a single place,
preventing future bugs of this nature.
In this commit, we update the brar logic in the channel state machine,
and also the brar itself to be aware of the new commitment format.
Similar to the unilateral close summary, we'll now blank out the
SingleTweak field in `NewBreachRetribution` if it's a tweakless
commitment. The brar will then use this to properly identify the
commitment type, to ensure we use the proper witness generation function
when we're handling our own breach.
In this commit, we update the funding workflow to be aware of the new
channel type that doesn't tweak the remote party's output within the
non-delay script on their commitment transaction. To do this, we now
allow the caller of `InnitChannelReservation` to signal if they want the
old or new (tweakless) commitment style.
The funding tests are also updated to test both funding variants, as
we'll still need to understand the legacy format for older nodes.
In this commit, we update the channel state machine to be aware of
tweakless commits. In several areas, we'll now check the channel's type
to see if it's `SingleFunderTweakless`. If so, then we'll opt to use the
remote party's non-delay based point directly in the script, skipping
any additional cryptographic operations. Along the way we move the
`validateCommitmentSanity` method to be defined _before_ it's used as is
cutomary within the codebase.
Notably, within the `NewUnilateralCloseSummary` method, we'll now _blank
out_ the `SingleTweak` value if the commitment is tweakless. This
indicates to callers the witness type they should map to, as the value
isn't needed at all any longer when sweeping a non-delay output.
We also update the signing+verification tests to also test that we're
able to properly generate a valid witness for the new tweakless
commitment format.
Instead of marking the database state when processing the channel
reestablishment message, we wait for the result of this processing to
arrive in the link, and mark it accordingly in the database here.
We do this move the logic determining whether we should force close the
channel or not, and what state to mark it in the DB, to the same place,
as these need to be consistent.
This commit converts the ErrCommitSyncLocalDataLoss error into a struct,
that also holds the received last unrevoked commit point from the remote
party.
Checks that we get ErrDoubleSpend as expected when publishing a
conflicting mempool transaction with the same fee as the existing one,
and that we can publish a replacement with a higher fee successfully.
error
Since btcwallet will return typed errors now, we can simplify the
matching logic in order to return ErrDoubleSpend.
In case a transaction cannot be published since it did not satisfy the
requirements for a valid replacement, return ErrDoubleSpend to indicate
it was not propagated.
The cache wasn't really serving a purpose as FetchInputInfo isn't known
to be a hot path. Also, with a planned addition of returning the
confirmation status of an output within FetchInputInfo in a later
commit, caching won't be useful as we'll have to go to disk anyway to
determine the confirmation status.
In this commit, we address an edge case that can happen a user rescans
w/ their seed, while retaining their existing `channel.db`. Once they
rescan, if they go to sign for a channel sweep for example, the
commitment key family (actually an account) may not yet have been
created, causing the signing attempt to fail.
We remedy this always creating the account if we go to sign, and the
account isn't found. The change has been structured to make this the
exception, so we'll avoid always needing to do 2 DB hits (check if
account exists, sign), each time we sign.
A new test has been added to exercise this behavior. If the diff from
the `signer.go` file is removed, then the test will fail.
This commit adds a SubtractFees option to the funding request, letting
the caller specify that the fees should be deducted from the funding
amount. This paves the way for letting the funding manager spend up to a
given amount when creating a channel, like the rest of the funds in the
wallet.
This makes the method independent of the ChannelContribution struct.
We also add a function closure to the return of selectCoinsAndChange,
that let is unlock the selected output in case of error.
In this commit, we fix an existing bug that would cause us to be unable
to derive the very first key in a key family if the wallet hadn't
already derived it in the past. This can happen if a user keeps their
same `channel.db`, but restores their wallet resulting in fresh
`wallet.db` state.
This is an existing issue due to the fact that we don't properly
distinguish between an empty key locator, and the very first key in a
`KeyFamily`: `(0, 0)`. Atm, `KeyLoactor{0, 0}.IsEmpty() == True`,
causing us to be unable to retrieve this key in certain cases since we
fall through and attempt address based derivation.
In order to remedy this, we add a new special case (until we upgrade
`KeyLoactor` formats, but needed for legacy reasons) to _try_ a regular
`KeyLoactor` based derivation if we fail to derive via address, and this
is an "empty" key loc. This has been tested in the field and shown to
work, with the one downside that in this "hot swap restoration" case,
we'll hit the database twice to derive the key.
In this commit, we fix a logic flaw in the testCreateSimpleTx test case
which emerged once we the bug fix for dust outputs landed. Before this
commit, we would erroneously fail during valid test execution.
In this commit we fix a hidden bug in the transaction creating logic
that was only manifested recently due to higher fees on Bitcoin's
mainnet. Before this commit, we would use the target fee rate to
determine if an output was dust or not. However, this is incorrect, as
instead the relay fee should be used as this matches the policy checks
widely deployed in Bitcoin full node today.
To fix this issue we now properly use the relay fee when computing dust.
This fixes the issue for the `EstimateFee` call, but the `SendOutputs`
call also has a similar issue. However, this must be fixed within
`btcwallet` itself, so it has been left out of this commit
Fixes#3217.
In this commit, we patch a small bug in the newly added raw tx hex field
for ListTransactions. We now ensure that we also set the raw tx hex
field for unconfirmed transactions.
In this commit, we fix a lingering TOOD statement in the channel arb.
Before this commitment, we would simply wipe our our local HTLC set of
the HTLC set that was on the remote commitment transaction on force
close. This was incorrect as if our commitment transaction had an HTLC
that the remote commitment didn't, then we would fail to cancel that
back, and cause both channels to time out on chain.
In order to remedy this, we introduce a new `HtlcSetKey` struct to track
all 3 possible in-flight set of HTLCs: ours, theirs, and their pending.
We also we start to tack on additional data to all the unilateral close
messages we send to subscribers. This new data is the CommitSet, or the
set of valid commitments at channel closure time. This new information
will be used by the channel arb in an upcoming commit to ensure it will
cancel back HTLCs in the case of split commitment state.
Finally, we start to thread through an optional *CommitSet to the
advanceState method. This additional information will give the channel
arb addition information it needs to ensure it properly cancels back
HTLCs that are about to time out or may time out depending on which
commitment is played.
Within the htlcswitch pakage, we modify the `SignNextCommitment` method
to return the new set of pending HTLCs for the remote party's commitment
transaction and `ReceiveRevocation` to return the latest set of
commitment transactions on the remote party's commitment as well. This
is a preparatory change which is part of a larger change to address a
lingering TODO in the cnct.
Additionally, rather than just send of the set of HTLCs after the we
revoke, we'll also send of the set of HTLCs after the remote party
revokes, and we create a pending commitment state for it.
Now that the success resolver preimage field is always populated by the
incoming contest resolver, preimage lookups earlier in the
process (channel and channel arbitrator) can mostly be removed.
This enables users to specify an external API for fee estimation.
The API is expected to return fees in the JSON format:
`{
fee_by_block_target: {
a: x,
b: y,
...
c: z
}
}`
where a, b, c are block targets and x, y, z are fees in sat/kb.
Note that a, b, c need not be contiguous.
In this commit, we add a new interface which will allow callers to drop
in an arbitrary Web API for fee estimation with an arbitrary
request/response schema.
Co-authored-by: Valentine Wallace <vwallace@protonmail.com>
In this commit, we modify the main `closeObserver` dispatch loop to only
look for the local force close if we didn't recover the channel. We do
this, as for a recovered channel, it isn't possible for us to force
close from a recovered channel.
