In this commit, we fix an existing bug that would result in some
payments getting “stuck”. This would happen if one side restarted
before the channel was fully locked in. In this case, since upon
re-connection, the link will get added to the switch with a *short
channel ID of zero*. If A then tries to make a multi-hop payment
through B, B will fail to forward the payment, as it’ll mistakenly
think that the payment originated from a local-subsystem as the channel
ID is zero. A short channel ID of zero is used to map local payments
back to their caller.
With fix this by allowing the funding manager to dynamically update the
short channel ID of a link after it discovers the short channel ID.
In this commit, we fix a second instance of reported “stuck” payments
by users.
This commit introduces a new Ticker interface, that can be used
to control when the batch timer should tick. This is done to be
able to more easily control the ticker during tests. The batch
timer is wrapped in the new BatchTicker struct, and made part
of the config together with BatchSize.
In this commit, we add 6 new integration tests to test the various
actions that may need to be performed when either side goes on-chain to
fully resolve HTLC’s. Many of the tests are mirrors of each other as
they test sweeping/resolving HTLC’s from both commitment transactions.
In this commit, we address a lingering TODO: before this if we had a
set of HTLC’s that we knew the pre-image to on our commitment
transaction after a restart, then we wouldn’t attempt to settle them.
With this new change, we’ll check that we didn’t already retransmit the
settles for them, and check the preimage cache to see if we already
know the preimage. If we do, then we’ll immediately settle them.
In this commit, we add some additional logic to the case when we
receive a pre-image from an upstream peer. We’ll immediately add it to
the witness cache, as an incoming HTLC might be waiting on-chain to
fully resolve the HTLC with knowledge of the newly discovered
pre-image.
Before this commit, if the htlcManager unexpectedly exited (due to a
protocol error, etc), the underlying block epoch notification intent
that was created for it would never be cancelled. This would result in
tens, or hundreds of goroutine leaks as the client would never consume
those notifications.
To fix this, we move cancellation of the block epoch intent from the
Stop() method of the channel link, to the defer statement at the top of
the htlcManager.
In this commit, we add an additional case when handling a failed
commitment signature. If we detect that it’s a InvalidCommitSigError,
then we’ll send over an lnwire.Error message with the full details. We
don’t yet properly dispatch this error on the reciting side, but that
will be done in a follow up a commit.
In this commit, we modify the way the link handles HTLC’s that it
detects is destined for itself. Before this commit if a payment hash
came across for an invoice we’d already settled, then we’d gladly
accept the payment _again_. As we’d like to enforce the norm that an
invoice is NEVER to be used twice, this commit modifies that behavior
to instead reject an incoming payment that attempts to re-use an
invoice.
Fixes#560.
This simplifies the pending payment handling code because it allows it
be handled in nearly the same way as forwarded HTLCs by treating an
empty channel ID as local dispatch.
The src/dest terminology for routing packets is kind of confusing
because the source HTLC may not be the source of the packet for
settles/fails traversing the circuit in the opposite direction. This
changes the nomenclature to incoming/outgoing and always references
the HTLCs themselves.
Previously, some methods on a LightningChannel like SettleHTLC and
FailHTLC would identify HTLCs by payment hash. This would not always
work correctly if there are multiple HTLCs with the same payment hash,
so instead we change these methods to identify HTLCs by their unique
identifiers instead.
This changes the circuit map internals and API to reference circuits
by a primary key of (channel ID, HTLC ID) instead of paymnet
hash. This is because each circuit has a unique offered HTLC, but
there may be multiple circuits for a payment hash with different
source or destination channels.
The constructor functions have no additional logic other than passing
function parameters into struct fields. Given the large function
signatures, it is more clear to directly construct the htlcPacket in
client code than call a function with lots of positional arguments.
In this commit, we modify the existing logic to handle
UpdateFailMalformedHLTC message from an incoming peer. Rather than fail
the Chanel if they give us an invalid failure code, we’ll instead treat
it as a temporary channel failure so we can continue to forward the
error.
This commit is a follow up to a prior commit which skipped sending the
commitment sig message (and sending out the update fee) message if the
channel wasn’t yet able to forward any HTLC’s. We’ll modify the prior
commit to not add the fee update to the channel at all. Otherwise, we
risk a state desynchronization.
This commit adds a check to `updateChannelFee` which skipssending the
`update_fee` message when the channel is not eligable for forwarding
messages (likely due to the channel's `RemoteNextRevocation` not yet
being set).
This addresses #470.
In this commit, we add a new method to the ChanneLink interface:
EligibleToForward. This method allows a link to be added to the switch,
but in an intermediate state which indicates that it isn’t yet ready to
forward any incoming HTLC’s.
In this commit we add a new case to the main select statement within a
channel link. This select statement will serve as a Sipping Bird which
will check the network fee rate (as returned by the fee estimator) and
compare that to the fee on the commitment transaction. Using the
shouldAdjustCommitFee function, we determine if we should update the
commitment fee. If so, then we’ll send an UpdateFee message and also
trigger a new commitment update.
We also add a new unit test: TestChannelLinkUpdateCommitFee to ensure
that we update the fee accordingly if the fee increases or decreases by
a large portion.
In this commit, we add a new helper function to the link which will be
utilized in a later commit. This helper function will help us determine
if we should update the commitment fee, in response to a change in the
network fee return by our fee estimators.
In this commit we add a quit case to the select statement that’s
entered once a link is created. Before this commit, upon restart it
would be possible that the deamon would never ben able to shutdown as
the link would be waiting for the messages to be sent by the other
side.
In this commit, we’ve re-written the process of syncing the state of
channels after we reconnect. This re-write ensure correctness, and also
simplified the existing logic which would attempt to launch another
goroutine to handle requests from the switch to ensure that it doesn’t
block. This is no longer necessary as the AddPacket method that the
switch indirectly calls is non-blocking.
In this commit, we modify the existing implementation of the
Bandwidth() method on the default ChannelLink implementation to use
much tighter accounting. Before this commit, there was a bug wherein if
the link restarted with pending un-settled HTLC’s, and one of them was
settled, then the bandwidth wouldn’t properly be updated to reflect
this fact.
To fix this, we’ve done away with the manual accounting and instead
grab the current balances from two sources: the set of active HTLC’s
within the overflow queue, and the report from the link itself which
includes the pending HTLC’s and factors in the amount we’d need to (or
not need to) pay in fees for each HTLC.
In this commit, we’ve modified the link and the switch to start to use
the new mailBox in place of the existing synchronous message send
directly into the link’s upstream/downstream channels. With his change,
we no longer need to spawn a new goroutine each time an HTLC needs to
be forwarded, or a user payment is initiated.
After addition of the retransmission logic in the channel link, we
should make the onion blobs persistant, the proper way to do this is
include the onion blobs in the payment descriptor rather than storing
them in the distinct struct in the channel link.
In this commit BOLT№2 retranmission logic for the channel link have
been added. Now if channel link have been initialised with the
'SyncState' field than it will send the lnwire.ChannelReestablish
message and will be waiting for receiving the same message from remote
side. Exchange of this message allow both sides understand which
updates they should exchange with each other in order sync their
states.
In order to be able to properly restart switch several times we should
have the sequential process of channel link stop. In other words if we
stopped the switch we should be sure that all channel links have been
stopped too. Addition of the goroutine during the force close was added
because of the deadlock:
Trace:
1. link:force_close_notification
2. link:wipe_channel
3. peer:switch_remove_link
4. switch:stop_link
5. link:wait <-- deadlock
In this commit, we relax the time lock verification when we realize
we’re an intermediate hop. We no longer directly assert that the time
lock we receive is _identical_, instead we allow slow slack and will
reject iff, the incoming timelock minus the outgoing time lock doesn’t
meet our delta requirements.
This commit renames the Deobfuscator interface to ErrorDecrypter and
the Obfuscator interface to ErrorEncrypter. With this rename, the
purpose of these two interfaces are a bit clearer.
Additionally, DecryptError (which was formerly Deobfuscate) now
directly returns an ForwardingError type instead of the
lnwire.FailureMessage.
This commit fixes an existing bug, wherein if we failed to account for
the fact that if we we’re unable to add an HTLC for any reason other
than an overflown commitment transaction, then we wouldn’t properly
re-add the available bandwidth of the offending HTLC.
In the case where the channelLink get started and the number of
updates on this channel is zero, this means no paymenys has been
done using this channel. This might mean that the fundingLocked
never was sent successfully, so we resend to make sure this
channel gets opened correctly.
This commit fixes a bug related to swallowing an error that should go
to the switch in the case of an insufficient balance error when
attempting to add a new HTLC to the channel state machine. In this
case, an error would never be returned back to the client/switch, and
the internal processing within the channelLink would loop forever,
attempting to add an HTLC that can’t be added due to insufficient
balance to state machine itself.
We fix this issue by only treating the lnwallet.ErrMaxHTLCNumber as the
only error that prompts adding an HTLC to the overflow queue rather
than sending the error directly back to the switch.
In this commit, we’ve moved away from the internal queryHandler within
the packetQueue entirely. We now use an internal queueLen variable
internally to allow callers to sample the queue’s size, and also for
synchronization purposes internally.
This commit also introduces a chan struct{} (freeSlots) that is used
internally as a semaphore. The current value of freeSlots reflects the
number of available slots within the commitment transaction. Within the
link, after an HTLC has been removed/modified, then a “slot” is freed
up. The main packetConsumer then interprets these messages as a signal
to attempt to free up a new slot within the queue itself by dumping off
to the commitment transaction.
This commit modifies the way the bandwidth of a given channel link is
tracked, and reported externally. The prior approach pushed most of the
logic for tracking channel bandwidth into the link itself, and relied
on a report from the queue in order to determine the total available
bandwidth. This approach at times could inadvertently introduce
deadlocks when working on new features as since the query was handled
internally, it required the link to be _active_ and non-blocked in
order to respond to.
We’ve now abandoned this approach in favor of lifting the bandwidth
accounting to the highest possible abstraction layer within the link
itself. We now maintain a availableBandwidth integer that’s used
atomically within the link in response to: us adding+settling an HTLC,
and the remote party failing one of our HTLC’s.
This commit completes a full re-write of the link’s packet overflow
queue with the goals of the making the code itself more understandable
and also allowing it to be more extensible in the future with various
algorithms for handling HTLC congestion avoidance and persistent queue
back pressure.
The new design is simpler and consumes much less coroutines (no longer
a new goroutine for each active HLTC). We now implement a simple
synchronized queue using a standard condition variable.
This commit adds a new debug mode for lnd
called hodlhtlc. This mode instructs a node
to refrain from settling incoming HTLCs for
which it is the exit node. We plan to use
this in testing to more precisely control
the states a node can take during
execution.
This commit fixes an existing bug in the way we perform validation of
the timelock information as the final hop in the route. Previously, we
would assert that the outgoing time lock in the per-hop payload would
exactly match our time lock delta.
Instead, we should be asserting two things:
1. That the time lock in the payload is >= the expected time lock
2. That timeout on the HTLC is exactly equal to the payload
This commit adds a new method to the HtlcSwitch:
UpdateForwardingPolicies. With this method callers are now able to
modify the forwarding policies of all, or some currently active links.
We also make a slight modification to the way that forwarding policy
updates are handled within the links themselves to ensure that we don’t
override with a zero value for any of the fields.
This commit implements a missing policy within the current ChannelLink
interface. If an HTLC arrives that is too close to the current block
height, then we’ll reject it. As otherwise, it may be possible for us
to lose an on-chain claim if they HTLC expires already or expires
before we’re able to get a commitment transaction in the chain.
As the exit node, we have a grace period that governs out decision. As
an intermediate node, we ensure that the HTLC isn’t close to expiry on
our outgoing link end if we forward it.
We can safely remove the initial revocation window extension as this
has gone away with the new state machine. We instead now just fill the
window once the channel has been opened, and then maintain a fixed
window size of 2 from there on.
Within the network, it's important that when an HTLC forwarding failure
occurs, the recipient is notified in a timely manner in order to ensure
that errors are graceful and not unknown. For that reason with
accordance to BOLT №4 onion failure obfuscation have been added.
This commit fixes a regression introduce in the prior commit which
added full verification of the per-hop payloads to the ChannelLink
interface. When this was initially implemented, the added checks
weren’t guarded on the existence of debughtlc’s. As a result,
debughtlc’s would be rejected as they don’t match the expected invoice
value.
This commit fixes that issue by only checking the hop payload if debug
HTLC mode isn’t on.
This commit adds a new method to the ChannelLink interface which is
meant to allow outside sub-system to update the forwarding policy of a
channel. This can be triggered either by a new RPC method, or
automatically by some sort of control system which seeks to optimize
fee revenue, or block off channels, etc.
This commit puts a missing piece in place by properly parsing and
validating the per hop payload received in incoming HTLC’s. When
forwarding HTLC’s we ensure that the payload recovered is consistent
with our current forwarding policy. Additionally, when we’re the “exit
node” for a payment, then we ensure that the HTLC extended matches up
with our expectation w.r.t the payment amount to be received.
This commit fixes a slight regression in the logic of the switch by
ensuring that the log commitment timer is only start _after_ we receive
a new commitment signature. Otherwise, the ticker will keep ticking and
possibly settle HTLC’s that’ve yet to be locked in, or waste a
signature causing us to be deprived of a revocation which is required
for us to initiate a new state transition.
Additionally, the commit performs a few minor post-merge clean ups.
This commit fixes an issue that would at times cause the htlcManager
which manages the link that’s the final hop to settle in an HTLC flow.
Previously, a case would arise wherein a set of HTLC’s were settled to,
but not properly committed to in the commitment transaction of the
remote node. This wasn’t an issue with HTLC’s which were added but
uncleared, as that batch was tracked independently.
In order to fix this issue, we now track pending HTLC settles
independently. This is a temporary fix, as has been noted in a TODO
within this commit.
In this commit usage of the pending packet queue have been added.
This queue will consume the downstream packets if state machine return
the error that we do not have enough capacity for htlc in commitment
transaction. Upon receiving settle/fail payment descriptors - add htlc
have been removed, we release the slot, and process pending add htlc
requests.
Because processing of onion blob have been moved in another place we
could get rid of the variables which are not needed any more.
NOTE: pendingBatch have been replaced with batchCounter variable, but
it should be removed at all, because number of pending batch updates
might be counted by the state machine itself.
Step №4 in making htlcManager (aka channelLink) testable:
This step consist of two:
1. Start using the hop iterator abstraction, the concrete
implementation of which will be added later, basically it will we the
same sphinx onion packet processor, but wrapped in hop iterator
abstraction.
2. The RevokAndAck processing part have been replaced by the
"processLockedInHtlcs" function which implement the same logic, but make
it a bit simpler.
Such changes will allow as to get rid of the the unnecessary variables.
Short: such abstraction give as ability to test the channel link in the
future.
Long: hop iterator represents the entity which is be able to give payment
route hop by hop. This interface will be used to have an abstraction
over the algorithm which we use to determine the next hope in htlc route
and also helps the unit test to create mock representation of such
algorithm which uses simple array of hops.
Step №2 in making htlcManager (aka channelLink) testable:
Implement the ChannelLink interface which is needed to use it in pair
with htlc switch. With this commit channel link impelements interface,
but isn't able to operate properly yet.
In this commit all initial code which will be transformed into channel
link have been added. Rather than changing the in the same commit is
better to create the standalone commit, in order to see the changes
which have been applied to relocated code.
