In this commit, we convert all the `Update` calls which are serial, to
use `Batch` calls which are optimistically batched together for
concurrent writers. This should increase performance slightly during the
initial graph sync, and also updates at tip as we can coalesce more of
these individual transactions into a single transaction.
If the max_htlc field is not found when fetching a ChannelEdgePolicy
from the DB, we treat this as an unknown policy.
This is done to ensure we won't propagate invalid data further. The data
will be overwritten with a valid one when we receive an update for this
channel.
It shouldn't be very common, but old data could be lingering in the DB
added before this field was validated.
Adding this field will allow us to persist an edge's
max HTLC to disk, thus preserving it between restarts.
Co-authored-by: Johan T. Halseth <johanth@gmail.com>
In this commit:
* we partition lnwire.ChanUpdateFlag into two (ChanUpdateChanFlags and
ChanUpdateMsgFlags), from a uint16 to a pair of uint8's
* we rename the ChannelUpdate.Flags to ChannelFlags and add an
additional MessageFlags field, which will be used to indicate the
presence of the optional field HtlcMaximumMsat within the ChannelUpdate.
* we partition ChannelEdgePolicy.Flags into message and channel flags.
This change corresponds to the partitioning of the ChannelUpdate's Flags
field into MessageFlags and ChannelFlags.
Co-authored-by: Johan T. Halseth <johanth@gmail.com>
Instead return ErrGraphNodeNotFound directly. If the node bucket was
created it would be empty, and the call delChannelByEdge ->
fetchChanEdgePolicies -> fetchChanEdgePolicy ->
deserializeChanEdgePolicy -> fetchLightningNode would return this error
anyway.
In this commit, we add a method to the ChannelGraph struct that
determines whether a node is seen as public based on graph's source
node's point of view.
In this commit, we account for the additional case wherein the
announcement hasn't yet been written with the extra zero byte to
indicate that there aren't any remaining bytes to be read. Before this
commit, we accounted for the case where the announcement was written
with the extra byte, but now we ensure that legacy nodes that upgrade
will be able to boot properly.
In this commit, we add a new limit on the largest number of extra opaque
bytes that we'll allow to be written per vertex/edge. We do this in
order to limit the amount of disk space that we expose, as it's possible
that nodes may start to pad their announcements adding an additional
externalized cost as nodes may need to continue to store and relay these
large announcements.
In this commit, we add a mirror set of fields to the ones we recently
added to the set of gossip wire messages. With these set of fields in
place, we ensure that we'll be able to properly store and re-validate
gossip messages that contain a set of extra/optional fields.
In this commit, we ensure that we de-duplicate the set of channel edges
returned from ChanUpdatesInHorizon. Other subsystems within lnd use this
method to retrieve and send all the channels with updates within a time
series to network peers. However, since the method looks at the edge
update index, which can include up to two entries per edge for each
policy, it's possible that we'd send channel announcements and updates
twice, causing extra bandwidth.
In this commit, we fix a lingering issue within the edge update index
where entries were not being properly pruned due to an incorrect
calculation of the offset of an edge's last update time. Since the
offset is being determined from the end to the start, we need to
subtract all the fields after an edge policy's last update time from the
total amount of bytes of the serialized edge policy to determine the
correct offset. This was also slightly off as the edge policy included
an extra byte, which has been fixed in the previous commit.
Instead of continuing the slicing approach however, we'll switch to
deserializing the raw bytes of an edge's policy to ensure this doesn't
happen in the future when/if the serialization methods change or extra
data is included.
In this commit, we fix an off-by-one error when slicing the public key
from the serialized node info byte slice. This would cause us to write
an extra byte to all edge policies. Even though the values were read
correctly, when attempting to calculate the offset of an edge's update
time going backwards, we'd always be incorrect, causing us to not
properly prune the edge update index.
In this commit we fix a minor logging artifact. After the switch to
EdgePoint, the FilteredChainView implementations will try to log the
struct directly, as prior they would have an outpoint object. We restore
this behavior by adding a String() method to EdgePoint which will simply
proxy through to the outpoint so we can log that directly.
In this commit, we add a new method to the ChannelEdgeInfo that will
allow the path finding logic to get the node opposite the pivot node
without first creating a new db transaction. The new method is able to
use an existing db transaction, or create a new one if needed.
The commit ensures that for every channel, there will always
be two entries in the edges bucket. If the policy from one or
both ends of the channel is unknown, it is marked as such.
This allows efficient lookup of incoming edges. This is
required for backwards payment path finding.
In this commit, we update the ChannelView method to be compatible with
the new set of interfaces that require the script to be passed in in
addition to the outpoint. In order to do this, we introduce a new
EdgePoint struct which packages together a channel point along with the
funding pkScript. Along the way, we've copied over a utility method from
the lnwallet package to avoid having to deal with an import cycle.
In this commit, we fix a slight race condition that can occur when we go
to add a shell node for a node announcement, but then right afterwards,
a new block arrives that causes us to prune an unconnected node. To
ensure this doesn't happen, we now add shell nodes within the same db
transaction as AddChannelEdge. This ensures that the state is fully
consistent and shell nodes will be added atomically along with the new
channel edge.
As a result of this change, we no longer need to add shell nodes within
the ChannelRouter, as the database will take care of this operation as
it should.
In this commit, we fix an existing bug in the pruneGraphNodes method.
Before this commit, if a node was involved in a channel, but only one of
the edges was advertised, then either it, or the other node would be
erroneously pruned from the graph. They shouldn't be pruned as there's
still an edge connecting the, although only 1/2 of the edge is actually
advertised.
In order to fix this, we'll now do two passes: the first pass will
populate a ref count map of all known nodes in the graph, the second
pass will increment the ref count each time a node is found in the
graph. With this two pass method, we ensure that nodes are only deleted
if there are absolutely no edges pointing to them within the graph.
In this commit, we fix an existing bug in the new graph query sync
feature. Before this commit, when a block is pruned, we would never
actually delete the update index entries. This is due to the fact that
we would attempt to delete the entries from the update index _after_ we
had already removed the edges from the update index.
We fix this by simply swapping the order: first we delete from the
update index, then we delete the edges themselves. A test ensuring that
the entires are cleared (which failed before this commit), has been
added.
In this commit, we ensure that all indexes for a particular channel have
any relevant keys deleted once a channel is removed from the database.
Before this commit, if we pruned a channel due to closing, then its
entry in the channel update index would ever be removed.
In this commit, we add a series of methods, and a new database index
that we'll use to implement the new discovery.ChannelGraphTimeSeries
interface interface. The primary change is that we now maintain two new
indexes tracking the last update time for each node, and the last update
time for each edge. These two indexes allow us to implement the
NodeUpdatesInHorizon and ChanUpdatesInHorizon methods. The remaining
methods added simply utilize the existing database indexes to allow us to
respond to any peer gossip range queries.
A set of new unit tests has been added to exercise the added logic.
