In this commit, we ensure that we de-duplicate the set of channel edges
returned from ChanUpdatesInHorizon. Other subsystems within lnd use this
method to retrieve and send all the channels with updates within a time
series to network peers. However, since the method looks at the edge
update index, which can include up to two entries per edge for each
policy, it's possible that we'd send channel announcements and updates
twice, causing extra bandwidth.
In this commit, we fix a lingering issue within the edge update index
where entries were not being properly pruned due to an incorrect
calculation of the offset of an edge's last update time. Since the
offset is being determined from the end to the start, we need to
subtract all the fields after an edge policy's last update time from the
total amount of bytes of the serialized edge policy to determine the
correct offset. This was also slightly off as the edge policy included
an extra byte, which has been fixed in the previous commit.
Instead of continuing the slicing approach however, we'll switch to
deserializing the raw bytes of an edge's policy to ensure this doesn't
happen in the future when/if the serialization methods change or extra
data is included.
In this commit, we fix an off-by-one error when slicing the public key
from the serialized node info byte slice. This would cause us to write
an extra byte to all edge policies. Even though the values were read
correctly, when attempting to calculate the offset of an edge's update
time going backwards, we'd always be incorrect, causing us to not
properly prune the edge update index.
In this commit we fix a minor logging artifact. After the switch to
EdgePoint, the FilteredChainView implementations will try to log the
struct directly, as prior they would have an outpoint object. We restore
this behavior by adding a String() method to EdgePoint which will simply
proxy through to the outpoint so we can log that directly.
In this commit, we add a new method to the ChannelEdgeInfo that will
allow the path finding logic to get the node opposite the pivot node
without first creating a new db transaction. The new method is able to
use an existing db transaction, or create a new one if needed.
The commit ensures that for every channel, there will always
be two entries in the edges bucket. If the policy from one or
both ends of the channel is unknown, it is marked as such.
This allows efficient lookup of incoming edges. This is
required for backwards payment path finding.
In this commit, we update the ChannelView method to be compatible with
the new set of interfaces that require the script to be passed in in
addition to the outpoint. In order to do this, we introduce a new
EdgePoint struct which packages together a channel point along with the
funding pkScript. Along the way, we've copied over a utility method from
the lnwallet package to avoid having to deal with an import cycle.
In this commit, we fix a slight race condition that can occur when we go
to add a shell node for a node announcement, but then right afterwards,
a new block arrives that causes us to prune an unconnected node. To
ensure this doesn't happen, we now add shell nodes within the same db
transaction as AddChannelEdge. This ensures that the state is fully
consistent and shell nodes will be added atomically along with the new
channel edge.
As a result of this change, we no longer need to add shell nodes within
the ChannelRouter, as the database will take care of this operation as
it should.
In this commit, we fix an existing bug in the pruneGraphNodes method.
Before this commit, if a node was involved in a channel, but only one of
the edges was advertised, then either it, or the other node would be
erroneously pruned from the graph. They shouldn't be pruned as there's
still an edge connecting the, although only 1/2 of the edge is actually
advertised.
In order to fix this, we'll now do two passes: the first pass will
populate a ref count map of all known nodes in the graph, the second
pass will increment the ref count each time a node is found in the
graph. With this two pass method, we ensure that nodes are only deleted
if there are absolutely no edges pointing to them within the graph.
In this commit, we fix an existing bug in the new graph query sync
feature. Before this commit, when a block is pruned, we would never
actually delete the update index entries. This is due to the fact that
we would attempt to delete the entries from the update index _after_ we
had already removed the edges from the update index.
We fix this by simply swapping the order: first we delete from the
update index, then we delete the edges themselves. A test ensuring that
the entires are cleared (which failed before this commit), has been
added.
In this commit, we ensure that all indexes for a particular channel have
any relevant keys deleted once a channel is removed from the database.
Before this commit, if we pruned a channel due to closing, then its
entry in the channel update index would ever be removed.
In this commit, we add a series of methods, and a new database index
that we'll use to implement the new discovery.ChannelGraphTimeSeries
interface interface. The primary change is that we now maintain two new
indexes tracking the last update time for each node, and the last update
time for each edge. These two indexes allow us to implement the
NodeUpdatesInHorizon and ChanUpdatesInHorizon methods. The remaining
methods added simply utilize the existing database indexes to allow us to
respond to any peer gossip range queries.
A set of new unit tests has been added to exercise the added logic.
In this commit, we made a series of modification to the way we handle
reading edges and vertexes from disk, in order to reduce the amount of
garbage generated:
1. Properly use PubKeyBytes are required rather than PubKey()
2. Return direct structs rather than pointers, and leave it to the
runtime to perform escape analysis.
3. In-line the former readSig() method when reading sigs from disk.
In this commit, we make an API change that’s meant to reduce the amount
of garbage we generate when doing pathfinding or syncing nodes with our
latest graph state. Before this commit, we would always have to fully
decode the public key and signatures when reading a edge or vertex
struct. For the edges, we may need several EC operations to fully
decode all the pubkeys. This has been seen to generate a ton of
garbage, as well as slow down path finding a good bit.
To remedy this, we’ll now only ever read the *raw* bytes from disk. In
the event that we actually need to verify a signature (or w/e), only
*then* will we fully decode everything.
Before this commit, we’d unnecessarily use a write transaction within
the FetchChannelEdgesByOutpoint. This is wasteful as the function only
actually reads items from the database, and doesn’t attempt any
mutations at all.
This commit adds the method DisconnectBlockAtHeight to the channel
graph database, making it possible to "rewind" the database in case
a block is disconnected from the main chain. To accomplish this,
a prune log is introduced, making it possible to keep track of the
point in time where the database was pruned. This is necessary for
the case where lnd might wake up on a stale branch, and must "walk
backwards" on this branch after it finds a common block fro the
graph database and the new main chain.
This commit modifies the recently modified logic for self-channel
retransmission to exclude pruning *our* channels which haven’t been
updated since the broadcastInterval. Instead, we only re-broadcast
channels of ours that haven’t been updated in 24 hours.
This commit modifies the node.ForEachChannel to *also* return the
incoming edge as well as the outgoing edge. We make this modification
as when we’re doing path finding, we need to return the incoming edge
as well, since we need to use the to properly compute the time lock and
fees for transit on that edge.
Adds a HaveNodeAnnouncement field to the LightningNode
struct, which is used to indicate if we have gotten
all the necessary information to fill the remaining
fields in the struct. If we haven't gotten a node
announcement for this specific node, then we only
know the pubkey, and can only fill that field in
the struct. Still, we should be able to add it to the
channel graph and use it for routes, as long as we
know about channels to this node.
This commit introduces a new method to the ChannelGraph struct:
ChannelView. This struct returns all the outpoints that represent the
set of active channels within the network. The set of items returned by
this new method will possibly shrink with each call to `PruneGraph`,
and possibly be expanded by each call to `AddChannelEdge`.
The graph pruning tests have been updated to ensure the description
above holds true.
