In this commit, we fix an existing bug in the new graph query sync
feature. Before this commit, when a block is pruned, we would never
actually delete the update index entries. This is due to the fact that
we would attempt to delete the entries from the update index _after_ we
had already removed the edges from the update index.
We fix this by simply swapping the order: first we delete from the
update index, then we delete the edges themselves. A test ensuring that
the entires are cleared (which failed before this commit), has been
added.
In this commit, we ensure that all indexes for a particular channel have
any relevant keys deleted once a channel is removed from the database.
Before this commit, if we pruned a channel due to closing, then its
entry in the channel update index would ever be removed.
In this commit, we add a series of methods, and a new database index
that we'll use to implement the new discovery.ChannelGraphTimeSeries
interface interface. The primary change is that we now maintain two new
indexes tracking the last update time for each node, and the last update
time for each edge. These two indexes allow us to implement the
NodeUpdatesInHorizon and ChanUpdatesInHorizon methods. The remaining
methods added simply utilize the existing database indexes to allow us to
respond to any peer gossip range queries.
A set of new unit tests has been added to exercise the added logic.
In this commit, we made a series of modification to the way we handle
reading edges and vertexes from disk, in order to reduce the amount of
garbage generated:
1. Properly use PubKeyBytes are required rather than PubKey()
2. Return direct structs rather than pointers, and leave it to the
runtime to perform escape analysis.
3. In-line the former readSig() method when reading sigs from disk.
In this commit, we make an API change that’s meant to reduce the amount
of garbage we generate when doing pathfinding or syncing nodes with our
latest graph state. Before this commit, we would always have to fully
decode the public key and signatures when reading a edge or vertex
struct. For the edges, we may need several EC operations to fully
decode all the pubkeys. This has been seen to generate a ton of
garbage, as well as slow down path finding a good bit.
To remedy this, we’ll now only ever read the *raw* bytes from disk. In
the event that we actually need to verify a signature (or w/e), only
*then* will we fully decode everything.
Before this commit, we’d unnecessarily use a write transaction within
the FetchChannelEdgesByOutpoint. This is wasteful as the function only
actually reads items from the database, and doesn’t attempt any
mutations at all.
This commit adds the method DisconnectBlockAtHeight to the channel
graph database, making it possible to "rewind" the database in case
a block is disconnected from the main chain. To accomplish this,
a prune log is introduced, making it possible to keep track of the
point in time where the database was pruned. This is necessary for
the case where lnd might wake up on a stale branch, and must "walk
backwards" on this branch after it finds a common block fro the
graph database and the new main chain.
This commit modifies the recently modified logic for self-channel
retransmission to exclude pruning *our* channels which haven’t been
updated since the broadcastInterval. Instead, we only re-broadcast
channels of ours that haven’t been updated in 24 hours.
This commit modifies the node.ForEachChannel to *also* return the
incoming edge as well as the outgoing edge. We make this modification
as when we’re doing path finding, we need to return the incoming edge
as well, since we need to use the to properly compute the time lock and
fees for transit on that edge.
Adds a HaveNodeAnnouncement field to the LightningNode
struct, which is used to indicate if we have gotten
all the necessary information to fill the remaining
fields in the struct. If we haven't gotten a node
announcement for this specific node, then we only
know the pubkey, and can only fill that field in
the struct. Still, we should be able to add it to the
channel graph and use it for routes, as long as we
know about channels to this node.
This commit introduces a new method to the ChannelGraph struct:
ChannelView. This struct returns all the outpoints that represent the
set of active channels within the network. The set of items returned by
this new method will possibly shrink with each call to `PruneGraph`,
and possibly be expanded by each call to `AddChannelEdge`.
The graph pruning tests have been updated to ensure the description
above holds true.
This commit modifies the ForEachNode on the ChannelGraph and
ForEachChannel on the LightningNode struct to accept a database
transaction as its first argument. With this change, it’ll now be
possible to implement graph traversals that typically required a nested
loop with all the vertex loaded into memory using the callback API
instead:
c.ForEachNode(nil, func(tx, node) {
node.ForEachChannel(tx, func(…) {
})
})
Originally we adding the edge without proof in order to able to use it
for payments path constrcution. This method will allow us to populate
the announcement proof after the exchange of the half proofs and
constrcutrion of full proof finished.
In case if the channel shouldn't be announced to the rest of the network
the proof, which is needed to announce the channel, will not be
populated, fot that reason the ability to store the empty proof has
been added.
This commit modifies address handling in the NodeAnnouncement struct,
switching from net.TCPAddr to []net.Addr. This enables more flexible
address handling with multiple types and multiple addresses for each
node. This commit addresses the first part of issue #131 .
This commit modifies the path finding routines to properly use the new
channel edge related API exposed by the database. Additionally, a new
type `ChannelHop` has been introduced which couples an edges routing
policy with the capacity and origin chain of the channel.
This commit alters the return value of PrunedGraph to me a bit more
useful: the function now returns all the channels that were closed when
processing the block (slice of spent outpoints). With this information,
callers gain greater visibility into exactly which channels were
closed. This can be used in higher levels to present detailed summaries
of how blocks affect closed channels.
This commit splits the prior ChannelEdge struct into two distinct
structs: ChannelEdgeInfo and ChannelEdgePolicy. The info structs stores
the _full_ information that was used to advertise the channel, while
the policy struct contains the information that’s needed in order to
use the information for routing purposes.
With this split we can eliminate a number of hacks within the rest of
the codebase that were added as a result of data unavailability if one
or neither edge was present.
Finally a bit of field renaming has taken place (Exipiry ->
TimeLockDelta), etc.
This commit fixes a panic that would arise when the daemon attempts to
query for a channel that doesn’t currently exist. The bug was the
result of a typo which checked for the nil existence of the incorrect
variable.
This commit fixes a prior bug in the graph database due to an invalid
assumption that both channel edges would _always_ be advertised. This
assumption is invalid, as it’s up to a node’s policy if the advertise
their direction of the channel.
The fix for this assumption is straight forward: ErrEdgeNotFound is no
longer a critical error, instead a nil pointer will now be passed into
the passed callback function.
This commit makes a large number of minor changes concerning API usage
within the deamon to match the latest version on the upstream btcsuite
libraries.
The major changes are the switch from wire.ShaHash to chainhash.Hash,
and that wire.NewMsgTx() now takes a paramter indicating the version of
the transaction to be created.
This commit modifies the error propagating behavior within the
HasChannelEdge struct. Rather than exiting the function early when a
single edge isn’t found, we instead continue to also possibly retrieve
the second directional edge.
With this change, we avoid a potential infinite gossiping loop in the
routing package that would result if we’d seen one edge but not the
other. In this case the timestamps returned for *both* edges would
always be zero, causing the daemon to always accept and rebroadcast the
announcement putting all other connected lnd’s into the same loop.
This commit modifies the FetchChannelEdgesByID slightly to use a
read-only transaction rather than a write-only transaction. As a result
we’ll no longer extraneously consume a writer’s slot when we’re only
reading data from the database.
This commit modifies the HasChannelEdge function to _always_ return
true if we know of the channel edge, meaning that it was previously
added on announce.
This change fixes a minor bug present in the code which would result in
extraneous re-transmissions of updates received by the new routing
package.
This commit adds a utility method whcih utilizes the edge index bucket
and allows caller to look up the channel ID of a channel by its funding
outpoint. This can be used to populate RPC’s with additional
information and also to provide users with an additional query
interface to build channel explorers, etc.
This commit fixes a minor bug in the ForEachChannel method of the
ChannelGraph struct. Rather than ErrGraphNoEdgesFound being returned if
either of the edge related buckets hadn’t been created yet,
ErrGraphNodesNotFound was being returned.
To fix this bug, we now properly return ErrGraphNoEdgesFound.
Additionally a mental note to roasbeef has been left as the current
code currently assumes that eventually both directions of the channel
edge will be advertised. However, this may not necessarily be the case
in a live network, since a side chooses to preferentially advertise a
channel or not.
This commit adds to new functions to the ChannelGraph struct which
allow the callers to query for the existence or non-existence of a
vertex (node) or edge (channel) within the graph. In addition to
returning whether the edge exists, the functions will also return the
last time the state has been modified for the edge or vertex. This will
allow callers to ensure that only the most up to date state is
committed to disk.
This commit adds support for channel graph pruning, which is the method
used to keep the channel graph in sync with the current UTXO state. As
the channel graph is essentially simply a subset of the UTXO set, by
evaluating the channel graph with the set of outfits spent within a
block, then we’re able to prune channels that’ve been closed by
spending their funding outpoint. A new method `PruneGraph` has been
provided which implements the described functionality.
Upon start up any upper routing layers should sync forward in the chain
pruning the channel graph with each newly found block. In order to
facilitate such channel graph reconciliation a new method `PruneTip`
has been added which allows callers to query current pruning state of
the channel graph.
This commit modifies the LightningNode.ForEachChannel method to give
the caller the option of re-using an existing database transaction
instead of always creating a new db transaction with each invocation.
Internally boltdb will run into an error/dead-lock if a nested
transaction is attempted.
Such an action might be attempted if one were to use the traversal
functions in a path finding algorithm. Therefore in order to avoid
that after, we now allow the re-use of transactions to facilitate
nested calls to ForEachChannel.
This commit introduces a new capability to the database: storage of an
on-disk directed channel graph. The on-disk representation of the graph
within boltdb is essentially a modified adjacency list which separates
the storage of the edge’s existence and the storage of the edge
information itself.
The new objects provided within he ChannelGraph carry an API which
facilitates easy graph traversal via their ForEach* methods. As a
result, path finding algorithms will be able to be expressed in a
natural way using the range methods as a for-range language extension
within Go.
Additionally caching will likely be added either at this layer or the
layer above (the RoutingManager) in order keep queries and outgoing
payments speedy. In a future commit a new set of RPC’s to query the
state of a particular edge or node will also be added.
