Commit Graph

74 Commits

Author SHA1 Message Date
Wilmer Paulino
a4f33ae63c
discovery: adhere to proper channel chunk splitting for ReplyChannelRange 2020-12-08 15:18:07 -08:00
Conner Fromknecht
edaa5d4308
channeldb/graph: add batched APIs for graph ingestion 2020-11-24 16:39:28 -08:00
Andras Banki-Horvath
d89f51d1d0
multi: add reset closure to kvdb.Update
Similarly as with kvdb.View this commits adds a reset closure to the
kvdb.Update call in order to be able to reset external state if the
underlying db backend needs to retry the transaction.
2020-11-05 17:57:12 +01:00
Andras Banki-Horvath
2a358327f4
multi: add reset closure to kvdb.View
This commit adds a reset() closure to the kvdb.View function which will
be called before each retry (including the first) of the view
transaction. The reset() closure can be used to reset external state
(eg slices or maps) where the view closure puts intermediate results.
2020-11-05 17:57:12 +01:00
Andras Banki-Horvath
cbce8e8872
channeldb: move makeTestDB out of test to make it available for other tests
This commit moves makeTestDB to db.go and exports it so that we'll be
able to use this function in other unit tests to make them testable with
etcd if needed.
2020-07-28 17:57:29 +02:00
Conner Fromknecht
d0d2ca403d
multi: rename ReadTx to RTx 2020-05-26 18:20:37 -07:00
Conner Fromknecht
90d3bf532d
channeldb+routing: remove tx arg for ForEachNode 2020-05-12 17:53:20 -07:00
Olaoluwa Osuntokun
f0911765af
channeldb: convert to uniformly use new kvdb abstractions
In this commit, we migrate all the code in `channeldb` to only reference
the new `kvdb` package rather than `bbolt` directly.

In many instances, we need to add two version to fetch a bucket as both
read and write when needed. As an example, we add a new
`fetchChanBucketRw` function. This function is identical to
`fetchChanBucket`, but it will be used to fetch the main channel bucket
for all _write_ transactions. We need a new method as you can pass a
write transaction where a read is accepted, but not the other way around
due to the stronger typing of the new `kvdb` package.
2020-03-18 19:34:49 -07:00
Joost Jager
95ddab57fa
channeldb: add tx argument for FetchLightningNode
To allow execution within an existing tx.
2019-12-20 10:17:01 +01:00
Joost Jager
37d9ee302c
channeldb: take serialized key to fetch lightning node
This prevents inefficient key conversions in a follow up commit that
change the inner pathfinding loop.
2019-12-19 12:44:00 +01:00
Conner Fromknecht
5e27b5022c
multi: remove LocalFeatures and GlobalFeatures 2019-11-08 05:32:00 -08:00
Joost Jager
a0e91b5d41
channeldb: add forward fee calculation function 2019-09-17 11:11:02 +02:00
Joost Jager
3d7de2ad39
multi: remove dead code 2019-09-10 17:21:59 +02:00
Olaoluwa Osuntokun
8c389d13f9
Merge pull request #3197 from breez/optimize_prune_zombie_channels
Optimize prune zombie channels
2019-07-18 20:56:56 -07:00
Conner Fromknecht
0223039fb4
channeldb/graph: add NumZombies method and tests 2019-07-16 16:32:27 -07:00
Roei Erez
7300f33fe2 graph: implement index for disabled channels.
This commit adds an index bucket, disabledEdgePolicyBucket, for those
ChannelEdgePolicy with disabled bit on.
The main purpose is to be able to iterate over these fast when prune is
needed without the need for iterating the whole graph.

The entry points for accessing this index are:
1. When updating ChannelEdgePolicy - insert an entry.
2. When deleting ChannelEdge - delete the associated entries.
3. When querying for disabled channels - implemented DisabledChannelIDs
function
2019-07-16 17:48:07 +03:00
Johan T. Halseth
21c989ffcc
channeldb/graph: don't nil curve of returned PubKey
Nilling the key would cause signature verification to crash if the
returned key was used.

A test to exercise the behavior is added.
2019-05-09 10:49:58 +02:00
Wilmer Paulino
0b0a9f4172
channeldb+routing: refactor DeleteChannelEdge to use ChannelID
In this commit, we refactor DeleteChannelEdge to use ChannelIDs rather
than ChannelPoints. We do this as the only use of DeleteChannelEdge is
when we are pruning zombie channels from our graph. When running under a
light client, we are unable to obtain the ChannelPoint of each edge due
to the expensive operations required to do so. As a stop-gap, we'll
resort towards using an edge's ChannelID instead, which is already
gossiped between nodes.
2019-04-18 21:57:41 -07:00
Conner Fromknecht
f82b7c9bac
channeldb/graph: remove MarkEdgeZombie
This commit removes the MarkEdgeZombie method from channeldb. This
method is currently not used in any live code paths in production, and
is only used in unit tests. However, incorrect usage of this method
could result in an edge being present in both the zombie and channel
indexes, which deviates from any state we would expect to see in
production. Removing the method will help mitigate the potential for
writing incorrect unit tests in the future, by forcing zombie edges to
be created via the relevant, production APIs, e.g. DeleteChannelEdge.

The existing unit tests that use this method have been modified to use
the DeleteChannelEdge instead. No regressions were discovered in the
process.
2019-04-09 22:12:02 -07:00
Conner Fromknecht
4a755435e6
channeldb/graph: skip unknown edges in FetchChanInfos
This commit modifies FetchChanInfos to skip any channels that are not in
the graph at the time of the call. Currently the entire call will fail
if the edge is not found, which stalls a gossip sync in the following
scenario:

 1. Remote peer queries for a channel range
 2. We return the set of channel ids in that range
 3. A channel from that set is removed from the graph, e.g. via close.
 4. Remote peer queries for removed edge, causing the query to fail.

To remedy this, we will now skip any edges that are not known in the
database at the time of the query. This prevents the syncer state
machines from halting, which otherwise could only be resolved by
disconnecting and reconnecting.
2019-04-09 17:35:58 -07:00
Conner Fromknecht
e91bacd1bc
channeldb/graph: filter zombie channels in FilterKnownChanIDs
This commit modifies FilterKnownChanIDs to skip edges that
we ourselves have deemed zombies. This prevents us from requesting
the updates from them, as this wastes bandwidth and cpu cycles.
2019-04-05 13:01:08 -07:00
Wilmer Paulino
c82d73a826
channeldb+routing: extend edge lookup methods with zombie index check
In this commit, we extend the graph's FetchChannelEdgesByID and
HasChannelEdge methods to also check the zombie index whenever the edge
to be looked up doesn't exist within the edge index. We do this to
signal to callers that the edge is known, but only as a zombie, and the
only information that we have about the edge are the node public keys of
the two parties involved in the edge.

In the event that an edge does exist within the zombie index, we make
an additional check on edge policies to ensure they are not within the
router's pruning window, indicating that it is a fresh update.
2019-03-27 13:06:57 -07:00
Wilmer Paulino
e98f4d6d9d
channeldb: extend DeleteChannelEdge to mark edge as zombie
We mark the edges as zombies when pruning them to ensure we don't
attempt to reprocess them later on. This also applies to channels that
have been removed from the graph due to being stale.
2019-03-27 13:06:34 -07:00
Wilmer Paulino
b780dfacdb
channeldb: add zombie edge index
In this commit, we add a zombie edge index to the database. This allows
us to quickly determine across restarts whether we're attempting to
process an edge we've previously deemed as zombie.
2019-03-27 13:06:12 -07:00
Johan T. Halseth
2b012b06a0
channeldb/graph test: add test for update policy for unknown edge 2019-02-14 14:21:18 +01:00
Johan T. Halseth
01e679786d
channeldb/graph_test: add TestEdgePolicyMissingMaxHtcl 2019-01-22 08:42:28 +01:00
Johan T. Halseth
8dd074ee57
channeldb/graph_test: assert MaxHTLC field correctness
Co-authored-by: Valentine Wallace <valentine.m.wallace@gmail.com>
2019-01-22 08:42:27 +01:00
Valentine Wallace
0fd6004958
multi: partition lnwire.ChanUpdateFlag into ChannelFlags and MessageFlags
In this commit:

* we partition lnwire.ChanUpdateFlag into two (ChanUpdateChanFlags and
ChanUpdateMsgFlags), from a uint16 to a pair of uint8's

* we rename the ChannelUpdate.Flags to ChannelFlags and add an
additional MessageFlags field, which will be used to indicate the
presence of the optional field HtlcMaximumMsat within the ChannelUpdate.

* we partition ChannelEdgePolicy.Flags into message and channel flags.
This change corresponds to the partitioning of the ChannelUpdate's Flags
field into MessageFlags and ChannelFlags.

Co-authored-by: Johan T. Halseth <johanth@gmail.com>
2019-01-22 08:42:26 +01:00
Olaoluwa Osuntokun
1fd3aac925
multi: switch from bolt packge to bbolt package for all imports 2018-11-29 20:33:49 -08:00
Wilmer Paulino
e795f7fce4
channeldb/graph: add method to determine if a node is public
In this commit, we add a method to the ChannelGraph struct that
determines whether a node is seen as public based on graph's source
node's point of view.
2018-10-24 17:28:53 -07:00
Olaoluwa Osuntokun
d8e6085c17
multi: update to go 1.11 gofmt 2018-10-07 14:37:38 +09:00
maurycy
ac24b12bf2
multi: fix various typos in comments 2018-09-07 06:51:49 +02:00
Olaoluwa Osuntokun
d2a7d910b7
Merge pull request #1825 from Roasbeef/extra-gossip-message-data
channeldb+discovery: ensure we store, validate and propagate announcements with opaque data
2018-09-05 17:53:02 -07:00
Olaoluwa Osuntokun
16b5a67c3a
Merge pull request #1842 from wpaulino/dedup-chan-updates-horizon
channeldb: dedup channel edges returned from ChanUpdatesInHorizon
2018-09-04 21:24:40 -07:00
Olaoluwa Osuntokun
a35bdd4233
channeldb: extend set of graph tests to include opque data where pertinent 2018-09-04 20:52:43 -07:00
Wilmer Paulino
d3cf3168d2
channeldb/graph_test: ensure policies for an edge have different
timestamps

In this commit, we ensure policies for edges we create in
TestChanUpdatesInHorizon have different update timestamps. This ensures
that there are two entries per edge in the edge update index. Because of
this, the test will fail because ChanUpdatesInHorizon will return
duplicate channel edges due to looking at all the entries within the
edge update index. This will be addressed in a future commit to allow
the set of tests to pass once again.
2018-09-04 18:36:25 -07:00
Wilmer Paulino
c1633da252
channeldb/graph_test: extend prune edge update index test to update edges
In this commit, we extend TestChannelEdgePruningUpdateIndexDeletion test
to include one more update for each edge. By doing this, we can
correctly determine whether old entries were properly pruned from the
index once a new update has arrived.
2018-09-04 18:33:41 -07:00
Wilmer Paulino
8dec659e10
channeldb/graph_test: properly check entries within edge update index
Due to entries within the edge update index having a nil value, the
tests need to be modified to account for this. Previously, we'd assume
that if we were unable to retrieve a value for a certain key that the
entry was non-existent, which is why the improper pruning bug was not
caught. Instead, we'll assert the number of entries to be the expected
value and populate a lookup map to determine whether the correct entries
exist within it.
2018-09-04 18:33:40 -07:00
Olaoluwa Osuntokun
06d03b55a3
channeldb: add new TestNodePruningUpdateIndexDeletion test
In this commit, we add a new test to expose a lurking bug within the
graph database code. As is, when we go to delete a node from the
database, we don't also remove the entries within the update index. As a
result, if a user attempted to call NodeUpdatesInHorizon (or typically
as part of the p2p handshake), we would error out, as we would try to
read a node that no longer existed in the graph, as it was pruned.
2018-08-16 18:20:58 -07:00
Joost Jager
6c918a1806 channeldb: store unknown policies in database
The commit ensures that for every channel, there will always
be two entries in the edges bucket. If the policy from one or
both ends of the channel is unknown, it is marked as such.

This allows efficient lookup of incoming edges. This is
required for backwards payment path finding.
2018-08-09 20:23:40 -07:00
Olaoluwa Osuntokun
d129e7c890
channeldb: update ChannelView to return new EdgePoint struct
In this commit, we update the ChannelView method to be compatible with
the new set of interfaces that require the script to be passed in in
addition to the outpoint. In order to do this, we introduce a new
EdgePoint struct which packages together a channel point along with the
funding pkScript. Along the way, we've copied over a utility method from
the lnwallet package to avoid having to deal with an import cycle.
2018-07-31 21:28:54 -07:00
Olaoluwa Osuntokun
ef4c7d2a78
channeldb: add new TestAddChannelEdgeShellNodes test 2018-07-22 21:05:52 -07:00
Olaoluwa Osuntokun
131eea4960
channeldb: modify TestPruneGraphNodes to test nodes w/ single edge
In this commit, we extend the TestPruneGraphNodes test to also test the
case of when a node is involved in a channel, but only a single edge for
that channel has been advertised. In order to test this, we add an
additional node to the graph, and also a new channel. However, this
channel will only have a single edge advertised. As result, when we
prune the set of edges, the only node remaining should be the node that
didn't have any edges at all.
2018-07-22 18:57:18 -07:00
Olaoluwa Osuntokun
0645261e5e
channeldb: add nwe PruneGraphNodes method to prune unconnected nodes 2018-07-21 19:49:59 -07:00
Wilmer Paulino
0e4d350e56
channeldb: prune nodes with no open channels left from the graph 2018-07-17 17:35:29 -07:00
Olaoluwa Osuntokun
6f60f139f4 multi: switch over import paths from roasbeef/* to btcsuite/* 2018-07-13 17:05:39 -07:00
Olaoluwa Osuntokun
dd4cab054d
channeldb: ensure items from the update index are actually deleted
In this commit, we fix an existing bug in the new graph query sync
feature. Before this commit, when a block is pruned, we would never
actually delete the update index entries. This is due to the fact that
we would attempt to delete the entries from the update index _after_ we
had already removed the edges from the update index.

We fix this by simply swapping the order: first we delete from the
update index, then we delete the edges themselves. A test ensuring that
the entires are cleared (which failed before this commit), has been
added.
2018-06-05 17:30:56 -07:00
Olaoluwa Osuntokun
baed4d1f47
channeldb: add new methods required to implement new discovery.ChannelGraphTimeSeries interface
In this commit, we add a series of methods, and a new database index
that we'll use to implement the new discovery.ChannelGraphTimeSeries
interface interface. The primary change is that we now maintain two new
indexes tracking the last update time for each node, and the last update
time for each edge. These two indexes allow us to implement the
NodeUpdatesInHorizon and ChanUpdatesInHorizon methods. The remaining
methods added simply utilize the existing database indexes to allow us to
respond to any peer gossip range queries.

A set of new unit tests has been added to exercise the added logic.
2018-05-31 16:30:54 -07:00
Dimitris Tsapakidis
4009f7f874 multi: fix typos in comments 2018-04-17 19:03:27 -07:00
practicalswift
663c396235 multi: fix a-vs-an typos 2018-04-17 19:02:04 -07:00