Commit Graph

580 Commits

Author SHA1 Message Date
Conner Fromknecht
af0ea3590b
channeldb/reject_cache: add rejectCache w/ randomized eviction 2019-04-01 16:25:31 -07:00
Conner Fromknecht
3c46ceec1d
channeldb/db: init one ChannelGraph per channeldb.DB 2019-04-01 16:25:30 -07:00
Conner Fromknecht
ecbb78651a
channeldb/graph: add newChannelGraph constructor 2019-04-01 16:25:30 -07:00
Conner Fromknecht
ac315fd051
channeldb/db: remove unused buffer pool 2019-04-01 16:25:30 -07:00
Conner Fromknecht
2f927493c8
Revert "channeldb: convert invoice settle/cancel calls to use Batch"
This reverts commit da76c34418a40761fdc99e11200755661699f7eb.
2019-04-01 16:25:30 -07:00
Conner Fromknecht
e35f676b4c
Revert "channeldb: convert concurrent channel state machine calls to use Batch"
This reverts commit e8da6dd0b48843a48cecdcff894f48e8020d940e.
2019-04-01 16:25:30 -07:00
Conner Fromknecht
f6d93c8e5d
Revert "channeldb: convert all Update calls to use Batch"
This reverts commit 3555d3dbd431fe3ddf2660d4dc270b5a595fe517.
2019-04-01 16:25:26 -07:00
Olaoluwa Osuntokun
b8cbe3a1f8
channeldb: in RestoreChannelShells don't exit if edge already exists
During the restore process, it may be possible that we have already
heard about our prior edge from a node on the network (or our channel
peers). As a result, we shouldn't exit if this happens, and instead
should continue with the rest of the restoration process.
2019-03-28 17:54:06 -07:00
Olaoluwa Osuntokun
64b8facf0e
channeldb: don't read/write funding transactions if a restore channel 2019-03-28 17:53:43 -07:00
Olaoluwa Osuntokun
19ef4bbcb9
channeldb: within AddrsForNode don't fail if no graph node is found
In this commit, we modify the `AddrsForNode` method to not fail if no
graph node is found. We do this as when the backup is being
restored/created, it's possible that we don't yet have a
NodeAnnouncement for that node.
2019-03-28 17:53:42 -07:00
Olaoluwa Osuntokun
b93ff26265
channeldb: set restored chan status within RestoreChannelShells
In this commit, we move the location where we restore the channel status
to within the `RestoreChannelShells` method itself. Before this commit,
we attempted to use `ApplyChanStatus` which creates a DB transaction and
relies on a fully populated channel state, which in the restoration
case, we don't yet have.
2019-03-28 17:53:41 -07:00
Olaoluwa Osuntokun
13e7244d14
channeldb: also restore channel capacity in RestoreChannelShells 2019-03-28 17:53:40 -07:00
Wilmer Paulino
c82d73a826
channeldb+routing: extend edge lookup methods with zombie index check
In this commit, we extend the graph's FetchChannelEdgesByID and
HasChannelEdge methods to also check the zombie index whenever the edge
to be looked up doesn't exist within the edge index. We do this to
signal to callers that the edge is known, but only as a zombie, and the
only information that we have about the edge are the node public keys of
the two parties involved in the edge.

In the event that an edge does exist within the zombie index, we make
an additional check on edge policies to ensure they are not within the
router's pruning window, indicating that it is a fresh update.
2019-03-27 13:06:57 -07:00
Wilmer Paulino
e98f4d6d9d
channeldb: extend DeleteChannelEdge to mark edge as zombie
We mark the edges as zombies when pruning them to ensure we don't
attempt to reprocess them later on. This also applies to channels that
have been removed from the graph due to being stale.
2019-03-27 13:06:34 -07:00
Wilmer Paulino
b780dfacdb
channeldb: add zombie edge index
In this commit, we add a zombie edge index to the database. This allows
us to quickly determine across restarts whether we're attempting to
process an edge we've previously deemed as zombie.
2019-03-27 13:06:12 -07:00
Wilmer Paulino
a26a643273
channeldb: remove unused buckets 2019-03-27 13:05:48 -07:00
Olaoluwa Osuntokun
da76c34418
channeldb: convert invoice settle/cancel calls to use Batch 2019-03-19 17:11:17 -07:00
Olaoluwa Osuntokun
e8da6dd0b4
channeldb: convert concurrent channel state machine calls to use Batch 2019-03-19 17:11:14 -07:00
Olaoluwa Osuntokun
3555d3dbd4 channeldb: convert all Update calls to use Batch
In this commit, we convert all the `Update` calls which are serial, to
use `Batch` calls which are optimistically batched together for
concurrent writers. This should increase performance slightly during the
initial graph sync, and also updates at tip as we can coalesce more of
these individual transactions into a single transaction.
2019-03-19 17:10:46 -07:00
Joost Jager
32f2b047e8
htlcswitch: hodl invoice
This commit modifies the invoice registry to handle invoices for which
the preimage is not known yet (hodl invoices). In that case, the
resolution channel passed in from links and resolvers is stored until we
either learn the preimage or want to cancel the htlc.
2019-03-15 10:09:17 +01:00
Joost Jager
19f79613df
channeldb: store hold invoice 2019-03-15 10:08:55 +01:00
Joost Jager
8392f6d28f
lnrpc/invoicesrpc: remove lnrpc type from add invoice 2019-03-15 10:08:52 +01:00
Olaoluwa Osuntokun
33ad645f8c
lnwallet: update TestChanSyncFailure to pass with new borked update restriction
In this commit, we update the `TestChanSyncFailure` method to pass given
the new behavior around updating borked channel states. In order to do
this, we add a new method to allow the test to clear an existing channel
state. This method may be of independent use in other areas in the
codebase in the future as well.
2019-03-08 19:15:10 -08:00
Olaoluwa Osuntokun
032eacb796
channeldb: prevent mutating on-disk commitment state if channel is borked 2019-03-08 19:15:04 -08:00
Olaoluwa Osuntokun
cbe0bf6a22
Merge pull request #2501 from cfromknecht/batch-preimage-writes
htlcswitch: batch preimage writes/consistency fix
2019-02-21 17:00:00 -08:00
Conner Fromknecht
0a3e1cfbe5
channeldb+witness_beacon: use sha256 lookup+delete witness 2019-02-19 17:06:42 -08:00
Conner Fromknecht
e8b7f1fca3
channeldb/witness_cache: create AddSha256Witnesses helper + test 2019-02-19 17:05:30 -08:00
Conner Fromknecht
56b6becc48
channeldb/witness_cache_test: test batch preimage insertion 2019-02-19 17:05:17 -08:00
Conner Fromknecht
30f61b7630
multi: make AddPreimage variadic, optimistically compute key
In this commit, we modify the WitnessCache's
AddPreimage method to accept a variadic number
of preimages. This enables callers to batch
preimage writes in performance critical areas
of the codebase, e.g. the htlcswitch.

Additionally, we lift the computation of the
witnesses' keys outside of the db transaction.
This saves us from having to do hashing inside
and blocking other callers, and limits extraneous
blocking at the call site.
2019-02-19 17:05:04 -08:00
Wilmer Paulino
9febc9cc04
channeldb: add gossiper message store key migration
In this commit, we introduce a migration for the message store
sub-bucket that will migrate all keys within it to a new key format.
This new key format is composed of the peer's public key, followed by
the short channel ID, followed by the message type. This migration is
needed in order to provide backwards-compatibility with messages that
were previously stored before the introduction of the new key format.
2019-02-14 18:29:39 -08:00
Johan T. Halseth
2b012b06a0
channeldb/graph test: add test for update policy for unknown edge 2019-02-14 14:21:18 +01:00
Joost Jager
1b87fbfab2
invoices+htlcswitch+lnrpc: cancel invoice 2019-02-06 07:29:18 +01:00
Joost Jager
436dd41c77
channeldb: move idempotency up the call tree
As a preparation for subscribing to single invoices, InvoiceRegistry
needs to become aware of settling a settled invoice.
2019-02-01 09:42:56 +01:00
Joost Jager
bacd92418a
invoices: use lntypes.Hash and lntypes.Preimage
Previously chainhash.Hash was used, which converts to/from string in
reversed format. Payment hashes and preimages are supposed to be
non-reversed.
2019-02-01 09:42:29 +01:00
Olaoluwa Osuntokun
9a66ac396b
Merge pull request #2493 from halseth/tor-addr-sanity-check
Validate addresses before writing them to DB
2019-01-31 17:15:22 -08:00
Olaoluwa Osuntokun
480ec3bbca
channeldb: ensure restored channels can't be mutated 2019-01-28 20:24:48 -08:00
Olaoluwa Osuntokun
b2b57314fa
channeldb: add new RestoreChannelShells method
In this commit, we add a new type (ChannelShell) along with a new
method, RestoreChannelShells which allows a caller to insert a series of
channel shells into the database. These channel shells will allow a
restored node to initiate the DLP protocol and recover their set of
existing channels.

When we insert a channel shell, we re-create the original link node, and
also add the outgoing edge to the channel graph. This way we can be sure
that upon start up, we attempt to connect to the remote peers, and that
the normal graph query commands will operate as expected.
2019-01-28 20:24:41 -08:00
Olaoluwa Osuntokun
fa30af0475
channeldb: when fetching/inserting commits check for ChanStatusRestored
If the ChanStatusRestored flag is set, then we don't need to write or
read the set of commits for a channel as they won't exist. This will be
the case when we restore a channel from an SCB.
2019-01-28 20:24:40 -08:00
Olaoluwa Osuntokun
c656788b0b
channeldb: add new methods to allow adding a new edge+policy w/ existing db transaction 2019-01-28 20:24:38 -08:00
Olaoluwa Osuntokun
11c6887ffa
channeldb: refactor syncPending to expose new syncNewChannel function
The new syncNewChannel function will allow callers to insert a new
channel given the OpenChannel struct, and set of addresses for the
channel peer. This new method will also create a new LinkNode for the
peer if one doesn't already exist.
2019-01-28 20:24:37 -08:00
Olaoluwa Osuntokun
7a88f580ea
channeldb: add new FetchChannelMethod
In this commit, we add a new method, FetchChannel which is required in
order to implement the new chanbackup.LiveChannelSource interface.
2019-01-28 20:24:36 -08:00
Olaoluwa Osuntokun
aaf6456e12
channeldb: add HasChanStatus and ApplyChanState methods to OpenChannel
These methods allow callers to properly query if a channel is in a
particular flag, and also modify the channel status in a thread safe
manner.
2019-01-28 20:24:35 -08:00
Olaoluwa Osuntokun
a410262dda
channeldb: modify the String() method of ChannelStatus reflect all flags
In this commit, we modify the String() method of the ChannelStatus type
to reflect the fact that it's a flag set. With these new changes, we'll
now print the variable name of each assigned bit with a bar delimiting
them all.
2019-01-28 20:24:33 -08:00
Olaoluwa Osuntokun
f57f40e767
channeldb: add prefix naming to ChannelStatus enum variables
In this commit, we add a prefix naming scheme to the ChannelStatus enum
variables. We do this as it enables outside callers to more easily
identify each individual enum variable as a part of the greater
enum-like type.
2019-01-28 20:24:32 -08:00
Olaoluwa Osuntokun
f4054d2a66 channeldb: add new AddrsForNode method
In this commit, we add a new AddrsForNode method. This method will allow
a wrapper sturct to implement the new chanbackup.LiveChannelSource
method which is required to implement the full SCB feature set.
2019-01-28 20:24:22 -08:00
Johan T. Halseth
7d34ce9d08
lnwire+multi: define HasMaxHtlc helper on msgFlags 2019-01-22 08:42:30 +01:00
Johan T. Halseth
01e679786d
channeldb/graph_test: add TestEdgePolicyMissingMaxHtcl 2019-01-22 08:42:28 +01:00
Johan T. Halseth
6fcc76fc68
channeldb/channel: ignore ChannelEdgePolicy with missing max_htlc
If the max_htlc field is not found when fetching a ChannelEdgePolicy
from the DB, we treat this as an unknown policy.

This is done to ensure we won't propagate invalid data further. The data
will be overwritten with a valid one when we receive an update for this
channel.

It shouldn't be very common, but old data could be lingering in the DB
added before this field was validated.
2019-01-22 08:42:28 +01:00
Johan T. Halseth
8dd074ee57
channeldb/graph_test: assert MaxHTLC field correctness
Co-authored-by: Valentine Wallace <valentine.m.wallace@gmail.com>
2019-01-22 08:42:27 +01:00
Valentine Wallace
69d4bf051f
channeldb/graph: add max HTLC to ChannelEdgePolicy
Adding this field will allow us to persist an edge's
max HTLC to disk, thus preserving it between restarts.

Co-authored-by: Johan T. Halseth <johanth@gmail.com>
2019-01-22 08:42:27 +01:00
Johan T. Halseth
b9c5248915
channeldb/graph: extract ChannelEdgePolicy serialization 2019-01-22 08:42:27 +01:00
Johan T. Halseth
91c9e45031
channeldb/graph: check correct bucket for nilness 2019-01-22 08:42:27 +01:00
Valentine Wallace
0fd6004958
multi: partition lnwire.ChanUpdateFlag into ChannelFlags and MessageFlags
In this commit:

* we partition lnwire.ChanUpdateFlag into two (ChanUpdateChanFlags and
ChanUpdateMsgFlags), from a uint16 to a pair of uint8's

* we rename the ChannelUpdate.Flags to ChannelFlags and add an
additional MessageFlags field, which will be used to indicate the
presence of the optional field HtlcMaximumMsat within the ChannelUpdate.

* we partition ChannelEdgePolicy.Flags into message and channel flags.
This change corresponds to the partitioning of the ChannelUpdate's Flags
field into MessageFlags and ChannelFlags.

Co-authored-by: Johan T. Halseth <johanth@gmail.com>
2019-01-22 08:42:26 +01:00
Johan T. Halseth
8af2473644
channeldb/addr_test: add tests for invalid onion addresses 2019-01-17 14:16:15 +01:00
Johan T. Halseth
6a3e1423d2
channeldb/addr: sanity check onion address length before writing to db 2019-01-17 14:16:15 +01:00
Johan T. Halseth
e2e00f9dd2
channeldb/addr_test: add invalid TCP cases
These are invalid length IP addresses that earlier would not fail to
serialize.
2019-01-17 14:16:15 +01:00
Johan T. Halseth
02b7bb356a
channeldb/addr: validate ip before writing 2019-01-17 14:16:12 +01:00
Johan T. Halseth
6f605b2de2
channeldb/addr_test: inspect error string
Lets us match on more than static errors.
2019-01-17 14:14:24 +01:00
Wilmer Paulino
b951f06456
multi: move CsvDelay into ChannelConstraints 2019-01-11 16:58:15 -08:00
Olaoluwa Osuntokun
2e2d5fcf54
Merge pull request #2354 from joostjager/invoice-state
channeldb+lnrpc: invoice state
2019-01-04 14:48:17 -08:00
Joost Jager
5515713b88
channeldb: convert settled boolean to state
This commit is a preparation for the addition of new invoice
states. A database migration is not needed because we keep
the same field length and values.
2019-01-04 07:53:55 +01:00
Wilmer Paulino
70d3fc640a channeldb/db: prevent filtering channels with unconfirmed close txs
In this commit, we address an issue with the FetchWaitingCloseChannels
method where it would not properly return channels that are unconfirmed
and also have an unconfirmed closing transaction because they were
filtered out. We fix this by fetching channels that remain unconfirmed
that are also waiting for a confirmed closing transaction.

This will allow the recently added test TestFetchWaitingCloseChannels to
pass.
2019-01-03 20:48:28 -05:00
Wilmer Paulino
409efd1361 channeldb/channel_test: add TestFetchWaitingCloseChannels
In this commit, we add a test case for FetchWaitingCloseChannels to
ensure it behaves as intended. Currently, this test fails due to not
fetching channels which are pending to be closed but are also pending to
be opened. This will be fixed in the following commit and should allow
the test to pass.
2019-01-03 20:48:28 -05:00
Wilmer Paulino
f6c02aec20 channeldb/channel_test: use random outpoint when creating new test channels
In this commit, we use random outpoints when creating new test channels
to ensure we can uniquely identify them.
2019-01-03 20:33:33 -05:00
Johan T. Halseth
70aeda1589
channeldb/graph: don't create node bucket in PruneGraph
Instead return ErrSourceNodeNotSet directly. This would be returned from
the call pruneGraphNodes -> sourceNode anyway.
2018-12-06 10:22:16 +01:00
Johan T. Halseth
9d467d3534
channeldb/graph: reuse nodes bucket in deleteLightningNode
Lets us avoid passing the tx from pruneGraphNodes
2018-12-06 10:22:16 +01:00
Johan T. Halseth
1b9ca06563
channeldb/graph: reuse nodes bucket when fetchin source node 2018-12-06 10:22:16 +01:00
Johan T. Halseth
86d40636cd
channeldb/graph: don't create bucket in delEdgeUpdateIndexEntry
We are deleting what's in the bucket, so if it doesn't exist, just
return early with a non-error.
2018-12-06 10:22:16 +01:00
Johan T. Halseth
84de318553
channeldb/graph: don't create bucket in UpdateEdgePolicy
instead return ErEdgeNotFound, which would be returned anyway when
querying the edgeIndex for the channel.
2018-12-06 10:22:15 +01:00
Johan T. Halseth
9722323161
channeldb/graph: don't create node bucket in DeleteChannelEdge
Instead return ErrGraphNodeNotFound directly. If the node bucket was
created it would be empty, and the call delChannelByEdge ->
fetchChanEdgePolicies -> fetchChanEdgePolicy ->
deserializeChanEdgePolicy -> fetchLightningNode would return this error
anyway.
2018-12-06 10:22:15 +01:00
Johan T. Halseth
351f4aab79
channeldb/graph: don't create buckets in DeleteChannelEdge
Instead return ErrEdgeNotFound if buckets don't exist. This would be
returned anyway, when the chanIndex is checked for the channel point in
question.
2018-12-06 10:22:15 +01:00
Johan T. Halseth
5d8d99b7bc
channeldb/graph: don't create bucket in UpdateChannelEdge
It would return ErrEdgeNotFound when edge not found in bucket anyway.
2018-12-06 10:22:15 +01:00
Johan T. Halseth
81fe6e73ed
router+graph: return ErrGraphNodesNotFound if no nodes to prune
Avoids creating a bucket unneccessarily.
2018-12-06 10:22:15 +01:00
Johan T. Halseth
3d7bbd6d72
channeldb/graph: write max 80 sig len for nodes 2018-12-05 09:31:26 +01:00
Olaoluwa Osuntokun
1fd3aac925
multi: switch from bolt packge to bbolt package for all imports 2018-11-29 20:33:49 -08:00
Johan T. Halseth
b1a35fc8f4
channeldb/migrations_test: add TestMigrateOptionalChannelCloseSummaryFields 2018-11-21 13:36:42 +01:00
Johan T. Halseth
da3f515f12
channeldb/db_test: add TestFetchClosedChannelForID 2018-11-21 10:28:56 +01:00
Johan T. Halseth
45b132a954
channeldb/db: define FetchClosedChannelForID
FetchClosedChannelForID is used to find the channel close summary given
only a channel ID.
2018-11-21 10:28:56 +01:00
Johan T. Halseth
0b9a323fcb
channeldb/channel: add LastChanSync field to CloseChannelSummary
This commit adds an optional field LastChanSyncMsg to the
CloseChannelSummary, which will be used to save the ChannelReestablish
message for the channel at the point of channel close.
2018-11-21 10:28:43 +01:00
Johan T. Halseth
46c961aa17
channeldb/migrations: migrate ChannelCloseSummary 2018-11-21 10:27:58 +01:00
Johan T. Halseth
28b15dcbbb
channeldb/channel: write boolean to indicate presence of ChannelCloseSummary fields 2018-11-21 10:27:58 +01:00
Johan T. Halseth
149a8ce94f
channeldb/legacy_serialization: add deserializeCloseChannelSummaryV6
This commit adds a new file legacy_serialization.go, where a copy of the
current deserializeCloseChannelSummary is made, called
deserializeCloseChannelSummaryV6.

The rationale is to keep old deserialization code around to be used
during migration, as it is hard maintaining compatibility with the old
format while changing the code in use.
2018-11-21 10:12:05 +01:00
Wilmer Paulino
e795f7fce4
channeldb/graph: add method to determine if a node is public
In this commit, we add a method to the ChannelGraph struct that
determines whether a node is seen as public based on graph's source
node's point of view.
2018-10-24 17:28:53 -07:00
Johan T. Halseth
9cba3e813a
Merge pull request #1992 from halseth/listinvoices-offset
Make reversed listinvoices consistent in edge cases
2018-10-24 17:25:54 -07:00
Wilmer Paulino
28cf413055
channeldb: add method to retrieve a node's node announcement 2018-10-23 18:42:25 -07:00
Olaoluwa Osuntokun
4c0ca37174
Merge pull request #1389 from Bluetegu/noprivate-describegraph-1037
Add --noprivate flag to describegraph rpc to filter out private channels
2018-10-16 20:01:35 -07:00
Olaoluwa Osuntokun
3fda1029c0
Merge pull request #1786 from Roasbeef/golang-1.1
build+docs: bump to golang 1.11
2018-10-08 13:26:07 +09:00
Olaoluwa Osuntokun
d8e6085c17
multi: update to go 1.11 gofmt 2018-10-07 14:37:38 +09:00
Conner Fromknecht
59b459674d
multi: init subsystem loggers via build pkg 2018-10-05 13:04:45 +09:00
bluetegu
bd75a56855 Add unit test for db Wipe. 2018-10-04 13:35:42 -04:00
Olaoluwa Osuntokun
13802b4218
Merge pull request #1963 from wpaulino/create-chan-buckets-once
channeldb: ensure channel buckets are only created once
2018-09-28 17:06:39 -07:00
Johan T. Halseth
22d0e5b43e
channeldb/invoice test: add more tests for edge cases 2018-09-28 13:00:57 +02:00
Johan T. Halseth
0c51e31d1c
channeldb/invoices+test: don't return invoices on reversed query at index 1
Previously a call to QueryInvoices with reversed=true and index_offset=1
would make the cursor point to the first available invoice (num 1) that
would be returned as part of the response. This is inconsistent with the
othre indexes, so we instead just return an empty list in this case.

A test case for this situation is also added.
2018-09-28 13:00:57 +02:00
Olaoluwa Osuntokun
6afee3d099
Merge pull request #1856 from maurycy/typos
multi: fix various typos in comments
2018-09-27 20:38:10 -07:00
Wilmer Paulino
f13c1d2787
channeldb: ensure channel buckets are only created once
In this commit, we ensure that we only create the sub-bucket for
channels once: at the time of creation. We do this as otherwise it's
possible that a method that mutates a channel's state is called after it
has already been closed on-chain, leading to the channel bucket being
recreated.
2018-09-21 17:14:59 -07:00
Joost Jager
ab67b9a4de
rpcserver+lnrpc+lncli: add AbandonChannel rpc call
Using AbandonChannel, a channel can be abandoned. This means
removing all state without any on-chain or off-chain action.
A close summary is the only thing that is stored in the db after
abandoning.

A specific close type Abandoned is added. Abandoned channels
can be retrieved via the ClosedChannels RPC.
2018-09-18 12:20:27 -07:00
Conner Fromknecht
974ec02d24
channeldb/migrations: improve performance of edge update migration
This commit modifies the edge update index
migration to avoid repetitive calls to
Cursor.First(). Instead, we construct a set
of all edge update keys to remove, and then do
a second pass to remove each from the bucket.
Additional log messages are included to notify
users on progress, as some have reported long
migration times.
2018-09-17 00:05:24 -07:00
Olaoluwa Osuntokun
3b2c807288
channeldb: fix bug in migration from 0.4 to 0.5
In this commit, we fix a bug in the latest database migration when
migrating from 0.4 to 0.5. There's an issue in bolt db where if one
deletes a bucket that has a key with a nil value, it thinks that's a sub
bucket and attempts a bucket deletion. This will fail as it's not
actually a sub-bucket.  We get around this by using a cursor to manually
delete items in the
bucket.

Fixes #1907.
2018-09-14 12:39:44 -07:00
Olaoluwa Osuntokun
9e99f13f8d
Merge pull request #1898 from Roasbeef/fix-migration
channeldb: don't use KeyN in latest migration
2018-09-13 14:12:52 -07:00
Wilmer Paulino
7e6eb44cf8
channeldb/invoice_test: refactor TestQueryInvoices and add reverse test cases 2018-09-12 22:11:57 -07:00
Wilmer Paulino
0fe35e0014
channeldb/invoices: extend invoice queries to allow backwards pagination 2018-09-12 22:11:56 -07:00
Olaoluwa Osuntokun
034198ffb7
channeldb: don't use KeyN in latest migration
In this commit, we fix a bug in the latest migration that could cause
the migration to end in a panic. Additionally, we modify the migration
to exit early if the bucket wasn't found, as in this case, no migration
is required.

Fixes #1874.
2018-09-12 22:06:11 -07:00
Olaoluwa Osuntokun
09924f3f2c
channeldb: relax bucket assumptions for latest db migration
In this commit, we no longer assume that the bucket hierarchy has been
created properly when applying the latest DB migration. On older nodes
that never obtained a channel graph, or updated _before_ the query sync
stuff was added, then they're missing buckets that the migration expects
them to have.

We fix this by simply creating the buckets as we go, if needed.
2018-09-11 16:09:25 -07:00
Olaoluwa Osuntokun
3e67321aa7
channeldb: fix bucket creation hierarchy in createChannelDB
In this commit, we fix a bug in the bucket creation code in
createChannelDB. This bug can cause migrations on older nodes to fail,
as we expect the bucket to already have been created. With this commit,
we ensure that all the buckets under the main node and edges bucket are
properly created. Otherwise, a set of the newer migrations will fail to
apply for nodes updating from 0.4.
2018-09-09 15:49:28 -07:00
Olaoluwa Osuntokun
874002022f
channeldb: remove unused fwdPackageLogBucket key
In this commit, we move the declaration of the key for an unused bucket.
In the past, this bucket was used to store the revocation forwarding
package log. However, this has been moved under the key
`fwdPackagesKey`.
2018-09-09 15:47:52 -07:00
maurycy
ac24b12bf2
multi: fix various typos in comments 2018-09-07 06:51:49 +02:00
Olaoluwa Osuntokun
d2a7d910b7
Merge pull request #1825 from Roasbeef/extra-gossip-message-data
channeldb+discovery: ensure we store, validate and propagate announcements with opaque data
2018-09-05 17:53:02 -07:00
Olaoluwa Osuntokun
785efcfaa2
channeldb: also ignore the EOF error when trying to read ExtraOpaqueBytes
In this commit, we account for the additional case wherein the
announcement hasn't yet been written with the extra zero byte to
indicate that there aren't any remaining bytes to be read. Before this
commit, we accounted for the case where the announcement was written
with the extra byte, but now we ensure that legacy nodes that upgrade
will be able to boot properly.
2018-09-05 16:42:23 -07:00
Olaoluwa Osuntokun
16b5a67c3a
Merge pull request #1842 from wpaulino/dedup-chan-updates-horizon
channeldb: dedup channel edges returned from ChanUpdatesInHorizon
2018-09-04 21:24:40 -07:00
Olaoluwa Osuntokun
48072f8e82
channeldb: add limit on the max number of opaque bytes per announcement
In this commit, we add a new limit on the largest number of extra opaque
bytes that we'll allow to be written per vertex/edge. We do this in
order to limit the amount of disk space that we expose, as it's possible
that nodes may start to pad their announcements adding an additional
externalized cost as nodes may need to continue to store and relay these
large announcements.
2018-09-04 20:52:45 -07:00
Olaoluwa Osuntokun
a35bdd4233
channeldb: extend set of graph tests to include opque data where pertinent 2018-09-04 20:52:43 -07:00
Olaoluwa Osuntokun
f5c582d8d7
channeldb: add new ExtraOpaqueData to edge policy+update and node ann
In this commit, we add a mirror set of fields to the ones we recently
added to the set of gossip wire messages. With these set of fields in
place, we ensure that we'll be able to properly store and re-validate
gossip messages that contain a set of extra/optional fields.
2018-09-04 20:52:43 -07:00
Wilmer Paulino
3f58c2dea4
channeldb/graph: dedup channel edges returned from ChanUpdatesInHorizon
In this commit, we ensure that we de-duplicate the set of channel edges
returned from ChanUpdatesInHorizon. Other subsystems within lnd use this
method to retrieve and send all the channels with updates within a time
series to network peers. However, since the method looks at the edge
update index, which can include up to two entries per edge for each
policy, it's possible that we'd send channel announcements and updates
twice, causing extra bandwidth.
2018-09-04 18:48:21 -07:00
Wilmer Paulino
d3cf3168d2
channeldb/graph_test: ensure policies for an edge have different
timestamps

In this commit, we ensure policies for edges we create in
TestChanUpdatesInHorizon have different update timestamps. This ensures
that there are two entries per edge in the edge update index. Because of
this, the test will fail because ChanUpdatesInHorizon will return
duplicate channel edges due to looking at all the entries within the
edge update index. This will be addressed in a future commit to allow
the set of tests to pass once again.
2018-09-04 18:36:25 -07:00
Wilmer Paulino
85ea08fd17
channeldb: add migration to properly prune edge update index
In this commit, we introduce a migration to fix some of the recent
issues found w.r.t. the edge update index. The migration attempts to fix
two things:

1) Edge policies include an extra byte at the end due to reading an
extra byte for the node's public key from the serialized node info.

2) Properly prune all stale entries within the edge update index.

As a result of this migration, nodes will have a slightly smaller in
size channeldb. We will also no longer send stale edges to our peers in
response to their gossip queries, which should also fix the fetching
channel announcement for closed channels issue.
2018-09-04 18:33:43 -07:00
Wilmer Paulino
c1633da252
channeldb/graph_test: extend prune edge update index test to update edges
In this commit, we extend TestChannelEdgePruningUpdateIndexDeletion test
to include one more update for each edge. By doing this, we can
correctly determine whether old entries were properly pruned from the
index once a new update has arrived.
2018-09-04 18:33:41 -07:00
Wilmer Paulino
8dec659e10
channeldb/graph_test: properly check entries within edge update index
Due to entries within the edge update index having a nil value, the
tests need to be modified to account for this. Previously, we'd assume
that if we were unable to retrieve a value for a certain key that the
entry was non-existent, which is why the improper pruning bug was not
caught. Instead, we'll assert the number of entries to be the expected
value and populate a lookup map to determine whether the correct entries
exist within it.
2018-09-04 18:33:40 -07:00
Wilmer Paulino
2f22e6c35f
channeldb/graph: properly determine old update timestamp for an edge
In this commit, we fix a lingering issue within the edge update index
where entries were not being properly pruned due to an incorrect
calculation of the offset of an edge's last update time. Since the
offset is being determined from the end to the start, we need to
subtract all the fields after an edge policy's last update time from the
total amount of bytes of the serialized edge policy to determine the
correct offset. This was also slightly off as the edge policy included
an extra byte, which has been fixed in the previous commit.

Instead of continuing the slicing approach however, we'll switch to
deserializing the raw bytes of an edge's policy to ensure this doesn't
happen in the future when/if the serialization methods change or extra
data is included.
2018-09-04 18:33:38 -07:00
Wilmer Paulino
492d581df6
channeldb/graph: fix off-by-one public key slice
In this commit, we fix an off-by-one error when slicing the public key
from the serialized node info byte slice. This would cause us to write
an extra byte to all edge policies. Even though the values were read
correctly, when attempting to calculate the offset of an edge's update
time going backwards, we'd always be incorrect, causing us to not
properly prune the edge update index.
2018-09-04 18:31:38 -07:00
Wilmer Paulino
06344da62e
channeldb/graph: refactor UpdateEdgePolicy to use existing db transaction 2018-09-04 18:31:37 -07:00
Wilmer Paulino
aa3e2b6ba4
channeldb/graph: identify edge chan id on failure 2018-09-04 18:31:36 -07:00
Olaoluwa Osuntokun
4f20905ac1
Merge pull request #1719 from cfromknecht/duplicate-payments
Control Tower: Sender-side checks for duplicate payments #2
2018-08-22 21:22:38 -07:00
Olaoluwa Osuntokun
9f4685228e
Merge pull request #1763 from valentinewallace/listinvoices-pagination
channeldb+rpc: add pagination of listinvoices
2018-08-21 20:14:18 -07:00
Conner Fromknecht
86b347c996
channeldb/payments: make payment status helper methods
This commit splits FetchPaymentStatus and
UpdatePaymentStatus, such that they each invoke
helper methods that can be composed into different
db txns. This enables us to improve performance on
send/receive, as we can remove the exclusive lock
from the control tower, and allow concurrent calls
to utilize Batch more effectively.
2018-08-21 19:23:25 -07:00
Conner Fromknecht
0865ac7cf6
channeldb/migrations_test: assert locally-sourced circuits...
in the circuit map are marked StatusInFlight. We also
check that hashes contained in forwarded circuits are
not updated.
2018-08-21 19:23:25 -07:00
Conner Fromknecht
11bb5685f9
channeldb/migrations: mark locally-sourced payments as InFlight...
by reading the payment hash from the circuit map.
2018-08-21 19:23:25 -07:00
Conner Fromknecht
e7c0f4c5dc
channeldb/payments: touch up docs 2018-08-21 19:23:23 -07:00
Conner Fromknecht
3a579f3305
channeldb/migrations_test: touch up docs 2018-08-21 19:23:23 -07:00
Conner Fromknecht
93e5f9a545
channeldb/migrations: touch up documentation 2018-08-21 19:23:23 -07:00
Conner Fromknecht
9f46727507
channeldb/meta_test: restore migration helper + godocs 2018-08-21 19:23:23 -07:00
Conner Fromknecht
090e97cd3b
channeldb/db_test: move migration helper back to meta_test 2018-08-21 19:23:23 -07:00
Conner Fromknecht
79a4203346
channeldb/db: add comment describing migration 2018-08-21 19:23:23 -07:00
Vadym Popov
41e31e0909
channeldb: applying payment statuses migration to current database 2018-08-21 19:23:22 -07:00
Vadym Popov
24e3310f13
channeldb: payment statuses migration test 2018-08-21 19:23:21 -07:00
Vadym Popov
f405376b8b
channeldb: add migration for payment statuses for completed payments 2018-08-21 19:23:21 -07:00
Vadym Popov
7f6fbd4533
channeldb: test of payment statuses transitions 2018-08-21 19:23:21 -07:00
Vadym Popov
7296cfb425
channeldb: add payment statuses: ground, in flight, completed 2018-08-21 19:23:18 -07:00
Wilmer Paulino
f315b5b0d1
channeldb: support querying for invoices within a specific time range
In this commit, we introduce support for querying the database for invoices
that occurred within a specific add index range. The query format includes an
index to start with and a limit on the number of returned results.

Co-authored-by: Valentine Wallace <valentine.m.wallace@gmail.com>
2018-08-21 18:00:08 -07:00
Conner Fromknecht
af6c4e5174
channeldb/channel: adds readLogKey for chanids
Adds helper method to parse short chanids used as keys in the forwarding package.
2018-08-21 00:30:26 -07:00
Conner Fromknecht
81778664a7
channeldb/channel: expose AckAddHtlc and AckSettleFail 2018-08-21 00:30:26 -07:00
Conner Fromknecht
215a47e6bf
channeldb/forwarding_package: loosen Add/SettleFailRef acking
This commit loosens the fwdpkg reference acking to be more tolerant
of prior deletions. Specifically, we won't fail if certain channels
are not found or fwdpkgs do not exist. This will make us more
tolerant to future changes where we:
 - remove fwdpkgs on channel close
 - defensively cleanup stray responses
2018-08-21 00:30:23 -07:00
Olaoluwa Osuntokun
9353ab00b1
channeldb: ensure that we remove update index entries for nodes upon deletion 2018-08-16 18:22:55 -07:00
Olaoluwa Osuntokun
06d03b55a3
channeldb: add new TestNodePruningUpdateIndexDeletion test
In this commit, we add a new test to expose a lurking bug within the
graph database code. As is, when we go to delete a node from the
database, we don't also remove the entries within the update index. As a
result, if a user attempted to call NodeUpdatesInHorizon (or typically
as part of the p2p handshake), we would error out, as we would try to
read a node that no longer existed in the graph, as it was pruned.
2018-08-16 18:20:58 -07:00
Olaoluwa Osuntokun
90cdc9da8f
channeldb: add String() method for EdgePoint
In this commit we fix a minor logging artifact. After the switch to
EdgePoint, the FilteredChainView implementations will try to log the
struct directly, as prior they would have an outpoint object. We restore
this behavior by adding a String() method to EdgePoint which will simply
proxy through to the outpoint so we can log that directly.
2018-08-15 21:43:39 -07:00
Olaoluwa Osuntokun
26032f5956
channeldb: extend TestFetchClosedChannels to populate local chan cfg properly 2018-08-14 19:11:41 -07:00
Olaoluwa Osuntokun
d5863d03ae
Merge pull request #1660 from lightningnetwork/fetch-allll-channels
channeldb/db: return all channels if multiple chains exist
2018-08-09 20:59:28 -07:00
Olaoluwa Osuntokun
0f3c0ccf85
channeldb: add new FetchOtherNode method to ChannelEdgeInfo
In this commit, we add a new method to the ChannelEdgeInfo that will
allow the path finding logic to get the node opposite the pivot node
without first creating a new db transaction. The new method is able to
use an existing db transaction, or create a new one if needed.
2018-08-09 20:47:01 -07:00
Olaoluwa Osuntokun
ce7bfae4f7
channeldb: ensure we always check error from fetchChanEdgePolicy 2018-08-09 20:46:04 -07:00
Olaoluwa Osuntokun
0109672835
channeldb: add new db field for ChannelEdgeInfo to allow new methods 2018-08-09 20:45:39 -07:00
Joost Jager
6c918a1806 channeldb: store unknown policies in database
The commit ensures that for every channel, there will always
be two entries in the edges bucket. If the policy from one or
both ends of the channel is unknown, it is marked as such.

This allows efficient lookup of incoming edges. This is
required for backwards payment path finding.
2018-08-09 20:23:40 -07:00