Commit Graph

591 Commits

Author SHA1 Message Date
Joost Jager
762609a169
channeldb: fix suppressed error 2019-09-04 19:20:05 +02:00
Olaoluwa Osuntokun
441b15055b
channeldb: properly compare payment attempts in serialization test 2019-08-23 12:50:25 -07:00
Olaoluwa Osuntokun
c78e3aaa9d
channeldb: update route.Hop serialization to include new EOB related fields
We also include a migration for the existing routes stored on disk.
2019-08-22 18:53:04 -07:00
Johan T. Halseth
21baa7bf18
multi: fix linter errors 2019-08-07 10:53:10 +02:00
Olaoluwa Osuntokun
7767eecbb8 Merge pull request #3164 from joostjager/persistent-mc
routing: persistent mission control
2019-08-02 15:45:49 -07:00
Olaoluwa Osuntokun
69c9e2b732
Merge pull request #3278 from Crypt-iQ/bbolt_options_0707
channeldb: specify freelist bbolt options by default
2019-07-30 16:52:28 -07:00
Joost Jager
eb4e65e54f
channeldb: export route serialization 2019-07-29 13:42:57 +02:00
Olaoluwa Osuntokun
8c389d13f9
Merge pull request #3197 from breez/optimize_prune_zombie_channels
Optimize prune zombie channels
2019-07-18 20:56:56 -07:00
Conner Fromknecht
0223039fb4
channeldb/graph: add NumZombies method and tests 2019-07-16 16:32:27 -07:00
Roei Erez
7300f33fe2 graph: implement index for disabled channels.
This commit adds an index bucket, disabledEdgePolicyBucket, for those
ChannelEdgePolicy with disabled bit on.
The main purpose is to be able to iterate over these fast when prune is
needed without the need for iterating the whole graph.

The entry points for accessing this index are:
1. When updating ChannelEdgePolicy - insert an entry.
2. When deleting ChannelEdge - delete the associated entries.
3. When querying for disabled channels - implemented DisabledChannelIDs
function
2019-07-16 17:48:07 +03:00
nsa
61a1ab08fb routing+channeldb: findPath uses Vertex instead of LightningNode
This commit modifies the nodeWithDist struct to use a route.Vertex
instead of a *channeldb.LightningNode. This change, coupled with
the new ForEachNodeChannel function, allows the findPath Djikstra's
algorithm to cut down on database lookups since we no longer need
to call the FetchOtherNode function.
2019-07-12 06:08:01 -04:00
Olaoluwa Osuntokun
ba5fbb3c27
Merge pull request #3156 from joostjager/extended-fail
routerrpc: add more failure reasons and route hints
2019-07-08 19:12:03 -07:00
nsa
66d15c8e76 channeldb + wtdb: specify freelist bbolt options by default
This commit specifies two bbolt options when opening the underlying
channel and watchtower databases so that there is reduced heap
pressure in case the bbolt database has a lot of free pages in the
B+ tree.
2019-07-08 18:41:25 -04:00
Joost Jager
5ce04091d8
routing+routerrpc+channeldb: return route on invalid payment details 2019-07-04 09:27:16 +02:00
Joost Jager
ae46fb00cb
routing+channeldb: add more failure reasons 2019-07-04 09:27:12 +02:00
Johan T. Halseth
97dda4f8c8
channeldb/migrations: skip migration for empty outgoing payments bucket
Previously the migration would fail if the source node was not set in
the database. Since we know that the source node must have been set
before making any payments, we check whether there actually are any
payments to migrate, and return early if not.
2019-06-18 21:14:16 +02:00
Johan T. Halseth
f98f452528
channeldb: avoid modifying bucket during cursor traversal
This is not safe according to bbolt documentation.
2019-06-14 21:09:03 +02:00
Johan T. Halseth
b7a37728c0
channeldb/migrations: avoid modifying bucket during ForEach loop
bbolt documenation state that this is not safe and leads to undefined
behavior.
2019-06-14 21:09:02 +02:00
Joost Jager
18c025151a
invoices+channeldb: move invoice state check to invoiceregistry 2019-06-10 16:12:08 +02:00
Joost Jager
2644759924
channeldb: add todo for more detailed failure reasons 2019-06-05 12:41:55 +02:00
Joost Jager
f03533c67a
routerrpc: convert sendpayment to async
Modify the routerrpc SendPayment api to asynchronous. This allows
callers to pick up a payment after the rpc connection was lost or lnd
was restarted.
2019-06-05 12:41:53 +02:00
Joost Jager
eb2647e8fc
channeldb: add subscription to control tower
Allows other sub-systems to subscribe to payment success and fail
events.
2019-06-05 12:41:49 +02:00
Joost Jager
87d3207baf
channeldb+routing: move control tower interface to routing
This commit creates an empty shall for control tower in the routing
package. It is a preparation for adding event notification.
2019-06-05 12:41:47 +02:00
Johan T. Halseth
7cb25a5afb
channeldb/control_tower test: add TestPaymentControlDeleteNonInFlight
TestPaymentControlDeleteNonInFlight checks that calling DeletaPayments
only deletes payments from the database that are not in-flight.
2019-05-27 20:19:00 +02:00
Johan T. Halseth
589f0fcc5d
control_tower: make initial bucket bucket creation stricter 2019-05-27 20:18:59 +02:00
Johan T. Halseth
95b4828780
channeldb: derive PaymentStatus implicitly from stored info 2019-05-27 20:18:59 +02:00
Johan T. Halseth
d705b8a013
channeldb+multi: rename Grounded->Unknown, Completed->Succeeded 2019-05-27 20:18:59 +02:00
Johan T. Halseth
d78d3f50b9
channeldb: move deprecated code to migration_09_legacy_serialization.go
And unexport deprecated code.
2019-05-27 20:18:59 +02:00
Johan T. Halseth
1b788904f0
channeldb+router: record payment failure reason in db 2019-05-27 20:18:59 +02:00
Johan T. Halseth
677cb018c9
routing/control_tower: add FetchInFlightPayments 2019-05-27 20:18:58 +02:00
Johan T. Halseth
de1bf8a518
routing/router: persist payment state machine
This commit makes the router use the ControlTower to drive the payment
life cycle state machine, to keep track of active payments across
restarts.  This lets the router resume payments on startup, such that
their final results can be handled and stored when ready.
2019-05-27 20:18:58 +02:00
Johan T. Halseth
59c2557cc9
channeldb/control_tower test: test new payment behavior 2019-05-27 20:18:57 +02:00
Johan T. Halseth
1af1832ff7
channeldb/control_tower: add payment information during state changes
This commit gives a new responsibility to the control tower, letting it
populate the payment bucket structure as the payment goes through
its different stages.

The payment will transition states Grounded->InFlight->Success/Failed,
where the CreationInfo/AttemptInfo/Preimage must be set accordingly.

This will be the main driver for the router state machine.
2019-05-27 20:18:57 +02:00
Johan T. Halseth
6d80661bbb
channeldb/payments: add StatusFailed 2019-05-27 20:18:57 +02:00
Johan T. Halseth
bb4aadd16c
channeldb/control_tower: remove non-strict option
Since we have performed a migration, the db should be in a consistent
state, and we can remove the non-strict option.
2019-05-27 20:18:57 +02:00
Johan T. Halseth
b7189ba028
channeldb test: add TestOutgoingPaymentsMigration 2019-05-27 20:18:57 +02:00
Johan T. Halseth
693807cf6e
channeldb/control_tower: add FetchSentPayments 2019-05-27 20:18:56 +02:00
Johan T. Halseth
801521ed2d
channeldb/migration: add migration for new payment bucket structure
migrateOutgoingPayments moves the OutgoingPayments into a new bucket format
where they all reside in a top-level bucket indexed by the payment hash. In
this sub-bucket we store information relevant to this payment, such as the
payment status.

To avoid that the router resend payments that have the status InFlight (we
cannot resume these payments for pre-migration payments) we delete those
statuses, so only Completed payments remain in the new bucket structure.
2019-05-27 20:18:56 +02:00
Johan T. Halseth
178996f0d3
channeldb/payments+control_tower: split OutgoingPayments
This commit changes the format used to store payments within the
DB. Previously this was serialized as one continuous struct
OutgoingPayment, which also contained an Invoice struct we where only
using a few fields of. We now split it up into two simpler sub-structs
CreationInfo, AttemptInfo and PaymentPreimage.

We also want to associate the payments more closely with payment
statuses, so we move to this hierarchy:

There's one top-level bucket "sentPaymentsBucket" which contains a set
of sub-buckets indexed by a payment's payment hash. Each such sub-bucket
contains several fields:
paymentStatusKey -> the payment's status
paymentCreationInfoKey -> the payment's CreationInfo.
paymentAttemptInfoKey -> the payment's AttemptInfo.
paymentSettleInfoKey -> the payment's preimage (or zeroes for
non-settled payments)

The CreationInfo is information that is static during the whole payment
lifcycle. The attempt info is set each time a new payment attempt
(route+paymentID) is sent on the network. The preimage is information
only known when a payment succeeds.  It therefore makes sense to split
them.

We keep legacy serialization code for migration puproses.
2019-05-27 20:18:56 +02:00
Johan T. Halseth
f022810f8b
channeldb/codec: add privateKey serialization 2019-05-27 20:18:56 +02:00
Johan T. Halseth
2417f40532
channeldb: put payment status in new bucket
We move the payment status to a new bucket hierarchy. Old buckets and
fetch methods are kept around for migration purposes.
2019-05-27 20:18:56 +02:00
Johan T. Halseth
d027e10201
htlcswitch+channeldb: move control tower to channeldb 2019-05-27 20:18:56 +02:00
Conner Fromknecht
28bf49807e
watchtower/wtdb: add CSessionStatus field to ClientSession
This commit adds persisted status bit-field to ClientSessions, that can
be used to modify behavior of their handling in the client. Currently,
only a default CSessionActive status is defined. However, the intention
is that this could later be used to signal that a session is abandoned
without needing to perform a db migration to add the field. As we move
forward with testing, this will likely be useful if a session gets
borked and we need a simple method of the client to temporarily ignore
certain sessions.

The field may be useful in signaling other types of status changes,
though this was the primary motivation that warranted the addition.
2019-05-24 18:24:41 -07:00
Conner Fromknecht
2a904cb69f
watchtower/wtdb: add Encode/Decode methods to wtclient structs 2019-05-23 20:48:08 -07:00
Johan T. Halseth
15bed506b1
routing/route+multi: remove redundant TotalFees field
Instead get it on demand using method TotalFees().
2019-05-16 23:58:32 +02:00
Joost Jager
064e8492de
cnct+htlcswitch+invoices: move invoice parameter check out of link
This commit is the final step in making the link unaware of invoices. It
now purely offers the htlc to the invoice registry and follows
instructions from the invoice registry about how and when to respond to
the htlc.

The change also fixes a bug where upon restart, hodl htlcs were
subjected to the invoice minimum cltv delta requirement again. If the
block height has increased in the mean while, the htlc would be canceled
back.

Furthermore the invoice registry interaction is aligned between link and
contract resolvers.
2019-05-15 14:42:12 +02:00
AdamISZ
6e054886d8 Typo corrections in various comments 2019-05-10 17:15:54 +02:00
Olaoluwa Osuntokun
fb1819bf7f
Merge pull request #3064 from halseth/route-serialization
[Reliable payments] add routing.Route (de)serialization + test
2019-05-09 16:46:14 -07:00
Johan T. Halseth
21c989ffcc
channeldb/graph: don't nil curve of returned PubKey
Nilling the key would cause signature verification to crash if the
returned key was used.

A test to exercise the behavior is added.
2019-05-09 10:49:58 +02:00
Johan T. Halseth
89fd43ebcb
channeldb: add Route (de)serialization + test
We will store the routes used during payment attempts, so we need
serialization code.
2019-05-09 09:59:43 +02:00
Conner Fromknecht
678ca9feff
channeldb/codec: add NewUnknownElementType constructor 2019-04-26 17:20:23 -07:00
Wilmer Paulino
0b0a9f4172
channeldb+routing: refactor DeleteChannelEdge to use ChannelID
In this commit, we refactor DeleteChannelEdge to use ChannelIDs rather
than ChannelPoints. We do this as the only use of DeleteChannelEdge is
when we are pruning zombie channels from our graph. When running under a
light client, we are unable to obtain the ChannelPoint of each edge due
to the expensive operations required to do so. As a stop-gap, we'll
resort towards using an edge's ChannelID instead, which is already
gossiped between nodes.
2019-04-18 21:57:41 -07:00
Wilmer Paulino
292defd6ba
channeldb: add IsDisabled method to ChannelEdgePolicy 2019-04-18 21:57:32 -07:00
Conner Fromknecht
f82b7c9bac
channeldb/graph: remove MarkEdgeZombie
This commit removes the MarkEdgeZombie method from channeldb. This
method is currently not used in any live code paths in production, and
is only used in unit tests. However, incorrect usage of this method
could result in an edge being present in both the zombie and channel
indexes, which deviates from any state we would expect to see in
production. Removing the method will help mitigate the potential for
writing incorrect unit tests in the future, by forcing zombie edges to
be created via the relevant, production APIs, e.g. DeleteChannelEdge.

The existing unit tests that use this method have been modified to use
the DeleteChannelEdge instead. No regressions were discovered in the
process.
2019-04-09 22:12:02 -07:00
Conner Fromknecht
4a755435e6
channeldb/graph: skip unknown edges in FetchChanInfos
This commit modifies FetchChanInfos to skip any channels that are not in
the graph at the time of the call. Currently the entire call will fail
if the edge is not found, which stalls a gossip sync in the following
scenario:

 1. Remote peer queries for a channel range
 2. We return the set of channel ids in that range
 3. A channel from that set is removed from the graph, e.g. via close.
 4. Remote peer queries for removed edge, causing the query to fail.

To remedy this, we will now skip any edges that are not known in the
database at the time of the query. This prevents the syncer state
machines from halting, which otherwise could only be resolved by
disconnecting and reconnecting.
2019-04-09 17:35:58 -07:00
Conner Fromknecht
e91bacd1bc
channeldb/graph: filter zombie channels in FilterKnownChanIDs
This commit modifies FilterKnownChanIDs to skip edges that
we ourselves have deemed zombies. This prevents us from requesting
the updates from them, as this wastes bandwidth and cpu cycles.
2019-04-05 13:01:08 -07:00
Conner Fromknecht
5d98a94d60
channeldb: write chan updates through reject+channel cache 2019-04-01 16:34:51 -07:00
Conner Fromknecht
6d3e081f7b
channeldb: accept cache sizes in ChannelGraph 2019-04-01 16:33:36 -07:00
Conner Fromknecht
baa968b1ff
channeldb/options: add Options w/ functional modifiers 2019-04-01 16:33:13 -07:00
Conner Fromknecht
ae3a00a5da
channeldb/graph+db: integrate reject and channel caches 2019-04-01 16:32:52 -07:00
Conner Fromknecht
b20a254faa
channeldb/channel_cache: add channelCache w/ randomized eviction 2019-04-01 16:25:31 -07:00
Conner Fromknecht
af0ea3590b
channeldb/reject_cache: add rejectCache w/ randomized eviction 2019-04-01 16:25:31 -07:00
Conner Fromknecht
3c46ceec1d
channeldb/db: init one ChannelGraph per channeldb.DB 2019-04-01 16:25:30 -07:00
Conner Fromknecht
ecbb78651a
channeldb/graph: add newChannelGraph constructor 2019-04-01 16:25:30 -07:00
Conner Fromknecht
ac315fd051
channeldb/db: remove unused buffer pool 2019-04-01 16:25:30 -07:00
Conner Fromknecht
2f927493c8
Revert "channeldb: convert invoice settle/cancel calls to use Batch"
This reverts commit da76c34418a40761fdc99e11200755661699f7eb.
2019-04-01 16:25:30 -07:00
Conner Fromknecht
e35f676b4c
Revert "channeldb: convert concurrent channel state machine calls to use Batch"
This reverts commit e8da6dd0b48843a48cecdcff894f48e8020d940e.
2019-04-01 16:25:30 -07:00
Conner Fromknecht
f6d93c8e5d
Revert "channeldb: convert all Update calls to use Batch"
This reverts commit 3555d3dbd431fe3ddf2660d4dc270b5a595fe517.
2019-04-01 16:25:26 -07:00
Olaoluwa Osuntokun
b8cbe3a1f8
channeldb: in RestoreChannelShells don't exit if edge already exists
During the restore process, it may be possible that we have already
heard about our prior edge from a node on the network (or our channel
peers). As a result, we shouldn't exit if this happens, and instead
should continue with the rest of the restoration process.
2019-03-28 17:54:06 -07:00
Olaoluwa Osuntokun
64b8facf0e
channeldb: don't read/write funding transactions if a restore channel 2019-03-28 17:53:43 -07:00
Olaoluwa Osuntokun
19ef4bbcb9
channeldb: within AddrsForNode don't fail if no graph node is found
In this commit, we modify the `AddrsForNode` method to not fail if no
graph node is found. We do this as when the backup is being
restored/created, it's possible that we don't yet have a
NodeAnnouncement for that node.
2019-03-28 17:53:42 -07:00
Olaoluwa Osuntokun
b93ff26265
channeldb: set restored chan status within RestoreChannelShells
In this commit, we move the location where we restore the channel status
to within the `RestoreChannelShells` method itself. Before this commit,
we attempted to use `ApplyChanStatus` which creates a DB transaction and
relies on a fully populated channel state, which in the restoration
case, we don't yet have.
2019-03-28 17:53:41 -07:00
Olaoluwa Osuntokun
13e7244d14
channeldb: also restore channel capacity in RestoreChannelShells 2019-03-28 17:53:40 -07:00
Wilmer Paulino
c82d73a826
channeldb+routing: extend edge lookup methods with zombie index check
In this commit, we extend the graph's FetchChannelEdgesByID and
HasChannelEdge methods to also check the zombie index whenever the edge
to be looked up doesn't exist within the edge index. We do this to
signal to callers that the edge is known, but only as a zombie, and the
only information that we have about the edge are the node public keys of
the two parties involved in the edge.

In the event that an edge does exist within the zombie index, we make
an additional check on edge policies to ensure they are not within the
router's pruning window, indicating that it is a fresh update.
2019-03-27 13:06:57 -07:00
Wilmer Paulino
e98f4d6d9d
channeldb: extend DeleteChannelEdge to mark edge as zombie
We mark the edges as zombies when pruning them to ensure we don't
attempt to reprocess them later on. This also applies to channels that
have been removed from the graph due to being stale.
2019-03-27 13:06:34 -07:00
Wilmer Paulino
b780dfacdb
channeldb: add zombie edge index
In this commit, we add a zombie edge index to the database. This allows
us to quickly determine across restarts whether we're attempting to
process an edge we've previously deemed as zombie.
2019-03-27 13:06:12 -07:00
Wilmer Paulino
a26a643273
channeldb: remove unused buckets 2019-03-27 13:05:48 -07:00
Olaoluwa Osuntokun
da76c34418
channeldb: convert invoice settle/cancel calls to use Batch 2019-03-19 17:11:17 -07:00
Olaoluwa Osuntokun
e8da6dd0b4
channeldb: convert concurrent channel state machine calls to use Batch 2019-03-19 17:11:14 -07:00
Olaoluwa Osuntokun
3555d3dbd4 channeldb: convert all Update calls to use Batch
In this commit, we convert all the `Update` calls which are serial, to
use `Batch` calls which are optimistically batched together for
concurrent writers. This should increase performance slightly during the
initial graph sync, and also updates at tip as we can coalesce more of
these individual transactions into a single transaction.
2019-03-19 17:10:46 -07:00
Joost Jager
32f2b047e8
htlcswitch: hodl invoice
This commit modifies the invoice registry to handle invoices for which
the preimage is not known yet (hodl invoices). In that case, the
resolution channel passed in from links and resolvers is stored until we
either learn the preimage or want to cancel the htlc.
2019-03-15 10:09:17 +01:00
Joost Jager
19f79613df
channeldb: store hold invoice 2019-03-15 10:08:55 +01:00
Joost Jager
8392f6d28f
lnrpc/invoicesrpc: remove lnrpc type from add invoice 2019-03-15 10:08:52 +01:00
Olaoluwa Osuntokun
33ad645f8c
lnwallet: update TestChanSyncFailure to pass with new borked update restriction
In this commit, we update the `TestChanSyncFailure` method to pass given
the new behavior around updating borked channel states. In order to do
this, we add a new method to allow the test to clear an existing channel
state. This method may be of independent use in other areas in the
codebase in the future as well.
2019-03-08 19:15:10 -08:00
Olaoluwa Osuntokun
032eacb796
channeldb: prevent mutating on-disk commitment state if channel is borked 2019-03-08 19:15:04 -08:00
Olaoluwa Osuntokun
cbe0bf6a22
Merge pull request #2501 from cfromknecht/batch-preimage-writes
htlcswitch: batch preimage writes/consistency fix
2019-02-21 17:00:00 -08:00
Conner Fromknecht
0a3e1cfbe5
channeldb+witness_beacon: use sha256 lookup+delete witness 2019-02-19 17:06:42 -08:00
Conner Fromknecht
e8b7f1fca3
channeldb/witness_cache: create AddSha256Witnesses helper + test 2019-02-19 17:05:30 -08:00
Conner Fromknecht
56b6becc48
channeldb/witness_cache_test: test batch preimage insertion 2019-02-19 17:05:17 -08:00
Conner Fromknecht
30f61b7630
multi: make AddPreimage variadic, optimistically compute key
In this commit, we modify the WitnessCache's
AddPreimage method to accept a variadic number
of preimages. This enables callers to batch
preimage writes in performance critical areas
of the codebase, e.g. the htlcswitch.

Additionally, we lift the computation of the
witnesses' keys outside of the db transaction.
This saves us from having to do hashing inside
and blocking other callers, and limits extraneous
blocking at the call site.
2019-02-19 17:05:04 -08:00
Wilmer Paulino
9febc9cc04
channeldb: add gossiper message store key migration
In this commit, we introduce a migration for the message store
sub-bucket that will migrate all keys within it to a new key format.
This new key format is composed of the peer's public key, followed by
the short channel ID, followed by the message type. This migration is
needed in order to provide backwards-compatibility with messages that
were previously stored before the introduction of the new key format.
2019-02-14 18:29:39 -08:00
Johan T. Halseth
2b012b06a0
channeldb/graph test: add test for update policy for unknown edge 2019-02-14 14:21:18 +01:00
Joost Jager
1b87fbfab2
invoices+htlcswitch+lnrpc: cancel invoice 2019-02-06 07:29:18 +01:00
Joost Jager
436dd41c77
channeldb: move idempotency up the call tree
As a preparation for subscribing to single invoices, InvoiceRegistry
needs to become aware of settling a settled invoice.
2019-02-01 09:42:56 +01:00
Joost Jager
bacd92418a
invoices: use lntypes.Hash and lntypes.Preimage
Previously chainhash.Hash was used, which converts to/from string in
reversed format. Payment hashes and preimages are supposed to be
non-reversed.
2019-02-01 09:42:29 +01:00
Olaoluwa Osuntokun
9a66ac396b
Merge pull request #2493 from halseth/tor-addr-sanity-check
Validate addresses before writing them to DB
2019-01-31 17:15:22 -08:00
Olaoluwa Osuntokun
480ec3bbca
channeldb: ensure restored channels can't be mutated 2019-01-28 20:24:48 -08:00
Olaoluwa Osuntokun
b2b57314fa
channeldb: add new RestoreChannelShells method
In this commit, we add a new type (ChannelShell) along with a new
method, RestoreChannelShells which allows a caller to insert a series of
channel shells into the database. These channel shells will allow a
restored node to initiate the DLP protocol and recover their set of
existing channels.

When we insert a channel shell, we re-create the original link node, and
also add the outgoing edge to the channel graph. This way we can be sure
that upon start up, we attempt to connect to the remote peers, and that
the normal graph query commands will operate as expected.
2019-01-28 20:24:41 -08:00
Olaoluwa Osuntokun
fa30af0475
channeldb: when fetching/inserting commits check for ChanStatusRestored
If the ChanStatusRestored flag is set, then we don't need to write or
read the set of commits for a channel as they won't exist. This will be
the case when we restore a channel from an SCB.
2019-01-28 20:24:40 -08:00
Olaoluwa Osuntokun
c656788b0b
channeldb: add new methods to allow adding a new edge+policy w/ existing db transaction 2019-01-28 20:24:38 -08:00