Commit Graph

3524 Commits

Author SHA1 Message Date
Olaoluwa Osuntokun
4cf4fd377f
peer: in chanMsgStream release lock for condition var after msg q pop
In this commit we fix an existing bug within the msgConsumer grouting
of the chanMsgStream that could result in a partial deadlock, as the
readHandler would no longer be able to add messages to the message
queue. The primary cause of this issue would be if we got an update for
a channel that “we don’t know of”. The main loop would continue,
leaving the mutex unlocked. We would then try to re-lock at the top of
the loop, leading to a deadlock.

We avoid this situation by properly unlocking the condition variable as
soon as we’re done modifying the condition itself.
2017-10-19 19:45:08 -07:00
Johan T. Halseth
00ee3afab8 lncli: add commands "create" and "unlock"
lncli create:
This command is used to set up a wallet encryption password for
use with lnd at first time use. It will ask fot the user to
confirm the chosen password, then do a call to the lnd RPC method
CreateWallet with the chosen password.

lncli unlock:
This command is used to unlock the wallet of a running lnd instance.
It calls the RPC method UnlockWallet with the provided password.

Both methods makes use of the terminal.ReadPassword method, to
securely read a password from user input without making it
replayable in the terminal.
2017-10-19 19:17:35 -07:00
Johan T. Halseth
2c2d18ee10 networktest: --noencryptwallet for test nodes
This commit adds the --noencryptwallet flag to integration test
nodes,  causing them to be operational during the tests without
having to provide a password over rpc.
2017-10-19 19:17:35 -07:00
Johan T. Halseth
824e0c2982 daemon: wait for encryption passoword at startup
This commit makes use of the UnlockerService within lnd, waiting
for the user to provide a wallet encryption passord over RPC at
startup. When the passoword is received, startup continues as
normal, either using the passowrd to create the wallet for first
time use, or unlocking an existing wallet database.

This can be skipped by setting the --noencryptwallet flag, causing
the wallet database to be encypted using the default passoword.
2017-10-19 19:17:35 -07:00
Johan T. Halseth
62dc6b81d2 chainregistry: take wallet password as parameter to newChainControlFromConfig
This let the caller of newChainControlFromConfig set the password
to be used when creating or unlocking the wallet database. The
provided password is used both as private and public password.
2017-10-19 19:17:35 -07:00
Johan T. Halseth
f1c7dc1cbd config: add --noencryptwallet flag
This commit add the --noencryptwallet flag to lnd config, to be
used by users wanting to disable wallet encryption.
2017-10-19 19:17:35 -07:00
Johan T. Halseth
b7ba2697c8 walletunlocker: add package walletunlocker
The walletunlocker package contains the UnlockerService, which
implements the lnrpc.WalletUnlocker interface. This service is
used for receiving a password from the user over RPC, and doing
simple validity checks like making sure the user is not trying
to create a new wallet if one already exists, and that in case
the wallet exists, the provided password is correct.

The service will the pass the passwords over the CreatePasswords
or UnlockPasswords channels, for use within lnd.go.
2017-10-19 19:17:35 -07:00
Johan T. Halseth
1f34bd815d lnrpc: add WalletUnlocker service.
This commit adds the service WalletUnlocker, which is to be used
for creating a wallet password at first time startup, and unlocking
the wallet. The service exposes the RPC methods CreateWallet and
UnlockWallet.
2017-10-19 19:17:35 -07:00
Johan T. Halseth
e6cc46d84e lnwallet: export btcwallet.NetworkDir 2017-10-19 19:17:35 -07:00
Jim Posen
b29af869c7 lnwallet: Support funding tx fee estimation for more input types.
The fee estimation for funding transactions now properly accounts for
different types of UTXOs spent, whereas previously it assumed all
inputs were spending native P2WKH outputs.
2017-10-19 17:37:53 -07:00
Jim Posen
3232fd71c2 lnwallet: Add TxWeightEstimator support for nested pay-to-witness. 2017-10-19 17:37:53 -07:00
Jim Posen
c94130328a lnwallet: Extend Utxo struct with AddressType.
The Utxo struct now includes the address type and redeem/witness
scripts. This is necessary for accurate fee estimation.
2017-10-19 17:37:53 -07:00
Jim Posen
10a336db46 lnwallet: Use TxWeightEstimator in breacharbiter. 2017-10-19 17:37:53 -07:00
Jim Posen
ced530f98e lnwallet: Use TxWeightEstimator for funding transaction. 2017-10-19 17:37:53 -07:00
Jim Posen
fb32c3f73d lnwallet: Create type to estimate transaction weights.
The new TxWeightEstimator type can be used to replace the fee
estimation code for funding transactions and transactions in
breacharbiter.
2017-10-19 17:37:53 -07:00
Olaoluwa Osuntokun
d894917458
server: add new shouldRequestGraphSync method to gate third feature bit
In this commit, we add a new method shouldRequestGraphSync which the
server will use in order to determine if we should request a full
channel graph sync from a newly connected remote peer. Atm, we’ll only
request a full sync iff, we have less than two peers. This is only the
initial basic logic, as we’ll later extend this to be more
comprehensive.

With this change, we’ll no longer be blasted by full channel graph
dumps for _each_ new connection after we deem that we’ve been
sufficiently bootstrapped to the network.
2017-10-18 15:18:13 -07:00
Olaoluwa Osuntokun
56d4c15914
peer: add localFeatures as parameter to newPeer
In this commit we add the set of local features advertised as a
parameter to the newPeer function. With this change, the server will be
able to programmatically determine _which_ bits should be set on a
connection basis, rather than re-using the same global set of bits for
each peer.
2017-10-18 15:16:09 -07:00
Olaoluwa Osuntokun
f6f983a13b
peer: rename theirLocalFeatures+theirGlobal features to remote prefix 2017-10-18 15:14:27 -07:00
Olaoluwa Osuntokun
46d4f7216e
test: in testBasicChannelFunding also wait for bob to see channel 2017-10-18 15:13:41 -07:00
Jim Posen
ee49cdd103 server: Respect the initial_routing_sync feature bit.
Only synchronize routing info with peer if they request it by setting
the appropriate local feature bit.
2017-10-17 22:47:20 -07:00
Jim Posen
9fd77a6e40 multi: Update lnd to use new feature vector API. 2017-10-17 22:47:20 -07:00
Jim Posen
af49752d4d peer: Update peer handling of received feature vectors.
This updates peer to be compatible with the new feature vector API.
2017-10-17 22:47:20 -07:00
Jim Posen
1633ab180f lnwire: New API for feature vectors.
This is a rewrite of feature vectors in lnwire. This has a few
benefits:

1) a simpler interface
2) separate structs for a plain set of feature bits and a feature
vector with associated feature names
their respective feature sets
3) loosened requirements that bits MUST be assigned in pairs
4) fix endianness of encoding/decoding
2017-10-17 22:47:20 -07:00
Olaoluwa Osuntokun
b4273d1eaa
routing: abstract path finding behind mission control
In this commit, from the PoV of the SendPayment method we now delegate
all path finding+verification to missionControl. This change doesn’t
materially affect anything, it simply expands the abstraction to make
way for future features that more heavily utilize mission control.
2017-10-17 19:44:17 -07:00
Olaoluwa Osuntokun
dff3ad05f0
config: add sanity check to ensure either BTC or LTC is specified 2017-10-17 19:44:15 -07:00
Olaoluwa Osuntokun
f953f94f71
peer: display reversed bytes of chain hash in message summaries 2017-10-17 19:44:15 -07:00
Olaoluwa Osuntokun
dc124baca1
peer: if cooperative closing transaction is rejected, return err to caller 2017-10-17 19:44:14 -07:00
Olaoluwa Osuntokun
03f9479eb4
discovery: only interpret remote announcements as pre-mature
This commit fixes a slight bug in the announcement processing logic
within the AuthenticatedGossiper. Before this commit, it was possible
for us to ignore one of our now announcements due to it being
pre-mature, rendering (atm) the channel unusable by the ChannelRouter
itself. To fix this, we know only check for a pre mature announcement
iff the message is coming from a remote node.
2017-10-17 19:44:13 -07:00
Olaoluwa Osuntokun
f3b8c912b4
test: bump up logging level for integration test nodes to debug
In light of the recent commit which added a new set of logging in the
debug verbosity and the commit which added exporting logs from test
execution to Travis, we’ve now bumped up the logging level to debug.
With this new set of logging infrastructure, we should be able to
quickly debug any issues or flakes in the integration tests running on
CI.
2017-10-17 19:44:12 -07:00
Olaoluwa Osuntokun
b7e193354a
peer: add message summaries of sent/recv'd message for debug log level
This commit fills in an existing logging gap by adding a new set of
message summaries that is shown for the debug logging level.

Before this commit, if a user wanted to get a close up feel for what
lnd was doing under the covers, they had to use the trace logging
level. Trace can be very verbose, so we now provide a debug logging
level with message “summaries”. The summaries may not contain all the
data in the message, hut have been crafted in order to provide
sufficient detail at a glance.
2017-10-17 19:44:12 -07:00
Sam Lewis
dd90a724e5 test: add upload of integration logs from TravisCI builds
This commit adds adds: 

- Functionality to `gotest.sh` to log the output of `lnd` instances used for
  integration testing to file.

- Uploading of those log files to [termbin.com ](http://www.termbin.com) from
  TravisCI integration builds.

As an example of a build with this enabled, you can look at [this
build](https://travis-ci.org/samvrlewis/lnd/builds/286942025), which is a build
from my `ci_logging_test` branch. That branch has the same diff as my
`ci_logging` branch but with a small inclusion to allow TravisCI builds to work
(as I described in [my
comment](https://github.com/lightningnetwork/lnd/issues/302#issuecomment-335133834)
the other day). If you scroll to the end of the job logs where `RACE=false`
you'll see a `after_script` section with an output like this:

```
output0.log uploaded to http://termbin.com/aloqr
output1.log uploaded to http://termbin.com/3ggh
output2.log uploaded to http://termbin.com/abb9
output3.log uploaded to http://termbin.com/rk9j
output4.log uploaded to http://termbin.com/956p
output5.log uploaded to http://termbin.com/iwwt
```

Fixes #302.
2017-10-17 15:42:22 -07:00
Olaoluwa Osuntokun
209fb98d0f
routing: properly use vertexDecay for vertexes in missionControl 2017-10-16 20:31:32 -07:00
Olaoluwa Osuntokun
f555f29c7f
build: update glide files to point to latest btcd
This commit updates the glide files to point to the latest btcd build
which has had stall detection temporary disabled in order to ensure
smooth neutrino syncs without the false positives currently present in
stall detection.
2017-10-16 20:30:49 -07:00
Olaoluwa Osuntokun
bbb34cebe0
routing: modify the TestSendPaymentRouteFailureFallback to clear missionControl between attempts
In order to maintain the original essence of the test, we need to clear
the state of missionControl with each attempt, essentially advancing
time between each payment attempt.
2017-10-16 19:07:40 -07:00
Olaoluwa Osuntokun
8ef829ed80
routing: modify SendPayment loop to be lazy, iterative, and use missionControl
In this commit we modify the SendPayment loop to optimize for
time-to-first-payment-success-or-failure. The prior logic would first
attempt to find at least 100 routes to the destination, then
iteratively prune them away as errors were encountered. In this commit,
we modify this approach to instead take a lazy approach: we first find
the current “best” path, attempt to send to that, and if an error
occurs we prune a section of the graph by reporting to missionControl,
then continue.

With this new approach, if the first known path has sufficient
capacity, and is available, then the payment speed is greatly improved
from the PoV of users. Additionally, we avoid the excessive computation
of crawling most of the graph in the k-shortest paths loop. With the
decay on missionControl, all routes will now feed information into the
central knowledge hung, allowing all payments to iteratively find out
the inactive portions of the payment graph.
2017-10-16 19:05:47 -07:00
Olaoluwa Osuntokun
276f2e467b
routing: end path finding on an additional set of critical-ish errors 2017-10-16 18:58:35 -07:00
Olaoluwa Osuntokun
e06177e55c
routing: introduce new missionControl system within ChannelRouter
This commit adds a new system within the ChannelRouter: missionControl.
The purpose of this system to is to act as a shared memory of sorts
between payment sending attempts, recording which edges/vertexes word
or didn’t work. Allowing execution attempts to pass on their iterative
knowledge of the graph to later attempts will reduce the number of
failures encountered, and generally lead to a better UX when sending
payments.

The current capabilities of missionControl are rather limited just to
introduce the new abstraction. Later follow up commits will also add
preferential treatment for reliable nodes, knowledge the impact that
target payments have on unbalancing the payment graph, etc.
2017-10-16 18:57:36 -07:00
Olaoluwa Osuntokun
ae6bde2d77
routing: avoid internal bolt db deadlock by reusing transaction in findPath
This commit fixes a bug that could lead to a deadlock inside bolt db
itself. In a recent commit we allowed a db transaction to be passed
directly into findPath, however, the initial call to graph.ForEachNode
instead passed a _nil_ transaction causing the method itself to create
a _new_ transaction, leading to a deadlock.

We fix this issue by instead re-using the transaction pointer.
2017-10-16 18:48:27 -07:00
Olaoluwa Osuntokun
0692d2d408
peer: log type of received unknown message 2017-10-16 18:43:19 -07:00
Olaoluwa Osuntokun
65c03c98d0
htlcswitch: always return an instance of ForwardingError from handleLocalDispatch
This commit modifies the errors that we return within the
handleLocalDispatch method. Rather than returning a regular error, or
simply the matching error code in some instances, we now _always_
return an instance of ForwardingError. This will allow the router to
make more intelligent decisions w.r.t routing HTLC’s as with this
information it will now be able to differentiate errors that occur
within the switch (before sending out the HTLC), from errors that occur
within the HTLC route itself.
2017-10-16 18:42:16 -07:00
Olaoluwa Osuntokun
61be23dc31
htlcswitch+server: add new field SelfKey to htlcswitch.Config
This commit adds a new field to the switch’s Config, namely the public
key of the backing lightning node. This field will soon be used to
return more detailed errors messages back to the ChannelRouter itself.
2017-10-16 18:39:17 -07:00
Olaoluwa Osuntokun
51d04e8922
htlcswitch: add new ExtraMsg field to ForwardingError
This commit adds a new field to the ForwardingError struct: ExtraMsg.
The purpose of this field is to allow the htlcswitch to tack on
additional error context to ForwardingError messages returned to the L3
router.
2017-10-16 18:15:51 -07:00
Olaoluwa Osuntokun
b29a73a0dd
routing: don't prune our own channels during zombie channel collection
This commit is a precautionary commit which ensure that we don’t
attempt to prune our _own_ channels during zombie channel collection.
2017-10-16 18:14:01 -07:00
Olaoluwa Osuntokun
9b0b945a3d
Revert "server: BroadcastMessage will no longer block until completion"
This reverts commit 6db90ef09ab974df0ff09aeaf75a3d80414d4f50.

The root cause was fixed by commit
f4e7c36c80e8e2be9edb78b3b317c69d28d6d78f. As a result, this commit is no
longer needed.
2017-10-16 14:54:30 -07:00
Olaoluwa Osuntokun
7300452225
Revert "server: make sendToPeer async"
This reverts commit b7704e2de3dc96a4aebc2f47908a8f9def1da7f6.

The root issue was fixed by commit
f4e7c36c80e8e2be9edb78b3b317c69d28d6d78f. As a result, this commit is no
longer needed.
2017-10-16 14:53:24 -07:00
Olaoluwa Osuntokun
f4e7c36c80
peer: eliminate excessive queueHandler spinning w/ blocked writeHandler
This commit fixes an existing bug within the iteration between the
queueHandler and the writeHandler. Under certain scenarios, if the
writeHandler was blocked for a non negligible period of time, then the
queueHandler would enter a very tight spinning loop. This was due to
the fact that the break statement in the inner select loop of the
queueHandler wouldn’t actually break the inner for loop, instead it
would cause the execution logic to re-enter that same select loop,
causing a very tight spin.

In this commit, we fix the issue by adding to things: we now label the
inner select loop so we can break out of it if we detect that the
writeHandler has blocked. Secondly, we introduce a new channel between
the queueHandler and the writeHandler to signal the queueHandler that
the writeHandler has finished processing the last message.
2017-10-15 15:19:57 -07:00
Olaoluwa Osuntokun
cbdf139696
peer: add an idle timer to the readHandler
In this commit, we add an idle timer to the readHandler itself. This
will serve to slowly prune away inactive TCP connections as a result of
remote peer being blocked either upon reading or writing to the socket.
Our ping timer interval is 1 minute, so an idle timer interval of 5
minutes seem reasonable.
2017-10-15 15:14:35 -07:00
Olaoluwa Osuntokun
7a2ce62346
config+lnd: add new option --cpuprofile for writing cpuprofile to file
In this commit we add a new option to lnd, cpuprofile. With this option
we add additional telemetry hooks into and, allowing users to generate
CPU profiling files to measure hot spots within the daemon.
2017-10-14 17:08:34 -07:00
Olaoluwa Osuntokun
b7704e2de3
server: make sendToPeer async
This commit removes another case of unnecessary blockage, by modifying
the sendToPeer method to be fully asynchronous. From the PoV of the
callers that utilize this method currently, there’s no reason to block
until the completion of this method. Additionally, as the graph grows
larger without more intelligent the number of messages sent during
initial dump will start to be prohibitive to waiting for full
completion before proceeding.
2017-10-14 17:07:08 -07:00
Olaoluwa Osuntokun
6db90ef09a
server: BroadcastMessage will no longer block until completion
In this commit, we make the BroadcastMessage method on the server more
asynchronous by abandoning the two wait groups that it used for
synchronization. It has been observed that a circular waiting loop
between the AuthenticatedGossiper and a peer’s readHandler can cause
the system to dead lock.

By removing this unnecessary synchronization, we avoid the deadlock
case and allow the gossiper itself to no longer block in this scenario.
2017-10-14 16:04:06 -07:00