Commit Graph

19 Commits

Author SHA1 Message Date
carla
e2c0604657
multi: add flap count and last flap time to listpeers 2020-09-08 15:01:06 +02:00
carla
6cf66aea47
chanfitness: cool down flap count for rate limiting
Since we store all-time flap count for a peer, we add a cooldown factor
which will discount poor flap counts in the past. This is only applied
to peers that have not flapped for at least a cooldown period, so that
we do not downgrade our rate limiting for badly behaved peers.
2020-09-08 15:01:02 +02:00
carla
a550ca3d64
multi: store peer flap rate on disk on best effort basis
Since we will use peer flap rate to determine how we rate limit, we
store this value on disk per peer per channel. This allows us to
restart with memory of our peers past behaviour, so we don't give badly
behaving peers have a fresh start on restart. Last flap timestamp is
stored with our flap count so that we can degrade this all time flap
count over time for peers that have not recently flapped.
2020-09-08 13:49:46 +02:00
carla
70bca1f350
chanfitness: add flap count based rate limiting
To prevent flapping peers from endlessly dos-ing us with online and
offline events, we rate limit the number of events we will store per
period using their flap rate to determine how often we will add their
events to our in memory list of online events.

Since we are tracking online events, we need to track the aggregate
change over the rate limited period, otherwise we will lose track of
a peer's current state. For example, if we store an online event, then
do not store the subsequent offline event, we will believe that the
peer is online when they actually aren't. To address this, we "stage"
a single event which keeps track of all the events that occurred while
we were rate limiting the peer. At the end of the rate limting period,
we will store the last state for that peer, thereby ensureing that
we maintain our record of their most recent state.
2020-09-08 13:47:20 +02:00
carla
8b09b2d716
chanfitness: record timestamped flap count for peers
In preparation for storing our flap count on disk, we start tracking
flap count per-peer.
2020-09-08 13:47:19 +02:00
carla
e05b4a8e2e
chanfitness: refactor to store channels by peer
When dealing with online events, we actually need to track our events
by peer, not by channel. All we need to track channels is to have a
set of online events for a peer which at least contain those events.
This change refactors chanfitness to track by peer.
2020-09-08 13:47:18 +02:00
carla
10f9ba952e
chanfitness: unify requests to store in single chan info struct
We currently query the store for uptime and lifespan individually. As
we add more fields, we will need to add more queries with this design.
This change combines requests into a single channel infor request so
that we do not need to add unnecessary boilerplate going forward.
2020-09-08 13:47:17 +02:00
carla
7930ef7cf4
chanfitness: make online period calculation tolerant of duplicates
To get our uptime, we first filter our event log to get online periods.
This change updates this code to be tolerant of consecutive online or
offline events in the log. This will be required for rate limiting,
because we will not record every event for anti-dos reasons, so we could
record an online event, ignore an offline event and then record another
offline event. We could just ignore this duplicate event, but we will
also need this tolerance for when we persist uptime and our peers
can have their last event before restart as an online event and record
another online event when we come back up.
2020-09-08 13:47:16 +02:00
carla
c33d0aad46
chanfitness/test: switchover to testify 2020-09-08 13:47:15 +02:00
carla
94accfb69d
chanfitness: pass clock in to chaneventstore for testing 2020-09-08 13:47:15 +02:00
carla
7afd113b9f
chanfitness: add test context for better testing
As we add more elements to the chanfitness subsystem, we will require
more complex testing. The current tests are built around the inability
to mock subscriptions, which is remedied by addition of our own mock.
This context allows us to run the full store in a test, rather than
having to manually spin up the main goroutine. Mocking our subscriptions
is required so that we can block our subscribe updates on consumption,
using the real package provides us with no guarantee that the client
receives the update before shutdown, which produces test flakes.

This change also makes a move towards separating out the testing of our
event store from testing the underlying event logs to prepare for
further refactoring.
2020-09-08 13:47:14 +02:00
carla
3aa008ab04
multi: add interface for subscribe client so it can be mocked
The current implementation of subscribe is difficult to mock because
the queue that we send updates on in unexported, so you cannot create
a subscribe.Client object and then add your own updates. While it is
possible to run a subscribe server in tests, subscribe servers will
shutdown before dispatching their udpates to all clients, which can be
flakey (and is difficult to workaround). In this commit, we add a
subscription interface so that these testing struggles can be addressed
with a mock.
2020-09-08 13:47:13 +02:00
carla
25b0c40d05
chanfitness: fix line wrapping
Original PR was written with 4 spaces instead of 8, do a once off fix
here rather than fixing bit-by bit in the subsequent commits and
cluttering them for review.
2020-09-08 13:47:12 +02:00
carla
384c6aa1f1
chanfitness: save initial online event for channels with online peers
This commit adds an initial peer online event for channels
that have peers that are online when they are created. This
addresses a race between the peer coming online and an
existing channel being added to the event store; if the
peer comes online first, the existing channel will not have
its initial online event added.
2020-01-07 16:35:23 +02:00
carla
eebc23a8ad
chanfitness: set open time for channels with offline peers
This commit addresses a bug in the channel event store
where the opened time of a channel event log was not
set for peers that were offline on startup.

Previously, opened time was set to the time of the first
event in the event log. This worked for online peers,
because the eventlog was created with an initial online
event. However, offline peers had no inital event so had
no open time set.

This commit simplifies the creation of an event log by
removing the initial event and setting open time for all
event logs. This has the effect of potentially introducing
a gap between opened time for a log and the first peer
online event for peers with channels that exist at startup
if a peer takes time to reconnect. However, the cost of this
is less than the benefit of reducing the bug-prone custom
code path that was previously in place.
2019-12-17 17:36:28 +02:00
carla
4f9795e8ae
chanfitness: switch to query by channel outpoint
In this commit, the channelEventStore in the channel
fitness subsystem is changed to identify channels
by their outpoint rather than short channel id. This
change is made made becuase outpoints are the preferred
way to expose references over rpc, and easier to perform
queries within lnd.
2019-12-17 17:36:28 +02:00
carla
ec12463a95
chanfitness: define and export ErrChannelNotFound 2019-12-17 17:36:21 +02:00
carla
1e86589bee chanfitness: Add channel event store
This commit adds a channel event store to the channel fitness
package which is used to manage tracking of a node's channels.
It adds tracking for channel open/closed and peer online/offline
events for all channels that a node has open.

Events are consumed from channelNotifier and peerNotifier event
subscriptions. If either of these subscriptions is cancelled,
channel scoring stops, because both subscriptions are expected
to run until node shutdown.

Two functions are exposed to allow external callers to get uptime
information about a channel. GetLifespan returns the period over
which the channel has been monitored. GetUptime returns the channel's
uptime over a specified period. Callers can use these functions to
get the channel's remote peer uptime over its entire lifetime, or
a subset of that period.
2019-10-25 09:55:09 +02:00
carla
744876003d chanfitness: Add channel event log structure
This commit adds a chanfitness package which will be used to track
channel health and performance metrics. It adds a channel event
structure which will be used to track channel opens/closes and peer
uptime.

The eventLog implements an uptime function which calcualtes uptime
over a given period and a lifespan function which returns the time
when the log began monitoring the channel and, if the channel is
closed, the time when it stopped moitoring it.
2019-10-25 09:51:07 +02:00