Matt Corallo [Sat, 27 Mar 2021 02:51:41 +0000 (22:51 -0400)]
[router] Do not fail if we are exactly (or 3x) limited by a maximum
The new MPP routing algorithm attempts to build paths with a higher
value than our payment to find paths which may allow us to pay a
fee to meet an htlc_minimum limit. This is great if we're
min-bounded, however it results in us rejecting paths where we are
bounded by a maximum near the end of the path, with fees, and then
bounded by a minimum earlier in the path. Further, it means we will
not find the cheapest path where paths have a lower relative fee
but a higher absolute fee.
Instead, we calculate routes using the actual amount we wish to
send. To maintain the previous behavior of searching for cheaper
paths where we can "pay the difference" to meet an htlc_minimum, we
detect if we were minimum-bounded during graph walking and, if we
are, we walk the graph again with a higher value.
Matt Corallo [Mon, 15 Mar 2021 23:49:51 +0000 (19:49 -0400)]
Enforce block connection ordering in unit and functional tests
This expands the assertions on block ordering to apply to
`#[cfg(test)]` builds in addition to normal builds, requiring that
unit and functional tests have syntactically-valid (ie the previous
block hash pointer and the heights match the blocks) blockchains.
This requires a reasonably nontrivial diff in the functional tests
however it is mostly straightforward changes.
Matt Corallo [Wed, 17 Mar 2021 17:11:48 +0000 (13:11 -0400)]
Fix block connection ordering in a number of functional tests
Many functional tests rely on being able to call block_connected
arbitrarily, jumping back in time to confirm a transaction at a
specific height. Instead, this takes us one step towards having a
well-formed blockchain in the functional tests.
We also take this opportunity to reduce the number of blocks
connected during tests, requiring a number of constant tweaks in
various functional tests.
Co-authored-by: Valentine Wallace <vwallace@protonmail.com> Co-authored-by: Matt Corallo <git@bluematt.me>
Matt Corallo [Fri, 5 Mar 2021 16:02:42 +0000 (11:02 -0500)]
Add assertions for in-order block [dis]connection in ChannelManager
Sadly the connected-in-order tests have to be skipped in our normal
test suite as many tests violate it. Luckily we can still enforce
it in the tests which run in other crates.
Co-authored-by: Matt Corallo <git@bluematt.me> Co-authored-by: Jeffrey Czyz <jkczyz@gmail.com>
Matt Corallo [Thu, 18 Mar 2021 03:12:47 +0000 (23:12 -0400)]
Ignore patch codecov as long as total coverage is within 1% of base
In some PRs, codecov gets mad that the coverage of the patch itself
is lower than the base. In most cases, we largely don't want a Big
Red X, at least as long as the total coverage has not gone down
substantially.
Matt Corallo [Wed, 17 Mar 2021 16:49:49 +0000 (12:49 -0400)]
Make cltv_expiry_delta configurable and reduce the min/default some
We allow users to configure the to_self_delay, which is analogous to
the cltv_expiry_delta in terms of its security context, so we should
allow users to specify both.
We similarly bound it on the lower end, but reduce that bound
somewhat now that it is configurable.
Matt Corallo [Wed, 17 Mar 2021 18:04:02 +0000 (14:04 -0400)]
Clean up some doc links in lightning_block_sync.
Relative HTML doc paths in doc links works locally, but breaks on
crates.io. Luckily, we can now use explicit full paths and rustdoc
will resolve them for us.
bmancini55 [Tue, 16 Mar 2021 20:30:22 +0000 (16:30 -0400)]
Simplify sequencing of handle_query_channel_range
Modify NetGraphMsgHandler::handle_query_channel_range to always use
first_blocknum=0 in replies. This is spec compliant after changes to
make sequence completion explicity using sync_complete.
bmancini55 [Sat, 13 Mar 2021 19:51:36 +0000 (14:51 -0500)]
Use constant MAX_REPLY_SCID
Modifies NetGraphMsgHandler::handle_query_channel_range to use a constant
max value in replies. Modifies tests to generate 8000 channels instead
of making this value configurable.
Matt Corallo [Fri, 5 Mar 2021 03:10:48 +0000 (22:10 -0500)]
Create new `InvoiceFeatures` object for Invoice-specific features
In the past we skipped doing this since invoice parsing occurs in a
different crate. However, we need to accept InvoiceFeatures in routing
now that we support MPP route collection, to detect if we can select
multiple paths or not. Further, we should probably take
rust-lightning-invoice as either a module or a subcrate in this repo.
Matt Corallo [Sun, 7 Mar 2021 18:00:11 +0000 (13:00 -0500)]
Make `get_outputs_to_watch` return a `Vec` instead of a `HashMap`
`get_outputs_to_watch` returned a reference to an existing
`HashMap` avoiding extra clones, but there isn't a huge reason to
do so now that we have to clone to copy it out of the
`ChannelMonitor` mutex. Instead, return a `Vec` since it may be
less memory and it allows us to have a bindings C mapping for the
function.
Co-authored-by: Jeffrey Czyz <jkczyz@gmail.com> Co-authored-by: Matt Corallo <git@bluematt.me>
Matt Corallo [Tue, 2 Mar 2021 15:25:37 +0000 (10:25 -0500)]
Make IgnoringMessageHandler and ErroringMessageHandler pub
This is largely useful for bindings, and the off-github discussion
around #814 concluded these should be pub, but the PR was not
updated to capture this. Now that the bindings support generation
for the structs, expose them.
Matt Corallo [Mon, 1 Mar 2021 22:07:29 +0000 (17:07 -0500)]
[bindings] Be explicit when calling pointer.is_null()
When the (somewhat anti-pattern)
`impl Deref for X { type Target = X; .. }` is used, this avoids an
infinite dereference exception trying to figure out what type to
resolve `is_null` against.
Matt Corallo [Fri, 26 Feb 2021 16:28:55 +0000 (11:28 -0500)]
Change ChannelManager::wait to be more descriptive
`wait` doesn't capture enough of what's going on, but also Java
Java doesn't accpet methods just called `wait`, as it conflicts
with existing sync primitives on all Objects.
Matt Corallo [Sun, 7 Mar 2021 17:58:14 +0000 (12:58 -0500)]
Add utility in `ChannelMonitor` to reload `chain::Filter` data
The deserialization process for `ChannelManager`/`ChannelMonitor`
data includes reloading any relevant `chain::Filter` with data
provided from the `ChannelMonitor`, but its nice if we adapt the
data to `chain::Filter` calls for users.
Jeffrey Czyz [Thu, 4 Mar 2021 02:33:54 +0000 (18:33 -0800)]
Correctly update the last block hash on disconnect
When a block is disconnected, the hash of the disconnected block was
used to update the last connected block. However, this amounts to a
no-op because these hashes should be equal. Successive disconnections
would update the hash but leave it one block off.
Normally, this not a problem because the last block_disconnected should
be followed by block_connected since the former is triggered by a chain
re-org. However, this assumes the user calls the API correctly and that
no failure occurs that would prevent block_connected from being called
(e.g., if fetching the connected block fails).
Instead, update the last block hash with the disconnected block's
previous block hash.
Jeffrey Czyz [Fri, 5 Mar 2021 20:37:50 +0000 (12:37 -0800)]
Hold ChannelManager locks independently
ChannelManager reads channel_state and last_block_hash while processing
funding_created and funding_signed messages. It writes these while
processing block_connected and block_disconnected events. To avoid any
potential deadlocks, have each site hold these locks independent of one
another and in a consistent order.
Additionally, use a RwLock instead of Mutex for last_block_hash since
exclusive access is not needed in funding_created / funding_signed and
cannot be guaranteed in block_connected / block_disconnected because of
the reads in the former.
Jeffrey Czyz [Fri, 5 Mar 2021 02:22:37 +0000 (18:22 -0800)]
Pass along ChannelManager's last_block_hash
ChannelMonitor keeps track of the last block connected. However, it is
initialized with the default block hash, which is a problem if the
ChannelMonitor is serialized before a block is connected. Instead, pass
ChannelManager's last_block_hash, which is initialized with a "birthday"
hash, when creating a new ChannelMonitor.
Jeffrey Czyz [Fri, 5 Mar 2021 01:39:21 +0000 (17:39 -0800)]
Remove last_block_connected from Channel
Tracking the last block was only used to de-duplicate block_connected
calls, but this is no longer required as of the previous commit.
Further, the ChannelManager can pass the latest block hash when needing
to create a ChannelMonitor rather than have each Channel maintain an
up-to-date copy. This is implemented in the next commit.
Jeffrey Czyz [Fri, 5 Mar 2021 00:58:26 +0000 (16:58 -0800)]
Remove block_connected de-duplication logic
Calling block_connected for the same block was necessary when clients
were expected to re-fetch filtered blocks with relevant transactions. At
the time, all listeners were notified of the connected block again for
re-scanning. Thus, de-duplication logic was put in place.
Jeffrey Czyz [Wed, 3 Mar 2021 19:24:55 +0000 (11:24 -0800)]
Parameterize ChannelManager::new with a block hash
When ChannelMonitors are persisted, they need to store the most recent
block hash seen. However, for newly created channels the default block
hash is used. If persisted before a block is connected, the funding
output may be missed when syncing after a restart. Instead, initialize
ChannelManager with a "birthday" hash so it can be used later when
creating channels.
bmancini55 [Wed, 3 Mar 2021 21:48:19 +0000 (16:48 -0500)]
Handle query_channel_range message from peer
Initial implementation of handling query_channel_range messages in
NetGraphMsgHandler. Enqueues a sequence of reply message in the pending
message events buffer.
bmancini55 [Fri, 19 Feb 2021 21:56:48 +0000 (16:56 -0500)]
Add SendReplyChannelRange message event
Creates a MessageSendEvent for sending a reply_channel_range message.
This event will be fired when handling inbound query_channel_range
messages in the NetGraphMessageHandler.
Matt Corallo [Fri, 26 Feb 2021 17:02:11 +0000 (12:02 -0500)]
Process monitor update events in block_[dis]connected asynchronously
The instructions for `ChannelManagerReadArgs` indicate that you need
to connect blocks on a newly-deserialized `ChannelManager` in a
separate pass from the newly-deserialized `ChannelMontiors` as the
`ChannelManager` assumes the ability to update the monitors during
block [dis]connected events, saying that users need to:
```
4) Reconnect blocks on your ChannelMonitors
5) Move the ChannelMonitors into your local chain::Watch.
6) Disconnect/connect blocks on the ChannelManager.
```
This is fine for `ChannelManager`'s purpose, but is very awkward
for users. Notably, our new `lightning-block-sync` implemented
on-load reconnection in the most obvious (and performant) way -
connecting the blocks all at once, violating the
`ChannelManagerReadArgs` API.
Luckily, the events in question really don't need to be processed
with the same urgency as most channel monitor updates. The only two
monitor updates which can occur in block_[dis]connected is either
a) in block_connected, we identify a now-confirmed commitment
transaction, closing one of our channels, or
b) in block_disconnected, the funding transaction is reorganized
out of the chain, making our channel no longer funded.
In the case of (a), sending a monitor update which broadcasts a
conflicting holder commitment transaction is far from
time-critical, though we should still ensure we do it. In the case
of (b), we should try to broadcast our holder commitment transaction
when we can, but within a few minutes is fine on the scale of
block mining anyway.
Note that in both cases cannot simply move the logic to
ChannelMonitor::block[dis]_connected, as this could result in us
broadcasting a commitment transaction from ChannelMonitor, then
revoking the now-broadcasted state, and only then receiving the
block_[dis]connected event in the ChannelManager.
Thus, we move both events into an internal invent queue and process
them in timer_chan_freshness_every_min().
Matt Corallo [Fri, 26 Feb 2021 02:55:30 +0000 (21:55 -0500)]
Change ShutdownResult type to better capture the possibilites
The return value from Channel::force_shutdown previously always
returned a `ChannelMonitorUpdate`, but expected it to only be
applied in the case that it *also* returned a Some for the funding
transaction output.
This is confusing, instead we move the `ChannelMontiorUpdate`
inside the Option, making it hold a tuple instead.
Matt Corallo [Sun, 7 Feb 2021 23:06:58 +0000 (18:06 -0500)]
Move functional tests involving reorgs to reorg_test
functional_tests.rs is huge, so anything we can do to split it up
some is helpful. This also exposes a somewhat glaring lack of
reorgs in our existing tests.