]> git.bitcoin.ninja Git - rust-lightning/log
rust-lightning
19 months agoFix scorer panic when available capacity is zero
Jeffrey Czyz [Thu, 5 Jan 2023 17:50:24 +0000 (11:50 -0600)]
Fix scorer panic when available capacity is zero

ProbabilisticScorer takes a ChannelUsage when computing a penalty for a
channel. The formula for calculating the liquidity penalty reduces the
maximum capacity by the amount of in-flight HTLCs (available capacity)
and adds one to prevent division by zero.

However, since the available capacity is passed to
DirectedChannelLiquidity as the capacity, other penalty formulas may use
the available (i.e., reduced) capacity inadvertently. In practice, this
has two ramifications for the historical liquidity penalty computation:

1. The bucket formula doesn't have a consistent denominator for a given
   channel.
2. The bucket formula may divide by zero when the in-flight HTLC amount
   equals or exceeds the effective capacity.

Fixing this involves only using the available capacity when appropriate.

20 months agoMerge pull request #2006 from TheBlueMatt/2023-02-no-recursive-read-locks v0.0.114-beta
Wilmer Paulino [Tue, 28 Feb 2023 08:24:16 +0000 (00:24 -0800)]
Merge pull request #2006 from TheBlueMatt/2023-02-no-recursive-read-locks

Refuse recursive read locks

20 months agoMerge pull request #2015 from TheBlueMatt/2023-02-no-dumb-redundant-fields
Matt Corallo [Tue, 28 Feb 2023 07:55:56 +0000 (07:55 +0000)]
Merge pull request #2015 from TheBlueMatt/2023-02-no-dumb-redundant-fields

20 months agoMake sure individual mutexes are constructed on different lines 2023-02-no-recursive-read-locks
Matt Corallo [Sun, 26 Feb 2023 20:22:28 +0000 (20:22 +0000)]
Make sure individual mutexes are constructed on different lines

Our lockdep logic (on Windows) identifies a mutex based on which
line it was constructed on. Thus, if we have two mutexes
constructed on the same line it will generate false positives.

20 months agoExport RUST_BACKTRACE=1 in `--feature backtrace` CI test
Matt Corallo [Fri, 24 Feb 2023 20:29:45 +0000 (20:29 +0000)]
Export RUST_BACKTRACE=1 in `--feature backtrace` CI test

as this test often fails on windows which is hard to debug locally
for most contributors.

20 months agoDisallow taking two instances of the same mutex at the same time
Matt Corallo [Wed, 22 Feb 2023 22:54:38 +0000 (22:54 +0000)]
Disallow taking two instances of the same mutex at the same time

Taking two instances of the same mutex may be totally fine, but it
requires a total lockorder that we cannot (trivially) check. Thus,
its generally unsafe to do if we can avoid it.

To discourage doing this, here we default to panicing on such locks
in our lockorder tests, with a separate lock function added that is
clearly labeled "unsafe" to allow doing so when we can guarantee a
total lockorder.

This requires adapting a number of sites to the new API, including
fixing a bug this turned up in `ChannelMonitor`'s `PartialEq` where
no lockorder was guaranteed.

20 months agoRefuse recursive read locks in lockorder testing
Matt Corallo [Thu, 2 Feb 2023 22:38:54 +0000 (22:38 +0000)]
Refuse recursive read locks in lockorder testing

Our existing lockorder tests assume that a read lock on a thread
that is already holding the same read lock is totally fine. This
isn't at all true. The `std` `RwLock` behavior is
platform-dependent - on most platforms readers can starve writers
as readers will never block for a pending writer. However, on
platforms where this is not the case, one thread trying to take a
write lock may deadlock with another thread that both already has,
and is attempting to take again, a read lock.

Worse, our in-tree `FairRwLock` exhibits this behavior explicitly
on all platforms to avoid the starvation issue.

Thus, we shouldn't have any special handling for allowing recursive
read locks, so we simply remove it here.

20 months agoDon't `per_peer_state` read locks recursively in monitor updating
Matt Corallo [Wed, 22 Feb 2023 22:10:46 +0000 (22:10 +0000)]
Don't `per_peer_state` read locks recursively in monitor updating

When handling a `ChannelMonitor` update via the new
`handle_new_monitor_update` macro, we always call the macro with
the `per_peer_state` read lock held and have the macro drop the
per-peer state lock. Then, when handling the resulting updates, we
may take the `per_peer_state` read lock again in another function.

In a coming commit, recursive read locks will be disallowed, so we
have to drop the `per_peer_state` read lock before calling
additional functions in `handle_new_monitor_update`, which we do
here.

20 months agoExpect callers to hold read locks before `channel_monitor_updated`
Matt Corallo [Fri, 3 Feb 2023 00:46:50 +0000 (00:46 +0000)]
Expect callers to hold read locks before `channel_monitor_updated`

Our existing lockorder tests assume that a read lock on a thread
that is already holding the same read lock is totally fine. This
isn't at all true. The `std` `RwLock` behavior is
platform-dependent - on most platforms readers can starve writers
as readers will never block for a pending writer. However, on
platforms where this is not the case, one thread trying to take a
write lock may deadlock with another thread that both already has,
and is attempting to take again, a read lock.

Worse, our in-tree `FairRwLock` exhibits this behavior explicitly
on all platforms to avoid the starvation issue.

Sadly, a user ended up hitting this deadlock in production in the
form of a call to `get_and_clear_pending_msg_events` which holds
the `ChannelManager::total_consistency_lock` before calling
`process_pending_monitor_events` and eventually
`channel_monitor_updated`, which tries to take the same read lock
again.

Luckily, the fix is trivial, simply remove the redundand read lock
in `channel_monitor_updated`.

Fixes #2000

20 months agoHold the `total_consistency_lock` while in `outbound_payment` fns
Matt Corallo [Fri, 3 Feb 2023 00:33:27 +0000 (00:33 +0000)]
Hold the `total_consistency_lock` while in `outbound_payment` fns

We previously avoided holding the `total_consistency_lock` while
doing crypto operations to build onions. However, now that we've
abstracted out the outbound payment logic into a utility module,
ensuring the state is consistent at all times is now abstracted
away from code authors and reviewers, making it likely to break.

Further, because we now call `send_payment_along_path` both with,
and without, the `total_consistency_lock`, and because recursive
read locks may deadlock, it would now be quite difficult to figure
out which paths through `outbound_payment` need the lock and which
don't.

While it may slow writes somewhat, it's not really worth trying to
figure out this mess, instead we just hold the
`total_consistency_lock` before going into `outbound_payment`
functions.

20 months agoRemove the `final_cltv_expiry_delta` in `RouteParameters` entirely 2023-02-no-dumb-redundant-fields
Matt Corallo [Mon, 6 Feb 2023 22:12:09 +0000 (22:12 +0000)]
Remove the `final_cltv_expiry_delta` in `RouteParameters` entirely

fbc08477e8dcdd8f3f2ada8ca77388b6185febe2 purported to "move" the
`final_cltv_expiry_delta` field to `PaymentParamters` from
`RouteParameters`. However, for naive backwards-compatibility
reasons it left the existing on in place and only added a new,
redundant field in `PaymentParameters`.

It turns out there's really no reason for this - if we take a more
critical eye towards backwards compatibility we can figure out the
correct value in every `PaymentParameters` while deserializing.

We do this here - making `PaymentParameters` a `ReadableArgs`
taking a "default" `cltv_expiry_delta` when it goes to read. This
allows existing `RouteParameters` objects to pass the read
`final_cltv_expiry_delta` field in to be used if the new field
wasn't present.

20 months agoSupport `ReadableArgs` types across in the TLV struct serialization
Matt Corallo [Mon, 6 Feb 2023 21:56:39 +0000 (21:56 +0000)]
Support `ReadableArgs` types across in the TLV struct serialization

This adds `required` support for trait-wrapped reading (e.g. for
objects read via `ReadableArgs`) as well as support for the
trait-wrapped reading syntax across the TLV struct/enum
serialization macros.

20 months agoRequire a non-0 number of non-empty paths when deserializing routes
Matt Corallo [Mon, 6 Feb 2023 21:43:10 +0000 (21:43 +0000)]
Require a non-0 number of non-empty paths when deserializing routes

When we read a `Route` (or a list of `RouteHop`s), we should never
have zero paths or zero `RouteHop`s in a path. As such, its fine to
simply reject these at deserialization-time. Technically this could
lead to something which we can generate not round-trip'ing
serialization, but that seems okay here.

20 months agoMerge pull request #2055 from TheBlueMatt/2023-02-fning-msrv
Matt Corallo [Mon, 27 Feb 2023 22:30:14 +0000 (22:30 +0000)]
Merge pull request #2055 from TheBlueMatt/2023-02-fning-msrv

Remove the lightning-transaction-sync MSRV

20 months agoMove `lightning-transaction-sync` out of the main workspace 2023-02-fning-msrv
Matt Corallo [Mon, 27 Feb 2023 17:32:07 +0000 (17:32 +0000)]
Move `lightning-transaction-sync` out of the main workspace

Because `lightning-transaction-sync` does not have an MSRV (and
because its dev-dependencies are huge), we can't build it by
default when devs run `cargo test`, so it is moved out of the
top-level workspace.

20 months agoRemove the `lightning-transaction-sync` MSRV
Matt Corallo [Mon, 27 Feb 2023 17:31:15 +0000 (17:31 +0000)]
Remove the `lightning-transaction-sync` MSRV

Apparently it has dependencies that don't track an MSRV at all, so
we can't practically enforce one in CI.

20 months agoFix silent rebase conflict in previous PR
Matt Corallo [Mon, 27 Feb 2023 18:28:26 +0000 (18:28 +0000)]
Fix silent rebase conflict in previous PR

20 months agoMerge pull request #2043 from valentinewallace/2023-02-initial-send-path-fails
Matt Corallo [Mon, 27 Feb 2023 18:10:27 +0000 (18:10 +0000)]
Merge pull request #2043 from valentinewallace/2023-02-initial-send-path-fails

`PaymentPathFailed`: add initial send error details

20 months agoMerge pull request #2033 from tnull/2023-02-make-esplora-sync-test-more-robust
Matt Corallo [Mon, 27 Feb 2023 17:55:43 +0000 (17:55 +0000)]
Merge pull request #2033 from tnull/2023-02-make-esplora-sync-test-more-robust

Make `test_esplora_syncs` more robust

20 months agoMerge pull request #2025 from TheBlueMatt/2023-02-no-pub-genesis-hashes
Matt Corallo [Mon, 27 Feb 2023 17:50:22 +0000 (17:50 +0000)]
Merge pull request #2025 from TheBlueMatt/2023-02-no-pub-genesis-hashes

Remove genesis block hash from public API

20 months agoFix PaymentPathFailed generation and scid on initial send
Valentine Wallace [Fri, 24 Feb 2023 03:46:45 +0000 (22:46 -0500)]
Fix PaymentPathFailed generation and scid on initial send

20 months agoChange PaymentPathFailed's optional network update to a Failure enum
Valentine Wallace [Mon, 13 Feb 2023 22:55:42 +0000 (17:55 -0500)]
Change PaymentPathFailed's optional network update to a Failure enum

This let us capture the errors when we fail without committing to an HTLC vs
failing via update_fail.

20 months agoImplement writeable for APIError
Valentine Wallace [Sun, 19 Feb 2023 23:32:29 +0000 (18:32 -0500)]
Implement writeable for APIError

20 months agoser_macros: Document behavior of upgradable_* variants
Valentine Wallace [Sat, 25 Feb 2023 21:03:43 +0000 (16:03 -0500)]
ser_macros: Document behavior of upgradable_* variants

20 months agoFix upgradable_required fields to actually be required in lower level macros
Valentine Wallace [Sun, 19 Feb 2023 21:52:22 +0000 (16:52 -0500)]
Fix upgradable_required fields to actually be required in lower level macros

When using lower level macros such as read_tlv_stream, upgradable_required
fields have been treated as regular options. This is incorrect, they should
either be upgradable_options or treated as required fields.

20 months agoMerge pull request #1977 from jkczyz/2023-01-offers-fuzz
Matt Corallo [Sat, 25 Feb 2023 18:03:51 +0000 (18:03 +0000)]
Merge pull request #1977 from jkczyz/2023-01-offers-fuzz

BOLT 12 deserialization fuzzers

20 months agoRename OptionDeserWrapper->RequiredWrapper
Valentine Wallace [Sat, 25 Feb 2023 00:50:24 +0000 (19:50 -0500)]
Rename OptionDeserWrapper->RequiredWrapper

Makes more sense and sets us up for the UpgradableRequired version of it

20 months agoDisambiguate ignorable and ignorable_option
Valentine Wallace [Mon, 20 Feb 2023 17:42:45 +0000 (12:42 -0500)]
Disambiguate ignorable and ignorable_option

Would rather not rename ignorable to ignorable_required, so rename both of them
to upgradable_*

20 months agoSupport deserializing an Option-al MaybeReadable
Valentine Wallace [Mon, 13 Feb 2023 21:49:22 +0000 (16:49 -0500)]
Support deserializing an Option-al MaybeReadable

Prior to this change, our impl_writeable_tlv_based macros only supported
deserializing a MaybeReadable if it's non-Optional.

20 months agoRemove all_paths_failed from PaymentPathFailed
Valentine Wallace [Sun, 12 Feb 2023 00:38:48 +0000 (19:38 -0500)]
Remove all_paths_failed from PaymentPathFailed

This field was previous useful in manual retries for users to know when all
paths of a payment have failed and it is safe to retry. Now that we support
automatic retries in ChannelManager and no longer support manual retries, the
field is no longer useful.

For backwards compat, we now always write false for this field. If we didn't do
this, previous versions would default this field's value to true, which can be
problematic because some clients have relied on the field to indicate when a
full payment retry is safe.

20 months agoMerge pull request #2050 from douglaz/import-lightning-io-futures
wpaulino [Fri, 24 Feb 2023 19:03:32 +0000 (11:03 -0800)]
Merge pull request #2050 from douglaz/import-lightning-io-futures

Only import lightning::io if futures are enabled

20 months agoOnly import lightning::io if futures are enabled
Allan Douglas R. de Oliveira [Fri, 24 Feb 2023 02:56:11 +0000 (02:56 +0000)]
Only import lightning::io if futures are enabled

20 months agoMake `test_esplora_syncs` more robust
Elias Rohrer [Wed, 15 Feb 2023 00:16:23 +0000 (18:16 -0600)]
Make `test_esplora_syncs` more robust

The test generally works and the parts in question *should* never fail.
However, they did, so we give the test server instances some leeway.

20 months agoFix amount overflow in Invoice building
Jeffrey Czyz [Thu, 16 Feb 2023 02:17:18 +0000 (20:17 -0600)]
Fix amount overflow in Invoice building

An overflow can occur when multiplying the offer amount by the requested
quantity when no amount is given in the request. Return an error instead
of overflowing.

20 months agoFix amount overflow in Offer parsing and building
Jeffrey Czyz [Wed, 15 Feb 2023 22:43:41 +0000 (16:43 -0600)]
Fix amount overflow in Offer parsing and building

An overflow can occur when multiplying the offer amount by the requested
quantity when checking if the given amount is enough. Return an error
instead of overflowing.

20 months agoFuzz test for bech32 decoding
Jeffrey Czyz [Thu, 9 Feb 2023 17:09:23 +0000 (11:09 -0600)]
Fuzz test for bech32 decoding

Fuzz testing bech32 decoding along with deserializing the underlying
message can result in overly exhaustive searches. Instead, the message
deserializations are now fuzzed separately. Add fuzzing for bech32
decoding.

20 months agoExpose Bech32Encode trait for fuzzing
Jeffrey Czyz [Thu, 9 Feb 2023 16:59:11 +0000 (10:59 -0600)]
Expose Bech32Encode trait for fuzzing

In order to fuzz test Bech32Encode parsing independent of the underlying
message deserialization, the trait needs to be exposed. Conditionally
expose it only for fuzzing.

20 months agoFuzz test for parsing Invoice
Jeffrey Czyz [Fri, 20 Jan 2023 22:30:45 +0000 (16:30 -0600)]
Fuzz test for parsing Invoice

An invoice is serialized as a TLV stream and encoded as bytes. Add a
fuzz test that parses the TLV stream and deserializes the underlying
Invoice. Then compare the original bytes with those obtained by
re-serializing the Invoice.

20 months agoFuzz test for parsing InvoiceRequest
Jeffrey Czyz [Fri, 20 Jan 2023 19:34:34 +0000 (13:34 -0600)]
Fuzz test for parsing InvoiceRequest

An invoice request is serialized as a TLV stream and encoded as bytes.
Add a fuzz test that parses the TLV stream and deserializes the
underlying InvoiceRequest. Then compare the original bytes with those
obtained by re-serializing the InvoiceRequest.

20 months agoRename `BestBlock::from_genesis` to `from_network` for clarity 2023-02-no-pub-genesis-hashes
Matt Corallo [Wed, 15 Feb 2023 06:09:00 +0000 (06:09 +0000)]
Rename `BestBlock::from_genesis` to `from_network` for clarity

20 months agoRemove genesis block hash from public API
Matt Corallo [Thu, 9 Feb 2023 19:20:22 +0000 (19:20 +0000)]
Remove genesis block hash from public API

Forcing users to pass a genesis block hash has ended up being
error-prone largely due to byte-swapping questions for bindings
users. Further, our API is currently inconsistent - in
`ChannelManager` we take a `Bitcoin::Network` but in `NetworkGraph`
we take the genesis block hash.

Luckily `NetworkGraph` is the only remaining place where we require
users pass the genesis block hash, so swapping it for a `Network`
is a simple change.

20 months agoAdd missing import path in ser macro
Valentine Wallace [Sun, 19 Feb 2023 23:32:06 +0000 (18:32 -0500)]
Add missing import path in ser macro

20 months agoMerge pull request #2014 from valentinewallace/2023-02-rework-partial-pmt-fail
Matt Corallo [Thu, 23 Feb 2023 21:54:16 +0000 (21:54 +0000)]
Merge pull request #2014 from valentinewallace/2023-02-rework-partial-pmt-fail

Rework auto-retry send errors

20 months agoClarify Retry::Timeout vs PaymentParams::expiry_time in docs
Valentine Wallace [Thu, 23 Feb 2023 20:49:44 +0000 (15:49 -0500)]
Clarify Retry::Timeout vs PaymentParams::expiry_time in docs

20 months agoFix outdated PendingOutboundPayment::Abandoned docs
Valentine Wallace [Sun, 12 Feb 2023 00:35:48 +0000 (19:35 -0500)]
Fix outdated PendingOutboundPayment::Abandoned docs

20 months agoOn initial send retries, avoid previously failed scids
Valentine Wallace [Thu, 16 Feb 2023 21:40:51 +0000 (16:40 -0500)]
On initial send retries, avoid previously failed scids

Previously, we could have tried the same failed channels over and over until
retries are exhausted.

20 months agoIn-line retry_with_route method
Valentine Wallace [Sun, 19 Feb 2023 21:12:24 +0000 (16:12 -0500)]
In-line retry_with_route method

Since it's only used one place now

20 months agoRework auto retry send errors
Valentine Wallace [Fri, 10 Feb 2023 21:09:01 +0000 (15:09 -0600)]
Rework auto retry send errors

Prior to this, we returned PaymentSendFailure from auto retry send payment
methods. This implied that we might return a PartialFailure from them, which
has never been the case. So it makes sense to rework the errors to be a better
fit for the methods.

We're taking error handling in a totally different direction now to make it
more asynchronous, see send_payment_internal for more information.

20 months agoFix InvalidRoute error to be ChannelUnavailable
Valentine Wallace [Wed, 22 Feb 2023 19:09:58 +0000 (14:09 -0500)]
Fix InvalidRoute error to be ChannelUnavailable

InvalidRoute is reserved for malformed routes, not routes where a channel or
its peer is unavailable

20 months agoMerge pull request #1897 from TheBlueMatt/2022-11-monitor-updates-always-async
Matt Corallo [Wed, 22 Feb 2023 19:12:31 +0000 (19:12 +0000)]
Merge pull request #1897 from TheBlueMatt/2022-11-monitor-updates-always-async

Always process `ChannelMonitorUpdate`s asynchronously

20 months agoDon't generate a `ChannelMonitorUpdate` for closed chans on shutdown 2022-11-monitor-updates-always-async
Matt Corallo [Sun, 19 Feb 2023 00:13:51 +0000 (00:13 +0000)]
Don't generate a `ChannelMonitorUpdate` for closed chans on shutdown

The `Channel::get_shutdown` docs are very clear - if the channel
jumps to `Shutdown` as a result of not being funded when we go to
initiate shutdown we should not generate a `ChannelMonitorUpdate`
as there's no need to bother with the shutdown script - we're
force-closing anyway.

However, this wasn't actually implemented, potentially causing a
spurious monitor update for no reason.

20 months agoUse the new monitor persistence flow for `funding_created` handling
Matt Corallo [Sat, 3 Dec 2022 03:15:04 +0000 (03:15 +0000)]
Use the new monitor persistence flow for `funding_created` handling

Building on the previous commits, this finishes our transition to
doing all message-sending in the monitor update completion
pipeline, unifying our immediate- and async- `ChannelMonitor`
update and persistence flows.

20 months agoUse new monitor persistence flow in funding_signed handling
Matt Corallo [Mon, 6 Feb 2023 23:03:38 +0000 (23:03 +0000)]
Use new monitor persistence flow in funding_signed handling

In the previous commit, we moved all our `ChannelMonitorUpdate`
pipelines to use a new async path via the
`handle_new_monitor_update` macro. This avoids having two message
sending pathways and simply sends messages in the "monitor update
completed" flow, which is shared between sync and async monitor
updates.

Here we reuse the new macro for handling `funding_signed` messages
when doing an initial `ChannelMonitor` persistence. This provides
a similar benefit, simplifying the code a trivial amount, but
importantly allows us to fully remove the original
`handle_monitor_update_res` macro.

20 months agoAlways process `ChannelMonitorUpdate`s asynchronously
Matt Corallo [Wed, 11 Jan 2023 21:37:57 +0000 (21:37 +0000)]
Always process `ChannelMonitorUpdate`s asynchronously

We currently have two codepaths on most channel update functions -
most methods return a set of messages to send a peer iff the
`ChannelMonitorUpdate` succeeds, but if it does not we push the
messages back into the `Channel` and then pull them back out when
the `ChannelMonitorUpdate` completes and send them then. This adds
a substantial amount of complexity in very critical codepaths.

Instead, here we swap all our channel update codepaths to
immediately set the channel-update-required flag and only return a
`ChannelMonitorUpdate` to the `ChannelManager`. Internally in the
`Channel` we store a queue of `ChannelMonitorUpdate`s, which will
become critical in future work to surface pending
`ChannelMonitorUpdate`s to users at startup so they can complete.

This leaves some redundant work in `Channel` to be cleaned up
later. Specifically, we still generate the messages which we will
now ignore and regenerate later.

This commit updates the `ChannelMonitorUpdate` pipeline across all
the places we generate them.

20 months agoMove TODO from `handle_monitor_update_res` into `Channel`
Matt Corallo [Sat, 3 Dec 2022 05:38:24 +0000 (05:38 +0000)]
Move TODO from `handle_monitor_update_res` into `Channel`

The TODO mentioned in `handle_monitor_update_res` about how we
might forget about HTLCs in case of permanent monitor update
failure still applies in spite of all our changes. If a channel is
drop'd in general, monitor-pending updates may be lost if the
monitor update failed to persist.

This was always the case, and is ultimately the general form of the
the specific TODO, so we simply leave comments there

20 months agoHandle `MonitorUpdateCompletionAction`s after monitor update sync
Matt Corallo [Fri, 27 Jan 2023 06:14:18 +0000 (06:14 +0000)]
Handle `MonitorUpdateCompletionAction`s after monitor update sync

In a previous PR, we added a `MonitorUpdateCompletionAction` enum
which described actions to take after a `ChannelMonitorUpdate`
persistence completes. At the time, it was only used to execute
actions in-line, however in the next commit we'll start (correctly)
leaving the existing actions until after monitor updates complete.

20 months agoMerge pull request #1988 from TheBlueMatt/2023-01-limited-chans
Matt Corallo [Wed, 22 Feb 2023 00:39:13 +0000 (00:39 +0000)]
Merge pull request #1988 from TheBlueMatt/2023-01-limited-chans

Limit the number of pending un-funded inbound channel

20 months agoLimit the number of pending un-funded inbound channel 2023-01-limited-chans
Matt Corallo [Thu, 26 Jan 2023 04:47:25 +0000 (04:47 +0000)]
Limit the number of pending un-funded inbound channel

Because we store some (not large, but not zero) state per-peer,
it's useful to limit the number of peers we have connected, at
least with some buffer.

Much more importantly, each channel has a relatively large cost,
especially around the `ChannelMonitor`s we have to build for each.

Thus, here, we limit the number of channels per-peer which aren't
(yet) on-chain, as well as limit the number of (inbound) peers
which don't have a (funded-on-chain) channel.

Fixes #1889

20 months agoAdd an `inbound` flag to the `peer_connected` message handlers
Matt Corallo [Thu, 26 Jan 2023 04:45:58 +0000 (04:45 +0000)]
Add an `inbound` flag to the `peer_connected` message handlers

Its useful for the message handlers to know if a peer is inbound
for DoS decision-making reasons.

20 months agoMerge pull request #2035 from TheBlueMatt/2023-02-fix-no-con-discon
Matt Corallo [Tue, 21 Feb 2023 21:28:05 +0000 (21:28 +0000)]
Merge pull request #2035 from TheBlueMatt/2023-02-fix-no-con-discon

Fix (and DRY) the conditionals before calling peer_disconnected

20 months agoDerive traits for InvoiceRequest and Invoice
Jeffrey Czyz [Wed, 8 Feb 2023 00:29:08 +0000 (18:29 -0600)]
Derive traits for InvoiceRequest and Invoice

Offer and Refund derive Debug, Clone, and PartialEq. For consistency,
derive these traits for InvoiceRequest and Invoice as well.

20 months agoMerge pull request #2040 from alecchendev/2023-02-indexed-map-btreeset-to-vec
Matt Corallo [Tue, 21 Feb 2023 19:57:51 +0000 (19:57 +0000)]
Merge pull request #2040 from alecchendev/2023-02-indexed-map-btreeset-to-vec

Replace `BTreeSet` in `IndexedMap` with sorted `Vec`

20 months agoRemove the `peer_disconnected` `no_connection_possible` flag 2023-02-fix-no-con-discon
Matt Corallo [Tue, 21 Feb 2023 19:10:43 +0000 (19:10 +0000)]
Remove the `peer_disconnected` `no_connection_possible` flag

Long ago, we used the `no_connection_possible` to signal that a
peer has some unknown feature set or some other condition prevents
us from ever connecting to the given peer. In that case we'd
automatically force-close all channels with the given peer. This
was somewhat surprising to users so we removed the automatic
force-close, leaving the flag serving no LDK-internal purpose.

Distilling the concept of "can we connect to this peer again in the
future" to a simple flag turns out to be ripe with edge cases, so
users actually using the flag to force-close channels would likely
cause surprising behavior.

Thus, there's really not a lot of reason to keep the flag,
especially given its untested and likely to be broken in subtle
ways anyway.

20 months agoAdd a further debug_assert that disconnecting peers are connected
Matt Corallo [Wed, 15 Feb 2023 06:29:29 +0000 (06:29 +0000)]
Add a further debug_assert that disconnecting peers are connected

20 months agoCorrect `funding_transaction_generated` err msg and fix fuzz check
Matt Corallo [Wed, 15 Feb 2023 01:23:20 +0000 (01:23 +0000)]
Correct `funding_transaction_generated` err msg and fix fuzz check

This fixes new errors in `full_stack_target` pointed out by
Chaincode's generous fuzzing infrastructure. Specifically, there's
no reason to check the error message in the
`funding_transaction_generated` return value - it can only return
a failure if the channel has closed since the funding transaction
was generated (which is fine) or if the signer refuses to sign
(which can't happen in fuzzing).

20 months agoCorrect the "is peer live" checks in `PeerManager`
Matt Corallo [Wed, 15 Feb 2023 01:20:38 +0000 (01:20 +0000)]
Correct the "is peer live" checks in `PeerManager`

In general, we should be checking if a `Peer` has `their_features`
set as the "is this peer connected and have they finished the
handshake" flag as it indicates an `Init` message was received.

While none of these appear to be reachable bugs, there were a
number of places where we checked other flags for this purpose,
which may lead to sending messages before `Init` in the future.

Here we clean these cases up to always use the correct check (via
the new util method).

20 months agoAdd test of an initial message other than `Init`
Matt Corallo [Wed, 15 Feb 2023 01:54:21 +0000 (01:54 +0000)]
Add test of an initial message other than `Init`

This test fails without the previous commit.

20 months agoFix (and DRY) the conditionals before calling `peer_disconnected`
Matt Corallo [Wed, 15 Feb 2023 01:13:57 +0000 (01:13 +0000)]
Fix (and DRY) the conditionals before calling `peer_disconnected`

If we have a peer that sends a non-`Init` first message, we'll call
`peer_disconnected` without ever having called `peer_connected`
(which has to wait until we have an `Init` message). This is a
violation of our API guarantees, though should generally not be an
issue.

Because this bug was repeated in a few places, we also take this
opportunity to DRY up the logic which checks the peer state before
calling `peer_disconnected`.

Found by the new `ChannelManager` assertions and the
`full_stack_target` fuzzer.

20 months agoReplace `BTreeSet` in `IndexedMap` with sorted `Vec`
Alec Chen [Thu, 16 Feb 2023 22:34:06 +0000 (16:34 -0600)]
Replace `BTreeSet` in `IndexedMap` with sorted `Vec`

The `Vec` is sorted not on `IndexedMap::insert`, but on
`IndexedMap::range` to avoid unnecessary work while reading a network
graph.

20 months agoPass pending_events into pay_internal
Valentine Wallace [Thu, 9 Feb 2023 19:34:37 +0000 (13:34 -0600)]
Pass pending_events into pay_internal

Useful for generating Payment(Path)Failed events in this method

20 months agoPass payment hash into pay_internal
Valentine Wallace [Thu, 9 Feb 2023 19:21:52 +0000 (13:21 -0600)]
Pass payment hash into pay_internal

Useful for generating Payment(Path)Failed events in this method

20 months agoMerge pull request #2026 from valentinewallace/2023-02-dedup-pending-forwardable-evs
Matt Corallo [Sun, 19 Feb 2023 18:25:03 +0000 (18:25 +0000)]
Merge pull request #2026 from valentinewallace/2023-02-dedup-pending-forwardable-evs

Deduplicate `PendingHTLCsForwardable` events on generation

20 months agoDeduplicate PendingHTLCsForwardable events when queueing
Valentine Wallace [Fri, 17 Feb 2023 22:41:21 +0000 (17:41 -0500)]
Deduplicate PendingHTLCsForwardable events when queueing

20 months agoOn retryable update_fail, don't queue redundant PendingHTLCsForwardable
Valentine Wallace [Fri, 17 Feb 2023 22:35:09 +0000 (17:35 -0500)]
On retryable update_fail, don't queue redundant PendingHTLCsForwardable

20 months agoCheck for abandon-able payments on startup
Valentine Wallace [Fri, 17 Feb 2023 22:14:43 +0000 (17:14 -0500)]
Check for abandon-able payments on startup

20 months agoAdd a new monitor update result handling macro
Matt Corallo [Sat, 3 Dec 2022 04:25:37 +0000 (04:25 +0000)]
Add a new monitor update result handling macro

Over the next few commits, this macro will replace the
`handle_monitor_update_res` macro. It takes a different approach -
instead of receiving the message(s) that need to be re-sent after
the monitor update completes and pushing them back into the
channel, we'll not get the messages from the channel at all until
we're ready for them.

This will unify our message sending into only actually fetching +
sending messages in the common monitor-update-completed code,
rather than both there *and* in the functions that call `Channel`
when new messages are originated.

20 months agoAdd storage for `ChannelMonitorUpdate`s in `Channel`s
Matt Corallo [Mon, 19 Dec 2022 20:41:42 +0000 (20:41 +0000)]
Add storage for `ChannelMonitorUpdate`s in `Channel`s

In order to support fully async `ChannelMonitor` updating, we need
to ensure that we can replay `ChannelMonitorUpdate`s if we shut
down after persisting a `ChannelManager` but without completing a
`ChannelMonitorUpdate` persistence. In order to support that we
(obviously) have to store the `ChannelMonitorUpdate`s in the
`ChannelManager`, which we do here inside the `Channel`.

We do so now because in the coming commits we will start using the
async persistence flow for all updates, and while we won't yet
support fully async monitor updating it's nice to get some of the
foundational structures in place now.

20 months agoTrack actions to execute after a `ChannelMonitor` is updated
Matt Corallo [Wed, 30 Nov 2022 18:49:44 +0000 (18:49 +0000)]
Track actions to execute after a `ChannelMonitor` is updated

When a `ChannelMonitor` update completes, we may need to take some
further action, such as exposing an `Event` to the user initiating
another `ChannelMonitor` update, etc. This commit adds the basic
structure to track such actions and serialize them as required.

Note that while this does introduce a new map which is written as
an even value which users cannot opt out of, the map is only filled
in when users use the asynchronous `ChannelMonitor` updates. As
these are still considered beta, breaking downgrades for such users
is considered acceptable here.

20 months agoAdd an infallible no-sign version of send_commitment_no_status_check
Matt Corallo [Thu, 1 Dec 2022 00:25:32 +0000 (00:25 +0000)]
Add an infallible no-sign version of send_commitment_no_status_check

In the coming commits we'll move to async `ChannelMonitorUpdate`
application, which means we'll want to generate a
`ChannelMonitorUpdate` (including a new counterparty commitment
transaction) before we actually send it to our counterparty. To do
that today we'd have to actually sign the commitment transaction
by calling the signer, then drop it, apply the
`ChannelMonitorUpdate`, then re-sign the commitment transaction to
send it to our peer.

In this commit we instead split `send_commitment_no_status_check`
and `send_commitment_no_state_update` into `build_` and `send_`
variants, allowing us to generate new counterparty commitment
transactions without actually signing, then build them for sending,
with signatures, later.

20 months agoFix the err msg provided when a send fails due to peer disconnected
Matt Corallo [Wed, 1 Feb 2023 20:54:10 +0000 (20:54 +0000)]
Fix the err msg provided when a send fails due to peer disconnected

We haven't had a `MonitorUpdateInProgress` check in
`Channel::is_live` for some time.

20 months agoMerge pull request #2009 from TheBlueMatt/2023-02-no-racey-retries
Matt Corallo [Thu, 16 Feb 2023 23:41:09 +0000 (23:41 +0000)]
Merge pull request #2009 from TheBlueMatt/2023-02-no-racey-retries

Fix (and test) threaded payment retries

20 months agoFix (and test) threaded payment retries 2023-02-no-racey-retries
Matt Corallo [Fri, 3 Feb 2023 23:05:58 +0000 (23:05 +0000)]
Fix (and test) threaded payment retries

The new in-`ChannelManager` retries logic does retries as two
separate steps, under two separate locks - first it calculates
the amount that needs to be retried, then it actually sends it.
Because the first step doesn't udpate the amount, a second thread
may come along and calculate the same amount and end up retrying
duplicatively.

Because we generally shouldn't ever be processing retries at the
same time, the fix is trivial - simply take a lock at the top of
the retry loop and hold it until we're done.

20 months agoAlways check `next_route` in `TestRouter` is fully consumed
Matt Corallo [Fri, 3 Feb 2023 02:17:41 +0000 (02:17 +0000)]
Always check `next_route` in `TestRouter` is fully consumed

...rather than only in std.

20 months agoTest if a given mutex is locked by the current thread in tests
Matt Corallo [Tue, 7 Feb 2023 19:46:08 +0000 (19:46 +0000)]
Test if a given mutex is locked by the current thread in tests

In anticipation of the next commit(s) adding threaded tests, we
need to ensure our lockorder checks work fine with multiple
threads. Sadly, currently we have tests in the form
`assert!(mutex.try_lock().is_ok())` to assert that a given mutex is
not locked by the caller to a function.

The fix is rather simple given we already track mutexes locked by a
thread in our `debug_sync` logic - simply replace the check with a
new extension trait which (for test builds) checks the locked state
by only looking at what was locked by the current thread.

20 months agoMerge pull request #2037 from valentinewallace/2023-02-fix-probe-leak
Matt Corallo [Thu, 16 Feb 2023 21:29:26 +0000 (21:29 +0000)]
Merge pull request #2037 from valentinewallace/2023-02-fix-probe-leak

Fix outbound payments probe leak

20 months agoMerge pull request #1921 from arik-so/2022-12-prohibit-old-rgs-updates
Matt Corallo [Thu, 16 Feb 2023 21:20:28 +0000 (21:20 +0000)]
Merge pull request #1921 from arik-so/2022-12-prohibit-old-rgs-updates

Throw error for RGS data that's more than two weeks old

20 months agoThrow error for RGS data that's more than two weeks old, fixing #1785
Arik Sosman [Fri, 16 Dec 2022 18:52:56 +0000 (10:52 -0800)]
Throw error for RGS data that's more than two weeks old, fixing #1785

20 months agoRemove pending probes on update_fail
Valentine Wallace [Wed, 15 Feb 2023 22:30:47 +0000 (17:30 -0500)]
Remove pending probes on update_fail

Previously we had a memory leak where probes would not be removed from
outbound_payments on htlc fail

20 months agoMerge pull request #2008 from valentinewallace/2023-02-remove-manual-retries
Matt Corallo [Thu, 16 Feb 2023 00:49:04 +0000 (00:49 +0000)]
Merge pull request #2008 from valentinewallace/2023-02-remove-manual-retries

Abandon payments on behalf of the user and remove manual retries

20 months agoReword and fix grammar in PartialFailure docs
Valentine Wallace [Mon, 13 Feb 2023 16:36:46 +0000 (11:36 -0500)]
Reword and fix grammar in PartialFailure docs

20 months agoRemove retry_payments method
Valentine Wallace [Sun, 5 Feb 2023 22:05:12 +0000 (17:05 -0500)]
Remove retry_payments method

We're no longer supporting manual retries since
ChannelManager::send_payment_with_retry can be parameterized by a retry
strategy

This commit also updates all docs related to retry_payment and abandon_payment.
Since these docs frequently overlap with changes in preceding commits where we
start abandoning payments on behalf of the user, all the docs are updated in
one go.

20 months agoWhen processing pending htlcs, abandon outbounds that are not retryable
Valentine Wallace [Sat, 4 Feb 2023 02:44:08 +0000 (21:44 -0500)]
When processing pending htlcs, abandon outbounds that are not retryable

20 months agoAbandon payment on behalf of the user on payment path failure
Valentine Wallace [Fri, 3 Feb 2023 17:53:01 +0000 (12:53 -0500)]
Abandon payment on behalf of the user on payment path failure

Removed retry_single_path_payment, it's replaced by automatic_retries with
AutoRetry::Success

20 months agoAbandon payment if retry fails in process_pending_htlcs
Valentine Wallace [Fri, 3 Feb 2023 17:49:07 +0000 (12:49 -0500)]
Abandon payment if retry fails in process_pending_htlcs

20 months agoMerge pull request #1832 from jkczyz/2022-11-composite-handler
Matt Corallo [Wed, 15 Feb 2023 17:45:39 +0000 (17:45 +0000)]
Merge pull request #1832 from jkczyz/2022-11-composite-handler

Macro for composing custom message handlers

20 months agoPass pending events to outbound_payments::abandon_payment
Valentine Wallace [Fri, 3 Feb 2023 17:44:14 +0000 (12:44 -0500)]
Pass pending events to outbound_payments::abandon_payment

This makes it uniform with the outbound payment methods that generate events
and set us up for abandoning payments on behalf of the user.

20 months agoFix indentation in outbound payment mark_abandoned
Valentine Wallace [Fri, 3 Feb 2023 16:53:44 +0000 (11:53 -0500)]
Fix indentation in outbound payment mark_abandoned

20 months agoMerge pull request #2020 from valentinewallace/2023-02-test-inflight-scoring
Matt Corallo [Wed, 15 Feb 2023 01:25:09 +0000 (01:25 +0000)]
Merge pull request #2020 from valentinewallace/2023-02-test-inflight-scoring

Fix and test in-flight HTLC scoring in between retries

20 months agoAdd CI testing for lightning-custom-message crate
Jeffrey Czyz [Thu, 2 Feb 2023 03:35:38 +0000 (21:35 -0600)]
Add CI testing for lightning-custom-message crate

20 months agoRe-write CustomMessageHandler documentation
Jeffrey Czyz [Tue, 31 Jan 2023 16:44:19 +0000 (10:44 -0600)]
Re-write CustomMessageHandler documentation

Documentation for CustomMessageHandler wasn't clear how it is related to
PeerManager and contained some grammatical and factual errors. Re-write
the docs and link to the lightning_custom_message crate.