Matt Corallo [Thu, 19 Jan 2023 18:24:30 +0000 (18:24 +0000)]
Update min-inbound-fee values on `NetworkGraph` load
Historically we've had various bugs in keeping the
`lowest_inbound_channel_fees` field in `NodeInfo` up-to-date as we
go. This leaves the A* routing less efficient as it can't prune
hops as aggressively.
In order to get accurate benchmarks, this commit updates the
minimum-inbound-fees field on load. This is not the most efficient
way of doing so, but suffices for fetching benchmarks and will be
removed in the coming commits.
Note that this is *slower* than the non-updating version in the
previous commit. While I haven't dug into this incredibly deeply,
the graph snapshot in use has min-fee info for only 9,618 of
20,818 nodes. Thus, it is my guess that with the graph snapshot
as-is the branch predictor is able to largely remove the A*
heuristic lookups, but with this change it is forced to wait for
A* heuristic map lookups to complete, causing a performance
regression.
```
test routing::router::benches::generate_mpp_routes_with_probabilistic_scorer ... bench: 182,980,059 ns/iter (+/- 32,662,047)
test routing::router::benches::generate_mpp_routes_with_zero_penalty_scorer ... bench: 151,170,457 ns/iter (+/- 75,351,011)
test routing::router::benches::generate_routes_with_probabilistic_scorer ... bench: 58,187,277 ns/iter (+/- 11,606,440)
test routing::router::benches::generate_routes_with_zero_penalty_scorer ... bench: 41,210,193 ns/iter (+/- 18,103,320)
```
Wilmer Paulino [Wed, 18 Jan 2023 21:43:32 +0000 (13:43 -0800)]
Remove NodeSigner::get_node_secret
Secrets should not be exposed in-memory at the interface level as it
would be impossible the implement it against a hardware security
module/secure element.
Matt Corallo [Tue, 17 Jan 2023 23:40:44 +0000 (23:40 +0000)]
Use `test`/`_test_utils` to enable single-threaded debug assertions
We have a number of debug assertions which are expected to never
fire when running in a single thread. This is just fine in tests,
and gives us good coverage of our lockorder requirements, but is
not-irregularly surprising to users, who may run with their own
debug assertions in test environments.
Instead, we gate these checks by the `cfg(test)` setting as well as
the `_test_utils` feature, ensuring they run in our own tests, but
not downstream tests.
Matt Corallo [Mon, 28 Nov 2022 01:00:38 +0000 (01:00 +0000)]
Use a variable-length integer for many serialization wrappers
The lightning protocol uses u16s for lengths in most cases. As our
serialization framework primarily targets that, we must as well.
However, because we may serialize objects that have more than 65K
entries, we want to be able to store larger values. Thus, we define
a variable length integer here which is backwards-compatible but
treats 0xffff as "read eight more bytes".
This doesn't address any specific known issue, but feels like good
practice just in case.
Matt Corallo [Tue, 17 Jan 2023 00:16:48 +0000 (00:16 +0000)]
Make `background-processor` no-std-friendly (ish)
This makes `background-processor` build without `std` at all. This
isn't particularly useful in the general no-std case as
`background-processor` is only useful with the `futures` feature,
and async will generally need `std` in some way or another. Still,
it ensures we don't end up reintroducing a dependency on the
current time, which breaks `wasm` use-cases.
Matt Corallo [Tue, 17 Jan 2023 00:03:43 +0000 (00:03 +0000)]
Use the user-provided `SleepFuture` for interval checks in BP
`background-processor` does a number of jobs on various timers.
Instead of doing those by interrogating `std::time::Instant`, this
change swaps to using the existing user-provided sleep future.
Matt Corallo [Mon, 16 Jan 2023 23:47:11 +0000 (23:47 +0000)]
Use flexible timer types in background processor's regular jobs
`background-processor` does a number of jobs on various timers.
Currently, those are all done by checking the timers every 100ms
by interrogating `std::time::Instant`. This is fine for the
threaded version, but we'd like more flexibility in the `futures`-
based `background-processor`.
Here we swap the `std::time::Instant` interrogation for a lambda
which we will switch out to the user-provided sleeper in the next
commit.
Elias Rohrer [Mon, 16 Jan 2023 22:54:53 +0000 (16:54 -0600)]
Make `InvoicePayer`'s `Router` `Deref`
As of HEAD the `ChannelManager` is parametrized by a `Router`, while
`InvoicePayer` also owns a `Router`. In order to allow for a single
object being reused, we make the `InvoicePayer` side `Deref`.
Matt Corallo [Sat, 12 Nov 2022 18:26:38 +0000 (18:26 +0000)]
Pass monitor updates by reference, not owned
In the next commit(s) we'll start holding `ChannelMonitorUpdate`s
that are being persisted in `Channel`s until they're done
persisting. In order to do that, switch to applying the updates by
reference instead of value.
Matt Corallo [Sun, 15 Jan 2023 23:37:00 +0000 (23:37 +0000)]
Use `ChannelUnavailable` for a peer disconnecting not `MisuseError`
This fixes a crash in the `full_stack_target` fuzz test (found by
Chaincode's generous fuzzing infrastructure!) but ultimately is a
better error code - a peer disconnecting before we can fund a
channel isn't a "misuse error" its an unavailable channel.
Matt Corallo [Fri, 6 Jan 2023 20:05:07 +0000 (20:05 +0000)]
Do not rely on auto-deref'ing when aaccessing a `Hash{Map,Set}`
In newer versions of `hashbrown` this code would be broken. While
we aren't updating `hashbrown` any time soon (as it requires an
MSRV bump), it is useful to swap for a newer `hashbrown` when
fuzzing, which this makes easier.
Matt Corallo [Fri, 6 Jan 2023 19:53:47 +0000 (19:53 +0000)]
Switch to hashbrown in fuzzing
hashbrown by default uses ahash, which may be a bit faster, but
more importantly, if we upgrade to hashbrown 0.13/ahash 0.8 we can
make it use a constant randomization factor, making fuzzers happier.
Matt Corallo [Mon, 28 Nov 2022 00:42:15 +0000 (00:42 +0000)]
Convert `Vec` de/serialization impl to a macro and impl for tuples
...to make it easier to add new implementations and implement it
for all tuples which implement `Readabe` + `Writeable`. Note that
we don't want to just convert to a blanket implementation as we'd
really like to keep our optimized `Vec<u8>` wrapper or we'll end
up spinning way too much when writing vecs of bytes.
Matt Corallo [Tue, 22 Nov 2022 22:41:13 +0000 (22:41 +0000)]
Add support for including `ignorable` types in enum de/ser macros
An enum implements de/serialization via
`impl_writeable_tlv_based_enum_upgradable` currently cannot contain
an object that only implements `MaybeReadable`.
This solves that by implementing the required blocks for
`ignorable`, opting to return `Ok(None)` in the top-level read in
cases where the inner field read returns `Ok(None)`.
Wilmer Paulino [Wed, 11 Jan 2023 18:21:29 +0000 (10:21 -0800)]
Use UserConfig to determine advertised InitFeatures by ChannelManager
This is purely a refactor that does not change the InitFeatures
advertised by a ChannelManager. This allows users to configure which
features should be advertised based on the values of `UserConfig`. While
there aren't any existing features currently leveraging this behavior,
it will be used by the upcoming anchors_zero_fee_htlc_tx feature.
The UserConfig dependency on provided_init_features caused most
callsites of the main test methods responsible for opening channels to
be updated. This commit foregos that completely by no longer requiring
the InitFeatures of each side to be provided to these methods. The
methods already require a reference to each node's ChannelManager to
open the channel, so we use that same reference to obtain their
InitFeatures. A way to override such features was required for some
tests, so a new `override_init_features` config option now exists on
the test harness.
Matt Corallo [Tue, 10 Jan 2023 06:37:39 +0000 (06:37 +0000)]
Make `debug_sync` regex more robust
On windows the symbol names appear to sometimes be truncated,
which causes the symbol name to not include the `::new` at the end.
This causes the regex to mis-match and track the wrong location
for the mutex construction, leading to bogus lockorder violations.
For example, in testing the following symbol name appeared on
Windows, without the function name itself:
Omer Yacine [Fri, 6 Jan 2023 08:18:26 +0000 (10:18 +0200)]
Expose `impl_writeable_tlv_based` macro
Every exported macro needed to have all the macros used inside it:
1- to be exported as well.
2- be called from the `$crate` namespace so it works in other crates.
Some structs in `lightning::util::ser` needed to be made public as they were used inside the exported macros.
Use the macros like this:
```Rust
lightning::impl_writeable_tlv_based!(...)
```
Jeffrey Czyz [Fri, 16 Dec 2022 20:06:33 +0000 (14:06 -0600)]
Define blinded hop features for use in BOLT 12
BOLT 12 invoices may contain blinded_payinfo for each hop in a blinded
path. Each blinded_payinfo contains features, whose length must be
encoded since there may be multiple hops.
Note these features are also needed in the BOLT 4 encrypted_data_tlv
stream. But since they are a single TLV record, the length must *not* be
encoded there.
Jeffrey Czyz [Fri, 16 Dec 2022 19:35:50 +0000 (13:35 -0600)]
Use explicit WithoutLength for BOLT 12 features
Most BOLT 12 features are used as the value of a TLV record and thus
don't use an explicit length. One exception is the features inside the
blinded payinfo subtype since the TLV record contains a list of them.
However, these features are also used in the BOLT 4 encrypted_data_tlv
TLV stream as a single record, where the length is implicit.
Implement Readable and Writeable for Features wrapped in WithoutLength
such that either serialization can be used where required.