Matt Corallo [Mon, 16 Jan 2023 23:18:39 +0000 (23:18 +0000)]
Calc decayed buckets to decide if we have valid historical points
When we're calculating if, once we apply the unupdated decays, the
historical data tracker has enough data to assign a score, we
previously calculated the decayed points while walking the buckets
as we don't use the decayed buckets anyway (to avoid losing
precision). That is fine, except that as written it decayed
individual buckets additional times.
Instead, here we actually calculate the full set of decayed buckets
and use those to decide if we have valid points. This adds some
additional stack space and may in fact be slower, but will be
useful in the next commit and shouldn't be a huge change.
Jeffrey Czyz [Thu, 19 Jan 2023 00:58:20 +0000 (18:58 -0600)]
Disallow offer_metadata in Refund
The offer_metadata was optional but is redundant with invreq_metadata
(i.e., payer_metadata) for refunds. It is now disallowed in the spec and
was already unsupported by RefundBuilder.
Jeffrey Czyz [Wed, 18 Jan 2023 23:29:31 +0000 (17:29 -0600)]
Allow quantity in Refund
The spec always allowed this but the reason was unclear. It's useful if
the refund is for an invoice paid for offer where a quantity was given
in the request. The description in the refund would be from the offer,
which may have given a unit for each item. So allowing a quantity makes
it clear how many items the refund is for.
Jeffrey Czyz [Wed, 18 Jan 2023 22:44:16 +0000 (16:44 -0600)]
Support explicit quantity_max = 1 in Offer
The spec was modified to allow setting offer_quantity_max explicitly to
one. This is to support a use case where more than one item is supported
but only one item is left in the inventory. Introduce a Quantity::One
variant to replace Quantity::Bounded(1) so the later can be used for the
explicit setting.
Matt Corallo [Thu, 26 Jan 2023 02:21:31 +0000 (02:21 +0000)]
Remove the `ChannelMonitor` secp context
`ChannelMonitor` indirectly already has a context - the
`OnchainTxHandler` has one. This makes it trivial to remove the
existing one, so we do so for a free memory usage reduction.
Matt Corallo [Thu, 26 Jan 2023 02:23:08 +0000 (02:23 +0000)]
Implement `PartialEq` for `ChannelMonitor`
It turns out `#[derive(PartialEq)]` will automatically bound the
`PartialEq` implementation by any bounds on the struct also being
`PartialEq`. This means to use an auto-derived `ChannelMonitor`
`PartialEq` the `EcdsaSigner` used must also be `PartialEq`, but
for the use-cases we have today for a `ChannelMonitor` `PartialEq`
it doesn't really matter - we use it internally in tests and
downstream users wanted similar test-only usage.
Matt Corallo [Wed, 25 Jan 2023 02:56:13 +0000 (02:56 +0000)]
Make `test_duplicate_payment_hash_one_failure_one_success` robust
`test_duplicate_payment_hash_one_failure_one_success` currently
fails if the "wrong" HTLC is picked to be claimed. Given the HTLCs
are identical, there's no way to figure out which we should claim.
The test instead relies on a magic value - the first one is the
right one....unless we change our CSPRNG implementation. When we
try to do so, the test randomly fails.
Here we change one HTLC to a lower amount so we can figure out
which transaction to broadcast to make the test robust against
CSPRNG changes.
Alec Chen [Wed, 25 Jan 2023 18:27:59 +0000 (12:27 -0600)]
Add FailureCode enum and ChannelManager::fail_htlc_backwards_with_reason
FailureCode is used to specify which error code and data to send
to peers when failing back an HTLC.
ChannelManager::fail_htlc_backwards_with_reason
allows a user to specify the error code and
corresponding data to send to peers when failing back an HTLC.
This function is mentioned in Event::PaymentClaimable docs.
ChannelManager::get_htlc_fail_reason_from_failure_code was also
added to assist with this function.
Decode onion fail outside of outbound_payments lock
It's not ideal to do all this computation while the lock is held. We also want
to decode the failure *before* taking the lock, so we can store the failed scid
in the relevant outbound for retry in the next commit(s).
Matt Corallo [Wed, 25 Jan 2023 17:42:20 +0000 (17:42 +0000)]
Clean up `compute_fees` and add a saturating variant
Often when we call `compute_fees` we really just want it to
saturate and we deal with `u64::max_value` later. In that case,
we're much better off doing the saturating in the `compute_fees` as
it can use CMOVs rather than branching at each step and then
`unwrap_or`ing at the callsite.
Matt Corallo [Thu, 19 Jan 2023 17:59:10 +0000 (17:59 +0000)]
Swap `IndexedMap` implementation for a `HashMap`+B-Tree
Our network graph has to be iterable in a deterministic order and
with the ability to iterate over a specific range. Thus,
historically, we've used a `BTreeMap` to do the iteration. This is
fine, except our map needs to also provide high performance lookups
in order to make route-finding fast. Sadly, `BTreeMap`s are quite
slow due to the branching penalty.
Here we replace the implementation of our `IndexedMap` with a
`HashMap` to store the elements itself and a `BTreeSet` to store
the keys set in sorted order for iteration.
As of this commit on the same hardware as the above few commits,
the benchmark results are:
```
test routing::router::benches::generate_mpp_routes_with_probabilistic_scorer ... bench: 109,544,993 ns/iter (+/- 27,553,574)
test routing::router::benches::generate_mpp_routes_with_zero_penalty_scorer ... bench: 81,164,590 ns/iter (+/- 55,422,930)
test routing::router::benches::generate_routes_with_probabilistic_scorer ... bench: 34,726,569 ns/iter (+/- 9,646,345)
test routing::router::benches::generate_routes_with_zero_penalty_scorer ... bench: 22,772,355 ns/iter (+/- 9,574,418)
```
Matt Corallo [Tue, 25 Oct 2022 03:50:07 +0000 (03:50 +0000)]
Add a new `IndexedMap` type and use it in network graph storage
Our network graph has to be iterable in a deterministic order and
with the ability to iterate over a specific range. Thus,
historically, we've used a `BTreeMap` to do the iteration. This is
fine, except our map needs to also provide high performance lookups
in order to make route-finding fast. Sadly, `BTreeMap`s are quite
slow due to the branching penalty.
Here we replace the `BTreeMap`s in the scorer with a dummy wrapper.
In the next commit the internals thereof will be replaced with a
`HashMap`-based implementation.
Matt Corallo [Tue, 25 Oct 2022 03:15:03 +0000 (03:15 +0000)]
Drop A* implementation in the router for simple Dijkstra's
As evidenced by the previous commit, it appears our A* router
does worse than a more naive approach. This isn't super surpsising,
as the A* heuristic calculation requires a map lookup, which is
relatively expensive.
```
test routing::router::benches::generate_mpp_routes_with_probabilistic_scorer ... bench: 169,991,943 ns/iter (+/- 30,838,048)
test routing::router::benches::generate_mpp_routes_with_zero_penalty_scorer ... bench: 122,144,987 ns/iter (+/- 61,708,911)
test routing::router::benches::generate_routes_with_probabilistic_scorer ... bench: 48,546,068 ns/iter (+/- 10,379,642)
test routing::router::benches::generate_routes_with_zero_penalty_scorer ... bench: 32,898,557 ns/iter (+/- 14,157,641)
```
Duncan Dean [Mon, 21 Nov 2022 12:53:52 +0000 (14:53 +0200)]
Add `min_final_cltv_expiry` parameter to invoice utils
All utility functions for invoice construction will now also accept an
Option<>al `min_final_cltv_expiry_delta` which is useful for things like
swaps etc. The `min_final_cltv_expiry_delta` will default back to
`MIN_FINAL_CLTV_EXPIRY_DELTA` if `None` is provided.
Duncan Dean [Tue, 29 Nov 2022 10:47:00 +0000 (12:47 +0200)]
Add `_delta` suffix to `min_final_cltv_expiry`
This matches the spec and helps avoid any confusion around
naming. We're also then consistent with `cltv_expiry` in an HTLC being
the actual block height value for the CLTV and not a delta.
Jeffrey Czyz [Fri, 13 Jan 2023 05:02:39 +0000 (23:02 -0600)]
Use SystemTime::now() for Invoice creation time
For std builds, Invoice::created_at can be automatically set upon
construction using SystemTime::now() offset by SystemTime::UNIX_EPOCH.
Change InvoiceRequest::respond_with and Refund::respond_with to only
take a created_at parameter in no-std builds.
Jeffrey Czyz [Tue, 20 Dec 2022 15:33:11 +0000 (09:33 -0600)]
Builder for creating invoices for refunds
Add a builder for creating invoices for a refund and required fields.
Other settings are optional and duplicative settings will override
previous settings. Building produces a semantically valid `invoice`
message for the refund, which then can be signed with the key associated
with the provided signing pubkey.
Jeffrey Czyz [Tue, 20 Dec 2022 04:23:39 +0000 (22:23 -0600)]
Builder for creating invoices for offers
Add a builder for creating invoices for an offer from a given request
and required fields. Other settings are optional and duplicative
settings will override previous settings. Building produces a
semantically valid `invoice` message for the offer, which then can be
signed with the key associated with the offer's signing pubkey.
Define an interface for BOLT 12 `invoice` messages. The underlying
format consists of the original bytes and the parsed contents.
The bytes are later needed for serialization. This is because it must
mirror all the `offer` and `invoice_request` TLV records, including
unknown ones, which aren't represented in the contents.
Invoices may be created for an Offer (from an InvoiceRequest) or for a
Refund. The primary difference is how the signing pubkey is given -- by
the writer of the offer or the reader of the refund.
Matt Corallo [Thu, 19 Jan 2023 18:24:30 +0000 (18:24 +0000)]
Update min-inbound-fee values on `NetworkGraph` load
Historically we've had various bugs in keeping the
`lowest_inbound_channel_fees` field in `NodeInfo` up-to-date as we
go. This leaves the A* routing less efficient as it can't prune
hops as aggressively.
In order to get accurate benchmarks, this commit updates the
minimum-inbound-fees field on load. This is not the most efficient
way of doing so, but suffices for fetching benchmarks and will be
removed in the coming commits.
Note that this is *slower* than the non-updating version in the
previous commit. While I haven't dug into this incredibly deeply,
the graph snapshot in use has min-fee info for only 9,618 of
20,818 nodes. Thus, it is my guess that with the graph snapshot
as-is the branch predictor is able to largely remove the A*
heuristic lookups, but with this change it is forced to wait for
A* heuristic map lookups to complete, causing a performance
regression.
```
test routing::router::benches::generate_mpp_routes_with_probabilistic_scorer ... bench: 182,980,059 ns/iter (+/- 32,662,047)
test routing::router::benches::generate_mpp_routes_with_zero_penalty_scorer ... bench: 151,170,457 ns/iter (+/- 75,351,011)
test routing::router::benches::generate_routes_with_probabilistic_scorer ... bench: 58,187,277 ns/iter (+/- 11,606,440)
test routing::router::benches::generate_routes_with_zero_penalty_scorer ... bench: 41,210,193 ns/iter (+/- 18,103,320)
```
Wilmer Paulino [Wed, 18 Jan 2023 21:43:32 +0000 (13:43 -0800)]
Remove NodeSigner::get_node_secret
Secrets should not be exposed in-memory at the interface level as it
would be impossible the implement it against a hardware security
module/secure element.
Jeffrey Czyz [Thu, 22 Dec 2022 15:33:41 +0000 (09:33 -0600)]
Encoding for TLV stream without signature records
When using bytes from an InvoiceRequest to constructing bytes for an
Invoice, any signature TLV records in the bytes must be excluded. Define
a wrapper for encoding such pre-serialized bytes in this manner. This
will allow the forthcoming InvoiceBuilder to construct bytes for an
Invoice properly.
Jeffrey Czyz [Thu, 22 Dec 2022 15:10:21 +0000 (09:10 -0600)]
Define TlvStream::skip_signatures
Provide a helper for skipping signature TLV records from a TLV stream.
This prevents needing to duplicate the check for signature TLV records
when writing a TLV stream without signatures in an upcoming commit.
Matt Corallo [Tue, 17 Jan 2023 23:40:44 +0000 (23:40 +0000)]
Use `test`/`_test_utils` to enable single-threaded debug assertions
We have a number of debug assertions which are expected to never
fire when running in a single thread. This is just fine in tests,
and gives us good coverage of our lockorder requirements, but is
not-irregularly surprising to users, who may run with their own
debug assertions in test environments.
Instead, we gate these checks by the `cfg(test)` setting as well as
the `_test_utils` feature, ensuring they run in our own tests, but
not downstream tests.
Matt Corallo [Mon, 28 Nov 2022 01:00:38 +0000 (01:00 +0000)]
Use a variable-length integer for many serialization wrappers
The lightning protocol uses u16s for lengths in most cases. As our
serialization framework primarily targets that, we must as well.
However, because we may serialize objects that have more than 65K
entries, we want to be able to store larger values. Thus, we define
a variable length integer here which is backwards-compatible but
treats 0xffff as "read eight more bytes".
This doesn't address any specific known issue, but feels like good
practice just in case.
Matt Corallo [Tue, 17 Jan 2023 00:16:48 +0000 (00:16 +0000)]
Make `background-processor` no-std-friendly (ish)
This makes `background-processor` build without `std` at all. This
isn't particularly useful in the general no-std case as
`background-processor` is only useful with the `futures` feature,
and async will generally need `std` in some way or another. Still,
it ensures we don't end up reintroducing a dependency on the
current time, which breaks `wasm` use-cases.