BOLT 12 messages are limited to a range of TLV record types. Refactor
decode_tlv_stream into a decode_tlv_stream_range macro for limiting
which types are parsed. Requires a SeekReadable trait for rewinding when
a type outside of the range is seen. This allows for composing TLV
streams of different ranges.
Updates offer parsing accordingly and adds a test demonstrating failure
if a type outside of the range is included.
Jeffrey Czyz [Thu, 11 Aug 2022 21:51:06 +0000 (16:51 -0500)]
Offer parsing from bech32 strings
Add common bech32 parsing for BOLT 12 messages. The encoding is similar
to bech32 only without a checksum and with support for continuing
messages across multiple parts.
Messages implementing Bech32Encode are parsed into a TLV stream, which
is converted to the desired message content while performing semantic
checks. Checking after conversion allows for more elaborate checks of
data composed of multiple TLV records and for more meaningful error
messages.
The parsed bytes are also saved to allow creating messages with mirrored
data, even if TLV records are unknown.
Jeffrey Czyz [Tue, 9 Aug 2022 22:40:26 +0000 (17:40 -0500)]
Builder for creating offers
Add a builder for creating offers given a required description and
node_id. Other settings are optional and duplicative settings will
override previous settings for non-Vec fields.
Jeffrey Czyz [Fri, 24 Jun 2022 21:27:42 +0000 (16:27 -0500)]
Serialization macro for TLV streams
BOLT 12's offer message is encoded as a TLV stream (i.e., a sequence of
TLV records). impl_writeable_tlv_based can't be used because it writes
the overall length of the struct, whereas TLV streams only include the
length of each TLV record. Add a `tlv_stream` macro for defining structs
used in encoding.
TLV records containing a single variable-length type should not encode
the types length in the value since it is redundant. Add a wrapper type
that can be used within a TLV stream to support the correct behavior
during serialization and de-serialization.
Jeffrey Czyz [Fri, 7 Oct 2022 04:12:48 +0000 (23:12 -0500)]
Add WithoutLength wrapper
When serializing variable-length types as part of a TLV stream, the
length does not need to be serialized as it is already encoded in TLV
records. Add a WithoutLength wrapper for this encoding. Replace
VecReadWrapper and VecWriteWrapper with this single type to avoid
redundant encoders.
Jeffrey Czyz [Tue, 9 Aug 2022 22:24:10 +0000 (17:24 -0500)]
Offer message interface and data format
Define an interface for BOLT 12 `offer` messages. The underlying format
consists of the original bytes and the parsed contents.
The bytes are later needed when constructing an `invoice_request`
message. This is because it must mirror all the `offer` TLV records,
including unknown ones, which aren't represented in the contents.
The contents will be used in `invoice_request` messages to avoid
duplication. Some fields while required in a typical user-pays-merchant
flow may not be necessary in the merchant-pays-user flow (i.e., refund).
Jeffrey Czyz [Thu, 6 Oct 2022 18:44:35 +0000 (13:44 -0500)]
Add PrintableString utility
Strings defined by third parties may contain control characters. Provide
a wrapper such that these are replaced when displayed. Useful in node
aliases and offer fields.
Jeffrey Czyz [Fri, 24 Jun 2022 21:40:49 +0000 (16:40 -0500)]
Offer features for BOLT 12
The offer message in BOLT 12 contains a features TLV record. Add a
corresponding OfferFeatures type where the length is not included in the
serialization as it would be redundant with the record length.
Otherwise, define the features to be the same as InvoiceFeatures.
Make `process_pending_htlc_forwards` more readable
Refactor `process_pending_htlc_forwards` to ensure that both branches
that fails `pending_forwards` are placed next to eachother for improved
readability.
Consider `channel_id`s in `short_to_chan_info` as unguaranteed
As the `short_to_chan_info` map has been removed from the
`channel_state`, there is no longer any consistency guarantees between
the `by_id` and `short_to_chan_info` maps. This commit ensures that we
don't force unwrap channels where the channel_id has been queried from
the `short_to_chan_info` map.
As the `short_to_chan_info` has been moved out of the `channel_state` to
a standalone lock, several macros no longer need the `channel_state`
passed into the macro.
This is part of moving the Router trait into ChannelManager, which will help
allow ChannelManager to fetch routes on-the-fly as part of supporting
trampoline payments.
Elias Rohrer [Wed, 26 Oct 2022 09:24:28 +0000 (11:24 +0200)]
Rename chan state `ChannelFunded` to `ChannelReady`
We rename `ChannelState::ChannelFunded` to `ChannelState::ChannelReady`
as we'll be in this state when both sides sent the `ChannelReady`
messages, which may also be before funding in the 0conf case.
Matt Corallo [Mon, 31 Oct 2022 03:23:02 +0000 (03:23 +0000)]
Add missing break when scoring a path with a missing channel
If we send payments over a path where a channel ended up being
closed, we'll remove it before we call
`ProbabilisticPaymentScorer::payment_path_failed`. This should be
fine, except that `payment_path_failed` does not break out of its
scoring loop if a channel is missing, causing it to assign a
minimum available-liquidity of the payment amount even to channels
which our attempt never arrived at.
The fix is simple - add the missing check and break.
Avoid generating redundant claims after initial confirmation
These claims will never be valid as a previous claim has already
confirmed. If a previous claim is reorged out of the chain, a new claim
will be generated bypassing the new behavior.
While this doesn't change much for our existing transaction-based
claims, as broadcasting an already confirmed transaction acts as a NOP,
it prevents us from yielding redundant event-based claims, which will be
introduced as part of the anchors patchset.
Matt Corallo [Wed, 2 Nov 2022 01:06:39 +0000 (01:06 +0000)]
Allow users to specify the `PaymentId` used in `InvoicePayer`
In order to allow users to pass a custom idempotency key to the
`send*` methods in `InvoicePayer`, we have to pipe the `PaymentId`
through to the `Payer` methods, which we do here.
By default, existing `InvoicePayer` methods use the `PaymentHash`
as the `PaymentId`, however we also add duplicate `send*_with_id`
methods which allow users to pass a custom `PaymentId`.
Finally, appropriate documentation updates are made to clarify
idempotency guarantees.
Matt Corallo [Sat, 8 Oct 2022 19:54:43 +0000 (19:54 +0000)]
Stop timing out payments automatically, requiring abandon_payment
When the `abandon_payment` flow was added there was some concern
that upgrading users may not migrate to the new flow, causing
memory leaks in the pending-payment tracking.
While this is true, now that we're relying on the
pending_outbound_payments map for `send_payment` idempotency, the
risk of removing a payment prematurely goes up from "spurious
retry failure" to "sending a duplicative payment", which is much
worse.
Thus, we simply remove the automated payment timeout here,
explicitly requiring that users call `abandon_payment` when they
give up retrying a payment.
Matt Corallo [Sat, 8 Oct 2022 23:26:18 +0000 (23:26 +0000)]
Delay removal of fulfilled outbound payments for a few timer ticks
Previously, once a fulfilled outbound payment completed and all
associated HTLCs were resolved, we'd immediately remove the payment
entry from the `pending_outbound_payments` map.
Now that we're using the `pending_outbound_payments` map for send
idempotency, this presents a race condition - if the user makes a
redundant `send_payment` call at the same time that the original
payment's last HTLC is resolved, the user would reasonably expect
the `send_payment` call to fail due to our idempotency guarantees.
However, because the `pending_outbound_payments` entry is being
removed, if it completes first the `send_payment` call will
succeed even though the user has not had a chance to see the
corresponding `Event::PaymentSent`.
Instead, here, we delay removal of `Fulfilled`
`pending_outbound_payments` entries until several timer ticks have
passed without any corresponding event or HTLC pending.
Matt Corallo [Thu, 6 Oct 2022 21:31:02 +0000 (21:31 +0000)]
Allow users to specify the `PaymentId` for new outbound payments
In c986e52ce83e9aeaa9447abebc5f6600470337cf, an `MppId` was added
to `HTLCSource` objects as a way of correlating HTLCs which belong
to the same payment when the `ChannelManager` sees an HTLC
succeed/fail. This allows it to have awareness of the state of all
HTLCs in a payment when it generates the ultimate user-facing
payment success/failure events. This was used in the same PR to
avoid generating duplicative success/failure events for a single
payment.
Because the field was only used as an internal token to correlate
HTLCs, and retries were not supported, it was generated randomly by
calling the `KeysInterface`'s 32-byte random-fetching function.
This also provided a backwards-compatibility story as the existing
HTLC randomization key was re-used for older clients.
In 28eea12bbe0d78d256f79ec725cf02366dce4e36 `MppId` was renamed to
the current `PaymentId` which was then used expose the
`retry_payment` interface, allowing users to send new HTLCs which
are considered a part of an existing payment.
At no point has the payment-sending API seriously considered
idempotency, a major drawback which leaves the API unsafe in most
deployments. Luckily, there is a simple solution - because the
`PaymentId` must be unique, and because payment information for a
given payment is held for several blocks after a payment
completes/fails, it represents an obvious idempotency token.
Here we simply require the user provide the `PaymentId` directly in
`send_payment`, allowing them to use whatever token they may
already have for a payment's idempotency token.
Onion messages: fix edge case where we are the introduction node
If we're sending straight to a blinded route with no unblinded intermediate
hops, and we are the introduction node, we need to advance the blinded route by
one hop so that the second hop is the new introduction node.
Matt Corallo [Sat, 29 Oct 2022 18:30:37 +0000 (18:30 +0000)]
Make `htlc_maximum_msat` in `EffectiveCapacity` non-`Option`al
Because we now never generate an `EffectiveCapacity` with an
`htlc_maximum_msat` set to `None`, making it non-`Option`al
effectively removes dead code, which we do here.
Matt Corallo [Sat, 29 Oct 2022 18:24:36 +0000 (18:24 +0000)]
Require directional updates for a `DirectionalChannelInfo`
We currently construct `DirectedChannelInfo`s for routing before
checking if the given direction has its directional info filled in.
We then always check for directional info before actually deciding
to route over a channel, as otherwise we assume the channel is not
online.
This makes for somewhat redundant checks, and `DirectedCHannelInfo`
isn't, by itself, a very useful API. Because fetching the HTLC-max
or effective channel capacity gives spurious data if no directional
info is available, there's little reason to have that data
available, and so we here check for directional info first. This
effectively merges `DirectionalChannelInfo` and
`DirectionalChannelInfoWithUpdate`.
Matt Corallo [Wed, 26 Oct 2022 19:23:10 +0000 (19:23 +0000)]
Give us a self when reading a custom onion message
+ remove MaybeReadableArgs trait as it is now unused
+ remove onion_utils::DecodeInput as it would've now needed to be parameterized
by the CustomOnionMessageHandler trait, and we'd like to avoid either
implementing DecodeInput in messenger or having onion_utils depend on
onion_message::*
Co-authored-by: Matt Corallo <git@bluematt.me> Co-authored-by: Valentine Wallace <vwallace@protonmail.com>
Matt Corallo [Wed, 26 Oct 2022 21:55:23 +0000 (21:55 +0000)]
Export io::ErrorKind in bindings
The bindings have exported `io::Error` as, basically,
`io::ErrorKind`, for quite some time, so there's little reason to
not just export `io::ErrorKind` as well.
Matt Corallo [Thu, 29 Sep 2022 15:22:47 +0000 (15:22 +0000)]
Add `WriteableScore` bindings impl for `MultiThreadedLockableScore`
In 56b07e52aabdaca521987e765f1fa864966a5d49 we made
`MultiThreadedLockableScore` fully bindings-compatible. However, it
did not add a `WriteableScore` implementation for it. This was an
oversight as it is a `WriteableScore` in Rust and needs to be for
use in other parts of the API.
Here we add the required impl in a way that the bindings generator
is able to handle it and add conversion utilities.
Implement sending and receiving custom onion messages
This uses the work done in the preceding commits to implement encoding a user's
custom TLV in outbound onion messages, and decoding custom TLVs in inbound
onion messages, to be provided to the new CustomOnionMessageHandler.
As we're moving towards monitor update async being a supported
use-case, we shouldn't call an async monitor update "failed", but
rather "in progress". This simply updates the internal channel.rs
enum name to reflect the new thinking.
Matt Corallo [Sat, 20 Aug 2022 01:03:27 +0000 (01:03 +0000)]
Handle async initial ChannelMonitor persistence failing on restart
If the initial ChannelMonitor persistence is done asynchronously
but does not complete before the node restarts (with a
ChannelManager persistence), we'll start back up with a channel
present but no corresponding ChannelMonitor.
Because the Channel is pending-monitor-update and has not yet
broadcasted its initial funding transaction or sent channel_ready,
this is not a violation of our API contract nor a safety violation.
However, the previous code would refuse to deserialize the
ChannelManager treating it as an API contract violation.
The solution is to test for this case explicitly and drop the
channel entirely as if the peer disconnected before we received
the funding_signed for outbound channels or before sending the
channel_ready for inbound channels.
Matt Corallo [Thu, 13 Oct 2022 21:57:09 +0000 (21:57 +0000)]
Include all channel route hints if no connected channels exist
Mobile clients often take a second or two before they reconnect to
their peers as its not always clear immediately that connections
have been killed (especially on iOS). This can cause us to
spuriously fail to include route hints in our invoices if we're on
mobile.
The fix is simple, if we're selecting channels to include in route
hints and we're not not connected to the peer for any of our
channels, we should simply include the hints for all channels, even
though we're disconencted.