Matt Corallo [Mon, 15 Aug 2022 19:30:32 +0000 (19:30 +0000)]
Provide guidance on ChannelMonitorUpdate serialized size
Users need to make decisions about storage sizing and we need to
have advice on the maximum size of various things users need to
store. ChannelMonitorUpdates are likely the worst case of this,
they're usually at max a few KB, but can get up to a few hundred
KB for commitment transactions that have 400+ HTLCs pending.
We had one user report an update (likely) going over 400 KiB, which
isn't immediately obvious to me is practical, but its within a few
multiples of trivially-reachable sizes, so its likely that did
occur. To be on the safe side, we simply recommend users ensure
they can support "upwards of 1 MiB" here.
Matt Corallo [Sat, 13 Aug 2022 17:29:06 +0000 (17:29 +0000)]
Correct the on-chain script checked in gossip verification
The `bitcoin_key_1` and `bitcoin_key_2` fields in
`channel_announcement` messages are sorted according to node_ids
rather than the keys themselves, however the on-chain funding
script is sorted according to the bitcoin keys themselves. Thus,
with some probability, we end up checking that the on-chain script
matches the wrong script and rejecting the channel announcement.
The correct solution is to use our existing channel funding script
generation function which ensure we always match what we generate.
This was found in testing the Java bindings, where a test checks
that retunring the generated funding script in `chain::Access`
results in the constructed channel ending up in our network graph.
Also update the fuzz ChaCha20Poly1305 to not mark as finished after a single
encrypt_in_place. This is because more bytes may still need to be encrypted,
causing us to panic at the assertion that finished == false when we go to
encrypt more.
Also fix unused_mut warning in messenger + add log on OM forward for testing
Matt Corallo [Sat, 16 Jul 2022 20:41:45 +0000 (20:41 +0000)]
Move per-HTLC logic out of get_claimable_balances into a helper
Val suggested this as an obvious cleanup to separate per_HTLC logic
from the total commitment transaction logic, separating the large
function into two.
Matt Corallo [Tue, 24 May 2022 23:57:56 +0000 (23:57 +0000)]
Expose counterparty-revoked-outputs in `get_claimable_balance`
This uses the various new tracking added in the prior commits to
expose a new `Balance` type - `CounterpartyRevokedOutputClaimable`.
Some nontrivial work is required, however, as we now have to track
HTLC outputs as spendable in a transaction that comes *after* an
HTLC-Success/HTLC-Timeout transaction, which we previously didn't
need to do. Thus, we have to check if an
`onchain_events_awaiting_threshold_conf` event spends a commitment
transaction's HTLC output while walking events. Further, because
we now need to track HTLC outputs after the
HTLC-Success/HTLC-Timeout confirms, and because we have to track
the counterparty's `to_self` output as a contentious output which
could be claimed by either party, we have to examine the
`OnchainTxHandler`'s set of outputs to spend when determining if
certain outputs are still spendable.
Two new tests are added which test various different transaction
formats, and hopefully provide good test coverage of the various
revoked output paths.
Matt Corallo [Tue, 17 May 2022 20:45:17 +0000 (20:45 +0000)]
Scan `onchain_events_awaiting_threshold_conf` once in balance calc
Instead of a series of different
`onchain_events_awaiting_threshold_conf.iter()...` calls to scan
for HTLC status in balance calculation, pull them all out into one
`for ... { match ... }` to do it once and simplify the code
somewhat.
Matt Corallo [Sat, 21 May 2022 01:11:52 +0000 (01:11 +0000)]
Track the txid that resolves HTLCs even after resolution completes
We need this information when we look up if we still need to spend
a revoked output from an HTLC-Success/HTLC-Timeout transaction for
balance calculation.
Matt Corallo [Thu, 19 May 2022 01:50:37 +0000 (01:50 +0000)]
Track HTLC-Success/HTLC-Timeout claims of revoked outputs
When a counterparty broadcasts a revoked commitment transaction,
followed immediately by HTLC-Success/-Timeout spends thereof, we'd
like to have an `onchain_events_awaiting_threshold_conf` entry
for them.
This does so using the `HTLCSpendConfirmation` entry, giving it
(slightly) new meaning. Because all existing uses of
`HTLCSpendConfirmation` already check if the relevant commitment
transaction is revoked first, this should be trivially backwards
compatible.
We will ultimately figure out if something is being spent via the
`OnchainTxHandler`, but to do so we need to look up the output via
the HTLC transaction txid, which this allows us to do.
Matt Corallo [Tue, 17 May 2022 23:57:52 +0000 (23:57 +0000)]
Fix off-by-one in test_onchain_htlc_claim_reorg_remote_commitment
The test intended to disconnect a transaction previously connected
but didn't disconnect enough blocks to do so, leading to it
confirming two conflicting transactions.
In the next few commits this will become an assertion failure.
Matt Corallo [Sat, 30 Apr 2022 20:29:31 +0000 (20:29 +0000)]
Track counterparty payout info in counterparty commitment txn
When handling a revoked counterparty commitment transaction which
was broadcast on-chain, we occasionally need to look up which
output (and its value) was to the counterparty (the `to_self`
output). This will allow us to generate `Balance`s for the user for
the revoked output.
Matt Corallo [Fri, 13 May 2022 05:11:14 +0000 (05:11 +0000)]
Store the full event transaction in `OnchainEvent` structs
When we see a transaction which generates some `OnchainEvent`, its
useful to have the full transaction around for later analysis.
Specifically, it lets us check the list of outputs which were spent
in the transaction, allowing us to look up, e.g. which HTLC
outpoint was spent in a transaction.
This will be used in a few commits to do exactly that - figure out
which HTLC a given `OnchainEvent` corresponds with.
Matt Corallo [Tue, 9 Aug 2022 21:26:16 +0000 (21:26 +0000)]
Backfill gossip without buffering directly in LDK
Instead of backfilling gossip by buffering (up to) ten messages at
a time, only buffer one message at a time, as the peers' outbound
socket buffer drains. This moves the outbound backfill messages out
of `PeerHandler` and into the operating system buffer, where it
arguably belongs.
Not buffering causes us to walk the gossip B-Trees somewhat more
often, but avoids allocating vecs for the responses. While its
probably (without having benchmarked it) a net performance loss, it
simplifies buffer tracking and leaves us with more room to play
with the buffer sizing constants as we add onion message forwarding
which is an important win.
Note that because we change how often we check if we're out of
messages to send before pinging, we slightly change how many
messages are exchanged at once, impacting the
`test_do_attempt_write_data` constants.
Elias Rohrer [Thu, 11 Aug 2022 12:27:45 +0000 (14:27 +0200)]
Drop return value from `Filter::register_output`
This commit removes the return value from `Filter::register_output` as
creating a suitable value almost always entails blocking operations
(e.g., lookups via network request), which however conflicts with the
requirement that user calls should avoid blocking calls at all cost.
Removing the return value also rendered quite a bit of test code for
dependent transaction handling superfluous, which is therefore also
removed with this commit.
Prior to this change, we could have failed to decode a valid payload of size
>253. This is because we were decoding the length (a BigSize, big-endian) as a
VarInt (little-endian).
Use util methods in `Peer` to decide when to forward
This consolidates our various checks on peer buffer space into the
`Peer` impl itself, making the thresholds at which we stop taking
various actions on a peer more readable as a whole.
This commit was primarily authored by `Valentine Wallace
<vwallace@protonmail.com>` with some amendments by `Matt Corallo
<git@bluematt.me>`.
Matt Corallo [Sun, 7 Aug 2022 19:02:33 +0000 (19:02 +0000)]
Update libfuzzer-sys to new upstream inclusion method
Dunno why they changed it, but the old "depend directly on git"
thing that cargo-fuzz used forever is now deprecated that that
repo is archived, they've now moved to another repo and publish
properly on crates.io.
Fix possible incomplete read bug on onion packet decode
Pre-existing to this PR, we were reading next packet bytes with io::Read::read,
which is not guaranteed to read all the bytes we need, only guaranteed to read
*some* bytes.
We fix this to be read_exact, which is guaranteed to read all the next hop
packet bytes.
This required adapting `onion_utils::decode_next_hop` to work for both payments
and onion messages.
Currently we just print out the path_id of any onion messages we receive. In
the future, these received onion messages will be redirected to their
respective handlers: i.e. an invoice_request will go to an InvoiceHandler,
custom onion messages will go to a custom handler, etc.
This adds several utilities in service of then adding
OnionMessenger::send_onion_message, which can send to either an unblinded
pubkey or a blinded route. Sending custom TLVs and sending an onion message
containing a reply path are not yet supported.
We also need to split the construct_keys_callback macro into two macros to
avoid an unused assignment warning.
This method will help us avoid retrieving our node secret, something we want to
get rid of entirely. It will be used in upcoming commits when decoding the
onion message packet, and in future PRs to help us get rid of
KeysInterface::get_node_secret usages across the codebase
Add onion_message::Packet and adapt construct_onion_packet_with_init_noise for it
We need to add a new Packet struct because onion message packet hop_data fields
can be of variable length, whereas regular payment packets are always 1366
bytes.
Add config support for 'their_channel_reserve_proportional_millionths' [#1498]
It is proportion of the channel value to configure as the
`their_channel_reserve_satoshis` for both outbound and inbound channels.
It decides the minimum balance that the other node has to maintain on their
side, at all times.
Don't return `channel_state` from `decode_update_add_htlc_onion`
Currently `decode_update_add_htlc_onion` returns the `channel_state`
lock to ensure that `internal_update_add_htlc` holds a single
`channel_state` lock in when the entire function execution. This is
unnecessary, and since we are moving the channel storage to the
`per_peer_state`, this no longer achieves the goal it was intended for.
We therefore avoid returning the `channel_state` from
`decode_update_add_htlc_onion`, and just retake the lock in
`internal_update_add_htlc` instead.
Add onion messages module + enable the construction of blinded routes
Blinded routes can be provided as destinations for onion messages, when the
recipient prefers to remain anonymous.
We also add supporting utilities for constructing blinded path keys, and
control TLVs structs representing blinded payloads prior to being
encoded/encrypted. These utilities and struct will be re-used in upcoming
commits for sending and receiving/forwarding onion messages.
Finally, add utilities for reading the padding from an onion message's
encrypted TLVs without an intermediate Vec.
Matt Corallo [Wed, 27 Jul 2022 03:53:47 +0000 (03:53 +0000)]
Expose `ChannelMonitor::get_counterparty_node_id`
This fixes an oversight in ac842ed9dd7a36a4a26eb6b856d80ab04eecf750
namely that it left users unable to implement their own
`ChainMonitor` from outside of the `rust-lightning` crate.
Matt Corallo [Thu, 21 Jul 2022 17:47:19 +0000 (17:47 +0000)]
Rename amount penalty to `liquidity_penalty_amount_multiplier_msat`
This makes our `ProbabilisticScorer` field names more consistent,
as we add more types of penalties, referring to a penalty as only
the "amount penalty" no longer makes sense - we not have several
amount multiplier penalties.
Matt Corallo [Thu, 14 Jul 2022 01:06:10 +0000 (01:06 +0000)]
Add a per-amount base penalty in the `ProbabilisticScorer`
There's not much reason to not have a per-hop-per-amount penalty in
the `ProbabilisticScorer` to go along with the per-hop penalty to
let it scale up to larger amounts, so we add one here.
Notably, we use a divisor of 2^30 instead of 2^20 (like the
equivalent liquidity penalty) as it allows for more flexibility,
and there's not really any reason to worry about us not being able
to create high enough penalties.
Matt Corallo [Thu, 21 Jul 2022 21:38:00 +0000 (21:38 +0000)]
Avoid blanket implementing FeeEstimator for Deref<FeeEstimator>
This simplifies things for bindings (and, to some extent,
downstream users) by exploiting the fact that we can always "clone"
a reference to a struct by dereferencing and then creating a new
reference.
Matt Corallo [Wed, 13 Jul 2022 16:52:27 +0000 (16:52 +0000)]
Use a separate (non-trait) fee-estimation fn in LowerBoundedEstimator
This should make it somewhat more difficult to accidentally use a
straight fee estimator when we actually want a
LowerBoundedFeeEstimator by not having the types be exchangeable at
all.
Matt Corallo [Fri, 22 Jul 2022 00:32:17 +0000 (00:32 +0000)]
Remove scary disconenct warnings on PeerManager new connection fns
In 4703d4e72565ddfd150b9368ea036f4973fd7590 we changed
PeerManager::socket_disconnected to no longer require that sockets
which the PeerManager decided to disconnect not be disconnected.
However, we forgot to remove the scary warnings on the
`new_{inbound,outbound}_connection` functions which warned of the
old behavior.
We add `HTLCHandlingFailedConditions` to express the failure parameters,
that will be enforced by a new macro, `expect_pending_htlcs_forwardable_conditions`.
Adds a HTLCHandlingFailed that expresses failure by our node to process
a specific HTLC. A HTLCDestination enum is defined to express the
possible cases that causes the handling to fail.
Matt Corallo [Tue, 12 Apr 2022 17:32:35 +0000 (17:32 +0000)]
Expand lockorder testing to look at mutexes, not specific instances
Our existing lockorder inversion checks look at specific instances
of mutexes rather than the general mutex itself. This changes that
behavior to look at the instruction pointer at which a mutex was
created and treat all mutexes which were created at the same
location as equivalent.
This allows us to detect lockorder inversions which occur across
tests, though it does substantially reduce parallelism during test
runs.
projects / rust-lightning / log