Matt Corallo [Mon, 22 Mar 2021 22:27:34 +0000 (18:27 -0400)]
Fix serialization expected lengths and check them in test/fuzzing
Matt Corallo [Tue, 27 Apr 2021 01:06:11 +0000 (01:06 +0000)]
Merge pull request #897 from TheBlueMatt/2021-04-fix-ci
Bump MSRV to 1.36
Matt Corallo [Mon, 26 Apr 2021 17:48:57 +0000 (17:48 +0000)]
Test lightning-persister on rustc 1.36 as well
Matt Corallo [Mon, 26 Apr 2021 17:47:47 +0000 (17:47 +0000)]
Bump MSRV to 1.36.0
Debian is shipping 1.41 on oldstable and rust-bitcoin will likely
move to 1.36 over the coming months, so there's little reason to
wait on this.
cc https://github.com/rust-bitcoin/rust-bitcoin/issues/510
Matt Corallo [Mon, 26 Apr 2021 19:59:32 +0000 (19:59 +0000)]
Merge pull request #894 from TheBlueMatt/2021-04-fix-docs
Fix doc resolution in rustc 1.48
Matt Corallo [Mon, 26 Apr 2021 18:00:33 +0000 (18:00 +0000)]
Merge pull request #876 from valentinewallace/invoice-features
Invoice features
Valentine Wallace [Fri, 16 Apr 2021 19:29:33 +0000 (15:29 -0400)]
Add Features feature to invoices.
Matt Corallo [Sat, 24 Apr 2021 00:03:42 +0000 (00:03 +0000)]
Merge pull request #890 from TheBlueMatt/2021-04-fix-chan-shutdown-crash
Fix (and test) panic when our counterparty uses a bogus funding tx
Matt Corallo [Wed, 21 Apr 2021 00:11:54 +0000 (00:11 +0000)]
Fix (and test) panic when our counterparty uses a bogus funding tx
During the block API refactor, we started calling
Channel::force_shutdown when a channel is closed due to a bogus
funding tx. However, we still set the channel's state to Shutdown
prior to doing so, leading to an assertion in force_shutdown (that
the channel is not already closed).
This removes the state-set call and adds a (long-overdue) test for
this case.
Fixes: 60b962a18ebcf494340ddc001870f8160c625968
Valentine Wallace [Fri, 16 Apr 2021 19:26:38 +0000 (15:26 -0400)]
Expose Features::from_le_bytes() method
Useful for constructing features objects from raw feature bytes.
Valentine Wallace [Tue, 13 Apr 2021 14:34:33 +0000 (10:34 -0400)]
Fix indentation in payment_secret function
Matt Corallo [Fri, 23 Apr 2021 20:42:11 +0000 (20:42 +0000)]
Fix doc resolution in rustc 1.48
Matt Corallo [Fri, 23 Apr 2021 19:13:23 +0000 (19:13 +0000)]
Merge pull request #889 from jkczyz/2021-04-electrum-trait
Define chain::Confirm trait for use by Electrum clients
Jeffrey Czyz [Thu, 22 Apr 2021 16:52:10 +0000 (09:52 -0700)]
Drop pub functions for ChainMonitor's Listen impl
Jeffrey Czyz [Thu, 22 Apr 2021 07:11:00 +0000 (00:11 -0700)]
Implement chain::Confirm for ChainMonitor
Jeffrey Czyz [Tue, 20 Apr 2021 20:39:00 +0000 (13:39 -0700)]
Implement chain::Confirm for relevant structs
Jeffrey Czyz [Tue, 20 Apr 2021 17:24:08 +0000 (10:24 -0700)]
Define chain::Confirm trait
Define a separate trait akin to chain::Listen for notifying when
transactions have been confirmed on chain or unconfirmed during a chain
reorganization. Whereas chain::Listen is used for block-oriented chain
sources, chain::Confirm is used for chain sources supplying data for
activity related to transactions and outputs registered via
chain::Filter.
Matt Corallo [Thu, 22 Apr 2021 14:16:05 +0000 (14:16 +0000)]
Merge pull request #891 from TheBlueMatt/2021-04-peer_handler_lock
[peer_handler] Take the peers lock before getting messages to send
Matt Corallo [Wed, 21 Apr 2021 21:50:41 +0000 (21:50 +0000)]
[peer_handler] Take the peers lock before getting messages to send
Previously, if a user simultaneously called
`PeerHandler::process_events()` from two threads, we'd race, which
ended up sending messages out-of-order in the real world.
Specifically, we first called `get_and_clear_pending_msg_events`,
then take the `peers` lock and push the messages we got into the
sending queue. Two threads may both get some set of messages to
send, but then race each other into the `peers` lock and send the
messages in random order.
Because we already hold the `peers` lock when calling most message
handler functions, we can simply take the lock before calling
`get_and_clear_pending_msg_events`, solving the race.
Matt Corallo [Wed, 21 Apr 2021 15:21:39 +0000 (15:21 +0000)]
Merge pull request #887 from valentinewallace/invoice-use-RL-routehint
invoice: swap RouteHop for RouteHint
Valentine Wallace [Tue, 20 Apr 2021 19:46:58 +0000 (15:46 -0400)]
Test lightning-invoice on 1.30.0
Valentine Wallace [Mon, 19 Apr 2021 22:22:21 +0000 (18:22 -0400)]
invoice: rename Route to RouteHint (which is more accurate)
Valentine Wallace [Mon, 19 Apr 2021 22:12:51 +0000 (18:12 -0400)]
Rename RouteHint to RouteHintHop (which is more accurate)
Valentine Wallace [Fri, 16 Apr 2021 20:17:11 +0000 (16:17 -0400)]
invoice: swap RouteHop for RouteHint
To prevent naming conflicts in bindings
Matt Corallo [Tue, 20 Apr 2021 00:45:30 +0000 (00:45 +0000)]
Merge pull request #844 from sr-gi/843-ln-signing
Adds lightning message signing/verification/pk_recovery
Sergi Delgado Segura [Wed, 31 Mar 2021 09:38:31 +0000 (11:38 +0200)]
Changes zbase32 crate from pub to pub(crate)
Matt Corallo [Mon, 22 Mar 2021 16:22:24 +0000 (12:22 -0400)]
Correct license header in zbase32 to match original work
The original work is licensed dual MIT+Apache-2 just like us, so
the license header should not only mention MIT.
Matt Corallo [Mon, 22 Mar 2021 16:20:26 +0000 (12:20 -0400)]
Replace spaces with tabs in zbase32.rs
Matt Corallo [Mon, 22 Mar 2021 16:19:28 +0000 (12:19 -0400)]
Fix a number of bugs in zbase32 and add a fuzzer which caught them.
Sergi Delgado Segura [Tue, 16 Mar 2021 14:55:44 +0000 (15:55 +0100)]
Adds lightning message signing/verification/pk_recovery
Matt Corallo [Thu, 15 Apr 2021 21:55:53 +0000 (21:55 +0000)]
Merge pull request #886 from TheBlueMatt/2021-04-pub-alias
Make the used-in-pub-interfaces TransactionOutputs type alias pub
Matt Corallo [Thu, 15 Apr 2021 20:11:19 +0000 (20:11 +0000)]
Merge pull request #884 from TheBlueMatt/2021-04-bp-bindings
Prep background-processor for bindings inclusion
Matt Corallo [Thu, 15 Apr 2021 20:10:21 +0000 (16:10 -0400)]
Make the used-in-pub-interfaces TransactionOutputs type alias pub
Matt Corallo [Tue, 13 Apr 2021 23:39:32 +0000 (19:39 -0400)]
Rename background-processor to lightning-... to match other crates
Matt Corallo [Tue, 13 Apr 2021 23:38:31 +0000 (19:38 -0400)]
Use a trait to handle ChannelManager persistence instead of an Fn
This avoids having to write new support for closures in the C
bindings generation but, more importantly, we really need to also
be handling Events in the same trait, so it makes sense to go ahead
and convert it.
For compatibility, the trait is implemented for any matching
closure.
Matt Corallo [Tue, 13 Apr 2021 20:04:17 +0000 (16:04 -0400)]
Loosen background-persister so bindings are happy
Matt Corallo [Thu, 15 Apr 2021 14:25:54 +0000 (14:25 +0000)]
Merge pull request #858 from jkczyz/2021-03-electrum-interface
Electrum interface for ChannelMonitor
Jeffrey Czyz [Mon, 12 Apr 2021 18:47:23 +0000 (11:47 -0700)]
Rename onchain_events_waiting_threshold_conf
Jeffrey Czyz [Sat, 10 Apr 2021 07:32:08 +0000 (00:32 -0700)]
Parameterize test_htlc_on_chain_timeout
This test failed when ConnectionStyle was set to a SkippingBlocks
variant because of a bug in ChannelMonitor::update_best_block.
Parameterize the test with these styles to catch any regressions.
Jeffrey Czyz [Wed, 7 Apr 2021 19:02:33 +0000 (12:02 -0700)]
Test ChainMonitor's Electrum interface
Jeffrey Czyz [Wed, 7 Apr 2021 18:43:32 +0000 (11:43 -0700)]
Reuse txdata in functional_test_utils.rs
Jeffrey Czyz [Fri, 2 Apr 2021 23:53:00 +0000 (16:53 -0700)]
Add an internal typedef for transaction outputs
Jeffrey Czyz [Fri, 2 Apr 2021 22:31:20 +0000 (15:31 -0700)]
Remove unnecessary script_pubkey clones
Jeffrey Czyz [Fri, 2 Apr 2021 22:03:41 +0000 (15:03 -0700)]
Electrum interface for ChainMonitor
Add an interface to ChainMonitor for Electrum users, delegating to the
corresponding methods in each ChannelMonitor.
Jeffrey Czyz [Mon, 5 Apr 2021 20:18:27 +0000 (13:18 -0700)]
Add ChannelMonitor::get_relevant_txids
Define an Electrum-friendly interface for ChannelMonitor where txids of
relevant transactions can be obtained. For any of these transactions
that are re-orged out of the chain, users must call
transaction_unconfirmed.
Jeffrey Czyz [Sat, 27 Mar 2021 20:26:45 +0000 (16:26 -0400)]
Add ChannelMonitor::transaction_unconfirmed
Define an Electrum-friendly interface for ChannelMonitor where
transactions are unconfirmed independently from updating the latest
block.
Jeffrey Czyz [Fri, 2 Apr 2021 02:44:46 +0000 (19:44 -0700)]
Add ChannelMonitor::update_best_block
Expose a way for Electrum users to update the best block on a
ChannelMonitor independently of confirming transactions.
Jeffrey Czyz [Wed, 24 Mar 2021 22:47:44 +0000 (18:47 -0400)]
Add ChannelMonitor::transactions_confirmed
Define an Electrum-friendly interface for ChannelMonitor where
transactions are confirmed independently of updating the last connected
block.
Jeffrey Czyz [Tue, 6 Apr 2021 18:34:17 +0000 (11:34 -0700)]
Track block height in ChannelMonitor
Jeffrey Czyz [Fri, 9 Apr 2021 06:36:30 +0000 (23:36 -0700)]
Combine ChannelManager's block hash and height
There is a possible race condition when both the latest block hash and
height are needed. Combine these in one struct and place them behind a
single lock.
Jeffrey Czyz [Wed, 31 Mar 2021 17:54:01 +0000 (13:54 -0400)]
Add txid to on-chain event tracking
When using Electrum, transactions are individually unconfirmed during a
reorg rather than by block. Store the txid of the transaction creating
the on-chain event so that it can be used to determine which events need
to be removed when a transaction is unconfirmed.
Jeffrey Czyz [Sat, 10 Apr 2021 01:38:03 +0000 (18:38 -0700)]
Check for duplicate HTLC events having matured
Jeffrey Czyz [Wed, 31 Mar 2021 17:23:57 +0000 (13:23 -0400)]
Flatten onchain_events_waiting_threshold_conf
Rather than mapping height to a vector of events, use a single vector
for all events. This allows for easily processing events by either
height or transaction. The latter will be used for an interface suitable
for Electrum.
Matt Corallo [Wed, 14 Apr 2021 18:50:52 +0000 (18:50 +0000)]
Merge pull request #853 from TheBlueMatt/2021-03-transaction_unconfirmed
Add method to note transaction unconfirmed/reorged-out
Matt Corallo [Mon, 22 Mar 2021 22:07:13 +0000 (18:07 -0400)]
Test new transactions_unconfirmed
Matt Corallo [Mon, 22 Mar 2021 21:59:59 +0000 (17:59 -0400)]
Expose ChannelManager's current config and use it in reload in tests
Matt Corallo [Mon, 22 Mar 2021 21:01:04 +0000 (17:01 -0400)]
Add method to note transaction unconfirmed/reorged-out
Matt Corallo [Wed, 14 Apr 2021 02:20:27 +0000 (02:20 +0000)]
Merge pull request #885 from p2pderivatives/use-secp256k1-consts-in-ser
Fix size check in Vec<Signature> serialization + use consts
Matt Corallo [Wed, 14 Apr 2021 01:51:33 +0000 (01:51 +0000)]
Merge pull request #875 from TheBlueMatt/2021-04-fix-bench
Fix benchmark compile warnings and errors
Tibo-lg [Wed, 14 Apr 2021 01:37:49 +0000 (10:37 +0900)]
Fix size check in Vec<Signature> serialization + use consts
Matt Corallo [Tue, 13 Apr 2021 00:33:53 +0000 (20:33 -0400)]
Fix compile warning with --cfg require_route_graph_test
Matt Corallo [Tue, 13 Apr 2021 01:37:52 +0000 (01:37 +0000)]
Merge pull request #872 from valentinewallace/timer-tick-in-bg-processor
Call timer_tick_occurred in BackgroundProcessor
Valentine Wallace [Fri, 9 Apr 2021 20:58:31 +0000 (16:58 -0400)]
Call peer_manager.timer_tick_occurred() in background processor
Valentine Wallace [Fri, 9 Apr 2021 20:55:10 +0000 (16:55 -0400)]
Rename timer_chan_freshness_every_min for uniformity with PeerManager
Matt Corallo [Mon, 12 Apr 2021 23:56:01 +0000 (23:56 +0000)]
Merge pull request #874 from TheBlueMatt/2021-04-persister-export
Prep lightning-persister for export in the C bindings
Matt Corallo [Mon, 12 Apr 2021 21:59:11 +0000 (17:59 -0400)]
Improve + fix indentation and style in lightning-persister
Matt Corallo [Mon, 12 Apr 2021 17:48:29 +0000 (13:48 -0400)]
Return ChannelMonitors in a Vec, not HashMap when loading from disk
There's little reason for the HashMap - the ChannelMonitors are
already unique (enforced by file names), and the eventual HashMap
that users need when deserializing the `ChannelManager` is a
slightly different form (it requires no BlockHash entry).
Matt Corallo [Mon, 12 Apr 2021 22:04:55 +0000 (18:04 -0400)]
Fix benchmark compile warnings and errors
Hopefully soon once a few more PRs get merged we can require this
in CI so we won't have any more regressions here.
Matt Corallo [Mon, 12 Apr 2021 16:20:38 +0000 (12:20 -0400)]
Expand persistence to all ChannelManagers, not just Arc-based ones
Matt Corallo [Sat, 10 Apr 2021 20:27:24 +0000 (20:27 +0000)]
Merge pull request #856 from TheBlueMatt/2021-03-check-tx
Take the full funding transaction from the user on generation
Matt Corallo [Sat, 10 Apr 2021 20:26:58 +0000 (20:26 +0000)]
Merge pull request #870 from valentinewallace/invoices-crate
Invoices crate
Matt Corallo [Sat, 10 Apr 2021 20:26:36 +0000 (20:26 +0000)]
Merge pull request #873 from TheBlueMatt/2021-861-missing-eof-eol
Add missing EOF newline
Matt Corallo [Fri, 26 Mar 2021 22:07:24 +0000 (18:07 -0400)]
Take the full funding transaction from the user on generation
Instead of relying on the user to ensure the funding transaction is
correct (and panicing when it is confirmed), we should check it is
correct when it is generated. By taking the full funding transaciton
from the user on generation, we can also handle broadcasting for
them instead of doing so via an event.
Matt Corallo [Fri, 9 Apr 2021 23:57:20 +0000 (23:57 +0000)]
Merge pull request #861 from lightning-signer/degenerify
De-generify Sign methods
Matt Corallo [Fri, 9 Apr 2021 23:57:12 +0000 (19:57 -0400)]
Add missing EOF newline
Matt Corallo [Fri, 9 Apr 2021 22:23:14 +0000 (22:23 +0000)]
Merge pull request #866 from TheBlueMatt/2021-04-log-err-node
Log the node generating an onion error
Matt Corallo [Fri, 9 Apr 2021 16:56:07 +0000 (16:56 +0000)]
Merge pull request #863 from valentinewallace/expose-read-chanmons-from-disk
persister: Expose method to read ChannelMonitors from disk
Valentine Wallace [Fri, 2 Apr 2021 18:42:03 +0000 (14:42 -0400)]
persister: Expose method to read ChannelMonitors from disk
Valentine Wallace [Wed, 7 Apr 2021 17:44:59 +0000 (13:44 -0400)]
Fix Windows
Valentine Wallace [Wed, 7 Apr 2021 17:17:22 +0000 (13:17 -0400)]
Clean up lightning-invoice Cargo.toml and README
Valentine Wallace [Wed, 7 Apr 2021 17:04:29 +0000 (13:04 -0400)]
Add lightning invoice fuzzing to CI
Valentine Wallace [Wed, 7 Apr 2021 17:04:04 +0000 (13:04 -0400)]
Clean up lightning-invoice CI, license, and add to workspaces
Sebastian [Wed, 7 Apr 2021 16:48:01 +0000 (12:48 -0400)]
Pure import of lightning-invoice crate
Original repo: https://github.com/rust-bitcoin/rust-lightning-invoice
Devrandom [Fri, 9 Apr 2021 09:16:22 +0000 (11:16 +0200)]
Test that BaseSign can have a vtable
Devrandom [Fri, 9 Apr 2021 09:18:45 +0000 (11:18 +0200)]
Move Writeable from BaseSign to Sign
Devrandom [Thu, 1 Apr 2021 10:24:21 +0000 (12:24 +0200)]
Separate Clone from Sign
Clone requires Sized, which prevents Sign from being a dyn object.
Devrandom [Thu, 1 Apr 2021 10:40:35 +0000 (12:40 +0200)]
Eliminate unnecessary generics from Sign
The generic methods prevent Sign from being a dyn object.
Use Secp256k1<All> as part of removing generics from Secp256k1 contexts passed into Sign methods.
Matt Corallo [Thu, 8 Apr 2021 15:07:24 +0000 (15:07 +0000)]
Merge pull request #871 from TheBlueMatt/2021-03-bench-sends
Run FS bench in CI as well
Matt Corallo [Thu, 8 Apr 2021 02:55:10 +0000 (22:55 -0400)]
Run FS bench in CI as well
Matt Corallo [Thu, 8 Apr 2021 02:06:13 +0000 (02:06 +0000)]
Merge pull request #815 from TheBlueMatt/2021-03-router-fuzzzzzzzz
Track full-path htlc-minimum-msat while routing
Matt Corallo [Thu, 1 Apr 2021 22:31:34 +0000 (18:31 -0400)]
[router] Add and clarify comments describing router internals
Matt Corallo [Sat, 27 Mar 2021 22:16:38 +0000 (18:16 -0400)]
Drop unreachable underflow-handling block in route calculation
See comment in the removed block, note that the subsequent
subtraction will underflow if the block would otherwise have been
reached.
Matt Corallo [Fri, 5 Mar 2021 02:42:42 +0000 (21:42 -0500)]
Don't clone Features during Dijkstras graph walk
We currently copy the features objects in each channel as we walk
the graph during route calculation. This implies a significant
amount of malloc traffic as the features flags object are stored
on the heap.
Instead, because they features being referenced are in the network
graph which we hold a reference to, we can simply store references
to them.
This nontrivially improves our get_route benchmark by around 5%.
Matt Corallo [Sat, 27 Mar 2021 16:49:42 +0000 (12:49 -0400)]
[router] Avoid re-processing peers when a score component decreases
While walking the graph doing Dijkstra's, we may decrease the
amount being sent along one path, and not others, based on the
htlc_minimum_msat value. This may result in a lower relative fees
on that path in comparison to others. In the extreme, this may
result in finding a second path to a node with a lower fee than the
first path found (normally impossible in a Dijkstra's
implementation, as we walk next hops in fee-order).
In such a case, we end up with parts of our state invalid as
further hops beyond that node have been filled in using the
original total fee information.
Instead, we simply track which nodes have been processed and ignore
and further updates to them (as it implies we've reduced the amount
we're sending to find a lower absolute fee, but a higher relative
fee, which isn't a better route).
We check that we are in exactly this case in test builds with new
in-line assertions. Note that these assertions successfully detect
the bug in the previous commit.
Sadly this requires an extra HashMap lookup every time we pop an
element off of our heap, though we can skip a number of heap pushes
during the channel walking.
Matt Corallo [Sat, 27 Mar 2021 01:50:54 +0000 (21:50 -0400)]
Add a random real-world-network-graph test for the router
This is the same code as was recently failing in our benchmarks,
adapted to use a random starting seed instead of a fixed one and
a smaller iteration to reduce runtime.
Matt Corallo [Sat, 27 Mar 2021 16:27:44 +0000 (12:27 -0400)]
[router] Calc min-to-node fees based on min value, not est value
When walking the network graph to calculate a route, we always
calculate the minimum fee which is required to make one further
hop. This avoids some extra hop processing at the end of each path
selection loop (saving around 10% runtime in our benchmarks).
However, if we use the real value which we expect
to send over a channel in that calculation, we may find an
alternate path to the same node which is more expensive but
capacity-constrained, resulting in us considering it cheaper as the
relative fee paid will be lower.
Instead, we can use the `minimal_value_contribution_msat`, which is
a constant through an entire path finding iteration, as the amount,
preventing any basis change in the relative fee paid.
Matt Corallo [Sat, 27 Mar 2021 02:56:37 +0000 (22:56 -0400)]
[router] Make Dijkstra's path scoring non-decreasing + consistent
Currently, the "best source" for a given node tracked during
Dijkstra's is updated with a different critera from the heap
sorting, resulting in loops in calculated paths.
This adds a test for the specific failure currently seen, utilizing
the new path-htlc-minimum tracking in the heap entries in place of
the per-hop htlc-minimum values which the MPP changeset partially
used.
Matt Corallo [Sat, 27 Mar 2021 02:31:57 +0000 (22:31 -0400)]
[router] Avoid re-selecting the same path over and over again
If we walk the network graph and find a route that meets are
payment amount and has a liquidiy limit far in excess of our
payment amount, we may select the same path several times in the
main path-gathering loop.
Instead, if the path we selected was not limited by the available
liquidity, we set the middle hop in the path's liquidity available
to zero, disabling that channel in future path finding steps.
Matt Corallo [Sat, 27 Mar 2021 02:53:35 +0000 (22:53 -0400)]
[router] Track full-path htlc-minimum-msat while graph-walking
Previously, we'd happily send funds through a path where, while
generating the path, in some middle hope we reduce the value being
sent to meet an htlc_maximum, making a later hop invalid due to it
no longer meeting its htlc_minimum. Instead, we need to track the
path's htlc-minimum while we're transiting the graph.
Matt Corallo [Sat, 27 Mar 2021 02:51:41 +0000 (22:51 -0400)]
[router] Do not fail if we are exactly (or 3x) limited by a maximum
The new MPP routing algorithm attempts to build paths with a higher
value than our payment to find paths which may allow us to pay a
fee to meet an htlc_minimum limit. This is great if we're
min-bounded, however it results in us rejecting paths where we are
bounded by a maximum near the end of the path, with fees, and then
bounded by a minimum earlier in the path. Further, it means we will
not find the cheapest path where paths have a lower relative fee
but a higher absolute fee.
Instead, we calculate routes using the actual amount we wish to
send. To maintain the previous behavior of searching for cheaper
paths where we can "pay the difference" to meet an htlc_minimum, we
detect if we were minimum-bounded during graph walking and, if we
are, we walk the graph again with a higher value.