]> git.bitcoin.ninja Git - rust-lightning/log
rust-lightning
3 years agoAdd a macro to implement serialization of enums using TLVs
Matt Corallo [Sun, 30 May 2021 22:27:37 +0000 (22:27 +0000)]
Add a macro to implement serialization of enums using TLVs

3 years agoGive OnchainEvent::HTLCUpdate individual fields instead of a pair
Matt Corallo [Sun, 30 May 2021 20:44:26 +0000 (20:44 +0000)]
Give OnchainEvent::HTLCUpdate individual fields instead of a pair

3 years agoMerge pull request #933 from TheBlueMatt/2021-05-ser-fast
Matt Corallo [Tue, 1 Jun 2021 16:11:24 +0000 (16:11 +0000)]
Merge pull request #933 from TheBlueMatt/2021-05-ser-fast

Speed up deserialization of secp256k1 objects significantly

3 years agoAdd additional inline hints in serialization methods 2021-05-ser-fast
Matt Corallo [Fri, 28 May 2021 14:07:39 +0000 (14:07 +0000)]
Add additional inline hints in serialization methods

This substantially improves deserialization performance when LLVM
decides not to inline many short methods, eg when not building
with LTO/codegen-units=1.

Even with the default bench params of LTO/codegen-units=1, the
serialization benchmarks on an Intel 2687W v3 take:

test routing::network_graph::benches::read_network_graph  ... bench: 1,955,616,225 ns/iter (+/- 4,135,777)
test routing::network_graph::benches::write_network_graph ... bench: 165,905,275 ns/iter (+/- 118,798)

3 years agoImpl `serialized_length()` without `LengthCalculatingWriter`
Matt Corallo [Sat, 29 May 2021 18:32:53 +0000 (18:32 +0000)]
Impl `serialized_length()` without `LengthCalculatingWriter`

With the new `serialized_length()` method potentially being
significantly more efficient than `LengthCalculatingWriter`, this
commit ensures we call `serialized_length()` when calculating
length of a larger struct.

Specifically, prior to this commit a call to
`serialized_length()` on a large object serialized with
`impl_writeable`, `impl_writeable_len_match`, or
`encode_varint_length_prefixed_tlv` (and
`impl_writeable_tlv_based`) would always serialize all inner fields
of that object using `LengthCalculatingWriter`. This would ignore
any `serialized_length()` overrides by inner fields. Instead, we
override `serialized_length()` on all of the above by calculating
the serialized size using calls to `serialized_length()` on inner
fields.

Further, writes to `LengthCalculatingWriter` should never fail as
its `write` method never returns an error. Thus, any write failures
indicate a bug in an object's write method or in our
object-creation sanity checking. We `.expect()` such write calls
here.

As of this commit, on an Intel 2687W v3, the serialization
benchmarks take:

test routing::network_graph::benches::read_network_graph  ... bench: 2,039,451,296 ns/iter (+/- 4,329,821)
test routing::network_graph::benches::write_network_graph ... bench: 166,685,412 ns/iter (+/- 352,537)

3 years agoAvoid calling libsecp serialization fns when calculating length
Matt Corallo [Sat, 29 May 2021 18:24:16 +0000 (18:24 +0000)]
Avoid calling libsecp serialization fns when calculating length

When writing out libsecp256k1 objects during serialization in a
TLV, we potentially calculate the TLV length twice before
performing the actual serialization (once when calculating the
total TLV-stream length and once when calculating the length of the
secp256k1-object-containing TLV). Because the lengths of secp256k1
objects is a constant, we'd ideally like LLVM to entirely optimize
out those calls and simply know the expected length. However,
without cross-language LTO, there is no way for LLVM to verify that
there are no side-effects of the calls to libsecp256k1, leaving
LLVM with no way to optimize them out.

This commit adds a new method to `Writeable` which returns the
length of an object once serialized. It is implemented by default
using `LengthCalculatingWriter` (which LLVM generally optimizes out
for Rust objects) and overrides it for libsecp256k1 objects.

As of this commit, on an Intel 2687W v3, the serialization
benchmarks take:

test routing::network_graph::benches::read_network_graph  ... bench: 2,035,402,164 ns/iter (+/- 1,855,357)
test routing::network_graph::benches::write_network_graph ... bench: 308,235,267 ns/iter (+/- 140,202)

3 years agoDrop byte_utils in favor of native `to/from_be_bytes` methods
Matt Corallo [Fri, 28 May 2021 01:40:22 +0000 (01:40 +0000)]
Drop byte_utils in favor of native `to/from_be_bytes` methods

Now that our MSRV supports the native methods, we have no need
for the helpers anymore. Because LLVM was already matching our
byte_utils methods as byteswap functions, this should have no
impact on generated (optimzied) code.

This removes most of the byte_utils usage, though some remains to
keep the patch size reasonable.

3 years agoAdd bench profiles to Cargo.toml to force codegen-units=1
Matt Corallo [Fri, 28 May 2021 14:16:20 +0000 (14:16 +0000)]
Add bench profiles to Cargo.toml to force codegen-units=1

This makes a small difference for NetworkGraph deserialization
as it enables more inlining across different files, hopefully
better matching user performance as well.

As of this commit, on an Intel 2687W v3, the serialization
benchmarks take:

test routing::network_graph::benches::read_network_graph  ... bench: 2,037,875,071 ns/iter (+/- 760,370)
test routing::network_graph::benches::write_network_graph ... bench: 320,561,557 ns/iter (+/- 176,343)

3 years agoAdd benchmark of deserializing a NetworkGraph.
Matt Corallo [Fri, 28 May 2021 00:47:11 +0000 (00:47 +0000)]
Add benchmark of deserializing a NetworkGraph.

NetworkGraph is one of the largest structures we generally
deserialize, so it makes for a good benchmark, even if it isn't the
most complicated one.

As of this commit, on an Intel 2687W v3, these benchmarks take:

test routing::network_graph::benches::read_network_graph  ... bench: 2,101,420,078 ns/iter (+/- 6,649,020)
test routing::network_graph::benches::write_network_graph ... bench: 344,696,835 ns/iter (+/- 229,061)

3 years agoMerge pull request #892 from TheBlueMatt/2021-04-fix-htlc-ser
Matt Corallo [Mon, 31 May 2021 23:52:22 +0000 (23:52 +0000)]
Merge pull request #892 from TheBlueMatt/2021-04-fix-htlc-ser

Correct Channel outbound HTLC serialization and expand fuzzing coverage

3 years agoMerge pull request #934 from TheBlueMatt/2021-05-new-tag
Matt Corallo [Mon, 31 May 2021 20:35:16 +0000 (20:35 +0000)]
Merge pull request #934 from TheBlueMatt/2021-05-new-tag

Add new `(C-not implementable)` tag on `EventsProvider`

3 years agoCorrect Channel outbound HTLC serialization 2021-04-fix-htlc-ser
Matt Corallo [Wed, 21 Apr 2021 20:47:24 +0000 (20:47 +0000)]
Correct Channel outbound HTLC serialization

Channel serialization should happen "as if
remove_uncommitted_htlcs_and_mark_paused had just been called".

This is true for the most part, but outbound RemoteRemoved HTLCs
were being serialized as normal, even though
`remote_uncommitted_htlcs_and_mark_paused` resets them to
`Committed`.

This led to a bug identified by the `chanmon_consistency_target`
fuzzer wherein, if we receive a update_*_htlc message bug not the
corresponding commitment_signed prior to a serialization roundtrip,
we'd force-close the channel due to the peer "attempting to
fail/claim an HTLC which was already failed/claimed".

3 years ago[fuzz] Expand chanmon_consistency to do single message delivery
Matt Corallo [Wed, 21 Apr 2021 17:03:57 +0000 (17:03 +0000)]
[fuzz] Expand chanmon_consistency to do single message delivery

While trying to debug the issue ultimately tracked down to a
`PeerHandler` locking bug in #891, the ability to deliver only
individual messages at a time in chanmon_consistency looked
important. Specifically, it initially appeared there may be a race
when an update_add_htlc was delivered, then a node sent a payment,
and only after that, the corresponding commitment-signed was
delivered.

This commit adds such an ability, greatly expanding the potential
for chanmon_consistency to identify channel state machine bugs.

3 years ago[fuzz] Fix hang due to double-lock in full_stack_target
Matt Corallo [Wed, 21 Apr 2021 01:35:12 +0000 (01:35 +0000)]
[fuzz] Fix hang due to double-lock in full_stack_target

3 years ago[fuzz] Do not fail in FST if a channel is closed before we fund it
Matt Corallo [Wed, 21 Apr 2021 00:35:23 +0000 (00:35 +0000)]
[fuzz] Do not fail in FST if a channel is closed before we fund it

3 years agoMerge pull request #931 from jkczyz/2021-05-http-client-reconnect
Matt Corallo [Mon, 31 May 2021 18:05:45 +0000 (18:05 +0000)]
Merge pull request #931 from jkczyz/2021-05-http-client-reconnect

Cache socket address in HttpClient for reconnect

3 years agoAdd new `(C-not implementable)` tag on `EventsProvider` 2021-05-new-tag
Matt Corallo [Sun, 30 May 2021 17:04:21 +0000 (17:04 +0000)]
Add new `(C-not implementable)` tag on `EventsProvider`

This makes it so that users cannot usefully implement their own
`EventsProvider`, which would require substantial new logic in the
bindings generator (for generic methods). In the case of
`EventsProvider`, because there are no Rust methods which accept an
`EventsProvider` as an argument, this is perfectly OK as the
generated code would be entirely unused anyway.

3 years agoMerge pull request #932 from TheBlueMatt/2021-05-broadcast-locktime-delay
Matt Corallo [Sat, 29 May 2021 02:07:33 +0000 (02:07 +0000)]
Merge pull request #932 from TheBlueMatt/2021-05-broadcast-locktime-delay

Broadcast transactions only after their timelock is up

3 years agopanic if locktime is violated when broadcasting in tests 2021-05-broadcast-locktime-delay
Matt Corallo [Wed, 26 May 2021 20:02:30 +0000 (20:02 +0000)]
panic if locktime is violated when broadcasting in tests

3 years agoDont broadcast HTLC-Timeouts when closing a channel until locktime
Matt Corallo [Wed, 26 May 2021 19:58:59 +0000 (19:58 +0000)]
Dont broadcast HTLC-Timeouts when closing a channel until locktime

3 years agoSkip transactions which are locktime'd when broadcasting in test
Matt Corallo [Wed, 26 May 2021 19:59:35 +0000 (19:59 +0000)]
Skip transactions which are locktime'd when broadcasting in test

3 years agoTrack the blocks a node has connected in the TestBroadcaster
Matt Corallo [Wed, 26 May 2021 19:05:00 +0000 (19:05 +0000)]
Track the blocks a node has connected in the TestBroadcaster

3 years agoDelay broadcast of PackageTemplate packages until their locktime
Matt Corallo [Wed, 19 May 2021 21:47:42 +0000 (21:47 +0000)]
Delay broadcast of PackageTemplate packages until their locktime

This stores transaction templates temporarily until their locktime
is reached, avoiding broadcasting (or RBF bumping) transactions
prior to their locktime. For those broadcasting transactions
(potentially indirectly) via Bitcoin Core RPC, this ensures no
automated rebroadcast of transactions on the client side is
required to get transactions confirmed.

3 years agoAlways pass height to OnchainTxHandler::update_claims_view
Matt Corallo [Wed, 19 May 2021 21:47:30 +0000 (21:47 +0000)]
Always pass height to OnchainTxHandler::update_claims_view

This simplifies logic somewhat and avoids duplicating the storage
of the current height in OnchainTxHandler.

3 years agoExpose the timelock of transaction packages
Matt Corallo [Fri, 21 May 2021 21:02:03 +0000 (21:02 +0000)]
Expose the timelock of transaction packages

3 years agoSimplify tx checks in functional tests to make later commits simpler
Matt Corallo [Sun, 23 May 2021 21:11:21 +0000 (21:11 +0000)]
Simplify tx checks in functional tests to make later commits simpler

This cleans up some of the transaction format verification and docs
to make it easier when we delay CLTV-locked transactions to update
the tests.

3 years agoSimplify HolderHTLCOutput constructor and track CLTV expiry
Matt Corallo [Wed, 26 May 2021 17:17:29 +0000 (17:17 +0000)]
Simplify HolderHTLCOutput constructor and track CLTV expiry

This allows us to interrogate a PackageTemplate for the CLTV
timelock of the resulting transaction.

3 years agoAdd assertions to ensure we don't use an invalid package_amount
Matt Corallo [Wed, 26 May 2021 15:47:29 +0000 (15:47 +0000)]
Add assertions to ensure we don't use an invalid package_amount

This somewhat cleans up the public API of PackageSolvingData to
make it harder to get an invalid amount and use it, adding further
debug assertion to check it at test-time.

3 years agoMerge pull request #928 from TheBlueMatt/2021-05-really-tlv-ser
Matt Corallo [Thu, 27 May 2021 23:05:47 +0000 (23:05 +0000)]
Merge pull request #928 from TheBlueMatt/2021-05-really-tlv-ser

Migrate some inner structs to TLVs

3 years agoUpdate net_graph used for benchmarks to use new ser format. 2021-05-really-tlv-ser
Matt Corallo [Thu, 27 May 2021 00:12:30 +0000 (00:12 +0000)]
Update net_graph used for benchmarks to use new ser format.

3 years agoMigrate packages sub-fields to TLV serialization
Matt Corallo [Thu, 27 May 2021 19:19:47 +0000 (19:19 +0000)]
Migrate packages sub-fields to TLV serialization

3 years agoUse TLV serialization for Commitment transaction structures
Matt Corallo [Thu, 27 May 2021 19:19:54 +0000 (19:19 +0000)]
Use TLV serialization for Commitment transaction structures

3 years agoAdd TLV suffix to PackageTemplate
Matt Corallo [Wed, 26 May 2021 03:06:12 +0000 (03:06 +0000)]
Add TLV suffix to PackageTemplate

3 years agoMove NetworkGraph inner structs to TLV storage
Matt Corallo [Tue, 25 May 2021 23:13:11 +0000 (23:13 +0000)]
Move NetworkGraph inner structs to TLV storage

3 years agoMove ChannelManager-inner structs to TLV storage
Matt Corallo [Tue, 25 May 2021 21:20:02 +0000 (21:20 +0000)]
Move ChannelManager-inner structs to TLV storage

Note that enums are left alone as we can use the type byte already
present for future compatibility.

3 years agoAdd dummy OutPoint constructor to be used when reading it in a TLV
Matt Corallo [Tue, 25 May 2021 21:18:49 +0000 (21:18 +0000)]
Add dummy OutPoint constructor to be used when reading it in a TLV

3 years agoAdd a macro which implements Readable/Writeable using TLVs only
Matt Corallo [Tue, 25 May 2021 21:18:30 +0000 (21:18 +0000)]
Add a macro which implements Readable/Writeable using TLVs only

This also includes a `VecWriteWrapper` and `VecReadWrapper` which
implements serialization for any `Readable`/`Writeable` type that is
in a Vec. We do this instead of implementing `Readable`/`Writeable`
directly as there isn't always a univerally-defined way to serialize
a Vec and this makes things more explicit.

Finally, this tweaks existing macros (and in the new macros) to
support a trailing `,` after a list, eg
`write_tlv_fields!(stream, {(0, a),}, {});` whereas previously the
trailing `,` after the `(0, a)` would be a compile-error.

3 years agoMerge pull request #924 from GeneFerneau/alloc
Matt Corallo [Thu, 27 May 2021 19:08:00 +0000 (19:08 +0000)]
Merge pull request #924 from GeneFerneau/alloc

Use alloc for no_std builds

3 years agoUse alloc for no_std builds
Gene Ferneau [Wed, 19 May 2021 04:21:39 +0000 (04:21 +0000)]
Use alloc for no_std builds

Replace std structs with alloc equivalents to support no_std builds

f use prelude::* credit @devrandom

3 years agoCache socket address in HttpClient for reconnect
Jeffrey Czyz [Thu, 27 May 2021 15:53:14 +0000 (08:53 -0700)]
Cache socket address in HttpClient for reconnect

If the HttpClient attempts to reconnect to bitcoind that is no longer
running, the client fails to get the address from the stream. Cache the
address when connecting to prevent this.

3 years agoFix compile warnings reading type-0 TLVs
Matt Corallo [Tue, 25 May 2021 20:29:09 +0000 (20:29 +0000)]
Fix compile warnings reading type-0 TLVs

3 years agoMerge pull request #929 from jkczyz/2021-05-json-rpc-error
Matt Corallo [Thu, 27 May 2021 00:20:08 +0000 (00:20 +0000)]
Merge pull request #929 from jkczyz/2021-05-json-rpc-error

Parse RPC errors as JSON content

3 years agoParse RPC errors as JSON content
Jeffrey Czyz [Wed, 26 May 2021 17:57:39 +0000 (10:57 -0700)]
Parse RPC errors as JSON content

Bitcoin Core's JSON RPC server returns errors as HTTP error responses
with JSON content in the body. Parse this content as JSON to give a more
meaningful error. Otherwise, "binary" is given because the content
contains ASCII control characters.

3 years agoDefine an HttpError for returning error contents
Jeffrey Czyz [Wed, 26 May 2021 17:51:16 +0000 (10:51 -0700)]
Define an HttpError for returning error contents

Return an HTTP error response as a status code and contents. This allows
clients to interpret the response as desired (e.g., the contents as a
JSON-formatted error).

3 years agoMerge pull request #642 from ariard/2020-06-anchor-backend-refacto
Matt Corallo [Wed, 26 May 2021 15:50:25 +0000 (15:50 +0000)]
Merge pull request #642 from ariard/2020-06-anchor-backend-refacto

Anchor-outputs (1/3): Refactoring chain backend to extract PackageTemplate

3 years agoAdd package template unit tests
Antoine Riard [Mon, 26 Apr 2021 23:23:43 +0000 (19:23 -0400)]
Add package template unit tests

3 years agoMove compute_output_value as part of package member functions
Antoine Riard [Tue, 18 May 2021 18:09:02 +0000 (14:09 -0400)]
Move compute_output_value as part of package member functions

3 years agoMove get_height_timer out of OnchainTxHandler
Antoine Riard [Mon, 26 Apr 2021 22:22:53 +0000 (18:22 -0400)]
Move get_height_timer out of OnchainTxHandler

3 years agoMove onchain* to chain/
Antoine Riard [Sun, 18 Apr 2021 23:18:30 +0000 (19:18 -0400)]
Move onchain* to chain/

3 years agoIntegrate PackageTemplate
Antoine Riard [Thu, 20 May 2021 16:44:41 +0000 (12:44 -0400)]
Integrate PackageTemplate

This commit replaces InputMaterial in both ChannelMonitor/
OnchainTxHandler.

This doesn't change behavior.

3 years agoDuplicate fee computation utilities out of OnchainTxHandler
Antoine Riard [Fri, 7 May 2021 23:51:40 +0000 (19:51 -0400)]
Duplicate fee computation utilities out of OnchainTxHandler

Duplicated code in onchain.rs is removed in next commits.

3 years agoIntroduce PackageTemplae, a replacement of InputMaterial
Antoine Riard [Fri, 7 May 2021 23:36:50 +0000 (19:36 -0400)]
Introduce PackageTemplae, a replacement of InputMaterial

PackageTemplate aims to replace InputMaterial, introducing a clean
interface to manipulate a wide range of claim types without
OnchainTxHandler aware of special content of each.

This is used in next commits.

3 years agoAdd package.rs file
Antoine Riard [Tue, 16 Jun 2020 00:11:01 +0000 (20:11 -0400)]
Add package.rs file

Package.rs aims to gather interfaces to communicate between
onchain channel transactions parser (ChannelMonitor) and outputs
claiming logic (OnchainTxHandler). These interfaces are data
structures, generated per-case by ChannelMonitor and consumed
blindly by OnchainTxHandler.

3 years agoMerge pull request #920 from TheBlueMatt/2021-05-tlv-ser
Matt Corallo [Tue, 25 May 2021 20:07:07 +0000 (20:07 +0000)]
Merge pull request #920 from TheBlueMatt/2021-05-tlv-ser

Rebroadcast channel_announcements when we broadcast a node_announce

3 years agoAdd note in CONTRIBUTING.md that some structs need prefix/suffixes
Matt Corallo [Thu, 20 May 2021 20:09:29 +0000 (20:09 +0000)]
Add note in CONTRIBUTING.md that some structs need prefix/suffixes

3 years agoRebroadcast channel_announcements when we broadcast a node_announce
Matt Corallo [Thu, 6 May 2021 01:31:39 +0000 (01:31 +0000)]
Rebroadcast channel_announcements when we broadcast a node_announce

3 years agoProcess announcement_signatures messages in Channel and store sigs
Matt Corallo [Thu, 6 May 2021 01:15:35 +0000 (01:15 +0000)]
Process announcement_signatures messages in Channel and store sigs

Previously we handled most of the logic of announcement_signatures
in ChannelManager, rather than Channel. This is somewhat unique as
far as our message processing goes, but it also avoided having to
pass the node_secret in to the Channel.

Eventually, we'll move the node_secret behind the signer anyway, so
there isn't much reason for this, and storing the
announcement_signatures-provided signatures in the Channel allows
us to recreate the channel_announcement later for rebroadcast,
which may be useful.

3 years agoUpdate net graph copy used in tests and bench for new format
Matt Corallo [Sat, 22 May 2021 01:10:27 +0000 (01:10 +0000)]
Update net graph copy used in tests and bench for new format

3 years agoAdd version and TLV suffix for more user-facing "major" structs
Matt Corallo [Fri, 7 May 2021 22:36:45 +0000 (22:36 +0000)]
Add version and TLV suffix for more user-facing "major" structs

3 years agoSupport serializing TLV fields which may or may not be present
Matt Corallo [Thu, 6 May 2021 00:23:08 +0000 (00:23 +0000)]
Support serializing TLV fields which may or may not be present

3 years agoAppend backwards-compat TLVs to serialization of larger structs
Matt Corallo [Wed, 5 May 2021 22:56:42 +0000 (22:56 +0000)]
Append backwards-compat TLVs to serialization of larger structs

Currently our serialization is very compact, and contains version
numbers to indicate which versions the code can read a given
serialized struct. However, if you want to add a new field without
needlessly breaking the ability of previous versions of the code to
read the struct, there is not a good way to do so.

This adds dummy, currently empty, TLVs to the major structs we
serialize out for users, providing an easy place to put new
optional fields without breaking previous versions.

3 years agoMerge pull request #920 from jkczyz/2021-05-event-processing
Matt Corallo [Tue, 25 May 2021 15:55:50 +0000 (15:55 +0000)]
Merge pull request #920 from jkczyz/2021-05-event-processing

Background processing of ChannelManager and ChannelMonitor events

3 years agoRemove event channel from lightning-net-tokio
Jeffrey Czyz [Tue, 18 May 2021 23:06:13 +0000 (16:06 -0700)]
Remove event channel from lightning-net-tokio

It is unnecessary since ChannelManager has a notification interface as
of 12c735ab3af8f09dc402f8147594afcc4569be61.

3 years agoProcess ChainMonitor events in the background
Jeffrey Czyz [Tue, 11 May 2021 15:34:57 +0000 (08:34 -0700)]
Process ChainMonitor events in the background

3 years agoProcess ChannelManager events in the background
Jeffrey Czyz [Tue, 11 May 2021 15:07:54 +0000 (08:07 -0700)]
Process ChannelManager events in the background

3 years agof - Persist when check_free_holding_cells has updates
Jeffrey Czyz [Tue, 25 May 2021 07:27:28 +0000 (00:27 -0700)]
f - Persist when check_free_holding_cells has updates

3 years agoGuard get_and_clear_pending_msg_events
Jeffrey Czyz [Wed, 19 May 2021 19:26:55 +0000 (12:26 -0700)]
Guard get_and_clear_pending_msg_events

3 years agoRemove unnecessary scope from ChannelManager
Jeffrey Czyz [Fri, 21 May 2021 18:59:54 +0000 (11:59 -0700)]
Remove unnecessary scope from ChannelManager

3 years agoRefactor EventsProvider to take an EventHandler
Jeffrey Czyz [Wed, 12 May 2021 07:34:30 +0000 (00:34 -0700)]
Refactor EventsProvider to take an EventHandler

3 years agoMerge pull request #851 from TheBlueMatt/2021-03-holding-cell-clear-msg-get
Matt Corallo [Mon, 24 May 2021 21:02:50 +0000 (21:02 +0000)]
Merge pull request #851 from TheBlueMatt/2021-03-holding-cell-clear-msg-get

Clean up and more liberally free holding cell HTLCs (without re-entrancy)

3 years agoMerge pull request #926 from GeneFerneau/core
Matt Corallo [Mon, 24 May 2021 17:49:54 +0000 (17:49 +0000)]
Merge pull request #926 from GeneFerneau/core

Use core replacements for std members

3 years agoUse core replacements for std members
Gene Ferneau [Sun, 23 May 2021 23:22:46 +0000 (23:22 +0000)]
Use core replacements for std members

In preparation for no_std build support, replace std structs and
functions with core equivalents

3 years agoAdd trace to ci-fuzz.sh to make debugging CI issues easier 2021-03-holding-cell-clear-msg-get
Matt Corallo [Wed, 21 Apr 2021 22:24:36 +0000 (22:24 +0000)]
Add trace to ci-fuzz.sh to make debugging CI issues easier

3 years agoLog info about HTLC failures when we fail them back
Matt Corallo [Tue, 20 Apr 2021 21:35:11 +0000 (21:35 +0000)]
Log info about HTLC failures when we fail them back

3 years agoRead monitors from our KeysInterface in chanmon_consistency_fuzz
Matt Corallo [Tue, 9 Feb 2021 20:22:44 +0000 (15:22 -0500)]
Read monitors from our KeysInterface in chanmon_consistency_fuzz

If the fuzz target is failing due to a channel force-close, the
immediately-visible error is that we're signing a stale state. This
is because the ChannelMonitorUpdateStep::ChannelForceClosed event
results in a signature in the test clone which was deserialized
using a OnlyReadsKeysInterface. Instead, we need to deserialize
using the full KeysInterface instance.

3 years agoAdd a test for HTLC freeing on monitor update restoration
Matt Corallo [Mon, 1 Mar 2021 02:00:46 +0000 (21:00 -0500)]
Add a test for HTLC freeing on monitor update restoration

3 years agoStop failing back HTLCs on peer disconnection
Matt Corallo [Fri, 20 Nov 2020 20:49:53 +0000 (15:49 -0500)]
Stop failing back HTLCs on peer disconnection

Previously, if we got disconnected from a peer while there were
HTLCs pending forwarding in the holding cell, we'd clear them and
fail them all backwards. This is largely fine, but since we now
have support for handling such HTLCs on reconnect, we might as
well not, instead relying on our timeout logic to fail them
backwards if it takes too long to forward them.

3 years ago[fuzz] Handle monitor updates during get_and_clear_pending_msg_events
Matt Corallo [Wed, 21 Apr 2021 02:37:02 +0000 (02:37 +0000)]
[fuzz] Handle monitor updates during get_and_clear_pending_msg_events

Because we may now generate a monitor update during
get_and_clear_pending_msg_events calls, we need to ensure we
re-serialize the relevant ChannelManager before attempting to
reload it, if such a monitor update occurred.

3 years agoFree holding cell on monitor-updating-restored when there's no upd
Matt Corallo [Thu, 18 Mar 2021 22:03:30 +0000 (18:03 -0400)]
Free holding cell on monitor-updating-restored when there's no upd

If there is no pending channel update messages when monitor updating
is restored (though there may be an RAA to send), and we're
connected to our peer and not awaiting a remote RAA, we need to
free anything in our holding cell.

However, we don't want to immediately free the holding cell during
channel_monitor_updated as it presents a somewhat bug-prone case of
reentrancy:
 a) it would re-enter user code around a monitor update while being
    called from user code notifying us of the same monitor being
    updated, making deadlocs very likely (in fact, our fuzzers
    would have a bug here!),
 b) the re-entrancy only occurs in a very rare case, making it
    likely users will not hit it in testing, only deadlocking in
    production.

Thus, we add a holding-cell-free pass over each channel in
get_and_clear_pending_msg_events. This fits up nicely with the
anticipated bug - users almost certainly need to process new
network messages immediately after monitor updating has been
restored to send messages which were not sent originally when the
monitor updating was paused.

Without this, chanmon_fail_consistency was able to find a stuck
condition where we sit on an HTLC failure in our holding cell and
don't ever handle it (at least until we have other actions to take
which empty the holding cell).

3 years agoDRY ChannelError conversion macros
Matt Corallo [Thu, 18 Mar 2021 22:23:05 +0000 (18:23 -0400)]
DRY ChannelError conversion macros

Both break_chan_entry and try_chan_entry do almost identical work,
only differing on if they `break` or `return` in response to an
error. Because we will now also need an option to do neither, we
break out the common code into a shared `convert_chan_err` macro.

3 years agoUse new chan_restoration macros in channel_reestablish handling.
Matt Corallo [Thu, 18 Mar 2021 16:44:31 +0000 (12:44 -0400)]
Use new chan_restoration macros in channel_reestablish handling.

This merges the code for restoring channel functionality between
channel monitor updating restored and peer reconnection, reducing
redundant code.

3 years ago[fuzz] Allow SendAnnouncementSigs events in chanmon_consistency
Matt Corallo [Tue, 24 Nov 2020 00:12:31 +0000 (19:12 -0500)]
[fuzz] Allow SendAnnouncementSigs events in chanmon_consistency

Because of the merge between peer reconnection and channel monitor
updating channel restoration code, we now sometimes generate
(somewhat spurious) announcement signatures when restoring channel
monitor updating. This should not result in a fuzzing failure.

3 years ago[fuzz] Be more strict about msg events in chanmon_consistency
Matt Corallo [Tue, 24 Nov 2020 00:12:19 +0000 (19:12 -0500)]
[fuzz] Be more strict about msg events in chanmon_consistency

This fails chanmon_consistency on IgnoreError error events and on
messages left over to be sent to a just-disconnected peer, which
should have been drained.

These should never appear, so consider them a fuzzer fail case.

3 years agoClean up channel updating macro somewhat
Matt Corallo [Tue, 15 Dec 2020 02:52:10 +0000 (21:52 -0500)]
Clean up channel updating macro somewhat

This mostly swaps some Vecs that can only ever contain one element
for Options.

3 years agoMove channel restoration after monitor update to a two-part macro
Matt Corallo [Fri, 20 Nov 2020 19:29:33 +0000 (14:29 -0500)]
Move channel restoration after monitor update to a two-part macro

The channel restoration code in channel monitor updating and peer
reconnection both do incredibly similar things, and there is
little reason to have them be separate. Sadly because they require
holding a lock with a reference to elements in the lock, its not
practical to make them utility functions, so instead we introduce
a two-step macro here which will eventually be used for both.

Because we still support pre-NLL Rust, the macro has to be in two
parts - one which runs with the channel_state lock, and one which
does not.

3 years agoDrop unused variable and loop in channel_monitor_updated
Matt Corallo [Fri, 20 Nov 2020 19:06:17 +0000 (14:06 -0500)]
Drop unused variable and loop in channel_monitor_updated

3 years ago[fuzz] Print the output of all failed test cases, not one test.
Matt Corallo [Mon, 23 Nov 2020 23:22:29 +0000 (18:22 -0500)]
[fuzz] Print the output of all failed test cases, not one test.

Our fuzz tests previously only printed the log output of the first
fuzz test case to fail. This commit changes that (with lots of
auto-generated updates) to ensure we print all log outputs.

3 years agoMerge pull request #918 from TheBlueMatt/2021-05-dup-claims
Matt Corallo [Thu, 20 May 2021 17:10:52 +0000 (17:10 +0000)]
Merge pull request #918 from TheBlueMatt/2021-05-dup-claims

Make payments not duplicatively fail/succeed on reload/reconnect

3 years ago[fuzz] Make get_secure_random_bytes in chanmon_consistency unique 2021-05-dup-claims
Matt Corallo [Thu, 20 May 2021 16:38:18 +0000 (16:38 +0000)]
[fuzz] Make get_secure_random_bytes in chanmon_consistency unique

3 years agoTest pending payments when resolved duplicatively via reconnect
Matt Corallo [Sun, 9 May 2021 17:53:55 +0000 (17:53 +0000)]
Test pending payments when resolved duplicatively via reconnect

3 years agoTest pending payments when duplicatively resolved on chain
Matt Corallo [Sat, 8 May 2021 22:54:26 +0000 (22:54 +0000)]
Test pending payments when duplicatively resolved on chain

3 years agoMake payments not duplicatively fail/succeed on reload/reconnect
Matt Corallo [Sun, 9 May 2021 19:19:11 +0000 (19:19 +0000)]
Make payments not duplicatively fail/succeed on reload/reconnect

We currently generate duplicative PaymentFailed/PaymentSent events
in two cases:

a) If we receive a update_fulfill_htlc message, followed by a
   disconnect, then a resend of the same update_fulfill_htlc
   message, we will generate a PaymentSent event for each message.

b) When a Channel is closed, any outbound HTLCs which were relayed
   through it are simply dropped when the Channel is. From there,
   the ChannelManager relies on the ChannelMonitor having a copy of
   the relevant fail-/claim-back data and processes the HTLC
   fail/claim when the ChannelMonitor tells it to.

   If, due to an on-chain event, an HTLC is failed/claimed, and
   then we serialize the ChannelManager, but do not re-serialize
   the relevant ChannelMonitor, we may end up getting a duplicative
   event.

In order to provide the expected consistency, we add explicit
tracking of pending outbound payments using their unique
session_priv field which is generated when the payment is sent.
Then, before generating PaymentFailed/PaymentSent events, we check
that the session_priv for the payment is still pending.

Thix fixes #209.

3 years agoMerge pull request #925 from valentinewallace/ignore-zlib-compressed-msgs
Matt Corallo [Thu, 20 May 2021 16:26:49 +0000 (16:26 +0000)]
Merge pull request #925 from valentinewallace/ignore-zlib-compressed-msgs

Ignore messages with zlib-compressed fields

3 years agoReturn new DecodeError::UnsupportedCompression
Valentine Wallace [Wed, 19 May 2021 19:15:41 +0000 (15:15 -0400)]
Return new DecodeError::UnsupportedCompression

if we receive a message with zlib-compressed values.

3 years agoAdd new DecodeError for messages that include zlib-compressed values
Valentine Wallace [Wed, 19 May 2021 18:05:03 +0000 (14:05 -0400)]
Add new DecodeError for messages that include zlib-compressed values

No need to disconnect peers if this error is encountered. It just means
we can't use some of their gossip messages.

3 years agoMerge pull request #923 from ariard/2021-05-split-sign-justice
Matt Corallo [Tue, 18 May 2021 15:24:57 +0000 (15:24 +0000)]
Merge pull request #923 from ariard/2021-05-split-sign-justice

Split `sign_justice_transaction` in two halves

3 years agokeysinterface: replace Result match with map_err
Antoine Riard [Mon, 17 May 2021 18:22:12 +0000 (14:22 -0400)]
keysinterface: replace Result match with map_err

3 years agoSplit `sign_justice_transaction` in two halves
Antoine Riard [Sat, 15 May 2021 21:20:10 +0000 (17:20 -0400)]
Split `sign_justice_transaction` in two halves

To avoid caller data struct storing HTLC-related information when
a revokeable output is claimed on top of a commitment/second-stage
HTLC transactions, we split `keysinterface::sign_justice_transaction`
in two new halves `keysinterfaces::sign_justice_revoked_output` and
`keysinterfaces::sign_justice_revoked_htlc`.

Further, this split offers more flexibility to signer policy as a
commitment revokeable output might be of a value far more significant
than HTLC ones.

3 years agoMerge pull request #916 from TheBlueMatt/2021-05-fix-disabled-announcements
Matt Corallo [Sat, 15 May 2021 00:44:40 +0000 (00:44 +0000)]
Merge pull request #916 from TheBlueMatt/2021-05-fix-disabled-announcements

Avoid persisting a ChannelManager after each timer tick and send update_channel re-enable messages

3 years agoSimplify+expand per-channel check in test_announce_disable_channels 2021-05-fix-disabled-announcements
Matt Corallo [Thu, 13 May 2021 15:41:05 +0000 (15:41 +0000)]
Simplify+expand per-channel check in test_announce_disable_channels