]>
git.bitcoin.ninja Git - rapid-gossip-sync-server/log
Arik Sosman [Tue, 2 Jul 2024 07:29:00 +0000 (00:29 -0700)]
Extract snapshot reminder inclusion check method.
We will need to determine whether or not a snapshot should include
reminders for both channel and node update messages. To prepare for
that, we extract the decision logic into its own method.
Arik Sosman [Tue, 2 Jul 2024 07:20:19 +0000 (00:20 -0700)]
Only consider node announcements from current graph.
We want to ignore any node announcements that have already been
pruned. To do so, we extract all the node IDs from the network graph,
and use those to filter our queries.
Arik Sosman [Mon, 24 Jun 2024 18:36:13 +0000 (14:36 -0400)]
Upgrade LDK to 0.0.124.
Arik Sosman [Fri, 28 Jun 2024 21:13:05 +0000 (17:13 -0400)]
Rename latest_details_after_seen to latest_details.
If the latest node details have already been seen by a client, we
still need to store them for correctly detecting reminder necessity
in the future.
Arik Sosman [Fri, 28 Jun 2024 17:57:48 +0000 (13:57 -0400)]
Parametrize prune interval.
Create a config variable representing the time interval whereafter
graph data gets pruned. This value should be used to limit lookup
time frames.
Matt Corallo [Fri, 7 Jun 2024 18:54:08 +0000 (11:54 -0700)]
Merge pull request #82 from arik-so/gossip-v2-bugfixes
Gossip v2 bugfixes
Arik Sosman [Fri, 7 Jun 2024 18:15:16 +0000 (11:15 -0700)]
Update unit tests.
Arik Sosman [Fri, 7 Jun 2024 18:15:06 +0000 (11:15 -0700)]
Serialize removal of all socket addresses in snapshots.
Arik Sosman [Fri, 7 Jun 2024 18:14:42 +0000 (11:14 -0700)]
Send addresses and features for hitherto unseen nodes.
Arik [Wed, 29 May 2024 16:42:56 +0000 (09:42 -0700)]
Merge pull request #81 from TheBlueMatt/main
Correct v2 symlink paths
Matt Corallo [Wed, 29 May 2024 14:54:57 +0000 (14:54 +0000)]
Correct v2 symlink paths
Matt Corallo [Wed, 29 May 2024 13:50:46 +0000 (06:50 -0700)]
Merge pull request #76 from arik-so/arik/2024/02/node-gossip
RGS v2: NodeAnnouncement Delta Serialization
Arik Sosman [Wed, 15 May 2024 21:56:08 +0000 (14:56 -0700)]
Store v1 and v2 snapshots.
Arik Sosman [Mon, 20 May 2024 06:02:39 +0000 (23:02 -0700)]
Version-aware serialization of node features and addresses in delta.
Arik Sosman [Thu, 25 Apr 2024 05:18:10 +0000 (22:18 -0700)]
Introduce method for comparing node announcements.
Arik Sosman [Wed, 15 May 2024 19:45:04 +0000 (12:45 -0700)]
Track db schema version 14.
Arik Sosman [Thu, 28 Mar 2024 06:48:08 +0000 (23:48 -0700)]
Test NodeAnnouncement persistence.
Matt Corallo [Wed, 15 May 2024 19:04:09 +0000 (12:04 -0700)]
Merge pull request #79 from arik-so/persist-node-data
Persist NodeAnnouncement Gossip
Arik Sosman [Thu, 28 Mar 2024 06:57:19 +0000 (23:57 -0700)]
Test storing node addresses.
Arik Sosman [Thu, 28 Mar 2024 06:48:08 +0000 (23:48 -0700)]
Test NodeAnnouncement persistence.
Arik Sosman [Wed, 15 May 2024 16:10:08 +0000 (09:10 -0700)]
Store serialized addresses and features.
Arik Sosman [Thu, 28 Mar 2024 06:45:05 +0000 (23:45 -0700)]
Update rust-lightning version.
Arik Sosman [Thu, 28 Mar 2024 06:28:41 +0000 (23:28 -0700)]
Update schema for node announcement and address storage.
Matt Corallo [Mon, 13 May 2024 21:14:50 +0000 (14:14 -0700)]
Merge pull request #78 from arik-so/conservative-announcements
Only consider channel announcements with bidirectional updates.
Arik Sosman [Fri, 10 May 2024 23:42:20 +0000 (16:42 -0700)]
Only consider channel announcements with bidirectional updates.
Matt Corallo [Fri, 10 May 2024 20:07:16 +0000 (13:07 -0700)]
Merge pull request #77 from arik-so/static-channel-reminders
Send reminders for non-mutating channels
Arik Sosman [Fri, 10 May 2024 16:41:30 +0000 (09:41 -0700)]
Increase reminder snapshot scope interval threshold.
Arik Sosman [Fri, 10 May 2024 01:55:54 +0000 (18:55 -0700)]
Use reference timestamp for reminder calculation.
Arik Sosman [Fri, 10 May 2024 01:40:37 +0000 (18:40 -0700)]
Allow passing a timestamp override to snapshot calculation.
Arik Sosman [Thu, 9 May 2024 23:18:10 +0000 (16:18 -0700)]
Make reminder inclusion more conservative.
Arik Sosman [Thu, 9 May 2024 06:58:52 +0000 (23:58 -0700)]
Update test to reflect new reminder mechanics.
Arik Sosman [Thu, 9 May 2024 06:58:29 +0000 (23:58 -0700)]
Detect beginning of static channel status streaks.
Previously, whenever a channel hasn't received updates in
6+ days, we would automatically send incremental reminders
that only contain the update flags.
However, if a channel has been received updates
within that six-day-timeframe, but all those updates were
identical, it would result in no updates being added to the
snapshot because the mutation set ended up empty and that
data would get purged from the serialization.
In order to avoid the reminder logic being duped by
channels simply being consistent, we now look up the
beginning of the latest continuous stretch of non-mutating
channel updates. If a channel's details have not been
altered in more than six days, we now send reminders no
matter the frequency with which channel updates have been
received since.
Arik Sosman [Wed, 8 May 2024 08:37:23 +0000 (01:37 -0700)]
Test that channels with non-changing updates result in missing reminders.
Arik Sosman [Wed, 24 Apr 2024 23:51:57 +0000 (16:51 -0700)]
Don't panic while panicking.
Arik Sosman [Wed, 8 May 2024 08:03:46 +0000 (01:03 -0700)]
Stop tracking Cargo.lock
Arik [Sun, 24 Mar 2024 01:01:04 +0000 (18:01 -0700)]
Merge pull request #74 from TheBlueMatt/main
Bump LDK to 0.0.121/rust-bitcoin 0.30
Matt Corallo [Sat, 23 Mar 2024 21:24:38 +0000 (21:24 +0000)]
Fix unused `Result` in tests
Matt Corallo [Sat, 23 Mar 2024 21:24:29 +0000 (21:24 +0000)]
Fix indentation in `test_ln_peers`
Matt Corallo [Wed, 13 Mar 2024 14:28:26 +0000 (14:28 +0000)]
Bump LDK to 0.0.121/rust-bitcoin 0.30, bumping MSRV to 1.63
Co-authored-by: Arik Sosman <git@arik.io>
Matt Corallo [Fri, 22 Mar 2024 22:44:41 +0000 (22:44 +0000)]
Merge pull request #73 from arik-so/arik/2024-03-tokio-runtime-split
Use separate Tokio runtime for gossip persistence.
Arik Sosman [Thu, 21 Mar 2024 00:41:34 +0000 (17:41 -0700)]
Simplify client creation.
Arik Sosman [Tue, 12 Mar 2024 06:41:27 +0000 (23:41 -0700)]
Use separate tokio runtime for gossip persistence.
Arik Sosman [Thu, 21 Mar 2024 00:45:18 +0000 (17:45 -0700)]
Fix CI compilation in beta environment.
Arik [Tue, 6 Feb 2024 19:42:10 +0000 (11:42 -0800)]
Merge pull request #71 from TheBlueMatt/main
Do DB insertions in parallel
Matt Corallo [Thu, 1 Feb 2024 17:39:10 +0000 (17:39 +0000)]
Merge pull request #72 from lightningdevkit/arik/2024/01/bitcoin-rest-error
Improve bitcoind RPC/Rest error message
Arik Sosman [Tue, 30 Jan 2024 19:27:22 +0000 (11:27 -0800)]
Improve bitcoind RPC/Rest error message.
Matt Corallo [Mon, 29 Jan 2024 17:24:32 +0000 (17:24 +0000)]
Do DB insertions in parallel
When inserting new gossip into the DB, we block the LDK peer
handling if we get behind. This is mostly okay, but can cause ping
timeouts and reconnections, which isn't ideal. To limit how often
we should see this, here we move to doing the new gossip insertions
in parallel.
Matt Corallo [Mon, 29 Jan 2024 17:27:14 +0000 (17:27 +0000)]
Merge pull request #70 from domZippilli/2024/01/load-peers-tolerating-trailing-comma
Ignore trailing commas when parsing peers, and log more on error.
Dominic Zippilli [Thu, 25 Jan 2024 06:02:54 +0000 (22:02 -0800)]
Ignore trailing commas when parsing peers, and log more on error.
Matt Corallo [Wed, 10 Jan 2024 17:38:14 +0000 (17:38 +0000)]
Merge pull request #68 from johncantrell97/118
bump to ldk 118
John Cantrell [Tue, 5 Dec 2023 21:42:47 +0000 (16:42 -0500)]
bump to ldk 118
Matt Corallo [Sat, 11 Nov 2023 17:27:16 +0000 (17:27 +0000)]
Merge pull request #65 from arik-so/announcement-history-tests
Update-recency-based inclusion logic tests
Arik Sosman [Sat, 4 Nov 2023 04:43:33 +0000 (21:43 -0700)]
Test announcement inclusion logic.
Arik Sosman [Sat, 4 Nov 2023 06:05:26 +0000 (23:05 -0700)]
Include old updates when necessary.
When a channel has only recently become bidirectional,
but there has not been a new update in the old direction
since the last sync, the latest update in the old direction
must still be included in full because it is the first time
the full channel is being snapshotted.
Arik Sosman [Sat, 4 Nov 2023 03:58:21 +0000 (20:58 -0700)]
Allow seen override for announcements.
Matt Corallo [Thu, 19 Oct 2023 19:50:38 +0000 (19:50 +0000)]
Merge pull request #63 from arik-so/2023/10/ordering_fix
Fix update recency issue with expanded test coverage
Arik Sosman [Thu, 19 Oct 2023 18:52:32 +0000 (11:52 -0700)]
Optimize indices for expired value fix lookup.
Arik Sosman [Thu, 19 Oct 2023 18:52:06 +0000 (11:52 -0700)]
Fix expired value being sent as the latest update.
Arik Sosman [Thu, 19 Oct 2023 18:49:27 +0000 (11:49 -0700)]
Reproduce test for expired value being sent as the latest update.
Arik Sosman [Wed, 18 Oct 2023 00:37:55 +0000 (17:37 -0700)]
Create snapshot generation test.
Arik Sosman [Tue, 17 Oct 2023 20:49:48 +0000 (13:49 -0700)]
Extract snapshot generation method.
Arik Sosman [Mon, 16 Oct 2023 15:27:45 +0000 (08:27 -0700)]
Replicate Mutiny test scenarios.
Arik Sosman [Sat, 14 Oct 2023 01:39:27 +0000 (18:39 -0700)]
Allow setting custom gossip seen timestamps.
Matt Corallo [Sun, 15 Oct 2023 19:54:49 +0000 (19:54 +0000)]
Merge pull request #62 from arik-so/2023/10/ordering_fix
Create Postgres step in CI.
Arik Sosman [Sat, 14 Oct 2023 01:01:21 +0000 (18:01 -0700)]
Assert that schema is uninitialized.
Arik Sosman [Fri, 13 Oct 2023 19:54:34 +0000 (12:54 -0700)]
Create Postgres step in CI.
Matt Corallo [Thu, 12 Oct 2023 17:35:33 +0000 (17:35 +0000)]
Merge pull request #60 from arik-so/2023/08/first_unit_test
First unit test
Arik Sosman [Tue, 12 Sep 2023 19:25:09 +0000 (12:25 -0700)]
Assert schema sanitization.
Arik Sosman [Fri, 8 Sep 2023 17:47:11 +0000 (10:47 -0700)]
Use schema as test logger id.
Arik Sosman [Tue, 15 Aug 2023 21:39:12 +0000 (14:39 -0700)]
Create first unit test.
Arik Sosman [Tue, 15 Aug 2023 21:38:46 +0000 (14:38 -0700)]
Upgrade to LDK 117.
Arik Sosman [Tue, 15 Aug 2023 21:38:17 +0000 (14:38 -0700)]
Create TestLogger.
Arik Sosman [Tue, 15 Aug 2023 21:37:25 +0000 (14:37 -0700)]
Create test-aware db env vars.
Matt Corallo [Wed, 6 Sep 2023 23:51:06 +0000 (23:51 +0000)]
Merge pull request #56 from arik-so/2023/08/connect-to-peers-asynchronously
Track gossip data prior to peer connection loop.
Arik Sosman [Wed, 6 Sep 2023 23:26:59 +0000 (16:26 -0700)]
Always wait 10 seconds prior to attempting reconnection.
Arik Sosman [Wed, 6 Sep 2023 04:28:15 +0000 (21:28 -0700)]
Don't special-case initial connection attempt.
Arik Sosman [Mon, 28 Aug 2023 21:40:35 +0000 (14:40 -0700)]
Connect peers independently.
Previously, we wouldn't start monitoring the gossip data until all
intended peers had either failed or succeeded to connect.
Instead, we're now spawn each peer connection independently, and only
await the first 5 successful peer connections.
Matt Corallo [Wed, 6 Sep 2023 22:16:26 +0000 (22:16 +0000)]
Merge pull request #57 from arik-so/2023/08/old-incremental-update-fix
Send full updates after old last seen updates.
Arik Sosman [Tue, 29 Aug 2023 01:01:03 +0000 (18:01 -0700)]
Send full updates after old last seen updates.
Previously, whenever we saw that there was a previous update that a
client would have seen, we simply calculated the delta set based on
which properties have changed, and would most likely send an
incremental update set (excepting the case of a new or newly sent
announcement, in which case all sent updates are full).
However, if the last seen update was old, and there's a chance that
a user may have run RGS since, it is possible that due to the
7-day-backdating-mechanism included on the client, the reference
update would no longer be present.
To fix that, anytime we see that a last seen update is more than six
days old, we automatically include a full update.
Matt Corallo [Sat, 2 Sep 2023 16:27:08 +0000 (16:27 +0000)]
Merge pull request #59 from arik-so/2023/08/custom-intervals
Distinction between snapshot and symlink intervals
Arik Sosman [Fri, 1 Sep 2023 21:42:40 +0000 (14:42 -0700)]
Dynamically adjust snapshot scopes.
Previously, we had hard-coded factors for the default snapshot
generation interval, which also served as the minimum snapshot
scope. In this commit, we substitute that with a doubling
mechanism that stops once it reaches or exceeds the
21-day-mark, which can be configured using an additional flag.
Arik Sosman [Thu, 31 Aug 2023 22:59:32 +0000 (15:59 -0700)]
Distinguish between snapshot and symlink intervals.
Snapshots may be generated every day or even every couple days,
but symlinks will be generated at the given symlink granularity.
To avoid gcd complications, all snapshot generation intervals
must be multiples of the symlink granularity interval.
Graham Krizek [Thu, 6 Jul 2023 05:31:24 +0000 (00:31 -0500)]
Allow the snapshot interval to be configurable in the environment variables
Matt Corallo [Mon, 28 Aug 2023 21:20:51 +0000 (21:20 +0000)]
Merge pull request #55 from arik-so/2023/08/multiplication-overflow-fix
Fix multiplication overflow bug
Arik Sosman [Mon, 28 Aug 2023 16:07:19 +0000 (09:07 -0700)]
Fix multiplication overflow bug.
The `snapshot_sync_day_factors` array is sorted
ascendingly, so find() will return on the first
iteration that is at least equal to the requested
interval.
However, the last value in the array is u64::max,
which means that multiplying it with DAY_SECONDS
will overflow. To avoid that, we use saturating_mul.
Matt Corallo [Thu, 17 Aug 2023 20:55:48 +0000 (20:55 +0000)]
Merge pull request #54 from arik-so/2023/08/incremental-update-fix
Only send full updates with announcements
Arik Sosman [Thu, 17 Aug 2023 20:14:18 +0000 (13:14 -0700)]
Create indices for schema version 12.
Arik Sosman [Thu, 17 Aug 2023 04:35:55 +0000 (21:35 -0700)]
Create method for obtaining UTC-prepared Postgres client.
Arik Sosman [Thu, 17 Aug 2023 04:50:43 +0000 (21:50 -0700)]
Send full updates whenever the channel announcement is sent.
Arik Sosman [Thu, 17 Aug 2023 03:55:51 +0000 (20:55 -0700)]
Fix ordering of intermediate commits to ensure the latest comes first and is used as the authoritative source of what's latest.
Matt Corallo [Thu, 17 Aug 2023 00:16:14 +0000 (00:16 +0000)]
Merge pull request #53 from arik-so/2023/08/query-timestamp-fix
Switch timestamp queries to use f64
Arik Sosman [Thu, 17 Aug 2023 00:04:06 +0000 (17:04 -0700)]
Switch timestamp reads to use i64.
Arik Sosman [Wed, 16 Aug 2023 23:18:40 +0000 (16:18 -0700)]
Switch timestamp queries to use f64.
Matt Corallo [Sun, 13 Aug 2023 16:37:54 +0000 (16:37 +0000)]
Merge pull request #49 from TheBlueMatt/main
Matt Corallo [Sun, 16 Jul 2023 20:41:28 +0000 (20:41 +0000)]
Make snapshotter a bit more amenable to changes in interval
By separating the concept of "seconds in a day" from the snapshot
interval.
Matt Corallo [Sun, 16 Jul 2023 20:41:00 +0000 (20:41 +0000)]
Only generate as many symlinks as we need (rather than 27 years worth)
Matt Corallo [Sun, 13 Aug 2023 05:30:42 +0000 (05:30 +0000)]
Merge pull request #51 from arik-so/2023/08/pre_test_refactors
Allow arbitrary logger types.
Arik Sosman [Fri, 4 Aug 2023 21:30:02 +0000 (14:30 -0700)]
Allow log threshold configuration.
Arik Sosman [Thu, 3 Aug 2023 01:34:01 +0000 (18:34 -0700)]
Remove println from verifier.rs
Arik Sosman [Thu, 3 Aug 2023 01:26:55 +0000 (18:26 -0700)]
Remove println from tracking.rs