]> git.bitcoin.ninja Git - rapid-gossip-sync-server/log
rapid-gossip-sync-server
7 weeks agoExtract snapshot reminder inclusion check method.
Arik Sosman [Tue, 2 Jul 2024 07:29:00 +0000 (00:29 -0700)]
Extract snapshot reminder inclusion check method.

We will need to determine whether or not a snapshot should include
reminders for both channel and node update messages. To prepare for
that, we extract the decision logic into its own method.

7 weeks agoOnly consider node announcements from current graph.
Arik Sosman [Tue, 2 Jul 2024 07:20:19 +0000 (00:20 -0700)]
Only consider node announcements from current graph.

We want to ignore any node announcements that have already been
pruned. To do so, we extract all the node IDs from the network graph,
and use those to filter our queries.

7 weeks agoUpgrade LDK to 0.0.124.
Arik Sosman [Mon, 24 Jun 2024 18:36:13 +0000 (14:36 -0400)]
Upgrade LDK to 0.0.124.

2 months agoRename latest_details_after_seen to latest_details.
Arik Sosman [Fri, 28 Jun 2024 21:13:05 +0000 (17:13 -0400)]
Rename latest_details_after_seen to latest_details.

If the latest node details have already been seen by a client, we
still need to store them for correctly detecting reminder necessity
in the future.

2 months agoParametrize prune interval.
Arik Sosman [Fri, 28 Jun 2024 17:57:48 +0000 (13:57 -0400)]
Parametrize prune interval.

Create a config variable representing the time interval whereafter
graph data gets pruned. This value should be used to limit lookup
time frames.

5 months agoMerge pull request #82 from arik-so/gossip-v2-bugfixes
Matt Corallo [Fri, 7 Jun 2024 18:54:08 +0000 (11:54 -0700)]
Merge pull request #82 from arik-so/gossip-v2-bugfixes

Gossip v2 bugfixes

5 months agoUpdate unit tests.
Arik Sosman [Fri, 7 Jun 2024 18:15:16 +0000 (11:15 -0700)]
Update unit tests.

5 months agoSerialize removal of all socket addresses in snapshots.
Arik Sosman [Fri, 7 Jun 2024 18:15:06 +0000 (11:15 -0700)]
Serialize removal of all socket addresses in snapshots.

5 months agoSend addresses and features for hitherto unseen nodes.
Arik Sosman [Fri, 7 Jun 2024 18:14:42 +0000 (11:14 -0700)]
Send addresses and features for hitherto unseen nodes.

5 months agoMerge pull request #81 from TheBlueMatt/main
Arik [Wed, 29 May 2024 16:42:56 +0000 (09:42 -0700)]
Merge pull request #81 from TheBlueMatt/main

Correct v2 symlink paths

5 months agoCorrect v2 symlink paths
Matt Corallo [Wed, 29 May 2024 14:54:57 +0000 (14:54 +0000)]
Correct v2 symlink paths

5 months agoMerge pull request #76 from arik-so/arik/2024/02/node-gossip
Matt Corallo [Wed, 29 May 2024 13:50:46 +0000 (06:50 -0700)]
Merge pull request #76 from arik-so/arik/2024/02/node-gossip

RGS v2: NodeAnnouncement Delta Serialization

5 months agoStore v1 and v2 snapshots.
Arik Sosman [Wed, 15 May 2024 21:56:08 +0000 (14:56 -0700)]
Store v1 and v2 snapshots.

5 months agoVersion-aware serialization of node features and addresses in delta.
Arik Sosman [Mon, 20 May 2024 06:02:39 +0000 (23:02 -0700)]
Version-aware serialization of node features and addresses in delta.

5 months agoIntroduce method for comparing node announcements.
Arik Sosman [Thu, 25 Apr 2024 05:18:10 +0000 (22:18 -0700)]
Introduce method for comparing node announcements.

5 months agoTrack db schema version 14.
Arik Sosman [Wed, 15 May 2024 19:45:04 +0000 (12:45 -0700)]
Track db schema version 14.

5 months agoTest NodeAnnouncement persistence.
Arik Sosman [Thu, 28 Mar 2024 06:48:08 +0000 (23:48 -0700)]
Test NodeAnnouncement persistence.

5 months agoMerge pull request #79 from arik-so/persist-node-data
Matt Corallo [Wed, 15 May 2024 19:04:09 +0000 (12:04 -0700)]
Merge pull request #79 from arik-so/persist-node-data

Persist NodeAnnouncement Gossip

5 months agoTest storing node addresses.
Arik Sosman [Thu, 28 Mar 2024 06:57:19 +0000 (23:57 -0700)]
Test storing node addresses.

5 months agoTest NodeAnnouncement persistence.
Arik Sosman [Thu, 28 Mar 2024 06:48:08 +0000 (23:48 -0700)]
Test NodeAnnouncement persistence.

5 months agoStore serialized addresses and features.
Arik Sosman [Wed, 15 May 2024 16:10:08 +0000 (09:10 -0700)]
Store serialized addresses and features.

5 months agoUpdate rust-lightning version.
Arik Sosman [Thu, 28 Mar 2024 06:45:05 +0000 (23:45 -0700)]
Update rust-lightning version.

5 months agoUpdate schema for node announcement and address storage.
Arik Sosman [Thu, 28 Mar 2024 06:28:41 +0000 (23:28 -0700)]
Update schema for node announcement and address storage.

5 months agoMerge pull request #78 from arik-so/conservative-announcements
Matt Corallo [Mon, 13 May 2024 21:14:50 +0000 (14:14 -0700)]
Merge pull request #78 from arik-so/conservative-announcements

Only consider channel announcements with bidirectional updates.

6 months agoOnly consider channel announcements with bidirectional updates.
Arik Sosman [Fri, 10 May 2024 23:42:20 +0000 (16:42 -0700)]
Only consider channel announcements with bidirectional updates.

6 months agoMerge pull request #77 from arik-so/static-channel-reminders
Matt Corallo [Fri, 10 May 2024 20:07:16 +0000 (13:07 -0700)]
Merge pull request #77 from arik-so/static-channel-reminders

Send reminders for non-mutating channels

6 months agoIncrease reminder snapshot scope interval threshold.
Arik Sosman [Fri, 10 May 2024 16:41:30 +0000 (09:41 -0700)]
Increase reminder snapshot scope interval threshold.

6 months agoUse reference timestamp for reminder calculation.
Arik Sosman [Fri, 10 May 2024 01:55:54 +0000 (18:55 -0700)]
Use reference timestamp for reminder calculation.

6 months agoAllow passing a timestamp override to snapshot calculation.
Arik Sosman [Fri, 10 May 2024 01:40:37 +0000 (18:40 -0700)]
Allow passing a timestamp override to snapshot calculation.

6 months agoMake reminder inclusion more conservative.
Arik Sosman [Thu, 9 May 2024 23:18:10 +0000 (16:18 -0700)]
Make reminder inclusion more conservative.

6 months agoUpdate test to reflect new reminder mechanics.
Arik Sosman [Thu, 9 May 2024 06:58:52 +0000 (23:58 -0700)]
Update test to reflect new reminder mechanics.

6 months agoDetect beginning of static channel status streaks.
Arik Sosman [Thu, 9 May 2024 06:58:29 +0000 (23:58 -0700)]
Detect beginning of static channel status streaks.

Previously, whenever a channel hasn't received updates in
6+ days, we would automatically send incremental reminders
that only contain the update flags.

However, if a channel has been received updates
within that six-day-timeframe, but all those updates were
identical, it would result in no updates being added to the
snapshot because the mutation set ended up empty and that
data would get purged from the serialization.

In order to avoid the reminder logic being duped by
channels simply being consistent, we now look up the
beginning of the latest continuous stretch of non-mutating
channel updates. If a channel's details have not been
altered in more than six days, we now send reminders no
matter the frequency with which channel updates have been
received since.

6 months agoTest that channels with non-changing updates result in missing reminders.
Arik Sosman [Wed, 8 May 2024 08:37:23 +0000 (01:37 -0700)]
Test that channels with non-changing updates result in missing reminders.

6 months agoDon't panic while panicking.
Arik Sosman [Wed, 24 Apr 2024 23:51:57 +0000 (16:51 -0700)]
Don't panic while panicking.

6 months agoStop tracking Cargo.lock
Arik Sosman [Wed, 8 May 2024 08:03:46 +0000 (01:03 -0700)]
Stop tracking Cargo.lock

7 months agoMerge pull request #74 from TheBlueMatt/main
Arik [Sun, 24 Mar 2024 01:01:04 +0000 (18:01 -0700)]
Merge pull request #74 from TheBlueMatt/main

Bump LDK to 0.0.121/rust-bitcoin 0.30

7 months agoFix unused `Result` in tests
Matt Corallo [Sat, 23 Mar 2024 21:24:38 +0000 (21:24 +0000)]
Fix unused `Result` in tests

7 months agoFix indentation in `test_ln_peers`
Matt Corallo [Sat, 23 Mar 2024 21:24:29 +0000 (21:24 +0000)]
Fix indentation in `test_ln_peers`

7 months agoBump LDK to 0.0.121/rust-bitcoin 0.30, bumping MSRV to 1.63
Matt Corallo [Wed, 13 Mar 2024 14:28:26 +0000 (14:28 +0000)]
Bump LDK to 0.0.121/rust-bitcoin 0.30, bumping MSRV to 1.63

Co-authored-by: Arik Sosman <git@arik.io>
7 months agoMerge pull request #73 from arik-so/arik/2024-03-tokio-runtime-split
Matt Corallo [Fri, 22 Mar 2024 22:44:41 +0000 (22:44 +0000)]
Merge pull request #73 from arik-so/arik/2024-03-tokio-runtime-split

Use separate Tokio runtime for gossip persistence.

7 months agoSimplify client creation.
Arik Sosman [Thu, 21 Mar 2024 00:41:34 +0000 (17:41 -0700)]
Simplify client creation.

7 months agoUse separate tokio runtime for gossip persistence.
Arik Sosman [Tue, 12 Mar 2024 06:41:27 +0000 (23:41 -0700)]
Use separate tokio runtime for gossip persistence.

7 months agoFix CI compilation in beta environment.
Arik Sosman [Thu, 21 Mar 2024 00:45:18 +0000 (17:45 -0700)]
Fix CI compilation in beta environment.

9 months agoMerge pull request #71 from TheBlueMatt/main
Arik [Tue, 6 Feb 2024 19:42:10 +0000 (11:42 -0800)]
Merge pull request #71 from TheBlueMatt/main

Do DB insertions in parallel

9 months agoMerge pull request #72 from lightningdevkit/arik/2024/01/bitcoin-rest-error
Matt Corallo [Thu, 1 Feb 2024 17:39:10 +0000 (17:39 +0000)]
Merge pull request #72 from lightningdevkit/arik/2024/01/bitcoin-rest-error

Improve bitcoind RPC/Rest error message

9 months agoImprove bitcoind RPC/Rest error message.
Arik Sosman [Tue, 30 Jan 2024 19:27:22 +0000 (11:27 -0800)]
Improve bitcoind RPC/Rest error message.

9 months agoDo DB insertions in parallel
Matt Corallo [Mon, 29 Jan 2024 17:24:32 +0000 (17:24 +0000)]
Do DB insertions in parallel

When inserting new gossip into the DB, we block the LDK peer
handling if we get behind. This is mostly okay, but can cause ping
timeouts and reconnections, which isn't ideal. To limit how often
we should see this, here we move to doing the new gossip insertions
in parallel.

9 months agoMerge pull request #70 from domZippilli/2024/01/load-peers-tolerating-trailing-comma
Matt Corallo [Mon, 29 Jan 2024 17:27:14 +0000 (17:27 +0000)]
Merge pull request #70 from domZippilli/2024/01/load-peers-tolerating-trailing-comma

Ignore trailing commas when parsing peers, and log more on error.

9 months agoIgnore trailing commas when parsing peers, and log more on error.
Dominic Zippilli [Thu, 25 Jan 2024 06:02:54 +0000 (22:02 -0800)]
Ignore trailing commas when parsing peers, and log more on error.

10 months agoMerge pull request #68 from johncantrell97/118
Matt Corallo [Wed, 10 Jan 2024 17:38:14 +0000 (17:38 +0000)]
Merge pull request #68 from johncantrell97/118

bump to ldk 118

11 months agobump to ldk 118
John Cantrell [Tue, 5 Dec 2023 21:42:47 +0000 (16:42 -0500)]
bump to ldk 118

12 months agoMerge pull request #65 from arik-so/announcement-history-tests
Matt Corallo [Sat, 11 Nov 2023 17:27:16 +0000 (17:27 +0000)]
Merge pull request #65 from arik-so/announcement-history-tests

Update-recency-based inclusion logic tests

12 months agoTest announcement inclusion logic.
Arik Sosman [Sat, 4 Nov 2023 04:43:33 +0000 (21:43 -0700)]
Test announcement inclusion logic.

12 months agoInclude old updates when necessary.
Arik Sosman [Sat, 4 Nov 2023 06:05:26 +0000 (23:05 -0700)]
Include old updates when necessary.

When a channel has only recently become bidirectional,
but there has not been a new update in the old direction
since the last sync, the latest update in the old direction
must still be included in full because it is the first time
the full channel is being snapshotted.

12 months agoAllow seen override for announcements.
Arik Sosman [Sat, 4 Nov 2023 03:58:21 +0000 (20:58 -0700)]
Allow seen override for announcements.

12 months agoMerge pull request #63 from arik-so/2023/10/ordering_fix
Matt Corallo [Thu, 19 Oct 2023 19:50:38 +0000 (19:50 +0000)]
Merge pull request #63 from arik-so/2023/10/ordering_fix

Fix update recency issue with expanded test coverage

12 months agoOptimize indices for expired value fix lookup.
Arik Sosman [Thu, 19 Oct 2023 18:52:32 +0000 (11:52 -0700)]
Optimize indices for expired value fix lookup.

12 months agoFix expired value being sent as the latest update.
Arik Sosman [Thu, 19 Oct 2023 18:52:06 +0000 (11:52 -0700)]
Fix expired value being sent as the latest update.

12 months agoReproduce test for expired value being sent as the latest update.
Arik Sosman [Thu, 19 Oct 2023 18:49:27 +0000 (11:49 -0700)]
Reproduce test for expired value being sent as the latest update.

12 months agoCreate snapshot generation test.
Arik Sosman [Wed, 18 Oct 2023 00:37:55 +0000 (17:37 -0700)]
Create snapshot generation test.

12 months agoExtract snapshot generation method.
Arik Sosman [Tue, 17 Oct 2023 20:49:48 +0000 (13:49 -0700)]
Extract snapshot generation method.

12 months agoReplicate Mutiny test scenarios.
Arik Sosman [Mon, 16 Oct 2023 15:27:45 +0000 (08:27 -0700)]
Replicate Mutiny test scenarios.

12 months agoAllow setting custom gossip seen timestamps.
Arik Sosman [Sat, 14 Oct 2023 01:39:27 +0000 (18:39 -0700)]
Allow setting custom gossip seen timestamps.

12 months agoMerge pull request #62 from arik-so/2023/10/ordering_fix
Matt Corallo [Sun, 15 Oct 2023 19:54:49 +0000 (19:54 +0000)]
Merge pull request #62 from arik-so/2023/10/ordering_fix

Create Postgres step in CI.

12 months agoAssert that schema is uninitialized.
Arik Sosman [Sat, 14 Oct 2023 01:01:21 +0000 (18:01 -0700)]
Assert that schema is uninitialized.

12 months agoCreate Postgres step in CI.
Arik Sosman [Fri, 13 Oct 2023 19:54:34 +0000 (12:54 -0700)]
Create Postgres step in CI.

13 months agoMerge pull request #60 from arik-so/2023/08/first_unit_test
Matt Corallo [Thu, 12 Oct 2023 17:35:33 +0000 (17:35 +0000)]
Merge pull request #60 from arik-so/2023/08/first_unit_test

First unit test

13 months agoAssert schema sanitization.
Arik Sosman [Tue, 12 Sep 2023 19:25:09 +0000 (12:25 -0700)]
Assert schema sanitization.

13 months agoUse schema as test logger id.
Arik Sosman [Fri, 8 Sep 2023 17:47:11 +0000 (10:47 -0700)]
Use schema as test logger id.

13 months agoCreate first unit test.
Arik Sosman [Tue, 15 Aug 2023 21:39:12 +0000 (14:39 -0700)]
Create first unit test.

13 months agoUpgrade to LDK 117.
Arik Sosman [Tue, 15 Aug 2023 21:38:46 +0000 (14:38 -0700)]
Upgrade to LDK 117.

14 months agoCreate TestLogger.
Arik Sosman [Tue, 15 Aug 2023 21:38:17 +0000 (14:38 -0700)]
Create TestLogger.

14 months agoCreate test-aware db env vars.
Arik Sosman [Tue, 15 Aug 2023 21:37:25 +0000 (14:37 -0700)]
Create test-aware db env vars.

14 months agoMerge pull request #56 from arik-so/2023/08/connect-to-peers-asynchronously
Matt Corallo [Wed, 6 Sep 2023 23:51:06 +0000 (23:51 +0000)]
Merge pull request #56 from arik-so/2023/08/connect-to-peers-asynchronously

Track gossip data prior to peer connection loop.

14 months agoAlways wait 10 seconds prior to attempting reconnection.
Arik Sosman [Wed, 6 Sep 2023 23:26:59 +0000 (16:26 -0700)]
Always wait 10 seconds prior to attempting reconnection.

14 months agoDon't special-case initial connection attempt.
Arik Sosman [Wed, 6 Sep 2023 04:28:15 +0000 (21:28 -0700)]
Don't special-case initial connection attempt.

14 months agoConnect peers independently.
Arik Sosman [Mon, 28 Aug 2023 21:40:35 +0000 (14:40 -0700)]
Connect peers independently.

Previously, we wouldn't start monitoring the gossip data until all
intended peers had either failed or succeeded to connect.

Instead, we're now spawn each peer connection independently, and only
await the first 5 successful peer connections.

14 months agoMerge pull request #57 from arik-so/2023/08/old-incremental-update-fix
Matt Corallo [Wed, 6 Sep 2023 22:16:26 +0000 (22:16 +0000)]
Merge pull request #57 from arik-so/2023/08/old-incremental-update-fix

Send full updates after old last seen updates.

14 months agoSend full updates after old last seen updates.
Arik Sosman [Tue, 29 Aug 2023 01:01:03 +0000 (18:01 -0700)]
Send full updates after old last seen updates.

Previously, whenever we saw that there was a previous update that a
client would have seen, we simply calculated the delta set based on
which properties have changed, and would most likely send an
incremental update set (excepting the case of a new or newly sent
announcement, in which case all sent updates are full).

However, if the last seen update was old, and there's a chance that
a user may have run RGS since, it is possible that due to the
7-day-backdating-mechanism included on the client, the reference
update would no longer be present.

To fix that, anytime we see that a last seen update is more than six
days old, we automatically include a full update.

14 months agoMerge pull request #59 from arik-so/2023/08/custom-intervals
Matt Corallo [Sat, 2 Sep 2023 16:27:08 +0000 (16:27 +0000)]
Merge pull request #59 from arik-so/2023/08/custom-intervals

Distinction between snapshot and symlink intervals

14 months agoDynamically adjust snapshot scopes.
Arik Sosman [Fri, 1 Sep 2023 21:42:40 +0000 (14:42 -0700)]
Dynamically adjust snapshot scopes.

Previously, we had hard-coded factors for the default snapshot
generation interval, which also served as the minimum snapshot
scope. In this commit, we substitute that with a doubling
mechanism that stops once it reaches or exceeds the
21-day-mark, which can be configured using an additional flag.

14 months agoDistinguish between snapshot and symlink intervals.
Arik Sosman [Thu, 31 Aug 2023 22:59:32 +0000 (15:59 -0700)]
Distinguish between snapshot and symlink intervals.

Snapshots may be generated every day or even every couple days,
but symlinks will be generated at the given symlink granularity.

To avoid gcd complications, all snapshot generation intervals
must be multiples of the symlink granularity interval.

14 months agoAllow the snapshot interval to be configurable in the environment variables
Graham Krizek [Thu, 6 Jul 2023 05:31:24 +0000 (00:31 -0500)]
Allow the snapshot interval to be configurable in the environment variables

14 months agoMerge pull request #55 from arik-so/2023/08/multiplication-overflow-fix
Matt Corallo [Mon, 28 Aug 2023 21:20:51 +0000 (21:20 +0000)]
Merge pull request #55 from arik-so/2023/08/multiplication-overflow-fix

Fix multiplication overflow bug

14 months agoFix multiplication overflow bug.
Arik Sosman [Mon, 28 Aug 2023 16:07:19 +0000 (09:07 -0700)]
Fix multiplication overflow bug.

The `snapshot_sync_day_factors` array is sorted
ascendingly, so find() will return on the first
iteration that is at least equal to the requested
interval.

However, the last value in the array is u64::max,
which means that multiplying it with DAY_SECONDS
will overflow. To avoid that, we use saturating_mul.

14 months agoMerge pull request #54 from arik-so/2023/08/incremental-update-fix
Matt Corallo [Thu, 17 Aug 2023 20:55:48 +0000 (20:55 +0000)]
Merge pull request #54 from arik-so/2023/08/incremental-update-fix

Only send full updates with announcements

14 months agoCreate indices for schema version 12.
Arik Sosman [Thu, 17 Aug 2023 20:14:18 +0000 (13:14 -0700)]
Create indices for schema version 12.

14 months agoCreate method for obtaining UTC-prepared Postgres client.
Arik Sosman [Thu, 17 Aug 2023 04:35:55 +0000 (21:35 -0700)]
Create method for obtaining UTC-prepared Postgres client.

14 months agoSend full updates whenever the channel announcement is sent.
Arik Sosman [Thu, 17 Aug 2023 04:50:43 +0000 (21:50 -0700)]
Send full updates whenever the channel announcement is sent.

14 months agoFix ordering of intermediate commits to ensure the latest comes first and is used...
Arik Sosman [Thu, 17 Aug 2023 03:55:51 +0000 (20:55 -0700)]
Fix ordering of intermediate commits to ensure the latest comes first and is used as the authoritative source of what's latest.

14 months agoMerge pull request #53 from arik-so/2023/08/query-timestamp-fix
Matt Corallo [Thu, 17 Aug 2023 00:16:14 +0000 (00:16 +0000)]
Merge pull request #53 from arik-so/2023/08/query-timestamp-fix

Switch timestamp queries to use f64

14 months agoSwitch timestamp reads to use i64.
Arik Sosman [Thu, 17 Aug 2023 00:04:06 +0000 (17:04 -0700)]
Switch timestamp reads to use i64.

14 months agoSwitch timestamp queries to use f64.
Arik Sosman [Wed, 16 Aug 2023 23:18:40 +0000 (16:18 -0700)]
Switch timestamp queries to use f64.

14 months agoMerge pull request #49 from TheBlueMatt/main
Matt Corallo [Sun, 13 Aug 2023 16:37:54 +0000 (16:37 +0000)]
Merge pull request #49 from TheBlueMatt/main

15 months agoMake snapshotter a bit more amenable to changes in interval
Matt Corallo [Sun, 16 Jul 2023 20:41:28 +0000 (20:41 +0000)]
Make snapshotter a bit more amenable to changes in interval

By separating the concept of "seconds in a day" from the snapshot
interval.

15 months agoOnly generate as many symlinks as we need (rather than 27 years worth)
Matt Corallo [Sun, 16 Jul 2023 20:41:00 +0000 (20:41 +0000)]
Only generate as many symlinks as we need (rather than 27 years worth)

15 months agoMerge pull request #51 from arik-so/2023/08/pre_test_refactors
Matt Corallo [Sun, 13 Aug 2023 05:30:42 +0000 (05:30 +0000)]
Merge pull request #51 from arik-so/2023/08/pre_test_refactors

Allow arbitrary logger types.

15 months agoAllow log threshold configuration.
Arik Sosman [Fri, 4 Aug 2023 21:30:02 +0000 (14:30 -0700)]
Allow log threshold configuration.

15 months agoRemove println from verifier.rs
Arik Sosman [Thu, 3 Aug 2023 01:34:01 +0000 (18:34 -0700)]
Remove println from verifier.rs

15 months agoRemove println from tracking.rs
Arik Sosman [Thu, 3 Aug 2023 01:26:55 +0000 (18:26 -0700)]
Remove println from tracking.rs