February 25, 2025

Introducing Network Health Weekly Reports for the Avail LC DHT

We are looking into the internals of Avail's LC DHT. Read on to find out important details.

The ProbeLab team is producing Network Health Weekly Reports for Avail’s Light Client (LC) DHT network as of January 2025. The reports are published at: https://probelab.io and the latest one, for Week 07, 2025 can be found here: https://probelab.io/avail/lc/mainnet/2025-07/ The ProbeLab team has taken up the task of identifying the most important metrics for the P2P layer of Avail’s LC DHT network and adapted its suite of tools to monitor the network.

In this post, we give an overview of the most important metrics and critical findings from the past month of publishing weekly reports and investigating Avail’s Data Availability P2P network.

General Setup and Network Information

We use the Nebula crawler to gather information about DHT server nodes in the Avail LC DHT P2P network. Our crawler runs every 2 hours and it takes around 3 mins to complete a crawl, resulting in a total of 84 crawls per week. This results to around 300k visits, on average, with a visit being defined as “an attempt to dial or connect to a peer”. Below is a high-level summary of our findings. Further explanations on definitions can be found at: https://probelab.io/avail/lc/mainnet/2025-07/#general-information, which shows results for Week 07 of 2025.

Note that the data and results presented in the reports and this blogpost refer to the LC DHT Server nodes and does not include LC DHT Client nodes, which are not visible through a crawler.

  • Number of unique peer IDs discovered in the DHT: 574
  • Number of unique IP addresses discovered: 10
  • Network Size: 553

We generally see a clear decline in the number of nodes participating in the network, compared to Week 4 of 2025, when numbers looked as follows: https://probelab.io/avail/lc/mainnet/2025-04/:

  • Number of unique peer IDs discovered in the DHT: 815
  • Number of unique IP addresses discovered: 22
  • Network Size: 762

This decline, however, does not translate into less actual nodes being simultaneously online, which stays stable at 400 throughout the last four weeks.

Observations and Concerns

Some other observations from these numbers are:

  • There are way less unique IP addresses seen than unique PeerIDs. For Week 07, 2025, we see 10 unique IP addresses and 574 unique Peer IDs.
  • The small number of IP addresses is due to the fact that, as noted earlier, ProbeLab’s crawler only captures LC DHT Server nodes and not LC DHT Client nodes. The Avail team operates 1 LC DHT Server node for every 20 LC DHT Client nodes (aka Light Clients). The ratio of 1:20 has been fixed according to libp2p’s routing table update settings.
  • The large discrepancy between unique IP addresses and Peer IDs means that either those 10 nodes rotate their PeerIDs from the same 10 physical machines, or that there are several light clients ran per machine/IP address, or both.
  • The number of connection errors (here), which is hovering around 150, justifies the difference between the 553 peers shown as "Network Size" at the top of the report and the 400 peers seen in the rest of the plots.

Although the increased number of errors doesn’t seem to be critical or causing any other issues to the network, it still consumes resources and increases RTTs as nodes attempt to discover peers or connect to unreachable addresses.

One of the possible effects of these "unreachable" nodes is a reduction in the DHT’s overall performance. With ~27% of the DHT servers marked as 'unreachable,' cell availability may decline over time. Public DHTs rely on a replication factor to adapt their routing to the network's node-churn dynamics. If this replication factor is reduced early on, it could interfere with or delay the retrieval of DA cells. Furthermore, since fewer nodes store and serve the data in the network, it can induce some overload on nodes that are providing popular DA calls, potentially leading to some congestion in the network.

The ProbeLab team has reported these observations in this GH issue.

Network Size & Stability

The reports show the agent distribution throughout the duration of a week over time [link to plot for Week 07, 2025]. The plot helps identify fluctuations in network node participation in the network. For instance, we saw in Week 06, 2025 a dip in the number of nodes, in particular on 2025-02-07 (shown below). The dip was due to a simultaneous re-deployment and restart of all nodes in the network. However, the network has recovered smoothly without impacting the DA operation.

One critical metric that relates to and affects network stability in a P2P network is the concept of node churn, i.e., how often do nodes leave and rejoin the network. ProbeLab’s weekly reports show the node churn in terms of the percentage of nodes that are expected to leave the network in the next 24 hours, based on the past week’s statistics.

The graph below shows, from Week 06, 2025, shows remarkable stability with less than ~0.005% of nodes expected to leave the network in the following 24 hrs.

Node Geolocation & Deployment

Nodes in the Avail LC DHT network seem to be primarily deployed in the Netherlands (87%, or 350 nodes) and the US (13%, or 50 nodes). All of the nodes are deployed in a single public public infrastructure provider (DME Hosting) [link].

DHT Key Space Density

Monitoring the DHT keyspace density is an important measure against Eclipse attacks. It’s not a mitigation technique, but rather a pro-active detection technique. ProbeLab’s specialised tooling is scanning the DHT keyspace and produces alerts when the density over some particular part of the keyspace goes above some threshold. Read the methodology and details at: https://probelab.io/avail/lc/keydensity/.

Based on our latest results, the Avail LC DHT key space is in a healthy state.

Misc

ProbeLab’s Weekly Network Health reports for the Avail LC DHT network are also presenting stats related to the protocols observed being supported by network nodes [link], the number of connection errors [link] and crawl errors [link] together with the error description.

Conclusions

The Avail Light Client DHT network presents a healthy state with nodes being stable overall with very little churn. The network could be more decentralised in terms of geographic diversity, as well as cloud infra node deployment. The number of no_ip_address connection errors remain concerning, although not creating an immediately visible impact to the network. Deeper investigation is needed in order to assess if and when these errors become a problem.

The links and plots provided in this post are from Weeks 06 and 07 of 2025. Some earlier reports can be found at: https://probelab.io. Reports are published weekly on Mondays at https://probelab.io, so make sure to follow along, or subscribe to receive updates: https://www.probelab.network/contact.

If you have metrics in mind that you would like to see included in the weekly reports, or ways in which our methodology can be improved, please reach out to us via the details in the above link.

Recent blog