Monitoring light nodes is not straightforward. Read on to find out how ProbeLab does it!
Capturing nodes that interact with a Distributed Hash Table (DHT) network as clients has traditionally been challenging, but also not a critical metric for the overall health and operation of the network. With modern-day Data Availability (DA) solutions, such as Celestia, being based on a DHT network and in particular on DHT client nodes, monitoring and metrics for those nodes becomes crucial.
The ProbeLab team has adjusted its network crawling tools to monitor Celestia DHT Server nodes since April 2024 [see Celestia forum post], but has recently also developed a specialised tool, called ants-watch, to capture DHT client nodes (aka Light Nodes, or LNs). Our tool collects important data from Celestia Light Nodes which we analyse to develop critical metrics and insights for the DA network’s operation.
In this post, we initially share some overall metrics for DHT Server nodes, then give a brief overview of ants-watch together with results and finally, we highlight findings that we were able to uncover by observing the network through our specialised tools.
We use the Nebula crawler to gather information about DHT server nodes in the Celestia DHT network. The crawler runs every 2 hours, resulting in a total of 84 crawls per week. This results to around 70k visits per week, on average, with a visit being defined as “an attempt to dial or connect to a peer”. Below is a high-level summary of our findings. Further explanations on definitions can be found at: https://probelab.io/celestia/dht/2025-08/#general-information , which shows results for Week 08 of 2025 (17th to 24th Feb 2025).
Note that this data refers to Celestia’s DHT Server nodes and does not include DHT Client nodes, which we present further down in this post.
149162131The reports show the agent distribution throughout the duration of a week over time [link to plot for Week 08, 2025], which helps identify fluctuations in network node participation.
The reports include an updated churn plot (compared to the one shown in the earlier Celestia forum post), which reveals a very stable base of DHT Server nodes (agent versions: full and bridge nodes) with churn rates of well below 5%. As expected, the browser-based lumina nodes present higher churn rates, which can reach up to 50%. Check the corresponding section of the reports [link] for a detailed definition of how we measure churn.
There has been a remarkable shift in the number of nodes deployed in the US: the number has jumped from 6.5% in Week 15 of 2024 [link] to 31% in the most recent report. The rest of the nodes are geolocated as shown in the following graph [link].
Apart from the above metrics, ProbeLab’s Weekly Network Health reports for the Celestia DHT network include stats related to the protocols observed being supported by network nodes [link], the number of connection errors [link] and crawl errors [link] together with the error description.
DHT Server nodes can be easily captured by standard network crawler functionality, but the case is not the same for DHT Client nodes. This is because each DHT Server node is keeping track of other server nodes in the network by inserting them in their routing table. The routing table of a server node can be queried and will return the addresses of other server nodes in the network. DHT Client nodes, however, are not kept in any other node’s table because they are not expected to be stable, present and participating in the network continuously. This is where extra, specialised functionality is needed to capture DHT Client nodes, and in this case Celestia Light Nodes.
ants-watch is a tool developed by the ProbeLab team specifically for capturing DHT Client nodes. The tool captures client nodes by exploiting some of the DHT’s (in this case libp2p’s kad-dht) internal operational details. More specifically, each DHT Client node needs to keep its routing table up to date and for this reason, it queries the k = 20 closest DHT Server nodes to itself (i.e., to its own PeerID) every so often (every 10 min in Celestia’s case). Therefore, by inserting one controlled DHT Server node for every k = 20 DHT Server nodes, our tool is able to receive routing table update requests from DHT Client nodes (i.e., Celestia Light Nodes). When a request hits one of our ants, we record information about the requesting peer like its agent version, supported protocols, IP addresses, and others.
This is a simplified explanation of how ProbeLab’s ants-watch tool works. For more information, methodology and deployment details head to the tool’s GH repo: https://github.com/probe-lab/ants-watch.
Several metrics can be shown out of the data we gather from our ants deployment, but for now we’re publishing the agent distribution over time (first plot) [link] and the adoption of new releases over time (second plot) [link].
The ants that we have deployed on the Celestia network can capture DHT Client nodes (i.e., Light Nodes), but can also capture DHT Server nodes (i.e., full and bridge nodes). We’ve done a comparison between the numbers we get from our Nebula crawler and those from our ants deployment.
Interestingly, we found that we can see way more DHT Server nodes from the ants deployment, than from the Nebula crawler. In particular, we see almost the double as much server nodes from ants than from Nebula. This finding leads to the conclusion that quite a few DHT Server nodes in the Celestia network are facing connectivity issues, most likely due to NATs. In the comparative plot below we see between 80 and 90 nodes seen by Nebula (left plot) and more than 150 seen by the ants (right plot). The issue has been reported to the Celestia team, which is looking into it and has been documented in this GH issue.
Overall, Celestia’s P2P network is in a healthy state with nodes distributed geographically and among different cloud providers. The network is stable with server nodes churning very little. The novel browser-based Light Node implementation (i.e., https://lumina.rs/) allows for easier deployment, but is also more prone to higher churn. However, judging from the population of Light Nodes seen in the network, higher churn does not risk Data Availability.
The ProbeLab team will continue to monitor the Celestia network and report results at: https://probelab.io. Reports are published weekly on Mondays. If there are important metrics that you believe should be part of these weekly reports, or have questions regarding the existing ones, please get in touch with the team.
