Talking on Slack with various folks, and looking briefly thru the Erlang p2p code, the hotspot code tries pretty hard to traverse NAT in various ways.
My goal in asking for this is to have a detailed document describing the P2P network activity of hotspots, how they deal with various kinds of firewalls and NAT so that an technical user can make informed tradeoffs to how they want to setup their network access for the Hotspot. This isn’t going to be needed for 99% of hotspot owners, but for the few of us that are network nerds, knowing the side effects of not letting any incoming, vs letting the hotspot do NAT-PMP with renewing PMP leases vs allowing allowing a port forward would be really really useful. Small details matter, like how the hotspot picks incoming ports in various situations, how other hotspots know how a given hotspot, how it reacts to changing ports, etc.
Giving some more background:
I’ve got a fairly complex network setup at home and I currently have two hotspots there. I’m going to be either using directional antennas to focus coverage or move one of the hotspots entirely. In the meantime, it’s really helpful to run both on my home network at the same time at least while I get new hotspots synced and proven.
Right now, it is really difficult to understand and debug the side effects (to syncing as well as the normal operation of the hotspot) of these choices (there may be others, this is off the top of my head):
allow no incoming connections
Let hotspot do uPnP or NAT-PMP, if so, what ranges do I allow it to have access to? How does it deal with NAT-PMP leases expiring and changing?
port forward 44158 for one hotspot and a different port for the other hotspot
Effect of the NAT being symmetric, aka NAT’ing port as well vs not.
Effect of two hotspots behind the same IP, even on different ports external ports.
I realize I’m asking for a lot here and it’ll be useful for maybe only a hotspot owners, but those owners may make great evangelists.
Thanks, Krby. You’re certainly asking for a lot of info here but we don’t explain most of this very well at the moment. Let me get with the team and see how we can approach this. I’ll keep you posted.
Hi, author of most of the NAT madness in libp2p here.
libp2p tries really hard to work in just about any networking environment, and I think it’s been largely successful. We assume the worst case (symmetric NAT or a firewall blocking all inbound connections) and then try to take advantage of any mitigating factors (static NAT, PMP/uPNP, a configured port mapping, etc) to improve the circumstances.
I’ll try to detail the cases you mention below:
allow no incoming connections
If libp2p can connect outbound, it will still be able to connect to the network and obtain gossiped peers and blocks. Additionally a relay address will get obtained to allow other peers to connect via existing connections. Relay addresses are not the most stable thing right now, so this can affect cases where reliable connectivity is important (challenges and consensus group membership). This is a very common configuration in the network and the one we tend to assume people have.
Let hotspot do uPnP or NAT-PMP, if so, what ranges do I allow it to have access to? How does it deal with NAT-PMP leases expiring and changing?
It will just try to use any port that it is allocated, the port range doesn’t matter.
port forward 44158 for one hotspot and a different port for the other hotspot
This is totally fine, the external port doesn’t matter. libp2p will figure it out and add it to its peer entry.
Effect of the NAT being symmetric, aka NAT’ing port as well vs not.
Truly symmetric NATs are a pain, but we do detect them and work around them. It’s equivalent to the ‘block all inbound connections’ case above.
Effect of two hotspots behind the same IP, even on different ports external ports.
This should work fine, they’ll both discover their NAT situation and any working external ports independently.
There’s still more work to be done (ipv6 support, reworking relays to be less fragile, etc) but overall the goal is that you should be able to get connected to the network in almost any network environment. Firewalled outbound connections (we run some nodes in AWS on port 443 to try to dodge this by pretending to be HTTPS but deep packet inspection won’t be fooled) and captive portal networks (think hotel wifi where you have to time your name/room number into a page before you can browse the internet) are the main cases you’'ll have problems. Adding a websocket transport might help with the former (depending on how draconian the firewall is) but the latter is not something we’re planning on solving any time soon.
Thanks for the detail here, that’s really good info to have.
In the case where it tries to figure out what ports have been statically forwarded to it, can you give more info on that? Does it ask something to probe it? If so, does it check incoming ports randomly, or in some sort of sequence? If not, how does it figure this out?
I have 2 hotspots behind the same IP, I’ve forwarded 44158 to one of them, but would like to forward a 2nd port to the other. If it does something simple such as check 44158, 44159, 44160… then I’ll just setup forwards accordingly.
Thanks for the writeup @Vagabond, it’s great! Some followup questions:
I think I’m missing some backgroud on how the p2p network works. Is there an up-to-date paper or page I can read? I don’t know what a relay address or gossip is. I can kind of infer based on the words, but knowing for sure is important for understanding.
What is driving this is that I was trying to troubleshoot a two-hotspot at home setup and it is very difficult to know whether the inbound and outbound being off in the diag report was something I did, or a temporary problem in the p2p network. When trying things like a static 44159 redirect being available to hotspot #2, it felt like it had more “needs attention” than hotspot #1. When trying to let them both NAT-PMP, with the router, I had more situations where they both complained of being offline and it wasn’t clear if I could improve things or I just needed to wait it out.
So, libp2p checks for what it calls “observed” addresses. These are the addresses that the peers it is connected to report as the address/port they see the hotspot has. If there’s enough agreement on the observed addresses, then the hotspot will advertise that IP/Port combination as its address that other peers can connect to.
If, instead, there’s no agreement, or we can’t get any inbound connections, we try to determine which flavor of NAT we are behind and try to obtain a relay address.
This means that you can forward any external port through to a hotspot and it will figure out what the external IP/port is. We use a consistent local port only for the purposes of making it easy to set up port forwards. Mapping to that “blessed” port 44158 is only otherwise helpful for other people who have whitelisted their outbound connections to only hosts on that port but since most hosts won’t actually be using that port on the outside, this will likely result in sub-par connectivity options for the firewalled node.
Right, so we are (somewhat loosely) based on the libp2p spec here: GitHub - libp2p/specs: Technical specifications for the libp2p networking stack . Unfortunately, as you might notice, the spec is quite incomplete. At the time we started our implementation the specs were even more incomplete, so we had to fill in a lot of the blanks ourselves.
Gossip is the mechanism by which updates propagate around the network. The primary pieces of data we gossip around are 1) peerbook entries and 2) new blocks.
Peerbook entries consist of the address(es) a node is reachable at, some metadata about the peer (block height among them) and the set of peers its connected to. This is how other nodes know how to connect to your node (and vice versa). Peers periodically emit new versions of their peerbook entry as they change or become stale.
There are some special nodes called ‘seed nodes’ that are hard-coded into the firmware that the node goes to to initially obtain a list of peers.
When a node sees a “new” block (ie. one that it didn’t have before, but has just successfully added to the chain) it also gossips those out to all its peers. This is how new blocks propagate out through the network.
So the “needs attention” thing is a bit of a “best effort” thing on our part. There’s some fairly coarse metrics that we use (the block height the node last challenged, the age of the peerbook entry, the block height in the peer’s metadata) and because the number of nodes on the network has gotten much higher, the passive gossip of peer book entries has slowed. This can push the age of the peerbook the API server sees beyond the thresholds we set up for inactivity and trigger a spurious ‘needs attention’ flag.
I believe the diagnostic report obtainable over bluetooth will be more accurate about the actual connectivity, but we are working to improve the API server’s view of the peerbook so it can try to refresh hotspot peerbook entries a bit more aggressively.
I assume you mean you can automatically forward any port with a traversal service like UPNP. A hotspot wouldn’t know about a manual port forward on any port other than 44158 without a portscan.
Say you have two hotspots sharing a single IP, but the only forwarding option is static forwards. If you forward port 44158 to hotspot A, will hotspot B with no forwards get a false positive on incoming connectivity?
I had a lot of trouble syncing a hotspot in this case. Consider that hotspot B will reach out and ask if it is connectable on 44158 and someone outside will connect back to see port 44158 is open and a hotspot responded, but it’s actually hotspot A.
In any case, it sure would be nice to be able to manually select a port.
I assume you mean you can automatically forward any port with a traversal service like UPNP. A hotspot wouldn’t know about a manual port forward on any port other than 44158 without a portscan.
Ah yes, you’re right in this case. We actually have a configuration variable called a ‘nat map’ that allows you to map internal to external ports for this case, but it’s not exposed over the GATT yet.
Say you have two hotspots sharing a single IP, but the only forwarding option is static forwards. If you forward port 44158 to hotspot A, will hotspot B with no forwards get a false positive on incoming connectivity?
No, this is not how it works, 44158 is not special at all from the node’s perspective (we used to bind to port 0 so the operating system would assign one randomly). If the external address/port the node’s peers report is consistent (eg. a bunch of peers report they see the same external address/port) the node will try to solicit connections on that address (from peers not currently connected to us). If those connections work, we consider it a working external address.
I had a lot of trouble syncing a hotspot in this case. Consider that hotspot B will reach out and ask if it is connectable on 44158 and someone outside will connect back to see port 44158 is open and a hotspot responded, but it’s actually hotspot A.
This, at least, should not happen because the node won’t be able to use those inbound connections to validate that port being open to itself.
In any case, it sure would be nice to be able to manually select a port.
