Monitor the network

vNet has three sources of information: the conductor’s state dump, the switches’ counters, and the logs. None needs extra software.

The state dump

The conductor writes its view of the network every 60 seconds (VNET_DUMP_SEC) and on SIGUSR1, to /var/vnet-dump.bin (VNET_DUMP_PATH). Read it with vnet-state; given the auth file, it also lists the machines that should be there and are not:

$ sudo systemctl kill -s USR1 vnet-conductor
$ sudo vnet-state /var/vnet-dump.bin /etc/vnet/auth.txt

The output is described in During an event and Programs, signals and services. vnet-state reads a copied file as well, so the dump can be watched from another machine.

The switch counters

Every switch prints one statistics line every 60 seconds (VNET_STATS_SEC; 0 turns it off), and on SIGUSR1 the same line followed by one line per machine:

$ sudo systemctl kill -s USR1 vnet-switch
$ journalctl -u vnet-switch -n 5
[Switch] Stats: agents=1 routes=1 local=16 to-switch=16 to-internet=2 waited-for-route=8 dropped=6 acl=6 link-full-drops=0 tun-write-errors=0 peers=1 pending-tokens=0 conductor=up
[Switch] Agents:
agent1 10.0.1.1 fd=7 in=30/2520B out=16/1344B dropped=4

The counters count since the switch started.

Field	Meaning
agents	machines attached to this switch now
routes	machines on other switches this switch knows how to reach
local	packets delivered to a machine of this switch
to-switch	packets sent to another switch
to-internet	packets sent to the internet
waited-for-route	packets held because the destination’s route was not known yet (The data plane)
dropped	packets dropped, all reasons together; then one field per reason that occurred: malformed, unknown-source, spoof, rate-limited, acl, unreachable
link-full-drops	packets dropped because the link to their destination was full
tun-write-errors	packets that could not be written to the internet interface
peers	links to other switches that are up or being dialed
pending-tokens	joins announced by the conductor that have not completed yet
conductor	up or down: the link to the conductor

And per machine:

Field	Meaning
name, address	the machine and its virtual address
fd	the socket of its link, as in the Connection closed lines
in	packets/bytes received from the machine
out	packets/bytes delivered to the machine
dropped	packets from this machine that were dropped

What healthy looks like

• conductor=up on every switch, and peers equal to the number of other switches that hold machines (a switch links to another one when it first learns a route through it).

• routes on each switch equal to the number of machines on the other switches (vnet-state gives both).

• pending-tokens back to 0 a few seconds after machines join.

• acl drops growing slowly: machines trying what the rules forbid (software updates, telemetry, a contestant exploring). A sudden rise from one machine is worth a look.

• No spoof, unknown-source or malformed: these mean a machine sends packets it should not be able to make, or a broken agent.

• rate-limited only on machines you expect to be busy.

• link-full-drops and tun-write-errors at 0. A growing value means a link or the uplink cannot keep up.

To watch the counters over time, collect the [Switch] Stats: lines from the journal of every switch; each line is self-contained.

Debug output

VNET_DEBUG=1 on a switch logs every dropped packet with its reason, every packet held for a route, and every write that failed. It is meant for a lab: on a busy switch it writes a line per packet. Set it in /etc/vnet/switch.env and restart the switch.