I’m running Cisco Modeling labs, which uses Ubuntu and QEMU under the hood. CML allows you to place external connectors in a lab that bridge virtual and physical lab devices. These external connectors are Linux bridges running on the CML controller hypervisor.
In my case the controller is itself virtualized under ESXi and is connected to a distributed vSwitch, which is in turn connected to a physical switch and the rest of the lab.
Now, out of the box the connectors work fine. You connect a virtual node to it and it shows up on your physical network. Unfortunately, you can’t have a trunk running between the physical and virtual lab nodes because the path between them is itself a trunk.
After some research, I came across Q-in-Q, aka dot1q tunneling aka 802.1ad aka nested vlans. The physical infrastructure (also Cisco) supports Q-in-Q by entering switchport mode dot1q-tunnel in interface config mode. Then you assign it to an outer vlan with switchport access vlan <vlan-number>. All traffic entering that port, regardless of if it’s tagged or not, gets this outer vlan tag, and the tag already present becomes the inner tag. The frame is switched according to the outer tag. Once the frame reaches the other side of the tunnel, the outer tag is stripped and the destination device only sees the correct inner vlan tag.
While this is easy enough to set up in IOS, I’m having no luck doing this with a linux bridge. They do support 802.1ad, which you can indicate when creating the bridge. The big issue is that the interface on the bridge the virtual node is connected to is created on the fly when you connect the node to the external connector. I can’t set up anything on the port itself, it has to inherit any settings from the bridge, and I’m not sure how to do that. Further, the interface on the bridge that faces the rest of the network must be configured as a trunk in order to allow the outer-vlan tagged traffic to pass to the vSwitch, and I’m not sure how to do that either.
There is a vlan_default_pvid option when creating a bridge, but that doesn’t seem to do what I want. The desired behavior is as follows:
- a frame is generated by the virtual lab node, with or without tags, and enters the ephemeral bridge interface created by CML.
- regardless of the presence or absence of a vlan tag, the bridge pushes an outer tag onto the frame, possibly resulting in two vlan tags.
- The frame is switched across the vSwitch and physical infrastructure, which is completely agnostic to this inner vlan if present.
- Before egressing the final switch before the physical lab node, the outer tag is stripped, leaving only the inner tag for the lab node to see.
- The lab node processes the frame as though it were directly connected to the virtual node.
For traffic in reverse, the process is similar. Double-tagged traffic traverses the link between the distributed vSwitch and the linux bridge. The bridge strips the outer tag, then pushes the traffic with just the inner tag to the virtual lab node.
You must log in or register to comment.
I think I understand what you’re trying to do, but I’m having trouble parsing this sentence:
The big issue is that the interface on the bridge the virtual node is connected to is created on the fly when you connect the node to the external connector
The topology looks like this:
VIRTUAL | PHYSICAL -------------------------|------------------------- 1 2 3 4 5 [VN] --- [LB] --- [VS] --|-- [P1] --- [P2] --- [PN]Nodes: VN = virtual lab node LB = linux bridge (external connector) VS = VMWare vSwitch P1 = 1st physical switch P2 = 2nd physical switch PN = physical lab node
Links:
1: link between virtual lab node and linux bridge. Created and destroyed dynamically. I can’t control the settings of this link and connected interfaces. Virtual lab nodes are nested VMs running in QEMU.
2: Link between the linux bridge and vSwitch. The linux bridge resides on the CML controller node, which is a VM in ESXi. This link must be a trunk.
3 and 4: both trunks, one from vSwitch to 1st physical switch, the second between 1st physical switch and 2nd physical switch.
5: End of the dot1q tunnel. Outer tag is stripped and inner traffic (tagged or not) is passed to the second lab node
Ok, the diagram has been very useful for me to understand.
My thinking is that for the Linux bridge (LB), you’ll have to create it with “protocol 802.1ad”. So something like this:
ip link add name ${BRIDGE} type bridge vlan_default_pvid 420 vlan_protocol 802.1ad
I presume that there is no problem with manually creating the bridge? Also, compared to the example from the link, I’ve modified it to set the default port VID (PVID) to 420. This means that if any frame comes into LB and does not have an S-tag, then an S-tag with the port VID will be applied to the frame upon ingress.
Using the standard nomenclature of 802.1ad Q-in-Q, an S-tag (S for “service provider”) is the outermost tag with TPID=0x88a8, and a C-tag (C for “customer”) is the innermost tag, which would be a standard 802.1q VLAN tag with TPID=0x8100. That said, when I say “innermost” and “outermost”, that only applies to double-tagged traffic. It is entirely possible to single-tag using an S-tag, and that’s exactly what will happen if a VN sends an untagged frame to LB, as an S-tag with VID 420 will be added. But there wouldn’t be an inner C-tag.
If VN sends an 802.1q C-tagged frame to LB, then that’s still not an S-tag and so LB will also add an S-tag upon ingress, making this frame now doubly tagged.
I also omitted “vlan_filtering 1” because that would cause all traffic with an unknown S-tag VID to be dropped. Dropping such traffic is the default for a standard switch, but it may aid your troubleshooting if LB is configured as permissively as possible. At the very least, it means that unknown S-tags can at least be inspected using tcpdump upon ingress, rather than being dropped outright.
For link 1, there’s nothing we can do, but we also shouldn’t have to do anything. If VNs are attached to the LB as a subordinate port, then they would be exchanging either untagged or single C-tagged traffic. And that’s fine because that’s exactly what LB is prepared to handle.
For LB’s port connected to link 2, this must be configured as a trunk port for VID 420. This will cause LB to pass doubly tagged frames to VS, which is what we want. For link 2 facing VS, no special configuration for 802.1ad or 802.1q is necessary, since the frames will enter VS as if standard Ethernet; the TPID takes up the same position in the header as an EtherType, and standard switch behavior is to forward any unknown Ethertypes. If VS has any configurable options, any options that are VLAN-aware ought to be disabled.
For link 3 facing VS, this isn’t anything special to configure, as described above. But for the P1 port connecting to link 3, this must be configured as a trunk port for VID 420.
Between P1 and P2, this is also a trunk for VID 420.
On P2’s port connecting to PN, this will be an untagged 802.1ad port for VID 420, with the PVID set for 420. By being an untagged port, P2 will strip the S-tag from any frame egressing to PN, thus yielding either an untagged Ethernet frame or a singly C-tagged 802.1q frame. And when PN sends untagged or C-tagged traffic that ingresses P2, the S-tag for PVID 420 will be added.
Such ingress traffic will propagate from P1 as either S+C double tags or S-only single tag, all the way back to LB. But when proceeding to the subordinate port that connects to a VN, the LB will know to strip the S-tag, thus yielding either C-tagged or untagged traffic.
In this way, the whole network can be traversed by wrapping all of VN’s traffic with an S-tag, irrespective of any C-tags that traffic already had. You may have to play around with the exact mechanics of the Linux bridge to fit your circumstance, but this should yield a working setup.
Good luck!
ip link add name ${BRIDGE} type bridge vlan_default_pvid 420 vlan_protocol 802.1ad
I have been doing this, but what I’ve also been setting vlan_filtering to 1, so I think that’s what’s keeping this from working. Clarifying the terminology (s tags and c tags) should also help in future troubleshooting. Thanks.