A common description

It’s easy to find pieces like these:

  • Web-scale NOS with traditional L2/L3 functions and management, with an API structure for third-party apps and integration with provisioning and orchestration systems.

  • Broadcom ICOS/FASTPATH is an off-the-shelf (Linux based) network operating system (NOS), providing traditional L2 and L3 functions and management, with an API-structure for value-added applications and integration with provisioning and orchestration systems.

  • ICOS is a robust, programmable full featured and most widely deployed networking operating system for enterprise and data center networking. Its built-in arbitration between conventional and SDN flows allowing customers seamless integration to Next-Gen Open programmable networks.

  • FASTPATH production-ready networking software helps manufacturers achieve record-setting time-to-market performance for introducing new Ethernet products. In part, this advantage comes from the extensive feature set and integration capabilities that give Broadcom’s software unprecedented application flexibility.

A general praise to its wholesome, next-gen design, apps and APIs. Typical marketing stuff that gives no real understanding of the product.

What Is ICOS?

As we mentioned in our previous article, a Network OS (NOS) has many important functions in managing switch hardware.

In short, it looks like this:

NOS architecture

A more detailed view may looks like this:

NOS architecture in details

On all images, all these layers of software are separated from the hardware because of one major reason, and it’s not the importance of disaggregation. Every networking device looks like that from the inside, even the most monolithic boxes from big vendors.

An ASIC that does all the heavy load in a switch does not ask the control plane every time it receives a packet. It follows the rules set by the control plane. And the control plane is responsible for rules to be following standards. Protocols.

A control plane implements the logic behind protocols and pushes it to the HW.

ICOS/FASTPATH is a collection of protocol stacks that can be easily implemented on silicon from Broadcom. And it’s a big one. One can find almost anything they need to make a switch for any market:

  • Switching

  • Stacking

  • Routing

  • IPv6 Routing

  • Management

  • Quality of Service

  • Multicast

  • BGP4

  • Metro

  • Data Center

  • Service Provider

Every module contains protocols, standards, and MIBs that are ready to use. One can get a whole stack or in pieces.

In details it looks like this:

L2 protocol stack
 L2 stack

A bunch of submodules you may use or not - only a few items are mandatory. That’s why comparison of L2 switches is such a mess.

In the system description it will look like:

Additional Packages............................
QOS
Multicast
Stacking
Routing

Who is using it?

Many companies. Even if they don’t mention it.

HW vendors (just names we know):

  • Dell in its PowerConnect line

  • Ubiquiti

  • Netgear
  • Quanta for its QNOS

  • Edge-Core

  • Netapp (yes!) for its CN1610 switch

  • Lenovo/IBM for its Flex System switches (maybe for other as well, no docs available)
  • Penguin Computing

  • A countless battalion of L2/L3-light switch vendors

Cloud mega-scalers - we are sure of Ali and Baidu.

You?

Industry-standard CLI

AKA Cisco-like CLI.

Yes, it does embrace very similar syntax. The learning curve needed to master the environment is much easier with that. It makes network professionals who spent ages working on Cisco devices feel more comfortable in case of transition. And the amount of training material for the Cisco CLI is immense. Moving to a completely different CLI concept is much harder (like on Juniper devices or inventing your own structure).

You can access the CLI by using a direct serial connection or by using a remote logical connection with telnet or SSH. In case of ICOS running as a service, you can get in via Linux shell commands.

The CLI group commands into modes according to the command function. Each of the command modes supports specific software commands. The commands in one mode are not available until you switch to that particular mode, with the exception of the User EXEC mode commands.

Autocompletion, shortcuts, scripting, you name it.

Over a thousand pages of available commands to manage it all.

Linux shell capabilities

We have a piece about ICOS and Linux Shell Management, but it about the idea and main difference with the embedded version.

Getting straight into the Linux shell means that you can use all Linux commands to install/remove packets, manage system settings, etc. Any experienced Linux user will be happy here.

There are still a few tricks you need to remember, they are covered in the related manuals. Here we mention the main things you can do.

It is still not possible to manage a switch port just using Linux commands on a single entity, they are divided into physical interfaces, routing interfaces, vlan interfaces, and bonding interfaces. You will need to apply configurations in several steps to /etc/network/interfaces, like:

Configuring Physical Interface Properties

auto fpti1_0_1
iface fpti1_0_1 inet static
       address 0.0.0.0
       mtu 1540
       pre-up /sbin/ethtool -s fpti1_0_1 speed 10 duplex half
       pre-down /sbin/ethtool -s fpti1_0_1 speed 10 duplex half

Configuring a Port-based Routing Interface

auto rt1_0_2
iface rt1_0_2 inet static
     mtu 200
      address 4.2.2.2
      netmask 255.255.255.0
      up ip addr add 3.3.3.3/24 dev rt1_0_2
     up ip addr add 2001::1/64 dev rt1_0_2

Configuring a VLAN Routing Interface

auto rt_v20
iface rt_v20 inet static
     mtu 200
      address 4.2.2.2
      netmask 255.255.255.0
      up ip addr add 3.3.3.3/24 dev rt_v20
     up ip addr add 2001::1/64 dev rt_v20

Configuring Routes

auto rt_v20
iface rt_v20 inet static
     mtu 200
      address 4.2.2.2
      netmask 255.255.255.0
      up route add -net 5.5.5.0 netmask 255.255.255.0 gw 4.2.2.3
     up ip route add 6.6.6.0/24 via 4.2.2.3

Configuring a LAG Interface

auto bond1
iface bond1 inet static
        address 0.0.0.0
        mtu 2000
        up ip link set fpti1_0_1 master bond1
      up echo 4 > /sys/class/net/bond1/bonding/mode
      up echo 2 > /sys/class/net/bond1/bonding/ min_links

Layer 3 features

Working with routes from Linux shell:

Install IPv4 Route

root@host:/home/admin# ip route add 9.9.9.0/24 via 2.0.0.3
root@host:/home/admin# route add -net 4.4.4.0 netmask 255.255.255.0 gw 7.0.0.5
root@host:/home/admin#
root@host:/home/admin# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.130.84.129 0.0.0.0 UG 100 0 0 eth0
10.130.84.128 0.0.0.0 255.255.255.128 U 0 0 0 eth0
7.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 rt1_0_11
2.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 rt1_0_18
4.4.4.0 7.0.0.5 255.255.255.0 UG 0 0 0 rt1_0_11
9.9.9.0 2.0.0.3 255.255.255.0 UG 0 0 0 rt1_0_18
root@host:/home/admin#
root@host:/home/admin#icos-cli
(host) #
(host) #show ip route
Route Codes: R - RIP Derived, O - OSPF Derived, C - Connected, S - Static
       B - BGP Derived, IA - OSPF Inter Area
       E1 - OSPF External Type 1, E2 - OSPF External Type 2
       N1 - OSPF NSSA External Type 1, N2 - OSPF NSSA External Type 2
       S U - Unnumbered Peer, L - Leaked Route, K - Kernel
C 7.0.0.0/24 [0/1] directly connected, 0/11
C 2.0.0.0/24 [0/1] directly connected, 0/18
K 9.9.9.0/24 [1/6493] via 2.0.0.3, 00d:00h:1m, 0/18
K 4.4.4.0/24 [1/6493] via 7.0.0.5, 00d:00h:1m, 0/11
(host) #

Adding IPv6 ECMP Routes

root@host:/home/admin#
root@host:/home/admin# route add -6 3001:33:3::/64 gw 2009:1::13
root@host:/home/admin# route add -6 3001:33:3::/64 gw 2044:1::14
root@host:/home/admin# route add -6 3001:33:3::/64 gw 2044:1::15
root@host:/home/admin#
root@host:/home/admin# route -6 -n
Kernel IPv6 routing table
Destination      Next Hop       Flag Met Ref Use If
2009:1::/64      ::             U    256 0   6   rt1_0_11
2044:1::/64      ::             U    256 0   7   rt1_0_13
3001:33:3::/64   2009:1::13     UG   1   0   0   rt1_0_11
3001:33:3::/64   2044:1::14     UG   1   0   0   rt1_0_13
3001:33:3::/64   2044:1::15     UG   1   0   0   rt1_0_13
root@host:/home/admin#
root@host:/home/admin#
root@host:/home/admin#icos-cli
(host) #
(host) #show ipv6 route
IPv6 Routing Table - 3 entries
Codes: C - connected, S - static, 6To4 - 6to4 Route, B - BGP Derived
O - OSPF Intra, OI - OSPF Inter, OE1 - OSPF Ext 1, OE2 - OSPF Ext 2
ON1 - OSPF NSSA Ext Type 1, ON2 - OSPF NSSA Ext Type 2, K - kernel
C 2009:1::/64 [0/0]
   via ::, 0/11
C 2044:1::/64 [0/0]
   via ::, 0/13
K 3001:33:3::/64 [1/0]
   via 2009:1::13, 00h:07m:25s, 0/11
   via 2044:1::14, 00h:07m:25s, 0/13
   via 2044:1::15, 00h:07m:25s, 0/13
(host) #

Redistribution of IPv4 Kernel Routes via BGP is possible.

root@host:/home/admin # route add -net 6.5.5.0 netmask 255.255.255.0 gw 2.2.2.9
root@host:/home/admin # route -n
Kernel IP routing table
Destination     Gateway       Genmask          Flags Metric Ref Use Iface
0.0.0.0         10.130.187.1  0.0.0.0          UG    16     0   0   eth0
2.2.2.0         0.0.0.0       255.255.255.0    U     0      0   0   rt1_0_9
6.5.5.0         2.2.2.10      255.255.255.0    UG    0      0   0   rt1_0_9
10.130.187.0    0.0.0.0       255.255.255.128  U     0      0   0   eth0
1 27.0.0.0      0.0.0.0       255.0.0.0        U     0      0   0   lo
root@host:/home/admin # exit

Quagga

You can install Quagga from http://download.savannah.gnu.org/releases/quagga/ and manage routes from its CLI:

(localhost) #telnet 127.0.0.1 2601
Trying 127.0.0.1...
Connected to 127.0.0.1
Hello, this is Quagga (version 0.99.18).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
User Access Verification
Password:
Password:
Router>
Router> en
Password:
Password:
Router#
Router# config t
Router(config)# ip route 7.3.6.0 255.255.255.0 1.2.3.5
14 Dec 18 15:32:39 localhost-1 VR_AGENT[procLOG]: rto_netlink.c(1121) 651
%% Successfully added
kernel route ip 7.3.6.0 mask 255.255.255.0 via 1.2.3.5 on IntIf 25.
Router(config)# exit
Router# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF,
I - ISIS, B - BGP, > - selected route, * - FIB route
K>* 0.0.0.0/0 via 10.130.171.129, eth0
C>* 1.2.3.0/24 is directly connected, rt1_0_25
S>* 7.3.6.0/24 [1/0] via 1.2.3.5, rt1_0_25
C>* 10.130.171.128/25 is directly connected, eth0
C>* 127.0.0.0/8 is directly connected, lo
C>* 250.0.0.0/30 is directly connected, rfc5549if

Quagga routes are added to the kernel:

root@localhost:/home/admin# ip route
default via 10.130.171.129 dev eth0 metric 100
1.2.3.0/24 dev rt1_0_25 proto kernel scope link src 1.2.3.4
7.3.6.0/24 via 1.2.3.5 dev rt1_0_25 proto zebra
10.130.171.128/25 dev eth0 proto kernel scope link src 10.130.171.168
250.0.0.0/30 dev rfc5549if proto kernel scope link src 250.0.0.1

Route gets propagated from kernel to ICOS OS, and then to hardware (from ICOS CLI):

(host) #show ip route
Route Codes: R - RIP Derived, O - OSPF Derived, C - Connected, S - Static
B - BGP Derived, IA - OSPF Inter Area
E1 - OSPF External Type 1, E2 - OSPF External Type 2
N1 - OSPF NSSA External Type 1, N2 - OSPF NSSA External Type 2
S U - Unnumbered Peer, L - Leaked Route, K - Kernel
C 1.2.3.0/24 [0/0] directly connected, 0/25
K 7.3.6.0/24 [1/56] via 1.2.3.5, 00h:02m:11s, 0/25