Solaris 11 NWAM
2011-11-14

Solaris 11 completely changes how network interfaces are configured and managed. The Network Auto-Magic Daemon (NWAM) is now responsible for automatically configuring the network (all interfaces, wired and/or wifi) based on the combination of a specified network configuration profile and a location profile. These profiles can range from very complex to very simple.

A network configuration profile (NCP) defines the overall network configuration of the system. NCPs are made up of network configuration units (NCU) that define how individual physical links and interfaces are configured. There are two types of NCUs, Link NCUs and Interface NCUs. Link NCUs define layer 2 (PHY) settings and Interface NCUs define layer 3 (IP) settings.

The default NCP is named Automatic and this NCP cannot be modified or destroyed:

Automatic NCP

  • defines a link and interface NCU for every interface
  • prefers connected wired links over wireless
  • plumbs both IPv4 and IPv6 for each interface
  • DHCP is enabled for IPv4
  • IPv6 stateless autoconf is enabled for IPv6
  • configuration changes dynamically when links brought up/down

Here is the configuration for the Automatic NCP and my system with eight interfaces:

# netadm list -p ncp Automatic
TYPE        PROFILE        STATE
ncp         Automatic      online
ncu:phys    net0           online
ncu:ip      net0           online
ncu:phys    net1           offline
ncu:ip      net1           offline
ncu:phys    net2           offline
ncu:ip      net2           offline
ncu:phys    net3           online
ncu:ip      net3           offline*
ncu:phys    net4           online
ncu:ip      net4           offline*
ncu:phys    net5           offline
ncu:ip      net5           offline
ncu:phys    net6           online
ncu:ip      net6           online
ncu:phys    net7           offline
ncu:ip      net7           offline

I think the stars mean there is link but no usable address assigned...?

The location profile is used to provide additional system-wide networking details after the interfaces have been brought up and IP addresses assigned. Examples include name services (nsswitch) and firewall (ipfilter) configurations.

There are two default location profiles defined by NWAM:

NoNet location

  • activated when no interfaces are assigned an IP address
  • no name services are defined
  • firewall enabled to block everything except loopback

Automatic location

  • activated when any interface is assigned an IP address
  • name services configured to allow DNS
  • firewall disabled

Here is the configuration for the Automatic and NoNet locations:

# netcfg list loc NoNet
loc:NoNet
        activation-mode                 system
        enabled                         false
        nameservices                    files
        nameservices-config-file        "/etc/nsswitch.files"
        dns-nameservice-configsrc       dhcp
        ipfilter-config-file            "/etc/nwam/loc/NoNet/ipf.conf"
        ipfilter-v6-config-file         "/etc/nwam/loc/NoNet/ipf6.conf"

# netcfg list loc Automatic
loc:Automatic
        activation-mode                 system
        enabled                         true
        nameservices                    dns
        nameservices-config-file        "/etc/nsswitch.dns"
        dns-nameservice-configsrc       dhcp

NOTE: ifconfig is being phased out in favor of the following tools. ifconfig can still be used for legacy network configuration but ipadm is preferred. At some point ifconfig will be changed to be read only.

  • dladm - data link administration (like linux ethtool)
  • ipadm - IP interface administration (like original ifconfig)
  • netadm - NCP profile administration (view, enable, disable)
  • netcfg - NCP/NCU/Location administration (create, modify, destroy)

Interface names are now automatically renamed to netX where X is some number. No longer do you see bge0, bge1, bnx0, bnx1, bnxe0, etc. For example on my system with 4x 5709 LOMs, 1x 5718 NIC, and 1x 57810 NIC:

# dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net5              Ethernet             down       0      unknown   bge1
net0              Ethernet             up         1000   full      bnx0
net4              Ethernet             up         1000   full      bge0
net3              Ethernet             up         1000   full      bnx3
net1              Ethernet             down       0      half      bnx1
net2              Ethernet             down       0      half      bnx2
net6              Ethernet             up         10000  full      bnxe0
net7              Ethernet             down       0      half      bnxe1

But the really cool thing is you can rename an interface name to whatever you want! The name must end with an interface index value which in turn can be whatever you want. One caveat though... don't go over 999 which is VLAN territory.

# dladm rename-link net0 dohickey0
# dladm rename-link net6 bnxe0
# dladm rename-link net7 fcoe99
# dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net5              Ethernet             down       0      unknown   bge1
dohickey0         Ethernet             up         1000   full      bnx0
net4              Ethernet             up         1000   full      bge0
net3              Ethernet             up         1000   full      bnx3
net1              Ethernet             down       0      half      bnx1
net2              Ethernet             down       0      half      bnx2
bnxe0             Ethernet             up         10000  full      bnxe0
fcoe99            Ethernet             down       0      half      bnxe1

When the Automatic NCP is enabled you cannot make any network changes to the system. Instead you must create your own NCP using netcfg and enable it using netadm. Here I'll walk through an example which includes only three interfaces (IPv4 only): DHCP on net0/bnx0, static on net4/bge0, and DHCP on net6/bnxe0.

First off, here is the current network configuration as performed by the Automatic NCP. We can see that NWAM received an address via DHCP for both net0/bnx0 and net6/bnxe0.

# ipadm show-addr
ADDROBJ           TYPE     STATE        ADDR
lo0/v4            static   ok           127.0.0.1/8
net0/_b           dhcp     ok           10.13.107.254/23
net3/_b           dhcp     ok           ?
net4/_b           dhcp     ok           ?
net6/_b           dhcp     ok           192.168.0.123/16
lo0/v6            static   ok           ::1/128
net0/_a           addrconf ok           fe80::d685:64ff:fe69:33b2/10
net3/_a           addrconf ok           fe80::d685:64ff:fe69:33b8/10
net4/_a           addrconf ok           fe80::210:18ff:fe6f:cb82/10
net6/_a           addrconf ok           fe80::210:18ff:fea7:1df0/10

Now lets create a new NCP. This is done using the netcfg tool and it can be scripted (i.e. via command line arguments) or done interactively. Here is an interactive session:

# netcfg
netcfg>
netcfg> help
commands:
        cancel
        clear <prop-name>
        commit
        create [-t <template>] <object-type> [<class>] <object-name>
        destroy {-a | <object-type> [<class>] <object-name>}
        end
        exit
        export [-d] [-f <output-file>] [<object-type> [<class>] <object-name>]
        get [-V] <prop-name>
        help [command-name]
        list [-a] [<object-type> [<class>] <object-name>]
        revert
        select <object-type> [<class>] <object-name>
        set <prop-name>=<value1>[,<value2>...]
        verify
        walkprop [-a]
netcfg>
netcfg> create ncp insanum
netcfg:ncp:insanum> list
netcfg:ncp:insanum> create ncu phys net0
Created ncu 'net0'.  Walking properties ...
activation-mode (manual) [manual|prioritized]>
link-mac-addr>
link-autopush>
link-mtu>
netcfg:ncp:insanum:ncu:net0> list
ncu:net0
        type                    link
        class                   phys
        parent                  "insanum"
        activation-mode         manual
        enabled                 true
netcfg:ncp:insanum:ncu:net0> end
Committed changes
netcfg:ncp:insanum> create ncu ip net0
Created ncu 'net0'.  Walking properties ...
ip-version (ipv4,ipv6) [ipv4|ipv6]> ipv4
ipv4-addrsrc (dhcp) [dhcp|static]>
ipv4-default-route>
netcfg:ncp:insanum:ncu:net0> list
ncu:net0
        type                    interface
        class                   ip
        parent                  "insanum"
        enabled                 true
        ip-version              ipv4
        ipv4-addrsrc            dhcp
        ipv6-addrsrc            dhcp,autoconf
netcfg:ncp:insanum:ncu:net0> end
Committed changes
netcfg:ncp:insanum> list
NCUs:
        phys    net0
        ip      net0
netcfg:ncp:insanum>
netcfg:ncp:insanum> create ncu phys net4
Created ncu 'net4'.  Walking properties ...
activation-mode (manual) [manual|prioritized]>
link-mac-addr>
link-autopush>
link-mtu>
netcfg:ncp:insanum:ncu:net4> list
ncu:net4
        type                    link
        class                   phys
        parent                  "insanum"
        activation-mode         manual
        enabled                 true
netcfg:ncp:insanum:ncu:net4> end
Committed changes
netcfg:ncp:insanum> create ncu ip net4
Created ncu 'net4'.  Walking properties ...
ip-version (ipv4,ipv6) [ipv4|ipv6]> ipv4
ipv4-addrsrc (dhcp) [dhcp|static]> static
ipv4-addr> 172.16.99.44/24
ipv4-default-route>
netcfg:ncp:insanum:ncu:net4> list
ncu:net4
        type                    interface
        class                   ip
        parent                  "insanum"
        enabled                 true
        ip-version              ipv4
        ipv4-addrsrc            static
        ipv4-addr               "172.16.99.44/24"
        ipv6-addrsrc            dhcp,autoconf
netcfg:ncp:insanum:ncu:net4> end
Committed changes
netcfg:ncp:insanum> list
NCUs:
        phys    net0
        ip      net0
        phys    net4
        ip      net4
netcfg:ncp:insanum> create ncu phys net6
Created ncu 'net6'.  Walking properties ...
activation-mode (manual) [manual|prioritized]>
link-mac-addr>
link-autopush>
link-mtu>
netcfg:ncp:insanum:ncu:net6> list
ncu:net6
        type                    link
        class                   phys
        parent                  "insanum"
        activation-mode         manual
        enabled                 true
netcfg:ncp:insanum:ncu:net6> end
Committed changes
netcfg:ncp:insanum> create ncu ip net6
Created ncu 'net6'.  Walking properties ...
ip-version (ipv4,ipv6) [ipv4|ipv6]> ipv4
ipv4-addrsrc (dhcp) [dhcp|static]>
ipv4-default-route>
netcfg:ncp:insanum:ncu:net6> list
ncu:net6
        type                    interface
        class                   ip
        parent                  "insanum"
        enabled                 true
        ip-version              ipv4
        ipv4-addrsrc            dhcp
        ipv6-addrsrc            dhcp,autoconf
netcfg:ncp:insanum:ncu:net6> end
Committed changes
netcfg:ncp:insanum> list
NCUs:
        phys    net0
        ip      net0
        phys    net4
        ip      net4
        phys    net6
        ip      net6
netcfg:ncp:insanum> end
netcfg> end

For comparison, the follow sequence of commands create the same configuration (easy scripting and these commands could be combined if needed):

# netcfg create ncp insanum
# netcfg select ncp insanum \; create ncu phys net0
# netcfg select ncp insanum \; create ncu ip net0
# netcfg select ncp insanum \; select ncu ip net0 \; set ip-version=ipv4
# netcfg select ncp insanum \; create ncu phys net4
# netcfg select ncp insanum \; create ncu ip net4
# netcfg select ncp insanum \; select ncu ip net4 \; set ip-version=ipv4
# netcfg select ncp insanum \; select ncu ip net4 \; set ipv4-addrsrc=static
# netcfg select ncp insanum \; select ncu ip net4 \; set ipv4-addr=172.16.99.44/24
# netcfg select ncp insanum \; create ncu phys net6
# netcfg select ncp insanum \; create ncu ip net6
# netcfg select ncp insanum \; select ncu ip net6 \; set ip-version=ipv4

Now that we've created a user NCP we need to enable it. First lets look at the status of the profiles:

# netadm list -x
TYPE        PROFILE        STATE          AUXILIARY STATE
ncp         Automatic      online         active
ncu:phys    net0           online         interface/link is up
ncu:ip      net0           online         interface/link is up
ncu:phys    net1           offline        interface/link is down
ncu:ip      net1           offline        conditions for activation are unmet
ncu:phys    net2           offline        interface/link is down
ncu:ip      net2           offline        conditions for activation are unmet
ncu:phys    net3           online         interface/link is up
ncu:ip      net3           offline*       DHCP wait timeout, still trying...
ncu:phys    net4           online         interface/link is up
ncu:ip      net4           offline*       DHCP wait timeout, still trying...
ncu:phys    net5           offline        interface/link is down
ncu:ip      net5           offline        conditions for activation are unmet
ncu:phys    net6           online         interface/link is up
ncu:ip      net6           online         interface/link is up
ncu:phys    net7           offline        interface/link is down
ncu:ip      net7           offline        conditions for activation are unmet
ncp         insanum        disabled       disabled by administrator
loc         Automatic      online         active
loc         NoNet          offline        conditions for activation are unmet

Lets enable the insanum NCP and see what happens...

# netadm enable insanum

# ipadm show-addr
ADDROBJ           TYPE     STATE        ADDR
lo0/v4            static   ok           127.0.0.1/8
net0/_a           dhcp     ok           10.13.107.254/23
net4/_a           static   ok           172.16.99.44/24
net6/_a           dhcp     ok           192.168.0.123/16
lo0/v6            static   ok           ::1/128

# netadm list -x
TYPE        PROFILE        STATE          AUXILIARY STATE
ncp         Automatic      disabled       disabled by administrator
ncp         insanum        online         active
ncu:phys    net0           online         interface/link is up
ncu:ip      net0           online         interface/link is up
ncu:phys    net4           online         interface/link is up
ncu:ip      net4           online         interface/link is up
ncu:phys    net6           online         interface/link is up
ncu:ip      net6           online         interface/link is up
loc         Automatic      online         active
loc         NoNet          offline        conditions for activation are unmet

Perfect! :-)

Only with a user NCP are you allowed to bring down interfaces, for example unplumbing any necessary interfaces needed to install a new driver binary. Based on the insanum profile above the following steps will bring down the bnxe driver (i.e. net6/net7):

# dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net5              Ethernet             unknown    0      unknown   bge1
net0              Ethernet             up         1000   full      bnx0
net4              Ethernet             up         1000   full      bge0
net3              Ethernet             unknown    0      unknown   bnx3
net1              Ethernet             unknown    0      unknown   bnx1
net2              Ethernet             unknown    0      unknown   bnx2
net6              Ethernet             up         10000  full      bnxe0
net7              Ethernet             unknown    0      unknown   bnxe1

# netadm disable -p ncu -c ip net6
# netadm disable -p ncu -c phys net6

# netadm list -x
TYPE        PROFILE        STATE          AUXILIARY STATE
ncp         Automatic      disabled       disabled by administrator
ncp         insanum        online         active
ncu:phys    net0           online         interface/link is up
ncu:ip      net0           online         interface/link is up
ncu:phys    net4           online         interface/link is up
ncu:ip      net4           online         interface/link is up
ncu:phys    net6           disabled       disabled by administrator
ncu:ip      net6           disabled       disabled by administrator
loc         Automatic      online         active
loc         NoNet          offline        conditions for activation are unmet

# pkgrm BRCMbnxe
# pkgadd -d <BRCMbnxe.pkg file>

# netadm enable -p ncu -c phys net6
# netadm enable -p ncu -c ip net6

# ipadm show-addr
ipadm show-addr
ADDROBJ           TYPE     STATE        ADDR
lo0/v4            static   ok           127.0.0.1/8
net0/_a           dhcp     ok           10.13.107.254/23
net4/_a           static   ok           172.16.99.44/24
net6/_a           dhcp     ok           192.168.0.123/16
lo0/v6            static   ok           ::1/128

# netadm list -x
netadm list -x
TYPE        PROFILE        STATE          AUXILIARY STATE
ncp         Automatic      disabled       disabled by administrator
ncp         insanum        online         active
ncu:phys    net0           online         interface/link is up
ncu:ip      net0           online         interface/link is up
ncu:phys    net4           online         interface/link is up
ncu:ip      net4           online         interface/link is up
ncu:phys    net6           online         interface/link is up
ncu:ip      net6           online         interface/link is up
loc         Automatic      online         active
loc         NoNet          offline        conditions for activation are unmet

Now lets look at the location profile. The Automatic location profile is still in effect and has updated the /etc/nsswitch.conf file for DNS. Specifically the hosts and ipnodes lines are:

  • hosts: files dns mdns
  • ipnodes: files dns mdns

If you're like me and your Solaris system lives in a Windows environment you like to use Samba and set up name resolution via WINS. I do this not because I use Windows (I DON'T!), but simply because I like my Solaris systems registering their names on the network making it easier for me to remember and access them from whatever client system I'm working from.

To enable WINS we have to add winbind to the /etc/nsswitch.conf file. This is achieved in one of two ways. First, and easiest, is to add winbind to the hosts and ipnodes lines in the /etc/nsswitch.dns file which is used by the Automatic location to create /etc/nsswitch.conf (and then performing a netadm disable and enable command on the Automatic location). The second and more formal way is to create a new location profile:

# netcfg
netcfg> create loc insanum
Created loc 'insanum'.  Walking properties ...
activation-mode (manual) [manual|conditional-any|conditional-all]>
nameservices (dns) [dns|files|nis|ldap]>
nameservices-config-file ("/etc/nsswitch.dns")> /etc/nsswitch.dns_wins
dns-nameservice-configsrc (dhcp) [manual|dhcp]>
nfsv4-domain>
ipfilter-config-file>
ipfilter-v6-config-file>
ipnat-config-file>
ippool-config-file>
ike-config-file>
ipsecpolicy-config-file>
netcfg:loc:insanum>
netcfg:loc:insanum> list
loc:insanum
        activation-mode                 manual
        enabled                         false
        nameservices                    dns
        nameservices-config-file        "/etc/nsswitch.dns_wins"
        dns-nameservice-configsrc       dhcp
netcfg:loc:insanum> end
Committed changes
netcfg> end

Or:

# netcfg create loc insanum
# netcfg select loc insanum \; set nameservices-config-file=/etc/nsswitch.dns_wins

Now create a copy of /etc/nsswitch.dns to /etc/nsswitch.dns_wins and add winbind to the hosts and ipnodes lines.

Lets enable the insanum location and see what happens...

# netadm list -x
TYPE        PROFILE        STATE          AUXILIARY STATE
ncp         Automatic      disabled       disabled by administrator
ncp         insanum        online         active
ncu:phys    net0           online         interface/link is up
ncu:ip      net0           online         interface/link is up
ncu:phys    net4           online         interface/link is up
ncu:ip      net4           online         interface/link is up
ncu:phys    net6           online         interface/link is up
ncu:ip      net6           online         interface/link is up
loc         Automatic      online         active
loc         insanum        disabled       disabled by administrator
loc         NoNet          offline        conditions for activation are unmet

# netadm enable -p loc insanum

# netadm list -x
TYPE        PROFILE        STATE          AUXILIARY STATE
ncp         Automatic      disabled       disabled by administrator
ncp         insanum        online         active
ncu:phys    net0           online         interface/link is up
ncu:ip      net0           online         interface/link is up
ncu:phys    net4           online         interface/link is up
ncu:ip      net4           online         interface/link is up
ncu:phys    net6           online         interface/link is up
ncu:ip      net6           online         interface/link is up
loc         Automatic      offline        conditions for activation are unmet
loc         insanum        online         active
loc         NoNet          offline        conditions for activation are unmet

Now you'll see the proper /etc/nsswitch.conf file in place.

This is only a small snippet of what you can do with NWAM. See the "Oracle System Administration: Network Interfaces and Network Virtualization" manual for all the gory details.

Here is a script I cooked up that wraps all the netadm and netcfg operations you normally do as an admin or during development. See the usage for how easy it is to administer network profiles and interfaces.

#!/bin/bash

SCR=nwam

function Usage()
{
    echo "
Usage: $SCR [ -h | <prof> <cmd> [args] ]

With NO arguments the current interface state is displayed.

<prof>:

ncpAuto | locAuto | locNoNet | [ncp|loc]<name>
- profile names must be prefixed with 'ncp' or 'loc'

<cmd>:

list
- show netadm profile state

dump
- dump netcfg profile configuration

create <if>,[<ip/mask>|dhcp] ...
- create an ncp or loc profile
- ncu(s) are only added for an ncp profile

destroy
- destroy an ncp or loc profile

add <if>,[<ip/mask>|dhcp] ...
- add ncu(s) to an ncp

remove <if> ...
- remove ncu(s) from an ncp

enable
- enable an ncp

EXAMPLES:

$SCR
$SCR ncpFoo create net0,dhcp net4,172.16.99.5/24
$SCR ncpFoo dump
$SCR ncpFoo enable
$SCR ncpFoo add net6,192.168.0.5/24
$SCR ncpFoo remove net4 net6
$SCR ncpAuto enable
$SCR ncpFoo destroy
"
    exit
}

if [ -z $1 ]; then
    netadm list -x
    echo
    ipadm show-addr
    echo
    dladm show-phys
    exit
fi

if [ $1 = -h ]; then Usage; fi

if    [ $1 =  ncpAuto  ];  then P="ncp"; PROF="Automatic"
elif  [ $1 =  locAuto  ];  then P="loc"; PROF="Automatic"
elif  [ $1 =  locNoNet ];  then P="loc"; PROF="NoNet"
elif [[ $1 =~ ^ncp     ]]; then P="ncp"; PROF=$1
elif [[ $1 =~ ^loc     ]]; then P="loc"; PROF=$1
else echo "ERROR: Invalid profile ($1)"; exit
fi

shift

function DumpProfile()
{
    if [ $P = loc ]; then
        netcfg select loc $PROF \; list
    else
        OIFS=$IFS
        IFS=$'\n'
        for NCU in `netcfg select ncp $PROF \; list | tail +2`; do
            netcfg select ncp $PROF \; select ncu $NCU \; list
        done
        IFS=$OIFS
    fi
}

function AddNCU()
{
    # <if>,[<ip/mask>|dhcp] ...
    while [ -n ${1:-''} ]; do
        v=(${1//,/ })
        if [ -z ${v[0]} ] || [ -z ${v[1]} ]; then
            echo "ERROR: Invalid argument ($1)"
            exit
        fi

        netcfg select ncp $PROF \; create ncu phys ${v[0]}
        netcfg select ncp $PROF \; create ncu ip ${v[0]}

        if [ ${v[1]} = dhcp ]; then
            netcfg select ncp $PROF \; select ncu ip ${v[0]} \; \
                   set ip-version=ipv4 \; set ipv4-addrsrc=dhcp \; \
                   clear ipv4-addr
        else
            netcfg select ncp $PROF \; select ncu ip ${v[0]} \; \
                   set ip-version=ipv4 \; set ipv4-addrsrc=static \; \
                   set ipv4-addr=${v[1]}
        fi

        shift
    done
}

function RemoveNCU()
{
    # <if> ...
    while [ -n ${1:-''} ]; do
        netcfg select ncp $PROF \; destroy ncu phys $1
        netcfg select ncp $PROF \; destroy ncu ip $1
        shift
    done
}

if [ $1 != create ]; then
    netcfg select $P $PROF &> /dev/null
    if [ $? -ne 0 ]; then
        echo "ERROR: Invalid profile ($PROF)"
        exit
    fi
fi

if [ $1 = list ]; then
    netadm list -x -p $P $PROF
elif [ $1 = dump ]; then
    DumpProfile
elif [ $1 = create ]; then
    netcfg create $P $PROF
    if [ $P = ncp ]; then shift; AddNCU $*; fi
elif [ $1 = destroy ]; then
    netcfg destroy $P $PROF
elif [ $1 = add ]; then
    shift; AddNCU $*
elif [ $1 = remove ]; then
    shift; RemoveNCU $*
elif [ $1 = enable ]; then
    netadm enable -p $P $PROF
else
    echo "ERROR: Invalid command ($1)"
fi