~paelzer/serverguide/serverguide-chrony-18.04

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.3//EN" 
	"http://www.oasis-open.org/docbook/xml/4.3/docbookx.dtd" [
<!ENTITY % globalent SYSTEM "../../libs/global.ent">
%globalent;
<!ENTITY % xinclude SYSTEM "../../libs/xinclude.mod">
%xinclude;
<!ENTITY language "&EnglishAmerican;">
]>
<chapter id="networking" status="review">
	<title>Networking</title>
	<para>
	Networks consist of two or more devices, such as computer systems, printers, 
	and related equipment which are connected by either physical cabling or 
	wireless links for the purpose of sharing and distributing information among 
	the connected devices. 
  	</para>
  	<para>
	This section provides general and specific information pertaining
        to networking, including an overview of network concepts and detailed 
	discussion of popular network protocols.
	</para>

	<sect1 id="network-configuration" status="review">
	<title>Network Configuration</title>
	<para>
	Ubuntu ships with a number of graphical utilities to configure your 
	network devices.  This document is geared toward server administrators 
	and will focus on managing your	network on the command line.
    </para>

		<sect2 id="ethernet-interfaces" status="review">
		<title>Ethernet Interfaces</title>
		<para>
		Ethernet interfaces are identified by the system using the naming convention of 
		<emphasis role="italix">ethX</emphasis>, where <emphasis role="italic">X</emphasis> 
		represents a numeric value.  The first Ethernet interface is typically identified 
		as <emphasis role="italic">eth0</emphasis>, the second as 
		<emphasis role="italic">eth1</emphasis>, and all others should move up in 
		numerical order.
		</para>

			<sect3 id="identify-ethernet-interfaces" status="review">
			<title>Identify Ethernet Interfaces</title>
			<para>
			To quickly identify all available Ethernet interfaces, you can use the 
			<application>ifconfig</application> command as shown below.
			</para>
<screen>
<command>ifconfig -a | grep eth</command>
<computeroutput>eth0      Link encap:Ethernet  HWaddr 00:15:c5:4a:16:5a</computeroutput>
</screen>
			<para>
			Another application that can help identify all network interfaces available to your system 
			is the <application>lshw</application> command.  In the example below, <application>lshw</application> 
			shows a single Ethernet interface with the logical name of <emphasis role="italic">eth0</emphasis>
			along with bus information, driver details and all supported capabilities.
			</para>
<screen>
<command>sudo lshw -class network</command>
<computeroutput>  *-network
       description: Ethernet interface
       product: BCM4401-B0 100Base-TX
       vendor: Broadcom Corporation
       physical id: 0
       bus info: pci@0000:03:00.0
       logical name: eth0
       version: 02
       serial: 00:15:c5:4a:16:5a
       size: 10MB/s
       capacity: 100MB/s
       width: 32 bits
       clock: 33MHz
       capabilities: (snipped for brevity)
       configuration: (snipped for brevity)
       resources: irq:17 memory:ef9fe000-ef9fffff</computeroutput>
</screen>
			</sect3>

			<sect3 id="ethernet-interface-names" status="review">
			<title>Ethernet Interface Logical Names</title>
			<para>
			Interface logical names are configured in the file 
			<filename>/etc/udev/rules.d/70-persistent-net.rules.</filename>  If you would 
			like control which interface receives a particular logical name, find the line 
			matching the interfaces physical MAC address and modify the value of 
			<emphasis role="italic">NAME=ethX</emphasis> to the desired logical name. 
			Reboot the system to commit your changes.
			</para>
			</sect3>

			<sect3 id="ethernet-interface-settings" status="review">
			<title>Ethernet Interface Settings</title>
			<para>
			<application>ethtool</application> is a program that displays and changes Ethernet 
			card settings such as auto-negotiation, port speed, duplex mode, and Wake-on-LAN. It
			is not installed by default, but is available for installation in the repositories.
			</para>
<screen>
<command>sudo apt install ethtool</command>
</screen>
			<para>
			The following is an example of how to view supported features and configured 
			settings of an Ethernet interface.
			</para>
<screen>
<command>sudo ethtool eth0</command>
<computeroutput>Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Half 1000baseT/Full 
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Half 1000baseT/Full 
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: g
        Wake-on: d
        Current message level: 0x000000ff (255)
        Link detected: yes</computeroutput>
</screen>
			<para>
			Changes made with the <application>ethtool</application> command are temporary 
			and will be lost after a reboot. If you would like to retain settings, simply add 
			the desired <application>ethtool</application> command to a <emphasis role="italic">pre-up</emphasis> 
			statement in the interface configuration file <filename>/etc/network/interfaces</filename>. 
			</para>
			<para>
			The following is an example of how the interface identified as <emphasis role="italic">eth0</emphasis>
			could be permanently configured with a port speed of 1000Mb/s running in full duplex mode.
			</para>
<programlisting>
auto eth0
iface eth0 inet static
pre-up /sbin/ethtool -s eth0 speed 1000 duplex full
</programlisting>
			<note>
			<para>
			Although the example above shows the interface configured to use the 
			<emphasis role="italic">static</emphasis> method, it actually works with other 
			methods as well, such as DHCP.  The example is meant to demonstrate only proper 
			placement of the <emphasis role="italic">pre-up</emphasis> statement in relation 
			to the rest of the interface configuration.
			</para>
			</note>
			</sect3>
		</sect2>

		<sect2 id="ip-addressing" status="review">
		<title>IP Addressing</title>
		<para>
		The following section describes the process of configuring your systems IP address
		and default gateway needed for communicating on a local area network and the
		Internet.
		</para>

			<sect3 id="temp-ip-assignment" status="review">
			<title>Temporary IP Address Assignment</title>
			<para>
			For temporary network configurations, you can use standard commands 
			such as <application>ip</application>, <application>ifconfig</application> 
			and <application>route</application>, which are also found on most other 
			GNU/Linux operating systems.  These commands allow you to configure settings
			which take effect immediately, however they are not persistent and will
			be lost after a reboot.
			</para>
			<para>
			To temporarily configure an IP address, you can use the <application>ifconfig</application> 
			command in the following manner. Just modify the IP address and subnet mask to match your 
			network requirements.
			</para>
<screen>
<command>sudo ifconfig eth0 10.0.0.100 netmask 255.255.255.0</command>
</screen>
			<para>
			To verify the IP address configuration of <application>eth0</application>, 
			you can use the <application>ifconfig</application> command in the following manner.
			</para>
<screen>
<command>ifconfig eth0</command>
<computeroutput>eth0      Link encap:Ethernet  HWaddr 00:15:c5:4a:16:5a  
          inet addr:10.0.0.100  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::215:c5ff:fe4a:165a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:466475604 errors:0 dropped:0 overruns:0 frame:0
          TX packets:403172654 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:2574778386 (2.5 GB)  TX bytes:1618367329 (1.6 GB)
          Interrupt:16</computeroutput> 
</screen>
			<para>
			To configure a default gateway, you can use the <application>route</application> 
			command in the following manner.  Modify the default gateway address to match 
			your network requirements.
			</para>
<screen>
<command>sudo route add default gw 10.0.0.1 eth0</command>
</screen>
			<para>
			To verify your default gateway configuration, you can use the <application>route</application> 
			command in the following manner.
			</para>
<screen>
<command>route -n</command>
<computeroutput>Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.0.0.0        0.0.0.0         255.255.255.0   U     1      0        0 eth0
0.0.0.0         10.0.0.1        0.0.0.0         UG    0      0        0 eth0</computeroutput>
</screen>
			<para>
			If you require DNS for your temporary network configuration, you can add DNS server
			IP addresses in the file <filename>/etc/resolv.conf</filename>. In general, editing
                        <filename>/etc/resolv.conf</filename> directly is not recommanded, but this is a
                        temporary and non-persistent configuration. The example below 
			shows how to enter two DNS servers to <filename>/etc/resolv.conf</filename>, which 
			should be changed to servers appropriate for your network. A more lengthy description 
			of the proper persistent way to do DNS client configuration is in a following section.
			</para>
<programlisting>
nameserver 8.8.8.8
nameserver 8.8.4.4
</programlisting>
			<para>
			If you no longer need this configuration and wish to purge all IP configuration from
			an interface, you can use the <application>ip</application> command with the flush option
			as shown below.  
			</para>
<screen>
<command>ip addr flush eth0</command>
</screen>
			<note>
			<para>
			Flushing the IP configuration using the <application>ip</application> command does not clear the 
			contents of <filename>/etc/resolv.conf</filename>. You must remove or modify those entries manually,
                        or re-boot which should also cause <filename>/etc/resolv.conf</filename>, which is actually now a
                        symlink to <filename>/run/resolvconf/resolv.conf</filename>, to be re-written.
			</para>
			</note>
			</sect3>

			<sect3 id="dynamic-ip-addressing" status="review">
			<title>Dynamic IP Address Assignment (DHCP Client)</title>
			<para>
			To configure your server to use DHCP for dynamic address assignment, add the
			<emphasis role="italic">dhcp</emphasis> method to the inet address family statement 
			for the appropriate interface in the file <filename>/etc/network/interfaces</filename>.
			The example below assumes you are configuring your first Ethernet interface identified as 
			<emphasis role="italic">eth0</emphasis>.
			</para>
<programlisting>
auto eth0
iface eth0 inet dhcp
</programlisting>
			<para>
			By adding an interface configuration as shown above, you can manually enable the 
			interface through the <application>ifup</application> command which initiates the 
			DHCP process via <application>dhclient</application>.
			</para>
<screen>
<command>sudo ifup eth0</command>
</screen>
			<para>
			To manually disable the	interface, you can use the <application>ifdown</application> 
			command, which in turn will initiate the DHCP release process and shut down the 
			interface.
			</para>
<screen>
<command>sudo ifdown eth0</command>
</screen>
			</sect3>

			<sect3 id="static-ip-addressing" status="review">
			<title>Static IP Address Assignment</title>
			<para>
			To configure your system to use a static IP address assignment, add the 
			<emphasis role="italic">static</emphasis> method to the inet address family statement 
			for the appropriate interface in the file <filename>/etc/network/interfaces</filename>. 
			The example below assumes you are configuring your first Ethernet interface identified as 
			<emphasis role="italic">eth0</emphasis>.  Change the <emphasis role="italic">address</emphasis>, 
			<emphasis role="italic">netmask</emphasis>, and <emphasis role="italic">gateway</emphasis> 
			values to meet the requirements of your network.
			</para>
<programlisting>
auto eth0
iface eth0 inet static
address 10.0.0.100
netmask 255.255.255.0
gateway 10.0.0.1
</programlisting>
			<para>
			By adding an interface configuration as shown above, you can manually enable the 
			interface through the <application>ifup</application> command.
			</para>
<screen>
<command>sudo ifup eth0</command>
</screen>
			<para>
			To manually disable the	interface, you can use the <application>ifdown</application> 
			command.
			</para>
<screen>
<command>sudo ifdown eth0</command>
</screen>
			</sect3>

			<sect3 id="loopback-interface" status="review">
			<title>Loopback Interface</title>
			<para>
			The loopback interface is identified by the system as <emphasis role="italic">lo</emphasis>
			and has a default IP address of 127.0.0.1.  It can be viewed using the ifconfig command. 
			</para>
<screen>
<command>ifconfig lo</command>
<computeroutput>lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:2718 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2718 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:183308 (183.3 KB)  TX bytes:183308 (183.3 KB)</computeroutput>
</screen>
			<para>
			By default, there should be two lines in <filename>/etc/network/interfaces</filename> 
			responsible for automatically configuring your loopback interface. It is recommended that you 
			keep the default settings unless you have a specific purpose for changing them.  An example of 
			the two default lines are shown below.
			</para>
<programlisting>
auto lo
iface lo inet loopback
</programlisting>
			</sect3>
		</sect2>


		<sect2 id="name-resolution" status="review">
		<title>Name Resolution</title>
		<para>
		Name resolution as it relates to IP networking is the process of mapping IP addresses 
		to hostnames, making it easier to identify resources on a network.  The following section
		will explain how to properly configure your system for name resolution using DNS and static
		hostname records.
		</para>

			<sect3 id="dns-client-configuration" status="review">
			<title>DNS Client Configuration</title>
			<para>
				Traditionally, the file <filename>/etc/resolv.conf</filename> was a static configuration file that rarely
			needed to be changed or automatically changed via DCHP client hooks. Nowadays, a computer can switch from
			one network to another quite often and the <emphasis>resolvconf</emphasis> framework is now being used to track
			these changes and update the resolver's configuration automatically.  It acts as an intermediary between programs
			that supply nameserver information and applications that need nameserver information. Resolvconf gets populated with information
			by a set of hook scripts related to network interface configuration. The most notable difference for the
			user is that any change manually done to <filename>/etc/resolv.conf</filename> will be lost as it gets overwritten each time
			something triggers resolvconf. Instead, resolvconf uses DHCP client hooks, and <filename>/etc/network/interfaces</filename> to
			generate a list of nameservers and domains to put in <filename>/etc/resolv.conf</filename>, which is now a symlink:

<programlisting>
/etc/resolv.conf -> ../run/resolvconf/resolv.conf
</programlisting>

			To configure the resolver, add the IP addresses of the nameservers that
			are appropriate for your network in the file <filename>/etc/network/interfaces</filename>. You can also
			add an optional DNS suffix search-lists to match your network domain names. For each other valid
			resolv.conf configuration option, you can include, in the stanza, one line beginning with that
			option name with a <emphasis role="bold">dns-</emphasis> prefix. The resulting file might look like the following:
			</para>

<programlisting>
iface eth0 inet static
    address 192.168.3.3
    netmask 255.255.255.0
    gateway 192.168.3.1
    dns-search example.com
    dns-nameservers 192.168.3.45 192.168.8.10
</programlisting>

			<para>
			The <emphasis role="italic">search</emphasis> option can also be used with multiple domain names 
			so that DNS queries will be appended in the order in which they are entered. For example, your 
			network may have multiple sub-domains to search; a parent domain of <emphasis role="italic">example.com</emphasis>,
			and two sub-domains, <emphasis role="italic">sales.example.com</emphasis> and <emphasis role="italic">dev.example.com</emphasis>.  
			</para>

			<para>
			If you have multiple domains you wish to search, your configuration might look like the following:
			</para>

<programlisting>
iface eth0 inet static
    address 192.168.3.3
    netmask 255.255.255.0
    gateway 192.168.3.1
    dns-search example.com sales.example.com dev.example.com
    dns-nameservers 192.168.3.45 192.168.8.10
</programlisting>

			<para>
			If you try to ping a host with the name of <emphasis role="italic">server1</emphasis>, your system 
			will automatically query DNS for its Fully Qualified Domain Name (FQDN) in the following order:
			</para>

			<orderedlist>
			<listitem>
			<para>
			server1<emphasis role="bold">.example.com</emphasis>
			</para>
			</listitem>
			<listitem>
			<para>
			server1<emphasis role="bold">.sales.example.com</emphasis>
			</para>
			</listitem>
			<listitem>
			<para>
			server1<emphasis role="bold">.dev.example.com</emphasis>
			</para>
			</listitem>
			</orderedlist>
			<para>
			If no matches are found, the DNS server will provide a result of 
			<emphasis role="italic">notfound</emphasis> and the DNS query will fail.
			</para>
			</sect3>

			<sect3 id="static-hostnames" status="review">
			<title>Static Hostnames</title>
			<para>
			Static hostnames are locally defined hostname-to-IP mappings located in the file <filename>/etc/hosts</filename>.
			Entries in the <filename>hosts</filename> file will have precedence over DNS by default. This means that if your
			system tries to resolve a hostname and it matches an entry in /etc/hosts, it will not attempt to look up the
			record in DNS.  In some configurations, especially when Internet access is not required, servers that 
			communicate with a limited number of resources can be conveniently set to use static hostnames instead of DNS.
			</para>
			<para>
			The following is an example of a <filename>hosts</filename> file where a number of local servers 
			have been identified by simple hostnames, aliases and their equivalent Fully Qualified Domain Names (FQDN's).
			</para>
<programlisting>
127.0.0.1	localhost
127.0.1.1	ubuntu-server
10.0.0.11	server1 server1.example.com vpn
10.0.0.12	server2 server2.example.com mail
10.0.0.13	server3 server3.example.com www
10.0.0.14	server4 server4.example.com file
</programlisting>
			<note>
			<para>
			In the above example, notice that each of the servers have been given aliases in addition to their 
			proper names and FQDN's. <emphasis role="italic">Server1</emphasis> has been mapped to the name 
			<emphasis role="italic">vpn</emphasis>, <emphasis role="italic">server2</emphasis> is referred 
			to as <emphasis role="italic">mail</emphasis>, <emphasis role="italic">server3</emphasis> as 
			<emphasis role="italic">www</emphasis>, and <emphasis role="italic">server4</emphasis> as 
			<emphasis role="italic">file</emphasis>.
			</para>
			</note>
			</sect3>

			<sect3 id="name-service-switch-config" status="review">
			<title>Name Service Switch Configuration</title>
			<para>
			The order in which your system selects a method of resolving hostnames to IP addresses is
			controlled by the Name Service Switch (NSS) configuration file <filename>/etc/nsswitch.conf</filename>.
			As mentioned in the previous section, typically static hostnames defined in the systems 
			<filename>/etc/hosts</filename> file have precedence over names resolved from DNS. The following 
			is an example of the line responsible for this order of hostname lookups in the file 
			<filename>/etc/nsswitch.conf</filename>.
			</para>
<programlisting>
hosts:          files mdns4_minimal [NOTFOUND=return] dns mdns4
</programlisting>

			<itemizedlist>
			<listitem>
			<para>
			<emphasis role="bold">files</emphasis> first tries to resolve static hostnames located in 
			<filename>/etc/hosts</filename>.
			</para>
			</listitem>
			<listitem>
			<para>
			<emphasis role="bold">mdns4_minimal</emphasis> attempts to resolve the name using Multicast DNS.
			</para>
			</listitem>
			<listitem>
			<para>
			<emphasis role="bold">[NOTFOUND=return]</emphasis> means that any response of 
			<emphasis role="italic">notfound</emphasis> by the preceding 
			<emphasis role="italic">mdns4_minimal</emphasis> process should be treated as 
			authoritative and that the system should not try to continue hunting for an answer.
			</para>
			</listitem>
			<listitem>
			<para>
			<emphasis role="bold">dns</emphasis> represents a legacy unicast DNS query.
			</para>
			</listitem>
			<listitem>
			<para>
			<emphasis role="bold">mdns4</emphasis> represents a Multicast DNS query.
			</para>
			</listitem>
			</itemizedlist>

			<para>
			To modify the order of the above mentioned name resolution methods, you can
			simply change the <emphasis role="italic">hosts:</emphasis> string to the value 
			of your choosing. For example, if you prefer to use legacy Unicast DNS versus 
			Multicast DNS, you can change the string in <filename>/etc/nsswitch.conf</filename> 
			as shown below.
			</para>
<programlisting>
hosts:          files dns [NOTFOUND=return] mdns4_minimal mdns4
</programlisting>
			</sect3>
		</sect2>

    <sect2 id="bridging" status="review">
      <title>Bridging</title>

      <para>
      Bridging multiple interfaces is a more advanced configuration, but is very useful in multiple scenarios.  
      One scenario is setting up a bridge with multiple network interfaces, then using a firewall to filter traffic
      between two network segments.  Another scenario is using bridge on a system with one interface to allow virtual
      machines direct access to the outside network.  The following example covers the latter scenario.
      </para>

      <para>
      Before configuring a bridge you will need to install the <application>bridge-utils</application> package.  To install the 
      package, in a terminal enter:
      </para>

<screen>
<command>sudo apt install bridge-utils</command>
</screen>

      <para>
      Next, configure the bridge by editing <filename>/etc/network/interfaces</filename>:
      </para>

<programlisting>
auto lo
iface lo inet loopback

auto br0
iface br0 inet static
        address 192.168.0.10
        network 192.168.0.0
        netmask 255.255.255.0
        broadcast 192.168.0.255
        gateway 192.168.0.1
        bridge_ports eth0
        bridge_fd 9
        bridge_hello 2
        bridge_maxage 12
        bridge_stp off
</programlisting>

      <note>
        <para>
        Enter the appropriate values for your physical interface and network.
        </para>
      </note>

      <para>
      Now bring up the bridge:
      </para>

<screen>
<command>sudo ifup br0</command>
</screen>
        <para>
        The new bridge interface should now be up and running.  The <application>brctl</application> provides useful information
        about the state of the bridge, controls which interfaces are part of the bridge, etc.  See <command>man brctl</command> 
        for more information.        
        </para>

      </sect2>
      <sect2 id="network-config-resources" status="review">
        <title>Resources</title>
    
        <para>
        
        </para>

        <itemizedlist>
          <listitem>
            <para>
            The <ulink url="https://help.ubuntu.com/community/Network">Ubuntu Wiki Network page</ulink> has 
            links to articles covering more advanced network configuration.
            </para>
          </listitem>
          <listitem>
            <para>
            The <ulink url="http://manpages.ubuntu.com/manpages/man8/resolvconf.8.html">resolvconf man page</ulink> has 
            more information on resolvconf.
            </para>
          </listitem>
          <listitem>
            <para>
            The <ulink url="http://manpages.ubuntu.com/manpages/man5/interfaces.5.html">interfaces man page</ulink> has 
            details on more options for <filename>/etc/network/interfaces</filename>.
            </para>
          </listitem>
          <listitem>
            <para>
            The <ulink url="http://manpages.ubuntu.com/manpages/man8/dhclient.8.html">dhclient man page</ulink> has 
            details on more options for configuring DHCP client settings.
            </para>
          </listitem>
          <listitem>
            <para>
            For more information on DNS client configuration see the 
            <ulink url="http://manpages.ubuntu.com/manpages/man5/resolver.5.html">resolver man page</ulink>.  Also, Chapter 
            6 of O'Reilly's <ulink url="http://oreilly.com/catalog/linag2/book/ch06.html">Linux Network Administrator's Guide</ulink> is 
            a good source of resolver and name service configuration information.
            </para>
          </listitem>
          <listitem>
            <para>
            For more information on <emphasis>bridging</emphasis> see the 
            <ulink url="http://manpages.ubuntu.com/manpages/man8/brctl.8.html">brctl man page</ulink> and the Linux Foundation's
            <ulink url="http://www.linuxfoundation.org/collaborate/workgroups/networking/bridge">Networking-Bridge</ulink> page.
            </para>
          </listitem>
        </itemizedlist>

      </sect2>
    </sect1>
	<sect1 id="tcpip" status="review">
		<title>TCP/IP</title>
          <para>
            The Transmission Control Protocol and Internet Protocol (TCP/IP) is a standard 
			set of protocols developed in the late 1970s by the Defense Advanced Research 
			Projects Agency (DARPA) as a means of communication between different types of 
			computers and computer networks. TCP/IP is the driving force of the Internet, 
			and thus it is the most popular set of network protocols on Earth. 
          </para>
      <sect2 id="tcpip-introduction" status="review">
        <title>TCP/IP Introduction</title> 
          <para>
            The two protocol components of TCP/IP deal with different aspects of computer networking.
            <emphasis>Internet Protocol</emphasis>, the "IP" of TCP/IP is a 
			connectionless protocol which deals only with network packet routing using the <emphasis 
			role="italics">IP Datagram</emphasis> as the basic unit of networking information.  The 
			IP Datagram consists of a header followed by a message.  The <emphasis>
			Transmission Control Protocol</emphasis> is the "TCP" of TCP/IP and enables network hosts 
			to establish connections which may be used to exchange data streams.  TCP also guarantees 
			that the data between connections is delivered and that it arrives at one network host in 
			the same order as sent from another network host.
          </para>
          </sect2>
        <sect2 id="tcpip-configuration" status="review">
          <title>TCP/IP Configuration</title>
            <para>
            The TCP/IP protocol configuration consists of several elements which must be set by 
			editing the appropriate configuration files, or deploying solutions such as the Dynamic 
			Host Configuration Protocol (DHCP) server which in turn, can be configured to provide the 
			proper TCP/IP configuration settings to network clients automatically. These 
			configuration values must be set correctly in order to facilitate the proper network 
			operation of your Ubuntu system.
            </para>
            <para>
            The common configuration elements of TCP/IP and their purposes are as follows:
            <itemizedlist>
               <listitem>
                  <para>
                  <emphasis role="bold">IP address</emphasis> The IP address is a unique
                  identifying string expressed as four decimal numbers ranging from zero (0)
                  to two-hundred and fifty-five (255), separated by periods,
                  with each of the four numbers representing eight (8) bits of the address for a
                  total length of thirty-two (32) bits for the whole address. This format is called
                  <emphasis>dotted quad
                  notation</emphasis>.</para>
               </listitem>
               <listitem>
                  <para>
                  <emphasis role="bold">Netmask</emphasis> The Subnet Mask (or simply,
                  <emphasis>netmask</emphasis>) is a local bit mask, or set of flags
                  which separate the portions of an IP address significant to the network from the
                  bits significant to the <emphasis>subnetwork</emphasis>.  For example,
                  in a Class C network, the standard netmask is 255.255.255.0 which masks the first
                  three bytes of the IP address and allows the last byte of the IP address to
                  remain available for specifying hosts on the subnetwork.
                  </para>
               </listitem>
               <listitem>
                  <para>
                  <emphasis role="bold">Network Address</emphasis> The Network Address represents the
                  bytes comprising the network portion of an IP address.  For example, the host 12.128.1.2
                  in a Class A network would use 12.0.0.0 as the network address, where twelve (12)
                  represents the first byte of the IP address, (the network part) and zeroes (0) 
                  in all of the remaining three bytes to represent the potential host values.  A network
                  host using the private IP address 192.168.1.100
                  would in turn use a Network Address of 192.168.1.0, which specifies the first three
                  bytes of the Class C 192.168.1 network and a zero (0) for all the possible hosts on the
                  network.
                  </para>
               </listitem>
               <listitem>
                  <para>
                  <emphasis role="bold">Broadcast Address</emphasis> The Broadcast Address is an IP address
                  which allows network data to be sent simultaneously to all hosts on a given subnetwork rather
                  than specifying a particular host.  The standard general broadcast address for IP networks is
                  255.255.255.255, but this broadcast address cannot be used to send a broadcast message to
                  every host on the Internet because routers block it.  A more appropriate broadcast address
                  is set to match a specific subnetwork.  For example, on the private Class C IP network,
                  192.168.1.0, the broadcast address is 192.168.1.255. Broadcast messages are
                  typically produced by network protocols such as the Address Resolution Protocol (ARP) and the 
                  Routing Information Protocol (RIP).             
                  </para>
               </listitem>
               <listitem>
                  <para>
                  <emphasis role="bold">Gateway Address</emphasis> A Gateway Address is the IP address through which
                  a particular network, or host on a network, may be reached.  If one network host wishes to communicate
                  with another network host, and that host is not located on the same network, then a 
                  <emphasis>gateway</emphasis> must be used. In many cases, the Gateway Address will be that
                  of a router on the same network, which will in turn pass traffic on to other networks or hosts, such as
                  Internet hosts.  The value of the Gateway Address setting must be correct, or your system will not be able
                  to reach any hosts beyond those on the same network.                  
                  </para>
               </listitem>
               <listitem>
                  <para>
                  <emphasis role="bold">Nameserver Address</emphasis> Nameserver Addresses represent the IP addresses of
                  Domain Name Service (DNS) systems, which resolve network hostnames into IP addresses.  There are three levels of
                  Nameserver Addresses, which may be specified in order of precedence: The 
                  <emphasis>Primary</emphasis>
                  Nameserver, the <emphasis>Secondary</emphasis> Nameserver, and the 
                  <emphasis>Tertiary</emphasis>
                  Nameserver. In order for your system to be able to resolve network hostnames into their
                  corresponding IP addresses, you must specify valid Nameserver Addresses which you are authorized to use
                  in your system's TCP/IP configuration.  In many cases these addresses can and will be provided by your
                  network service provider, but many free and publicly accessible nameservers are available for use, such as
                  the Level3 (Verizon) servers with IP addresses from
                  4.2.2.1 to 4.2.2.6. </para>
                     <tip>
                        <para>
                        The IP address, Netmask, Network Address, Broadcast Address, Gateway Address, and Nameserver
                        Addresses are typically specified via the appropriate directives in the file
                        <filename>/etc/network/interfaces</filename>. For more information, view the system manual
                        page for <filename>interfaces</filename>, with the following command typed at a terminal prompt:
                        </para>
                     </tip>
                    <para>
                    Access the system manual page for <filename>interfaces</filename> with the following command:
                    </para>
                    <para>
<screen>
<command>man interfaces</command>
</screen>
                    </para>
               </listitem>
            </itemizedlist>
            </para>
        </sect2>
        <sect2 id="ip-routing" status="review">
          <title>IP Routing</title>
            <para>
              IP routing is a means of specifying and discovering paths in a TCP/IP network along which
              network data may be sent.  Routing uses a set of <emphasis>routing tables</emphasis>
              to direct the forwarding of network data packets from their source to the destination, often
              via many intermediary network nodes known as <emphasis>routers</emphasis>.
              There are two primary forms of
              IP routing: <emphasis>Static Routing</emphasis> and 
              <emphasis>Dynamic Routing.</emphasis>
            </para>
            <para>
            Static routing involves manually adding IP routes to the system's routing table, and this is usually
            done by manipulating the routing table with the <application>route</application> command. Static routing enjoys
            many advantages over dynamic routing, such as simplicity of implementation on smaller networks, 
            predictability (the routing table is always computed in advance, and thus the route is precisely the 
            same each time it is used), and low overhead on other routers and network links due to the lack of a
            dynamic routing protocol.  However, static routing does present some disadvantages as well.  For example,
            static routing is limited to small networks and does not scale well.  Static routing also fails completely
            to adapt to network outages and failures along the route due to the fixed nature of the route. 
            </para>
            <para>
            Dynamic routing depends on large networks with multiple possible IP routes from a source to a destination
            and makes use of special routing protocols, such as the Router Information Protocol (RIP), which handle
            the automatic adjustments in routing tables that make dynamic routing possible.  Dynamic routing
            has several advantages over static routing, such as superior scalability and the ability to adapt
            to failures and outages along network routes. Additionally, there is less manual configuration of the
            routing tables, since routers learn from one another about their existence and available routes. This trait
            also eliminates the possibility of introducing mistakes in the routing tables via human error.
            Dynamic routing is not perfect, however, and presents disadvantages such as heightened complexity and
            additional network overhead from router communications, which does not immediately benefit the end users,
            but still consumes network bandwidth.
            </para>
        </sect2>
        <sect2 id="tcp-and-udp" status="review">
          <title>TCP and UDP</title>
            <para>
              TCP is a connection-based protocol, offering error correction and guaranteed delivery of
              data via what is known as <emphasis>flow control</emphasis>. Flow control
              determines when the flow of a data stream needs to be stopped, and previously sent data
              packets should to be re-sent due to problems such as <emphasis>collisions</emphasis>,
              for example, thus ensuring complete and accurate delivery of the data.  TCP is typically
              used in the exchange of important information such as database transactions. 
            </para>
            <para>
              The User Datagram Protocol (UDP), on the other hand, is a <emphasis>connectionless</emphasis>
              protocol which seldom deals with the transmission of important data because it lacks flow
              control or any other method to ensure reliable delivery of the data.  UDP is commonly used
              in such applications as audio and video streaming, where it is considerably faster than
              TCP due to the lack of error correction and flow control, and where the loss of a few packets
			  is not generally catastrophic.
            </para>
        </sect2>
        <sect2 id="icmp" status="review">
          <title>ICMP</title>
            <para>
              The Internet Control Messaging Protocol (ICMP) is an extension to the Internet Protocol (IP) as defined
              in the Request For Comments (RFC) #792 and supports network packets containing control, error, and
              informational messages.  ICMP is used by such network applications as the <application>ping</application>
              utility, which can determine the availability of a network host or device.  Examples of some error messages
              returned by ICMP which are useful to both network hosts and devices such as routers, include 
              <emphasis>Destination Unreachable</emphasis> and <emphasis>Time Exceeded</emphasis>.
            </para>
        </sect2>
        <sect2 id="daemons" status="review">
          <title>Daemons</title>
            <para>
              Daemons are special system applications which typically execute continuously in the background and
              await requests for the functions they provide from other applications.  Many daemons are network-centric; that is,
              a large number of daemons executing in the background on an Ubuntu system may provide network-related functionality.
              Some examples of such network daemons include the <emphasis>Hyper Text Transport Protocol Daemon</emphasis> 
              (httpd), which provides web server functionality; the <emphasis>Secure SHell Daemon</emphasis> (sshd), which
              provides secure remote login shell and file transfer capabilities; and the <emphasis>Internet Message Access 
              Protocol Daemon</emphasis> (imapd), which provides E-Mail services.
            </para>
        </sect2>
        <sect2 id="tcpip-resources" status="review">
          <title>Resources</title>

          <itemizedlist>
            <listitem>
              <para>
              There are man pages for <ulink url="http://manpages.ubuntu.com/manpages/&distro-short-codename;/en/man7/tcp.7.html">TCP</ulink> and
              <ulink url="http://manpages.ubuntu.com/manpages/&distro-short-codename;/man7/ip.7.html">IP</ulink> that contain more useful information.
              </para>
            </listitem>
            <listitem>
              <para>
              Also, see the <ulink url="http://www.redbooks.ibm.com/abstracts/gg243376.html">TCP/IP Tutorial and Technical Overview</ulink>
              IBM Redbook.
              </para>
            </listitem>
            <listitem>
              <para>
              Another resource is O'Reilly's <ulink url="http://oreilly.com/catalog/9780596002978/">TCP/IP Network Administration</ulink>.
              </para>
            </listitem>
          </itemizedlist>
        </sect2>
    </sect1>

	<sect1 id="dhcp" status="review">
		<title>Dynamic Host Configuration Protocol (DHCP)</title>
            <para>
	    The Dynamic Host Configuration Protocol (DHCP) is a network service that enables 
	    host computers to be automatically assigned settings from a server as opposed to
            manually configuring each network host. Computers configured to be DHCP clients have
            no control over the settings they receive from the DHCP server, and the 
	    configuration is transparent to the computer's user.
	    </para>
            <para>
	    The most common settings provided by a DHCP server to DHCP clients include:
	    </para>
            <itemizedlist spacing="compact">
                <listitem>
                    <para>IP address and netmask</para>
                </listitem>
                <listitem>
                    <para>IP address of the default-gateway to use</para>
                </listitem>
                <listitem>
                    <para>IP adresses of the DNS servers to use</para>
                </listitem>
            </itemizedlist>
            <para>
	    However, a DHCP server can also supply configuration properties such as:
	    </para>
            <itemizedlist spacing="compact">
                <listitem>
                    <para>Host Name</para>
                </listitem>
                <listitem>
                    <para>Domain Name</para>
                </listitem>
                <listitem>
                    <para>Time Server</para>
                </listitem>
                <listitem>
                    <para>Print Server</para>
                </listitem>
            </itemizedlist>
            <para>
	    The advantage of using DHCP is that changes to the network, for example a change
            in the address of the DNS server, need only be changed at the DHCP server, and all
            network hosts will be reconfigured the next time their DHCP clients poll the
            DHCP server. As an added advantage, it is also easier to integrate new computers 
	    into the network, as there is no need to check for the availability of an IP 
	    address.  Conflicts in IP address allocation are also reduced.
	    </para>
            <para>
	    A DHCP server can provide configuration settings using the following methods:
	    </para>
            <variablelist>
                <varlistentry>
                    <term>Manual allocation (MAC address)</term>
                    <listitem>
                        <para>
			This method entails using DHCP to identify the unique hardware address
                        of each network card connected to the network and then continually
                        supplying a constant configuration each time the DHCP client makes a
                        request to the DHCP server using that network device. This ensures that
			a particular address is assigned automatically to that network card, 
			based on it's MAC address.
			</para>
                    </listitem>
                </varlistentry>
                <varlistentry>
                    <term>Dynamic allocation (address pool)</term>
                    <listitem>
                        <para>
			In this method, the DHCP server will assign an IP address from a pool of addresses
			(sometimes also called a range or scope) for a period of time or lease, that is 
			configured on the server or until the client informs the server that it doesn't 
			need the address anymore. This way, the clients will be receiving their configuration
			properties dynamically and on a "first come, first served" basis. When a DHCP client
			is no longer on the network for a specified period, the configuration is expired and 
			released back to the address pool for use by other DHCP Clients. This way, an address
			can be leased or used for a period of time. After this period, the client has to 
			renegociate the lease with the server to maintain use of the address.
			</para>
                    </listitem>
                </varlistentry>
                <varlistentry>
                    <term>Automatic allocation</term>
                    <listitem>
                        <para>
			Using this method, the DHCP automatically assigns an IP address permanently to a device,
			selecting it from a pool of available addresses. Usually DHCP is used to assign a 
			temporary address to a client, but a DHCP server can allow an infinite lease time.
			</para>
                    </listitem>
                </varlistentry>
            </variablelist>
            <para>
	    The last two methods can be considered "automatic" because in each case the DHCP server
            assigns an address with no extra intervention needed. The only difference between them
	    is in how long the IP address is leased, in other words whether a client's address varies
	    over time.

	    Ubuntu is shipped with both DHCP server and client. The server is
            <application>dhcpd</application> (dynamic host configuration protocol daemon).
            The client provided with Ubuntu is <application>dhclient</application> and should 
	    be installed on all computers required to be automatically configured. Both 
	    programs are easy to install and configure and will be automatically started at
	    system boot.
	    </para>
      <sect2 id="dhcp-installation" status="review">
        <title>Installation</title>
          <para>
	  At a terminal prompt, enter the following command to install
          <application>dhcpd</application>:
	  </para>
<screen>
<command>sudo apt install isc-dhcp-server</command>
</screen>
          <para>
	  You will probably need to change the default configuration
	  by editing /etc/dhcp/dhcpd.conf to suit your needs and particular configuration.
	  </para>
          <para>
	  You also may need to edit /etc/default/isc-dhcp-server to specify the interfaces dhcpd
	  should listen to.
	  </para>
          <para>
	  NOTE: dhcpd's messages are being sent to syslog. Look there for
	  diagnostics messages.
	  </para>
      </sect2>
      <sect2 id="dhcp-configuration" status="review">
        <title>Configuration</title>
            <para>
	    The error message the installation ends with might be a little confusing, but the
            following steps will help you configure the service:
	    </para>
            <para>
	    Most commonly, what you want to do is assign an IP address randomly. This can be
            done with settings as follows:
	    </para>
<programlisting>
# minimal sample /etc/dhcp/dhcpd.conf
default-lease-time 600;
max-lease-time 7200;

subnet 192.168.1.0 netmask 255.255.255.0 {
 range 192.168.1.150 192.168.1.200;
 option routers 192.168.1.254;
 option domain-name-servers 192.168.1.1, 192.168.1.2;
 option domain-name "mydomain.example";
} 
</programlisting>
            <para>
	    This will result in the DHCP server giving clients an IP address from the range
            192.168.1.150-192.168.1.200. It will lease an IP
            address for 600 seconds if the client doesn't ask for a specific time frame. 
	    Otherwise the maximum (allowed) lease will be 7200 seconds. The server will also 
	    "advise" the client to use 192.168.1.254 as the default-gateway and 
	    192.168.1.1 and 192.168.1.2 as its DNS servers. 
	    </para>
          <para>
	  After changing the config file you have to restart the
          <application>dhcpd</application>:
	  </para>
<screen>
<command>sudo systemctl restart isc-dhcp-server.service</command>
</screen>
      </sect2>
      <sect2 id="dhcp-references" status="review">
        <title>References</title>

          <itemizedlist>
            <listitem>
              <para>
              The <ulink url="https://help.ubuntu.com/community/dhcp3-server">dhcp3-server Ubuntu Wiki</ulink> page has more information.
              </para>
            </listitem>
            <listitem>
              <para>
              For more <filename>/etc/dhcp/dhcpd.conf</filename> options see the 
              <ulink url="http://manpages.ubuntu.com/manpages/&distro-short-codename;/en/man5/dhcpd.conf.5.html">dhcpd.conf man page</ulink>.
              </para>
            </listitem>
            <listitem>
              <para>
              <ulink url="http://www.isc.org/software/dhcp">ISC dhcp-server</ulink>
              </para>
            </listitem>
          </itemizedlist>

      </sect2>
    </sect1>

	<sect1 id="NTP" status="review">
		<title>Time Synchronisation</title>
	<para>
NTP is a TCP/IP protocol for synchronising time over a network. Basically a client requests the current time from a server, and uses it to set its own clock.  
</para>
	<para>
Behind this simple description, there is a lot of complexity - there are tiers of NTP servers, with the tier one NTP servers connected to atomic clocks, and tier two and three servers spreading the load of actually handling requests across the Internet. Also the client software is a lot more complex than you might think - it has to factor out communication delays, and adjust the time in a way that does not upset all the other processes that run on the server. But luckily all that complexity is hidden from you! 
</para>
	<para>
        Ubuntu by default uses <emphasis>timedatectl / timesyncd</emphasis> to synchronize time and users can optionally use chrony to <xref linkend="timeservers"/>.
</para>

<sect2 id="timedate-info" status="review">
<title>Synchronizing your systems time</title>
	<para>
		Since Ubuntu 16.04 <emphasis>timedatectl / timesyncd</emphasis> (which are part of systemd) replace most of <emphasis>ntpdate / ntp</emphasis>.
	</para>
	<para>
        <application>timesyncd</application> is available by default and replaces not only <application>ntpdate</application>, but also the client portion of <application>chrony</application> (or formerly <application>ntpd</application>).
		So on top of the one-shot action that <application>ntpdate</application> provided on boot and network activation, now <application>timesyncd</application> by default regularly checks and keeps your local time in sync.
		It also stores time updates locally, so that after reboots monotonically advances if applicable.
	</para>
	<para>
		If <application>chrony</application> is installed <application>timedatectl</application> steps back to let chrony do the time keeping.
        That shall ensure that no two time syncing services are fighting.
        While no more recommended to be userd, this also applies to <application>ntpd</application> being installed to retain any kind of old behaviour/config that you had through an upgrade.
		But it also implies that on an upgrade from a former release ntp/ntpdate might still be installed and therefore renders the new systemd based services disabled.
	</para>
	<para>
        <application>ntpdate</application> is considered deprecated in favour of <application>timedatectl</application> (or <application>chrony</application>) and thereby no more installed by default.
        timesyncd will generally do the right thing keeping your time in sync, and <application>chrony</application> will help with more complex cases.
        But if you had oen of a few known special ntpdate use cases, consider the following:
        <itemizedlist>
          <listitem>
            <para>
              If you require a one-shot sync use: <command>chronyd -q</command>
            </para>
          </listitem>
          <listitem>
            <para>
              If you require a one-shot time check, without setting the time use: <command>chronyd -Q</command>
            </para>
          </listitem>
        </itemizedlist>
	</para>

<sect3 id="timedate-config" status="review">
<title>Configuring timedatectl and timesyncd</title>
	<para>
		The current status of time and time configuration via <application>timedatectl</application> and <application>timesyncd</application> can be checked with <command>timedatectl status</command>.
	</para>
      <screen>
$ timedatectl status
                       Local time: Fr 2018-02-23 08:47:13 UTC
                   Universal time: Fr 2018-02-23 08:47:13 UTC
                         RTC time: Fr 2018-02-23 08:47:13
                        Time zone: Etc/UTC (UTC, +0000)
        System clock synchronized: yes
 systemd-timesyncd.service active: yes
                  RTC in local TZ: no

If chrony is running it will automatically switch to:
[...]
 systemd-timesyncd.service active: no
      </screen>
<para>
Via <application>timedatectl</application> an admin can control the timezone, how the system clock should relate to the hwclock and if permanent synronization should be enabled or not.
See <command>man timedatectl</command> for more details.
</para>
<para>
        timesyncd itself is still a normal service, so you can check its status also more in detail via.
<screen>
$ systemctl status systemd-timesyncd
  systemd-timesyncd.service - Network Time Synchronization
   Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2018-02-23 08:55:46 UTC; 10s ago
     Docs: man:systemd-timesyncd.service(8)
 Main PID: 3744 (systemd-timesyn)
   Status: "Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com)."
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/systemd-timesyncd.service
           └─3744 /lib/systemd/systemd-timesyncd

Feb 23 08:55:46 bionic-test systemd[1]: Starting Network Time Synchronization...
Feb 23 08:55:46 bionic-test systemd[1]: Started Network Time Synchronization.
Feb 23 08:55:46 bionic-test systemd-timesyncd[3744]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).
</screen>
</para>
<para>
        The nameserver to fetch time for <application>timedatectl</application> and <application>timesyncd</application> from can be specified in <filename>/etc/systemd/timesyncd.conf</filename> and additional config files can be stored in <filename>/etc/systemd/timesyncd.conf.d/</filename>.
        The entries for NTP= and FallbackNTP= are space separated lists.
        See <command>man timesyncd.conf</command> for more.
</para>
</sect3>

</sect2>

<sect2 id="timeservers" status="review">
   <title>Serve the Network Time Protocol</title>
   <para>
       If in addition to synchronizing your system you also want to serve NTP information you need an NTP server. There are several options with <application>chrony</application>, <application>ntpd</application> and <application>open-ntp</application>.
       The recommended solution <application>chrony</application>.
   </para>

<sect3 id="chrony" status="review">
   <title>chrony(d)</title>
   <para>
   The NTP daemon chronyd calculates the drift and offset of your system clock and continuously adjusts it, so there are no large corrections that could
   lead to inconsistent logs for instance. The cost is a little processing power and memory, but for a modern server this is usually negligible.
   </para>
</sect3>

<sect3 id="chrony-installation" status="review">
   <title>Installation</title>
   <para>
   To install chrony, from a terminal prompt enter:
   </para>
<screen>
<command>sudo apt install chrony</command>
</screen>
   <para>
       This will provide two binaries:
        <itemizedlist>
          <listitem>
            <para>
              chronyd - the actual daemon to sync and serve via the NTP protocol
            </para>
          </listitem>
          <listitem>
            <para>
              chronyc - command-line interface for chrony daemon
            </para>
          </listitem>
        </itemizedlist>
   </para>
</sect3>

<sect3 id="timeservers-conf" status="review">
  <title>Chronyd Configuration</title>

  <para>
  Edit <filename>/etc/ntp.conf</filename> to add/remove server lines.
  By default these servers are configured:
  </para>

<programlisting>
# Use servers from the NTP Pool Project. Approved by Ubuntu Technical Board
# on 2011-02-08 (LP: #104525). See http://www.pool.ntp.org/join.html for
# more information.
server 0.ubuntu.pool.ntp.org
server 1.ubuntu.pool.ntp.org
server 2.ubuntu.pool.ntp.org
server 3.ubuntu.pool.ntp.org
</programlisting>

          <para>
	  After changing the config file you have to reload the
          <application>ntpd</application>:
	  </para>
<screen>
<command>sudo systemctl reload ntp.service</command>
</screen>
<para>
	   Of the pool number 2.ubuntu.pool.ntp.org as well as ntp.ubuntu.com also support ipv6 if needed.
	   If one needs to force ipv6 there also is ipv6.ntp.ubuntu.com which is not configured by default.
</para>

</sect3>

<sect3 id="ntp-status" status="review">
  <title>View status</title>
  <para>
  Use ntpq to see more info: 
  </para>
<screen>
<command># sudo ntpq -p</command>
<computeroutput>     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+stratum2-2.NTP. 129.70.130.70    2 u    5   64  377   68.461  -44.274 110.334
+ntp2.m-online.n 212.18.1.106     2 u    5   64  377   54.629  -27.318  78.882
*145.253.66.170  .DCFa.           1 u   10   64  377   83.607  -30.159  68.343
+stratum2-3.NTP. 129.70.130.70    2 u    5   64  357   68.795  -68.168 104.612
+europium.canoni 193.79.237.14    2 u   63   64  337   81.534  -67.968  92.792</computeroutput>
</screen>

</sect3>

<sect3 id="ntp-pps" status="review">
  <title>PPS Support</title>
  <para>
Since 16.04 ntp supports PPS discipline which can be used to augment ntp with local timesources for better accuracy.
For more details on configuration see the external pps ressource listed below.
  </para>
</sect3>
</sect2>


  <sect2 id="ntp-references" status="review">
    <title>References</title>

	<itemizedlist>
	  <listitem>
  	    <para>
          See the <ulink url="https://help.ubuntu.com/community/UbuntuTime">Ubuntu Time</ulink> wiki page for more information.
        </para>
      </listitem>
	  <listitem>
  	    <para>
          <ulink url="http://www.ntp.org/">ntp.org, home of the Network Time Protocol project</ulink>
        </para>
      </listitem>
	  <listitem>
  	    <para>
            <ulink url="https://www.freedesktop.org/software/systemd/man/timedatectl.html">Freedesktop.org info on timedatectl</ulink>
        </para>
      </listitem>
	  <listitem>
  	    <para>
            <ulink url="https://www.freedesktop.org/software/systemd/man/systemd-timesyncd.service.html#">Freedesktop.org info on systemd-timesyncd service</ulink>
        </para>
      </listitem>
	  <listitem>
  	    <para>
		    <ulink url="http://www.ntp.org/ntpfaq/NTP-s-config-adv.htm#S-CONFIG-ADV-PPS">ntp.org faq on configuring PPS</ulink>
        </para>
      </listitem>
    </itemizedlist>

  </sect2>
  </sect1>

  <sect1 id="DPDK" status="review">
	  <title>Data Plane Development Kit</title>
	  <para>
		  The DPDK is a set of libraries and drivers for fast packet processing and runs mostly in Linux userland.
		  It is a set of libraries that provide the so called "Environment Abstraction Layer" (EAL).
		  The EAL hides the details of the environment and provides a standard programming interface.
		  Common use cases are around special solutions for instance network function virtualization
		  and advanced high-throughput network switching.
		  The DPDK uses a run-to-completion model for fast data plane performance and accesses devices via polling
		  to eliminate the latency of interrupt processing at the tradeoff of higher cpu consumption.
		  It was designed to run on any processors. The first supported CPU was Intel x86 and it is
		  now extended to IBM Power 8, EZchip TILE-Gx and ARM.
	  </para>
	  <para>
		  Ubuntu currently supports DPDK version 2.2 and provides some infrastructure to ease its usability.
	  </para>

	  <sect2 id="dpdk-prerequisites" status="review">
		  <title>Prerequisites</title>
		  <para>
			  This package is currently compiled for the lowest possible CPU requirements. Which still requires at least SSE3 to be supported by the CPU.
		  </para>
		  <para>
			  The list of upstream DPDK supported network cards can be found at <ulink url="http://dpdk.org/doc/nics">supported NICs</ulink>.
			  But a lot of those are disabled by default in the upstream Project as they are not yet in a stable state.
			  The subset of network cards that DPDK has enabled in the package as available in Ubuntu 16.04 is:
		  </para>
		  <para>
			  Intel
		  </para>
			  <itemizedlist>
				  <listitem>
					  <para><ulink url="http://dpdk.org/doc/guides/nics/e1000em.html">e1000</ulink> (82540, 82545, 82546)</para>
				  </listitem>
				  <listitem>
					  <para><ulink url="http://dpdk.org/browse/dpdk/tree/drivers/net/e1000/">e1000e</ulink> (82571..82574, 82583, ICH8..ICH10, PCH..PCH2)</para>
				  </listitem>
				  <listitem>
					  <para><ulink url="http://dpdk.org/browse/dpdk/tree/drivers/net/e1000/">igb</ulink> (82575..82576, 82580, I210, I211, I350, I354, DH89xx)</para>
				  </listitem>
				  <listitem>
					  <para><ulink url="http://dpdk.org/doc/guides/nics/ixgbe.html">ixgbe</ulink> (82598..82599, X540, X550)</para>
				  </listitem>
				  <listitem>
					  <para><ulink url="http://dpdk.org/browse/dpdk/tree/drivers/net/i40e/">i40e</ulink> (X710, XL710, X722)</para>
				  </listitem>
				  <listitem>
					  <para><ulink url="http://dpdk.org/doc/guides/nics/fm10k.html">fm10k</ulink> (FM10420)</para>
				  </listitem>
			  </itemizedlist>
		  <para>
			  Chelsio
		  </para>
			  <itemizedlist>
				  <listitem>
					  <para><ulink url="http://dpdk.org/doc/guides/nics/cxgbe.html">cxgbe</ulink> (Terminator 5)</para>
				  </listitem>
			  </itemizedlist>
		  <para>
			  Cisco
		  </para>
			  <itemizedlist>
				  <listitem>
					  <para><ulink url="http://dpdk.org/browse/dpdk/tree/drivers/net/enic">enic</ulink> (UCS Virtual Interface Card)</para>
				  </listitem>
			  </itemizedlist>
		  <para>
			  Paravirtualization
		  </para>
			  <itemizedlist>
				  <listitem>
					  <para><ulink url="http://dpdk.org/doc/guides/nics/virtio.html">virtio-net</ulink> (QEMU)</para>
				  </listitem>
				  <listitem>
					  <para><ulink url="http://dpdk.org/doc/guides/nics/vmxnet3.html">vmxnet3</ulink></para>
				  </listitem>
			  </itemizedlist>
		  <para>
			  Others
		  </para>
			  <itemizedlist>
				  <listitem>
					  <para><ulink url="http://dpdk.org/browse/dpdk/tree/drivers/net/af_packet">af_packet</ulink> (Linux AF_PACKET socket)</para>
				  </listitem>
				  <listitem>
					  <para><ulink url="http://dpdk.org/doc/guides/nics/pcap_ring.html#rings-based-pmd">ring</ulink> (memory)</para>
				  </listitem>
			  </itemizedlist>
		  <para>
			  On top it experimentally enables the following two PMD drivers as they represent (virtual) devices that are very accessible to end users.
		  </para>
		  <para>
			  Paravirtualization
		  </para>
			  <itemizedlist>
				  <listitem>
					  <para><ulink url="http://dpdk.org/doc/guides/xen/pkt_switch.html#xen-pmd-frontend-prerequisites">xenvirt</ulink> (Xen)</para>
				  </listitem>
			  </itemizedlist>
		  <para>
			  Others
		  </para>
			  <itemizedlist>
				  <listitem>
					  <para><ulink url="http://dpdk.org/doc/guides/nics/pcap_ring.html#libpcap-based-pmd">pcap</ulink> (file or kernel driver)</para>
				  </listitem>
			  </itemizedlist>
		  <para>
			  Cards have to be unassigned from their kernel driver and instead be assigned to uio_pci_generic of vfio-pci.
			  uio_pci_generic is older and usually getting to work more easily.
		  </para>
		  <para>
			  The newer vfio-pci requires that you activate the following kernel parameters to enable iommu.
		  </para>
		  <programlisting>
iommu=pt intel_iommu=on
		  </programlisting>
		  <para>
			  On top for vfio-pci you then have to configure and assign the iommu groups accordingly.
		  </para>
		  <para>
			  Note: In virtio based environment it is enough to "unassign" devices from the kernel driver.
			  Without that DPDK will reject to use the device to avoid issues with kernel and DPDK working on the device at the same time.
			  Since DPDK can work directly on virtio devices it is not required to assign e.g. uio_pci_generic to those devices.
		  </para>
		  <para>
			  Manual configuration and status checks can be done via sysfs or with the tool dpdk_nic_bind
		  </para>
		  <programlisting>
dpdk_nic_bind --help

Usage:
------

     dpdk_nic_bind [options] DEVICE1 DEVICE2 ....

     where DEVICE1, DEVICE2 etc, are specified via PCI "domain:bus:slot.func" syntax
     or "bus:slot.func" syntax. For devices bound to Linux kernel drivers, they may
     also be referred to by Linux interface name e.g. eth0, eth1, em0, em1, etc.

     Options:
         --help, --usage:
         Display usage information and quit

     -s, --status:
             Print the current status of all known network interfaces.
             For each device, it displays the PCI domain, bus, slot and function,
             along with a text description of the device. Depending upon whether the
             device is being used by a kernel driver, the igb_uio driver, or no
             driver, other relevant information will be displayed:
             * the Linux interface name e.g. if=eth0
             * the driver being used e.g. drv=igb_uio
             * any suitable drivers not currently using that device
                 e.g. unused=igb_uio
         NOTE: if this flag is passed along with a bind/unbind option, the status
         display will always occur after the other operations have taken place.

     -b driver, --bind=driver:
             Select the driver to use or "none" to unbind the device

         -u, --unbind:
         Unbind a device (Equivalent to "-b none")

     --force:
             By default, devices which are used by Linux - as indicated by having
             routes in the routing table - cannot be modified. Using the --force
             flag overrides this behavior, allowing active links to be forcibly
             unbound.
             WARNING: This can lead to loss of network connection and should be used
             with caution.

     Examples:
     ---------

     To display current device status:
             dpdk_nic_bind --status

     To bind eth1 from the current driver and move to use igb_uio
             dpdk_nic_bind --bind=igb_uio eth1

     To unbind 0000:01:00.0 from using any driver
             dpdk_nic_bind -u 0000:01:00.0

     To bind 0000:02:00.0 and 0000:02:00.1 to the ixgbe kernel driver
             dpdk_nic_bind -b ixgbe 02:00.0 02:00.
</programlisting>
	  </sect2>

	  <sect2 id="dpdk-config-dev" status="review">
		  <title>DPDK Device configuration</title>
		  <para>
              The package <emphasis>dpdk</emphasis> provides init scripts that ease configuration of device assignment and huge pages.
			  It also makes them persistent accross reboots.
		  </para>
		  <para>
			  The following is an example of the file /etc/dpdk/interfaces configuring two ports of a network card.
			  One with uio_pci_generic and the other one with vfio-pci
		  </para>
		  <programlisting>
# &lt;bus&gt;         Currently only &quot;pci&quot; is supported
# &lt;id&gt;          Device ID on the specified bus
# &lt;driver&gt;      Driver to bind against (vfio-pci or uio_pci_generic)
#
# Be aware that the two DPDK compatible drivers uio_pci_generic and vfio-pci are
# part of linux-image-extra-&lt;VERSION&gt; package.
# This package is not always installed by default - for example in cloud-images.
# So please install it in case you run into missing module issues.
#
# &lt;bus&gt; &lt;id&gt;     &lt;driver&gt;
pci 0000:04:00.0 uio_pci_generic
pci 0000:04:00.1 vfio-pci
		  </programlisting>
		  <para>
			  Cards are identified by their PCI-ID. If you are unsure you might use the tool dpdk_nic_bind to show the current available devices and the drivers they are assigned to.
		  </para>
		  <screen>
<command>dpdk_nic_bind --status</command>
<computeroutput>
Network devices using DPDK-compatible driver
============================================
0000:04:00.0 &apos;Ethernet Controller 10-Gigabit X540-AT2&apos; drv=uio_pci_generic unused=ixgbe

Network devices using kernel driver
===================================
0000:02:00.0 &apos;NetXtreme BCM5719 Gigabit Ethernet PCIe&apos; if=eth0 drv=tg3 unused=uio_pci_generic *Active*
0000:02:00.1 &apos;NetXtreme BCM5719 Gigabit Ethernet PCIe&apos; if=eth1 drv=tg3 unused=uio_pci_generic 
0000:02:00.2 &apos;NetXtreme BCM5719 Gigabit Ethernet PCIe&apos; if=eth2 drv=tg3 unused=uio_pci_generic 
0000:02:00.3 &apos;NetXtreme BCM5719 Gigabit Ethernet PCIe&apos; if=eth3 drv=tg3 unused=uio_pci_generic 
0000:04:00.1 &apos;Ethernet Controller 10-Gigabit X540-AT2&apos; if=eth5 drv=ixgbe unused=uio_pci_generic 

Other network devices
=====================
&lt;none&gt;
</computeroutput>
		  </screen>
	  </sect2>

	  <sect2 id="dpdk-config-hp" status="review">
		  <title>DPDK HugePage configuration</title>
		  <para>
			  DPDK makes heavy use of huge pages to eliminate pressure on the TLB.
			  Therefore hugepages have to be configured in your system.
		  </para>
		  <para>
              The <emphasis>dpdk</emphasis> package has a config file and scripts that try to ease hugepage configuration for DPDK in the form of <emphasis>/etc/dpdk/dpdk.conf</emphasis>.
			  If you have more consumers of hugepages than just DPDK in your system or very special requirements how your hugepages are going to be set up you likely want to allocate/control them by yourself.
			  If not this can be a great simplification to get DPDK configured for your needs.
		  </para>
		  <para>
			  Here an example configuring 1024 Hugepages of 2M each and 4 1G pages.
		  </para>
		  <programlisting>
NR_2M_PAGES=1024
NR_1G_PAGES=4
		  </programlisting>
		  <para>
			  As shown this supports configuring 2M and the larger 1G hugepages (or a mix of both).
			  It will make sure there are proper hugetlbfs mountpoints for DPDK to find both sizes no matter what your default huge page size is.
			  The config file itself holds more details on certain corner cases and a few hints if you want to allocate hugepages manually via a kernel parameter.
		  </para>
		  <para>
			  It depends on your needs which size you want - 1G pages are certainly more effective regarding TLB pressure.
			  But there were reports of them fragmenting inside the DPDK memory alloactions.
			  Also it can be harder to grab enough free space to set up a certain amount of 1G pages later in the lifecycle of a system.
		  </para>
	  </sect2>

	  <sect2 id="dpdk-apps" status="review">
		  <title>Compile DPDK Applications</title>
		  <para>
			  Currently there are not a lot consumers of the DPDK library that are stable and released.
			  OpenVswitch-DPDK being an exception to that (see below), but in general it is very likely that you might want / have to compile an app against the library.
		  </para>
		  <para>
			  You will often find guides that tell you to fetch the DPDK sources, build them to your needs and eventually build your application based on DPDK by setting values RTE_* for the build system.
			  Since Ubunutu provides an already compiled DPDK for you can can skip all that.
			  To simplify setting the proper variables you can source the file /usr/share/dpdk/dpdk-sdk-env.sh before building your application.
			  Here an excerpt building the l2fwd example application delivered with the dpdk-doc package.
		  </para>
		  <programlisting>
sudo apt-get install dpdk-dev libdpdk-dev
. /usr/share/dpdk/dpdk-sdk-env.sh
make -C /usr/share/dpdk/examples/l2fwd
		  </programlisting>
		  <para>
			  Depending on what you build it might be a good addition to install all of DPDK build dependencies before the make.
		  </para>
		  <programlisting>
sudo apt-get install build-dep dpdk
		  </programlisting>
	  </sect2>

	  <sect2 id="dpdk-openvswitch" status="review">
		  <title>OpenVswitch-DPDK</title>
		  <para>
			  Being a library it doesn't do a lot on its own, so it depends on emerging projects making use of it.
			  One consumer of the library that already is bundled in the Ubuntu 16.04 release is OpenVswitch with DPDK support in the package openvswitch-switch-dpdk.
		  </para>
		  <para>
			  Here an example how to install and configure a basic OpenVswitch using DPDK for later use via libvirt/qemu-kvm.
		  </para>
		  <programlisting>
sudo apt-get install openvswitch-switch-dpdk
sudo update-alternatives --set ovs-vswitchd /usr/lib/openvswitch-switch-dpdk/ovs-vswitchd-dpdk
echo "DPDK_OPTS='--dpdk -c 0x1 -n 4 -m 2048 --vhost-owner libvirt-qemu:kvm --vhost-perm 0664'" | sudo tee -a /etc/default/openvswitch-switch
sudo service openvswitch-switch restart
		  </programlisting>
		  <para>
			  Please remember that you have to assign devices to DPDK compatible drivers (see above) before restarting.
		  </para>
		  <para>
			  The section <emphasis>--vhost-owner libvirt-qemu:kvm --vhost-perm 0664</emphasis> will set vhost_user ports up with owner/permissions to be compatible with Ubuntus way of running qemu-kvm/libvirt with reduced privileges for more security.
		  </para>
		  <para>
			  Please note that the section <emphasis>-m 2048</emphasis> is the most basic numa setup for a single socket system.
			  If you have multiple sockets you might want to define how to split your memory among them, for example <emphasis>-m 1024, 1024</emphasis>.
			  Please be aware that DPDK will try to work only with local memory to the network cards it works with (for performance reasons).
			  That said if you have multiple nodes, but all network cards on one, you should consider spreading your cards.
			  If not at least allocate your memory to the node where the cards reside, for example in a two node all to node #2: <emphasis>-m 0, 2048</emphasis>.
			  You can use the tool <emphasis>lstopo</emphasis> from the package <emphasis>hwloc-nox</emphasis> to see on which socket your cards are located.
		  </para>
		  <para>
			  The OpenVswitch you now started supports all port types OpenVswitch usually does, plus DPDK port types.
			  Here an example how to create a bridge and - instead of a normal external port - add an external DPDK port to it.
		  </para>
		  <programlisting>
ovs-vsctl add-br ovsdpdkbr0 -- set bridge ovsdpdkbr0 datapath_type=netdev
ovs-vsctl add-port ovsdpdkbr0 dpdk0 -- set Interface dpdk0 type=dpdk
		  </programlisting>
          <note>
		    <para>
                The enablement of DPDK in Open vSwitch has changed in version 2.6.
                So for users of releases >=16.10, but also for users of the <ulink url="https://wiki.ubuntu.com/OpenStack/CloudArchive">Ubuntu Cloud Archive</ulink> >=neutron the enablement has changed compared to that for users of Ubuntu 16.04.
                The options formerly passed via <emphasis>DPDK_OPTS</emphasis> are now configured via ovs-vsctl into the Open vSwitch configuration database.
            </para>
		    <para>
                The same example as above would in the new way look like:
            </para>
		  <programlisting>
# Enable DPDK
ovs-vsctl set Open_vSwitch . "other_config:dpdk-init=true"
# run on core 0
ovs-vsctl set Open_vSwitch . "other_config:dpdk-lcore-mask=0x1"
# Allocate 2G huge pages (not Numa node aware)
ovs-vsctl set Open_vSwitch . "other_config:dpdk-alloc-mem=2048"
# group/permissions for vhost-user sockets (required to work with libvirt/qemu)
ovs-vsctl set Open_vSwitch . \
   "other_config:dpdk-extra=--vhost-owner libvirt-qemu:kvm --vhost-perm 0666"
		  </programlisting>
		    <para>
                Please see the associated upstream documentation and the man page of the
                vswitch configuration as provided by the package for more details:
                <itemizedlist>
                    <listitem><para><filename>/usr/share/doc/openvswitch-common/INSTALL.DPDK.md.gz</filename></para></listitem>
                    <listitem><para><filename>/usr/share/doc/openvswitch-common/INSTALL.DPDK-ADVANCED.md.gz</filename></para></listitem>
                    <listitem><para><command>man ovs-vswitchd.conf.db</command></para></listitem>
                </itemizedlist>
            </para>
          </note>
	  </sect2>

	  <sect2 id="dpdk-openvswitch-guest" status="review">
		  <title>OpenVswitch DPDK to KVM Guests</title>
		  <para>
			  If you are not building some sort of SDN switch or NFV on top of DPDK it is very likely that you want to forward traffic to KVM guests.
			  The good news is, that with the new qemu/libvirt/dpdk/openvswitch versions in Ubuntu 16.04 this is no more about manually appending commandline string.
			  This chapter covers a basic configuration how to connect a KVM guest to a OpenVswitch-DPDK instance.
		  </para>
		  <para>
			  The Guest has to be backed by shared hugepages for DPDK/vhost_user to work.
			  To ensure in general that libvirt/qemu-kvm finds a proper hugepage mountpoint you can just enable KVM_HUGEPAGES in /etc/default/qemu-kvm.
			  Afterwards restart the service to pick up the changed configuration.
		  </para>
		  <programlisting>
sed -ri -e 's,(KVM_HUGEPAGES=).*,\11,' /etc/default/qemu-kvm
service qemu-kvm restart
		  </programlisting>
		  <para>
			  To let a guest be backed by hugepages is now also supported via recent libvirt, just add the following snippet to your virsh xml (or the equivalent libvirt interface you use).
			  Those xmls can also be used as templates to easily spawn guests with "uvt-kvm create".
		  </para>
		  <programlisting>
&lt;numa&gt;
&lt;cell id=&apos;0&apos; cpus=&apos;0&apos; memory=&apos;6291456&apos; unit=&apos;KiB&apos; memAccess=&apos;shared&apos;/&gt;
&lt;/numa&gt;
[...]
&lt;memoryBacking&gt;
&lt;hugepages&gt;
&lt;page size="2" unit="M" nodeset="0"/&gt;
&lt;/hugepages&gt;
&lt;/memoryBacking&gt;
		  </programlisting>
		  <para>
			  The new and recommended way to get to a KVM guest is using vhost_user.
			  This will cause DPDK to create a socket that qemu will connect the guest to.
			  Here an example how to add such a port to the bridge you created (see above).
		  </para>
		  <programlisting>
ovs-vsctl add-port ovsdpdkbr0 vhost-user-1 -- set Interface vhost-user-1 type=dpdkvhostuser
		  </programlisting>
		  <para>
			  This will create a vhost_user socket at /var/run/openvswitch/vhost-user-1
		  </para>
		  <para>
			  To let libvirt/kvm consume this socket and create a guest virtio network device for it add a snippet like this to your guest definition as the network definition.
		  </para>
		  <programlisting>
&lt;interface type=&apos;vhostuser&apos;&gt;
&lt;source type=&apos;unix&apos;
path=&apos;/var/run/openvswitch/vhost-user-1&apos;
mode=&apos;client&apos;/&gt;
&lt;model type=&apos;virtio&apos;/&gt;
&lt;/interface&gt;
		  </programlisting>
	  </sect2>

	  <sect2 id="dpdk-in-guest" status="review">
		  <title>DPDK in KVM Guests</title>
		  <para>
			  If you have no access to DPDK supported network cards you can still work with DPDK by using its support for virtio.
			  To do so you have to create guests backed by hugepages (see above).
		  </para>
		  <para>
			  On top of that there it is required to have at least SSE3.
			  The default CPU model qemu/libvirt uses is only up to SSE2.
			  So you will have to define a model that passed the proper feature flag - and of course have a Host system that supportes it.
			  An example can be found in following snippet to your virsh xml (or the equivalent virsh interface you use).
		  </para>
		  <programlisting>
&lt;cpu mode=&apos;host-passthrough&apos;&gt;
		  </programlisting>
		  <para>
			  This example is rather offensive and passes all host features.
			  That in turn makes the guest not very migratable as the target would need all the features as well.
			  A "softer" way is to just add sse3 to the default model like the following example.
		  </para>
		  <programlisting>
&lt;cpu mode=&apos;custom&apos; match=&apos;exact&apos;&gt;
&lt;model fallback=&apos;allow&apos;&gt;qemu64&lt;/model&gt;
&lt;feature policy=&apos;require&apos; name=&apos;ssse3&apos;/&gt;
&lt;/cpu&gt;
		  </programlisting>
		  <para>
			  Also virtio nowadays supports multiqueue which DPDK in turn can exploit for better speed.
			  To modify a normal virtio definition to have multiple queues add the following to your interface definition.
			  This is about enhancing a normal virtio nic to have multiple queues, to later on be consumed e.g. by DPDK in the guest.
		  </para>
		  <programlisting>
&lt;driver name=&quot;vhost&quot; queues=&quot;4&quot;/&gt;
		  </programlisting>
	  </sect2>

	  <sect2 id="ovs-dpdk-tuning" status="review">
		  <title>Tuning Openvswitch-DPDK</title>
		  <para>
			  DPDK has plenty of options - in combination with Openvswitch-DPDK the two most commonly used are:
		  </para>
		  <programlisting>
ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=2
ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x6
		  </programlisting>
		  <para>
			  The first select how many rx queues are to be used for each DPDK interface, while the second controls how many and where to run PMD threads.
			  The example above will utilize two rx queues and run PMD threads on CPU 1 and 2.
			  See the referred links to "EAL Command-line Options" and "OpenVswitch DPDK installation" at the end of this document for more.
		  </para>
		  <para>
			  As usual with tunings you have to know your system and workload really well - so please verify any tunings with workloads matching your real use case.
		  </para>
	  </sect2>

	  <sect2 id="dpdk-support" status="review">
		  <title>Support and Troubleshooting</title>
		  <para>
			  DPDK is a fast evolving project.
			  In any case of a search for support and further guides it is highly recommended to first check if they apply to the current version.
		  </para>

		  <itemizedlist>
			  <listitem>
				  <para>
					  <ulink url="http://dpdk.org/ml">DPDK Mailing Lists</ulink>
				  </para>
			  </listitem>
			  <listitem>
				  <para>
					  For OpenVswitch-DPDK <ulink url="http://openvswitch.org/mlists">OpenStack Mailing Lists</ulink>
				  </para>
			  </listitem>
			  <listitem>
				  <para>
					  Known issues in <ulink url="https://bugs.launchpad.net/ubuntu/+source/dpdk">DPDK Launchpad Area</ulink>
				  </para>
			  </listitem>
			  <listitem>
				  <para>
					  Join the IRC channels #DPDK or #openvswitch on freenode.
				  </para>
			  </listitem>
		  </itemizedlist>

		  <para>
		Issues are often due to missing small details in the general setup. Later on,
		these missing details cause problems which can be hard to track down to their
		root cause.

		A common case seems to be the "could not open network device dpdk0 (No such device)" issue.
		This occurs rather late when setting up a port in Open vSwitch with DPDK.
		But the root cause most of the time is very early in the setup and initialization.
		Here an example how a proper initialization of a device looks - this can be found in
		the syslog/journal when starting Open vSwitch with DPDK enabled.
		  </para>
		  <programlisting>
ovs-ctl[3560]: EAL: PCI device 0000:04:00.1 on NUMA socket 0
ovs-ctl[3560]: EAL:   probe driver: 8086:1528 rte_ixgbe_pmd
ovs-ctl[3560]: EAL:   PCI memory mapped at 0x7f2140000000
ovs-ctl[3560]: EAL:   PCI memory mapped at 0x7f2140200000
		  </programlisting>

		  <para>
		If this is missing, either by ignored cards, failed initialization
		or other reasons, later on there will be no DPDK device to refer to.
		Unfortunately the logging is spread across syslog/journal and the openvswitch log.
		To allow some cross checking here an example what can be found in these logs,
		relative to the entered command.
		  </para>

		  <programlisting>
#Note: This log was taken with dpdk 2.2 and openvswitch 2.5
Captions:
CMD: that you enter
SYSLOG: (Inlcuding EAL and OVS Messages)
OVS-LOG: (Openvswitch messages)

#PREPARATION
Bind an interface to DPDK UIO drivers, make Hugepages available, enable DPDK on OVS

CMD: sudo service openvswitch-switch restart

SYSLOG:
2016-01-22T08:58:31.372Z|00003|daemon_unix(monitor)|INFO|pid 3329 died, killed (Terminated), exiting
2016-01-22T08:58:33.377Z|00002|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2016-01-22T08:58:33.381Z|00003|ovs_numa|INFO|Discovered 12 CPU cores on NUMA node 0
2016-01-22T08:58:33.381Z|00004|ovs_numa|INFO|Discovered 1 NUMA nodes and 12 CPU cores
2016-01-22T08:58:33.381Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2016-01-22T08:58:33.383Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected
2016-01-22T08:58:33.386Z|00007|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.5.0

OVS-LOG:
systemd[1]: Stopping Open vSwitch...
systemd[1]: Stopped Open vSwitch.
systemd[1]: Stopping Open vSwitch Internal Unit...
ovs-ctl[3541]: * Killing ovs-vswitchd (3329)
ovs-ctl[3541]: * Killing ovsdb-server (3318)
systemd[1]: Stopped Open vSwitch Internal Unit.
systemd[1]: Starting Open vSwitch Internal Unit...
ovs-ctl[3560]: * Starting ovsdb-server
ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait -- init -- set Open_vSwitch . db-version=7.12.1
ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait set Open_vSwitch . ovs-version=2.5.0 "external-ids:system-id=\"e7c5ba80-bb14-45c1-b8eb-628f3ad03903\"" "system-type=\"Ubuntu\"" "system-version=\"16.04-xenial\""
ovs-ctl[3560]: * Configuring Open vSwitch system IDs
ovs-ctl[3560]: 2016-01-22T08:58:31Z|00001|dpdk|INFO|No -vhost_sock_dir provided - defaulting to /var/run/openvswitch
ovs-vswitchd: ovs|00001|dpdk|INFO|No -vhost_sock_dir provided - defaulting to /var/run/openvswitch
ovs-ctl[3560]: EAL: Detected lcore 0 as core 0 on socket 0
ovs-ctl[3560]: EAL: Detected lcore 1 as core 1 on socket 0
ovs-ctl[3560]: EAL: Detected lcore 2 as core 2 on socket 0
ovs-ctl[3560]: EAL: Detected lcore 3 as core 3 on socket 0
ovs-ctl[3560]: EAL: Detected lcore 4 as core 4 on socket 0
ovs-ctl[3560]: EAL: Detected lcore 5 as core 5 on socket 0
ovs-ctl[3560]: EAL: Detected lcore 6 as core 0 on socket 0
ovs-ctl[3560]: EAL: Detected lcore 7 as core 1 on socket 0
ovs-ctl[3560]: EAL: Detected lcore 8 as core 2 on socket 0
ovs-ctl[3560]: EAL: Detected lcore 9 as core 3 on socket 0
ovs-ctl[3560]: EAL: Detected lcore 10 as core 4 on socket 0
ovs-ctl[3560]: EAL: Detected lcore 11 as core 5 on socket 0
ovs-ctl[3560]: EAL: Support maximum 128 logical core(s) by configuration.
ovs-ctl[3560]: EAL: Detected 12 lcore(s)
ovs-ctl[3560]: EAL: VFIO modules not all loaded, skip VFIO support...
ovs-ctl[3560]: EAL: Setting up physically contiguous memory...
ovs-ctl[3560]: EAL: Ask a virtual area of 0x100000000 bytes
ovs-ctl[3560]: EAL: Virtual area found at 0x7f2040000000 (size = 0x100000000)
ovs-ctl[3560]: EAL: Requesting 4 pages of size 1024MB from socket 0
ovs-ctl[3560]: EAL: TSC frequency is ~2397202 KHz
ovs-vswitchd[3592]: EAL: TSC frequency is ~2397202 KHz
ovs-vswitchd[3592]: EAL: Master lcore 0 is ready (tid=fc6cbb00;cpuset=[0])
ovs-vswitchd[3592]: EAL: PCI device 0000:04:00.0 on NUMA socket 0
ovs-vswitchd[3592]: EAL:   probe driver: 8086:1528 rte_ixgbe_pmd
ovs-vswitchd[3592]: EAL:   Not managed by a supported kernel driver, skipped
ovs-vswitchd[3592]: EAL: PCI device 0000:04:00.1 on NUMA socket 0
ovs-vswitchd[3592]: EAL:   probe driver: 8086:1528 rte_ixgbe_pmd
ovs-vswitchd[3592]: EAL:   PCI memory mapped at 0x7f2140000000
ovs-vswitchd[3592]: EAL:   PCI memory mapped at 0x7f2140200000
ovs-ctl[3560]: EAL: Master lcore 0 is ready (tid=fc6cbb00;cpuset=[0])
ovs-ctl[3560]: EAL: PCI device 0000:04:00.0 on NUMA socket 0
ovs-ctl[3560]: EAL:   probe driver: 8086:1528 rte_ixgbe_pmd
ovs-ctl[3560]: EAL:   Not managed by a supported kernel driver, skipped
ovs-ctl[3560]: EAL: PCI device 0000:04:00.1 on NUMA socket 0
ovs-ctl[3560]: EAL:   probe driver: 8086:1528 rte_ixgbe_pmd
ovs-ctl[3560]: EAL:   PCI memory mapped at 0x7f2140000000
ovs-ctl[3560]: EAL:   PCI memory mapped at 0x7f2140200000
ovs-vswitchd[3592]: PMD: eth_ixgbe_dev_init(): MAC: 4, PHY: 3
ovs-vswitchd[3592]: PMD: eth_ixgbe_dev_init(): port 0 vendorID=0x8086 deviceID=0x1528
ovs-ctl[3560]: PMD: eth_ixgbe_dev_init(): MAC: 4, PHY: 3
ovs-ctl[3560]: PMD: eth_ixgbe_dev_init(): port 0 vendorID=0x8086 deviceID=0x1528
ovs-ctl[3560]: Zone 0: name:&lt;RG_MP_log_history&gt;, phys:0x83fffdec0, len:0x2080, virt:0x7f213fffdec0, socket_id:0, flags:0
ovs-ctl[3560]: Zone 1: name:&lt;MP_log_history&gt;, phys:0x83fd73d40, len:0x28a0c0, virt:0x7f213fd73d40, socket_id:0, flags:0
ovs-ctl[3560]: Zone 2: name:&lt;rte_eth_dev_data&gt;, phys:0x83fd43380, len:0x2f700, virt:0x7f213fd43380, socket_id:0, flags:0
ovs-ctl[3560]: * Starting ovs-vswitchd
ovs-ctl[3560]: * Enabling remote OVSDB managers
systemd[1]: Started Open vSwitch Internal Unit.
systemd[1]: Starting Open vSwitch...
systemd[1]: Started Open vSwitch.


CMD: sudo ovs-vsctl add-br ovsdpdkbr0 -- set bridge ovsdpdkbr0 datapath_type=netdev

SYSLOG:
2016-01-22T08:58:56.344Z|00008|memory|INFO|37256 kB peak resident set size after 24.5 seconds
2016-01-22T08:58:56.346Z|00009|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports recirculation
2016-01-22T08:58:56.346Z|00010|ofproto_dpif|INFO|netdev@ovs-netdev: MPLS label stack length probed as 3
2016-01-22T08:58:56.346Z|00011|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports unique flow ids
2016-01-22T08:58:56.346Z|00012|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_state
2016-01-22T08:58:56.346Z|00013|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_zone
2016-01-22T08:58:56.346Z|00014|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_mark
2016-01-22T08:58:56.346Z|00015|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath does not support ct_label
2016-01-22T08:58:56.360Z|00016|bridge|INFO|bridge ovsdpdkbr0: added interface ovsdpdkbr0 on port 65534
2016-01-22T08:58:56.361Z|00017|bridge|INFO|bridge ovsdpdkbr0: using datapath ID 00005a4a1ed0a14d
2016-01-22T08:58:56.361Z|00018|connmgr|INFO|ovsdpdkbr0: added service controller "punix:/var/run/openvswitch/ovsdpdkbr0.mgmt"

OVS-LOG:
ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl add-br ovsdpdkbr0 -- set bridge ovsdpdkbr0 datapath_type=netdev
systemd-udevd[3607]: Could not generate persistent MAC address for ovs-netdev: No such file or directory
kernel: [50165.886554] device ovs-netdev entered promiscuous mode
kernel: [50165.901261] device ovsdpdkbr0 entered promiscuous mode


CMD: sudo ovs-vsctl add-port ovsdpdkbr0 dpdk0 -- set Interface dpdk0 type=dpdk

SYSLOG:
2016-01-22T08:59:06.369Z|00019|memory|INFO|peak resident set size grew 155% in last 10.0 seconds, from 37256 kB to 95008 kB
2016-01-22T08:59:06.369Z|00020|memory|INFO|handlers:4 ports:1 revalidators:2 rules:5
2016-01-22T08:59:30.989Z|00021|dpdk|INFO|Port 0: 8c:dc:d4:b3:6d:e9
2016-01-22T08:59:31.520Z|00022|dpdk|INFO|Port 0: 8c:dc:d4:b3:6d:e9
2016-01-22T08:59:31.521Z|00023|dpif_netdev|INFO|Created 1 pmd threads on numa node 0
2016-01-22T08:59:31.522Z|00001|dpif_netdev(pmd16)|INFO|Core 0 processing port 'dpdk0'
2016-01-22T08:59:31.522Z|00024|bridge|INFO|bridge ovsdpdkbr0: added interface dpdk0 on port 1
2016-01-22T08:59:31.522Z|00025|bridge|INFO|bridge ovsdpdkbr0: using datapath ID 00008cdcd4b36de9
2016-01-22T08:59:31.523Z|00002|dpif_netdev(pmd16)|INFO|Core 0 processing port 'dpdk0'

OVS-LOG:
ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl add-port ovsdpdkbr0 dpdk0 -- set Interface dpdk0 type=dpdk
ovs-vswitchd[3595]: PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f211a79ebc0 hw_ring=0x7f211a7a6c00 dma_addr=0x81a7a6c00
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Using simple tx code path
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Vector tx enabled.
ovs-vswitchd[3595]: PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7f211a78a6c0 sw_sc_ring=0x7f211a786580 hw_ring=0x7f211a78e800 dma_addr=0x81a78e800
ovs-vswitchd[3595]: PMD: ixgbe_set_rx_function(): Vector rx enabled, please make sure RX burst size no less than 4 (port=0).
ovs-vswitchd[3595]: PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f211a79ebc0 hw_ring=0x7f211a7a6c00 dma_addr=0x81a7a6c00
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Using simple tx code path
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Vector tx enabled.
ovs-vswitchd[3595]: PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f211a76e4c0 hw_ring=0x7f211a776500 dma_addr=0x81a776500
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Using simple tx code path
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Vector tx enabled.
ovs-vswitchd[3595]: PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f211a756440 hw_ring=0x7f211a75e480 dma_addr=0x81a75e480
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Using simple tx code path
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Vector tx enabled.
ovs-vswitchd[3595]: PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f211a73e3c0 hw_ring=0x7f211a746400 dma_addr=0x81a746400
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Using simple tx code path
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Vector tx enabled.
ovs-vswitchd[3595]: PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f211a726340 hw_ring=0x7f211a72e380 dma_addr=0x81a72e380
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Using simple tx code path
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Vector tx enabled.
ovs-vswitchd[3595]: PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f211a70e2c0 hw_ring=0x7f211a716300 dma_addr=0x81a716300
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Using simple tx code path
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Vector tx enabled.
ovs-vswitchd[3595]: PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f211a6f6240 hw_ring=0x7f211a6fe280 dma_addr=0x81a6fe280
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Using simple tx code path
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Vector tx enabled.
ovs-vswitchd[3595]: PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f211a6de1c0 hw_ring=0x7f211a6e6200 dma_addr=0x81a6e6200
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Using simple tx code path
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Vector tx enabled.
ovs-vswitchd[3595]: PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f211a6c6140 hw_ring=0x7f211a6ce180 dma_addr=0x81a6ce180
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Using simple tx code path
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Vector tx enabled.
ovs-vswitchd[3595]: PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f211a6ae0c0 hw_ring=0x7f211a6b6100 dma_addr=0x81a6b6100
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Using simple tx code path
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Vector tx enabled.
ovs-vswitchd[3595]: PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f211a696040 hw_ring=0x7f211a69e080 dma_addr=0x81a69e080
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Using simple tx code path
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Vector tx enabled.
ovs-vswitchd[3595]: PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f211a67dfc0 hw_ring=0x7f211a686000 dma_addr=0x81a686000
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Using simple tx code path
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Vector tx enabled.
ovs-vswitchd[3595]: PMD: ixgbe_dev_tx_queue_setup(): sw_ring=0x7f211a665e40 hw_ring=0x7f211a66de80 dma_addr=0x81a66de80
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Using simple tx code path
ovs-vswitchd[3595]: PMD: ixgbe_set_tx_function(): Vector tx enabled.
ovs-vswitchd[3595]: PMD: ixgbe_dev_rx_queue_setup(): sw_ring=0x7f211a78a6c0 sw_sc_ring=0x7f211a786580 hw_ring=0x7f211a78e800 dma_addr=0x81a78e800
ovs-vswitchd[3595]: PMD: ixgbe_set_rx_function(): Vector rx enabled, please make sure RX burst size no less than 4 (port=0).


CMD: sudo ovs-vsctl add-port ovsdpdkbr0 vhost-user-1 -- set Interface vhost-user-1 type=dpdkvhostuser

OVS-LOG:
2016-01-22T09:00:35.145Z|00026|dpdk|INFO|Socket /var/run/openvswitch/vhost-user-1 created for vhost-user port vhost-user-1
2016-01-22T09:00:35.145Z|00003|dpif_netdev(pmd16)|INFO|Core 0 processing port 'dpdk0'
2016-01-22T09:00:35.145Z|00004|dpif_netdev(pmd16)|INFO|Core 0 processing port 'vhost-user-1'
2016-01-22T09:00:35.145Z|00027|bridge|INFO|bridge ovsdpdkbr0: added interface vhost-user-1 on port 2

SYSLOG:
ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl add-port ovsdpdkbr0 vhost-user-1 -- set Interface vhost-user-1 type=dpdkvhostuser
ovs-vswitchd[3595]: VHOST_CONFIG: socket created, fd:46
ovs-vswitchd[3595]: VHOST_CONFIG: bind to /var/run/openvswitch/vhost-user-1

Eventually we can see the poll thread in top
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 3595 root      10 -10 4975344 103936   9916 S 100.0  0.3  33:13.56 ovs-vswitchd
		  </programlisting>

	  </sect2>

	  <sect2 id="dpdk-references" status="review">
		  <title>Resources</title>

		  <itemizedlist>
			  <listitem>
				  <para>
					  <ulink url="http://dpdk.org/doc">DPDK Documentation</ulink>
				  </para>
			  </listitem>
			  <listitem>
				  <para>
					  <ulink url="http://dpdk.org/doc/guides/rel_notes/release_2_2.html">Release Notes matching the version packages in Ubuntu 16.04</ulink>
				  </para>
			  </listitem>
			  <listitem>
				  <para>
					  <ulink url="http://dpdk.org/doc/guides/linux_gsg/index.html">Linux DPDK User Getting Started</ulink>
				  </para>
			  </listitem>
			  <listitem>
				  <para>
					  <ulink url="http://dpdk.org/doc/guides/testpmd_app_ug/run_app.html">EAL Command-line Options</ulink>
				  </para>
			  </listitem>
			  <listitem>
				  <para>
					  <ulink url="http://dpdk.org/doc/api/">DPDK Api Documentation</ulink>
				  </para>
			  </listitem>
			  <listitem>
				  <para>
					  <ulink url="https://github.com/openvswitch/ovs/blob/branch-2.5/INSTALL.DPDK.md">OpenVswitch DPDK installation</ulink>
				  </para>
			  </listitem>
			  <listitem>
				  <para>
					  <ulink url="https://en.wikipedia.org/wiki/Data_Plane_Development_Kit">Wikipedias definition of DPDK</ulink>
				  </para>
			  </listitem>
		  </itemizedlist>

	  </sect2>
  </sect1>

  </chapter>