~hardware-certification/maas-cert-server/trunk
» Revision
39.1.1
: documentation/Self-Test_Guide.rst
« back to all changes in this revision
Viewing changes to documentation/Self-Test_Guide.rst
- browse files at revision 39.1.1
- compare with another revision
- download diff
- download tarball
- view history from revision 39.1.1
- Committer: Rod Smith
- Date: 2015-11-12 20:59:43 UTC
- mto: This revision was merged to the branch mainline in revision 40.
- Revision ID: rod.smith@canonical.com-20151112205943-pwuqga9v71ve98az
Replaced documentation with a dependency on the certification-docs package. Made related changes in maniacs-setup and elsewhere.
- files removed:
- documentation
- documentation/images
- documentation/styles
- files modified:
- debian/changelog
added
removed
documentation/Self-Test_Guide.rst
1
=================================================================
2
Ubuntu Server Certified Hardware Self-Testing Guide (14.04 LTS)
3
=================================================================
4
5
.. header:: |ubuntu_logo|
6
7
.. |ubuntu_logo| image:: images/logo-ubuntu_su-white_orange-hex.png
8
:scale: 20%
9
10
.. footer:: |canonical_logo|
11
12
.. |canonical_logo| image:: images/logo-canonical_no-tm-white-hex.png
13
:scale: 10%
14
15
.. raw:: pdf
16
17
PageBreak oneColumn
18
19
.. contents::
20
21
.. raw:: pdf
22
23
PageBreak
24
25
Introduction
26
============
27
28
The aim of this document is to provide the information needed to certify
29
a server on-site using the Canonical Server Test Suite without requiring
30
an Internet connection.
31
32
Glossary
33
========
34
35
The following definitions apply to terms used in this document.
36
37
1Gbps
38
1 Gigabit per second -- Network speed for Gigabit Ethernet (1000Mbps).
39
40
10Gbps
41
10 Gigabits per second -- Network speed for 10 Gigabit Ethernet
42
(10,000Mbps).
43
44
BMC
45
Baseboard Management Controller -- A device in many server models
46
that enables remote in- and out-of-band management of hardware.
47
48
DHCP
49
Dynamic Host Control Protocol -- A method for providing IP
50
addresses to the SUT and Targets.
51
52
Greylist test
53
A test that must be performed but will not affect the
54
granting of a certified status.
55
56
KVM
57
Kernel Virtual Machine -- A system for running virtual machines on
58
Ubuntu Server.
59
60
IPMI
61
Intelligent Platform Management Interface -- A technology for
62
remotely connecting to a computer to perform management functions.
63
64
JBOD
65
Just a bunch of disks -- A non-RAID disk configuration.
66
67
LAN
68
Local Area Network -- The network to which your SUT and Targets are
69
connected. The LAN does not need to be Internet accessible (though that
70
is preferable if possible).
71
72
.. raw:: pdf
73
74
PageBreak
75
76
MAAS
77
Metal as a Service -- A Canonical product for provisioning systems
78
quickly and easily.
79
80
NIC
81
Network Interface Card -- The network device(s).
82
83
PXE
84
Pre-boot Execution Environment -- A technology that enables you to
85
boot a computer using remote images for easy deployment or network-based
86
installation.
87
88
RAID
89
Redundant Array of Independent Disks - Multi-disk storage
90
providing redundancy, parity checking, and data integrity.
91
92
RAM
93
Random Access Memory -- System memory.
94
95
SAN
96
Storage Area Network -- Usually FibreChannel.
97
98
SUT
99
System Under Test -- The machine you are testing for certification.
100
101
Target
102
A computer on the test LAN that the SUT can use for network testing. The
103
Target must be running an ``iperf`` server, but otherwise does not need
104
any special configuration. The MAAS server may double as the Target.
105
106
Test case
107
A test to be executed as part of the certification test
108
suite. Test cases include things such as "CPU Stress" and "CPU
109
Topology."
110
111
Whitelist test
112
A test that *must* pass for the SUT to be granted a certified status.
113
114
Overview of the Certification Process
115
=====================================
116
117
The certification process has certain prerequisites and procedures with
118
which you should be familiar before beginning. Specifically, you should
119
be aware of hardware requirements and the needs of the network test
120
environment. There are steps you should perform before running the
121
certification tests and before uploading the results. The following
122
sub-sections briefly describe these requirements. Detailed descriptions
123
appear later in this document.
124
125
Hardware Requirements
126
---------------------
127
128
- SUTs must contain at least 4GiB of RAM and one hard disk, but using
129
the maximum amount of RAM and the maximum number of disks is
130
preferable. Likewise, if configurable, using the maximum number of
131
CPUs is desirable.
132
133
- CPUs should support virtualization (VMX/SVM), when supported by CPU
134
architecture.
135
136
- All firmware (BIOS/UEFI, NIC, storage controller, etc) should be
137
shipping level, *not* development level.
138
139
- A monitor and keyboard for the SUT are helpful because they will
140
enable you to monitor its activities. If necessary, however,
141
certification can be done without these items.
142
143
Network Test Environment
144
------------------------
145
146
- In addition to the SUT, the network must contain at least one other
147
machine, which will run MAAS and an ``iperf`` server. The MAAS Advanced
148
NUC Installation and Configuration (MANIAC) document (available from
149
https://certification.canonical.com) describes how to configure a MAAS
150
server. This server may be a standard part of the testing network or
151
something you bring with you for testing purposes alone. A laptop or a
152
small portable computer such as an Intel NUC is sufficient. MAAS version
153
1.7 or later is strongly preferred for certification work; the older
154
MAAS 1.5 lacks certain features that are becoming increasingly
155
important. This document describes use of MAAS 1.8. If you use MAAS 1.7,
156
some procedures will differ slightly.
157
158
- The MAAS server computer should run Ubuntu 14.04 (Trusty Tahr), and
159
should be configured to deliver Ubuntu 14.04 images to its clients.
160
Testing with fixed point releases, as described in the "Configuring
161
MAAS to Deliver Fixed Point Releases" section of the MANIAC document,
162
is required.
163
164
- Ideally, the network should have few or no other computers;
165
extraneous network traffic can negatively impact the network tests.
166
167
- Ideally, the MAAS server system should handle DNS and DHCP for the
168
network. If other computers manage these tasks, be sure that they're
169
configured to work with the MAAS server so that the SUT obtains its
170
PXE-boot images from the MAAS server.
171
172
- Network cabling, switches, and the ``iperf`` server should be capable of
173
at least the SUT's best speed. For instance, if the SUT has 1Gbps
174
Ethernet, the other network components should be capable of 1Gbps or
175
faster speeds. If the local network used for testing is less capable
176
than the best network interfaces on the SUT, those interfaces must be
177
tested later on a more-capable network.
178
179
- If desired, the MAAS server may be run inside a virtual machine;
180
however, it is advisable to run the ``iperf`` server on "real" hardware
181
so as to minimize the risk of network tests failing because of
182
virtualization issues.
183
184
Before Running Test Cases
185
-------------------------
186
187
- The SUT must be properly configured and cabled.
188
189
- The SUT must have Ubuntu 14.04 installed on it. (This must be done via
190
MAAS.) See the `Installing Ubuntu on the System`_ section below for
191
details on how to do this.
192
193
- The ``canonical-certification-server`` package must be installed on the
194
SUT together with all its dependencies. (This will normally be done via
195
MAAS.) For more information about how to get all the packages, please
196
refer to the `Installing the Server Test Suite Packages`_ section below.
197
198
- Be sure to have all the items described in the `Equipment to
199
Bring`_ section below.
200
201
- Some manual configuration will be necessary to perform
202
network testing. See the `Installing Ubuntu on the
203
System`_ section below.
204
205
Before Uploading Test Case Results
206
----------------------------------
207
208
Prior to uploading results to Canonical's certification site, you must
209
handle some preliminary tasks:
210
211
- You will need an account at the certification web site,
212
https://certification.canonical.com.
213
214
- The certification web site is the location where all the test case
215
results will be uploaded once your testing session has finished.
216
217
- *If you do not have an account for your company on the private
218
certification web site, please contact your account manager who will
219
work with the Server Certification Team to establish the account.*
220
221
- You must create a hardware entry (unless one already exists for the
222
SUT) at the certification web site with a secure ID.
223
224
- The secure ID is a string made from 15 alphanumeric characters that
225
is used to make sure that only authorized parties upload results to
226
the site and also to associate the results to the hardware being
227
tested. This can be found on the hardware entry page on the
228
Certification web site:
229
230
.. image:: images/secure_id.png
231
:alt: The Secure ID can be obtained from the Ceritification web site.
232
:align: left
233
:width: 50%
234
235
- For more information on creating the hardware entry, please see
236
`Creating a Hardware Entry on C3` below.
237
238
- If the SUT lacks Internet access, the package
239
``canonical-certification-submit`` should be installed on the system
240
from which you plan to submit results. Please see the section below
241
titled `Manually Uploading Test Results to the Certification Site`_ for
242
more information on this topic.
243
244
Initial Setup
245
=============
246
247
Before you certify the hardware, you must perform some initial setup
248
steps. These steps are preparing the hardware you'll bring, configuring
249
the SUT for testing, installing Ubuntu, and installing the Server Test
250
Suite.
251
252
Equipment to Bring
253
------------------
254
255
The requirements for running the tests for a server are minimal. Ensure
256
that you have:
257
258
- A writable USB stick with enough free space (> 256MB). It must contain a
259
*single partition* with a *writable FAT* filesystem on it. If you need
260
to test more than one computer then *bring one USB stick per system*.
261
Note that a USB stick with multiple partitions may cause problems, so if
262
necessary you should repartitition your device to have a single
263
partition.
264
265
- A data CD with some files written to it. This is required to test the
266
system's optical drive read capabilities. Note that a movie DVD or an
267
audio CD won't be useful in this case, as they are not in the right
268
format for the test. If you need to test more than one computer then
269
*bring one medium per system*.
270
271
- A computer to function as a MAAS server and ``iperf`` target on the test
272
LAN. This server will provision the SUT. The MAAS server can be a normal
273
part of the test LAN or can be brought in specifically for testing SUTs
274
on the test LAN. (Note, however, that the MAAS server for certification
275
testing should ideally be configured to automatically install the Server
276
Test Suite on the SUT, which will not be the case for a "generic" MAAS
277
server.)
278
279
- A USB flash drive that holds the Server Test Suite software. (This is a
280
precautionary measure in case MAAS can't install the Server Test Suite
281
packages and your LAN lacks Internet access.) The `Bringing the Server
282
Test Suite Packages With You`_ section of Appendix A describes how to
283
obtain the software.
284
285
Hardware Setup
286
--------------
287
288
The following should be considered the minimum requirements for setting
289
up the SUT and test environment:
290
291
- Minimum loadout
292
293
- Minimum of 4GiB RAM
294
295
- 1 HDD (2 with minimal RAID)
296
297
- 1 CPU of a supported type
298
299
- Recommended (preferred) loadout
300
301
- Maximum supported number of HDDs or SSDs, especially if you can
302
configure multiple RAID levels (e.g. 2 for RAID 0, 3 for RAID 5, and
303
6 for RAID 50)
304
305
- Maximum amount of supported RAM
306
307
- Maximum number of supported CPUs
308
309
- The SUT should not contain any extraneous PCI devices that are not
310
part of the certification.
311
312
- This includes things like network, SAN and iSCSI cards.
313
314
- Hardware RAID cards are allowed if they are used to provide RAID
315
services to the SUT's onboard storage.
316
317
- The SUT should be running a release level BIOS/UEFI configured using
318
factory default settings, with the following exceptions:
319
320
- If the hardware virtualization options in the BIOS/UEFI are not
321
enabled, enable them, save the settings and allow the SUT to reboot.
322
323
- The SUT must be configured to PXE-boot by default.
324
325
- Storage should be properly configured.
326
327
- If the SUT is to be tested using RAID, then the proper hardware RAID
328
configuration should be set up prior to testing.
329
330
- Software RAID, including firmware-supported software RAID (aka "fake
331
RAID") should *not* be used. If hardware RAID is not available,
332
configure the server for JBOD.
333
334
- Any additional HDDs or logical drives should be partitioned and
335
mounted prior to testing. Partitions on those additional HDDs should,
336
preferably, be a single partition that spans the entire disk.
337
338
- Some computers may have problems booting from disks over 2TiB in
339
size. If the SUT fails for this reason, it may pass with smaller
340
disks (or a smaller RAID array). In some cases, configuring the SUT
341
to boot in UEFI mode may also solve this problem; but consult the
342
Canonical Certification Team before changing firmware settings to
343
overcome such a problem.
344
345
- If possible, as many processors as the SUT will support should be
346
installed.
347
348
- Note that systems that ship with processors from different families
349
(e.g Sandy Bridge vs. Haswell) will require extra testing.
350
351
- CPU speed bumps and die shrinks do not require extra testing.
352
353
- Disks with 4,096-byte *logical* sector sizes may require booting in
354
EFI/UEFI mode, and MAAS's support for this is still new, so be alert to
355
potential problems. For instance, some computers support EFI-mode
356
booting but not EFI-mode PXE-booting. Note that disks with 4,096-byte
357
*physical* sector sizes are fine, so long as the disk's firmware
358
translates those sectors into 512-byte logical sectors.
359
360
- The test environment should have a working network setup. Internet
361
access is not required, and testing should work on any private or
362
segregated LAN.
363
364
- If possible, the test LAN's speed should match or exceed the network
365
speed of the SUT. For instance, a SUT with 1Gbps onboard Ethernet
366
should be connected to a LAN capable of at least 1Gbps and a system
367
with 10Gbps Ethernet should be connected to a LAN capable of at least
368
10Gbps. Connecting a SUT to a network with greater network speed is
369
acceptable.
370
371
- If the primary test network cannot meet these requirements,
372
re-running the network tests in an environment that does match these
373
requirements will be necessary.
374
375
- Every network port must be cabled to the LAN and properly configured
376
with either DHCP or static addressing. If a SUT has 4 NIC ports, then
377
all 4 must be connected to the LAN.
378
379
- It is very strongly recommended that SUT and Target machines be on a
380
clean network (that is, one that is not full of other traffic), as
381
extraneous network traffic could impact the network testing results.
382
383
- The test LAN must have a working MAAS server that can provision and run
384
the tests on the SUT. The MAAS Advanced NUC Installation and
385
Configuration (MANIAC) document, available at
386
https://certification.canonical.com, describes the basics of the MAAS
387
setup, but you may need to refer to additional documentation to complete
388
the task if you're not already familiar with MAAS.
389
390
- The test LAN must have one system available to act as a Target for
391
network testing with ``iperf``. Note that accessing an ``iperf`` server
392
that's reachable only via a router may not work, because routing tables
393
are temporarily lost during network testing. The ``iperf`` server is
394
normally the same as the MAAS server, but this does not need to be the
395
case. If the SUT has a faster network interface than the MAAS server,
396
you should set up another computer that matches the SUT's network
397
interface speed to function as an ``iperf`` server.
398
399
- The SUT's BMC, if present, may be configured via DHCP or with a static
400
IP address. MAAS will set up its own BMC user account (``maas``) when
401
enlisting the SUT.
402
403
Installing Ubuntu on the System
404
-------------------------------
405
406
Beginning with Ubuntu 14.04 (Trusty Tahr), server certification requires
407
that the SUT be installable via MAAS. Therefore, the following procedure
408
assumes the presence of a properly-configured MAAS server. The MAAS
409
Advanced NUC Installation and Configuration (MANIAC) document describes how
410
to set up a MAAS server for certification testing purposes. This document
411
describes use of MAAS 1.8. Using MAAS 1.7 is also acceptable, but some user
412
interface details differ.
413
414
Once the SUT and MAAS server are both connected to the network, you can
415
install Ubuntu on the SUT as follows:
416
417
#. Unplug any USB flash drives or external hard disks from the SUT.
418
(MAAS will attempt to install to a USB flash drive if it's detected
419
before the hard disk. This is obviously undesirable.)
420
421
#. Power on the SUT and allow it to PXE-boot.
422
423
- The SUT should boot the MAAS enlistment image and then power off.
424
425
- You should see the SUT appear as a newly-enlisted computer in your
426
MAAS server's node list. (You may need to refresh your browser to see
427
the new entry.)
428
429
#. Check and verify the following items in the MAAS server's node details
430
page:
431
432
- If desired, set a node name for the SUT.
433
434
- Check the SUT's power type and ensure it's set correctly (IPMI, AMT,
435
etc.). If the SUT has no BMC, you can leave this section blank or
436
enter "dummy" data. If the power control information was detected
437
incorrectly or incompletely, you should consult the Canonical
438
Certification Team for advice.
439
440
#. Commission the node by clicking Take Action followed by Commission
441
and then Go.
442
443
- If the SUT has a BMC, the computer should power up, pass more
444
information about itself to the MAAS server, and then power down
445
again.
446
447
- If the SUT does not have a BMC, you should manually power on the SUT
448
after clicking the Commission Node button. The SUT should power up,
449
pass more information about itself to the MAAS server, and then power
450
down again.
451
452
- Note that manual power control is acceptable only on low-end servers
453
that lack BMCs. If MAAS fails to detect a BMC that is present or if
454
MAAS cannot control a BMC that is present, please consult the
455
Canonical Certification Team.
456
457
#. On the MAAS server, verify that the SUT's Status is listed as Ready
458
in the node list or on the node's details page. You may need to
459
refresh the page to see the status update.
460
461
#. Click Take Action followed by Deploy. Options to select the OS version
462
to deploy should appear.
463
464
#. Select the Ubuntu release you want to deploy. Normally, you'll pick a
465
point release that you installed as described in the MANIAC document.
466
This image will appear as an OS type of Custom" and a description that
467
you gave it. The normal procedure is to test with 14.04 GA followed by
468
the latest point release. `Appendix D - Point Release Testing`_,
469
elaborates on this policy.
470
471
#. Click Go to begin deployment.
472
473
- If the SUT has a BMC, it should power up and install Ubuntu. This
474
process can take several minutes.
475
476
- If the SUT does not have a BMC, you should power it on manually after
477
clicking Acquire and Start Node. The SUT should then boot and install
478
Ubuntu. This process can take several minutes.
479
480
If MAAS has problems in any of the preceding steps, the SUT might not
481
pass certification. For instance, certification requires that MAAS be
482
able to detect the SUT and set its power type information automatically.
483
If you have problems with any of these steps, contact the
484
Canonical Server Certification Team to learn how to proceed; you might
485
have run into a simple misconfiguration, or the server might need
486
enablement work.
487
488
Logging Into the SUT
489
--------------------
490
491
Once the SUT is installed, you should be able to log into it using SSH from
492
the MAAS server. Check the node details page to learn its primary IP
493
address. (Using a hostname will also work if DNS is properly configured,
494
but this can be fragile.) The username on the node is ``ubuntu``, and you
495
should require no password when logging in from the MAAS server or from any
496
other computer and account whose SSH key you've registered with the MAAS
497
server.
498
499
You should keep some details in mind as you continue to access the SUT:
500
501
- You should *not* install updates to the SUT unless they are absolutely
502
necessary to pass the certification. In that case, the Canonical
503
Certification Team will make the determination of what updates should be
504
applied.
505
506
- You should verify your SUT's version by typing ``lsb_release -a``. The
507
result includes both the main release version (such as 14.04) and the
508
point release version (such as 14.04.2, on the *Description* line). You
509
can also check your kernel version by typing ``uname -r``. The kernel
510
version changes with the Ubuntu release. Both 14.04 and 14.04.1 ship
511
with 3.13-series kernels, while 14.04.2 ships with a 3.16-series kernel.
512
513
- By default, MAAS provides a DHCP server, and the SUT should use it to
514
obtain an IP address. If necessary for your environment, you may
515
manually change these settings on the SUT to use a static IP address.
516
517
- If you want to log in at the console or from another computer, you
518
must set a password::
519
520
$ sudo passwd ubuntu
521
522
After prompting, this command changes the password for the user called
523
``ubuntu``, which is the default user created by MAAS. Testing at the
524
console has certain advantages (described shortly).
525
526
- A MAAS installation configured for certification testing should
527
provision the SUT with the Server Test Suite and related packages. If
528
you're using a more "generic" MAAS setup, you'll have to install the
529
certification software yourself, as described in `Appendix A -
530
Installing the Server Test Suite Manually`_.
531
532
Ensure all network devices are configured in the file
533
``/etc/network/interfaces``:
534
535
.. figure:: images/interfaces.png
536
:alt: The /etc/network/interfaces file must be properly configured
537
for your system's interfaces.
538
:width: 100%
539
540
Be sure to start networking on all the interfaces. (The network tests will
541
*not* bring up a network interface, even if it's defined in
542
``/etc/network/interfaces``.) In most cases, typing ``sudo ifup eth1``, and
543
so on for additional entries, will do the job. (The ``eth0`` interface is
544
normally enabled by default on the first boot and so need not be explicitly
545
brought up, but this isn't always the case.) Once you've done this, typing
546
``ifconfig`` should show that all the interfaces have IP addresses.
547
548
Installing the Server Test Suite Packages
549
-----------------------------------------
550
551
Three methods of installing the Server Test Suite exist:
552
553
- Automatically by the MAAS server
554
555
- Using APT to retrieve the Server Test Suite packages on a SUT with
556
full Internet access
557
558
- By loading the Server Test Suite Debian packages from a USB flash
559
drive or other medium you bring with you to the test site
560
561
If MAAS is fully configured as described in the `MAAS Advanced NUC
562
Installation and Configuration (MANIAC)` document, it should deploy the
563
Server Test Suite automatically. If MAAS doesn't deploy the Server Test
564
Suite properly, you can do so manually, as described in `Appendix A -
565
Installing the Server Test Suite Manually`_.
566
567
Creating a Hardware Entry on C3
568
===============================
569
570
In order to upload test results to the certification web site
571
(http://certification.canonical.com, or C3 for short) you need to create a
572
hardware entry for the system which you will be certifying. If the SUT has
573
no direct Internet connection, you can put off creating the C3 entry until
574
after the test (although doing it before testing is fine, too). If you
575
don't plan to submit the results, you should not create a C3 entry for the
576
machine. To create an entry you can go directly to:
577
578
https://certification.canonical.com/hardware/create-system
579
580
If you have problems accessing this site, contact your account manager.
581
582
When creating an entry, you must enter assorted pieces of information:
583
584
#. Fill in the details:
585
586
* **Account** -- The name of your account. If the account is incorrect
587
or can't be set, please contact your account manager for assistance.
588
This field is never published; it is for internal use only.
589
590
* **Make** -- The manufacturer of the system, e.g. Dell, HP, as you
591
would like it to appear on the public web site.
592
593
* **Model** -- The name of the system itself, e.g ProLiant DL630 or
594
PowerEdge R210, as you would like it to appear on the public web site.
595
596
* **Aliases** -- This is used for alternate marketing names for a
597
server. This field is only accessible to the Canonical Server
598
Certification Team. If you need to add items to this field, please
599
contact your account manager. These do appear publicly as separate
600
entries in the database (e.g. Server1000, Alias1001 and Alias1002 all
601
point to the same system, but appear as three separate entries on the
602
public web site).
603
604
* **Codenames** -- This is for your internal reference and use and is
605
for the internal code name associated with the SUT. This data is
606
*never* published and is visible only to you and to Canonical.
607
608
* **Web site** -- Optional, link to the system info on the
609
manufacturer's web site. This field is published publicly and is a way
610
for potential customers to directly access information about your
611
hardware on your own web site.
612
613
* **Comment** -- Optional, any comment you want to make about the
614
hardware, including things like tester name, test location, etc.
615
These comments are never made public, they are for internal use only.
616
617
* **Form factor** -- The type of system: Laptop, Server, etc. This is
618
not published directly, but determines where your system is displayed
619
on the public site. Client form factors appear in one place while
620
server form factors appear elsewhere on the public certification site.
621
You may select any of the Server form factors you like except for
622
Server SoC, which is reserved for System on Chip certifications.
623
624
* **Architecture** -- The CPU architecture of the SUT. This is used
625
internally and is not published.
626
627
* **Confidential** -- Defaults to False (unchecked). Check the box if
628
the system has not been publicly announced yet or should remain
629
unpublished for any reason. This will cause the entire entry to *not*
630
be published to the public web site.
631
632
#. Click Submit.
633
634
#. Note the "Secure ID for testing purposes" value. You'll need this
635
when submitting the test results. (Note that this value is unique for
636
each machine.)
637
638
Running the Certification Tests
639
===============================
640
641
To initiate a testing session in a server:
642
643
#. Connect to the server via SSH or log in at the console. A standard MAAS
644
installation creates a user called ``ubuntu``, as noted earlier. You can
645
test using either a direct console login or SSH, but an SSH login may be
646
disconnected by the network tests or for other reasons.
647
648
#. Before testing you must ensure that all network ports are cabled to a
649
working LAN and configured in ``/etc/network/interfaces`` using the
650
appropriate configuration (static or DHCP) for your test environment.
651
If you edit this file, either reboot or bring up the interfaces you
652
add with ``ifup`` before running tests.
653
654
#. If the SUT provides the suitable ports and drives, plug in a USB 2
655
stick, plug in a USB 3 stick, and insert a suitable data CD in the
656
optical drive. Note that USB testing is not required for blades that
657
provide USB ports only via specialized dongles. These media must remain
658
inserted *throughout the test run*, because the media tests will be
659
kicked off partway through the run.
660
661
#. If the system doesn't have Internet access:
662
663
* Copy the image you downloaded from
664
http://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-i386-disk1.img (as
665
noted in `Appendix A`) to any directory of the SUT.
666
667
* Supply the full path under the section labeled "environment" in
668
``/etc/xdg/canonical-certification.conf``. For example::
669
670
[environment]
671
KVM_TIMEOUT:
672
KVM_IMAGE: /home/ubuntu/trusty-server-cloudimg-i386-disk1.img
673
674
#. If necessary, edit the ``/etc/xdg/canonical-certification.conf`` file on
675
the SUT so as to specify your ``iperf`` server. For example::
676
677
TEST_TARGET_FTP = your-ftp-server.example.com
678
TEST_USER = anonymous
679
TEST_PASS =
680
TEST_TARGET_IPERF = 192.168.0.2
681
682
If you configured your MAAS server as described in the MANIAC document,
683
the ``TEST_TARGET_IPERF`` line should point to your SUT's gateway address,
684
which is normally also your MAAS server; but if your network
685
configuration deviates from the one described in MANIAC, you may need to
686
adjust this value. The Server Test Suite does not currently use FTP, but
687
these lines must be uncommented. They may be left at their default
688
values.
689
690
#. While editing ``/etc/xdg/canonical-certification.conf``, you may
691
optionally enter the SUT's Secure ID in the ``[sru]`` section. This can
692
simplify submission of results at the end of the test; however, this
693
will work only if the SUT has full Internet access.
694
695
#. Launch ``iperf`` on the server identified in the SUT's
696
``/etc/xdg/canonical-certification.conf`` file by typing::
697
698
$ iperf -s
699
700
#. If you're running the test via SSH, type screen on the SUT to ensure
701
that you can reconnect to your session should your link to the SUT go
702
down, as may happen when running the network tests. If you're
703
disconnected, you can reconnect to your session by logging in and
704
typing ``screen -r``. This step is not important if you're running the
705
Server Test Suite at the console.
706
707
#. Run::
708
709
$ canonical-certification-server
710
711
#. A welcome message will be displayed. Make sure to read the message
712
and follow its instructions.
713
714
#. Press the Enter key. The system will display a Suite Selection
715
screen:
716
717
.. figure:: images/some_tests.png
718
:alt: The suite selection screen enables you to pick which
719
tests to run
720
:width: 100%
721
722
#. Select the *Server-full-14.04* item and deselect the other items.
723
(These other suites exist to enable easy re-running of subsets of
724
tests that often fail in some environments.)
725
726
#. Use the arrow keys to highlight the *<OK>* option and then press
727
Enter.
728
729
#. After a few seconds, a test selection screen will appear, as shown
730
below. You should ordinarily leave all the tests selected. (Tests that
731
are irrelevant for a given computer, such as tests of the optical drive
732
on computers that lack this hardware, are automatically ignored.) If a
733
test is hanging or otherwise causing problems, please contact the
734
Canonical Server Certification Team for advice on how to proceed. Using
735
this screen is fairly straightforward, but `Appendix C - Using the Test
736
Selection Screen`_ covers the details.
737
738
.. figure:: images/ccs_tests.png
739
:alt: The suite selection screen enables you to pick which
740
tests to run
741
:width: 100%
742
743
#. Press the *T* key to start testing. The screen will begin displaying a
744
scrolling set of technical details about the tests as they are
745
performed.
746
747
#. The full test suite can take several hours to complete, depending on
748
the hardware configuration (amount of RAM, disk space, etc). During
749
this time the computer may be unresponsive. This is due to the
750
inclusion of some stress test cases. These are deliberately
751
intensive and produce high load on the system's resources.
752
753
#. If at any time during the execution you are *sure* the computer has
754
crashed (or it reboots spontaneously) then after the system comes back
755
up you should run the ``canonical-certification-server`` command again
756
and respond `y` when asked if you want to resume the previous session.
757
758
#. When the test run is complete, you should see a summary of tests run, a
759
note about where the ``submission.xml``, ``results.html``, and
760
``results.xlsx`` files have been stored, and a prompt to submit the
761
results to ``certification.canonical.com``. If you're connected to the
762
Internet, typing ``y`` at this query should cause the results to be
763
submitted. You will need either a Secure ID value or to have already
764
entered this value in the ``/etc/xdg/canonical-certification.conf``
765
file.
766
767
#. Copying the results files off of the SUT is advisable. This is most
768
important if the automatic submission of results fails; however,
769
having the results available as a backup can be useful because it
770
enables you to review the results off-line or in case of submission
771
problems that aren't immediately obvious. The results are stored in
772
the ``~/.local/share/plainbox`` directory.
773
774
#. Copy the whole directory to an external medium and bring it with you
775
after certifying the system. It contains a file called
776
``submission.xml`` which is the results of the testing as well as a file
777
called ``results.html`` that, if loaded in a web browser, will show you
778
the results.
779
780
If you review your results by loading ``results.html`` in a web browser,
781
you can quickly spot failed tests because they're highlighted in red with a
782
"FAILED" notation in the Result column, whereas passed tests acquire a
783
green color, with the word "PASSED." Note, however, that *a failed test
784
does not necessarily denote a failed certification*. Reasons a test might
785
fail but still enable a certification to pass include the following:
786
787
- A test may be a greylist test, as described in the `Ubuntu Server
788
Hardware Certification Coverage` document, available from
789
https://certification.canonical.com.
790
791
- Some tests are known to produce occasional false positives -- that
792
is, they claim that problems exist when in fact they don't.
793
794
- Some test environments are sub-optimal, necessitating that specific
795
tests be re-run. This can happen with network tests or if the tester
796
forgot to insert a removable medium. In such cases, the specific test
797
can be re-run rather than the entire test suite.
798
799
Consult your account manager if you have questions about specific test
800
results.
801
802
Manually Uploading Test Results to the Certification Site
803
=========================================================
804
805
If you can't upload test results to the certification site from the
806
certification program itself, you must do so manually, perhaps from
807
another computer. To upload the results, you should have the Server Test
808
Suite and ``canonical-certification-submit`` installed on the system from
809
which you plan to submit results. The Server Test Suite is part of the
810
default install on all Ubuntu Desktop systems. At this time, there is no
811
mechanism for submitting results from an OS other than Ubuntu.
812
813
To add the Hardware Certification PPA, install
814
``canonical-certification-submit``, and submit the results, follow these
815
instructions:
816
817
#. Add the Hardware Certification PPA::
818
819
$ sudo apt-add-repository ppa:hardware-certification/public
820
$ sudo apt-get update
821
822
#. Install the package::
823
824
$ sudo apt-get install canonical-certification-submit
825
826
#. Run the following command::
827
828
$ canonical-certification-submit --secure_id <SUT_SECURE_ID> \
829
<PATH_TO>/submission.xml
830
831
where:
832
833
- ``<SUT_SECURE_ID>`` can be found on your system's page on the
834
certification web site (http://certification.canonical.com) by
835
looking next to "Secure ID for testing purposes":
836
837
.. image:: images/secure_id.png
838
:alt: The Secure ID can be obtained from the Ceritification web site.
839
:align: left
840
:width: 50%
841
842
- ``<PATH_TO>`` refers to the location of the ``submission.xml file``
843
(which should be contained in the ``~/.local/share/plainbox``
844
directory you copied to the USB key).
845
846
- Older versions of this tool used ``\-\-hwid`` rather than
847
``\-\-secure_id``.
848
849
You should see output similar to the following for a successful
850
submission::
851
852
$ canonical-certification-submit --hwid a00D000000LU9Ji \
853
~/.local/share/plainbox/submission.xml
854
2012-03-28 11:05:30,575 INFO Preparing to submit results using Hardware
855
ID: a00D000000LU9Ji
856
2012-03-28 11:05:30,576 INFO Getting messages
857
2012-03-28 11:05:56,726 INFO Exchanged 8 of 8 messages
858
2012-03-28 11:05:56,726 INFO Results have been successfully submitted.
859
To review your test results now, please
860
go to the following URL:
861
https://certification.canonical.com/submission/eFXnst3rVbBvdu2
862
863
Once results submission is complete, use the provided link in the output
864
to review the results and confirm that they are correct.
865
866
Requesting a Certificate
867
========================
868
869
Once you've uploaded the data to the certification site, you should
870
review it in the web interface. If you're satisfied that there are no
871
problems, you can request a certificate:
872
873
#. Click the date link under the Created column in the Submissions
874
section. The result should be a page showing most of the same
875
information as the previous page, but in a different format, and
876
restricted to that one test run.
877
878
#. Click the Request Certificate link. The result should be a page with
879
a few radio buttons in which you can enter information:
880
881
- Status is fixed at In Progress.
882
883
- Release indicates the Ubuntu release used for testing, and for which
884
the certificate will be issued.
885
886
- Level indicates the type of certification:
887
888
- Certified is for for hardware that's ready to be deployed with
889
Ubuntu.
890
891
- Certified Pre-installed is for hardware that ships with a (possibly
892
customized) version of Ubuntu.
893
894
- Is Private should be checked if the certification should be kept
895
private. Note that this check box affects the certificate only, not
896
the entry for the computer as a whole on
897
http://certification.canonical.com. Other public pre-existing
898
certificates, or those issued in the future, will remain public.
899
900
#. Click Submit. You'll see a new screen in which you can (and in one
901
case *must*) enter more information. In particular, you can click:
902
903
- Link Bug to link to a bug on https://bugs.launchpad.net.
904
This option is available only to Canonical engineers.
905
906
- Create Note or Create Note from Template to create a note. Most
907
systems will have at least two notes:
908
909
- *A note titled "Tester" with the name of the person who did the
910
testing is required.*
911
912
- A note titled "Test Notes" is usually present. It describes
913
test-specific quirks, such as why a failure should be ignored
914
(say, if a network test failed because of local network problems
915
but succeeded on re-testing). If the
916
*miscellanea/get-maas-version* test fails, be sure to specify the
917
version of MAAS used to deploy the SUT.
918
919
In most cases, the "Private" check box should be checked for your
920
notes.
921
922
.. raw:: pdf
923
924
PageBreak
925
926
Appendix A - Installing the Server Test Suite Manually
927
======================================================
928
929
Ordinarily, MAAS will install the Server Test Suite onto the SUT as part
930
of the provisioning process, as described in the main body of this
931
document. If the MAAS server is not configured to do this, though, you
932
have two additional options for installing the Server Test Suite: You
933
may use APT if the SUT has full Internet access or you may install
934
the Server Test Suite from a tarball that you bring with you.
935
936
Installing the Server Test Suite via APT
937
----------------------------------------
938
939
If your lab setup has Internet access, getting the testing tools is a
940
pretty straightforward process, because you can install the necessary
941
tools from the Ubuntu Hardware Certification PPA.
942
943
Log in to the server locally or via SSH or KVM and run the following
944
commands::
945
946
$ sudo apt-add-repository ppa:hardware-certification/public
947
$ sudo apt-add-repository ppa:firmware-testing-team/ppa-fwts-stable
948
$ sudo apt-get update
949
$ sudo apt-get install canonical-certification-server
950
951
.. The ppa:hardware-certification/public should be stable. For
952
the development PPA, instead use ppa:checkbox-dev/ppa.
953
954
If for some reason you want to run the test suite from an Ubuntu live
955
medium, you must also enable the universe repository::
956
957
$ sudo apt-add-repository universe
958
959
Note that running the test suite from a live medium is not accepted for any
960
certification attempt; this information is provided to help in unusual
961
situations or when debugging problems that necessitate booting in this way.
962
963
During the installation, you may be prompted for a password for ``mysql``.
964
This can be set to anything you wish; it will not be used during testing.
965
966
At this point, you should have the test suite and dependencies installed
967
and be ready to begin testing.
968
969
Bringing the Server Test Suite Packages With You
970
------------------------------------------------
971
972
If you do not have Internet access from your LAN, you can find the
973
pre-built tarball including the Server Test Suite packages under:
974
975
https://certification.canonical.com/offline
976
977
Note that you will be asked for your account credentials when you access
978
that URL.
979
980
Copy the appropriate ``.tar.gz`` file to a USB stick and bring this with
981
you when testing the system. You will also need to obtain and bring a copy
982
of a bootable Ubuntu Cloud Image for the virtualization portion of the
983
certification test. You can obtain that here:
984
985
http://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-i386-disk1.img
986
987
You can use another release of Ubuntu if this is convenient. An i386
988
image is used even when you're testing 64-bit hardware. The point is to
989
test that virtualization features work, not that a specific Ubuntu
990
version or architecture is supported. Because the i386 image is more
991
general, it's the one that we use in testing.
992
993
To install the server certification packages from the copied tarball,
994
perform the following steps on the SUT:
995
996
#. Insert the USB drive on which you placed the tarball.
997
998
#. Extract the contents of the ``.tar.gz`` file from the USB stick to a
999
temporary directory (such as ``/tmp/``)::
1000
1001
$ tar -C /tmp -xzf ubuntu-14.04-server-amd64.tar.gz
1002
$ cd /tmp
1003
1004
#. Look for a directory whose name begins with ``apt-repo`` and switch to
1005
it (note this is just an example, your exact directory name may be
1006
different)::
1007
1008
$ cd apt-repo-ubuntu-14.04.1-server-amd64.iso-20140901-canonical-certification-ser
1009
1010
4. Use a provided helper script to add a local package repository::
1011
1012
$ sudo ./add_offline_repository -u
1013
1014
5. Install the Server Test Suite::
1015
1016
$ sudo apt-get install canonical-certification-server
1017
1018
During the installation, you may be prompted for a password for ``mysql``.
1019
This can be set to anything you wish; it will not be used during testing.
1020
If you're asked about mail server configuration, respond that the mail
1021
server should not be configured.
1022
1023
.. raw:: pdf
1024
1025
PageBreak
1026
1027
Appendix B - Re-Testing and Installing Updated Tests
1028
====================================================
1029
1030
Occasionally, a test will fail, necessitating re-testing a feature. For
1031
instance, if you forget to insert a USB flash drive, the relevant USB
1032
tests will fail. The same thing will happen if a USB flash drive is
1033
defective or improperly prepared. Another common source of problems is
1034
network tests, which can fail because of busy LANs, flaky switches, bad
1035
cables, and so on. When this happens, you must re-run the relevant
1036
test(s).
1037
1038
Although it's often possible to re-run a test by directly executing a
1039
single test script, the preferred method is:
1040
1041
#. Re-run ``canonical-certification-server``.
1042
1043
#. Use one of the abbreviated testing whitelists (such as *Network-only*)
1044
or adjust the set of tests to be run (as described in `Appendix C`).
1045
1046
#. Submit the resulting ``submission.xml`` file to the C3 site.
1047
1048
You can then request a certificate based on the main results (the one with
1049
the most passed tests) and refer to the secondary set of results in the
1050
certificate notes. This procedure ensures that all the necessary data will
1051
be present on C3. It also ensures that (sometimes subtle) problems will be
1052
avoided; for instance, network tests may not be valid if network ports that
1053
are not being tested are active. The ``canonical-certification-server``
1054
framework ensures that such potential problems are avoided.
1055
1056
From time to time, a test will be found to contain a bug or need to be
1057
updated to deal with a problem. In such cases, it is often impractical
1058
to wait for the fix to work its way down through Ubuntu's packaging
1059
system, or even through the PPAs in which some of the relevant tools are
1060
distributed. In such cases, the usual procedure for replacing the script
1061
or file is as follows:
1062
1063
#. Consult with the Server Certification Team about the problem; *do
1064
not* install an updated script from some other source!
1065
1066
#. Obtain the updated file (typically a script) from the Server
1067
Certification Team. Store it on the SUT in the home directory of the
1068
test account. For instance, the new script might be
1069
``/home/ubuntu/newscript``. If necessary, give the new file execute
1070
permissions.
1071
1072
#. On the SUT, rename or delete the original file, as in::
1073
1074
$ sudo rm /usr/lib/2013.canonical.com\:checkbox/bin/oldscript
1075
1076
#. Create a symbolic link from the new script to the original name, as
1077
in::
1078
1079
$ sudo ln -s /home/ubuntu/newscript \
1080
/usr/lib/2013.canonical.com\:checkbox/bin/oldscript
1081
1082
#. Run the tests again, using the ``canonical-certification-server`` user
1083
interface.
1084
1085
In some cases, another procedure might be necessary; for instance, a bug
1086
fix might require installing a new Debian package with the dpkg command,
1087
or you might need to edit a configuration file. The Canonical Server
1088
Certification Team can advise you about such requirements.
1089
1090
.. raw:: pdf
1091
1092
PageBreak
1093
1094
Appendix C - Using the Test Selection Screen
1095
============================================
1096
1097
It may be necessary for you to deselect some of the tests which are to
1098
be run for certification. This is unlikely, though, and you should only
1099
do it when so instructed.
1100
1101
The test selection screen looks like this:
1102
1103
.. figure:: images/ccs_tests2.png
1104
:alt: The test selection screen enables you to select the tests
1105
you want to run.
1106
:width: 100%
1107
1108
Every test suite name is preceded by a brackets that contain either an
1109
*X* character or nothing to identify whether the test suite has been
1110
selected or not.
1111
1112
Tests are arranged hierarchically. When a top-level test is highlighted,
1113
you can hide or reveal the details by pressing the Enter key; or you can
1114
select or deselect all the tests in that category by pressing the
1115
Spacebar. For instance, with *Benchmark tests* highlighted, as in the
1116
figure, pressing Enter will hide the lines up to *CPU tests*; and pressing
1117
the Spacebar will deactivate (or re-activate) all of the benchmark
1118
tests. You can select or de-select an individual test, such as the
1119
*hdparm-read_sda* test, by highlighting it and pressing the Spacebar.
1120
1121
Once you've selected all the tests, press the *T* key to begin the testing
1122
process.
1123
1124
.. raw:: pdf
1125
1126
PageBreak
1127
1128
Appendix D - Point Release Testing
1129
==================================
1130
1131
Ordinarily, 14.04 certification requires testing two releases:
1132
1133
- Ubuntu 14.04 GA -- That is, the version that was released in April of
1134
2014.
1135
1136
- The current point release -- That is, version 14.04.2 or whatever is
1137
the latest release in the 14.04 series.
1138
1139
In theory, compatibility will only improve with time, so a server might
1140
fail testing with 14.04 GA because it uses new hardware that had not
1141
been supported in April of 2014, but pass with the latest version. Such
1142
a server would be certified for that latest version, but not for the
1143
original GA release. If such a situation arises, testing should also be
1144
done with intervening releases so as to determine the earliest working
1145
version of Ubuntu.
1146
1147
If a server fails certification with a more recent release but works
1148
with an earlier one, this situation is treated as a regression; a bug
1149
report should be filed and note made of the problem in the certificate
1150
request. Please notify your TPM about such problems to facilitate their
1151
resolution.
1152
1153
Because Ubuntu 14.04.1 uses the same 3.13 kernel series as 14.04 GA,
1154
testing 14.04.1 is required only if 14.04 GA fails. (Although 14.04 GA
1155
and 14.04.1 use the same kernel series, 14.04.1 ships with a later
1156
kernel within that series, so it might fix a bug that blocks 14.04 GA
1157
certification.)
1158
1159
If the procedure for installing point releases, as described in
1160
the MANIAC document (available from https://certification.canonical.com),
1161
fails, then you should consult the Server Certification Team.
1162
1163
.. raw:: pdf
1164
1165
PageBreak
1166
1167
Appendix E - Troubleshooting Tips
1168
=================================
1169
1170
Submitting Results
1171
------------------
1172
1173
If submitting results from the Server Test Suite itself fails, you can use
1174
the ``canonical-certification-submit`` program, as described earlier, in
1175
`Manually Uploading Test Results to the Certification Site`. You can try
1176
this on the SUT, but if network problems prevented a successful submission,
1177
you may need to bring the files out on a USB flash drive or other removable
1178
medium and submit them from a computer with better Internet connectivity.
1179
1180
Inconsistent Message when Submitting Results
1181
--------------------------------------------
1182
1183
If you receive a message that looks like the following when using
1184
``canonical-certification-submit``, please be sure to save the
1185
``submission.xml`` file and contact your account manager::
1186
1187
2014-04-28 10:55:33,894 CRITICAL Error: Inconsistent message
1188
1189
Network Problems
1190
----------------
1191
1192
Network problems are common in testing. These problems can manifest as
1193
complete failures of all network tests or as failures of just some
1194
tests. Specific suggestions for fixing these problems include:
1195
1196
- **Check cables and other hardware** -- Yes, this is very basic; but bad
1197
cables can cause problems. For instance, one bad cable at Canonical
1198
resulted in connections at 100Mbps rather than 1Gbps, and therefore
1199
failures. Some of these failures were identified in the output as the
1200
lack of a route to the host. Similarly, if a switch connecting the SUT
1201
to the ``iperf`` server is deficient, it will affect the network test
1202
results.
1203
1204
- **Use the simplest possible network** -- Complex network setups and those
1205
with heavy traffic from computers uninvolved in the testing or those
1206
with multiple switches, bridges, etc., can create problems for
1207
network testing. Simplifying the network in whatever way is practical
1208
can improve matters.
1209
1210
- **Check the iperf server** -- Ensure that the server computer is up and
1211
that the ``iperf`` server program is running on it. Also ensure that the
1212
computer has no issues. For instance, some versions of ``iperf``, when
1213
run in daemon mode, cause the load average to go up every time a client
1214
disconnects. This can bring even a powerful computer to its knees quite
1215
quickly!
1216
1217
- **Ensure the iperf server is on the SUT's local network** -- The
1218
network tests temporarily remove the default route from the routing
1219
table, so the ``iperf`` server must be on the same network segment as
1220
the SUT.
1221
1222
- **Check the SUT's network configuration** -- A failure to configure the
1223
network ports in ``/etc/network/interfaces`` will cause a failure of the
1224
network tests. Likewise, a failure to bring up a network interface
1225
before testing will cause the test to fail, even if
1226
``canonical-certification-server`` detects the interface.
1227
1228
- **Check your DHCP server** -- A sluggish or otherwise malfunctioning
1229
DHCP server can delay bringing up the SUT's network interfaces (which
1230
repeatedly go down and come up during testing). This in turn can cause
1231
network testing failures.
1232
1233
If you end up having to re-run the network tests, either do so from within
1234
``canonical-certification-server`` or be sure to bring down all the network
1235
interfaces except the one you're testing before using ``iperf`` manually.
1236
The way Linux manages network interfaces makes it difficult to ensure that
1237
network traffic will be restricted to a single network port if more than
1238
one is active.
1239
1240
Issues During Testing
1241
---------------------
1242
1243
The testing process should be straightforward and complete without issue.
1244
Should you encounter issues during testing, please contact your account
1245
manager. Be sure to save the ``~/.local/share/plainbox`` and
1246
``~/.cache/plainbox`` directory trees as they will contain logs and other
1247
data that will help the Server Certification Team determine if the issue is
1248
a testing issue or a hardware issue that will affect the certification
1249
outcome.
1250
1251
If possible, please also save a copy of any terminal output or
1252
tracebacks you notice to a text file and save that along with the
1253
previously-noted directories. (Feel free to send us a photo of the
1254
screen taken with a digital camera.)
1255
Loggerhead is a web-based interface for Breezy
Version: 2.0.1