The  Linux sg_dd utility

Introduction
dd like features
sg_dd extras
Continue on error (coe)
Recovered errors
Verbose
Conclusion
 

Introduction

The sg_dd utility is a variant of the standard Unix command dd which copies files. The sg_dd utility is specialized for devices that use the SCSI command set in the Linux operating system. The sg_dd utility is found in the sg3_utils package which targets the linux kernel 2.4 and 2.6 series.

Not all SCSI device types are supported by the sg_dd utility. Obviously those device types that deal with enclosures and medium changers don't have addressable blocks and are not supported. The supported device types are direct access devices (e.g. disks) and cd/dvd devices. The SCSI tape device type is not be supported (but could be to a limited degree). When instructed, the sg_dd utility issues SCSI commands that are defined in SPC-3 (primary commands), SBC-2 (commands for direct access devices (e.g. disks)) and MMC-4 (commands for CD/DVD devices). These SCSI command sets can be found at www.t10.org .

It is becoming common for non-SCSI devices (e.g. ATA disks) to appear as SCSI devices to an operating system via a protocol conversion in an external enclosure and via some transport such as USB or IEEE 1394. The sg_dd utility should work with most of these devices as it tends to use exactly the same SCSI commands that the normal block layer would uses. However, advanced options (e.g. using the 'cdbsz' and 'fua') most likely will be ignored. Apart from CD players over 10 years old, almost all CD/DVD players use the Multi Media Command set (MMC or  MMC-2,3,4,5) as their native command set. The fact that the most common transport is ATAPI (i.e. the ATA packet interface) is irrelevant, to the sg_dd utility they are SCSI devices.

This page outlines the features of the sg_dd utility version 5.42 found in the sg3_utils version 1.16 package. This was released on the 9th August 2005.

dd like features

The basic syntax of the sg_dd utility is the same as the dd command in Unix. That said, the syntax of the dd command in Unix is different from almost all other standard Unix commands. Those familiar with the dd command should not be too surprised by the syntax and semantic of the sg_dd utility. Those not familiar with the dd syntax should be very careful, especially with the 'of' and 'seek' options, both with dd and sg_dd. The recent GNU implementation of the dd command is used as a reference point.

The main options of dd are:
Both dd and sg_dd given these options with suitable arguments will copy (bs * count) bytes from the beginning of <input_file> to the beginning of <output_file>. One restriction that sg_dd imposes when either the <input_file> or <output_file> are accessed via SCSI commands is that <block_size> must match that of the device. Further, if both the <input_file> and <output_file> are accessed via SCSI commands then <block_size> must match that of both devices. In both dd and sg_dd if the if=<input_file> option is not given then stdin is assumed. In both dd and sg_dd if of=<output_file> is not given then stdout is assumed.

The following extensions are found in sg_dd. An <input_file> of "-" is interpreted as stdin; an <output_file> of "-" is interpreted as stdout while an <output_file> of "." is interpreted as /dev/null. [dd interprets input and output file names of "-" literally. dd interprets an output file of "." as the current directory and will not accept it.]

If the 'count' option is not given then an attempt is made to determine the remaining blocks in the file, device or partition. If the input file is stdin and no count is given then a copy will continue until an EOF is detected on the input stream (or something else goes wrong). If the 'count' option is not given then the remaining blocks on both the input and output files are determined (if possible) and if both are found then the minimum of the two counts is used.  The 'skip' option for an input file and the 'seek' option for an output file are taken into account when calculating the remaining number of blocks in a file, device or partition.

If the 'count' option is given then no further checks regarding the remaining length of the input and output files are done and the sg_dd will attempt to copy that number of blocks. A 'count=0' option is valid and all the normal preparations are made including files being opened but no copy takes place. Hence the 'count=0' option can be used to check the syntax is in order and the files are present (see the "Verbose" section below).

Other dd options supported by sg_dd:
If ibs and obs are given to sg_dd then they must be the same as bs (i.e. they are essentially dummies). Both seek and skip are origin zero and default to zero (i.e the start of the output and input files respectively). The 'skip' option cannot be used on an input stream (e.g. stdin) while the 'seek' option cannot be used on an output stream (e.g. stdout).

All numeric arguments can take a multiplier suffix. From sg3_utils version 1.13 sg_dd's multiplier prefixes have been brought into line with those of GNU's dd (in coreutils post a change on  2001-12-18):

Multiplier
 Meaning
x<n>
 *<n>
c
 *1
w
 *2
b
 *512
k  K  KiB
 *1024
KB
 *1000
m  M  MiB
 *1048576
MB
 *1000000
g  G  GiB
 2**30
GB
 10**9
t  T  TiB
 2**40
TB
 10**12

The pattern that starts with "k" and proceeds to "m", "g" and "t" then to "p", "e", "z" and "y"  (not shown in above table). sg_dd only implements as far as "p" (10**15 or 2**50). sg_dd only allows multipliers based on "t" and "p" for 'count', 'skip' and 'seek'.

sg_dd allows numeric arguments to be given in hexadecimal in which case they must be prefixed by either "0x" or "0X". A numeric argument cannot both be in hex and have a suffix multiplier. Hence "0x9" is interpreted as hexadecimal 9 [not (0 * 9)==0]. This is valid "2x4x0xa" and yields 80 (but it isn't very clear).

The following dd options are not supported by sg_dd:
Basically sg_dd does not supported the conversion features of dd. If a conversion is really needed then sg_dd could be piped through dd (or vice versa).

sg_dd extras

The extra options of sg_dd (not found in GNU's dd) are:

extra options in sg_dd
 default
 Brief description
append=0|1
 0
 append to output file (rather than overwrite)
blk_sgio=0|1
 0
 when set access devices via SCSI commands (SG_IO ioctl)
bpt
 128 or 32
 blocks_per_transfer (granularity of each IO). Default is 128 when bs < 2048 (bytes) else the default is 32
cdbsz=6|10|12|16
 10 or 16
 cdb size of SCSI READ and/or WRITE commands
coe=0|1
 0
 continue_on_error when set
dio=0|1
 0
 direct IO (only via sg device nodes)
fua=0|1|2|3
 0
 force_unit_access, 1->if, 2->of, 3->if+of
odir=0|1
 0
 O_DIRECT flag on open() when set
sync=0|1
 0
 SYNCHRONIZE CACHE SCSI command after transfer when set
time=0|1
 0
 when set print elapsed time and throughput after copy
verbose=<n>
 0
 larger <n> is the greater the debug output
--version
print the version number and release date of sg_dd then exit


The sg_dd utility examines the files it is given and treats them differently depending on the category and, in some case,s the command line options given:

File type
 open [when input]
 open [when output]
 IO method
 Notes
normal
 O_RDONLY

 O_WRONLY | O_CREAT
[add O_APPEND if 'append=1']
 Unix read() write()
 Add O_DIRECT if
'odir=1'
stdin or stdout
 [do nothing]
 [do nothing]
 Unix read() write()
/dev/null
 O_RDONLY [do nothing]
 Unix read() if input
 if output then no IO
block device
 O_RDONLY O_WRONLY | O_CREAT
[add O_APPEND if 'append=1']		 Unix read() write() Add O_DIRECT if
'odir=1'
block device [blk_sgio=1]
 O_RDWR or O_RDONLY O_RDWR SCSI commands Opens input O_RDONLY
if O_RDWR fails. Adds
O_DIRECT if 'odir=1'
sg device
 O_RDWR or O_RDONLY
 O_RDWR
 SCSI commands
 Opens input O_RDONLY
if O_RDWR fails
raw device
 O_RDONLY O_WRONLY
 Unix read() write()
scsi tape device
 x
 x
 no IO
 error reported

Some of the above combinations are not sensible (e.g. 'append=1' on a block device). Raw devices were introduced in the lk 2.4 series and are still available in the lk 2.6 series but opening a block device or a normal file with O_DIRECT (in sg_dd using the option 'odir=1') is preferred in the lk 2.6 series. If either the input or output file is a raw device, or 'odir=1' is given then the internal buffers used by sg_dd are aligned to a memory page boundary. A memory page is 4 kilobytes in the i386 architecture. This memory alignment is a technical requirement of both raw devices and normal block devices that implement O_DIRECT.

The 'blk_sgio=1' option is only valid in the lk 2.6 series. This option instructs the sg_dd utility to use the SG_IO ioctl to issue SCSI commands even though the file type is not a scsi generic (sg) device. Currently this will work for all cd/dvd devices irrespective of the transport (except for some very old devices ( > 10 years old)) and block devices representing SCSI disks (e.g. /dev/sda). Even though SCSI tape (st) devices now accept the SG_IO ioctl in the lk 2.6 series, the sg_dd utility expects direct access devices (whereas tapes are sequential access) so sg_dd generates an error if a st device is detected. When the input or output file is a sg device then the 'blk_sgio' option is ignored as IO with that file is always via SCSI commands. When 'blk_sgio=1' is used with a block device and the value given to 'bpt' is too high then the first read will report an EIO (input/output) error; the solution is to lower the 'bpt' value. [The EIO error occurred often with CD/DVD drives which have 2048 bytes sectors, so the 'bpt' default value was reduced for block size >= 2048 bytes.]

If a partition of block device is accessed (e.g. /dev/sda2) when 'blk_sgio=0' (or is not given) then logical block address 0 for the purposes of sg_dd (and its skip and seek options) is the beginning of that partition while the calculated count (e.g. when a 'count' option is not given) is the extent of that partition. However if a partition of block device is accessed (e.g. /dev/sda2) when 'blk_sgio=1' then the partition is ignored and the underlying device is accessed. This means logical block address 0 for the purposes of sg_dd (and its skip and seek options) is the beginning of the device while the calculated count (e.g. when a 'count' option is not given) is the extent of that device.

Note about CDs and count (perhaps DVDs don't have this problem). When a CD is accessed via SCSI commands and the 'count' option is not given then the READ CAPACITY SCSI command is used by sg_dd to determine the number of blocks to read. Unfortunately some MMC standards allow a few of the final blocks to be invalid ("run out" sectors) and this can result in IO errors when the sg_dd utility is used to copy such a CD. The isosize utility can be used to determine the actual size of an ISO9660 file system residing on such a CD. It is often a few blocks shorter than READ CAPACITY reports (for good reason).

When an input or output file is being accessed via SCSI commands the following "extra" options being discussed in this section are active:
When an input or output file is being accessed via normal Unix read() and write() commands the following "extra" options being discussed in this section are active:
If a SIGUSR1 signal is sent to the process identifier (pid) of a running sg_dd utility then the number of blocks copies to that point is output. The copy continues.

When the 'time' option is set, the elapsed time for the copy plus the throughput measured in megabytes (10**6 bytes)  per second is output when the copy is complete (or an error stops the copy). If a SIGUSR1 signal is sent to the process identifier (pid) of a running sg_dd utility which has the 'time' option set the elapsed time and the throughput of the copy to that point is output and the copy continues.

Continue on error (coe)

Recent additions to the sg_dd utility allow it to be used as a copy "of last resort" from failing media. Read errors on an input file taking SCSI commands are "tolerated" when the 'coe' option is set. Write errors from SCSI commands are reported and ignored and the sg_dd utility continues when the 'coe' option is set. [In the case where media errors are causing write errors the reader should check the setting of the AWRE bit in the SCSI "read write error recovery" mode page (see SBC-2 at http://www.t10.org).] As for errors in normal files the reader may wish to examine the "conv=noerror" option of the dd command.

When a SCSI READ command detects an unrecoverable read error it  responds with a sense key of MEDIUM ERROR or HARDWARE ERROR. Additionally it responds with the logical block address of the first (lowest) block that it failed to read in the current READ command. Any valid blocks prior to the "bad" block may or may not have been transferred (depending on several other settings). If 'coe=0' then the sg_dd utility will simply terminate at this point (with a reasonable amount of debug information sent to stderr). If 'coe=1' then the first thing it will try to do is a truncated read up to but not including the bad block. The remaining discussion in this section assumes 'coe=1'.

The "bad" block may or may not be readable by the device. If it is readable then most likely the associated ECC data indicates that it is corrupt and the ECC data cannot recover it. [It is possible that all the corruption is in the ECC data and the payload data is ok.] Such blocks can be retrieved with the READ LONG command which is what the sg_dd utility does. If the READ LONG fails then a block of zeroes is substituted for the bad block in the transfer buffer. Only SBC-2 devices (primarily SCSI disks) optionally support READ LONG. CD/DVD devices (covered in the MMC-4 (draft) standard) do not so a block of zeroes is substituted for them.

There still remains blocks after the "bad" block in the initial transfer to be fetched. Further bad blocks may be detected and if so the algorithm in the last two paragraph is repeated. The result of this process is an imperfect copy with blocks that were read properly placed in the correct relative position in the output. When the 'coe=1' option is given two addition counters are output on completion:
The first counter is the number of unrecovered (read) errors encountered. Any number greater than zero flags an imperfect copy. The second counter (always less than or equal to the first) is the number of "successful" READ LONG SCSI commands performed. If the source media is a CD or DVD then this number will be zero.

Recovered errors

Often errors are recovered using ECC data or by the device retrying (usually re-reading) the media. Also at the first sign of trouble, recoverable errors lead to the block in question being reassigned to another location on the media (automatically when the AWRE and ARRE bits are set in the "read write error recovery" mode page).  The user of such a disk may be blissfully unaware that the disk may be reaching the end of its useful life. Error counters are maintained in the "Read error counter" and "Write error counter" logs pages which can be viewed with smartctl (from smartmontools) and sg_logs (from the same sg3_utils package that sg_dd comes from). Any block that is automatically or manually re-assigned add a new entry to the "grown" defect list which cal be viewed with 'sginfo -G' (also found in the sg3_utils package).

A disk can be instructed to report RECOVERED ERRORs by setting the PER bit in the "read write error recover" mode page. Most often this bit is clear. When sg_dd detects RECOVERED ERRORs it reports them, counts them and continues the copy. Only the lba of the last recovered error in a READ or WRITE SCSI command is reported so there could be more than one recovered error per SCSI command. The 'bpt=1' option could be chosen to limit every SCSI command to a single block transfer (but that would slow things down a fair amount). If the count of recovered errors is greater than zero at the end of the copy then this count is output as well.

There can be other reasons for a sense key of RECOVERED ERROR not directly related to data being currently read or written. SMART alerts (called in SCSI documents "Informational Exceptions") can be conveyed via a RECOVERED ERROR sense key (see the MRIE field in the Informational Exceptions mode page). Such alerts have additional sense codes like "Failure prediction threshold exceeded" and those that contain "impending failure".

Verbose

In the Unix style, sg_dd doesn't output anything (to stderr) during large IO transfers. To get a progress report the SIGUSR1 signal can be sent to the sg_dd process. In the Unix dd command style, sg_dd outputs two lines on completion that show the number of full and partial records in (on the first line) and out (on the second line). Some modern versions of the dd command also output elapsed time and throughput figures (to get this with sg_dd use the "time=1" option).

The sg_dd has a 'verbose' option whose default value is zero. When set this option has the following actions:
  1. show categorization and INQUIRY data (where applicable) for the input and output files. For files, other than streams, the file/device size (and device block size) are output. For SCSI devices mode data relevant to error recovery and caching is output.
  2. same output as 1 plus data for Unix and SCSI commands (cdbs) that are not repeated (i.e. other than Unix read/write and SCSI READ/WRITE commands). Increased error reporting for all SCSI commands
  3. same output as 2 plus data for Unix and SCSI commands (cdbs) that are repeated. For a large copy this will be a lot of output.
  4. maximum amount of debug output. For a large copy this will be a lot of output.
All verbose output is sent to stderr (so that sg_dd usage with "of=-" (copy output to stdout) is not corrupted).

Following is an example of using "verbose=1" to find information about /dev/sda . If no copy is required then setting "count=0" will see to that. Since "dev/sda" is a block device then it would normally be accessed via Unix system commands. The "verbose=1" output is relatively short when "blk_sgio=0" (its default value). The second invocation is with "blk_sgio=1" and a lot more is output. That includes INQUIRY standard response data (e.g. "SEAGATE ..." line) followed by the setting of various mode pages relevant to error processing and caching. See the SBC-2 drafts at www.t10.org for more information. As an example a AWRE bit of 1 means that the disk will automatically reallocate a logical block address (to another sector) if a recoverable error is detected on a WRITE operation. RECOVERED ERRORs are only report on read and write operations when the PER [Post Error] bit is set. The values shown in brackets are:

$ sg_dd if=/dev/sda of=. bs=512 verbose=1 count=0 blk_sgio=0
 >> Input file type: block device
        open input, flags=0x0
 >> Output file type: null device
      [bgs64] number of blocks=17781521 [0x10f5311], block size=512
0+0 records in
0+0 records out


$ sg_dd if=/dev/sda of=. bs=512 verbose=1 count=0 blk_sgio=1
 >> Input file type: block device
        open input(sg_io), flags=0x2
    /dev/sda: SEAGATE   ST39173LC         1234  [pdt=0]
  Read write error recovery mode page:
    AWRE:      1  [cha: y, def: 1, sav: 1]
    ARRE:      1  [cha: y, def: 1, sav: 1]
    RC:        0  [cha: y, def: 0, sav: 0]
    EER:       1  [cha: y, def: 0, sav: 1]
    PER:       1  [cha: y, def: 0, sav: 1]
    DTE:       0  [cha: y, def: 0, sav: 0]
    DCR:       0  [cha: y, def: 0, sav: 0]
  Caching mode page:
    WRE:       1  [cha: y, def: 1, sav: 1]
    RCD:       0  [cha: y, def: 0, sav: 0]
  Control mode page:
    SWP:       0  [cha: n, def: 0, sav: 0]
 >> Output file type: null device
      number of blocks=17781521 [0x10f5311], block size=512
0+0 records in
0+0 records out

Some of the output above is out of order, the "number of blocks" line relates to the input file.

Mode page settings can be examined and changed with a utility like sdparm .

Conclusion

The sg_dd utility maintains syntactic and semantic compatibility with the Unix dd command while allowing precise control over SCSI devices such as SCSI disks and CD/DVD drives. Amongst other uses it can copy data from failing media, continuing as best it can when medium errors are encountered.

Return to main page.

Last updated: 8th August 2005