|
|
Abstract
NDMJOB is a NDMP-compatible backup/restore package, reference implementation, and conformance
test. It is furnished in source code form to the public by Traakan,
Inc. and other contributors free of charge or restriction. NDMP
(Network Data Management Protocol) is an open protocol for network
based backup enabling multi-vendor backup solutions. NDMJOB may
be used as a ready-to-run backup package, used in conjunction
with other NDMP products, as the basis for new products, as a
verification of NDMP products, and as a reference of the NDMP
protocol standards and conventions.
This NDMJOB User Manual explains the capabilities, command usage and options,
and provides examples and guidance.
|
| http://www.traakan.com/ndmjob
|
NDMJOB
latest source and docs |
| http://www.ndmp.org
|
NDMP
protocol status and info |
| ndmp-tech@ndmp.org
|
Questions,
comments and bug reports. |
Table of Contents
Introduction
What is NDMJOB
Background, Purpose, and Motivation
Terminology - "Agents" vs "Client/Server"
Four agents - CONTROL, DATA, TAPE, and ROBOT
Resident and remote agents
NDMJOB Capabilities
Limitations
Command Overview
NDMJOB media (tapes)
Tape records and file marks
Tape labels
Multiple tapes with robot
Media indexes and merging
Cookbook
-- Recipes for Getting Started
Basics of NDMJOB
Command line
Recuring command options
in examples
When not using a robot
When using a robot
Query NDMP agents (-q)
Query resident agents
Query remote agents
Query ROBOT agents
Initialize media (tape) labels (-o init-labels)
Initialize label on local
(resident) tape drive
Initialize label on remote
tape drive
Initialize label(s) using
a robot
List tape labels (-l)
List label on local (resident)
tape drive
List label on remote tape
drive
List label(s) using a robot
Creating a backup (-c)
Creating with resident agents
Creating with resident
DATA remote TAPE
Creating with remote
DATA resident TAPE
Creating with remote
DATA and TAPE
Creating with file index
generation
Creating with a robot
Creating with other options
Listing a backup (-t)
Listing with resident agents
Listing with resident
DATA remote TAPE
Listing with remote
DATA resident TAPE
Listing with remote
DATA and TAPE
Listing with file index
generation
Listing with file index
guidance
Listing with a robot
Recovering (extracting) a backup (-x)
Recovering with resident
agents
Recovering with resident
DATA remote TAPE
Recovering with remote
DATA resident TAPE
Recovering with remote
DATA and TAPE
Recovering with file index
generation
Recovering with file index
guidance
Recovering with a robot
Cleaning up robot messes (-Z)
Logging and debugging (-L, -d)
Using command macros (--MACRO)
Conformance tests (-o test-tape, -o test-mover)
Test NDMP_TAPE without
robot
Test NDMP_TAPE with
robot
Test NDMP_MOVER without
robot
Test NDMP_MOVER with
robot
Running as a daemon
Command line
reference -- ndmjob [OPTION ...] [FILE ...]
Modes (exactly one required)
-c -- create a backup
-t -- list contents of a backup
-x -- extract from a backup
-o init-labels -- init
media labels
-l -- list media labels
-q -- query agent(s)
-Z -- clean up zee mess (put robot
right)
-o init-labels -- init
media labels
-o daemon -- spawn server
for incomming connections
-o test-tape -- test
TAPE agent NDMP_TAPE functions
-o test-mover -- test
TAPE agent NDMP_MOVER functions
CONTROL of DATA agent parameters
-D AGENT -- data agent
-B BUTYPE -- set backup format
(default tar)
-C DIR -- change directory on data
agent before operation
-e PATN -- exclude files matching
pattern
-E NAME=VAL -- add to data agent
environment
-U USER -- user rights to use on
data agent
CONTROL of TAPE agent parameters
-T AGENT -- tape agent if different
than -D
-f TAPE -- tape drive device name
-b N -- block size in 512-byte
records (default 20)
-o tape-timeout=SECONDS
-- how long to retry opening drive (await tape)
-o use-eject=N -- use
eject when unloading tapes (default 0)
CONTROL of ROBOT agent parameters
-R AGENT -- robot agent if different
than -T
-r SCSI -- tape robot target
-o tape-addr=ELEMADDR
-- robot element address of drive (default first)
-o tape-scsi=SCSI --
tape drive SCSI target
Logging options
-d N -- set debug level to N (default
0, max 9)
-L FILE -- set log file (default
stderr, includes debug)
File index options
-I FILE -- set output index file,
enable FILEHIST
-J FILE -- set input index file
Media management options
-m MEDIA -- add entry to media
table (see below)
General options
-n -- no-op, just show how args
were handled
-v -- verbose
--MACRO -- expand argument
MACRO
Definitions
AGENT -- HOST[:PORT][/FLAGS][,USERNAME,PASSWORD]
SCSI -- ADAPTER,[CNUM,]SID,LUN
FILE ...
Argument Macros
(--MACRO)
The ndmjob-args file
How macros are expanded
Creating a macro
Good organization
Example
Log file and index
file line types
#D, #T, #R -- snoop lines
LDl, LTl, LRl -- NDMP_LOG_LOG lines
LD__, LT__, LR__ -- NDMP_LOG_DEBUG lines
LDf, LTf, LRf -- NDMP_LOG_FILE lines (file
recovered)
CL -- Control (ndmjob) log lines
LDp, LDn, LDd -- NDMP_FH_ADD_UNIX_xxx index
lines
CM -- Control (ndmjob) media line
DEc, DEi, DEa -- Post-backup data environment
Introduction
What is NDMJOB
NDMJOB is a command which manages
tape backup and recover operations using NDMP (Network Data Management
Protocol). NDMJOB, and NDMP in general, is a framework for managing backups
and not a backup method. There is no NDMJOB tape format, for example.
Rather, NDMJOB is a "wrapper" around programs which do the real work,
and provides an interoperable way to orchestrate backup/recover operations.
Background, Purpose, and
Motivation
NDMJOB was originally developed
by Traakan, Inc as a testing tool. It gradually became more sophisticatd
to test tape robotics, and directed access recovery. Further, it became
a vehicle for identifying and resolving ambiguities in the NDMP specification.
As NDMJOB evolved, it became a generally useful utility as well as a testing
tool.
Traakan released NDMJOB
into the public domain in the spirit of fostering interoperability,
and fostering the deployment of NDMP-based backup systems. The hope
is that a ready-to-go NDMP implementation with straight forward porting
requirements will result in widely deployed, compatible, and interoperable
NDMP packages.
NDMJOB is more library than
command, and this library may serve as the foundation for more elaborate
CONTROL packages and/or better user interfaces. The NDMP behaviour and
conventions are isolated in operating system generic modules. Operating
system specific portions, such as device access and manipulation, are
isolated in per-system modules.
Please e-mail comments and
contributions to ndmjob@traakan.com and/or ndmp-tech@ndmp.org.
Terminology - "Agents" vs
"Client/Server"
The NDMJOB documentation and
code refer to "Agents", while the NDMP specification refers to "Clients"
and "Servers". The client/server model -- or duality -- is the most commonly
used for networking protocols. Peer-to-peer and agents are other terms
frequently used.
If you say "NDMP Client"
to a mixed audience of backup folks, half of them are thinking the same
thing you are and the other half aren't. The conversation invariably
comes to a halt while everybody syncs up on terminology. If you say
"NDMP Control agent" to an audience, everybody knows what you mean.
As NDMJOB and NDMP are introduced into more elaborate packages -- and
the client/server duality becomes difficult to apply -- the "agent"
terminology will continue to work.
The NDMP specification refers
to client/server. This is heavily influenced by the direction of connection.
The client-side initiates activity, connects to one or more servers,
the uses the NDMP to coordinate activities. To people accustomed to
working with networking protocols, this terminology is crystal clear
and easily recognized: the elements which "connect" are clients, and
the elements which "listen" are the servers. Yet, ignore how connections
are established for a moment, and for NDMP the recognition gets more
difficult.
The promotional materials
and personel of commercial backup packages also refer to client/server,
and the meaning is reversed from that of the NDMP specification. This
has an interesting history which we won't explore here. To these folks
the servers are the elements closest to objective, and the clients are
elements which interact and support the server. The objective, again
to these folks, is the management of backup schedules, file indexes,
recovery request queues, media aging and reassigment, etc, etc, etc.
A tape drive, or anything that helps talk to it, is a supportive role
to the server.
The client/server duality
breaks down for backup packages precisely because backup packages have
more than two elements. Although NDMJOB is a stand-alone CONTROL agent,
the CONTROL agent for commercial backup packages is encapsulated within
standing processes ("servers") which await connections from administrative
and user interfaces, and peer processes running the same package on
other machines. At appropriate times, these packages initiate and orchestrate
backup activity. It's easy to imagine backup packages with three, four,
or even five key elements. Architectures and systems with five elements
defy traditional client/server duality.
NDMJOB introduces its own
wrinkle with its "resident" agents. A resident agent does not involve
connecting and listening, nor any network connection at all. Resident
agents are accessed as a subroutine package.
The term "agent" is used
for SNMP (Simple Network Management Protocol) because large SNMP systems
also have more than two key elements. The issues surrounding NDMP and
backup packages are similar to those of SNMP, so we adopted the "agents"
terminology rather than inventing a new one. At times, client/server
is a handy short hand for what's connecting and what's listening.
Four agents - CONTROL, DATA,
TAPE, and ROBOT
The NDMP architecture
has four roles, or agents, which NDMJOB implements. Each agent may occur
on a different host. TCP/IP connections are used to control and coordinate. Each agent may be filled by a different NDMP-compatible package. NDMJOB
conforms to the NDMP specifications and conventions, and will interoperate with other conforming packages. The four agents are:
Resident and remote agents
An NDMJOB process can simultaneously
implement all four agents. These are called "resident" agents.
NDMJOB can act as a CONTROL
agent and direct "remote" agents.
Finally, NDMJOB can be
run as a daemon which awaits connections then acts as one or more
of the DATA, TAPE, and ROBOT agents under the direction of the connecting
CONTROL agent.
NDMJOB Capabilities
The capabilities, in broad terms,
are:
- Create, list, and recover
backups.
- Generation of file index
during create and list operations.
- Rapid seek (indexed)
recovery of files.
- Backups may span multiple
tapes (with tape robot).
- A tape may be used for
multiple backups (filemark skipping).
- Basic tape robot handling,
including parameter and status queries.
- Tape labels may be established,
and later enforced during backup operations.
- Tape label listings (inventory)
of tapes in robot (all or selectively).
- Query NDMP agents for
capabilities.
- Arbitrary specification
of a variety of NDMP parameters, including backup type, block size,
"environment" variables, etc.
- Macros may be defined
making command lines easier.
- Comprehensive logging
capabilities in selectable details. Illustrates the NDMP traffic for
specific tasks.
- Conformance tests of
NDMP agents.
- Supports both NDMPv2
and NDMPv3 simultaneously, and can coordinate the activities of two
agents with different versions.
Limitations
The limitations are:
- No scheduling of backups.
NDMJOB performs operations immediately.
- No scheduling, aging,
or selection of media. NDMJOB uses media as specified by the command
parameters.
- No provisions for multi-tape
operations without a robot.
- No directed recovery
based on indexes generated by NDMP_FH_NODE (dump-type indexes).
Command Overview
The output from ndmjob -help
gives a summary of the options:
ndmjob -v -- print version and configuration info
ndmjob OPTIONS ... FILES ...
Modes (exactly one required)
-c -- create a backup
-t -- list contents on a backup
-x -- extract from a backup
-l -- list media labels
-q -- query agent(s)
-Z -- clean up zee mess (put robot right)
-o init-labels -- init media labels
-o daemon -- launch session for incomming connections
-o test-tape -- test TAPE agent NDMP_TAPE functions
-o test-mover -- test TAPE agent NDMP_MOVER functions
General and Logging parameters
--MACRO -- expand MACRO from ndmjob-args file
-d N -- set debug level to N (default 0, max 9)
-L FILE -- set log file (default stderr, includes debug)
-n -- no-op, just show how args were handled
-v -- verbose, same messages as -d1 to standard out
CONTROL of DATA agent parameters
-D AGENT -- data agent (see AGENT below)
-B TYPE -- set backup format (default tar)
-C DIR -- change directory on data agent before operation
-e PATN -- exclude files matching pattern
-E NAME=VAL -- add to data agent environment
-F FILE -- add FILE arg (used to not confuse arg processing)
-I FILE -- set output index file, enable FILEHIST (default to log)
-J FILE -- set input index file (default none)
-U USER -- user rights to use on data agent
-o rules=RULES -- apply RULES to job (see RULES below)
CONTROL of TAPE agent parameters
-T AGENT -- tape agent if different than -D (see AGENT below)
-b N -- block size in 512-byte records (default 20)
-f TAPE -- tape drive device name
-o tape-timeout=SECONDS
-- how long to retry opening drive (await tape)
-o use-eject=N
-- use eject when unloading tapes (default 0)
CONTROL of ROBOT agent parameters
-R AGENT -- robot agent if different than -T (see AGENT below)
-m MEDIA -- add entry to media table (see below)
-o tape-addr=ELEMADDR
-- robot element address of drive (default first)
-o tape-scsi=SCSI
-- tape drive SCSI target (see below)
-r SCSI -- tape robot target (see below)
Definitions:
AGENT HOST[:PORT][/FLAGS][,USERNAME,PASSWORD]
FLAGS [23][ntm] 2->v2 3->v3 n->AUTH_NONE t->TEXT m->MD5
AGENT . (resident)
SCSI ADAPTER,[CNUM,]SID,LUN
MEDIA [TAPE-LABEL][+SKIP-FILEMARKS][/WINDOW-SIZE][@ELEMADDR]
RULES:
none -- no additional rules (default)
|
NDMJOB media (tapes)
Tape records and file
marks
Tape labels
Multiple tapes with robot
Media indexes and merging
Cookbook -- Recipes for Getting
Started
Basics of NDMJOB
Command line
Recuring command options in examples
The examples that follow contain recuring command line options. We explain them here:
| -D. |
Use the resident DATA agent. Disk files are on the local
system on which NDMJOB is running. |
| -D filebox,ndmp,sesame |
Use the remote DATA agent on system "filebox", user "ndmp",
password "sesame". The disk files are on that remote system.
|
| -B tar |
The backup format is "tar". |
| -C /u1 |
The DATA agent should change directory to "/u1". All disk
file path names, whether backup or recovery, are relative to
this directory. |
| -T. |
Use the resident TAPE agent. The tape drive is on the local
system on which NDMJOB is running. |
| -T jookbox,ndmp,shazam |
Use the remote TAPE agent on system "jookbox", user "ndmp",
password "shazam". The tape drive is on that remote system.
|
| -f /dev/rtape0 |
The tape drive device name is "/dev/rtape0". |
| -R tomservo,ndmp,mocha |
Use the remote ROBOT agent on system "tomservo", user "ndmp",
password "mocha". The robot (SCSI Media Changer) is on a SCSI
bus on that remote system. This sometimes happens when there
is one robot with multiple tape drives. The tape drives may
each be connected to a different system (via SCSI), but the
robot is connected to only one of them. |
| -r aha1,5,1 |
The tape robot is on device "aha1", SCSI ID (SID) "5", logical
unit (LUN) "1". Notice that the controller number is not specified.
|
| -r /dev/scsi,1,3,0 |
The tape robot is on device "/dev/scsi", controller or bus
"1", SCSI ID (SID) "3", logical unit (LUN) "0". |
| -x |
|
| -x |
|
When not using a robot
When using a robot
Query NDMP agents (-q)
The query (-q) operation simply
connects, authenticates, and retrieves a bit of NDMP information from
the agents. Which agents are queried is determined by which agents were
specified on the command line (-D, -T, -R).
When querying a DATA agent
(-D), the backup type (-B) is significant. Capability information of
the DATA agent for the particular backup type is obtained. Expect an
error if the backup format isn't supported.
When querying a ROBOT agent
(-R), the robot device must be specified (-r). The parameters and status
of the robot are queried and reported, such as slot addresses, tapes
present, etc.
Query resident agents
The first example always works.
The second shows an error for an unsupported backup type. The third
shows a usage error.
% ndmjob -q -D. -T.
% ndmjob -q -D. -B bogus
% ndmjob -q -R.
Query remote agents
This is a handy way to verify
the availability of remote agents, and to verify authentication (user/password)
parameters.
% ndmjob -q -D filebox,ndmp,sesame
% ndmjob -q -T jookbox,ndmp,shazam
% ndmjob -q -D filebox,bad,bad
Query ROBOT agents
Use this query to verify availability,
authentication, and interoperability with a robot (SCSI Media Changer).
The results of querying
a robot contain slot address and content information robot. Note that
all the smarts for manipulating the robot are in NDMJOB itself. The
ROBOT agent is little more than a SCSI pass-thru.
The NDMJOB parameters
for using media in a robot require the bona-fide addresses, so this
is also a handy way to discover them. For example, the first tape
slot might be at address 100. When you want to use the first tape
slot (-m option), you must use 100 as its address (-m@100).
The first example is when
the robot is on the same as the tape drive. The second example is
when the robot is on a different host, and thus separate TAPE and
ROBOT agents are required.
% ndmjob -q -T jookbox,ndmp,shazam -r aha1,5,1
% ndmjob -q -R tomserver,ndmp,mocha -r aha1,5,1
Initialize media (tape)
labels (-o init-labels)
The initialize labels (-o init-labels)
operation writes a one record tape file with a short name at the begining
of the tape. This name should be unique for each tape. During other operations,
the tape labels may be enforced. Such operations only proceed if the tape
label(s) is as specified by the -m options.
The -m option is used to
specify the label for the tape. When using a robot, multiple -m options
may be given, and each -m option must specify the slot address.
For label operations, the
TAPE agent is little more than a pass-thru. The composition of the tape
label is done by NDMJOB (the CONTROL agent).
Initialize label on local
(resident) tape drive
% ndmjob -o init-labels -T. -f /dev/rtape0 -mT123
Initialize label on remote
tape drive
% ndmjob -o init-labels -T jookbox,ndmp,shazam -f /dev/rtape0 -mT123
Initialize label(s) using
a robot
In this example, separate
TAPE and ROBOT agents are being used. Also, the verbose (-v) option
has been added so that NDMJOB tattles as it makes progress. This is
somewhat comforting during the long delays while tape drive and robot
mechanics are under way.
% ndmjob -o init-labels -v \
-T jookbox,ndmp,shazam -f /dev/rtape0 \
-R tomservo,ndmp,mocha -r aha1,5,1 \
-mT123@100 -mT124@101 -mT125@102
List tape labels (-l)
The list labels (-l) operation
simply reads, audits, and reports the label on the tape. The label must
have been written by NDMJOB.
For label operations, the
TAPE agent is little more than a pass-thru. The composition of the tape
label is done by NDMJOB (the CONTROL agent).
List label on local (resident)
tape drive
% ndmjob -l -T. -f /dev/rtape0
List label on remote tape
drive
% ndmjob -l -T jookbox,ndmp,shazam -f /dev/rtape0
List label(s) using a
robot
The first example will report
the label of all tapes in the robot. There is no -m directive. The robot
status information which indicates which slots have a tape and which
don't is respected.
The second example only
checks the slots specified by the -m options.
% ndmjob -l -v \
-T jookbox,ndmp,shazam -f /dev/rtape0 \
-R tomservo,ndmp,mocha -r aha1,5,1
% ndmjob -l -v \
-T jookbox,ndmp,shazam -f /dev/rtape0 \
-R tomservo,ndmp,mocha -r aha1,5,1 \
-m@100 -m@102 -m@103
Creating a backup (-c)
Creating with resident
agents
% ndmjob -c -D. -f /dev/rtape0 /u1/home
% ndmjob -c -D. -f /dev/rtape0 -F /u1/home
% ndmjob -c -D. -F /u1/home -f /dev/rtape0
% ndmjob -c -D. -F /u1/home -f /dev/rtape0 -mT123
Creating with resident
DATA remote TAPE
% ndmjob -c \
-D. -F /u1/home \
-T jookbox,ndmp,shazam -f /dev/rtape0 \
-mT123
Creating with remote DATA
resident TAPE
% ndmjob -c \
-D filebox,ndmp,sesame -F /u1/home \
-T . -f /dev/rtape0 \
-mT123
Creating with remote DATA
and TAPE
% ndmjob -c \
-D filebox,ndmp,sesame -F /u1/home \
-T jookbox,ndmp,shazam -f /dev/rtape0 \
-mT123
% ndmjob -c \
-D filebox,ndmp,sesame -F /u1/home \
-f /dev/rtape0 \
-mT123
Creating with file index
generation
% ndmjob -c \
-D. -F /u1/home \
-f /dev/rtape0 \
-mT123 \
-I file-index
Creating with a robot
% ndmjob -c \
-D. -F /u1/home \
-T jookbox,ndmp,shazam -f /dev/rtape0 \
-r aha1,5,1 \
-mT123@100 -mT124@101 \
-I file-index
% ndmjob -c \
-D. -F /u1/home \
-T jookbox,ndmp,shazam -f /dev/rtape0 \
-R tomservo,ndmp,mocha -r aha1,5,1 \
-mT123@100 -mT124@101 \
-I file-index
% ndmjob -c \
-D filebox,ndmp,sesame -F /u1/home \
-T jookbox,ndmp,shazam -f /dev/rtape0 \
-R tomservo,ndmp,mocha -r aha1,5,1 \
-mT123+3/100m@100 -mT124@101 \
-I file-index
Creating with other options
% ndmjob -c -D. -f /dev/rtape0 -C /u1 home
% ndmjob -c -D. -f /dev/rtape0 \
-e '*.o' -e 'core' -e '*.core' \
-C /u1 home
Listing a backup (-t)
Listing with resident
agents
Listing with resident
DATA remote TAPE
Listing with remote DATA
resident TAPE
Listing with remote DATA
and TAPE
Listing with file index
generation
Listing with file index
guidance
Listing with a robot
Recovering (extracting)
a backup (-x)
Recovering with resident
agents
Recovering with resident
DATA remote TAPE
Recovering with remote
DATA resident TAPE
Recovering with remote
DATA and TAPE
Recovering with file index
generation
Recovering with file index
guidance
Recovering with a robot
Cleaning up robot messes
(-Z)
Logging and debugging (-L,
-d)
Using command macros (--MACRO)
Conformance tests (-o test-tape,
-o test-mover)
Test NDMP_TAPE without
robot
Test NDMP_TAPE with robot
Test NDMP_MOVER without
robot
Test NDMP_MOVER with robot
Running as a daemon
Command line reference --
ndmjob [OPTION ...] [FILE ...]
Modes (exactly one required)
-c -- create a backup
The processing sequence is:
- Expand command line
argument macros
- Process and audit command
line arguments
- If a robot is being
used (see -r), make sure the robot is ready and try to remedy if
not
- Connect to the DATA
agent
- Connect to the TAPE
agent
- Load first media item
(see -m)
- Start the NDMP_MOVER
interface (NDMP_MOVER_LISTEN)
- Capture the image window
begin parameters
- Compose and issue the
NDMP_DATA_START_BACKUP request
- Wait for the operation
to start, run, and complete. Log and index information is captured.
Tape changes are handled (see -m) if a robot is being used. This
monitoring continues until something (DATA or TAPE agent) halts.
- Capture the post-backup
environment (NDMP_DATA_GET_ENV).
- Capture the image window
end parameters (for last tape) (see log line type CM)
- Unload the last media
item (see -m)
The DATA environment is
constructed from -E, -C, -B, -e, etc. The bu_type is taken from -B.
The mover_addr is determined by the return from NDMP_MOVER_LISTEN.
This will be NDMP_ADDR_LOCAL if no -T option was given, otherwise
NDMP_ADDR_TCP.
-t -- list contents of
a backup
The processing sequence is:
- Expand command line
argument macros
- Process and audit command
line arguments
- If a robot is being
used (see -r), make sure the robot is ready and try to remedy if
not
- Digest the input index
(see -J) to determine the backup image offsets for each FILE argument,
and to determine the media and windows
- Connect to the data
agent
- Connect to the tape
agent
- Load first media item
(see -m)
- Start the NDMP_MOVER
interface (NDMP_MOVER_LISTEN)
- Set the image window
begin/end parameters
- Compose and issue the
NDMP_DATA_START_RECOVER_FILEHIST request
- Wait for the operation
to start, run, and complete. Log and index information is captured.
Tape changes are handled (see -m) if a robot is being used. This
monitoring continues until something (DATA or TAPE agent) halts.
- Unload the last media
item (see -m)
Notice "Non-standard" \ "NDMP_DATA_START_RECOVER_FILEHIST
is a Traakan extension to the NDMPv2 protocol. It is evidently being
adopted as part of NDMPv3. Because it is a separate request type, there
is no risk of incompatiblity with other NDMPv2 CONTROL agents. NDMPv2
DATA and TAPE agents which do not support NDMP_DATA_START_RECOVER_FILEHIST
should simply respond with NDMP_NOT_SUPPORTED_ERR. If they don't, simply
don't use -t on such agents." DetailTableBegin DetailTableEnter NDMP_DATA_START_RECOVER_FILEHIST
The DATA environment is constructed from -E, -C, -B, -e, etc., and from
DATA Environment (DE) records in the -J input index, if any. The bu_type
is taken from -B. The nlist is left empty. The mover_addr is determined
by the return from NDMP_MOVER_LISTEN. This will be NDMP_ADDR_LOCAL if
no -T option was given, otherwise NDMP_ADDR_TCP. DetailTableEnd
-x -- extract from a backup
The processing sequence is:
- Expand command line
argument macros
- Process and audit command
line arguments
- If a robot is being
used (see -r), make sure the robot is ready and try to remedy if
not
- Digest the input index
(see -J) to determine the backup image offsets for each FILE argument,
and to determine the media and windows
- Connect to the data
agent
- Connect to the tape
agent
- Load first media item
(see -m)
- Start the NDMP_MOVER
interface (NDMP_MOVER_LISTEN)
- Set the image window
begin/end parameters
- Compose and issue the
NDMP_DATA_START_RECOVER_FILEHIST request
- Wait for the operation
to start, run, and complete. Log and index information is captured.
Tape changes are handled (see -m) if a robot is being used. This
monitoring continues until something (DATA or TAPE agent) halts.
- Unload the last media
item (see -m)
DetailTableBegin DetailTableEnter
NDMP_DATA_START_RECOVER The DATA environment is constructed from -E,
-C, -B, -e, etc., and from DATA Environment (DE) records in the -J input
index, if any. The bu_type is taken from -B. The nlist is determined
by the FILE arguments. The fh_info is each nlist entry is determined
from the -J input index. See the discussion of -J for more details.
The mover_addr is determined by the return from NDMP_MOVER_LISTEN. This
will be NDMP_ADDR_LOCAL if no -T option was given, otherwise NDMP_ADDR_TCP.
DetailTableEnd
-o init-labels -- init
media labels
The processing sequence is:
- Expand command line
argument macros
- Process and audit command
line arguments
- If a robot is being
used (see -r), make sure the robot is ready and try to remedy if
not
- Connect to the tape
agent
- Load first media item
(see -m)
- Use NDMP_TAPE_WRITE
to write a 512-byte tape label according to the LABEL portion of
the -m argument
- Unload the media
- Repeat the process for
each -m argument
-l -- list media labels
The processing sequence is:
- Expand command line
argument macros
- Process and audit command
line arguments
- If a robot is being
used (see -r), make sure the robot is ready and try to remedy if
not
- Connect to the tape
agent
- Load first media item
(see -m). If no -m arguments are given and a robot is being used,
then -m arguments for each slot containing a tape are sythesized.
If no -m arguments are given and there is no robot, then one -m
argument is synthesized for the tape currently in the drive.
- Use NDMP_TAPE_READ to
read the 512-byte tape label
- Report what was found
- Unload the media
- Repeat the process for
each -m argument
-q -- query agent(s)
-Z -- clean up zee mess
(put robot right)
The processing sequence is:
- Expand command line
argument macros
- Process and audit command
line arguments
- If a robot is being
used (see -r), make sure the robot is ready and try to remedy if
not
-o init-labels -- init
media labels
-o daemon -- spawn server
for incomming connections
-o test-tape -- test TAPE
agent NDMP_TAPE functions
-o test-mover -- test
TAPE agent NDMP_MOVER functions
CONTROL of DATA agent parameters
-D AGENT -- data agent
-B BUTYPE -- set backup
format (default tar)
This sets the backup type
which is specified to certain DATA agents requests. This also sets TYPE=BUTYPE
in the DATA agent's environment.
-C DIR -- change directory
on data agent before operation
This sets PREFIX=DIR
in the DATA agent's environment. For recovery operations, this also
specifies the "top" directory to receive the recovered files. The DIR
is simply preceded to the recovery path names.
-e PATN -- exclude files
matching pattern
Each -e option adds an EXCLUDE=PATN
entry in the DATA agent's environment. More than one may appear. Some
DATA agents will not recognize multiple environment variables with the
same name.
-E NAME=VAL -- add to
data agent environment
-U USER -- user rights
to use on data agent
This sets USER=USER
in the DATA agent's environment. It is equivalent to -E USER=USER.
Absence of option -U causes no USER= variable to be set, but the USER=
may still be specified by -E.
CONTROL of TAPE agent parameters
-T AGENT -- tape agent
if different than -D
-f TAPE -- tape drive
device name
-b N -- block size in
512-byte records (default 20)
-o tape-timeout=SECONDS
-- how long to retry opening drive (await tape)
-o use-eject=N -- use
eject when unloading tapes (default 0)
CONTROL of ROBOT agent parameters
-R AGENT -- robot agent
if different than -T
-r SCSI -- tape robot
target
-o tape-addr=ELEMADDR
-- robot element address of drive (default first)
-o tape-scsi=SCSI -- tape
drive SCSI target
Logging options
-d N -- set debug level to N (default 0, max 9)
| Level |
Description |
| 0 |
Messages of a critical nature. Level 0 messages are always
delivered. |
| 1 |
Progress messages. These are messages deemed useful when
-v is given. |
| 2 |
Detailed progress messages. These are messaged deemed always
worth logging when -L is given. |
| 3 |
Extremely detailed progress messages. These are the first
level of real debugging. Such messages indicate when specific
device operations are being commenced. |
| 4 |
No specific meaning. |
| 5 |
Summary protocol logging. Protocol message summaries are
logged showing the times of each NDMP request and reply. No
message parameters are displayed. This is a good level of detail
to see how the overall operation is implemented via the NDMP
protocol. |
| 6 |
Detailed protocol logging. Protocol messages, including the
message bodies, are reported. |
| 7 |
No specific meaning. |
| 8 |
Low level "syscall" reporting. Shows when read(2) and write(2)
calls occur with byte-counts, and return values. This only applies
to the NDMP control connections. Normal file activity (like
the log file) is not so reported. |
| 9 |
Painful detail. Reports hex dumps of the data sent and received
on the NDMP control connection. |
-L FILE -- set log file (default stderr, includes debug)
Sets the output file for log messages. When -L is given, the debug level is raised to at least 2.
A subsequent -d option may lower it. All log messages from the DATA,
TAPE, and ROBOT agents are delivered to the log file, or to stderr of
no log file is set by -L.
See the Log
file and index file line types section for a description of log
file contents.
File index options
-I FILE -- set output
index file, enable FILEHIST
Sets the output file for index
records. This also has the side effect of setting HIST=Yes in the DATA
agent's environment. In the absence of a -I option, HIST=No is set in
the DATA agent's environment. The FILE may be a dash (-) to indicate
the a file index should be generated and the records should be saved
in the same file as the log (see -L).
-J FILE -- set input index
file
Media management options
-m MEDIA -- add entry
to media table (see below)
The MEDIA argument follows
the synopsis:
[TAPE-LABEL][+SKIP-FILEMARKS][/WINDOW-SIZE][@SLOT]
The TAPE-LABEL, if present,
must be first. The other three fields may occur in any order.
TAPE-LABEL
A short name, up to thirty
characters, which uniquely identifies the tape. This should consist
of only alphanumeric characters.
For normal operations
(-c, -t, -x, etc), the tape label must match before a tape is used.
If no tape label is given, the tape is used unconditionally.
For label initialization
(-o init-labels), the tape label is initialized as specified.
+SKIP-FILEMARKS
A decimal value for the
number of filemarks to skip before using the tape. If not specified,
the default value is 1 if there is a tape label, 0 if no tape label.
/WINDOW-SIZE
A decimal value, optionally
followed by a scale letter (k, m, or g), specifying the number of
usuable bytes in the tape file. If not specified, the tape is used
until an exceptional condition occurs (end-of-medium, end-of-file,
etc). This value must be a multiple of the block size (see -b option).
@SLOT
A decimal value for the element address of the slot containing the tape in the tape robot.
This is the absolute address, not a slot number. This is required
if a tape robot is being used (see -r option). It is ignored if there
is no tape robot.
| -m T103+2/200m |
-m T509+3/90m |
-m T201/400m |
| TAPE LABEL: T103 |
| (filemark) |
| (content 1) |
| (filemark) |
Window, 200mb
0-200mb |
| (filemark) |
| . . . |
| (filemark) |
| (filemark) |
|
| TAPE LABEL: T509 |
| (filemark) |
| (content 1) |
| (filemark) |
| (content 2) |
| (filemark) |
Window, 90mb
200-290mb |
| (filemark) |
| . . . |
| (filemark) |
| (filemark) |
|
| TAPE LABEL: T201 |
| (filemark) |
Window, 400mb
290-490mb |
| (filemark) |
| . . . |
| (filemark) |
| (filemark) |
|
|
General options
-n -- no-op, just show
how args were handled
-v -- verbose
Messages of level 1 (see -d)
are delivered to standard output. The -v option may be given more than
once, increasing the level of detail. One -v serves to monitor the progress
of the operation.
--MACRO -- expand argument
MACRO
A --MACRO command line argument
is used as a shorthand for often repeated, tedious, series of options.
Macros are fully described in the Argument Macros
section.
The macros are defined
in a file. The macro name, MACRO, is used as a key in this file. All
matching definitions replace the --MACRO on the command line exactly
where the --MACRO appears.
Macros are fully processed
before the command line is interpretted; it is not done incrementally
on an argument by argument basis. The entire command line is scanned
for --MACRO forms, which are expanded. Then the arguments are interpretted.
The --MACRO forms must be used carefully when a macro defined FILE
arguments. If the --MACRO appears before the end of the options, the
options which follow it will not be recognized as options but rather
as FILE arguments.
Definitions
AGENT -- HOST[:PORT][/FLAGS][,USERNAME,PASSWORD]
SCSI -- ADAPTER,[CNUM,]SID,LUN
FILE ...
Argument Macros (--MACRO)
The ndmjob-args file
The sh(1) environment variable
NDMJOB_ARGFILE may contain a pathname to the file containing the macro
definitions. If NDMJOB_ARGFILE is not set, the file ndmjob-args in the
current directory is used.
Lines in the file have the
synopsis:
MACRO: ARG ...
The MACRO name must be at the
begining of the line (no leading whitespace), and must immediately be
followed by a colon (:).
Blank lines and lines begining
with # (pound sign) are ignored.
How macros are expanded
Macros are expanded prior to
command line argument processing. It is not done incrementally. Because
of this, there is no way to specify the path to the ndmjob-args file on
the command line (chicken and egg).
For example, given a set
of macros like:
create: -cv -Ic-index
server: -D filebox,ndmp,sesame -C /u1
tape: -T jookbox,ndmp,shazam -f /dev/rtape0
and given a command line like:
% ndmjob --create --server --tape databases
the command arguments actually
processed are
% ndmjob -cv -Ic-index \
-D filebox,ndmp,sesame -C /u1 \
-T jookbox,ndmp,shazam -f /dev/rtape0 \
databases
The motivation of the -n
option is to see what actual command arguments, and their interpretation,
took place after macro expansion.
For each --MACRO command
argument, the ndmjob-args file is scanned for lines that begin with
MACRO: (the name immediately followed by a colon). It is an error if
no matching lines are found. Each matching line is parsed according
to white space into separate command arguments. The result is inserted
into the command argument list at the position of the original --MACRO
argument, and the resulting argument list is rescanned.
Because of the rescanning,
macros may invoke other macros.
No sh(1)-like variable expansion,
wild card (*?) expansion, nor quoting are supported.
Creating a macro
Good organization
Example
# ndmjob example macros file
# AGENTS -- one place to change for hostnames, usernames, and passwords
agent-filebox: filebox,ndmp,sesame
agent-jookbox: jookbox,ndmp,shazam
agent-tomservo: tomservo,ndmp,mocha
# ROBOT TYPES (quirks)
# TODO: still need to establish options for quirks (e.g. -o park-pos)
robot-quirk-archive-28849: -o use-eject=0
robot-quirk-exabyte-exb10e:
robot-quirk-exabyte-exb210:
# ROBOTS
# Notice the use of a macro for the -R option
robot-tomservo-1: -R --agent-tomservo
robot-tomservo-1: -r aha1,5,1
robot-tomservo-1: --robot-quirk-archive-28849
# TAPE DRIVES
# Put -T off till the last minute
tape-jookbox-1: -f /dev/rtape0
# LOG/INDEX short hands
log+index: -L ndmjob.log -I ndmjob.ind
just-log: -L ndmjob.log -I-
log-no-index: -L ndmjob.log
dbg: -L dbg.log -I dbg.ind -d5
# EXCLUDES
# These patterns are very common.
std-excl: -e *.core
std-excl: -e BUILD*/*.o
std-excl: -e home/*/.netscape/cache
# This example shows a specific exclude for a large directory containing
# easily recovered stuff.
bilbo-excl: -e home/bilbo/public-imports
# DEVICE COMBOS -- Handy since using a tape in a robot w/o the
# robot is hard. The -T is not given here since, when
# using -D --jookbox, it would force using the TAPE agent
# in remote mode (NDMP_ADDR_TCP). Whenever possible, we want to
# use the TAPE agent in local mode (NDMP_ADDR_LOCAL) for performance
_robotape-jookbox-1: --robot-jookbox-1 --tape-jookbox-1
robotape-jookbox-1: -T --agent jookbox --_robot-jookbox-1
# FILE SETS -- Combinations of DATA agent, root (-C), and exclude patterns
files-filebox-/u1: -D --agent-filebox
files-filebox-/u1: -C /u1
files-filebox-/u1: --std-excl --bilbo-excl
# BACKUP CONFIGURATIONS
bu-u1: --files-filebox-/u1
bu-u1: --robotape-jookbox-1
bu-home-dirs: --bu-u1
bu-home-dirs: -F home
# All set for requests like:
# ndmjob -c --log+index --bu-home-dirs -mT103+3@3
# ndmjob -cv --dbg --bu-u1 -mT100@2 home/bilbo
# ndmjob -q -D --agent-filebox
# ndmjob -q -T --agent-jookbox
# ndmjob -q --robotape-jookbox-1
# ndmjob -lv --robotape-jookbox-1 -m@2 -m@4
Log file and index file line
types
This section describes the contents
of the log file (ref -L) and the index file (ref -I).
The files are simple text files consisting of ordinary lines. All lines
begin with a short two or three character tag. This makes examining the
files with simple tools like grep(1) convenient.
Time stamps have the form
H:MM:SS.MMM, which are relative to the start time of the ndmjob
run.
All tags start with one of
D, T, R, or C, which indicated the origin of the information: D for the
DATA agent, T for the TAPE agent, R for the ROBOT agent, and C for the
CONTROL (ndmjob) agent.
#D, #T, #R -- snoop lines
Snoop lines show communication,
using the NDMP protocol, between the CONTROL (ndmjob) agent and the other
agents, from the point of view of the CONTROL agent. The detail shown
depends on the debug level (ref -d).
The snoop lines all begin
with #, and are easily ignored by using grep -v '^#'.
#x H:MM:SS.MMM C NDMP_... SEQ Send
These lines indicate a request
(C for call, a snoop convention) from the CONTROL agent to agent x.
The type of request is shown by NDMP_... The SEQ sequence number is
a packet number which, per the NDMP specification, starts at one (1)
and counts up. The SEQ number is a count of packets sent by the CONTROL
agent to agent x. Depending on the detail, an initial line of this type
may be followed by the parameters in the request.
#x H:MM:SS.MMM R NDMP_... REPLY-SEQ (SEQ Recv
These lines indicate a reply
(R for return, a snoop convention) to the CONTROL agent from agent x.
The type of reply is shown by NDMP_... The REPLY-SEQ is the packet number
of the original request (CONTROL to x). The SEQ number is a count of
packets sent by agent x to the CONTROL agent. Depending on the detail,
an initial line of this type may be followed by the parameters in the
reply.
#x H:MM:SS.MMM C NDMP_... SEQ Recv
These lines indicate a request
(C for call, a snoop convention) from agent x to the CONTROL agent.
The type of request is shown by NDMP_... The SEQ number is a count of
packets sent by agent x to the CONTROL agent. Depending on the detail,
an initial line of this type may be followed by the parameters in the
request. These requests never have replies, and contain NDMP_LOG_...,
NDMP_FH_..., or NDMP_NOTIFY_... requests.
LDl, LTl, LRl -- NDMP_LOG_LOG
lines
Lxl H:MM:SS.MMM ...
These lines indicate an ordinary log message from agent x.
LD__, LT__, LR__ -- NDMP_LOG_DEBUG
lines
Lx__ H:MM:SS.MMM ...
| __ |
level |
|
| ui |
NDMP_DBG_USER_INFO |
| us |
NDMP_DBG_USER_SUMMARY |
| ud |
NDMP_DBG_USER_DETAIL |
| di |
NDMP_DBG_DIAG_INFO |
| ds |
NDMP_DBG_DIAG_SUMMARY |
| dd |
NDMP_DBG_DIAG_DETAIL |
| pi |
NDMP_DBG_PROG_INFO |
| ps |
NDMP_DBG_PROG_SUMMARY |
| pd |
NDMP_DBG_PROG_DETAIL |
LDf, LTf, LRf -- NDMP_LOG_FILE
lines (file recovered)
LDf H:MM:SS.MMM ...
These lines indicate a file
recovered log message from the DATA agent. Such messages from the TAPE
or ROBOT agent, while recognized and recorded by ndmjob, would be meaningless.
These messages are sent during recovery operations (NDMP_DATA_START_RECOVER)
to indicate the disposition of each item (nlist) of the requested recovery.
CL -- Control (ndmjob) log
lines
LDp, LDn, LDd -- NDMP_FH_ADD_UNIX_xxx
index lines
These line types contain file
index information sent to the CONTROL agent from the DATA agent. Such
messages from the TAPE or ROBOT agent, while recognized and recorded by
ndmjob, would be meaningless. These messages are sent during backup operations
(NDMP_DATA_START_RECOVER) and during recover file history operations (NDMP_DATA_START_RECOVER_FILEHIST).
Be sure to see the notice
about Non-standard under the discussion of the -t
option.
IDp Tmode UID GID SIZE MTIME ATIME CTIME @FHINFO PATH
(NDMP_FH_ADD_UNIX_PATH)
IDd NODE PARENT NAME
(NDMP_FH_ADD_UNIX_DIR)
IDn Tmode UID GID SIZE MTIME ATIME CTIME @FHINFO NODE
(NDMP_FH_ADD_UNIX_NODE)
The common fields are:
| Field |
Description |
| Tmode |
T is the type (- d c b p l s). mode is low 12-bits of mode in
octal (eg 0666) |
| UID/GID |
decimal ID number |
| SIZE |
decimal size in bytes |
| MTIME/ATIME/CTIME |
hex 32-bit UNIX time (GMT) |
| @FHINFO |
decimal offset in backup byte-stream |
| PATH |
A path name, spaces are possible |
| NAME |
A name, spaces are possible |
| NODE/PARENT |
decimal 32-bit node number |
CM -- Control (ndmjob) media line
CM TAPENO MEDIA
These lines are constructed
by the CONTROL (ndmjob) agent during a tape change and when the backup
operation is complete. They are only constructed when creating a backup.
TAPENO is the tape number
in the sequence begining with 1 (one). The MEDIA is a media argument
compatible with -m, and contains the tape label (if any), the filemark
offset (if any), and the window size (always). It never contains the
robot slot address.
The window size is determined
by the data_written field in the NDMP_MOVER_GET_STATE reply.
DEc, DEi, DEa -- Post-backup
data environment
DE NAME=VAL
|
