FSOBJIMPORT

Section: FSCLI (1)
Updated: December 2018


NAME

fsobjimport – import object storage media and ingest content into the Tertiary Manager system

SYNOPSIS

fsobjimport [-d directory] [-D delimiter] [-f manifest-file] [-g group] -m mediaid [-M mode] [-p prefix] [-t files|list|media] [-T time] [-u user] [-U] [-W]
fsobjimport -C [-m mediaid]
fsobjimport -R -m mediaid
fsobjimport -s [-F type] [-l] [-m mediaid]
fsobjimport -S -i requestid|-m mediaid
fsobjimport -h

DESCRIPTION

fsobjimport(1)is used to import object storage media and to ingest content into the Tertiary Manager system. It also provides administrative operations for scanning and listing the contents of object storage media and for reporting the status of import requests.

The media must be known to the Tertiary Manager system before the import can be requested and must be in the protect state in order for it to be used for import.

For file imports, the contents of the files found on the media will be copied into a destination directory. If the file already exists in the destination directory, a numeric suffix will be appended to the name of the copied file, increasing from zero, until a file with a unique name can be created. Otherwise, option -W can be specified to request that the file be overwritten if it already exists. Once the import has completed, no references to the media are retained and the media can be removed from the Tertiary Manager system unless needed for other purposes.

For media imports, media is retained in the Tertiary Manager system, and the metadata required to map the contents of an object storage bucket into a StorNext file system will be created for each file found on the media. The created metadata allows the data to be retrieved through the Tertiary Manager system. Once the import is complete, the media state must be changed from the protect state to allow access to the files.

Specifying the list import type will generate a list of the files on an object storage medium. The file list will be created in JSON format. The contents of the file can be modified and then used for file or media import.

fsobjimport(1) will log file error information for the current operation to /usr/adic/TSM/logs/reports/import.mediaid. The status information for an import operation can be displayed by specifying the -s option of the fsobjimport(1) command.

If an import request terminates before successful completion, it can be resumed by specifying the -R option with the fsobjimport(1) command. In this case, the -m option must also be specified to identify the failed import request. The import request will be resumed from the failure point.

If an import request has been terminated and is no longer needed, the -U option can be used to undo the import. If neither an undo nor resume of a terminated import is required, the checkpoint metadata for this media should be cleared with the -C option of the fsobjimport(1) command. The -m option must also be specified to identify the import request to clear.

An import request can be tuned by modifying system parameters which define the number of threads used on the request. System parameters IMP_META_NUM_THREADS and IMP_DBASE_NUM_THREADS can be modified to optimize the media import processing for your system. IMP_META_NUM_THREADS defines the number of threads used to create the file metadata and IMP_DBASE_NUM_THREADS the number of threads used to create the database metadata. Parameter IMP_COPY_NUM_THREADS defines the number of threads used to copy data on file imports and IMP_REMOVE_NUM_THREADS defines the number of threads used to remove the imported files or metadata for an undo request. By default, five threads are used for each thread type. For further descriptions of each system parameter, see the fs_sysparm.README file.

OPTIONS

-C
Clear the import request information for the specified medium or for all terminated import requests. After using this option, the import request can no longer be resumed.
-d directory
Ingest data into the destination directory directory. The destination directory must be associated with a relation point when ingesting media. The destination directory may be associated with a relation point if ingesting files. Note that if file ingest is chosen and the destination directory is not under a relation point, the files will be ingested but will not be managed by the Tertiary Manager system.
-D delimiter
Specifies the character representing a directory delimiter in the object storage names.
-f manifest-file
If specified with -t list, output a list of the files on an object storage medium to this file. The file list will be output in JSON format and will include file attributes. If specified with -t media or files, ingest the files listed in file manifest-file. The file attribute values can be modified from their defaults before using the file list for ingest.
-F type
Sets the output format to the specified type. Valid values are TEXT (default), XML, or JSON.
TEXT is the “legacy” textual format.
XML (Extensible Markup Language) is a set of rules for encoding documents in machine-readable form. XML’s design goals emphasize simplicity, generality, and usability over the Internet. It is a textual data format. Although the design of XML focuses on documents, it is widely used for the representation of arbitrary data structures, for example in web services. You may validate the XML output using the XSD (see fsxsd(1) for details). See http://en.wikipedia.org/wiki/XML for more information.
JSON (JavaScript Object Notation) is a lightweight text-based open standard designed for human-readable data interchange. It is derived from the JavaScript programming language for representing simple data structures and associative arrays, called objects. Despite its relationship to JavaScript, it is language-independent, with parsers available for virtually every programming language. The JSON format is often used for serializing and transmitting structured data over a network connection. It is primarily used to transmit data between a server and web application, serving as an alternative to XML. See http://json.org for more information.
-g group
Set ingested files to group group where group is the group name or numeric group ID. Defaults to 0.
-h
Print a usage message and exit.
-i
Specify the request ID of the import request to suspend. The fsobjimport(1) command, specified with options -s and -l, can be used to display the request IDs of pending imports.
-l
Long status report. For the specified import request, this option provides all of the import option values, the state of the import, import identifiers, and the progress of the import.
-m mediaid
Operate on the medium mediaid. The medium must be known to the Tertiary Manager system.
-M mode
Set the permission bits for ingested files to mode. If this option is not specified, for file imports, the mode will default to 0440, and for media imports, the mode will be inherited from the destination directory, with the execute bits set to zero.
-p prefix
Import only those files beginning with the specified prefix.
-R
Restart the media or file import processing from the failure point using the checkpoint metadata. The request is identified by the media ID specified with the -m option. Restarts of failed list requests are not supported.
-s
Display the status of the import request identified by the media ID specified with the -m option, or, in the case where the -m option is omitted, display all pending or failed import requests.
-S
Suspend an import request. Either the -m option or the -i option must be used to identify the import request to suspend. A suspended import can be restarted by specifying the -R option with the media ID. If an import is no longer needed, the -U option can be specified to undo the partially completed import. If neither an import resume nor undo is needed, the checkpoint metadata should be cleared from the Storage Manager by specifying the -C option with the media ID of the suspended request.
-t type
Identify the import request type. Supported types are:

files
Copy contents of medium to destination directory
list
Generate a list of files residing on medium
media
Create metadata for each file residing on medium, which will allow file content to be retrieved
If the type is files or media, then the -d option must also be specified. If the type is list, an output file must be specified with option -f.
-T time
Import only those files created after the specified time. Times are expressed in local time and must be specified in format YYYY-MM-DD HH:MM:SS.
-u user
Set ingested files to owner user where user is the user name or numeric user ID. Defaults to 0.
-U
Undo a media or file import request. An undo of a media import will delete the metadata and database information associated with the files found on the imported media. An undo of a file import will remove all files copied into the destination directory during the import. The options specified with an undo must match those used on the import to return to the pre-import state. If this is an undo of a terminated import, the checkpoint metadata will be used to undo the request.
-W
On file imports, overwrite a file if it already exists.

EXIT VALUES

Exit codes for the fsobjimport(1) command are:

EEXIST
Import file or manifest file already exists.
EFAULT
Bad address specified in file system request.
EINTR
The import request was interrupted or suspended.
EINVAL
Invalid command arguments, manifest file, or object storage data.
EIO
An I/O error occurred while transferring object storage data or when accessing the manifest file or database.
ENAMETOOLONG
Object ID or path length exceeds the system maximum.
ENODIR
Destination path is not a directory.
ENOENT
The specified media has not been configured, a path to the media has not been configured, the specified import request is not pending, or the destination directory does not exist.
ENOMEM
Insufficient memory to process command.
ENOSPC
No space on file system.
ENOSYS
Object storage import not supported by provider.
ENOTCONN
Cannot connect to object storage system or database.
EPERM
The media is write-protected, an import request is already active for the media, or you do not have permission to access the object storage system or to access a file.

The fsobjimport(1) command may also fail and return errno for the errors documented in fopen(3)

EXAMPLES

Example 1

Import files from medium AWS0000, copying the files to directory /stornext/snfs1/import1. Create the files with an owner ID of 101, a group ID of 120, and with file permissions of 0660. Pre-existing files in the destination directory which have the same name as an imported file will be overwritten:

fsobjimport -m AWS0000 -t files -d /stornext/snfs1/import1 -u 101 \
            -g 120 -M 0660 -W

Example 2

Undo the import request from Example 1, removing all imported files from directory /stornext/snfs1/import1:

fsobjimport -m AWS0000 -t files -d /stornext/snfs1/import1 -u 101 \
            -g 120 -M 0660 -W -U

Example 3

Import medium STS0001 into the Tertiary Manager system, creating the metadata which allows access to the file contents. Import only those files which were created after 7:30 P.M., August 10, 2017:

fsobjimport -m STS0001 -t media -d /stornext/snfs1/import2 \
            -T "2017-08-10 19:30:00"

Example 4

Generate a list of files found on medium AWS0000, which are prefixed with accounts, but do not ingest the contents. Replace the object storage delimiter ‘:’ with a forward slash (‘/’) in the destination file path:

fsobjimport -m AWS0000 -t list -f manifest-file -p accounts -D :

The file created with the list option can then be modified and used to create a manifest file that is used for ingesting content:

fsobjimport -m AWS0000 -t files -d /stornext/snfs1/import3 \
            -f manifest-file

Example 5

Check the status of all running fsobjimport(1) processes:

fsobjimport -s

Example 6

Suspend the import of medium AWS0000:

fsobjimport -m AWS0000 -S

Example 7

Restart a suspended import of medium AWS0000:

fsobjimport -m AWS0000 -R

Example 8

Undo a suspended import of medium AWS0000:

fsobjimport -m AWS0000 -U

FILES

/usr/adic/TSM/logs/reports/import.<mediaid>
objimport.json

SEE ALSO

fsobjcfg(1), fschmedstate(1) objimport.json(5)