2.3.10. vacumm.misc.remote – Remote files

Attributes:
Classes:
digraph inheritance1bdd584a37 { rankdir=TB; ratio=compress; size="6.0, 8.0"; "vacumm.misc.remote.InputWorkFiles" [URL="#vacumm.misc.remote.InputWorkFiles",color=orange,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.6,shape=ellipse,style=filled,target="_top",tooltip="A class to deal with input remote files"]; "vacumm.misc.remote.WorkFile" -> "vacumm.misc.remote.InputWorkFiles" [arrowsize=0.5,penwidth=1,style="setlinewidth(0.5)"]; "vacumm.misc.remote.OutputWorkFile" [URL="#vacumm.misc.remote.OutputWorkFile",color=orange,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.6,shape=ellipse,style=filled,target="_top",tooltip="A class to deal with an output remote file"]; "vacumm.misc.remote.WorkFile" -> "vacumm.misc.remote.OutputWorkFile" [arrowsize=0.5,penwidth=1,style="setlinewidth(0.5)"]; "vacumm.misc.remote.SSHBank" [URL="#vacumm.misc.remote.SSHBank",color=orange,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.6,shape=ellipse,style=filled,target="_top",tooltip="Interface to handle a bank of SSH/SFTP connections"]; "vacumm.misc.remote.WorkFile" [URL="#vacumm.misc.remote.WorkFile",color=orange,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.6,shape=ellipse,style=filled,target="_top",tooltip="Base class for :class:`InputWorkFiles` and class:`OutputWorkFile`"]; "vacumm.misc.remote.WorkFileException" [color=orange,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.6,shape=ellipse,style=filled]; }

Working with remote files

Warning

Yoo need paramiko to use this module.

class SSHBank[source]

Bases: object

Interface to handle a bank of SSH/SFTP connections

Example:
>>> sshbank = SSHBank()
>>> client = sshbank('camarpor')
>>> ssh = client('ssh')
>>> res = ssh.exec_command('ls')
>>> ftp = client('ftp')
>>> ftp.get('remote_file', 'local_file')
>>> ssh2 = sshbank.ssh('myhost:4022')
>>> ftp2 = sshbank.ftp('user@myhost')
ftp(host)[source]

Get the SFTP agent

ssh(host)[source]

Get the SSH agent

sshbank = <vacumm.misc.remote.SSHBank object>

Ready to use SSH bank

class WorkFile(logger, ssh, umask=0, dmode=509, fmode=436, raise_error=True)[source]

Bases: object

Base class for InputWorkFiles and class:OutputWorkFile

Params:
  • logger: A Logger (or subclass) instance
  • ssh: A ssh connexion (for instance created using :class`SSHBank` connected to an host).
  • umask, optional: Argument to os.umask()
  • dmode, optional: Directory unix mode (see os.chmod())
  • fmode, optional: File unix mode (see os.chmod())
debug(msg)[source]

Send a DEBUG message

error(msg, warn=None)[source]

Send an ERROR message and exit

classmethod expand_path(path, subst=None)[source]

Expand ‘~’, environment and other variables in a path

Params:
  • path: Path to be expanded
  • subst, optional: Dictionnary of variables that may be used for expansion
info(msg)[source]

Send an INFO message

classmethod parse_path(path, rootdir=None, subst=None, mode=None)[source]

Get the (inputdir, outputdir, path) from path (and rootdir)

Params:

  • path: Simple or complex path with remote and local part. The generic form is : "[<prefix>][(<remote_dir>><local_dir>)[<pattern>]" where <pattern> is global expression pattern (see glob.glob()). It can also take the form of a tuple. Remote paths follow specifications from RFC1738: http://tools.ietf.org/html/rfc1738.html.

    Warning

    Be careful when using <prefix> with a remote path.

  • rootdir, optional: Optional prefix to prepend to path.

  • subst, optional: Dictionary of variables used for string isubstitutions on path. All environnement variables (taken from os.environ) are also substituted.
Example:
>>> obj.parse_path('(sftp://user@host.fr:1022/my/path>/local/path)/to/data.nc', mode='get')
>>> obj.parse_path(('sftp://user@host.fr:1022/my/path','/local/path','/to/data.nc'), mode='get')
>>> obj.obj.parse_path('/local/path/data.nc')
>>> obj.obj.parse_path('/home10(/user1>/user2)/data.nc')
remote_exec(cmd)[source]

Execute a remote command and return result as list of lines

Example:
>>> files = workfile.remote_exec('ls')
warning(msg)[source]

Send a WARNING message

class InputWorkFiles(path, rootdir=None, logger=None, sshbank=None, transfer=False, subst=None, check=2, sort=None, filter=None, raise_error=True, **kwargs)[source]

Bases: vacumm.misc.remote.WorkFile

A class to deal with input remote files

Params:
  • path: path in the form "<prefix>(<remote_dir>><local_dir>)<pattern>". (see parse_path())
Params:
  • logger: a Logger (or subclass) instance
  • sshbank: a :class`SSHBank` instance
  • transfer: automatically start the transfer after initialization
Example: 
>>> wfile = InputWorkFiles('(caparmor-sftp:/home125>/home200/caparmor)toto*/data/file*.nc')
>>> wfile.get() # update
>>> wfile = InputWorkFiles('/home510/toto/toto.nc') # does nothing !
>>> wfile = InputWorkFiles(('/home15>/home12)toto/toto.nc') # local copy only
>>> print wfile.local_files()
>>> print wfile.remote_files()
get(check=2, ifile=None)[source]

Download remote files when needed

Params:

check: checks

  • 0: check nothing => force the transfer
  • 1: only check the existence of the local file
  • 2: check existence and compare local and remote dates
local_files(ifile=None, update=None)[source]

List of local (working) files

remote_files(ifile=None)[source]

List of remote files

class OutputWorkFile(path, rootdir='', logger=None, sshbank=None, subst=None, raise_error=True, **kwargs)[source]

Bases: vacumm.misc.remote.WorkFile

A class to deal with an output remote file

Params:
  • path: path in the form "<prefix>(<remote_dir>><local_dir>)<pattern>"
    (see parse_path())
Params:
  • logger: a Logger (or subclass) instance
  • sshbank: a :class`SSHBank` instance
Example: 
>>> wfile = OutputWorkFile('(sftp://caparmor-sftp/home125>/home200/caparmor)toto/data/file.png')
>>> wfile = OutputWorkFile('(/home200/caparmor>sftp://username@my.host.fr:1022/prefix)/toto/data/file.png')
>>> wfile = OutputWorkFile('/home510/toto/toto.png') # does nothing !
>>> wfile = OutputWorkFile('(/home15>/home12)toto/toto.png') # local copy only
>>> pylab.savefig(wfile.local_file)
>>> wfile.put() # send or copy
put(checkdir=True)[source]

Send the file