2.3.1. vacumm.misc.misc– Very generic tools

Functions:

Cfg2Att() MV2_axisConcatenate() MV2_concatenate() N_choose() auto_scale() basic_auto_scale() bound_ops() broadcast() check_case() check_def_atts() checkdir() closeto() cp_atts() cp_props() create_selector() deg2str() deg_from_dec() deplab() dict_aliases() dict_check_defaults() dict_copy_items() dict_filter() dict_filter_out() dict_merge()

dicttree_get() dicttree_set() dirsize() filter_level_selector() filter_selector() geo_scale() geodir() get_atts() get_svn_revision() grow_depth() grow_lat() grow_variables() history() indices2slices() intersect() is_iterable() isempty() ismasked() isnumber() iterable() kwfilter() kwfilterout() latlab() lonlab()

lunique() main_geodir() makeiter() mask_nan() match_atts() match_string() minbox() numod() phaselab() rm_html_tags() scalebox() selector2str() set_atts() set_lang() set_lang_fr() split_selector() splitidx() squarebox() squeeze_variable() tunique() write_ascii_time1d() xls_style() zoombox()

Classes:

ArgList get() put() ArgTuple get() put() Att CaseChecker check() error() isvalid() FileTree default_patterns file_list() scan() set_exclude() set_include() set_patterns()

Misc tools

Note

You can import it directly for example like this:

>>> from vacumm.misc import auto_scale

class ArgList(argsi)

Bases: object

Utility to always manage arguments as list and return results as input

Examples:

>>> a = 'a'
>>> al = ArgList(a)
>>> al.get() # input for function as list
['a']
>>> al.put(['aa']) # output as input
'aa'

>>> a = ['a','b']
>>> al = ArgList(a)
>>> al.get()
['a', 'b']
>>> al.put(['aa'])
['aa']

get()

put(argso)

class ArgTuple(argsi)

Bases: object

Utility to always manage arguments as tuple and return results as input

Examples:

>>> a = 'a'
>>> al = ArgTuple(a)
>>> al.get() # input for function as tuple
['a']
>>> al.put(['aa']) # output as input
'aa'

>>> a = ['a','b']
>>> al = ArgTuple(a)
>>> al.get()
['a', 'b']
>>> al.put(['aa'])
['aa']

get()

put(argso)

class Att

Bases: dict

Class to create a dictionnary and access and set keys as attributes.

You initialize and manage it as using dict.

Example:

>>> dd = Att(toto=3)
>>> print dd['toto']
3
>>> print dd.toto
3
>>> dd.toto = 5
>>> print dd['toto']
5

class CaseChecker(valid_cases, errmode='raise', ic=True, casename='parameter')

Bases: object

Check that a string that represents a “case” is valid

check(case)

Check that case is a valid string or None and make an error if not

error(message)

Exit with an error

isvalid(case)

Check that case is a valid string or None

Cfg2Att(cfg)

Convert a ConfigObj object to an arborescence of Att objects

class FileTree(input_dir, relative=False, scan=True, **kwargs)

Bases: object

Build a file tree

Params:

• input_dir: Input directory.
• patterns, optional: A string (or a list of strings) indicating which REGULAR EXPRESSION patterns files must match (using glob.glob) [default: ‘.*’]
• exclude, optional:: A string (or a list of strings) indicating which REGULAR EXPRESSION patterns files must not match (using re.search) [default: [‘CVS/’,’.svn/’]]
• include, optional:: A string (or a list of strings) indicating which REGULAR EXPRESSION patterns files must match if they are excluded by ‘exclude’ patterns (using re.search) [default: None]
• relative, optional:: Return file names relative to input_dir [default: False]

default_patterns = {'exclude': ['~$', '^CVS$/', '^\\.svn$/', '^\\.DS_Store$', '\\.db$', '\\.ini$'], 'include': [], 'patterns': ['.*']}

file_list(**kwargs)

scan(**kwargs)

Recursive scan to list files

set_exclude(value, **kwargs)

set_include(value, **kwargs)

set_patterns(value, **kwargs)

MV2_axisConcatenate(axes, id=None, attributes=None, copy=True)

Concatenate the axes, return a transient axis.

MV2_concatenate(arrays, axis=0, axisid=None, axisattributes=None, copy=True)

Concatenate the arrays along the given axis. Give the extended axis the id and attributes provided - by default, those of the first array.

N_choose(a, choices, out=None, mode='raise')

Robust version to numpy.choose() valid also for masked arrays.

The func:numpy.choose function fails when length of choices is greater than 32! You can verify it with:

>>> numpy.choose([0], numpy.zeros((50, 1))) ! FAIL
>>> N_choose([0], numpy.zeros((50, 1))) ! SUCCESS

Note

Performances of this function can be improved, even if it should not take alot of time to run in the vast majority of cases.

auto_scale(data=None, nmax=None, vmin=None, vmax=None, separators=None, fractions=False, symetric=False, keepminmax=False, **kwargs)

Computes levels according to a dataset and its range of values. Locators are on a 10-base. Different scaling can be used with this version.

Params:

• data: The dataset
• vmax, optional: Replaces max(data)
• vmin, optional: // min(data)
• symetric, optional: min_value and max_value are made symetric with respect to zero [default: False]
• nmax, optional: Maximal number of levels
• separators, optional: Subdivision for irregular levels
• fractions, optional: Consider separators as fractions (]0.,1.[)
• steps, optional: Base 10 steps for finding levels [default: [1,2,2.5,5,10]]
• geo, optional: Treat levels as geographical degrees [default: False]
• Other parameters are given to MaxNLocator

Return:Array of levels See also:basic_auto_scale() geo_scale()

basic_auto_scale(vmin, vmax, nmax=7, steps=[1, 2, 2.5, 5, 10], geo=False, minutes=False, **kwargs)

Computes levels according to a dataset and its range of values. Locators are on a 10-base.

Params:

• vmin/vmax: Find levels around this range.
• nmax, optional: Maximal number of steps.
• steps, optional: Base 10 steps for finding levels [default: [1,2,2.5,5,10]]
• geo, optional: Assume longitude or latitude degrees [default: False].
• minutes, optional: If geo, find suitable levels to match nice minutes locations when locations have floating values (like 1.2) [default: False].

See also:

auto_scale() geo_scale()

bound_ops(bounds)

Get operators that must be used for checking inclusion within boundaries. Returned operators (ops) must be used in the following way to return True if value is inside bounds:

Params:

• bounds: Boundary closing indicator (like ‘co’)

Example:

>>> ops[0](value,lower_bound)
>>> ops[1](value,upper_bound)

Return:

A 2-element tuple of operators taken within (operator.ge,operator.gt,operator.le,operator.lt).

broadcast(set, n, mode='last', **kwargs)

Broadcast set to the specified length n

Params:

• set: A single element or a sequence.

• n: Final requested length.

• mode, optional: Filling mode.

  • "last": Use the last element.
  • "first": Use the first element.
  • "cycle": Cycle through set.
  • if fillvalue is passed as a keyword, it is used to fill.

Example:

>>> broadcast([2,4], 5)
>>> broadcast(5, 4)
>>> broadcast((2,3), 3)
>>> broadcast((2,3), 3, mode='first')
>>> broadcast([2,3], 3, mode='value', fillvalue=999)

check_case(cases, case, mode='isvalid', **kwargs)

Check that case is valid using :class:``CaseChecker` and allowed cases

check_def_atts(obj, **defaults)

Check defaults attributes and set them if empty

See also:get_atts() cp_atts() set_atts()

checkdir(path, asfile=None, chmod=None)

Make sure that a directory exists and has right access

Params:

• path: Directory of file path.
• asfile, optional: Interpret path as file name instead of a directory name. When None, it is set to True if it has a dot (".") inside.

Return:The directory name.

closeto(a, b, rtol=1e-05, atol=1e-08)

Check which values of a numeric array are close to those of another array

Params:

• a: A numpy.ndarray variable.
• b: Another array.

See also:

allclose()

cp_atts(var1, var2, overwrite=True, select=None, exclude=None, extra=None, **kwargs)

Copy all atributes from one variable to another

Params:

• var1/var2: Copy var1 attributes to var2
• id, optional: Also copy id [default: False]
• overwrite, optional: Overwrite attributes of var2?
• select, optional: Copy only these attributes.
• exclude, optional: Attributes that must be excuded from copy.
• Other keywords are set as attributes

See also:

get_atts() set_atts() check_def_atts()

cp_props(var1, var2, axes=None, grid=True, atts=None, exaxes=None, exatts=None, owatts=True)

Copy properties of a variable to another variabes

Proporties are attributes, axes and grid.

Params:

• var1/var2: cdms variables.
• axes, optional: Position of axes to copy. Positions are converted to relative position to that last dimension to handle the case of variables that don’t have the same number of dimensions. Set it to False to not copy axes.
• exaxes, optional: Position of axes to not copy.
• grid, optional: Also copy the grid?
• atts, optional: Attributes to copy. Set it to False to not copy attributes.
• exatts, optional: Attributes to exclude from copy.
• owatts, optional: Overwrite attributes?

Examples:

>>> cp_props(var1, var2, atts=False)
>>> cp_props(var1, var2, axes=[1,-1], atts=['units','long_name'])
>>> cp_props(var1, var2, exaxes=0, grid=False)
>>> cp_props(var1, var2, axes=False, exatts='units', owatts=False)

create_selector(*args, **kwargs)

Create a cdms2.selectors.Selector

Params:

• args: Each item of args is treated in a special way:

  • A dictionary is treated using: selector.refine(**args[i])
  • A list is treated using: selector.refine(*args[i])
  • Else (single slice, tuple, etc): selector.refine(args[i])

• kwargs: Items are integrated using: selector.refine(**kwargs)

Examples:

>>> create_selector((time1,time2),slice(0,1),lat=(lat1,lat2))
>>> create_selector([(time1,time2),(lat1,lat2)],(lon1,lon2))
>>> selector2 = create_selector(selector1, lon=(lon1,lon2))

deg2str(*args, **kwargs)

deg_from_dec(dec)

Convert from decimal degrees to degrees,minutes,seconds

deplab(depths, fmt='%gm', auto=False, nosign=False)

Return well formatted depth labels

Params:

• depths: Numerical depths
• fmt, optional: Numeric format of the string (including units) [default: ‘%gm’]
• auto, optional: If True, find the ticks according to the range of depths [default: False]
• nosign, optional: Absolute values are used.

See also:

lonlab() latlab()

dict_aliases(kwargs, aliases)

Remove duplicate entries in a dictionnary according to a list of aliases. The first alias has priority over the following.

Example:

>>> kwargs = dict(min=4, title='Title', label='Label')
>>> dict_aliases(kwargs, ['label', 'title'])
{'min': 4, 'label': 'Label'}

dict_check_defaults(kwargs, **defs)

Check that a dictionary has some default values

dict_copy_items(ddi, ddo, keys)

Copy existing items of an array to another one

Params:

• ddi: Dictionary from which items are taken.
• ddo: Dictionary in which items are put.
• keys: single or list of keys.

dict_filter(kwargs, filters, defaults=None, copy=False, short=False, keep=False, **kwadd)

Filter out kwargs (typically extra calling keywords)

Params:

• kwargs: Dictionnary to filter.
• filters: Single or list of prefixes.
• defaults: dictionnary of default values for output fictionnary.
• copy: Simply copy items, do not remove them from kwargs.
• short: Allow prefixes to not end with "_".
• keep: Keep prefix filter in output keys.

Example:

>>> kwargs = {'basemap':'f', 'basemap_fillcontinents':True, 'quiet':False,'basemap_plot':False}
>>> print kwfilter(kwargs,'basemap', defaults=dict(drawcoastlines=True,plot=True),good=True)
{'plot': False, 'fillcontinents': True, 'good': True, 'basemap': 'f', 'drawcoastlines': True}
>>> print kwargs
{'quiet': False}

dict_filter_out(kwargs, filters, copy=False, mode='start')

Remove entries from a dictionary

Params:

• kwargs: Valid dictionary.

• filters: Single or list of prefixes. Entries with a key starting with one of these filters are removed.

• copy, optional: Create a copy or work on original dict?

• mode, optional: Matching mode.

  • "start": remove if key starts with filter.
  • "end": remove if key ends with filter.
  • "regexp": remove if key match regexp match filter.
  • else remove if key is exactly filter.

dict_merge(*dd, **kwargs)

Merge dictionaries

First dictionaries have priority over next

Params:

• dd: Argument are interpreted as dictionary to merge. Those who are not dictionaries are skipped.
• mergesubdicts, optional: Also merge dictionary items (like in a tree) [default: True].
• mergetuples, optional: Also merge tuple items [default: False].
• mergelists, optional: Also merge list items [default: False].
• unique, optional: Uniquify lists and tuples [default: True].
• skipnones, optional: Skip Nones [default: True].
• skipempty, optional: Skip everything that is not converted to False using bool [default: False].
• cls, optional: Class to use. Default to the first class found in arguments that is not a dict, else defaults to dict.

Example:

>>> d1 = dict(a=3, b=5)
>>> d2 = dict(a=5, c=7)
>>> print dict_merge(d1,d2)
{'a': 3, 'c': 7, 'b': 5}

dicttree_get(dd, *keys, **kwargs)

Get value of tree of dicts

dicttree_set(dd, *keys, **items)

Set values in a tree of dicts

dd: dict

Dict of dicts to fill

keys: strings items: dict

To set

>>> dd = {}
>>> dicttree_set(dd, 'sec', 'subsec', 'subsubsec', option=15)
>>> dicttree_set(dd, value1=30, value2=20)

dirsize(folder, units='b')

Get the size of a directory

Params:

• folder: path of the directory

• units, optional:

  • "b": bytes
  • "k": Kb
  • "m": Mb
  • "g": Gb
  • "t": Tb

filter_level_selector(selector, ids=None, **kwargs)

Filter a cdms2.selectors.Selector instance to keep or remove z dimension

filter_selector(selector, ids=None, copy=True, out=False, keeppos=False, noslice=False)

Filter a cdms2.selectors.Selector instance to keep or remove only a list of ids

Params:

• selector: A cdms2.selectors.Selector instance.
• ids, optional: Allowed ids.
• copy, optional: Filter original or copy?
• out, optional: Filter in (keep target selections) or out(remove target selections)?
• keeppos, optional: Keep positional components?
• noslice, optional: Remove slices?

geo_scale(*args, **kwargs)

auto_scale() with geo=True

See also:basic_auto_scale() auto_scale()

geodir(direction, from_north=True, inverse=False)

Return a direction in degrees in the form ‘WNW’

get_atts(var, id=True, extra=None, **kwargs)

Get all attributes from a variable

Params:

• var: Target variable
• id, optional: Get also id [default: False]
• Other keywords are set as attributes

See also:

cp_atts() set_atts() check_def_atts()

get_svn_revision(path, max=False)

Get the revision number of a path

Adapted from numpy.disutils.misc_util.Configuration

grow_depth(var, depth=None, mode=None, default=0.0, getvar=True)

Make depth and variable have the same shape

Params:

• var: A MV2 array with a depth dimension.

• depth, optional: Depth axis or array. If None, it is guessed from var. If not found, it defaults to default.

• default, optional: Default value for depth if not found.

• mode, optional: How to handle case where shape of var is changed:

  • None: Nothing happens
  • "warn": A warning with warn().
  • "raise": Raise a VACUMMError exception.

Return:

var, depth where depth may be a scalar equal to default

grow_lat(var, lat=None, mode=None, default=None, getvar=True)

Make latitude and variable have the same shape

Params:

• var: A MV2 array with a lat dimension.

• depth, optional: Latitude axis or array. If None, it is guessed from var.

• default, optional: Default value for lat if not found.

• mode, optional: How to handle case where shape of var is changed:

  • None: Nothing happens
  • "warn": A warning with warn().
  • "raise": Raise a VACUMMError exception.

Return:

var, lat where lat may be ``None

grow_variables(var1, var2)

Grow dimensions of var1 and var2 until their match

It is an improved version genutil.grower.grower().

history(nbcommand=None)

Display the command history for the interactive python.

indices2slices(indices)

Convert a list of indices to a list of slices

intersect(seg1, seg2, length=False)

Intersection of two segments

Example:

>>> intersect([1,10],(5,12))
[5, 10]

is_iterable(obj, nostr=True, nogen=True)

Check if an object is iterable or not.

Params:

• obj: Object to check

Return:

True/False

isempty(x)

Check if empty

ismasked(arr)

isnumber(var)

iterable(obj, nostr=True, nogen=True)

Check if an object is iterable or not.

Params:

• obj: Object to check

Return:

True/False

kwfilter(*args, **kwargs)

Alias for dict_filter()

kwfilterout(kwargs, filters, copy=False, mode='start')

Remove entries from a dictionary

Params:

• kwargs: Valid dictionary.

• filters: Single or list of prefixes. Entries with a key starting with one of these filters are removed.

• copy, optional: Create a copy or work on original dict?

• mode, optional: Matching mode.

  • "start": remove if key starts with filter.
  • "end": remove if key ends with filter.
  • "regexp": remove if key match regexp match filter.
  • else remove if key is exactly filter.

latlab(latitudes, **kwargs)

Return nice latitude labels

Params:

• latitudes: Value of latitudes
• decimal, optional: Use decimal representation
• fmt, optional: Format for decimal representation
• no_seconds, optional: Do not add seconds to degrees representation
• tex, optional: Use Tex degree symbol (depends on rcParams['text.usetex'] by default)
• no_symbol, optional: Don’t use degree symbol
• no_zeros, optional: Don’t insert zeros when integer < 10
• auto, optional: If True, find the ticks according to the range of values
• auto_minutes, optional: Automatically suppress degrees if value is not exactly a degree (just display minutes and seconds) else display degree
• bfdeg, optional: Use bold face for degrees when alone if auto_minutes and rcParams['text.usetex'] are True.

See also:

phaselab() lonlab() deplab()

lonlab(longitudes, **kwargs)

Return nice longitude labels

Params:

• longitudes: Value of longitudes
• decimal, optional: Use decimal representation
• fmt, optional: Format for decimal representation
• no_seconds, optional: Do not add seconds to degrees representation
• tex, optional: Use Tex degree symbol (depends on rcParams['text.usetex'] by default)
• no_symbol, optional: Don’t use degree symbol
• no_zeros, optional: Don’t insert zeros when integer < 10
• auto, optional: If True, find the ticks according to the range of values
• auto_minutes, optional: Automatically suppress degrees if value is not exactly a degree (just display minutes and seconds) else display degree
• bfdeg, optional: Use bold face for degrees when alone if auto_minutes and rcParams['text.usetex'] are True.

See also:

phaselab() latlab() deplab()

lunique(mylist)

Uniquify a list to a new list

main_geodir(directions, amp=None, num=False, res=22.5, getamp=False, **kwargs)

Return the dominant direction from a set of directions in degrees

makeiter(var)

Make var iterable as a list if not ietrable or a string

mask_nan(input)

Mask NaN from a variable

Params:

• input: Array (numpy, numpy.ma or MV2)

Note

If input is pure numpy, it is converted to numpy.ma

Example:

>>> var = N.array([1,N.nan])
>>> print mask_nan(var)
[1.0 --]

match_atts(obj, checks, id=True, ignorecase=True, transform=None)

Check that at least one of the attributes of an object matches check list

Params:

• obj: An object
• checks: A dictionary of (attributes name, checklist)

match_string(ss, checks, ignorecase=True, transform=None)

Check that a string verify a check list that consists of a list of either strings or callables

minbox(box, dxmin=None, dymin=None)

Ensure that a box as minimal extents

numod(*vv)

Get the needed numeric module to be able to handle all specified variables

Params:Scalars or numeric arrays. Return:numpy, numpy.ma or MV2.

phaselab(vals, fmt='%.5g', label=None, decimal=True, tex=None, auto_minutes=False, no_seconds=False, no_symbol=False, no_zeros=False, auto=False, bfdeg=False, **kwargs)

Return a nice label for degrees

Inspired from Basemap toolkit of Matplotlib

rm_html_tags(str)

Remove html tags from a string

Params:

• str: A string

Return:

Cleaned string

Example:

>>> rm_html_tags('<title>My title</title>')
My title

scalebox(box, factor, square=False, xmargin=None, ymargin=None)

Alter box limits with a zoom factor

Params:

• box: A list of [xmin,ymin,xmax,ymax].
• factor: Zoom factor where 1 means no change, and a factor < 1 means a smaller box.

Example:

>>> scalebox([0,0,1,2], 1.1)
[-0.55, -1.10, 1.55, 2.55]

selector2str(selector)

Convert a selector to a generic string where <obj …> labels are removed

Example:

>>> sel = cdms2.selectors.Selector(slice(2,5),time=(4,6))
>>> selector2str(sel)
'((slice(2, 5, None)),("time", (4, 6)))'

set_atts(var, atts=None, overwrite=True, **kwargs)

Set attributes

Params:

• var: Change attributes of var
• atts, optional: A dictionary of attributes
• overwrite, optional: Overwrite attributes of var2?
• Other keywords are set as attributes

See also:

cp_atts() get_atts() check_def_atts()

set_lang(default='en_US.UTF-8', num=None)

Set the default and numeric languages

The numeric language defaults to the default language.

set_lang_fr(ennum=True)

Set lang to french, except the numeric lang which is set to english by default

split_selector(selector)

Split a selector into (positionalComponents, axisComponents).

Params:

• selector: cdms2.selectors.Selector

Return:

a tuple of: - the list of positionalComponents - the OrderedDict of axisComponents

Exemple:

>>> selector = cdms2.selectors.Selector(slice(2,5),time=(4,6))
>>> split_selector(selector)
((slice(2, 5, None),), {'time': (4, 6)})

splitidx(arr, crit)

Return a list of split indices for a 1d array according to a criteria

arr: 1D array crit: int, list of int, int array, float

If a positive int, it refers to the number of almost equal intervals. If a negative int, it refers to the max number of item per interval. If a list or array of int, it is directly intepreted as split indices, with some checks. If a float, the input array is monotically increasing and the intervals are at least of size crit.

idx: list of ints

Indices that are compatible ith numpy.array_split().

squarebox(box, scale=1)

Get an approximate [xmin, ymin, xmax, ymax] that is square in meters

squeeze_variable(var, spec=True, asmv=False)

Remove singleton axes from a MV2 variable

Params:

• var: MV2 array.

• spec, optional: Squeeze specification.

  • True or None: Simply squeeze out all singletons.
  • False: Does nothing (return the same variable).
  • A string containing geo letters like ‘xyzt’ to remove axes according to their type.

• asmv, optional: If True and input is a MV2 array, always return a MV2 array, even if all result is a scalar value or masked.

Examples:

>>> squeeze_variable(var)
>>> squeeze_variable(var, spec='tx')

tunique(mytuple)

Uniquify a tuple to a new tuple

write_ascii_time1d(var, file, fmt='%g')

Write an ascii file in the following format: YYY/MM/DD HH:MN:SS DATA where DATA is one column.

Params:

• var: a cdms variable WITH A TIME AXIS
• file: output file name
• fmt, optional: format of DATA [default: ‘%g’]

xls_style(style=None, b=None, i=None, u=None, c=None, o=None, bd=None, fmt=None, va=None, ha=None, f=None, s=None, n=None, copy=True, **kwargs)

Excel style sheet for pyExcelerator cell objects

Params:

• style, optional: Style object to update.
• scopy, optional: Make a copy of style if passed.
• b, optional: Bold [True/False].
• i, optional: italic[True/False].
• u, optional: Underline [True/False].
• c, optional: Colour index [from 0 to 82!, with 0=black, 1=white, 2=red, 3=green, 4=blue, 5=yellow, 6=magenta, 7=cyan].
• o, optional: Outline [True/False].
• bd, optional: Borders [True/int/False/dict(top=int…)].
• fmt, optional: Format [‘general’,‘0.0’,…].
• va, optional: Vertical alignment [‘top’,’bottom’,’center’,’justified’].
• ha, optional: Horizontal alignment [‘left’,’right’,’center’,’justified’].

Example:

>>> style_num = xls_style(fmt='0.00')
>>> style_num_left = xls_style(style_num, left=2, copy=True)
>>> style_full = xls_style(c=2, b=True, va='center', ha='justified')

Tutorial:

Les fichiers Excel

zoombox(box, factor, square=False, xmargin=None, ymargin=None)

Alias for scalebox() with 1/factor as zoom factor.