pylib.fun package

Module contents

exception pylib.fun.ArgTypeError

Bases: exceptions.Exception

exception pylib.fun.CsvSqlError

Bases: exceptions.Exception

exception pylib.fun.DaemonError

Bases: exceptions.Exception

pylib.fun.Inf = inf

Use this for testing against infinite.

exception pylib.fun.InitConfError

Bases: exceptions.Exception

class pylib.fun.Machine(hostname)

Bases: tuple

hostname

Alias for field number 0

class pylib.fun.Namespace(**kwds)

Bases: object

Generic container with dot-notation indexing

keys()
exception pylib.fun.NonZeroError

Bases: exceptions.Exception

class pylib.fun.Program(file, base, root, ext, dir, md5, args)

Bases: tuple

args

Alias for field number 6

base

Alias for field number 1

dir

Alias for field number 4

ext

Alias for field number 3

file

Alias for field number 0

md5

Alias for field number 5

root

Alias for field number 2

class pylib.fun.Summary(launcher, pid, stdout, stderr, status, env, command)

Bases: tuple

command

Alias for field number 6

env

Alias for field number 5

launcher

Alias for field number 0

pid

Alias for field number 1

status

Alias for field number 4

stderr

Alias for field number 3

stdout

Alias for field number 2

pylib.fun.WriteError(Exception)
pylib.fun.atomic_write(filename, content, overwrite=True, permissions=420)

Write a file atomically by writing the file content to a temporary location, then renaming the file

Parameters:
  • filename (str) – Filename
  • content (str) – File content
  • overwrite (bool) – Overwrite
  • permissions (octal) – Octal permissions
pylib.fun.cast(s)

Cast a basestring to a more appropriate type.

Example::
>>> from pylib.fun import cast
>>> type(cast("999"))
<type 'int'>
Parameters:s (basestring) – String
Returns:Casted string
Return type:int|float|str
pylib.fun.cleanup(src, dest=None)

Move or remove a file or directory. The source is removed if no destination is supplied, or if the move operation fails.

Parameters:
  • src (str) – Source file or directory name
  • dest (str) – Destination file or directory name
pylib.fun.crc32file(f)

Get the CRC32 checksum of a file.

Parameters:f (str) – Filename
Returns:CRC32 digest
Return type:str
pylib.fun.csvtosql(reader, table_name, db_conn=None, date_formats=None, rename=None)
Convert a CSV file to an in-memory database. The first row is always assumed
to be column headers.
Example usage::
>>> from pylib.fun import csvtosql
>>> reader = open("file.csv", "rb")
>>> conn = csvtosql(reader, "table1", date_formats={'Date': '%Y-%m-%d'})
>>> cursor = conn.cursor()
>>> cursor.execute("SELECT foo FROM table1")
>>> for row in cursor:
...    print(row["foo"])
Parameters:
  • reader (_csv.reader) – CSV reader
  • table_name (str) – Datbase table name
  • db_conn (sqlite3.Connection) – Use existing database connection
  • date_formats (dict) – Date formatting
Returns:

sqlite3 database connection

Return type:

sqlite3.Connection

pylib.fun.curry

Alias for functools.partial

alias of partial

pylib.fun.daemonize(pidfile, stdin='/dev/null', stdout='/dev/null', stderr='/dev/null', wdir='/', sigterm=True)
Use double-fork trick to daemonize a program. The second fork only ensures
that the daemon cannot reacquire a controlling terminal.
Parameters:
  • pidfile (str) – Process ID file
  • stdin (str) – File to use for standard input
  • stdout (str) – File to use for standard output
  • stderr (str) – File to use for standard error
  • wdir (str) – Working directory
  • sigterm (bool) – Register a dummy SIGTERM handler
pylib.fun.depcheck(deps)

Check and return a list of lost or found dependencies.

Parameters:deps (dict) – Dependencies {“files”: [...], “execs”: [...]}
Returns:Tuple of (False, summary) or (True, summary)
Return type:tuple
pylib.fun.execute(command, kill=False)

Simple(r) function to execute a command.

Parameters:
  • command (str|list) – Command
  • kill (bool) – Raise exception if the subprocess returncode is non-zero
Returns:

Summary of ‘stdout’, ‘stderr’, ‘status’, ‘env’, and ‘command’

Return type:

pylib.fun.Summary

pylib.fun.execute_async(command)

Execute a command as a background process, returning a handle to it that can be polled to later produce a summary.

Parameters:command (str|list) – Command
Returns:Summary of ‘stdout’, ‘stderr’, ‘status’, ‘env’, and ‘command’
Return type:subprocess.Popen
pylib.fun.expand(f, links=True, evars=True)

Expand a file or directory name completely.

Example::
>>> from pylib.fun import expand
>>> expand("~/sym/link/$DIR", links=True, evars=True)
'/home/user/real/path'
Parameters:
  • f (str) – File name
  • links (bool) – Resolve symbolic links
  • evars (bool) – Resolve environment variables
Returns:

Expanded file or directory name

Return type:

str

pylib.fun.flatten(l)

Flatten a 2-dimensional list into a 1-dimensional list.

Parameters:l (list) – 2-D list
Returns:Flattened list
Return type:list
pylib.fun.initconf(config, name)

Initialize a Namespace with arguments from a XML file.

Parameters:
  • config (str) – Configuration file
  • name (str) – Configuration section name
Returns:

Namespace

Return type:

pylib.fun.Namespace

pylib.fun.isexec(f)

Test if a file shows all of the signs of being an executable.

Parameters:f (str) – Filename
Returns:Is file an executable
Return type:bool
pylib.fun.iso8601(tz='UTC')

Get ISO 8601 timestamp as a string.

Parameters:tz (str) – Timezone
Returns:Timestamp
Return type:str
pylib.fun.local(command, stdout=None, stderr=None, kill=False, **kwargs)

Launch a command locally.

Parameters:
  • command (list) – Command
  • stdout (str) – Output log filename
  • stderr (str) – Error log filename
  • kill (bool) – Raise exception if subprocess exits abnormally
Returns:

Command summary

Return type:

pylib.fun.Summary

pylib.fun.machine()

Get machine information.

Returns:Machine of ‘hostname’
Return type:pylib.fun.Machine
pylib.fun.matlabpath()

Get the current user’s Matlab PATH as list.

Returns:Matlab path
Return type:list
pylib.fun.md5file(f)

Get the md5sum of a file.

Parameters:f (str) – Filename
Returns:hex digest
Return type:str
pylib.fun.memoize(fn)
Memoization wrapper that can handle function attributes, mutable arguments,
and can be applied either as a decorator or at runtime.
Parameters:fn (function) – Function
Returns:Memoized function
Return type:function
pylib.fun.older(f, days=1)

Check if a file modification is older than N days.

Parameters:
  • f (str) – Filename
  • days (int) – Number of days
Returns:

True or False

Return type:

bool

pylib.fun.pbsubmit(command, stdout=None, stderr=None, kill=False, **kwargs)

Launch a command through pbsubmit.

Parameters:
  • command (list) – Command
  • queue (str) – Queue e.g., “max50”
  • mem (int) – Required memory (MB)
  • stdout (str) – Output log filename
  • stderr (str) – Error log filename
  • kill (bool) – Raise exception if subprocess exits abnormally
Returns:

Command summary

Return type:

pylib.fun.Summary

pylib.fun.program()

Get program information.

Returns:Program of ‘file’, ‘base’, ‘root’, ‘ext’, ‘dir’, ‘md5’, and ‘args’
Return type:pylib.fun.Program
pylib.fun.py3k = False

Use this to test if in a Python 3 context.

pylib.fun.repeat(fun, every)

Repeat a function at a user-defined interval within a separate process.

Example usage::

import time from pylib.fun import repeat

def myfun():
print(“Hello, World!”)

p = repeat(myfun, 1) p.start() time.sleep(5) p.terminate()

Parameters:
  • fun (function) – Function
  • every (int) – Interval in seconds
Returns:

Process handle

Return type:

multiprocessing.Process

pylib.fun.sbatch(command, stdout=None, stderr=None, kill=False, **kwargs)

Launch a command through sbatch.

Parameters:
  • command (list) – Command
  • queue (str) – Partition e.g., “ncf”
  • mem (float) – Required memory (MB)
  • tmp (float) – Temporary storage required (MB)
  • cpus (int) – Number of CPUs required
  • time (str) – Execution time limit in “hour:min:sec” format
  • stdout (str) – Output log filename
  • stderr (str) – Error log filename
  • kill (bool) – Raise exception if subprocess exits abnormally
Returns:

Command summary

Return type:

pylib.fun.Summary

pylib.fun.spinner = <itertools.cycle object>

Use this to render a command-line spinning pinwheel cursor.

pylib.fun.splitext(f)

Split filename into (base, extension).

Note

As opposed to os.path.splitext this function handles .nii.gz as a single extension.

Parameters:f (str) – Filename
Returns:Tuple of (root, ext)
Return type:tuple
pylib.fun.srun(command, stdout=None, stderr=None, **kwargs)

Launch a command through srun, inside of an sbatch context.

Parameters:
  • command (list) – Command
  • stdout (str) – Output log filename
  • stderr (str) – Error log filename
Returns:

Command summary

Return type:

pylib.fun.Summary

pylib.fun.summarize_async(pending_tasks)

Poll against a list of commands and summarize them, returning an iterator over results.

Parameters:pending_tasks (list) – List of tuples of (launcher, provenance, process)
Returns:List of summaries of ‘stdout’, ‘stderr’, ‘status’, ‘env’, and ‘command’
Return type:pylib.fun.Summary or list
pylib.fun.which(x)

Equivalent to POSIX which command.

Parameters:x (str) – Executable
Returns:Path to executable
Return type:str