tramway.inference package

tramway.inference.base module

class tramway.inference.base.Local(index, data, center=None, span=None, boundary=None)

Bases: tramway.core.lazy.Lazy

Spatially local subset of elements (e.g. translocations). Abstract class.

index

This cell’s index as referenced in FiniteElements.

Type:int
data

Elements, either terminal or not.

Type:collection of terminal elements or Local
center

Cell center coordinates.

Type:array-like
span

Difference vectors from this cell’s center to adjacent centers.

Type:array-like
boundary/hull

Convex hull of the contained locations.

Type:scipy.spatial.qhull.ConvexHull
boundary

any, property

Cell boundary/hull.

center
data
dim

int, property

Dimension of the terminal elements.

get_localization_error(_kwargs=None, _default_value=None, _localization_error_is_sigma=False, **kwargs)

Return the localization error as \(sigma^2\).

Parameters:
  • _kwargs (dict) – mutable keyword arguments; ‘localization_error’, ‘sigma’ and ‘sigma2’ are popped out.
  • _default_value (float) – default localization error (subject to _localization_error_is_sigma).
  • _localization_error_is_sigma (bool) – argument ‘localization_error’ in _kwargs or kwargs is considered as \(sigma\).
Returns:

\(sigma^2\).

Return type:

float

hull

scipy.spatial.qhull.ConvexHull (or any if not convex), property

Alias for boundary.

index
span
tcount

int, property

Total number of terminal elements (e.g. translocations).

volume

float, property

Surface area (2D) or volume (3D+) of the convex hull.

Note that the hull should be convex so that the volume can be transparently extracted. In other cases, volume can be manually set just like a normal attribute.

class tramway.inference.base.Distributed(cells, adjacency, index=None, center=None, span=None, central=None, boundary=None)

Bases: tramway.inference.base.Local

central

margin cells are not central.

Type:array of bools
adjacency

scipy.sparse.csr_matrix, rw property

Cell adjacency matrix. Row and column indices are to be mapped with indices.

any_cell()
ccount

int, ro property

Total number of terminal cells. Duplicates are ignored.

cells

list or OrderedDict, rw property for data

Collection of Local. Indices may not match with the global index attribute of the elements, but match with attributes central, adjacency and degree.

central
clear_caches()
degree

array of ints, ro lazy property

Number of adjacent cells.

dim

int, ro property

Dimension of the terminal points.

flatten()
grad(i, X, index_map=None, **kwargs)

Local spatial gradient.

Sub-classes are free to return multi-component gradients as matrices provided that they exhibit as many columns as there are dimensions in the (trans-)location data. grad_sum() is then responsible for summing all the elements of such matrices.

Parameters:
  • i (int) – cell index at which the gradient is evaluated.
  • X (numpy.ndarray) – vector of a scalar measurement at every cell.
  • index_map (numpy.ndarray) – index map that converts cell indices to indices in X.
Returns:

gradient vector with as many elements as spatial dimensions.

Return type:

numpy.ndarray

See also grad1() and documentation section Gradients.

grad_sum(i, grad2, index_map=None)

Sum for local penalties (e.g. local gradient vector with squared elements) at a given cell.

Parameters:
  • i (int) – cell index.
  • grad2 (numpy.ndarray) – local penalties (e.g. local gradient vector with squared elements).
  • index_map (numpy.ndarray) – index mapping, useful to convert cell indices to positional indices in an optimization array for example.
Returns:

weighted sum of the elements of grad2.

Return type:

float

group(ngroups=None, max_cell_count=None, cell_centers=None, adjacency_margin=2, connected=False)

Make groups of cells.

This builds up an extra hierarchical level. For example if self is a FiniteElements of Cell, then this returns a FiniteElements of FiniteElements (the groups) of Cell.

Several grouping strategies are proposed.

Parameters:
  • ngroups (int) – number of groups.
  • max_cell_count (int) – maximum number of cells per group.
  • cell_centers (array-like) – spatial centers of the groups. Cells are sorted by nearest neighbours. If not provided, group() will use a k-means approach to positionning the centers.
  • adjacency_margin (int) – groups are dilated to the adjacent cells adjacency_margin times. Defaults to 2.
  • connected (bool) – separates the connected components from one another. Conflicts with the other arguments.
Returns:

a new object of the same type as self that contains other such objects as cells.

Return type:

FiniteElements

indices
items()
keys()
local_variation(i, X, index_map=None, **kwargs)

Local spatial variation, gradient-like, aimed at penalizing spatial variations.

See also delta0().

As of version 0.3.8: new; called for spatial regularization in stochastic_dv.

As of version 0.4: default implementation becomes delta0().

May become the new default for spatial regularization.

Parameters:
  • i (int) – cell index at which the gradient is evaluated.
  • X (numpy.ndarray) – vector of a scalar measurement at every cell.
  • index_map (numpy.ndarray) – index map that converts cell indices to indices in X.
Returns:

delta vector with as many elements as there are spatial dimensions.

Return type:

numpy.ndarray

neighbours(i)

Indices of neighbour cells.

Argument:

i (int): cell index.
Returns:indices of the neighbour cells of cell i.
Return type:numpy.ndarray
reverse

dict of ints, ro lazy property

Get “local” indices from global ones.

run(function, *args, **kwargs)

Apply a function that takes a group (FiniteElements) of terminal cells as input argument, plus args and kwargs, and must return a pandas.DataFrame array with the indices referring to cells.

function is called for each group of terminal cells, adjacency margins are removed if any, and the resulting DataFrame are merged into a single DataFrame.

function may instead be applied to self in the case self.cells has been overloaded to exhibit the output features as attributes.

Parameters:
  • function (callable) – the function to be called on each terminal FiniteElements. Its first argument is the FiniteElements object. It should return maps as a DataFrame and optionally posteriors as Series or DataFrame.
  • args (list) – positional arguments for function after the first one.
  • kwargs (dict) – keyword arguments for function; the following arguments are popped out of kwargs:
  • returns (list) – attributes to be collected from all the individual cells as return values; if defined, the values returned by function are ignored.
  • worker_count (int) – number of simultaneously working processing units, if the self.cells .
  • profile (bool or str or tuple) – profile each child job if any; if str, dump the output stats into .prof files; if tuple, print a report with print_stats() and tuple elements as input arguments.
  • function_worker_count (int) – if worker_count is a reserved keyword argument for function, function_worker_count can be specified and is passed to function with name/keyword worker_count; in this case, worker_count must be a multiple of function_worker_count.
Returns:

single merged array of maps. If function returns two output arguments, run() also returns a second merged array of posterior probabilities(?).

Return type:

pandas.DataFrame

space_cols

list of ints or strings, ro property

Column indices for coordinates of the terminal points.

tcount

int, ro property

Total number of terminal points. Duplicates are ignored.

values()
class tramway.inference.base.Cell(index, data, center=None, span=None, boundary=None)

Bases: tramway.inference.base.Local

Spatially constrained subset of (trans-)locations with associated intermediate calculations.

cache

Depending on the inference approach and objective, caching an intermediate result may avoid repeating many times a same computation. Usage of this cache is totally free and comes without support for concurrency.

Type:any
cache
clear_cache()
dim

int, property

Dimension of the terminal elements.

fuzzy
space_cols

list of ints or strings, lazy

Column indices for coordinates.

space_data
tcount

int

Number of (trans-)locations.

time_col

int or string, lazy

Column index for time.

time_data
class tramway.inference.base.Locations(index, data, center=None, span=None, boundary=None)

Bases: tramway.inference.base.Cell

locations

array-like, property

Locations as a matrix of coordinates and times with as many columns as dimensions; this is an alias for data.

r

array-like, ro property

Location coordinates; xy is an alias

t

array-like, ro property

Location timestamps.

xy
class tramway.inference.base.Translocations(index, translocations, center=None, span=None, boundary=None)

Bases: tramway.inference.base.Cell

origins

Initial locations (both spatial coordinates and times).

Type:array-like, ro property
destinations

Final locations (both spatial coordinates and times).

Type:array-like, ro property
destinations
dr

array-like, ro property

Translocation displacements in space; dxy is an alias.

dt

array-like, ro property

Translocation durations.

dxy
origins
r
t

array-like, ro property

Initial timestamps.

translocations

array-like, property

Translocations as a matrix of variations of coordinate and time with as many columns as dimensions; this is an alias for data.

class tramway.inference.base.Maps(maps, mode=None, posteriors=None)

Bases: tramway.core.lazy.Lazy

Basic container for maps, posteriors and the associated input parameters used to generate the maps.

Attributes distributed_translocations, partition_file, tessellation_program, version are deprecated and will be removed.

Functional dependencies:

  • setting maps unsets _features
defattr(attr, val)
features

list or None

List of different feature names without the coordinate suffix.

For example, if maps has columns diffusivity, force x and force y, features will return diffusivity and force.

maps
sub(ix, reindex=False)

Sub-map.

Operates like loc on a DataFrame.

Parameters:
  • ix (slice or array-like) – cell indices or boolean array.
  • reindex (bool) – make index range from 0 without missing integer.
Returns:

sub-map.

Return type:

tramway.inference.base.Maps

variables

Alias for features.

tramway.inference.base.FiniteElement

alias of tramway.inference.base.Cell

tramway.inference.base.FiniteElements

alias of tramway.inference.base.Distributed

tramway.inference.base.identify_columns(points, trajectory_col=True)

Identify columns by type.

Parameters:
  • points (array-like) – location coordinates and trajectory index.
  • trajectory_col (bool or int or str) – trajectory column index or name, or False or None if there is no trajectory information is to be found or extracted, or True to automatically identify such a column.
Returns:

(coord_cols, trajectory_col, get_var, get_point)

Return type:

tuple

coord_cols is an array of column indices/names other than the trajectory column. If the data contains typical translocation column names such as ‘dx’, ‘dy’, ‘dt’, coord_cols is a tuple with, as first element, an array of location column names, and as second element an array of translocation column names.

trajectory_col is the column index/name that contains tracking information.

get_var is a callable that takes an array like points and column identifiers and returns the corresponding sub-matrix.

get_point is a callable that takes an array like points and row indices and returns the corresponding sub-matrix.

tramway.inference.base.get_locations(points, index=None, coord_cols=None, get_var=None, get_point=None)

Make helpers for manipulating the point data.

Parameters:
  • points (array-like) – location coordinates and trajectory index.
  • index (ndarray or pair of ndarrrays or sparse matrix) – point-cell association (see CellStats documentation or cell_index()).
Returns:

(locations, location_cell, get_point)

Return type:

tuple

locations are point coordinates (same format as points with the trajectory column removed.

location_cell is either an array of cell indices (same size as locations) or a callable that takes a cell index and returns a boolean array with Trues for locations associated with the specified cell and Falses elsewhere.

get_point is a callable that takes an array like points or locations and row indices and returns the corresponding rows in the same format.

tramway.inference.base.get_translocations(points, index=None, coord_cols=None, trajectory_col=True, get_var=None, get_point=None)

Identify translocations as initial and final points.

Parameters:
  • points (array-like) – location coordinates and trajectory index.
  • index (ndarray or pair of ndarrays or sparse matrix) – point-cell association (see CellStats documentation or cell_index()).
Returns:

(initial_point, final_point, initial_cell, final_cell, get_point)

Return type:

tuple

initial_point and final_point are point coordinates (same format as points with the trajectory index column removed) respectively of the initial and final displacement locations.

initial_cell and final_cell are either arrays of cell indices (same size as initial_point) or callables that take a cell index and return a boolean array with Trues for displacements/translocations associated with the specified cell and Falses elsewhere.

get_point is a callable that takes an array like points and row indices and returns the corresponding rows in the same format.

tramway.inference.base.distributed(cells, new_cell=None, new_group=<class 'tramway.inference.base.Distributed'>, fuzzy=None, new_cell_kwargs={}, new_group_kwargs={}, fuzzy_kwargs={}, new=None, include_empty_cells=False, verbose=False)

Build a FiniteElements-like object from a CellStats object.

Cells with no (trans-)locations are discarded in addition to those with null or negative label.

Parameters:
  • cells (CellStats) – tessellation and partition; must contain (trans-)location data.
  • new_cell (callable) – cell constructor; default is Locations or Translocations depending on the data in cells.
  • new_group (callable) – constructor for groups of cell; default is FiniteElements.
  • fuzzy (callable) – (trans-)location-cell weighting function; the default for translocations considers the initial cell, no weight.
  • new_cell_kwargs (dict) – keyword arguments for new_cell.
  • new_group_kwargs (dict) – keyword arguments for new_group.
  • fuzzy_kwargs (dict) – keyword arguments for fuzzy.
  • new (callable) – legacy argument; use new_group instead.
  • include_empty_cells (bool) – do not discard cells with no (trans-)locations.

fuzzy takes a Tessellation object, a cell index (int), location coordinates (array-like), cell indices (array-like or callable) and the get_point function returned by get_locations or get_translocations.

If the data are translocations, fuzzy takes a pair of arrays as location coordinates (initial and final locations respectively) and a pair of arrays or callables as cell indices (initial and final cells respectively).

fuzzy returns an array of booleans or values of any scalar type that can be evaluated logically. The returned array contains as many elements as input (trans-)locations.

class tramway.inference.base.TrackedMolecules(index, translocations, center=None, span=None, boundary=None)

Bases: tramway.inference.base.Translocations

n

Trajectory indices.

Type:numpy.ndarray
n
class tramway.inference.base.DistributeMerge(cells, adjacency, min_location_count=None)

Bases: tramway.inference.base.Distributed

merged
exception tramway.inference.base.DiffusivityWarning(diffusivity, lower_bound=None)

Bases: RuntimeWarning

exception tramway.inference.base.OptimizationWarning

Bases: RuntimeWarning

tramway.inference.base.smooth_infer_init(cells, min_diffusivity=None, jeffreys_prior=None, **kwargs)

Initialize the diffusivity array for translocations.

This function is a collection of checks and initial computations suitable for infer functions in inference plugins.

Parameters:
  • cells (FiniteElements) – first input argument of the infer function.
  • min_diffusivity (float) – minimum allowed diffusivity value.
  • jeffreys_prior (bool) – activate Jeffreys’ prior.

Returns:

  • index (numpy.ndarray) – cell indices corresponding to arrays n, dt_mean and D_initial.
  • reverse_index (numpy.ndarray) – element indices in arrays index, n, dt_mean, D_initial for each cell index.
  • n (numpy.ndarray) – translocation count for each cell.
  • dt_mean (numpy.ndarray) – mean translocation duration for each cell.
  • D_initial (numpy.ndarray) – initial diffusivity array.
  • min_diffusivity (float or None) – minimum diffusivity value; the returned value depends on jeffreys_prior.
  • D_bounds (list) – list of (lower bound, upper bound) couples; suitable as bounds input argument for the scipy.optimize.minimize() function.
  • border (numpy.ndarray) – MxD boolean matrix with M the number of cells and D the number of space dimensions

tramway.inference.d plugin

tramway.inference.d.infer_D(cells, diffusivity_prior=None, jeffreys_prior=None, min_diffusivity=None, max_iter=None, epsilon=None, rgrad=None, **kwargs)

tramway.inference.degraded_d plugin

tramway.inference.standard_d plugin

class tramway.inference.standard_d.Result(result, success, message, ncalls)

Bases: tuple

message

Alias for field number 2

ncalls

Alias for field number 3

result

Alias for field number 0

success

Alias for field number 1

tramway.inference.standard_d.d_neg_posterior1(diffusivity, cells, sigma2, diffusivity_prior, jeffreys_prior, dt_mean, min_diffusivity, index, reverse_index, grad_kwargs)

Similar to smooth_d_neg_posterior(). The smoothing prior features an alternative spatial “gradient” implemented using local_variation() instead of grad().

tramway.inference.standard_d.infer_smooth_D(cells, diffusivity_prior=None, jeffreys_prior=None, min_diffusivity=None, max_iter=None, epsilon=None, rgrad=None, verbose=False, **kwargs)
tramway.inference.standard_d.interruptible_minimize(func, prms, args, **kwargs)
tramway.inference.standard_d.smooth_d_neg_posterior(diffusivity, cells, sigma2, diffusivity_prior, jeffreys_prior, dt_mean, min_diffusivity, index, reverse_index, grad_kwargs)

Adapted from InferenceMAP’s dDDPosterior procedure:

for (int a = 0; a < NUMBER_OF_ZONES; a++) {
    ZONES[a].gradDx = dvGradDx(DD,a);
    ZONES[a].gradDy = dvGradDy(DD,a);
    ZONES[a].priorActive = true;
}

for (int z = 0; z < NUMBER_OF_ZONES; z++) {
    const double gradDx = ZONES[z].gradDx;
    const double gradDy = ZONES[z].gradDy;
    const double D = DD[z];

    for (int j = 0; j < ZONES[z].translocations; j++) {
        const double dt = ZONES[z].dt[j];
        const double dx = ZONES[z].dx[j];
        const double dy = ZONES[z].dy[j];
        const double Dnoise = LOCALIZATION_ERROR*LOCALIZATION_ERROR/dt;

        result += - log(4.0*PI*(D + Dnoise)*dt) - ( dx*dx + dy*dy)/(4.0*(D+Dnoise)*dt);
    }

    if (ZONES[z].priorActive == true) {
        result -= D_PRIOR*(gradDx*gradDx*ZONES[z].areaX + gradDy*gradDy*ZONES[z].areaY);
        if (JEFFREYS_PRIOR == 1) {
            result += 2.0*log(D) - 2.0*log(D*ZONES[z].dtMean + LOCALIZATION_ERROR*LOCALIZATION_ERROR);
        }
    }
}

return -result;

tramway.inference.d_conj_prior plugin

tramway.inference.d_conj_prior.infer_d_conj_prior(cells, alpha=0.95, return_zeta_spurious=True, trust=False, **kwargs)

Infer diffusivity MAP and confidence interval using a conjugate prior [Serov et al. 2019]

Parameters:
  • cells (tramway.inference.base.Distributed) – distributed cells.
  • alpha (float) – confidence level for CI estimation.
  • return_zeta_spurious (bool) – add variable zeta_spurious to the returned dataframe.
  • trust (bool) – if False, silently catch any exception raised in get_D_confidence_interval.
Returns:

maps including diffusivity MAPs and confidence intervals, and SNR extensions.

Return type:

pandas.DataFrame

Other valid input arguments are localization- and gradient-related arguments.

See also get_D_confidence_interval(), get_localization_error() and get_grad_kwargs().

tramway.inference.d_conj_prior.infer_d_map_and_ci(*args, **kwargs)

Backward-compatibility alias for infer_d_conj_prior().

tramway.inference.ddrift plugin

tramway.inference.ddrift.infer_DD(cells, diffusivity_prior=None, drift_prior=None, jeffreys_prior=False, min_diffusivity=None, max_iter=None, epsilon=None, rgrad=None, **kwargs)

tramway.inference.degraded_ddrift plugin

tramway.inference.standard_ddrift plugin

tramway.inference.standard_ddrift.dd_neg_posterior1(x, dd, cells, sigma2, diffusivity_prior, drift_prior, jeffreys_prior, dt_mean, min_diffusivity, index, reverse_index, grad_kwargs)
tramway.inference.standard_ddrift.infer_smooth_DD(cells, diffusivity_prior=None, drift_prior=None, jeffreys_prior=False, min_diffusivity=None, max_iter=None, epsilon=None, rgrad=None, verbose=False, **kwargs)
tramway.inference.standard_ddrift.smooth_dd_neg_posterior(x, dd, cells, sigma2, diffusivity_prior, drift_prior, jeffreys_prior, dt_mean, min_diffusivity, index, reverse_index, grad_kwargs)

tramway.inference.density plugin

tramway.inference.density.infer_density(cells, convex_hull='fallback', trust_volume=False, scale=False, min_volume=None, volume=None, **kwargs)

If knn was defined for the partition, use convex_hull='always'.

With the Voronoi-based tessellations (random, kmeans, gwr), do not set trust_volume to True. With the other tessellations (grid, hexagon, kdtree), use trust_volume=True if convex_hull evaluates to True.

With the regular tessellations (grid, hexagon), scale can be True. With the other tessellations, do not set scale to True.

tramway.inference.df plugin

tramway.inference.df.infer_DF(cells, diffusivity_prior=None, force_prior=None, jeffreys_prior=False, min_diffusivity=None, max_iter=None, epsilon=None, rgrad=None, **kwargs)

tramway.inference.degraded_df plugin

tramway.inference.standard_df plugin

tramway.inference.standard_df.df_neg_posterior1(x, df, cells, sigma2, diffusivity_prior, force_prior, jeffreys_prior, dt_mean, min_diffusivity, index, reverse_index, grad_kwargs)
tramway.inference.standard_df.infer_smooth_DF(cells, diffusivity_prior=None, force_prior=None, potential_prior=None, jeffreys_prior=False, min_diffusivity=None, max_iter=None, epsilon=None, rgrad=None, verbose=False, **kwargs)

Argument potential_prior is an alias for force_prior which penalizes the large force amplitudes.

tramway.inference.standard_df.smooth_df_neg_posterior(x, df, cells, sigma2, diffusivity_prior, force_prior, jeffreys_prior, dt_mean, min_diffusivity, index, reverse_index, grad_kwargs)

tramway.inference.dv plugin

class tramway.inference.dv.DV(diffusivity, potential, diffusivity_prior=None, potential_prior=None, minimum_diffusivity=None, positive_diffusivity=None, prior_include=None)

Bases: tramway.core.chain.ChainArray

Convenience class for handling the D+V parameter vector and related priors.

Objects of this class are not supposed to be serialized in .rwa files.

D
V
diffusivity_prior(j)
minimum_diffusivity
potential_prior(j)
prior_include
tramway.inference.dv.dv_neg_posterior(x, dv, cells, sigma2, jeffreys_prior, dt_mean, index, reverse_index, grad_kwargs, y0, verbose, posteriors)

Adapted from InferenceMAP’s dvPosterior procedure modified:

for (int a = 0; a < NUMBER_OF_ZONES; a++) {
    ZONES[a].gradVx = dvGradVx(DV,a);
    ZONES[a].gradVy = dvGradVy(DV,a);
    ZONES[a].gradDx = dvGradDx(DV,a);
    ZONES[a].gradDy = dvGradDy(DV,a);
    ZONES[a].priorActive = true;
}


for (int z = 0; z < NUMBER_OF_ZONES; z++) {
    const double gradVx = ZONES[z].gradVx;
    const double gradVy = ZONES[z].gradVy;
    const double gradDx = ZONES[z].gradDx;
    const double gradDy = ZONES[z].gradDy;

    const double D = DV[2*z];

    for (int j = 0; j < ZONES[z].translocations; j++) {
        const double dt = ZONES[z].dt[j];
        const double dx = ZONES[z].dx[j];
        const double dy = ZONES[z].dy[j];
        const double  Dnoise = LOCALIZATION_ERROR*LOCALIZATION_ERROR/dt;

        result += - log(4.0*PI*(D + Dnoise)*dt) - ((dx-D*gradVx*dt)*(dx-D*gradVx*dt) + (dy-D*gradVy*dt)*(dy-D*gradVy*dt))/(4.0*(D+Dnoise)*dt);
    }

    if (ZONES[z].priorActive == true) {
        result -= V_PRIOR*(gradVx*gradVx*ZONES[z].areaX + gradVy*gradVy*ZONES[z].areaY);
        result -= D_PRIOR*(gradDx*gradDx*ZONES[z].areaX + gradDy*gradDy*ZONES[z].areaY);
        if (JEFFREYS_PRIOR == 1) {
            result += 2.0*log(D*1.00) - 2.0*log(D*ZONES[z].dtMean + LOCALIZATION_ERROR*LOCALIZATION_ERROR);
    }
}

with dx-D*gradVx*dt and dy-D*gradVy*dt modified as dx+D*gradVx*dt and dy+D*gradVy*dt respectively.

See also DV.

tramway.inference.dv.dv_neg_posterior1(x, dv, cells, sigma2, jeffreys_prior, dt_mean, index, reverse_index, grad_kwargs, y0, verbose, posteriors)

Similar to dv_neg_posterior(). The smoothing priors feature an alternative spatial “gradient” implemented using local_variation() instead of grad().

tramway.inference.dv.inferDV(cells, diffusivity_prior=None, potential_prior=None, jeffreys_prior=False, min_diffusivity=None, max_iter=None, epsilon=None, export_centers=False, verbose=True, compatibility=False, D0=None, V0=None, rgrad=None, **kwargs)

tramway.inference.gradient module

tramway.inference.gradient.get_grad_kwargs(_kwargs=None, gradient=None, grad_epsilon=None, grad_selection_angle=None, compatibility=None, grad=None, epsilon=None, **kwargs)

Parse grad1() keyworded arguments.

Parameters:
  • _kwargs (dict) – mutable keyword arguments; all the keywords below are popped out.
  • gradient (str) – either grad1 or gradn.
  • grad_epsilon (float) – eps argument for grad1() (or neighbours_per_axis()).
  • grad_selection_angle (float) – selection_angle argument for grad1() (or neighbours_per_axis()).
  • compatibility (bool) – backward compatibility with InferenceMAP.
  • grad (str) – alias for gradient.
  • epsilon (float) – alias for grad_epsilon.
Returns:

keyworded arguments to grad().

Return type:

dict

Note: grad and gradient are currently ignored.

tramway.inference.gradient.neighbours_per_axis(i, cells, centers=None, eps=None, selection_angle=None)

See also grad1().

tramway.inference.gradient.grad1(cells, i, X, index_map=None, eps=None, selection_angle=None, na=nan)

Local gradient by 2 degree polynomial interpolation along each dimension independently.

Considering spatial coordinate \(x\), bin \(i\) and its neighbour bins \(\\mathcal{N}_i\):

\[\begin{split}\left.X'_i\right|_x = \left\{ \begin{array}{ll} \frac{X_i - \overline{X}_{\mathcal{N}_i}}{x_i - \overline{x}_{\mathcal{N}_i}} & \textrm{ if either } \mathcal{N}_i^- \textrm{ or } \mathcal{N}_i^+ \textrm{ is } \emptyset \\ b + 2 c x_{i} & \textrm{ with } \left[ \begin{array}{ccc} 1 & \overline{x}_{\mathcal{N}_i^-} & \overline{x}_{\mathcal{N}_i^-}^2 \\ 1 & x_i & x_i^2 \\ 1 & \overline{x}_{\mathcal{N}_i^+} & \overline{x}_{\mathcal{N}_i^+}^2 \end{array} \right] . \left[ \begin{array}{c} a \\ b \\ c \end{array} \right] = \left[ \begin{array}{c}\overline{X}_{\mathcal{N}_i^-} \\ X_i \\ \overline{X}_{\mathcal{N}_i^+}\end{array} \right] \textrm{ otherwise } \\ \end{array} \right.\end{split}\]

Claims cache variable grad1.

Supports the InferenceMAP way of dividing the space in non-overlapping regions (quadrants in 2D) so that each neighbour is involved in the calculation of a single component of the gradient.

This is the default behaviour in cases where the adjacent attribute contains adequate labels, i.e. -j and j for each dimension j, representing one side and the other side of cell-i’s center. This is also the default behaviour otherwise, when eps is None.

If eps is defined, the calculation of a gradient component may recruit all the points minus those at a projected distance smaller than this value.

If selection_angle is defined, neighbours are selected in two symmetric hypercones which top angle is selection_angle times \(\pi\) radians.

If get_grad_kwargs() is called (most if not all the inference modes call this function), the default selection behaviour is equal to selection_angle=0.9. Otherwise, grad1() defaults to selection_angle=0.5 in 2D.

See also

neighbours_per_axis.

Parameters:
  • cells (tramway.inference.base.Distributed) – distributed cells.
  • i (int) – cell index at which the gradient is evaluated.
  • X (array) – vector of a scalar measurement at every cell.
  • index_map (array) – index map that converts cell indices to indices in X; for a given cell, should be constant.
  • eps (float) – minimum projected distance from cell-i’s center for neighbours to be considered in the calculation of the gradient; if eps is None, the space is instead divided in twice as many non-overlapping regions as dimensions, and each neighbour is assigned to a single region (counts against a single dimension). Incompatible with selection_angle.
  • selection_angle (float) – top angle of the neighbour selection hypercones; should be in the \([0.5, 1.0[\) range. Incompatible with eps.
  • na (float) – value for undefined components (no neighbours).
Returns:

local gradient vector with as many elements as there are dimensions in the (trans-)location data.

Return type:

array

tramway.inference.gradient.gradn(cells, i, X, index_map=None)

Local gradient by multi-dimensional 2 degree polynomial interpolation.

Parameters:
  • cells (tramway.inference.base.Distributed) – distributed cells.
  • i (int) – cell index at which the gradient is evaluated.
  • X (array) – vector of a scalar measurement at every cell.
  • index_map (array) – index map that converts cell indices to indices in X.
Returns:

local gradient vector with as many elements as there are dimensions in the (trans-)location data.

Return type:

array

tramway.inference.gradient.delta0(cells, i, X, index_map=None, **kwargs)

Differences with neighbour values:

\[\Delta X_i = \frac{1}{\sqrt{|\mathcal{N}_i|}} \left[ \frac{X_i-X_j}{|| \textbf{x}_i-\textbf{x}_j ||} \right]_{j \in \mathcal{N}_i}\]

The above scaling is chosen so that combining the element-wise square of local_variation() with grad_sum() results in the following scalar penalty:

\[\Delta X_i^2 = \frac{1}{ | \mathcal{N}_i | } \sum_{j \in \mathcal{N}_i} \left( \frac{X_i-X_j}{|| \textbf{x}_i-\textbf{x}_j ||} \right)^2\]

Claims cache variable ‘delta0’.

Parameters:
  • cells (tramway.inference.base.Distributed) – distributed cells.
  • i (int) – cell index at which the differences are evaluated.
  • X (array) – vector of a scalar measurement at every cell.
  • index_map (array) – index map that converts cell indices to indices in X.
Returns:

difference vector with as many elements as there are neighbours.

Return type:

array

tramway.inference.gradient.delta0_without_scaling(cells, i, X, index_map=None, **kwargs)
tramway.inference.gradient.delta1(cells, i, X, index_map=None, eps=None, selection_angle=None)

Local spatial variation.

Similar to grad1. Considering spatial coordinate \(x\), bin \(i\) and its neighbour bins \(\\mathcal{N}_i\):

\[\begin{split}\left.\Delta X_i\right|_x = \left( \begin{array}{ll} \frac{X_i - \overline{X}_{\mathcal{N}_i^-}}{x_i - \overline{x}_{\mathcal{N}_i^-}} & \textrm{ or } 0 \textrm{ if } \mathcal{N}_i^- \textrm{ is } \emptyset \\ \frac{X_i - \overline{X}_{\mathcal{N}_i^+}}{x_i - \overline{x}_{\mathcal{N}_i^+}} & \textrm{ or } 0 \textrm{ if } \mathcal{N}_i^+ \textrm{ is } \emptyset \\ \end{array} \right)\end{split}\]

Also claims cache variable grad1 in a compatible way.

Parameters:
  • i (int) – cell index at which the gradient is evaluated.
  • X (numpy.ndarray) – vector of a scalar measurement at every cell.
  • index_map (numpy.ndarray) – index map that converts cell indices to indices in X.
Returns:

delta vector with as many elements as there are spatial dimensions.

Return type:

numpy.ndarray

tramway.inference.gradient.setup_with_grad_arguments(setup)

Add grad() related arguments to inference plugin setup.

Input argument setup is modified inplace.

tramway.inference.gradient.gradient_map(cells, feature=None, **kwargs)

tramway.inference.optimization module

class tramway.inference.optimization.BFGSResult(x, H, resolution, niter, f, df, projg, cumtime, err, diagnosis, ncalls)

Bases: tuple

H

Alias for field number 1

cumtime

Alias for field number 7

df

Alias for field number 5

diagnosis

Alias for field number 9

err

Alias for field number 8

f

Alias for field number 4

ncalls

Alias for field number 10

niter

Alias for field number 3

projg

Alias for field number 6

resolution

Alias for field number 2

x

Alias for field number 0

tramway.inference.optimization.minimize_sparse_bfgs(fun, x0, component, covariate, gradient_subspace, descent_subspace, args=(), bounds=None, _sum=<function sum>, gradient_sum=None, gradient_covariate=None, memory=10, eps=1e-06, ftol=1e-06, gtol=1e-10, low_df_rate=0.9, low_dg_rate=0.9, step_scale=1.0, max_iter=None, regul=None, regul_decay=1e-05, ls_kwargs={}, ls_regul=None, ls_step_max=None, ls_step_max_decay=None, ls_iter_max=None, ls_armijo_max=None, ls_wolfe=None, ls_failure_rate=0.9, fix_ls=None, fix_ls_trigger=5, gradient_initial_step=1e-08, Component=<class 'tramway.inference.optimization.Component'>, independent_components=False, newton=True, verbose=False, diagnosis=None, returns=(), max_runtime=None, update_timeout=None, **kwargs)

Let the objective function \(f(x) = \sum_{i \in C} f_{i}(x) \forall x \in \Theta\) be a linear function of sparse components \(f_{i}\) such that \(\forall j \notin G_{i}, \forall x \in \Theta, {\partial f_{i}}{\partial x_{j}}(x) = 0\).

Let the components also covary sparsely: \(\forall i \in C, \exists C_{i} \subset C | i \in C_{i}, \exists D_{i} \subset G_{i}, \forall j \in D_{i}, \forall x \in \Theta, \frac{\\partial f}{\partial x_{j}}(x) = \sum_{i' \in C_{i}} \frac{\partial f_{i'}}{\partial x_{j}}(x)\).

We may additionally need that \(\forall i \in C, D_{i} = \bigcap_{i' \in C_{i}} G_{i'}\), \(\bigcup_{i \in C} D_{i} = J\) and \(D_{i} \cap D_{j} = \emptyset \forall i, j \in C^{2}\) with \(J\) the indices of parameter vector \(x = \lvert x_{j}\rvert_{j \in J}\) (to be checked).

At iteration \(k\), let choose component \(i\) and minimize \(f\) wrt parameters \(\{x_{j} | j \in D_{i}\}\).

Compute gradient \(g_{i}(x) = \lvert\frac{\partial f}{\partial x_{j}}(x)\rvert_{j} = \sum_{i' \in C_{i}} \lvert\frac{\partial f_{i'}}{\partial x_{j}}(x) \rvert_{j}\) with (again) \(g_{i}(x)\rvert_{j} = 0 \forall j \notin G_{i}\).

Perform a Wolfe line search along descent direction restricted to subspace \(D_{i}\). Update \(x\) and compute the gradient again.

The inverse Hessian matrix must also be updated twice: before and after the update.

Parameters:
  • fun (callable) – takes a component index (int) and the parameter vector (numpy.ndarray) and returns a scalar float.
  • x0 (numpy.ndarray) – initial parameter vector.
  • component (callable) – takes an iteration index (int) and returns a component index (int); all the components are expected to be drawn exactly once per epoch; if component is an int, will be interpreted as the number of components.
  • covariate (callable) – takes a component index i (int) and returns a sequence of indices of the components that covary with i, including i.
  • gradient_subspace (callable) – takes a component index (int) and returns a sequence of indices of the parameters spanning the gradient subspace; if None, the gradient ‘subspace’ is the full space.
  • descent_subspace (callable) – takes a component index (int) and returns a sequence of indices of the parameters spanning the descent subspace; if None, the descent subspace equals to the gradient subspace.
  • args (tuple or list) – sequence of positional arguments to fun.
  • bounds (tuple or list) – sequence of pairs of (lower, upper) bounds on the parameters.
  • _sum (callable) – takes a list of float values and returns a float.
  • gradient_sum (callable) – replacement for _sum in the calculation of the gradient.
  • gradient_covariate (callable) – takes a parameter index (int) and returns a sequence of the components affected by this parameter.
  • memory (int) – number of memorized pairs of H updates in quasi-Newton mode.
  • eps (float) – initial scaling of the descent direction.
  • ftol (float) – maximum decrease in the local objective.
  • gtol (float) – maximum decrease in the projected gradient.
  • low_df_rate (float) – epoch-wise rate of low decrease in local objective (below ftol); at the end of an epoch, if this rate has been reached, then the iteration stops.
  • low_dg_rate (float) – epoch-wise rate of low decrease in the projected gradient (below gtol); at the end of an epoch, if this rate has been reached, then the iteration stops.
  • step_scale (float) – reduction factor on the line-searched step.
  • max_iter (int) – maximum number of iterations.
  • regul (float) – regularization trade-off coefficient for the L2-norm of the parameter vector.
  • regul_decay (float) – decay parameter for regul.
  • ls_kwargs (dict) – keyword arguments to wolfe_line_search().
  • ls_regul (float) – regularization trade-off coefficient for the L2-norm of the parameter vector update.
  • ls_step_max (float) – maximum L-infinite norm of the parameter vector update; if ls_step_max_decay is defined, ls_step_max can be an (initial, final) tuple instead.
  • ls_step_max_decay (float) – decay parameter for ls_step_max.
  • ls_iter_max (int) – maximum number of linesearch iterations.
  • ls_armijo_max (float) – maximum expected decrease in the local objective; throttles the Armijo threshold.
  • ls_wolfe (tuple) – (c2, c3) pair for wolfe_line_search().
  • ls_failure_rate (float) – epoch-wise rate of linesearch failure; at the end of an epoch, if this rate has been reached, then the iteration stops.
  • fix_ls (callable) – takes a component index (int) and the parameter vector (numpy.ndarray) to be modified inplace in the case of recurrent linesearch failures for this component.
  • fix_ls_trigger (int) – minimum number of iterations with successive linesearch failures on a given component for fix_ls to be triggered.
  • Component (type) – component constructor.
  • independent_components (bool) – whether to represent the local inverse Hessian submatrix for a component independently of the other components.
  • newton (bool) – quasi-Newton (BFGS) mode; if False, use vanilla gradient descent instead.
  • verbose (bool or logging.Logger) – verbose mode or logger.
  • diagnosis (callable) – function of the iteration number and the current and candidate-new components.
  • returns (sequence of str) – any subset of {‘f’, ‘df’, ‘projg’, ‘err’, ‘ncalls’, ‘diagnosis’} or ‘all’.
  • xref (numpy.ndarray) – reference final parameter vector; if defined, at each iteration, the L2-norm of the difference between this and the current parameter vector is evaluated and returned as attribute err; note that this computation may add quite some overhead.
Returns:

final parameter vector.

Return type:

BFGSResult

See also SparseFunction and wolfe_line_search().

tramway.inference.optimization.minimize_sparse_bfgs0(fun, x0, component, covariate, gradient_subspace, descent_subspace, args=(), bounds=None, _sum=<function sum>, gradient_sum=None, gradient_covariate=None, memory=10, eps=1e-06, ftol=1e-06, gtol=1e-10, low_df_rate=0.9, low_dg_rate=0.9, step_scale=1.0, max_iter=None, regul=None, regul_decay=1e-05, ls_regul=None, ls_step_max=None, ls_step_max_decay=None, ls_iter_max=None, ls_armijo_max=None, ls_wolfe=None, ls_failure_rate=0.9, fix_ls=None, fix_ls_trigger=5, gradient_initial_step=1e-08, independent_components=True, newton=True, verbose=False)

Let the objective function \(f(x) = \sum_{i \in C} f_{i}(x) \forall x \in \Theta\) be a linear function of sparse components \(f_{i}\) such that \(\forall j \notin G_{i}, \forall x \in \Theta, {\partial f_{i}}{\partial x_{j}}(x) = 0\).

Let the components also covary sparsely: \(\forall i \in C, \exists C_{i} \subset C | i \in C_{i}, \exists D_{i} \subset G_{i}, \forall j \in D_{i}, \forall x \in \Theta, \frac{\\partial f}{\partial x_{j}}(x) = \sum_{i' \in C_{i}} \frac{\partial f_{i'}}{\partial x_{j}}(x)\).

We may additionally need that \(\forall i \in C, D_{i} = \bigcap_{i' \in C_{i}} G_{i'}\), \(\bigcup_{i \in C} D_{i} = J\) and \(D_{i} \cap D_{j} = \emptyset \\forall i, j \in C^{2}\) with \(J\) the indices of parameter vector \(x = \lvert x_{j}\rvert_{j \in J}\) (to be checked).

At iteration \(k\), let choose component \(i\) and minimize \(f\) wrt parameters \(\{x_{j} | j \in D_{i}\}\).

Compute gradient \(g_{i}(x) = \lvert\frac{\partial f}{\partial x_{j}}(x)\rvert_{j} = \sum_{i' \in C_{i}} \lvert\frac{\partial f_{i'}}{\partial x_{j}}(x) \rvert_{j}\) with (again) \(g_{i}(x)\rvert_{j} = 0 \forall j \notin G_{i}\).

Perform a Wolfe line search along descent direction restricted to subspace \(D_{i}\). Update \(x\) and compute the gradient again.

The inverse Hessian matrix must also be updated twice: before and after the update.

Parameters:
  • fun (callable) – takes a component index (int) and the parameter vector (numpy.ndarray) and returns a scalar float.
  • x0 (numpy.ndarray) – initial parameter vector.
  • component (callable) – takes an iteration index (int) and returns a component index (int); all the components are expected to be drawn exactly once per epoch; if component is an int, will be interpreted as the number of components.
  • covariate (callable) – takes a component index i (int) and returns a sequence of indices of the components that covary with i, including i.
  • gradient_subspace (callable) – takes a component index (int) and returns a sequence of indices of the parameters spanning the gradient subspace; if None, the gradient ‘subspace’ is the full space.
  • descent_subspace (callable) – takes a component index (int) and returns a sequence of indices of the parameters spanning the descent subspace; if None, the descent subspace equals to the gradient subspace.
  • args (tuple or list) – sequence of positional arguments to fun.
  • bounds (tuple or list) – sequence of pairs of (lower, upper) bounds on the parameters.
  • _sum (callable) – takes a list of float values and returns a float.
  • gradient_sum (callable) – replacement for _sum in the calculation of the gradient.
  • gradient_covariate (callable) – takes a parameter index (int) and returns a sequence of the components affected by this parameter.
  • memory (int) – number of memorized pairs of H updates in quasi-Newton mode.
  • eps (float) – initial scaling of the descent direction.
  • ftol (float) – maximum decrease in the local objective.
  • gtol (float) – maximum decrease in the projected gradient.
  • low_df_rate (float) – epoch-wise rate of low decrease in local objective (below ftol); at the end of an epoch, if this rate has been reached, then the iteration stops.
  • low_dg_rate (float) – epoch-wise rate of low decrease in the projected gradient (below gtol); at the end of an epoch, if this rate has been reached, then the iteration stops.
  • step_scale (float) – reduction factor on the line-searched step.
  • max_iter (int) – maximum number of iterations.
  • regul (float) – regularization trade-off coefficient for the L2-norm of the parameter vector.
  • regul_decay (float) – decay parameter for regul.
  • ls_regul (float) – regularization trade-off coefficient for the L2-norm of the parameter vector update.
  • ls_step_max (float) – maximum L-infinite norm of the parameter vector update; if ls_step_max_decay is defined, ls_step_max can be an (initial, final) tuple instead.
  • ls_step_max_decay (float) – decay parameter for ls_step_max.
  • ls_iter_max (int) – maximum number of linesearch iterations.
  • ls_armijo_max (float) – maximum expected decrease in the local objective; throttles the Armijo threshold.
  • ls_wolfe (tuple) – (c2, c3) pair for wolfe_line_search().
  • ls_failure_rate (float) – epoch-wise rate of linesearch failure; at the end of an epoch, if this rate has been reached, then the iteration stops.
  • fix_ls (callable) – takes a component index (int) and the parameter vector (numpy.ndarray) to be modified inplace in the case of recurrent linesearch failures for this component.
  • fix_ls_trigger (int) – minimum number of iterations with successive linesearch failures on a given component for fix_ls to be triggered.
  • independent_components (bool) – whether to represent the local inverse Hessian submatrix for a component independently of the other components.
  • newton (bool) – quasi-Newton (BFGS) mode; if False, use vanilla gradient descent instead.
  • verbose (bool or logging.Logger) – verbose mode or logger.
Returns:

final parameter vector.

Return type:

BFGSResult

See also SparseFunction and wolfe_line_search().

tramway.inference.optimization.minimize_sparse_bfgs1(fun, x0, component, covariate, gradient_subspace, descent_subspace, args=(), bounds=None, _sum=<function sum>, gradient_sum=None, gradient_covariate=None, memory=10, eps=1e-06, ftol=1e-06, gtol=1e-10, low_df_rate=0.9, low_dg_rate=0.9, step_scale=1.0, max_iter=None, regul=None, regul_decay=1e-05, ls_kwargs={}, ls_regul=None, ls_step_max=None, ls_step_max_decay=None, ls_iter_max=None, ls_armijo_max=None, ls_wolfe=None, ls_failure_rate=0.9, fix_ls=None, fix_ls_trigger=5, gradient_initial_step=1e-08, Component=<class 'tramway.inference.optimization.Component'>, independent_components=False, newton=True, verbose=False, diagnosis=None, returns=(), max_runtime=None, update_timeout=None, **kwargs)

Let the objective function \(f(x) = \sum_{i \in C} f_{i}(x) \forall x \in \Theta\) be a linear function of sparse components \(f_{i}\) such that \(\forall j \notin G_{i}, \forall x \in \Theta, {\partial f_{i}}{\partial x_{j}}(x) = 0\).

Let the components also covary sparsely: \(\forall i \in C, \exists C_{i} \subset C | i \in C_{i}, \exists D_{i} \subset G_{i}, \forall j \in D_{i}, \forall x \in \Theta, \frac{\\partial f}{\partial x_{j}}(x) = \sum_{i' \in C_{i}} \frac{\partial f_{i'}}{\partial x_{j}}(x)\).

We may additionally need that \(\forall i \in C, D_{i} = \bigcap_{i' \in C_{i}} G_{i'}\), \(\bigcup_{i \in C} D_{i} = J\) and \(D_{i} \cap D_{j} = \emptyset \forall i, j \in C^{2}\) with \(J\) the indices of parameter vector \(x = \lvert x_{j}\rvert_{j \in J}\) (to be checked).

At iteration \(k\), let choose component \(i\) and minimize \(f\) wrt parameters \(\{x_{j} | j \in D_{i}\}\).

Compute gradient \(g_{i}(x) = \lvert\frac{\partial f}{\partial x_{j}}(x)\rvert_{j} = \sum_{i' \in C_{i}} \lvert\frac{\partial f_{i'}}{\partial x_{j}}(x) \rvert_{j}\) with (again) \(g_{i}(x)\rvert_{j} = 0 \forall j \notin G_{i}\).

Perform a Wolfe line search along descent direction restricted to subspace \(D_{i}\). Update \(x\) and compute the gradient again.

The inverse Hessian matrix must also be updated twice: before and after the update.

Parameters:
  • fun (callable) – takes a component index (int) and the parameter vector (numpy.ndarray) and returns a scalar float.
  • x0 (numpy.ndarray) – initial parameter vector.
  • component (callable) – takes an iteration index (int) and returns a component index (int); all the components are expected to be drawn exactly once per epoch; if component is an int, will be interpreted as the number of components.
  • covariate (callable) – takes a component index i (int) and returns a sequence of indices of the components that covary with i, including i.
  • gradient_subspace (callable) – takes a component index (int) and returns a sequence of indices of the parameters spanning the gradient subspace; if None, the gradient ‘subspace’ is the full space.
  • descent_subspace (callable) – takes a component index (int) and returns a sequence of indices of the parameters spanning the descent subspace; if None, the descent subspace equals to the gradient subspace.
  • args (tuple or list) – sequence of positional arguments to fun.
  • bounds (tuple or list) – sequence of pairs of (lower, upper) bounds on the parameters.
  • _sum (callable) – takes a list of float values and returns a float.
  • gradient_sum (callable) – replacement for _sum in the calculation of the gradient.
  • gradient_covariate (callable) – takes a parameter index (int) and returns a sequence of the components affected by this parameter.
  • memory (int) – number of memorized pairs of H updates in quasi-Newton mode.
  • eps (float) – initial scaling of the descent direction.
  • ftol (float) – maximum decrease in the local objective.
  • gtol (float) – maximum decrease in the projected gradient.
  • low_df_rate (float) – epoch-wise rate of low decrease in local objective (below ftol); at the end of an epoch, if this rate has been reached, then the iteration stops.
  • low_dg_rate (float) – epoch-wise rate of low decrease in the projected gradient (below gtol); at the end of an epoch, if this rate has been reached, then the iteration stops.
  • step_scale (float) – reduction factor on the line-searched step.
  • max_iter (int) – maximum number of iterations.
  • regul (float) – regularization trade-off coefficient for the L2-norm of the parameter vector.
  • regul_decay (float) – decay parameter for regul.
  • ls_kwargs (dict) – keyword arguments to wolfe_line_search().
  • ls_regul (float) – regularization trade-off coefficient for the L2-norm of the parameter vector update.
  • ls_step_max (float) – maximum L-infinite norm of the parameter vector update; if ls_step_max_decay is defined, ls_step_max can be an (initial, final) tuple instead.
  • ls_step_max_decay (float) – decay parameter for ls_step_max.
  • ls_iter_max (int) – maximum number of linesearch iterations.
  • ls_armijo_max (float) – maximum expected decrease in the local objective; throttles the Armijo threshold.
  • ls_wolfe (tuple) – (c2, c3) pair for wolfe_line_search().
  • ls_failure_rate (float) – epoch-wise rate of linesearch failure; at the end of an epoch, if this rate has been reached, then the iteration stops.
  • fix_ls (callable) – takes a component index (int) and the parameter vector (numpy.ndarray) to be modified inplace in the case of recurrent linesearch failures for this component.
  • fix_ls_trigger (int) – minimum number of iterations with successive linesearch failures on a given component for fix_ls to be triggered.
  • Component (type) – component constructor.
  • independent_components (bool) – whether to represent the local inverse Hessian submatrix for a component independently of the other components.
  • newton (bool) – quasi-Newton (BFGS) mode; if False, use vanilla gradient descent instead.
  • verbose (bool or logging.Logger) – verbose mode or logger.
  • diagnosis (callable) – function of the iteration number and the current and candidate-new components.
  • returns (sequence of str) – any subset of {‘f’, ‘df’, ‘projg’, ‘err’, ‘ncalls’, ‘diagnosis’} or ‘all’.
  • xref (numpy.ndarray) – reference final parameter vector; if defined, at each iteration, the L2-norm of the difference between this and the current parameter vector is evaluated and returned as attribute err; note that this computation may add quite some overhead.
Returns:

final parameter vector.

Return type:

BFGSResult

See also SparseFunction and wolfe_line_search().

class tramway.inference.optimization.SparseFunction(x, covariate, gradient_subspace, descent_subspace, eps, fun, _sum, args, regul, bounds, h0)

Bases: tramway.core.parallel.Workspace

Parameter singleton.

x

working copy of the parameter vector.

Type:numpy.ndarray
covariate

returns the components that covary with the input component (all indices).

Type:callable
gradient_subspace

takes a component index and returns the indices of the related parameters.

Type:callable
descent_subspace

takes a component index and returns the indices of the related parameters.

Type:callable
eps

initial scaling of the descent direction when no curvature information is available.

Type:float
fun

local cost function (takes a component index before the parameter vector).

Type:callable
_sum

mixture function for local costs.

Type:callable
args

positional input arguments to fun.

Type:sequence
regul

regularization coefficient on the parameters.

Type:float
bounds

(lower, upper) bounds as a couple of lists.

Type:tuple
h0

gradient initial step.

Type:float
ncalls

number of calls to fun.

Type:int

See also minimize_sparse_bfgs().

fun(*args, **kwargs)
update(component)
x

Wolfe line search along direction p possibly restricted to a subspace.

If defined, argument subspace is passed to gradient function g as last positional argument, unless already present in args_g.

args is an alias for args_f. If args_g is None, then it falls back to args_f.

f(x_k + eta_k p_k) <= f(x_k) + c1 eta_k p_k^T g(x_k) # Armijo rule
c2 * p_k^T g(x_k) <= -p_k^T g(x_k + eta_k p_k) <= -c3 p_k^T g(x_k) # weak (c2=None) or strong (c2=c3) Wolfe condition
c4 eta_k <= eta_{k+1} <= c0 eta_k
eta_min = c5 eta_max
tramway.inference.optimization.sparse_grad(fun, x, active_i, active_j, args=(), _sum=<function sum>, regul=None, bounds=None, h0=1e-08)

Compute the derivative of a function.

tramway.inference.stochastic_dv plugin

class tramway.inference.stochastic_dv.LocalDV(diffusivity, potential, diffusivity_spatial_prior=None, potential_spatial_prior=None, diffusivity_time_prior=None, potential_time_prior=None, minimum_diffusivity=None, positive_diffusivity=None, prior_include=None, regions=None, prior_delay=None, logger=True)

Bases: tramway.inference.dv.DV

diffusivity_indices(cell_ids)
diffusivity_prior(i)
diffusivity_spatial_prior(i)
diffusivity_time_prior(i)
indices(cell_ids)
logger
pop_workspace_update()
potential_indices(cell_ids)
potential_prior(i)
potential_spatial_prior(i)
potential_time_prior(i)
prior_delay
push_workspace_update(update)
region(i)
regions
undefined_grad(i, feature='')
undefined_time_derivative(i, feature='')
verbose
tramway.inference.stochastic_dv.infer_stochastic_DV(cells, diffusivity_spatial_prior=None, potential_spatial_prior=None, diffusivity_time_prior=None, potential_time_prior=None, jeffreys_prior=False, min_diffusivity=None, max_iter=None, compatibility=False, export_centers=False, verbose=True, superlocal=True, stochastic=True, D0=None, V0=None, x0=None, rgrad=None, debug=False, fulltime=False, diffusion_prior=None, diffusion_spatial_prior=None, diffusion_time_prior=None, prior_delay=None, return_struct=False, posterior_max_count=None, diffusivity_prior=None, potential_prior=None, time_prior=None, allow_negative_potential=False, **kwargs)
Parameters:
  • ..
  • time_prior (float or tuple) – one of two regularization coefficients that penalize the temporal derivative of diffusivity and potential energy; the first coefficient applies to diffusivity (and is multiplied by diffusivity_prior) and the second applies to potential energy (and is multiplied by potential_prior).
  • diffusivity_spatial_prior/diffusion_spatial_prior – alias for diffusivity_prior.
  • diffusivity_time_prior/diffusion_time_prior – penalizes the temporal derivative of diffusivity; this coefficient does NOT multiply with diffusivity_prior.
  • potential_spatial_prior – alias for potential_prior.
  • potential_time_prior – penalizes the temporal derivative of potential; this coefficient does NOT multiply with potential_prior.
  • stochastic (bool) – if False, emulate standard DV with sparse gradient.
  • D0 (float or ndarray) – initial diffusivity value(s).
  • V0 (float or ndarray) – initial potential energy value(s).
  • x0 (ndarray) – initial parameter vector [diffusivities, potentials].
  • rgrad (str) – either ‘grad’/’grad1’, ‘gradn’ (none recommended), ‘delta’/’delta0’ or ‘delta1’; see the corresponding functions in module gradient.
  • debug (bool or sequence of str) – any subset of {‘ncalls’, ‘f_history’, ‘df_history’, ‘projg_history’, ‘error’, ‘diagnoses’}.
  • xref (ndarray) – reference parameter vector [diffusivities, potentials]; required to compute the ‘error’ debug variable.
  • diagnosis (callable) – function of the iteration number, current component and candidate updated component; required to compute the ‘diagnoses’ debug variable.
  • prior_delay/return_struct/posterior_max_count – all deprecated.
  • allow_negative_potential (bool) – do not offset the whole potential map towards all-positive values.
  • ..

See also minimize_sparse_bfgs().

tramway.inference.stochastic_dv.local_dv_neg_posterior(j, x, dv, cells, sigma2, jeffreys_prior, dt_mean, index, reverse_index, grad_kwargs, posterior_info=None, iter_num=None, verbose=False)
tramway.inference.stochastic_dv.lookup_space_cells(cells)
tramway.inference.stochastic_dv.make_regions(cells, index, reverse_index, size=1)

tramway.inference.bayes_factors subpackage

tramway.inference.bayes_factors.calculate_bayes_factors(zeta_ts, zeta_sps, ns, Vs, Vs_pi, loc_error, dim=2, B_threshold=10, verbose=True)

Calculate the Bayes factor for a set of bins given a uniform localization error.

Input: zeta_ts — signal-to-noise ratios for the total force = dx_mean / sqrt(var(dx)) in bins. Size: M x 2, zeta_sps — signal-to-noise ratios for the spurious force = grad(D) / sqrt(var(dx)) in bins. Size: M x 2, ns — number of jumps in each bin. Size: M x 1, Vs — jump variance in bins = E((dx - dx_mean) ** 2). Size: M x 1, Vs_pi — jump variance in all other bins relative to the current bin. Size: M x 1, loc_error — localization error. Same units as variance. Set to 0 if localization error can be ignored; dim — dimensionality of the problem; B_threshold — the values of Bayes factor for thresholding.

Output: Bs, forces, min_ns

lg_Bs — log_10 Bayes factor values in the bins. Size: M x 1, forces — Returns 1 if there is strong evidence for the presence of a conservative forces, -1 for strong evidence for a spurious force, and 0 if the is not enough evidence. Size: M x 1; min_ns — minimum number of data points in a bin to support the currently favored model at the required evidence level. Size: M x 1.

Notation: M — number of bins.

tramway.inference.bayes_factors.calculate_bayes_factors_for_one_cell(cell, loc_error, dim=2, B_threshold=10, verbose=True)

Calculate Bayes factors for one cell. The results are saved to cell properties.

tramway.inference.bayes_factors.calculate_bayes_factors module

exception tramway.inference.bayes_factors.calculate_bayes_factors.NaNInputError

Bases: ValueError

tramway.inference.bayes_factors.calculate_bayes_factors.calculate_bayes_factors(zeta_ts, zeta_sps, ns, Vs, Vs_pi, loc_error, dim=2, B_threshold=10, verbose=True)

Calculate the Bayes factor for a set of bins given a uniform localization error.

Input: zeta_ts — signal-to-noise ratios for the total force = dx_mean / sqrt(var(dx)) in bins. Size: M x 2, zeta_sps — signal-to-noise ratios for the spurious force = grad(D) / sqrt(var(dx)) in bins. Size: M x 2, ns — number of jumps in each bin. Size: M x 1, Vs — jump variance in bins = E((dx - dx_mean) ** 2). Size: M x 1, Vs_pi — jump variance in all other bins relative to the current bin. Size: M x 1, loc_error — localization error. Same units as variance. Set to 0 if localization error can be ignored; dim — dimensionality of the problem; B_threshold — the values of Bayes factor for thresholding.

Output: Bs, forces, min_ns

lg_Bs — log_10 Bayes factor values in the bins. Size: M x 1, forces — Returns 1 if there is strong evidence for the presence of a conservative forces, -1 for strong evidence for a spurious force, and 0 if the is not enough evidence. Size: M x 1; min_ns — minimum number of data points in a bin to support the currently favored model at the required evidence level. Size: M x 1.

Notation: M — number of bins.

tramway.inference.bayes_factors.calculate_bayes_factors.calculate_bayes_factors_for_one_cell(cell, loc_error, dim=2, B_threshold=10, verbose=True)

Calculate Bayes factors for one cell. The results are saved to cell properties.

tramway.inference.bayes_factors.calculate_bayes_factors.calculate_minimal_n(zeta_t, zeta_sp, n0, V, V_pi, loc_error, dim=2, B_threshold=10)

Calculate the minimal number of jumps per bin needed to obtain strong evidence for the active force or the spurious force model.

Input: zeta_t, zeta_sp - – vectors of length 2 for one bin n0 - – initial number of jumps(evidence already available). This way the “next” strong evidence can be found, i.e. the minimal number of data points to support the current conclusion

Output: min_n - – minimal number of jumps to obtain strong evidence for the conservative force model. Return - 1 if unable to find the min_n

tramway.inference.bayes_factors.calculate_bayes_factors.check_dimensionality(dim)
tramway.inference.bayes_factors.calculate_bayes_factors.check_for_nan(*args)

Return ‘nan’ if a nan value is present in any of the variables. Return ‘None’ if any of the variables is None. Return ‘ok’ otherwise

tramway.inference.bayes_factors.calculate_marginalized_integral module

tramway.inference.bayes_factors.calculate_marginalized_integral.calculate_any_lambda_integral(func, break_points=[])

DEPRECATED: calculate ratios instead. This function allows to calculate more complicated lambda integrals, such as 1D posteriors in 2D.

Input: func(lambda) - function of lambda to integrate, break_points - special integration points (i.e. zeta_t/zeta_sp), optional.

tramway.inference.bayes_factors.calculate_marginalized_integral.calculate_integral_ratio(arg_func_up, arg_func_down, pow_up, pow_down, v, rel_loc_error, break_points=[], lamb='marg', rtol=1e-06, atol=1e-32)

Calculate the ratio of two similar lambda integrals, each of which has the form int_0^1 dlambda arg_func(lambda)**(-pow) * gammainc(pow, rel_loc_error * arg_func(lambda))

Input: v — zeta-independent term inside the argument functions. Must be the same upstairs and downstairs. In Bayes factor calculations, this is v := (1 + n_pi * V_pi / n / V).

Return: Natural logarithm of the integral ratio

Details: Calculations are performed differently for the case with and without localization error. This avoids overflow errors. In these cases, both the integrated equations and prefactors are different. The integrals are taken over lambda if lamb == ‘marg’, otherwise they are evaluated at the given lambda.

tramway.inference.bayes_factors.calculate_marginalized_integral.calculate_marginalized_integral(zeta_t, zeta_sp, p, v, E, rel_loc_error, zeta_a=[0, 0], factor_za=0, lamb='int')

Calculate the marginalized lambda integral >>> Integrate[gamma_inc[p, arg * rel_loc_error] * arg ** (-p), {lambda, 0, 1}] >>> for the given values of zeta_t and zeta_sp.

Here: arg = (v + factor_za * zeta_a**2 + E * (zeta_t - zeta_a - lambda * zeta_sp)**2); IMPORTANT: gamma_inc — is the normalized lower incomplete gamma function; rel_loc_error = n * V / (4 * sigma_L^2) — inverse relative localization error,

Input: zeta_t and zeta_sp are vectors of length dim (x, y, z). All other parameters are scalars. abs_tol — absolute tolerance for integral calculations. lamb = float in [0, 1] or ‘int’. If a float is given, the integrated function is just evaluated at the given value of lambda, without integrating.

tramway.inference.bayes_factors.calculate_marginalized_integral.sum_series(term_func, rtol, *args)

Sum any series with the first argument of the term_func being the term number. Input: function, return - float

tramway.inference.bayes_factors.calculate_posteriors module

The file contains posterior calculation functions.

tramway.inference.bayes_factors.calculate_posteriors.calculate_one_1D_posterior_in_2D(zeta_a, zeta_t, zeta_sp, n, V, V_pi, loc_error, lamb='marg', axis='x')

Calculate the posterior for a projection of zeta_a for a given inference convention for one bin in 2D

Input: lambda: float in [0, 1] or ‘marg’; for different conventions axis: [‘x’, ‘y’]; for which projection to calculate the posterior zeta_a: float; a projection on the selected axis

tramway.inference.bayes_factors.calculate_posteriors.calculate_one_1D_prior_in_2D(zeta_a, V_pi, sigma2)

Calculate the prior for a projection of zeta_a for any convention for one bin in 2D

Input: zeta_a: float; a projection on the selected axis

tramway.inference.bayes_factors.calculate_posteriors.calculate_one_2D_posterior(zeta_t, zeta_sp, zeta_a, n, V, V_pi, loc_error, dim=2, lamb='marg')

Calculate the posterior for zeta_a for a given method for one bin in 2D

Input: lambda — float in [0, 1] or ‘marg’ — for different conventions

tramway.inference.bayes_factors.calculate_posteriors.get_lambda_MAP(zeta_t, zeta_sp)

Calculate the maximum a posteriori value of lambda for given zetas in 1D and 2D

tramway.inference.bayes_factors.convenience_functions module

Small functions used throughout the package

tramway.inference.bayes_factors.convenience_functions.n_pi_func(dim)
tramway.inference.bayes_factors.convenience_functions.p(n, dim)

tramway.inference.bayes_factors.find_marginalized_zeta_t_roots module

tramway.inference.bayes_factors.find_marginalized_zeta_t_roots.find_marginalized_zeta_t_roots(zeta_sp_par, n, n_pi, B, u, dim, zeta_t_perp, sigma2)

Find marginalized roots zeta_t. I have proven that the min B for the marginalized inference is achieved at zeta_t = zeta_sp/2 (for the parallel component and under the condition that the orthogonal component is 0). sigma2 — localization error. Same units as variance. Set to 0 if localization error can be ignored;

tramway.inference.bayes_factors.get_D_posterior module

Functions allowing to calculate the D posterior and MAP(D) as (will be) described in the Ito-Stratonovich article. Provides 3 functions: - get_D_posterior - get_MAP_D - get_D_confidence_interval

If unsure, use get_D_confidence_interval, which provides MAP(D) and a confidence interval for D for the given confidence level. If need the D posterior, use get_D_posterior.

tramway.inference.bayes_factors.get_D_posterior.get_D_confidence_interval(alpha, n, zeta_t, V, V_pi, dt, sigma2, dim)

Returns MAP_D and a confidence interval corresponding to confidence level alpha, i.e. the values of D giving the values [(1-alpha)/2, 1-(1-alpha)/2] for the D posterior integrals

Input: alpha - confidence level, n - number of jumps in bin, zeta_t - signal-to-noise ratio for the total force: <dr>/sqrt(V), V - (biased) variance of jumps in the current bin, V_pi - (biased) variance of jumps in all other bins excluding the current one, dt - time step sigma2 - localization error (in the units of variance), dim - dimensionality of the problem

Output: MAP_D - MAP value of the diffusivity, CI - a confidence interval for the diffusivity, 2x1 np.array

tramway.inference.bayes_factors.get_D_posterior.get_D_posterior(n, zeta_t, V, V_pi, dt, sigma2, dim, _MAP_D=False)

Return the diffusivity posterior as a function. If _MAP_D flag is supplied, returns instead a single MAP(D) value

Input: V - (biased) variance of jumps in the current bin V_pi - (biased) variance of jumps in all other bins excluding the current one dt - time step sigma2 - localization error (in the units of variance) dim - dimensionality of the problem _MAP_D - if True, returns MAP_D instead of the posterior

Output: posterior - function object accepting D as the only argument

tramway.inference.bayes_factors.get_D_posterior.get_MAP_D(n, zeta_t, V, V_pi, dt, sigma2, dim)