natcap.invest.utils

InVEST specific code utils.

class natcap.invest.utils.ThreadFilter(thread_name)

Bases: Filter

Filters out log messages issued by the given thread.

Any log messages generated by a thread with the name matching the threadname provided to the constructor will be excluded.

filter(record)

Filter the given log record.

Parameters:

record (log record) – The log record to filter.

Returns:

True if the record should be included, false if not.

natcap.invest.utils.base_model_id(model_id: str) → str

Strip version info from an annotated model ID.

Useful for translating from a key in the Workbench’s settingsStore.plugins to a key in natcap.invest.models.model_id_to_pyname.

Returns the substring of model_id from position 0 up to, but not including, the @. If model_id contains no @, model_id is returned unchanged.

natcap.invest.utils.capture_gdal_logging()

Context manager for logging GDAL errors with python logging.

While we are now using gdal.UseExceptions(), we still need this to handle GDAL logging that does not get raised as an exception. GDAL error messages are logged via python’s logging system, at a severity that corresponds to a log level in logging. Error messages are logged with the osgeo.gdal logger.

Parameters:

None

Returns:

None

natcap.invest.utils.copy_spatial_files(spatial_filepath, target_dir)

Copy spatial files to a new directory.

Parameters:

  • spatial_filepath (str) – The filepath to a GDAL-supported file.

  • target_dir (str) – The directory where all component files of spatial_filepath should be copied. If this directory does not exist, it will be created.

Returns:

The path to a representative file copied into the target_dir. If possible, this will match the basename of spatial_filepath, so if someone provides an ESRI Shapefile called my_vector.shp, the return value will be os.path.join(target_dir, my_vector.shp).

Return type:

filepath (str)

natcap.invest.utils.create_coordinate_transformer(base_ref, target_ref, osr_axis_mapping_strategy=0)

Create a spatial reference coordinate transformation function.

Parameters:

  • base_ref (osr spatial reference) – A defined spatial reference to transform FROM

  • target_ref (osr spatial reference) – A defined spatial reference to transform TO

  • osr_axis_mapping_strategy (int) – OSR axis mapping strategy for SpatialReference objects. Defaults to utils.DEFAULT_OSR_AXIS_MAPPING_STRATEGY. This parameter should not be changed unless you know what you are doing.

Returns:

An OSR Coordinate Transformation object

natcap.invest.utils.evaluate_expression(expression, variable_map)

Evaluate a python expression.

The expression must be able to be evaluated as a python expression.

Parameters:

  • expression (string) – A string expression that returns a value.

  • variable_map (dict) – A dict mapping string variable names to their python object values. This is the variable map that will be used when evaluating the expression.

Returns:

Whatever value is returned from evaluating expression with the variables stored in variable_map.

natcap.invest.utils.expand_path(path, base_path)

Check if a path is relative, and if so, expand it using the base path.

Parameters:

  • path (string) – path to check and expand if necessary

  • base_path (string) – path to expand the first path relative to

Returns:

path as an absolute path

natcap.invest.utils.format_args_dict(args_dict, model_id)

Nicely format an arguments dictionary for writing to a stream.

If printed to a console, the returned string will be aligned in two columns representing each key and value in the arg dict. Keys are in ascending, sorted order. Both columns are left-aligned.

Parameters:

  • args_dict (dict) – The args dictionary to format.

  • model_id (string) – The model ID (e.g. carbon)

Returns:

A formatted, unicode string.

natcap.invest.utils.has_utf8_bom(textfile_path)

Determine if the text file has a UTF-8 byte-order marker.

Parameters:

textfile_path (str) – The path to a file on disk.

Returns:

A bool indicating whether the textfile has a BOM. If True, a BOM is present.

natcap.invest.utils.log_to_file(logfile, exclude_threads=None, logging_level=0, log_fmt='%(asctime)s (%(name)s) %(module)s.%(funcName)s(%(lineno)d) %(levelname)s %(message)s', date_fmt=None)

Log all messages within this context to a file.

Parameters:

  • logfile (string) – The path to where the logfile will be written. If there is already a file at this location, it will be overwritten.

  • exclude_threads=None (list) – If None, logging from all threads will be included in the log. If a list, it must be a list of string thread names that should be excluded from logging in this file.

  • logging_level=logging.NOTSET (int) – The logging threshold. Log messages with a level less than this will be automatically excluded from the logfile. The default value (logging.NOTSET) will cause all logging to be captured.

  • log_fmt=LOG_FMT (string) – The logging format string to use. If not provided, utils.LOG_FMT will be used.

  • date_fmt (string) – The logging date format string to use. If not provided, ISO8601 format will be used.

Yields:

handler –

An instance of logging.FileHandler that

represents the file that is being written to.

Returns:

None

natcap.invest.utils.matches_format_string(test_string, format_string)

Assert that a given string matches a given format string.

This means that the given test string could be derived from the given format string by replacing replacement fields with any text. For example, the string ‘Value “foo” is invalid.’ matches the format string ‘Value “{value}” is invalid.’

Parameters:

  • test_string (str) – string to test.

  • format_string (str) – format string, which may contain curly-brace delimited replacement fields

Returns:

True if test_string matches format_string, False if not.

natcap.invest.utils.mean_pixel_size_and_area(pixel_size_tuple)

Convert to mean and raise Exception if they are not close.

Parameter:
pixel_size_tuple (tuple): a 2 tuple indicating the x/y size of a

pixel.

Returns:

tuple of (mean absolute average of pixel_size, area of pixel size)

Raises:

ValueError if the dimensions of pixel_size_tuple are not almost – square.

natcap.invest.utils.prepare_workspace(workspace, model_id, logging_level=0, exclude_threads=None)

Prepare the workspace.

natcap.invest.utils.read_csv_to_dataframe(path, **kwargs)

Return a dataframe representation of the CSV.

Wrapper around pandas.read_csv that performs some common data cleaning. Column names are lowercased and whitespace is stripped off. Empty rows and columns are dropped. Sets custom defaults for some kwargs passed to pandas.read_csv, which you can override with kwargs:

  • sep=None: lets the Python engine infer the separator

  • engine=’python’: The ‘python’ engine supports the sep=None option.

  • encoding=’utf-8-sig’: ‘utf-8-sig’ handles UTF-8 with or without BOM.

  • index_col=False: force pandas not to index by any column, useful in

    case of trailing separators

Parameters:

  • path (str) – path to a CSV file

  • **kwargs – additional kwargs will be passed to pandas.read_csv

Returns:

pandas.DataFrame with the contents of the given CSV

natcap.invest.utils.reclassify_raster(raster_path_band, value_map, target_raster_path, target_datatype, target_nodata, error_details)

A wrapper function for calling pygeoprocessing.reclassify_raster.

This wrapper function is helpful when added as a TaskGraph.task so a better error message can be provided to the users if a pygeoprocessing.ReclassificationMissingValuesError is raised.

Parameters:

  • raster_path_band (tuple) – a tuple including file path to a raster and the band index to operate over. ex: (path, band_index)

  • value_map (dictionary) – a dictionary of values of {source_value: dest_value, …} where source_value’s type is the same as the values in base_raster_path at band band_index. Must contain at least one value.

  • target_raster_path (string) – target raster output path; overwritten if it exists

  • target_datatype (gdal type) – the numerical type for the target raster

  • target_nodata (numerical type) – the nodata value for the target raster Must be the same type as target_datatype

  • error_details (dict) –

    a dictionary with key value pairs that provide more context for a raised pygeoprocessing.ReclassificationMissingValuesError. keys must be {‘raster_name’, ‘column_name’, ‘table_name’}. Values each key represent:

    ’raster_name’ - string for the raster name being reclassified ‘column_name’ - name of the table column that value_map dictionary keys came from. ‘table_name’ - table name that value_map came from.

Returns:

None

Raises:

  • ValueError –

    • if values_required is True and a pixel value from raster_path_band is not a key in value_map.

  • TypeError –

    • if there is a None or NA key in value_map.

natcap.invest.utils.resample_population_raster(source_population_raster_path, target_population_raster_path, target_pixel_size, target_bb, target_projection_wkt, working_dir)

Resample a population raster without losing or gaining people.

Population rasters are an interesting special case where the data are neither continuous nor categorical, and the total population count typically matters. Common resampling methods for continuous (interpolation) and categorical (nearest-neighbor) datasets leave room for the total population of a resampled raster to significantly change. This function resamples a population raster with the following steps:

  1. Convert a population count raster to population density per pixel

  2. Warp the population density raster to the target spatial reference and pixel size using bilinear interpolation.

  3. Convert the warped density raster back to population counts.

Parameters:

  • source_population_raster_path (string) – The source population raster. Pixel values represent the number of people occupying the pixel. Must be linearly projected in meters.

  • target_population_raster_path (string) – The path to where the target, warped population raster will live on disk.

  • target_pixel_size (tuple) – A tuple of the pixel size for the target raster. Passed directly to pygeoprocessing.warp_raster.

  • target_bb (tuple) – A tuple of the bounding box for the target raster. Passed directly to pygeoprocessing.warp_raster.

  • target_projection_wkt (string) – The Well-Known Text of the target spatial reference for the target raster. Passed directly to pygeoprocessing.warp_raster. Assumed to be a linear projection in meters.

  • working_dir (string) – The path to a directory on disk. If the directory does not already exist, it will be created. A temporary subdirectory will be generated within this location to store temporary files, which is automatically removed upon successful completion of the function.

Returns:

None