hf_hydrodata.gridded module

Functions to access gridded data from the data catalog index of the GPFS files.

hf_hydrodata.gridded.get_date_range(*args, **kwargs) → Tuple[datetime, datetime]

Get the date range of the dataset specified by the options.

The parameters to the function can be specified either by passing a dict with the parameter values or by passing named parameters to the function. You can pass any parameters used by get_gridded_data() or get_data_catalog_entry(), but only the dataset option is used.

Parameters:

dataset -- A dataset name (see Gridded Data documentation).

Returns:

A tuple with (dataset_start_date, dataset_end_date) or None if no date range is available.

Example:

import hf_hydrodata as hf

options = {"dataset": "NLDAS2", "temporal_resolution": "daily", "variable": "precipitation",
           "date_start":"2005-09-30", "date_end":"2005-10-03",
           "grid_bounds":[200, 200, 300, 250]
}
range = hf.get_date_range(options)
assert range[0] == datetime.datetime(2002, 10, 1)
assert range[1] == datetime.datetime(2006, 9, 30)
hf_hydrodata.gridded.get_file_path(entry, *args, **kwargs) → str

This function is deprecated.

Use the function get_path() instead.

hf_hydrodata.gridded.get_file_paths(entry, *args, **kwargs) → List[str]

This function is deprecated.

Use the function get_paths() instead.

hf_hydrodata.gridded.get_gridded_data(*args, **kwargs) → ndarray

Get a numpy ndarray from files in /hydroframe. with the applied data filters.

The parameters to the function can be specified either by passing a dict with the parameter values or by passing named parameters to the function.

Parameters:

  • dataset -- A dataset name (see Gridded Data documentation).

  • variable -- A variable from a dataset.

  • temporal_resolution -- Time resolution of a the dataset variable. Must be hourly, daily, weekly, monthly.

  • grid -- A grid supported by a dataset (e.g. conus1 or conus2). Normally this is determined by the dataset.

  • aggregation -- One of mean, max, min. Normally, only needed for temperature variables.

  • date_start -- A time as either a datetime object or a string in the form YYYY-MM-DD. Start of the date range for data (start_time is also accepted for backward compatibility).

  • date_end -- A time as either a datetime object or a string in the form YYYY-MM-DD. End of the date range for data (end_time is also accepted for backward compatibility).

  • grid_bounds -- An array (or string representing an array) of points [left, bottom, right, top] in xy grid corridates in the grid of the data.

  • latlng_bounds -- An array (or string representing an array) of points [left, bottom, right, top] in lat/lng coordinates mapped with the grid of the data.

  • grid_point -- An array (or string representing an array) of points [x, y] in grid corridates of a point in the grid.

  • latlng_point -- An array (or string representing an array) of points [lat, lon] in lat/lng coordinates of a point in the grid.

  • huc_id -- A comma seperated list of HUC id that specifies the grid_bounds using the HUC bounding box.

  • z -- A value of the z dimension to be used as a filter for this dismension when loading data.

  • level -- A HUC level integer when reading HUC boundary files. Must be 2, 4, 6, 8, or 10.

  • site_id -- Used when reading data associated with an observation site.

  • data_catalog_entry_id -- Optional. The id of an entry in the data catalog to identify an entry.

Returns:

A numpy ndarray containing the data loaded from the files identified by the entry and sliced by the data filter options.

Raises:

  • ValueError -- If both grid_bounds and latlng_bounds are specified as data filters.

  • ValueError -- If no data catalog entry is found associated with the filter parameters.

  • ValueError -- If any filter parameters are invalid.

For gridded results the returned numpy array has dimensions:

  • [hour, y, x] temporal_resolution is hourly without z dimension

  • [day, y, x] temporal_resolution is daily without z dimension

  • [month, y, x] temporal_resolution is monthly without z dimension

  • [y, x] temporal_resolution is static or blank without z dimension

  • [hour, z, y, x] temporal_resolution is hourly with z dimension

  • [day, z, y, x] temporal_resolution is daily with z dimension

  • [month, z, y, x] temporal_resolution is monthly with z dimension

  • [z, y, x] temporal_resolution is static or blank with z dimension

If the dataset has ensembles then there is an ensemble dimension at the beginning.

Both start_time and end_time must be in the form "YYYY-MM-DD HH:MM:SS" or "YYYY-MM-DD" or a datetime object.

If only start_time is specified than only that month/day/hour is returned. The start_time is inclusive the end_time is exclusive (data returned less than that time).

If either grid_bounds or latlng_bounds is specified then the result is sliced by the x,y values in the bounds. If grid_point or latlon_point is specified this is mapped to a grid_bounds of size 1x1 at that point.

If z is specified then the result is sliced by the z dimension.

For example, to get data from the 3 daily files bewteen 9/30/2005 and 10/3/2005.

Example:

import hf_hydrodata as hf

options = {
    "dataset": "NLDAS2", "temporal_resolution": "daily", "variable": "precipitation",
    "date_start":"2005-09-30", "date_end":"2005-10-03",
    "grid_bounds":[200, 200, 300, 250]
}
# The result has 3 days in the time dimension
# The result is sliced to x,y size 100x50 in the conus1 grid.
data = hf.get_gridded_data(options)
assert data.shape == (3, 50, 100)

metadata = dc.get_catalog_entry(options)
hf_hydrodata.gridded.get_gridded_file(file_path: str, options: dict)

Query the hf_hydrodata data_catalog for data matching the options and download the data into file_path. This function allows you to perform the same query as get_gridded_data(), but write it to a file with projection information instead of just returning a numpy array.

Note - this function is new as of hf_hydrodata version 1.4.7. This function replaces the old get_gridded_files() function.

Parameters:

  • file_path -- A file path to write the data downloaded using the options.

  • options -- A dict with the same options allowed by the get_gridded_data() function.

Raises:

ValueError -- If an error occurs while downloading and creating files.

The file_path must have an extension of either ".tiff", ".tif", ".nc" or ".pfb". The format of the file written depends on the extension. Format: "TIFF", "NetCDF" or "parflow binary pfb" file.

The file will contain projection information and x, y, time coordinate information to allow the file to be viewed by ARCGIS or QGIS if this is possible (pfb files do not hold projection info). Tiff files are only written with 2D data so data with multiple time dimensions will not be stored. You can write a file with a time dimension to a tiff as long as you only select one time in the query.

The file_path may optionally contain one or more of the substitution options listed below. If this makes the file_path different for the time in the downloaded data then multiple files may be be written.

Substitutable parameters available for file_path:

  • dataset: The dataset name from the options.

  • variable: The variable being downloaded from options or the variables list.

  • dataset_var: The data catalog entry dataset_var.

  • hour_start: The starting hour of the data in the saved file (starting at 0).

  • hour_end: The ending hour of the data in the saved file.

  • daynum: The day number of the data in the saved file (starting at 0).

  • wy: The water year of the data in the saved file.

  • wy_daynum: The day number of the water year.

  • wy_start_24hr: The 24-hour start hour of the water year in the file.

  • wy_end_24hr: The 24-hour end hour of the water year in the file.

  • mdy: The date as month-day-year of data in the saved file.

  • ymd: The date as year-month-day of data in the saved file.

The example below will create a .tiff file with a subset of the 30m resolution WTD data.

import hf_hydrodata as hf

options = {"dataset": "ma_2025", "variable": "water_table_depth",
           "grid": "conus2_wtd.30", "huc_id": "14010002"}
hf.get_gridded_file("wtd.tiff", options)

The example below will create two NetCDF files, one for each water year in the query because of the {wy} in the file_path.

import hf_hydrodata as hf

options = {
    "dataset": "NLDAS2", "temporal_resolution": "hourly", "variable": "precipitation",
    "date_start":"2005-09-01", "date_end":"2005-11-01",
    "grid_bounds":[200, 200, 300, 250]
}
hf.get_gridded_file("{dataset}_{variable}_{wy}.nc", options)

The example below will create 3 .pfb files in two water year directories with hourly data and one file per day. Note the {wy} in the directory part of the file_path and the {wy_start_24hr} in the name part of the file_path.

import hf_hydrodata as hf

options = {
    "dataset": "NLDAS2", "temporal_resolution": "hourly", "variable": "precipitation",
    "date_start":"2009-09-28", "date_end":"2009-10-04",
    "grid_bounds":[200, 200, 300, 250]
}
file_path = "WY{wy}/{dataset}_{variable}.{wy_start_24hr:06d}_to_{wy_end_24hr:06}.pfb"
hf.get_gridded_file(file_path, options)
hf_hydrodata.gridded.get_gridded_files(options: dict, filename_template: str = None, variables=None, verbose=False)

Deprecated. Use the new get_gridded_file() function instead.

hf_hydrodata.gridded.get_huc_bbox(grid: str, huc_id_list: List[str]) → List[int]

Get the grid bounding box containing all the HUC ids.

Parameters:

  • grid -- A grid id from the data catalog (e.g. conus1 or conus2)

  • huc_id_list -- A list of HUC id strings of HUCs in the grid (or comma seperated list of HUC id).

Returns:

A bounding box in grid coordinates as a list of int (i_min, j_min, i_max, j_max)

Raises:

  • ValueError -- if all the HUC id are not at the same level (same length).

  • ValueError -- if grid is not valid or a HUC_VERSION environment variable is not valid.

If the environment variable HUC_VERSION is set this will cause the function to use the HUC boundaries for that dataset_version. The versions 2025_06, 2025_01, 2024_11 are supported as well as blank to get latest.

Example:

import hf_hydrodata as hf

bbox = hf.get_huc_bbox("conus1", ["181001"])
assert bbox == (1, 167, 180, 378)
hf_hydrodata.gridded.get_huc_from_latlon(grid: str, level: int, lat: float, lon: float) → str

Get a HUC id at a lat/lon point for a given grid and level.

Parameters:

  • grid -- grid name (e.g. conus1 or conus2)

  • level -- HUC level (length of HUC id to be returned). Must be 2, 4, 6, 8, or 10.

  • lat -- lattitude of point

  • lon -- longitude of point

Returns:

The HUC id string containing the lat/lon point or None.

Example:

import hf_hydrodata as hf

huc_id = hf.get_huc_from_latlon("conus1", 6, 34.48, -115.63)
assert huc_id == "181001"
hf_hydrodata.gridded.get_huc_from_xy(grid: str, level: int, x: int, y: int) → str

Get a HUC id at an xy point for a given grid and level.

Parameters:

  • grid -- grid name (e.g. conus1 or conus2)

  • level -- HUC level (length of HUC id to be returned). Must be 2, 4, 6, 8, or 10.

  • x -- x coordinate in the grid

  • y -- y coordinate in the grid

Returns:

The HUC id string containing the lat/lon point or None.

Example:

import hf_hydrodata as hf

huc_id = hf.get_huc_from_xy("conus1", 6, 300, 100)
assert huc_id == "181001"
hf_hydrodata.gridded.get_numpy(*args, **kwargs)

Deprecated. Return an error.

hf_hydrodata.gridded.get_path(*args, **kwargs) → str

Get the file path within the data catalog for the specified filter options.

Parameters can be specified either by passing a dict with parameter values or by passing named parameters to the function.

Parameters:

  • dataset (str) -- A dataset name (see Gridded Data documentation).

  • variable (str) -- A variable from a dataset.

  • temporal_resolution (str) -- Time resolution of the dataset variable. Must be hourly, daily, weekly, or monthly.

  • grid (str) -- A grid supported by a dataset (e.g. conus1 or conus2). Normally this is determined by the dataset.

  • aggregation (str) -- One of mean, max, or min. Normally only needed for temperature variables.

  • date_start (str or datetime) -- A time as either a datetime object or a string in YYYY-MM-DD format. Start of the date range for data. (start_time is also accepted for backward compatibility).

  • date_end (str or datetime) -- A time as either a datetime object or a string in YYYY-MM-DD format. End of the date range for data. (end_time is also accepted for backward compatibility).

  • grid_bounds (list) -- An array of points [left, bottom, right, top] in xy grid coordinates.

  • latlng_bounds (list) -- An array of points [lat_min, lon_min, lat_max, lon_max] in lat/lng coordinates.

  • grid_point (list) -- An array of points [x, y] in grid coordinates.

  • latlng_point (list) -- An array of points [lat, lon] in lat/lng coordinates.

  • huc_id (str or list) -- A comma-separated list of HUC ids that specifies the grid_bounds using the HUC bounding box.

  • z (int) -- A value of the z dimension to be used as a filter for this dimension when loading data.

  • level (int) -- A HUC level integer when reading HUC boundary files. Must be 2, 4, 6, 8, or 10.

  • site_id (str) -- Used when reading data associated with an observation site.

Returns:

Absolute path name to the file location on the GPFS file system.

Return type:

str

Raises:

ValueError -- If no data catalog entry is found for the provided filter options.

Example:

import hf_hydrodata as hf

options = {
    "dataset": "NLDAS2", "temporal_resolution": "daily", "variable": "precipitation",
    "date_start":"2005-09-30"
}
path = hf.get_path(options)
hf_hydrodata.gridded.get_paths(*args, **kwargs) → List[str]

Get the file paths within the data catalog for the specified filter options.

Parameters can be specified either by passing a dict with parameter values or by passing named parameters to the function.

Parameters:

  • dataset (str) -- A dataset name (see Gridded Data documentation).

  • variable (str) -- A variable from a dataset.

  • temporal_resolution (str) -- Time resolution of the dataset variable. Must be hourly, daily, weekly, or monthly.

  • grid (str) -- A grid supported by a dataset (e.g. conus1 or conus2). Normally this is determined by the dataset.

  • aggregation (str) -- One of mean, max, or min. Normally only needed for temperature variables.

  • date_start (str or datetime) -- A time as either a datetime object or a string in YYYY-MM-DD format. Start of the date range for data. (start_time is also accepted for backward compatibility).

  • date_end (str or datetime) -- A time as either a datetime object or a string in YYYY-MM-DD format. End of the date range for data. (end_time is also accepted for backward compatibility).

  • grid_bounds (list) -- An array of points [left, bottom, right, top] in xy grid coordinates.

  • latlng_bounds (list) -- An array of points [lat_min, lon_min, lat_max, lon_max] in lat/lng coordinates.

  • grid_point (list) -- An array of points [x, y] in grid coordinates.

  • latlng_point (list) -- An array of points [lat, lon] in lat/lng coordinates.

  • huc_id (str or list) -- A comma-separated list of HUC ids that specifies the grid_bounds using the HUC bounding box.

  • z (int) -- A value of the z dimension to be used as a filter for this dimension when loading data.

  • level (int) -- A HUC level integer when reading HUC boundary files. Must be 2, 4, 6, 8, or 10.

  • site_id (str) -- Used when reading data associated with an observation site.

Returns:

Absolute path names to the file locations on the GPFS file system.

Return type:

list

Raises:

ValueError -- If no data catalog entry is found for the provided filter options.

Example:

import hf_hydrodata as hf

options = {
    "dataset": "NLDAS2", "temporal_resolution": "daily", "variable": "precipitation",
     "date_start":"2005-09-30", "date_end": "2005-10-3"
}
paths = hf.get_paths(options)
assert len(paths) == 5    # 5 days
hf_hydrodata.gridded.get_raw_file(filepath: str, *args, **kwargs)

Get the hydroframe file that is selected by the options.

This can be used to get smaller files such as huc_mapping file with the HUC ids for points in conus2. This can also be used to get large files (24 GB) such as the 30m water table depth file or wtd_uncertainty file.

Note - the 30m resolution full datasets are large - 25 GB for the cog and 35 GB for the tiff. It can take about 20-40 minutes to download these large files. If your connection drops you can restart the function writing to the same file and it will continue where it left off. If multiple people are trying to download a large file at the same time your request may be queued and take longer per chunking request during your download.

Parameters:

  • filepath -- A path name to the file to store the downloaded data.

  • options -- Any hf_hydrodata filter options. This must identify a single file. You may need to specify a date_start option if the catalog entry has a temporal resolution and the data is stored in multiple files.

Returns:

None

Raises:

ValueError -- If there are multiple paths selected from hydroframe.

Examples:

import hf_hydrodata as hf
options = {
    "dataset": "huc_mapping", "grid": "conus2", "level": "4"}
}
hf.get_raw_file("huc4.tiff", options)

options = {"dataset": "ma_2025", "variable": "water_table_depth", "grid": "conus2_wtd.30"}
hf.get_raw_file("wtd.tiff", options)

options = {"dataset": "ma_2025_cog", "variable": "water_table_depth"}
hf.get_raw_file("wtd.cog.tiff", options)
hf_hydrodata.gridded.read_fast_pfb(pfb_files: List[str], pfb_constraints: dict = None)

Fast PFB file reader supporting parallel reads and optional spatial constraints.

Read a list of PFB files and subset all the files with the optional constraints. This runs much faster than parflow.read_pfb() since it loads multiple files in parallel and uses fewer IO operations by reading fewer subgrid headers. It is especially faster for reading many files with small subgrid constraints.

Parameters:

  • pfb_files (list or str) -- A list of PFB files to read or a single PFB file name.

  • pfb_constraints (dict or list, optional) -- A constraint that filters the read of the PFB files by the x, y, z dimensions. Default is None (reads entire contents of all files).

For example,

constraint = [10, 20, 50, 50]
data = read_fast_pfb("a.pfb", constraint)

constraint = {"x":{"start": 10, "stop": 50}, "y": {"start": 20, "stop", 50}, "z": {"start": 0, "stop": 0}}
data = read_fast_pfb(["a.pfb", "b.pfb"], constraint)

constraint = [10, 20, 50, 50]
data = read_fast_pfb(["a.pfb", "b.pfb"], constraint)

constraint = [[10, 20], [50, 50]]
data = read_fast_pfb(["a.pfb", "b.pfb"], constraint)

If the pfb files have dimensions (25, 3247, 4222) then the return numpy array is (2, 25, 30, 40).