lgdo.lh5 package¶

Routines from reading and writing LEGEND Data Objects in HDF5 files. Currently the primary on-disk format for LGDO object is LEGEND HDF5 (LH5) files. IO is done via the class store.LH5Store. LH5 files can also be browsed easily in python like any HDF5 file using h5py.

Subpackages¶

lgdo.lh5._serializers package
- Subpackages
  - lgdo.lh5._serializers.read package
  - lgdo.lh5._serializers.write package

Submodules¶

lgdo.lh5.concat module¶

lgdo.lh5.concat._get_lgdos(file, obj_list)¶: Get name of LGDO objects.

lgdo.lh5.concat._get_obj_list(lh5_files, include_list=None, exclude_list=None)¶

Extract a list of lh5 objects to concatenate.

Parameters:

lh5_files (list) – list of input files to concatenate.
include_list (list | None) – patterns for tables to include.
exclude_list (list | None) – patterns for tables to exclude.

Return type:

list[str]

lgdo.lh5.concat._inplace_table_filter(name, table, obj_list)¶: filter objects nested in this LGDO

lgdo.lh5.concat._remove_nested_fields(lgdos, obj_list)¶

Remove (nested) table fields based on obj_list.

lgdo.lh5.concat._slice(obj, n_rows)¶

lgdo.lh5.concat.lh5concat(lh5_files, output, overwrite=False, *, include_list=None, exclude_list=None)¶

Concatenate LGDO Arrays, VectorOfVectors and Tables in LH5 files.

Parameters:

lh5_files (list) – list of input files to concatenate.
output (str) – path to the output file
include_list (list | None) – patterns for tables to include.
exclude_list (list | None) – patterns for tables to exclude.

lgdo.lh5.core module¶

lgdo.lh5.core.read(name, lh5_file, start_row=0, n_rows=9223372036854775807, idx=None, use_h5idx=False, field_mask=None, obj_buf=None, obj_buf_start=0, decompress=True, locking=False)¶

Read LH5 object data from a file.

Note

Use the idx parameter to read out particular rows of the data. The use_h5idx flag controls whether only those rows are read from disk or if the rows are indexed after reading the entire object. Reading individual rows can be orders of magnitude slower than reading the whole object and then indexing the desired rows. The default behavior (use_h5idx=False) is to use slightly more memory for a much faster read. See legend-pydataobj/issues/#29 for additional information.

Parameters:

name (str) – Name of the LH5 object to be read (including its group path).
lh5_file (str | File | Sequence[str | File]) – The file(s) containing the object to be read out. If a list of files, array-like object data will be concatenated into the output object.
start_row (int) – Starting entry for the object read (for array-like objects). For a list of files, only applies to the first file.
n_rows (int) – The maximum number of rows to read (for array-like objects). The actual number of rows read will be returned as one of the return values (see below).
idx (Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | bool | int | float | complex | str | bytes | _NestedSequence[bool | int | float | complex | str | bytes]) – For NumPy-style “fancying indexing” for the read to select only some rows, e.g. after applying some cuts to particular columns. Only selection along the first axis is supported, so tuple arguments must be one-tuples. If n_rows is not false, idx will be truncated to n_rows before reading. To use with a list of files, can pass in a list of idx’s (one for each file) or use a long contiguous list (e.g. built from a previous identical read). If used in conjunction with start_row and n_rows, will be sliced to obey those constraints, where n_rows is interpreted as the (max) number of selected values (in idx) to be read out. Note that the use_h5idx parameter controls some behaviour of the read and that the default behavior (use_h5idx=False) prioritizes speed over a small memory penalty.
use_h5idx (bool) – True will directly pass the idx parameter to the underlying h5py call such that only the selected rows are read directly into memory, which conserves memory at the cost of speed. There can be a significant penalty to speed for larger files (1 - 2 orders of magnitude longer time). False (default) will read the entire object into memory before performing the indexing. The default is much faster but requires additional memory, though a relatively small amount in the typical use case. It is recommended to leave this parameter as its default.
field_mask (Mapping[str, bool] | Sequence[str] | None) – For tables and structs, determines which fields get read out. Only applies to immediate fields of the requested objects. If a dict is used, a default dict will be made with the default set to the opposite of the first element in the dict. This way if one specifies a few fields at False, all but those fields will be read out, while if one specifies just a few fields as True, only those fields will be read out. If a list is provided, the listed fields will be set to True, while the rest will default to False.
obj_buf (LGDO) – Read directly into memory provided in obj_buf. Note: the buffer will be expanded to accommodate the data requested. To maintain the buffer length, send in n_rows = len(obj_buf).
obj_buf_start (int) – Start location in obj_buf for read. For concatenating data to array-like objects.
decompress (bool) – Decompress data encoded with LGDO’s compression routines right after reading. The option has no effect on data encoded with HDF5 built-in filters, which is always decompressed upstream by HDF5.
locking (bool) – Lock HDF5 file while reading

Returns:

(object, n_rows_read) – object is the read-out object n_rows_read is the number of rows successfully read out. Essential for arrays when the amount of data is smaller than the object buffer. For scalars and structs n_rows_read will be``1``. For tables it is redundant with table.loc. If obj_buf is None, only object is returned.

Return type:

LGDO | tuple[LGDO, int]

lgdo.lh5.core.read_as(name, lh5_file, library, **kwargs)¶

Read LH5 data from disk straight into a third-party data format view.

This function is nothing more than a shortcut chained call to read() and to LGDO.view_as().

Parameters:

name (str) – LH5 object name on disk.
lh5_file (str | File | Sequence[str | File]) – LH5 file name.
library (str) – string ID of the third-party data format library (np, pd, ak, etc).

Return type:

Any

See also

read, LGDO.view_as

lgdo.lh5.core.write(obj, name, lh5_file, group='/', start_row=0, n_rows=None, wo_mode='append', write_start=0, page_buffer=0, **h5py_kwargs)¶

Write an LGDO into an LH5 file.

If the obj LGDO has a compression attribute, its value is interpreted as the algorithm to be used to compress obj before writing to disk. The type of compression can be:

string, kwargs dictionary, hdf5plugin filter: interpreted as the name of a built-in or custom HDF5 compression filter ("gzip", "lzf", hdf5plugin filter object etc.) and passed directly to h5py.Group.create_dataset().
WaveformCodec object: If obj is a WaveformTable and obj.values holds the attribute, compress values using this algorithm. More documentation about the supported waveform compression algorithms at lgdo.compression.

If the obj LGDO has a hdf5_settings attribute holding a dictionary, it is interpreted as a list of keyword arguments to be forwarded directly to h5py.Group.create_dataset() (exactly like the first format of compression above). This is the preferred way to specify HDF5 dataset options such as chunking etc. If compression options are specified, they take precedence over those set with the compression attribute.

Note

The compression LGDO attribute takes precedence over the default HDF5 compression settings. The hdf5_settings attribute takes precedence over compression. These attributes are not written to disk.

Note

HDF5 compression is skipped for the encoded_data.flattened_data dataset of VectorOfEncodedVectors and ArrayOfEncodedEqualSizedArrays.

Parameters:

obj (LGDO) – LH5 object. if object is array-like, writes n_rows starting from start_row in obj.
name (str) – name of the object in the output HDF5 file.
lh5_file (str | File) – HDF5 file name or h5py.File object.
group (str | Group) – HDF5 group name or h5py.Group object in which obj should be written.
start_row (int) – first row in obj to be written.
n_rows (int | None) – number of rows in obj to be written.
wo_mode (str) –
- write_safe or w: only proceed with writing if the object does not already exist in the file.
- append or a: append along axis 0 (the first dimension) of array-like objects and array-like subfields of structs. Scalar objects get overwritten.
- overwrite or o: replace data in the file if present, starting from write_start. Note: overwriting with write_start = end of array is the same as append.
- overwrite_file or of: delete file if present prior to writing to it. write_start should be 0 (its ignored).
- append_column or ac: append columns from an Table obj only if there is an existing Table in the lh5_file with the same name and size. If the sizes don’t match, or if there are matching fields, it errors out.
write_start (int) – row in the output file (if already existing) to start overwriting from.
page_buffer (int) – enable paged aggregation with a buffer of this size in bytes Only used when creating a new file. Useful when writing a file with a large number of small datasets. This is a short-hand for (fs_stragety="page", fs_pagesize=[page_buffer])
**h5py_kwargs – additional keyword arguments forwarded to h5py.Group.create_dataset() to specify, for example, an HDF5 compression filter to be applied before writing non-scalar datasets. Note: `compression` Ignored if compression is specified as an `obj` attribute.

lgdo.lh5.datatype module¶

lgdo.lh5.datatype._lgdo_datatype_map: dict[str, LGDO] = {<class 'lgdo.types.array.Array'>: '^array<\\d+>\\{.+\\}$', <class 'lgdo.types.arrayofequalsizedarrays.ArrayOfEqualSizedArrays'>: '^array_of_equalsized_arrays<1,1>\\{.+\\}$', <class 'lgdo.types.encoded.ArrayOfEncodedEqualSizedArrays'>: '^array_of_encoded_equalsized_arrays<1,1>\\{.+\\}$', <class 'lgdo.types.encoded.VectorOfEncodedVectors'>: '^array<1>\\{encoded_array<1>\\{.+\\}\\}$', <class 'lgdo.types.fixedsizearray.FixedSizeArray'>: '^fixedsize_array<\\d+>\\{.+\\}$', <class 'lgdo.types.histogram.Histogram'>: '^struct\\{binning,weights,isdensity\\}$', <class 'lgdo.types.scalar.Scalar'>: '^real$|^bool$|^complex$|^bool$|^string$', <class 'lgdo.types.struct.Struct'>: '^struct\\{.*\\}$', <class 'lgdo.types.table.Table'>: '^table\\{.*\\}$', <class 'lgdo.types.vectorofvectors.VectorOfVectors'>: '^array<1>\\{array<1>\\{.+\\}\\}$'}¶: Mapping between LGDO types and regular expression defining the corresponding datatype string

lgdo.lh5.datatype.datatype(expr)¶

Return the LGDO type corresponding to a datatype string.

Return type:: type

lgdo.lh5.datatype.get_nested_datatype_string(expr)¶

Matches the content of the outermost curly brackets.

Return type:: str

lgdo.lh5.datatype.get_struct_fields(expr)¶

Returns a list of Struct fields, given its datatype string.

Return type:: list[str]

lgdo.lh5.exceptions module¶

exception lgdo.lh5.exceptions.LH5DecodeError(message, fname, oname)¶

Bases: Exception

exception lgdo.lh5.exceptions.LH5EncodeError(message, file, group, name)¶

Bases: Exception

lgdo.lh5.iterator module¶

class lgdo.lh5.iterator.LH5Iterator(lh5_files, groups, base_path='', entry_list=None, entry_mask=None, field_mask=None, buffer_len='100*MB', file_cache=10, file_map=None, friend=None)¶

Bases: Iterator

A class for iterating through one or more LH5 files, one block of entries at a time. This also accepts an entry list/mask to enable event selection, and a field mask.

This can be used as an iterator:

>>> for lh5_obj, i_entry, n_rows in LH5Iterator(...):
>>>    # do the thing!

This is intended for if you are reading a large quantity of data. This will ensure that you traverse files efficiently to minimize caching time and will limit your memory usage (particularly when reading in waveforms!). The lh5_obj that is read by this class is reused in order to avoid reallocation of memory; this means that if you want to hold on to data between reads, you will have to copy it somewhere!

When defining an LH5Iterator, you must give it a list of files and the hdf5 groups containing the data tables you are reading. You may also provide a field mask, and an entry list or mask, specifying which entries to read from the files. You may also pair it with a friend iterator, which contains a parallel group of files which will be simultaneously read. In addition to accessing requested data via lh5_obj, several properties exist to tell you where that data came from:

lh5_it.current_local_entries: get the entry numbers relative to the file the data came from
lh5_it.current_global_entries: get the entry number relative to the full dataset
lh5_it.current_files: get the file name corresponding to each entry
lh5_it.current_groups: get the group name corresponding to each entry

This class can also be used either for random access:

>>> lh5_obj, n_rows = lh5_it.read(i_entry)

to read the block of entries starting at i_entry. In case of multiple files or the use of an event selection, i_entry refers to a global event index across files and does not count events that are excluded by the selection.

Parameters:

lh5_files (str | list[str]) – file or files to read from. May include wildcards and environment variables.
groups (str | list[str] | list[list[str]]) – HDF5 group(s) to read. If a list of strings is provided, use same groups for each file. If a list of lists is provided, size of outer list must match size of file list, and each inner list will apply to a single file (or set of wildcarded files)
entry_list (list[int] | list[list[int]] | None) – list of entry numbers to read. If a nested list is provided, expect one top-level list for each file, containing a list of local entries. If a list of ints is provided, use global entries.
entry_mask (list[bool] | list[list[bool]] | None) – mask of entries to read. If a list of arrays is provided, expect one for each file. Ignore if a selection list is provided.
field_mask (dict[str, bool] | list[str] | tuple[str] | None) – mask of which fields to read. See LH5Store.read() for more details.
buffer_len (int) – number of entries to read at a time while iterating through files.
file_cache (int) – maximum number of files to keep open at a time
file_map (NDArray[int]) – cumulative file/group entries. This can be provided on construction to speed up random or sparse access; otherwise, we sequentially read the size of each group. WARNING: no checks for accuracy are performed so only use this if you know what you are doing!
friend (Iterator | None) – a “friend” LH5Iterator that will be read in parallel with this. The friend should have the same length and entry list. A single LH5 table containing columns from both iterators will be returned. Note that buffer_len will be set to the minimum of the two.

_get_file_cumentries(i_file)¶

Helper to get cumulative iterator entries in file

Return type:: int

_get_file_cumlen(i_file)¶

Helper to get cumulative file length of file

Return type:: int

property current_entry: int¶: deprecated alias for current_i_entry

property current_files: ndarray[tuple[int, ...], dtype[str]]¶: Return list of file names for entries in buffer

property current_global_entries: ndarray[tuple[int, ...], dtype[int]]¶: Return list of local file entries in buffer

property current_groups: ndarray[tuple[int, ...], dtype[str]]¶: Return list of group names for entries in buffer

property current_local_entries: ndarray[tuple[int, ...], dtype[int]]¶: Return list of local file entries in buffer

get_file_entrylist(i_file)¶

Helper to get entry list for file

Return type:: ndarray

get_global_entrylist()¶

Get global entry list, constructing it if needed

Return type:: ndarray

read(i_entry)¶

Read the nextlocal chunk of events, starting at i_entry. Return the LH5 buffer and number of rows read.

Return type:: tuple[Array | Scalar | Struct | VectorOfVectors, int]

reset_field_mask(mask)¶: Replaces the field mask of this iterator and any friends with mask

lgdo.lh5.store module¶

This module implements routines from reading and writing LEGEND Data Objects in HDF5 files.

class lgdo.lh5.store.LH5Store(base_path='', keep_open=False, locking=False)¶

Bases: object

Class to represent a store of LEGEND HDF5 files. The two main methods implemented by the class are read() and write().

Examples

>>> from lgdo import LH5Store
>>> store = LH5Store()
>>> obj, _ = store.read("/geds/waveform", "file.lh5")
>>> type(obj)
lgdo.waveformtable.WaveformTable

Parameters:

base_path (str) – directory path to prepend to LH5 files.
keep_open (bool) – whether to keep files open by storing the h5py objects as class attributes. If keep_open is an int, keep only the n most recently opened files; if True, no limit
locking (bool) – whether to lock files when reading

get_buffer(name, lh5_file, size=None, field_mask=None)¶

Returns an LH5 object appropriate for use as a pre-allocated buffer in a read loop. Sets size to size if object has a size.

Return type:: LGDO

gimme_file(lh5_file, mode='r', page_buffer=0, **file_kwargs)¶

Returns a h5py file object from the store or creates a new one.

Parameters:

lh5_file (str | File) – LH5 file name.
mode (str) – mode in which to open file. See h5py.File documentation.
page_buffer (int) – enable paged aggregation with a buffer of this size in bytes Only used when creating a new file. Useful when writing a file with a large number of small datasets. This is a short-hand for (fs_stragety="page", fs_pagesize=[page_buffer])
file_kwargs – Keyword arguments for h5py.File

Return type:

File

gimme_group(group, base_group, grp_attrs=None, overwrite=False)¶

Returns an existing h5py group from a base group or creates a new one.

See also

lh5.utils.get_h5_group

Return type:: Group

read(name, lh5_file, start_row=0, n_rows=9223372036854775807, idx=None, use_h5idx=False, field_mask=None, obj_buf=None, obj_buf_start=0, decompress=True, **file_kwargs)¶

Read LH5 object data from a file in the store.

See also

lh5.core.read

Return type:: tuple[LGDO, int]

read_n_rows(name, lh5_file)¶

Look up the number of rows in an Array-like object called name in lh5_file.

Return None if it is a Scalar or a Struct.

Return type:: int | None

read_size_in_bytes(name, lh5_file)¶

Look up the size (in B) of the object in memory. Will recursively crawl through all objects in a Struct or Table

Return type:: int

write(obj, name, lh5_file, group='/', start_row=0, n_rows=None, wo_mode='append', write_start=0, page_buffer=0, **h5py_kwargs)¶

Write an LGDO into an LH5 file.

See also

lh5.core.write

lgdo.lh5.tools module¶

lgdo.lh5.tools.load_dfs(f_list, par_list, lh5_group='', idx_list=None)¶

Build a pandas.DataFrame from LH5 data.

Given a list of files (can use wildcards), a list of LH5 columns, and optionally the group path, return a pandas.DataFrame with all values for each parameter.

See also

load_nda()

Returns:: dataframe – contains columns for each parameter in par_list, and rows containing all data for the associated parameters concatenated over all files in f_list.
Return type:: DataFrame

lgdo.lh5.tools.load_nda(f_list, par_list, lh5_group='', idx_list=None)¶

Build a dictionary of numpy.ndarrays from LH5 data.

Given a list of files, a list of LH5 table parameters, and an optional group path, return a NumPy array with all values for each parameter.

Parameters:

f_list (str | list[str]) – A list of files. Can contain wildcards.
par_list (list[str]) – A list of parameters to read from each file.
lh5_group (str) – group path within which to find the specified parameters.
idx_list (list[ndarray[tuple[int, ...], dtype[_ScalarType_co]] | list | tuple] | None) – for fancy-indexed reads. Must be one index array for each file in f_list.

Returns:

par_data – A dictionary of the parameter data keyed by the elements of par_list. Each entry contains the data for the specified parameter concatenated over all files in f_list.

Return type:

dict[str, ndarray[tuple[int, …], dtype[_ScalarType_co]]]

lgdo.lh5.tools.ls(lh5_file, lh5_group='', recursive=False)¶

Return a list of LH5 groups in the input file and group, similar to ls or h5ls. Supports wildcards in group names.

Parameters:

lh5_file (str | Group) – name of file.
lh5_group (str) – group to search. add a / to the end of the group name if you want to list all objects inside that group.
recursive (bool) – if True, recurse into subgroups.

Return type:

list[str]

lgdo.lh5.tools.show(lh5_file, lh5_group='/', attrs=False, indent='', header=True, depth=None, detail=False)¶

Print a tree of LH5 file contents with LGDO datatype.

Parameters:

lh5_file (str | Group) – the LH5 file.
lh5_group (str) – print only contents of this HDF5 group.
attrs (bool) – print the HDF5 attributes too.
indent (str) – indent the diagram with this string.
header (bool) – print lh5_group at the top of the diagram.
depth (int | None) – maximum tree depth of groups to print
detail (bool) – whether to print additional information about how the data is stored

Examples

>>> from lgdo import show
>>> show("file.lh5", "/geds/raw")
/geds/raw
├── channel · array<1>{real}
├── energy · array<1>{real}
├── timestamp · array<1>{real}
├── waveform · table{t0,dt,values}
│   ├── dt · array<1>{real}
│   ├── t0 · array<1>{real}
│   └── values · array_of_equalsized_arrays<1,1>{real}
└── wf_std · array<1>{real}

lgdo.lh5.utils module¶

Implements utilities for LEGEND Data Objects.

lgdo.lh5.utils.expand_path(path, substitute=None, list=False, base_path=None)¶

Expand (environment) variables and wildcards to return absolute paths.

Parameters:

path (str) – name of path, which may include environment variables and wildcards.
list (bool) – if True, return a list. If False, return a string; if False and a unique file is not found, raise an exception.
substitute (dict[str, str] | None) – use this dictionary to substitute variables. Environment variables take precedence.
base_path (str | None) – name of base path. Returned paths will be relative to base.

Returns:

path or list of paths – Unique absolute path, or list of all absolute paths

Return type:

str | list

lgdo.lh5.utils.expand_vars(expr, substitute=None)¶

Expand (environment) variables.

Note

Malformed variable names and references to non-existing variables are left unchanged.

Parameters:

expr (str) – string expression, which may include (environment) variables prefixed by $.
substitute (dict[str, str] | None) – use this dictionary to substitute variables. Takes precedence over environment variables.

Return type:

str

lgdo.lh5.utils.fmtbytes(num, suffix='B')¶: Returns formatted f-string for printing human-readable number of bytes.

lgdo.lh5.utils.get_buffer(name, lh5_file, size=None, field_mask=None)¶

Returns an LGDO appropriate for use as a pre-allocated buffer.

Sets size to size if object has a size.

Return type:: LGDO

lgdo.lh5.utils.get_h5_group(group, base_group, grp_attrs=None, overwrite=False)¶

Returns an existing h5py group from a base group or creates a new one. Can also set (or replace) group attributes.

Parameters:

group (str | Group) – name of the HDF5 group.
base_group (Group) – HDF5 group to be used as a base.
grp_attrs (Mapping[str, Any] | None) – HDF5 group attributes.
overwrite (bool) – whether overwrite group attributes, ignored if grp_attrs is None.

Return type:

Group

lgdo.lh5.utils.read_n_rows(name, h5f)¶

Look up the number of rows in an Array-like LGDO object on disk.

Return None if name is a Scalar or a Struct.

Return type:: int | None

lgdo.lh5.utils.read_size_in_bytes(name, h5f)¶

Look up the size (in B) in an LGDO object in memory. Will crawl recursively through members of a Struct or Table

Return type:: int | None