tatami_layered Namespace Reference

Create layered sparse matrices for tatami. More...

Classes
struct	ConvertToLayeredSparseOptions
	Options for convert_to_layered_sparse(). More...
struct	ReadLayeredSparseFromMatrixMarketOptions

Functions
template<typename ValueOut_ = double, typename IndexOut_ = int, typename ColumnIndex_ = std::uint16_t, typename ValueIn_ , typename IndexIn_ >
std::shared_ptr< tatami::Matrix< ValueOut_, IndexOut_ > >	convert_to_layered_sparse (const tatami::Matrix< ValueIn_, IndexIn_ > &mat, const ConvertToLayeredSparseOptions &options)
template<typename Value_ = double, typename Index_ = int, typename ColumnIndex_ = std::uint16_t>
std::shared_ptr< tatami::Matrix< Value_, Index_ > >	read_layered_sparse_from_matrix_market_text_file (const char *filepath, const ReadLayeredSparseFromMatrixMarketOptions &options)
template<typename Value_ = double, typename Index_ = int, typename ColumnIndex_ = std::uint16_t>
std::shared_ptr< tatami::Matrix< Value_, Index_ > >	read_layered_sparse_from_matrix_market_some_file (const char *filepath, const ReadLayeredSparseFromMatrixMarketOptions &options)
template<typename Value_ = double, typename Index_ = int, typename ColumnIndex_ = std::uint16_t>
std::shared_ptr< tatami::Matrix< Value_, Index_ > >	read_layered_sparse_from_matrix_market_gzip_file (const char *filepath, const ReadLayeredSparseFromMatrixMarketOptions &options)
template<typename Value_ = double, typename Index_ = int, typename ColumnIndex_ = std::uint16_t>
std::shared_ptr< tatami::Matrix< Value_, Index_ > >	read_layered_sparse_from_matrix_market_text_buffer (const unsigned char *contents, std::size_t length, const ReadLayeredSparseFromMatrixMarketOptions &options)
template<typename Value_ = double, typename Index_ = int, typename ColumnIndex_ = std::uint16_t>
std::shared_ptr< tatami::Matrix< Value_, Index_ > >	read_layered_sparse_from_matrix_market_some_buffer (const unsigned char *contents, std::size_t length, const ReadLayeredSparseFromMatrixMarketOptions &options)
template<typename Value_ = double, typename Index_ = int, typename ColumnIndex_ = std::uint16_t>
std::shared_ptr< tatami::Matrix< Value_, Index_ > >	read_layered_sparse_from_matrix_market_zlib_buffer (const unsigned char *contents, std::size_t length, const ReadLayeredSparseFromMatrixMarketOptions &options)

Detailed Description

Create layered sparse matrices for tatami.

Function Documentation

convert_to_layered_sparse()

template<typename ValueOut_ = double, typename IndexOut_ = int, typename ColumnIndex_ = std::uint16_t, typename ValueIn_ , typename IndexIn_ >

std::shared_ptr< tatami::Matrix< ValueOut_, IndexOut_ > > tatami_layered::convert_to_layered_sparse	(	const tatami::Matrix< ValueIn_, IndexIn_ > &	mat,
		const ConvertToLayeredSparseOptions &	options )

Parameters

mat	A tatami::Matrix object containing non-negative integers.
options	Further options.

Returns

A tatami::Matrix object containing a layered sparse matrix.

Template Parameters

ValueOut_	Type of data value for the output tatami::Matrix interface.
IndexOut_	Integer type for the row/column indices of the output.
ColumnIndex_	Integer type for the stored column indices.
ValueIn_	Type of data value for the input.
IndexIn_	Integer type for the row/column indices of the input.

This function converts an existing sparse matrix of non-negative integers into a layered sparse matrix. The aim is to reduce memory usage by storing each row's data in the smallest unsigned integer type that can hold them. To create a layered sparse matrix:

1. We split the input matrix into chunks of options.chunk_size contiguous columns.
2. Within each chunk, we identify the maximum integer for each row.
3. Data for each row are stored in one of three sparse matrices using 8, 16, or 32-bit unsigned integers as the data type, depending on the row's maximum value.
4. The three sparse matrices are combined together with tatami::DelayedBind, with some use of tatami::DelayedSubset to restore the original order of rows.
5. Each chunk is then combined together with tatami::DelayedBind.

We improve the chances of being able to use small types by splitting the matrix columns into chunks. This ensures that a few large values in a particular row only cause promotion to a larger integer type for the chunks in which they occur.

Setting ColumnIndex_ to the smallest type that can hold options.chunk_size - 1 can be used to further reduce memory usage. If ColumnIndex_ is not able to hold options.chunk_size - 1, the chunk size is automatically set to the largest value that can be represented by ColumnIndex_ plus 1. For example, if ColumnIndex_ was set to an unsigned 8-bit integer, chunk_size would be automatically reduced to 256.

read_layered_sparse_from_matrix_market_text_file()

template<typename Value_ = double, typename Index_ = int, typename ColumnIndex_ = std::uint16_t>

std::shared_ptr< tatami::Matrix< Value_, Index_ > > tatami_layered::read_layered_sparse_from_matrix_market_text_file	(	const char *	filepath,
		const ReadLayeredSparseFromMatrixMarketOptions &	options )

Parameters

filepath	Path to an uncompressed Matrix Market text file.
options	Further options.

Returns

A tatami::Matrix object containing a layered sparse matrix.

Template Parameters

Value_	Type of data value for the output tatami::Matrix interface.
Index_	Integer type for the row/column indices of the output.
ColumnIndex_	Integer type for the stored column indices.

This function loads a layered sparse integer matrix from a Matrix Market file. The aim is to reduce memory usage by storing each gene's counts in the smallest unsigned integer type that can hold them. See convert_to_layered_sparse() for more details.

read_layered_sparse_from_matrix_market_some_file()

template<typename Value_ = double, typename Index_ = int, typename ColumnIndex_ = std::uint16_t>

std::shared_ptr< tatami::Matrix< Value_, Index_ > > tatami_layered::read_layered_sparse_from_matrix_market_some_file	(	const char *	filepath,
		const ReadLayeredSparseFromMatrixMarketOptions &	options )

Parameters

filepath	Path to a (possibly Gzip-compressed) Matrix Market file.
options	Further options.

Returns

A tatami::Matrix object containing a layered sparse matrix.

Template Parameters

Value_	Type of data value for the output tatami::Matrix interface.
Index_	Integer type for the row/column indices of the output.
ColumnIndex_	Integer type for the stored column indices.

This function loads a layered sparse integer matrix from a Matrix Market file. The aim is to reduce memory usage by storing each gene's counts in the smallest unsigned integer type that can hold them. See convert_to_layered_sparse() for more details.

read_layered_sparse_from_matrix_market_gzip_file()

template<typename Value_ = double, typename Index_ = int, typename ColumnIndex_ = std::uint16_t>

std::shared_ptr< tatami::Matrix< Value_, Index_ > > tatami_layered::read_layered_sparse_from_matrix_market_gzip_file	(	const char *	filepath,
		const ReadLayeredSparseFromMatrixMarketOptions &	options )

Parameters

filepath	Path to a Gzip-compressed Matrix Market file.
options	Further options.

Returns

A tatami::Matrix object containing a layered sparse matrix.

Template Parameters

Value_	Type of data value for the output tatami::Matrix interface.
Index_	Integer type for the row/column indices of the output.
ColumnIndex_	Integer type for the stored column indices.

This function loads a layered sparse integer matrix from a Matrix Market file. The aim is to reduce memory usage by storing each gene's counts in the smallest unsigned integer type that can hold them. See convert_to_layered_sparse() for more details.

read_layered_sparse_from_matrix_market_text_buffer()

template<typename Value_ = double, typename Index_ = int, typename ColumnIndex_ = std::uint16_t>

std::shared_ptr< tatami::Matrix< Value_, Index_ > > tatami_layered::read_layered_sparse_from_matrix_market_text_buffer	(	const unsigned char *	contents,
		std::size_t	length,
		const ReadLayeredSparseFromMatrixMarketOptions &	options )

Parameters

contents	Array containing the contents of an uncompressed Matrix Market text file.
length	Length of the array.
options	Further options.

Returns

A tatami::Matrix object containing a layered sparse matrix.

Template Parameters

Value_	Type of data value for the output tatami::Matrix interface.
Index_	Integer type for the row/column indices of the output.
ColumnIndex_	Integer type for the stored column indices.

This function loads a layered sparse integer matrix from a buffer with the contents of a Matrix Market file. The aim is to reduce memory usage by storing each gene's counts in the smallest unsigned integer type that can hold them. See convert_to_layered_sparse() for more details.

read_layered_sparse_from_matrix_market_some_buffer()

template<typename Value_ = double, typename Index_ = int, typename ColumnIndex_ = std::uint16_t>

std::shared_ptr< tatami::Matrix< Value_, Index_ > > tatami_layered::read_layered_sparse_from_matrix_market_some_buffer	(	const unsigned char *	contents,
		std::size_t	length,
		const ReadLayeredSparseFromMatrixMarketOptions &	options )

Parameters

contents	Array containing the contents of a (possibly Gzip/Zlib-compressed) Matrix Market file.
length	Length of the array.
options	Further options.

Returns

A tatami::Matrix object containing a layered sparse matrix.

Template Parameters

Value_	Type of data value for the output tatami::Matrix interface.
Index_	Integer type for the row/column indices of the output.
ColumnIndex_	Integer type for the stored column indices.

This function loads a layered sparse integer matrix from a buffer with the contents of a Matrix Market file. The aim is to reduce memory usage by storing each gene's counts in the smallest unsigned integer type that can hold them. See convert_to_layered_sparse() for more details.

read_layered_sparse_from_matrix_market_zlib_buffer()

template<typename Value_ = double, typename Index_ = int, typename ColumnIndex_ = std::uint16_t>

std::shared_ptr< tatami::Matrix< Value_, Index_ > > tatami_layered::read_layered_sparse_from_matrix_market_zlib_buffer	(	const unsigned char *	contents,
		std::size_t	length,
		const ReadLayeredSparseFromMatrixMarketOptions &	options )

Parameters

contents	Array containing the contents of a Gzip/Zlib-compressed Matrix Market file.
length	Length of the array.
options	Further options.

Returns

A tatami::Matrix object containing a layered sparse matrix.

Template Parameters

Value_	Type of data value for the output tatami::Matrix interface.
Index_	Integer type for the row/column indices of the output.
ColumnIndex_	Integer type for the stored column indices.

This function loads a layered sparse integer matrix from a buffer with the contents of a Matrix Market file. The aim is to reduce memory usage by storing each gene's counts in the smallest unsigned integer type that can hold them. See convert_to_layered_sparse() for more details.