cellarr.build package

Submodules

cellarr.build.build_cellarr_steps module

class cellarr.build.build_cellarr_steps.SlurmBuilder(output_dir, log_dir, temp_dir, memory_gb=64, time_hours=24, cpus_per_task=4)

Bases: object

SLURM-based builder for CellArrDataset.

__init__(output_dir, log_dir, temp_dir, memory_gb=64, time_hours=24, cpus_per_task=4)

Initialize the SLURM builder.

Parameters:

• output_dir (str) – Path to final output directory.

• log_dir (str) – Path to store SLURM logs.

• temp_dir (str) – Path for temporary files.

• memory_gb (int) – Memory per job in GB.

• time_hours (int) – Time limit per job in hours.

• cpus_per_task (int) – CPUs per task.

create_array_script(job_name, python_script, args, n_tasks, dependencies=None, python_env='', sbatch_extra_args='')

Create a SLURM array job submission script.

Return type:

str

create_slurm_script(job_name, python_script, args, dependencies=None, python_env='', sbatch_extra_args='')

Create a SLURM job submission script.

Return type:

str

submit_cell_metadata_job(files, cell_options, dependency, python_env, sbatch_extra_args)

Submit cell metadata processing job.

Return type:

str

submit_final_assembly(matrix_names, dependencies, python_env, sbatch_extra_args)

Submit final assembly job.

Return type:

str

submit_gene_annotation_job(files, gene_options, python_env, sbatch_extra_args)

Submit gene annotation processing job.

Return type:

str

submit_job(script_path)

Submit a SLURM job and return job ID.

Return type:

str

submit_matrix_processing(files, matrix_options, dependency, python_env, sbatch_extra_args)

Submit matrix processing as SLURM array job.

Return type:

Tuple[str, str]

submit_sample_metadata_job(files, sample_options, dependency, python_env, sbatch_extra_args)

Submit sample metadata processing job.

Return type:

str

cellarr.build.build_cellarr_steps.main()

cellarr.build.build_cellarrdataset module

Build the CellArrDatset.

The CellArrDataset method is designed to store single-cell RNA-seq datasets but can be generalized to store any 2-dimensional experimental data.

This method creates four TileDB files in the directory specified by output_path:

• gene_annotation: A TileDB file containing feature/gene annotations.

• sample_metadata: A TileDB file containing sample metadata.

• cell_metadata: A TileDB file containing cell metadata including mapping to the samples

they are tagged with in sample_metadata. - An assay TileDB group containing various matrices. This allows the package to store multiple different matrices, e.g. ‘counts’, ‘normalized’, ‘scaled’ for the same sample/cell and gene attributes.

The TileDB matrix file is stored in a cell X gene orientation. This orientation is chosen because the fastest-changing dimension as new files are added to the collection is usually the cells rather than genes.

Process:

1. Scan the Collection: Scan the entire collection of files to create a unique set of feature ids (e.g. gene symbols). Store this set as the gene_annotation TileDB file.

2. Sample Metadata: Store sample metadata in sample_metadata TileDB file. Each file is typically considered a sample, and an automatic mapping is created between files and samples.

3. Store Cell Metadata: Store cell metadata in the cell_metadata TileDB file.

4. Remap and Orient Data: For each dataset in the collection, remap and orient the feature dimension using the feature set from Step 1. This step ensures consistency in gene measurement and order, even if some genes are unmeasured or ordered differently in the original experiments.

Example

import anndata
import numpy as np
import tempfile
from cellarr import (
    build_cellarrdataset,
    CellArrDataset,
    MatrixOptions,
)

# Create a temporary directory
tempdir = tempfile.mkdtemp()

# Read AnnData objects
adata1 = anndata.read_h5ad(
    "path/to/object1.h5ad",
    "r",
)
# or just provide the path
adata2 = "path/to/object2.h5ad"

# Build CellArrDataset
dataset = build_cellarrdataset(
    output_path=tempdir,
    files=[
        adata1,
        adata2,
    ],
    matrix_options=MatrixOptions(
        dtype=np.float32
    ),
)

cellarr.build.build_cellarrdataset.build_cellarrdataset(files, output_path, gene_annotation=None, sample_metadata=None, cell_metadata=None, sample_metadata_options=SampleMetadataOptions(skip=False, dtype=<class 'numpy.uint32'>, tiledb_store_name='sample_metadata', column_types=None), cell_metadata_options=CellMetadataOptions(skip=False, dtype=<class 'numpy.uint32'>, tiledb_store_name='cell_metadata', column_types=None), gene_annotation_options=GeneAnnotationOptions(skip=False, feature_column='index', dtype=<class 'numpy.uint32'>, tiledb_store_name='gene_annotation', column_types=None), matrix_options=MatrixOptions(skip=False, consolidate_duplicate_gene_func=<built-in function sum>, matrix_name='counts', matrix_attr_name='data', dtype=<class 'numpy.uint16'>, tiledb_store_name='counts'), optimize_tiledb=True, num_threads=1)

Create the CellArrDataset from a list of single-cell experiment objects.

All files are expected to be consistent and any modifications to make them consistent is outside the scope of this function and package.

There’s a few assumptions this process makes: - If object in files is an AnnData or H5AD object, these must contain an assay matrix in the layers slot of the object named as layer_matrix_name parameter. - Feature information must contain a column defined by the parameter feature_column in the that contains feature ids or gene symbols across all files. - If no cell_metadata is provided, we scan to count the number of cells and create a simple range index. - Each file is considered a sample and a mapping between cells and samples is automatically created. Hence the sample information provided must match the number of input files and is expected to be in the same order.

Parameters:

• files (List[Union[str, AnnData]]) – List of file paths to H5AD or AnnData objects.

• output_path (str) – Path to where the output TileDB files should be stored.

• gene_annotation (Union[List[str], str, DataFrame]) –

  A DataFrame containing the feature/gene annotations across all objects.

  Alternatively, may provide a path to the file containing a concatenated gene annotations across all datasets. In this case, the first row is expected to contain the column names and an index column containing the feature ids or gene symbols.

  Alternatively, a list or a dictionary of gene symbols.

  Irrespective of the input, the object will be appended with a cellarr_gene_index column that contains numerical gene index across all objects.

  Defaults to None, then a gene set is generated by scanning all objects in files.

  Additional options may be specified by gene_annotations_options.

• sample_metadata (Union[DataFrame, str]) –

  A DataFrame containing the sample metadata for each file in files. Hences the number of rows in the dataframe must match the number of files.

  Alternatively, may provide path to the file containing a concatenated sample metadata across all cells. In this case, the first row is expected to contain the column names.

  Additionally, the order of rows is expected to be in the same order as the input list of files.

  Irrespective of the input, this object is appended with a cellarr_original_gene_set column that contains the original set of feature ids (or gene symbols) from the dataset to differentiate between zero-expressed vs unmeasured genes. Additional columns are added to help with slicing and accessing chunks.

  Defaults to None, in which case, we create a simple sample metadata dataframe containing the list of datasets. Each dataset is named as sample_{i} where i refers to the index position of the object in files.

  Additional options may be specified by sample_metadata_options.

• cell_metadata (Union[DataFrame, str]) –

  A DataFrame containing the cell metadata for cells across files. Hences the number of rows in the dataframe must match the number of cells across all files.

  Alternatively, may provide path to the file containing a concatenated cell metadata across all cells. In this case, the first row is expected to contain the column names.

  Additionally, the order of cells is expected to be in the same order as the input list of files. If the input is a path, the file is expected to contain mappings between cells and datasets (or samples).

  Defaults to None, we scan all files to count the number of cells, then create a simple cell metadata DataFrame containing mappings from cells to their associated datasets. Each dataset is named as sample_{i} where i refers to the index position of the object in files.

  Additional options may be specified by cell_metadata_options.

• sample_metadata_options (SampleMetadataOptions) – Optional parameters when generating sample_metadata store.

• cell_metadata_options (CellMetadataOptions) – Optional parameters when generating cell_metadata store.

• gene_annotation_options (GeneAnnotationOptions) – Optional parameters when generating gene_annotation store.

• matrix_options (Union[MatrixOptions, List[MatrixOptions]]) – Optional parameters when generating matrix store.

• optimize_tiledb (bool) – Whether to run TileDB’s vaccum and consolidation (may take long).

• num_threads (int) – Number of threads. Defaults to 1.

cellarr.build.build_cellarrdataset.generate_metadata_tiledb_csv(output_uri, input, column_dtype=None, index_col=False, chunksize=1000)

Generate a metadata TileDB from csv.

The difference between this and generate_metadata_tiledb_frame is when the csv is super large and it won’t fit into memory.

Parameters:

• output_uri (str) – TileDB URI or path to save the file.

• input (str) – Path to the csv file. The first row is expected to contain the column names.

• column_dtype (Dict[str, dtype]) – Dtype for each of the columns. Defaults to None.

• chunksize – Chunk size to read the dataframe. Defaults to 1000.

cellarr.build.build_cellarrdataset.generate_metadata_tiledb_frame(output_uri, input, column_types=None)

Generate metadata TileDB from a DataFrame.

Parameters:

• output_uri (str) – TileDB URI or path to save the file.

• input (DataFrame) – Input dataframe.

• column_types (dict) –

  You can specify type of each column name to cast into. “ascii” or str works best for most scenarios.

  Defaults to None.

cellarr.build.build_options module

class cellarr.build.build_options.CellMetadataOptions(skip=False, dtype=<class 'numpy.uint32'>, tiledb_store_name='cell_metadata', column_types=None)

Bases: object

Optional arguments for the cell_metadata store for build_cellarrdataset().

skip

Whether to skip generating cell metadata TileDB. Defaults to False.

dtype

NumPy dtype for the cell dimension. Defaults to np.uint32.

Note: make sure the number of cells fit within the integer limits of unsigned-int32.

tiledb_store_name

Name of the TileDB file. Defaults to “cell_metadata”.

column_names

List of cell metadata columns to extract from each data object. If a column is not available, it is represented as ‘NA’.

column_types

A dictionary containing column names as keys and the value representing the type to in the TileDB. The TileDB will only contain the columns listed here. If the column is not present in a dataset, it is represented as ‘NA’.

__annotations__ = {'column_types': typing.Dict[str, numpy.dtype], 'dtype': <class 'numpy.dtype'>, 'skip': <class 'bool'>, 'tiledb_store_name': <class 'str'>}

__dataclass_fields__ = {'column_types': Field(name='column_types',type=typing.Dict[str, numpy.dtype],default=None,default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'dtype': Field(name='dtype',type=<class 'numpy.dtype'>,default=<class 'numpy.uint32'>,default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'skip': Field(name='skip',type=<class 'bool'>,default=False,default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'tiledb_store_name': Field(name='tiledb_store_name',type=<class 'str'>,default='cell_metadata',default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD)}

__dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,order=False,unsafe_hash=False,frozen=False)

__eq__(other)

Return self==value.

__hash__ = None

__init__(skip=False, dtype=<class 'numpy.uint32'>, tiledb_store_name='cell_metadata', column_types=None)

__match_args__ = ('skip', 'dtype', 'tiledb_store_name', 'column_types')

__repr__()

Return repr(self).

column_types: Dict[str, dtype] = None

dtype

alias of uint32

skip: bool = False

tiledb_store_name: str = 'cell_metadata'

class cellarr.build.build_options.GeneAnnotationOptions(skip=False, feature_column='index', dtype=<class 'numpy.uint32'>, tiledb_store_name='gene_annotation', column_types=None)

Bases: object

Optional arguments for the gene_annotation store for build_cellarrdataset().

feature_column

Column in var containing the feature ids (e.g. gene symbols). Defaults to the index of the var slot.

skip

Whether to skip generating gene annotation TileDB. Defaults to False.

dtype

NumPy dtype for the gene dimension. Defaults to np.uint32.

Note: make sure the number of genes fit within the integer limits of unsigned-int32.

tiledb_store_name

Name of the TileDB file. Defaults to “gene_annotation”.

column_types

A dictionary containing column names as keys and the value representing the type to in the TileDB.

If None, all columns are cast as ‘ascii’.

__annotations__ = {'column_types': typing.Dict[str, numpy.dtype], 'dtype': <class 'numpy.dtype'>, 'feature_column': <class 'str'>, 'skip': <class 'bool'>, 'tiledb_store_name': <class 'str'>}

__dataclass_fields__ = {'column_types': Field(name='column_types',type=typing.Dict[str, numpy.dtype],default=None,default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'dtype': Field(name='dtype',type=<class 'numpy.dtype'>,default=<class 'numpy.uint32'>,default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'feature_column': Field(name='feature_column',type=<class 'str'>,default='index',default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'skip': Field(name='skip',type=<class 'bool'>,default=False,default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'tiledb_store_name': Field(name='tiledb_store_name',type=<class 'str'>,default='gene_annotation',default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD)}

__dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,order=False,unsafe_hash=False,frozen=False)

__eq__(other)

Return self==value.

__hash__ = None

__init__(skip=False, feature_column='index', dtype=<class 'numpy.uint32'>, tiledb_store_name='gene_annotation', column_types=None)

__match_args__ = ('skip', 'feature_column', 'dtype', 'tiledb_store_name', 'column_types')

__repr__()

Return repr(self).

column_types: Dict[str, dtype] = None

dtype

alias of uint32

feature_column: str = 'index'

skip: bool = False

tiledb_store_name: str = 'gene_annotation'

class cellarr.build.build_options.MatrixOptions(skip=False, consolidate_duplicate_gene_func=<built-in function sum>, matrix_name='counts', matrix_attr_name='data', dtype=<class 'numpy.uint16'>, tiledb_store_name='counts')

Bases: object

Optional arguments for the matrix store for build_cellarrdataset().

matrix_name

Matrix name from layers slot to add to TileDB. Must be consistent across all objects in files.

Defaults to “counts”.

matrix_attr_name

Name of the matrix to be stored in the TileDB file. Defaults to “data”.

consolidate_duplicate_gene_func

Function to consolidate when the AnnData object contains multiple rows with the same feature id or gene symbol.

Defaults to sum().

skip

Whether to skip generating matrix TileDB. Defaults to False.

dtype

NumPy dtype for the values in the matrix. Defaults to np.uint16.

Note: make sure the matrix values fit within the range limits of unsigned-int16.

tiledb_store_name

Name of the TileDB file. Defaults to counts.

__annotations__ = {'consolidate_duplicate_gene_func': <built-in function callable>, 'dtype': <class 'numpy.dtype'>, 'matrix_attr_name': <class 'str'>, 'matrix_name': <class 'str'>, 'skip': <class 'bool'>, 'tiledb_store_name': <class 'str'>}

__dataclass_fields__ = {'consolidate_duplicate_gene_func': Field(name='consolidate_duplicate_gene_func',type=<built-in function callable>,default=<built-in function sum>,default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'dtype': Field(name='dtype',type=<class 'numpy.dtype'>,default=<class 'numpy.uint16'>,default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'matrix_attr_name': Field(name='matrix_attr_name',type=<class 'str'>,default='data',default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'matrix_name': Field(name='matrix_name',type=<class 'str'>,default='counts',default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'skip': Field(name='skip',type=<class 'bool'>,default=False,default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'tiledb_store_name': Field(name='tiledb_store_name',type=<class 'str'>,default='counts',default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD)}

__dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,order=False,unsafe_hash=False,frozen=False)

__eq__(other)

Return self==value.

__hash__ = None

__init__(skip=False, consolidate_duplicate_gene_func=<built-in function sum>, matrix_name='counts', matrix_attr_name='data', dtype=<class 'numpy.uint16'>, tiledb_store_name='counts')

__match_args__ = ('skip', 'consolidate_duplicate_gene_func', 'matrix_name', 'matrix_attr_name', 'dtype', 'tiledb_store_name')

__repr__()

Return repr(self).

consolidate_duplicate_gene_func(iterable, /, start=0)

Return the sum of a ‘start’ value (default: 0) plus an iterable of numbers

When the iterable is empty, return the start value. This function is intended specifically for use with numeric values and may reject non-numeric types.

dtype

alias of uint16

matrix_attr_name: str = 'data'

matrix_name: str = 'counts'

skip: bool = False

tiledb_store_name: str = 'counts'

class cellarr.build.build_options.SampleMetadataOptions(skip=False, dtype=<class 'numpy.uint32'>, tiledb_store_name='sample_metadata', column_types=None)

Bases: object

Optional arguments for the sample store for build_cellarrdataset().

skip

Whether to skip generating sample TileDB. Defaults to False.

dtype

NumPy dtype for the sample dimension. Defaults to np.uint32.

Note: make sure the number of samples fit within the integer limits of unsigned-int32.

tiledb_store_name

Name of the TileDB file. Defaults to “sample_metadata”.

column_types

A dictionary containing column names as keys and the value representing the type to in the TileDB.

If None, all columns are cast as ‘ascii’.

__annotations__ = {'column_types': typing.Dict[str, numpy.dtype], 'dtype': <class 'numpy.dtype'>, 'skip': <class 'bool'>, 'tiledb_store_name': <class 'str'>}

__dataclass_fields__ = {'column_types': Field(name='column_types',type=typing.Dict[str, numpy.dtype],default=None,default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'dtype': Field(name='dtype',type=<class 'numpy.dtype'>,default=<class 'numpy.uint32'>,default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'skip': Field(name='skip',type=<class 'bool'>,default=False,default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD), 'tiledb_store_name': Field(name='tiledb_store_name',type=<class 'str'>,default='sample_metadata',default_factory=<dataclasses._MISSING_TYPE object>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),kw_only=False,_field_type=_FIELD)}

__dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,order=False,unsafe_hash=False,frozen=False)

__eq__(other)

Return self==value.

__hash__ = None

__init__(skip=False, dtype=<class 'numpy.uint32'>, tiledb_store_name='sample_metadata', column_types=None)

__match_args__ = ('skip', 'dtype', 'tiledb_store_name', 'column_types')

__repr__()

Return repr(self).

column_types: Dict[str, dtype] = None

dtype

alias of uint32

skip: bool = False

tiledb_store_name: str = 'sample_metadata'

cellarr.build.buildutils_tiledb_array module

cellarr.build.buildutils_tiledb_array.create_group(output_path, group_name)

cellarr.build.buildutils_tiledb_array.create_tiledb_array(tiledb_uri_path, x_dim_length=None, y_dim_length=None, x_dim_name='cell_index', y_dim_name='gene_index', matrix_attr_name='data', x_dim_dtype=<class 'numpy.uint32'>, y_dim_dtype=<class 'numpy.uint32'>, matrix_dim_dtype=<class 'numpy.uint32'>, is_sparse=True)

Create a TileDB file with the provided attributes to persistent storage.

This will materialize the array directory and all related schema files.

Parameters:

• tiledb_uri_path (str) – Path to create the array TileDB file.

• x_dim_length (int) – Number of entries along the x/fastest-changing dimension. e.g. Number of cells. Defaults to None, in which case, the max integer value of x_dim_dtype is used.

• y_dim_length (int) – Number of entries along the y dimension. e.g. Number of genes. Defaults to None, in which case, the max integer value of y_dim_dtype is used.

• x_dim_name (str) – Name for the x-dimension. Defaults to “cell_index”.

• y_dim_name (str) – Name for the y-dimension. Defaults to “gene_index”.

• matrix_attr_name (str) – Name for the attribute in the array. Defaults to “data”.

• x_dim_dtype (dtype) – NumPy dtype for the x-dimension. Defaults to np.uint32.

• y_dim_dtype (dtype) – NumPy dtype for the y-dimension. Defaults to np.uint32.

• matrix_dim_dtype (dtype) – NumPy dtype for the values in the matrix. Defaults to np.uint32.

• is_sparse (bool) – Whether the matrix is sparse. Defaults to True.

cellarr.build.buildutils_tiledb_array.optimize_tiledb_array(tiledb_array_uri, verbose=True)

Consolidate TileDB fragments.

cellarr.build.buildutils_tiledb_array.write_csr_matrix_to_tiledb(tiledb_array_uri, matrix, value_dtype=<class 'numpy.uint32'>, row_offset=0, batch_size=25000)

Append and save a csr_matrix to TileDB.

Parameters:

• tiledb_array_uri (Union[str, SparseArray]) – TileDB array object or path to a TileDB object.

• matrix (csr_matrix) – Input matrix to write to TileDB, must be a csr_matrix matrix.

• value_dtype (dtype) – NumPy dtype to reformat the matrix values. Defaults to uint32.

• row_offset (int) – Offset row number to append to matrix. Defaults to 0.

• batch_size (int) – Batch size. Defaults to 25000.

cellarr.build.buildutils_tiledb_frame module

cellarr.build.buildutils_tiledb_frame.append_to_tiledb_frame(tiledb_uri_path, frame, row_offset=0)

Create a TileDB file with the provided attributes to persistent storage.

This will materialize the array directory and all related schema files.

Parameters:

• tiledb_uri_path (str) – Path to create the metadata TileDB file.

• frame (DataFrame) – Pandas Dataframe to append to TileDB.

• row_offset (int) – Row offset to append new rows to. Defaults to 0.

cellarr.build.buildutils_tiledb_frame.create_tiledb_frame_from_chunk(tiledb_uri_path, chunk, column_types)

Create a TileDB file from the DataFrame chunk, to persistent storage. This is used by the importer for large datasets stored in csv.

This will materialize the array directory and all related schema files.

Parameters:

• tiledb_uri_path (str) – Path to create the metadata TileDB file.

• chunk (DataFrame) – Pandas data frame.

• column_types (Dict[str, dtype]) – Dictionary specifying the column types for each column in the frame.

cellarr.build.buildutils_tiledb_frame.create_tiledb_frame_from_column_names(tiledb_uri_path, column_names, column_types)

Create a TileDB file with the provided attributes to persistent storage.

This will materialize the array directory and all related schema files.

Parameters:

• tiledb_uri_path (str) – Path to create the metadata TileDB file.

• column_names (List[str]) – Column names of the data frame.

• column_types (Dict[str, dtype]) – Dictionary specifying the column types for each column in the frame.

cellarr.build.buildutils_tiledb_frame.create_tiledb_frame_from_dataframe(tiledb_uri_path, frame, column_types=None)

Create a TileDB file with the provided attributes to persistent storage.

This will materialize the array directory and all related schema files.

Parameters:

• tiledb_uri_path (str) – Path to create the metadata TileDB file.

• column_names – Column names of the data frame.

• column_types (dict) – Dictionary specifying the column types for each column in the frame.

cellarr.build.buildutils_tiledb_frame.infer_column_types(frame, col_types)

Infer column types based on pandas types for each column.

Note: Currently sets all columns to ‘ascii’.

Parameters:

frame (DataFrame) – DataFrame to infer column types from.

Return type:

Dict[str, str]

Returns:

Dictionary containing column names as keys and value representing the column types.

Module contents