storage sub-package

storage sub-package#

Methods to interact with the user storage resources.

class ExportDestination(*values)#

Bases: Enum

AWS_S3 = 'aws-s3'#
class StorageOperation(api, id_)#

Bases: object

Represents a storage operation running remotely via Inductiva API.

__init__(api, id_)#
classmethod from_api_response(api, response)#
wait(poll_s: int = 2)#

Wait for the operation to complete.

Parameters:

poll_s – Time in seconds between calls to the API to update the status of the operation.

class ZipArchiveInfo(size: int, files: List[ZipFileInfo])#

Bases: object

Represents the total ZIP size and file contents of a ZIP archive.

__init__(size: int, files: List[ZipFileInfo]) → None#
files: List[ZipFileInfo]#
size: int#
class ZipFileInfo(name: str, size: int, compressed_size: int, range_start: int | None, compress_type: int | None)#

Bases: object

Represents information about a file within a ZIP archive.

__init__(name: str, size: int, compressed_size: int, range_start: int | None, compress_type: int | None) → None#
compress_type: int | None#
compressed_size: int#
name: str#
range_start: int | None#
size: int#
download(remote_path: str, local_dir: str = '', decompress: bool = True)#

Downloads a file or folder from storage to a local directory, optionally decompressing the contents.

Parameters:

  • remote_path (str) – The path of the file or folder on the remote server to download.

  • local_dir (str, optional) – The local directory where the file or folder will be saved. Defaults to the current working directory.

  • decompress (bool, optional) – Whether to decompress the downloaded file or folder if it is compressed. Defaults to True.

Examples

Download a folder from a remote server to the current directory:

inductiva.storage.download(remote_path="/path/to/remote/folder/")

Download a file and save it to a local directory without decompressing:

inductiva.storage.download(remote_path="/path/to/remote/file.zip",
                           local_dir="/local/directory",
                           decompress=False)

Download a file inside a zip archive:

inductiva.storage.download(
    remote_path="/some_task_id/output.zip/stdout.txt"
)

Note

It is not possible to download folders that are inside zip archives.

export(path_to_export: str, export_to: ExportDestination, bucket_name: str, file_name: str | None = None, part_size: int = 128)#
export_to_aws_s3(path_to_export, part_size, filename, bucket_name)#
get_signed_urls(paths: List[str], operation: Literal['upload', 'download']) → List[str]#
get_space_used()#

Returns the occupied storage size in GB.

get_zip_contents(path: str, zip_relative_path: str = '') → ZipArchiveInfo#

Retrieve the contents of a ZIP archive from a given path.

Parameters:

  • path (str) – The full path to the ZIP archive.

  • zip_relative_path (str, optional) – A relative path inside the ZIP archive to filter the contents. Defaults to an empty string, which lists all files within the archive.

Returns:

An object containing the total size of the ZIP archive

and a list of ZipFileInfo objects representing the files within the specified ZIP archive.

Return type:

ZipArchiveInfo

listdir(path='/', max_results: int = 10, order_by: Literal['size', 'creation_time'] = 'creation_time', sort_order: Literal['asc', 'desc'] = 'desc', print_results: bool = True)#

List and display the contents of the user’s storage. :param path: Storage directory to list. Default is root. :type path: str :param max_results: The maximum number of results to return. :type max_results: int :param order_by: The field to sort the contents by. :type order_by: str :param sort_order: Whether to sort the contents in ascending or :type sort_order: str :param descending order.:

Returns:

A list of dictionaries containing information about the size, the name and the creation time of each content that can easily be converted to a dataframe.

Return type:

list of dict

This function prints a table with the storage content information:

Name Size Creation Time 1234 5.68 MiB 29 Sep, 14:12:00 12345 374.85 KiB 29 Sep, 14:13:10 1234567 97.59 KiB 29 Sep, 14:13:24 123 0 B 29 Sep, 14:13:29 123456 0 B 29 Sep, 14:13:18

You can use this information to delete the contents you don’t need anymore and further inspect task outputs and logs using the Task class.

multipart_upload(path, parts_size, upload_parts, complete_multipart_url)#

Perform the multipart upload using the server.

remove_workspace(remote_dir) → bool#

Removes path from a remote directory.

Parameters: - remote_dir (str): The path to the remote directory.

upload(local_path: str, remote_dir: str)#

Upload a local file or directory to a specified remote directory.

Parameters:

  • local_path (str) – The path to the local file or directory to be uploaded.

  • remote_dir (str) – The remote directory where the file will be uploaded.

Example

Upload a file to a remote directory:

inductiva.storage.upload('local/path/file.txt', 'my_data')

Upload a directory to a remote location:

inductiva.storage.upload('local/path/folder', 'my_data')
upload_from_url(url: str, remote_dir: str, file_name: str | None = None)#

Upload a file from a given URL to a specified remote directory.

If no file name is provided, the function extracts the name from the URL.

Parameters:

  • url (str) – The URL of the file to be uploaded.

  • remote_dir (str) – The path to the remote directory where the file will be stored.

  • file_name (str, optional) – The name to save the uploaded file as. If not specified, the name will be extracted from the URL.

Contents