Datasets

This page explains how to work with Synativ Datasets. Synativ enables customers to fine-tune foundation models on their own datasets.

Create Dataset

Creating a Dataset will register and upload your dataset with Synativ. When you create a Synativ Dataset, you receive a DatasetId. You can use this DatasetId for fine-tuning or inference later.

You can create a Dataset by calling create_dataset:

synativ_api.create_dataset(
    dataset_name='my_dataset',
    dataset_dir='<local_path_of_your_dataset>',
)

Your local dataset_dir will be zipped automatically before uploading. You can give your dataset a friendly name of choice.

This method will return something like:

Registered a new Synativ Dataset: synativ-dataset-a2be9b2b-15a0-46e8-bc85-59c52fbdf522
Uploading <local_path_of_your_dataset> to synativ-dataset-a2be9b2b-15a0-46e8-bc85-59c52fbdf522

You will receive a Dataset object as response:

Dataset(
    creation_time='2023-09-11 09:02:00.285938',
    name='my_dataset',
    id='synativ-dataset-a2be9b2b-15a0-46e8-bc85-59c52fbdf522'
)

List Datasets

You can list your existing Datasets by calling list_datasets:

synativ_api.list_datasets()

You will receive a list of datasets that you have registered before as response:

ListDatasetsResponse(
	datasets=[
        Dataset(
            creation_time='2023-09-11 09:02:00.285938',
            name='my_dataset_1',
            id='synativ-dataset-a2be9b2b-15a0-46e8-bc85-59c52fbdf522'
        ),
		Dataset(
            creation_time='2023-09-15 14:22:39.029121',
            name='my_dataset_2',
            id='synativ-dataset-569b573b-c5d2-4f1a-bdc8-5235f74fa6c4'
        ),
		Dataset(
            creation_time='2023-09-19 15:33:08.601124',
            name='my_dataset_3',
            id='synativ-dataset-b269b4fb-02d1-4ddd-b3f3-f5536ff57089'
        )
	]
)

Get Dataset

You can get the details of a specific dataset by calling get_dataset with the respective DatasetId:

synativ_api.get_dataset(
    dataset_id="synativ-dataset-a2be9b2b-15a0-46e8-bc85-59c52fbdf522"
)

You will receive a Dataset object as response:

Dataset(
    creation_time='2023-09-11 09:02:00.285938',
    name='my_dataset',
    id='synativ-dataset-a2be9b2b-15a0-46e8-bc85-59c52fbdf522'
)

Delete Dataset

You can always delete a Dataset by calling delete_dataset with the respective DatasetId:

synativ_api.delete_dataset(
    dataset_id="synativ-dataset-a2be9b2b-15a0-46e8-bc85-59c52fbdf522"
)

Deleting a Dataset will completely delete all records and files of this dataset from our servers. We will have no way, whatsoever, to retrieve it.