Quickstart

This quickstart will walk you through:
The key concepts and goals of Aquarium
Getting data into Aquarium, starting with a standard open dataset
Providing your own data, including strategies for securely accessing data
Recommendations for working with your different machine learning tasks
At the end, you should be all ready to start working with your own datasets and models!
Account Setup and Team Onboarding
Initial Onboarding for Your Organization
Reach out to Aquarium via slack or email (pgao at aquarium-learn.com) and we will:
Configure a new tenant for your organization
Provide initial user accounts.
Inviting your Team Members
Anyone in your organization can self-serve invite additional members to your team.
While logged in to the app, click on your username in the top right of the screen
Click Organization Settings -> Invite User
Enter the user's email address and they will receive a password reset prompt to their email and can create an account.
Provisioning API Keys
Once you've successfully logged in to the app and invited your team members, you should provision a key to interact with the API and upload datasets.
While logged in to the app, click on your username in the top right of the screen
Click User Settings -> Generate New API Key
Toy Dog Dataset Ingestion
To highlight the core concepts and interactions, we're going to work with an open source dataset and computer vision task: classifying pet breeds from a photo. We'll be working with the oxford-iiit-pets dataset, which contains 6000 labeled images.
# Overall data structure
├── imgs
│   ├── Abyssinian_100.jpg
│   ├── american_bulldog_125.jpg
│   └── american_pit_bull_terrier_191.jpg
├── inferences.json
└── labels.json
​
# All images are mirrored online at
# https://storage.googleapis.com/aquarium-public/quickstart/pets/imgs/<filename>.jpg
​
# Format of labels.json
[
  {
    "file_name": "Sphynx_158.jpg",
    "class_id": 33,
    "species_id": 0,
    "class_name": "sphynx",
    "species_name": "cat",
    "split_name": "train"
  },
  ...
]
​
# Format of inferences.json
[
  {
    "confidence": 1,
    "frame_id": "Sphynx_158",
    "class_id": 33,
    "class_name": "sphynx"
  },
  ...
]
You can download the dataset (231MB) at this link.
Python Client Library
The aquariumlearning package requires Python >= 3.6.
Aquarium provides a python client library to simplify integration into your existing ML data workflows. In addition to wrapping API requests, it also handles common needs such as efficiently encoding uploaded data or using disk space to work with datasets larger than available system memory.
!pip install aquariumlearning
​
import aquariumlearning as al
al_client = al.Client()
al_client.set_credentials(api_key=YOUR_API_KEY)
Projects
Projects are the highest level grouping in Aquarium, and can be compared to "spaces" in other systems. A project is expected to contain data for the same core task -- in this case, pet breed detection.
To create a project, we specify a name, what the full set of valid classifications are, and optionally hints about the primary task being performed.
import string
import random
import json
​
# Project names are globally unique namespaces, similar to
# a bucket name in S3 or GCS. For this quickstart, we make sure
# we create a unique project name.
random_str = ''.join(random.choices(string.ascii_lowercase, k=8))
AL_PROJECT = 'pets_quickstart_{}'.format(random_str)
​
with open('./classnames.json') as f:
    classnames = json.load(f)
​
al_client.create_project(
    AL_PROJECT, 
    al.LabelClassMap.from_classnames(classnames), 
    primary_task="CLASSIFICATION"
)
Labeled Datasets
Often just called "datasets" for short, these are versioned snapshots of input data and ground truth labels. They belong to a Project, and consist of multiple Labeled Frames. 
A Labeled Frame is one logical "frame" of data, such as an image from a camera stream. They can contain one or more media/sensor inputs, zero or more ground truth labels, and arbitrary user provided metadata.
with open('./labels.json') as f:
    label_entries = json.load(f)
​
dataset = al.LabeledDataset()
for entry in label_entries:
    # Create a frame object, using the filename as an id
    frame_id = entry['file_name'].split('.jpg')[0]
    frame = al.LabeledFrame(frame_id=frame_id)
​
    # Add arbitrary metadata, such as the train vs test split
    frame.add_user_metadata('split_name', entry['split_name'])
    
    # You can also add arbitrary metadata as a list, such as a tag list
    frame.add_user_metadata_list('data_tags', ['pets', entry['species_name']])
​
    # Add an image to the frame
    image_url = "https://storage.googleapis.com/aquarium-public/quickstart/pets/imgs/" + entry['file_name']
    frame.add_image(image_url=image_url)
​
    # Add the ground truth classification label to the frame
    label_id = frame_id + '_gt'
    frame.add_label_2d_classification(
        label_id=label_id, 
        classification=entry['class_name']
    )
​
    # Add the frame to the dataset collection
    dataset.add_frame(frame)
Submit the Dataset!
Now we have everything all set up, and let’s submit it! Aquarium does some processing and analysis of datasets when they’re submitted, so this process will take several minutes. 
To spot check our data immediately, we can set the preview_first_frame flag to True, letting us confirm that it looks right before we get started.
<TODO: Insert image>
Looks good! Let's press enter and kick off the full job.
AL_DATASET = 'fullset_0'
al_client.create_dataset(
    AL_PROJECT, 
    AL_DATASET, 
    dataset=dataset, 
    # Poll for completion of the processing job
    wait_until_finish=True, 
    # Preview the first frame before submission to catch mistakes
    preview_first_frame=True
)
Once that's all done, we can go and see it in the app:
<TODO: Insert image>
Inferences
Before we move on from the code, let's also get those sample model inferences in there. Like datasets, Inference sets are immutable snapshots made up of Inference Frames, which contain inferred values (in this case, the predicted pet breed).
with open('./inferences.json') as f:
    inference_entries = json.load(f)
​
inferences = al.Inferences()
for entry in inference_entries:
    # Create a frame object, using the same id
    frame_id = entry['frame_id']
    inf_frame = al.InferencesFrame(frame_id=frame_id)
​
    # Add the inferred classification label to the frame
    inf_label_id = frame_id + '_inf'
    inf_frame.add_inference_2d_classification(
        label_id=inf_label_id, 
        classification=entry['class_name'],
        confidence=entry['confidence']
    )
​
    # Add the frame to the inferences collection
    inferences.add_frame(inf_frame)
We can then submit it in much the same way.
al_client.create_inferences(
    AL_PROJECT, 
    AL_DATASET, 
    inferences=inferences, 
    inferences_id='demo_model_1', 
    wait_until_finish=True
)
And there we go! We've taken an open-source dataset and loaded its labels, metadata, and a sample set of inferences into Aquarium. Feel free to poke around the data in the app now, or move on to getting your own data in.
Your data!
Ok, we worked through a small set of toy data, but your situation is more complex -- you do instance segmentation on secure imagery with some domain-specific evaluation criteria, or some equally tricky setup. In this section, we'll walk through our most common data sharing schemes, along with common data and label formats.
If you don’t see your setup here, don’t fret -- we support all sorts of ML workflows, not just those in this tutorial.
Data Sharing
Before anything else, we should figure out data sharing, since you probably aren't working with public images of puppies. Aquarium offers several easy ways to securely work with your data assets on our platform, from re-hosting it with us through locked down data that we won’t ever get to touch. If you don't see a solution here, reach out -- we’ve worked with many enterprise IT teams on unique security schemes too.
Options with an asterisk (*) will only allow your users to view raw data (images, point clouds, etc.) and will never make them accessible to Aquarium's servers.
This increased data privacy does mean that some features around clustering and similarity search will require users to provide their own "image embeddings," as we won't be able to compute them for you.
Public URLs
 Aquarium Hosted
Aquarium Mirrored
Local Bucket Credentials*
Network Restricted*
Local Files*
Your data is free to share publicly.
This is the easiest, assuming you're working on data without any security restrictions. Anywhere you need to provide a URL to an asset, just provide any URL that's accessible on the public internet. For example, the above toy dataset example uses public URLs like the following:
image_url = "https://storage.googleapis.com/aquarium-public/quickstart/pets/imgs/Abyssinian_1.jpg"
Your data needs to be kept secure, and it's ok for Aquarium to host a copy. 
By allowing Aquarium to issue you a storage bucket and host your data, it allows Aquarium to manage a lot of the annoying details that go into a producing a snappy user experience. Browser cache control headers, CORS settings, user access credentials -- we'll handle all of these for you. 
We'll issue you a Google Cloud Storage bucket of the form gs://aquarium-customer-abcd1234/ and a Google Cloud service account with read/write permissions for that bucket. If your organization uses Google Cloud, we can also directly grant permissions to your admin users. 
To upload data:
# Install gsutil, google cloud's storage CLI utility
# https://cloud.google.com/storage/docs/gsutil_install#sdk-install
​
gsutil --version
​
# Activate the service account credentials
# https://cloud.google.com/sdk/gcloud/reference/auth/activate-service-account
​
gcloud auth activate-service-account \
    [email protected] \
    --key-file /path/to/credentials.json
​
# Copy directories up! This command will recursively copy a directory
# Feel free to check out the docs for more options:
#    https://cloud.google.com/storage/docs/gsutil/commands/cp
​
gsutil -m rsync -r \
    ./projectA/imgs/ gs://aquarium-customer-abcd1234/projectA/imgs/
​
To reference the data:
image_url = "https://storage.cloud.google.com/aquarium-customer-abcd1234/path/to/image.png"
​
Your data needs to be kept secure, and it's ok for Aquarium to mirror a copy. 
By signing data urls so that they can be accessed by Aquarium, we can mirror these assets in a secure bucket without the need of uploading your images into an Aquarium bucket directly. 
To upload:
 frame.add_image(   
    sensor_id='camera_1',
    image_url='',
    # Flag that tells Aquarium to mirror the image
    mirror_asset=True,
)
The mirror_asset flag is available for all Data Formats seen below.
Your data is in a private storage bucket, and you don't want Aquarium to ever access the raw data.
Because the underlying raw data is only needed for visualization purposes, you can provide bucket paths that point to secure resources. Then, when your users want to view them in the application, they can use local credentials to view the data. Neither the credentials nor the data will ever leave your users' browser / local device.
To reference the data, simply use the bucket path as the data url:
image_url = "s3://yourbucket/path/to/img.jpg"
image_url = "gs://yourbucket/path/to/img.jpg"
Then, each user can go to https://illume.aquariumlearning.com/settings/user to point to a local credentials file. This file is expected to match the formats provided by your cloud provider's admin/IAM console. Reach out if you have questions about the format!
Local device credentials files
Doing this securely requires modern browser capabilities that aren't available in all browsers yet. We recommend upgrading to the latest version of Google Chrome or Microsoft Edge if you want to use this data sharing scheme.
Your team locks down resources through a corporate VPN.
Because the underlying raw data is only needed for visualization purposes, you can provide URLs that will only resolve for users on your corporate network. Because your users will be accessing Aquarium on that network, they'll be able to access those resources without them being accessible to Aquarium's servers. 
The best option depends on your specific infrastructure setup, but common options include:
An image server / URL signing service only accessible on your network.
Bucket-level IP allow permissions for your VPN's known IP Addresses.
If you're planning on this path, reach out to the Aquarium team and we'd be happy to chat with your IT/Infra team to find the easiest solution!
You just want to work with files on your file system (or a network mounted storage device that's like your file system).
Aquarium's only requirement is that images are available via an HTTP request from the browser, which means users can host data with a simple local file server. There are countless ways to do this, and here are some easy ways with python:
If you're working with semantic segmentation or point cloud data, you'll need to use a local file server that supports Cross-Origin Resource Sharing (CORS). The recommended NPM package below supports it as written. 
If you are unable to use an NPM package, please reach out and we can get you set up.
# Python 3
python3 -m http.server 5000
​
# Python 2
python -m SimpleHTTPServer 5000
​
# NPM
npx http-server --cors='*' --port=5000
To reference the data:
image_url = "http://localhost:5000/path/to/img/.jpg"
Project Configuration
In Aquarium, a project is a space that contains datasets that are all for the same machine learning task. For example, a company might have one model doing classification of aerial imagery and another model doing classification of social media photos. These would be two different projects in Aquarium.
Projects are defined by two properties: their label class map and their primary task.
Label Class Maps
Your project predicts labels for things. At its most basic, a label class map defines how to map between integer ids, display name strings, and colors for rendering. At its most complex, it can also contain information about how to map between ground truth classes and inferred classes, and how each class should be interpreted when computing metrics. 
I just have classes
I care about colors
My inferred classes != ground truth classes
I want to change my colors
If you just have a list of classes with no strong opinions on presentation, we make that easy:
CLASSNAMES = ['dog', 'cat', 'horse', 'hamster']
# Attempts to assign presentational colors intelligently
label_class_map = al.LabelClassMap.from_classnames(CLASSNAMES)
If you have specific requirements around presentation, such as consistent coloring with your product, you can also explicitly define your class map and its colors.
CLASSNAMES = ['dog', 'cat', 'horse', 'hamster']
# Explicitly specify handling of each class
label_class_map = al.LabelClassMap(entries=[
    al.ClassMapEntry(
        name='dog',
        class_id=0,
        color=(255,0,0)
    ),
    al.ClassMapEntry(
        name='cat',
        class_id=1,
        color=(0,255,0)
    ),
    al.ClassMapEntry(
        name='horse',
        class_id=2,
        color=(0,0,255)
    ),
    al.ClassMapEntry(
        name='hamster',
        class_id=3,
        color=(0,128,128)
    )
])
Aquarium also allows you to express more complex relationships between your ground truth classes and inferred classes. The following example aims to highlight more complex configurations.
If you don't see how to model your setup, please reach out!
​
# Explicitly specify handling of each class
#
# Here, we have three pet GT classes, which get
# predicted as either large or small pet. We also have
# an "ignore" class which may exist in the labels,
# but should be ignored in evaluation without a
# corresponding predicted label. 
​
label_class_map = al.LabelClassMap(entries=[
    al.ClassMapEntry(
        name='dog',
        class_id=0,
        color=(255,0,0),
        train_name='small_pet',
        train_id=0,
        train_color=(200,200,0)
    ),
    al.ClassMapEntry(
        name='cat',
        class_id=1,
        color=(0,255,0),
        train_name='small_pet',
        train_id=0,
        train_color=(200,200,0)
    ),
    al.ClassMapEntry(
        name='horse',
        class_id=2,
        color=(0,0,255),
        train_name='large_pet',
        train_id=1,
        train_color=(0,0,200)
    ),
    al.ClassMapEntry(
        name='ignore',
        class_id=255,
        color=(0,0,0),
        train_name="",
        train_id=255,
        ignore_in_eval=True
    )
])
Aquarium allows you to change the class colors for a project after they have been created as well.
# You can refer to a specific label by id or by name
label_class_changes = [
    al.ClassMapUpdateEntry(class_id=5, color=(0, 255, 255)),
    al.ClassMapUpdateEntry(name='jellyfish', train_color=(255, 0, 255))
]
​
al_client.update_label_class_map_colors(AL_PROJECT, label_class_changes)
Primary Tasks
Deep learning tasks vary quite a bit, and the ideal presentation depends on the specific task. Projects can specify a "primary task", which will be used to fine tune the UI for that type of task. 
Object Detection
Classification
Semantic Segmentation
Multi-Label Classification
Object detection is our default view, and currently requires no configuration.
al_client.create_project(AL_PROJECT, class_map)
Your task is fundamentally a classification task, where each data point has exactly one ground truth label, and one predicted label. 
al_client.create_project(AL_PROJECT, class_map, primary_task="CLASSIFICATION")
Your task is a 2D semantic segmentation task, where you have per-pixel labels.
al_client.create_project(AL_PROJECT, class_map, primary_task="2D_SEMSEG")
Your task is fundamentally a classification task, where each data point has zero or more labels, and can be predicted as zero or more classes.
al_client.create_project(AL_PROJECT, class_map, primary_task="MULTI_LABEL_CLASSIFICATION")
Data Formats
Each labeled frame in your dataset can contain one or more input pieces of data. In a many computer vision tasks, this may be a single image. In a robotics or self-driving task, this may be a full suite of camera images, lidar point clouds, and radar scans. 
Here are some common data types, their expected formats, and how to work with them in Aquarium:
Image
Metadata
Geospatial
Audio
Point Cloud
3D Geometry
3D Coordinate Frames
​
frame.add_image(
    # A URL to load the image by
    image_url='',
    # A URL to a compressed form of the image for faster loading in browsers.
    # It must be the same pixel dimensions as the original image.
    preview_url='',
    # Optional: ISO formatted date-time string
    date_captured='',
    # Optional: width of image in pixels, will be inferred otherwise
    width=1280,
    # Optional: height of image in pixels, will be inferred otherwise
    height=720
)
​
​
frame.add_user_metadata('deployment_id', value)
​
# In the case of nullable values, you can also provide an explicit type
frame.add_user_metadata('nullable_field', maybe_null, val_type='int')
Metadata is indexed on Aquarium's side, so make sure you aren't sharing any private information.
​
# EPSG:4326 WGS84 Latitute Longitude coordinates
frame.add_geo_latlong_data(lat, lon)
Geospatial metadata is indexed on Aquarium's side, so make sure you aren't sharing any private information.
Aquarium supports audio files that are natively playable in browsers. For maximum compatibility, we recommend providing .mp3 files.
If your model also works against spectrograms, you can provide both an audio data input and an image. The Aquarium UI will then present both alongside each other.
frame.add_audio(
    # A URL to load the mp3 file from
    audio_url='',
    # Optional: ISO formatted date-time string
    date_captured=''
)
​
Unfortunately, point cloud formats haven't standardized as much as image data yet. These are our currently supported formats, but please reach out if you have a different representation. We'd be more than happy to support your in-house representation. 
PCL / PCD
Aquarium supports the *.pcd file format used by the PCL library, including the binary and compressed binary encodings. Numeric values for the following column names are expected: x, y, z, intensity (optional), range (optional).
frame.add_point_cloud_pcd(
    pcd_url='',
    # Optional: If your point cloud is relative to a specific
    # coordinate frame, you can reference it by name here.
    coord_frame_id="",
    # Optional: ISO formatted date-time string
    date_captured=''
)
KITTI-like binary files
Similar to the raw KITTI lidar formats, we can also take in raw, dense binary files of little-endian values. This is in many ways more fragile, but also requires no third party libraries. 
frame.add_point_cloud_bins(
    # URL for the point positions: 
    #    float32 [x1, y1, z1, x2, y2, z2, ...]
    point_cloud_url='',
    # URL for the point intensities:
    #    unsigned int32 [i1, i2, i3, ...]
    intensity_url='',
    # URL for the point ranges
    #    float32 [r1, r2, r3, ...]
    range_url'',
    # Optional: If your point cloud is relative to a specific
    # coordinate frame, you can reference it by name here.
    coord_frame_id="",
    # Optional: ISO formatted date-time string
    date_captured=''
)
​
Aquarium supports rendering basic 3D geometry meshes. Please reach out if you have any needs that aren't captured here.
.OBJ (Wavefront) Files
frame.add_obj(
    # A url to a *.obj formatted text
    obj_url='',
    # Optional: If your object geometry is relative to a specific
    # coordinate frame, you can reference it by name here.
    coord_frame_id="",
    # Optional: ISO formatted date-time string
    date_captured=''
)
​
In robotics applications, you often have multiple sensors in multiple coordinate frames. Aquarium supports specifying different coordinate frames, which will be used when interpreting 3D data inputs and labels. 
frame.add_coordinate_frame_3d(
    coord_frame_id='robot_ego_frame',
    # Position offset of this coordinate frame
    position={'x': 0, 'y': 0, 'z': 0},
    # Rotation/Orientation of this coordinate frame,
    # represented as a quaternion
    orientation={'w': 1, 'x': 0, 'y': 0, 'z': 0},
    # Optional: string ID of the parent coordinate frame
    # that this one is relative to.
    parent_frame_id=""
)
​
Label Formats
Each labeled frame in your dataset can contain zero or more ground truth labels. These vary from task to task, and may occasionally be combined together. 
Here are some common label types, their expected formats, and how to work with them in Aquarium:
Classification
2D Bounding Box
3D Cuboid
2D Semseg
2D Polygon Lists
​
# Standard 2D case
frame.add_label_2d_classification(
    # A unique id across all other labels in this dataset
    label_id='unique_id_for_this_label',
    classification='dog'
)
​
# 3D classification
frame.add_label_3d_classification(
    # A unique id across all other labels in this dataset
    label_id='unique_id_for_this_label',
    classification='dog',        
    # Optional, defaults to implicit WORLD coordinate frame
    coord_frame_id='robot_ego_frame',
)
​
​
frame.add_label_2d_bbox(
    # A unique id across all other labels in this dataset
    label_id='unique_id_for_this_label',
    classification='dog',
    # Coordinates are in absolute pixel space
    top=200,
    left=300,
    width=250,
    height=150
)
​
Aquarium supports 3D cuboid labels, with 6-DOF position and orientation.
frame.add_label_3d_cuboid(
    label_id="unique_id_for_this_label",
    classification="car",
    # XYZ dimensions of this cuboid
    dimensions=[1.0, 0.5, 0.5],
    # XYZ position of the center of this object
    position=[2.0, 2.0, 1.0],
    # An XYZW ordered object rotation quaternion
    rotation=[0.0, 0.0, 0.0, 1.0],
    # Optional: If your cuboid is relative to a specific
    # coordinate frame, you can reference it by name here.
    coord_frame_id="robot_ego_frame"
)
​
3D Cuboid Image Projection (Optional)
Example of 3D cuboids projected into image
If you have images attached to your frames alongside 3D cuboids, we can project the cuboids onto the images! First you will need to add a 2D coordinate frame:
frame.add_coordinate_frame_2d(
    # String identifier for this coordinate frame.
    coord_frame_id="camera_1_coordinate_frame",
    # focal length x in pixels.
    fx=fx,
    # focal length y in pixels.
    fy=fy,
    # Optional: Either "fisheye" for the fisheye model,
    # or "brown_conrady" for the pinhole model with
    # Brown-Conrady distortion.
    camera_model=camera_model
    # Optional: Dict of the form {x, y, z}.
    position=position,
    # Optional: Quaternion rotation dict of the form {w, x, y, z}.
    orientation=orientation,
    # Optional: 4x4 row major order camera matrix mapping
    # 3d world space to camera space (x right, y down, z forward).
    # Keep in mind, if you pass in the camera matrix it will stack
    # on top of the position/orientation you pass in as well. This
    # is only needed if you cannot properly represent your camera
    # using the position/orientation parameters.
    camera_matrix=camera_matrix,
    # Optional: optical center pixel x coordinate.
    cx=cx,
    # Optional: optical center pixel y coordinate.
    cy=cy,
    # Optional: k1 radial distortion coefficient (Brown-Conrady, fisheye).
    k1=k1,
    # Optional: k2 radial distortion coefficient (Brown-Conrady, fisheye).
    k2=k2,
    # Optional: k3 radial distortion coefficient (Brown-Conrady, fisheye).
    k3-k3,
    # Optional: k4 radial distortion coefficient (Brown-Conrady, fisheye).
    k4=k4,
    # Optional: k5 radial distortion coefficient (Brown-Conrady).
    k5=k5,
    # Optional: k6 radial distortion coefficient (Brown-Conrady).
    k6=k6,
    # Optional: p1 tangential distortion coefficient (Brown-Conrady).
    p1=p1,
    # Optional: p2 tangential distortion coefficient (Brown-Conrady).
    p2=p2,
    # Optional: s1 thin prism distortion coefficient (Brown-Conrady).
    s1=s1,
    # Optional: s2 thin prism distortion coefficient (Brown-Conrady).
    s2=s2,
    # Optional: s3 thin prism distortion coefficient (Brown-Conrady).
    s3=s3,
    # Optional: s4 thin prism distortion coefficient (Brown-Conrady).
    s4=s4,
    # Optional: camera skew coefficient (fisheye).
    skew=skew,
    # Optional: String id of the parent coordinate frame.
    parent_frame_id=parent_frame_id
)
View the Python API docs for more information on the parameters to add_coordinate_frame_2d
And then you can use this 2D coordinate frame when adding your image to the frame:
frame.add_image(
    # A unique name to refer to this image by
    sensor_id='camera_1',
    # A URL to load the image by
    image_url='',
    # A URL to a compressed form of the image for faster loading in browsers.
    # It must be the same pixel dimensions as the original image.
    preview_url='',
    # Optional: ISO formatted date-time string
    date_captured='',
    # Optional: width of image in pixels, will be inferred otherwise
    width=1280,
    # Optional: height of image in pixels, will be inferred otherwise
    height=720
    # Optional: 2D coordinate frame to use for this image
    coord_frame_id="camera_1_coordinate_frame",
)
Now you will be able to see your 3D cuboids projected onto your images and have camera distortion properly accounted for!
2D Semantic Segmentation labels are represented by an image mask, where each pixel is assigned an integer value in the range of [0,255]. For efficient representation across both servers and browsers, Aquarium expects label masks to be encoded as grey-scale PNGs of the same dimension as the underlying image.
If you have your label masks in the form of a numpy ndarray, we recommend using the pillow python library to convert it into a PNG:
! pip3 install pillow
​
from PIL import Image
...
​
# 2D array, where each value is [0,255] corresponding to a class_id
# in the project's label_class_map.
int_arr = your_2d_ndarray.astype('uint8')
​
Image.fromarray(int_arr).save(f"{imagename}.png")
Because this will be loaded dynamically by the web-app for visualization, this image mask will need to be hosted somewhere. To upload it as an asset to Aquarium, you can use the following utility:
mask_url = al_client.upload_asset_from_filepath(project_id, dataset_id, filepath)
This utility hosts and stores a copy of the label mask (not the underlying RGB image) with Aquarium. If you would like your label masks to remain outside of Aquarium, chat with us and we'll help figure out a good setup.
Now, we add the label to the frame like any other label type:
frame.add_label_2d_semseg(
    # The sensor id of the image this label corresponds to
    sensor_id='some_camera',
    # A unique id across all other labels in this dataset
    label_id='unique_id_for_this_label',
    # Expected to be a PNG, with values in [0,255] that correspond
    # to the class_id of classes in the label_class_map
    mask_url='url_to_greyscale_png'
)
​
Aquarium represents instance segmentation labels as 2D Polygon Lists. Each label is represented by one or more polygons, which do not need to be connected.
frame.add_label_2d_polygon_list(
    # A unique id across all other labels in this dataset
    label_id='unique_id_for_this_label',
    classification='dog',
    # All coordinates are in absolute pixel space
    #
    # These are polygon vertices, not a line string. This means
    # that no vertices are duplicated in the lists.
    polygons=[
        {'vertices': [(x1, y1), (x2, y2), ...]},
        {'vertices': [(x1, y1), (x2, y2), ...]}
    ],
    # Optional: indicate the center position of the object
    center: [center_x, center_y]
)
​
Inference Formats
Inference frames can contain zero or more inference objects, which are labels produced by your model. They generally correspond 1:1 with a label format, with the addition of a confidence parameter. 
Using 2D bounding boxes as an example:
frame.add_label_2d_bbox(
    sensor_id='some_camera',
    label_id='abcd_label',
    classification='dog',
    top=200,
    left=300,
    width=250,
    height=150
)
​
inference_frame.add_inference_2d_bbox(
    sensor_id='some_camera',
    label_id='abcd_inference',
    classification='cat',
    top=200,
    left=300,
    width=250,
    height=150,
    confidence=0.85
)
Multiple Sensor IDs
Sensor IDs are used reference data points that exist on a frame.  They are usually omitted in frames and labels, but become necessary if there exists more than one type of data point on a single frame.  A good example of this is a frame with multiple camera view points.  
frame.add_image(sensor_id='camera_front', image_url='')
frame.add_image(sensor_id='camera_right', image_url='')
frame.add_image(sensor_id='camera_left', image_url='')
​
# 2D BBOX label on the `camera_front` image
frame.add_label_2d_bbox(
    sensor_id='camera_front',
    label_id='unique_id_for_this_label',
    classification='dog',
    top=200,
    left=300,
    width=250,
    height=150
)
​
# 2D BBOX label on the `camera_left` image
frame.add_label_2d_bbox(
    sensor_id='camera_left',
    label_id='unique_id_for_this_label',
    classification='cat',
    top=200,
    left=300,
    width=250,
    height=150
)
​
# Inferences MUST match the same sensor id as the
# corresponding base frame sensor id.
inference_frame.add_inference_2d_bbox(
    sensor_id='camera_front',
    label_id='abcd_inference',
    classification='cat',
    top=200,
    left=300,
    width=250,
    height=150,
    confidence=0.85
)
Custom Metrics
Aquarium automatically computes standard performance metrics for your inferences, but you may have custom performance metrics. In that case, you can provide your own custom metrics, either as an objective function, or a confusion matrix:
al_client.create_project(..., custom_metrics=[
    al.CustomMetricsDefinition(name='my_score', metrics_type='objective'),
    al.CustomMetricsDefinition(name='my_conf_matrix', metrics_type='confusion_matrix'),
])
​
...
​
inferences_frame.add_custom_metric('my_score', 3.32)
inferences_frame.add_custom_metric('my_conf_matrix', [
    [6,2],
    [1,5]
])
Custom Embeddings
Aquarium uses neural network embeddings to enable features like clustering and similarity search. They can be thought of as lists of numbers that represent the essential visual qualities of an image. By default, Aquarium will try to compute embeddings for you by using our standard neural network. 
You may also wish to provide your own embeddings, especially if you have an unusual data domain. Our python library makes it easy to attach your own values instead:
# Embedding vectors are expected to by python lists of float values.
#
# You can provide both frame-level embeddings (for the entire image or scene)
# and crop/label-level embeddings (for that specific object)
​
frame.add_frame_embedding(embedding=[1.0,2.0,3.0, ...])
for label_id, label_embedding in label_embeddings.items():
    frame.add_crop_embedding(label_id=label_id, embedding=label_embedding)
​
# Similar APIs exist for inferences
inference_frame.add_frame_embedding(embedding=[1.0,2.0,3.0, ...])
for inf_id, inf_embedding in inference_embeddings.items():
    inference_frame.add_crop_embedding(label_id=inf_id, embedding=inf_embedding)
​
If you chose a data sharing scheme that doesn't allow Aquarium to access your data, then you'll have to provide your own embeddings to enable certain features. Check out our full page on embeddings for some sample code, along with guidance for using your own models.
Colab Notebook Example
Some folks may want a code example to get started with. To demonstrate how to upload your data with our Python client, we've provided the following Colab notebooks:
​Quickstart Notebook --- This walks through the process of:
Creating a project
Uploading a labeled dataset (based on the KITTI dataset)
Uploading model inferences
​
​Quickstart Notebook (code-only) --- This is a condensed version of the notebook above, with minimal comments.
Common Workflows
To skip straight to some common workflows for spotting and tracking label quality issues, analyzing model performance on different segments of your data, and quickly sampling new data to improve your model, see the Common Workflows section.
​