Stratified Metrics

Often you want to get a more nuanced understanding of how your model performs on specific slices of the labels. For example, in a self driving use case, you might want to understand how you perform on near vs far objects. Or you might have a fish-eye camera and what to understand how your model performs as objects approach the edge of the image.

With stratified metrics, you can filter the metrics view based on per-label attributes, looking at a specific 'stratum' of your label set. For example, if you attach an attribute called β€œrange,” you can evaluate performance on just large or small labels independently.

Example Usage

stratified_metrics_definitions = [
al.StratifiedMetricsDefinition(
name='range_bucket',
ordered_values=['0-10', '10-20', '20-30', '30-40', '40-50', '50+']
)
]
​
...
​
range_bucket = get_cuboid_range_bucket((0,0,0), label['position'])
frame.add_label_3d_cuboid(user_attrs={'range_bucket': range_bucket}, **label)
​
...
​
range_bucket = get_cuboid_range_bucket((0,0,0), inference['position'])
inf_frame.add_inference_3d_cuboid(user_attrs={'range_bucket': range_bucket}, **inference)
​
...
​
al_client.create_project(
aquarium_project,
label_class_map,
stratified_metrics=stratified_metrics_definitions
)

​