Data Labeling: The Authoritative Guide

Computer vision is a field in artificial intelligence focused on understanding data from 2D images, videos, or 3D inputs and making predictions or recommendations based on that data. The human vision system is particularly advanced, and humans are very good at computer vision tasks. 

In this chapter, we explore the most relevant types of data labeling for computer vision and provide best practices for labeling each type of data.

1. Bounding Box

The most commonly used and simplest data label, bounding boxes are rectangular boxes that identify the position of an object in an image or video

Data labelers draw a rectangular box over an object of interest, such as a car or street sign. This box defines the object's X and Y coordinates.

By "bounding" an object with this type of label, machine learning models have a more precise feature set from which to extract specific object attributes to help them conserve computing resources and more accurately detect objects of a particular type.

Object detection is the process of categorizing objects along with their location in an image. These X and Y coordinates can then be output in a machine-readable format such as JSON.

Typical Bounding Box Applications:

• Autonomous driving and robotics to detect objects such as cars, people, or houses
• Identifying damage or defects in manufactured objects
• Household object detection for augmented reality applications
• Anomaly detection in medical diagnostic imaging

Best Practices:

1. Hug the border as tightly as possible. Accurate labels will capture the entire object and match the edges as closely as possible to the object's edges to reduce confusion for your model.
2. Avoid item overlap. Due to IoU, bounding boxes work best when there is minimal overlap between objects. If objects overlap significantly, using polygon or segmentation annotations may be better.
3. Object size: Smaller objects are better suited for bounding boxes, while larger objects are better suited for instance segmentation. However, annotating tiny objects may require more advanced techniques.
4. Avoid Diagonal Lines: Bounding boxes perform poorly with diagonal lines such as walkways, bridges, or train tracks as boxes cannot tightly hug the borders. Polygons and instance segmentation are better approaches in these cases.

2. Classification

Object classification means applying a label to an entire image based on predefined categories, known as classes. Labeling images as containing a particular class such as "Dog," "Dress," or "Car" helps train an ML model to accurately predict objects of the same class when run on new data.

Typical Classification Applications:

• Activity Classification
• Product Categorization
• Image Sentiment Analysis
• Hot Dog vs. Not Hot Dog

Best Practices:

1. Create clearly defined, easily understandable categories that are relevant to the dataset.
2. Provide sufficient examples and training to your data labelers so that the requirements are clear and ambiguity between classes is minimized.
3. Create benchmark tests to ensure label quality.

3. Cuboids

Cuboids are 3-dimensional labels that identify the width, height, and depth of an object, as well as the object's location.

Data labelers draw a cuboid over the object of interest such as a building, car, or household object, which defines the object's X, Y, and Z coordinates. These coordinates are then output in a machine-readable format such as JSON.

Cuboids enable models to precisely understand an object's position in 3D space, which is essential in applications such as autonomous driving, indoor robotics, or 3D room planners. Reducing these objects to geometric primitives also makes understanding an entire scene more manageable and efficient.

Typical Cuboid Applications:

• Develop prediction and planning models for autonomous vehicles using cuboids on pedestrians and cars to determine predicted behavior and intent.
• Indoor objects such as furniture for room planners
• Picking, safety, or defect detection applications in manufacturing facilities

Best Practices:

1. Capture the corners and edges accurately. Like bounding boxes, ensure that you capture the entire object in the cuboid while keeping the label as tight to the object as possible.
2. Avoid Overlapping labels where possible. Clean, non-overlapping cuboid data annotations will help your model improve object predictions and localizations in 3D space.
3. Axis alignment is critical. Ensure that the alignment of your bounding boxes is on the same axis for objects of the same class.
4. Keep your camera intrinsics in mind. Applying cuboids without understanding the camera's location will lead to poor prediction results when objects are not in the same position related to the camera in the future. The front face of a "true" cuboid will likely not be a perfect 90-degree rectangle, especially if it isn't facing the camera head-on. Furthermore, the edges of a cuboid parallel to the ground should converge to the horizon, while the top and bottom edges of the right side of the above annotation are parallel.
5. Pair 2D Data with 3D Depth Data such as LiDAR.2D images inherently lack depth information, so pairing your 2D data with 3D depth data such as LiDAR will yield the best results for applications dependent on depth accuracy. See the section below on 3D Sensor fusion for more information on this topic.

4. 3D Sensor Fusion

3D sensor fusion refers to the method of combining the data from multiple sensors to accommodate for the weaknesses of each sensor. 2D images alone are not enough for current machine learning models to make sense of entire scenes. Estimating depth from a 2D image is challenging, and occlusion and limited field of view make relying on 2D images tricky. While some approaches to autonomous driving rely solely on cameras, a more robust approach is to overcome the limitations of 2D by supplementing with 3D systems using sensors such as LiDAR and Radar.

LiDAR (Light Detection and Ranging) is a method for determining the distance of objects with a laser to determine the depth of objects in scenes and create 3D representations of the scene.

Radar (Radio detection and ranging) uses radio waves to determine objects' distance, angle, and radial velocity.

This demo provides an interactive 3D sensor fused scene, and the video below gives a high-level overview of a similar scene.

Typical 3D Sensor Fusion Applications

• Autonomous Vehicles
• Geospatial and mapping applications
• Robotics and automation

Best Practices

1. Ensure that your data labeling platform is calibrated to your sensor intrinsics (or better yet, ensure that your tooling is sensor agnostic) and supports different lens and sensor types, for example, fisheye and panoramic cameras.
2. Look for a data labeling platform that can support large scenes, ideally with support for infinitely long scenes.
3. Ensure that object tracking is consistent throughout a scene, even when an object leaves and returns to the scene. 
4. Include attribute support for understanding correlations between objects, such as truck cabs and trailers. 
5. Leverage linked instance IDs describing the same object across the 2D and 3D modalities.

5. Ellipses

Ellipses are oval data labels that identify the position of objects in an image. Data labelers draw an ellipse label on an object of interest such as wheels, faces, eyes, or fruit. This annotation defines the object's location in 2D space. The X and Y coordinates of the four extremal vertices of the ellipse can then be output in a machine-readable format such as JSON to fully define the location of the ellipse.

Applications

• Face Detection
• Medical Imaging Diagnosis
• Wheel Detection

Best Practices:

1. The data to be labeled should be oval or circular; i.e., it is not helpful to label rectangular boxes with ellipses when a bounding box will yield better results.
2. Use ellipses where there would be high overlap for bounding boxes or where objects are tightly clustered or occluded, such as in bunches of fruit. Ellipses can tightly hug the borders of these objects and provide a more targeted geometry to your model.

6. Lines

Lines identify the position of linear objects like roadway markers. Data labelers draw lines over areas of interest, which define the vertices of a line. Labeling images with lines helps to train your model to identify boundaries more accurately. The X and Y coordinates of the vertices of the lines can then be output in JSON.

Typical Lines Applications

• Label roadway markers with straight or curved lines for autonomous vehicles
• Horizon lines for AR/VR applications
• Define boundaries for sporting fields

Best Practices

1. Label only the lines that matter most to your application.
2. Match the lines to the shape of the lines in the image as closely as possible.
3. Depending on the use case, it could be important for lines not to intersect. 
4. Center the line annotation within the line in the image to improve model performance.

7. Points

Points are spatial locations in an image used to define important features of an object. Data labelers place a point on each location of interest, representing that location's X and Y coordinates. These points may be related to each other, such as when annotating a human shoulder, elbow, and wrist to identify the larger moving parts of an arm. These labels help machine learning models more accurately determine pose estimations or detect essential features of an object.

Typical Points Applications

• Pose estimation for fitness or health applications or activity recognition
• facial feature points for face detection

Best Practices

1. Label only the points that are most critical to your application. For instance, if you are building a face detection application, focus on labeling salient points on the eyes, nose, mouth, eyebrows, and the outline of the face.
2. Group points into structures (hand, face, and skeletal keypoints), and the labeling interface should make it efficient for taskers to visualize the interconnections between points in these structures.

8. Polygons

While bounding boxes are quick and easy for data labelers to draw, they are not precise in mapping to irregular shapes and can leave large gaps around an object. There is a tradeoff between accuracy and efficiency in using bounding boxes and polygons. For many applications, bounding boxes provide sufficient accuracy for a machine learning model with minimal effort. However, some applications require the increased accuracy of polygons at the expense of a more costly and less efficient annotation.

Data labelers draw a polygon shape over an object of interest by clicking on relevant points of the object to complete an entirely connected annotation. These points define the vertices of the polygon. The X and Y coordinates of these vertices are then output in JSON.

Typical Polygons Applications

• Irregular objects such as buildings, vehicles, or trees for autonomous vehicles
• Satellite imagery of houses, pools, industrial facilities, planes, or landmarks
• Fruit detection for agricultural applications

Best Practices

1. Objects with holes or those split into multiple polygons due to occlusion (a car behind a tree, for example) require special treatment. Subtract the area of each hole from the object.
2. Avoid slight overlaps between polygons that are next to each other.
3. Zoom in closely to each object to ensure that you place points close to each object's borders.
4. Pay close attention to curved edges, making sure to add more vertices to 'smooth' these edges as much as possible.
5. Leverage the Auto Annotate Polygon tool to efficiently label objects. Automatically and quickly generate high-precision polygon annotations by highlighting specific objects of interest with an initial, approximate bounding box.

Follow these steps to achieve success with the Auto Annotate Polygon tool:

• Include all parts of the object of interest.
• Exclude overlapping object instances and other objects as much as possible.
• Keep the bounding box tight to the borders of the object.
• Use click to include/exclude to refine the automatically-generated polygon by instantly performing local edits - include and exclude specific areas of interest.
• Further, refine the polygon by increasing or decreasing vertex density to smooth curved edges.

9. Segmentation

Segmentation labels relate to pixel-wise labels on an image and come in three common types, semantic segmentation, instance segmentation, and panoptic segmentation.

Semantic Segmentation

Label each pixel of an image with a class of what is being represented, such as a car, human, or foliage. Referred to as "dense prediction," this is a time-consuming and tedious process.

With semantic segmentation, you do not distinguish between separate objects of the same class (see instance segmentation for this).

Instance segmentation

Label every pixel of each distinct object of an image. Unlike semantic segmentation, instance segmentation distinguishes between separate objects of the same class (i.e., identifying car 1 as separate from car 2)

Panoptic Segmentation

Panoptic segmentation is the combination of instance segmentation and semantic segmentation. Each point in an image is assigned a class label (semantic segmentation) AND an instance label (instance segmentation). Each instance can represent distinct objects such as cars, people, or regions such as the road or sky. Panoptic segmentation provides more context than instance segmentation and is more detailed than semantic segmentation, making them useful for deeper scene understanding.

Typical Segmentation Applications

• Autonomous Vehicles and Robotics: Identify pedestrians, cars, trees
• Medical Diagnostic imaging: tumors, abscesses in diagnostic imaging
• Clothing: Fashion retail

Best Practices

1. Carefully trace the outlines of each shape to ensure that all pixels of each object are labeled.
2. Use ML-assisted tooling like the boundary tool to quickly segment borders and objects of interest.
3. After segmenting borders, use the flood fill tool to fill in and complete segmentation masks quickly.
4. Use active tools like Autosegment to increase the efficiency and accuracy of your labelers.

Explore Coco-Stuff on Nucleus for a large collection of data with segmentation labels!

10. Special considerations for Video Labeling

You can apply many of the same labels to images and videos, but there are some special considerations for video labeling.

Temporal linking of labels

Video annotations add the dimension of time to the data fed to your models, so your model needs a way to understand related objects and labels between frames.

Manually tracking objects and adding labels through many video frames is time and resource intensive. You can leverage several techniques to increase the efficiency and accuracy of temporally linked labels.

1. First, you can leverage video Interpolation to interpolate between frames of your video to smooth out the images and labels to make it easier to track labels through the video.
2. You should also look for tools that automatically duplicate annotations between frames, minimizing human intervention needed to correct for misaligned labels.
3. If you are working with videos, ensure that the tools you are using can handle the storage capacity of the video file and can stitch together hour-long videos so that you retain the same context no matter how long the video.

In videos, objects may leave the camera view and may return at a later time. Leverage tools that enable you to track these objects automatically or make sure to remember to annotate these objects with the same unique ids.

Multimodal

Multimodal machine learning attempts to understand the world from multiple, related modalities such as 2D images, audio, and text.

Combining multiple labeling types such as human keypoints, bounding boxes, and transcribed audio with entity recognition all connected in rich scenes.

Typical Multimodal Applications

• AR/VR full scene understanding Video/GIF/Image (Object Detection + Human Keypoints + Audio Transcription + Entity Recognition)
• Sentiment analysis by combining video gestures and voice data

Best Practices

1. Incorporate temporal linking to ensure that models fully understand the entire breadth of each scene.
2. Identify which modalities are best suited for your application. For instance, if you are working on sentiment analysis for AR/VR applications, you will want to consider not only 2D video object or human keypoint labels but also audio transcription and entity recognition in addition to sentiment classification so that you have a rich understanding of the entire scene and how the individuals in the scene contribute towards a particular sentiment (i.e., if the person is yelling and gesturing wildly you can determine the sentiment is "upset").
3. Include Human in the loop to ensure consistency across modalities. Assign complex scenes to only the most experienced taskers.

Synthetic Data

Synthetic data is digitally-generated data that mimics real-world data. This data is often generated from artists using graphics computer tooling or generated programmatically from models like Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), or Neural Radiance Fields (NERFs). Synthetic data includes perfect ground truth labels automatically, without requiring additional human intervention to label the data.

Typical Synthetic Data Applications

• Digital Humans for autonomous vehicles and robotics, particularly in long-tail edge cases such as pedestrians walking on shoulders
• Digital humans for fitness and health applications. Getting enough real-world human pose data is difficult and expensive, whereas synthetic data is relatively easy to generate and cheaper.
• Manufacturing defect detection

Best Practices

• If using a mix of Synthetic and real-world data, ensure that the labels of your real data are as accurate as possible. Synthetic data generates perfectly accurate labels, so any label inaccuracies in your real data will degrade your model's predictive capabilities.
• Leverage synthetic data for data that is difficult to collect due to privacy concerns, rare edge cases, to avoid bias, or prohibitively expensive data collection and labeling methods.
• Integrate synthetic data with your existing data pipelines to maximize your ROI
• Curate your data using best-in-class tools to ensure that you are surfacing the edge cases for which you need more data. Ideally, your dataset curation tool will also integrate into your labeling pipelines.