Proposal: segmentation masks in Croissant

[marcenacp]

Introduction

In computer vision, segmentation is the ML task to recognize objects in an image at the pixel level. The annotations are "segmentation masks", i.e. segments or the zones of interest in the image. Segmentation is a wide-spread enough task in ML for segmentation masks to deserve a first-class place in Croissant JSON-LD definition.
Here is a good example by fchollet@.

Usual format for segmentation masks:

• An image with 0 or 1:
  • 0 = the pixel is outside of the mask.
  • 1 = the pixel is inside the mask.
• A polygon with (x, y) coordinates that describes the boundary of the segmentation mask.
  • (x, y) can be percentages on the image.
  • (x, y) can be numbers of pixels.

Examples

Example 1: YOLO

YOLO declares *.txt files where each line is:

class_index x_1 y_1 x_2 y_2 ... x_n y_n

where x_i and y_i are the (x, y) coordinates in percentage.

Example 2: COCO

COCO uses two different formats:

• Stuff segmentation with annotations in both JSON and PNG.
• Panoptic segmentation with a PNG that stores the class-agnostic image segmentation.

Croissant standard proposal

We saw that we need to support segmentation masks as both list of coordinates and images.

We could define the following types:

• ml:SegmentationMask for the image representation of segmentation masks.
• ml:Polygon for the polygon representation of segmentation masks. ml:Polygon can be later re-used for other tasks. Another proposition is to re-use the existing sc:GeoCoordinates.

Segmentation masks as images

For the image representation, we can already define an image in a field:

{
    "@type": "ml:Field",
    "name": "segmentation_mask",
    "description": "The segmentation mask as a PNG.",
    "dataType": ["sc:ImageObject", "ml:SegmentationMask"],
    "source": {
        "dataExtraction": {
            "fileProperty": "content"
        },
        "distribution": "images"
    }
},

Segmentation masks as polygons

For the polygon representation, we propose to define SubFields:

{
    "@type": "ml:Field",
    "name": "segmentation_mask",
    "description": "The segmentation mask as a polygon.",
    "dataType": ["ml:RecordSet", "ml:Polygon"],
    "subField": [
        {
            "@type": "ml:Field",
            "name": "x",
            "source": {
                "applyTransform": [
                    {
                        "separator": " "
                    },
                    {
                        "jsonPath": "$.[1::2]"
                    }
                ],
                "dataExtraction": {
                    "fileProperty": "line"
                },
                "distribution": "annotations"
            }
        },
        {
            "@type": "ml:Field",
            "name": "y",
            "source": {
                "applyTransform": [
                    {
                        "separator": " "
                    },
                    {
                        "jsonPath": "$.[2::2]"
                    }
                ],
                "dataExtraction": {
                    "fileProperty": "line"
                },
                "distribution": "annotations"
            }
        }
    ]
}

The problem here is the data extraction method. So we use the source to:

1. Read the file line by line ("fileProperty": "line").
2. For every line, separate each word with ("separator": " ").
3. For every word, take one every two words except the first one (which is the class index) ("jsonPath": "$.[1::2]").

[benjelloun]

How about sc:GeoShape instead of ml:polygon? It has a polygon property.

The subField approach seems overly complicated...

[marcenacp]

I agree we can re-use sc:GeoShape.

The subField approach seems overly complicated...

This was not meant to be another approach, but the same approach. We need to parse the information from a text/json/csv representation to the Croissant sc:GeoShape representation. I used applyTransform for this.

How would you do?