Metadata format

Metadata allows a set of keywords (tags) to be stored in the index along with each image. The keywords can describe the content of the image (e.g., tree, house, people) or can contain a classification of the image according to a given criterion (e.g., photo versus illustration). A list of attributes (e.g., data type, data encoding) can be associated with each keyword. The keywords can be provided in a hierarchical or a flat structure.

The keyword search can be done using one or multiple keywords linked by operators (e.g., AND, OR). The keywords and the operators are combined using a hierarchical structure to form query requests in a coherent fashion. The search can be done using query keywords (keyword search) or using query keywords and query colors (keyword and color search). Associated keywords can be returned along with each match if necessary.

The keywords can be provided using XML or JSON format. Note that JSON format does not support attributes the same way XML does. For this reason, when using JSON all attributes must be embeded into the tree hierarchy. Each keyword can be followed by a set of attributes and a special empty tag containing its value or its children.

Indexing using metadata

Metadata can be stored in the index along with collection images using the POST add method. Metadata can be provided to POST add method using the metadata argument.

Format

JSON (metadata stored with an image)

"type": "Photo",
"category": "Royalty-free",

"id": {
    "action": "return",
    "type": "int",
    "": "4365456"
},

"key": {
    "action": "return",
    "type": "binary",
    "encoding": "base64",
    "": "NQysf3CgRuXyWSOMkf8I"
},

"price": {
    "action": "mixed",
    "type": "float",
    "": "120.89"
},

"available": {
    "action": "mixed",
    "type": "bool",
    "": "True"
},

"clothes": {
    "t-shirt": {
        "manufacturer": "Idee",
        "color": ["blue", "white"],

        "size": {
            "type": "int",
            "": "12"
        }
    }
}

XML (metadata stored with an image)

<type>Photo</type>
<category>Royalty-free</category>

<id action="return" type="int">4365456</id>
<key action="return" type="binary" encoding="base64">NQhsf6CgRuXkWSOMkf8I</key>

<price action="mixed" type="float">120.89</price>
<available action="mixed" type="bool">True</available>

<clothes>
    <t-shirt>
        <manufacturer>Idee</manufacturer>

        <color>
            <blue/>
            <white/>
        </color>

        <size type="int">12</size>
    </t-shirt>
</clothes>

The tags located on the first level of the tree hierarchy are called root tags (e.g., type, category, id, key, price, available, clothes). All the tags located in the subsequent levels in the hierarchy are regular tags. A list of attributes can be specified for each tag. The accepted attributes are action, type and encoding and are case sensitive. An error message will be returned if an unknown tag attribute is provided for an image. Note that if any error occurs during an POST add request the image will not be indexed. The error must be corrected and the POST add request must be sent again.

Action attributes

The action attribute can be only used for root tags. An error message will be returned if an action attribute is used for a regular tag. The accepted values for the action attributes are search, return and mixed. If an action attribute is not provided for a root tag the search action will be assumed by default.

  • search, the tag and all its descendents in the tree hierarchy can be used for keyword searching
  • return, the tag and its value can be returned along with each match
  • mixed, the tag and its value can be used for keyword searching and also can be returned along with each match

Tags with return and mixed actions cannot have descendents but must have a value (e.g., ‘“price”: “120.89”’). On the other hand, tags with a search action may have descendents organized in a tree hierarchy. The only limitation is that the number of levels in the hierarchy cannot exceed 255.

Type attributes

The type attribute defines the data type for the tag’s children. The accepted values for the type attributes are string, binary, bool, char, uchar, short, ushort, int, uint, long, ulong, float and double. If a type attribute is not provided, the string data type will be assumed by default. Note that all root tags must be strings.

  • string, the tag’s children are strings (variable length)
  • binary, the tag’s children contain encoded (see Encoding attributes section) binary data (variable length)
  • bool, the tag’s value is boolean, only “True”, “true”, “1” and “False”, “false”, “0” values are accepted
  • char, the tag’s value is a number of type char (length: 1 byte)
  • uchar, the tag’s value is a number of type unsigned char (length: 1 byte)
  • short, the tag’s value is a number of type short (length: 2 bytes)
  • ushort, the tag’s value is a number of type unsigned short (length: 2 bytes)
  • int, the tag’s value is a number of type int (length: 4 bytes)
  • uint, the tag’s value is a number of type unsigned int (length: 4 bytes)
  • long, the tag’s value is a number of type long (length: 8 bytes)
  • ulong, the tag’s value is a number of type unsigned long (length: 8 bytes)
  • float, the tag’s value is a number of type float (length: 4 bytes)
  • double, the tag’s value is a number of type double (length: 8 bytes)

Tags with bool, char, uchar, short, ushort, int, uint, long, ulong, float and double data types cannot have descendents. They can be composed only from a name and a value (e.g., ‘“id”: “4365456”’). On the other hand, tags with string and binary data type can have descendents organized in a tree hierarchy. The only limitation is that the number of levels in the hierarchy cannot exceed 255.

Tags with string and binary data types have variable length. They can be between 1 and 255 characters long. An error message will be returned if the tag’s length exceeds 255 characters. Note that for the binary data type these limitations are enforced only after the tag’s data was decoded.

Encoding attributes

The encoding attribute can be used only for binary data types. Currently, the only encoding format supported is Base64 (e.g., ‘“encoding”: “base64”’). Note that additional to storing binary data the binary type can be used for storing reserved words (see Comparison operators and Logical operators sections).

Searching using metadata

A keyword search can be done alone or combined with a color search (using POST color_search method). The tags and the operators can be provided to the POST color_search method using the metadata argument.

Format

JSON

"_and_operator_": {
    "type": {
        "weight": "2.0",
        "": "Photo"
    },

    "_less_operator_": {
        "price": "150.0"
    },

    "_or_operator_": {
        "weight": "2.0",
        "": [
            {
                "clothes": {
                    "weight": "4.0",
                    "": {
                        "t-shirt": {
                            "color": "red"
                        }
                    }
                }
            },

            {
                "clothes": {
                    "weight": "3.0",
                    "": {
                        "t-shirt": {
                            "color": "blue"
                        }
                    }
                }
            }
        ]
    },

    "_not_operator_": {
        "weight": "1.0",
        "": {
            "clothes": {
                "t-shirt": {
                    "manufacturer": "Idee"
                }
            }
        }
    },

    "_greater_less_operator_": {
        "clothes": {
            "t-shirt": {
                "size": "10 14"
            }
        }
    }
}

XML

<_and_operator_>
    <type weight="2.0">Photo</type>

    <_less_operator_>
        <price>150.0</price>
    </_less_operator_>

    <_or_operator_ weight="2.0">
        <clothes weight="4.0">
            <t-shirt>
                <color>red</color>
            </t-shirt>
        </clothes>

        <clothes weight="3.0">
            <t-shirt>
                <color>blue</color>
            </t-shirt>
        </clothes>
    </_or_operator_>

    <_not_operator_ weight="1.0">
        <clothes>
            <t-shirt>
                <manufacturer>Idee</manufacturer>
            </t-shirt>
        </clothes>
    </_not_operator_>

    <_greater_less_operator_>
        <clothes>
            <t-shirt>
                <size>10 14</size>
            </t-shirt>
        </clothes>
    </_greater_less_operator_>
</_and_operator_>

All tags used for a keyword search must have a search or a mixed action attributes associated with them in the index. An error message will be returned if a tag with a return action attribute is encountered.

The API accepts two types of entities:

  • Operands, a unique and fully qualified tag name (in this context fully qualified tag means a tag and all its predecessors in the tree hierarchy)
  • Operators, specifies the action to be applied to one or more operands

The API supports two types of operators:

  • Comparison operators, used to determine equality or difference between values. The comparison operators can be applied only to numeric data types (char, uchar, short, ushort, int, uint, long, ulong, float and double). However, some of the comparison operators can be applied to boolean data type (bool) as well.
  • Logic operators, used to determine the merging logic of multiple operands. These operands can be applied to non numeric data types (string and binary) and to other operators.

Each operator and operand can have an associated weight attribute. For operands the weight attribute can be specified only for their root tags. If a weight attribute is not specified for an operator or an operand then the default weight assumed is 0. The value of a weight attribute must be a floating number greater or equal with 0. The weights of operators and operands are combined using addition and an overall metadata score is computed for the results. If all the weights are 0 (or not specified) then a metadata score is not computed.

Comparison operators

A comparison operator can be applied to only one operand. The data type of the operand’s end tag in the tree hierarchy must be numeric (or boolean for few operators). Because the end tag has a numeric data type it must be followed by one number (for regular comparison operators) or two space separated numbers (for range comparison operators) as shown in the flowing example.

JSON

"_and_operator_": {
    "_less_operator_": {
        "price": "150.0"
    },

    "_greater_less_operator_": {
        "clothes": {
            "t-shirt": {
                "size": "10 14"
            }
        }
    }
}

XML

<_and_operator_>
    <_less_operator_>
        <price>150.0</price>
    </_less_operator_>

    <_greater_less_operator_>
        <clothes>
            <t-shirt>
                <size>10 14</size>
            </t-shirt>
        </clothes>
    </_greater_less_operator_>
</_and_operator_>

The regular comparison operators are:

  • _equal_operator_, the Equal To operator. Can be applied to boolean, numeric and non numeric data types. Note that using this operator with float and double data types should be avoided (because of the way floats and doubles are represented internally).
  • _not_equal_operator_, the Not Equal To operator. Can be applied to boolean and numeric data types.
  • _greater_operator_, the Greater Than operator. Can be applied only to numeric data types.
  • _less_operator_, the Less Than operator. Can be applied only to numeric data types.
  • _greater_equal_operator_, the Greater Than Or Equal To operator. Can be applied only to numeric data types.
  • _less_equal_operator_, the Less Than Or Equal To operator. Can be applied only to numeric data types.

The range comparison operators are:

  • _greater_less_operator_, combines Greater Than and Less Than operators using an AND logic operator (Greater than X AND less than Y). Can be applied only to numeric data types.
  • _greater_equal_less_operator_, combines Greater Than Or Equal To and Less Than operators using an AND logic operator (Greater than or equal to X AND less than Y). Can be applied only to numeric data types.
  • _greater_less_equal_operator_, combines Greater Than and Less Than Or Equal To operators using an AND logic operator (Greater than X AND less than or equal to Y). Can be applied only to numeric data types.
  • _greater_equal_less_equal_operator_, combines Greater Than Or Equal To and Less Than Or Equal To operators using an AND logic operator (Greater than or equal to X AND less than or equal to Y). Can be applied only to numeric data types.

Logical operators

A logic operator can be applied to operands and to operators (including other logic operators). Some of the logic operators can be applied to only one operand or operator (like logical NOT). Other logic operators can be applied to two or more operands and/or operators (like logical AND and logical OR).

The logic operators are:

  • _and_operator_, the Logical AND operator. Can be applied to two or more operands and/or operators and computes the intersection between the results generated by each of these operands and operators.
  • _or_operator_, the Logical OR operator. Can be applied to two or more operands and/or operators and computes the union between the results generated by each of these operands and operators.
  • _not_operator_, the Logical NOT operator. Can be applied to only one operand or operator and negates the results of this operand or operator (negating a list of results A means generating a new list of results B containing all images present in the collection but not present in list A).

The name of the comparison and logic operators are reserved and cannot be used as tag names when employing string data types. However, they can be Base64 encoded and used as tag names when employing binary data types.

Returning metadata

Metadata can be returned along with each match when performing a color search (using the POST color_search method). The metadata to be returned can be provided to the POST color_search method using the return_metadata argument.

JSON

[
    {
        "price": {
            "sort": "asc",
            "": null
        }
    },
    {
        "id": {
            "sort": "desc",
            "": null
        }
    },
    "key"
]

XML

<price sort="asc"/>
<id sort="desc"/>
<key/>

All tags to be returned must have a return or a mixed action attribute associated with them in the index. This means that these tags have a return value but do not have a descendant (see Action attributes section). An error message will be returned if a tag with search action attribute is encountered.

The return tags can have sort attributes that determine the sorting order of the results. If multiple tags have sort attributes then the precedence will be determined by the order tags are listed in the return-metadata descriptor. Tags at the beginning of the list will have higher sorting precedence than tags at the end of the list.

The accepted values for the sort attribute are:

  • asc: sort the results in ascending order using the value of the associated tag
  • desc: sort the results in descending order using the value of the associated tag
  • none: don’t use the associated tag to sort the results

If a sort attribute is not provided the none sorting will be assumed by default. Following is the algorithm used to determine the sorting order of two results:

  • compare the results using the first tag from the return-metadata list with a valid sort attribute (asc or desc) and determine the sorting order
  • if a sorting order was found stop; if the order cannot be determined because of equality, then the next tag from the return-metadata list with a valid sort attribute will be used
  • if a sorting order was found stop; otherwise repeat the previous step until a sorting order is found or the end of the list was reached
  • if the sorting order cannot be determined using the return tags because of equality or because they don’t have sort attributes, then the match score or the metadata score generated during the search will be used (see sort_metadata argument for POST color_search method)
  • if the sorting order cannot be determined using the match score, then the results are sorted using the order they are stored in the index

Note that for binary data types encoded in Base64 format sorting is done using the decoded data.

Character escaping

The metadata string tags can contain any character. However, some characters need to be escaped when using the JSON or XML protocol.

JSON

When using JSON tags the following characters need to be escaped:

ASCII value Character name Character Escape sequence
34 double quote " \"
92 backslash \ \\
8 backspace BS \b
9 horizontal tab TAB \t
10 line feed NL \n
12 form feed NP \f
13 carriage return CR \r

XML

When using XML value tags the following characters need to be escaped:

ASCII value Character name Character Escape sequence
38 ampersand & &amp;
60 less angle bracket < &lt;
62 greater angle bracket > &gt;

When using XML name tags the first character can be a letter, a colon : or an underscore _. The following characters in the name tag can be letters, digits, colons :, hyphens -, and underscores _. All other ASCII characters present in a name tag must be escaped using the following sequence _x00HH_, where HH represents the hexadecimal representation of their ASCII value (e.g., the space character must be escaped using _x0020_ sequence). The underscore _ character doesn’t need to be escaped unless it is followed by a x00HH_ sequence that can be misinterpreted as an escaped sequence. In this case the underscore character _ must be escaped using _x005F_ sequence.

Note that binary Base64 encoded data may end with one or two equal signs = and can contain plus signs + and forward slashes /. If the binary data is part of a name tag then the equal signs = must be escaped using _x003D_ sequences, the plus signs + using _x002B_ sequences and the forward slashes / using _x002F_ sequences. If the binary data is part of a value tag then those characters don’t need to be escaped.

Handling unicode

Special care needs to be taken when working with Unicode characters. If sending Unicode characters using JSON, metadata search responses should be requested in JSON to avoid encoding problems. Similarly, if sending Unicode characters with XML, metadata search responses should be requested in XML.

JSON

In JSON, Unicode characters may be either UTF-8 encoded, or escaped using their six-character representation: a backslash, followed by the lowercase letter u, followed by the character’s four-hexadecimal-digit code-point representation.

For example, the c-with-cedilla character ç (U+00E7) can be represented as \u00E7, and the G clef character (U+1D11E) can be represented as \uD834\uDD1E.

XML

In XML, Unicode characters should be escaped using the representation: underscore, followed by the lowercase letter x, followed by the followed by the character’s four-hexadecimal-digit code-point representation, followed by an underscore.

For example, the c-with-cedilla character ç (U+00E7) should be represented as _x00E7_, and the G clef character (U+1D11E) should be represented as _xD834__xDD1E_.