attributes.html

---
# Copyright Vespa.ai. All rights reserved.
title: "Attributes"
redirect_from:
- /en/attributes.html
---

<p>
  An <em>attribute</em> is a <a href="../reference/schemas/schemas.html#attribute">schema</a> keyword,
  specifying the indexing for a field:
</p>
<pre>
field price type int {
    indexing: attribute
}
</pre>
<p>
  Attribute properties and use cases:
</p>
<ul>
  <li>Flexible <a href="../reference/schemas/schemas.html#match">match modes</a>
    including exact match, prefix match, and case-sensitive matching, but not text matching (tokenization and linguistic processing).</li>
  <li>High sustained update rates (avoiding read-apply-write patterns). Any mutating operation against an attribute field 
    is written to Vespa's <a href="proton.html#transaction-log">transaction log</a> and persisted, but appending to the log is sequential access, not random.
    Read more in <a href="../writing/partial-updates.html">partial updates</a>.</li>
  <li>Instant query updates - values are immediately searchable.</li>
  <li><a href="../querying/document-summaries.html">Document Summaries</a> are memory-only operations if all fields are attributes.</li>
  <li><a href="../reference/querying/yql.html#numeric">Numerical range queries</a>.
<pre>where price > 100</pre></li>
  <li><a href="../querying/grouping.html">Grouping</a> - aggregate results into groups -
    it is also great for generating diversity in results.
<pre>
all(group(customer) each(max(3) each(output(summary()))))
</pre></li>
  <li><a href="../basics/ranking.html">Ranking</a> - use attribute values directly in rank functions.
<pre>
rank-profile rank_fresh {
    first-phase {
        expression { freshness(timestamp) }
    }
}
</pre></li>
  <li><a href="../reference/querying/sorting-language.html">Sorting</a> - order results by attribute value.
<pre>
order by price asc, release_date desc
</pre></li>
  <li><a href="../schemas/parent-child.html">Parent/child</a> - import attribute values from global parent documents.
<pre>
import field advertiser_ref.name as advertiser_name {}
</pre></li>

</ul>

<p>
  The other field option is <em>index</em> -
  use <a href="proton.html#index">index</a> for fields used for
  <a href="../querying/text-matching.html">text search</a>,
  with <a href="../linguistics/linguistics-opennlp.html#stemming">stemming</a> and
  <a href="../linguistics/linguistics-opennlp.html#normalization">normalization</a>.
</p>
<p>
  An attribute is an in-memory data structure.
  Attributes speed up query execution and <a href="../writing/partial-updates.html">document updates</a>, trading off memory.
  As data structures are regularly optimized, consider both static and temporary resource usage -
  see <a href="#attribute-memory-usage">attribute memory usage</a> below.
  Use attributes in document summaries to limit access to storage to generate result sets.
</p>
<img src="/assets/img/attributes-update.svg" width="788px" height="auto"
     alt="Attribute is an in-memory data structure" />
<p>
  Configuration overview:
</p>
<table class="table">
  <tr>
    <th style="white-space:nowrap;">fast-search</th>
    <td>
      <p id="fast-search">
      Also see the <a href="../reference/schemas/schemas.html#attribute">reference</a>.
      Add an <a href="#index-structures">index structure</a> to improve query performance:
      </p>
<pre>
field titles type array&lt;string&gt; {
    indexing : summary | attribute
    attribute: fast-search
}
</pre>
    </td>
  </tr>
  <tr>
    <th style="white-space:nowrap;">fast-access</th>
    <td>
      <p id="fast-access">
      For high-throughput updates, all nodes with a replica should have the attribute loaded in memory.
      Depending on replication factor and other configuration, this is not always the case.
      Use <a href="../reference/schemas/schemas.html#attribute">fast-access</a>
      to increase feed rates by having replicas on all nodes in memory -
      see the <a href="../reference/schemas/schemas.html#attribute">reference</a>
      and  <a href="../performance/sizing-feeding.html">sizing feeding</a>.
      </p>
<pre>
field titles type array&lt;string&gt; {
    indexing : summary | attribute
    attribute: fast-access
}
</pre>
    </td>
  </tr><tr>
    <th style="white-space:nowrap;">distance-metric</th>
    <td>
      <p id="distance-metric">
      Features like <a href="../querying/nearest-neighbor-search">nearest neighbor search</a>
      require a <a href="../reference/schemas/schemas.html#distance-metric">distance-metric</a>,
      and can also have an <code>hsnw index</code> to speed up queries.
      Read more in <a href="../querying/approximate-nn-hnsw">approximate nearest neighbor</a>.
      Pay attention to the field's <code>index</code> setting to enable the index:
      </p>
<pre>
field image_sift_encoding type tensor&lt;float&gt;(x[128]) {
    indexing: summary | attribute | <span class="pre-hilite">index</span>
    attribute {
        distance-metric: euclidean
    }
    <span class="pre-hilite">index</span> {
        hnsw {
            max-links-per-node: 16
            neighbors-to-explore-at-insert: 500
        }
    }
}
</pre>
    </td>
  </tr>
</table>



<h2 id="data-structures">Data structures</h2>
<p>
  The attribute field's data type decides which data structures are used by the attribute
  to store values for that field across all documents on a content node.
  For some data types, a combination of data structures is used:
</p>
<ul>
  <li><em>Attribute Multivalue Mapping</em> stores arrays of values for array and weighed set types.</li>
  <li><em>Attribute Enum Store</em> stores unique strings for all string attributes and unique values for
  attributes with <a href="attributes.html#fast-search">fast-search</a>.</li>
  <li><em>Attribute Tensor Store</em> stores tensor values for all tensor attributes.</li>
</ul>
<p>
  In the following illustration, a row represents a document, while a named column represents an attribute.
</p>
<img src="/assets/img/attributes.svg" width="668px" height="auto"
     alt="Attribute in-memory stores"/>
<p>
  Attributes can be:
</p>
<table class="table">
  <tr>
    <th>Type</th>
    <th>Size</th>
    <th>Description</th>
  </tr>
  <tr>
    <td style="white-space:nowrap;">Single-valued</td>
    <td>Fixed</td>
    <td>
      Like the "A" attribute, example <code>int</code>.
      The element size is the size of the type, like 4 bytes for an integer.
      A memory buffer (indexed by Local ID) holds all values directly.
    </td>
  </tr><tr>
    <td>Multi-valued</td>
    <td>Fixed</td>
   <td>
      Like the "B" attribute, example <code>array&lt;int&gt;</code>.
      A memory buffer (indexed by Local ID) is holding references (32 bit) to where in the <em>Multivalue Mapping</em> the arrays are stored.
      The <em>Multivalue Mapping</em> consists of multiple memory buffers, where arrays of the same size are co-located in the same buffer.
    </td>
  </tr><tr>
    <td>Multi-valued</td>
    <td>Variable</td>
  <td>
      Like the "B" attribute, example <code>array&lt;string&gt;</code>.
      A memory buffer (indexed by Local ID) is holding references (32 bit) to where in the <em>Multivalue Mapping</em> the arrays are stored.
      The unique strings are stored in the <em>Enum Store</em>, and the arrays in the <em>Multivalue Mapping</em>
      stores the references (32 bit) to the strings in the <em>Enum Store</em>.
      The <em>Enum Store</em> consists of multiple memory buffers.
    </td>
  </tr><tr>
    <td>Single-valued</td>
    <td>Variable</td>
    <td>
      Like the "C" attribute, example <code>string</code>.
      A memory buffer (indexed by Local ID) is holding references (32 bit) to where in the <em>Enum Store</em> the strings are stored.
    </td>
  </tr><tr>
    <td>Tensor</td>
    <td>Fixed / Variable</td>
    <td>
      Like the "D" attribute, example <code>tensor&lt;float&gt;(x{},y[64])</code>.
      A memory buffer (indexed by Local ID) is holding references (32 bit) to where in the <em>Tensor Store</em> the tensor values are stored.
      The memory layout in the <em>Tensor Store</em> depends on the tensor type.
    </td>
  </tr>
</table>
<p>
  The "A", "B", "C" and "D" attribute memory buffers have attribute values or references in Local ID (LID) order -
  see <a href="#document-meta-store">document meta store</a>.
</p>
<p>
  When updating an attribute, the full value is written.
  This also applies to <a href="../basics/schemas.html#document-fields">multivalue</a> fields -
  example adding an item to an array:
</p>
<ol>
  <li>Space for the new array is reserved in a memory buffer</li>
  <li>The current value is copied</li>
  <li>The new element is written</li>
</ol>
<p>
  This means that larger fields will copy more data at updates.
  It also implies that updates to <a href="../reference/schemas/schemas.html#weightedset">weighted sets</a>
  are faster when using numeric keys (less memory and easier comparisons).
</p>
<p>
  Data stored in the <em>Multivalue Mapping</em>, <em>Enum Store</em> and <em>Tensor Store</em>
  is referenced using 32 bit references.
  This address space can go full, and then feeding is blocked - <a href="../writing/feed-block.html">learn more</a>.
  For array or weighted set attributes,
  the max limit on the number of documents that can have the same number of values is approx 2 billion per node.
  For string attributes or attributes with
  <a href="attributes.html#fast-search">fast-search</a>,
  the max limit on the number of unique values is approx 2 billion per node.
</p>



<h2 id="index-structures">Index structures</h2>
<p>
  Without <code>fast-search</code>, attribute access is a memory lookup,
  being one value or all values, depending on query execution.
  An attribute is a linear array-like data structure -
  matching documents potentially means scanning <em>all</em> attribute values.
</p>
<p>
  Setting <a href="../reference/schemas/schemas.html#attribute">fast-search</a>
  creates an index structure for quicker lookup and search.
  This consists of a <a href="../reference/schemas/schemas.html#dictionary">dictionary</a> pointing to posting lists.
  This uses more memory, and also more CPU when updating documents.
  It increases steady state memory usage for all attribute types
  and also add initialization overhead for numeric types.
</p>
<p>
  The default dictionary is a b-tree of attribute <em>values</em>,
  pointing to an <em>occurrence</em> b-tree (posting list) of local doc IDs for each value, exemplified in the A-attribute below.
  Using <code>dictionary: hash</code> on the attribute generates a hash table of attributes values pointing to the posting lists,
  as in the C-attribute (short posting lists are represented as arrays instead of b-trees):
</p>
<img src="/assets/img/attributes-indexes.svg" width="951px" height="auto"
     alt="Attribute index structures"/>
<p>
Notes:
</p>
<ul>
  <li>If a value occurs in many documents, the <em>occurrence</em> b-tree grows large.
    For such values, a boolean-occurrence list (i.e. bitvector) is generated in addition to the b-tree.</li>
  <li>Setting <code>fast-search</code> is not observable in the files on disk, other than size.</li>
  <li><code>fast-search</code> causes a memory increase even for empty fields,
    due to the extra index structures created.
    E.g. single value fields will have the "undefined value" when empty,
    and there is a posting list for this value.</li>
  <li>The <em>value</em> b-tree enables fast range-searches in numerical attributes.
    This is also available for <code>hash</code>-based dictionaries, but slower as a full scan is needed.</li>
</ul>
<p>
  Using <code>fast-search</code> has many implications, read more in
  <a href="../performance/feature-tuning.html#when-to-use-fast-search-for-attribute-fields">when to use fast-search</a>.
</p>



<h2 id="attribute-memory-usage">Attribute memory usage</h2>
<p>
  Attribute structures are regularly optimized, and this causes temporary resource usage -
  read more in <a href="proton.html#proton-maintenance-jobs">maintenance jobs</a>.
  The memory footprint of an attribute depends on a few factors,
  data type being the most important:
</p>
<ul>
  <li>Numeric (int, long, byte, and double) and Boolean (bit) types  -
    fixed length and fix cost per document</li>
  <li>String type - the footprint depends on the length of the
    strings and how many unique strings that needs to be
    stored.</li>
</ul>
<p>
  Collection types like array and weighted sets increases the memory
  usage some, but the main factor is the average number of values per document.
  String attributes are typically the largest attributes,
  and requires most memory during initialization - use boolean/numeric types where possible.
  Example, refer to formulas below:
</p>
<pre>
schema foo {
    document bar {
        field titles type array&lt;string&gt; {
            indexing: summary | attribute
        }
    }
}
</pre>
<ul>
  <li>Assume average 10 values per document, average string length 15, 100k unique strings and 20M documents.</li>
  <li>Steady state memory usage is approx 1 GB (20M*4*(6/5) + 20M*10*4*(6/5) + 100k*(15+1+4+4)*(6/5)).</li>
  <li>During initialization (loading attribute from disk) an additional 2.4 GB is allocated (20M*10*(4+4+4),
    for each value:
    <ul>
      <li>local document id</li>
      <li>enum value</li>
      <li>weight</li>
    </ul>
  </li>
  <li>Increasing the average number of values per document to 20 (double) will also double the memory
    footprint during initialization (4.8 GB).</li>
</ul>
<p>
  When doing the capacity planning, keep in mind the maximum footprint, which occurs during initialization.
  For the steady state footprint, the number of unique values is important for string attributes.
</p><p>
  Check the <a href="../../assets/attribute-memory-Vespa.xls">Example attribute sizing spreadsheet</a>,
  with various data types and collection types.
  It also contains estimates for how many documents a 48 GB RAM node can hold, taking initialization into account.
</p>
<p>
  <a href="../basics/schemas.html#document-fields">Multivalue</a> attributes use an
  adaptive approach in how data is stored in memory,
  and up to 2 billion documents per node is supported.
</p>
<p>
  <strong>Pro-tip:</strong> The proton <em>/state/v1/</em> interface can be explored for attribute memory usage.
  This is an undocumented debug-interface, subject to change at any moment - example:
  <em>http://localhost:19110/state/v1/custom/component/documentdb/music/subdb/ready/attribute/artist</em>
</p>



<h2 id="attribute-file-usage">Attribute file usage</h2>
<p>Attribute data is stored in two locations on disk:</p>
<ul>
  <li>
    <p>
      The attribute store in memory, which is regularly flushed to disk.
      At startup, the flushed files are used to quickly populate the memory structures,
      resulting in a much quicker startup compared to generating the attribute store from the source in the document store.
    </p>
    <p>
      The attribute store will temporarily double its disk usage when generating a new flush file,
      see <a href="/en/content/proton.html#attribute-flush">attribute flush</a>.
    </p>
  </li>
  <li>
    <p>
      The document store on disk.
      Documents here are used to (re)generate index structures,
      as well as being the source for replica generation across nodes.
    </p>
    <p>
      Note that the attribute data is stored in the document store regardless of
      the <a href="/en/querying/document-summaries.html">summary</a> configuration.
    </p>
  </li>
</ul>
<p>
  The different field types use various data types for storage, see below,
  a conservative rule of thumb for steady-state disk usage is hence twice the data size.
</p>



<h2 id="sizing">Sizing</h2>
<p>
  Attribute sizing is not an exact science but rather an  approximation.
  The reason is that they vary in size.
  Both the number of documents, number of values, and uniqueness of the values are variable.
  The components of the attributes that occupy memory are:
</p>
<table class="table">
  <thead>
  <tr>
    <th>Abbreviation</th>
    <th>Concept</th>
    <th>Comment</th>
  </tr>
  </thead>
  <tbody>
  <tr>
    <td>D</td>
    <td>Number of documents</td>
    <td>Number of documents on the node, or rather the maximum number of local document ids allocated</td>
  </tr>
  <tr>
    <td>V</td>
    <td>Average number of values per document</td>
    <td>Only applicable for arrays and weighted sets</td>
  </tr>
  <tr>
    <td>U</td>
    <td>Number of unique values</td>
    <td>Only applies for strings or if
      <a href="../reference/schemas/schemas.html#attribute">fast-search</a> is set</td>
  </tr>
  <tr>
    <td>FW</td>
    <td>Fixed data width</td>
    <td>sizeof(T) for numerics, 1 byte for strings, 1 bit for boolean</td>
  </tr>
  <tr>
    <td>WW</td>
    <td>Weight width</td>
    <td>Width of the weight in a weighted set, 4 bytes. 0 bytes for arrays.</td>
  </tr>
  <tr>
    <td>EIW</td>
    <td>Enum index width</td>
    <td>Width of the index into the enum store, 4 bytes. Used by all strings and other attributes if
      <a href="../reference/schemas/schemas.html#attribute">fast-search</a> is set</td>
  </tr>
  <tr>
    <td>VW</td>
    <td>Variable data width</td>
    <td>strlen(s) for strings, 0 bytes for the rest</td>
  </tr>
  <tr>
    <td>PW</td>
    <td>Posting entry width</td>
    <td>Width of a posting list entry, 4 bytes for singlevalue, 8 bytes for array and weighted sets.
      Only applies if <a href="../reference/schemas/schemas.html#attribute">fast-search</a> is set.</td>
  </tr>
  <tr>
    <td>PIW</td>
    <td>Posting index width</td>
    <td>Width of the index into the store of posting lists; 4 bytes</td>
  </tr>
  <tr>
    <td>MIW</td>
    <td>Multivalue index width</td>
    <td>Width of the index into the multivalue mapping; 4 bytes</td>
  </tr>
  <tr>
    <td>ROF</td>
    <td>Resize overhead factor</td>
    <td>Default is 6/5. This is the average overhead in any dynamic vector due to
      resizing strategy. Resize strategy is 50% indicating that
      structure is 5/6 full on average.</td>
  </tr>
  </tbody>
</table>


<h3 id="components">Components</h3>
<table class="table">
  <thead>
  <tr>
    <th>Component</th>
    <th>Formula</th>
    <th>Approx Factor</th>
    <th>Applies to</th>
  </tr>
  </thead>

  <tbody>
  <tr>
    <td>Document vector</td>
    <td>D * ((FW or EIW) or MIW)</td>
    <td>ROF</td>
    <td>FW for singlevalue numeric attributes and MIW for
      multivalue attributes. EIW for singlevalue string or if the
      attribute is singlevalue fast-search</td>
  </tr>
  <tr>
    <td>Multivalue mapping</td>
    <td>D * V * ((FW or EIW) + WW)</td>
    <td>ROF</td>
    <td>Applicable only for array or weighted sets. EIW if string or fast-search</td>
  </tr>
  <tr>
    <td>Enum store</td>
    <td>U * ((FW + VW) + 4 + ((EIW + PIW) or EIW))</td>
    <td>ROF</td>
    <td>Applicable for strings or if fast-search is set.
      (EIW + PIW) if fast-search is set, EIW otherwise.</td>
  </tr>
  <tr>
    <td>Posting list</td>
    <td>D * V * PW</td>
    <td>ROF</td>
    <td>Applicable if fast-search is set</td>
  </tr>
  </tbody>
</table>


<h3 id="variants">Variants</h3>
<table class="table">
  <thead>
  <tr>
    <th>Type</th>
    <th>Components</th>
    <th>Formula</th>
  </tr>
  </thead>

  <tbody>
  <tr>
    <td>Numeric singlevalue plain</td>
    <td>Document vector</td>
    <td>D * FW * ROF</td>
  </tr>
  <tr>
    <td>Numeric multivalue value plain</td>
    <td>Document vector, Multivalue mapping</td>
    <td>D * MIW * ROF + D * V * (FW+WW) * ROF </td>
  </tr>
  <tr>
    <td>Numeric singlevalue fast-search</td>
    <td>Document vector, Enum store, Posting List</td>
    <td>D * EIW * ROF + U * (FW+4+EIW+PIW) * ROF + D * PW * ROF</td>
  </tr>
  <tr>
    <td>Numeric multivalue value fast-search</td>
    <td>Document vector, Multivalue mapping, Enum store, Posting List</td>
    <td>D * MIW * ROF + D * V * (EIW+WW) * ROF + U * (FW+4+EIW+PIW) * ROF + D * V * PW * ROF</td>
  </tr>
  <tr>
    <td>Singlevalue string plain</td>
    <td>Document vector, Enum store</td>
    <td>D * EIW * ROF + U * (FW+VW+4+EIW) * ROF</td>
  </tr>
  <tr>
    <td>Singlevalue string fast-search</td>
    <td>Document vector, Enum store, Posting List</td>
    <td>D * EIW * ROF + U * (FW+VW+4+EIW+PIW) * ROF + D * PW * ROF</td>
  </tr>
  <tr>
    <td>Multivalue string plain</td>
    <td>Document vector, Multivalue mapping, Enum store</td>
    <td>D * MIW * ROF + D * V * (EIW+WW) * ROF + U * (FW+VW+4+EIW) * ROF</td>
  </tr>
  <tr>
    <td>Multivalue string fast-search</td>
    <td>Document vector, Multivalue mapping, Enum store, Posting list</td>
    <td>D * MIW * ROF + D * V * (EIW+WW) * ROF + U * (FW+VW+4+EIW+PIW) * ROF + D * V * PW * ROF</td>
  </tr>
  <tr>
    <td>Boolean singlevalue</td>
    <td>Document vector</td>
    <td>D * FW * ROF</td>
  </tr>
  </tbody>
</table>



<h2 id="paged-attributes">Paged attributes</h2>
<p>
  Regular attribute fields are guaranteed to be in-memory, while the
  <a href="../reference/schemas/schemas.html#attribute">paged</a>
  attribute setting allows paging the attribute data out of memory to disk.
  The <code>paged</code> setting is <em>not</em> supported for the following types:
</p>
  <ul>
    <li><a href="../reference/schemas/schemas.html#tensor">tensor</a> with
        <a href="../reference/schemas/schemas.html#attribute">fast-rank</a>.</li>
    <li><a href="../reference/schemas/schemas.html#predicate">predicate</a>.</li>
  </ul>
<p>
  For attribute fields using
  <a href="../reference/schemas/schemas.html#attribute">fast-search</a>,
  the memory needed for dictionary and index structures are never paged out to disk.
</p>
<p>
  Using the <code>paged</code> setting for attributes
  is an alternative when there are memory resource constraints
  and the attribute data is only accessed by a limited number of hits per query during ranking.
  E.g. a dense tensor attribute which is only used during a <a href="../ranking/phased-ranking.html">re-ranking phase</a>,
  where the number of attribute accesses are limited by the re-ranking phase count.
</p>
<p>
  For example using a second phase <a href="../reference/schemas/schemas.html#secondphase-total-rerank-count">total-rerank-count</a>
  of 100 will limit the maximum number of page-ins/disk access per query to 100.
  Running at 100 QPS would need up to 10K disk accesses per second.
  This is the worst case if none of the accessed attribute data were paged into memory already.
  This depends on access locality and memory pressure (size of the attribute data versus available memory).
</p>
<p>
  In this example, we have a dense tensor with
  1024 <a href="../reference/ranking/tensor.html#tensor-type-spec">int8</a> values.
  The tensor attribute is only accessed during re-ranking (second-phase ranking expression):
</p>
<pre>
schema foo {
    document foo {
        field tensordata type tensor&lt;int8&gt;(x[1024]) {
            indexing: attribute
            attribute: paged
        }
    }
    rank-profile foo {
        first-phase {}
        second-phase {
            total-rerank-count: 100
            expression: sum(attribute(tensordata))
        }
    }
}
</pre>
<p>
  For some use cases where serving latency SLA is not strict and query throughput is low,
  the <code>paged</code> attribute setting might be a tuning alternative,
  as it allows storing more data per node.
</p>


<h3 id="paged-attributes-disadvantages">Paged attributes disadvantages</h3>
<p>
  The disadvantages of using <em>paged</em> attributes are many:
</p>
<ul>
<li>
Unpredictable query latency as attribute access might touch disk. Limited queries per second throughput
per node (depends on the locality of document re-ranking requests).
</li>
<li>
Paged attributes are implemented by file-backed memory mappings. The performance depends on
the <a href="https://tldp.org/LDP/tlk/mm/memory.html" data-proofer-ignore>Linux virtual memory management</a> ability to page
data in and out. Using many threads per search/high query throughput might cause high system (kernel) CPU and
system unresponsiveness.
</li>
<li>
The content node's total memory utilization will be close to 100% when using paged attributes.
It's up to the Linux kernel to determine what part of the attribute data is paged into memory based on access patterns.
A good understanding of how the Linux virtual memory management system works is recommended before enabling paged attributes.
</li>
<li>
The
<a href="/en/performance/sizing-search.html#metrics-for-vespa-sizing">memory usage metrics</a>
from content nodes are not reflecting the reality when using paged attributes.
They can indicate a usage that is much higher than the available memory on the node.
This is because attribute memory usage is reported as the amount of data contained in the attribute,
and whether this data is paged out to disk is controlled by the Linux kernel.
</li>
<li>
Using paged attributes doubles the disk usage of attribute data.
For example if the original attribute size is 92 GB (100M documents of the above 1024 int8 per document schema),
using the <code>paged</code> setting will double the attribute disk usage to close to 200 GB.
</li>
<li>
Changing the <code>paged</code> setting (e.g. removing the option) on a running system might cause hard out-of-memory situations as
without <code>paged</code>, the content nodes will attempt loading the attribute into memory without the option for page outs.
</li>
<li>
Using a paged attribute in <a href="../ranking/phased-ranking.html">first-phase</a> ranking
can result in extremely high query latency if a large amount of the corpus is retrieved by the query.
The number of disk accesses will, in the worst case, be equal to the number of hits the query produces.
A similar problem can occur if running a query that searches a paged attribute.
</li>
<li>
Using <code>paged</code> in combination with <a href="../querying/approximate-nn-hnsw">HNSW indexing</a> is <em>strongly</em> discouraged.
<em>HNSW</em> indexing
also searches and reads tensors during indexing, causing random access during feeding. Once the system memory usage reaches 100%,
the Linux kernel will start paging pages in and out of memory. This can cause a high system (kernel) CPU and slows down HNSW indexing
throughput significantly.
</li>
</ul>



<h2 id="mutable-attributes">Mutable attributes</h2>
<p>
  <a href="../reference/schemas/schemas.html#mutate">Mutable attributes</a> is document metadata
  for matching and ranking performance per document.
</p>
<p>
  The attribute values are mutated as part of query execution, as defined in rank profiles -
  see <a href="../ranking/phased-ranking.html#rank-phase-statistics">rank phase statistics</a> for details.
</p>



<h2 id="document-meta-store">Document meta store</h2>
<p>
  The document meta store is an in-memory data structure for all documents on a node.
  It is an <em>implicit attribute</em>, and is
  <a href="proton.html#lid-space-compaction">compacted</a> and
  <a href="proton.html#attribute-flush">flushed</a>.
  Memory usage for applications with small documents / no other attributes can be dominated by this attribute.
</p><p>
  The document meta store scales linearly with number of documents -
  using approximately 30 bytes per document.
  The metric <em>content.proton.documentdb.ready.attribute.memory_usage.allocated_bytes</em> for
  <code>"field": "[documentmetastore]"</code> is the size of the document meta store in memory - use the
  <a href="../reference/api/state-v1.html#state-v1-metrics">metric API</a> to find the size -
  in this example, the node has 9M ready documents with 52 bytes in memory per document:
</p>
<pre>
{
    "name": "content.proton.documentdb.ready.attribute.memory_usage.allocated_bytes",
    "description": "The number of allocated bytes",
    "values": {
        "average": 4.69736008E8,
        "count": 12,
        "rate": 0.2,
        "min": 469736008,
        "max": 469736008,
        <span class="pre-hilite">"last": 469736008</span>
    },
    "dimensions": {
        "documenttype": "doctype",
        <span class="pre-hilite">"field": "[documentmetastore]"</span>
    }
},
</pre>
<p>
  The above is for the <em>ready</em> documents,
  also check <em>removed</em> and <em>notready</em> -
  refer to <a href="proton.html#sub-databases">sub-databases</a>.
</p>