Datomic Adaptive Indexing

24 February 2014

We are pleased to announce today new Adaptive Indexing support for Datomic. Adaptive indexing involves both a new index format and algorithm for how indexes are maintained. It should prove especially beneficial to those with large databases, high write loads, or large imports.

Some of the benefits:

There will be fewer index segments written, which should take less time and/or allow for reduced write provisioning when using DynamoDB.

  • Reduced memory requirements
We have revised downwards both the default and recommended thresholds for memory-index-threshold (to 32m) and memory-index-max (to 512m), for all workloads. This will also reduce memory pressure on peers, which are similarly configured. In addition, indexing itself uses less memory and has better GC characteristics. You will not approach memory-index-max except during imports and bulk loads.

  • Simpler configuration and less variability
You should rarely need to diverge from the defaults, and generally can use the same configuration for imports and ongoing production.

  • Sustainable import rates independent of db size
You will see indexing job times flatten out even as your db size grows linearly, as there is a sub-linear worst-case relationship between db size and indexing job size All of this is done while minimizing the amount of merging done during reads.

TL;DR - much better and more predictable performance, using fewer resources.

We expect adaptive indexing to make a tangible difference to most customers, and look forward to your feedback. Be sure to read the release notices for important details.