If you have an application that is struggling with performance, it might be due to poor data locality, the physical distribution of relevant information. A query that is searching for sparsely distributed data will find few results per storage read. Conversely, queries can efficiently slurp relevant data when it is densely colocated. Datomic’s partitions allow applications to group entities together and gain control over locality. Entities that share a partition will be colocated in the E-leading indexes, EAVT and AEVT. Because each partition requires a name and installation prior to its use, it can become cumbersome to build schemes that use many partitions.
Today’s release of Datomic On-Prem includes a new feature: Implicit Partitions. These partitions are identified by an integer, and they are usable without an installation step. There are 524288 (2^19) implicit partitions in each Datomic database, and these facilitate the use of algorithmic partition assignment strategies, e.g. deterministically assigning each customer to a particular partition, or assigning time-windowed data to different partitions.
Applications can transact entities into implicit partitions by passing one to d/tempid:
(d/transact conn [{:db/id (d/tempid (d/implicit-part 55))
:movie/title "The Great Gatsby"}])We are also introducing two new transaction data directives that facilitate partition assignment. To request put a new entity into a particular partition, you can include its tempid in the :db/force-partition map. This can appear anywhere in a transaction map form. Here is the same transaction using this directive with a string tempid:
(d/transact conn [{:db/id "movie"
:movie/title "The Great Gatsby"
:db/force-partition {"movie" (d/implicit-part 55)}}])The :db/match-partition directive says that an entity should have the same partition as another entity. Here, both items will be in the same partition as its order, which is in implicit partition 15555:
(d/transact conn
[{:db/id "order"
:order/id 123
:order/items ["item1" "item2"]}
{:db/id "item1"
:item/name "cereal"
:item/sku "A150"}
{:db/id "item2"
:item/name "milk"
:item/sku "T770"}
{:db/force-partition {"order" (d/implicit-part 15555)}
:db/match-partition {"item1" "order"
"item2" "order"}}])To learn more about implicit partitions, read the documentation for Implicit Partitions and Controlling Partition Assignment, and the API documentation for Datomic On-Prem.
