10 July 2016

The most recent Datomic Release provides access to the Datomic Log API for memory databases. I would like to take this opportunity to describe some of the features and uses of the Datomic Log API.

The transactional log is a fundamental component of ACID database systems, a durable record of the transactions performed by the database. In addition to its critical function in ensuring ACID semantics, the Datomic log, as a sequential (in database time, t) record of all transactions, also functions as a time-ordered index to the data stored in a Datomic database.

Datomic provides access to the transaction log directly via the tx-range function and from within query using the tx-ids and tx-data functions.

An example using the log in query is now available in the Day of Datomic repo. Our example database records the streets on which our three protagonists, John, Mary, and Joe, live.

Let’s find out when Joe moved to Broadway:

(d/q '[:find ?tx
 :in $ ?name ?street
 :where
 [?e :person/name ?name]
 [?e :person/street ?street ?tx true]]
 (d/history (d/db conn)) "Joe" "Broadway")

This query returns 13194139534317, the transaction ID of the transaction that asserted Joe’s street is Broadway. As in all Datomic databases, every transaction also records a timestamp, the :db/txInstant. Let’s see what wall-clock time is associated with this transaction entity:

(d/pull (d/db conn) '[:db/txInstant] 13194139534317)

So Joe moved to Broadway in 1983.

Issuing queries against the transaction log is a powerful approach for auditing the operational history of a database. Because every transaction is an entity in Datomic, we can easily retrieve the entire set of datoms for a given transaction entity. Let’s find out what else happened in the ‘Joe moves to Broadway’ transaction. This query returns all the datoms associated with the given transaction:

(d/q '[:find ?e ?a ?v ?tx ?op
 :in ?log ?tx
 :where [(tx-data ?log ?tx)[[?e ?a ?v _ ?op]]]]
 (d/log conn) 13194139534317)

;; result:
#{[17592186045420 64 "2nd" 13194139534317 true]
 [17592186045419 64 "Broadway" 13194139534317 true]
 [13194139534317 50 #inst"1983-01-01T00:00:00.000-00:00" 13194139534317 true]
 [17592186045420 64 "Elm" 13194139534317 false]
 [17592186045419 64 "1st" 13194139534317 false]}

Note that we see the same wall clock time we found previously as well as 4 other datoms. One is the assertion of Joe moving to Broadway, one is the retraction of his previous street (1st), and the remaining two datoms are about someone else entirely. Let’s find out who:

(d/pull (d/db conn) '[*] 17592186045420)

;; result:
{:db/id 17592186045420, :person/name "Mary", :person/street "2nd"}

By using the log, we’ve determined that Mary’s move to 2nd Street was recorded at the same time (during the same transaction) as Joe's move to Broadway.

The ability to query the Datomic transaction log directly is a powerful tool for managing, administering, and using a Datomic database. The addition of the Log API to memory databases enables low-overhead testing and evaluation of Datomic’s Log API feature. We hope you find the Log API on memory databases a helpful addition for lightweight development and unit testing.

---