Datomic is a general purpose database system designed for data-of-record applications. A Datomic database is a set of immutable atomic facts called datoms. Datomic transactions add datoms, never updating or removing them, so you have a complete audit trail for data and the ability to query “as of” points in time. Datomic transactions are serialized in a total order, providing strong ACID guarantees.
Datomic has a powerful and flexible information model. Each Datomic database declares an attribute-level schema, and any entity can possess any attribute. You can query your data with datalog (a logic-based query) and navigate arbitrary hierarchies with pull. Datomic’s indexes automatically support many access patterns common in SQL, column, K/V, hierarchical, and graph databases.
Datomic is developed by Nubank, one of the world’s largest digital financial services platforms, serving over 100 million customers across Brazil, Mexico, and Colombia. At Nubank we use Datomic for almost all of our suite of products resulting in an average of 2.5 billion Datomic transactions being processed each day!
What is Jepsen?
Jepsen is a small consultancy focused on distributed systems safety. They write public reports, maintain open-source tooling, and offer training courses to make databases, queues, and other systems safer. Jepsen works with vendors like Nubank to evaluate their documentation and system behavior under a variety of failure modes, helps those vendors identify and fix bugs and documentation errors, and writes an analysis presenting their findings and recommendations for both vendors and users.
Jepsen Analysis Outcome
Datomic aced the inter-transaction test and then some. Jepsen concluded:
We found no behavior which violated Datomic’s core safety claims. Transactions appeared to execute as if they had been applied in
a total order, and that order was consistent with the local order of operations on each peer. Histories restricted to just those
transactions performing writes, and histories in which reads used (d/sync conn) to obtain a current copy of the database, were
consistent with real-time order.
Indeed, we believe Datomic’s inter-transaction safety properties are stronger than promised.
Jepsen’s intra-transaction tests also confirmed Datomic’s semantics. These tests required a great deal of explanation in the report. Most databases and formalisms for serializability provide sequential execution semantics within a transaction. Datomic applies most elements of a transaction in a semantically parallel manner. As a result, we updated our docs to better explain Datomic’s transactions, how they are composed, and how they compare to other systems.
Jepsen now tests Datomic in our CI system!
Conclusion
Our partnership with Jepsen validated that Datomic is safe under a wide variety of circumstances, motivated valuable documentation improvements, and introduced a new battery of correctness tests that is now being used in our CI suite to continually ensure Datomic maintains the same high level of safety as the product evolves.
Our Building Nubank blog has more content like this, dedicated to cutting-edge technologies!
Please check out Careers at Nu if you are interested in joining our team!
..
