Datomic Adaptive Indexing

24 February 2014

We are pleased to announce today new Adaptive Indexing support for Datomic. Adaptive indexing involves both a new index format and algorithm for how indexes are maintained. It should prove especially beneficial to those with large databases, high write loads, or large imports.

Some of the benefits:

Reduced work to be done per indexing job

There will be fewer index segments written, which should take less time and/or allow for reduced write provisioning when using DynamoDB.

Reduced memory requirements

We have revised downwards both the default and recommended thresholds for memory-index-threshold (to 32m) and memory-index-max (to 512m), for all workloads. This will also reduce memory pressure on peers, which are similarly configured. In addition, indexing itself uses less memory and has better GC characteristics. You will not approach memory-index-max except during imports and bulk loads.

Simpler configuration and less variability

You should rarely need to diverge from the defaults, and generally can use the same configuration for imports and ongoing production.

Sustainable import rates independent of db size

You will see indexing job times flatten out even as your db size grows linearly, as there is a sub-linear worst-case relationship between db size and indexing job size All of this is done while minimizing the amount of merging done during reads.

TL;DR - much better and more predictable performance, using fewer resources.

We expect adaptive indexing to make a tangible difference to most customers, and look forward to your feedback. Be sure to read the release notices for important details.

Datomic Lookup Refs

13 January 2014

Datomic's new lookup refs provide an easy way to identify entities by domain unique identifiers.

Rationale

Datomic supports several approaches to identity, including internal identifiers (entity IDs that are unique within a database) and external identifiers (domain attributes that are designated to be :db.unique/identity). Lookup refs provide a concise data representation that accords to external ids the powers of internal ids, and can be used interchangeably with internal ids in the Datomic API. This is powerful, since often, e.g. in the front-end of a system or during imports, you already have the external ids in hand. With lookups refs you can often avoid having to incorporate a lookup in application code since the lookup will be done implicitly when resolving the lookup ref (thus the name).

Implementation

A lookup ref is simply a java.util.List containing two elements: a unique attribute and a value for that attribute. Encoded in edn, a Lookup ref looks like this:

[:org/email "info@datomic.com"]

Lookup refs can be used in the entity API, or in any of the database index APIs. For example, the following code retrieves an entity:

// Groovy
db.entity([':org/email', 'info@datomic.com'])

;; Clojure
(d/entity db [:org/email "info@datomic.com"])

Lookup refs can also be used to refer to existing entities in transaction data. This allows updating transactions to specify :db/id directly instead of using a dummy upsert, as shown below:

;; edn transaction data, using temp :db/id and dummy upsert
[{:db/id #db/id[:db.part/user]
 :org/email "info@datomic.com",
 :org/favoriteColor :blue}]

;; edn transaction data, using lookup ref
[{:db/id [:org/email "info@datomic.com"],
 :org/favoriteColor :blue}]

Similarly, they can be used in transactions to build ref relationships.

We expect that lookup refs will expedite a wide variety of use cases, including imports, interactive exploration, and client application development.

We look forward to your feedback.

Schema Alteration

23 December 2013

Datomic is a database that has flexible, minimal schema. Starting with version 0.9.4470, available here, we have added the ability to alter existing schema attributes after they are first defined. You can alter schema to

rename attributes
rename your own programmatic identities (uses of :db/ident)
add or remove indexes
add or remove uniqueness constraints
change attribute cardinality
change whether history is retained for an attribute
change whether an attribute is treated as a component

Schema alterations use the same transaction API as all other transactions, just as schema installation does. All schema alterations can be performed while a database is online, without requiring database downtime. Most schema changes are effective immediately, at the end of the transaction. There is one exception: adding an index requires a background job to build the new index. You can use the new syncSchema API for detecting when a schema change is available.

When renaming an attribute or identity, you can continue to use the old name as long as you haven't repurposed it. This allows for incremental application updating.

See the schema alteration docs for the details.

Schema alteration has been our most requested enhancement. We hope you find it useful and look forward to your feedback.

Datomic 2013 Recap

08 December 2013

2013 was a great year for Datomic. The value of a flexible information model and immutable data have proven themselves time and again. Customers have built a variety of powerful systems, taking advantage of

ACID transactions
pluggable SQL/NoSQL/cloud storage
complete access to the history of information
the Datalog query language
elastic read scalability
a granular information model

Over the course of the year, we produced over 40 Datomic releases. The API has been remarkably stable: Our commitment to a strong architecture has allowed us to focus on adding features and fleshing out the vision, without the churn of revisiting past decisions.

A major new feature is the Datomic Console, a graphical UI for exploring Datomic databases. The console provides a great visual introduction to the Datomic information model. It supports exploring schema, building and executing queries, navigating entities, examining transaction history, and walking raw indexes.

We made several API additions:

Excision, a sound model (and API) for permanent removal of data, with auditability.
The log API provides the ability to access the log, which is more properly viewed as a time index.
The seekDatoms and entidAt APIs provide advanced capability for accessing Datomic's indexes, augmenting the datoms API.
The sync API allows multiple processes to coordinate around points in time-of-record, or relative to local process time.
Transaction map expansion automates the creation of arbitrarily nested data.

We also made a number of operational improvements:

We added Cassandra and to the list of supported storage, in addition to the existing options of DynamoDB, SQL, filesystem, CouchBase, Infinispan, and Riak.
The Starter Edition of the Datomic Pro license makes all storages available, for free.
We have added a number of new CloudWatch metrics, and a pluggable metrics API for integration with other systems.
The MusicBrainz sample database is a great dataset for exploring Datomic.
We continue to track AWS best practices, now supporting IAM roles for distributing credentials and DynamoDB local for testing.

We are looking forward to an equally exciting 2014. We will be delivering a number of new features requested by users, plus a few big surprises.

Many thanks to our customers and early adopters for your support and feedback.

Happy New Year!

Cassandra Support

18 November 2013

We are pleased to announce alpha support for Cassandra as a storage service for Datomic, now available in version 0.9.4384.

Cassandra is an elastically scalable, distributed, redundant and highly-available column store. Recent versions of Cassandra added support for compare-and-swap operations via lightweight
transactions using an implementation of the Paxos protocol. Datomic leverages this mechanism to manage a small number of keys per database that require coordinated access, while the bulk of a database's content is written as immutable data to a quorum of replicas in the cluster.

Cassandra support requires a minimum Apache Cassandra 2.0.2 or newer, or equivalent. Native CQL protocol support must be enabled. Cross-data center deployments are not supported. Cassandra internal security is supported, but optional.

This release represents preliminary support based on requests from users. We are very interested in feedback.

For instructions on configuring Cassandra for use with Datomic, see Setting Up Storage Services.

PREVIOUS 9 of 13 NEXT

Datomic Adaptive Indexing

Some of the benefits:

Datomic Lookup Refs

Rationale

Implementation

Schema Alteration

Datomic 2013 Recap

Cassandra Support

Datomic Pro

Datomic Cloud

Resources

Company