Progressively enforce a document type

Learn: Schema

Fauna supports progressive enforcement for document types. Early in an application’s development, you can use collections with a schemaless or permissive document type to allow ad hoc fields in documents.

As your application evolves, you can use zero-downtime migrations to add stricter field definitions and to normalize field values. This lets you iteratively move from a permissive document type to a strict document type (or the reverse) based on your application’s needs.

In this tutorial, you’ll progressively define and enforce a document type for an example e-commerce application.

Before you start

To complete this tutorial, you’ll need:

Setup

Set up a database for the application.

  1. Log into Fauna using the CLI:

    fauna cloud-login

  2. Create an ecommerce database:

    fauna create-database ecommerce

  3. Create an ecommerce directory and navigate to it:

    mkdir ecommerce
cd ecommerce

  4. Initialize a project:

    fauna project init

    When prompted, use:

    • schema as the schema directory. The command creates the directory.

    • dev as the environment name.

    • The default endpoint.

    • ecommerce as the database.

Create a schemaless collection

Create a schemaless collection. A schemaless collection has no predefined document fields. Instead, you can include ad hoc fields of any type in the collection’s documents.

  1. Navigate to the schema directory:

    cd schema

  2. Create a collections.fsl file and add the following collection schema to it:

    // Stores product data for the e-commerce application.
collection Product {
  // Contains no field definitions.

  // The wildcard constraint accepts
  // ad hoc fields of any type.
  *: Any
  // If a collection schema has no field definitions
  // and no wildcard constraint, it has an implicit
  // wildcard constraint of `*: Any`.

  // Defines the `sortedByQuantity()` index.
  // Use the index to get `Product` documents sorted by
  // ascending `quantity` field value.
  index sortedByQuantity {
    values [.quantity]
  }
}

  3. Save collections.fsl. Then push the schema to Fauna:

    fauna schema push

    When prompted, accept and push the changes. This creates the collection.

Add sample data

Add documents to the Product collection.

  1. Start a shell session in the Fauna CLI:

    fauna shell

    The shell runs with the default admin key you created when you logged in using fauna cloud-login.

  2. Enter editor mode to run multi-line queries:

    > .editor

  3. Run the following FQL query:

    // Creates a `Product` collection document.
// In a schemaless collection,
// documents can contain any field of any type.
Product.create({
  name: "pinata",
  description: "Original Classic Donkey Pinata",
  price: 24.99,
  quantity: 40
})

// In a schemaless collection, a field
// can contain different types across documents.
Product.create({
  name: "cups",
  description: "Translucent 9 Oz, 100 ct",
  price: 6.98,
  // This document's `quantity` field uses a
  // different type than the previous document.
  quantity: "foo",
  // The previous document didn't include
  // the `backorder` field.
  backorder: {
    limit: 5,
    backordered: false
  }
})

    The query creates all documents but only returns the last document.

  4. Press Ctrl+D to exit the shell.

Migrate to a permissive document type

As application development continues, stricter data requirements typically evolve. In this case, the e-commerce application wants to use the name field to fetch specific Product documents.

Edit the Product collection schema to add:

  • A field definition for the name field

  • A migrations block

  • A byName() index definition

  1. Edit collections.fsl as follows:

    collection Product {
  // Field definition for the `name` field.
  // `name` only accepts `String` values.
  name: String
  // Adds the `typeConflicts` field as a catch-all field for
  // existing `name` values that aren't `String` or `null`.
  // Because `typeConflicts` is used in a `move_conflicts` statement,
  // it must have a type of `{ *: Any }?`.
  typeConflicts: { *: Any }?

  // The schema now includes field definitions.
  // The wildcard constraint is required to
  // accept documents with ad hoc fields.
  // You can only remove a wildcard constraint
  // with a migration.
  *: Any

  // Instructs Fauna on how to handle the field and
  // wildcard constraint updates.
  migrations {
    // Migration #1 (Current)
    // Adds the `typeConflicts` field.
    add .typeConflicts
    // Adds the `name` field.
    add .name
    // Nests non-conforming `name` and `typeConflicts` field
    // values in the `typeConflicts` field.
    move_conflicts .typeConflicts
    // Sets `name` to `"Default"` for existing documents
    // with a non-string `name` value.
    backfill .name = "Default"
  }

  // Defines the `byName()` index.
  // Use the index to get `Product` documents by `name` value.
  index byName {
    terms [.name]
  }

  index sortedByQuantity {
    values [.quantity]
  }
}

  2. Save collections.fsl. Then push the schema to Fauna:

    fauna schema push

    When prompted, accept and push the changes. This runs the migration. Changes from the migration are immediately visible in any subsequent queries.

    New Product collection documents must now contain a name field with a String value. The documents can also contain other ad hoc fields.

Review existing data

Run a few queries to review changes the migration made to the collection’s documents.

  1. Start a shell session:

    fauna shell

  2. Run the following query:

    Product.sortedByQuantity()

    The query returns all Product collection documents, sorted by ascending quantity value:

    {
  data: [
    {
      id: "400414078704549921",
      coll: Product,
      ts: Time("2099-06-11T16:31:12.790Z"),
      name: "pinata",
      description: "Original Classic Donkey Pinata",
      price: 24.99,
      quantity: 40
    },
    {
      id: "400414078705598497",
      coll: Product,
      ts: Time("2099-06-11T16:31:12.790Z"),
      name: "cups",
      description: "Translucent 9 Oz, 100 ct",
      price: 6.98,
      quantity: "foo",
      backorder: {
        limit: 5,
        backordered: false
      }
    }
  ]
}

    The migration made no changes to existing documents. The documents already contain a name field with a String value.

Add new data

New Product collection documents must contain a name field with a String value. Attempt to add some non-conforming documents to the collection.

  1. In the shell session, run the following query in editor mode:

    // Create a `Product` collection document
// without a `name` field.
Product.create({
  description: "Conventional Hass, 4ct bag",
  price: 3.99,
  quantity: 95
})

    The query returns an error. Product collection documents must contain a name field.

  2. Run the following query in editor mode:

    // Create a `Product` collection document
// with a non-String `name` value
Product.create({
  name: 12345,
  description: "Conventional Hass, 4ct bag",
  price: 3.99,
  quantity: 95
})

    The query returns an error. The name field must contain a String.

  3. Run the following query in editor mode:

    // Create a `Product` collection document
// with a `name` String
Product.create({
  name: "avocados",
  description: "Conventional Hass, 4ct bag",
  price: 3.99,
  quantity: 95
})

    The query runs successfully and creates a new Product document.

  4. Press Ctrl+D to exit the shell.

Add more field definitions

You can perform iterative migrations to add more field definitions.

Edit the Product collection schema to:

  • Add field definitions for the description and price fields

  • Update the migrations block

  1. Edit collections.fsl as follows:

    collection Product {
  name: String
  // Adds the `description` field.
  // Only accepts `String` values.
  description: String
  // Adds the `price` field.
  // Only accepts `Number` values.
  price: Number
  typeConflicts: { *: Any }?

  *: Any

  migrations {
    // Migration #1 (Previous)
    // Already run. Fauna ignores
    // previously run migration statements.
    add .typeConflicts
    add .name
    move_conflicts .typeConflicts
    backfill .name = "Default"

    // Migration #2 (Current)
    // New migration statements. Fauna
    // only runs these statements.
    add .description
    add .price
    // Uses the existing `typeConflicts` field
    // as a catch-all field.
    move_conflicts .typeConflicts
    backfill .description = "Default"
    backfill .price = 0.00
  }

  index byName {
    terms [.name]
  }

  index sortedByQuantity {
    values [.quantity]
  }
}

  2. Save collections.fsl. Then push the schema to Fauna:

    fauna schema push

    When prompted, accept and push the changes. This runs the migration.

    New Product collection documents must now contain:

    • A name field with a String value

    • A description field with a String value

    • A price field with a Number value

    The documents can also contain other ad hoc fields.

Review existing data

Run a few queries to review changes the migration made to the collection’s documents.

  1. Start a shell session:

    fauna shell

  2. Run the following query:

    Product.sortedByQuantity()

    The query returns all Product collection documents, sorted by ascending quantity value:

    {
  data: [
    {
      id: "400414078704549921",
      coll: Product,
      ts: Time("2099-06-11T16:31:12.790Z"),
      name: "pinata",
      description: "Original Classic Donkey Pinata",
      price: 24.99,
      quantity: 40
    },
    ...
  ]
}

    The migration made no changes to the existing documents.

  3. Press Ctrl+D to exit the shell.

Handle non-conforming values

Ad hoc fields may contain different types across documents in a collection. You can add a field definition to normalize an existing ad hoc field and narrow its accepted types.

A move_conflicts migration statement lets you assign non-conforming field values to a catch-all field.

Edit the Product collection schema to:

  • Add a field definition for the quantity field

  • Update migrations block

  1. Edit collections.fsl as follows:

    collection Product {
  name: String
  description: String
  price: Number
  // Adds the `quantity` field.
  // Only accepts `Int` values.
  quantity: Int
  typeConflicts: { *: Any }?

  *: Any

  migrations {
    // Migration #1 (Previous)
    // Already run. Ignored.
    add .typeConflicts
    add .name
    move_conflicts .typeConflicts
    backfill .name = "Default"

    // Migration #2 (Previous)
    // Already run. Ignored.
    add .description
    add .price
    move_conflicts .typeConflicts
    backfill .description = "Default"
    backfill .price = 0.00

    // Migration #3 (Current)
    // New migration statements.
    add .quantity
    // Uses the existing `typeConflicts` field
    // as a catch-all field.
    move_conflicts .typeConflicts
    backfill .quantity = 0
  }

  index byName {
    terms [.name]
  }

  index sortedByQuantity {
    values [.quantity]
  }
}

  2. Save collections.fsl. Then push the schema to Fauna:

    fauna schema push

    When prompted, accept and push the changes. This runs the migration.

    New Product collection documents must now contain:

    • A name field with a String value

    • A description field with a String value

    • A price field with a Number value

    • A quantity field with an Int value

    The documents can also contain other ad hoc fields.

Review existing data

Run a query to see how the migration affected a document with a non-conforming quantity value.

  1. Start a shell session:

    fauna shell

  2. Run the following query:

    Product.byName("cups").first()

    The query returns:

    {
  id: "400417408871825441",
  coll: Product,
  ts: Time("2099-06-11T17:24:08.690Z"),
  name: "cups",
  description: "Translucent 9 Oz, 100 ct",
  price: 6.98,
  backorder: {
    limit: 5,
    backordered: false
  },
  quantity: 0,
  typeConflicts: {
    quantity: "foo"
  }
}

    During the migration, Fauna nested the document’s non-conforming quantity value in the typeConflicts catch-all field, which was specified by the move_conflicts statement.

    The migration then filled in the specified backfill value of 0 so the document conforms to the field definition.

  3. Press Ctrl+D to exit the shell.

Migrate to a strict document type

As the application scales and its data model stabilizes, storage costs and data consistency become more important than flexibility. It no longer makes sense to accept and store ad hoc fields.

Edit the Product collection schema to remove the wildcard constraint.

  1. Edit collections.fsl as follows:

    collection Product {
  name: String
  description: String
  price: Number
  quantity: Int
  typeConflicts: { *: Any }?

  // Removes the `*: Any` wildcard constraint.
  // The collection no longer accepts
  // documents with ad hoc fields.

  migrations {
    // Migration #1 (Previous)
    // Already run. Ignored.
    add .typeConflicts
    add .name
    move_conflicts .typeConflicts
    backfill .name = "Default"

    // Migration #2 (Previous)
    // Already run. Ignored.
    add .description
    add .price
    move_conflicts .typeConflicts
    backfill .description = "Default"
    backfill .price = 0.00

    // Migration #3 (Previous)
    // Already run. Ignored.
    add .quantity
    move_conflicts .typeConflicts
    backfill .quantity = 0

    // Migration #4 (Current)
    // New migration statements.
    // Moves any existing ad hoc fields to
    // the `typeConflicts` catch-all field.
    move_wildcard .typeConflicts
  }

  index byName {
    terms [.name]
  }

  index sortedByQuantity {
    values [.quantity]
  }
}

  2. Save collections.fsl. Then push the schema to Fauna:

    fauna schema push

    When prompted, accept and push the changes. This runs the migration.

    New Product collection documents must now contain:

    • A name field with a String value

    • A description field with a String value

    • A price field with a Number value

    • A quantity field with an Int value

    The documents can’t contain other fields.

Review existing data

Run a query to see how the migration affected a document that contained an ad hoc field without a field definition.

  1. Start a shell session:

    fauna shell

  2. Run the following query in editor mode:

    // Gets `Product` collection documents
// with a `name` of `cups`. Then get the
// first document.
Product.byName("cups").first()

    The query returns:

    {
  id: "400417408871825441",
  coll: Product,
  ts: Time("2099-06-11T17:24:08.690Z"),
  name: "cups",
  description: "Translucent 9 Oz, 100 ct",
  price: 6.98,
  quantity: 0,
  typeConflicts: {
    quantity: "foo",
    backorder: {
      limit: 5,
      backordered: false
    }
  }
}

    During the migration, Fauna nested the document’s ad hoc backorder field value in the typeConflicts catch-all field, which specified by the move_wildcard statement.

Attempt to add invalid data

Attempt to add non-conforming documents to the Product collection. New documents can’t contain ad hoc fields. Field values must conform to their field definitions.

  1. In the shell session, run the following query in editor mode:

    // Create a `Product` collection document
// with an ad hoc `stocked` field.
Product.create({
  name: "limes",
  description: "Conventional, 1 ct",
  price: 0.35,
  quantity: 100,
  stocked: true
})

    The query returns an error. The Product collection no longer accepts documents with ad hoc fields.

  2. Run the following query in editor mode:

    // Create a `Product` collection document
// with a missing `quantity` field.
Product.create({
  name: "limes",
  description: "Conventional, 1 ct",
  price: 0.35
})

    The query returns an error. The quantity field must be present in new documents.

  3. Run the following query in editor mode:

    // Create a `Product` collection document
// with all required fields and no ad hoc fields.
Product.create({
  name: "limes",
  description: "Conventional, 1 ct",
  price: 0.35,
  quantity: 100
})

    The query runs successfully and creates a new Product document.

  4. Press Ctrl+D to exit the shell.

Is this article helpful? 

Tell Fauna how the article can be improved:
 Visit Fauna's forums or email docs@fauna.com

Thank you for your feedback!