Progressively enforce a document type

Learn: Schema

Fauna supports progressive enforcement for document types. Early in an application’s development, you can use collections with a schemaless or permissive document type to allow ad hoc fields in documents.

As your application evolves, you can use zero-downtime migrations to add stricter field definitions and to normalize field values. This lets you iteratively move from a permissive document type to a strict document type (or the reverse) based on your application’s needs.

In this tutorial, you’ll progressively define and enforce a document type for an example e-commerce application.

Before you start

To complete this tutorial, you’ll need:

Setup

Set up a database for the application.

  1. Log in to Fauna using the CLI:

    fauna cloud-login

    When prompted, enter:

    • Endpoint name: cloud (Press Enter)

      An endpoint defines the settings the CLI uses to run API requests against a Fauna account or database. See Endpoints.

    • Email address: The email address for your Fauna account.

    • Password: The password for your Fauna account.

    • Which endpoint would you like to set as default? The cloud-* endpoint for your preferred Region Group. For example, to use the US Region Group, use cloud-us.

    fauna cloud-login requires an email and password login. If you log in to Fauna using GitHub or Netlify, you can enable email and password login using the Forgot Password workflow.

  2. Create an ecommerce directory and navigate to it:

    mkdir ecommerce
    cd ecommerce
  3. Create an ecommerce database:

    fauna create-database ecommerce
  4. Initialize a project directory:

    fauna project init

    When prompted, use:

    • schema as the schema directory. The command creates the directory.

    • dev as the environment name.

    • The default endpoint.

    • ecommerce as the database.

Create a schemaless collection

Create a schemaless collection. A schemaless collection has no predefined document fields. Instead, you can include ad hoc fields of any type in the collection’s documents.

  1. Navigate to the schema directory:

    cd schema
  2. Create a collections.fsl file and add the following collection schema to it:

    // Stores product data for the e-commerce application.
    collection Product {
      // Contains no field definitions.
    
      // The wildcard constraint accepts
      // ad hoc fields of any type.
      *: Any
      // If a collection schema has no field definitions
      // and no wildcard constraint, it has an implicit
      // wildcard constraint of `*: Any`.
    
      // Defines the `sortedByStock()` index.
      // Use the index to get `Product` documents sorted by
      // ascending `stock` field value.
      index sortedByStock {
        values [.stock]
      }
    }
  3. Save collections.fsl. Then push the schema to Fauna:

    fauna schema push

    When prompted, accept and stage the schema.

  4. Check the status of the staged schema:

    fauna schema status
  5. When the status is ready, commit the staged schema to the database:

    fauna schema commit

    The commit applies the staged schema to the database. This creates the collection.

Add sample data

Add documents to the Product collection.

  1. Start a shell session in the Fauna CLI:

    fauna shell

    The shell runs with the default admin key you created when you logged in using fauna cloud-login.

  2. Enter editor mode to run multi-line queries:

    > .editor
  3. Run the following FQL query:

    // Creates a `Product` collection document.
    // In a schemaless collection,
    // documents can contain any field of any type.
    Product.create({
      name: "pinata",
      description: "Original Classic Donkey Pinata",
      price: 24_99,
      stock: 40
    })
    
    // In a schemaless collection, a field
    // can contain different types across documents.
    Product.create({
      name: "cups",
      description: "Translucent 9 Oz, 100 ct",
      price: 6_98,
      // This document's `stock` field uses a
      // different type than the previous document.
      stock: "foo",
      // The previous document didn't include
      // the `backorder` field.
      backorder: {
        limit: 5,
        backordered: false
      }
    })

    The query creates all documents but only returns the last document.

  4. Press Ctrl+D to exit the shell.

Migrate to a permissive document type

As application development continues, stricter data requirements typically evolve. In this case, the e-commerce application wants to use the name field to fetch specific Product documents.

Edit the Product collection schema to add:

  • A field definition for the name field

  • A migrations block

  • A byName() index definition

  1. Edit collections.fsl as follows:

    collection Product {
      // Field definition for the `name` field.
      // `name` only accepts `String` values.
      name: String
      // Adds the `typeConflicts` field as a catch-all field for
      // existing `name` values that aren't `String` or `null`.
      // Because `typeConflicts` is used in a `move_conflicts` statement,
      // it must have a type of `{ *: Any }?`.
      typeConflicts: { *: Any }?
    
      // The schema now includes field definitions.
      // The wildcard constraint is required to
      // accept documents with ad hoc fields.
      // You can only remove a wildcard constraint
      // with a migration.
      *: Any
    
      // Instructs Fauna on how to handle the field and
      // wildcard constraint updates.
      migrations {
        // Migration #1 (Current)
        // Adds the `typeConflicts` field.
        add .typeConflicts
        // Adds the `name` field.
        add .name
        // Nests non-conforming `name` and `typeConflicts` field
        // values in the `typeConflicts` field.
        move_conflicts .typeConflicts
        // Sets `name` to `"Default"` for existing documents
        // with a non-string `name` value.
        backfill .name = "Default"
      }
    
      // Defines the `byName()` index.
      // Use the index to get `Product` documents by `name` value.
      index byName {
        terms [.name]
      }
    
      index sortedByStock {
        values [.stock]
      }
    }
  2. Save collections.fsl. Then push the schema to Fauna:

    fauna schema push

    When prompted, accept and stage the schema.

  3. Check the status of the staged schema:

    fauna schema status
  4. When the status is ready, commit the staged schema to the database:

    fauna schema commit

    This runs the migration.

    New Product collection documents must now contain a name field with a String value. The documents can also contain other ad hoc fields.

Review existing data

Run a few queries to review changes the migration made to the collection’s documents.

  1. Start a shell session:

    fauna shell
  2. Run the following query:

    Product.sortedByStock()

    The query returns all Product collection documents, sorted by ascending stock value:

    {
      data: [
        {
          id: "400414078704549921",
          coll: Product,
          ts: Time("2099-06-11T16:31:12.790Z"),
          name: "pinata",
          description: "Original Classic Donkey Pinata",
          price: 2499,
          stock: 40
        },
        {
          id: "400414078705598497",
          coll: Product,
          ts: Time("2099-06-11T16:31:12.790Z"),
          name: "cups",
          description: "Translucent 9 Oz, 100 ct",
          price: 698,
          stock: "foo",
          backorder: {
            limit: 5,
            backordered: false
          }
        }
      ]
    }

    The migration made no changes to existing documents. The documents already contain a name field with a String value.

Add new data

New Product collection documents must contain a name field with a String value. Attempt to add some non-conforming documents to the collection.

  1. In the shell session, run the following query in editor mode:

    // Create a `Product` collection document
    // without a `name` field.
    Product.create({
      description: "Conventional Hass, 4ct bag",
      price: 3_99,
      stock: 95
    })

    The query returns an error. Product collection documents must contain a name field.

  2. Run the following query in editor mode:

    // Create a `Product` collection document
    // with a non-String `name` value
    Product.create({
      name: 12345,
      description: "Conventional Hass, 4ct bag",
      price: 3_99,
      stock: 95
    })

    The query returns an error. The name field must contain a String.

  3. Run the following query in editor mode:

    // Create a `Product` collection document
    // with a `name` String
    Product.create({
      name: "avocados",
      description: "Conventional Hass, 4ct bag",
      price: 3_99,
      stock: 95
    })

    The query runs successfully and creates a new Product document.

  4. Press Ctrl+D to exit the shell.

Add more field definitions

You can perform iterative migrations to add more field definitions.

Edit the Product collection schema to:

  • Add field definitions for the description and price fields

  • Update the migrations block

  1. Edit collections.fsl as follows:

    collection Product {
      name: String
      // Adds the `description` field.
      // Only accepts `String` values.
      description: String
      // Adds the `price` field.
      // Only accepts `Int` values.
      price: Int
      typeConflicts: { *: Any }?
    
      *: Any
    
      migrations {
        // Migration #1 (Previous)
        // Already run. Fauna ignores
        // previously run migration statements.
        add .typeConflicts
        add .name
        move_conflicts .typeConflicts
        backfill .name = "Default"
    
        // Migration #2 (Current)
        // New migration statements. Fauna
        // only runs these statements.
        add .description
        add .price
        // Uses the existing `typeConflicts` field
        // as a catch-all field.
        move_conflicts .typeConflicts
        backfill .description = "Default"
        backfill .price = 1
      }
    
      index byName {
        terms [.name]
      }
    
      index sortedByStock {
        values [.stock]
      }
    }
  2. Save collections.fsl. Then push the schema to Fauna:

    fauna schema push

    When prompted, accept and stage the schema.

  3. Check the status of the staged schema:

    fauna schema status
  4. When the status is ready, commit the staged schema to the database:

    fauna schema commit

    This runs the migration.

    New Product collection documents must now contain:

    • A name field with a String value

    • A description field with a String value

    • A price field with an Int value

    The documents can also contain other ad hoc fields.

Review existing data

Run a few queries to review changes the migration made to the collection’s documents.

  1. Start a shell session:

    fauna shell
  2. Run the following query:

    Product.sortedByStock()

    The query returns all Product collection documents, sorted by ascending stock value:

    {
      data: [
        {
          id: "400414078704549921",
          coll: Product,
          ts: Time("2099-06-11T16:31:12.790Z"),
          name: "pinata",
          description: "Original Classic Donkey Pinata",
          price: 2499,
          stock: 40
        },
        ...
      ]
    }

    The migration made no changes to the existing documents.

  3. Press Ctrl+D to exit the shell.

Handle non-conforming values

Ad hoc fields may contain different types across documents in a collection. You can add a field definition to normalize an existing ad hoc field and narrow its accepted types.

A move_conflicts migration statement lets you assign non-conforming field values to a catch-all field.

Edit the Product collection schema to:

  • Add a field definition for the stock field

  • Update migrations block

  1. Edit collections.fsl as follows:

    collection Product {
      name: String
      description: String
      price: Int
      // Adds the `stock` field.
      // Only accepts `Int` values.
      stock: Int
      typeConflicts: { *: Any }?
    
      *: Any
    
      migrations {
        // Migration #1 (Previous)
        // Already run. Ignored.
        add .typeConflicts
        add .name
        move_conflicts .typeConflicts
        backfill .name = "Default"
    
        // Migration #2 (Previous)
        // Already run. Ignored.
        add .description
        add .price
        move_conflicts .typeConflicts
        backfill .description = "Default"
        backfill .price = 1
    
        // Migration #3 (Current)
        // New migration statements.
        add .stock
        // Uses the existing `typeConflicts` field
        // as a catch-all field.
        move_conflicts .typeConflicts
        backfill .stock = 0
      }
    
      index byName {
        terms [.name]
      }
    
      index sortedByStock {
        values [.stock]
      }
    }
  2. Save collections.fsl. Then push the schema to Fauna:

    fauna schema push

    When prompted, accept and stage the schema.

  3. Check the status of the staged schema:

    fauna schema status
  4. When the status is ready, commit the staged schema to the database:

    fauna schema commit

    This runs the migration.

    New Product collection documents must now contain:

    • A name field with a String value

    • A description field with a String value

    • A price field with an Int value

    • A stock field with an Int value

    The documents can also contain other ad hoc fields.

Review existing data

Run a query to see how the migration affected a document with a non-conforming stock value.

  1. Start a shell session:

    fauna shell
  2. Run the following query:

    Product.byName("cups").first()

    The query returns:

    {
      id: "400414078705598497",
      coll: Product,
      ts: Time("2099-06-11T17:24:08.690Z"),
      name: "cups",
      description: "Translucent 9 Oz, 100 ct",
      price: 698,
      stock: 0,
      backorder: {
        limit: 5,
        backordered: false
      },
      typeConflicts: {
        stock: "foo"
      }
    }

    During the migration, Fauna nested the document’s non-conforming stock value in the typeConflicts catch-all field, which was specified by the move_conflicts statement.

    The migration then filled in the specified backfill value of 0 so the document conforms to the field definition.

  3. Press Ctrl+D to exit the shell.

Migrate to a strict document type

As the application scales and its data model stabilizes, storage costs and data consistency become more important than flexibility. It no longer makes sense to accept and store ad hoc fields.

Edit the Product collection schema to remove the wildcard constraint.

  1. Edit collections.fsl as follows:

    collection Product {
      name: String
      description: String
      price: Int
      stock: Int
      typeConflicts: { *: Any }?
    
      // Removes the `*: Any` wildcard constraint.
      // The collection no longer accepts
      // documents with ad hoc fields.
    
      migrations {
        // Migration #1 (Previous)
        // Already run. Ignored.
        add .typeConflicts
        add .name
        move_conflicts .typeConflicts
        backfill .name = "Default"
    
        // Migration #2 (Previous)
        // Already run. Ignored.
        add .description
        add .price
        move_conflicts .typeConflicts
        backfill .description = "Default"
        backfill .price = 1
    
        // Migration #3 (Previous)
        // Already run. Ignored.
        add .stock
        move_conflicts .typeConflicts
        backfill .stock = 0
    
        // Migration #4 (Current)
        // New migration statements.
        // Moves any existing ad hoc fields to
        // the `typeConflicts` catch-all field.
        move_wildcard .typeConflicts
      }
    
      index byName {
        terms [.name]
      }
    
      index sortedByStock {
        values [.stock]
      }
    }
  2. Save collections.fsl. Then push the schema to Fauna:

    fauna schema push

    When prompted, accept and stage the schema.

  3. Check the status of the staged schema:

    fauna schema status
  4. When the status is ready, commit the staged schema to the database:

    fauna schema commit

    This runs the migration.

    New Product collection documents must now contain:

    • A name field with a String value

    • A description field with a String value

    • A price field with an Int value

    • A stock field with an Int value

    The documents can’t contain other fields.

Review existing data

Run a query to see how the migration affected a document that contained an ad hoc field without a field definition.

  1. Start a shell session:

    fauna shell
  2. Run the following query in editor mode:

    // Gets `Product` collection documents
    // with a `name` of `cups`. Then get the
    // first document.
    Product.byName("cups").first()

    The query returns:

    {
      id: "400414078705598497",
      coll: Product,
      ts: Time("2099-06-11T17:24:08.690Z"),
      name: "cups",
      description: "Translucent 9 Oz, 100 ct",
      price: 698,
      stock: 0,
      typeConflicts: {
        stock: "foo",
        backorder: {
          limit: 5,
          backordered: false
        }
      }
    }

    During the migration, Fauna nested the document’s ad hoc backorder field value in the typeConflicts catch-all field, which specified by the move_wildcard statement.

Attempt to add invalid data

Attempt to add non-conforming documents to the Product collection. New documents can’t contain ad hoc fields. Field values must conform to their field definitions.

  1. In the shell session, run the following query in editor mode:

    // Create a `Product` collection document
    // with an ad hoc `stocked` field.
    Product.create({
      name: "limes",
      description: "Conventional, 1 ct",
      price: 35,
      stock: 100,
      stocked: true
    })

    The query returns an error. The Product collection no longer accepts documents with ad hoc fields.

  2. Run the following query in editor mode:

    // Create a `Product` collection document
    // with a missing `stock` field.
    Product.create({
      name: "limes",
      description: "Conventional, 1 ct",
      price: 35
    })

    The query returns an error. The stock field must be present in new documents.

  3. Run the following query in editor mode:

    // Create a `Product` collection document
    // with all required fields and no ad hoc fields.
    Product.create({
      name: "limes",
      description: "Conventional, 1 ct",
      price: 35,
      stock: 100
    })

    The query runs successfully and creates a new Product document.

  4. Press Ctrl+D to exit the shell.

Is this article helpful? 

Tell Fauna how the article can be improved:
Visit Fauna's forums or email docs@fauna.com

Thank you for your feedback!