Snapshot export

Fauna’s snapshot export feature lets you create a point-in-time snapshot of a database or specified user-defined collections and store their document data as JSON files in an AWS S3 bucket. You can use exports to:

Load Fauna data into third-party systems.
Store Fauna data externally for compliance or data control.

You can combine exports with event feeds to track database changes in other systems for change data capture (CDC).

We don’t recommend using exports to restore data in Fauna. To back up and restore databases, see Database backup and restore.

Requirements

To use exports, you must:

Have an Enterprise plan.
Be an owner, admin, or developer for your Fauna account. See Team management .
Have an AWS S3 bucket to store exported data. The bucket must grant permissions to Fauna’s IAM role. See S3 bucket permissions.

S3 bucket permissions

To create an export, you must specify an AWS S3 bucket path where the exported data will be stored. The bucket path must contain no existing objects.

The S3 bucket must grant the following permissions to Fauna’s IAM Role ARN (arn:aws:iam::209479277009:role/FaunaExport):

s3:GetBucketLocation
s3:ListBucket
s3:PutObject
s3:PutObjectAcl (if bucket ACLs are enabled)

Example bucket policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Allow Fauna export operations",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::209479277009:role/FaunaExport"
      },
      "Action": [
        "s3:GetBucketLocation",
        "s3:ListBucket",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket",
        "arn:aws:s3:::your-bucket/*"
      ]
    },
    {
      "Sid": "Allow Fauna to set bucket owner full control (if ACLs enabled)",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::209479277009:role/FaunaExport"
      },
      "Action": "s3:PutObjectAcl",
      "Resource": "arn:aws:s3:::your-bucket/*",
      "Condition": {
        "StringEquals": {
          "s3:x-amz-acl": "bucket-owner-full-control"
        }
      }
    }
  ]
}

All file uploads to the bucket use S3 standard encryption.

Create and manage exports

You create and manage exports using any of the following:

The Fauna CLI
The Fauna Account HTTP API

Fauna CLI

The following examples create and manage exports using the Fauna CLI’s fauna export commands.

Create an export

To start an export, use the fauna export create s3 command. In the command, specify:

The --database, including its region group identifier and hierarchy. Shorthand Region Group identifiers are supported. Child databases are not included in the export.
One or more user-defined collections to export, passed using the --collection flag. Pass multiple collections as a space-separated list, such as --collection Product Category. To export all user-defined collections, omit the --collection flag. You can’t export system collections.
An optional data --format used to convert exported document data to JSON. Defaults to the simple format.

The S3 --bucket and --path used to store export files.

Double-check your S3 destination before submitting a request. Once requested, you can’t cancel an export. Fauna can’t delete exported data in an S3 bucket.

We recommend using a unique S3 bucket path for each export. To avoid overwriting existing files, Fauna requires the path to contain no existing objects. If the path contains any existing objects, the export fails.

Exports from databases in Fauna’s Europe (eu-std) region group must be stored in an S3 bucket located in an EU AWS region.

# Export the the 'Product' and 'Category' collections in
# the 'us/parent_db/child_db' database. Store the export
# in the 'fauna_exports/parent_db/child_db/2099-12-31'
# path of the 'doc-example-bucket' S3 bucket. Format FQL data
# using the 'simple' data format.
fauna export create s3 \
  --database us/parent_db/child_db \
  --collection Product Category \
  --bucket doc-example-bucket \ # Replace with your bucket.
  --path fauna_exports/parent_db/child_db/2099-12-31 \
  --format simple

The command starts an asynchronous export operation that runs in the background. Export requests made using the CLI aren’t idempotent. Fauna processes one export per account at a time.

The command output includes a unique ID for the export:

123456789

Get exports

You can pass the export ID to the fauna export get command:

# Get an export with an ID of '123456789'.
fauna export get 123456789

The response includes the current state of the export:

id: "123456789"
state: Complete
database: us-std/parent_db/child_db
format: simple
destination:
  s3:
    bucket: doc-example-bucket
    path: fauna_exports/parent_db/child_db/2099-12-31
created_at: 2099-12-31T19:07:25.642703Z
updated_at: 2099-12-31T19:07:25.642703Z
collections:
  - Product
  - Category
destination_uri: s3://doc-example-bucket/fauna_exports/parent_db/child_db/2099-12-31

Possible state values:

State Description

State	Description
`Pending`	The export request has been received but is not yet in progress.
`InProgress`	The export is in progress. This includes copying collection files to the S3 bucket.
`Complete`	The export is complete. Collection files are available in the S3 bucket. This is a terminal state. The export’s state will no longer change.
`Failed`	There was an error processing the export. You can safely retry the export. This is a terminal state. The export’s state will no longer change.

Pending

The export request has been received but is not yet in progress.

InProgress

The export is in progress. This includes copying collection files to the S3 bucket.

Complete

The export is complete. Collection files are available in the S3 bucket. This is a terminal state. The export’s state will no longer change.

Failed

There was an error processing the export. You can safely retry the export. This is a terminal state. The export’s state will no longer change.

Alternatively, you can use the fauna export list command to get a list of all exports and their states for an account:

fauna export list

You can optionally filter the export list by --state. You can pass multiple --state values as a space-separated list.

# List exports in the 'Pending' or
# 'Complete' state.
fauna export list \
  --state Pending Complete

Account API

The following examples create and manage exports using the Fauna Account HTTP API’s Export endpoints.

Create an account key

To authenticate the Fauna Account API requests, you must have an account key. You can create an account key in the Fauna Dashboard.

Log in to the Fauna Dashboard and click Account in the left navigation.
Click Account Keys.
Click Create Key.
Enter a Name and an optional TTL. The TTL is the number of days until the account key expires.
Click Create.
Copy the Key Secret.

You can pass the secret as a Bearer token in Fauna Account API requests.

Create an export

To start an export, make a request to the Create Export endpoint. In the request, specify:

The database, including its region group identifier and hierarchy. Shorthand region group identifiers are not supported. Child databases are not included in the export.
One or more user-defined collections to export. To export all user-defined collections, omit the collections field. You can’t export system collections.
The data format used to convert exported FQL document data to JSON.

The destination URI for the S3 bucket and path used to store export files.

Double-check your S3 destination before submitting a request. Once requested, you can’t cancel an export. Fauna can’t delete exported data in an S3 bucket.

Exports from databases in Fauna’s Europe (eu-std) region group must be stored in an S3 bucket located in an EU AWS region.

# Export the 'Product' and 'Category' collections in
# the 'us-std/parent_db/child_db' database. Store the export in the
# 's3://doc-example-bucket/fauna_exports/parent_db/child_db/2099-12-3'
# S3 destination URI. Format FQL data using the 'simple' data format.
curl -X POST \
  'https://account.fauna.com/v2/exports' \
  -H "Authorization: Bearer $ACCOUNT_KEY_SECRET" \
  -H 'Content-Type: application/json' \
  -d '{
    "database": "us-std/parent_db/child_db",
    "collections": [
      "Product",
      "Category"
    ],
    "format": "simple",
    # Replace with your S3 destination URI.
    "destination": "s3://doc-example-bucket/fauna_exports/parent_db/child_db/2099-12-31/"
  }'

The request starts an asynchronous export operation that runs in the background.

The response includes a unique ID for the export and the export’s current state. If is_terminal is true, the state is final and will no longer change.

{
  "response": {
    "id": 123456789,
    "state": "Pending",
    "database": "us-std/parent_db/child_db",
    "format": "simple",
    "destination": {
      "s3": {
        "bucket": "doc-example-bucket",
        "path": "fauna_exports/my_db/2099-12-31"
      },
      "uri": "s3://doc-example-bucket/fauna_exports/my_db/2099-12-31/"
    },
    "created_at": "2099-12-31T21:46:12.580Z",
    "updated_at": "2099-12-31T21:46:12.580Z",
    "is_terminal": false,
    "collections": [
      "Product",
      "Category"
    ]
  }
}

Possible state values:

State Description

State	Description
`Pending`	The export request has been received but is not yet in progress.
`InProgress`	The export is in progress. This includes copying collection files to the S3 bucket.
`Complete`	The export is complete. Collection files are available in the S3 bucket. This is a terminal state. The export’s state will no longer change.
`Failed`	There was an error processing the export. You can safely retry the export. This is a terminal state. The export’s state will no longer change.

Pending

The export request has been received but is not yet in progress.

InProgress

The export is in progress. This includes copying collection files to the S3 bucket.

Complete

The export is complete. Collection files are available in the S3 bucket. This is a terminal state. The export’s state will no longer change.

Failed

There was an error processing the export. You can safely retry the export. This is a terminal state. The export’s state will no longer change.

Idempotent export requests

By default, export requests are not idempotent. To ensure idempotency, include a unique string in the Idempotency-Key HTTP request header.

If you retry a request with the same idempotency key within 24 hours, the API will return the result of the original request instead of processing the new request.

Idempotency keys are scoped to the request’s account API key. Using the same idempotency key with different API keys will not result in idempotent requests.

# Use the `Idempotency-Key` HTTP request header to
# prevent duplicate exports. If the same key is uses
# within 24 hours, Fauna returns the response of
# the original request.
curl -X POST \
  'https://account.fauna.com/v2/exports' \
  -H "Authorization: Bearer $ACCOUNT_KEY_SECRET" \
  -H "Idempotency-Key: f47ac10b-58cc-4372-a567-0e02b2c3d479" \
  -H 'Content-Type: application/json' \
  -d '{
    "database": "us-std/parent_db/child_db",
    "collections": [
      "Product",
      "Category"
    ],
    "format": "simple",
    # Replace with your S3 destination URI.
    "destination": "s3://doc-example-bucket/fauna_exports/parent_db/child_db/2099-12-31/"
  }'

The Idempotent-Replayed response header indicates whether the request was a replay. If true, the response is from a previous request with the same idempotency key. The new request was not processed. If false, the new request was processed.

Get exports

You can pass the export ID to the Get Export endpoint to get the current state of an export:

curl -X GET \
  'https://account.fauna.com/v2/exports/<EXPORT_ID>' \
  -H "Authorization: Bearer $ACCOUNT_KEY_SECRET"

Alternatively, you can use the List Exports endpoint to get a list of all exports and their states for an account:

curl -X GET \
  'https://account.fauna.com/v2/exports' \
  -H "Authorization: Bearer $ACCOUNT_KEY_SECRET"

You can optionally filter the export list by state and database. You can pass the state query parameter multiple times to filter on multiple states.

# Gets exports in the 'Pending' or 'Complete' state
# for the 'us-std/parent_db/child_db' database.
curl -X GET \
  --get \
  'https://account.fauna.com/v2/exports' \
  -H "Authorization: Bearer $ACCOUNT_KEY_SECRET" \
  --data-urlencode 'state=Pending' \
  --data-urlencode 'state=Complete' \
  --data-urlencode 'database=us-std/parent_db/child_db'

Export files

The export operation writes the following files to the specified S3 bucket path:

Collection files
Manifest file
Permissions validated file

Collection files

Fauna exports document data for each collection as one or more JSON files. Exported collection files use the following naming convention:

collections/<COLLECTION_NAME>/<COLLECTION_NAME>_<HOST_ID>_<FILE_IDX>.json

Where:

<COLLECTION_NAME> is the name of the collection.
<HOST_ID> is an identifier, ranging from 00 to 99 for the Fauna host that uploaded the file. An export may have multiple hosts.
<FILE_IDX> is an index for the file, ranging from 000000 to 999999. Fauna writes collection files sequentially in ascending index order, starting with 000000 for each host.

Collection file names can vary between exports, even exports of the same database with the same data.

Each collection export file contains an array of JSON objects, where each object represents a collection document. Documents are ordered by ascending document ID. Document data is converted from FQL to JSON based on the data format specified in the request. Example using the tagged format:

[
  {
    "id": "419646277065637921",
    "coll": {
      "@mod": "Category"
    },
    "ts": {
      "@time": "2099-01-09T23:18:46.510Z"
    },
    "name": "party",
    "description": "Party Supplies"
  },
  ...
]

Manifest file

When the export is complete, Fauna uploads a manifest.json file to the S3 bucket path. The manifest.json file contains metadata about the export and describes the structure and details of the exported data.

{
  "export_id": "123456789",
  "snapshot_ts": "2099-01-10T17:35:38.097961Z",
  "datafile_compression": false,
  "datafile_format": "json",
  "document_format": "tagged",
  "object_count": 10,
  "object_keys": [
    "fauna_exports/parent_db/child_db/2099-12-31/collections/Category/Category_00_000000.json",
    "fauna_exports/parent_db/child_db/2099-12-31/collections/Product/Product_00_000000.json",
    ...
  ]
}

Field Type Description

Field	Type	Description
`export_id`	String	Unique ID for the export. The ID is a string-encoded integer.
`snapshot_ts`	String	The timestamp of the export’s snapshot in ISO 8601 format. Exports capture a consistent state of the database’s document data at the `snapshot_ts`.
`datafile_compression`	Boolean	Indicates whether the export’s collection files are compressed.
`datafile_format`	String	Format for the export’s collection files. Must be `json`.
`document_format`	String	Data format used to convert the export’s FQL document data to JSON. Valid values are `tagged` and `simple`.
`object_count`	Integer	Total number of the export’s collection files.
`object_keys`	Array of strings	Paths to the export’s collection files.

export_id

String

Unique ID for the export. The ID is a string-encoded integer.

snapshot_ts

String

The timestamp of the export’s snapshot in ISO 8601 format. Exports capture a consistent state of the database’s document data at the snapshot_ts.

datafile_compression

Boolean

Indicates whether the export’s collection files are compressed.

datafile_format

String

Format for the export’s collection files. Must be json.

document_format

String

Data format used to convert the export’s FQL document data to JSON. Valid values are tagged and simple.

object_count

Integer

Total number of the export’s collection files.

object_keys

Array of strings

Paths to the export’s collection files.

Permissions validated file

At the start of an export, Fauna uploads a .perms_validated.txt file to verify that Fauna has appropriate permissions access for the provided S3 bucket path. Fauna doesn’t use the file after its creation. You can safely delete it.

Billing

Fauna charges Transactional Compute Operations (TCOs) for each GB of delivered export data. See Snapshot export delivery in the Fauna Billing docs.

Limitations

Exports do not include:
- Computed field values
- Document history
- Indexed historical data
- Database schema or .fsl schema files
- System collections
- Changes made to the database after the export’s snapshot_ts
Once requested, you can’t cancel an export.