Snapshot export
Fauna’s snapshot export feature lets you create a point-in-time snapshot of a database or specified user-defined collections and store their document data as JSON files in an AWS S3 bucket. You can use exports to:
-
Load Fauna data into third-party systems.
-
Store Fauna data externally for compliance or data control.
You can combine exports with event feeds to track database changes in other systems for change data capture (CDC).
We don’t recommend using exports to restore data in Fauna. To back up and restore databases, see Database backup and restore. |
Requirements
To use exports, you must:
-
Have an Enterprise plan.
-
Be an owner, admin, or developer for your Fauna account. See Team management .
-
Have an AWS S3 bucket to store exported data. The bucket must grant permissions to Fauna’s IAM role. See S3 bucket permissions.
S3 bucket permissions
To create an export, you must specify an AWS S3 bucket path where the exported data will be stored. The bucket path must contain no existing objects.
The S3 bucket must grant the following permissions to Fauna’s IAM Role ARN
(arn:aws:iam::209479277009:role/FaunaExport
):
-
s3:GetBucketLocation
-
s3:ListBucket
-
s3:PutObject
-
s3:PutObjectAcl
(if bucket ACLs are enabled)
Example bucket policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Allow Fauna export operations",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::209479277009:role/FaunaExport"
},
"Action": [
"s3:GetBucketLocation",
"s3:ListBucket",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::your-bucket",
"arn:aws:s3:::your-bucket/*"
]
},
{
"Sid": "Allow Fauna to set bucket owner full control (if ACLs enabled)",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::209479277009:role/FaunaExport"
},
"Action": "s3:PutObjectAcl",
"Resource": "arn:aws:s3:::your-bucket/*",
"Condition": {
"StringEquals": {
"s3:x-amz-acl": "bucket-owner-full-control"
}
}
}
]
}
All file uploads to the bucket use S3 standard encryption.
Create and manage exports
You create and manage exports using any of the following:
-
The Fauna CLI
Fauna CLI
The following examples create and manage exports using the Fauna CLI’s
fauna export
commands.
Create an export
To start an export, use the fauna export create s3
command. In the command, specify:
-
The
--database
, including its region group identifier and hierarchy. Shorthand Region Group identifiers are supported. Child databases are not included in the export. -
One or more user-defined collections to export, passed using the
--collection
flag. Pass multiple collections as a space-separated list, such as--collection Product Category
. To export all user-defined collections, omit the--collection
flag. You can’t export system collections. -
An optional data
--format
used to convert exported document data to JSON. Defaults to thesimple
format. -
The S3
--bucket
and--path
used to store export files.Double-check your S3 destination before submitting a request. Once requested, you can’t cancel an export. Fauna can’t delete exported data in an S3 bucket.
We recommend using a unique S3 bucket path for each export. To avoid overwriting existing files, Fauna requires the path to contain no existing objects. If the path contains any existing objects, the export fails.
Exports from databases in Fauna’s Europe (
eu-std
) region group must be stored in an S3 bucket located in an EU AWS region.
# Export the the 'Product' and 'Category' collections in
# the 'us/parent_db/child_db' database. Store the export
# in the 'fauna_exports/parent_db/child_db/2099-12-31'
# path of the 'doc-example-bucket' S3 bucket. Format FQL data
# using the 'simple' data format.
fauna export create s3 \
--database us/parent_db/child_db \
--collection Product Category \
--bucket doc-example-bucket \ # Replace with your bucket.
--path fauna_exports/parent_db/child_db/2099-12-31 \
--format simple
The command starts an asynchronous export operation that runs in the background. Export requests made using the CLI aren’t idempotent. Fauna processes one export per account at a time.
The command output includes a unique ID for the export:
123456789
Get exports
You can pass the export ID to the fauna export get
command:
# Get an export with an ID of '123456789'.
fauna export get 123456789
The response includes the current state of the export:
id: "123456789"
state: Complete
database: us-std/parent_db/child_db
format: simple
destination:
s3:
bucket: doc-example-bucket
path: fauna_exports/parent_db/child_db/2099-12-31
created_at: 2099-12-31T19:07:25.642703Z
updated_at: 2099-12-31T19:07:25.642703Z
collections:
- Product
- Category
destination_uri: s3://doc-example-bucket/fauna_exports/parent_db/child_db/2099-12-31
Possible state
values:
State | Description |
---|---|
|
The export request has been received but is not yet in progress. |
|
The export is in progress. This includes copying collection files to the S3 bucket. |
|
The export is complete. Collection files are available in the S3 bucket. This is a terminal state. The export’s state will no longer change. |
|
There was an error processing the export. You can safely retry the export. This is a terminal state. The export’s state will no longer change. |
Alternatively, you can use the fauna export list
command to get a list of all exports and their states for an account:
fauna export list
You can optionally filter the export list by --state
. You can pass multiple
--state
values as a space-separated list.
# List exports in the 'Pending' or
# 'Complete' state.
fauna export list \
--state Pending Complete
Account API
The following examples create and manage exports using the Fauna Account HTTP API’s Export endpoints.
Create an account key
To authenticate the Fauna Account API requests, you must have an account key. You can create an account key in the Fauna Dashboard.
-
Log in to the Fauna Dashboard and click Account in the left navigation.
-
Click Account Keys.
-
Click Create Key.
-
Enter a Name and an optional TTL. The TTL is the number of days until the account key expires.
-
Click Create.
-
Copy the Key Secret.
You can pass the secret as a Bearer token in Fauna Account API requests.
Create an export
To start an export, make a request to the Create Export endpoint. In the request, specify:
-
The
database
, including its region group identifier and hierarchy. Shorthand region group identifiers are not supported. Child databases are not included in the export. -
One or more user-defined
collections
to export. To export all user-defined collections, omit thecollections
field. You can’t export system collections. -
The data
format
used to convert exported FQL document data to JSON. -
The
destination
URI for the S3 bucket and path used to store export files.Double-check your S3 destination before submitting a request. Once requested, you can’t cancel an export. Fauna can’t delete exported data in an S3 bucket.
We recommend using a unique S3 bucket path for each export. To avoid overwriting existing files, Fauna requires the path to contain no existing objects. If the path contains any existing objects, the export fails.
Exports from databases in Fauna’s Europe (
eu-std
) region group must be stored in an S3 bucket located in an EU AWS region.
# Export the 'Product' and 'Category' collections in
# the 'us-std/parent_db/child_db' database. Store the export in the
# 's3://doc-example-bucket/fauna_exports/parent_db/child_db/2099-12-3'
# S3 destination URI. Format FQL data using the 'simple' data format.
curl -X POST \
'https://account.fauna.com/v2/exports' \
-H "Authorization: Bearer $ACCOUNT_KEY_SECRET" \
-H 'Content-Type: application/json' \
-d '{
"database": "us-std/parent_db/child_db",
"collections": [
"Product",
"Category"
],
"format": "simple",
# Replace with your S3 destination URI.
"destination": "s3://doc-example-bucket/fauna_exports/parent_db/child_db/2099-12-31/"
}'
The request starts an asynchronous export operation that runs in the background.
The response includes a unique ID for the export and the export’s current
state
. If is_terminal
is true
, the state
is final and will no longer
change.
{
"response": {
"id": 123456789,
"state": "Pending",
"database": "us-std/parent_db/child_db",
"format": "simple",
"destination": {
"s3": {
"bucket": "doc-example-bucket",
"path": "fauna_exports/my_db/2099-12-31"
},
"uri": "s3://doc-example-bucket/fauna_exports/my_db/2099-12-31/"
},
"created_at": "2099-12-31T21:46:12.580Z",
"updated_at": "2099-12-31T21:46:12.580Z",
"is_terminal": false,
"collections": [
"Product",
"Category"
]
}
}
Possible state
values:
State | Description |
---|---|
|
The export request has been received but is not yet in progress. |
|
The export is in progress. This includes copying collection files to the S3 bucket. |
|
The export is complete. Collection files are available in the S3 bucket. This is a terminal state. The export’s state will no longer change. |
|
There was an error processing the export. You can safely retry the export. This is a terminal state. The export’s state will no longer change. |
Idempotent export requests
By default, export requests are not idempotent. To ensure idempotency, include
a unique string in the Idempotency-Key
HTTP request header.
If you retry a request with the same idempotency key within 24 hours, the API will return the result of the original request instead of processing the new request.
Idempotency keys are scoped to the request’s account API key. Using the same idempotency key with different API keys will not result in idempotent requests. |
# Use the `Idempotency-Key` HTTP request header to
# prevent duplicate exports. If the same key is uses
# within 24 hours, Fauna returns the response of
# the original request.
curl -X POST \
'https://account.fauna.com/v2/exports' \
-H "Authorization: Bearer $ACCOUNT_KEY_SECRET" \
-H "Idempotency-Key: f47ac10b-58cc-4372-a567-0e02b2c3d479" \
-H 'Content-Type: application/json' \
-d '{
"database": "us-std/parent_db/child_db",
"collections": [
"Product",
"Category"
],
"format": "simple",
# Replace with your S3 destination URI.
"destination": "s3://doc-example-bucket/fauna_exports/parent_db/child_db/2099-12-31/"
}'
The Idempotent-Replayed
response header indicates whether the request was a
replay. If true
, the response is from a previous request with the same
idempotency key. The new request was not processed. If false
, the new request
was processed.
Get exports
You can pass the export ID to the Get Export endpoint to get the current state of an export:
curl -X GET \
'https://account.fauna.com/v2/exports/<EXPORT_ID>' \
-H "Authorization: Bearer $ACCOUNT_KEY_SECRET"
Alternatively, you can use the List Exports endpoint to get a list of all exports and their states for an account:
curl -X GET \
'https://account.fauna.com/v2/exports' \
-H "Authorization: Bearer $ACCOUNT_KEY_SECRET"
You can optionally filter the export list by state
and database
. You can
pass the state
query parameter multiple times to filter on multiple states.
# Gets exports in the 'Pending' or 'Complete' state
# for the 'us-std/parent_db/child_db' database.
curl -X GET \
--get \
'https://account.fauna.com/v2/exports' \
-H "Authorization: Bearer $ACCOUNT_KEY_SECRET" \
--data-urlencode 'state=Pending' \
--data-urlencode 'state=Complete' \
--data-urlencode 'database=us-std/parent_db/child_db'
Export files
The export operation writes the following files to the specified S3 bucket path:
Collection files
Fauna exports document data for each collection as one or more JSON files. Exported collection files use the following naming convention:
collections/<COLLECTION_NAME>/<COLLECTION_NAME>_<HOST_ID>_<FILE_IDX>.json
Where:
-
<COLLECTION_NAME>
is the name of the collection. -
<HOST_ID>
is an identifier, ranging from00
to99
for the Fauna host that uploaded the file. An export may have multiple hosts. -
<FILE_IDX>
is an index for the file, ranging from000000
to999999
. Fauna writes collection files sequentially in ascending index order, starting with000000
for each host.
Collection file names can vary between exports, even exports of the same database with the same data.
Each collection export file contains an array of JSON objects, where each object
represents a collection document. Documents are ordered by ascending document
ID. Document data is converted from FQL to JSON based on the
data format specified
in the request. Example using the
tagged
format:
[
{
"id": "419646277065637921",
"coll": {
"@mod": "Category"
},
"ts": {
"@time": "2099-01-09T23:18:46.510Z"
},
"name": "party",
"description": "Party Supplies"
},
...
]
Manifest file
When the export is complete, Fauna uploads a manifest.json
file to the S3
bucket path. The manifest.json
file contains metadata about the export and
describes the structure and details of the exported data.
{
"export_id": "123456789",
"snapshot_ts": "2099-01-10T17:35:38.097961Z",
"datafile_compression": false,
"datafile_format": "json",
"document_format": "tagged",
"object_count": 10,
"object_keys": [
"fauna_exports/parent_db/child_db/2099-12-31/collections/Category/Category_00_000000.json",
"fauna_exports/parent_db/child_db/2099-12-31/collections/Product/Product_00_000000.json",
...
]
}
Field | Type | Description |
---|---|---|
|
String |
Unique ID for the export. The ID is a string-encoded integer. |
|
String |
The timestamp of the export’s snapshot in ISO 8601 format. Exports capture a
consistent state of the database’s
document data at the |
|
Boolean |
Indicates whether the export’s collection files are compressed. |
|
String |
Format for the export’s collection files. Must be |
|
String |
Data format used to
convert the export’s FQL document data to JSON. Valid values are
|
|
Integer |
Total number of the export’s collection files. |
|
Array of strings |
Paths to the export’s collection files. |
Billing
Fauna charges Transactional Compute Operations (TCOs) for each GB of delivered export data. See Snapshot export delivery in the Fauna Billing docs.
Limitations
-
Exports do not include:
-
Computed field values
-
Database schema or
.fsl
schema files -
Changes made to the database after the export’s
snapshot_ts
-
-
Once requested, you can’t cancel an export.