DynamoDB datasource

Only available for Node.js.

The DynamoDB datasource connects Amazon DynamoDB tables to Forest. It introspects each table or reads a schema you declare, exposes the partition key and the sort key as a composite primary key, and turns arrays of maps into collections of their own.

Installation

npm install @forestadmin/datasource-dynamodb

The package brings the AWS SDK clients it needs.

Basic usage

import { createAgent } from '@forestadmin/agent';
import { createDynamoDBDataSource } from '@forestadmin/datasource-dynamodb';

const agent = createAgent(options);

// Every table of the account and region, introspected
agent.addDataSource(createDynamoDBDataSource({ region: 'eu-west-1' }));

Credentials and region

DynamoDB has no connection string. The AWS SDK resolves the credentials from its standard chain, in this order: environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY), a shared profile (AWS_PROFILE), then an IAM role. The region option sets the region, and AWS_REGION does the same when the option is absent.

Option	Description
`region`	AWS region of the tables.
`credentials`	Credentials passed to the AWS SDK, instead of the standard chain.
`endpoint`	Endpoint URL, for DynamoDB Local.
`clientConfig`	Extra configuration passed to the `DynamoDBClient`.
`client`	A `DynamoDBClient` or `DynamoDBDocumentClient` the project already has.

The datasource calls ListTables and DescribeTable to introspect, Query, Scan, GetItem and BatchGetItem to read, PutItem, UpdateItem, DeleteItem and BatchWriteItem to write, and ExecuteStatement for native queries. A read-only back-office needs only the read operations.

Choose the tables and adjust the collections

By default the datasource introspects every table of the account and region and names each collection after its table. List the tables to restrict the set:

createDynamoDBDataSource({
  region: 'eu-west-1',
  tables: ['Customers', 'Orders'],
});

Use the builder to adjust collections one table at a time: to name a collection, tune its introspection or share one table between several collections. The datasource introspects the tables the builder names once, through the builder, and keeps introspecting the other tables:

createDynamoDBDataSource({
  region: 'eu-west-1',
  builder: configurator =>
    configurator
      .addCollectionFromTable({ name: 'users', tableName: 'App', entity: { sortKeyPrefix: 'USER#' } })
      .addCollectionFromTable({ name: 'orders', tableName: 'App', entity: { sortKeyPrefix: 'ORDER#' } }),
});

In this example App becomes two collections and every other table keeps its own. Add tables: [] to expose the builder collections only.

Schema introspection

DescribeTable returns the keys and the indexes of a table. DynamoDB does not type the other attributes, so the datasource infers them from a sample of items.

Option	Default	Description
`sampleSize`	`100`	Items read per table to infer its attributes.
`maxDepth`	`5`	Deepest nested map exposed as separate columns.

Set both defaults for all tables with introspectionConfig: { sampleSize, maxDepth }. A table scan has no order, so the sample does not hold the latest items. Nested maps become columns named with @@@, such as address@@@city, which Forest shows as a path. An attribute absent or null in part of the sample is nullable. An attribute seen with several types becomes a Json column.

Save or write the introspection

introspect samples the tables and describes them as collection definitions. Save the result and give it back as the introspection of the datasource: the agent then starts without calling DescribeTable or sampling the tables.

import { createDynamoDBDataSource, introspect } from '@forestadmin/datasource-dynamodb';

// Once, from a script
const introspection = await introspect({ region: 'eu-west-1' });
await fs.promises.writeFile('introspection.json', JSON.stringify(introspection, null, 2));

// At startup
agent.addDataSource(
  createDynamoDBDataSource(
    { region: 'eu-west-1' },
    { introspection: JSON.parse(await fs.promises.readFile('introspection.json', 'utf8')) },
  ),
);

The file is plain JSON. Edit it to add what the sampling does not know, such as a nullable attribute, an index that covers all items or a set, or write the whole file by hand. An introspection does not combine with a builder.

createDynamoDBDataSource(
  { region: 'eu-west-1' },
  {
    introspection: {
      version: 1,
      collections: [
        {
          name: 'orders',
          tableName: 'Orders',
          partitionKey: { name: 'customerId', type: 'string' },
          sortKey: { name: 'orderId', type: 'string' },
          indexes: [
            {
              name: 'by-status',
              type: 'GSI',
              partitionKey: { name: 'status', type: 'string' },
              sortKey: { name: 'createdAt', type: 'string' },
              coversAllItems: true,
            },
          ],
          fields: [
            { name: 'total', type: 'number' },
            { name: 'status', type: 'string', nullable: true },
            { name: 'tags', type: 'array', subType: 'string', set: true },
            { name: 'address', type: 'object', fields: [{ name: 'city', type: 'string' }] },
          ],
        },
      ],
    },
  },
);

Keys are strings or numbers. The field types are string, number, boolean, date, dateonly, timeonly, object, array, point and binary. date and dateonly hold ISO 8601 strings. binary holds base64 text and Forest filters it on the items the datasource reads. point holds GeoJSON, which Forest shows as Json. timeonly appears as Forest’s Time. An object without nested fields stays one Json column. An index with a sort key leaves out the items that lack that attribute. The datasource reads such an index only when the filter has a condition on its sort key. Declare coversAllItems: true when every item holds the attribute, and the datasource reads the index whenever a filter pins its partition key. A string or number set reads as an array. The datasource writes it back as a set when the introspection saw the attribute as a set, or when the file declares set: true on the array. Emptying a set in Forest removes the attribute, because DynamoDB holds no empty set. A number beyond 2^53 reads as text, which keeps every digit.

Composite keys

The partition key and the sort key are both primary key columns of the collection, so Forest identifies a record by the two. Keys do not change once a record exists: editing a key returns an error, and moving a record means deleting it and creating another. Forest greys the key fields in the creation form. When a string partition key or sort key stays empty, the datasource generates a UUID, and a sort key keeps its entity prefix when the collection has one. A numeric key has no generated value, so the record needs it from another source.

Single-table design

Several collections can share one table, each restricted to its own items. Pick the items with a sort key prefix or with a discriminator attribute:

configurator
  .addCollectionFromTable({
    name: 'users',
    tableName: 'App',
    entity: { sortKeyPrefix: 'USER#' }, // or { attribute: 'entityType', value: 'USER' }
  })
  .addCollectionFromTable({
    name: 'orders',
    tableName: 'App',
    entity: { sortKeyPrefix: 'ORDER#' },
  });

The sort key prefix also serves as the sort key condition of queries, and the datasource checks it when it creates a record.

Arrays as collections

Expose an array of maps as a collection of its own with virtualCollections:

createDynamoDBDataSource({
  virtualCollections: [
    { parentCollection: 'orders', collectionName: 'order_lines', arrayField: 'lines' },
  ],
});

Each element of the array becomes a record. Its identity is the key of the table item plus its index in the array. Creating a record appends an element, updating sets the attributes of the element, and deleting removes it, so the index of the following elements shifts. arrayField points inside a map with @@@, as in meta@@@items. An array inside an array becomes a collection whose parent is the first virtual collection, declared after it. The position in the outer array becomes a key column named <parent>Index, and creating an inner element needs it:

virtualCollections: [
  { parentCollection: 'orders', collectionName: 'order_lines', arrayField: 'lines' },
  { parentCollection: 'order_lines', collectionName: 'line_parts', arrayField: 'parts' },
];

What Forest filters and sorts on

DynamoDB reads fast by key and scans otherwise, and it cannot sort, count or aggregate on the server. The datasource declares to Forest only what DynamoDB does itself, so that nothing in the interface reads a whole table to answer in memory.

Feature	Behavior
Filters	A column offers the operators DynamoDB evaluates: `Equal`, `In`, comparisons, `StartsWith`, `Contains`, `Blank` and `Present`. `IContains`, `Like`, `EndsWith` and the relative date operators are not offered. A filter on a key uses the key. A filter on another attribute is a filter expression, which still reads the table.
Access path	A `Query` on the table or on an index when an equality pins the partition key, a `BatchGetItem` when the filter names whole keys, a `Scan` otherwise.
Sort	Only the sort key of the table or of an index sorts, because DynamoDB orders by it. Forest hides the sort of other attributes and the datasource refuses a request for one.
Pagination	DynamoDB has no offset. The datasource remembers the key where earlier pages stopped and resumes from the closest one. `maxOffset` (default `100000`) caps how deep a page goes.
Count and charts	The datasource computes them on the items it reads, in parallel segments for a table. It keeps results in memory for `chartCacheTtlMs` (default 30 seconds) and drops them when a write goes through the agent. Set `approximateCount: true` to answer an unfiltered count from the table description, which DynamoDB refreshes about every 6 hours.
Search	Forest searches the string columns with `Contains`, which is case-sensitive, and it scans.
Relations	DynamoDB has none. Declare them with `addManyToOneRelation` and the other relation methods: a lookup by key costs one `BatchGetItem`.
Array collections	The attributes of an element live inside the item, so only the keys of the item and the position of the element filter. The other columns neither filter nor sort.

DynamoDB orders and compares dates as text. Store dates in one offset, UTC, when a date is a sort key or when you filter on ranges of it.

Allow in-memory processing

Set inMemoryProcessing: true to bring back every operator Forest knows (IContains, Like, relative dates), a sort on any attribute and filters on the attributes of array elements. The datasource then reads the matching items and does the work itself, bounded by maxScannedItems and governed by scanPolicy. The setting is off by default because it reads more than it returns.

createDynamoDBDataSource({
  region: 'eu-west-1',
  inMemoryProcessing: true,
});

Control read costs

A request that no key or index serves reads the whole table. scanPolicy decides what happens:

Value	Behavior
`'warn'` (default)	Scans and logs a warning that names the table and the filtered fields, never the values.
`'allow'`	Scans without a warning.
`'deny'`	Refuses the request with an explicit error.

The datasource refuses any other value at startup. Set scanPolicy on the datasource or on a single collection. maxScannedItems (default 1000000) stops a request that reads more items than the limit. The request ends with an error, never with a partial result. Set it on the datasource or on a collection. logQueries logs the expressions sent to DynamoDB, and logConsumedCapacity logs the capacity units each request consumed. The AWS SDK retries throttled requests in adaptive mode, up to 10 attempts.

Native queries

Set liveQueryConnection to offer PartiQL SELECT statements as native queries, with $name placeholders:

createDynamoDBDataSource({
  region: 'eu-west-1',
  liveQueryConnection: 'dynamo',
});

SELECT * FROM "Orders" WHERE customerId = $customer

The datasource refuses any statement that does not start with SELECT. PartiQL cannot aggregate, so build charts from the aggregations of a collection instead.

Limitations

Collections backed by a secondary index are not available. The datasource still reads an index as an access path of its table’s collection.
Updates have no optimistic locking: two users who edit one record at the same time both succeed, and the last write wins.
Binary partition keys and sort keys are not supported.
Counts and charts read the items. An exact count on a large table reads the whole table.

Source code

This connector is open source. Browse the code or contribute on GitHub: @forestadmin/datasource-dynamodb.

Overview

Connect

Process

Build

Execute

Collaborate

Manage

Embed

Control

Project Settings

Installation

Basic usage

Credentials and region

Choose the tables and adjust the collections

Schema introspection

Save or write the introspection

Composite keys

Single-table design

Arrays as collections

What Forest filters and sorts on

Allow in-memory processing

Control read costs

Native queries

Limitations

Source code

​Installation

​Basic usage

​Credentials and region

​Choose the tables and adjust the collections

​Schema introspection

​Save or write the introspection

​Composite keys

​Single-table design

​Arrays as collections

​What Forest filters and sorts on

​Allow in-memory processing

​Control read costs

​Native queries

​Limitations

​Source code

Installation

Basic usage

Credentials and region

Choose the tables and adjust the collections

Schema introspection

Save or write the introspection

Composite keys

Single-table design

Arrays as collections

What Forest filters and sorts on

Allow in-memory processing

Control read costs

Native queries

Limitations

Source code