How to export an existing dynamo table schema to json?

Question

I'd like to replicate some dynamodb tables, schema only, into my local environment for testing purposes. First I've tried:

aws dynamodb describe-table --table-name Foo > FooTable.json

But it's obvious that the output schema is not compliant to the input schema from the create-table command:

aws dynamodb create-table --cli-input-json file://FooTable.json --endpoint=http://localhost:8000

What I'm trying to avoid is to generate dozens of skeletons with aws dynamodb create-table --generate-cli-skeleton and fill them manually :/

Is there a way to get the table schema in a format that is "useful" for recreation? I find it unbelievable that there are no straightforward way of doing it through the web graphic interface or the standard aws command line - after hearing how "good" was their service.

Answer 1

I just managed to do a complete dump and "restore" using bchew/dynamodump:

git clone git@github.com:bchew/dynamodump.git

Notice the --schemaOnly option in the documentation https://github.com/bchew/dynamodump. Command was:

./dynamodump.py -m backup --schemaOnly --region foo-region --host localhost --srcTable '*' --port 8000 --accessKey fooKey --secretKey barKey

Then you can use the -m restore mode to put the data or schema back into a local dynamodb or wherever desired :)

With that said, I still find it unbelievable how bad is the amazon dynamodb tool-chain. Come on guys.

Answer 2

DynamoDB is a NoSQL database that, with the exception of the primary key and secondary indexes doesn't have a schema. You may notice in the describe-table that it only shows the keys and indexes, not any other columns.

I usually write code to create the table and keys in some sort of initialization code and then just begin using the table. It won't hurt if the table is already there. Later if I try to insert data and the "column" doesn't exist it doesn't matter as DynamoDB will create it automatically.

If you need to export the data, there are tools for that that will allow you to export (and some allow import too) from DynamoDB.

Answer 3

This takes aws dynamodb describe-table output, and transforms it into the input-format of aws dynamodb create-table --cli-input-json:

AWS_PROFILE=xyz aws dynamodb describe-table --table-name MyTable > mytable_full.json

# Pull out just what is minimally needed for table-creation:
#
#  TableName
#  KeySchema
#  AttributeDefinitions (must only contain attributes used in keys)
#  Global/Local Secondary Indexes
#  Defaults BillingMode to PAY_PER_REQUEST
#   (any provisioning can be set up manually based on load)

jq <mytable_full.json '.Table | {TableName, KeySchema, AttributeDefinitions} + (try {LocalSecondaryIndexes: [ .LocalSecondaryIndexes[] | {IndexName, KeySchema, Projection} ]} // {}) + (try {GlobalSecondaryIndexes: [ .GlobalSecondaryIndexes[] | {IndexName, KeySchema, Projection} ]} // {}) + {BillingMode: "PAY_PER_REQUEST"}' ​>mytable.json

AWS_PROFILE=xyz aws dynamodb create-table --cli-input-json file://mytable.json

You can also paste the json into python (the python dict syntax closely matches json) eg

import boto3

dynamodb = boto3.resource("dynamodb")

tabledef = {
    "TableName": "MyTable",
    "KeySchema": [
...
}

table = dynamodb.create_table(**tabledef)
print("Table status: ", table.table_status)

References:

https://docs.aws.amazon.com/cli/latest/reference/dynamodb/describe-table.html https://docs.aws.amazon.com/cli/latest/reference/dynamodb/create-table.html https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_CreateTable.html https://boto3.amazonaws.com/v1/documentation/api/latest/guide/dynamodb.html#creating-a-new-table

Answer 4

Here is a version using C#, AWS CLI and Newtonsoft JSON on Windows. Start by running this command: -

aws dynamodb describe-table --table-name TheTable --profile SourceAWSCredsProfile > TheTable.json

Pick up the file, deserialize and serialize to the --cli-input-json friendly class: -

TableContainer tableContainer;

string sourceFile = "TheTable.json";
string destFile = "TheTable.cli-input.json";

using (StreamReader file = File.OpenText(sourceFile))
{
    JsonSerializer serializer = new JsonSerializer();
    tableContainer = (TableContainer)serializer.Deserialize(file, typeof(TableContainer));
}

File.WriteAllText(destFile, JsonConvert.SerializeObject(tableContainer.Table, Formatting.Indented, new JsonSerializerSettings
    {
        NullValueHandling = NullValueHandling.Ignore
    }
));

Now run this command to import the table definition: -

aws dynamodb create-table --cli-input-json file://TheTable.cli-input.json --profile DestinationAWSCredsProfile

The TableContainer class definition is below. The absence of certain properties cleans out everything that the --cli-input-json parameter doesn't need. You can recreate this class anytime by running: -

aws dynamodb create-table --generate-cli-skeleton

Then copy and paste the output into a new class file, using the very handy Paste Special... Paste JSON as Classes feature in Visual Studio.

public class TableContainer
{
    public DynamoTableCLIJSON Table { get; set; }
}

public class DynamoTableCLIJSON
{
    public Attributedefinition[] AttributeDefinitions { get; set; }
    public string TableName { get; set; }
    public Keyschema[] KeySchema { get; set; }
    public Localsecondaryindex[] LocalSecondaryIndexes { get; set; }
    public Globalsecondaryindex[] GlobalSecondaryIndexes { get; set; }
    public string BillingMode { get; set; }
    public Provisionedthroughput ProvisionedThroughput { get; set; }
    public Streamspecification StreamSpecification { get; set; }
    public Ssespecification SSESpecification { get; set; }
    public Tag[] Tags { get; set; }
}

public class Provisionedthroughput
{
    public int ReadCapacityUnits { get; set; }
    public int WriteCapacityUnits { get; set; }
}

public class Streamspecification
{
    public bool StreamEnabled { get; set; }
    public string StreamViewType { get; set; }
}

public class Ssespecification
{
    public bool Enabled { get; set; }
    public string SSEType { get; set; }
    public string KMSMasterKeyId { get; set; }
}

public class Attributedefinition
{
    public string AttributeName { get; set; }
    public string AttributeType { get; set; }
}

public class Keyschema
{
    public string AttributeName { get; set; }
    public string KeyType { get; set; }
}

public class Localsecondaryindex
{
    public string IndexName { get; set; }
    public Keyschema1[] KeySchema { get; set; }
    public Projection Projection { get; set; }
}

public class Projection
{
    public string ProjectionType { get; set; }
    public string[] NonKeyAttributes { get; set; }
}

public class Keyschema1
{
    public string AttributeName { get; set; }
    public string KeyType { get; set; }
}

public class Globalsecondaryindex
{
    public string IndexName { get; set; }
    public Keyschema2[] KeySchema { get; set; }
    public Projection1 Projection { get; set; }
    public Provisionedthroughput1 ProvisionedThroughput { get; set; }
}

public class Projection1
{
    public string ProjectionType { get; set; }
    public string[] NonKeyAttributes { get; set; }
}

public class Provisionedthroughput1
{
    public int ReadCapacityUnits { get; set; }
    public int WriteCapacityUnits { get; set; }
}

public class Keyschema2
{
    public string AttributeName { get; set; }
    public string KeyType { get; set; }
}

public class Tag
{
    public string Key { get; set; }
    public string Value { get; set; }
}