Compressing Python dictionary objects before storing in json S3 files.

Here's a quick little script I wrote since I need to test uploading files into s3. In this case the file generated will be 78 bytes. When unzipped 170 Bytes. The reason I wrote this is because I have to upload large amounts of data in json form into S3. Saving space in S3 results in pretty great savings. Here is the code:

#!/usr/bin/env python3
import boto3, json
from io import BytesIO
from gzip import GzipFile

data = {
    "foo1": "bar",
    "harry1": "salad",
    "foo2": "bar",
    "harry2": "salad",
    "foo3": "bar",
    "harry3": "salad",
    "foo4": "bar",
    "harry4": "salad",
    "foo5": "bar",
    "harry5": "salad",
}

gz_body = BytesIO()
gz = GzipFile(None, 'wb', 9, gz_body)
gz.write(json.dumps(data).encode('utf-8'))
gz.close()

s3 = boto3.resource('s3')
bucket = "<your datalake bucket>"
s3_key = "test/test.gz"
try:
    f = s3.Object(bucket, s3_key).put(Body=gz_body.getvalue())
except Exception as e:
    print("Error: ", e)

This code takes a python dictionary and loads it compressed into S3 into s3://<your datalake bucket>/test/test.gz.