Easy Amazon Glacier Restore for Files and Folders

I love AWS Glacier, it is one of the cheapest and easiest ways to store your important files long term. Also with S3 lifecycles it has never been easier to move files into long term (cheap) storage. All that said, doing an Amazon Glacier Restore on folders is a real pain. In the AWS console you can do a single object restore in a few clicks. But if you need to restore a folder or entire directory AWS basically says figure it out yourself with code. After years of doing it with scripts I figured I’d just write myself a simple application to do this with a few copy and paste settings.

How to Restore Folders from Glacier to S3 (Application)

AWS does not provide an easy way to restore entire folders or directories, luckily I do provide one for Amazon Glacier Restore jobs 🙂 Without this program restoring folders can be done manually but requires knowledge of scripting with the AWS CLI or Programming through the AWS SDK. Don’t know how to code? No idea what an SDK is? Or are you like me just tired of having to script this out every time you need a folder restored? No problem, you can download this simple open source Python App that will do that all for you.

Application Files –

• Windows .EXE – https://github.com/nathanielkam/python-glacier-restore/raw/master/dist/glacier-restore-to-s3.exe
• MacOS App – https://github.com/nathanielkam/python-glacier-restore/raw/master/dist/glacier-restore-to-s3.app.zip

The GitHub project is open source with instructions in the readme on how to run directly from Python scripts OR to rebundle your own binaries using Pyinstaller.

Prerequisites –

The only thing you need to run this app is IAM keys (with restoreObject permissions) and knowledge of where your folder is located in S3. If your account is managed using IAM you will need to get these keys / permissions from your administrator. If you own your AWS account (ie you login with an email address) then you can get / create keys for yourself in the My Security Credentials section of your account settings (use the link to jump you there directly).

Amazon Glacier Restore Instructions –

Who should use this program for Amazon Glacier Restore?

Anyone who does not have the time or experience to code their own solution should use this program to do their folder restores.

How to Restore Files from Amazon Glacier (Web Console)

AWS makes it fairly trivial to restore files ONE AT A TIME through the web console, but they don’t make it easy to restore folders.

To restore single files from Amazon Glacier you login to your AWS Console, go to S3, find your file, right click and fill out the restore options dialog.

You can use the search bar under “services” to find s3 easily. You can also use this link to jump directly to the s3 services page, https://s3.console.aws.amazon.com/s3/home?region=us-west-1

Once in the S3 Web Console you will need to find the object you want to restore. Do that by browsing through the folder structure just like you do on a computer. When you find the file you want to restore, right-click and go to “Restore”

Choose the options you want to restore your file. Note you will be charged for all retrieval requests with an extra charge on Expedited retrieval. For current retrieval pricing see – https://aws.amazon.com/glacier/pricing/

What is the AWS Glacier Restore Time?

There are 3 tiers of Glacier Restore jobs, Bulk, Standard, and Expedited. When using the Expedited tier (which does cost extra) files are restored in 1-5 minutes. When using standard retrieval files will be available in 3-5 hours. When using bulk retrieval, files will be ready between 5-12 hours.

How does my AWS Glacier Restore Application Work?

My application is written in Python 3, uses Tkinter for the GUI, and the official AWS SDK Boto3. Developer tutorial below,

Environment Breakdown

First download the repo from GitHub and navigate to it in terminal or powershell – https://github.com/nathanielkam/python-glacier-restore

Next, create a virtual environment to install your dependencies, I prefer venv (use python3 if you have multiple version on your computer)

pip install virtualenv
virtualenv venv

Now that you have a venv you will need to activate it

• Linux / MacOSX: Run source venv/bin/activate
• Windows: Run venv\env\Scripts\activate.bat

You can now install the project requirements into your venv

pip install -r requirements.txt

Code Breakdown

Import tkinter for UI, boto3 for AWS, and webbrowser for hyperlinks.

from tkinter import *
from tkinter import messagebox
import boto3
import webbrowser
import tkinter as tk

Initialize Global Variables –

We will use fields to build out form UI

# Initialize
fields = ('AWS Access Key', 'AWS Secret Key','S3 Bucket (Format: bucketname)', 'Folder to Restore (Format: folder1/folder2/)', 'Bucket Region (Example: us-east-1)',)

Define Python Hyperlink Handling Function –

# Enable Hyperlinking
def hyperlink(url):
    webbrowser.open_new(url)

Define Function to Create a Boto3 Client –

This is technically not needed and boto3.client can be created on the fly. It is very important we use sessions to connect to AWS rather than using the standard method of AWS settings files (used by AWS CLI) as this app is intended for on the fly requests by users who DO NOT have AWS CLI and would not understand the config process.

# Create an S3 Client Session
def create_client(entries):
   client = boto3.client(
      's3', 
      aws_access_key_id=entries['AWS Access Key'].get(), 
      aws_secret_access_key=entries['AWS Secret Key'].get(),
      region_name=entries['Bucket Region (Example: us-east-1)'].get()
   )
   return client

Get list of glacier objects and trigger restore jobs

This section is the actual meat of the program that does the work. The most important part of getting object lists in the AWS SDK is handling for more than 1000 keys (objects). You can also see there is a default 14 day restoration lifespan. You can edit this to whatever you want. Additionally, if you wanted to add an expedited retrieval class you could easily do this by duplicating the Bulk v Standard Control statements.

def restore_glacier_contents(entries, type):

   client = create_client(entries)    

   bucket = entries['S3 Bucket (Format: bucketname)'].get()
   prefix = entries['Folder to Restore (Format: folder1/folder2/)'].get()

   # Try to get S3 list
   try:
      s3_result = client.list_objects_v2(
         Bucket=bucket,
         Prefix=prefix
      )
   except:
      messagebox.showerror("Error", "You are unable to list these files.")
      exit()

   if 'Contents' not in s3_result:
      return []

   file_list = []

   # Get list of Contents
   for file in s3_result['Contents']:
      # Only Log Glacier Files
      if file['StorageClass'] == 'GLACIER':
         if type=="Bulk":
            response = client.restore_object(Bucket=bucket, Key=file['Key'],RestoreRequest={'Days': 14, 'GlacierJobParameters': {
            'Tier': 'Bulk'}})
         elif type=="Standard":
            response = client.restore_object(Bucket=bucket, Key=file['Key'],RestoreRequest={'Days': 14, 'GlacierJobParameters': {
            'Tier': 'Standard'}})
         file_list.append(file['Key'])

   # Get list of Contents when More than 1000 Items
   while s3_result['IsTruncated']:
      continuation_key = s3_result['NextContinuationToken']
      s3_result = s3_conn.list_objects_v2(Bucket=bucket, Prefix=prefix, Delimiter="/", ContinuationToken=continuation_key)
      for file in s3_result['Contents']:
            # Only Log Glacier Files
         if file['StorageClass'] == 'GLACIER':
            if type=="Bulk":
                  response = client.restore_object(Bucket=bucket, Key=file['Key'],RestoreRequest={'Days': 14, 'GlacierJobParameters': {
               'Tier': 'Bulk'}})
            elif type=="Standard":
               response = client.restore_object(Bucket=bucket, Key=file['Key'],RestoreRequest={'Days': 14, 'GlacierJobParameters': {
               'Tier': 'Standard'}})
            file_list.append(file['Key'])

   # Return File List
   return file_list

Route the Restore Request

Depending on what type of restore request was submitted we will send different parameters to the restore code as well as return different responses.

def glacier_restore(entries, type):

   if type=="Bulk":
      files = restore_glacier_contents(entries, type)
      #print(files)
      # If Bulk Success
      messagebox.showinfo("Bulk Request Success!", "Bulk Restore Request Successful! Your files will be restored in 5-12 hours.")

   elif type=="Standard":
      files = restore_glacier_contents(entries, type)
      #print(files)
      # If Standard Success
      messagebox.showinfo("Standard Request Success!", "Standard Restore Request Successful! Your files will be restored in 3-5 hours.")

   else:
      # If Failure
      messagebox.showerror("Error", "There was an issue requesting restore of files, please contact your S3 administrator.")

Make the Form and Field Entries

The code below creates the form and field entries, there are a couple of custom cases like “AWS Secret Key” that require specific settings like obfuscating text entries with “*”.

def makeform(root, fields):
       
   # Initialize Entries
   entries = {}
   
   for field in fields:
      row = Frame(root)

      # Add Field Title
      lab = Label(row, width=50, text=field+": ", anchor='w')

      # Add Field Entry
      if field=="AWS Secret Key":
         # AWS Secret Key is sensitive data and should be obscured
         ent = Entry(row, width=61, show="*")

      else:  
         ent = Entry(row, width=61)

      # Add Styling
      row.pack(side = TOP, fill = X, padx = 5 , pady = 5)
      lab.pack(side = LEFT)
      ent.pack(side = LEFT, expand = YES, fill = X)
      
      # Save to Entries
      entries[field] = ent
   return entries

Complete the UI and Tie Buttons to Functions

Here we are just completing the UI by adding headings, text, the form, and then the buttons which we tie to functions defined earlier in the doc.

if __name__ == '__main__':
       
   # Init TK
   root = Tk()

   # Set Window Title
   root.title("AWS Glacier Restore to S3")
   lab1 = Label(root, text="AWS Glacier Restore to S3", font=('Helvetica', 14))
   lab1.pack(side = TOP)

   lab2 = Label(root, text="Restoration will take a few minutes as files restore 1 by 1, program is working even if it says it is Not Responding.", font=('Helvetica', 10))
   lab2.pack(side = TOP)

   # Add Hyperlink to AWS Region Documentation
   link2 = Label(root, text="GitHub", fg="blue", cursor="hand2")
   link2.pack(side = BOTTOM, fill = X, padx = 5 , pady = 5)
   link2.bind("<Button-1>", lambda e: hyperlink("https://github.com/nathanielkam/python-glacier-restore"))

   # Add Hyperlink to AWS Region Documentation
   link1 = Label(root, text="Documentation: AWS Region Codes", fg="blue", cursor="hand2")
   link1.pack(side = TOP, fill = X, padx = 5 , pady = 5)
   link1.bind("<Button-1>", lambda e: hyperlink("https://docs.aws.amazon.com/general/latest/gr/rande.html"))

   # Init the Form
   ents = makeform(root, fields)

   # Add Quit Button
   b3 = Button(root, text = 'Quit', command = root.quit)
   b3.pack(side = RIGHT, padx = 5, pady = 5)

   # Add Bulk Retrieval Button
   b1 = Button(root, text = 'Bulk Retrieval (5-12 Hours)',
   command=(lambda e = ents: glacier_restore(e, "Bulk")))
   b1.pack(side = RIGHT, padx = 5, pady = 5)

   # Add Standard Retrieval Button
   b2 = Button(root, text='Standard Retrieval (3-5 Hours)',
   command=(lambda e = ents: glacier_restore(e, "Standard")))
   b2.pack(side = RIGHT, padx = 5, pady = 5)

   # Run
   root.mainloop()

My Personal Projects

I am working on publishing all personal projects / problem solving I do with code / technology. Check out my other personal projects here – https://nathanielkam.com/tag/personal-projects/

Summary

Author Rating

Aggregate Rating

5 based on 37 votes

Software Name

AWS Glacier Restore - Folders to S3

Operating System

MacOSX, Windows 10

Software Category

AWS Glacier Restore

Price

USD 0.00

Landing Page

https://github.com/nathanielkam/python-glacier-restore