Easy Amazon Glacier Restore for Files and Folders

I love AWS Glacier, it is one of the cheapest and easiest ways to store your important files long term. Also with S3 lifecycles it has never been easier to move files into long term (cheap) storage. All that said, doing an Amazon Glacier Restore on folders is a real pain. In the AWS console you can do a single object restore in a few clicks. But if you need to restore a folder or entire directory AWS basically says figure it out yourself with code. After years of doing it with scripts I figured I’d just write myself a simple application to do this with a few copy and paste settings.

How to Restore Folders from Glacier to S3 (Application)

AWS does not provide an easy way to restore entire folders or directories, luckily I do provide one for Amazon Glacier Restore jobs 🙂 Without this program restoring folders can be done manually but requires knowledge of scripting with the AWS CLI or Programming through the AWS SDK. Don’t know how to code? No idea what an SDK is? Or are you like me just tired of having to script this out every time you need a folder restored? No problem, you can download this simple open source Python App that will do that all for you.

Application Files –

The GitHub project is open source with instructions in the readme on how to run directly from Python scripts OR to rebundle your own binaries using Pyinstaller.

Prerequisites –

The only thing you need to run this app is IAM keys (with restoreObject permissions) and knowledge of where your folder is located in S3. If your account is managed using IAM you will need to get these keys / permissions from your administrator. If you own your AWS account (ie you login with an email address) then you can get / create keys for yourself in the My Security Credentials section of your account settings (use the link to jump you there directly).

AWS - Create Access key
AWS – Create Full Access Key with All Permissions

Amazon Glacier Restore Instructions –

Time needed: 5 minutes.

Follow these simple 5 minute instructions to submit an Amazon Glacier Restore job for Folders.

  1. Download the right version for your Operating System

    Windows .EXEhttps://github.com/nathanielkam/python-glacier-restore/raw/master/dist/glacier-restore-to-s3.exe

    MacOS Apphttps://github.com/nathanielkam/python-glacier-restore/raw/master/dist/glacier-restore-to-s3.app.zip

  2. Open the Application by Double Clicking it

  3. Fill out the restore job form

    Pay attention to the example and format requirements listed.
    Example AWS Glacier Restore to S3

  4. Choose a Retrieval Tier Option (Standard / Bulk)

    I did not put expedited into the app because expedited retrievals have additional fees.

  5. Wait for a few depending on number of files being restored

    The program is non threaded so it will send 1 restore request per file. In general it takes about 2 minutes to restore 1000 files.

  6. A success popup will appear when restore request is complete

    Technically you send restore request that are then processed over the SLA time period your chose. You do not directly restore files.

  7. After X hours you may download files in S3 console or S3 client of your choosing

    After the SLA is fulfilled download the S3 files in your web console or favorite S3 client, I prefer CrossFTP which has a pro trial allowing for massively threaded downloads.

Who should use this program for Amazon Glacier Restore?

Anyone who does not have the time or experience to code their own solution should use this program to do their folder restores.

How to Restore Files from Amazon Glacier (Web Console)

AWS makes it fairly trivial to restore files ONE AT A TIME through the web console, but they don’t make it easy to restore folders.

To restore single files from Amazon Glacier you login to your AWS Console, go to S3, find your file, right click and fill out the restore options dialog.

You can use the search bar under “services” to find s3 easily. You can also use this link to jump directly to the s3 services page, https://s3.console.aws.amazon.com/s3/home?region=us-west-1

Amazon Glacier Restore - AWS Console S3
Amazon Glacier Restore – AWS Console S3

Once in the S3 Web Console you will need to find the object you want to restore. Do that by browsing through the folder structure just like you do on a computer. When you find the file you want to restore, right-click and go to “Restore”

Amazon Glacier Restore - S3 Restore Menu
Amazon Glacier Restore – S3 Restore Menu

Choose the options you want to restore your file. Note you will be charged for all retrieval requests with an extra charge on Expedited retrieval. For current retrieval pricing see – https://aws.amazon.com/glacier/pricing/

Amazon Glacier Restore - Glacier Restore Options
Amazon Glacier Restore – Glacier Restore Options

What is the AWS Glacier Restore Time?

There are 3 tiers of Glacier Restore jobs, Bulk, Standard, and Expedited. When using the Expedited tier (which does cost extra) files are restored in 1-5 minutes. When using standard retrieval files will be available in 3-5 hours. When using bulk retrieval, files will be ready between 5-12 hours.

How does my AWS Glacier Restore Application Work?

My application is written in Python 3, uses Tkinter for the GUI, and the official AWS SDK Boto3. Developer tutorial below,

Environment Breakdown

First download the repo from GitHub and navigate to it in terminal or powershellhttps://github.com/nathanielkam/python-glacier-restore

Next, create a virtual environment to install your dependencies, I prefer venv (use python3 if you have multiple version on your computer)

pip install virtualenv
virtualenv venv

Now that you have a venv you will need to activate it

  • Linux / MacOSX: Run source venv/bin/activate
  • Windows: Run venv\env\Scripts\activate.bat

You can now install the project requirements into your venv

pip install -r requirements.txt

Code Breakdown

Import tkinter for UI, boto3 for AWS, and webbrowser for hyperlinks.

from tkinter import *
from tkinter import messagebox
import boto3
import webbrowser
import tkinter as tk

Initialize Global Variables –

We will use fields to build out form UI

# Initialize
fields = ('AWS Access Key', 'AWS Secret Key','S3 Bucket (Format: bucketname)', 'Folder to Restore (Format: folder1/folder2/)', 'Bucket Region (Example: us-east-1)',)

Define Python Hyperlink Handling Function –

# Enable Hyperlinking
def hyperlink(url):
    webbrowser.open_new(url)

Define Function to Create a Boto3 Client –

This is technically not needed and boto3.client can be created on the fly. It is very important we use sessions to connect to AWS rather than using the standard method of AWS settings files (used by AWS CLI) as this app is intended for on the fly requests by users who DO NOT have AWS CLI and would not understand the config process.

# Create an S3 Client Session
def create_client(entries):
   client = boto3.client(
      's3', 
      aws_access_key_id=entries['AWS Access Key'].get(), 
      aws_secret_access_key=entries['AWS Secret Key'].get(),
      region_name=entries['Bucket Region (Example: us-east-1)'].get()
   )
   return client

Get list of glacier objects and trigger restore jobs

This section is the actual meat of the program that does the work. The most important part of getting object lists in the AWS SDK is handling for more than 1000 keys (objects). You can also see there is a default 14 day restoration lifespan. You can edit this to whatever you want. Additionally, if you wanted to add an expedited retrieval class you could easily do this by duplicating the Bulk v Standard Control statements.

def restore_glacier_contents(entries, type):

   client = create_client(entries)    

   bucket = entries['S3 Bucket (Format: bucketname)'].get()
   prefix = entries['Folder to Restore (Format: folder1/folder2/)'].get()

   # Try to get S3 list
   try:
      s3_result = client.list_objects_v2(
         Bucket=bucket,
         Prefix=prefix
      )
   except:
      messagebox.showerror("Error", "You are unable to list these files.")
      exit()

   if 'Contents' not in s3_result:
      return []

   file_list = []

   # Get list of Contents
   for file in s3_result['Contents']:
      # Only Log Glacier Files
      if file['StorageClass'] == 'GLACIER':
         if type=="Bulk":
            response = client.restore_object(Bucket=bucket, Key=file['Key'],RestoreRequest={'Days': 14, 'GlacierJobParameters': {
            'Tier': 'Bulk'}})
         elif type=="Standard":
            response = client.restore_object(Bucket=bucket, Key=file['Key'],RestoreRequest={'Days': 14, 'GlacierJobParameters': {
            'Tier': 'Standard'}})
         file_list.append(file['Key'])

   # Get list of Contents when More than 1000 Items
   while s3_result['IsTruncated']:
      continuation_key = s3_result['NextContinuationToken']
      s3_result = s3_conn.list_objects_v2(Bucket=bucket, Prefix=prefix, Delimiter="/", ContinuationToken=continuation_key)
      for file in s3_result['Contents']:
            # Only Log Glacier Files
         if file['StorageClass'] == 'GLACIER':
            if type=="Bulk":
                  response = client.restore_object(Bucket=bucket, Key=file['Key'],RestoreRequest={'Days': 14, 'GlacierJobParameters': {
               'Tier': 'Bulk'}})
            elif type=="Standard":
               response = client.restore_object(Bucket=bucket, Key=file['Key'],RestoreRequest={'Days': 14, 'GlacierJobParameters': {
               'Tier': 'Standard'}})
            file_list.append(file['Key'])

   # Return File List
   return file_list

Route the Restore Request

Depending on what type of restore request was submitted we will send different parameters to the restore code as well as return different responses.

def glacier_restore(entries, type):

   if type=="Bulk":
      files = restore_glacier_contents(entries, type)
      #print(files)
      # If Bulk Success
      messagebox.showinfo("Bulk Request Success!", "Bulk Restore Request Successful! Your files will be restored in 5-12 hours.")

   elif type=="Standard":
      files = restore_glacier_contents(entries, type)
      #print(files)
      # If Standard Success
      messagebox.showinfo("Standard Request Success!", "Standard Restore Request Successful! Your files will be restored in 3-5 hours.")

   else:
      # If Failure
      messagebox.showerror("Error", "There was an issue requesting restore of files, please contact your S3 administrator.")

Make the Form and Field Entries

The code below creates the form and field entries, there are a couple of custom cases like “AWS Secret Key” that require specific settings like obfuscating text entries with “*”.

def makeform(root, fields):
       
   # Initialize Entries
   entries = {}
   
   for field in fields:
      row = Frame(root)

      # Add Field Title
      lab = Label(row, width=50, text=field+": ", anchor='w')

      # Add Field Entry
      if field=="AWS Secret Key":
         # AWS Secret Key is sensitive data and should be obscured
         ent = Entry(row, width=61, show="*")

      else:  
         ent = Entry(row, width=61)

      # Add Styling
      row.pack(side = TOP, fill = X, padx = 5 , pady = 5)
      lab.pack(side = LEFT)
      ent.pack(side = LEFT, expand = YES, fill = X)
      
      # Save to Entries
      entries[field] = ent
   return entries

Complete the UI and Tie Buttons to Functions

Here we are just completing the UI by adding headings, text, the form, and then the buttons which we tie to functions defined earlier in the doc.

if __name__ == '__main__':
       
   # Init TK
   root = Tk()

   # Set Window Title
   root.title("AWS Glacier Restore to S3")
   lab1 = Label(root, text="AWS Glacier Restore to S3", font=('Helvetica', 14))
   lab1.pack(side = TOP)

   lab2 = Label(root, text="Restoration will take a few minutes as files restore 1 by 1, program is working even if it says it is Not Responding.", font=('Helvetica', 10))
   lab2.pack(side = TOP)

   # Add Hyperlink to AWS Region Documentation
   link2 = Label(root, text="GitHub", fg="blue", cursor="hand2")
   link2.pack(side = BOTTOM, fill = X, padx = 5 , pady = 5)
   link2.bind("<Button-1>", lambda e: hyperlink("https://github.com/nathanielkam/python-glacier-restore"))

   # Add Hyperlink to AWS Region Documentation
   link1 = Label(root, text="Documentation: AWS Region Codes", fg="blue", cursor="hand2")
   link1.pack(side = TOP, fill = X, padx = 5 , pady = 5)
   link1.bind("<Button-1>", lambda e: hyperlink("https://docs.aws.amazon.com/general/latest/gr/rande.html"))

   # Init the Form
   ents = makeform(root, fields)

   # Add Quit Button
   b3 = Button(root, text = 'Quit', command = root.quit)
   b3.pack(side = RIGHT, padx = 5, pady = 5)

   # Add Bulk Retrieval Button
   b1 = Button(root, text = 'Bulk Retrieval (5-12 Hours)',
   command=(lambda e = ents: glacier_restore(e, "Bulk")))
   b1.pack(side = RIGHT, padx = 5, pady = 5)

   # Add Standard Retrieval Button
   b2 = Button(root, text='Standard Retrieval (3-5 Hours)',
   command=(lambda e = ents: glacier_restore(e, "Standard")))
   b2.pack(side = RIGHT, padx = 5, pady = 5)

   # Run
   root.mainloop()

My Personal Projects

I am working on publishing all personal projects / problem solving I do with code / technology. Check out my other personal projects here – https://nathanielkam.com/tag/personal-projects/

Summary
software image
Author Rating
1star1star1star1star1star
Aggregate Rating
5 based on 37 votes
Software Name
AWS Glacier Restore - Folders to S3
Operating System
MacOSX, Windows 10
Software Category
AWS Glacier Restore
Price
USD 0.00
Landing Page
Share This:

4 thoughts on “Easy Amazon Glacier Restore for Files and Folders”

  1. This tool sounds like the silver bullet I’ve been looking for for while now. However, when trying the windows version, I end up with an error “You are unable to list these files.” I can see in the code where this exception is thrown but I cannot see why. Is there a way to get a more detailed exception message?
    Thx in advance!

    1. Hey Jan! Are you on windows 10, 64 bit? Also did you verify you’re entering folder and bucket paths in the exact format as the examples? Outside of that it is possible that you have a IAM restriction, you would at least need listbucket, read and write files, and create restore jobs in AWS to run the tool. Let me know and can try to help out more.

  2. Hello – I’m wondering if you can help me figure out why the app won’t open on my Mac. I’m downloaded the MacOS version and I’m trying to run it on High Sierra but it won’t actually open any new windows or show up in the drop down menu section. I’ve unzipped it and added it to my applications but it won’t open. Any suggestions for how to get this app to run on my system? Thanks!

    1. Hey Ben, I have also had some varying luck on MacOS, I suspect it is due to the OS wall and python tooling issues. I think I might have a High Sierra mac I can try to recompile on. Will try to add a new version this weekend for ya.

Leave a Reply

Your email address will not be published. Required fields are marked *