Leveraging Frame Admin API to Stick to your Public Cloud Budget: User Volumes

April 10, 2023 · 12 min read

Senior Solutions Architect

Leveraging Frame Admin API to Stick to your Public Cloud Budget

In a previous blog post, I showed how you could use Frame Admin API to monitor Frame accounts that had running instances because of misconfigured account capacity settings. This could lead to spending more on your cloud infrastructure than is needed. In this blog, I will demonstrate how you can use the Frame Admin API to identify and delete user volumes that have not been used in a while. Periodically removing unused volumes can help keep cloud storage costs down and can make sure that your data retention policies are being appropriately followed.

User Volumes

User Volumes in Frame come in two types: Personal Drives and Enterprise Profile disks. Use of these disks is described in our Personal Drive and Enterprise Profiles documentation, respectively. These volumes are often used for external contractor or classroom situations where users need to use the volumes for a period of time but are not a part of the core data management environment. In these use cases, the volumes are not long term storage locations for corporate data and once users are done with their work, the volumes can be deleted. Deletion of these old volumes can be done by a Frame administrator within Frame Console, but if a Frame administrator forgets, these volumes end up increasing the storage bill for the cloud account. This script was created to identify and delete these volumes and notify a Slack channel. To use the Frame Admin API, you will first need to setup API credentials. This process is documented here. Since I want to check the entire Frame “customer” for running machines, I made sure I set up my credentials at the customer level and I gave the credentials the “Customer Administrator” role. You will also need to grab the Customer ID, which can be found in the url from the API page.

Script Setup

As in the previous blog, you will need to collect some credentials for the Frame API and Slack.

You will also need to set up a few variables to define when the script should warn and delete the volumes, as well as what volumes should be ignored.

The following new variables need to be defined.

_idledays

This is the number of days that a volume is not used before the script will start to warn via Slack that the volume will be deleted.

_warningdays

Assuming the script is run daily, this is the number of days the script will send a Slack notification about the pending deletion of this volume. This gives the administrator time to warn the user about the pending deletion (the user simply has to start a Frame session to reset the idle clock) or mark the volume as one of the exceptions. If neither of these things are done, the script will delete the idle volume after the specified number of days.

_exceptions

This is a list of strings that has the “id” of volumes that should not be checked for deletion. These ids are in the format of gateway-prod.xxxxxx where the x’s are integers. This id will be provided as a part of the warning to make it easier for administrators to determine which volumes should be exempted.

A full list of the script variables (including those from the previous script) is given below.

## _clnt_id = "<ClientID>"
## _clnt_secret = "<ClientSecret>"
## _cust_id = "<CustomerID>"
## _slack_url = "<Slack Web Hook Url>"
## _idledays = <integer of the number of days since last used to consider the volume for deletion>
## _warningsdays = <integer number of days to warn an administrator that the volume will be deleted>
## _exceptions = <string array of any volumes that should be excluded from possible deletion 'gateway-prod.1234'>

Since we are going to do some time additions and subtractions, you will also need to import some functions from the datetime module

from datetime import datetime,date,timedelta

API DELETE requests

The Frame Admin API follows REST API best practices and uses different types of HTTPS requests to perform different functions. This is important to this task, because the API to delete volumes leverages an HTTPS DELETE request instead of an HTTPS GET request. So this requires us to create another function to make this request. This delete_FrameAPICall is shown below.

def delete_FrameAPICall (api_url,jsonbody):
    # Create signature
    timestamp = int(time.time())
    to_sign = "%s%s" % (timestamp, _clnt_id)
    signature = hmac.new(_clnt_secret, to_sign.encode('utf-8'), hashlib.sha256).hexdigest()
    # Prepare http request headers
    headers = { "X-Frame-ClientId": _clnt_id, "X-Frame-Timestamp": str(timestamp), "X-Frame-Signature": signature }
    # Make delete request
    r = requests.delete(api_url, headers=headers, data = jsonbody)
    if (r.status_code == 200) :
      return (r.content)
    elif (r.status_code == 401):
      return ("Unauthorized")
    elif (r.status_code == 404):
      return ("Not Found")
    elif (r.status_code == 403):
      return ("Bad Request")
    return(r.status_code)

Python Script explanation

OK the script starts out much like the capacity check script with nested loops to cover all accounts in a customer entity.

#_______________________________
# Main part of the python Script
#_______________________________
alert_to_slack("Checking Frame Education Customers for unused Volumes\n_________")
# Get a list of Organizations under the Frame customer
orgs=get_FrameAPICall("https://api.console.nutanix.com/api/rest/v1/organizations?" + _cust_id + "=&show_deleted=false")
# Convert the Response to JSON
orgs_json=json.loads(orgs)
# Iterate through each Org
for org in orgs_json :
# Get a list of accounts under a specific organization
accts=get_FrameAPICall("https://api.console.nutanix.com/api/rest/v1/accounts/?organization_id=" + str(org['id']) + "&active=true")
# Convert the Response to JSON
    	accts_json=json.loads(accts)

Now for each account, we need to enumerate the user volumes in that account. However, we only want to see the volumes that are in the “detached” state (because we do not want to try deleting volumes that are in use). We can do this by adding a query filter at the end of the API request URL. We will also need to convert the response to JSON to make further processing easier.

    for acct in accts_json :
        # Get a list of the user volumes under the account with a status of detached
        volumes=get_FrameAPICall("https://api.console.nutanix.com/api/rest/v1/accounts/" + str(acct['id']) + "/user_volumes/?statuses=detached")
        # Convert the Response to JSON
        volumes_json=json.loads(volumes)

Now we iterate through the volumes and the first thing we do is see if the volume is on the exception list. If it is, we set skip to true.

        for vol in volumes_json['user_volumes'] :
            # Initialize the variable that will be set to true if this is a volume in the exceptions list
            skip=False
            # Loop through the exceptions list and set skip to true if the id of the volume is in the list
            for skipit in _exceptions :
                if (vol['id'] == skipit):
                    print ("Skipping "+ skipit)
                    skip=True

Once we verify it is not on the exception list, we look at the last_used_time and subtract it from the current time. Then we convert that difference to an integer (intdaysbetween) which is easier to be used in future comparisons.

            # If skip was not set to true above get the volume's last used time
            if (skip is not True):
              if (vol['last_used_time'] is not None):
                lastdate=datetime.strptime(vol['last_used_time'], '%Y-%m-%dT%H:%M:%S.%fZ').date()
                # Get the integer number of days since the volume was last used.
                intdaysbetween=(date.today()-lastdate).days

Now it is two more if statements: one to confirm we need to either warn or delete and then a second one to determine which one.

               # check to see if it has been idle more that the minimum number of days.
                if (_idledays < intdaysbetween):
                  # if it has been idle for the minimum number of days are we warning? or deleting
                  if ((_idledays+_warningdays) < intdaysbetween):

If both of these are true, we need to delete the user volume. That involves using the delete_FrameAPIcall function defined before to create a delete task. We can then monitor for that task to be complete and send a message to slack when completed.

                    	# Start the delete task
                   	volid_json={"user_volume_ids" : vol['id']}
deltask=delete_FrameAPICall("https://api.console.nutanix.com/api/rest/v1/accounts/"+ str(acct['id']) +"/user_volumes?",volid_json)
                    	deltask_json=json.loads(deltask)
                    	# Monitor the task ID (checking every five seconds) for completion of the delete task
taskcheck=get_FrameAPICall("https://api.console.nutanix.com/api/rest/v1/accounts/"+ str(acct['id']) +"/task/"+str(deltask_json['id']))
                    	taskcheck_json=json.loads(taskcheck)
                    	while (taskcheck_json['stage']!="done"):
                       	time.sleep(5)
taskcheck=get_FrameAPICall("https://api.console.nutanix.com/api/rest/v1/accounts/"+ str(acct['id']) + "/task/"+str(deltask_json['id']))
                       	taskcheck_json=json.loads(taskcheck)
                    	alert_to_slack("Deleted "+ vol['name'] + " " + vol['type'] +" with id of `" + vol['id']+"`")

If the first is true and the second is false, it is just a warning.

             	else:
                    		# if it is within the warning period, print out the warning
                    		alert_to_slack("User volume "+ vol['name'] + " " + vol['type'] +" will be deleted in " +str(_idledays+_warningdays+1-intdaysbetween)+" days. Place the id `" + vol['id'] + "` in the exceptions list to avoid deletion.")

That is about all there is to it.

Complete Script

Use the dropdown menu below to review and copy the entire python script.

VolumeCheck.py

#!/bin/env python
import hashlib
import hmac
import time
import requests
import base64
import json
from datetime import datetime,date,timedelta
#Parameter content type application/json
## _clnt_id = "<ClientID>"
## _clnt_secret = "<ClientSecret>"
## _cust_id = "<CustomerID>"
## _slack_url = "<Slack Web Hook Url>"
## _idledays = <integer of the number of days since last used to consider the volume for deletion>
## _warnningsdays = <integer number of days to warn an administrator that the volume will be deleted>
## _exceptions = <string array of any volumes that should be excluded from possible deletion 'gateway-prod.1234'>
## Function to send alert to the Frame Education Slack Channel
# msg_text: Markup formated text to send to Slack
def alert_to_slack (msg_text):
    # use the slack_sdk Webhookclient
    from slack_sdk.webhook import WebhookClient
    # Create and send the formated message
    webhook = WebhookClient(_slack_url)
    response = webhook.send(
        text="fallback",
        blocks=[
            {
                "type": "section",
                "text": {
                    "type": "mrkdwn",
                    "text": msg_text
                }
            }
        ]
    )
## Function to use GET to get Frame API information
# api_url: the url of the API formated with the right request information
def get_FrameAPICall (api_url):
    # Create signature
    timestamp = int(time.time())
    to_sign = "%s%s" % (timestamp, _clnt_id)
    signature = hmac.new(_clnt_secret, to_sign.encode('utf-8'), hashlib.sha256).hexdigest()
    # Prepare http request headers
    headers = { "X-Frame-ClientId": _clnt_id, "X-Frame-Timestamp": str(timestamp), "X-Frame-Signature": signature }
    # Make request
    r = requests.get(api_url, headers=headers)
    if (r.status_code == 200) :
      return (r.content)
    elif (r.status_code == 401):
      return ("Unauthorized")
    elif (r.status_code == 404):
      return ("Not Found")
    elif (r.status_code == 403):
      return ("Bad Request")
    return(r.status_code)
## Function to use DELETE to get have the Frame API remove some resource
# api_url: the url of the API formated with the right request information
# jsonbody: the json formatted body of the DELETE request
def delete_FrameAPICall (api_url,jsonbody):
    # Create signature
    timestamp = int(time.time())
    to_sign = "%s%s" % (timestamp, _clnt_id)
    signature = hmac.new(_clnt_secret, to_sign.encode('utf-8'), hashlib.sha256).hexdigest()
    # Prepare http request headers
    headers = { "X-Frame-ClientId": _clnt_id, "X-Frame-Timestamp": str(timestamp), "X-Frame-Signature": signature }
    # Make request
    r = requests.delete(api_url, headers=headers, data = jsonbody)
    if (r.status_code == 200) :
      return (r.content)
    elif (r.status_code == 401):
      return ("Unauthorized")
    elif (r.status_code == 404):
      return ("Not Found")
    elif (r.status_code == 403):
      return ("Bad Request")
    return(r.status_code)
#_______________________________
# Main part of the python Script
#_______________________________
alert_to_slack("Checking Frame Education Customers for unused Volumes\n_________")
# Get a list of Organizations under the Frame customer
orgs=get_FrameAPICall("https://api.console.nutanix.com/api/rest/v1/organizations?" + _cust_id + "=&show_deleted=false")
# Convert the Response to JSON
orgs_json=json.loads(orgs)
# Iterate through each Org
for org in orgs_json :
    # Get a list of accounts under a specific organization
    accts=get_FrameAPICall("https://api.console.nutanix.com/api/rest/v1/accounts/?organization_id=" + str(org['id']) + "&active=true")
    # Convert the Response to JSON
    accts_json=json.loads(accts)
    for acct in accts_json :
        # Get a list of the user volumes under the account with a status of detached
        volumes=get_FrameAPICall("https://api.console.nutanix.com/api/rest/v1/accounts/" + str(acct['id']) + "/user_volumes/?statuses=detached")
        # Convert the Response to JSON
        volumes_json=json.loads(volumes)
        for vol in volumes_json['user_volumes'] :
            # Initialize the variable that will be set to true if this is a volume in the exceptions list
            skip=False
            # Loop through the exceptions list and set skip to true if the id of the volume is in the list
            for skipit in _exceptions :
                if (vol['id'] == skipit):
                    print ("Skipping "+ skipit)
                    skip=True
            # If skip was not set to true above get the volume's last used time
            if (skip is not True):
              if (vol['last_used_time'] is not None):
                lastdate=datetime.strptime(vol['last_used_time'], '%Y-%m-%dT%H:%M:%S.%fZ').date()
                # Get the integer number of days since the volume was last used.
                intdaysbetween=(date.today()-lastdate).days
                # check to see if it has been idle more that the minimum number of days.
                if (_idledays < intdaysbetween):
                  # if it has been idle for the minimum number of days are we warning? or deleting
                  if ((_idledays+_warningdays) < intdaysbetween):
                    # Start the delete task
                    volid_json={"user_volume_ids" : vol['id']}
                    deltask=delete_FrameAPICall("https://api.console.nutanix.com/api/rest/v1/accounts/"+ str(acct['id']) +"/user_volumes?",volid_json)
                    deltask_json=json.loads(deltask)
                    # Monitor the task ID (checking every five seconds) for completion of the delete task
                    taskcheck=get_FrameAPICall("https://api.console.nutanix.com/api/rest/v1/accounts/"+ str(acct['id']) + "/task/"+str(deltask_json['id']))
                    taskcheck_json=json.loads(taskcheck)
                    while (taskcheck_json['stage']!="done"):
                        time.sleep(5)
                        taskcheck=get_FrameAPICall("https://api.console.nutanix.com/api/rest/v1/accounts/"+ str(acct['id']) + "/task/"+str(deltask_json['id']))
                        taskcheck_json=json.loads(taskcheck)
                    alert_to_slack("Deleted "+ vol['name'] + " " + vol['type'] +" with id of `" + vol['id']+"`")
                  else:
                    # if it is within the warning period, print out the warning
                    alert_to_slack("User volume "+ vol['name'] + " " + vol['type'] +" will be deleted in " +str(_idledays+_warningdays+1-intdaysbetween)+" days. Place the id `" + vol['id'] + "` in the exceptions list to avoid deletion.")
                else:
                  # For volumes in active use don't do anything.
                  #print (acct['name'])
                  #print ("\t"+vol['name']+"\t"+vol['type']+"\t"+str(vol['size'])+"GB\n")
alert_to_slack("_________\nCompleted user volume check for Frame Education")

Conclusion

The script is intended to be run daily, but other time frames could work. Of course, you can play around with the variables to meet the needs of your organization. The body of this script could also be easily integrated into my previous script to minimize the number of places you have API credentials. If you do plan to use these scripts in a production environment, I would recommend implementing more error checking/recovery to deal with some of the edge situations where things don’t go as planned. Please also note this script is deleting volumes so user data can be at risk. The script should only be used in environments where important corporate data is being managed via a full lifecycle data retention, backup, and destruction policy.

About the Author

David Horvath

Senior Solutions Architect

User Volumes​

Script Setup​

_idledays​

_warningdays​

_exceptions​

API DELETE requests​

Python Script explanation​

Complete Script​

Conclusion​

About the Author​