Key Recovery

Yesterday I was able to setup a process to backup encryption keys into the cloud. I don’t have access to a key management service (KMS), so I’m storing the keys into Dropbox, automating the process with a Cron job, and interacting with the Dropbox API.

Todays goals are to restore encryption keys from the cloud, as well as moving the configuration settings into the secrets manager and the current state of whats one dropbox. Currently the secrets are stored on the build server, and written to an .htaccess file to be available as environment variables.

Let’s actually think this through. All of my keys have been deleted, lost, or altered. What are the steps to recovery? What do I need access to?

  • Generate a new encryption key
  • Create an encrypted database connection using that key
  • Set SECRET_2FA secret to an empty string
  • Setup Two-Factor Authentication
  • Grab the recovery key from a password manager or other secure location
  • Create/Update secret SECRET_RECOVERY_KEY
  • Create/Update secret SECRET_RECOVERY_API_TOKEN
  • Create/Update secret SECRET_LAST_BACKUP_FILE
  • Create/Update secret SECRET_LAST_RECOVERED_FILE to an empty value
  • Kickoff the import process

We have a vulnerability. For a brief moment, Two-Factor authentication is disabled… actually, we can’t get that far. If the local encryption key is lost/doesn’t work, I can’t set the SECRET_2FA to an empty string. A safeguard is built in to deny access even when the 2FA secret is present, but can not be decrypted.

Rather than going to various endpoints recovering one step at a time, what I’ll need to do is setup a full blown recovery page where you must provide a valid database connection, dropbox api key, and a new 2FA secret. If any fail, then don’t apply any of them as new settings until the database, dropbox, and 2FA secret are all verified. This removes the possibility that 2FA is disabled for a brief moment, and lets me do it all in one go. Even if 2FA is disabled, this process would enable it by default.

I changed how the backup process works by encrypting one key at a time as its own individual file, using the backup encryption key. I then iterated through each key and backed all of them up.

Done…

Encryption Key Restoration Prompt: Microsoft Designer

show me a drawing where many encryption keys on another server in the cloud/internet are downloaded to a webserver during a recovery process. During this time, engineers are working effortlessly around the clock addressing errors and bugs. At the end of the day, the database for the secrets manager comes back online and everyone is happy.

What a night. I had gotten a simple form setup to input all of the necessary data to recover from losing everything.

The form was fairly simple to setup. I worked though all of the steps necessary to verify everything was correct and connections to the database and dropbox were verified before modifying any configuration changes. The brunt of the work wasn’t figuring out how to do it – but ensuring that everything was done correctly.

I initially started out downloading the entire folder as a zip file and extracting all the files. For the purpose of scalability, I decided to download the files one at a time instead. The first file would be downloaded by the restoration process, while a cron job would continue to download the remaining files. I may change it to download five or 10 files at a time to speed up the recovery, but for now I’m keeping it simple.

Recovery PHP
<?php
require_once "../common/Secrets.php";
require_once "../common/Show.php";
require_once '../common/PostedJson.php';
require_once "../common/Database.php";
require_once "../common/Otp.php";
require_once "../common/Dropbox.php";
require_once "../common/replace_env.php";

function main()
{
    $posted = new PostedJson(2);

    if (!$posted->keysExist(
        'db_hostname',
        'db_username',
        'db_password',
        'db_database',
        'recovery_api_token',
        'recovery_key',
        'otp_secret',
        'otp'
    )) {
        Show::error($posted->lastError(), $posted->lastErrorCode());
        exit;
    }

    $key_dir = Secrets::key_dir();
    if (empty($key_dir) || !is_dir($key_dir)) {
        Show::error("Encryption directory unknown: $key_dir");
        exit;
    }

    $db_hostname = $posted->getValue('db_hostname');
    $db_username = $posted->getValue('db_username');
    $db_password = $posted->getValue('db_password');
    $db_database = $posted->getValue('db_database');

    $recovery_api_token = $posted->getValue('recovery_api_token');
    $recovery_key = $posted->getValue('recovery_key');
    if (empty($recovery_key)) {
        Show::error("Recovery key is empty");
        exit;
    }

    $recovery_key_bin = base64_decode($recovery_key);
    if ($recovery_key_bin === false) {
        Show::error("Recovery key was not base64 encoded");
        exit;
    }

    if (strlen($recovery_key_bin) !== 32) {
        Show::error("Recovery key was not 256-bit (Got " . (strlen($recovery_key_bin) * 8) . ")");
        exit;
    }

    $otp_secret = $posted->getValue('otp_secret');
    $otp = $posted->getValue('otp');

    // OTP
    $otpAuth = new Otp($otp_secret);
    if (!(
        $otp === $otpAuth->otp() ||
        $otp === $otpAuth->get_relative_otp(-1) ||
        $otp === $otpAuth->get_relative_otp(1)
    )) {
        Show::error("OTP Failed.");
        exit;
    }

    // Database
    $db_credentials = [
        'hostname' => $db_hostname,
        'username' => $db_username,
        'password' => $db_password,
        'database' => $db_database,
    ];

    try {
        $db = new Database($db_credentials);
    } catch (Exception $e) {
        Show::error($e->getMessage());
        exit;
    }

    // Dropbox

    $server = $_SERVER['SERVER_NAME'];
    $dropbox = new Dropbox($recovery_api_token);
    $result = $dropbox->list_first_file("/$server/keys");
    $first = $result['entries'][0];
    $first_name = $first['name'];
    $first_full_path = $first['path_lower'];
    $cursor = $result['cursor'];
    $has_more = $result['has_more'];
    $first_encrypted = $dropbox->download_as_binary($first_full_path);
    $key_dir = Secrets::key_dir();
    $first_decrypted = Secrets::decrypt($first_encrypted, $recovery_key_bin);

    // Everything has been verified - database, Dropbox, recovery key, decryption and OTP

    // Now start saving

    // decrypted key

    $name = preg_replace('/\.enc$/', '', $first_name);
    file_put_contents($key_dir . DIRECTORY_SEPARATOR . $name, $first_decrypted);

    // New Key
    $new_key_path = Secrets::generateKey();
    replace_env(Secrets::key_path_key(), $new_key_path);
    Secrets::change_key($new_key_path);

    // Database
    $json = json_encode($db_credentials, JSON_PRETTY_PRINT);
    $encrypted = Secrets::encryptValue($json);
    replace_env(Secrets::database_key(), $encrypted);
    Secrets::change_db($db_credentials);

    // OTP
    Secrets::keep(Secrets::otp_key(), $otp_secret);
    $secret_copy = Secrets::reveal(Secrets::otp_key());
    if ($otp_secret !== $secret_copy) {
        Show::error("Unable to store/retrieve secrets");
        exit;
    }

    // Dropbox
    Secrets::keep("SECRETS_RECOVERY_API_TOKEN", $recovery_api_token);
    Secrets::keep("SECRETS_RECOVERY_KEY", $recovery_key);
    // Assume last backup file is out of sync and start over
    // Secrets::keep("SECRETS_LAST_BACKUP_FILE", "");

    if ($has_more) {
        Secrets::keep("SECRETS_RECOVERY_CURSOR", $cursor);
    } else {
        Secrets::keep("SECRETS_RECOVERY_CURSOR", "");
    }
    Show::message("Recovery started.");
    exit;
}
function decrypt_latest_key(
    string $encrypted,
    #[SensitiveParameter] $recovery_key
) {

    $recovery_bin = base64_decode($recovery_key);
    if ($recovery_bin === false) {
        throw new Exception("Recovery key not base64 formatted");
    }
    $encrypted_bin = base64_decode($encrypted);

    $decrypted = Secrets::decrypt($encrypted, $encrypted_bin);
    if ($decrypted === false) {
        throw new Exception("Unable to decrypt");
    }
    return $decrypted;
}
try {
    if ($_SERVER['REQUEST_METHOD'] === 'POST') {
        main();
        exit;
    } else {
        $status = 'Failed';
        if (Secrets::db_verified()) {
            $status = 'Verified';
        }
        ;
    }
} catch (Exception $e) {
    Show::error("An unexpected error has occurred. " . $e->getMessage());
    exit;
}
$otp_secret = Otp::generate_secret();

?>
<fieldset>
  <legend>Database</legend>
  <label for="hostname">
    Hostname: <input id="db_hostname" value="localhost">
  </label><br>
  <label for="username">
    Username: <input id="db_username">
  </label><br>
  <label for="password">
    Password: <input id="db_password" type="password">
  </label><br>
  <label for="database">
    Database: <input id="db_database">
  </lable><br>
</fieldset>
<fieldset>
  <legend>Key Backup</legend>
  <label for="rocovery_api_token">
    API Token: <input id="recovery_api_token">
  </label><br>
  <label for="recovery_key">
    Recovery Key: <input id="recovery_key">
  </label>
</fieldset>
<fieldset>
  <legend>Two-Factor Authentication</legend>
  <label for="otp_secret">
    OTP Secret: <?php echo $otp_secret ?>
    <input id="otp_secret" type="hidden" value="<?php echo $otp_secret ?>">
  </label><br>
  <label for="otp_secret">
    One-time Password:
    <input id="otp">
  </label><br>
</fieldset>
<button id="submit">Recover</button><br />
<hr>
<textarea id="result" cols="60" rows="10"></textarea>
<script>
document.getElementById('submit').addEventListener('click', async () => {
  const db_hostname = document.getElementById('db_hostname').value;
  const db_username = document.getElementById('db_username').value;
  const db_password = document.getElementById('db_password').value;
  const db_database = document.getElementById('db_database').value;
  const recovery_api_token = document.getElementById('recovery_api_token').value;
  const recovery_key = document.getElementById('recovery_key').value;
  const otp_secret = document.getElementById('otp_secret').value;
  const otp = document.getElementById('otp').value;
  try {
    document.getElementById('result').value = 'Fetching...';

      const response = await fetch(window.location.href, {
          method: 'POST',
          headers: {
              'Content-Type': 'application/json'
          },
          body: JSON.stringify({
            db_hostname,
            db_username,
            db_password,
            db_database,
            recovery_api_token,
            recovery_key,
            otp_secret,
            otp
        })
      });

      const json = await response.json();
      document.getElementById('result').value = JSON.stringify(json, null, 2);
  } catch (error) {
      document.getElementById('result').value = 'Error: ' + error;
  }
  });
</script>

Disaster Recovery (DR) Testing

All of my keys were previously backed up to dropbox yesterday. I decided to actually simulate a loss of the keys by moving all keys to another folder. Holy smokes. My eyes were opened to how much trouble I would have been in if I had never tested the recovery process.

One of the things I discovered is that the Secrets manager was throwing up errors here and there because the configured encryption key was nowhere to be found on the file system. I reworked the code so that the manager would still work without the presence of the key, and could still tell me what folder the key should be in so that generating a new key works.

Without the configured key, encrypting the database connection was giving me problems as well. The quick fix was to pass in the key to encrypt the connection with. However, I needed to encrypt plenty of stuff, so I created a new method to change_key. From that point on, the secrets manager would use the overridden key.

Once the database connection was encrypted, I also had to tell the secrets manager to change_db since it was trying to use the configured database credentials when the php script first loaded.

From this point on, I could overwrite all of the secret configurations needed to work with dropbox, two-factor authentication, and setup the backup key.

After a few hours, I had a successful response. The next stop was to import my keys. I took a look at the backup-keys endpoint and renamed it to transfer-keys since it was changing to move keys both to and from dropbox. I set it up so that if the site was currently in a recovery process, it would download the next key from dropbox. Otherwise it would upload any new keys that were recently generated.

Transfer-Keys PHP
<?php
require_once "../common/Secrets.php";
require_once "../common/Show.php";
require_once "../common/Dropbox.php";
require_once '../common/PostedJson.php';
require_once '../common/HTTP_STATUS.php';

function get_next_file_to_backup(string $key_dir, string $last_file)
{
    $files = scandir($key_dir);
    if ($files === false) {
        Show::error("Failed to read files");
        exit;
    }
    if ($last_file === end($files)) {
        return false;
    }
    if (count($files) <= 2) {
        Show::error("Empty");
        exit;
    }
    $files = array_values(array_filter($files, function ($file) use ($key_dir) {
        if (is_dir($key_dir . DIRECTORY_SEPARATOR . $file)) {
            return false;
        }
        return preg_match('/^\d{10}_/', $file) === 1;
    }));
    if (empty($files)) {
        Show::error("Encryption keys missing");
        exit;
    }

    if ($last_file === false) {
        return $files[0];
    }
    if ($last_file === end($files)) {
        return false;
    }
    $index = array_search($last_file, $files);
    if ($index === false) {
        return $files[0];
    }
    return $files[$index + 1];
}
function encrypt_file(string $path, string $file, string $recovery_key)
{
    $encryption_key_path = getenv(Secrets::key_path_key());
    $encryption_key = basename($encryption_key_path);
    $contents = file_get_contents($path . DIRECTORY_SEPARATOR . $file);
    return Secrets::encrypt($contents, $recovery_key);
}
function backup_keys(
    string $key_dir,
    #[SensitiveParameter] string $recovery_key,
    #[SensitiveParameter] string $api_token
) {
    $last_file = Secrets::reveal("SECRETS_LAST_BACKUP_FILE");
    $next_file = get_next_file_to_backup($key_dir, $last_file);

    if (!$next_file) {
        Show::message("Nothing to transfer");
        exit;
    }

    $encrypted = encrypt_file($key_dir, $next_file, $recovery_key);

    $server = $_SERVER['SERVER_NAME'];
    $remote_path = "/$server/keys/$next_file.enc";
    $dropbox = new Dropbox($api_token);
    $dropbox->upload_contents($encrypted, $remote_path);

    Secrets::keep("SECRETS_LAST_BACKUP_FILE", $next_file);

    Show::message("File sent $next_file");
}
function restore_keys(
    string $key_dir,
    #[SensitiveParameter] string $recovery_key_bin,
    #[SensitiveParameter] string $api_token,
    string $recovery_cursor
) {
    $dropbox = new Dropbox($api_token);
    $result = $dropbox->list_next($recovery_cursor);
    if ($result === false) {
        Show::error("Unable to get the next file");
        return;
    }

    $has_more = $result['has_more'];
    if (!$has_more) {
        Secrets::keep("SECRETS_RECOVERY_CURSOR", "");
    } else {
        $cursor = $result['cursor'];
        Secrets::keep("SECRETS_RECOVERY_CURSOR", $cursor);
    }

    $entries = $result['entries'];
    $count = count($entries);
    if ($count === 0) {
        Show::error("No files returned.");
        exit;
    }

    $files = [];

    for ($i = 0; $i < $count; $i++) {
        $name = $entries[$i]['name'];
        $result = import_key(
            $dropbox,
            $entries[$i],
            $recovery_key_bin,
            $key_dir
        );
        if ($result !== true) {
            $files[$name] = $result;
        } else {
            $files[$name] = "Restored";
        }
    }

    Show::data($files);
    exit;
}
function import_key(
    object $dropbox,
    array $entry,
    #[SensitiveParameter] string $recovery_key_bin,
    string $key_dir
) {
    $name = $entry['name'];
    $path_lower = $entry['path_lower'];
    $encrypted = $dropbox->download_as_binary($path_lower);
    if ($encrypted === false) {
        return "Failed to retrieve file '$name'";
    }
    $key_dir = Secrets::key_dir();
    try {
        $decrypted = Secrets::decrypt($encrypted, $recovery_key_bin);
    } catch (Exception $e) {
        return "Failed to decrypt '$name'. " . $e->getMessage();
    }
    if ($decrypted === false) {
        return "Failed to decrypt file '$name'";
    }

    $name = preg_replace('/\.enc$/', '', $name);
    $bytes_written = file_put_contents($key_dir . DIRECTORY_SEPARATOR . $name, $decrypted);
    if ($bytes_written === false) {
        $error === error_get_last();
        if ($error !== null) {
            return "Failed to write file '$name' Error: $error->message";
        } else {
            return "Failed to write file '$name'";
        }
    }
    return true;
}
try {
    $key_dir = Secrets::key_dir();
    if (empty($key_dir) || !is_dir($key_dir)) {
        Show::error("Encryption not configured");
        exit;
    }
    $recovery_key = Secrets::reveal("SECRETS_RECOVERY_KEY");
    if ($recovery_key === false) {
        $errors = Secrets::get_errors();
        if ($errors === '') {
            Show::error("Recovery key false");
        } else {
            Show::error(['reason' => "Recovery key is falss", 'errors' => $errors]);
        }
        exit;
    }
    if (empty($recovery_key)) {
        Show::error("Recovery key empty");
        exit;
    }
    $recovery_key_bin = base64_decode($recovery_key);
    if ($recovery_key_bin === false) {
        Show::error("Recovery key not formatted correctly");
        exit;
    }
    if (strlen($recovery_key_bin) !== 32) {
        Show::error("Incorrect recovery key size");
        exit;
    }

    $api_token = Secrets::reveal("SECRETS_RECOVERY_API_TOKEN");
    if ($api_token === false || empty($api_token)) {
        Show::error("Recovery api token not configured");
        exit;
    }

    $recovery_cursor = Secrets::reveal("SECRETS_RECOVERY_CURSOR");

    if (!empty($recovery_cursor)) {
        restore_keys($key_dir, $recovery_key_bin, $api_token, $recovery_cursor);
    } else {
        backup_keys($key_dir, $recovery_key_bin, $api_token);
    }
} catch (Exception $e) {
    $errors = Secrets::get_errors();
    if (count($errors) !== 0) {
        Show::error([
            'unhandled' => $e->getMessage(),
            'secret_errors' => $errors,
        ]);
    } else {
        Show::error("An unexpected error has occurred. " . $e->getMessage());
    }
    exit;
}

Thus began many hours of troubleshooting small changes here and there. I spent plenty of time trying to figure out why the database credentials couldn’t be read, or why the new secret values were not being decrypted. Although I had changed the key in the secrets manager, I was still saving the configured key with the encrypted data rather than the path to the newly created key. On top of that, my dropbox api token kept expiring every hour or two and threw me in a loop trying to troubleshoot what was going on.

I went through the secrets class, logging reasons why methods were returning false so that they could be retrieved with calls to get_last_error. There were so many that I added in another call to get_errors to return them as an array.

Memcache Deprecation

Looking at the errors, I noticed that my memcache extension was unavailable. I suspect that the configuration of PHP extensions do not transfer when changing servers. I’ll add that to a list of grievances that I have with Hostinger. On a side note, I haven’t experience the FTP errors recently. Once I turned on memcache, I started getting deprecation errors that I hadn’t seen before.

PHP Deprecated: Creation of dynamic property Memcache::$connection is deprecated

I took a look at the line in my code. I just didn’t see what it’s complaining about.

self::$cache = new Memcache;
$result = self::$cache->connect($host, $port);

I’m calling the connect method itself. I’m not assigning $connection. My guess is that Memcache may be doing a dynamic property assignment internally.

Another thing I’ve done is separate the Dropbox calls into their own class. It’s not complete, or pretty, but it gets the job done. Trying to consolidate the requests into one method, I found it difficult since sometimes a JSON object is expected as the posted content, and other times it was expected as a Dropbox-API-Arg header. Various endpoints swap content types between an application/octet-stream or application/json. So far the pattern I found was that the sub domain of “content” or “api” seemed to dictate how to send the content type and arguments. Personally, I’d like a bit of consistency between the two sub domains.

Dropbox PHP Class
<?php
class Dropbox
{
    private string $token;
    public function __construct(
        #[SensitiveParameter] string $token
    ) {
        $this->token = $token;
    }
    public function get_current_account()
    {
        return $this->request("api", "users/get_current_account")['api_result'];
    }
    public function list_first_file($path)
    {
        $args = [
            'path' => $path,
            'recursive' => false,
            'limit' => 1,
            'include_non_downloadable_files' => false,
        ];
        return $this->request("api", "files/list_folder", $args)['api_result'];

    }
    public function list_next($cursor)
    {
        $args = [
            'cursor' => $cursor,
        ];
        return $this->request("api", "files/list_folder/continue", $args)['api_result'];
    }
    public function download_as_binary(string $path)
    {
        $args = [
            'path' => $path,
        ];
        return $this->request("content", "files/download", $args)['response'];
    }
    public function download_folder(string $path, string $target_path)
    {
        $args = ['path' => $path];
        $response = $this->request("content", "files/download_zip", $args);

        if ($response['size'] === 0) {
            throw new Exception("File created without content");
        }
        $temp_zip = "$target_path.zip";
        $zip = new ZipArchive();
        $result = $zip->open("zip://$temp_zip", ZipArchive::CREATE);
        if ($result !== true) {
            switch ($result) {
                case ZipArchive::ER_NOENT:
                    throw new Exception("Zip file not found");
                case ZipArchive::ER_INVAL:
                    throw new Exception("Invalid argument passed to open()");
                case ZipArchive::ER_READ:
                    throw new Exception("Error reading the zip archive.");
                default:
                    throw new Exception("Unknown error while opening zip: $result");
            }
        }
        $result = $zip->extractTo($target_path);
        if (!$result) {
            $status = $zip->getStatus();
            if ($status === ZipArchive::ER_OK) {
                throw new Exception("Extraction succeeded with warnings.");
            } else {
                $status_string = $zip->getStatusString();
                $error = $zip->getLastError();
                throw new Exception("Error during extraction: $status_string " . $error);
            }
        }
        $zip->close();
        unlink($temp_zip);
        return $response['api_result'];
    }
    public function upload_contents(string $contents, string $path)
    {
        $args = [
            'mode' => [".tag" => "overwrite"],
            'path' => $path,
        ];
        $api_result = $this->request("content", "files/upload", $args, $contents)['api_result'];
        if ($api_result['size'] === 0) {
            throw new Exception("File created without content");
        }
        return $api_result;
    }
    private function request(
        string $subdomain,
        string $endpoint,
        ?array $args = null,
        ?string $content = null) {
        $url = "https://$subdomain.dropboxapi.com/2/$endpoint";
        $headers = [
            "Authorization: Bearer $this->token",
        ];
        if ($subdomain === "content") {
            $headers[] = 'Content-Type: application/octet-stream';
        } elseif ($args !== null) {
            $headers[] = 'Content-Type: application/json';
        }
        if ($subdomain === "content") {
            $headers[] = "Dropbox-API-Arg: " . json_encode($args);
        } elseif ($args !== null && $content !== null) {
            $headers[] = "Dropbox-API-Arg: " . json_encode($args);
        }
        $ch = curl_init($url);
        curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
        curl_setopt($ch, CURLOPT_POST, true);
        if ($content !== null) {
            curl_setopt($ch, CURLOPT_POSTFIELDS, $content);
        } else if ($subdomain !== "content" && $args !== null) {
            curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($args));
        }
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
        curl_setopt($ch, CURLOPT_HEADER, true);
        $response = curl_exec($ch);

        // Separate headers
        $header_size = curl_getinfo($ch, CURLINFO_HEADER_SIZE);
        $headers = substr($response, 0, $header_size);
        $response = substr($response, $header_size);

        $error = curl_error($ch);
        $api_result = null;
        foreach (explode('\r\n', $headers) as $header) {
            if (strpos($header, 'Dropbox-Api-Result:') !== false) {
                $json = trim(explode(':', $header)[1]);
                Show::message($json);
                exit;
                $api_result = json_decode($json, true);
                break;
            }
        }
        if ($api_result === null) {
            $api_result = json_decode($response, true);
        }
        curl_close($ch);
        // try {
        $this->process_error($error, $api_result, $response, $url);
        // } catch (Exception $e) {
        //     $api_result = ['error' => $e->getMessage()];
        // }
        return [
            'api_result' => $api_result,
            'response' => $response,
        ];
    }

    private function process_error($error, $api_result, $response, $endpoint)
    {
        if ($error) {
            throw $error;
        }
        if ($api_result === false || $api_result === null) {
            // if (strpos($response, "Error") !== false) {
            //     throw new Exception("Unable to process api result for $endpoint. " . $response);
            // }
            return;
            // throw new Exception("Unable to process api result for $endpoint. Response: " . $response);
        }
        if (isset($api_result['error_summary'])) {
            throw new Exception($api_result['error_summary']);
        }
        if (!isset($api_result['error'])) {
            return;
        }
        if (isset($api_result['error']['message'])) {
            throw new Exception($api_result['error']['message']);
        }
        if (isset($api_result['error']['.tag'])) {
            $tag = $api_result['error']['.tag'];
            if (isset($api_result['error'][$tag]) && isset($api_result['error'][$tag][".tag"])) {
                $sub_tag = $api_result['error'][$tag][".tag"];
            }
            switch ($tag) {
                case "path":
                    if (isset($sub_tag)) {
                        switch ($sub_tag) {
                            case "not_found":
                                throw new Exception("Path not found.");
                            default:
                                throw new Exception("Droptox API exception: Path: $sub_tag");
                        }
                    } else {
                        throw new Exception("Droptox API exception: Path error");
                    }
                case "expired_access_token":
                    throw new Exception("Dropbox API exception: Access token expired");
                default:
                    if (isset($sub_tag)) {
                        throw new Exception("Droptox API exception: $tag: $sub_tag");
                    }
                    throw new Exception("Droptox API exception: $tag");
            }
        }
        if (isset($api_result['error']['reason'])) {
            throw new Exception($api_result['error']['reason']);
        }
        throw new Exception($api_result['error']);
    }

}

I’ve worked through the entire night. Once 6AM rolled around, I was finally in the final stage walking through importing each key. After running to the end of the file list, the next request to the same page started backing up all the keys to Dropbox, and finally, it reached the end with nothing left to do.

I reviewed some of the encrypted secrets in the database and found that “FOO” was referencing one of the keys that were restored. I ran over to the get endpoint and entered in the key. bar?

The recovery process is complete.

What have we done

  • Separated Dropbox API calls into its own class
  • Download a file from Dropbox
  • Download a folder as a zip file from Dropbox
  • Extract a zip file on the file system using ZipArchive
  • Request a list of files in a dropbox folder
  • Request the next batch of files listed in a dropbox folder
  • Allow the secrets manager to connect to a different database
  • Override the default key in the secrets manager
  • Swap between backup and restoration modes with the cron job
  • Added lots of error logging in the secrets manager and methods to get the last, or all errors.
  • Complete the recovery process
  • Access data using encryption keys recovered from dropbox

Discover more from Lewis Moten

Subscribe now to keep reading and get access to the full archive.

Continue reading