With the current setup, I’m storing secrets on the file system as a JSON file, and I’m about to add a slew of keys for encrypting data in a secured database. Getting a good nights rest gave me some time to think through the issues this causes with management of the individual secrets, backup, and restoration. Particularly with the secured database keys, I can’t afford to lose them. It’s a nightmare waiting to happen.
What I need to do is to expose them as a separate service. I can create a new subdomain and database to hold the information. Databases are easy to backup and restore. I could create a web interface to manage it. In turn, I could move the individual subdomain to another host if I needed to distribute the load.
JSON to Database Prompt
Show how an encrypted JSON file is being converted into an encrypted database of secrets, and exposed as a service that applications can all with API endpoints.
The problem here is that we would be sending sensitive information over the wire with some overhead. Caching addresses the overhead, and SSL encryption addresses the problem with prying eyes. Another problem is authorization. We need to ensure that only authorized services can access the secrets.
With a service, I could have separate accounts for the dev-api and api domains so that they can access the same variables with different values. It reduces the number of places that I need to track where I need to manage/backup/restore data. I could even use the service for local development as well.
In all – we will have three databases and services
- api.periplux.io – The primary interface/database
- vault.periplux.io – Keeps sensitive information for the application
- secrets.periplux.io – Keeps sensitive configuration information
Over Engineering
I started to create a service to keep my secrets. I was over engineering it to the point that it was wasting too much time designing the architecture. I had the thought that I could separate things out for local development vs dev and production environments, add in permissions to access individual secrets or manage the account, and generate tokens with the same permissions. It was pretty robust, but too time consuming in the implementation.
- Encrypted login credentials and email
- Encrypted account name and secret key names so you couldn’t tell who’s secrets they were
- Log when a secret is accessed, changed, deleted
- Allow each account to have its own encryption keys
- Hash email and username with pepper values for indexed lookup
- Associate external session id with users
- Allow accounts to have multiple users
- Allow users to have multiple accounts
- Auto lockout for an hour after 5 failed login attempts
- Delete/restore accounts
- Log when encryption keys were last rotated
- Maintain a history of why a user or account was locked
- Allow tokens to have an expiration data and an early revocation date
The whole solution was potentially adding a bit of overhead having to look up authorization, logging access, and looking up various encryption keys.
In a nut shell, I was designing an enterprise system that could be sold as a service to other companies. There are many well known and trusted companies already offering similar services, so the idea of selling the service wouldn’t take off easily.
I had to rethink what it was that I needed.
- Store data as key/value pairs
- Encrypted
- Shared between sub-domains using the same key names
Secrets Simplified
My secrets are already encrypted on the file system as a JSON file. It shouldn’t be difficult to setup a simple database with one table that has a name and a value. These are often referred to as key-value stores or NoSQL database. The JSON file itself is a key-value store. Memcached and Redis are key-value stores. I’ve designed a system like this in the past with MySQL as the back-end.
Let’s change our secrets manager so that it still uses Memcache for caching, but gets the secrets from the database if it’s not already available in memcache.
CREATE TABLE key_value_store (
name VARCHAR(64) PRIMARY KEY,
value BLOB
)
Reducing 17 tables down to one is pretty efficient. The database is oblivious to any encryption. Just like the enterprise version, it’s up to my implementation to perform the encryption outside of the database. We can prefix the values with the initialization vectors and just save as one binary value.
Secrets Normalized
Another day or two has passed. Much has changed. Let’s start with the database. I added two additional fields – scope and encryption key. The scope is the websites FQDN, and the encryption key is just a path to where the key is located, relative to the websites root folder (ie ../../secret/over/here.bin).
Scope allows me to stop hashing the names to prevent collisions and visually see which values I need to delete. Yes… the reason being is that encryption has gotten corrupted quite a number of times during testing. The encryption key path was added so that I no longer had to hunt and play guessing games with the corrupted data. This means the key used to decrypt the database credentials used to connect to the database may be different than the key used to encrypt each individual record – which was usually the case when testing went awry.
With the new fields, I decided to normalize the database into three separate tables to decrease the redundancy. That’s a 200% increase in complexity…
From here, I have four separate procedure calls that hide the underlying data structure to list, set, get, and check if a key name exists.
- sp_secret_get(scope, name) selects the value
- sp_secret_has(scope, name) selects 1 if scope has name; otherwise 0
- sp_secret_list(scope) selects 100 names, sorted
- sp_secret_set(scope, name, encryption key, value)
- inserts scope if it doesn’t exist
- inserts encryption key if it doesn’t exist
- deletes value if value is null
- updates value if name exists
- inserts value if name doesn’t exist
- Selects “Success”; otherwise the failure reason
CALL sp_secret_has('dev-api.perplux.io', 'HELLO');
-- 0
CALL sp_secret_set('dev-api.perplux.io', 'HELLO', '../key.bin', 0x01);
-- Success
CALL sp_secret_list('dev-api.perplux.io');
-- HELLO, 0x01, '../key.bin'
CALL sp_secret_has('dev-api.perplux.io', 'HELLO');
-- 1
CALL sp_secret_get('dev-api.perplux.io', 'HELLO');
-- 0x01, '../key.bin'
CALL sp_secret_set('dev-api.perplux.io', 'HELLO', '../key.bin', null);
-- Success
CALL sp_secret_has('dev-api.perplux.io', 'HELLO');
-- 0
View Normalized Secrets Database Script
DROP PROCEDURE IF EXISTS sp_secret_get;
DROP PROCEDURE IF EXISTS sp_secret_has;
DROP PROCEDURE IF EXISTS sp_secret_list;
DROP PROCEDURE IF EXISTS sp_secret_set;
DROP TABLE IF EXISTS key_value_store;
DROP TABLE IF EXISTS scopes;
DROP TABLE IF EXISTS encryption_keys;
CREATE TABLE encryption_keys (
`id` INT AUTO_INCREMENT PRIMARY KEY,
`path` VARCHAR(1024) UNIQUE
);
CREATE TABLE scopes (
`id` INT AUTO_INCREMENT PRIMARY KEY,
`name` VARCHAR(64) UNIQUE
);
CREATE TABLE key_value_store (
`id` INT AUTO_INCREMENT PRIMARY KEY,
`scope_id` INT,
`name` VARCHAR(64),
`encryption_key_id` INT,
`value` BLOB,
FOREIGN KEY(`scope_id`) REFERENCES `scopes`(`id`),
FOREIGN KEY(`encryption_key_id`) REFERENCES `encryption_keys`(`id`),
index(`scope_id`),
unique(`scope_id`, `name`)
);
DROP PROCEDURE IF EXISTS sp_secret_get;
DELIMITER //
CREATE PROCEDURE sp_secret_get (
IN p_scope VARCHAR(64),
IN p_name VARCHAR(64)
)
this_proc: BEGIN
DECLARE v_error_message TEXT;
DECLARE EXIT HANDLER FOR SQLEXCEPTION
BEGIN
-- SELECT 'SQL Exception' AS `proc_message`;
GET DIAGNOSTICS CONDITION 1 v_error_message = MESSAGE_TEXT;
SELECT v_error_message AS `proc_message`;
END;
IF p_scope IS NULL THEN
SELECT 'Scope can not be null' AS `proc_message`;
END IF;
IF p_name IS NULL THEN
SELECT 'Name can not be null' AS `proc_message`;
END IF;
IF p_scope = '' THEN
SELECT 'Scope can not be empty' AS `proc_message`;
END IF;
IF p_name = '' THEN
SELECT 'Name can not be empty' AS `proc_message`;
END IF;
SELECT
`value`,
ek.`path` AS `encryption_key`
FROM
`key_value_store` AS kvs
INNER JOIN `scopes` AS s ON
kvs.`scope_id` = s.`id`
INNER JOIN `encryption_keys` AS ek ON
kvs.`encryption_key_id` = ek.`id`
WHERE
s.`name` = p_scope
AND kvs.`name` = p_name
LIMIT 1;
END
//
DELIMITER ;
DROP PROCEDURE IF EXISTS sp_secret_has;
DELIMITER //
CREATE PROCEDURE sp_secret_has (
IN p_scope VARCHAR(64),
IN p_name VARCHAR(64)
)
this_proc: BEGIN
DECLARE v_error_message TEXT;
DECLARE EXIT HANDLER FOR SQLEXCEPTION
BEGIN
-- SELECT 'SQL Exception' AS `proc_message`;
GET DIAGNOSTICS CONDITION 1 v_error_message = MESSAGE_TEXT;
SELECT v_error_message AS `proc_message`;
END;
IF EXISTS(
SELECT
1
FROM
`key_value_store` AS kvs
INNER JOIN `scopes` AS s ON
kvs.`scope_id` = s.`id`
WHERE
s.`name` = p_scope
AND kvs.`name` = p_name
LIMIT 1
) THEN
SELECT 1 AS `has_key`;
ELSE
SELECT 0 AS `has_key`;
END IF;
END
//
DELIMITER ;
DROP PROCEDURE IF EXISTS sp_secret_list;
DELIMITER //
CREATE PROCEDURE sp_secret_list (
IN p_scope VARCHAR(64)
)
this_proc: BEGIN
DECLARE v_error_message TEXT;
DECLARE EXIT HANDLER FOR SQLEXCEPTION
BEGIN
-- SELECT 'SQL Exception' AS `proc_message`;
GET DIAGNOSTICS CONDITION 1 v_error_message = MESSAGE_TEXT;
SELECT v_error_message AS `proc_message`;
END;
SELECT
kvs.`name`,
kvs.`value`,
ek.`path` AS encryption_key
FROM
`key_value_store` AS kvs
INNER JOIN `scopes` AS s ON
kvs.`scope_id` = s.`id`
INNER JOIN `encryption_keys` AS ek ON
kvs.`encryption_key_id` = ek.`id`
WHERE
s.`name` = p_scope
ORDER BY
kvs.`name`
LIMIT 100;
END
//
DELIMITER ;
DROP PROCEDURE IF EXISTS sp_secret_set;
DELIMITER //
CREATE PROCEDURE sp_secret_set (
IN p_scope VARCHAR(64),
IN p_name VARCHAR(64),
IN p_encryption_key VARCHAR(1024),
IN p_value BLOB
)
this_proc: BEGIN
DECLARE v_error_message TEXT;
DECLARE v_affected_rows INT;
DECLARE v_encryption_key_id INT;
DECLARE v_scope_id INT;
DECLARE EXIT HANDLER FOR SQLEXCEPTION
BEGIN
-- SELECT 'SQL Exception' AS `proc_message`;
GET DIAGNOSTICS CONDITION 1 v_error_message = MESSAGE_TEXT;
SELECT v_error_message AS `proc_message`;
END;
SELECT `id` INTO v_scope_id
FROM `scopes`
WHERE `name` = p_scope
LIMIT 1;
IF p_value IS NULL THEN
DELETE FROM `key_value_store`
WHERE
`name` = p_name
AND `scope_id` = v_scope_id
LIMIT 1;
IF ROW_COUNT() = 1 THEN
SELECT 'Success' AS `proc_message`;
ELSE
SELECT 'Not found' AS `proc_message`;
END IF;
LEAVE this_proc;
END IF;
SELECT `id` INTO v_encryption_key_id
FROM `encryption_keys`
WHERE `path` = p_encryption_key
LIMIT 1;
IF v_encryption_key_id IS NULL THEN
INSERT INTO `encryption_keys` (`path`) VALUES (p_encryption_key);
SET v_encryption_key_id = LAST_INSERT_ID();
END IF;
IF v_encryption_key_id IS NULL THEN
SELECT 'Failed creating encryption key' AS `proc_message`;
LEAVE this_proc;
END IF;
IF EXISTS(
SELECT 1 FROM `key_value_store`
WHERE `scope_id` = v_scope_id AND `name` = p_name
LIMIT 1
) THEN
UPDATE `key_value_store` AS kvs
SET
`encryption_key_id` = v_encryption_key_id,
`value` = p_value
WHERE
kvs.`scope_id` = v_scope_id
AND kvs.`name` = p_name
LIMIT 1;
SET v_affected_rows = ROW_COUNT();
IF v_affected_rows = 1 THEN
SELECT 'Success' AS `proc_message`;
ELSE
SELECT CONCAT('Nothing changed') AS `proc_message`;
END IF;
LEAVE this_proc;
END IF;
IF v_scope_id IS NULL THEN
INSERT INTO `scopes` (`name`) VALUES (p_scope);
SET v_scope_id = LAST_INSERT_ID();
END IF;
IF v_scope_id IS NULL THEN
SELECT 'Failed creating scope' AS `proc_message`;
LEAVE this_proc;
END IF;
INSERT INTO `key_value_store` (
`scope_id`,
`name`,
`encryption_key_id`,
`value`
) VALUES (
v_scope_id,
p_name,
v_encryption_key_id,
p_value
);
IF ROW_COUNT() = 1 THEN
SELECT 'Success' AS `proc_message`;
ELSE
SELECT 'Failed' AS `proc_message`;
END IF;
END
//
DELIMITER ;
More work needs to be done in the database… but later.
- List total number of names for the given scope
- Paginate past the first 100 keys
- Dates if needed: upated_at, rotated_at, created_at
Secrets API
Ok… we’ve got all of that. Now what’s going on with the actual server? Glad you asked. Throughout my testing, I setup multiple endpoints and a simple web page to test each one. Things have changed a few things, but they have stabilized.
- /status – tells me if the various parts of the secrets (encryption, database, cache, 2FA) are working, and why they are failing (without throwing exceptions).
- /encryption {otp?} – creates a new encryption key. Updates the environment variables for the current key path, and updates the encrypted database credentials so that it can be decrypted with the new key
- /database {hostname, username, password, database, otp?} – Tests database connection with provided credentials and updates the environment variables
- /pepper {otp?} – Creates a new random value to be hashed with the name to store on the cache server (memcache)
- /oauth {secret, new_otp, otp?} – Sets up Two-Factor Authentication (Time Based OTP)
- /rotate {otp?} – updates all encrypted values in the current scope with the current encryption key and clears the cache server. If the key hasn’t changed, the initialization vector still changes for each encrypted value.
- /get {name, otp?} – decrypts a value for the given name
- /set {name, value, otp?}- updates/deletes a value for the given name
With all of this, most endpoints are protected once Two-Factor Authentication is setup. You can’t modify anything without providing an OTP token. You can still view the status, which is where I went to confirm what part of the system was failing, and why.
The testing interface is made of very simple tools.
What does bad decryption look like? One failure is when only the key was updated, but not the database credentials.
{
"encryption": "Ready",
"database": "Unhandled exception. Exception: Unable to decrypt: error:1C800064:Provider routines::bad decrypt in \/home\/u1\/domains\/periplux.io\/public_html\/dev-api\/common\/Secrets.php:684\nStack trace:\n#0 \/home\/u1\/domains\/periplux.io\/public_html\/dev-api\/common\/Secrets.php(498): Secrets::decrypt()\n#1 \/home\/u1\/domains\/periplux.io\/public_html\/dev-api\/secrets\/status.php(59): Secrets::decryptValue()\n#2 {main}",
"pepper": "Ready",
"cache": "Ready",
"oauth": "Depends on database"
}
I’ve got to clean up the excessive logging, but it’s on specifically to troubleshoot this issue. I’ve tried detecting a few things before I actually try to decrypt.
Status Checklist
- encryption
- key value missing
- key value empty
- key file does not exist
- key file is a directory
- error loading key
- file contents are empty
- key is not 32 bytes
- database
- credentials missing
- credentials empty
- not base64 encoded
- zero bytes
- initialization vector missing
- initialization vector too small
- only contains initialization vector (no encrypted data)
- corrupted encryption, byte length not divisible by key length
- decryption failed
- json parsing failed
- missing hostname, username, database, password
- empty hostname, username, database, password
- connection failed
- pepper
- pepper missing
- pepper empty
- not base64 encoded
- zero bytes
- not 256 bits
- cache
- disabled
- memcache extension not loaded
- host/port configuration is empty
- port is not an integer
- port is less than zero
- failed to instantiate
- failed to connect
- unable to get statistics
- create value
- get value
- update value
- get updated value
- delete value
- attempt to get deleted value
- pepper dependency
- oauth
- depends on encryption
- depends on database
- disabled (not in secrets)
- perform same key checks as encryption
- perform same encrypted value checks as database
- decryption failed
- not base32 encoded
- less than 128 bit
With all of the checks on the encryption key and encrypted data, I then pass the encrypted data over to openssl_decrypt and … boom! “bad decrypt”. I would have expected the output to be binary gibberish. Somehow OpenSSL is able to detect that the encrypted contents is not valid when using the wrong key.
PKCS-PAD
From some quick research, it appears that AES encryption uses a padding scheme (PKCS-PAD) which can be used to detect if the data was tampered with from its original encrypted state.
Looking at the specification, it appears that it pads the last block of binary data to take up the full encryption block. If your encryption block is 16 bytes, but you only have enough data to take up 16 bytes, then the encryption algorithm will pad the remaining bytes with the byte value of 0x10 (16 in decimal) to not only populate the entire block, but also to give it a validation check.
| Text | H | e | l | l | o | ! | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Decimal | 72 | 101 | 108 | 108 | 111 | 33 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 |
| Hex | 48 | 65 | 6C | 6C | 6F | 21 | 0A | 0A | 0A | 0A | 0A | 0A | 0A | 0A | 0A | 0A |
When decrypting the last block of data, the decryption algorithm looks at the last byte to determine how much padding is present. In php, I’m using AES-256 which uses 32 byte blocks. The last byte will be between 0x01 and 0x20 (1 to 32 in decimal). The validation comes in where it confirms that “N” number of bytes are the same value “N”. (Two bytes of 0x02, seven bytes of 0x07, or sixteen bytes of 0x10).
Prompt to demonstrate PKCS Padding
Demonstrate how a block of text is padded at the end to fill an encryption block. If the text only occupies half of the block, the remaining block is padded with the same number, indicating how many bytes that the text does not occupy.
The other day I wast testing AES-128 encryption within MySQL (Read: Secure Database / Vault) and noticed that when encrypting text with exactly 32 characters, it was adding an extra 16 character block of encryption. Without an understanding of the inner workings of AES, I had assumed it was encoding a null string terminator. Well… it was, but it wasn’t. It was sort-of adding its own null string terminator. Not in the way that I had imagined it. I had assumed the last block encrypted 0x00 as the null string terminator, and probably padded the rest of the block with 0x00 (null) values. It turns out that it was adding a whole block of 0x10 (16 in decimal) stating that the string ended at the end of the prior block.
What happens if I remove the last block of data? The previous block wasn’t setup with padding. If the last decrypted byte just so happened to be 0x01 (1 in 255 chance), then I suspect you would get all of your data except the last byte. If the last two values are 0x02, 0x02 (65,536 chance), then you would get everything except the last two bytes. Binary data has better odds than text when it comes to passing the validation check. With ASCII text, the first printable character is 0x20 (32 in decimal) which is a space. If you had 32 spaces, then the validation would pass and cut off those last 32 spaces. All other printable characters are out of range.
Decryption Failures
Besides performing the decryption by hand, there is no other way for me to detect this padding failure prior to passing the encrypted value to OpenSSL. Actually, since I control the data prior to going in, I could prefix the encrypted value with a hash of the encryption key, initialization vector, and encrypted data. That would be similar to a digital signature. If the hash doesn’t match, then I wouldn’t bother sending it to OpenSSL for decryption.
But is it really necessary? No. I already know that decryption is failing. My confusion was over the “why” it was failing, and if I could have validated something on my end prior to decryption. OpenSSL is fairly cryptic about the “why” part.
Unable to decrypt: error:1C800064:Provider routines::bad decrypt
I think I can safely assume that the padding validation in the last block didn’t match.
Managing Secrets
With all of this talk, how has the Secrets class itself changed? (Read Keeping Secrets). Rather than storing encrypted secrets on the file system in the same folder as the keys (not a good practice), the secrets are now stored in a separate database. There are three databases in all
- Secrets (Configuration)
- Vault (Credentials, PII, Financial, and Sensitive information)
- Primary Application Database.
If you prefer, there is actually a fourth database, memcache, used for caching.
Other things that have changed.
- No longer storing keys as PEM files. It’s not standard for symmetric keys and not using anything that makes use of the format.
- Lots of validation checking.
- Cache server is configured via environment variables.
- Each encrypted value stores the path to its encryption key
- Forgetting to update secrets in the build server doesn’t bring the website down after deployment
Secrets.php Class
<?php
require_once "Show.php";
require_once "DatabaseHelper.php";
// NOTE: This documentation hasn't been updated and is out of sync.
// Secrets::reveal(name) - unencrypted secret if exists; otherwise false
// Secrets::rotate() - changes to most recent encryption key and a new initialization vector
// Secrets::import() - imports new values from .secrets.json
// Secrets::encryptValue(value) - encrypts the value and returns as base64 encoded
// Secrets::keep(name, value) - stores encrypted secret
// Secrets::generateKey() - creates a new key in the same folder as the current key, prefixed with a unix timestamp
// Env Dependencies
// SECRETS_KEY_PATH: folder of key
// SECRETS_DATABASE = encrypted json of database credentials base64 encoded {hostname, database, username, password}
// ENVIRONMENT = Errors are obscure if PRODUCTION
// Stores decrypted values in memcache if available to perist between requests.
// Stores decrypted values in class vars for same in-process requests
final class Secrets
{
private static $BIT_LENGTH = 256;
private static $method = "aes-256-cbc";
private static $keyFile = "aes-256-key.bin";
private static $dataFile = 'data.json';
private static $secretCache = null;
private static $keyCache = null;
private static $decryptedCache = [];
private static $timestamp;
private static $use_cache = true;
private static $cache = null;
private static $db = null;
private static $pepperCache = null;
public static function bit_length()
{
return self::$BIT_LENGTH;
}
private function __construct()
{
}
public static function key_path_key()
{
return "SECRETS_KEY_PATH";
}
public static function database_key()
{
return "SECRETS_DATABASE";
}
public static function pepper_key()
{
return "SECRETS_CACHE_PEPPER";
}
public static function cache_host_key()
{
return "SECRETS_CACHE_HOST";
}
public static function cache_port_key()
{
return "SECRETS_CACHE_PORT";
}
public static function oauth_key()
{
return "SECRET_2FA";
}
public static function pepper()
{
if (self::$pepperCache) {
return self::$pepperCache;
}
$encoded = getenv(self::pepper_key());
if (empty($encoded)) {
self::$pepperCache = false;
return false;
}
$decoded = base64_decode($encoded);
if ($decoded === false) {
self::$pepperCache = false;
return false;
}
if (strlen($decoded) !== 32) {
return false;
}
return $decoded;
}
public static function pepper_changed()
{
self::reset_cache();
}
public static function pepper_exists()
{
return !empty(self::pepper());
}
private static function is_production()
{
return getenv("ENVIRONMENT") === "PRODUCTION";
}
private static function scope()
{
return $_SERVER['SERVER_NAME'];
}
public static function cache_enabled()
{
return self::$use_cache;
}
private static function cache_available()
{
$available = self::cache_enabled() &&
extension_loaded('memcache');
if (!$available) {
return false;
}
if (self::pepper() === false) {
return false;
}
$host = getenv(self::cache_host_key());
$port = getenv(self::cache_port_key());
if (empty($host) || empty($port)) {
return false;
}
if (!settype($port, 'integer')) {
return false;
}
if ($port < 0) {
return false;
}
self::$cache = new Memcache;
$result = self::$cache->connect($host, $port);
if ($result === false) {
throw new Exception("Memcache failed");
}
return true;
}
public static function valid_name(string $name)
{
if (empty($name)) {
return false;
}
if (strlen($name) > 64) {
return false;
}
// AA, A9, A_A, A-A, A.A = good
// A, 9A, A_, A-, A. = bad
$pattern = '/^[a-zA-Z][a-zA-Z0-9_.-]*[a-zA-Z0-9]$/';
if (!preg_match($pattern, $name)) {
return false;
}
return true;
}
public static function hashed_name(string $name, ?string $pepper = null)
{
if (!self::valid_name($name)) {
return false;
}
if (empty($pepper)) {
$pepper = self::pepper();
if ($pepper === false) {
return false;
}
}
// apcu - all php websites on same host server can see keys/values
// memcache - all websites have same access, but can't see keys/values
return hash('sha256', self::scope() . $pepper . $name);
}
private static function get_cache(string $name)
{
if (!self::cache_available()) {
return false;
}
$hashed_name = self::hashed_name($name);
if ($hashed_name === false) {
return false;
}
$encrypted = self::$cache->get($hashed_name);
if ($encrypted === false) {
return false;
}
return self::decryptValue($encrypted);
}
private static function set_cache(string $name, #[SensitiveParameter] ?string $value)
{
if (!self::cache_available()) {
return false;
}
$hashed_name = self::hashed_name($name);
if ($hashed_name === false) {
return false;
}
if ($value === null) {
$success = self::$cache->set($hashed_name, null);
return $success;
}
$encrypted = self::encryptValue($value);
if (!$encrypted) {
return false;
}
$success = self::$cache->set($hashed_name, $encrypted);
return $success;
}
private static function db()
{
$name = self::database_key();
if (self::$db !== null) {
return self::$db;
}
$credentials = null;
if (array_key_exists($name, self::$decryptedCache)) {
$credentials = self::$decryptedCache[$name];
} else {
$value = getenv($name);
if (empty($value)) {
return false;
}
$key = self::loadKey();
if ($key === false) {
return false;
}
try {
$decrypted = self::decrypt($value, $key);
if ($decrypted === false) {
return false;
}
} catch (Exception) {
return false;
}
$credentials = json_decode($decrypted, true);
if ($credentials === false || $credentials === null) {
return false;
}
self::$decryptedCache[$name] = $credentials;
}
try {
self::$db = new DatabaseHelper($credentials);
} catch (Exception) {
return false;
}
return self::$db;
}
public static function db_verified()
{
return self::db() !== false;
}
private static function get_db(string $name)
{
if (!self::valid_name($name)) {
return false;
}
$db = self::db();
if ($db === false) {
return false;
}
$rows = $db->selectRows(
"CALL sp_secret_get(?, ?)",
"ss",
self::scope(),
$name
);
if ($rows === false || count($rows) === 0) {
return false;
}
$row = $rows[0];
if (in_array('proc_message', $row)) {
return false;
}
$encryption_key = $row['encryption_key'];
$encrypted = $row['value'];
if ($encrypted === null) {
return false;
}
$key = self::loadKeyFromPath($encryption_key);
if ($key === false) {
return false;
}
return self::decrypt($encrypted, $key);
}
public static function has_key(string $name)
{
if (!self::valid_name($name)) {
return false;
}
if (array_key_exists($name, self::$decryptedCache)) {
return true;
}
if (self::get_cache($name) !== false) {
return true;
}
$db = self::db();
if ($db === false) {
return false;
}
$result = $db->selectScalar(
"CALL sp_secret_has(?, ?)",
"ss",
self::scope(),
$name
);
return $result === 1;
}
public static function trace(string $name)
{
if (!self::valid_name($name)) {
return "Name is not valid. Limited to alphanumeric and _.-";
}
if (!self::has_key($name)) {
return "Not found";
}
$location = "Not found";
$value;
if (array_key_exists($name, self::$decryptedCache)) {
$value = self::$decryptedCache[$value];
$location = "In-process cache";
} else if (($value = self::get_cache($name)) !== false) {
$location = "Cache server";
} else {
$location = "Database";
$db = self::db();
if ($db === false) {
return false;
}
$rows = $db->selectRows(
"CALL sp_secret_get(?, ?)",
"ss",
self::scope(),
$name
);
if ($rows === false) {
return "Query exception";
}
if (count($rows) === 0) {
return "Database returned no matches";
}
$encryption_key = $rows[0]['encryption_key'];
$value = $rows[0]['value'];
if (empty($value)) {
return "Value is empty";
}
$root = $_SERVER['DOCUMENT_ROOT'];
$path = $root . DIRECTORY_SEPARATOR . $encryption_key;
if (!file_exists($path)) {
return "Encryption key file not found";
}
if (is_dir($path)) {
return "Encryption key file is actually a directory";
}
$key = file_get_contents($path);
if ($key === false) {
return "Unable to read key file";
}
if (strlen($key) * 8 !== self::$BIT_LENGTH) {
return "Key file has wrong bit length";
}
$iv_length = self::iv_length();
$encryptedWithIv = base64_decode($value);
if ($encryptedWithIv === false) {
return "Value is not properly base64 encoded";
}
$iv = substr($encryptedWithIv, 0, $iv_length);
if (strlen($iv) !== $iv_length) {
return "Incorrect IV length.";
}
$encrypted = substr($encryptedWithIv, $iv_length);
$encryptedByteCount = strlen($encrypted);
if ($encryptedByteCount === 0) {
return "Encrypted value missing.";
}
if ($encryptedByteCount < $iv_length) {
return "Encrypted value corrupted. Too short.";
}
if ($encryptedByteCount % $iv_length !== 0) {
return "Encrypted value corrupted. Incorrect block size.";
}
$decrypted = openssl_decrypt($encrypted, self::$method, $key, OPENSSL_RAW_DATA, $iv);
if ($decrypted === false) {
return "Decryption error: " . openssl_error_string();
}
}
return "Retrieved from $location";
}
public static function rotate()
{
$names = self::names();
if ($names === false) {
return false;
}
$total = count($names);
$success = 0;
foreach ($names as $name) {
$value = self::get_db($name);
if ($value !== false) {
$result = self::set_db($name, $value);
if ($result !== false) {
$success++;
}
}
}
return ['total' => $total, 'success' => $success];
}
public static function names()
{
$db = self::db();
if ($db === false) {
return false;
}
$rows = $db->selectRows(
"CALL sp_secret_list(?)",
"s",
self::scope()
);
if ($rows === false || count($rows) === 0) {
return false;
}
$names = [];
foreach ($rows as $row) {
$names[] = $row['name'];
}
return $names;
}
private static function reset_cache()
{
if (self::cache_available()) {
$names = self::names();
if ($names === false) {
return false;
}
foreach ($names as $name) {
self::set_cache($name, null);
}
}
}
private static function set_db($name, #[SensitiveParameter] $value)
{
if (!self::valid_name($name)) {
return false;
}
$db = self::db();
if ($db === false) {
return false;
}
$encryption_key = getenv(self::key_path_key());
$key = self::loadKey();
$encrypted = self::encrypt($value, $key);
if ($encrypted === false) {
self::haltExecution("Failed to encrypt");
return false;
}
$result = $db->selectScalar(
"CALL sp_secret_set(?, ?, ?, ?)",
"ssss",
self::scope(),
$name,
$encryption_key,
$encrypted
);
if ($result === false) {
$db->rollback();
throw $db->get_last_exception("Failed executing.");
}
if ($result !== 'Success') {
throw new Exception("Failed to update value. $result");
}
return $result;
}
public static function reveal(string $name)
{
if (!self::valid_name($name)) {
return false;
}
if (array_key_exists($name, self::$decryptedCache)) {
return self::$decryptedCache[$name];
}
$value = self::get_cache($name);
if ($value !== false) {
self::$decryptedCache[$name] = $value;
return $value;
}
$value = self::get_db($name);
if ($value === false) {
return false;
}
self::$decryptedCache[$name] = $value;
self::set_cache($name, $value);
return $value;
}
public static function decryptValue(string $encrypted)
{
if (empty($encrypted)) {
return false;
}
$key = self::loadKey();
if ($key === false) {
return false;
}
return self::decrypt($encrypted, $key);
}
private static function loadKey()
{
if (self::$keyCache === null) {
$key = self::loadKeyFromPath(self::fullPathToKey());
if ($key === false) {
return false;
}
self::checkKey($key);
self::$keyCache = $key;
}
return self::$keyCache;
}
public static function key_exists(?string $path = null)
{
return file_exists(self::fullPathToKey($path));
}
private static function loadKeyFromPath($path)
{
$key = self::loadIfExists(self::fullPathToKey($path));
if ($key === false) {
return false;
}
return $key;
}
private static function fullPathToKey($path = null)
{
$root = $_SERVER['DOCUMENT_ROOT'];
if (empty($path)) {
$path = getenv(self::key_path_key());
}
if (strpos($path, '/') === 0) {
return $path;
}
$separator = DIRECTORY_SEPARATOR;
$fullPath = realpath($root . $separator . $path);
return $fullPath;
}
private static function loadIfExists($path)
{
if (file_exists($path)) {
if (is_dir($path)) {
throw new Exception("Expected a file: $path");
}
$contents = file_get_contents($path);
if ($contents === false) {
self::haltExecution("Error reading '$path'.");
}
return $contents;
}
return false;
}
private static function haltExecution($message, $additionalMessage = '')
{
if (self::is_production()) {
Show::error("An error occurred");
} else {
$fullMessage = $message;
if (!empty($additionalMessage)) {
$fullMessage .= ": $additionalMessage";
}
$lastError = error_get_last();
if ($lastError !== null) {
$fullMessage .= ": " . $lastError['message'];
}
Show::error($fullMessage);
}
exit();
}
private static function checkKey(#[SensitiveParameter] $key)
{
$actual_bits = strlen($key) * 8;
if ($actual_bits === self::$BIT_LENGTH) {
return;
}
$actual_bits = strlen($parsedKey) * 8;
if ($actual_bits !== self::$BIT_LENGTH) {
self::haltExecution("$actual_bits bit key expected to be " . self::$BIT_LENGTH . " bit.");
}
}
private static function save($path, #[SensitiveParameter] $data)
{
self::ensureDirExists($path);
$result = file_put_contents($path, $data);
if ($result === false) {
self::haltExecution("Unable to write to $path.");
}
}
public static function rotateValue(#[SensitiveParameter] string $encrypted, #[SensitiveParameter] string $old_path, #[SensitiveParameter] string $new_path)
{
$old_key = self::loadKeyFromPath($old_path);
if ($old_key === false) {
return false;
}
$new_key = self::loadKeyFromPath($new_path);
if ($new_key === false) {
return false;
}
$decrypted = self::decryptValue($encrypted, $old_key);
if ($decrypted === false) {
return false;
}
return self::encryptValue($decrypted, $new_key);
}
private static function ensureDirExists($path)
{
$dir = pathinfo($path, PATHINFO_DIRNAME);
if (!is_dir($dir)) {
// all for owner, read/execute for others
$permissions = 0755;
$result = mkdir($path, $permissions, true);
if ($result === false) {
self::haltExecution("Unable to make directory for $path.");
}
}
}
public static function iv_length()
{
return openssl_cipher_iv_length(self::$method);
}
private static function decrypt(#[SensitiveParameter] string $encrypted, #[SensitiveParameter] string $key)
{
if (empty($encrypted)) {
throw new Exception("Empty parameter: encrypted");
return false;
}
self::checkKey($key);
$iv_length = self::iv_length();
$encryptedWithIv = base64_decode($encrypted);
if ($encryptedWithIv === false) {
throw new Exception("Malformed base64 encoding.");
}
$iv = substr($encryptedWithIv, 0, $iv_length);
if (strlen($iv) !== $iv_length) {
throw new Exception("Incorrect IV length prevents decryption.");
}
$encrypted = substr($encryptedWithIv, $iv_length);
$encryptedByteCount = strlen($encrypted);
if ($encryptedByteCount === 0) {
throw new Exception("Encrypted value missing.");
}
if ($encryptedByteCount < $iv_length) {
throw new Exception("Encrypted value corrupted. Too short.");
}
if ($encryptedByteCount % $iv_length !== 0) {
throw new Exception("Encrypted value corrupted. Incorrect block size.");
}
$decrypted = openssl_decrypt($encrypted, self::$method, $key, OPENSSL_RAW_DATA, $iv);
if ($decrypted === false) {
throw new Exception("Unable to decrypt: " . openssl_error_string());
}
return $decrypted;
}
public static function encryptValue(#[SensitiveParameter] $value)
{
$key = self::loadKey();
return self::encrypt($value, $key);
}
private static function encrypt(#[SensitiveParameter] $value, #[SensitiveParameter] $key)
{
self::checkKey($key);
$iv_length = openssl_cipher_iv_length(self::$method);
$iv = openssl_random_pseudo_bytes($iv_length);
$encrypted = openssl_encrypt($value, self::$method, $key, OPENSSL_RAW_DATA, $iv);
if ($encrypted === false) {
self::haltExecution("Unable to encrypt.", openssl_error_string());
}
return base64_encode($iv . $encrypted);
}
public static function generatePepper()
{
return base64_encode(openssl_random_pseudo_bytes(32));
}
public static function generateKey()
{
$fullPath = self::fullPathToKey();
$directory = pathinfo($fullPath, PATHINFO_DIRNAME);
$fileName = pathinfo($fullPath, PATHINFO_BASENAME);
$pattern = '/^\d{10}_/';
if (preg_match($pattern, $fileName)) {
$fileName = preg_replace($pattern, '', $fileName);
}
$fileName = time() . '_' . $fileName;
$fullPath = $directory . DIRECTORY_SEPARATOR . $fileName;
$key = openssl_random_pseudo_bytes(self::$BIT_LENGTH / 8);
if ($key === false) {
self::haltExecution("Failed to generate secure random key.", openssl_error_string());
}
self::save($fullPath, $key);
$fullPath = realpath($fullPath);
return self::getRelativePath($fullPath);
}
private static function getRelativePath($path)
{
$root = $_SERVER['DOCUMENT_ROOT'];
$rootParts = explode(DIRECTORY_SEPARATOR, $root);
$pathParts = explode(DIRECTORY_SEPARATOR, $path);
$rootPartCount = count($rootParts);
$same = 0;
$maxIterations = min($rootPartCount, count($pathParts));
for ($i = 0; $i < $maxIterations; $i++) {
if ($rootParts[$i] === $pathParts[$i]) {
$same++;
} else {
break;
}
}
$remainingRootFolders = $rootPartCount - $same;
$relativePath = '';
for ($i = 0; $i < $remainingRootFolders; $i++) {
$relativePath .= '..' . DIRECTORY_SEPARATOR;
}
$relativePath .= implode(DIRECTORY_SEPARATOR, array_slice($pathParts, $same));
return $relativePath;
}
public static function keep(string $name, #[SensitiveParameter] ?string $value)
{
if (!self::valid_name($name)) {
return false;
}
self::$decryptedCache[$name] = $value;
self::set_cache($name, $value);
$result = self::set_db($name, $value);
if ($result === false) {
unset(self::$decryptedCache[$name]);
self::set_cache($name, null);
self::haltExecution("Failed to store in database");
return false;
}
}
}
What’s left?
- Setup api tokens (big effort)
- Setup login accounts (big effort)
- Move UI to secrets.periplux.io
- Configure build-server/YAML interact with secrets api during deployment
- Ensure cron job calls correct endpoint with credentials to rotate secrets
- Pagination for secret rotation
- Endpoint to list secret names
- consider if secrets/get endpoint can/should be removed.
One response to “Separating Services”
[…] I briefly mentioned a two-factor authentication scheme when separating my services and moving the secrets JSON file with encrypted data into its own separate database. Two-Step […]