Because I have a cPanel account with a few web hosts, I’m always limited to one account for managing multiple domains. With one of my domains, this is getting a bit unmanageable, as I often create many subdomains. It’s to the point where the file system feels a bit spammy and fragmented.

I also wanted to start looking into any way to make it easier to automate the common process of adding sub-domains.

1. Create sub-domain
2. Remove self-signed certificate
3. Use Auto-SSL to create a certificate via Let’s Encrypt

I decided it was time to convert that domain into its own account. This would allow me to control permissions, narrow down the features needed, and get a bit more control over resource usage with CloudLinux as well. There are multiple wins here.

I’m more familiar with the client/reseller side of WebHost Manager (WHM), where people use the Control Panel (cPanel). WHM is extensive, and I often research what it can do. I found the feature “Convert Addon Domain to Account” under Transfers. I saw a list of every domain under the same user, with an option to convert.

I found that I could maintain ownership of the account via “Preserve Account Ownership”, so that I could still manage it as a reseller. At the bottom, a message said the feature was still a work in progress and that it migrates most, but not all, of the account data. It linked to more information, and under “Data not migrated” were listed the subdomains.

What? That is the whole reason.

The recommendation was to back up the subdomains’ data and delete the subdomains before conversion. With 15 subdomains, it wouldn’t be fun. I found I could bulk-update the subdomains to enforce HTTPS redirects, but no feature existed to remove them.

Removing multiple subdomains is a pain point. I have many domains and subdomains, so I would first restrict the list to those matching the domain name. Once I deleted the domain, I was returned to the unfiltered list. I would filter and tried command-clicking to view each domain to delete it, but cPanel was one step ahead and treated it as a simple click.

A sub-domain doesn’t delete quickly either. I tried kicking off the delete, rushing back, filtering the list, finding the next subdomain to delete (skipping over the one currently being deleted), and opening the window – only to have the deletion for the prior subdomain complete and force me back to the unfiltered domain list again. It felt like the software was actively working against me and was not set up with guards to disable buttons while deletion was in progress. The fact that it didn’t retain the filtered list was also frustrating. All the while, I’m getting email pings as each subdomain is deleted.

Finally, I initiated the transfer, which took place in 12 seconds. It was successful, but when I checked the details, I got an error when I tried to copy an SSL certificate that supported every domain except the one I was transferring.

To me, the new account looked empty. No files on the file system. No SSL certificate. Just a blank slate except for the database. Now I’d have to add all 15 subdomains back. Adding them was quick. cPanel has both a “Submit” button and a “Submit And Create Another” button. The tough part would be transferring the files over. The build server often does that for me, so it may be as simple as re-running the last known successful job.

Next was the SSL certificate. Since one account controlled everything, I could start over and figure out why I didn’t have a certificate that covered all subdomains with a wildcard. When I validated it, I got an error: Domain Control Validation failed: DNS-based DCV failed. I first tried getting an SSL certificate for just the regular domain, which went through fine. I tried to get the wildcard again, but the validation failed again.

Domain Control Validation is different for wildcards compared to normal subdomains. Since a subdomain doesn’t exist, the system often adds a TXT _acme-challenge record to the DNS zone during validation and then removes it. The value changes with each validation attempt, and something wasn’t working. The name servers are on the same server, and the SSL Wizard should have done this automatically.

For now, I was able to assign a certificate to each individual sub-domain to get past this issue.

Next, set up the build servers. They were still pointing to the old path for the old account on the server and using the old account credentials. I would need an SSH key for rsync transfers.

Well, it seemed that my sub-account didn’t have terminal access. A quick jump over to WHM and I was able to upgrade/downgrade the account to a feature package with terminal access.

# Create a hidden directory for ssh keys
mkdir -p ~/.ssh
# Only the owner can read & write as a directory
chmod 700 ~/.ssh
# create an ssh file using ed25519 for the build server
ssh-keygen -t ed25519 \
  -f ~/.ssh/forgejo_ed25519 \
  -C "forgejo runner deployment"
# When asked, set no password for unattended builds
# Show private key contents
cat ~/.ssh/forgejo_ed25519
# Copy & Paste private key above
# as a build servers secret
# create the authorized_keys file if it does't exist
touch ~/.ssh/authorized_keys
# Set so only the owner can read & write
chmod 600 ~/.ssh/authorized_keys
grep -qxF -f ~/.ssh/forgejo_ed25519.pub ~/.ssh/authorized_keys ||
  cat ~/.ssh/forgejo_ed25519.pub >> ~/.ssh/authorized_keys
# VERIFY IT WORKS BEFORE THE NEXT STEP
# delete private key
rm ~/.ssh/forgejo_ed25519
# keep the public key

From there, I updated the organization’s secrets and variables in Forgejo and updated one project to deploy to the new users folder. I ran the most recent deploy job and visited the website afterward to verify that everything deployed.

Success!

14 more to go…

After that, I had to zip up a few sites and move files for the blog, the internet streaming site’s audio files, and the main website. I did the same for the primary home page and found that the files had actually transferred over. Woo Hoo!

For WordPress, I found that I needed to create a feature package with database access via phpMyAdmin. I then renamed the database to replace the old account prefix in both the database and the user account, and then updated the wp-config.php file to match.

Delete Everything!

I started working on a web app that would let me create subdomains quickly and automate them via a webhook or one of my agents. I set up a hashed password and protected the error log and other files. I even set up a deployment script. For such a small website with only a few files, I was a bit shocked by how long it took.

I opened the log file, and my heart sank. The first line said it all, followed by confirmation of what I had done:

rsync: [generator] failed to set times on "/.": Operation not permitted (1)
rsync: [generator] delete_file: rmdir(var/www/html/.well-known/acme-challenge) failed: Permission denied (13)
rsync: [generator] delete_file: rmdir(var/www/html/.well-known) failed: Permission denied (13)
rsync: [generator] delete_file: unlink(var/www/html/index.html) failed: Permission denied (13)
rsync: [generator] delete_file: unlink(var/www/html/cptechdomain.shtml) failed: Permission denied (13)

In my infinite wisdom, I set up the build workflow but forgot to set a DEPLOY_PATH variable. It was hard to scroll through the log because the browser kept saying the script was busy and asking if I wanted to wait. When all was said and done, in 1 minute and 17 seconds, the log file showed over 224,141 attempts to delete files and directories that were denied due to permissions.

Now would be a good time to say I have a shirt that says I don’t always test my code, but when I do, I do it in production. Only a 10x programmer shoots from the hip in the dark. *sigh*

I went to the file manager and found nothing in the accounts home folder. The good news is that this was a segregated account. Normally, I add new subdomains with my primary account, but I wanted to start segregating things. My concern is that I don’t know what all it was able to delete. Any file or folder that account could see and write to is gone. Hopefully it’s just the files under their own home folder. However, I do wonder…

The good news is that I have backups among backups. User accounts are backed up every night, both locally and via JetBackup. The server itself is also backed up daily as well. Before I started moving everything around, I created a 3.5 GB zip file of the primary account I use.

I tried to add the 15 folders back and upload the archived data for the blog, radio files, and primary domain name (thank goodness I made a small backup of the main website when I didn’t need to!). I kept getting an error: “Failed to exec”.

I went down a rabbit hole with ChatGPT, trying to diagnose the issue and see whether I could rebuild the account. I felt like I was fighting it, as it kept diagnosing that a folder didn’t exist or that the index.php or index.html files were missing. Every time it asked me to confirm, I knew the verdict, but I kept pasting the results it asked for, hoping for a potential way out of this mess. It checked .htaccess files and all kinds of configurations. Having walked through the process before the deletion and again afterward, I knew everything that should be there was there.

I ran over to Google Gemini and explained the situation – a new user deleted all their files via rsync before the account was ever backed up, and nothing existed in their home directory. Gemini was more promising, suggesting I manually set up scaffolding and rebuild the account, but it referred to a script that no longer existed (or never existed) for rebuilding the user account itself.

I knew what I had to do. I needed to terminate the account and recreate it. But first, I needed to export the WordPress blog database, since it moved when the account was split off.

Setting everything back up was much quicker now that I had been through the process once (sometimes twice) today. I knew what I was doing, and configuration was already set up on the different repositories for the build runner… almost.

I learn from my frustrations and mistakes. This was NOT going to happen again while I was in charge. I needed a safeguard to prevent this from happening again. Within the Forgejo organization, I added a variable as a safeguard.

DEPLOY_PATH = /home/kognabo/000-missing-deploy-path

If I ever set up a repository again and forget to add a DEPLOY_PATH variable, it will fall back to the organization variable and show up as a folder indicating that the repository is missing the deploy path. My websites also produce a build manifest when deploying code, so it would further help me track down which repository caused that folder to appear on the web server. The “000” prefix is just so it appears at the top of the list in alphabetical order, rather than being buried when scrolling through a long list of folders. I also setup the build script to fail if expected variables are not set.

I got back to where I was before the deletion and deployed the subdomain creation app. I noticed one of my Agents made changes to the repository, citing rsync’s potential to blow away everything when misconfigured, and also noted that their local .git folder and the application error log file were deleted.

A bit too late, but at least someone, or something, is thinking ahead and testing things first.

Subdomains & SSL Certificates

After I deployed the application, I ran into a few problems with basic authentication, so instead of letting Apache authenticate against actual user accounts, I switched to an HTML-based login form and posted the credentials. It’s one less thing going across the wire in the clear.

I had a few problems testing the API, but I finally got something working in cPanel to create the new subdomains and folders. I started addressing issues with existing domains and folders, and I even set up a temporary placeholder instead of showing the directory list for new subdomains.

The process was finally lean. I log in, type in a subdomain, and everything is automated.

Almost.

I noticed a response of “is_excluded: 1”. I checked the certificates created and found a familiar problem. The subdomains were created with a self-signed certificate, so AutoSSL excludes them. I don’t understand why it does that, but the only workaround in the past was to delete the self-signed certificates and ask AutoSSL to assign a certificate.

[global]
# Show include/exclude controls
# in SSL unified interface (SSL Status)
ssl.autossl_in_unified = true
# Accept Let's Encrypt terms on behalf of cPanel users
autossl.exclude_new_domains = false

[panel.ini]
comment=cPanel UI/feature customization file
paths=/var/cpanel/panel.ini

cagefsctl --force-update
cagefsctl -m user1
cagefsctl -m user2

I went into WHM under “Tweak Settings”, found the “Security” tab, and disabled “Generate a self-signed SSL certificate”. It’s actually recommended not to do this because the server’s certificate will be shown for any website on the server until it gets its own certificate. This is more of a temporary band-aid to fix the symptom rather than the cause.

I learned that I could remove AutoSSL-excluded domains through the cPanel API so that AutoSSL can pick them up. I don’t know how they get added to that list during creation. Since I found a solution closer to the root cause, I re-enabled the setting to generate self-signed SSL certificates.

What happened was a failure… and a success.

DNS DCV: No local authority: “bar.kognabo.com”; HTTP DCV: The system queried for a temporary file at “http://bar.kognabo.com/.well-known/acme-challenge/HX-FFNSYK9EP0YP5GL72CTBV7-5Q2E01”, but the web server responded with the following error: 404 (Not Found). A DNS (Domain Name System) or web server misconfiguration may exist.

I opened the file manager and found that the domains folder had been created. I drilled down into the .well-known/acme-challenge folder and found it empty. Something was happening with AutoSSL, but the validation file was missing. I checked whether the subdomain had a self-signed certificate. Much to my surprise, it had both the self-signed certificate and the Let’s Encrypt certificate. When I went to the website, the Let’s Encrypt certificate came through.

Now I have an issue: I am getting an error, but the results were successful. It seems the first error/status response isn’t the final verdict, while propagation is still settling after the first attempt.

As I understand it, AutoSSL doesn’t just keep retrying. I don’t know why this is successful while reporting an error. As a further measure, if AutoSSL fails, I start polling the new subdomain to see what certificate comes back. Perhaps the problem was with DNS. I also added a 10-second delay after domain creation before using AutoSSL, possibly to help with domain propagation – but no one has requested the domain yet, or knows of its existence to cause propagation.

The website finally detected that the certificate had been resolved after 40 seconds. I changed the web app to log each attempt to show what was happening at each step, and learned that the website goes through different states. I also wanted to confirm whether the problem was DNS propagation, the file not existing, permissions issues, or the site not being fully online yet. Since Let’s Encrypt was telling me what file it was looking for, I could verify it both physically on the storage, as well as a local loopback address while specifying a host header.

I discovered that the file no longer existed by the time AutoSSL finished. Upon further tweaks, I found that AutoSSL wasn’t exactly “done,” but simply not active – which I had assumed meant that an inactive instance of AutoSSL, along with an error message, meant that something had failed, when in fact it was part of the Domain Control Validation (DCV) lifecycle. I also saw the lifecycle of how the domain gets its certificates set up – first a default certificate for the server itself (www.cat.regaldragondancepartycom), and shortly after a temporary self-signed certificate.

1. 9:07:43 PM Creating the subdomain.
2. 9:07:53 PM The DNS publication wait is complete; requesting AutoSSL.
3. 9:07:55 PM Local loopback preflight reached this hostname and returned HTTP 200.
4. 9:07:55 PM Local HTTPS/SNI preflight reached this hostname and returned HTTP 200.
5. 9:07:55 PM Subdomain created and AutoSSL was requested. Checking its status.
6. 9:07:55 PM cPanel is reporting an active AutoSSL account check.
7. 9:07:55 PM Most recently recorded AutoSSL/DCV problem: DNS DCV: No local authority: “foo6.kognabo.com”; HTTP DCV: The system queried for a temporary file at “http://foo6.kognabo.com/.well-known/acme-challenge/PZ_ZS-46QHKDOLU3QY0PG03ZEAF73ZTO”, but the web server responded with the following error: 404 (Not Found). A DNS (Domain Name System) or web server misconfiguration may exist.
8. 9:07:55 PM The challenge file /.well-known/acme-challenge/PZ_ZS-46QHKDOLU3QY0PG03ZEAF73ZTO is not present in the document root now. The local web-server check returned HTTP 404. This is only a snapshot: AutoSSL may not have created the temporary file yet, or may already have removed it after an attempt. cPanel retains its most recent DCV problem between runs, so this does not prove the file was missing during the recorded validation.
9. 9:07:55 PM HTTPS is serving a CA-signed certificate for a different hostname: CN=www.cat.regaldragondanceparty.com.
10. 9:08:06 PM cPanel is no longer reporting an active AutoSSL account check; waiting for certificate installation or another queued worker phase.
11. 9:08:06 PM HTTPS is still serving a temporary self-signed certificate.
12. 9:08:38 PM cPanel is reporting an active AutoSSL account check.
13. 9:08:49 PM cPanel is no longer reporting an active AutoSSL account check; waiting for certificate installation or another queued worker phase.
14. 9:08:49 PM Certificate confirmed: cPanel reports it is active and eligible for renewal.

In Closing

Overall, from subdomain creation to an issued SSL certificate, it takes about a minute. This is quicker than the manual steps I would go through to create a domain, delete the self-issued certificate, run AutoSSL myself, and verify it worked without skipping a step. With this one domain, I create many subdomains for each building block to test and demonstrate them, so creating them through a simple interface is convenient. I may allow the agents to go ahead and create subdomains on their own instead of waiting on me to create their build targets for new repositories.