This is a walk through, describing a problem, reporting a bug, and the steps taken to work-around the issue. In short, I can’t use my clipboard on an unsecured website in my private network. I assign IP addresses vid DHCP, open up ports in a few routers, call my ISP to allow requests to my IP address, setup dynamic DNS, get a certificate issued, and install the certificate.
Not much else is here except linux commands, screenshots, thought process, and issues that came up during the whole process.
Microsoft Designer: Image Creator Prompt
Show an image of a giant man poking holes in two walls in front of each other, allowing access for the world to talk to a robot behind the last wall. In addition, show that they can only talk to the robot over a secure connection.
I’ve dug more into the never ending features of Open WebUI, and found a bug along the way. I was watching a video by Matt Williams (who worked on Open WebUI), called Is Open Webui The Ultimate Frontend Choice? He walked through the steps of creating a prompt with a variable, and makes use of the clipboard.
write a short summary of the following text for someone who is [age] years old: {{CLIPBOARD}}
I created the same prompt, went over to the chat window, typed forward slash, and saw my prompt appear.
Pressing the [ENTER] key did nothing. Clicking it did nothing. The [TAB] and arrow keys did nothing… I went to the browsers developer tools and saw an error on the console. I worked out that because I was accessing the website over a connection that isn’t secure (http vs https), and that the website was not on the same computer I was using (localhost, 127.0.0.1, 0.0.0.0), That Google Chrome does not make the clipboard available to JavaScript as a security measure.
In the video, Matt is seen accessing the site over http, but it’s via localhost. The browser is unfamiliar as well. I looked at the code itself, and found that although it was catching errors from a promise, it wasn’t using option chaining or ternary operators to check if navigator.clipboard was defined. I filed a bug report with full details at 6332 Slash commands throw console error for {{CLIPBOARD}} on unsecured http (non-localhost) It looks like they closed the issue and created discussion 6336 after six hours. I also provided an example of a potential, untested fix. I didn’t dig into their coding style, and I didn’t even download the repo locally. I simply edited a few lines of code via GitHub.
So what’s the workaround? Remove {{CLIPBOARD}} from my prompt, or setup Open WebUI to run over a secure connection.
Certificate
To make any website secure, you need a certificate. Most people use Let’s Encrypt as a certificate authority (CA) to issue certificates since its free. The issue here is that it requires a public domain like example.com. However, we are on an internal network using cjpi.local. I could get a domain name, or create an existing subdomain over to my IP address. It would also mean that I would be exposing the Raspberry Pi to the internet for anyone to access, and I would need to setup the router to forward the ports to that device. Let’s look into what’s involved.
Dynamic DNS
I don’t pay extra for a static IPv4 address, so I would need a dynamic DNS service to update the “A” record for the domain/sub-domain each time it changes. A few services are No-IP, Dynu, ClouDNS. These services usually require you to install a client to keep them informed of your latest IPv4 address. On top of that, the free versions usually require that you manually confirm that you are still using the service. I’ve used Dynu back in 2004 when it was a paid service for my personal blog.
So, let’s try cjpi.freeddns.org with a free account. The interface seems simple.
The IP is already populated with my own IPv4 address without setting up any client software.
Router Ports
Well, we now have a dynamic domain name pointing at our home ip address. We need to log into the router and forward the ports to the Raspberry Pi. My home system actually has a couple of routers to go through, before you reach the pi. We also need to setup DHCP so that the Raspberry Pi will always be assigned the same IPv4 address based on its mac address.
Let’s go to the router that the Pi connects directly to. We need to get its current IP and MAC address. Viewing “Online Devices”, this router shows the current IP address assigned on the local network as 192.168.23.198. The mac address isn’t listed though. Let’s go to the pi and ask what it’s MAC address is. Since we are connected over WiFi, we are looking for the wlan0 entry.
sudo apt install net-tools
ifconfig
wlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.23.198 netmask 255.255.255.0 broadcast 192.168.23.255
inet6 fe80::e65f:1ff:fef8:ca77 prefixlen 64 scopeid 0x20<link>
ether e4:5f:01:f8:ca:77 txqueuelen 1000 (Ethernet)
RX packets 6143618 bytes 8983693571 (8.9 GB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2465582 bytes 263473607 (263.4 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
It looks like “ether” is the MAC address – e4:5f:01:f8:ca:77
Let’s go back to the router and set up IP-MAC Binding. For my router, it’s under “Advanced”.
From now on, any time the Raspberry Pi asks the WiFi router to assign it an IP, the DHCP server will provide 192.168.23.198.
The next step is port forwarding. Since we want everything to work over a “standard” port for https, I’m going to expose the website as port 443. We need to tell the router that any request over port 443 should go to the Pi’s Open WebUI service running on port 3000.
This wasn’t the case 20 years ago, but most routers today already have support for dynamic DNS baked in so that you don’t have to install any special software on your PC. 20 years… for some reason I was recalling this as if I had set this up a couple years ago. but 20? And routers do much more as well, including media streaming and network attached storage. I think most people are oblivious to what their router is capable of if you just hook a USB drive directly to it. People occasionally ask me about what to buy to setup a NAS or media streaming service at home. People talk about a RAID setup, Gigabit networking, etc. Just buy a USB hard drive and connect it to your router for starters. A Raspberry Pi running Plex is the next cheapest option before building out a full blown server.
Anyhow, let’s enable the service.
Well, it appears that the host that I chose is not listed. Let’s jump over to the front facing router to see what it lists for service providers. Give me a second while I swap the network that I’m connected to…
Well, it looks like we have limitations if we want to have one of the routers update the IP address on their own. Wait … before we get this far, we need to setup MAC-IP binding and port forwarding for the other router. This forward facing router needs to be told that the other router should receive traffic from 443 and pass to 443.
I grabbed a photo of the Tenda AC1200 Smart Dual-band WiFi Router, which included the MAC address. Looking at the tp-link, I went under Network, DHCP Server, and scrolled down to the DCP Client List. On the page 5 we have Tenda listed as the last item with its mac address. E8-65-D4-4A-96-40
Scroll up and we can add an address reservation.
Now we need port forwarding so that traffic to https (port 443) goes to the other router. For this we go under NAT Forwarding -> Port Forwarding.
That should be everything to expose the Pi to the internet. Here is an overview of how traffic reaches the Pi.
Internet -> [443] tp-link Router -> [443] Tenda Router -> [3000] Raspberry Pi
-> [3000] Docker -> [8080] Open WebUI
Does it work? Let’s find out. We need an external service to try it out on. Let’s setup a PHP script to download content.
<?php
$url = "http://cjpi.freeddns.org:443";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
$response = curl_exec($ch);
if (curl_errno($ch)) {
echo 'cURL error: ' . curl_error($ch);
} else {
echo htmlspecialchars($response);
}
curl_close($ch);
cURL error: Failed to connect to cjpi.freeddns.org port 443 after 131960 ms: Couldn’t connect to server
Well that took forever. Let’s try to do it by IP, and add a short timeout.
cURL error: Connection timed out after 10002 milliseconds
Something isn’t configured. Lots of hoops to jump through here. Wait… I had reset both routers. Can I even access it locally? Yes. http://cjpi.local:3000/ works fine. Can I ping it and get the same address thats in the router? 192.168.23.198 – yes, that works. If I try to access it on 443, does the local router redirect to 3000? http://cjpi.local:443/ no… http://192.168.23.198:443/ no… Wait – I’m not going to the router to forward the port. Let’s try connecting to the first router instead to experience the port forwarding. In this scenario, I need to make a request to the second routers ip address on 443, and it will forward it to the pi on port 3000.
http://192.168.54.143:443 -> http://192.168.23.198:3000/
Yes! That works. That means the last router in the chain is configured appropriately. So the next step – can I make a request to the first router over port 443, while connected to that router?
http://192.168.54.1:443/ -> http://192.168.54.143:443 -> http://192.168.23.198:3000/
Http Error 400 (Bad Request). The router doesn’t have local management enabled, which runs over https. I wonder if it’s blocking port 443 as a security measure, regardless of port forwarding rules. Let’s try exposing a different port externally.
http://192.168.54.1:3001/ -> http://192.168.54.143:443 -> http://192.168.23.198:3000/
The error is different. ERR_CONNECTION_REFUSED. Let’s try updating the php script on the remote website to use the new port. 0ms response. Couldn’t connect to server. Let’s try removing 443 from all devices.
http://192.168.54.1:3002/ -> http://192.168.54.143:3001 -> http://192.168.23.198:3000/
cURL error: Failed to connect to 204.111.141.3 port 3002 after 0 ms: Couldn’t connect to server
Hmm… Could this have something to do with the optical network terminal (ONT)? It’s a Nokia XS-010X-Q. Can I log into it and configure port forwarding as well? Nothing appears on int like a sticker or anything with login credentials. I’m noticing that my first routers IP address isn’t my actual IP address externally. Can I log into the default gateway at http://100.64.192.1 ? Nothing… Let’s see. I’m finding specifications for XS-010X-Q. Web GUI on port 80 admin/1234 and ssh/telnet on port 23, same credentials. Why is the IP specified as 192.168.100.1? I’m unable to SSH into it. Do I need to connect to it directly rather than through the router? lets try ssh on my external ip instead. no… Ok, I need to go into the depths of the basement again. I’ve got a laptop, and a hub that can connect to an ethernet cable. Well… that particular hub needs a USB-C cable at both ends. I’m a bit confused where it had run off to. I did a quick search regarding my ISP and found that they usually don’t block ports.
Let’s think about this. The php code running on an external server changed its behavior once I setup port forwarding on all devices to use numbers other than 443. It may be the case that the issue lies with the Raspberry Pi itself.
Looking back at my php code, I had setup the port as https. Well… that’s an easy fix, but I’m still getting an error. Let’s open the router up for remote management. Maybe it’s not the Pi server. Nothing… I tried to allow connections over port 80 and 443, but I couldn’t reach the router from an external server. Perhaps it is the ONT getting in the way. Let’s go down and see if I can find something else to connect.
I was able to connect to the ethernet port with the old Windows Box. I tried to view with a web browser and ssh into the IP address of the gateway 100.64.192.1 but it just timed out. Was I supposed to try 192.168.100.1? One moment… nope. Nothing. I was thinking maybe it had some kind of internal routing to handle both 192.168.100.* and 100.64.192.* addresses. I’m getting the impression that the device is locked down by the ISP. I can’t find much about it except that one web page, and it didn’t seem to mention anything about blocking ports or a web interface.
I’m a bit stuck. Let’s look at the entire chain
- Internet
- Internet Service Provider
- Home Network
- Nokia XS-010X-Q Optical Network Terminal
- Terminal http://x.x.x.x:3002
- Internal Network 100.64.192.*
- tp-link AX6600 Tri-Band Wi-Fi 6 Router
- http://100.64.192.135:3002
- Internal Network 192.168.54.*
- Tenda AC6: AC1200 Smart Dual-band WiFi Router
- http://192.168.54.143:3001
- Internal Network 192.168.23.*
- Nokia XS-010X-Q Optical Network Terminal
- Raspberry Pi 4
- http://192.168.23.198:3000
- Docker
- http://0.0.0.0:3000
- Internal Network 172.18.0.*
- Open WebUI container
It feels like the optical network terminal may be the culprit. Although it only has one ethernet port, I get the inclination that it doesn’t automatically forward all traffic to the first IP it assigns… wait. First ip. So, it has an internal DHCP server that assigns those IP addresses. When I connected directly with my PC, it assigned me 100.64.198.252. My router gets an IP address of 100.64.198.135. The question is – what was the first IP ever assigned, and is it still reserved to receive all incoming traffic? For some reason, I’m suspecting the second WiFi router (Tenda) was originally connected to the ONT. I can’t log into the ONT via ssh or the web interface. Perhaps I can call my ISP and ask them to clear or reset the DHCP leases. I’ve gone through blackouts a few times, but the routers are on a dedicated battery supply that can last for days. Let’s try power cycling the device first. No… same problem. My external IP is still the same. If I try to hit http://x.x.x.x the request times out. If I try to hit http://x.x.x.x:3002 I immediately get denied.
Ugh… I don’t like when the only phone number available is in letters. It takes forever to work out what each letter maps to on the number pad. Estimated wait time of 10 minutes. Maybe I should shoot an email over to support and see which one responds first.
Someone picked up after 5 minutes. My first approach was to start talking as if the person knew about network terminology… and they did! Holy smokes. Not only did they understand, but they interrupted and told me that they knew what the problem was, and that I needed to be assigned a public reserved IP on the CG Nat (Carrier Grade NAT). The guy seemed happy about knowing about what my issue was. I got the impression that it was untapped knowledge that he rarely gets to use while usually answering basic level 1 customer support requests that can be answered with checking if the router is plugged in, and power cycling.
Here I was expecting to get some kind of upsell to increase speed and purchase a static ip address while still having the same problem when all was said and done. While we waited for the new IP address to be provisioned, we got to talking about how he knows so much, and that he used to work on building multi-million dollar airplane simulations. I responded in kind about the rig I had setup in my basement, as well as my experience with working on large simulations to train harbormasters. And then that got us talking about Baltimore with the Francis Scott Key Bridge collapsing when the ship ran into it when a harbormaster should have known how to navigate the area. Apparently we are both from the Baltimore county area – and now living way out in Front Royal, VA and Strasburg, VA. Wow…
Anyhow, I now have a new public IP. Somehow the CG Nat doesn’t know how to forward traffic to the private IP since the port is already open (or something similar of that nature, as he explained it). Although the IPv4 address is “private”, it’s still publicly available. I suppose this is both a security precaution, and a way to distribute a limited number of IP addresses to a larger number of people. Although the IP address hasn’t changed since it was first assigned, it can change. He said it was unlikely to change for residential customers, unless a business wanted that IP address. I inquired if they were going to upgrade to IPv6, and he simply said the guys at the top have been talking about it for years, but he’s not privy to any timelines.
Regarding the ONT DHCP server and how it directs traffic with port forwarding, he wasn’t too certain about it. He didn’t have access to login to the ONT himself. Eventually we saw the IP address had been assigned, and I was no longer getting timeout messages once I changed my php code to point to the new IP address.
Does it work… maybe? I’m having trouble and decided to open my routers remote management again and got a response.
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC
"-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta
http-equiv="refresh"
content="0;
URL=/webpages/index.html" />
</head>
</html>
Well, let’s update the URL and see what that page gets us.
/webpages/index.html
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<meta name="viewport" content="width=device-width,initial-scale=1,minimum-scale=1,maximum-scale=1,user-scalable=0">
<meta name="apple-touch-fullscreen" content="yes">
<meta name="apple-mobile-web-app-capable" content="yes">
<meta name="apple-mobile-web-app-status-bar-style" content="black">
<meta name="format-detection" content="telephone=no">
<link rel="shortcut icon" href="favicon.ico?t=f56578a8">
<link href="themes/default/css/perfect-scrollbar.css?t=f56578a8" rel="stylesheet">
<link href="themes/default/css/spectrum.css?t=f56578a8" rel="stylesheet">
<link href="themes/default/css/jquery.Jcrop.css?t=f56578a8" rel="stylesheet">
<link id="baseCss" href="themes/default/css/base.css?t=f56578a8" rel="stylesheet">
<!--[if IE 9]>
<link type="text/css" href="themes/default/css/ie9.css" rel="stylesheet" />
<![endif]-->
<!--[if lt IE 9]>
<link type="text/css" href="themes/default/css/total.ie8.css?t=f56578a8" rel="stylesheet" />
<![endif]-->
<title id="title">Opening...</title>
<noscript>
<meta http-equiv="refresh" content="0; url=error.html?t=f56578a8"/>
</noscript>
</head>
<body ontouchstart="">
<div id="main-container"></div>
<script src="js/libs/jquery.min.js?t=f56578a8"></script>
<script src="js/libs/jquery.backgroundSize.js?t=f56578a8"></script>
<script src="js/libs/base64.js?t=f56578a8"></script>
<script src="js/libs/encrypt.js?t=f56578a8"></script>
<script src="js/libs/cryptoJS.min.js?t=f56578a8"></script>
<script src="js/libs/tpEncrypt.js?t=f56578a8"></script>
<script src="js/libs/polyfill.js?t=f56578a8"></script>
<script src="js/libs/jquery.Jcrop.js?t=f56578a8"></script>
<script src="js/libs/spectrum.js?t=f56578a8"></script>
<script src="js/app/url.js?t=f56578a8"></script>
<script src="js/su/char.js?t=f56578a8"></script>
<script src="js/su/language.js?t=f56578a8"></script>
<script>
try{
if(!localStorage||!localStorage.setItem)
throw"unsupported browser"
}catch(r){
location.href="./error.html?t=f56578a8"
}
try
{
$.su.language=new $.su.Language
}catch(r){
location.href="./error.html?t=f56578a8"
}
GLOBAL_STYLE="default"
</script>
<script src="js/su/frame.js?t=f56578a8"></script>
<script src="locale/ispAutoConf.js?t=f56578a8"></script>
<script>
$.su.isMobile()
? $('<link
type="text/css"
href="themes/default/css/mobile.css?t=f56578a8"
rel="stylesheet" />')
.insertAfter("#baseCss")
: $('<link
type="text/css"
href="themes/default/css/total.css?t=f56578a8"
rel="stylesheet" />').insertAfter("#baseCss")
</script>
<!--[if lt IE 9]>
<script src="js/libs/respond.min.js?t=f56578a8"></script>
<![endif]-->
<script>
$(document).ready(function(n){
App=new $.su.App,App.setContainer("main-container"),
App.init().done(function(){App.launch()})})
</script>
</body>
</html>
Nothing discernable. The title “Opening…” doesn’t even convey anything about what I’m connecting to. Let’s try it on my phone over the cellular connection only. Why didn’t I just use my phone from the start? I’ve got a bad signal in my home, so I’m either crossing my fingers that it works, running outside and waving it up at a few mountain ranges, or running off to a local restaurant.
I’m getting a warning about the certificate, but It’s not letting me do anything when I tell it to proceed anyway. However, the certificate appears to be from my forward-facing router.
That’s enough for me to confirm that my router is being hit from an external network over port 443. That’s much further than I got prior. So what’s going on with port 3002? Why can’t I get it to respond? Let’s turn off remote management for the router, and assign ports 80 and 443 to go over to the internal router 3001.
Port 80 & 443 work. Not only that, port 3002 also works on my phone. Why isn’t the PHP code picking up on port 3002?
Could it be that my external web server is blocking requests to that port as well? It’s running on BlueHost. Let’s try a sever running on Hostinger instead. Well hey – it works! BlueHost has been a thorn in my side with a lack of features, running outdated software, and customer support that isn’t well versed in technology. The move to hostinger seems to be revalidated when I run into problems like this. I had just assumed that BlueHost would naturally let you request web pages over any port other than the standard 80/443. That’s generally how many websites communicate with each other to access a multitude of services running behind the same IP address.
Internet security is a big deal. I understand my own private network is a bit more complex than most people. Rather than testing direct from an external provided such as my cell phone, I tested from within another network that was clamping down on security pretty hard.
Dynamic DNS
With verification that we can reach the internal Open WebUI service from the outside world, we can now return to setting up Dynamic DNS. But first, our outward facing IP address has changed, and DynU needs to be informed about it.
The web interface isn’t that great to update the IPv4 address. When first setting up the account address, the IPv4 address was filled in for us based on our current IP address. Now, we need to enter it in manually. Once saved, I ran to my phone and entered http://cjpi.freeddns.org:3002 and was brought to the login page of Open WebUI.
Before moving onto setting up HTTPS, we still need to set something up to keep this IP address updated. While both of our routers support Dynamic DNS, they are limited to the following providers:
| Tenda | TP-Link |
|---|---|
| oray.com | TP-Link |
| 88ip.cn | No-IP |
| dyn.com | DynDNS |
- Tenda
- oray.com – Chinese site some kind of tech company
- 88ip.cn – Chinese site “Profesional and stable enterprise-level dynamic domain name resolution service”
- dyn.com – redirects to oracle.com talking abut OCI DNS
- TP-Link
- TP-Link
- No-IP – free DDNS, must confirm every 30 days
- DynDNS – 7 day free trial
It seems that DDNS isn’t standardized in a way that routers can be configured to use any DDNS service. Tenda seems to be focused more on services in China.
Let’s see what’s involved with the TP-Link DDNS. First, I needed to bind the router to my TP-Link ID. Done. Now I can go back to Dynamic DNS and assign a host name. Done.
And http://cjpi.tplinkdns.com:3002 is now pointing to the site. That was pretty simple. No costs?
Here is the problem. I’m locked into this router. I like to upgrade routers over time to get the latest features. Let’s stick with Dynu DDNS, but setup the Raspberry Pi to update the IP address on our behalf.
sudo apt update sudo apt upgrade sudo apt install ddclient
Oh boy, flashback to 80’s installation programs
Dynu isn’t listed. Is dyndns1/2 part of it? Ok, lets ditch the dynu account and go for one of the main ones on the initial dialog.
- no-ip.com
- freedns.afraid.org
- duckdns.org
- domains.google
- http://www.dyndns.com
- http://www.easydns.com
- http://www.dslreports.com
- http://www.zoneedit.com
We’ve already glanced over no-ip, and “afraid” seems… odd. What is duck dns? Free Dynamic DNS hosted on AWS.
Ok… cute, simple, free. Lot’s of sign-in options here. Although very limited in what to do here, the interface is much easier to use. I decided to go with my name as the subdomain. I could add more services later, and all requests are not limited to just the Raspberry Pi.
Now… how do I setup the Pi to keep it updated?
How do I go back to the first page of ddclient? The [ESC] key seems to proceed. Control + [C] isn’t quitting. Let’s follow through, uninstall, and reinstall the ddclient.
sudo apt remove ddclient sudo apt install ddclient
The install screen didn’t show up. It must have save a configuration file somewhere. I found a file in /etc. Let’s try this again.
sudo rm /etc/ddclient.conf sudo apt remove ddclient sudo apt install ddclient
It’s showing up, but it skipped some screens.
I entered the token on the Duck DNS page. It didn’t ask for anything else. Let’s look at the configuration and modify it.
sudo nano /etc/ddclient.conf
Well, I’m not sure what exactly to configure for DuckDNS. Looking for documentation, I’m finding references to intermittent support for DuckDNS and a few problems. Odd, since it’s featured on the main page of the installer.
Fortunately, DuckDNS has an install page that gives a ton of different installation methods, but none cover ddclient specifically. I’m going to uninstall ddclinet and try the first option to setup a cron job to run a shell script on Linux. Cron jobs are fairly strait forward and simple to use.
mkdir ~/duckdns cd ~/duckdns nano duck.sh chmod 700 duck.sh crontab -e ./duck.sh cat duck.log
duck.sh
echo url="https://www.duckdns.org/update?domains=lewismoten&token=MY_TOKEN&ip=" | curl -k -o ~/duckdns/duck.log -K -
crontab
*/5 * * * * ~/duckdns/duck.sh >/dev/null 2>&1
I ran everything through. A log was created and the contents are “OK”. I completely missed it at first using “CAT” since it wrote “ok” inline, before the prompt in my terminal. At this point we are now updating the dynamic DNS record every 5 minutes. If the IP address changes, I wont be able to access the website for that long of a period – max, as long as the Raspberry Pi is running, as well as its chron jobs.
Getting a Certificate
Now it’s time to get a certificate for my newly created domain. There is one place that just about everyone and his brother talks about, and that is “Let’s Encrypt”. I don’t see a way to create an account. The Getting Started page mentions certbot, but nothing else. Running over to pimylifeup, they have an article to get the Raspberry Pi up and running with python3-certbot-apache.
Now here is a problem. I just realized that Open WebUI’s website is running in docker. I’m finding referneces to use nginx and a proxy manager for Nginx (Engine X). Let’s update the docker compose.yaml file and add nginx
cd /opt/stacks/openwebui sudo nano compose.yaml docker compose pull docker compose up -d
services:
open-webui:
image: ghcr.io/open-webui/open-webui:main
container_name: open-webui
volumes:
- ./data:/app/backend/data
ports:
- 3000:8080
extra_hosts:
- host.docker.internal:host-gateway
restart: unless-stopped
nginxproxymanager:
image: 'jc21/nginx-proxy-manager:latest'
container_name: nginxproxymanager
restart: unless-stopped
ports:
- 80:80
- 81:81
- 443:443
volumes:
- ./nginx/data:/data
- ./nginx/letsencrypt:/etc/letsencrypt
Where is the Admin panel? No links to it. Maybe port 81 or 443? 81 it is!
Wait… I don’t have any credentials. It turns out that Nginx Proxy Manager has a quick start page as well. u: admin@example.com p:changeme. It works. Credentials changed. The main thing I’m after is an SSL certificate.
Testing the server reachability is failing. It mentioned an NPM server… ah, right. Nginx Proxy Manager (NPM). Well, I feel silly. Let’s forward those ports to expose the server. The documentation seems to want both ports 80 and 443 pointing to it from the router. Let’s go and update each router to forward the traffic over to the server.
I’m still running into trouble. I can access the site from the first router, so port forwarding is setup correctly. Let’s try to see what hostinger gives us – and let’s try with the IP address. A quick hit against https://periplux.io/llm.php and … timeout. Does my ISP simply have problems with exposing port 80? Port 443 works, but accessing via IP results as an unrecognized name.
cURL error: OpenSSL/1.1.1w: error:14094458:SSL routines:ssl3_read_bytes:tlsv1 unrecognized name
What’s this message from Nginx aout a DNS challenge? Was all of this stuff to expose ports and setup a dynamic DNS unnecessary?
It’s thinking…
Failed.
CommandError: Saving debug log to /tmp/letsencrypt-log/letsencrypt.log
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /tmp/letsencrypt-log/letsencrypt.log or re-run Certbot with -v for more details.
at /app/lib/utils.js:16:13
at ChildProcess.exithandler (node:child_process:430:5)
at ChildProcess.emit (node:events:519:28)
at maybeClose (node:internal/child_process:1105:16)
at Socket. (node:internal/child_process:457:11)
at Socket.emit (node:events:519:28)
at Pipe. (node:net:339:12)
Let’s shell into the docker container and find this file.
docker ps # CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES # 3be164990a9d jc21/nginx-proxy-manager:latest "/init" 43 minutes ago Up 43 minutes 0.0.0.0:80-81->80-81/tcp, :::80-81->80-81/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp openwebui-nginx-1 # cfb03ecdcb70 ghcr.io/open-webui/open-webui:main "bash start.sh" 2 days ago Up About an hour (healthy) 0.0.0.0:3000->8080/tcp, [::]:3000->8080/tcp cfb03ecdcb70_open-webui docker exec -it 3be164990a9d bash cat /tmp/letsencrypt-log/letsencrypt.log
The file is pretty big. Here is what I’ve found.
Certbot failed to authenticate some domains (authenticator: dns-duckdns). The Certificate Authority reported these problems:
Domain: lewismoten.duckdns.org
Type: unauthorized
Detail: Incorrect TXT record “” found at _acme-challenge.lewismoten.duckdns.org
Hint: The Certificate Authority failed to verify the DNS TXT records created by –dns-duckdns. Ensure the above domains are hosted by this DNS provider, or try increasing –dns-duckdns-propagation-seconds (currently 30 seconds).
Incorrect TXT record…. it’s blank. The DuckDNS spec mentions a TXT Record API, and how to change the value. Here is the URL to change the record:
https://www.duckdns.org/update?domains={DOMAIN}&token={TOKEN}&txt={TXT}[&verbose=true][&clear=true]
Let’s see what’s there rite now.
pi@cjpi:/opt/stacks/openwebui$ dig lewismoten.duckdns.org TXT ;; communications error to 127.0.0.53#53: timed out ; <<>> DiG 9.18.28-0ubuntu0.24.04.1-Ubuntu <<>> lewismoten.duckdns.org TXT ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37037 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 65494 ;; QUESTION SECTION: ;lewismoten.duckdns.org. IN TXT ;; ANSWER SECTION: lewismoten.duckdns.org. 60 IN TXT "" ;; Query time: 1033 msec ;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP) ;; WHEN: Wed Oct 23 22:07:54 UTC 2024 ;; MSG SIZE rcvd: 64
Yes… it’s blank. Can we update it?
curl https://www.duckdns.org/update?domains=lewismoten&token={TOKEN}&txt=hello-world&verbose=true
Knock Out – I got a KO response. I copied the token from the website. Does it want the full domain instead? Still knocked out.
I’m looking up what other people run into. I can specify the propagation seconds to wait for the DNS record to be updated. Second – I can add wildcards for the certificate so that it will work with chat.lewismoten.duckdns.org and nginx.lewismoten.duckdns.org. Let’s see if this works.
And I’m still getting errors. Is the problem with DuckDNS itself? I can’t update the record, so why wold nginx be able to update the record?
So I’m in a weird predicament here. It seems my ISP blocks requests to port 80, and DuckDNS has problems updating TXT records. I have control over a few domains. Can I use one of my domains to do this? Let’s create an “A” record of “home” and “*.home” pointing to my home ip.
Well, it seems that the DNS provider list doesn’t have BlueHost or Hostinger. However, I do see Dynu in the list, and it wants a token. Dynu has a section for API Credentials. Let’s try it…
It took about a minute, but it went through!
Well, since we are using Dynu, we need to change our chron job to update the IP address for the cjpi.freeddns.org record at Dynu.
mkdir ~/dynu cd ~/dynu nano dynu.sh chmod 700 dynu.sh crontab -e ./dynu.sh cat dynu.log
dynu.sh
MY_USERNAME="lewismoten@gmail.com"
MY_PASSWORD="{{password}}"
MY_HOSTNAME="cjpi.freeddns.org"
MY_IPv4=$(curl -s https://api.ipify.org)
MY_IPv6="no";
curl -u MY_USERNAME:MY_PASSWORD "https://api.dynu.com/nic/update?hostname=$MY_HOSTNAME&myip=$MY_IPv4&myipv6=$MY_IPv6"
echo "$(date) - IP address updated to: $MY_IPv4" >> dynu.log
crontab
*/5 * * * * ~/dynu/dynu.sh >/dev/null 2>&1
Hmm… after a few attempts, I got it to work with the old API over basic authentication. Ugh…
Installing the Certificate
We exposed our service to the internet by ensuring it can always be reached by the same IP on our network, setup port forwarding for traffic, contacted our ISP to allow incoming traffic, setup a dynamic DNS domain name, setup a cron job to keep the DNS record updated, and acquired a certificate. I’m exhausted just saying all of that. We are at the very last part. We now need to serve Open WebUI’s content with the certificate.
What else does nginx do? Let’s see what hosts are all about. Hosts -> add proxy host. I think we can use the container_name from compose.yaml for our hostname. I’m going to try with the following settings:
It’s not working… Let’s try enabling HTTP/2 Support. No. I notice that Ngix Proxy Manager has a link under “Source”, and it’s to the site without any protocol info – which means it’s trying to access everything over port 80/443. In this case, its https, so port 443. Oh wait… I am missing a “d” in freeddns. That changed things. 502 Bad Gateway.
Doh! I realized what it was. Looking at compose.yaml, Open WebUI runs on port 8080 internally.
Guess What?
Chicken Butt!!
It pasted everything in!
Here is a dumb question… what if I go to cjpi.local over 443? Nothing.
I think I’m catching onto whats happening. I think (I could be wrong), that nginx looks at the host header of http/https requests, and then acts as a proxy between different IP addresses/ports. That’s why it depends on port 80/443 to be open. It sits in the middle of the requests. If I want to add more services, I need to get a wildcard setup with Dynu and get a certificate issued for it.
Let’s see… I can already hit http://what-ever-i-want.cjpi.freeddns.org:3002/ and reach Open WebUI
Let’s get our wildcard setup for certificates.
Hmm… I think Ideally, I want both certificates to appear as one entry.
Let’s see if we can setup nginx as a sub-subdomain.
Let’s see if I can hit this… none of the routers are setup to forward port 81. Yea… I’m getting errors. I need to either open op that port, or setup some kind of DNS… holdup. It did have an error, but now I got through? But its serving the page from port 80. Wait… but how? The ISP is blocking 80. What is going on. And my phone can reach it too. I figured it out. I had enabled HTTP/2 Support. Disabling it, I was then able to get into it via my phone once I refreshed.
So… we are setup. Wow. What a crazy ride. And to think, with all the stuff nginx offers, I don’t need to keep opening new ports for any new service.
Wait… if nginx is accessible via port 80 now, does that mean that the normal validation works?
Well that’s something… let’s try it out.
Ooff!
Well, let’s take a look at where we are at.
Recap
One thing of note – I don’t have a certificate for https://cjpi.local – I think I would need to create a local certificate authority to issue a certificate, or self sign a certificate to get that to work. However, the problem I was attempting to solve was to have the site running over https – and it is. So to recap, this is a short list of what I went through to do it:
- Setup NAT/port forwarding on WiFi routers
- Setup IP-MAC binding for DHCP on WiFi routers
- Contact ISP to allow inbound traffic
- Setup a Dynamic DNS domain name
- Setup Cron job to update Dynamic DNS service
- Setup a reverse proxy server (nginx)
- Create certificates for domain and wildcards via Let’s Encrypt
- Setup a reverse proxy to redirect to different internal services
