Well, here we are again, going over the considerations of different NAS software, and looking at their strengths. I was originally thinking that the two major contenders were TrueNAS and Unraid. I briefly looked at Open Media Vault (OMV), but the interface looked fairly simple and limited. As I started specking out a budget NAS operated by a Raspberry Pi, I quickly caught on that OMV was actually a competitive option, as its the primary software used to manage Raspberry Pi’s since TrueNAS and Unraid only support x86 CPUs using Intel or AMD, and do not support the ARM architecture that Raspberry Pi’s and many single board systems use.

I’ve seen a post where someone got TrueNAS to work on a Raspberry Pi after making some modifications, but it an hour-long boot process before the server was available online. So let’s look again at the strengths of each NAS software.

	TrueNAS	Unraid	Open Media Vault
Cost	$0	Multi-Tiers: $49 + $36/year, $109 + $36/year, $249 lifetime	$0
License	Open-Source BSD	Proprietary: Basic, Plus, Pro	Open-Source GPL
OS	Yes	Yes	No. Install on Debian/Ubuntu
Architecture	x86 64-bit	x86 64-bit	x86 64-bit, ARM 32/64-bit
File System	CORE/SCALE: ZFS (default) SCALE: EXT4, XFS, Btrfs	XFS (default) Btrfs (for cache drives) ReiserFS (deprecated for XFS)	EXT4 (default) XFS: primarily for large files/video Btrfs: snapshots & integrity checks. ZFS: via plugins. complex setup. JFS: uncommon NTFS: external drives
Minimum RAM	8 GB 16 Recommended 32 for high performance, VMs, ZFS data deduplication, but 64 recommended. Ideally 1 GB per 1 TB storage in ZFS.	2 GB 4-8 Recommended for docker containers, file sharing, and light media streaming. 16 for multiple docker containers, VMs, and Plex media streaming	1 GB 2-4 Recommended 8 for heavy usage or ZFS 8-16 for Docker and VMs
OS Medium	Any internal storage other than USB	USB Flash Drive	Internal storage Avoid microSD/USB eMMC preferable
Drives	Smallest capacity governs max utilized. 1x1TB + 5x2TB = 6TB available, 5TB unused.	Can mix & max disks 1x1TB + 5x2TB = 11TB available	Can mix & max disks with ZFS, but not recommended as striping to distribute data is not optimal and parity data isn’t distributed evenly across all disks when using RAID-Z/Z2
Target	Hobbyists, Small & Large Business	Low-End/repurposed equipment with limitations	Low Budget

File Systems

So – file systems are a big deal. Various file systems are supported by default in different operating systems, and it seem that these different platforms have a particular affinity to one over the other as the default choice. Let’s take a look at the differences.

• ZFS: Zettabyte File System. Developed by Sun Microsystems in early 2000s. Used for high scalability and data integrity with checksums to ensure that corrupted data (bit rot) can be detected and repaired. You can combine multiple storage devices into one or more pools, in which each pool can be configured as a redundant array of independent disks (RAID), where one or more disks could fail without losing data. A raid can also be configured to improve the read/write speed of data. It also has snapshots and clones, allowing for point-in-time copies of data without the overhead of storing the entire file systems content with each snapshot. You are only saving a copy of what has changed since the last snapshot. The system has built-in compression optimized for speed. The disadvantages are the dependency on large amounts of RAM and complexity in setting it up to utilize optimal configurations with cache drives and secondary logs.
• XFS: Extended File System. Developed by Silicon Graphics in ’93. Designed for high performance with large files and parallel read/write operations. Uses journaling to protect against corrupted data during crashes and power failures. Great at utilizing space and scalability. Natively lacks snapshots, but external tools can add that capability. Not as robust as ZFS, and not used as much as EXT4.
• EXT4: Fourth Extended File System. Developed by Theodore Ts’o and Linux kernel developers in 2003. First EXT originally developed by Rémy Card in 1992. Widely used in the Linux community and the default file system for many Linux distributions. Provides journaling with checksumming for further reduce corruption to improve data integrity and crash recovery. Expands upon directory indexing supporting millions of files. Has features to reduce fragmenting files across disks. Supports files up to 16 TB in size and file systems up to 1 EB (1 million times the size of a terabyte). Supports features from EXT3 (journaling, directory indexing via B-tree) and EXT2 (block allocation, inode metadata such as permissions, ownership, timestamps) without required reformatting. Stable and well tested with good performance. Easier to manage than ZFS. Lacks advanced data integrity features, snapshots, and cloning. Limited scalability compared to ZFS (ie – big data centers have large storage capacity)

My primary focus is on data integrity, but I still desire to be able transfer files fast. ZFS seems like the likely system to satisfy my goals. OMV appears to support ZFS via plugins. I’m curious if TrueNAS would be able to work with a ZFS file system created by OMV if/when I upgrade to an x86 processor.

eMMC Storage

Microsoft Designer: Image Creator Prompt

A man is looking at an object that appears to be a key. In reality, it’s a PCB board that fits inside a microSD slot on one end, and has an eMMC microchip. On the table before him, he is building a NAS with a raspberry Pi and five 3.5″ hard drives. Spread out on the table are various storage devices that he’s been considering such as external hard drives, usb flash drives, 2.5″ SSD drives, M.2 SSD sticks, MicroSD cards, and more. He’s been calculating the speed of the different storage devices and determining how many devices can be used to build a budget system.

eMMC is a new one to me. I stared looking into what this is. eMMC stands for Embedded MultiMediaCard. It combines flash memory with a memory controller as one chip, being optimal for small devices with limited space for internal components. Prior to its popularity, people had external flash memory for their devices. It’s slower than an SSD, but typically faster than most microSD cards due to the memory controller optimizing its performance. It has a low cost and reliable.

eMMC overall has a high endurance compared to microSD cards. It can sustain it’s performance over long periods of time compared to microSD, and has a better read/write performance when accessing random data.

Reflecting back, I’m trying to think if I’ve worked with eMMC. I’ve worked with a few EPROM chips, but this seems different as if it is both read/writable. I’ve worked with variations of Arduino, but that uses built-in flash memory of a few KB. An ESP32 board might offer higher memory, but its still not eMMC. The Raspberry Pi doesn’t have this, but the compute module does with 8, 16, and 32 GB capacities. The compute module is a minimalistic Raspberry Pi without the IO and power connection. A separate I/O board is available to interact with it like a normal Raspberry Pi. There is also a microSD EMMC Module that looks like a key for the Pi that offers 16 & 32 GB of eMMC, but its speed is limited by the microSD interface on the Pi.

Using EMMC via microSD seems nice for convenience, and has a longer lifespan than a standard microsSD card. The microSD interface is limited to UHS-1, so write speeds would be around 30 MB/s max, rather than 200 MB/s. Even at those speeds, EMMS 5.1 is theoretically capable of writing at 250 MB/s. It seems that using a USB 3 port with an EMMC adapter would have been a better option to take advantage of the speed that EMMC offers.

MKS EMMC 32G Memory Expansion Card & MKS EMMC Adapter USB 3.0 Card Reader

$17

If you are going to use a USB 3 port, then an external SSD drive would probably be the better option anyway as you could write at 400 MB/s, and it’s fairly obvious to most people how to use an external USB drive. The most optimal solution would be if the Raspberry Pi already offered eMMC onboard – which is what the Compute module does.

Here is a generalized comparison of various storage mediums and theoretical speeds. These speeds can’t be trusted though. I’ve quickly thrown this together and haven’t followed through on cross referencing or confirming speeds. Some of the speeds are typical speeds rather than the theoretical max. These are ballpark estimates to get a grasp on general speeds of various storage medium. You need to check with the manufacturer of all storage medium as not every device is capable of delivering the maximum values, or to maintain them over a sustained period of time.

Storage / Interface	Notes	Bus Speed (theoretical)	Max Read	Max Write
eMMC 4.41		200 MB/s	200 MB/s	50 MB/s
eMMC 4.5		200 MB/s	200 MB/s	100 MB/s
eMMC 5.0		400 MB/s	400 MB/s	150 MB/s
eMMC 5.1	Raspberry Pi Compute Module	400 MB/s	400 MB/s	250 MB/s
eMMC 5.1 (enchanced)		600 MB/s	600 MB/s	300 MB/s
Plug and Play EMMC Module: Pi 5	Limited by microSD interface	400 MB/s	45 MB/s	36 MB/s
microSD Class 2		2 MB/s	30 MB/s	2 MB/s
microSD Class 4		4 MB/s	30 MB/s	4 MB/s
microSD Class 6		6 MB/s	30 MB/s	6 MB/s
microSD UHS-I U1		104 MB/s	95 MB/s	10 MB/s
microSD UHS-I U3 (Pi 5)		104 MB/s	100 MB/s	30 MB/s
microSD UHS-II		312 MB/s	300 MB/s	200 MB/s
SSD: USB 2.0		60 MB/s	60 MB/s	30 MB/s
SSD: USB 3.0		625 MB/s	500 MB/s	400 MB/s
SSD: USB 3.1 Gen 1		625 MB/s	500 MB/s	400 MB/s
SSD: USB 3.1 Gen 2		1,250 MB/s	1,050 MB/s	1,000 MB/s
SSD: USB 3.2 Gen 1		625 MB/s	500 MB/s	400 MB/s
SSD: USB 3.2 Gen 2		1,250 MB/s	1,050 MB/s	1,000 MB/s
SSD: USB 3.2 Gen 2 (2 lanes)		2,500 MB/s	2,000 MB/s	1,500 MB/s
SSD: USB 4		5,000 MB/s	7,000 MB/s	7,000 MB/s
HDD: SATA I		150 MB/s	120 MB/s	120 MB/s
HDD: SATA II		300 MB/s	200 MB/s	200 MB/s
HDD: SATA III		600 MB/s	250 MB/s	250 MB/s
SSD: SATA I		150 MB/s	150 MB/s	150 MB/s
SSD: SATA II		300 MB/s	300 MB/s	300 MB/s
SSD: SATA III		600 MB/s	550 MB/s	550 MB/s
M.2 SSD: PCIe 2.0		635 MB/s	1,500 MB/s	1,000 MB/s
M.2 SSD: PCIe 3.0	Raspberry Pi 5	4,000 MB/s	3,500 MB/s	3,000 MB/s
M.2 SSD: PCIe 4.0		8,000 MB/s	7,000 MB/s	5,000 MB/s
M.2 SSD: PCIe 5.0		16,000 MB/s	14,000 MB/s	12,000 MB/s
Pi 1, 2, 3 RAM: LPDDR2		450 MB/s	450 MB/s	350 MB/s
Pi 4 RAM: LPDDR4		2,133 MB/s	2,133 MB/s	2,000 MB/s
Pi 5 RAM: LPDDR4X		2,133 MB/s	2,133 MB/s	2,000 MB/s

Let’s try to turn that data into information.

Well, that gives sort of a broad picture to compare speeds of different mediums when sorting by the read speed. Let’s see if we can drop a few of the versions, as some of this is just historical comparisons, and the RAM and PCIe 4/5 doesn’t apply to the Raspberry Pi.

Ah… that is much better. All of the “noise” is gone. We just see the types of storage and interfaces that apply to us. Surprisingly, a microSD card is slower than a hard drive via SATA III. Being flash memory, I thought it was much faster. In this case utilizing eMMC would be optimal either through a direct line, USB 3, SATA III, or PCIe 2.0+. I could find a way to use eMMC for the OS, but an SSD over USB seems the more optimal choice to boot the Raspberry Pi from a USB Drive. Even if I choose to use the compute module with the eMMC already integrated, if it fails, then I’m stuck in a position where I can not replace it.

Since I’ll be using HDDs for my data array with ZFS, I’d like to use SSDs via USB 3, SATA, or PCIe as a Cache drive and Secondary Log. These specific roles require read/write speeds faster than the hard drives themselves to see any significant improvement in speed. The cache drives serve as a place to store data that’s still important, but not as statistically important that the files cached in RAM waiting to be served. The Secondary Log (SLOG) serves as a place to write information about files about to be read/written, rather than writing into the data pool itself. Without the SLOG, the HDD is written two a few times before data has been committed.

So – that means I need 3 SSDs – OS, Cache, SLOG. The Raspberry Pi only has two USB 3.0 ports. If I want a 2.5 Gbps network connection, that’s one less USB port available. I could get a hub, but then I’m sharing the speed and reducing available power (although SSD’s don’t use much power). Since the OS isn’t used heavily, it could be paired with one of the drives.

So… what about splitting the PCIe 3.0 connection? There are adapters that can split it into two or 4 lanes. Theoretically, it seems that I could support up to 20 SATA connections using a few of the SATA Shields.

Well, let’s think about that. The Raspberry Pi 5 has PCIe 3.0. That’s a 4,000 MB/s bus. Split between 20 drives being accessed at the same time, that’s 200 MB/s per drive. HDD usually communicate at speeds of 250 MB/s. If all drives were being written to, or read from at once, the bus would be saturated at 100%. I’m not sure about hardware communication on the bus, but given the role of cache drives and SLOGs, maybe those drives would be talking directly to the other, reducing the saturation. In general, you would have a bottleneck. Reducing it down to 15 SATA drives gives us an average of 267 MB/s for each drive – which meets the HDD max read/write speeds. So 15 seems plausible…

What about the PCIe itself? What happens with the splitter? Does it downgrade PCIe version, reduce the lanes, or do some kind of fancy switching between whats active? Let’s look at some of the product details:

• Dual PCIe Expansion — Effortlessly expand your Raspberry Pi 5’s FPC PCIe interface into two, providing users with the convenience of connecting multiple PCIe devices simultaneously for increased productivity and versatility.
• Daisy-Chaining Capability — Connect multiple B12 units in a cascading fashion to create a multi-tiered PCIe configuration, allowing for expanded connectivity options and customized setups tailored to your specific needs.
• Enhanced Connectivity — Unlock the potential to connect a wide range of PCIe devices, such as SSDs, GPUs, network cards, and more, expanding the capabilities of your Raspberry Pi 5 for diverse applications and projects.
• Robust and Reliable Design — Built with quality and durability in mind, the B12 Raspberry Pi HAT ensures stable and reliable performance, providing a dependable solution for your PCIe expansion needs.
• Dual Board Expansion: This feature allows for the connection of two additional Pineberry Pi boards, providing users with the flexibility to expand their projects’ scope and functionality.
• ASMedia PCIe Switch Integration — The board incorporates an ASMedia PCIe Switch (Gen2 variant), facilitating reliable and efficient data transfer between connected devices.
• Included FPC Ribbon Cables — Three FPC ribbon cables are provided, simplifying the connection process and offering flexibility in setup and configuration.

Daisy chaining – so we can just keep adding SATA shields to have 20, 100, 1000 drives… ASMedia PCIe Switch integration (Gen2 variant). Well, I think that’s the clue I was looking for with switching. What does that entail, and does it effect overall speed?

ASM1824-PCIe Gen2 Packet Switching Chips，24 Lane / 12 Port

PCIe Gen2 Packet Switch

ASMedia PCIe product ASM1824, a low latency, low cost and low power 24 lane , maximum 12 downstream ports packet switch. With upstream PCIe Gen2x8 bandwidth, ASM1824 can enable users to build up various high speed IO systems, including server, system storage or communication platforms.

Apparently there is a limit of 12 devices, and it looks like it downgrades to PCIe 2.0. So – that’s a max bandwidth of 450 MB/s. More than enough to fully saturate 2.5 GB/s Ethernet transfers over USB with an adapter.

A close inspection of the RP1 specs indicate it uses 4 lanes of PCIe Gen 2.0 from the BCM2712 SoC… meaning the board actually has five lanes of PCIe bandwidth available. But only one lane is exposed to the user via the FPC connector pictured above.

Testing PCIe on the Raspberry Pi 5

Ok – why is he talking about PCIe 2.0 when most people talk about 3. Is it because 3 isn’t enabled by default?

And the connection is certified for Gen 2.0 speed (5 GT/sec), but you can force it to Gen 3.0 (10 GT/sec) if you add the following line after:

dtparam=pciex1_gen=3

Question answered. So PCIe 3 isn’t certified… what does that mean in terms of enabling it? Are there any limitations or down sides?

I did that in much of my testing and never encountered any problems, so I’m guessing that outside of more exotic testing, you can run devices at PCIe Gen 3.0 speeds if you test and they run stable.

So yea… seems he’s wondering the same thing.

I was able to get about 450 MB/sec under the default PCIe Gen 2.0 speed, and very nearly 900 MB/sec forcing the unsupported Gen 3.0—almost exactly a 2x speedup.

Ok – that changes my calculations drastically. I had PCIe 3 at 3,500 MB/s. That also affects the average speed per SATA port as well. I just don’t have real numbers to work with, or know where to get them – specifically for the Raspberry Pi. A HDD can read/write at about 250 MB/s. With PCIe 3 at 900 MB/s, it’s fully saturated if all drives are accessed at once. However, if we have a 2.5 Gb/s ethernet connection, we are limited to just 250 MB/s in data transfer anyway unless something on the Raspberry Pi is doing some intensive read/write operations (Scrubbing? Deduplication? Resilvering?). Strictly as a NAS, the saturation of the PCIe port may not be that big of an issue.

Once in place, the card was immediately recognized, and at PCIe Gen 3, I was able to get 5.5-6 Gbps. I presume there may be a 10G NIC out there that will squeeze out closer to 10 Gbps through a Gen 3 x1 link, but I haven’t found it.

So that’s about 550 MB/s throughput. Good to know that speeds higher than 1 Gb/s are not limited to USB 3.0 port alone. However, would a 5.5 Gb/s transfer be hindered by having both the NIC and SATA ports on the same PCIe lane?

I didn’t have time yet for full debugging, though, and the same thing happened with the 9460-16i, so I will take up the gauntlet again soon. I’d love to try building another DIY NAS on the Pi 5 to see if I can resoundingly break the gigabit-write-over-the-network barrier (Samba held back speeds on the BCM2711!).

I’ve seen videos of the Pi 5 getting speeds up to 2.5 GB/s via USB 3 adapter to Ethernet and SATA drives over PCIe, so this barrier seems to have been broken.

So I just went back to the Manufacturers website for the Asmedia chip for the PCIe Gen2 Packet switch. Apparently you can scroll down and see more chips. It just sends the same signal from the motherboard to all devices that have been split. Wait… so PCIe is just a bus, and a ton of devices can be connected to that same bus? What’s the purpose of the chip then? Does it act similar to a managed switch or router? Let’s ask Google Gemini:

While it might seem logical to connect multiple devices to a single lane, there are several reasons why this isn’t practical:

1. Data Congestion: Multiple devices sharing a single lane can lead to significant data congestion, especially under heavy load. This can result in reduced performance, increased latency, and potential data corruption.
2. Signal Integrity: PCIe is a high-speed, complex protocol. Sharing a lane between multiple devices can introduce signal integrity issues, such as crosstalk and impedance mismatches, which can further degrade performance and reliability.
3. Device Compatibility: Different devices may have different power requirements, timing constraints, and other specific needs. Sharing a lane can make it difficult to ensure that all devices receive the necessary resources.
4. Future Expansion: By using a PCIe switch like the ASM1182e, you can easily add additional devices in the future without needing to replace the existing devices or the motherboard.

In essence, the ASM1182e chip allows for a more flexible and efficient use of PCIe lanes, enabling the connection of multiple devices while maintaining optimal performance and reliability.

I started going on asking about the theoretical max number of drives that the Pi with the chip could support, and it kept recommending SSD’s and Raid controllers, or dancing around vague expressions such as “handful” and “large”, but eventually I was able to get it to determine what the Raspberry Pi 5 “might” be capable of supporting, in terms of the number of HDDs and the forced PCIe 3.0 compatibility.

Given the limitations of a single PCIe 2.0 x1 lane, even with forcing it to Gen 3.0, a small array of 2-5 drives would be a more realistic estimate for a Raspberry Pi 5.

Remember, this is a rough estimate and actual performance will depend on various factors such as the specific SATA controller, the efficiency of the PCIe switch, and the workload on the system. For larger RAID arrays, a dedicated hardware RAID controller with multiple PCIe lanes would be a more suitable solution.

So this response seems like the likely outcome I was thinking of, but wanted to see what the Pi was capable of. It all comes down to bandwidth of the PCIe lane, the drives themselves, and the network. In regards to raid controllers, I don’t like them. I don’t feel like I can trust them, as you are locked in once you start using them. If the hardware fails, you have to hope that the vendor still makes a compatible card or find one on the used market. They can also become inefficient over time as new standards become available. CPU’s have sped up over the years that enable software to run just as fast or faster, and it’s much easier to update software. Although hardware now has firmware these days. However… I can see how a raid controller can reduce the overall bandwidth on the PCIe bus. Rather than software writing the same data and parity information to multiple drives for redundancy, the software just sends one copy of the data to the raid controller itself, and the controller takes over doing all of the parity calculations and sending the data to multiple drives with it’s own direct connections to the drives.

Ok, so back to the OS drive. I hate to say it, but maybe the EMMC Module via microSD slot isn’t such a bad idea. Sure – we are severely limited in speed going directly through the microSD port. However, it’s just the OS. Other than installation and updates, it’s not data intensive, and doesn’t end up using a USB or PCIe slot. The primary benefit here is that it has a better longevity vs microSD cards, and speed is only limited by the Pi’s hardware, rather than the storage medium. The other benefit is that it’s easy to replace when it fails, either with a spare MicroSD card, or having a spare RasPiKey prepared to take over. If I went with the compute module – the eMMC can not be removed, and I’d need another board to get to the I/O. It can expose the PCIe port, but its slower. I believe the hardware of the Compute 4 module is based on the Raspberry Pi 4 specifications. Either the RasPiKey, EMMC Module, or a similar EMMC to MicroSD adapter is the optimal solution here.

The thing that gets me is that the benchmark tests they post on the product page of the EMMC Module makes it appear much slower than microSD cards theoretical read/write max speeds. However, they end up showing that most microSD cards are slower as well. My numbers are way off anyway. Who should I trust? Well, someone did a comparison of the RasPiKey vs 14 SD Cards, but it was done for the Raspberry Pi 4. Well, lucky for us, the Pi 4 and 5 use the same MicroSD interface standard of UHS-I. While the microSD cards look like they perform best with sequential writes, they didn’t do well when it came to random read/write. The key seemed to outperform all microSD cards with all metrics. Here are a few of the highlights from his numbers, and later I added the EMMC Module numbers from their product page – but I’m uncertain about the random read IO/s:

	RasPiKey	EMMC Module	Sandisk Extreme Pro 64GB A2	Brave Eagle 32GB	Alert Seal	Kingston 64 GB Canvas Go Plus A2	Sandisk 64 GB Extreme A2
Sequential Write KB/s	43,059	36,944	38,035	21,896	27,970	32,251	32,204
Random Write IO/s	6,625	6,315	969	1,123	1,095	1,521	894
Random Read IO/s	5,538	10,806	2,548	3,825	3,796	3,621	2,077

Visually…

So yes – we will see some improvements in speed. Given that it’s just the OS to boot up on a freshly installed eMMC chip, I’m assuming we will mostly be dealing with sequential reads. That’s specifically the one thing that I don’t see with his benchmark tests. However, given all other metrics, the RasPiKey outperforms all other microSD cards that he tested in all categories.

You know, I bet that’s the main issue. When installing an operating system, who knows the order of files to be loaded once it is in operation? What would be fairly impressive is if there was a way to monitor the order in which files were accessed, and then “bake” that order somewhere to be one large sequential read for the next time that the operating system boots up. It would sort of be like defragging in a sense – but only to increase the boot. Perhaps it could be applied to other common tasks that may need to access the same files each time they start, such as spinning up a service, virtual machine, or docker container.

Well, the other day, I planned to get a 128 GB MicroSD card at $9. The size was a bit overkill. If we look at the RasPiKey, they come in 16 and 32 GB for $25 and $37. Having 16 GB would be sufficient for the bootable os, system updates, and logs. 32 GB would be more comfortable. That $9 line item just increased to $37.