Is Your FreeNAS Pool Encrypted With AES-XTS 128?

I have recently upgraded my pool of mirrors to double my storage space and, upon rebooting the machine, I received this email from the FreeNAS server:

freenas.local kernel log messages:
> GEOM_ELI: Device ada0p1.eli destroyed.
> GEOM_ELI: Detached ada0p1.eli on last close.
> GEOM_ELI: Device ada2p1.eli created.
> GEOM_ELI: Encryption: AES-XTS 128
> GEOM_ELI: Crypto: software
> GEOM_ELI: Device ada1p1.eli destroyed.
> GEOM_ELI: Detached ada1p1.eli on last close.
> GEOM_ELI: Device ada3p1.eli created.
> GEOM_ELI: Encryption: AES-XTS 128
> GEOM_ELI: Crypto: software

-- End of security output --

The interesting thing is that I have never enabled encryption for this pool, so this looked weird.

Continue reading

What Does Freed UMA keg Mean in FreeNAS

FreeNAS is full of secrets, and some of its log messages are quite cryptic, even for the more experienced part of the community. The most recent one I have stumbled upon myself is the Freed UMA keg message, always followed by something along the lines of Lost n pages of memory.

Continue reading

What Are bridge And epair Interfaces In FreeNAS?

I was taking a look at the network configuration of one of my FreeNAS machines, and I noticed a few network interfaces that I did not immediately recognize, something I am sure I had not created manually myself. These interfaces are named bridge0, epair0a and epair1a.

Continue reading

Disappearing Unreadable Sectors In FreeNAS Drives

Some weeks ago I experienced something a bit weird on my FreeNAS system. I received an email at 02:40 AM with this content:

The volume Disk1 (ZFS) state is DEGRADED: One or more devices has been removed by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state.

From the kernel log messages (also received via email) about half an hour later, I see that the first drive was disconnected:

ahcich0: Timeout on slot 26 port 0
ahcich0: is 00000000 cs 04000000 ss 00000000 rs 04000000 tfd c0 serr 00000000 cmd 0000fa17
(ada0:ahcich0:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
(ada0:ahcich0:0:0:0): CAM status: Command timeout
(ada0:ahcich0:0:0:0): Retrying command
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <ST31000333AS SD35> s/n 6TE03QP9 detached
GEOM_ELI: Device ada0p1.eli destroyed.
GEOM_ELI: Detached ada0p1.eli on last close.
(ada0:ahcich0:0:0:0): Periph destroyed

Not a problem, I thought, luckily this is a mirrored volume so I will just put in another drive and I won’t lose anything (hoping that nothing goes wrong during the resilvering process of course). First, though, I took a backup of my data and, while doing so, I receive a third email:

Device: /dev/ada0, 10 Currently unreadable (pending) sectors
Device: /dev/ada0, 10 Offline uncorrectable sectors

This really looked like the drive’s life was coming to an end. At the end of the backup, however, I decided to play around with the system a bit as I had never seen a drive fail before in FreeNAS. So I force a scrub and I run short and long smart test on the drive. Boy was I surprised.

Continue reading

How To Configure FreeNAS To Successfully Boot In XenServer

From the FreeNAS documentation page on VMware ESXi:

If you are running ESX 5.0, Workstation 8.0, or Fusion 4.0 or higher, additional configuration is needed so that the virtual HPET setting does not prevent the virtual machine from booting.

I have found that this holds true for Citrix XenServer as well. If you don’t touch the HPET setting, FreeNAS 9.3 will fail to boot on XenServer 6.5

I have also found this bug report against FreeNAS 9.3 and XenServer 6.2, which is currently in resolved status. However, since I have experienced this same issue on XenServer 6.5, I have decided to put together some screenshots and instructions on how to solve this issue and allow FreeNAS 9.3 to boot successfully on XenServer 6.5.

There are two different phases to allow this to work:

  1. Disable HPET from GRUB to allow FreeNAS to boot successfully the first time
  2. Disable HPET within FreeNAS to avoid stumbling upon the same issue on following reboots

Continue reading

How To Deal With Checksum Errors In ZFS – FreeNAS

Some days ago I received an email from one of my FreeNAS boxes letting me know that a pool scrub was starting:

starting scrub of pool 'Disk1'

About 3 hours after this, I received another email with an excerpt from the kernel logs:

freenas.local kernel log messages:
(ada1:ahcich1:0:0:0): READ_FPDMA_QUEUED. ACB: 60 d8 80 df 4d 40 46 00 00 00 00 00
(ada1:ahcich1:0:0:0): CAM status: ATA Status Error
(ada1:ahcich1:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
(ada1:ahcich1:0:0:0): RES: 51 40 98 df 4d 40 46 00 00 c0 00
(ada1:ahcich1:0:0:0): Retrying command
(ada1:ahcich1:0:0:0): READ_FPDMA_QUEUED. ACB: 60 d8 80 df 4d 40 46 00 00 00 00 00
(ada1:ahcich1:0:0:0): CAM status: ATA Status Error
(ada1:ahcich1:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
(ada1:ahcich1:0:0:0): RES: 51 40 98 df 4d 40 46 00 00 c0 00
(ada1:ahcich1:0:0:0): Retrying command
(ada1:ahcich1:0:0:0): READ_FPDMA_QUEUED. ACB: 60 d8 80 df 4d 40 46 00 00 00 00 00
(ada1:ahcich1:0:0:0): CAM status: ATA Status Error
(ada1:ahcich1:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
(ada1:ahcich1:0:0:0): RES: 51 40 98 df 4d 40 46 00 00 c0 00
(ada1:ahcich1:0:0:0): Retrying command
(ada1:ahcich1:0:0:0): READ_FPDMA_QUEUED. ACB: 60 d8 80 df 4d 40 46 00 00 00 00 00
(ada1:ahcich1:0:0:0): CAM status: ATA Status Error
(ada1:ahcich1:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
(ada1:ahcich1:0:0:0): RES: 51 40 98 df 4d 40 46 00 00 c0 00
(ada1:ahcich1:0:0:0): Retrying command
(ada1:ahcich1:0:0:0): READ_FPDMA_QUEUED. ACB: 60 d8 80 df 4d 40 46 00 00 00 00 00
(ada1:ahcich1:0:0:0): CAM status: ATA Status Error
(ada1:ahcich1:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
(ada1:ahcich1:0:0:0): RES: 51 40 98 df 4d 40 46 00 00 c0 00
(ada1:ahcich1:0:0:0): Error 5, Retries exhausted

-- End of security output --

Immediately followed by a third email, with a status summary of my pool:

Checking status of zfs pools:
Disk1   928G   494G   434G    53%  1.00x  ONLINE  /mnt

 pool: Disk1
state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
  see: http://illumos.org/msg/ZFS-8000-9P
 scan: scrub repaired 96K in 1h38m with 0 errors on Sun Aug  9 01:38:43 2015

	NAME                                            STATE     READ WRITE CKSUM
	Disk1                                           ONLINE       0     0     0
	  mirror-0                                      ONLINE       0     0     0
	    gptid/4f54a386-8f88-11e4-a049-009c02975356  ONLINE       0     0     0
	    gptid/62701945-eb48-11e4-bdc4-009c02975356  ONLINE       0     0     2

errors: No known data errors

-- End of daily output --

So the summary of the situation is that one of the two drives reported 2 checksum errors, and these are likely connected to the read errors reported by the kernel logs.

Often, this is because the drive is failing, but it could also be a SATA cable issue, or perhaps it could be something temporary. In my case, I was told that there had been a blackout recently, so when you see an output like this it doesn’t necessarily mean you need to go out and replace the disk straight away.

From the FreeNAS error message page:

ind the device with a non-zero error count for READ, WRITE, or CKSUM. This indicates that the device has experienced a read I/O error, write I/O error, or checksum validation error. Because the device is part of a mirror or RAID-Z device, ZFS was able to recover from the error and subsequently repair the damaged data.

If these errors persist over a period of time, ZFS may determine the device is faulty and mark it as such. However, these error counts may or may not indicate that the device is unusable. It depends on how the errors were caused, which the administrator can determine in advance of any ZFS diagnosis. For example, the following cases will all produce errors that do not indicate potential device failure:

  • A network attached device lost connectivity but has now recovered
  • A device suffered from a bit flip, an expected event over long periods of time
  • An administrator accidentally wrote over a portion of the disk using another program

In these cases, the presence of errors does not indicate that the device is likely to fail in the future, and therefore does not need to be replaced.

However, there are some things you can do to find out whether this is indeed a hard drive issue or not:

  1. Clear the error count:
    zpool clear pool_name

    This will reset all the read, write and checksum error counters.

  2. You can either wait for a while, wait for the next scrub to take place or, if you are impatient, you can force one immediately:
    zpool scrub pool_name
  3. Check the output of
    zpool status pool_name

    If the error counters are still 0, the issue you experienced was likely a temporary one. If no issue appears over the next few days, you hard drive should be fine for the foreseeable future.

  4. If you still see errors (or even if you really want to make sure everything is ok), you can check the SMART status of the drive. To run a short SMART check, type
    smartctl -t short /dev/ada0

    To run a long SMART check, type

    smartctl -t long /dev/ada0

    instead. Of course, replace


    with your drive identifier. Note: a short test will take a minute, a long test will take around 4 hours.

  5. Check the output of
    smartctl –a /dev/ada0

    for errors.

  6. At this point, if you don’t see any error messages, your drives might be ok. You can force a scrub again with
    zpool scrub pool_name

    and you should be good.

This is all to say that, even though this type of message usually points to a failing drive, it doesn’t necessarily have to be a hardware issue. Go through this list before deciding to buy a new drive and monitor the status of your system for a while after this happens for the first time to make sure this wasn’t just a temporary issue.

How to Convert a Stripe into a Mirror in FreeNAS

Ideally, with FreeNAS, if you plan on using mirrored drives, you should start directly with 2 physical drives. This holds for every combination of drives actually, because modifying existing vdevs is not allowed (you can only add new vdevs to an existing zpool).

The only exception consists in converting a striped vdev into a mirrored vdev. In my case, I started using FreeNAS in a test machine with a single physical drive, but then decided to add a second drive to keep testing it a little bit more thoroughly. Obviously, one solution consisted in taking a copy of my data, wipe out both disks, create a mirror vdev and then reimport the data from the backup, but why go through the hassle when you can do this quickly and easily from the CLI?

These instructions are taken from a post on the official FreeNAS forum. I have decided to include them here because there are several discussions on the topic on the forum, but not all of them include comprehensive instructions on how to do this.

gpart create -s gpt /dev/ada1
gpart add -i 1 -b 128 -t freebsd-swap -s 2g /dev/ada1
gpart add -i 2 -t freebsd-zfs /dev/ada1


zpool status

and note the gptid of the existing disk

glabel status

and find the gptid of the newly created partition. It is the gptid associated with ada1p2.

zpool attach tank /dev/gptid/[gptid_of_the_existing_disk] /dev/gptid/[gptid_of_the_new_partition]

In FreeNAS, hard drives are normally named ada<n>, with <n> starting from 0 and being incremented by one every time a new hard drive is added. Therefore, if you are only using one hard drive and want to add a second one, your new drive will likely be called ada1.

Also, replace tank with your zpool name.

I vehemently recommend testing this in a virtualized environment before attempting it on your physical system. Dealing with vdevs and zpools is a delicate operation, and you really don’t want to risk losing your data just because of a wrong command.

Note: if the new hard drive already contains some data, you need to wipe it first. This can be easily done from the GUI:

If everything went fine, you should see the resilvering process take place after running a zpool status:


And at the end of the resilvering process you should see something like this:

Screenshot 2015-04-25 16.26.37

© 2018 Daniel's TechBlog

Theme by Anders NorénUp ↑

%d bloggers like this: