Build_ZFS_with_QAT_supported

Build ZFS with QAT supported

0. System environment

OS: Ubuntu 16.04 LTS

Kernel: linux 4.4.1

QAT hardware: DH895XCC

QAT driver: qatmux.l.2.6.0-60

ZFS: 0.7.0

1. Build & install QAT driver

  1. Download QAT driver from intel:

    1
    wget https://01.org/sites/default/files/page/qatmux.l.2.6.0-60.tgz

    Note: I tried to install qat1.7 and qat1.6, but I failed.

  2. Create a directory and extract driver to here:

    1
    2
    3
    mkdir QAT
    cd QAT
    tar zxvf ../qatmux.l.2.6.0-60.tgz
  3. There are some places need to modify:

    1. QAT1.6/quickassist/utilities/osal/src/linux/user_space/OsalCryptoInterface.c:

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      77 OSAL_STATUS
      78 osalHashSHA1(UINT8 *in, UINT8 *out)
      79 {
      80 SHA_CTX ctx;
      81 if(!SHA_Init(&ctx))
      82 {
      83 return OSAL_FAIL;
      84 }
      85 SHA1_Transform(&ctx, in);
      86 memcpy(out, &ctx, SHA_DIGEST_LENGTH);
      87 return OSAL_SUCCESS;
      88 }

      change SHA_INIT to SHA1_INIT.

    2. QAT1.6/quickassist/adf/drivers/common/linux/src/adf_proc_debug.c:

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      static int adf_debug_show(struct seq_file *sfile, void *v)
      {
      debug_file_info_t* file_info = sfile->private;
      if (file_info && file_info->seq_read && file_info->page) {
      int ret = 0, old_offset = file_info->offset;
      file_info->offset =
      file_info->seq_read(file_info->private_data,
      file_info->page, PAGE_SIZE - 1,
      file_info->offset);
      // ret = seq_puts(sfile, (char*)file_info->page);
      seq_puts(sfile, (char*)file_info->page);
      if (ret) {
      /* run out of space - need to reprint */
      file_info->offset = old_offset;
      }
      }
      return 0;
      }

      removeret.

    3. QAT1.6/quickassist/adf/user/user_proxy/src/adf_user_ETring_mgr_dp.c:

      Remove INLINE before these functions:

      • icp_adf_getQueueNext

      • icp_adf_getSingleQueueAddr

      • icp_adf_updateQueueTail

      • icp_adf_pollQueue

      • icp_adf_queueDataToSend

      • icp_adf_getQueueMemory

  4. Enter root mode:

    1
    sudo su
  5. Run installer:

    1
    ./installer

    choose build then install.

  6. Put in-tree qat modules to blacklist by add following items to /etc/modprobe.d/blacklist.conf:

    1
    2
    3
    4
    5
    blacklist qat_dh895xcc

    blacklist intel_qat

    blacklist usdm_drv

2. Build & install ZFS

  1. Get spl source code from github:

    1
    git clone https://github.com/zfsonlinux/spl
  2. Compile spl code:

    1
    2
    3
    4
    5
    cd spl
    git checkout master
    sh autogen.sh
    ./configure
    make -s -j$(nproc)
  3. Get zfs source code from github:

    1
    git clone https://github.com/zfsonlinux/zfs
  4. Optional: To verify it is QAT that take effects, add some printk to zfs source code:

    module/zfs/qat_compress.c:

    1
    2
    3
    4
    5
    qat_compress(qat_compress_dir_t dir, char *src, int src_len,
    char *dst, int dst_len, size_t *c_len)
    {
    printk("calling qat_compress...\n");
    int ret;
  5. Compile and install zfs:

    1
    2
    3
    4
    5
    6
    cd ../zfs
    git checkout master
    sh autogen.sh
    ./configure --with-qat=<parent directory of QAT source>/QAT/QAT1.6
    make -s -j$(nproc)
    make install

3. Verify ZFS & QAT

  1. Simulate a disk with file:

    1
    2
    3
    4
    5
    6
    dd if=/dev/zero of=zfs_disk bs=1M count=4096
    losetup /dev/loop0 zfs_disk
    # verify zfs status
    zpool status
    # set compression for zfs_test
    zfs set compression=gzip zfs_test
  2. Create a pool:

    1
    2
    zpool create zfs_test /dev/loop0
    zpool status
  3. Set compression for zfs_test:

    1
    zfs set compression=gzip zfs_test
  4. Verify zfs_test parameters:

    1
    zfs get all
  5. Write something to zfs_test to verify QAT is working:

    1
    cp qatmux.l.2.6.0-60.tgz /zfs_test

    Note: We should not verify by writing zero to zfs_test, because zfs will check this situation and handle it directly without calling compression function. So, don’t run something like this:

    1
    dd if=/dev/zero of=a bs=1M count=1024
  6. If everything is OK, you will see “calling qat_compress…” in dmesg output.