AN2007_03_iSCSI < DULG

---+ Using AMCC PPC 440 processors for SAN applications over iSCSI

---++ Abstract

This document describes how to set up AMCC PPC 440 based systems
running DENX Linux for SAN applications over iSCSI transport.

It explains the process of setting up the iSCSI target and initiator
to use RAID arrays created on "katmai" (440SPe) and "taishan" (440GX)
boards as SAN appliances.

Building the iSCSI target and initiator kernel components (included
into the release tarballs) require the access to the kernel source
tree.

---++ Author

This document was written by Sergei Poselenov (sposelenov {at} emcraft {dot} com)

---++ Configuring and building target components:

For the target, the linux-2.6-denx tree was used, version 2.6.21-rc3

   1 Build the target kernel
     %BR%
     The following options should be enabled:
     <verbatim>
"Cryptographic API" (CONFIG_CRYPTO)
"CRC32c CRC algorithm" (CONFIG_CRYPTO_CRC32C)
     </verbatim>
     For a LSI SAS controller:
     <verbatim>
CONFIG_FUSION=y
# CONFIG_FUSION_SPI is not set
# CONFIG_FUSION_FC is not set
CONFIG_FUSION_SAS=y
CONFIG_FUSION_MAX_SGE=128
CONFIG_FUSION_CTL=y
     </verbatim>
     For Linux RAID support:
     <verbatim>
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_LINEAR=y
CONFIG_MD_RAID0=y
CONFIG_MD_RAID1=y
CONFIG_MD_RAID10=y
CONFIG_MD_RAID456=y
CONFIG_MD_RAID5_RESHAPE=y
# CONFIG_MD_MULTIPATH is not set
# CONFIG_MD_FAULTY is not set
# CONFIG_BLK_DEV_DM is not set
     </verbatim>
     Also, enable the SCSI disk option.
     %BR%
     %X% *Note:* Make sure to *disable* the ==CONFIG_DEBUG_MUTEXES== option.
      * [[%ATTACHURL%/katmai_iscsi.config][katmai_iscsi.config]]: iSCSI kernel configuration for "katmai" board
      * [[%ATTACHURL%/taishan_iscsi.config][taishan_iscsi.config]]: iSCSI kernel configuration for "taishan" board
      * [[%ATTACHURL%/uImage_katmai_iscsi][uImage_katmai_iscsi]]: Linux kernel image for "katmai" board
      * [[%ATTACHURL%/uImage_taishan_iscsi][uImage_taishan_iscsi]]: Linux kernel image for "taishan" board

   1 Build the iSCSI Target software (IET):
      * Download the [[http://belnet.dl.sourceforge.net/sourceforge/iscsitarget/iscsitarget-0.4.14.tar.gz][iscsitarget-0.4.14.tar.gz ]]
        tarball from
        http://iscsitarget.sourceforge.net
      * Unpack and apply patches:
         * [[%ATTACHURL%/compat-2.6.14-2.6.18.patch][compat-2.6.14-2.6.18.patch]] - apply with "-R" switch.
         * [[%ATTACHURL%/iscsitarget-ppc.patch][iscsitarget-ppc.patch]]
      * Build:
        <verbatim>
PATH=/opt/eldk/usr/ppc-linux/bin:/opt/eldk/usr/bin:$PATH CROSS_COMPILE=ppc_4xx- ARCH=ppc make KSRC=/home/wd/iscsi/linux-2.6-denx
        </verbatim>
      * Install (being root)
        <verbatim>
PATH=/opt/eldk/usr/ppc-linux/bin:/opt/eldk/usr/bin:$PATH CROSS_COMPILE=ppc_4xx- ARCH=ppc make KSRC=/home/wd/iscsi/linux-2.6-denx DISTDIR=/opt/eldk/ppc_4xx
        </verbatim>
        <verbatim>
cp etc/ietd.conf /opt/eldk/ppc_4xx/etc
        </verbatim>
   1 Configure the iscsitarget. Edit =/opt/eldk-4.1/ppc_4xx/etc/ietd.conf= .
     Set the target name appropriately. It may be left unchanged, though.
     For this example, the target name was set to the following:
     <verbatim>
Target iqn.2007-03.domain:storage.disk1
     </verbatim>
     Edit the following string to set the "Lun" to be the Linux RAID device:
     <verbatim>
Lun 0 Path=/dev/md0,Type=fileio
     </verbatim>
   1 Boot the target. Note the LSI SAS device and connected discs are
     recognized:
     <verbatim>
...
Fusion MPT base driver 3.04.04
Copyright (c) 1999-2007 LSI Logic Corporation
Fusion MPT FC Host driver 3.04.04
Fusion MPT SAS Host driver 3.04.04
mptbase: Initiating ioc0 bringup
ioc0: SAS1068: Capabilities={Initiator}
scsi0 : ioc0: LSISAS1068, FwRev=01060000h, Ports=1, MaxQ=511, IRQ=24
scsi 0:0:0:0: Direct-Access     ATA      Maxtor 6Y250M0   1EW0 PQ: 0 ANSI: 5
SCSI device sda: 490234752 512-byte hdwr sectors (251000 MB)
sda: Write Protect is off
SCSI device sda: write cache: enabled, read cache: enabled, doesn't support DPO or FUA
SCSI device sda: 490234752 512-byte hdwr sectors (251000 MB)
sda: Write Protect is off
SCSI device sda: write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sda: sda1
sd 0:0:0:0: Attached scsi disk sda
sd 0:0:0:0: Attached scsi generic sg0 type 0
scsi 0:0:1:0: Direct-Access     ATA      WDC WD1200JS-00M 1C03 PQ: 0 ANSI: 5
SCSI device sdb: 234441648 512-byte hdwr sectors (120034 MB)
sdb: Write Protect is off
SCSI device sdb: write cache: enabled, read cache: enabled, doesn't support DPO or FUA
SCSI device sdb: 234441648 512-byte hdwr sectors (120034 MB)
sdb: Write Protect is off
SCSI device sdb: write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sdb: unknown partition table
sd 0:0:1:0: Attached scsi disk sdb
sd 0:0:1:0: Attached scsi generic sg1 type 0
scsi 0:0:2:0: Direct-Access     ATA      WDC WD1200JS-00M 1C03 PQ: 0 ANSI: 5
SCSI device sdc: 234441648 512-byte hdwr sectors (120034 MB)
sdc: Write Protect is off
SCSI device sdc: write cache: enabled, read cache: enabled, doesn't support DPO or FUA
SCSI device sdc: 234441648 512-byte hdwr sectors (120034 MB)
sdc: Write Protect is off
SCSI device sdc: write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sdc: unknown partition table
sd 0:0:2:0: Attached scsi disk sdc
sd 0:0:2:0: Attached scsi generic sg2 type 0
scsi 0:0:3:0: Direct-Access     ATA      WDC WD1200JS-00M 1C03 PQ: 0 ANSI: 5
SCSI device sdd: 234441648 512-byte hdwr sectors (120034 MB)
sdd: Write Protect is off
SCSI device sdd: write cache: enabled, read cache: enabled, doesn't support DPO or FUA
SCSI device sdd: 234441648 512-byte hdwr sectors (120034 MB)
sdd: Write Protect is off
SCSI device sdd: write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sdd: sdd1
sd 0:0:3:0: Attached scsi disk sdd
sd 0:0:3:0: Attached scsi generic sg3 type 0
Fusion MPT misc device (ioctl) driver 3.04.04
mptctl: Registered with Fusion MPT base driver
mptctl: /dev/mptctl @ (major,minor=10,220)
...
     </verbatim>
   1  On the target, create the RAID5 array using the "mdadm" utility:
     <verbatim>
mknod /dev/md0 b 9 0
mdadm -C /dev/md0 -c 32 -l 5 -n 4 /dev/sd[a-d]
     </verbatim>
   1 Start the iSCSI target service:
     <verbatim>
depmod -a
/etc/init.d/iscsi-target start
     </verbatim>

---++ Building and configuring the iSCSI Initiator

In this example, a standard PC running Fedora Core 5 Linux was used as host
system.

%X% *Note:* We also tried to use the =iscsi-initiator-utils= package
from the Fedora Core 6 distribution to set up the iSCSI initiator on
a Fedora Core 6 host. This package contains only user-level
components; kernel modules used are from the mainline Fedora Core 6
kernel. Such a combination did not work in our tests; the target gets
discovered but doesn't appear as a local SCSI virtual drive. We ended
up with the =iscsi_tcp= module stuck and being unloadable from the
kernel, so we had to reboot the system.

There is note at
=http://www.open-iscsi.org/cgi-bin/wiki.pl/Supported_Kernels= that
kernel mainline modules should not be used, so we downloaded and built
the package:
   1 Download [[%ATTACHURL%/open-iscsi-2.0-754.tar.gz][open-iscsi-2.0-754.tar.gz]]
     from http://www.open-iscsi.org/
   1 Unpack and build:
     <verbatim>
make KSRC=/usr/src/kernels/2.6.17-1.2187_FC5-i686
     </verbatim>
   1 Install (being root)
     <verbatim>
make KSRC=/usr/src/kernels/2.6.17-1.2187_FC5-i686 install
depmod -a
     </verbatim>
   1 Create the following file:
     <verbatim>
echo "InitiatorName=iqn.2005-03.com.max:01.507a17"> /etc/iscsi/initiatorname.iscsi
     </verbatim>
   1 Start the initiator (being root)
     <verbatim>
/etc/init.d/open-iscsi start
     </verbatim>
   1 By default, the initiator is not configured for auto-connect the target.
     To automate the discovery/login, see =open-iscsi-2.0-754/README=
     %BR% %BR%
     First, run the discovery process (=192.168.80.17= is the IP address
     of the "taishan" board):
     <verbatim>
# iscsiadm -m discovery -t sendtargets -p 192.168.80.17:3260
192.168.80.17:3260,1 iqn.2007-03.domain:storage.disk1
     </verbatim>
     Login (connect) to the discovered target:
     <verbatim>
iscsiadm -m node -T iqn.2007-03.domain:storage.disk1 -p 192.168.80.17:3260 -l
     </verbatim>
     After this, you should be able to find the
     virtual appliance in =/proc/scsi/scsi=:
     <verbatim>
# cat /proc/scsi/scsi
Attached devices:
Host: scsi24 Channel: 00 Id: 00 Lun: 00
  Vendor: IET      Model: VIRTUAL-DISK     Rev: 0
  Type:   Direct-Access                    ANSI SCSI revision: 04
     </verbatim>
     Examine "dmesg" output for the virtual disk name to use:
     <verbatim>
...
csi24 : iSCSI Initiator over TCP/IP
  Vendor: IET       Model: VIRTUAL-DISK      Rev: 0
  Type:   Direct-Access                      ANSI SCSI revision: 04
SCSI device sda: 703324416 512-byte hdwr sectors (360102 MB)
sda: Write Protect is off
sda: Mode Sense: 77 00 00 08
SCSI device sda: drive cache: write through
SCSI device sda: 703324416 512-byte hdwr sectors (360102 MB)
sda: Write Protect is off
sda: Mode Sense: 77 00 00 08
SCSI device sda: drive cache: write through
 sda: sda1
sd 24:0:0:0: Attached scsi disk sda
sd 24:0:0:0: Attached scsi generic sg0 type 0
...
     </verbatim>
   1 Partition, format and mount the disk:
     <verbatim>
# fdisk /dev/sda
....
# mkfs -t ext3 /dev/sda1
...
# mount /dev/sda1 /mnt/iscsi
# cp -a /usr /mnt/iscsi
# cp -a /lib /mnt/iscsi
# ls -l /mnt/iscsi
total 24
drwxr-xr-x 11 psl  psl   4096 Mar 13 10:31 lib
drwx------  2 root root 16384 Mar 13 11:46 lost+found
drwxr-xr-x 15 psl  psl   4096 Mar 13 10:31 usr
     </verbatim>
   1 After use, unmount the virtual disk and disconnect from the target:
     <verbatim>
# umount /mnt/iscsi
# iscsiadm -m node -T iqn.2007-03.domain:storage.disk1 -p 192.168.80.17:3260 -u
     </verbatim>

[[%ATTACHURL%/iscsi-target-installed.tgz][Here]] you can find a
a tarball with daemon, utilities, kernel modules and
configuration files as used for this demo.
It works both on katmai and taishan.

-- Main.WolfgangDenk - 14 Mar 2007