Netapp Full Labs


 

 

root@robert:/media/CDROM# ./setup.sh

 

Script version 22 (18/Sep/2007)

Where to install to? [/sim]:

Would you like to install as a cluster? [no]:

Would you like full HTML/PDF FilerView documentation to be installed [yes]:

Continue with installation? [no]: yes

Creating /sim

Unpacking sim.tgz to /sim

Configured the simulators mac address to be [00:50:56:17:5:bf]

Please ensure the simulator is not running.

Your simulator has 3 disk(s). How many more would you like to add? [0]: 40

Too high. Must be between 0 and 25.

Your simulator has 3 disk(s). How many more would you like to add? [0]: 25

 

The following disk types are available in MB:

Real (Usable)

a – 43 ( 14)

b – 62 ( 30)

c – 78 ( 45)

d – 129 ( 90)

e – 535 (450)

f – 1024 (900)

 

If you are unsure choose the default option a

What disk size would you like to use? [a]:

Disk adapter to put disks on? [0]:

Use DHCP on first boot? [yes]:

Ask for floppy boot? [no]:

Checking the default route…

You have a single network interface called eth0 (default route) . You will not be able to access the simulator from this Linux host. If this interface is marked DOWN in ifconfig then your simulator will crash.

Which network interface should the simulator use? [default]:

Your system has 1803MB of free memory. The smallest simulator memory you should choose is 110MB. The maximum simulator memory is 1763MB.

The recommended memory is 512MB.

How much memory would you like the simulator to use? [512]:

Create a new log for each session? [no]:

Overwrite the single log each time? [yes]:

Adding 25 additional disk(s).

Complete. Run /sim/runsim.sh to start the simulator.

root@robert:/media/CDROM# /sim/runsim.sh

runsim.sh script version Script version 22 (18/Sep/2007)

This session is logged in /sim/sessionlogs/log

 

NetApp Release 7.3.4: Thu May 27 15:43:26 PDT 2010

Copyright (c) 1992-2010 NetApp.

Starting boot on Tue Sep 21 12:25:44 GMT 2010

 

Tue Sep 21 12:26:31 GMT [iomem.init.fail:CRITICAL]: Failed to initialize acceleration card (model name X1938A-R5, serial number 5012345670, part number virgo-simulator) in slot 1.

Tue Sep 21 12:26:31 GMT [fmm.domain.card.failure:error]: PAM II in slot 1 (model name X1938A-R5, serial number 5012345670, part number virgo-simulator): Flash device failed and needs to be updated or repaired or replaced.

Tue Sep 21 12:26:39 GMT [fmmb.current.lock.disk:info]: Disk v4.16 is a local HA mailbox disk.

Tue Sep 21 12:26:39 GMT [fmmb.current.lock.disk:info]: Disk v4.17 is a local HA mailbox disk.

Tue Sep 21 12:26:39 GMT [fmmb.instStat.change:info]: normal mailbox instance on local side.

Tue Sep 21 12:26:42 GMT [raid.cksum.replay.summary:info]: Replayed 0 checksum blocks.

Tue Sep 21 12:26:42 GMT [raid.stripe.replay.summary:info]: Replayed 0 stripes.

sparse volume upgrade done. num vol 0.

Vdisk Snap Table for host:0 is initialized

Tue Sep 21 12:26:44 GMT [vol.language.unspecified:info]: Language not set on volume vol0. Using language config “C”. Use vol lang to set language.

Tue Sep 21 12:26:44 GMT [rc:notice]: The system was down for 5325399 seconds

Tue Sep 21 12:26:44 GMT [useradmin.added.deleted:info]: The role ‘compliance’ has been added.

Tue Sep 21 12:26:45 GMT [useradmin.added.deleted:info]: The group ‘Backup Operators’ has been modified.

/etc/rc is missing. Running configuration dialog.

 

Tue Sep 21 12:26:52 GMT [dhcp.contacting.Server:info]: Contacting DHCP server.

Configuring onboard ethernet ns0.

Contacting DHCP server.

Ctrl-X to skip DHCP search …

Tue Sep 21 12:26:52 GMT [netif.linkUp:info]: Ethernet ns0: Link up.

Configuring ns0 using IP 192.168.1.105 netmask 255.255.255.0.

ifconfig ns0 up netmask 255.255.255.0 192.168.1.105

add net default: gateway 192.168.1.1

Tue Sep 21 12:26:58 GMT [dhcp.Config.Success:info]: DHCP configuration was successful.

Tue Sep 21 12:26:58 GMT [snmp.agent.msg.access.denied:warning]: Permission denied for SNMPv3 requests from root. Reason: Password is too short (SNMPv3 requires at least 8 characters).

    NetApp Release 7.3.4: Thu May 27 15:43:26 PDT 2010

    System ID: 0099929383 ()

    System Serial Number: 987654-32-0 ()

    System Storage Configuration: Multi-Path

    System ACP Connectivity: NA

    Model Name: Simulator

    Processors: 1

    slot 0: NetApp Virtual SCSI Host Adapter v0

28 Disks: 1.2GB

        3 shelves with LRC

    slot 1: NetApp Virtual SCSI Host Adapter v1

    slot 2: NetApp Virtual SCSI Host Adapter v2

    slot 3: NetApp Virtual SCSI Host Adapter v3

    slot 4: NetApp Virtual SCSI Host Adapter v4

28 Disks: 1.2GB

        3 shelves with LRC

    slot 5: NetApp Virtual SCSI Host Adapter v5

    slot 6: NetApp Virtual SCSI Host Adapter v6

    slot 7: NetApp Virtual SCSI Host Adapter v7

    slot 8: NetApp Virtual SCSI Host Adapter v8

4 Tapes: VT-100MB

VT-100MB

VT-100MB

VT-100MB

Please enter the new hostname []: Tue Sep 21 12:26:59 GMT [shelf.config.multipath:info]: All attached storage on the system is multi-pathed.

 

Please enter the new hostname []: robert

Do you want to enable IPv6? [n]:

Do you want to configure virtual network interfaces? [n]:

Please enter the IP address for Network Interface ns0 [192.168.1.105]: 192.168.1.102

Please enter the netmask for Network Interface ns0 [255.255.255.0]:

Please enter media type for ns0 {100tx-fd, auto} [auto]:

Please enter the IP address for Network Interface ns1 []:

Would you like to continue setup through the web interface? [n]:

Please enter the name or IP address of the IPv4 default gateway [192.168.1.1]:

    The administration host is given root access to the filer’s

    /etc files for system administration. To allow /etc root access

    to all NFS clients enter RETURN below.

Please enter the name or IP address of the administration host:

Please enter timezone [GMT]:

Where is the filer located? []:

What language will be used for multi-protocol files (Type ? for list)?:

language not set

Do you want to run DNS resolver? [n]:

Do you want to run NIS client? [n]:

    The Shelf Alternate Control Path Management process provides the ability

    to recover from certain SAS shelf module failures and provides a level of

    availability that is higher than systems not using the Alternate Control

    Path Management process.

Do you want to configure the Shelf Alternate Control Path Management interface for SAS shelves [n]: Setting the administrative (root) password for robert …

 

New password:

Retype new password:

Tue Sep 21 12:29:05 GMT [passwd.changed:info]: passwd for user ‘root’ changed.

Tue Sep 21 12:29:05 GMT [tapemc.alias.addOK:info]: Alias st0 automatically added for tape device WWN[0:042:424200:000000].

Tue Sep 21 12:29:05 GMT [dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives

Tue Sep 21 12:29:05 GMT [sfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk shelves.

Tue Sep 21 12:29:05 GMT [tapemc.alias.addOK:info]: Alias st1 automatically added for tape device WWN[0:142:424200:000000].

Tue Sep 21 12:29:05 GMT [tapemc.alias.addOK:info]: Alias st2 automatically added for tape device WWN[0:242:424200:000000].

Tue Sep 21 12:29:05 GMT [tapemc.alias.addOK:info]: Alias st3 automatically added for tape device WWN[0:342:424200:000000].

add net default: gateway 192.168.1.1: entry already exists

There are 25 spare disks; you may want to use the vol or aggr command

to create new volumes or aggregates or add disks to the existing aggregate.

Tue Sep 21 12:29:06 GMT [perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 195 counters.

Tue Sep 21 12:29:06 GMT [rc:info]: Registry is being upgraded to improve storing of local changes.

Tue Sep 21 12:29:06 GMT [rc:info]: Registry upgrade successful.

Tue Sep 21 12:29:07 GMT [mgr.boot.disk_done:info]: NetApp Release 7.3.4 boot complete. Last disk update written at Wed Jul 21 21:10:04 GMT 2010

Tue Sep 21 12:29:07 GMT [mgr.boot.reason_ok:notice]: System rebooted after a halt command.

This process will enable CIFS access to the filer from a Windows(R) system.

Use “?” for help at any prompt and Ctrl-C to exit without committing changes.

 

Your filer does not have WINS configured and is visible only to

clients on the same subnet.

Do you want to make the system visible via WINS? [n]:

A filer can be configured for multiprotocol access, or as an NTFS-only

filer. Since multiple protocols are currently licensed on this filer,

we recommend that you configure this filer as a multiprotocol filer

 

(1) Multiprotocol filer

(2) NTFS-only filer

 

Selection (1-2)? [1]:

CIFS requires local /etc/passwd and /etc/group files and default files

will be created. The default passwd file contains entries for ‘root’,

‘pcuser’, and ‘nobody’.

Enter the password for the root user []:

Retype the password:

The default name for this CIFS server is ‘ROBERT’.

Would you like to change this name? [n]:

Data ONTAP CIFS services support four styles of user authentication.

Choose the one from the list below that best suits your situation.

 

(1) Active Directory domain authentication (Active Directory domains only)

(2) Windows NT 4 domain authentication (Windows NT or Active Directory domains)

(3) Windows Workgroup authentication using the filer’s local user accounts

(4) /etc/passwd and/or NIS/LDAP authentication

 

Selection (1-4)? [1]: 3

What is the name of the Workgroup? [WORKGROUP]:

Tue Sep 21 12:30:03 GMT [passwd.changed:info]: passwd for user ‘root’ changed.

CIFS – Starting SMB protocol…

It is recommended that you create the local administrator account

(ROBERT\administrator) for this filer.

Do you want to create the ROBERT\administrator account? [y]:

Enter the new password for ROBERT\administrator:

Retype the password:

Welcome to the WORKGROUP Windows(R) workgroup

 

CIFS local server is running.

 

 

Data ONTAP (robert.)

 


the basic unit for snapvault backup and restore: qtree

the basic unit of an open systems snapvault backup: the directory

 

try to mount the exported NFS

:

on another vmware Ubuntu host:
mount -t nfs 192.168.1.102:/vol/vol0 /mnt/robert

 

root@robert:~# df -k

……

192.168.1.102:/vol/vol0

247680 88384 159296 36% /mnt/Robert

 

Try to mount iscsi device in another Ubuntu from netapp simulator

  1. Enable iscsi:

 

  1. Create a volume:

  1. Create a Lun

  1. Create a initiator group

在要准备用这个Lun的iSCSI initiator Ubuntu里做这个命令:

iscsiadm -m discovery -t st -p 192.168.1.102

可以在这个有Netapp Simulator的Ubuntu里看到类似的:

Tue Sep 21 13:00:04 GMT [iscsi.notice:notice]: ISCSI: New session from initiator iqn.1993-08.org.debian:01:f66d57c1c83 at IP addr 192.168.1.103

 

  1. Lun Map 到这个Initiator Group:

 

root@robert:~# apt-get install open-iscsi

正在读取软件包列表… 完成

正在分析软件包的依赖关系树

正在读取状态信息… 完成

将会安装下列额外的软件包:

open-iscsi-utils

下列【新】软件包将被安装:

open-iscsi open-iscsi-utils

升级了 0 个软件包,新安装了 2 个软件包,要卸载 0 个软件包,有 9 个软件包未被升级。

需要下载 604kB 的软件包。

解压缩后会消耗掉 1,499kB 的额外空间。

您希望继续执行吗?[Y/n]y

获取:1 http://mirrors.163.com/ubuntu/ lucid/main open-iscsi-utils 2.0.871-0ubuntu4 [98.1kB]

获取:2 http://mirrors.163.com/ubuntu/ lucid/main open-iscsi 2.0.871-0ubuntu4 [506kB]

下载 604kB,耗时 2秒 (226kB/s)

选中了曾被取消选择的软件包 open-iscsi-utils。

(正在读取数据库 … 系统当前总共安装有 202476 个文件和目录。)

正在解压缩 open-iscsi-utils (从 …/open-iscsi-utils_2.0.871-0ubuntu4_i386.deb) …

选中了曾被取消选择的软件包 open-iscsi。

正在解压缩 open-iscsi (从 …/open-iscsi_2.0.871-0ubuntu4_i386.deb) …

正在处理用于 man-db 的触发器…

正在处理用于 ureadahead 的触发器…

正在设置 open-iscsi-utils (2.0.871-0ubuntu4) …

正在设置 open-iscsi (2.0.871-0ubuntu4) …

update-rc.d: warning: open-iscsi stop runlevel arguments (0 1 6) do not match LSB Default-Stop values (0 6)

* Starting iSCSI initiator service iscsid

…done.

* Setting up iSCSI targets

…done.

 

192.168.1.102 是装了Netapp Simulator 的Ubuntu虚拟机:

root@robert:/etc/iscsi# iscsiadm -m discovery -t st -p 192.168.1.102

192.168.1.102:3260,1000 iqn.1992-08.com.netapp:sn.99929383

 

 

robert> lun show -v

/vol/iscsi1/lun 200m (209715200) (r/w, online)

Comment: “iscsi lun for ubuntu”

Serial#: nBM6goZBGr06

Share: none

Space Reservation: disabled

Multiprotocol Type: linux

robert> Tue Sep 21 12:53:35 GMT [iscsi.notice:notice]: ISCSI: New session from initiator iqn.1993-08.org.debian:01:f66d57c1c83 at IP addr 192.168.1.103

Tue Sep 21 12:58:20 GMT [lun.map:info]: LUN /vol/iscsi1/lun was mapped to initiator group iscsi1=1

Tue Sep 21 12:58:58 GMT [iscsi.notice:notice]: ISCSI: New session from initiator iqn.1993-08.org.debian:01:f66d57c1c83 at IP addr 192.168.1.103

Tue Sep 21 13:00:00 GMT [kern.uptime.filer:info]: 1:00pm up 34 mins, 0 NFS ops, 0 CIFS ops, 95 HTTP ops, 0 FCP ops, 0 iSCSI ops

Tue Sep 21 13:00:04 GMT [iscsi.notice:notice]: ISCSI: New session from initiator iqn.1993-08.org.debian:01:f66d57c1c83 at IP addr 192.168.1.103

 

igroup show

iscsi1 (iSCSI) (ostype: linux):

iqn.1993-08.org.debian:01:f66d57c1c83 (not logged in)

 

root@robert:~# iscsiadm -m node -T iqn.1992-08.com.netapp:sn.99929383 -p 192.168.1.102:3260 –login

Logging in to [iface: default, target: iqn.1992-08.com.netapp:sn.99929383, portal: 192.168.1.102,3260]

Login to [iface: default, target: iqn.1992-08.com.netapp:sn.99929383, portal: 192.168.1.102,3260]: successful

root@robert:~# tail /var/log/messages

Sep 21 21:01:16 robert kernel: [ 2493.444170] iscsi: registered transport (tcp)

Sep 21 21:01:16 robert kernel: [ 2493.480777] iscsi: registered transport (iser)

Sep 21 22:05:30 robert kernel: [ 6345.258467] scsi3 : iSCSI Initiator over TCP/IP

Sep 21 22:05:31 robert kernel: [ 6345.571521] scsi 3:0:0:1: Direct-Access NETAPP LUN 7340 PQ: 0 ANSI: 4

Sep 21 22:05:31 robert kernel: [ 6345.571963] sd 3:0:0:1: Attached scsi generic sg2 type 0

Sep 21 22:05:31 robert kernel: [ 6345.576135] sd 3:0:0:1: [sdb] 409600 512-byte logical blocks: (209 MB/200 MiB)

Sep 21 22:05:31 robert kernel: [ 6345.576942] sd 3:0:0:1: [sdb] Write Protect is off

Sep 21 22:05:31 robert kernel: [ 6345.578286] sd 3:0:0:1: [sdb] Write cache: disabled, read cache: enabled, doesn’t support DPO or FUA

Sep 21 22:05:31 robert kernel: [ 6345.581779] sdb: unknown partition table

Sep 21 22:05:31 robert kernel: [ 6345.591769] sd 3:0:0:1: [sdb] Attached SCSI disk

 

root@robert:~# fdisk -l

 

Disk /dev/sda: 21.5 GB, 21474836480 bytes

255 heads, 63 sectors/track, 2610 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x000adc00

 

Device Boot Start End Blocks Id System

/dev/sda1 * 1 2497 20051968 83 Linux

/dev/sda2 2497 2611 916481 5 Extended

/dev/sda5 2497 2611 916480 82 Linux swap / Solaris

 

Disk /dev/sdb: 209 MB, 209715200 bytes

7 heads, 58 sectors/track, 1008 cylinders

Units = cylinders of 406 * 512 = 207872 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0×00000000

 

Disk /dev/sdb doesn’t contain a valid partition table

root@robert:~# fdisk /dev/sdb

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel

Building a new DOS disklabel with disk identifier 0x18bcde61.

Changes will remain in memory only, until you decide to write them.

After that, of course, the previous content won’t be recoverable.

 

Warning: invalid flag 0×0000 of partition table 4 will be corrected by w(rite)

 

WARNING: DOS-compatible mode is deprecated. It’s strongly recommended to

switch off the mode (command ‘c’) and change display units to

sectors (command ‘u’).

 

Command (m for help): m

Command action

a toggle a bootable flag

b edit bsd disklabel

c toggle the dos compatibility flag

d delete a partition

l list known partition types

m print this menu

n add a new partition

o create a new empty DOS partition table

p print the partition table

q quit without saving changes

s create a new empty Sun disklabel

t change a partition’s system id

u change display/entry units

v verify the partition table

w write table to disk and exit

x extra functionality (experts only)

 

Command (m for help): l

 

0 Empty 24 NEC DOS 81 Minix / old Lin bf Solaris

1 FAT12 39 Plan 9 82 Linux swap / So c1 DRDOS/sec (FAT-

2 XENIX root 3c PartitionMagic 83 Linux c4 DRDOS/sec (FAT-

3 XENIX usr 40 Venix 80286 84 OS/2 hidden C: c6 DRDOS/sec (FAT-

4 FAT16 <32M 41 PPC PReP Boot 85 Linux extended c7 Syrinx

5 Extended 42 SFS 86 NTFS volume set da Non-FS data

6 FAT16 4d QNX4.x 87 NTFS volume set db CP/M / CTOS / .

7 HPFS/NTFS 4e QNX4.x 2nd part 88 Linux plaintext de Dell Utility

8 AIX 4f QNX4.x 3rd part 8e Linux LVM df BootIt

9 AIX bootable 50 OnTrack DM 93 Amoeba e1 DOS access

a OS/2 Boot Manag 51 OnTrack DM6 Aux 94 Amoeba BBT e3 DOS R/O

b W95 FAT32 52 CP/M 9f BSD/OS e4 SpeedStor

c W95 FAT32 (LBA) 53 OnTrack DM6 Aux a0 IBM Thinkpad hi eb BeOS fs

e W95 FAT16 (LBA) 54 OnTrackDM6 a5 FreeBSD ee GPT

f W95 Ext’d (LBA) 55 EZ-Drive a6 OpenBSD ef EFI (FAT-12/16/

10 OPUS 56 Golden Bow a7 NeXTSTEP f0 Linux/PA-RISC b

11 Hidden FAT12 5c Priam Edisk a8 Darwin UFS f1 SpeedStor

12 Compaq diagnost 61 SpeedStor a9 NetBSD f4 SpeedStor

14 Hidden FAT16 <3 63 GNU HURD or Sys ab Darwin boot f2 DOS secondary

16 Hidden FAT16 64 Novell Netware af HFS / HFS+ fb VMware VMFS

17 Hidden HPFS/NTF 65 Novell Netware b7 BSDI fs fc VMware VMKCORE

18 AST SmartSleep 70 DiskSecure Mult b8 BSDI swap fd Linux raid auto

1b Hidden W95 FAT3 75 PC/IX bb Boot Wizard hid fe LANstep

1c Hidden W95 FAT3 80 Old Minix be Solaris boot ff BBT

1e Hidden W95 FAT1

 

Command (m for help): n

Command action

e extended

p primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-1008, default 1):

Using default value 1

Last cylinder, +cylinders or +size{K,M,G} (1-1008, default 1008):

Using default value 1008

 

Command (m for help): m

Command action

a toggle a bootable flag

b edit bsd disklabel

c toggle the dos compatibility flag

d delete a partition

l list known partition types

m print this menu

n add a new partition

o create a new empty DOS partition table

p print the partition table

q quit without saving changes

s create a new empty Sun disklabel

t change a partition’s system id

u change display/entry units

v verify the partition table

w write table to disk and exit

x extra functionality (experts only)

 

Command (m for help): w

The partition table has been altered!

 

Calling ioctl() to re-read partition table.

Syncing disks.

root@robert:~# fdisk -l

 

Disk /dev/sda: 21.5 GB, 21474836480 bytes

255 heads, 63 sectors/track, 2610 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x000adc00

 

Device Boot Start End Blocks Id System

/dev/sda1 * 1 2497 20051968 83 Linux

/dev/sda2 2497 2611 916481 5 Extended

/dev/sda5 2497 2611 916480 82 Linux swap / Solaris

 

Disk /dev/sdb: 209 MB, 209715200 bytes

7 heads, 58 sectors/track, 1008 cylinders

Units = cylinders of 406 * 512 = 207872 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x18bcde61

 

Device Boot Start End Blocks Id System

/dev/sdb1 1 1008 204595 83 Linux

 

 

root@robert:~# mkfs.ext3 /dev/sdb1

mke2fs 1.41.11 (14-Mar-2010)

文件系统标签=

操作系统:Linux

块大小=1024 (log=0)

分块大小=1024 (log=0)

Stride=0 blocks, Stripe width=0 blocks

51200 inodes, 204592 blocks

10229 blocks (5.00%) reserved for the super user

第一个数据块=1

Maximum filesystem blocks=67371008

25 block groups

8192 blocks per group, 8192 fragments per group

2048 inodes per group

Superblock backups stored on blocks:

8193, 24577, 40961, 57345, 73729

 

root@robert:~# mount /dev/sdb1 /mnt

 

正在写入inode表: 完成

Creating journal (4096 blocks): 完成

Writing superblocks and filesystem accounting information: 完成

 

This filesystem will be automatically checked every 21 mounts or

180 days, whichever comes first. Use tune2fs -c or -i to override.

 

iscsi initiator show

Initiators connected:

TSIH TPGroup Initiator/ISID/IGroup

9 1000 robert (iqn.1993-08.org.debian:01:f66d57c1c83 / 00:02:3d:01:00:00 / iscsi1)

robert> igroup show

iscsi1 (iSCSI) (ostype: linux):

iqn.1993-08.org.debian:01:f66d57c1c83 (logged in on: ns0)

 

 

CIFS usage

 

netapp-robert> cifs shares

Name Mount Point Description

—- ———– ———–

ETC$ /etc Remote Administration

BUILTIN\Administrators / Full Control

HOME /vol/vol0/home Default Share

everyone / Full Control

C$ / Remote Administration

BUILTIN\Administrators / Full Control

 

netapp-robert> qtree create /vol/vol0/cifs_tree

netapp-robert> qtree security /vol/vol0/cifs_tree ntfs

Wed Sep 22 01:02:21 GMT [wafl.quota.sec.change:notice]: security style for /vol/vol0/cifs_tree changed from unix to ntfs

netapp-robert> qtree status

Volume Tree Style Oplocks Status

——– ——– —– ——– ———

vol0 unix enabled normal

vol0 cifs_tree ntfs enabled normal

netapp-robert> cifs shares -add cifs_tree /vol/vol0/cifs_tree

The share name ‘cifs_tree’ will not be accessible by some MS-DOS workstations

netapp-robert> cifs shares cifs_tree

Name Mount Point Description

—- ———– ———–

cifs_tree /vol/vol0/cifs_tree

everyone / Full Control

 

工具/映射网络驱动器/ 输入Administrator 密码:

Enable SSH

 

 

 

Read file: rdfile

Write file: wrfile

Can edit /etc files with \\$storage-name\C$

 

Security admin

netapp-robert> useradmin group list

Name: Administrators

Info: Members can fully administer the filer

Rid: 544

Roles: admin

 

Name: Backup Operators

Info: Members can bypass file security to backup files

Rid: 551

Roles: backup

 

Name: Compliance Administrators

Info: Members can perform compliance operations

Rid: 131072

Roles: compliance

 

Name: Guests

Info: Users granted Guest Access

Rid: 546

Roles: none

 

Name: Power Users

Info: Members that can share directories

Rid: 547

Roles: power

 

Name: Replicators

Info: not supported

Rid: 552

Roles: none

 

Name: Users

Info: Ordinary Users

Rid: 545

Roles: audit

 

netapp-robert> options security

security.admin.authentication internal

security.admin.nsswitchgroup

security.passwd.firstlogin.enable off

security.passwd.lockout.numtries 4294967295

security.passwd.rootaccess.enable on

security.passwd.rules.enable on

security.passwd.rules.everyone off

security.passwd.rules.history 0

security.passwd.rules.maximum 256

security.passwd.rules.minimum 8

security.passwd.rules.minimum.alphabetic 2

security.passwd.rules.minimum.digit 1

security.passwd.rules.minimum.symbol 0

 

netapp-robert> useradmin user list

Name: root

Info: Default system administrator.

Rid: 0

Groups:

 

Name: administrator

Info: Built-in account for administering the filer

Rid: 500

Groups: Administrators

 

netapp-robert> sysconfig -r

Aggregate aggr0 (online, raid0) (zoned checksums)

Plex /aggr0/plex0 (online, normal, active)

RAID group /aggr0/plex0/rg0 (normal)

 

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

——— —— ————- —- —- —- —– ————– ————–

data v4.16 v4 1 0 FC:B – FCAL N/A 120/246784 127/261248

data v4.17 v4 1 1 FC:B – FCAL N/A 120/246784 127/261248

data v4.18 v4 1 2 FC:B – FCAL N/A 120/246784 127/261248

 

 

Spare disks

 

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

——— —— ————- —- —- —- —– ————– ————–

Spare disks for zoned checksum traditional volumes or aggregates only

……

spare v4.48 v4 3 0 FC:B – FCAL N/A 70/144384 77/158848

 

Network Admin

netapp-robert> rdfile /etc/rc

#Auto-generated by setup Wed Sep 22 00:34:09 GMT 2010

hostname netapp-robert

ifconfig ns0 `hostname`-ns0 mediatype auto

ifconfig ns1 `hostname`-ns1 mediatype auto

route add default 192.168.1.1 1

routed on

options dns.enable off

options nis.enable off

netapp-robert> ifconfig -a

ns0: flags=0x2c48867<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 192.168.1.110 netmask-or-prefix 0xffffff00 broadcast 192.168.1.255

ether 00:50:56:02:0a:ff (auto-100tx-fd-up)

ns1: flags=0x2c48867<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 192.168.1.210 netmask-or-prefix 0xffffff00 broadcast 192.168.1.255

ether 00:50:56:03:0a:ff (auto-100tx-fd-up)

lo: flags=0×1948049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 9188

inet 127.0.0.1 netmask-or-prefix 0xff000000 broadcast 127.0.0.1

 

virtual network interface is for standby or improve bandwidth (multimode).

 

Disk admin/physical storage admin

netapp-robert> fcstat device_map

Loop Map for channel v0:

Translated Map: Port Count 29

7 35 48 21 40 20 19 24 33 17 25 22 44 36 18 45

16 28 29 27 42 41 32 38 34 37 26 43 39

Shelf mapping:

Shelf 1: 29 28 27 26 25 24 XXX 22 21 20 19 18 17 16

Shelf 2: 45 44 43 42 41 40 39 38 37 36 35 34 33 32

Shelf 3: XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX 48

 

 

Cannot complete operation on channel v1; link is DOWN

Cannot complete operation on channel v2; link is DOWN

Cannot complete operation on channel v3; link is DOWN

Loop Map for channel v4:

Translated Map: Port Count 29

7 35 48 21 40 20 19 24 33 17 25 22 44 36 18 45

16 28 29 27 42 41 32 38 34 37 26 43 39

Shelf mapping:

Shelf 1: 29 28 27 26 25 24 XXX 22 21 20 19 18 17 16

Shelf 2: 45 44 43 42 41 40 39 38 37 36 35 34 33 32

Shelf 3: XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX XXX 48

 

 

Cannot complete operation on channel v5; link is DOWN

Cannot complete operation on channel v6; link is DOWN

Cannot complete operation on channel v7; link is DOWN

Loop Map for channel v8:

Translated Map: Port Count 5

7 0 1 2 3

Shelf mapping:

 

 

Shelf Unknown: 0 1 2 3

 

 

Aggregate = one or more raid groups = collection of disks

 

Logical storage Admin

WAFL: write anywhere file layout

 

Volumes is a collection of stored data within an aggregate

Tradditional volumes is contained by a single, dedicated aggregate.

 

Multiple FlexVol volumes can per aggregate

 

Flexclone: multiple, instant data-set clones

 

4995 qtrees in a volume

 

Snapshot admin

A volume can have 255 snapshots

 

netapp-robert> options cifs.show_snapshot

cifs.show_snapshot off

netapp-robert> options cifs.show_snapshot on

 

打开后,可以看到 .snapshot目录:

 

For another ubuntu, we can delete file and copy from .snapshot:

root@robert:~# cd /mnt

root@robert:/mnt# mkdir netapp-robert

root@robert:/mnt# cd netapp-robert/

root@robert:/mnt/netapp-robert# mkdir vol0

root@robert:/mnt/netapp-robert# mount 192.168.1.110:/vol/vol0 /mnt/netapp-robert/vol0

root@robert:/mnt/netapp-robert# cd ./vol0/etc

root@robert:/mnt/netapp-robert/vol0/etc# pwd

/mnt/netapp-robert/vol0/etc

root@robert:/mnt/netapp-robert/vol0/etc# rm hosts.bak

root@robert:/mnt/netapp-robert/vol0/etc# cd /mnt/netapp-robert/vol0/.snapshot/

root@robert:/mnt/netapp-robert/vol0/.snapshot# ls

hourly.0

root@robert:/mnt/netapp-robert/vol0/.snapshot# cd hourly.0/

root@robert:/mnt/netapp-robert/vol0/.snapshot/hourly.0# cd etc

root@robert:/mnt/netapp-robert/vol0/.snapshot/hourly.0/etc# cp hosts.bak /mnt/netapp-robert/vol0/etc

 

Cifs and snap test:

  1. Check snap status and create snap1

netapp-robert> snap sched vol0

Volume vol0: 0 2 6@8,12,16,20

netapp-robert> snap create vol0 snap1

netapp-robert> df -h

Filesystem total used avail capacity Mounted on

/vol/vol0/ 241MB 102MB 138MB 43% /vol/vol0/

/vol/vol0/.snapshot 0MB 11MB 0MB —% /vol/vol0/.snapshot

netapp-robert> snap list vol0

Volume vol0

working…

 

%/used %/total date name

———- ———- ———— ——–

0% ( 0%) 0% ( 0%) Sep 25 02:40 snap1

6% ( 6%) 2% ( 2%) Sep 22 12:00 hourly.0

11% ( 6%) 5% ( 2%) Sep 22 08:00 hourly.1

  1. Copy a 10m file to z:/home

netapp-robert> df -h

Filesystem total used avail capacity Mounted on

/vol/vol0/ 241MB 113MB 128MB 47% /vol/vol0/

/vol/vol0/.snapshot 0MB 11MB 0MB —% /vol/vol0/.snapshot

 

  1. Create snap2

netapp-robert> snap create vol0 snap2

netapp-robert> snap list vol0

Volume vol0

working…

 

%/used %/total date name

———- ———- ———— ——–

0% ( 0%) 0% ( 0%) Sep 25 02:42 snap2

0% ( 0%) 0% ( 0%) Sep 25 02:40 snap1

5% ( 5%) 2% ( 2%) Sep 22 12:00 hourly.0

10% ( 5%) 5% ( 2%) Sep 22 08:00 hourly.1

 

netapp-robert> snap delta vol0

 

Volume vol0

working…

 

From Snapshot To KB changed Time Rate (KB/hour)

————— ——————– ———– ———— —————

snap2 Active File System 92 36s 9200.000

snap1 snap2 10960 0d 00:01 358690.909

hourly.0 snap1 6340 2d 14:39 101.174

hourly.1 hourly.0 6192 0d 03:59 1548.107

 

Summary…

 

From Snapshot To KB changed Time Rate (KB/hour)

————— ——————– ———– ———— —————

hourly.1 Active File System 23584 2d 18:42 353.561

 

netapp-robert> snap restore -t vol -s hourly.0 /vol/vol0

 

WARNING! This will revert the volume to a previous snapshot.

All modifications to the volume after the snapshot will be

irrevocably lost.

 

Volume vol0 is the root volume. All system settings are

stored in the root volume and will revert to their value at

the time of the snapshot. As such, note that you may need to

re-enter license codes after the revert. The system will be

rebooted to complete the reversion process.

 

Are you sure you want to do this? yes

 

You have selected volume vol0, snapshot hourly.0

 

Reboot system and proceed with revert? Yes

 

After reboot:

netapp-robert> snap list vol0

Volume vol0

working…

 

%/used %/total date name

———- ———- ———— ——–

1% ( 1%) 0% ( 0%) Sep 22 12:00 hourly.0

6% ( 6%) 2% ( 2%) Sep 22 08:00 hourly.1

netapp-robert>

 

恢复后,发现拷贝的文件没有了。Snapshot 会指向一些旧数据,所以会占据空间。

 

Snapmirror lab

  1. Install license

 

  1. Create aggr1:

 

 

 

 

第一个Ubuntu Netapp simulator IP is 192.168.1.110, startup the second Ubuntu netapp simulator, run this to change IP address of the second one:

root@robert:~# /sim/setup.sh

 

two hosts:

netapp-robert> rdfile /etc/rc

#Auto-generated by setup Wed Sep 22 00:34:09 GMT 2010

hostname netapp-robert

ifconfig ns0 `hostname`-ns0 mediatype auto

ifconfig ns1 `hostname`-ns1 mediatype auto

route add default 192.168.1.1 1

routed on

options dns.enable off

options nis.enable off

 

netapp-robert2> rdfile /etc/rc

#Auto-generated by setup Sat Sep 25 07:01:24 GMT 2010

hostname netapp-robert2

ifconfig ns0 `hostname`-ns0 mediatype auto

ifconfig ns1 `hostname`-ns1 mediatype auto

route add default 192.168.1.1 1

routed on

options dns.enable off

options nis.enable off

 

host1 can ping hosts2:

netapp-robert> ping netapp-robert2

netapp-robert2 is alive

netapp-robert> rdfile /etc/hosts

#Auto-generated by setup Wed Sep 22 00:34:09 GMT 2010

127.0.0.1 localhost

192.168.1.110 netapp-robert netapp-robert-ns0

192.168.1.210 netapp-robert-ns1

192.168.1.111 netapp-robert2 netapp-robert2-ns0

 

host2 can ping host1:

netapp-robert2> ping netapp-robert

netapp-robert is alive

netapp-robert2> rdfile /etc/hosts

#Auto-generated by setup Sat Sep 25 07:01:24 GMT 2010

127.0.0.1 localhost

192.168.1.111 netapp-robert2 netapp-robert2-ns0

192.168.1.211 netapp-robert2-ns1

192.168.1.110 netapp-robert netapp-robert-ns0

 

On source

netapp-robert> options snapmirror.access host=netapp-robert2

netapp-robert> options snapmirror.access

snapmirror.access host=netapp-robert2

 

netapp-robert> vol create vol3 -s none aggr1 40m

Creation of volume ‘vol3′ with size 40m on containing aggregate

‘aggr1′ has completed.

netapp-robert> vol status vol3

Volume State Status Options

vol3 online raid_dp, flex create_ucode=on,

convert_ucode=on, guarantee=none,

fractional_reserve=0

Containing aggregate: ‘aggr1′

netapp-robert> cifs shares -add vol3 /vol/vol3

netapp-robert> cifs shares vol3

Name Mount Point Description

—- ———– ———–

vol3 /vol/vol3

everyone / Full Control

 

on destination:

netapp-robert2> snapmirror status

Snapmirror is on.

netapp-robert2> Sat Sep 25 07:10:53 GMT [rshd_0:notice]: snapmirror_sync licensed

netapp-robert2> options snapmirror.access host=netapp-robert

netapp-robert2> options snapmirror.access

snapmirror.access host=netapp-robert

netapp-robert2> vol create vol3 -s none aggr1 40m

Creation of volume ‘vol3′ with size 40m on containing aggregate

‘aggr1′ has completed.

netapp-robert2> vol status vol3

Volume State Status Options

vol3 online raid_dp, flex create_ucode=on,

convert_ucode=on, guarantee=none,

fractional_reserve=0

Containing aggregate: ‘aggr1′

netapp-robert2> cifs shares -add vol3 /vol/vol3

netapp-robert2> cifs shares vol3

Name Mount Point Description

—- ———– ———–

vol3 /vol/vol3

everyone / Full Control

netapp-robert2> vol restrict vol3

Share vol3 disabled while volume vol3 is offline.

Volume ‘vol3′ is now restricted.

 

 

 

netapp-robert2> snapmirror initialize -S netapp-robert:vol3 netapp-robert2:vol3

Sat Sep 25 08:21:42 GMT [netapp-robert2: replication.dst.err:error]: SnapMirror: destination transfer from netapp-robert:vol3 to vol3 : cannot connect to source filer.

Transfer aborted: cannot connect to source filer. 一直遇到这个错误,运行以下命令就可以了:

Source 也运行:

netapp-robert> options snapmirror.access host=netapp-robert,netapp-robert2

 

netapp-robert2> options snapmirror.access

snapmirror.access host=netapp-robert

netapp-robert2> options snapmirror.access host=netapp-robert,netapp-robert2

netapp-robert2> options snapmirror.access

snapmirror.access host=netapp-robert,netapp-robert2

netapp-robert2> snapmirror initialize -S netapp-robert:vol3 netapp-robert2:vol3

Transfer started.

Monitor progress with ‘snapmirror status’ or the snapmirror log.

netapp-robert2> Share vol3 activated.

Sat Sep 25 08:24:45 GMT [netapp-robert2: cifs.shares.activated:info]: Activated 1 CIFS share on the volume vol3.

 

netapp-robert2> rdfile /etc/snapmirror.conf

/etc/snapmirror.conf: No such file or directory

netapp-robert2> snapmirror status -l

Snapmirror is on.

 

Source: netapp-robert:vol3

Destination: netapp-robert2:vol3

Status: Idle

Progress: -

State: Snapmirrored

Lag: 00:02:28

Mirror Timestamp: Sat Sep 25 08:24:40 GMT 2010

Base Snapshot: netapp-robert2(0099929383)_vol3.1

Current Transfer Type: -

Current Transfer Error: -

Contents: Replica

Last Transfer Type: Initialize

Last Transfer Size: 76 KB

Last Transfer Duration: 00:00:05

Last Transfer From: netapp-robert:vol3

 

netapp-robert2> vol status vol3

Volume State Status Options

vol3 online raid_dp, flex snapmirrored=on, create_ucode=on,

snapmirrored convert_ucode=on,

read-only fs_size_fixed=on, guarantee=none,

Containing aggregate: ‘aggr1′

 

 

netapp-robert> snapmirror status -l

Snapmirror is on.

 

Source: netapp-robert:vol3

Destination: netapp-robert2:vol3

Status: Idle

Progress: -

State: Source

Lag: 00:04:08

Mirror Timestamp: Sat Sep 25 08:24:40 GMT 2010

Base Snapshot: netapp-robert2(0099929383)_vol3.1

Current Transfer Type: -

Current Transfer Error: -

Contents: -

Last Transfer Type: -

Last Transfer Size: 76 KB

Last Transfer Duration: 00:00:05

Last Transfer From: -

netapp-robert> snap list

Volume vol0

working…

 

%/used %/total date name

———- ———- ———— ——–

2% ( 2%) 1% ( 1%) Sep 25 08:00 hourly.0

7% ( 6%) 3% ( 2%) Sep 22 12:00 hourly.1

12% ( 6%) 5% ( 2%) Sep 22 08:00 hourly.2

 

Volume vol3

working…

 

%/used %/total date name

———- ———- ———— ——–

30% (30%) 0% ( 0%) Sep 25 08:24 netapp-robert2(0099929383)_vol3.1 (snapmirror)

 

After setup the snapmirror, copy some files to z: like below, check w:, the files also are there too.

 

While setup sync mode, got this error:

netapp-robert> Sat Sep 25 08:42:01 GMT [netapp-robert: snapmirror.src.sync.FvolSyncTooSmall:error]: The flexible volume Synchronous SnapMirror source vol3 is 40 MB, which is smaller than the minimum supported size of 10240 MB.

 

Cluster mode testing

root@robert:/media/CDROM# ./setup.sh

 

Script version 22 (18/Sep/2007)

Where to install to? [/sim]:

Would you like to install as a cluster? [no]: yes

Continue with installation? [no]: yes

Creating /sim

Unpacking sim.tgz to /sim/node1

Configured the simulators mac address to be [00:50:56:f:27:d]

Unpacking sim.tgz to /sim/node2

Configured the simulators mac address to be [00:50:56:2:7b:b5]

Building cluster half for node /sim/node1

璁板綍浜024+0 鐨勮鍏

璁板綍浜024+0 鐨勫啓鍑

1048576瀛楄妭(1.0 MB)宸插鍒讹紝0.00758874 绉掞紝138 MB/绉

Building cluster half for node /sim/node2

璁板綍浜024+0 鐨勮鍏

璁板綍浜024+0 鐨勫啓鍑

1048576瀛楄妭(1.0 MB)宸插鍒讹紝0.00800686 绉掞紝131 MB/绉

Please ensure the simulator is not running.

Your simulator has 3 disk(s). How many more would you like to add? [0]: 40

Too high. Must be between 0 and 25.

Your simulator has 3 disk(s). How many more would you like to add? [0]: 25

 

The following disk types are available in MB:

Real (Usable)

a – 43 ( 14)

b – 62 ( 30)

c – 78 ( 45)

d – 129 ( 90)

e – 535 (450)

f – 1024 (900)

 

If you are unsure choose the default option a

What disk size would you like to use? [a]: d

Disk adapter to put disks on? [0]:

Use DHCP on first boot? [yes]:

Ask for floppy boot? [no]:

Checking the default route…

You have a single network interface called eth0 (default route) . You will not be able to access the simulator from this Linux host. If this interface is marked DOWN in ifconfig then your simulator

will crash.

Which network interface should the simulator use? [default]:

Your system has 1807MB of free memory. The smallest simulator memory you should choose is 220MB. The maximum simulator memory is 1767MB.

The recommended memory is 512MB.

How much memory would you like the simulator to use? [512]:

Create a new log for each session? [no]:

Overwrite the single log each time? [yes]:

Adding 25 additional disk(s).

Complete. Run the script /sim/node1/runsim.sh to start the first node, license clustering, then repeat with /sim/node2/runsim.sh.

 

 

Node1, runsim.sh:

Please enter the IP address for Network Interface ns0 [192.168.1.101]: 192.168.1.121

Please enter the netmask for Network Interface ns0 [255.255.255.0]:

Should interface ns0 take over a partner IP address during failover? [n]: y

The clustered failover software is not yet licensed. To enable

network failover, you should run the ‘license’ command for

clustered failover.

Please enter the IPv4 address or interface name to be taken over by ns0 []: 192.168.1.122

Please enter media type for ns0 {100tx-fd, auto} [auto]:

Please enter the IP address for Network Interface ns1 []: 192.168.1.221

Please enter the netmask for Network Interface ns1 [255.255.255.0]:

Should interface ns1 take over a partner IP address during failover? [n]: y

Please enter the IPv4 address or interface name to be taken over by ns1 []: 192.168.1.222

Please enter media type for ns1 {100tx-fd, auto} [auto]:

Would you like to continue setup through the web interface? [n]:

Please enter the name or IP address of the IPv4 default gateway [192.168.1.1]:

The administration host is given root access to the filer’s

/etc files for system administration. To allow /etc root access

to all NFS clients enter RETURN below.

 

 

A cluster site license has been installed.

    Clustered Failover will be enabled upon reboot.

    Make sure that each individual service is licensed

    on both nodes or on neither node. Remember to configure

    the network interfaces for the other node.

 

On node2:

Do you want to configure virtual network interfaces? [n]:

Please enter the IP address for Network Interface ns0 [192.168.1.103]: 192.168.1.122

Please enter the netmask for Network Interface ns0 [255.255.255.0]:

Should interface ns0 take over a partner IP address during failover? [n]: y

The clustered failover software is not yet licensed. To enable

network failover, you should run the ‘license’ command for

clustered failover.

Please enter the IPv4 address or interface name to be taken over by ns0 []: 192.168.1.121

Please enter media type for ns0 {100tx-fd, auto} [auto]:

Please enter the IP address for Network Interface ns1 []: 192.168.1.222

Please enter the netmask for Network Interface ns1 [255.255.255.0]:

Should interface ns1 take over a partner IP address during failover? [n]: y

Please enter the IPv4 address or interface name to be taken over by ns1 []: 192.168.1.221

 

login: Sun Sep 26 09:04:00 GMT [netapp2: monitor.globalStatus.critical:CRITICAL]: Cluster failover of netapp1 is not possible: cluster takeover disabled.

 

In two nodes, 都运行:cf enable:

netapp1> cf enable

Sun Sep 26 09:26:08 GMT [netapp1: cf.misc.operatorEnable:warning]: Cluster monitor: operator initiated enabling of cluster

netapp1> Sun Sep 26 09:26:08 GMT [netapp1: cf.fsm.takeoverOfPartnerDisabled:notice]: Cluster monitor: takeover of netapp2 disabled (cluster takeover disabled by partner)

Sun Sep 26 09:26:08 GMT [netapp1: cf.fsm.takeoverByPartnerDisabled:notice]: Cluster monitor: takeover of netapp1 by netapp2 disabled (unsynchronized log)

Sun Sep 26 09:26:09 GMT [netapp1: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of netapp1 by netapp2 enabled

Sun Sep 26 09:26:09 GMT [netapp1: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of netapp2 enabled

 

netapp1> cf status

Cluster enabled, netapp2 is up.

 

netapp1> ifconfig -a

ns0: flags=0x2c48867<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 192.168.1.121 netmask-or-prefix 0xffffff00 broadcast 192.168.1.255

partner inet 192.168.1.122 (not in use)

ether 00:50:56:0f:27:0d (auto-100tx-fd-up)

ns1: flags=0x2c48867<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 192.168.1.221 netmask-or-prefix 0xffffff00 broadcast 192.168.1.255

partner inet 192.168.1.222 (not in use)

ether 00:50:56:10:27:0d (auto-100tx-fd-up)

lo: flags=0×1948049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 4064

inet 127.0.0.1 netmask-or-prefix 0xff000000 broadcast 127.0.0.1

ether 00:00:00:00:00:00 (Shared memory)

 

netapp1> sysconfig -a

NetApp Release 7.3.4: Thu May 27 15:43:26 PDT 2010

System ID: 0099929383 (netapp1); partner ID: 0099923498 (netapp2)

System Serial Number: 987654-32-0 (netapp1)

 

netapp1> sysconfig -r

Aggregate aggr0 (online, raid0) (zoned checksums)

Plex /aggr0/plex0 (online, normal, active)

RAID group /aggr0/plex0/rg0 (normal)

 

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

——— —— ————- —- —- —- —– ————– ————–

data v0.16 v0 1 0 FC:A – FCAL N/A 120/246784 127/261248

data v0.17 v0 1 1 FC:A – FCAL N/A 120/246784 127/261248

data v0.18 v0 1 2 FC:A – FCAL N/A 120/246784 127/261248

 

 

Spare disks

 

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

——— —— ————- —- —- —- —– ————– ————–

Spare disks for zoned checksum traditional volumes or aggregates only

spare v0.19 v0 1 3 FC:A – FCAL N/A 120/246784 127/261248

spare v0.20 v0 1 4 FC:A – FCAL N/A 120/246784 127/261248

spare v0.21 v0 1 5 FC:A – FCAL N/A 120/246784 127/261248

spare v0.22 v0 1 6 FC:A – FCAL N/A 120/246784 127/261248

spare v0.24 v0 1 8 FC:A – FCAL N/A 120/246784 127/261248

spare v0.25 v0 1 9 FC:A – FCAL N/A 120/246784 127/261248

spare v0.26 v0 1 10 FC:A – FCAL N/A 120/246784 127/261248

spare v0.27 v0 1 11 FC:A – FCAL N/A 120/246784 127/261248

spare v0.28 v0 1 12 FC:A – FCAL N/A 120/246784 127/261248

spare v0.29 v0 1 13 FC:A – FCAL N/A 120/246784 127/261248

spare v0.32 v0 2 0 FC:A – FCAL N/A 120/246784 127/261248

spare v0.33 v0 2 1 FC:A – FCAL N/A 120/246784 127/261248

spare v0.34 v0 2 2 FC:A – FCAL N/A 120/246784 127/261248

spare v0.35 v0 2 3 FC:A – FCAL N/A 120/246784 127/261248

spare v0.36 v0 2 4 FC:A – FCAL N/A 120/246784 127/261248

spare v0.37 v0 2 5 FC:A – FCAL N/A 120/246784 127/261248

spare v0.38 v0 2 6 FC:A – FCAL N/A 120/246784 127/261248

spare v0.39 v0 2 7 FC:A – FCAL N/A 120/246784 127/261248

spare v0.40 v0 2 8 FC:A – FCAL N/A 120/246784 127/261248

spare v0.41 v0 2 9 FC:A – FCAL N/A 120/246784 127/261248

spare v0.42 v0 2 10 FC:A – FCAL N/A 120/246784 127/261248

spare v0.43 v0 2 11 FC:A – FCAL N/A 120/246784 127/261248

spare v0.44 v0 2 12 FC:A – FCAL N/A 120/246784 127/261248

spare v0.45 v0 2 13 FC:A – FCAL N/A 120/246784 127/261248

spare v0.48 v0 3 0 FC:A – FCAL N/A 120/246784 127/261248

 

Partner disks

 

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

——— —— ————- —- —- —- —– ————– ————–

partner v4.16 v4 1 0 FC:B – FCAL N/A 0/0 127/261248

partner v4.17 v4 1 1 FC:B – FCAL N/A 0/0 127/261248

 

netapp2> sysconfig -r

Aggregate aggr0 (online, raid0) (zoned checksums)

Plex /aggr0/plex0 (online, normal, active)

RAID group /aggr0/plex0/rg0 (normal)

 

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

——— —— ————- —- —- —- —– ————– ————–

data v0.16 v0 1 0 FC:A – FCAL N/A 120/246784 127/261248

data v0.17 v0 1 1 FC:A – FCAL N/A 120/246784 127/261248

 

 

Spare disks (empty)

 

Partner disks

 

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

——— —— ————- —- —- —- —– ————– ————–

partner v4.38 v4 2 6 FC:B – FCAL N/A 0/0 127/261248

partner v4.32 v4 2 0 FC:B – FCAL N/A 0/0 127/261248

partner v4.48 v4 3 0 FC:B – FCAL N/A 0/0 127/261248

partner v4.34 v4 2 2 FC:B – FCAL N/A 0/0 127/261248

partner v4.33 v4 2 1 FC:B – FCAL N/A 0/0 127/261248

partner v4.26 v4 1 10 FC:B – FCAL N/A 0/0 127/261248

partner v4.28 v4 1 12 FC:B – FCAL N/A 0/0 127/261248

partner v4.16 v4 1 0 FC:B – FCAL N/A 0/0 127/261248

partner v4.21 v4 1 5 FC:B – FCAL N/A 0/0 127/261248

partner v4.42 v4 2 10 FC:B – FCAL N/A 0/0 127/261248

partner v4.37 v4 2 5 FC:B – FCAL N/A 0/0 127/261248

partner v4.18 v4 1 2 FC:B – FCAL N/A 0/0 127/261248

partner v4.39 v4 2 7 FC:B – FCAL N/A 0/0 127/261248

partner v4.45 v4 2 13 FC:B – FCAL N/A 0/0 127/261248

partner v4.25 v4 1 9 FC:B – FCAL N/A 0/0 127/261248

partner v4.22 v4 1 6 FC:B – FCAL N/A 0/0 127/261248

partner v4.41 v4 2 9 FC:B – FCAL N/A 0/0 127/261248

partner v4.29 v4 1 13 FC:B – FCAL N/A 0/0 127/261248

partner v4.27 v4 1 11 FC:B – FCAL N/A 0/0 127/261248

partner v4.24 v4 1 8 FC:B – FCAL N/A 0/0 127/261248

partner v4.19 v4 1 3 FC:B – FCAL N/A 0/0 127/261248

partner v4.17 v4 1 1 FC:B – FCAL N/A 0/0 127/261248

partner v4.20 v4 1 4 FC:B – FCAL N/A 0/0 127/261248

partner v4.36 v4 2 4 FC:B – FCAL N/A 0/0 127/261248

partner v4.43 v4 2 11 FC:B – FCAL N/A 0/0 127/261248

partner v4.44 v4 2 12 FC:B – FCAL N/A 0/0 127/261248

partner v4.35 v4 2 3 FC:B – FCAL N/A 0/0 127/261248

partner v4.40 v4 2 8 FC:B – FCAL N/A 0/0 127/261248

 

netapp2/netapp1> partner

Logoff from partner shell: netapp2

netapp1(takeover)> Sun Sep 26 10:41:58 GMT [netapp1 (takeover): cf.partner.logoff:notice]: Logoff from partner shell: netapp2

 

netapp1(takeover)> cf giveback

Sun Sep 26 10:42:02 GMT [netapp1 (takeover): cf.misc.operatorGiveback:info]: Cluster monitor: giveback initiated by operator

Sun Sep 26 10:42:02 GMT [netapp1: cf.fm.givebackStarted:warning]: Cluster monitor: giveback started

netapp1(takeover)>

CIFS partner server is shutting down…

 

CIFS partner server has shut down…

Sun Sep 26 10:42:03 GMT [netapp1: cf.rsrc.transitTime:notice]: Top Giveback transit times wafl=470 {giveback_sync=240, sync_clean=220, forget=10, mark_abort=0, wait_offline=0, wait_create=0, vol_refs=0, abort_scans=0, drain_msgs=0, zombie_wait=0}, wafl_gb_sync=430, ndmpd=340, raid=90, registry_giveback=40, nfsd=11, route=9, priority_ha=0, fmfsm_reserve=0, raid_disaster_early=0

Sun Sep 26 10:42:03 GMT [netapp1: asup.msg.giveback.delayed:info]: giveback AutoSupport delayed 5 minutes (until after the giveback process is complete).

Sun Sep 26 10:42:04 GMT [netapp1: cf.fm.givebackComplete:warning]: Cluster monitor: giveback completed

Sun Sep 26 10:42:04 GMT [netapp1: cf.fm.givebackDuration:warning]: Cluster monitor: giveback duration time is 2 seconds

Sun Sep 26 10:42:04 GMT [netapp1: cf.fsm.stateTransit:warning]: Cluster monitor: TAKEOVER –> UP

Sun Sep 26 10:42:04 GMT [netapp1: cf.fsm.takeoverByPartnerDisabled:notice]: Cluster monitor: takeover of netapp1 by netapp2 disabled (unsynchronized log)

Sun Sep 26 10:42:09 GMT [netapp1: cf.fsm.partnerNotResponding:notice]: Cluster monitor: partner not responding

Sun Sep 26 10:42:13 GMT [netapp1: cf.fm.timeMasterStatus:info]: Acting as cluster time slave

Sun Sep 26 10:42:13 GMT [netapp1: cf.fsm.partnerOk:notice]: Cluster monitor: partner ok

Sun Sep 26 10:42:13 GMT [netapp1: cf.fsm.takeoverOfPartnerDisabled:notice]: Cluster monitor: takeover of netapp2 disabled (partner booting)

Sun Sep 26 10:42:14 GMT [netapp1: cf.fsm.takeoverOfPartnerDisabled:notice]: Cluster monitor: takeover of netapp2 disabled (unsynchronized log)

Sun Sep 26 10:42:16 GMT [netapp1: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of netapp1 by netapp2 enabled

Sun Sep 26 10:42:17 GMT [netapp1: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of netapp2 enabled

 

 

Monitoring & performance

netapp-robert> sysstat -c 10 -s 5

CPU NFS CIFS HTTP Net kB/s Disk kB/s Tape kB/s Cache

in out read write read write age

0% 0 0 0 0 0 0 0 0 0 >60

 

netapp-robert> priv set advanced

Warning: These advanced commands are potentially dangerous; use

them only when directed to do so by NetApp

personnel.

netapp-robert*> statit -b

netapp-robert*> statit -e

 

Hostname: netapp-robert ID: 0099929383 Memory: 512 MB

NetApp Release 7.3.4: Thu May 27 15:43:26 PDT 2010

<L>

Start time: Wed Sep 22 10:51:32 GMT 2010

 

CPU Statistics

4.335463 time (seconds) 100 %

0.021975 system time 1 %

0.000434 rupt time 0 % (434 rupts x 1 usec/rupt)

0.021541 non-rupt system time 0 %

4.313488 idle time 99 %

 

0.200114 time in CP 5 % 100 %

0.000020 rupt time in CP 0 % (20 rupts x 1 usec/rupt)

……

 

netapp1> cf takeover

cf: takeover initiated by operator

netapp1> Sun Sep 26 10:32:02 GMT [netapp1: cf.misc.operatorTakeover:warning]: Cluster monitor: takeover initiated by operator

Sun Sep 26 10:32:02 GMT [netapp1: cf.fsm.nfo.acceptTakeoverReq:warning]: Negotiated failover: accepting takeover request by partner, reason: operator initiated cf takeover. Asking partner to shutdown gracefully; will takeover in at most 180 seconds.

Sun Sep 26 10:32:08 GMT [netapp1: cf.fsm.firmwareStatus:info]: Cluster monitor: partner rebooting

Sun Sep 26 10:32:08 GMT [netapp1: cf.fsm.nfo.partnerShutdown:warning]: Negotiated failover: partner has shutdown

Sun Sep 26 10:32:08 GMT [netapp1: cf.fsm.takeover.nfo:info]: Cluster monitor: takeover attempted after ‘cf takeover’. command

Sun Sep 26 10:32:08 GMT [netapp1: cf.fsm.stateTransit:warning]: Cluster monitor: UP –> TAKEOVER

Sun Sep 26 10:32:08 GMT [netapp1: cf.fm.takeoverStarted:warning]: Cluster monitor: takeover started

Sun Sep 26 10:32:09 GMT [netapp1: nv.partner.disabled:info]: NVRAM takeover: Partner NVRAM was disabled.

Replaying takeover WAFL log

Sun Sep 26 10:32:11 GMT [netapp2/netapp1: wafl.takeover.nvram.missing:info]: WAFL takeover: No WAFL nvlog records were found to replay.

Sun Sep 26 10:32:11 GMT [netapp2/netapp1: wafl.replay.done:info]: WAFL log replay completed, 0 seconds

Vdisk Snap Table for host:1 is initialized

Sun Sep 26 10:32:12 GMT [netapp2/netapp1: vol.language.unspecified:info]: Language not set on volume vol0. Using language config “C”. Use vol lang to set language.

Sun Sep 26 10:33:36 GMT [netapp1 (takeover): cf.rsrc.transitTime:notice]: Top Takeover transit times registry_postrc_phase1=52451, rc=30319 {ifconfig=10470, options=10400, hostname=9100, always_do_just_after_etc_rc=90, always_do_just_before_etc_rc=80, ifconfig=60, options=40, raid_config_prerc=20, ems_postrc=19}, wafl=1200 {prvol_to_done=740, pvvols_to_done=460, paggrs_to_done=0, prvol_mnt_end=0, part_vols_mnt_end=0}, registry_prerc=740, raid=480, wafl_sync=360, wafl_restart=260 {restarters=180, vdisk=80},

Sun Sep 26 10:33:36 GMT [netapp1 (takeover): cf.fm.takeoverComplete:warning]: Cluster monitor: takeover completed

Sun Sep 26 10:33:36 GMT [netapp1 (takeover): cf.fm.takeoverDuration:warning]: Cluster monitor: takeover duration time is 88 seconds

 

netapp1(takeover)> partner

Login to partner shell: netapp2

netapp2/netapp1> Sun Sep 26 10:33:58 GMT [netapp1 (takeover): cf.partner.login:notice]: Login to partner shell: netapp2

Sun Sep 26 10:34:00 GMT [netapp1 (takeover): monitor.globalStatus.critical:CRITICAL]: This node has taken over netapp2.

Sun Sep 26 10:34:00 GMT [netapp2/netapp1: monitor.globalStatus.critical:CRITICAL]: netapp1 has taken over this node.

 

netapp2/netapp1> cf status

netapp2 has been taken over by netapp1.

 

netapp2/netapp1> sysconfig -r

*** This system has failed.

Any adapters shown below are those of the live partner, netapp1

Aggregate aggr0 (online, raid0) (zoned checksums)

Plex /aggr0/plex0 (online, normal, active)

RAID group /aggr0/plex0/rg0 (normal)

 

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

——— —— ————- —- —- —- —– ————– ————–

data v4.16 v4 1 0 FC:B – FCAL N/A 120/246784 127/261248

data v4.17 v4 1 1 FC:B – FCAL N/A 120/246784 127/261248

 

 

Spare disks (empty)

 

Partner disks

 

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

——— —— ————- —- —- —- —– ————– ————–

partner v0.38 v0 2 6 FC:A – FCAL N/A 120/246784 127/261248

partner v0.32 v0 2 0 FC:A – FCAL N/A 120/246784 127/261248

partner v0.48 v0 3 0 FC:A – FCAL N/A 120/246784 127/261248

partner v0.34 v0 2 2 FC:A – FCAL N/A 120/246784 127/261248

partner v0.33 v0 2 1 FC:A – FCAL N/A 120/246784 127/261248

partner v0.26 v0 1 10 FC:A – FCAL N/A 120/246784 127/261248

partner v0.28 v0 1 12 FC:A – FCAL N/A 120/246784 127/261248

partner v0.16 v0 1 0 FC:A – FCAL N/A 120/246784 127/261248

partner v0.21 v0 1 5 FC:A – FCAL N/A 120/246784 127/261248

partner v0.42 v0 2 10 FC:A – FCAL N/A 120/246784 127/261248

partner v0.37 v0 2 5 FC:A – FCAL N/A 120/246784 127/261248

partner v0.18 v0 1 2 FC:A – FCAL N/A 120/246784 127/261248

partner v0.39 v0 2 7 FC:A – FCAL N/A 120/246784 127/261248

partner v0.45 v0 2 13 FC:A – FCAL N/A 120/246784 127/261248

partner v0.25 v0 1 9 FC:A – FCAL N/A 120/246784 127/261248

partner v0.22 v0 1 6 FC:A – FCAL N/A 120/246784 127/261248

partner v0.41 v0 2 9 FC:A – FCAL N/A 120/246784 127/261248

partner v0.29 v0 1 13 FC:A – FCAL N/A 120/246784 127/261248

partner v0.27 v0 1 11 FC:A – FCAL N/A 120/246784 127/261248

partner v0.24 v0 1 8 FC:A – FCAL N/A 120/246784 127/261248

partner v0.19 v0 1 3 FC:A – FCAL N/A 120/246784 127/261248

partner v0.17 v0 1 1 FC:A – FCAL N/A 120/246784 127/261248

partner v0.20 v0 1 4 FC:A – FCAL N/A 120/246784 127/261248

partner v0.36 v0 2 4 FC:A – FCAL N/A 120/246784 127/261248

partner v0.43 v0 2 11 FC:A – FCAL N/A 120/246784 127/261248

partner v0.44 v0 2 12 FC:A – FCAL N/A 120/246784 127/261248

partner v0.35 v0 2 3 FC:A – FCAL N/A 120/246784 127/261248

partner v0.40 v0 2 8 FC:A – FCAL N/A 120/246784 127/261248

1. What is iSCSI?

It is a network storage protocol above TCP/IP. This protocol encapsulates SCSI data into TCP packets. iSCSI allows us to connect a host to a storage array via a simple Ethernet connection (tape drive). This solution is cheaper than the Fibre Channel SAN (Fibre channel HBAs and switches are expensive). From the host view the user sees the storage array LUNs like a local disks. iSCSI devices should not be confused with the NAS devices (for example NFS). The most important difference is that NFS volumes can be accessed by multiple hosts, but one iSCSI volume can by accessed by one host. It is similar to SCSIi protocol: usually only one host has access to one SCSI disk (the difference is the cluster enviroment). The iSCSI protocol is defined in the RFC3720 document by the IETF (Internet Engineering Task Force).

Some critics said that iSCSI has a worse performance comparing to Fibre Channel and causes high CPU load at the host machines. I think if we use Gigabit ethernet, the speed can be enough. To overcome the high CPU load, some vendors developed the iSCSI TOE-s (TCP Offload Engine). It means that the card has a built in network chip, which creates and computes the tcp frames. The Linux kernel doesn’t support directly this and the card vendors write their own drivers for the OS.

The most important iscsi terms:

Initiator:

The initiator is the name of the iSCSI client. The iSCSI client has a block level access to the iSCSI devices, which can be a disk, tape drive, DVD/CD writer. One client can use multiple iSCSI devices.

Target:

The target is the name of the iSCSI server. The iSCSI server offers its devices (disks, tape, dvd/cd … etc.) to the clients. One device can by accessed by one client.

Discovery:

Discovery is the process which shows the targets for the initiator.

 

Discovery method:

Describes the way in which the iSCSI targets can be found.The methods are currently available:

  • Internet Storage Name Service (iSNS) – Potential targets are discovered by interacting with one or more iSNS servers.
  • SendTargets – Potential targets are discovered by using a discovery-address.
  • SLP – Discover targets via Service Location protocol (RFC 4018)
  • Static – Static target adress is specified.

    iSCSI naming:

    The RFC document also covers the iSCSI names.The iSCSI name consists of two parts: type string and unique name string.

    The type string can be the following:

  • iqn. : iscsi qualifiled name
  • eui. : eui-64 bit identifier

    Most of the implementations use the iqn format. Let’S see our initiator name: iqn.1993-08.org.debian:01.35ef13adb6d

    iqn            : we use iSCSI qualified name adress.
    1993-08   : the year of the month on which the naming authority acquired the domain name which is used in the iSCSI name.
    org.debian : reversed dns name which defines the organizational naming authority.
    01.35ef13adb6d    : this string is defined by the naming authority.

    Our target name is similar (iqn.1992-08.com.netapp:sn.84211978). The difference is that contains the serial number of Netapp filer. Both names are user editable (initiator,target). We need also two ip adresses for the the target and for the initator, too.

    The following figure shows our demo environment. It consists of one Debian host which is the iSCSI initiator, and accesses the
    iSCSI disk via 
    /dev/sdb device. The Netapp filer is our iSCSI target device, which offers /vol/iscsivol/tesztlun0 disk or lun for the Debian Linux host. The iSCSI session consists of login phase, then the data exchange phase.

     

     

    2. iSCSI support on other Unix platforms

    The Cisco iSCSI Driver is one of the earliest software iSCSI initiator implementations.This driver supports all of the major commercial Unix systems and their versions (HPUX:10.20,11,11i, AIX:4.3.3,5.1,5.2, Solaris: 2.6,7,8,9). The earliest release can be dated back to 2001. Currently each Unix vendor implements its own driver, and we investigate these drivers.

    Solaris:

    Solaris 10 (from 1/06 release) supports iSCSI. The initiator driver can do the following:

  • Multiple sessions to one target support: this feature enables that one client can create more iSCSI sessions to one target as needed, and it increases the performance.
  • Multipathing: with the help of Solaris Mpxio or IPMP feature we can create redundant paths to the targets.
  • 2 Tb disks and CHAP authentication are also supported. The Solaris driver can use the three discovery methods (SLP can’t). iSCSI disks can be
    accessed by the format program.

    HPUX:

    HP supported the iSCSI from the HP11i v1 os. This driver can discover the targets via SLP (Service Location Protocol) which is also defined by IETF (RFC 4018). This means that the iSCSI initiator and targets register themselves at the SLP Directory agent. After the registration the iSCSIi initiator queries only the Directory agent. HPUX driver implements all of the discovery methods. The CHAP authentication is also implemented and the OS multipath tools (PVLinks) also supported. The HPUX driver provides transport statistics, too.

    AIX:

    From 5.2 AIX supports iSCSI.The driver implements the static target discovery only.We can use the iSCSI disks with AIX multi pathing called MPIO. The CHAP authentication is also supported.

    None of the drivers allows us to boot from iSCSI. This can be a next step in the driver development.

    3. iscsi Linux implementations

    Initiator implementations:

    Cisco also released Linux driver but it is quite old.

    The Intel iSCSI implementation contains both target and initiator drivers and a handy tool for generating workloads.

    UNH-iSCSI is an initiator and target implementation of the University of New hampshire.

    The Open-iSCSI project is the newest implementation.It can be used with 2.6.11 kernels and up. We will test this driver with the Debian host. It contains kernel modules and an iscsid daemon.

    The iscsid can be started with the following command:

    /etc/init.d/open-scsi start

    The iSCSI operations can be controlled with the iscsiadm command. The command can discover the targets, login/logout to the target, and displays the session information.

    The configuration files are under the /etc/iscsi directory:

  • iscsid.conf:         Configuration file for the iscsi daemon. It is read at startup.
  • initiatorname.iscsi:    The name of initator, which the daemon reds at the startup.
  • nodes directory:         The directory contains the nodes and their targets.
  • send_targets directory: The directory contains the discovered targets.

    The installation process is quite simple. Issue:

    apt-get install open-iscsi

    This driver implements currently the Sendtargets discovery method.

    Target implementations:

    iSCSI enterprise target is the open source target implementation for Linux. It based on the Ardis iSCSI Linux implementation and requires the 2.6.14 kernel.

    Openfiler is a quite popular Linux NAS implementation, and offers a Linux based NAS software with a web based GUI.

    Many other companies offer software-based commercial iSCSI target drivers (Amgeon, Mayastor, Chelsio).

    The storage array manufacturers offer also a native support for iSCSI (EMC, Netapp, etc.).

    We have chosen Netapp FAS filer for the testing, but you can test it with a free software. There is a link at the bottom of the article which shows how can we do it with Openfiler.

    4. Setting up the iSCSI Linux demo environment

    Our demo environment contains one Debian Linux host and one Netapp filer. The Debain host is the initiator, and the Netapp filer is the target.

    The setup process is the following briefly:

  1. We should set up the tcp/ip connection between Debian and Netapp filer. The initiator and target must ping each other. We assume that the open-iscsi package is already installed on Debian.
  2. The Debian host must discover the Netapp targets. It is called a “discovery” process. Then the target sends the target lists.
  3. The target must enable to the initator to access the LUN. On the Netapp side, it means that we should create one initiator group, which is a logical binding between the hosts and the luns. The initiator group contains the lun and one Debian host which can access this lun.
  4. When the initator gets the target lists, it must “login” to the target.
  5. When the “login” process completes successfully and Netapp filer allows the access, the initiator can use the iSCSI disk as the normal disk. It appears under /dev/sdx devices and you can format, mount it like the normal disk.

    Here are the detailed steps:

    1. We ping the Netapp filer from the Linux host:

    debian:~# ping nasa

    PING nasa (192.168.2.222) 56(84) bytes of data.

    64 bytes from nasa (192.168.2.222): icmp_seq=1 ttl=255 time=0.716 ms
    64 bytes from nasa (192.168.2.222): icmp_seq=2 ttl=255 time=0.620 ms

    It is successful.

    2. We discover the netapp filer iSCSI LUNs with the iscsiadm command. We have choosen the st (sendtargets) discovery method. Currently it is implemented with this driver:

    debian:~# iscsiadm -m discovery -t st -p 192.168.2.222

    192.168.2.222:3260 via sendtargets

    Let’s see the discovered targets:

    debian:~# iscsiadm -m node

    192.168.2.222:3260,1000 iqn.1992-08.com.netapp:sn.84211978

    3. We have to prepare the Netapp side: In this example we will create one 4GB LUN (part of the RAID group), and assign it to the Debian host. We should check the free space:

    nasa> df -k

    Filesystem               total       used      avail capacity  Mounted on
    /vol/vol0/           8388608KB   476784KB  7911824KB       6%  /vol/vol0/
    /vol/vol0/.snapshot  2097152KB    10952KB  2086200KB       1%  /vol/vol0/.snapshot
    /vol/iscsiLunVol/   31457280KB 20181396KB 11275884KB      64%  /vol/iscsiLunVol/
    /vol/iscsiLunVol/.snapshot        0KB      232KB        0KB     —%  /vol/iscsiLunVol/.snapshotunVol/testlun1

    The following command creates one 4GB Lun on the iscsiLunVol volume:

    nasa> lun create -s 4g -t linux /vol/iscsiLunVol/testlun1

    Check it:

    nasa> lun show

            /vol/iscsiLunVol/iscsitestlun    7.0g (7526131200)    (r/w, online, mapped)
    /vol/iscsiLunVol/iscsitestlun2    7.0g (7526131200)   (r/w, online, mapped)
    /vol/iscsiLunVol/testlun1      4g (4294967296)        (r/w, online)

    We should check if the Debian host is visible from netapp host:

    nasa> iscsi initiator show

    Initiators connected:
    TSIH  TPGroup  Initiator
    19    1000   debian (iqn.1993-08.org.debian:01.35ef13adb6d / 00:02:3d:00:00:00)

    Ok, we see the Debain host. Let’s create the initiator group, called Debian2.

    nasa> igroup create -i -t linux Debian2 iqn.1993-08.org.debian:01.35ef13adb6d

    nasa> igroup show

        Debian2 (iSCSI) (ostype: linux):
    iqn.1993-08.org.debian:01.35ef13adb6d (logged in on: e0a)

    We should assign the newly created Lun to the Debian2 hosts.

    nasa> lun map /vol/iscsiLunVol/testlun1 Debian2

    lun map: auto-assigned Debian2=2

    The check command:

    nasa> lun show -v

            /vol/iscsiLunVol/testlun1      4g (4294967296)    (r/w, online, mapped)
    Serial#: hpGBe4AZsnLV
    Share: none
    Space Reservation: enabled
    Multiprotocol Type: linux
    Maps: Debian2=2

    4. Lets go back to our initator host. Now everything is prepared to access the 4GB lun. The following command makes the disk accessible from the Linux host.

     iscsiadm -m node -T iqn.1992-08.com.netapp:sn.84211978 -p 192.168.2.222:3260 –login

     

    We should see the following entries in the messages file:

    debian:~# tail /var/log/messages

    Apr 13 00:31:34 debian kernel: scsi: unknown device type 31
    Apr 13 00:31:34 debian kernel:   Vendor: NETAPP    Model: LUN               Rev: 0.2
    Apr 13 00:31:34 debian kernel:   Type:   Unknown                            ANSI SCSI revision: 04
    Apr 13 00:31:34 debian kernel:   Vendor: NETAPP    Model: LUN               Rev: 0.2
    Apr 13 00:31:34 debian kernel:   Type:   Direct-Access                      ANSI SCSI revision: 04
    Apr 13 00:31:34 debian kernel: SCSI device sdb: 8388608 512-byte hdwr sectors (4295 MB)
    Apr 13 00:31:34 debian kernel: sdb: Write Protect is off
    Apr 13 00:31:34 debian kernel: SCSI device sdb: drive cache: write through
    Apr 13 00:31:34 debian kernel: sd 1:0:0:2: Attached scsi disk sdb

    The disk appears as the sdb device (/dev/sdb).

    5. We can use it as the normal disk. You can create one partion, and you can easily mount it.

    debian:~# fdisk /dev/sdb

    debian:~# mkfs /dev/sdb1 ; mount /dev/sdb1 /mnt

    If you want to use sdb after the next reboot, you should change the following entry:

    node.conn[0].startup = manual to automatic

    in the /etc/iscsi/nodes/<iscsi target name>/<ip address> file. After you change it the iSCSI daemon will login to this target. Adding an automatic mount entry (/dev/sdb1 /mnt) in the /etc/fstab file doesn’t work, because the open-iscsi daemon will start later than the mounting of filesystems. One simple script can solve this problem, which does the automatic mounting after the iSCSI daemon starts.

    The open-iscsi initiator implementation tolerates network errors well. If you disconnect the Ethernet cable and connect it again, you must start the io process again, but the reconnection occurs automatically.

    Another good solution is for the network failures, if you create multiple paths for the one LUN (For example: /dev/sdb, /dev/sdc), the initator logs in to two locations (two RAID controllers) and you make the two disks as a single logical disk using Linux multipath software (dmsetup).

    I recommend another alternative for iSCSI target implementation: Openfiler (if you cant test on the Netapp box). It is a free Linux based NAS sofware, which can be managed with a web based GUI.

    The iSCSI setup process is quite similar in the case of other Unix implementations.

    5. Summary and results

    iSCSI is a good solution for a cheap disaster recovery site.You shouldn’t buy an expensive Fibre Channel card at the disaster recovery site, you can use the Ethernet and iSCSI. You can also use it for connecting hosts to disk arrays without Fibre Channel host adapters (if the arrays are iSCSI capable).

    During the test I ran the Debian host in the Vmware player program, and my network connection was 100 Mbit/s. I cannot reach more than 15 MB/s read/write performance but it isn’t relevant. With Gigabit Ethernet you can reach much better performance, the only drawback is that it increases the CPU load (CPU must build and compute TCP frames).

     

     

     

    redhat es5中安装使用netapp simulator


    redhat es5中安装使用netapp simulator
    1、Linux中安装netapp-simulator
    netapp-simulator的安装过程并不复杂,可以直接按照提示一步步进行安装。
    安装完成后,执行以下命令
    Password:
    netapp01> Wed Mar  1 00:08:51 IST [console_login_mgr:info]: root logged in from console
    //*This will create an aggregate of 8 GB comprising of 8 disks each of 1 GB.
    It will take some while(around 15 mins) to initialize the disks. *//
    netapp01> aggr create aggr1 -t raid4 8
    Creation of an aggregate with 8 disks has been initiated.  The disks need
    to be zeroed before addition to the aggregate. The process has been initiated
    and you will be notified via the system log as disks are added.
    //*Below you can view aggregate is getting initialized.
    You can take a coffee break here !!!*//
    netapp01> aggr status aggr1
    Aggr State      Status            Options
    aggr1 creating   raid4, aggr       snapshot_autodelete=off,
    initializing      lost_write_protect=off
    Volumes:
    Plex /aggr1/plex0: offline, empty, active
    //*Once the aggregate is initialized following would be the status.*//
    netapp01> aggr status aggr1
    Aggr State      Status            Options
    aggr1 online     raid4, aggr
    Volumes:
    Plex /aggr1/plex0: online, normal, active
    RAID group /aggr1/plex0/rg0: normal
    //*You can disable snaps creatation to maximize the performance.*//
    netapp01> aggr options aggr1 nosnap on
    netapp01> aggr status aggr1

    Aggr State      Status            Options
    aggr1 online     raid4, aggr       nosnap=on
    Volumes:
    Plex /aggr1/plex0: online, normal, active
    RAID group /aggr1/plex0/rg0: normal
    //*Now you create the volume(vol1) of 2GB. This can dynamically extended as per your requirement.
    2GB would be sufficient for creating the seed database.*//
    netapp01> vol create vol1 aggr1 2g
    Creation of volume ‘vol1′ with size 2g on containing aggregate
    ‘aggr1′ has completed.
    //*Status of the volume can be checked with following command.*//
    netapp01> vol status vol1
    Volume State      Status            Options
    vol1 online     raid4, flex       create_ucode=on,
    convert_ucode=on
    Containing aggregate: ‘aggr1′

    netapp01> lun setup

    This setup will take you through the steps needed to create LUNs
    and to make them accessible by initiators. You can type ^C (Control-C)
    at any time to abort the setup and no unconfirmed changes will be made
    to the system.
    Do you want to create a LUN? [y]: y
    Multiprotocol type of LUN (image/solaris/windows/hpux/aix/linux/netware/vmware)
    [image]: linux
    A LUN path must be absolute. A LUN can only reside in a volume or
    qtree root. For example, to create a LUN with name “lun0″ in the
    qtree root /vol/vol1/q0, specify the path as “/vol/vol1/q0/lun0″.
    Enter LUN path: /vol/vol1/lun0
    A LUN can be created with or without space reservations being enabled.
    Space reservation guarantees that data writes to that LUN will never
    fail.
    Do you want the LUN to be space reserved? [y]: y
    Size for a LUN is specified in bytes. You can use single-character
    multiplier suffixes: b(sectors), k(KB), m(MB), g(GB) or t(TB).
    Enter LUN size: 2g
    You can add a comment string to describe the contents of the LUN.
    Please type a string (without quotes), or hit ENTER if you don’t
    want to supply a comment.
    Enter comment string: oradata01
    The LUN will be accessible to an initiator group. You can use an
    existing group name, or supply a new name to create a new initiator
    group. Enter ‘?’ to see existing initiator group names.
    Name of initiator group []: ora
    Type of initiator group ora (FCP/iSCSI) [iSCSI]: iSCSI
    An iSCSI initiator group is a collection of initiator node names.Each
    node name can begin with either ‘eui.’ or ‘iqn.’ and should be in the
    following formats: eui.{EUI-64 address} or iqn.yyyy-mm.{reversed domain
    name}:{optional string composed of alphanumeric characters, ‘-’, ‘.’
    and ‘:’}
    Eg: iqn.2001-04.com.acme:storage.tape.sys1.xyz or eui.02004567A425678D
    You can separate node names by commas. Enter ‘?’ to display a list of
    connected initiators. Hit ENTER when you are done adding node names to
    this group.
    Enter comma separated nodenames: iqn.1987-05.com.cisco:calvin
    Enter comma separated nodenames: iqn.1987-05.com.cisco:hobbes
    Enter comma separated nodenames:
    The initiator group has an associated OS type. The following are
    currently supported: solaris, windows, hpux, aix, linux, netware, vmware
    or default.
    OS type of initiator group “ora” [linux]:
    The LUN will be accessible to all the initiators in the
    initiator group. Enter ‘?’ to display LUNs already in use
    by one or more initiators in group “ora”.
    LUN ID at which initiator group “ora” sees “/vol/vol1/lun0″ [0]:
    LUN Path                          : /vol/vol1/lun0
    OS Type                           : linux
    Size                              : 2g (2097152000)
    Comment                           : oradata01
    Initiator Group                   : ora
    Initiator Group Type              : iSCSI
    Initiator Group Members           : iqn.1987-05.com.cisco:calvin
    : iqn.1987-05.com.cisco:hobbes
    Mapped to LUN-ID                  : 0
    Do you want to accept this configuration? [y]: y
    Do you want to create another LUN? [n]: n
    //*Check the status of the iscsi service on the NetApp Filer.*//
    netapp01> iscsi status
    iSCSI service is not running
    //*Start the service on the NetApp Filer.*//
    netapp01> iscsi start
    Wed Mar  1 23:13:42 IST [iscsi.adapter.online:notice]: ISCSI: iswta, Adapter brought online.
    iSCSI service started
    Wed Mar  1 23:13:42 IST [iscsi.service.startup:info]: iSCSI service startup
    //*Recheck the status of the iscsi service.*//
    netapp01> iscsi status
    iSCSI service is running
    //*Check the status of the LUN on the NetApp filer.*//
    netapp01> lun show
    /vol/vol1/lun0               1.6g (1712324608)    (r/w, online, mapped)
    2、在客户端连接iscsi
    在redhat es5版本中连接iscsi与之前版本有较大差异:
    # rpm -ivh iscsi-initiator-utils-6.2.0.742-0.5.el5.rpm
    (RHEL5的光盘(1/5)中有这个文件)
    # chkconfig iscsid start
    # /etc/init.d/iscsid start
    # iscsiadm -m discovery -t sendtargets -p 192.168.122.1:3260
    172.16.122.1:3260,1 iqn.2001-04.com.example:storage.disk2.sys1.xyz
    (iqn.2001……….这部份应该和iscsi服务器中ietd.conf中的标记一样)
    # iscsiadm -m node -T iqn.2001-04.com.example:storage.disk2.sys1.xyz -p 172.16.122.1:3260 -l
    (以上两个iscsiadm的操作只在第一次连接iscsi服务器时要执行,以后每当iscsid启动都会自动连接)
    # fdisk -l
    (应该可以看到多出来一个/dev/sdx的分区)
    明天要弄明白的是:如果两台机器同时连上了之后,oracle rac是如何安装的呢?

     

     

     

    文章

     


     


    就想UNIX的文件系统一样,WAFL有下面几个特点

    WAFL 是基于数据块的,使用没有片段的4KB 大小数据块
    WAFL
    使用inodes 来描述其文件
    WAFL
    文件系统中,目录是格式特殊的文件

    WAFL 使用文件来储存元数据(Meta Data)。 WAFL三个最重要的元数据文件是:
    inode
    文件,包含文件系统所有的inode
    块图文件,用以识别空闲块
    inode
    图文件,用以识别空闲inode

    每一个WAFL inode包含16个块指针,用以表明哪一个数据块属于该文件。一个WAFL inode的所有块指针指向的是相同层次上的数据块。这样,对于小于64KB的文件,inode使用16个块指针指向文件数据块;大于64KB的文件, inode使用块指针指向间接数据块,间接数据块再指向实际的文件数据块。
    较大文件的inode指向更多的间接数据块。
    对于十分小的文件,数据储存在 inode本身。

     

    说到inode,看来NETAPP已经把WAFL要用的大小给定死了。不过数量上是可以增加的,要不以后扩容就没办法了。

    可以把WAFL 理解为数据块树型结构,在树的根部是root inoderoot inode 是用以描述inode 文件的特殊inodeinode 文件包含描述系统中其它文件的inode,包括块图和inode 图文件。WAFL 文件系统数据块树型结构的树叶是所有文件的数据块。

     

    DATA ONTAP系统启动后,需要挂接WAFL的文件系统。这个时候就需要知道root inode的位置。因此root inode的位置必须要固定。当系统找到root inode后,根据树型结构才可能查找到其他文件的inode,然后再找到包含文件的具体数据块的位置。比较大的文件需要和实际数据块之间存在额外的间接层次。

    据说大机上的DB2类似于这种结构。呵呵,说是要8年的时间去学大机可能才算有个结果。分db2OS390,网络和CICS.太复杂了。

    以前听说UNISYS的大机的课程,使用磁盘的方式和我们平时见的有很大的不同,感觉空间都被浪费了。现在可能只有航空售票那个地方还在用这套系统。

     

    优化了写入性能的WAFL文件系统
    如果有安装,配置存储设备的经验的话。大家就知道在中端的EMC   CX系列产品我们会把尽量多的cache分配给写操作,而高端的DMX系列据说是微码自动优化的。那个SYMMETRIX的内核应该是只有开发系统的几个以色列人最清楚,国内的离开了LIB的支持基本出RED BOX后就抓瞎了
             
    为了有最好的写入性能最好能达到几个方面:
    1
    。将多数据块写入到同一条带中的方式,大大提高RAID 的效率与性能。
    2
    。将相关的块写入到相近的位置而大大缩短了寻道时间,硬盘的悬臂不用来回摆动,积少成多还是可观的时间,特别对压力大的系统。
    3
    。将顺序的块放置在RAID 阵列中某个单一磁盘上的方式,减少了大文件读出时
    的磁头冲突。

    安装提出的要求来实现一套新的文件系统,WAFL就是这个实现的结果。

    WAFL 能把文件系统的任何数据块 (除包含root inode 的数据块外,前面提到过这个是在硬盘的固定位置 )写到磁盘中的任何位置上。
    而在一般的文件系统中,诸如inode、位图和元数据被保存在磁盘中的固定位置上,这就妨碍了文件系统写入性能。硬盘的悬臂先回到元数据的位置,再移动到数据存储的位置。这个动作来回重复总比不动要用的时间多,很好理解。
    例如,对某一新近更新的文件,不能将它的数据块和inode 在磁盘上的紧邻位置加以存储。而WAFL 则相反,它可以在磁盘上的任何地方加以写入,从而创造性的优化了写入性能。

    NVRAM 是带有电池的独特内存,即使切断外来电源,仍然可以由电池供电保证存储数据的安全。
    在不正常停机之后,WAFL重演所有日志以防丢失。日志类似于数据库线上交易处理 (OLTP) 数据库系统的Log 记录。WAFL 采用内存,而不是硬盘来保存日志(用硬盘的话不如直接写到数据盘里),因为比硬盘快一千倍,提供极高的响应速度。即使有大量的交易,响应速度也不会变慢。当使用者的写入要求送到NVRAM 并完成时,马上可以接受下一笔的写入要求,又可同时保护metadata data 的交易纪录。当NVRAM 收集这些写入要求到达某种条件时,WAFL 会先经过计算并整理过后再批次写入到硬盘。
    现在的阵列很多都带有电池,能保证市电中断的情况下把CACHE中的所有数据全部写入到磁盘中后,阵列再自行关闭。再加上数据库的特性,基本可以保证数据的完整性。极端的情况听说过cache坏掉的,特别是没有mirror的板坏了以后很惨。毕竟存储的设计都是提交到cache就算是成功,这个时候坏了不丢数据很难啊。

    为在磁盘上保存完整的文件系统一致性版本,WAFL 文件系统至少每10 秒会生成一个内部数据快照SnapShot,称为一个一致点(consistency point)。当时Filer 启动时,WAFL 总是使用时间最近的一致点版本,这意味着即使发生掉电或者其它严重系统错误后,系统再次投入正常运转也无需耗时的文件系统检查工作。Filer 在仅仅1 2 分钟的时间内即可正常启动,而这段时间也主要是花费在磁盘定位和内存检查上。
    恢复时间根据应用的需要来确定了,几年前北京机场的那次故障恢复时间就够很多人受了。

    WAFL 通过创建内部SnapShot 来避免系统在不正常停机后对文件系统进行检查,这种特殊的SnapShot 叫做一致点,每过几秒钟就会自动创建一次。不同于其它SnapShot 快照的是,一个一致点没有名字,而且不能访问它。同于其它SnapShot 快照的是,一致点是整个文件系统的一个完全自一致的映像。当WAFL 重新启动时,它简单地回放为最近的一致点。

    在一致点之间,WAFL 也在把数据写到磁盘中,但是它仅仅写到不使用的块中,如此呈现的数据块树型结构的最近一致点仍然是完全不变的。在两个一致点之间,WAFL 需要处理数百或者成千上万的写入请求,因所以磁盘上面的文件系统映像在数秒内会保持不变,直到WAFL 写入新的一致点。磁盘上面的文件系统映像也会自动更新。

    前面说到了cache坏掉的情况,那如果NVRAM坏了咋办?
    Filer
    中同时有cacheNVRAM。一起坏的可能性比较小吧,这个问题还要找高手出来解答一下。

    NOW上查了一下,说是当两块电池都失效时,系统会crash掉。电池电量低的时候会在24小时后关机。平时可以用
    “sysconfig -v”
    命令来监控看电池的容量是否正常。
    另外看了更换电池的PDF,如果是两个filerHA,还行。关掉一个换电池就是了,但看那个步骤还是有电麻烦。以后是不是该改成在面板上直接插拔的,那样才方便。

    NVRAM 收集这些写入要求到达某种条件时,WAFL 会先经过计算并整理过后再批次写入到硬盘。这种方式就提高了写入的效率,就像是数据库对日志进行commit的动作。
    1
    。当Filer 正常关闭时,如果NVRAM 不包含任何未处理的写入请求,它被自动关闭以延长其电池寿命
    2
    。在正常系统关闭情况下,文件管理器先关闭NFSCIFS等对外提供的服务,将所有缓存中的操作写入磁盘,然后关闭NVRAM
    3
    。在系统掉电或者其它非正常状况下,系统将会自动把尚未转存到磁盘上的NFS CIFS 请求的数据保存在NVRAM 中,等下次加电时再写入磁盘。这个断电时间是有限制的,等电池耗尽,数据就不完整了。WAFL 实际上把NVRAM 划分成为两份独立的日志使用。
    当一份日志写满,WAFL 转换到另一份日志,并且开始写一致点,将与第一份日志的差异安全地记录到硬盘上。即使日志没有写满,WAFL 10 秒也要写一次新的一致点,以防止硬盘中存储的文件系统映像太过陈旧。

     

    利用NVRAM 记录NFS请求的方式比传统的利用NVRAM 为磁盘做读写缓存的方式有很多好处。
    1
    NetApp Filer 使用NVRAM 来存储未提交NFS 请求的方式则与其它一些UNIX 产品十分不同:
    其它操作系统NVRAM 作为磁盘缓存,并在磁盘层加以使用时,包含的是关键的文件系统一致性数据。如果NVRAM 发生故障,即使采用fsck 也无法恢复系统状态。
    2
    WAFL 使用NVRAM 作为文件系统的遍历来源(日志?),而不是需要利用硬盘来加以更新的高速缓存。WAFL NVRAM 空间的使用是非常有效的,例如,要求文件系统创建一个文件的请求仅需使用数百个字节即可描述。而实际的文件创建操作则可能包含着众多的数据块改变。因为WAFL 使用NVRAM 记录操作的遍历信息,而不是操作结果本身,这就使可以在Filer NVRAM 日志文件中执行得数千个操作。

    上两种机制做一下比较:
    利用硬盘当作日志,只能保障硬盘区块的运作,不能确保网络文件系统交易的安全与完整性,当遭遇网络暂时中断或系统宕机,恢复联机与运作后,中断的网络文件系统交易已经流失。不仅速度无法与NVRAM 匹敌,也无法同时保护metadata data 的交易纪录不会流失。
    别的厂家是处理NFS 请求并将其结果写入磁盘缓存。而NetApp NVRAM 采用的,对那些需要重演的请求,记录下其日志信息。
    记录NFS 请求的最终优势是它大大提高了NFS 响应时间。为了响应NFS 请求,没有NVRAM的文件系统必须更新它的内存数据结构、为新数据分配磁盘空间,但是它们只是把修改后的数据复制到NVRAM 中,而不是磁盘中。相比之下,WAFL 能更迅速准确的响应NFS 请求,因为它只需要更新其内存数据结构并且记录访问要求,而不需要为新数据分配磁盘空间,也不需要把修改后的数据复制到NVRAM 中。
    这样看来考虑WAFL的时候是充分利用的类似数据库日志的功能来实现NVRAM,并利用它远高于硬盘的速度,来回应应用。然后再后台做大数据量的数据更新动作,充分利用阵列的CPU和磁盘组的性能。Netapp 所采用的RAID技术
    以前给客户培训的时候,看客户水平,要是以前没有接触过就得从RAID讲起。不过只涉及到RAID 0,RAID 1,RAID 1+0,RAID 5.甚至RAID S。但别得RAID级别几乎不讲,因为在一般情况下阵列多采用上面几种。
    NetApp
    采用的增强型RAID4与以上介绍的WAFL 文件系统结合非常紧密,并经过了专门的优化。通过同时优化文件系统与RAID机制,NetApp RAID 在提供RAID 保护机制的同时,没有带来类似RAID 5 的性能损耗。
    同时,RAID结构可以迅速、简便的加以动态扩充。
    通过增强型RAID 4,如果一块磁盘上的一个数据块损坏,盘组中的校验盘将重新计算该数据块上的数据。并且该数据块将被映射到磁盘中新的位置上。如果某个磁盘整体损坏,校验盘能防止任何数据丢失。当替换损坏的磁盘时,校验盘被将自动重新计算其内容。
    增强型RAID 4 阵列在同一个盘组中使用一个磁盘作为校验盘,其它作为数据盘。
    每个磁盘均划分为4KB 大小的数据块。一道磁盘条带包括每个数据磁盘的一个数据块和校验盘的数据块。条带中任意一个数据块的数据丢失,均能通过校验计算重新得出。

    WAFL 文件系统利用其灵活的任意位置写入文件布局特性,继承了RAID 4 固有的优势,并且避免了其校验盘瓶颈,最大限度的发挥了RAID 4 的效率。
    WAFL
    文件系统的设计方式确保能够最大限度地减少校验盘寻址操作。WAFL 总是把相关的数据块写到彼此邻近的条带中,消除校验盘上的长时间寻址操作。只要可能,WAFL 也把多重数据块写到同样的条带中,从而进一步减少校验盘上的阻塞。

    RAID4为传统的单一奇偶校验RAID 技术,能为单个故障磁盘驱动器提供保护。在重建故障磁盘时必须确保不能发生其他磁盘故障,而且不能在读操作过程中出现不可修复的误码。如果在重建过程中出现上面任何一种事件,那么保留在RAID 阵列或卷中的部分或所有数据就可能会丢失。随着现代更大的磁盘存储介质的出现,由于磁盘容量显著增加,而误码率仍是原来的水平,这就使得出现不可修复的误码的可能性相当高。因此,传统的单一奇偶校验RAID 保护数据的方法已经超出了其保护极限。更高一级的RAID 数据保护已经发展到RAID 双奇偶校验,也称作RAID DP,它可以应用于整个网络用数据存储产品线。

    现在的硬盘FC的正常配300GSATA500G。只要盘一坏,数据的重建也是很需要注意的问题。毕竟盘太大了,而且在线系统中重建的优先级一般都很低,一个盘同步一天的情况也不少见。不像以前9G,18GSCSI硬盘的时代了。例如EMC早期BCV的限制,一有盘坏了,BCV就不能用了,这个是BCV设计上的缺陷。现在一般都采用的是CLONE的方式,就消除了这个问题。
    盘大了,数据丢失的风险就更大了。一个盘做保护不够就多再加一个。就是现在所谓的RAID6了。当然做MIRROR是最好的方法,但是投入太大了。
          
    各个厂家都有自己的解决方案,NETAPP的叫RAID DP。多加的一块盘做对角线的数据校验保护,还是很有意思的算法。具体的说明坛子里有人转过一篇文章。我就不多写了。

    最核心的WAFL和为了实现WAFL而设计的NVRAMRAID都已经介绍过了。公开资料的最低层的东西就差不多了。
    接下来说下WAFL带来的一个非常使用的功能SNAPSHOT
    现在厂家都能实现这个功能的,但每家都有自己不同的特点。NETAPP的就个人观点实现的比较好,开销相对来说要小一些。

    一份SnapShot 是文件系统的在线只读拷贝。创建文件系统的一份SnapShot 仅仅需要几秒种的时间,并且除非原始文件被删除或者更改,数据快照并不占用额外的磁盘空间。
    这种只有当数据块发生改动时才进行数据块复制的技术被称作“copy-on-write”,只有修改活动文件系统中的数据块并写入磁盘中新的位置时,SnapShot 才会占用额外的磁盘空间。
    多数情况下客户都是将快照用于做文件系统,或者数据库的备份到离线存储系统TL,VTL中。

    data ontap很小,很强大

    但是有些命令也很傻,哈哈

    now网站是个好东西,但是需要有个帐号

    我觉得snapshot是最好的快照软件,netapp白送,挺厚道的

    第一步snapshot的创建:
    WAFL
    通过对所要做做快照的文件的node 做一个完全相同的拷贝来建立新的SnapShot 快照。这个复制而成的inode 就是代表SnapShot 数据块树状结构的inode,和实际文件系统的inode 结构相同。当创建了SnapShot inode 之后,它所指向的数据块与实际文件系统inode所指向的数据块完全一致。所以除了inode 本身占用的空间之外,新建的SnapShot 并不会占用额外的空间。
    这个过程只是拷贝inode的过程,数据量非常小。

    第二步,当原始数据变化时,如何保证SNAPSHOT的数据还是快照执行时的数据
    当一个用户修改原有一个数据块时,文件系统中发生的变化。 WAFL 在新的一个数据块上面写入新的数据,并将活动文件系统指向新的数据块。而SnapShot 仍然指向原有的未经修改的数据块。随着写入动作文件系统中的文件越来越多的被加以修改,SnapShot 中所包含的活动文件系统不再使用的数据块也就越来越多。文件变动的频度决定了SnapShot 可以在磁盘上保留的时间长短,以免耗费过多的磁盘空间。
    snapshot
    适用在数据变换量不大的应用环境下,目前就到的最多的就是做了7snapshot的。一个研发中心,每天固定时间做一个snapshot。每周循环。不在原有数据块上做更新,而使用新的数据块作为改动后的数据存放点有什么好处?
          
    这样做是为了避免更新数据块的时候还需要更新该数据块相关的校验数据,那样又多出了校验数据的读出和写入的时间。当然新的数据块写入,这个时间是省不了的。
         
    别的厂商有采用在数据更新前把要更新的数据块拷贝到保留区域来实现SNAPSHOT功能的方法,比较下来,那个方法的开销要大于NETAPP所采用的方法。可能是NETAPP申请了专利,别的厂家不得已而为之吧。

    按照理论上来说快照可以产生无数个,但在NETAPP的系统中限制了最多255个。实际上也够用了。
    snapshot
    的功能是WAFL的副产品,使用非常简便。在别的厂家实现snapshot的时候往往需要在阵列初始化配置的时候要保留一定的空间作为保留区域。对于改配置比较麻烦的阵列,例如EMCSYMM,改次BIN的费用就不知道是多少了。
    这样比较下来NETAPP的方式很灵活,SNAPSHOT的功能想用就用,不用了占用的空间也可以马上释放出来。不需要做额外的配置工作。

    说到这里就顺便说一下利用snapshot的快速恢复功能。正如前面说到的,在netapp中做snapshot的时候,只拷贝了inode的信息,并且在新写入数据后,也是写到新的未使用的空间上。原来数据位置没有变化。如果要恢复到做snapshot的那个时间的数据,当然也就很简单快捷了。

     

    DATA ONTAP

     

    前面说的WAFL,NVRAM,RAID,SNAPSHOT都是些低层概念上的东西,后面就要开始学些比较实际的东西了。


    看了一下包括创建Aggregates,卷,LUNQtree以及如何把建好的东西提供出来供用户使用,无论是CIFS,NFSFTPNAS的方式,还是FC,ISCSISAN方式。

     

    DATA ONTAP的管理接口

     

    设备有了怎么用啊?总要有个接口让我们去做配置吧。DATA ONTAP提供的接口就比较多。


    从很方便的WEB接口的Operations Manager FilerView到我们这帮做UNIX的人习惯的Telnet,SSH命令行。当然还有最传统的串口,别小看串口,难免有时候会拿来救命的。

    web接口做的很好,很多任务都可以通过向导来完成,给人感觉非常简单。呵呵,为什么要搞那么复杂呢,像SYMM那个黑匣子。

     

    priv set 命令

     

    使用priv set命令可以切换命令的级别,格式是:

    priv set level

    切换到advanced命令级别

    >priv set advanced

    *>

    提示符前多了个*号,带有一些危险的命令是要求在高级模式下才能使用的。后面列表的可用命令可以和普通模式下的比较一下,命令多了很多。

     

    切换到管理员命令级别

    *>priv set admin

    >

     

    options

     

    DATA NOTAP默认的很多设置是关闭的,用options命令就可以查看到当前有哪些选项,会把系统中所有可用的都列出。在options 后面加一个你想改的参数的头字母就可以过滤出相关的,缩小范围。

    可以使用ON/OFF这两个值来打开/关闭需要的选项。

    例如,打开RSH的支持:

    options rsh.enable on

    当参数的值是字符串的时候需要用“”来把字符串给定义范围。

    ---------------------------------------------

    DR> options c

    cf.giveback.auto.cifs.terminate.minutes 5

    cf.giveback.auto.enable off

    sysconfig 命令

    sysconfig列出当前系统中硬件的配置信息。维护的时候经常使用到的命令。
    -a  
    列出所有IO设备非常详细的信息
    -v  
    -a少些。别的啥差别没看出来,呵呵。

    ----------------------------------------------
    DR> sysconfig

    Aggregates

     

    Aggregates是一个或一组Plex组成的。Plex是由一个或者多个RAID4或者RAID DP组成的 。

    在Aggregates可以做mirror保护。但需要有SyncMirror 的许可.

    当创建一个aggregate的时候,DATA ONTAP分配数据盘和校验盘到一个RAID组中,有RAID级别和大小等选项可以选择。在一个aggregate中可以创建一个或多个flexvol卷,为了扩充容量可以在现有的RAID组中增加盘或者把新加入的盘建立成新的RAID组再添加到aggregate中。

    Aggregate的大小限制是从10G到16TB。

    aggregate 相关命令

    aggr status -s

    aggr status aggr_name  -r

    DATA NOTAP支持把不同转速的盘混合起来使用,需要打开raid.rpm.ata.enableraid.rpm.fcal.enable选项。

    aggr offline
    aggr online
    aggr restrict
    aggr rename
    aggr destroy

    sysmconfig -r
    disk show -v 这个可以列出所有已分配和未分配的盘。

    DATA ONTAP中的volume

     

    在DATA ONTAP中volume分为两种:traditional volumes 和 flexible volumes

     

    traditional volumes 顾名思义,是系统开发初期采用的。有诸多限制,例如不能减小,只能在一个aggregate中,只能通过添加整个新硬盘的容量来扩容。

    flexible volumes是随技术的发展以及应用的需求开发出来的新的volume克服了一些traditional volumes的限制,一个aggreate中可以建立多个flexible volume,可以动态的按照MB的大小来调整大小等等。

    我的理解就是netapp的LVM有了一次新的突破。现在很多LVM都可以这么做了。traditional volumes就像是一个老的文件系统,必须占整个aggregate的空间,占了以后就不能再减小,要减小的方法就是把数据迁移了后删除重建。而flexible volume就灵活了许多。

    volume是包含了用户数据的文件系统,可以通过DATA ONTAP所支持的访问协议,无论是NFS,CIFS,HTTP,FTP,FCP或ISCSI提供给外部设备来访问。可以为每个flexible volume建立其snapshot,方便不同的用户根据需要来控制其数据的snapshot,避免互相影响。

     

    空间利用率的问题

     

    traditional volumes模式,10%的空间需要给WAFL overhead.剩下的90%空间中20%用于snapshot reserve,另外80%用于文件系统的空间。

    flexible volumes模式,同样10%的空间需要给WAFL overhead.剩下的90%的空间中的5%用于整个aggregate snapshot reserve.剩下的95%的空间可以用于建立一个或多个flexvol。每个flexvol建立后会有20%的空间用于它自己的snapshot reserve.

     

    Root Volumes

     

    每个存储系统都只有一个root volume,就像前面说到的,别的都是从这个根上分出去的。root volume是用来启动存储系统的,这个是唯一个拥有”root”属性的volume,在它的/etc目录中包含了所有的用户配置信息。

    其他的卷的路径名都是以/vol开头的,/vol不是个目录。它是个特殊的虚拟路径用于mount其他的目录。

     

    上面的说法有问题,已经给冬瓜头指出,有不对的地方高手要多多指教,为后来的人答疑解惑么。呵呵。·

     

    /vol/vol0 The volume name is vol0.

     

    /vol/users/cheryl Cheryl is a directory on a volume named users.

     

    作镜像就用syncmirror,两个plex做景象。

    另外,root vol不是什么”从这个根分出去的”那个根,它与其它vol差不多,只不过这个vol还存放ontap的配置信息、日志、firmware等等东西。如果愿意,你可以往vol0里面存数据。

     

    aggr 命令

     

    R1> aggr

    The following commands are available; for more information

    type “aggr help <command>”

    add mirror rename split

    copy offline restrict status

    create online scrub undestroy

    destroy options show_space verify

    media_scrub

    R1> aggr status

    Aggr State Status Options

    aggr0 online raid0, aggr root

    R1>

    R1> aggr create myaggr -t raid4 5

    Mon Mar 24 02:40:55 EDT [raid.vol.disk.add.done:notice]: Addition of Disk /myagg

     

    r/plex0/rg0/v4.39 Shelf 2 Bay 7 [NETAPP VD-100MB-FZ-520 0042] S/N [89570621]

     

    to aggregate myaggr has completed successfully

    Mon Mar 24 02:40:55 EDT [raid.vol.disk.add.done:notice]: Addition of Disk /myagg

     

    r/plex0/rg0/v4.24 Shelf 1 Bay 8 [NETAPP VD-100MB-FZ-520 0042] S/N [89570508]

     

    to aggregate myaggr has completed successfully

    Mon Mar 24 02:40:55 EDT [raid.vol.disk.add.done:notice]: Addition of Disk /myagg

     

    r/plex0/rg0/v4.38 Shelf 2 Bay 6 [NETAPP VD-100MB-FZ-520 0042] S/N [89570620]

     

    to aggregate myaggr has completed successfully

    Mon Mar 24 02:40:55 EDT [raid.vol.disk.add.done:notice]: Addition of Disk /myagg

     

    r/plex0/rg0/v4.23 Shelf 1 Bay 7 [NETAPP VD-100MB-FZ-520 0042] S/N [89570507]

     

    to aggregate myaggr has completed successfully

    Mon Mar 24 02:40:55 EDT [raid.vol.disk.add.done:notice]: Addition of Disk /myagg

     

    r/plex0/rg0/v4.37 Shelf 2 Bay 5 [NETAPP VD-100MB-FZ-520 0042] S/N [89570619]

     

    to aggregate myaggr has completed successfully

    Creation of an aggregate with 5 disks has completed.

    R1> Mon Mar 24 02:40:56 EDT [wafl.vol.add:notice]: Aggregate myaggr has been add

     

    ed to the system.

    R1> aggr status

    Aggr State Status Options

    myaggr online raid4, aggr

    aggr0 online raid0, aggr root

    -t 指定RAID类型,然后加上包含了校验盘的RAID组需要的磁盘数目。

     

    R1> aggr destroy myaggr

    aggr destroy: Aggregate ‘myaggr’ cannot be destroyed because it is online.

    You will need to take it offline before destroying it.

    R1> aggr offline myaggr

    Aggregate ‘myaggr’ is now offline.

    R1> Mon Mar 24 02:44:38 EDT [volaggr.offline:CRITICAL]: Some aggregates are offl

     

    ine. Volume creation could cause duplicate FSIDs.

     

    R1> aggr destroy myaggr

    Are you sure you want to destroy this aggregate? y

    Mon Mar 24 02:44:54 EDT [raid.config.vol.destroyed:info]: Aggregate ‘myaggr’ des

     

    troyed.

    Aggregate ‘myaggr’ destroyed.

     

    vol命令

     

    R1> vol create mytravol -t raid4 4

    Mon Mar 24 02:47:32 EDT [raid.vol.disk.add.done:notice]: Addition of Disk /mytra

     

    vol/plex0/rg0/v4.26 Shelf 1 Bay 10 [NETAPP VD-100MB-FZ-520 0042] S/N [8957051

     

    0] to volume mytravol has completed successfully

    Mon Mar 24 02:47:32 EDT [raid.vol.disk.add.done:notice]: Addition of Disk /mytra

     

    vol/plex0/rg0/v4.41 Shelf 2 Bay 9 [NETAPP VD-100MB-FZ-520 0042] S/N [89570623

     

    ] to volume mytravol has completed successfully

    Mon Mar 24 02:47:32 EDT [raid.vol.disk.add.done:notice]: Addition of Disk /mytra

     

    vol/plex0/rg0/v4.25 Shelf 1 Bay 9 [NETAPP VD-100MB-FZ-520 0042] S/N [89570509

     

    ] to volume mytravol has completed successfully

    Mon Mar 24 02:47:32 EDT [raid.vol.disk.add.done:notice]: Addition of Disk /mytra

     

    vol/plex0/rg0/v4.40 Shelf 2 Bay 8 [NETAPP VD-100MB-FZ-520 0042] S/N [89570622

     

    ] to volume mytravol has completed successfully

    Creation of a volume with 4 disks has completed.

    R1> Mon Mar 24 02:47:33 EDT [wafl.vol.add:notice]: Volume mytravol has been adde

     

    d to the system.

    Mon Mar 24 02:47:34 EDT [vol.language.changed:info]: Language on volume mytravol

     

    changed to en_US

     

    The new language mappings will be available after reboot

    Mon Mar 24 02:47:34 EDT [config_async_0:notice]: XL – Language of Volume mytravo

     

    l has been changed to en_US.

     

    R1> aggr status

    Aggr State Status Options

    mytravol online raid4, trad

    aggr0 online raid0, aggr root

     

    建立了一个traditional volume后,直接指定RAID类型和盘数。这个时候看traditional volume就是个和它名字一样的aggr.

     

    R1> aggr status -b

    Aggr Block Size (bytes) Aggr Size (blocks) FS Size (blocks)

    —— —————— —————— —————-

    myaggr 4096 102400 102400

    mytravol 4096 76800 76800

    aggr0 4096 102400 102400

     

    R1> vol status -b

    Volume Block Size (bytes) Vol Size (blocks) FS Size (blocks)

    —— —————— —————— —————-

    vol0 4096 65184 65184

    vvol1 4096 12800 12800

    svpri 4096 5120 5120

    mytravol 4096 76800 76800

    myflexvol01 4096 25600 25600

    myflexvol02 4096 5120 5120

     

    mytravol实际已经占用了整个aggr的空间。试试看能不能减小下:

    R1> vol size mytravol -50M

    vol size: Invalid operation on traditional volume ‘mytravol’

    R1>

    R1> vol add mytravol 1

    Mon Mar 24 03:08:13 EDT [raid.vol.disk.add.done:notice]: Addition of Disk /mytra

     

    vol/plex0/rg0/v4.44 Shelf 2 Bay 12 [NETAPP VD-100MB-FZ-520 0042] S/N [8957062

     

    6] to volume mytravol has completed successfully

    Addition of 1 disk to the volume has completed.

    R1> vol status -b

    Volume Block Size (bytes) Vol Size (blocks) FS Size (blocks)

    —— —————— —————— —————-

    vol0 4096 65184 65184

    vvol1 4096 12800 12800

    svpri 4096 5120 5120

    mytravol 4096 102400 102400

    myflexvol01 4096 25600 25600

    myflexvol02 4096 5120 5120

    R1> aggr status -b

    Aggr Block Size (bytes) Aggr Size (blocks) FS Size (blocks)

    —— —————— —————— —————-

    myaggr 4096 102400 102400

    mytravol 4096 102400 102400

    aggr0 4096 102400 102400

    R1>

    要增加容量就是加入整个新盘的容量。

     

    R1> vol status -b

    Volume Block Size (bytes) Vol Size (blocks) FS Size (blocks)

    —— —————— —————— —————-

    vol0 4096 65184 65184

    vvol1 4096 12800 12800

    svpri 4096 5120 5120

    mytravol 4096 102400 102400

    myflexvol01 4096 25600 25600

    myflexvol02 4096 5120 5120

    R1> vol size myflexvol01 -30m

    vol size: Flexible volume ‘myflexvol01′ size set to 70m.

    R1> vol status -b

    Volume Block Size (bytes) Vol Size (blocks) FS Size (blocks)

    —— —————— —————— —————-

    vol0 4096 65184 65184

    vvol1 4096 12800 12800

    svpri 4096 5120 5120

    mytravol 4096 102400 102400

    myflexvol01 4096 17920 17920

    myflexvol02 4096 5120 5120

    R1> vol status

    Volume State Status Options

    mytravol online raid4, trad create_ucode=on,

    convert_ucode=on

    myflexvol01 online raid4, flex create_ucode=on,

    convert_ucode=on

    vol0 online raid0, flex root, no_atime_update=on,

    create_ucode=on,

    convert_ucode=on,

    maxdirsize=2621

    myflexvol02 online raid4, flex create_ucode=on,

    convert_ucode=on

     

    flexvol就灵活多了,可以很简单的通过”+,-”加容量来调整大小。

     

    Foreign Volume

     

    用拔盘的方式做数据迁移?这个比较有意思。

     

    等我用模拟器试试看行不行,等待更新中。。。。。。

    R2> aggr create myforaggr -t raid4 3

    Sun Mar 23 00:43:22 EDT [raid.vol.disk.add.done:notice]: Addition of Disk /myforaggr/plex0/rg0/v4.35 Shelf 2 Bay 3 [NETAPP VD-100MB-FZ-520 0042] S/N [89570517] to aggregate myforaggr has completed successfully

    Sun Mar 23 00:43:22 EDT [raid.vol.disk.add.done:notice]: Addition of Disk /myforaggr/plex0/rg0/v4.20 Shelf 1 Bay 4 [NETAPP VD-100MB-FZ-520 0042] S/N [89570504] to aggregate myforaggr has completed successfully

    Sun Mar 23 00:43:22 EDT [raid.vol.disk.add.done:notice]: Addition of Disk /myforaggr/plex0/rg0/v4.34 Shelf 2 Bay 2 [NETAPP VD-100MB-FZ-520 0042] S/N [89570516] to aggregate myforaggr has completed successfully

    Creation of an aggregate with 3 disks has completed.

    R2> Sun Mar 23 00:43:22 EDT [wafl.vol.add:notice]: Aggregate myforaggr has been added to the system.

    R2> vol create myforvol001 myforaggr 20M

    Sun Mar 23 00:43:58 EDT [vol.language.changed:info]: Language on volume myforvol001 changed to en_US

     

    The new language mappings will be available after reboot

    Sun Mar 23 00:43:58 EDT [vv_config_worker:notice]: XL – Language of Volume myforvol001 has been changed to en_US.

    Creation of volume ‘myforvol001′ with size 20m on containing aggregate

    ‘myforaggr’ has completed.

    R2> vol create myforvol002 myforaggr 30M

    Sun Mar 23 00:44:09 EDT [vol.language.changed:info]: Language on volume myforvol002 changed to en_US

     

    The new language mappings will be available after reboot

    Sun Mar 23 00:44:09 EDT [vv_config_worker:notice]: XL – Language of Volume myforvol002 has been changed to en_US.

    Creation of volume ‘myforvol002′ with size 30m on containing aggregate

    ‘myforaggr’ has completed.

    创建一个名叫myforaggr的aggregate,然后在上面建立两个flexvol,20M的myforvol001和30M的myforvol002

    这个实验有点麻烦,在模拟器下迁移到另一个模拟器看来是不行

     

    查出myforaggr实际使用了哪些disks.在建立的时候没有指定盘,是由系统自己挑选的。也不晓得有什么规律。

     

    Aggregate myforaggr (online, raid4) (block checksums)

    Plex /myforaggr/plex0 (online, normal, active)

    RAID group /myforaggr/plex0/rg0 (normal)

     

    RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

    ——— —— ————- —- —- —- —– ————– ————–

    parity v4.34 v4 2 2 FC:B – FCAL N/A 120/246784 126/259200

    data v4.22 v4 1 6 FC:B – FCAL N/A 120/246784 126/259200

    data v0.35 v0 2 3 FC:A – FCAL N/A 120/246784 126/259200

    在linux系统的/sim/,disks下面找到这几个盘的模拟文件

     

    [root@DOT ,disks]# pwd

    /sim/,disks

    [root@DOT ,disks]# ls

    ,pulled v0.29:NETAPP__:VD-100MB-FZ-520_:89570513:259200

    ,reservations v0.32:NETAPP__:VD-100MB-FZ-520_:89570514:259200

    ShelfiskShelf14 v0.33:NETAPP__:VD-100MB-FZ-520_:89570515:259200

    v0.16:NETAPP__:VD-100MB-FZ-520_:72355100:259200 v0.34:NETAPP__:VD-100MB-FZ-520_:89570516:259200

    v0.17:NETAPP__:VD-100MB-FZ-520_:72355101:259200 v0.35:NETAPP__:VD-100MB-FZ-520_:89570517:259200

    v0.18:NETAPP__:VD-100MB-FZ-520_:72355102:259200 v0.36:NETAPP__:VD-100MB-FZ-520_:89570518:259200

    v0.19:NETAPP__:VD-100MB-FZ-520_:89570503:259200 v0.37:NETAPP__:VD-100MB-FZ-520_:89570619:259200

    v0.20:NETAPP__:VD-100MB-FZ-520_:89570504:259200 v0.38:NETAPP__:VD-100MB-FZ-520_:89570620:259200

    v0.21:NETAPP__:VD-100MB-FZ-520_:89570505:259200 v0.39:NETAPP__:VD-100MB-FZ-520_:89570621:259200

    v0.22:NETAPP__:VD-100MB-FZ-520_:89570506:259200 v0.40:NETAPP__:VD-100MB-FZ-520_:89570622:259200

    v0.23:NETAPP__:VD-100MB-FZ-520_:89570507:259200 v0.41:NETAPP__:VD-100MB-FZ-520_:89570623:259200

    v0.24:NETAPP__:VD-100MB-FZ-520_:89570508:259200 v0.42:NETAPP__:VD-100MB-FZ-520_:89570624:259200

    v0.25:NETAPP__:VD-100MB-FZ-520_:89570509:259200 v0.43:NETAPP__:VD-100MB-FZ-520_:89570625:259200

    v0.26:NETAPP__:VD-100MB-FZ-520_:89570510:259200 v0.44:NETAPP__:VD-100MB-FZ-520_:89570626:259200

    v0.27:NETAPP__:VD-100MB-FZ-520_:89570511:259200 v0.45:NETAPP__:VD-100MB-FZ-520_:89570627:259200

    v0.28:NETAPP__:VD-100MB-FZ-520_:89570512:259200

     

    用tar cvf 打包这几个文件,发现不行诶。硬盘开始狂转了,这几个文件应该是有些东东值得研究下。可能只能是把这几个盘覆盖另外几个试试了。用mv把v22,v34,v35覆盖到v43,v44,v45,启动系统

     

    [root@DOT sim]# ./runsim.sh

    runsim.sh script version Script version 19 (24/Nov/2005)

    This session is logged in /sim/sessionlogs/log

    Old style simulator disk name ,disks/disk.tar found. Ignoring …

     

    PANIC: Can’t find device with WWN 0×2304181510. Remove ‘,disks/,reservations’ and restart.

    version: Data ONTAP Release 7.2.1P1: Thu Jan 4 02:52:47 PST 2007 (IBM)

    cc flags: L

    dumpcore: Can’t find device with WWN 0×2304181510. Remove ‘,disks/,reservations’ and restart.

     

    删掉,disks/,reservations系统可以启动。不知道这个WWN是哪里来的。

     

    再查询aggr的信息

    Aggregate myforaggr (online, raid4) (block checksums)

    Plex /myforaggr/plex0 (online, normal, active)

    RAID group /myforaggr/plex0/rg0 (normal)

     

    RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

    ——— —— ————- —- —- —- —– ————– ————–

    parity v4.44 v4 2 12 FC:B – FCAL N/A 120/246784 126/259200

    data v4.43 v4 2 11 FC:B – FCAL N/A 120/246784 126/259200

    data v4.45 v4 2 13 FC:B – FCAL N/A 120/246784 126/259200

     

    R2*> aggr read_fsid myforaggr

    Aggregate myforaggr has an FSID of 0x48da97cb.

     

     

    顺便做的一个RAID4拔盘的实验

     

    R2*> disk simpull v4.20

    R2*> Sun Mar 23 01:43:26 EDT [raid.disk.missing:info]: Disk /myforaggr/plex0/rg0/v4.20 Shelf 1 Bay 4 [NETAPP VD-100MB-FZ-520 0042] S/N [89570504] is missing from the system

    Sun Mar 23 01:43:26 EDT [raid.config.filesystem.disk.missing:info]: File system Disk /myforaggr/plex0/rg0/v4.20 Shelf 1 Bay 4 [NETAPP VD-100MB-FZ-520 0042] S/N [89570504] is missing.

    Sun Mar 23 01:43:26 EDT [raid.rg.recons.missing:notice]: RAID group /myforaggr/plex0/rg0 is missing 1 disk(s).

    Sun Mar 23 01:43:26 EDT [raid.rg.recons.info:notice]: Spare disk v4.21 will be used to reconstruct one missing disk in RAID group /myforaggr/plex0/rg0.

    Sun Mar 23 01:43:26 EDT [raid.rg.recons.start:notice]: /myforaggr/plex0/rg0: starting reconstruction, using disk v4.21

    Sun Mar 23 01:43:27 EDT [ses.channel.rescanInitiated:info]: Initiating rescan on channel v0

    Sun Mar 23 01:43:27 EDT [ses.channel.rescanInitiated:info]: Initiating rescan on channel v4

     

    R2*> aggr status

    Aggr State Status Options

    aggr0 online raid0, aggr root

    myforaggr online raid4, aggr

    reconstruct

    R2*> aggr status -r

    Aggregate aggr0 (online, raid0) (block checksums)

    Plex /aggr0/plex0 (online, normal, active)

    RAID group /aggr0/plex0/rg0 (normal)

     

    RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

    ——— —— ————- —- —- —- —– ————– ————–

    data v4.16 v4 1 0 FC:B – FCAL N/A 120/246784 126/259200

    data v0.17 v0 1 1 FC:A – FCAL N/A 120/246784 126/259200

    data v0.18 v0 1 2 FC:A – FCAL N/A 120/246784 126/259200

    data v4.32 v4 2 0 FC:B – FCAL N/A 120/246784 126/259200

     

    Aggregate myforaggr (online, raid4, reconstruct) (block checksums)

    Plex /myforaggr/plex0 (online, normal, active)

    RAID group /myforaggr/plex0/rg0 (reconstruction 64% completed)

     

    RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

    ——— —— ————- —- —- —- —– ————– ————–

    parity v4.34 v4 2 2 FC:B – FCAL N/A 120/246784 126/259200

    data v0.21 v0 1 5 FC:A – FCAL N/A 120/246784 126/259200 (reconstruction 64% completed)

    data v0.35 v0 2 3 FC:A – FCAL N/A 120/246784 126/259200

     

     

    Spare disks

     

    RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

    ——— —— ————- —- —- —- —– ————– ————–

    Spare disks for block or zoned checksum traditional volumes or aggregates

    spare v0.19 v0 1 3 FC:A – FCAL N/A 120/246784 126/259200 (not zeroed)

    spare v0.33 v0 2 1 FC:A – FCAL N/A 120/246784 126/259200 (not zeroed)

    spare v0.37 v0 2 5 FC:A – FCAL N/A 120/246784 126/259200

    spare v0.38 v0 2 6 FC:A – FCAL N/A 120/246784 126/259200

    spare v0.40 v0 2 8 FC:A – FCAL N/A 120/246784 126/259200

    spare v0.42 v0 2 10 FC:A – FCAL N/A 120/246784 126/259200

    spare v0.44 v0 2 12 FC:A – FCAL N/A 120/246784 126/259200

    spare v4.22 v4 1 6 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.23 v4 1 7 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.24 v4 1 8 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.25 v4 1 9 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.26 v4 1 10 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.27 v4 1 11 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.28 v4 1 12 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.29 v4 1 13 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.36 v4 2 4 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.39 v4 2 7 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.41 v4 2 9 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.43 v4 2 11 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.45 v4 2 13 FC:B – FCAL N/A 120/246784 126/259200

    R2*>

     

    备用盘中的一个盘已经顶上来了,当把盘放回去的时候。新的盘作为spare的盘,不会去把以前的盘给替换回来。

    R2*> disk simpull v0.21:NETAPP__:VD-100MB-FZ-520_:89570505:259200

    disk simpull: Disk v0.21:NETAPP__:VD-100MB-FZ-520_:89570505:259200 does not exist.

    R2*> disk simpush v0.21:NETAPP__:VD-100MB-FZ-520_:89570505:259200

    R2*> Sun Mar 23 01:50:45 EDT [ses.status.driveOk:info]: The error on drive 6 on DiskShelf14 shelf v0.1 has been corrected.

    Sun Mar 23 01:50:46 EDT [raid.disk.inserted:info]: Disk v0.21 Shelf 1 Bay 5 [NETAPP VD-100MB-FZ-520 0042] S/N [89570505] has been inserted into the system

    Sun Mar 23 01:50:47 EDT [raid.assim.disk.spare:notice]: Sparing Disk /v0.21 Shelf 1 Bay 5 [NETAPP VD-100MB-FZ-520 0042] S/N [89570505], because volume is online and complete

    Sun Mar 23 01:50:56 EDT [ses.channel.rescanInitiated:info]: Initiating rescan on channel v0

    Sun Mar 23 01:50:56 EDT [ses.channel.rescanInitiated:info]: Initiating rescan on channel v4

    Sun Mar 23 01:51:16 EDT [sfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk shelves.

    在EMC SYMM中热备盘的位置很重要,换盘后会需要把热备盘恢复到以前的热备状态。这里感觉不太一样。

     

    zeroing 使用过的盘

     

    以前在一个aggregate中使用过的盘,再次使用之前可以对其做做zero的动作,下次加入新的aggregate时可以减少延时。

     

     

    Spare disks

     

    RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

    ——— —— ————- —- —- —- —– ————– ————–

    Spare disks for block or zoned checksum traditional volumes or aggregates

    spare v4.19 v4 1 3 FC:B – FCAL N/A 120/246784 126/259200 (not zeroed)

    spare v4.20 v4 1 4 FC:B – FCAL N/A 120/246784 126/259200 (not zeroed)

    spare v4.21 v4 1 5 FC:B – FCAL N/A 120/246784 126/259200 (not zeroed)

    spare v4.23 v4 1 7 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.24 v4 1 8 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.25 v4 1 9 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.26 v4 1 10 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.27 v4 1 11 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.28 v4 1 12 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.29 v4 1 13 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.33 v4 2 1 FC:B – FCAL N/A 120/246784 126/259200 (not zeroed)

    spare v4.36 v4 2 4 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.37 v4 2 5 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.38 v4 2 6 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.39 v4 2 7 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.40 v4 2 8 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.41 v4 2 9 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.42 v4 2 10 FC:B – FCAL N/A 120/246784 126/259200

    R2> disk zero spares

     

    Spare disks

     

    RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)

    ——— —— ————- —- —- —- —– ————– ————–

    Spare disks for block or zoned checksum traditional volumes or aggregates

    spare v4.19 v4 1 3 FC:B – FCAL N/A 120/246784 126/259200 (zeroing, 8% done)

    spare v4.20 v4 1 4 FC:B – FCAL N/A 120/246784 126/259200 (zeroing, 7% done)

    spare v4.21 v4 1 5 FC:B – FCAL N/A 120/246784 126/259200 (zeroing, 7% done)

    spare v4.23 v4 1 7 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.24 v4 1 8 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.25 v4 1 9 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.26 v4 1 10 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.27 v4 1 11 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.28 v4 1 12 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.29 v4 1 13 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.33 v4 2 1 FC:B – FCAL N/A 120/246784 126/259200 (zeroing, 6% done)

    spare v4.36 v4 2 4 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.37 v4 2 5 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.38 v4 2 6 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.39 v4 2 7 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.40 v4 2 8 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.41 v4 2 9 FC:B – FCAL N/A 120/246784 126/259200

    spare v4.42 v4 2 10 FC:B – FCAL N/A 120/246784 126/259200

     

    增强的内部数据可靠性保护

     

    保证数据的可靠是任何一个存储系统都必须做到的,除了前面介绍到的以后NETAPP还提供下面两种方式来增强数据可靠性

    RAID-level checksums 检验读出的数据量和写入的数据量,当它们相等的时候,确认无数据丢失。??

    Multipath I/O setup 提供内部互联的冗余。为到同一个盘提供了第二条路径,防止了单点失败的可能。

     

    RAID Group Size

     

    一个RAID Group可能包含2到28块磁盘,是由硬件平台和RAID类型来确定的。

    CX,一般一个RAID Group用5块或9块盘的情况比较多,相对来说性能较好。SYMM体系完全不一样,是用hyper来组成的Meta,做BIN的时候会去综合考虑。Netapp的情况不是太清楚了。

     

    回答ender不解的问题:

    raid level checksum其实就是三种磁盘上的checksum。netapp从初始到现在一共根据情况实现了3中checksum。即block checksum,zone checksum,8/9checksum。具体就不多讲了。BCK提一下,就是520字节每扇区,8字节的checksum不仅存放校验数据,而且还存放lost write保护信息,即fbn和tree id。其它两种checksum方式都是512字节每扇区,然后用对应的策略实现。

     

    关于multipath io,这个词用于netapp一般指的是在同一个机头上用两个口连接相同的柜子,这样会增加loop的吞吐量和iosp,仅当loop saturation的情况下能提高性能,其次还可以做为HA冗余,默认磁盘对半分给两个口,但是属于一个头。

     

    NetApp从来没说过5、9块磁盘性能最佳的说法,请不要套用emc的规则。

     

    DATA ONTAP 对网络的支持

     

    存储系统支持以下几种接口:

    10/100Base-T

    Gigabit Ethernet

    10G bps Ethernet

    FDDI

    ATM

    接口的命名规则是基于几个方面的:

    网络类型

    slot number

    port number(如果是multiport接口)

     

    模拟器中比较简单,只提供了两个接口。看系统的一个模块上都有4个口,e0a,e0b,e0c,e0d

    和上面的命名规则对应,ethernet接口,0号slot,4个接口用字符区分。

    DR> ifconfig -a

    ns0: flags=848043<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

    inet 192.168.222.150 netmask 0xffffff00 broadcast 192.168.222.255

    ether 00:50:56:aa:91:1c (Linux AF_PACKET socket)

    ns1: flags=848043<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

    inet 192.168.222.151 netmask 0xffffff00 broadcast 192.168.222.255

    ether 00:50:56:ab:91:1c (Linux AF_PACKET socket)

    lo: flags=1948049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 9188

    inet 127.0.0.1 netmask 0xff000000 broadcast 127.0.0.1

     

    系统中和网络相关的文件

     

    在Data ONTAP中,和网络相关的配置文件都保存在/etc目录下。在windows平台下,可以通过运行\\IP地址\etc$来访问这些文件。

    在没有启用NIS,DNS的情况下和UNIX类似都是通过/etc/hosts文件来做解析。

     

    /etc/rc

    /etc/syslog.conf

    /etc/dgateways

    /etc/exports

    /etc/netgroup

    /etc/hosts.equiv

     

    如果要启用DNS或NIS来解析的话,可以按照filerview提供的向导来做。按提示填入实际环境中的相关信息就可以了。这个比较简单。就无详细记录了。

     

    Virtual Interface

     

    关于trunking

    trunking是把多个网络接口或LINKS组成为一个逻辑接口的机制。在存储设备上4个端口可以组成一个trunk.

    支持trunking功能的一组以太网接口,可以在cluster环境中提供安全的takeover,以及更高的带宽。

    filer支持在同一个trunking中的端口来自于不同的物理网卡。

    有两种类型的trunk:

    single-mode

    enable one link of the trunk to take over for a failed link

    multiple-mode

    enable faster throughput by having links share network loads

     

    Single-Mode & Multimode

     

    在Single-Mode 的配置情况下,只有一个接口是处于激活状态,其他接口都是处于准备接管的状态(standby)。

    在mulitmode的配置情况下,所有接口都处于活动状态,它们共享一个MAC地址。

     

     

    还可以把它们结合起来用,先建立两个mulitmode 的VIF,然后再配置它们为一个single mode的VIF。这个实验看以后学cluster的时候,能不能把两个模拟器组合起来做测试了。以后再补。

     

    single-mode测试记录

     

    DR*> ifconfig -a

    ns0: flags=808042<BROADCAST,RUNNING,MULTICAST> mtu 1500

    ether 00:50:56:aa:91:1c (Linux AF_PACKET socket)

    ns1: flags=808042<BROADCAST,RUNNING,MULTICAST> mtu 1500

    ether 00:50:56:ab:91:1c (Linux AF_PACKET socket)

    lo: flags=1948049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 9188

    inet 127.0.0.1 netmask 0xff000000 broadcast 127.0.0.1

    DR*> vif create single singletest ns0

    DR*> vif add singletest ns1

    DR*> ifconfig -a

    ns0: flags=808042<BROADCAST,RUNNING,MULTICAST> mtu 1500

    ether 02:50:56:aa:91:1c (Linux AF_PACKET socket)

    trunked singletest

    ns1: flags=808042<BROADCAST,RUNNING,MULTICAST> mtu 1500

    ether 02:50:56:aa:91:1c (Linux AF_PACKET socket)

    trunked singletest

    lo: flags=1948049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 9188

    inet 127.0.0.1 netmask 0xff000000 broadcast 127.0.0.1

    singletest: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500

    ether 02:50:56:aa:91:1c (Disabled virtual interface)

    DR*> ifconfig singletest 192.168.222.150

    DR*> Thu Jun 26 03:42:36 EDT [pvif.switchLink:warning]: singletest: switching to ns1

     

    DR*> ifconfig -a

    ns0: flags=848043<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

    ether 02:50:56:aa:91:1c (Linux AF_PACKET socket)

    trunked singletest

    ns1: flags=848043<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

    ether 02:50:56:aa:91:1c (Linux AF_PACKET socket)

    trunked singletest

    lo: flags=1948049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 9188

    inet 127.0.0.1 netmask 0xff000000 broadcast 127.0.0.1

    singletest: flags=848043<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

    inet 192.168.222.150 netmask 0xffffff00 broadcast 192.168.222.255

    ether 02:50:56:aa:91:1c (Enabled virtual interface)

    DR*>

    DR*> vif status singletest

    default: transmit ‘IP Load balancing’, VIF Type ‘multi_mode’, fail ‘log’

    singletest: 1 link, transmit ‘none’, VIF Type ‘single_mode’ fail ‘default’

    VIF Status Up Addr_set

    up:

    ns1: state up, since 26Jun2008 03:42:36 (00:01:55)

    mediatype: Linux AF_PACKET socket

    flags: enabled

    input packets 0, input bytes 0

    output packets 42, output bytes 3488

    output probe packets 0, input probe packets 0

    up indications 1, broken indications 0

    drops (if) 0, drops (link) 0

    indication: up at 26Jun2008 03:42:36

    consecutive 114, transitions 1

    down:

    ns0: state down, since 26Jun2008 03:42:36 (00:01:55)

    mediatype: Linux AF_PACKET socket

    flags: enabled

    input packets 0, input bytes 0

    output packets 0, output bytes 0

    output probe packets 0, input probe packets 0

    up indications 1, broken indications 0

    drops (if) 0, drops (link) 0

    indication: up at 26Jun2008 03:42:36

    consecutive 114, transitions 1

     

    给VIF指定IP后,具体使用哪个端口是随机的,可以用vif favor 来切换。切换后系统会有提示。

     

    *> Thu Jun 26 03:57:37 EDT [pvif.switchLink:warning]: singletest: switching to ns0

     

    mutil-mode 测试

     

    省略了前面几个创建的命令:

     

    DR*> vif status multitest

    default: transmit ‘IP Load balancing’, VIF Type ‘multi_mode’, fail ‘log’

    multitest: 2 links, transmit ‘IP Load balancing’, VIF Type ‘multi_mode’ fail ‘default’

    VIF Status Up Addr_set

    up:

    ns1: state up, since 26Jun2008 04:07:19 (00:00:17)

    mediatype: Linux AF_PACKET socket

    flags: enabled

    input packets 0, input bytes 0

    output packets 7, output bytes 294

    up indications 1, broken indications 0

    drops (if) 0, drops (link) 0

    indication: up at 26Jun2008 04:07:19

    consecutive 17, transitions 1

    ns0: state up, since 26Jun2008 04:07:19 (00:00:17)

    mediatype: Linux AF_PACKET socket

    flags: enabled

    input packets 0, input bytes 0

    output packets 11, output bytes 690

    up indications 1, broken indications 0

    drops (if) 0, drops (link) 0

    indication: up at 26Jun2008 04:07:19

    consecutive 17, transitions 1

     

    应该是Load balancing了,但看结果NS1上的多很多,也许是模拟器的问题,也许是只用了两个ping来做测试,方法不对,呵呵。

    DR*> vif status multitest

    default: transmit ‘IP Load balancing’, VIF Type ‘multi_mode’, fail ‘log’

    multitest: 2 links, transmit ‘IP Load balancing’, VIF Type ‘multi_mode’ fail ‘default’

    VIF Status Up Addr_set

    up:

    ns1: state up, since 26Jun2008 04:07:19 (00:05:01)

    mediatype: Linux AF_PACKET socket

    flags: enabled

    input packets 0, input bytes 0

    output packets 18, output bytes 756

    up indications 1, broken indications 0

    drops (if) 0, drops (link) 0

    indication: up at 26Jun2008 04:07:19

    consecutive 299, transitions 1

    ns0: state up, since 26Jun2008 04:07:19 (00:05:01)

    mediatype: Linux AF_PACKET socket

    flags: enabled

    input packets 0, input bytes 0

    output packets 267, output bytes 19958

    up indications 1, broken indications 0

    drops (if) 0, drops (link) 0

    indication: up at 26Jun2008 04:07:19

    consecutive 299, transitions 1

     

    Message Logging

     

    系统的各项日志工作都是由daemon syslogd来王成的 。

    在/etc/syslog.conf配置文件决定了系统的日志如何被记录

    -----------------------------------------

    # $Id: //depot/prod/ontap/R7.2.1x/files/syslog.conf.sample#1 $

    # Copyright (c) 1994-1996 Network Appliance.

    # All rights reserved.

    # Sample syslog.conf file. Copy to /etc/syslog.conf to use.

    # You must use TABS for separators between fields.

     

    # Log messages of priority info or higher to the console and to /etc/messages

    *.info /dev/console

    *.info /etc/messages

     

    # Edit and uncomment following line to log all messages of priority

    # err or higher and all kernel messages to a remote host, e.g. adminhost

    # *.err;kern.* @adminhost

    ----------------------------------------

     

    信息可以被发送到console,文件,或者其他外部的机器。

    默认情况下信息是被发送到/etc/messages中。

     

     

    第三部分 NFS

     

    NFS概括

    NFS是一个由SUN公司开发的分布式的文件系统。在1980年代,用于解决在分布网路环境下资源共享的问题。NFS避免了在计算机间将文件物理的移动过程。

    NFS的服务器提供一个或多个文件系统可以被外部的客户端mount后使用。在NFS的客户端,NFS提供的网络盘类似于本地磁盘。一台NETAPP的存储设备可以做为一个NFS的服务器。NFS有三个版本v2,v3,v4.

    NFS的客户端可以通过mount命令来挂接NFS服务器上export出来的盘,为了防止重启后丢失,可以把信息写入/etc/fstab来保证重启后自动挂接设备。

     

    要提供NFS的服务功能,需要存储系统提供的服务包括:mount daemon,network lock manager,network file system daemon,status,rpcbin,rquotad and portmapper.

    上面说到的这些服务是每个NFS进程都需要的。例如rpcbind不运行的话。在NFS的客户端和服务器端不能建立连接。

     

     

    ------------------------------------------------------

    NFS

    由于员工队伍中包括原始 NFS 开发团队成员之一、NFS 版本 3 和版本 4 的合著者、NFS 版本 4 IETF(英文) 工作组共同主席以及 Linux NFS 维护人员,NetApp 成为 NFS 社区中的驱动力量。我们提供的开源包括 Linux 的 NFS 实施(英文)以及允许并行数据访问的 pNFS(英文) Linux 客户端。

    NFS是一个分布式文件系统,它容许用户访问远程系统上的文件和目录,这些文件和目录对用户来说好像是在本地。用户可以使用OS命令创建,删除,读写远程文件和目录,以及设置远程文件和目录的属性,而这一切都是在本地完成的。NFS使用RPC而不受本地机器类型、OS和网络结构的限制。

    NFS基于客户机/服务器模式。一个NFS服务器拥有本地磁盘上的文件,客户机通过NFS访问它。拥有实际磁盘,并通过NFS将这些磁盘共享出来的主机叫NFS服务器,通过NFS访问文件系统的叫NFS客户机。服务器给客户机共享文件系统的操作叫做导出(export ).客户机要访问服务器导出的文件系统之前,必须在本地先挂接(mount)这个文件系统。

    NFS服务器导出的目录叫做远程资源,并通常写做:hostname:pathname 格式,其中hostname是NFS服务器的主机名,pathname是服务器上被导出目录的绝对路径。在客户机上安装远程资源的那个目录叫做挂接点(mount point)。

    RPC

     

    远程过程调用(RPC)是一种通过网络从远程计算机程序上请求服务,而不需要了解底层网络技术的协议。RPC 协议假定某些传输协议的存在,如 TCP 或 UDP,为通信程序之间携带信息数据。在 OSI 网络通信模型中,RPC 跨越了传输层和应用层。RPC 使得开发包括网络分布式多程序在内的应用程序更加容易。

     

    RPC 采用客户机/服务器模式。请求程序就是一个客户机,而服务提供程序就是一个服务器。首先,调用进程发送一个有进程参数的调用信息到服务进程,然后等待应答信息。在服务器端,进程保持睡眠状态直到调用信息的到达为止。当一个调用信息到达,服务器获得进程参数,计算结果,发送答复信息,然后等待下一个调用信息,最后,客户端调用过程接收答复信息,获得进程结果,然后调用执行继续进行。

    portmap

     

    portmap进程是一种网络服务,它容许客户程序按照标准的方法查找与一个特定程序相关的一个端口号。portmap进程的主要功能就是吧PRC程序号转化为internet的端口号。

    当一个PRC服务器启动时,会选择一个空闲的端口号并在上面监听(每次启动后的端口号各不相同),同时它作为一个可用的服务会在portmap进程注册。一个PRC服务器对应唯一一个PRC程序号,PRC服务器告诉portmap进程它在哪个端口号上监听连接请求和为哪个PRC程序号提供服务。

    portmap进程在端口111上监听,以便客户进程容易与它取得联系。NFS服务默认的端口号是2049。

    file locking & Lock Manager & Automounter

     

    file locking 是防止多个用户同时对一个文件进行更改。

     

    Lock Manager NFS使用NLM来提供一个类似于UNIX的file lock的功能来管理客户机和服务器进程。

     

    Automounter是一个程序,可以按要求自动挂接(mount)文件系统。或者当文件系统超过某个时限后自动卸载(umount)文件系统。现在部分linux环境下对光盘和USB设备管理就是这样,方便使用。

    NFS 4.1 /pNFS

     

    在网上看到,现在开始做新版本的NFS了。V4的一个新版本,pNFS.

    Parallel NFS (pNFS) is an extension to NFS v4 that allows clients to access storage devices directly and in parallel thus eliminating the scalability and performance issues associated with NFS servers in deployment today. This is achieved by the separation of data and metadata, and moving the metadata server out of the data path.

    新技术总是很多,能活下来的不多。看三年后有没有实际的产品出现就是了。

     

    options nfs

     

    DR*> options nfs

    nfs.acache.persistence.enabled on

    nfs.assist.queue.limit 40

    nfs.export.allow_provisional_access on

     

    export and mount

     

    服务器端:

    /etc/exports文件包含了已经被export给客户端的目录和文件

    最多包含10240个条目(entry),对条目的大小没有限制

    一个条目可以分成几行

    条目包含了路径名和选项

    当options nfs.export.auto-update 设置为on的情况下,用命令exportfs或者对volume做了创建,重命名,删除等更新动作时,系统会自动更新/etc/exports文件

     

    客户端:

    mount命令被用来连接服务器端的文件系统

    需要在本机上选择或建立一个挂接点(mountpoint ),一个目录

    卸载的话使用命令umount

    已经mount的资源都包含在系统的mount table中。

    访问限制

     

    可以通过/etc/exports文件中的条目来定义,或者通过exportfs命令来指定访问限制

    哪个主机可以mount资源

    可以对mount后的文件是可以读写,还是只读

    客户端的root用户是否可以访问这些资源

    文件是否可以被带有SETUID位创建

    访问资源的用户ID(UID)

    当多个限制条件加于某个资源时,最严格的限制优先起作用

     

    /etc/exports 中各选项的解释

     

    DR*> rdfile /etc/exports

    #Auto-generated by setup Sat Mar 22 09:08:52 EDT 2008

    /vol/vol0 -sec=sys,rw,anon=0,nosuid

    /vol/vol0/home -sec=sys,rw,nosuid

    /vol/vol1 -sec=sys,rw,nosuid

     

    在/etc/exports文件中,一个被导出的NFS目录占据一行或多行,就是上面说的一个条目。格式上面例子中可以看到。

    一个带全路径的目录(绝对目录)。选项是下面说到的一个或多个设置的组合。

    ro 如果指定客户端,那该客户端以只读的方式访问被导出的目录。如果没有指定,那导出的目录对所有客户端都只读。

    rw 如果指定客户端,那该客户端以读写的方式访问被导出的目录。如果没有指定,那导出的目录对所有客户端都可读写。

    root 如果指定主机,就有root的权限。如果没有指定,那表示任何主机都没有root访问权限。

    anon 默认情况,该anon选项指定一个UID 65534。如果UID被指定为65535,root用户的访问将被禁止。如果UID被指定为0,客户端的访问都将被视为root用户。如果用一个具体的用户名来替代UID,系统会去根据/etc/nsswitch.conf中定义的查找顺序到相关配置中查询用户信息来确定UID.

    NFS mount 过程
    Client

    Process

    Server

    Here is my session ID (XID)

    Portmapper

    I see your session ID (XID)

    I want to mount

    Mount

    Who are you?

            
            
    I am UID0 and group ID (GID) All

    Mount

    OK, here is my fhandle (FSID)

    Should I use v3 TCP?

    Portmapper

    Yes on port 1024

    Can you hear me on port 1024 (SYN)?

    TCP

    Yes, let us negotiate window size (ACK)

    TCP is established (SYN, ACK)

    TCP

    OK (SYN, FIN,ACK)

    Here is my fhandle (FSID) I want to mount

    NFS

    You are connected

    一次nfs mount的全过程。

     

    在Data ONTAP中启用NFS

     

    1.需要输入NFS的license,rpc.mountd和nfsd daemons被启动

    DR*> license

    cifs site DZDACHD

    cluster not licensed

    cluster_remote not licensed

    disk_sanitization not licensed

    fcp not licensed

    flex_cache not licensed

    flex_clone not licensed

    gateway not licensed

    gateway_hitachi not licensed

    http site NAZOMKC

    iscsi site BSLRLTG

    multistore not licensed

    nearstore_asis1 not licensed

    nearstore_option not licensed

    nfs site BQOEAZL

    2.启动NFS服务

    DR*> nfs on

    NFS server is running.

    3.导出(export)文件系统供客户机使用

    4.在客户机上挂接(mount)文件系统

    exportfs 举例

     

    把/vol/vol0 export给所有客户端,并都有读写权限

    exportfs -p /vol/vol0

    exportfs -io rw /vol/vol0

     

    把/vol/vol0 export给所有客户端,并都有只读权限

    exportfs -p ro /vol/vol0

    exportfs -io ro /vol/vol0

     

    把/vol/vol0 export给10.45.67.0网段上所有的客户端,并都有读写权限

    exportfs -io rw=10.45.67.0/24 /vol/vol0

    exportfs -io rw=”network 10.45.67.0 netmask 255.255.255.0″ /vol/vol0

    exportfs -io rw=”10.45.67.0 255.255.255.0″ /vol/vol0

     

    把 /vol/vol0 export给trusted netgroup并有root访问权限, 在friendly netgroup 中的客户端有读写权限,其他客户端有只读权限:

    exportfs -io ro,root=@trusted,rw=@friendly /vol/vol0

    exportfs命令几个常用参数

     

    exportfs

    显示已经被导出的文件系统路径

     

    exportfs [ -c ] host name [ro | rw | root ]

    检查当前访问缓存中的主机权限

     

    exportfs -s pathname

    检验哪个卷被导出

     

    exportfs -r

    确认只有绑定了的exports被引导

     

    exportfs -a

    导出在/etc/exports文件中所有的条目

     

    exportfs -r

    导出在/etc/exports文件中所有的条目,将没在其中的unexport掉

    准备NFS测试环境

     

    准备NFS测试环境.前面说了很多理论上的NFS的东西,还是实践一下认识比较深刻一些.

     

    NFS服务器: Data ONTAP simulator,ip为192.168.222.100

    LINUX客户端:

    host1 192.168.222.2

    host2 192.168.222.20

    host3 192.168.200.2

     

    R1> vol create /vol/nfstest myaggr 30M

    Sat Mar 29 00:40:01 EDT [vol.language.changed:info]: Language on volume nfstest changed to en_US

    The new language mappings will be available after reboot

    Sat Mar 29 00:40:01 EDT [vv_config_worker:notice]: XL – Language of Volume nfstest has been changed to en_US.

    Creation of volume ‘nfstest’ with size 30m on containing aggregate

    ‘myaggr’ has completed.

    R1> rdfile /etc/exports

    #Auto-generated by setup Fri Mar 21 20:07:49 EDT 2008

    /vol/vol0 -sec=sys,rw,anon=0,nosuid

    /vol/vol0/home -sec=sys,rw,nosuid

    /vol/nfstest -sec=sys,rw,nosuid

     

    可见当一个新的flexvol,nfstest被创建后系统自动把其加入到/etc/exports文件中.

    这个时候在客户端可以mount.

    [root@host1 root]# mount 192.168.222.100:/vol/nfstest /mnt

    [root@host1 root]# df

    Filesystem 1K-blocks Used Available Use% Mounted on

    /dev/sda2 15891788 3839788 11244736 26% /

    /dev/sda1 101089 9426 86444 10% /boot

    none 127632 0 127632 0% /dev/shm

    192.168.222.100:/vol/nfstest

    24576 88 24488 1% /mnt

     

    但是这个时候主机是没有写权限的

    [root@host2 root]# mount 192.168.222.100:/vol/nfstest /mnt

    [root@host2 root]# touch /mnt/testfile

    touch: creating `/mnt/testfile’: Permission denied

    [root@host2 root]#

    NFS测试(续)

     

    在主机上继续mount一个NFS,/vol/vol0,这次可以写.

    [root@host2 root]# mount 192.168.222.100:/vol/vol0 /nfs

    [root@host2 root]# touch /nfs/lll

    [root@host2 root]# ls /nfs

    etc home lll

    [root@host2 root]#

    检查对比一下在/etc/exports中,/vol/vol0和/vol/nfstest有何不同

     

    R1> rdfile /etc/exports

    #Auto-generated by setup Fri Mar 21 20:07:49 EDT 2008

    /vol/vol0 -sec=sys,rw,anon=0,nosuid

    /vol/vol0/home -sec=sys,rw,nosuid

    /vol/nfstest -sec=sys,rw,nosuid

    从这里看配置应该是可以读写的.没搞明白.呵呵.

    There are two options in the “exportfs” command.One is “root=”, the other is “anon=”.

     

    1.If you use “root=<hostname or ip address>” option in the “exportfs” command, then the filer will assign UID 0 to the root user of the client host which is included in the “root=<>” option.

     

    2.If you don’t use the “root=” option while exportfs,but you use “anon=501″, then the filer will assign the UID 501 to the root user logined from eny clients

     

    3.If you use “root=” and also use “anon=” at the same time, the root user who login from the host included in the “root=” option will be assigned with the UID 0,and the root user who login from clients not included in the “root=” will be assigned with UID <anon>.

     

    4.If you don’t use the “root=” and “anon=” options , the root user of all clients will be assigned with UID 65534

     

    5.all none-root user will be assigned with the original UID of the client’s UID in the filer.

    options 中NFS版本相关项目

     

    options nfs.v2.df_2gb_lim

    Causes the storage system to return replies to the “file system statistics” NFS v2 request that shows no more than (2**31)-1 (or 2,147,483,647) total, free, or available bytes (i.e., 2 GB) on the file system.

    options nfs.v3.enable

    When enabled, the NFS server supports NFS v3. Disable this option if there is a problem with some client when using NFS v3, and that client cannot be configured to use NFS v2. Valid values for this option are on (enabled) or off (disabled).

    options nfs.v4.enable

    When enabled, the NFS server supports NFS v4. NFS v4 support is only over the TCP protocol. Valid values for this option are on (enabled) or off (disabled).

    options nfs.v4.id.domain

    This option controls the domain portion of the string form of user and group names as defined in the NFS v4 protocol. The domain name is normally taken from the NIS domain in use, or otherwise from the Domain Name System (DNS). However if this option is set, it will override this default behavior. When the option nfs.webnfs.enable is ‘on’, the NFS server supports WebNFS lookups. Valid values for this option are on (enabled) or off (disabled).

     

     

    options nfs.v4.read_delegation

    Read delegations allow NFS v4 clients to do read operations locally without contacting the server. These include open for read, read locks, and file read operations. Both the server and client must support read delegations for this feature to work. When enabled, read delegations are supported for NFS v4. This feature is not supported for NFS v2 or v3.

    options nfs.v4.write_delegation

    Write delegations allow NFS v4 clients to do write operations locally without contacting the server. These include open for write, write locks, and writing to files. Both the server and client must support write delegations for this feature to work. When enabled, write delegations are supported for NFS v4. This feature is not supported for NFS v 2 and v3. Valid values for this option are on (enabled) or off (disabled).

     

    有帖子说默认情况下V3,V4版本都是enable的,但在模拟器中默认的是V2,V3 enable.

    NFS的性能调整(一)

     

    资料里还说了一大堆关于nfs的options 看了看平时应该都用不上,看选项的名称大概也猜的出来是做什么用途的。有需要再去查就是了。

     

    从网上搜到一篇关于NFS性能调整的文章,觉的还不错。这里对照着服务器端为NETAPP的FILER来更新一下:

    首先我们需要从客户机和服务器端分别对整个NFS的环境做个仔细的分析。我们先把从客户端放在前面开始考虑。这里面没有讨论到网络上的很多设置,那个讨论起来就扯的太远了。

    一.设置合适的block size,优化传输速度

    在客户端,一个最重要优化设置就是NFS数据的传输buffer sizes,特别是 mount命令的选项 rsize and wsize.

    多数情况下mount命令默认使用4K作为这个值。在前面说到NFS版本限制的时候也说到过V2的诸多限制中有一个值就是V2最多支持rsize and wsize为8K。到了V3版本后这个值变为了32K。NETAPP对客户端的这两个参数的推荐值,linux,solaris,aix都是rsize=32768,wsize=32768。

    也可以自己在系统中用不同的值做个测试。分别用unix的特殊文件/dev/zero和/dev/null做写入和读出的测试。

    例如:写入的例子,建议是生成文件大小为系统内存的2倍为宜。

    # time dd if=/dev/zero of=/mnt/home/testfile bs=16k count=16384

    读出的例子。

    # time dd if=/mnt/home/testfile of=/dev/null bs=16k

    然后把这个NFS的文件系统umount后再 mount才用不同的值来测试。

     

    这个实验在模拟器上做就没有太大意思了,不会比系统里那个硬盘快的。

    NFS的性能调整(二)

     

    几种常用UNIX环境NETAPP针对数据库使用NFS的情况下,推荐使用的mount参数,

     

    Linux:

    rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3

     

    Solaris:

    rw,bg,hard,intr,rsize=32768,wsize=32768,proto=tcp,forcedirectio,noac,vers=3

     

    AIX:

    cio,rw,bg,hard,intr,proto=tcp,vers=3,rsize=32768,wsize=32768

     

    bg 是指background挂接,如果挂接失败将重试安装。

    hard 是指当NFS服务器没有响应的时候客户端会继续重试直到NFS服务器响应为止。与之相对,soft指没响应的时候系统会返回一个错误。

    proto是指使用TCP还是UDP协议,当然是使用TCP了。

    intr 允许通知中断一个NFS调用。当服务器没有应答需要放弃的时候有用处。 这点LINUX的要求比较奇怪。

    forcedirectio,cio这些是特定厂商自己提供的功能。都是为了更好的提供数据的数据完整性,和数据吞吐等而设定的。非数据库使用方式,不建议使用。

    ---------------------------------------------------

    Concurrent I/O provides the best performance for Oracle Databases since it:

    ! Bypasses Virtual Memory Manager (VMM) module code

    ! Avoids caching of file data in the kernel

    ! Avoids contention on the per file write lock that blocks readers, therefore relying on the

    applications to do file access synchronization

    Concurrent I/O is enabled with the “cio” mount option. More information on “cio” is available in the IBM

    paper Improving Database Performance With AIX Concurrent I/O.1

    While that paper describes concurrent I/O in relation to the JFS2 filesystem, the concepts are

    applicable to NFS starting with AIX release 5.3 ML-03.

    摘自IBM资料,关于CIO

    --------------------------------------------------

    acregmin=n:设定最小的在文件更新之前cache时间,默认是3

    acregmax=n:设定最大的在文件更新之前cache时间,默认是60

    acdirmin=n:设定最小的在目录更新之前cache时间,默认是30

    acdirmax=n:设定最大的在目录更新之前cache时间,默认是60

    actimeo=n:将acregmin、acregmax、acdirmin、acdirmax设定为同一个数值,默认是没有启用。

    LINUX的actimeo参数

    NFS的性能调整(三)

     

    在客户端有一个容易忽视掉的问题,那就是网卡的驱动程序

    最好升级网卡的驱动,特别是LINUX。多数情况下都会选择安装完整的LINUX,现在的LINUX版本都可以识别多数的网卡。但是使用的多是一个通用的驱动程序,对FC HBA也是一样的情况。建议到相关厂家网站上下载最新的驱动安装。还可以避免很多莫明其妙的问题,有些OEM的卡的驱动也特别注意。很可能是被该动过的。

    NFS的性能调整(四)

     

    在客户端和服务器端都可以使用nfsstat命令来对NFS的操作进行监控及分析。

     

     

     

    Check the following parameters:

    timeout>5%. Requests timing out before the server can answer them.

    badxid~timeout. Server slow. Check nfsstat –m.

    badxid~0 and timeouts > 3%. Packets lost in the network; check netstat. If this number is the same as bad calls, the network is congested.

    retrans. May indicate network or routing problem if retransmit >5%.

    null>0. Automounter timing out. Increase the timeout parameter on the Automounter configuration.

    In the output of the nfsstat –m command, thefollowing parameters are critical:

    srtt. Smoothed round-trip time

     

    dev. Estimated deviation

    NFS的性能调整(五)

     

    前面说到的都是从客户端来进行性能的优化,调整等。而NETAPP 的FILER做为NFS的服务器端的优化和性能调整就显示的更加重要一些。服务器端的问题会影响到所有的客户端。

    CPU的速度直接决定了对NFS请求的处理和响应。

    Memory 这个当然是越多越好。

    System bus 服务器的内部数据都是走这个。在部分阵列的宣传资料中常看到说系统内部带宽达到多少多少。

    Network 客户端所有的读写请求都要通过网络来实现。相信现在多数的网络建设都应该在千兆以上了,亲戚的网吧现在都用的是HUAWEI的千兆交换机来组网的。呵呵。

     

    NFS的性能调整(六)

     

    在DATA ONTAP上,系统提供了大量的命令来对性能数据进行收集和分析。

    sysstat, nfsstat, netstat, ifstat, nfs_hist, nfs-top, stats, statit, netdiag 和 wafl_susp。

    还有pktt在被集成在DATA ONTAP中,其收集的信息主要提供给NETAPP的技术支持使用。

     

    sysstat

     

    The sysstat command is used to report storage system aggregate performance statistics. The report includes the current CPU utilization, amount of network input/output traffic, disk I/O, and tape I/O. When invoked without any arguments, the printout is repeated every 15 seconds.

    Examples

    sysstat 1

    Displays the default output every second; requires control-C to terminate.

    sysstat -s 1

    Displays the default output every second; upon control-C termination, prints out the summary statistics.

    sysstat -c 10

    Displays the default output every 15 seconds, stopping after the 10th iteration.

    sysstat –u –c 10 –s 2

    Displays the utilization output format every 2 seconds, stopping after the 10th iteration; upon completion, prints out the summary statistics.

    sysstat –x –s 5

    Displays the extended (full) output every 5 seconds; upon control-C termination, prints out the summary statistics

     

    这些命令都有众多的参数,看一个人是否有丰富的经验也可以通过对参数的灵活应用来看哈。

    平时用的不多也记不住,用的时候man看一下就是了。

    NFS的性能调整(七)

     

    nfsstat

     

    The nfsstat command displays statistical information about the NFSand remote procedure call (RPC) interfaces to the kernel. It can also be usedto reinitialize this information. If no options are given the default is

    NetApp>nfsstat, whichwillrequest the listing of every statistical data

    The following table describes the output from the nfsstat command as server RPC displaysfields with separate values for TCP and User Datagram Protocol (UDP):

     

     

      Header
      Description
      calls
      Total RPC calls received
      badcalls
      Total number of calls rejected by the RPC layer
      nullrecv
      Number of times an RPC call was not available even though  it was believed to have been received
      badlen
      Number of RPC calls with a length shorter than that  allowed for RPC calls
      xdrcall
      Number of RPC calls whose header could not be decoded by  External Data Representation (XDR)

    The server NFS display shows the number ofNFS calls received (calls) and rejected (badcalls) and the countsand percentages for the various calls that were made.

     

    nfsstat 同样可以带很多不同的参数,其中最有用的是 nfsstat -d

    Command to Decode Mount and Export problemsThe nfsstat –d

    command displays reply cache statistics as well as incomingmessages, including allocated mbufs. This diagnostic option allows fordebugging of all NFS-related traffic on the network.

    NFS的性能调整(八)

     

    ifstat

    Displays device-level statistics for network interfaces

    Syntax ifstat [ -z ] -a | interface_name

    The ifstat command displays statistics about packets received and sent on a specified network interface or on all network interfaces. The statistics are cumulative since the storage system was booted.

    The -z argument clears the statistics. The -a argument displays statistics for all network interfaces including the virtual host and the loopback address. If you don’t use the -a argument, specify the name of a network interface.

    nfs_hist

    nfs_hist is a priv set advanced command that displays information on the processing time for NFS requests which could not be processed immediately by WAFL (i.e., they were suspended at least once). The information is given in the form of a delay time distribution, which gives the number of messages of a given type that were delayed for that amount of time (in milliseconds).

    In addition, delay distributions are given, which indicate the time various messages spent waiting to be processed in the system. Separate distributions are given for messages waiting on a queue to be processed by WAFL and those waiting on a queue to be processed by all other processes. While these delay distributions include non-NFS messages, they may be helpful in understanding how the system is working when one is attempting to understand NFS performance issues.

    The information displayed includes messages processed since the delay distributions were last zeroed with the -z option (or since reboot if the delay distributions have not been zeroed).

    netdiag

    Performs network diagnostics

    Syntax netdiag [ -s|v|d ] [ -nbate ] [ -p [ -I interface ] ]

    The netdiag command analyzes the statistics continuously gathered by the network protocol code and (if required) performs various tests to ensure the sanity of operation of the protocol code. It displays the results of the analysis (and any tests performed) along with suggested remedial actions (if any problems are found). It analyzes almost all of the statistics displayed by the many forms of the netstat command for aberrant values.

    The first form presented allows the user to specify what subset(s) of the networking subsystem to address and what kind of output to produce. The various options that influence this command form are described in detail below.

    The second form displays the usage message.

     

    NFS常见问题及解决方法:Stale NFS File handle

     

    Stale NFS File handle

     

    检查到存储系统的连接。

    检查mountpoint。

    检查客户端的 vfstab 或者 fstab 文件。

    在客户端用showmount -e filerx检查

    在服务器端用 exportfs检查

    检查服务器端 /etc/exports文件

     

    NFS常见问题及解决方法:NFS server not responding

     

    使用ping 来测试客户端和服务器端的网络连接是否正常。

    在服务器端使用ifconfig来确定网卡状态

    确认正确的NFS版本被激活

    检查所有的nfs options

    检查/etc/rc文件中和nfs options项目条目

    检查NFS的许可

     

    NFS常见问题及解决方法:Permission Denied

     

    在客户端检查showmount -e

    另外创建一个mountpoint再mount尝试一下

    在服务器端检查exportfs的输出

    检查exportfs -a的/etc/log/auditlog

    检查服务器端的文件系统是否被正确export出来,exportfs -s

    flush access cache重新exportfs,再尝试mount文件系统。

     

    对前面的部分一个小节,及个人的一点感想与建议。

     

    NFS的部分总算基本结束了。以前对NFS的使用只是知道怎么去用,并没有关心太多的东西。按照资料研究下来实际内容还不少。

    以前的工作主要是搭建SAN环境下的存储架构。见过最多的还是IBM的机器,呵呵。自己都买了一台小机器来做实验。HP和Solaris也很多,总体觉的还是IBM的用的习惯一些。也可能是98年就考了AIX的系统管理员,对这个东西先入为主了吧。还有些比较少见的像IRIX,TRU64实际上也是很不错的系统,只是可惜市场没做好。当然还有些用WINDOWS和LINUX的机器来访问EMC上的盘,往往要不了多少容量,但成本就很高了。

    开始学Netapp的东西后发现,如果是从这个产品学起的话,东西实在是不少。如果以前看过《UNIX网络编程卷2 –进程间通讯》对NFS的RPC这块就很好理解了,如果对网络很熟悉就明白为什么NETAPP都推荐NFS用TCP而不用UDP。而后面就要讲到的CIFS对MS的AD之类的知识又提出了要求。到了SAN(FC,ISCSI)后对UNIX的LVM比较熟悉那是最好不过。

    另外存储到了SAN环境,多数都是数据库应用。对数据库有个整体的认识,对于SAN的规划,阵列内部的RAID规划都有很好的帮助作用。建议要学数据库就学ORACLE,2000年自学考过的DB2虽然后面没怎么用过但对数据库的整体概念掌握的还可以,现在都没落了。学就要学ORACLE这种主流的数据库,东西很全,用途很广。

     

    CIFS是什么

     

    前面讨论了UNIX世界中我们使用NAS主要是NFS,接下来讨论在日常办公和中小企业中使用更为普遍的NAS访问的方法CIFS。

    CIFS是Common Internet FileSystem的首字母缩写,也就是通用Internet文件系统。它是一种基于服务器消息块协议SMB(Server Message Blockprotocol)的网络文件系统访问协议,最先由Microsoft公司设计并应用,主要使用在Windows客户端与服务器之间的文件访问请求。现在,CIFS协议的其他实现方式也可以使其他类型的客户端和服务器与Microsoft操作系统之间实现通信交互和操作交互。

     

    CIFS所服务的环境

     

    Netapp filer可以被加入到下面几种服务环境中:

     

    Windows workgroup

    Domain,又分为两种情况

    Windows NT 4.0

    Windows 2000 (or later) Active Directory?

    Non-Windows workgroup.

    注意: The non-Windows workgroup is also referred to as “UNIX? clear text password workgroup” and the “etc/passwd-style workgroup.”

    后面会对每种情况都做一个说明,并比较它们各自的优点和缺点.

     

    本来这部分不想讲那么细,有的东西太简单了.但为了给初学者一个完整的帮助,我把资料上关于CIFS要用到的基础知识也一并贴上来.免的去看MS那些写的绕来绕去的教材.

    Workgroup

     

    A Windows workgroup is a simple, logical group of networked machines (computers) that share resources, such as folders and files.

    Each machine has its own Security Accounts Manager database (for Windows NT) or a local security database (for Windows 2000 or later) that is used to perform user authentication and user authorization.

    Each user that wants to access resources on a machine must have a user account on that machine.

    貌似WIN2K安装好了默认的工作组名字叫”WORKGROUP”.

     

    Joining a Workgroup

     

    A machine “joins” a workgroup by broadcasting its identity to machines on the same subnet. Machine-C joins the workgroup by broadcasting its identity to Machine-A and Machine-B. The master browser in Machine-A responds by capturing the broadcast, updating the master browse list that contains all workgroup machine names, and then broadcasting to all workgroup machines that the updated list is available.

    There is a delay (up to 15 minutes) from when the master browser receives the new identity and broadcasts to the workgroup. Workgroup Machine-B and Machine-C then pull the updated master browse list to their local machine browse list. A user can find other machine names on the subnet in the browse list.

    Machine-Name Resolution in a Workgroup

     

    How does workgroup machine-name resolution work?

    A user broadcasts a name query with the requested machine name to the other machines on the subnet. Machine-C broadcasts a query for the IP address of Machine-B. Machine-B responds to the name query by broadcasting its IP address to Machine-C.

    User Authentication in a Workgroup

     

    Users are added locally to each machine.Local-user accounts are created with user names, passwords, group information,and user rights. Machine-C user has a local-user logon.

    User authentication is performed locallyon each machine. The local-user logon requires a user name and password. Usersession authentication is performed with the user name and password. When auser wants to access another machine for resources (as in a client-serverrelationship), successful user authentication establishes a session.

    Machine-B user requests user sessionauthentication with Machine-C. Machine-C authenticates Machine-B user by usinghis user name and password found in the Machine-C local-user account. AfterMachine-B user successfully authenticates, a session is established withMachine-B user and Machine-C.

    Storage System Joins a Workgroup

     

    When a storage system “joins” aworkgroup, it becomes a server that provides services to clients. In thisexample, the storage system broadcasts its identity to machines on the samesubnet.

    The master browser captures the storagesystem machine-name broadcast, updates the master browse list, and broadcaststo all workgroup machines that the updated list is available.

    Workgroup machines pull the updatedmaster browse list to their local machine. The storage system does not pull themaster browse list because its role is always a server and it does not need tofind other machines.

    User Authentication on a Storage System in a Workgroup

     

    Users (local-user accounts) are added toa storage system and user authentication is performed locally on the storagesystem. User session authentication with a user name and password authenticatesa user in order to establish a session with the storage system.

    Data access on a storage system requiresa network logon to the storage system. A user can administer a storage systemthrough the network (for example, a Telnet session) using a local account onthe storage system; however, a user cannot logon locally to a storage system toaccess data.

    Machine-B user requests user sessionauthentication with the storage system. The storage system authenticatesMachine-B user by using his user name and password found in the storage systemlocal-user account. After Machine-B user successfully authenticates, a sessionis established with Machine-B user and the storage system.

    Authenticated users can browse a storagesystem for available resources, but must be authorized to access a share andresources in a share.

    Workgroup Disadvantages

     

    The disadvantages of a workgroup are the following:

    A user must have a user account on each machine where he wants to gain access.

    Any changes to a user account (for example, passwords) must be made on each machine in the workgroup.

    Machines that join or leave (for example, machine shutdown) a workgroup must be broadcast by the master browser.

    There is a delay (up to 15 minutes) from when the master browser receives a machine broadcast and then broadcasts the availability of the updated list to the workgroup.

    A browse list cannot span subnets.

    The reliance on broadcast messaging is what restricts workgroups to the local subnet because broadcast traffic is not passed to remote subnets.

    What Is a Domain?

     

    A Windows domain is a logical group ofnetworked machines that share a central directory database located on a domaincontroller.

    The domain controller centralizes themanagement of a user’s access to the network including:

     

    User logon

     

    User authentication

     

    User group information foraccess to directory and shared resources.

    Machine-Name Resolution in a Domain

     

    Machine names and their IP addresses areadded to (or registered with) a machine-name-resolution server. This may beperformed dynamically or manually by a system administrator.

    The machine-name-resolution serverresolves machine names to IP addresses. When a user wants to access a share orresources on a machine, he sends the machine name to themachine-name-resolution server and the server returns the IP address of themachine.

    Machine-B user wants to access anothermachine and sends the machine name query to the Machine-Name Resolution Server.This server sends the resolved IP address of the machine to the Machine-B user.

    Joining a Domain

     

    Machine-B joins a domain by going to adomain controller. The domain controller adds the machine account with themachine name to the directory database.

    User Authentication in a Domain

     

    Users are added to the central directory database in the domain controller for user logon and user session authentication.

    User authentication is performed on the domain controller. The domain controller checks the directory for the user name and password to authenticate a user when a user requests the following:

    User logon access (using the domain-user account).

    User session authentication to establish a session.

     

     

    Machine-B user requests user logon access to Machine-B. The domain controller checks the Machine-B user name and password in the domain-user account. When the user information authenticates successfully, then Machine-B user has logon access to Machine-B.

    Machine-A user requests user session authentication for a session with Machine-B. Machine-B goes to the domain controller to authenticate Machine-A user. The domain controller successfully authenticates Machine-A user and a session is established with Machine-A user and Machine-B.

    Typical Machines in a Domain

     

    A typical Windows domain has the following types of machines:

    Clients: Machines that request services of a server.

    Member servers: Windows servers that are not configured as domain controllers, but are members of the domain and configured to provide resources to clients.

    Domain controllers (DCs): Windows servers configured to store and maintain a copy of a directory that has the following:

    Machine accounts (machine names) registered with the domain.

    User names and passwords for authenticating domain users.

    Domain machine-name-resolution servers: Windows servers configured to resolve machine names to IP addresses. They provide the following types of services:

    WINS (Windows Internet Naming Service) for Windows NT 4.0 domains.

    DNS (Domain Name System) for Windows 2000 (or later) domains.

    Storage System Joins a Domain

     

    When a storage system joins a domain, it becomes a member server that provides services to clients. The storage system (member server) goes to a domain controller and the domain controller adds the machine account to the directory database.

    User Authentication on a Storage System in a Domain

     

    Domain users (already added to the domain controller) can browse the storage system for available shares and then request access to the storage system and its shares and resources in a share.

    User session authentication with a user name and password is performed centrally on the domain controller; this establishes a user session with the storage system. Users must be authorized to access a share and resources in a share.

    Data access on a storage system requires a network logon to the storage system. A user can administer a storage system through the network (for example, a Telnet session) using a local account on the storage system; however, a user cannot logon locally to a storage system to access data.

    Client-B user requests user session authentication with the member server (storage system). The member server goes to the domain controller to authenticate Client-B user. The domain controller authenticates Client-B user and a session is established with Client-B user and the member server (storage system).

    Storage System Joins Windows NT 4.0 Domain 古董??

     

    1. The storage system registers with the Windows Internet Naming Service (WINS) server.

    2. The storage system asks WINS for the name and IP address of the Primary Domain Controller (PDC).

    3. The system administrator pre-creates a machine account for the storage system on the PDC before the storage system joins the domain.

    4. The storage system goes to the domain controller and joins the domain.

    5. The domain controller returns to the storage system its credentials.

    The storage system acts as a domain member server.

    Storage System Joins Windows 2000 (or later) Domain

     

    1. The storage system registers with the Domain Name System (DNS) server or a system administrator can manually add it.

    2. The storage system asks the DNS for the name and IP address of the following; domain controller in its domain, LDAP server, and Kerberos? server.

    3. The storage system goes to the domain controller and joins the domain.

    4. The domain controller returns to the storage system its credentials.

    The storage system acts as a domain member server.

    User Authentication on a Storage System in a Windows 2000 (or later) Domain

     

    The storage system acts as a domain member server. The storage system asks the DNS server to locate the domain controller in its domain. The domain controller performs the Active Directory user authentication using Kerberos. Kerberos V5 is an Internet standard security protocol for handling authentication of a user or system identity.

     

    Windows Mixed Domain Environment

     

    In a mixed Windows NT 4.0 and Windows 2000 (or later) domain environment, machine names are resolved to IP addresses using WINS and DNS.

    Both WINS and DNS can run at the same time in a Windows 2000 (or later) Active Directory-based domain.

    The older Windows clients rely on WINS only to resolve machine names. These are NetBIOS-based clients: Windows 95, Windows 98, and Windows NT.

    Non-Windows Workgroup

     

    A non-Windows workgroup is a logical group of networked machines that share resources with Windows client users; the networked machines are neither members of a Windows workgroup nor a Windows domain.

    This server environment is also called:

    UNIX clear text password workgroup

    /etc/passwd-style workgroup

     

    这个说法只看到NETAPP的资料里是这样

    Storage System as a Non-Windows Workgroup Server

     

    When a storage system becomes a non-Windows workgroup server, it provides services to clients. An example is an all-UNIX work environment with many UNIX workstations and a few Windows clients with users that need CIFS resources. Note that any UNX reference also includes LINUX.

    Servers functioning in the role of a directory store for user information (user names, passwords, and group information):

    Storage system’s local /etc/password file

    Network Information Services (NIS) server

    Lightweight Directory Access Protocol (LDAP) server

    Servers that can provide machine (host) name resolution:

    Storage system’s local /etc/hosts file

    NIS server

    Domain Name System (DNS) server

    NIS Server vs. LDAP and DNS Servers

     

    An NIS server provides machine-name resolution to IP addresses, and user names, passwords, and group information for user authentication and authorization.

    In an LDAP and DNS server environment, a DNS server provides machine-name resolution to IP addresses, and an LDAP server provides user names, passwords, and group information for user authentication and authorization.

    NOTE: An LDAP server requires a DNS server to handle machine name resolution.

     

     

    用NIS一个就行,LDAP要和DNS搭配才行

    User Authentication on a Storage System in a Non-Windows Workgroup

     

    The storage system performs user authentication with user information.

    User and group information can be added to one or both of the following:

    Locally to the storage system /etc/passwd file (user names and passwords) and /etc/group (group information).

    Centrally to an NIS or LDAP server.

    The /etc/nsswitch.conf file sets the order of precedence for where a storage system goes to search for user and group information.

    三种方式不同的优点

     

    Workgroup Advantages

     

     

    Does not require a machinerunning a Windows server to hold centralized security information.

     

    Saves money by not having tobuy Microsoft server licenses.

     

    Simple to design and implement.

     

    Convenient for a limited numberof machines in close proximity.

     

    Storage system access limitedto 96 local clients.

     

    Domain Advantages

    Centralized administration ofall user information.

     

    User information is modified ina central location rather than being replicated throughout the environment.

     

    A single authentication sitefor all domain account logons for local and user session authentication; a usermust also have permission to access network resources.

     

    More scalability because ofcentralized administration.

     

    Non-Windows Workgroup Advantages

    In an all-UNIX environment,CIFS can be licensed on a storage system with CIFS shares made available to thefew Windows client users.

     

    User authentication can beperformed with existing authentication mechanisms in a UNIX environment, usingan NIS or LDAPserver, or locally in the /etc/passwd file as a directory store for user andgroup information.

     

    Machine name resolution can behandled by an NISor DNS server.

    CIFS Features in Windows Workgroups and Domains

     

    The following are some CIFS features available in a Windows workgroup and domain:

    Network browsing to locate machines within a domain or workgroup (provided by a browse list) and shares that are available on each machine (provided by that machine).

    Network printing

    NOTE: This feature is not supported by the storage system.

    User authentication to establish sessions.

     

    Share access authorization(share permissions)

     

    Folder and file authorization(NTFS or local permissions)

     

    Basic file attributes of read-only,archive, system, and hidden file attributes

     

    Extended NTFS file attributesof indexing, compression, and encryption

    NOTE: These attributes are not supported by the storage system.

     

    Unicode support

    Universal character encodingstandard provides a unique number for every character no matter what theplatform, program, or language.

    Characters represented by morethan 8 bits.

     

    Dialect negotiation

    Each protocol version isreferred to as a “dialect” and assigned a unique string identifier.

     

    Opportunistic locks (oplocksfor domains only)

    Guarantee to the client thatfile content is not allowed to be changed by the server, or if some change isimminent, the client is notified before the change proceeds.

    Used to synchronize data andenhance performance.

    Client-Server Communications for Session, Share Access, and File Authorization

     

    This example demonstrates client-server communications for session, share access, and file authorization. The following are the basic steps.

    1. The client contacts the server and requests CIFS dialect.

    2. The server responds with supported CIFS dialect and next logon step.

    3. The client responds with username and password.

    4. The server sends a UID (User ID) if the username and password are accepted or an error if not accepted.

    5. The client requests access to a share.

    The storage system caches all security IDs (SIDs) and usernames received from the domain controller at boot time.

    6
    The server responds with a treeID to the requested share (if access is allowed).
    7.
    The client requests to open afile on a share.
    8.
    If access is allowed, theserver responds with the ID of the requested file.
    9.
    The client requests that theserver read the data and return its contents.
    10.
    The server sends the requesteddata.
    During this process, the Access ControlLists (ACLs) are checked for permissions.

     

    NETAPP FILER上使用CIFS

    CIFS要用的基本知识在上面也讲过了,实际对于所有使用CIFS的存储这些东西都是一样的.

    下面就开始在simulator中记录学习和实验的过程.

    给CIFS功能加入许可

     

    R1> license add DZDACHD

    A cifs site license has been installed.

    Run cifs setup to enable cifs.

    R1> Sat Mar 29 02:50:32 EDT [rc:notice]: cifs licensed

     

    NETAPP的功能在出厂的时候都有了,只要加入相应的LICENSE就可以了 .

     

    EMC 的CX 的部分功能还要去装个enabler,然后SP会重启,呵呵.这个还是有点风险的.还是把功能都加入系统中,然后根据需要购买后添加比较好,免得在现场的工程师对SP重启提心吊胆的,呵呵.

    配置WORKGROUP认证方式

     

    输入了lincense后,就可以根据需要来配置CIFS的用户认证方式了.最常用也是比较简单的是选用workgroup的方式.

     

    R2> cifs setup

    This process will enable CIFS access to the filer from a Windows(R) system.

    Use “?” for help at any prompt and Ctrl-C to exit without committing changes.

    Do you want to continue and change the current filer account information? [n]: y

    Your filer does not have WINS configured and is visible only to

    clients on the same subnet.

    Do you want to make the system visible via WINS? [n]:

    This filer is currently configured as a multiprotocol filer.

    Would you like to reconfigure this filer to be an NTFS-only filer? [n]:

    The default name for this CIFS server is ‘R2′.

    Would you like to change this name? [n]:

    Data ONTAP CIFS services support four styles of user authentication.

    Choose the one from the list below that best suits your situation.

     

    (1) Active Directory domain authentication (Active Directory domains only)

    (2) Windows NT 4 domain authentication (Windows NT or Active Directory domains)

    (3) Windows Workgroup authentication using the filer’s local user accounts

    (4) /etc/passwd and/or NIS/LDAP authentication

     

    Selection (1-4)? [1]: 3

    What is the name of the Workgroup? [WORKGROUP]:

     

    CIFS – Starting SMB protocol…

    Welcome to the WORKGROUP Windows(R) workgroup

     

    CIFS local server is running.

    Workgroup方式中的用户

     

    当对CIFS的服务采用WORKGROUP的方式的时候,在存储系统上我们必须要有合适的用户名和口令才可以访问存储.

    这就需要在存储上采用命令useradmin来创建和管理了

    R2*> useradmin

    Usage:

    useradmin user add <login_name> [options]

    modify <login_name> [options]

    delete <login_name>

    list [options]

    group add <group_name> [options]

    modify <group_name> [options]

    delete <group_name> [options]

    list [options]

    role add <role_name> [options]

    modify <role_name> [options]

    delete <role_name> [options]

    list [options]

    domainuser add <user_name> [options]

    delete [options]

    list [options]

    load <filename>

    For more detailed information about each subcommand, use:

     

    useradmin help { user | group | role | domainuser }

    R2*>

    建立一个普通用户

     

    R2*> useradmin user add testcifs -g Users

    New password:

    Retype new password:

    User <testcifs> added.

    R2*> Thu Jul 3 02:38:59 EDT [useradmin.added.deleted:info]: The user ‘testcifs’ has been added.

     

    在这个例子中建立了一个用户testcifs,组是Users.

    每个组都有不同的roles来限制该组用户可执行的操作

    R2*> useradmin group list

    Name: Administrators

    Info: Members can fully administer the filer

    Rid: 544

    Roles: admin

     

    Name: Backup Operators

    Info: Members can bypass file security to backup files

    Rid: 551

    Roles: none

     

    Name: Guests

    Info: Users granted Guest Access

    Rid: 546

    Roles: none

     

    Name: Power Users

    Info: Members that can share directories

    Rid: 547

    Roles: power

     

    Name: Replicators

    Info: not supported

    Rid: 552

    Roles: none

     

    Name: Users

    Info: Ordinary Users

    Rid: 545

    Roles: audit

    Predefined Groups
      Predefined Group     Default Role     Default Privileges  
      Administrators     admin   Grants all CLI, Application Program  Interface (API), login, and security capabilities
         
      Power Users   power   Grants the ability to invoke all cifsexportfsnfsuseradmin CLI commands, and to make all cifs and nfs API calls log in to telnet, HTTP, RSH, and  Secure Shell (SSH) sessions 
            
      Backup Operators   none   None
            
      Users   audit   Grants the ability to make snmp-get and snmp-get-next API calls.
            
      Guests   none   None
            
      Everyone   none   None

     

    在WINDOWS客户端切换不同用户访问存储

     

    Microsoft Windows XP [Version 5.1.2600]

    (C) Copyright 1985-2001 Microsoft Corp.

     

    C:\Documents and Settings\Administrator.LYHP>net use

    New connections will be remembered.

     

     

    Status Local Remote Network

     

    ——————————————————————————-

    OK \\192.168.222.150\home Microsoft Windows Network

    The command completed successfully.

     

     

    C:\Documents and Settings\Administrator.LYHP>net use \\192.168.222.150\home /dele

    te

    There are open files and/or incomplete directory searches pending on the connect

    ion to \\192.168.222.150\home.

     

    Is it OK to continue disconnecting and force them closed? (Y/N) [N]: y

    \\192.168.222.150\home was deleted successfully.

     

    使用例子中的两条命令就可以在WINDOWS的客户端使用不同的用户来访问存储。

    刚才建立的testcifs用户是没有权限去访问存储的\\storageIP\etc$目录的,而切换到 管理用户即可。

    Active Directory domain authentication

     

    准备好WINDOWS2000 ADV SERVER,建立一个名为ender.cn的域.过程略去.

     

    要在VMWARE中同时运行LINUX+NETAPP SIMULATOR和WIN2000 ADV SERVER的话内存是越大越好,硬盘速度是越快越好.呵呵.笔记本同时跑这些实在是吃力.

    更改用户的认证方式为AD,先要把CIFS的服务中断.因为以前采用的是WORKGROUP的认证方式.

     

    R2> cifs terminate

    CIFS local server is shutting down…

    CIFS local server has shut down…

    Multiprotocol or NTFS-only 补充一下

     

    In the previous cifs setup examples, the storage system was configured for NTFS-only.

    This means that the new default security style for a volume (shares and files) is Windows NTFS. Windows users only can access the volumes.

    A requirement is that the CIFS protocol is licensed.

     

    A storage system can also be configured for multiprotocol access.

    This means that a single set of data could be accessed by way of the NFS (Network File System) protocol (UNIX users) and CIFS protocol (Windows users).

    A requirement is that both NFS and CIFS protocols are licensed.

     

    要注意的是NETAPP SIMULATOR的时间和AD服务器的时间差别要在5分钟内,否则认证是通不过的

    R2> cifs setup

    This process will enable CIFS access to the filer from a Windows(R) system.

    Use “?” for help at any prompt and Ctrl-C to exit without committing changes.

    This filer is currently a member of the Windows-style workgroup

    ‘WORKGROUP’.

    Do you want to continue and change the current filer account information? [n]: y

    Your filer does not have WINS configured and is visible only to

    clients on the same subnet.

    Do you want to make the system visible via WINS? [n]:

    This filer is currently configured as a multiprotocol filer.

    Would you like to reconfigure this filer to be an NTFS-only filer? [n]:

    The default name for this CIFS server is ‘R2′.

    Would you like to change this name? [n]:

    Data ONTAP CIFS services support four styles of user authentication.

    Choose the one from the list below that best suits your situation.

    (1) Active Directory domain authentication (Active Directory domains only)

    (2) Windows NT 4 domain authentication (Windows NT or Active Directory domains)

    (3) Windows Workgroup authentication using the filer’s local user accounts

    (4) /etc/passwd and/or NIS/LDAP authentication

    Selection (1-4)? [1]:

    What is the name of the Active Directory domain? [ender.cn]:

    In order to create an Active Directory machine account for the filer,

    you must supply the name and password of a Windows account with

    sufficient privileges to add computers to the ENDER.CN domain.

    Enter the name of the Windows user [Administrator@ENDER.CN]:

    Password for Administrator@ENDER.CN:

    CIFS – Logged in as Administrator@ENDER.CN.

    An account that matches the name ‘R2′ already exists in Active

    Directory: ‘cn=r2,cn=computers,dc=ender,dc=cn’. This is normal if you

    are re-running CIFS Setup. You may continue by using this account or

    changing the name of this CIFS server.

    Do you want to re-use this machine account? [y]:

    CIFS – Starting SMB protocol…

    Welcome to the ENDER.CN (ENDER) Active Directory(R) domain.

    CIFS local server is running.

     

    R2> cifs testdc

    Using Established configuration

    Current Mode of NBT is B Mode

    Netbios scope “”

    Registered names…

    ENDER < 0> Broadcast

    Testing all Primary Domain Controllers

    found 1 unique addresses

    ..Not able to communicate with PDC 192.168.222.150

    trying 192.168.222.150…192.168.222.150 is alive

    Testing all Domain Controllers

    found 1 unique addresses

    ..Not able to communicate with DC 192.168.222.150

    trying 192.168.222.150…192.168.222.150 is alive

     

    几个和DC有关的cifs 子命令

     

    R2> cifs domaininfo

    NetBios Domain: ENDER

    Windows 2000 Domain Name: ender.cn

    Type: Windows 2000

    Filer AD Site: none

     

    Current Connected DCs: \\ADTEST

    Total DC addresses found: 1

    Preferred Addresses:

    None

    Favored Addresses:

    None

    Other Addresses:

    192.168.222.150 ADTEST PDC

     

    Not currently connected to any AD LDAP server

    Preferred Addresses:

    None

    Favored Addresses:

    None

    Other Addresses:

    None

    R2> cifs prefdc

    Usage:

    cifs prefdc — print usage information

    cifs prefdc print [<domain>] — print preferred DCs for domain

    cifs prefdc add <domain> <dclist> — set preferred DCs for domain

    cifs prefdc delete <domain> — clear perferred DCs for domain

    R2> cifs prefdc print

    No preferred Domain Controllers configured.

    DCs will be automatically discovered.

    R2> cifs resetdc

    Disconnecting from domain ENDER…

    Reconnecting to domain ENDER…

    Mon Jul 14 16:15:14 EDT [cifs.server.infoMsg:info]: CIFS: Warning for server [url=file://\\ADTEST]\\ADTEST[/url]: Connection terminated.

    Mon Jul 14 16:15:14 EDT [auth.dc.trace.DCConnection.statusMsg:info]: AUTH: TraceDC- Starting DC address discovery for ENDER.

    Mon Jul 14 16:15:14 EDT [auth.dc.trace.DCConnection.statusMsg:info]: AUTH: TraceDC- Filer is not a member of a site.

    Mon Jul 14 16:15:14 EDT [auth.dc.trace.DCConnection.statusMsg:info]: AUTH: TraceDC- Found 1 addresses using generic DNS query.

    Mon Jul 14 16:15:14 EDT [auth.dc.trace.DCConnection.statusMsg:info]: AUTH: TraceDC- DC address discovery for ENDER complete. 1 unique addresses found.

    Reconnection failed!

    Mon Jul 14 16:15:14 EDT [cifs.server.infoMsg:info]: CIFS: Warning for server [url=file://\\ADTEST]\\ADTEST[/url]: Could not make TCP connection.

    Mon Jul 14 16:15:14 EDT [auth.ldap.trace.LDAPConnection.statusMsg:info]: AUTH: TraceLDAPServer- Starting AD LDAP server address discovery for ENDER.CN.

    Mon Jul 14 16:15:14 EDT [auth.ldap.trace.LDAPConnection.statusMsg:info]: AUTH: TraceLDAPServer- Found 1 AD LDAP server addresses using generic DNS query.

    Mon Jul 14 16:15:14 EDT [auth.ldap.trace.LDAPConnection.statusMsg:info]: AUTH: TraceLDAPServer- AD LDAP server address discovery for ENDER.CN complete. 1 unique addresses found.

     

    共享资源的认证

     

    Non-windows workgroup和NT的认证方式都没有查到详细的介绍,NT的方式估计是只有部分很老的客户还在使用,新上的客户至少都是用2000了吧.Non-windows workgroup等以后有机会使用的时候补充.

     

    通过了WORKGROUP或AD的认证后,才完成151号帖子中图示的第四步.

    下面是关于资源共享的认证.

     

    Share-level Access

     

    A share is an object that points to a physical folder (directory), and it makes the folder available to other users on the network that have the appropriate permissions to access the share.

    Share-level ACL (Access Control List) are the permissions to access the share.

    ? Typically shares are set to full control for the group “Everyone.”

    ? It can be used to restrict access to various users and groups.

     

    Windows Share-level permissions are the following:

    read-only, full control, and change

    Each permission is marked with “deny” or “allow.” If all the permissions are denied, then there is “no access.”

     

    UNIX permissions are a combination of the following:

    r (read), w (write), x (execute)

    The – is also used to designate that permission is not given for r, w, or x, whether for the UNIX user, group, or other roles.

    File Permission Attributes with Data ONTAP

     

    When a file is created, a list of access permissions (ACL) is generated. Data ONTAP maps common permissions (for example, reading and writing) to the file.

    When a user tries to access a file, Data ONTAP uses the permissions list to determine whether to permit or deny access based upon:

    Operation being performed

    User account, user group or netgroup, client protocol, client IP address, and file type

    Using a lookup service, LDAP (Lightweight Data Access Protocol), NIS (Network Information Service), or a local storage system, Data ONTAP determines who is trying to access the resource, and verifies that the permission list indicates operations that the user may perform.

    Files Created to Support the CIFS Environment

     

    Files Created to Support the CIFS EnvironmentDuring the CLI cifs setup script or FilerView CIFS Setup Wizard, CIFS support and configuration files are created in the /etc directory. The number and content of the files are dependent on the environment.

    The following are files that are common to all environments:

    /etc/cifsconfig_setup.cfg (stores the CIFS setup configuration)

    /etc/usermap.cfg (multiprotocol support for mapping users of NFS and CIFS)

    /etc/passwd (multiprotocol and UNIX workgroup)

    /etc/cifsconfig_shares.cfg (default shares definitions)

    Additional files are created depending on the environment as in a workgroup (Windows/non-Windows) or a Windows domain.

    CIFS Default Shares Created as a Result of cifs setup

     

    R2*> rdfile /etc/cifsconfig_share.cfg

    #Generated automatically by cifs commands

     

    cifs shares -add “ETC$” “/etc” “Remote Administration”

    cifs access “ETC$” S-1-5-32-544 Full Control

    cifs shares -add “HOME” “/vol/vol0/home” “Default Share”

    cifs access “HOME” S-NONE “nosd”

    cifs shares -add “C$” “/” “Remote Administration”

    cifs access “C$” S-1-5-32-544 Full Control

    R2*>

    所有对存储的CIFS的共享控制都会记录在这个文件中.上面几个是在运行cifs setup后系统自动生成的.

    NETAPP FILER的administration host:

     

    在系统初始化的时候,安装向导会询问你关于administration host:的信息,并且也给出了administration host:所起到的作用的说明.

     

    Please enter the name or IP address of the default gateway:

    The administration host is given root access to the filer’s /etc files for system administration. To allow /etc root access to all NFS clients enter RETURN below. Please enter the name or IP address of the administration host:

     

    Purpose of a Domain Administrator and Local Administrator Accounts

     

    On the storage system, the domain administrators group and the local administrator account are part of the BUILTIN\Administrators group. They can do the following:

    Provide a text editor to edit configuration files.Data ONTAP does not include an editor.

    Provide the ability to administer a storage system and hence have access to the root file system (C$ and ETC$).

    Modify the share access for C$ and ETC$ to grant additional users access.

    The local administrator can setup local users on the storage system with the useradmin user add command.

    cifs shares

     

    The CLI cifs shares command displays, adds, changes, and deletes share definitions.

    To display all shares:

    cifs shares

    To add a share:

    cifs shares –add sharename path [options [argument]]

    To change a share:

    cifs shares –change sharename [options [argument]]

    To delete a share:

    cifs shares –delete sharename

     

    wrfile就是写文件的东西了。

    如果rdfile /etc/hosts是能看到/etc/hosts文件的内容的,但你wrfile /etc/hosts的话是直接覆盖文件内容,所以,如果你的hosts文件内容是这样的话

    x.x.x.x name

     

    wr /etc/hosts会显示个空白,你要在hosts里添加一条y.y.y.y的话,就要这样写:

    x.x.x.x name

    y.y.y.y name2

    (注意,再多敲一个回车,然后用Ctrl+c退出)

     

    即原来的内容要重新写一次,否则原来的内容就没有了——netapp这一点做得实在不好

     

    所以推荐使用wrfile -a来追加写入。

    Share Management by Using the cifs access Command

     

    The CLI cifs access command sets or modifies the share-level access control list (ACL) to share definitions.

    To modify a share access:

    cifs access sharename [-g] user_rights

    To delete an ACL entry for a user on a share:

    cifs access -delete sharename [-g] user

    The -g option specifies that the user is the name of a UNIX group. Use this command when you have

    A UNIX group and a UNIX user or an NT user or group with the same name.

    Displaying CIFS Shares

     

    R2*> cifs shares

    Name Mount Point Description

    —- ———– ———–

    ETC$ /etc Remote Administration

    BUILTIN\Administrators / Full Control

    HOME /vol/vol0/home Default Share

    everyone / Full Control

    C$ / Remote Administration

    BUILTIN\Administrators / Full Control

    Creating a Share

     

    When you create a share, you must provide these items:

    The complete path name of an existing folder, qtree, or volume to be shared

    The name of the share entered by users when they connect to the share

    The permission for the share

    Optionally, a description of the share

    When creating a share from the Data ONTAP CLI, you can specify a variety of share properties, including group membership for files in the share, support for wide symbolic links, and disabling of virus scanning when files in the share are first opened.

    Virus scanning occurs when files are opened, renamed, and closed after being modified.

     

    After you have created a share, you can specify these share properties:

    Maximum number of users who can simultaneously access the share

    If you do not specify a number, the number of users is limited by storage system memory.

    Share-level Access Control List (ACL)

     

    R2*> qtree create /vol/myforvol001/testshare

    R2*> qtree status

    Volume Tree Style Oplocks Status

    ——– ——– —– ——– ———

    vol0 unix enabled normal

    myforvol001 unix enabled normal

    myforvol001 testshare unix enabled normal

    R2*> cifs shares -add testshare /vol/myforvol001 -comment “test cifs share command”

    The share name ‘testshare’ will not be accessible by some MS-DOS workstations

    Are you sure you want to use this share name? [n]:y

    R2*> cifs shares

    Name Mount Point Description

    —- ———– ———–

    ETC$ /etc Remote Administration

    BUILTIN\Administrators / Full Control

    HOME /vol/vol0/home Default Share

    everyone / Full Control

    C$ / Remote Administration

    BUILTIN\Administrators / Full Control

    testshare /vol/myforvol001 test cifs share command

    everyone / Full Control

     

    默认的权限是

    Full Control for Everyone

    cifs access修改用户

     

    R2*> qtree status

    Volume Tree Style Oplocks Status

    ——– ——– —– ——– ———

    vol0 unix enabled normal

    myforvol001 unix enabled normal

    myforvol001 testshare unix enabled normal

    R2*> cifs access testshare Administrator Full Control

    1 share(s) have been successfully modified

    R2*> cifs shares

    Name Mount Point Description

    —- ———– ———–

    ETC$ /etc Remote Administration

    BUILTIN\Administrators / Full Control

    HOME /vol/vol0/home Default Share

    everyone / Full Control

    C$ / Remote Administration

    BUILTIN\Administrators / Full Control

    testshare /vol/myforvol001 test cifs share command

    everyone / Full Control

    R2\administrator / Full Control

    R2*> cifs access -delete testshare everyone

    1 share(s) have been successfully modified

    R2*> cifs shares

    Name Mount Point Description

    —- ———– ———–

    ETC$ /etc Remote Administration

    BUILTIN\Administrators / Full Control

    HOME /vol/vol0/home Default Share

    everyone / Full Control

    C$ / Remote Administration

    BUILTIN\Administrators / Full Control

    testshare /vol/myforvol001 test cifs share command

    R2\administrator / Full Control

     

    修改后普通用户testcifs就不可以访问testshare这个文件夹了.

    CIFS Sessions

     

    R2*> cifs sessions

    Server Registers as ‘R2′ in workgroup ‘WORKGROUP’

    Filer is using en_US for DOS users

    Using Local Users authentication

    ====================================================

    PC IP(PC Name) (user) #shares #files

    192.168.222.1() (R2\testcifs – pcuser)

    2 0

    R2*> cifs sessions *

     

    users

    shares/files opened

     

    192.168.222.1() (R2\testcifs – pcuser)

    HOME

    testshare

    R2*> cifs sessions -s testcifs

     

    users

    Security Information

     

    192.168.222.1() (R2\testcifs – pcuser)

    ***************

    UNIX uid = 65534

     

    NT membership

    R2\testcifs

    BUILTIN\Users

    User is also a member of Everyone, Network Users,

    Authenticated Users

    ***************

    Options

    The -t optiondisplays the total count of CIFS sessions, open shares and open files.

    If you include the user argument, the command displays information about the specified user, along with the names and access level of files that user has opened. If you use * as the specified user, the command lists all users.

    Specifying the -c option with a user argument, will display the names of open directories and the number of active ChangeNotify requests against the directory.

    The -s option displays security information for a specified connected user. If you do not specify a user or workstation name, the command displays security information for all users.

     

    Stopping and Restarting CIFS Services

     

    To terminate CIFS service (a complete shutdown) where all CIFS sessions are ended:

    cifs terminate [-t minutes]

    To terminate CIFS service and sessions for a specified workstation:

    cifs terminate [workstation] [[-t] minutes]

    To restart CIFS service after terminating:

    cifs restart

    ———————————————————————————————————————————————————–

    R2*> cifs terminate

     

    CIFS local server is shutting down…

     

    CIFS local server has shut down…

    R2*> cifs restart

    CIFS local server is running.

    R2*>

     

    我的DC用起来不大正常,不知道啥原因

     

    R2*> cifs testdc

    Using Established configuration

    Current Mode of NBT is B Mode

     

    Netbios scope “”

    Registered names…

    ENDER < 0> Broadcast

     

    Testing all Primary Domain Controllers

    found 1 unique addresses

     

    ..Not able to communicate with PDC 192.168.222.150

    trying 192.168.222.150…192.168.222.150 is alive

     

    Testing all Domain Controllers

    found 1 unique addresses

     

    ..Not able to communicate with DC 192.168.222.150

    trying 192.168.222.150…192.168.222.150 is alive

    R2*> cifs domaininfo

    NetBios Domain: ENDER

    Windows 2000 Domain Name: ender.cn

    Type: Windows 2000

    Filer AD Site: none

     

    Not currently connected to any DCs

    Preferred Addresses:

    None

    Favored Addresses:

    None

    Other Addresses:

    192.168.222.150 ADTEST PDCBROKEN

     

    Connected AD LDAP Server: \\adtest.ender.cn

    Preferred Addresses:

    None

    Favored Addresses:

    None

    Other Addresses:

    192.168.222.150

    adtest.ender.cn

    R2*> cifs resetdc

    Disconnecting from domain ENDER…

    Reconnecting to domain ENDER…

    Mon Jul 14 19:06:50 EDT [auth.dc.trace.DCConnection.statusMsg:info]: AUTH: TraceDC- Starting DC address discovery for ENDER.

    Mon Jul 14 19:06:50 EDT [auth.dc.trace.DCConnection.statusMsg:info]: AUTH: TraceDC- Filer is not a member of a site.

    Mon Jul 14 19:06:50 EDT [auth.dc.trace.DCConnection.statusMsg:info]: AUTH: TraceDC- Found 1 addresses using generic DNS query.

    Mon Jul 14 19:06:50 EDT [auth.dc.trace.DCConnection.statusMsg:info]: AUTH: TraceDC- DC address discovery for ENDER complete. 1 unique addresses found.

    Mon Jul 14 19:06:50 EDT [cifs.server.infoMsg:info]: CIFS: Warning for server \\ADTEST: Could not make TCP connection.

    Reconnection failed!

    Mon Jul 14 19:06:50 EDT [auth.ldap.trace.LDAPConnection.statusMsg:info]: AUTH: TraceLDAPServer- Starting AD LDAP server address discovery for ENDER.CN.

    Mon Jul 14 19:06:50 EDT [auth.ldap.trace.LDAPConnection.statusMsg:info]: AUTH: TraceLDAPServer- Found 1 AD LDAP server addresses using generic DNS query.

    Mon Jul 14 19:06:50 EDT [auth.ldap.trace.LDAPConnection.statusMsg:info]: AUTH: TraceLDAPServer- AD LDAP server address discovery for ENDER.CN complete. 1 unique addresses found.

     

    Auto Home Share on the Storage System

     

    You can create user home directories on the storage system and configure Data ONTAP to offer each user a home directory share automatically. Each user can connect to the user’s home directory only, not to the home directories of other users.

    这个功能应该来说比较有用.免的用户乱存东西到别人的目录下.

    To specify the naming style used for matching home directories to users:

    options cifs.home_dir_namestyle {ntname | hidden | domain | mapped | “”}

    To specify whether members of the storage-system Builtin\Administrators group can connect to the CIFS home directories of other users:

    options cifs.homedirs_public_for_admin on

    When you create a user’s folder for their home directory, Data ONTAP automatically searches the paths in the cifs_homedir.cfg file for the user name that matches the login name, and dynamically creates the share for that user

     

    Auditing CIFS Events

     

    以前看到过有个贴子问怎么对用户行为做audit,测试了一下.

     

     

    You can enable auditing for the following categories of events:

    Logon and logoff events

    File access events

    These are the prerequisites for auditing file access events:

    The file or directory can be audited in a mixed or NTFS volume or qtree.

    If the cifs.audit.nfs.enable option is “on,” you can audit events for files in UNIX security- style qtrees.

    You must activate auditing for individual files and directories according to your Windows documentation.

     

    R2*> options cifs.audit.enable on

    R2*> Mon Jul 14 22:04:13 EDT [cifs.auditfile.enable.on:info]: ALF: CIFS auditing started.

     

    R2*> cifs terminate

     

    CIFS local server is shutting down…

     

    waiting for CIFS shut down (^C aborts)…

    Mon Jul 14 22:04:23 EDT [cifs.auditfile.enable.off:info]: ALF: CIFS auditing stopped.

     

    CIFS local server has shut down…

    R2*>

    R2*> cifs restart

    CIFS local server is running.

    Mon Jul 14 22:04:23 EDT [cifs.auditfile.enable.off:info]: ALF: CIFS auditing stopped.

    Mon Jul 14 22:04:28 EDT [cifs.auditfile.enable.on:info]: ALF: CIFS auditing started.

    R2*>

    R2*> options cifs.audit.file_access_events.enable on

    R2*> options cifs.audit.logon_events.enable on

    R2*> options cifs.audit.autosave.file.extension timestamp

    Mon Jul 14 22:16:13 EDT [cifs.auditfile.logSetting.notice:info]: ALF: change to option cifs.audit.autosave.file.extension: Autosave file naming policy is now based on time stamping.

    [size=-1]R2*> options cifs.audit.saveas /vol/vol0/home/testaudit.evt

     

     

    这个文件要用WINDOWS的事件查看器来看.

     

    CIFS Opportunistic Locks

     

    An opportunistic lock (also called an oplock) is a lock placed by a client on a file residing on a server. In most cases, a client requests an opportunistic lock so it can cache data locally, thus reducing network traffic and improving apparent response time. Opportunistic locks are used by network redirectors on clients with remote servers, as well as by client applications on local servers.

    Opportunistic locks coordinate data caching and coherency between clients and servers and among multiple clients. Data that is coherent is data that is the same across the network. In other words, if data is coherent, data on the server and all the clients is synchronized.

     

    Opportunistic locks are not commands by the client to the server. They are requests from the client to the server. From the point of view of the client, they are opportunistic. In other words, the server grants such locks whenever other factors make the locks possible.

     

    When a local application requests access to a remote file, the implementation of opportunistic locks is transparent to the application. The network redirector and the server involved open and close the opportunistic locks automatically. However, opportunistic locks can also be used when a local application requests access to a local file, and access by other applications and processes must be delegated to prevent corruption of the file. In this case, the local application directly requests an opportunistic lock from the local file system and caches the file locally. When used in this way, the opportunistic lock is effectively a semaphore managed by the local server, and is mainly used for the purposes of data coherency in the file and file access notification.

     

     

    默认情况下NETAPP FILER中options cifs.oplocks.enable on是打开的.

    但在几种情况下需关闭

     

    Using a databaseapplication whose documentation recommends oplocks be turned off.

     

    The CIFS clients are on anunreliable network.

     

    Handling critical data,and you cannot afford even the slightest data loss.

    Otherwise, leave CIFS oplocks on.

    杀毒

     

    当然DOT本身还没有这个功能,但可以通过接入一台装了杀毒软件的服务器来进行。

     

    CIFS virus protection is a Data ONTAP feature that enables a virus-scanning Windows server running compliant antivirus applications to provide on-access virus scanning of files on a storage system.

    On-access virus scanning means that a file is scanned before a CIFS client is allowed to open it.

     

    1. The scanner (Windows server) registers with the storage system, so no storage system configuration is required.

    2. At the storage system prompt, type the vscan on command to enable scanning.

    3. The scanner waits for requests to come from the storage system.

    Several scanners can register with the storage system. This is recommended for performance and reliability.

    A single scanner can scan multiple storage systems.

    4. The scanner pings the storage system from time to time to detect and recover from reboots and takeovers.

    CIFS性能分析工具之cifs stat

     

    The cifs stat command has two main forms.

     

    Ifyou specify the interval, the command continues displaying a summary ofCIFS activity until interrupted. The information is for the preceding intervalseconds. (The header line is repeated periodically.) The interval must be >=1.

    If you do not specify the interval, the command displayscounts and percentages of all CIFS operations as well as a number of internalstatistics that may be of use when diagnosing performance and other problems.

    Bydefault, the statistics displayed are cumulative for all clients. However, ifthe cifs.per_client_stats.enable option is on, a subset of the clients may beselected using the -u and/or -h options

    cifs_stat Options

     

    -u <user> If per-client stats are being gathered, selects a user account to match for stats reporting. More than one -u <user> option may be supplied. If more than one client matches the user, the values reported are the sum of all matching clients.

    The user specified may have a domain, which restricts matching to that domain, or the domain may be “*” or left blank to match any domain. The user account may be specified, or may be “*” to match any user.

    -h <host> If per-client stats are being gathered, specifies a host to match for stats reporting. More than one -h <host> option may be supplied. If more than one client matches the host, the values reported are the sum of all matching clients.

    The host may be an IP address in dot notation, or it may be any host name found using DNS if that is enabled on the storage system.

    -v[v] If per-client stats are being reported using the -u or -h options, it may be desirable to know which clients contributed to the total stats being reported. If -v is given, the count of the number of matching clients is printed prior to the stats themselves. If -vv is given, the actual matching clients are also printed prior to printing the stats themselves.

    -c Displays counts and percentages for non_blocking CIFS operations as well as block_ing, which is the default. This option is not available in combination with the perclient options.

    -z Zeroes all CIFS operation counters, including per-client counters, if any.

     

    cifs top

     

    The cifs top command is used to display CIFS client activity based on a number of different criteria. It can display which clients are generating large amounts of load, as well as help identify clients that may be behaving suspiciously.

    The default output is a sorted list of clients, one per line, showing the number of I/Os, number and size of READ and WRITE requests, the number of “suspicious” events, and the IP address and user account of the client. The statistics are normalized to values per second. A single client may have more than one entry if it is multiplexing multiple users on a single connection, as is frequently the case when a Windows Terminal Server connects to the storage system.

    This command relies on data collected when the cifs.per_client_stats.enable option is “on”, so it must be used in conjunction with that option. Administrators should be aware that there is overhead associated with collecting the per-client stats. This overhead may noticeably affect the storage system performance.

     

     

    cifs stat 和 cifs top是可以统计和CIFS相关的各种信息。系统另外提供的netstat , ifstat ,netdiag ,sysstat ,stats 等命令是和网络性能,系统处理性能相关。是综合影响NAS各种服务NFS,CIFS,HTTP,FTP的性能分析收集之用。

    NTFS-only 或者 Multiprotocol

     

    在运行CIFS SETUP的时候,如果只有CIFS的许可.那么系统会有一个提示

    A filer can be configured for multiprotocol access, or as an NTFS-only filer. Since NFS, DAFS, VLD, FCP, and iSCSI are not licensed on this filer, we recommend that you configure this filer as an NTFS-only filer

    建议把FILER配置成NTFP-only的.如果这样系统会对VOL进行转换

     

    Wed Jun 21 16:29:23 GMT [wafl.quota.sec.change:notice]: security style for /vol/vol0/ changed from unix to ntfs

     

    同时对options wafl.default_security_style 和 wafl.nt_admin_priv_map_to_root 的值进行更改.

    wafl.default_security_style

    ntfs

    wafl.nt_admin_priv_map_to_root off

    /etc/usermap.cfg and /etc/passwd

     

    The purpose of mapping Windows and UNIX users is the following:

    The UID and GID are used to check file-access permissions for the mapped Windows user name that wants to access UNIX files.

    The user and group SIDs are used to check file-access permissions (Access Control Lists) for the mapped UNIX user name that wants to access NTFS files

     

    If the mapping for a Windows user name is not found:

    The Windows user name is mapped to a specified default UNIX user name with the following options command:

    options wafl.default_unix_user <name>

    A typical default UNIX user name is “pcuser” with UID=65534 and GID=65534; it is stored in the /etc/passwd file.

    The Windows Administrator user has a special case processing where it is mapped to the UNIX user name “root” with UID=0 and GID=1, but only if the following options command is set “on”:

    options wafl.nt_admin_priv_map_to_root on

     

    If the mapping for the UNIX user name is not found:

    The UNIX user name is mapped to a specified default Windows (NT) user name:

    options wafl.default_nt_user <name>

    You can create a default Windows user name.

    If this options command is not set (and by default, it is not set), then the UNIX user is denied access to the NTFS files.

    /etc/passwd中的用户和前面说到的用系统命令useradmin add增加的用户是没有关系的. 现在看来主要就是做UNIX和WINDOWS之间用户的转换.

     

    FILER提供的用户转换

     

    FILER提供了对NFS,CIFS客户端用户的相互转换的功能,直接编辑/etc/usermap.cfg就可以了。

     

    The process of creating a CIFS cred from a user ID (UID), or a UNIX cred from a Windows account, always involves checking a user mapping file called /etc/usermap.cfg. The user mapping process allows much flexibility, but it also must be used carefully because it is possible to create confusing scenarios.

    NETAPP 中SAN和NAS的对比

    unified storage

     

    NETAPP把NAS和SAN的功能统一在Data Ontap下提供给客户使用,自己定义了这种方式叫”unified storage”。

     

    NAS provides access for CIFS (Common Internet File System),most commonly used by Microsoft Windows-based systems, and for NFS (NetworkFile System), most commonly used by UNIX-based operating systems.

    Network access to NAS on the storageappliance is typically via TCP/IP.

    SAN provides block access to LUNs (logical unit numbers), whichare treated as local disks by both Windows and UNIX-based operating systems. Thestorage system views a LUN as a logical representation of physical storage. Networkaccess to LUNs is via SCSI over Fibre Channel (FCP) or SCSI over TCP/IP (iSCSI).

     

    如果对别的厂家的阵列比较熟悉,从上个帖子的图中就可以发现NETAPP对SAN的支持要比别的厂家多了一个层次wafl。

    Initiator & Target

     

    Initiator & Target

    主机是发起请求到存储设备,FILER自己访问扩展柜的时候就变成了发起(Initiator)。那个FA的端口的属性是可以改的。

     

    how requests move from the host (initiator) to thestorage system (target). First, an application sends a request to the filesystem. The file system issues Input/Output (I/O) calls to the operatingsystem. The operating system then sends the I/O through its storage stack (SCSIdriver) to issue the SCSI commands. Next, these commands are encapsulated inFibre Channel frames or iSCSI IP packets. Once the request is received by thestorage system target, the Data ONTAP operating system converts requests fromthe initiator. Data ONTAP turns SCSI commands into WAFL operations. WAFL sendsthe request to the ONTAP RAID subsystem where RAID manages the data on thephysical disks where the LUN is located. Once processed, request responses moveback through the FC fabric or iSCSI-based。

    WWNN & WWPN

     

    做过SAN的人一定接触过WWN

    When discussing World Wide Names it is important to cover themat a high level before going into too much detail. The FC specification for thenaming of nodes and ports on those nodes can be fairly complicated. Each device is given a globally unique WorldWide Node Name (WWNN) and an associated World Wide Port Name (WWPN) for each port on the node. World Wide Namesare 64-bit addresses made up of 16 hexadecimal digits grouped together in twos witha colon separating each pair (e.g. 21:00:00:2b:34:26:a6:54).

    Thefirst number in the WWN defines what the other numbers in the WWN represent,according to the FC specification. The first number is generally a 1, 2, or 5.In the example of QLogic initiator HBAs, the first number is generally a 2. ForEmulex initiator HBAs, the first number is generally a 1.

     

    Qlogic的卡多见于HP和SUN的系统上,EMULEX的卡多见于IBM的机器上,在市场上买回来的卡Qlogic的见的很多。

    Qlogic的WWN多为21:00:00:00。。。。

    EMULEX的WWN多为10:00:00:00:….

    SYMMETRIX的WWN为50:06:….

    CX的一下想不起来了。。。。

    EMC的SYMM和CX的端口都有自己的编号。可以通过WWN来算是哪块卡上的哪个口子,有工具可以算,实际上是有个算法的,也可以说是EMC对SYMM的端口WWN定义。CX要简单一点,熟练的工程师看WWN就知道是哪个SP的哪个端口。

    FCP Protocol Topologies

     

    Direct-attach (Point-to-Point)

    Initially, Fibre Channel (FC) Point-to-Point topologies were seen as a replacement for the parallel SCSI bus, to overcome bandwidth and distance limitations. FC at 100Mb/sec was superior to SCSI at 10–20Mb/sec, and as SCSI progressed to 40, 80, then 160Mb/sec, FC stayed ahead with 200Mb/sec then 400Mb/sec. SCSI bandwidth was reaching a ceiling where FC at 200Mb/sec was just getting started. FC Point-to-Point also overcame the severe distance limitations of SCSI, but one limitation remained: It connected one initiator to one target, supporting only the simplest topology. This provides limited connectivity.

     

    采用这种方式和DAS实际就是一回事了,不过把DAS多用的SCSI的线缆换成了FC的光纤。

    FCP Protocol Topologies 续

     

    Fibre Channel Arbitrated Loop (FCAL)

    To overcome the connection limitation, the second generation of FC was developed, FC Arbitrated Loop (FCAL). FCAL could connect up to 127 ports on a shared loop using a two-place hexadecimal addressing scheme. Eventually, even this 127-port address was seen as too much of a limitation, and the bandwidth was shared.

    In an arbitrated loop, when devices are added or removed, all activity on the loop is disrupted. This occurs because there is a break in the ring. A FC hub may be used to connect devices and therefore bypass failed ports

    磁带库多是这种情况接入SAN

    FCP Protocol Topologies

     

    Switched Fabric

    The third generation of FC, called Switched Fabric, used a 24-bit addressing scheme with 64-bit WWPN and WWNN. This scheme has a possible 12 million addresses, and the initiator-target pair got a dedicated non-blocking path to ensure full bandwidth.

    In this configuration, all devices or loops are connected to FC switches.

     

     

    最常见的情况。

    这里说一下存储内部DAE的采用的多是switch方式,但NETAPP的叫ESH。

    图示SAN TOPO

     

    文字描述的东西始终不如一副图看的明白。

    NETAPP FILER支持的SAN TOPO

     

    Network Appliance differentiates between three basic SAN topologies which are possible when connecting Network Appliance Storage Appliances and server systems with FC:

    Direct-attached – The servers (or hosts) are directly attached to the NetApp storage controller.

    Single Fabric – The servers are attached to NetApp storage controllers through a single FC fabric. This fabric may consist of multiple FC switches.

    Dual Fabric – Each server is attached to two physically independent fabrics that are connected to the NetApp storage controllers.

     

    Dual Fabric是用的物理上的两个FABRIC来实现的,不是在一个交换机上划上不同的zone来是实现的,那样做不到防止单点故障,当然对提供到存储的带宽还是有用。绝大多数情况下还是两个FABRIC。

    ZONING

     

    A FC channel zone consists of a group of FC ports or nodesthat can communicate with each other. It can be thought of as a logical fabricsubset. Two FC nodes can communicate with one another only when they arecontained in the same zone. A node can be contained in multiple zones. Thereare two types of zoning, “hard” and “soft” zoning.

    Hard ZoningHard zoning physically restricts communication in aswitched fabric. Hard zoning is considered secure because it prevents zoningbreaches caused by bypassing the fabric name service.

    Soft ZoningSoftzoning limits visibility of Fibre Channel nodes across zones. When a nameserver receives a request from a port, it will only return information fromports within the same zone as the port that generated the request. Althoughdevices are separated at the name service level, soft zones do not preventcommunication between zones. Soft zoning is typically more flexible than hardzoning because this type of zoning does not consume physical resources on theswitch. Soft zones are also considered less secure than hard zones

     

     

    项目中用什么方式的ZONING要看厂家的意见,还有就是根据特殊产品的特殊要求,例如EMC的NAS就要要求DATAMOVE和SYMM或CX之间一定要采用WWN的方式。

    在实际项目中,如果项目的机器很多而且不是一批到,并且有别的厂家的设备也要用到你提供的交换机,那么还是用PORT来做好一些,要不客户要你专门飞一趟去做个ZONE???!!!自己都觉的太浪费了。

    另外,不建议PORT和WWN混合做,这个是有教训的。

    iSCSI TOPO

     

    说完了SAN ,下面说说ISCSI.

    There are two basic topologies which are supported by NetAppin iSCSI direct-attached and switched environment.

    Direct-Attached

    In direct-attached, servers (or hosts) are directly attachedto the NetApp controller using a crossover cable. Direct-attach to more thanone controller in an HA Configuration is not possible.

    Switched Environment

    In a switched environment, servers are attached to NetAppcontrollers through Ethernet switches. This network may consist of multipleEthernet switches in any configuration.

    There are two types of switched environments, dedicatedEthernet and shared Ethernet. In a dedicated Ethernet, there is no extraneousnetwork traffic. The network is totally dedicated to iSCSI and relatedmanagement traffic. Such a network is typically located in a secure datacenter. Direct-attached and dedicated Ethernet networks represent approximately90 percent of current iSCSI deployments. In a shared Ethernet, the network isshared with other traffic or a corporate Ethernet network. This typicallyintroduces firewalls, routers, and IPSEC into the Ethernet network.

     

    这个环境太常见见了,现在家庭都能搭建。呵呵。

    ISCSI initiator & target

     

    initiator可能是下面几种情况中的一种

    an iSCSI software initiator with a standard NIC

    an iSCSI software initiator with a TCP OffloadEngine (TOE)

    an iSCSI Host Bus Adapter (HBA) card

     

    target

    an iSCSI Software Target (ISWT) and a NIC

    an iSCSI Target Host Bus Adapter (Target HBA)card

    用软件实现的initiator啥网卡都能用,要是有比较大的利润。估计山寨网卡也快出来了

    PCI-X的扩展槽看来是可以给FILER加新的TARGET了。

    iscsi 主要命令

    iscsi alias: Manages controller iSCSI target aliases
    iscsi connection – Displays active iSCSI connections
    iscsi help: Displays a list of iSCSI commands
    iscsi initiator: Displays connected initiators
    iscsi interface: Manages iSCSI use of network interfaces
    iscsi isns: Configures iSNS service
    iscsi nodename: Manages controller iSCSI target node name
    iscsi portal: Displays controller iSCSI portals
    iscsi security: Configures security parameters
    iscsi start: Starts the iSCSI service
    iscsi session: Displays active iSCSI sessions
    iscsi show: Displays connected initiators
    iscsi stats: Displays or zeroes statistics
    iscsi status: Shows the status of the iSCSI service
    iscsi stop: Stops the iSCSI service
    iscsi tpgroup: Manages target portal groups

     

    FCP 主要命令

    fcp config: Configures adapters
    fcp help: Displays a list of fcp commands
    fcp nodename: Sets up the storage system node name
    fcp show: Displays information about adapters and initiators
    fcp start: Starts the FCP service
    fcp status: Displays the status of the FCP service
    fcp stop: Stops the FCP service
    fcp stats: Displays statistics about the adapters

     

    igroup 相关命令

    igroup help: Lists initiator group commands
    igroup add: Adds initiator(s) to the group
    igroup bind: Binds an “igroup” to a portset
    igroup create: Creates a new initiator group
    igroup destroy: Destroys an initiator group
    igroup remove: Removes initiator(s) from an “igroup”
    igroup set: Sets attributes on a group
    igroup show: Displays initiator groups
    igroup unbind: Unbinds an igroup from a portset
    igroup rename: Renames an igroup

     

    lun 相关命令

     

    lun help: Lists the LUN (Logical Unit of Block Storage) commands

    lun config_check: Checks all lun/igroup/fcp settings for correctness

    lun clone: Manages LUN cloning

    lun comment: Displays/changes descriptive comment strings

    lun create: Creates a LUN

    lun destroy: Destroys a LUN

    lun map: Maps a LUN to an initiator group

    lun maxsize: Finds the maximum size for a LUN on a volume or qtree

    lun move: Moves (renames) a LUN

    lun offline: Stops block protocol access to a LUN

    lun online: Restarts block protocol access to a LUN

    lun resize: Resizes a LUN

    lun select: Displays/changes a LUN selection attribute

    lun serial: Displays/changes a LUN serial number

    lun set: Manage LUN properties

    lun setup: Initialize/configure LUNs

    lun share: Configure NAS file-sharing properties

    lun show: Displays LUNs

    lun snap: Manages LUN and snapshot interactions

    lun stats: Displays or zeros read/write statistics for a LUN

    lun unmap: Removes LUN mapping

     

    IGROUP是把FCP或者ISCSI和LUN绑定在一起的一个组,绑好了就可以把LUN给客户端使用了.

    其实对LUN的访问控制都是这样,叫发不同而已.CX叫什么ACCESS LOGEX,SYMM是用 symmask来改VCMDB.

     

    上面几个ISCSI,FCP把SAN的两种方式的命令列出了,IGROUP列出了他们要用的组,LUN就是要访问的盘,当然是做了保护后在主机上看是物理盘的设备.

    Relationship of LUNs to Volumes

     

    A volume is the most inclusive of the logical containers. It can store the following:

    Files and directories

    Qtrees

    LUNs

    Qtrees

    A qtree is a subdirectory of the root directory of a volume. Qtrees can be used to subdivide a volume in order to group LUNs.

    LUNs

    A LUN is a logical representation of a physical unit of storage. It is a collection of, or a part of, physical or virtual disks configured as a single disk. When you create a LUN, it is automatically striped across many physical disks. Data ONTAP manages LUNs at the block level, so it cannot interpret the file system or data in a LUN. From the host, LUNs appear as local disks, allowing you to format and manage to store data on them.

     

     

    Data ONTAP manages LUNs at the block level 有点搞不清楚了.前面有资料说LUN在WAFL之上,现在说是BLOCK LEVEL.不过从性能上考虑是应该直接到BLOCK LEVEL要好.就像ORACLE自己也可以管理盘,不需要OS的LVM.

    详细说下VOLUME

     

    详细说下VOLUME,这个和LUN是息息相关的,当然是指正在NETAPP系统中.

    Volume name 最好有个命名规范,要不自己给自己找麻烦

    Volume size 最小20M,最大和存储系统有关系,要去查手册了

     

    Security style – The securitystyle determines whether a volume can contain files that use UNIX security,files that use Windows file system (NTFS) file security, or both types of files 这个前面在CIFS的配置的时候讨论过

    Option to designate the volume asone that uses CIFS oplocks (opportunistic locks) 这个也说过了

    Supported language type 这个不太清楚,对中文是不是有特殊要求了,没管这个参数做测试好像没啥问题.

    Level of space guaranteed (forFlexVol volumes only) 一会说

    Disk space and file limits(quotas)

    Snapshot schedule (optional;for information about the default snapshot schedule, see the Data ProtectionOnline Backup and Recovery Guide) snapshot可以说是WAFL的副产品,不过要是不用这个功能该把它关了,免的浪费空间.

    Option to designate the volumeas a SnapLock volume.把这个卷搞成一次刻录光盘一样,只能写一次

    Option to designate the volume as a root volume – All new storage systems with factory-installed Data ONTAP have apre-configured root volume. The root volume is named “vol0″ by default.

    Space Reservations

     

    Data ONTAP uses space reservation to guarantee that space is available for completing writes to a LUN or for overwriting data in a LUN. When you create a LUN, Data ONTAP reserves enough space in the traditional or FlexVol volume so that write operations to those LUNs do not fail because of a lack of disk space on the storage system. Other operations, such as taking a Snapshot copy or the creation of new LUNs, can occur only if there is enough available unreserved space; these operations are restricted from using reserved space.

    一般阵列做LUN的时候都是直接在一个RAID上直接格式化一块区域,没有保留的说法.NETAPP可能是综合多种用途的考虑才有这个Space Reservations的说法吧.

    仔细研究了下资料,对空间做保留在特定环境下还是比较有用的.如果一个LUN没有对空间做保留,那么空间可以用于其他用途,但这个时候LUN上的数据再增长可能会出现盘空间不足.这样对LUN的大小调整也可以实现减小的功能.

    其他阵列想要减小一个LUN的大小是不可能的,必须要做数据迁移了.

    创建LUN的几个建议

     

    Do not create any LUNs in the system’s root volume. Data ONTAP uses this volume to administer the storage system. The default root volume is /vol/vol0. 我看我师弟写的NETAPP安装报告,现在系统是安装在三块盘做的RAID DP上,还保留了一块盘给它做HOTSPARE,这样系统就用了4块盘,也太浪费了.

     

    Ensure that no other files or directories exist in a volume that contains a LUN. If this is not possible and you are storing LUNs and files in the same volume, use a separate qtree to contain the LUNs.

     

    If multiple hosts share the same volume, create a qtree on the volume to store all LUNs for the same host. This is a recommended best practice that simplifies LUN administration and tracking.

    有了上面两条,看样子最好建立一个QTREE然后在里面建立LUN比较好.

     

    Ensure that the volume option create_ucode is set to on. (vol options <volname> create_ucode on). Data ONTAP requires that the path of a volume or qtree containing a LUN is in the Unicode format. This option is Off by default when you create a volume. It is important to enable this option for volumes that will contain LUNs.

     

    To simplify management, use naming conventions for LUNs and volumes that reflect their ownership or the way that they are used.

     

    我在网上好像也搜索到过类似的说法,要不系统就用掉4块盘,现在盘都那么大,空间浪费也太惊人了.客户容量下的话估计都会觉的受骗了.

    在这点上NETAPP应该像CX学习下,用前几块盘的部分来做就好了.这样也少了对root vol的手动迁移,免的出现些问题.

     

    LUN SIZES

     

    LUN Sizes

    The minimum size for a LUN in a Solaris environment is 4MB. The minimum size for a LUN in a Windows environment is 31.5MB. The maximum size for a LUN should not exceed approximately 50 percent of the size of a volume, especially if Snapshot copies of the LUN are desired.

     

    EMC的GATEKEEPER是3M左右,在WIN上一样可以认出来.NETAPP的要求好像是不太一样,但现在几百M的LUN好像除了可以用于HA做心跳盘,用来放数据实在是太小了.

     

    而最大的LUN要看系统支持了,现在2T没有问题.

    Host Utilities

     

    Diagnostic Scripts – Network Appliance Support must have these scripts run on your system before they are able to diagnose a problem.

     

    Documentation – Documentation contains crucial setup information and known problems which will help with setup and troubleshooting.

     

    sanlun utility – The sanlun utility allows you to view information about LUNs and the host HBA. It is only available on UNIX operating systems.

     

    The AIX ODM definition package – Properly identifies LUNs as Network Appliance LUNs. The NetApp ODM package for IBM AIX&reg; is only available in the FCP Host Attach Kit for AIX.

    IBM的AIX ODM是要特殊一些,别的厂家做的东西还得写个ODM的东西才能被正确识别.

    The set tuneables scripts – The set tuneables script updates registry and WMI values. It is only available for Windows&reg;

    SnapDrive for UNIX

     

    SnapDrive for UNIX is a tool that simplifies data backup management so that you can recover should data accidentally be deleted or modified. SnapDrive for UNIX uses Snapshot technology to create an image of the data stored on a storage system attached to a UNIX host. You can then restore that data at a later time.

    In addition, SnapDrive for UNIX lets you provision storage on the storage system. SnapDrive for UNIX provides a number of storage features that enable you to manage the entire storage hierarchy, from the host-side application-visible file down through volume manager to the storage-system-side LUNs that provide the actual repository.

     

    在主机端提供了做SNAPSHOT的工具,不可能所有的用户都到FILER上用命令来做snapshot的创建,恢复等动作.麻烦且危险.在自己使用的机器上对自己的使用的空间做就可以了,根据自己的需要灵活使用.

    SnapDrive for Windows Multipathing

     

    Multipathing uses redundant paths between a Windows host and a LUN (virtual disk), thus eliminating the “single-point-of-failure” vulnerability that exists when a host connects to a storage system across a single, fixed physical path. SnapDrive multipathing establishes two or more physicalpaths between the host and the LUN. One of the paths is designated active and the others as passive (standby). If the active physical path fails, a passive(standby) path will take over and continue to maintain connectivity between the host and the LUN.

    SnapDrive facilitates multipath redundancy by integrating a Windows MPIO-device-specific module (ntapdsm.sys) with a trio of Microsoft software drivers (mpio.sys, mpdev.sys, and mspspfltr.sys). This multipathing solution, referred to as “Windows MPIO,” is managed through the SnapDrive plug-in under the MMC or the SnapDrive for Windows sdcli.exe command-line utility.

    如果没有这个软件,windows是会把一个LUN在每一个通道上都认一边,并且认为它们是不同的盘,那就麻烦了.

    从上面的资料看,只是把盘聚合了,但没有实现负载均衡.POWERPATH这方面很厉害,可以去看看快成精华的那个SYMM招聘贴.

    snapdrive for linux

     

    windows平台需要2003,等明天去找个VMWARE的安装好的版本再试。

     

    在LINUX上安装比较简单,LINUX版本是

    [root@localhost snapdrive]# uname -a

    Linux localhost.localdomain 2.6.9-67.ELsmp #1 SMP Fri Nov 16 12:48:03 EST 2007 i686 athlon i386 GNU/Linux

     

    安装好的netapp.snapdrive

    [root@localhost snapdrive]# rpm -aq |grep netapp

    netapp.snapdrive-3.0-1

    [root@localhost snapdrive]# pwd

    /opt/NetApp/snapdrive

    [root@localhost snapdrive]#

    关键的命令只有snapdirve一个,并且链接到/usr/sbin中,这样可以直接使用了。

    看snapdrive的MAN有很大很大一篇。等下慢慢看。

     

    [root@localhost snapdrive]# snapdrive

    snapdrive: For detailed syntax of individual commands, type

    snapdrive: ‘snapdrive command operation help’

    snapdrive: Supported commands and operations are:

    snapdrive snap show

    snapdrive snap list

    snapdrive snap create

    snapdrive snap delete

    snapdrive snap rename

    snapdrive snap connect

    snapdrive snap disconnect

    snapdrive snap restore

    snapdrive storage show

    snapdrive storage list

    snapdrive storage create

    snapdrive storage delete

    snapdrive storage resize

    snapdrive storage connect

    snapdrive storage disconnect

    snapdrive host connect

    snapdrive host disconnect

    snapdrive version

    snapdrive config access

    snapdrive config prepare

    snapdrive config check

    snapdrive config show

    snapdrive config set

    snapdrive config delete

    snapdrive config list

    Snapshot 快照的创建和管理

     

    在任何指定的时间在Filer 的每个数据卷都可存在255 个Snapshot 快照。这样一个对每个数据卷快照的限制适用于所有的数据卷而不管他包含多少个虚拟磁盘。SnapDrive 可以实施所有的SnapShot 管理功能。应用程序需允许快照功能从而发挥快照的优势。当一个快照备份启动后,应用程序必须在一个热备份模式下,并将其未完成的输入输出操作同步到磁盘上以保证快照数据的一致性。对于SAN 环境下,由于主机的文件系统控制着对磁盘的IO 操作,快照数据的一致性需要保证,而命令行手工创建快照或用FilerView接口创建快照并不管这么多,因此对于主机存储在Filer 的数据应该使用SnapDrive 来创建和管理应用系统的快照。

    LINUX下的ISCSI的环境搭建

     

    NETAPP的simulator不支持FC,那就只好用iscsi来做snapdrive的测试了.

    前面的贴子已经写到过了,在iscsi的环境中需要有发起方(initiator)和目标方(target).这个时候simulator就作为target,在LINUX的这端我们需要安装initiator.前面还提到过有三种方式,显然这里只能是用软件来实现的imitiator了.

     

    安装iscsi initiator

     

    在网上找到这个包,目前只是测试,所以也没有去查matrix.

    [root@localhost ~]# yum install iscsi-initiator-utils-4.0.3.0-6.i386.rpm

    Setting up Install Process

    Setting up repositories

    Reading repository metadata in from local files

    /etc/iscsi.conf

     

    装好iscsi的包后尝试启动iscsi服务.

    [root@localhost ~]# chkconfig –list iscsi

    iscsi 0ff 1ff 2ff 3:off 4:off 5:off 6:off

    [root@localhost ~]# service iscsi start

    Checking iscsi config: [FAILED]

     

    看样子还要去配置下/etc/iscsi.conf文件,more了一下,这个文件内容也不少.

     

    找到DiscoveryAddress=ISCSI_TARGET_HOST_OR_IP这行,编辑一下把simulator的IP地址加入

     

    #

    DiscoveryAddress=192.168.222.100

    # OutgoingUsername=fred

    # OutgoingPassword=uhyt6h

    # and/or

    #

    DiscoveryAddress=192.168.222.101

    # IncomingUsername=mary

    # IncomingPassword=kdhjkd9l

    #

    # —————

    # DIGEST SETTINGS

    # —————

    “/etc/iscsi.conf” 433L, 15087C written

    再在LINUX上运行启动ISCSI.

    [root@localhost ~]# service iscsi start

    Checking iscsi config: [ OK ]

    Loading iscsi driver: [ OK ]

    Starting iscsid: [ OK ]

    [root@localhost ~]#

     

     

    在simulator上可以看到相关的信息

     

    R1*> iscsi start

    Sun Apr 13 04:20:34 EDT [iscsi.service.startup:info]: iSCSI service startup

    iSCSI service started

    R1*> Sun Apr 13 04:20:43 EDT [iscsi.notice:notice]: ISCSI: New session from initiator iqn.1987-05.com.cisco:01.d0d449bf9fa6 at IP addr 192.168.222.109

    Sun Apr 13 04:20:43 EDT last message repeated 3 times

     

    R1*> iscsi initiator show

    Initiators connected:

    TSIH TPGroup Initiator

    3 1001 localhost.localdomain (iqn.1987-05.com.cisco:01.d0d449bf9fa6 / 00:02:3d:00:00:02)

    4 1000 localhost.localdomain (iqn.1987-05.com.cisco:01.d0d449bf9fa6 / 00:02:3d:00:00:01)

    建立LUN和IGROUP

     

    R1*> aggr create lunaggr -r 4 -t raid4 -d v4.20 v4.21 v4.22

    Creation of an aggregate with 3 disks has been initiated. The disks need

    to be zeroed before addition to the aggregate. The process has been initiated

    and you will be notified via the system log as disks are added.

    R1*> aggr status

    Aggr State Status Options

    lunaggr creating raid4, aggr raidsize=4,

    initializing snapshot_autodelete=off,

    lost_write_protect=off

    aggr0 online raid0, aggr root

    myaggr online raid4, aggr

     

    R1*> vol create lunvol lunaggr 150m

    Sun Apr 13 04:40:33 EDT [vol.language.changed:info]: Language on volume lunvol changed to en_US

     

    The new language mappings will be available after reboot

    Sun Apr 13 04:40:33 EDT [vv_config_worker:notice]: XL – Language of Volume lunvol has been changed to en_US.

    Creation of volume ‘lunvol’ with size 150m on containing aggregate

    ‘lunaggr’ has completed.

    R1*> vol status

    Volume State Status Options

    vol0 online raid0, flex root, no_atime_update=on,

    create_ucode=on,

    convert_ucode=on,

    maxdirsize=2621

    nfstest online raid4, flex create_ucode=on,

    convert_ucode=on

    lunvol online raid4, flex create_ucode=on,

    convert_ucode=on

    newnfs online raid4, flex create_ucode=on,

    convert_ucode=on

     

    R1*> lun create -s 100m -t linux /vol/lunvol/lun000

    R1*> lun show

    /vol/lunvol/lun000 100m (104857600) (r/w, online)

     

    LUN创建完成,下面做个IGROUP

     

    R1*> igroup create -i -t linux lun000group

    R1*> igroup show

    lun000group (iSCSI) (ostype: linux):

     

    R1*> igroup add lun000group iqn.1987-05.com.cisco:01.d0d449bf9fa6

    R1*> igroup show

    lun000group (iSCSI) (ostype: linux):

    iqn.1987-05.com.cisco:01.d0d449bf9fa6 (logged in on: ns0, ns1)

    R1*>

    R1*> lun map /vol/lunvol/lun000 lun000group

    lun map: auto-assigned lun000group=0

    linix识别FILER上的LUN

     

    先用fdisk看下系统中的盘,只看到系统盘.

    root@localhost ~]# fdisk -l

     

    Disk /dev/sda: 8589 MB, 8589934592 bytes

    255 heads, 63 sectors/track, 1044 cylinders

    Units = cylinders of 16065 * 512 = 8225280 bytes

     

    Device Boot Start End Blocks Id System

    /dev/sda1 * 1 13 104391 83 Linux

    /dev/sda2 14 1044 8281507+ 8e Linux LVM

     

    不重启机器就重启ISCSI的服务了.

    [root@localhost ~]# /etc/init.d/iscsi stop

    Searching for iscsi-based multipath maps

    Found 0 maps

    Stopping iscsid: [ OK ]

    Removing iscsi driver: [ OK ]

    [root@localhost ~]# chkconfig iscsi off

    [root@localhost ~]# chkconfig iscsi on

    [root@localhost ~]# /etc/init.d/iscsi start

    Checking iscsi config: [ OK ]

    Loading iscsi driver: [ OK ]

    Starting iscsid: [ OK ]

    [root@localhost ~]# fdisk -l

     

    Disk /dev/sda: 8589 MB, 8589934592 bytes

    255 heads, 63 sectors/track, 1044 cylinders

    Units = cylinders of 16065 * 512 = 8225280 bytes

     

    Device Boot Start End Blocks Id System

    /dev/sda1 * 1 13 104391 83 Linux

    /dev/sda2 14 1044 8281507+ 8e Linux LVM

     

    Disk /dev/sdb: 104 MB, 104857600 bytes

    4 heads, 50 sectors/track, 1024 cylinders

    Units = cylinders of 200 * 512 = 102400 bytes

     

    Disk /dev/sdb doesn’t contain a valid partition table

     

    Disk /dev/sdc: 104 MB, 104857600 bytes

    4 heads, 50 sectors/track, 1024 cylinders

    Units = cylinders of 200 * 512 = 102400 bytes

     

    Disk /dev/sdc doesn’t contain a valid partition table

    [root@localhost ~]#

     

    从LINUX到FILER是连接到了两个FILER的端口,认出了两块盘,这个正常.

    [root@localhost ~]# tail -f /var/log/messages

    Jul 29 04:44:42 localhost kernel: sdb: unknown partition table

    Jul 29 04:44:42 localhost kernel: Attached scsi disk sdb at scsi4, channel 0, id 0, lun 0

    Jul 29 04:44:42 localhost kernel: SCSI device sdc: 204800 512-byte hdwr sectors (105 MB)

    Jul 29 04:44:42 localhost scsi.agent[4564]: disk at /devices/platform/host4/target4:0:0/4:0:0:0

    Jul 29 04:44:42 localhost kernel: SCSI device sdc: drive cache: write through

    Jul 29 04:44:42 localhost kernel: SCSI device sdc: 204800 512-byte hdwr sectors (105 MB)

    Jul 29 04:44:42 localhost kernel: SCSI device sdc: drive cache: write through

    Jul 29 04:44:42 localhost kernel: sdc: unknown partition table

    Jul 29 04:44:42 localhost kernel: Attached scsi disk sdc at scsi3, channel 0, id 0, lun 0

    Jul 29 04:44:43 localhost scsi.agent[4597]: disk at /devices/platform/host3/target3:0:0/3:0:0:0

     

    [root@localhost ~]# find /sys/devices/platform/host* -name “block*”

    /sys/devices/platform/host3/target3:0:0/3:0:0:0/block

    /sys/devices/platform/host4/target4:0:0/4:0:0:0/block

    [root@localhost ~]#

    fdisk

     

    接下来使用fdisk对/dev/sdb做分区.

     

    完成后用fdisk -l 再查看

    [root@localhost ~]# fdisk -l

     

    Disk /dev/sda: 8589 MB, 8589934592 bytes

    255 heads, 63 sectors/track, 1044 cylinders

    Units = cylinders of 16065 * 512 = 8225280 bytes

     

    Device Boot Start End Blocks Id System

    /dev/sda1 * 1 13 104391 83 Linux

    /dev/sda2 14 1044 8281507+ 8e Linux LVM

     

    Disk /dev/sdb: 104 MB, 104857600 bytes

    4 heads, 50 sectors/track, 1024 cylinders

    Units = cylinders of 200 * 512 = 102400 bytes

     

    Device Boot Start End Blocks Id System

    /dev/sdb1 1 1024 102375 83 Linux

     

    Disk /dev/sdc: 104 MB, 104857600 bytes

    4 heads, 50 sectors/track, 1024 cylinders

    Units = cylinders of 200 * 512 = 102400 bytes

     

    Device Boot Start End Blocks Id System

    /dev/sdc1 1 1024 102375 83 Linux

     

    可以看出/dev/sdb,/dev/sdc信息都被改变.呵呵,它们实际上是同一个盘,不改变还要去做troubleshooting.

    在LUN上建立文件系统

     

    [root@localhost ~]# mkfs.ext3 /dev/sdc

    mke2fs 1.35 (28-Feb-2004)

    /dev/sdc is entire device, not just one partition!

    Proceed anyway? (y,n) y

    Filesystem label=

    OS type: Linux

    Block size=1024 (log=0)

    Fragment size=1024 (log=0)

    25688 inodes, 102400 blocks

    5120 blocks (5.00%) reserved for the super user

    First data block=1

    Maximum filesystem blocks=67371008

    13 block groups

    8192 blocks per group, 8192 fragments per group

    1976 inodes per group

    Superblock backups stored on blocks:

    8193, 24577, 40961, 57345, 73729

     

    Writing inode tables: done

    Creating journal (4096 blocks): done

    Writing superblocks and filesystem accounting information: done

     

    This filesystem will be automatically checked every 38 mounts or

    180 days, whichever comes first. Use tune2fs -c or -i to override.

    [root@localhost ~]# mkdir /iscsi

    [root@localhost ~]# mount /dev/sdb /iscsi

    [root@localhost ~]# cd /iscsi

    [root@localhost iscsi]# df

    Filesystem 1K-blocks Used Available Use% Mounted on

    /dev/mapper/VolGroup00-LogVol00

    7579576 1179128 6015424 17% /

    /dev/sda1 101086 13240 82627 14% /boot

    none 127364 0 127364 0% /dev/shm

    /dev/sdb 99150 5664 88366 7% /iscsi

    [root@localhost iscsi]# df -H

    Filesystem Size Used Avail Use% Mounted on

    /dev/mapper/VolGroup00-LogVol00

    7.8G 1.3G 6.2G 17% /

    /dev/sda1 104M 14M 85M 14% /boot

    none 131M 0 131M 0% /dev/shm

    /dev/sdb 102M 5.8M 91M 7% /iscsi

    [root@localhost iscsi]#

    [root@localhost iscsi]# touch fileonsdb

    [root@localhost iscsi]# cd ..

    [root@localhost /]# umount /iscsi

    [root@localhost /]# mount /dev/sdc /iscsi

    [root@localhost /]# ls -l /iscsi

    total 13

    -rw-r–r– 1 root root 0 Jul 29 04:57 fileonsdb

    drwx—— 2 root root 12288 Jul 29 04:55 lost+found

     

    可以看出对/dev/sdc做的文件系统,用另外一条路径去mount文件系统也是没有问题的.

    创建LUN的四种方式

     

    前面几个贴记录了用CLI来实现LUN的创建,IGROUP的创建及LUN的MAP。

    NETAPP的FILER提供了更多的方法来方便创建LUN,共有4种。

     

    lun create – The lun create command only creates the LUN. It does not create igroups or map LUNs to igroups.

    In addition lun create does not add portsets. A portset consists of a group of FCP target ports. You bind a portset to an igroup in order to make the LUN available only on a subset of the storage system’s target porsts. Any host in an igroup can access the LUNs only by connecting to the target ports in the portset.

    If an igroup is not bound to a portset, the LUNs mapped to the igroup are available on all of the storage system’s FCP target ports. By using portsets, you can selectively control which initiators can access LUNs and the ports on which they access LUNs.

     

    lun setup – The lun setup wizard is a command line interface (CLI) wizard that walks you through the process of creating and mapping the LUN.

     

    FilerView – FilerView performs the same function as lun setup through a GUI interface. FilerView is provided with the purchase of Data ONTAP.

     

    SnapDrive – SnapDrive was designed specifically for LUN management. If you use SnapDrive, it is recommended that you create and manage all of your LUNs within it.

    lun setup 建立LUN过程

     

    R1> lun setup

    This setup will take you through the steps needed to create LUNs

    and to make them accessible by initiators. You can type ^C (Control-C)

    at any time to abort the setup and no unconfirmed changes will be made

    to the system.

    Do you want to create a LUN? [y]:

    Multiprotocol type of LUN (solaris/windows/hpux/aix/linux/netware/vmware/windows_gpt)

    [linux]:

    A LUN path must be absolute. A LUN can only reside in a volume or

    qtree root. For example, to create a LUN with name “lun0″ in the

    qtree root /vol/vol1/q0, specify the path as “/vol/vol1/q0/lun0″.

    Enter LUN path: /vol/lunvol1/lun001

    A LUN can be created with or without space reservations being enabled.

    Space reservation guarantees that data writes to that LUN will never

    fail.

    Do you want the LUN to be space reserved? [y]: n

    Size for a LUN is specified in bytes. You can use single-character

    multiplier suffixes: b(sectors), k(KB), m(MB), g(GB) or t(TB).

    Enter LUN size: 100m

    The maximum size of a LUN that can be created in volume lunvol1 is

    79m (82837504)

    Enter LUN size: 79m

    You can add a comment string to describe the contents of the LUN.

    Please type a string (without quotes), or hit ENTER if you don’t

    want to supply a comment.

    Enter comment string:

    The LUN will be accessible to an initiator group. You can use an

    existing group name, or supply a new name to create a new initiator

    group. Enter ‘?’ to see existing initiator group names.

    Name of initiator group [lun000group]:

    lun000group (iSCSI) (ostype: linux):

    iqn.1987-05.com.cisco:01.d0d449bf9fa6 (not logged in)

    The LUN will be accessible to all the initiators in the

    initiator group. Enter ‘?’ to display LUNs already in use

    by one or more initiators in group “lun000group”.

    LUN ID at which initiator group “lun000group” sees “/vol/lunvol1/lun001″ [1]:

    LUN Path : /vol/lunvol1/lun001

    OS Type : linux

    Size : 79m (82837504)

    Initiator Group : lun000group

    Initiator Group Type : iSCSI

    Initiator Group Members : iqn.1987-05.com.cisco:01.d0d449bf9fa6

    Mapped to LUN-ID : 1

    Do you want to accept this configuration? [y]:

    Do you want to create another LUN? [n]:

    R1>

    简单看了一下,在UNIX下用snapdrive够麻烦的.在WINDOWS中和MMC集成了,还有LUN的创建向导,这个就简单了.

     

    艰难的snapdrive测试

     

    [root@localhost /]# snapdrive storage create -lun 192.168.222.100:/vol/lunvol2/lun002 -lunsize 50m

    0001-242 Admin error: Unable to connect using https to filer: 192.168.222.100. Enable SSL on the filer or retry after changing snapdrive.conf to use http for filer communication.

     

    出错了……按提示去把FILER上的options打开.

    R1> options http

    httpd.access legacy

    httpd.admin.access legacy

    httpd.admin.enable on

    httpd.admin.hostsequiv.enable off

    httpd.admin.max_connections 1023

    httpd.admin.ssl.enable off

    httpd.admin.top-page.authentication on

    httpd.autoindex.enable off

    httpd.enable off

    httpd.log.format common

    httpd.method.trace.enable off

    httpd.rootdir XXX

    httpd.timeout 300

    httpd.timewait.enable off

    R1> options httpd.admin.ssl.enable on

    SSL is not configured. Please use the command

    ‘secureadmin setup ssl’ to configure SSL.

    R1> secureadmin setup ssl

    Country Name (2 letter code) [US]:

    State or Province Name (full name) [California]:

    Locality Name (city, town, etc.) [Santa Clara]:

    Organization Name (company) [Your Company]:

    Organization Unit Name (division):

    Common Name (fully qualified domain name) [R1.testcifs.com]:

    Administrator email:

    Days until expires [5475] :

    Key length (bits) [512] :

    Sun Apr 13 12:28:51 EDT [rc:info]: Starting SSL with new certificate.

    R1> options httpd.admin.ssl.enable on

     

    再试

    [root@localhost /]# snapdrive storage create -lun 192.168.222.100:/vol/lunvol2/lun002 -lunsize 50m

    0001-136 Admin error: Unable to log on to filer: 192.168.222.100

    Please set user name and/or password for 192.168.222.100, i.e.

    snapdrive config set root 192.168.222.100

     

    FILER上的相关信息

    R1> Sun Apr 13 12:29:04 EDT [HTTPPool00:warning]: HTTP Authentication from 192.168.222.109 for ONTAPI system-get-ontapi-version failed

    艰难的snapdrive测试 二

     

    按照提示继续努力

    [root@localhost /]# snapdrive config set root 192.168.222.100

    Password for root:

    Retype password:

    [root@localhost /]# snapdrive storage create -lun 192.168.222.100:/vol/lunvol2/lun002 -lunsize 50m

     

    0001-678 Admin error: Cannot prepare for new LUNs — resulting total number (65544)

    would exceed maximum supported for OS (256).

     

    You must change host configuration to allow for more new LUNs

     

     

    找了半天也不知道问题出在哪里,看了些资料snapdrive用来在UNIX环境下建LUN应该不是个好选择。在WINDOWS平台下有图形界面的向导,要简单多了。还早到一个sql server和snapdrive搭配的例子,老是老了点不过还能看。

    不耽误进度,先跳过LINUX下snapdrive创建LUN部分,等以后看有实际成功例子的机会再补上。

    HBA的管理工具

     

    HBAnyware – HBAnyware is an HBA management GUI that is provided by Emulex.

    SANsurfer – SANsurfer is an HBA and switch management GUI provided by QLogic.

     

    有图形的,也有字符界面的。主要用在LINUX,SUN上做Bind host HBA to WWPN

     

    HBA的配置参数都在一个相应的文件中,存储对这个文件中的参数设置是有要求的,这个需要按照厂家提供的文档去配置。

    如何查找HBA的WWPN

     

    实际上很多啦。

    可以到交换机上去查。

    也可以用系统的命令来查。

    甚至可以到存储上去查,当然这个时候zone要是做好了的,或者是直连的。

    例如可以在SYMMWIN中的FA口上用inline命令的F8来看。

     

    NETAPP给的方法在下面

    AIX and HP-UX hosts

     

    WWPNs identify the physical ports on the HBA. You must supply the WWPNs when you create an FCP type igroup. To determine the WWPN on AIX and HP-UX hosts, use the command sanlun fcp show adapter –c. This displays the command you need to enter to create the igroup. If you do not include –c, the command will only list the HBAs and WWPNs.

    Note: Sanlun is a valuable utility available in the Host Utilities kits.

     

    Linux hosts

    To find the WWPN for the HBA installed on the Linux host, complete the following steps:

    Type modprobe. This command will load the driver.

    Example: If your driver is called qla23000, enter /sbin/modprobe qla23000. The system creates a /proc/scsi/qla23000 directory that contains a file for each QLogic HBA port. The WWPN is the file for that port.

    Look in each /proc/scsi/qla23000/HBA-port-num file and get the WWPN. The filename matches the HBA port number.

    Example: To get the WWPN for port 2, you would look in the file /proc/scsi/qla23000/2.

     

    LINUX的还比较有用。

     

    iSCSI Nodes

     

    说了FC的,现在来说说ISCSI的

     

    iSCSI target nodes can be connected in two ways:

    Standard Ethernet interfaces

    iSCSI target HBAs

    Each iSCSI node must have a node name. This node name functions much like the WWPN does for FCP. There are two possible node name formats. They are listed below.

    iqn-type designator

    The format of this nodename is conventionally “iqn.yyyy-mm.backward_naming_authority: unique_device_name.” This is the most popular node name format and is the default used by a NetApp storage system. Components of the logical name are the following:

    Type designator, iqn, followed by a period (.)

    The date when the naming authority acquired the domain name, followed by a period

    The name of the naming authority, optionally followed by a colon (

    A unique device name

    Example: “iqn.1992-08.com.netapp:sn.50400285″

    eui-type designator

    The format of this nodename is “eui.nnnnnnnnnnnnnnnn.” Components of the logical name are the following:

    The type designator itself, “eui”, followed by a period (.)

    Sixteen hexadecimal digits

    Example: “eui.123456789ABCDEF0″

    Filer Commands for Initiators and Targets

     

    Host Initiator HBAs

    The command fcp show initiator displays either all initiators, or as in the example above, an initiator of interest. Initiator information that is displayed includes port name and group.

    Storage System Target HBAs

    The command fcp show adapter displays all adapters, or when a specific FCP port is specified, an adapter port of interest. Information generated on the adapter includes slot, description, adapter type, status, WWNN, and WWPN

     

     

    simulator不能模拟FC的卡,哪位好心人贴个上面两命令的输出来给大家看看。

    Access LUNs on Solaris

     

    这个单独拿出来写,是因为SUN下面认盘相对要麻烦一些。

    LUNs created on the NetApp storage system that will be accessed via FCP must be configured on the SUN Solaris? host. Complete the following steps to access your LUNs on Solaris:

    Modify the /kernel/drv/sd.conf file: You must add an entry for each target ID in the sd.conf file so the host system knows which disks to probe when the system is rebooted.

    Discover LUNs: There are two methods which prompt the system to probe for new devices.

    Use the command /usr/sbin/devfadm.

    Reboot the host with the reconfigure option (reboot — -r).

    Run sanlun lun show to verify that the LUNs are visible.

    Use the format utility: The format utility will format and label new LUNs.

    Create a UNIX file system on the disk or use it as a raw device.

     

    Note: If your configuration uses Volume Management software, you must configure the LUNs so they are under the control of the Volume Manager.

     

    Host Booting from SAN or IP SAN

     

    SAN Boot Tasks – Windows FCP

     

    To use a LUN as a boot device, you must complete these steps:

    Configure the host’s BIOS boot order: If your host has an internal disk, you must enter BIOS setup to configure the host to boot from the LUN. You must ensure that the internal disk is not bootable through the BIOS boot order. Your HBA should be the first device in the boot order.

    Enable Boot BIOS on the HBA: BootBIOS enables the HBA to access the existing BIOS. It also enables you to designate an FC drive, such as a storage system LUN, as the host’s boot device. BootBIOS firmware is installed on your HBA. It is disabled by default. If you are using an Emulex HBA, use HBAnywhere or LP6DUTIL.EXE to enable BootBIOS. If you are using an QLogic HBA, use Fast!UTIL to enable BootBIOS.

    Obtain the WWPN of the host initiator HBA: The WWPN is required when you create an igroup for the Boot LUN. If you are using an Emulex HBA, use HBAnywhere or BootBIOS to obtain the WWPN. If you are using Qlogic, use SANsurfer or BootBIOS to obtain the WWPN.

    Cable the storage system so that only one path exists from the HBA to the boot LUN: MPIO (multipathing) drivers are not installed with the operating system and first boot. When preparing to install the operating system, there are special cabling instructions you must follow to ensure there is only one path from the HBA to the Boot LUN.

    Create the Boot LUN: After you obtain the WWPN for the HBA, you must create the LUN to use as a Boot LUN, map it to an initiator group, and assign a LUN ID. You must assign a LUN ID of 0 to a LUN that will be used as a boot device. LUNs with IDs other than 0 are not supported as boot devices.

    Configure the BootBIOS to use the Boot LUN: Follow steps that are specific to the BootBIOS of your HBA.

    Copy the Emulex or QLogic SAN boot driver from the Web: When you boot from a LUN, you must ensure that the operating system on the LUN has the required HBA driver for booting from a LUN. You must download these drivers from the Emulex or QLogic Web site. During the Windows? (2000, 2003) installation, you must install the driver as a third-party SCSI array driver from a floppy disk.

    Install Windows (2000, 2003) on the Boot LUN

    Install the FCP Windows Attach Kit software drivers

     

    第一台装好了以后的就简单了。

    SAN Boot Tasks – Solaris FCP

     

    Configure the host and storage system with supported firmware: Software and firmware on the host must be supported by the Attach Kit.

    Install the host operating system on a local disk: The disk on which the operating system resides must use a file system type that matches that of host.

    Download and install OpenBoot firmware on the HBA: OpenBoot firmware is available on the Emulex Web site.

    Create the Boot LUN: Use standard storage system commands and procedures to create and map a LUN assigned to LUN ID 0. You must partition the LUN to match the partitions on the host boot device. To do this complete these steps:

    Display information about the host boot device

    Modify the bootable LUN to model the partition layout of the host boot device

    Copy bootblks and boot data: The bootblk contains startup information, required by the Solaris host that you must install onto the raw bootable LUN. Installing the bootblk involves these actions:

    Use the uname –a command to determine the directory in /usr/platform where the bootblk is located

    Install the bootblk onto the bootable LUN

     

    Copy the boot data to the bootable LUN involves these actions:

    Create and mounting a file system on the bootable LUN. The file system must match the file system type on the host boot device.

    Move boot data from the host device onto the bootable LUN

    Edit the vfstab file to reference the bootable LUN

    Modify OpenBoot: OpenBoot is the firmware that the Emulex software uses to startup the system. OpenBoot firmware also includes the hardware-level user interface that you use to configure the bootable LUN. Modifying OpenBoot involves these actions:

    Validate the OpenBoot version

    Set the topology (point-to-point or arbitrated loop) to the bootable LUN

    Bind the HBA to the bootable LUN

    Create an alias for the bootable LUN (The alias substitutes for the device address.)

    Host Boot from IP SAN 之QLOGIC

     

    Verify system requirements: Your configuration must meet host OS and host platform version requirements.

    Install the QLogic SANsurfer iSCSI HBA Manager interface on any Windows? computer: This interface will be used to update BootBIOS and firmware. It can also be used to change the HBA configuration settings to support SAN booting. A card can be updated in a computer other than the IP SAN booting host and then placed in that host.

    Create a device driver diskette: Create the QLogic device driver diskette for the QLogic card and operating system combination that you are using. During the OS installation, you are prompted to install the device driver from the diskette. This enables the OS to communicate with the HBA.

    Disable the primary hard drive: It is only necessary to disable the primary hard drive if the OS installed on the host. To disable the hard drive, use the system BIOS to physically remove the drive. If the OS installer detects a boot.ini file on the primary hard drive, it will not install the required boot files to boot the LUN. This means that if you disable or remove the primary hard drive at a later date, the host will not be able to boot from the LUN.

    Make sure the HBA has an IP address

     

    Update the HBA BootBIOS and firmware to theNetApp supported version levels: Before you configure the HBA to boot from theSAN, you may need to flash the Boot BIOS and firmware to the HBA in order to beat NetApp supported version levels. You can update the Boot BIOS and firmwareusing the QLogic SANsurfer iSCSI HBA Manager interface or a DOS-bootablediskette.

     

    Obtain the initiator and target node names: TheiSCSI node name is required when you create the igroup and map the LUN to thatigroup.

     

    Create the Boot LUN: After you obtain aninitiator node name, you create the LUN that will be used as a boot device, mapit to an igroup, and assign a LUN ID.

     

    Configure the HBA BIOS to boot from the Boot LUN:Use the QLogic Fast!Util utility (Ctrl-Q during host boot) to configure the HBABIOS. Enable the Spinup Delay option. It is disabled by default on a factory-shippediSCSI HBA.

     

    Configure the BIOS boot order: After youconfigure the HBA BootBIOS, you enter system BIOS setup to configure the hostto boot from the LUN and ensure that the internal disk is not bootable throughthe system BIOS boot order.

     

    Install the operating system on the Boot LUN

     

     

    Alter settings to support SAN Boot: You mustchange these settings to support SAN Boot:

    QLogic configuration settings

    Windows configuration settings

    Multipathing

     

    Multipathing is the term used to describe an FC or IP SAN solution that has been designed to include at least two distinct physical paths from a host to a target LUN. Multiple paths from a particular host to a particular target LUN generally reduce the risk of a single point of failure in the Fibre Channel fabric or Ethernet network and contribute to a highly available (HA) SAN solution. However, this is not a requirement. For example, if two physical paths from a host to a target LUN ran through the same switch then the solution would not be considered to be highly available or without a single point of failure. If the switch failed, then both paths would be lost.

    Multipathing is implemented for two reasons.

     

    Multipathing provides highly available (redundant) paths. It is implemented to eliminate a single point of failure in the FC fabric or Ethernet network. This includes the interfaces on the hosts and target controllers.

    Some forms of multipathing increase throughput using multiple physical connections. This allows the host to simultaneously send data across more than one path to target controllers.

    In active/active clustering, operating systems, applications, or services running on a cluster can access the same resources at the same time. In active/passive clustering, one node is preferred (active) and the operating system, application, or service cannot access the resources until a failure occurs and the passive node takes over for the active node.

     

    中低端的存储多为active/passive,高端的当然是active/active.

    Active/Active Storage Controller Failover

     

    There are two types of active/active storage controller configurations: standard and mirrored.

    资料中只有standard的,mirrored的没有.

     

    A standard active/active storage system configuration contains two paths to the Fibre Channel disk shelves, a local controller, and a partner controller (except the FAS270c, which does not require two sets of disk shelves). Both controllers must be the same model and they should be running the same version of Data ONTAP.The controllers are connected to each other via a cluster interconnect (IC). Each controller continually monitors its partner, mirroring the data from its partner’s NVRAM.

    以前EMC的SYMM的cache是没有镜像的,后面迫于竞争压力才加入的

     

    There are many benefits to an active/active storage controller configuration, including:

    High-availability data-clustering solution: This configuration protects against controller failure by transferring the data service from the failed storage system controller to its partner controller. In addition to controller failure, active/active storage controller failover can also protect against other hardware failures, such as network interface. Controller failover is also an effective tool for reducing planned downtime of one of the nodes.

    Nondisruptive controller and disk maintenance: When you halt one controller and allow takeover, the partner controller continues to serve data for the halted controller while you replace or repair hardware in the halted controller.

    Nondisruptive software upgrades: When you halt one controller and allow takeover, the other controller continues to serve data for the halted controller while you upgrade the halted controller.

    Single Point of Failure

     

    A single point of failure represents the failure of a single hardware component that can lead to the loss of data access or potential loss of data. A single point of failure does not include multiple or rolling hardware errors, such as double disk failure without RAID-DP. All hardware components have demonstrated very good reliability with low failure rates. If a hardware component, such as a storage controller or adapter, fails, then you can use an active/active storage controller configuration to provide continuous data availability and preserve data integrity for client applications and users.

    Active/Active Storage Controller Configuration

     

    In a clustered architecture, a pair of storage controllers of the same model are connected to one another through a cluster interconnect adapter. The cluster interconnect (IC) allows the storage controllers to perform these operations:

    Determine whether or not the other storage controller is functioning (heartbeat)

    Mirror the log data to each other’s NVRAM

    Pass data across the IC between a host and the partner controller in the event of a switch or fabric failure (dependent on cfmode). Any path utilizing the IC to pass data is known as a proxy path.

    In the diagram, storage controllers use FC-AL adapters to manage their disks. The cable from the FC-AL adapter in each storage controller connects to the interface module in the disk shelf. Each storage controller’s disks are on an A Loop, and the storage controller’s partner’s disks can be accessed on takeover on a B Loop. The storage controllers must have redundant network connections so that each storage controller can assume its partner’s network identity.

    An active/active storage controller configuration (cluster) allows one storage controller to take over for the other if the second storage controller fails. This means that data from the failed storage controller’s disks can still be served through the functioning storage controller.

     

    据说NETAPP的SIMULATOR可以做这个测试,晚上回来继续了.

    呵呵,直接帖原文了阿。

    值得说明的是,用snapdrive可以实现基于dsm的multipath,可以fc和iscsi hba混用共同访问一个LUN。

    基于iscsi层的multipath,这个在ms的iscsi initiator就自带了,叫做mcs,multi connection per session。

    两种不同层次的multipath

    cfmode – Cluster Failover Mode

     

    Cfmode only applies to Fibre Channelenvironments in an active/active NetApp storage controller configuration. Thecfmode determines how target ports do the following:

     

    Log into the fabric

     

    Handle local and partnertraffic for a cluster

     

    Provide access to local andpartner LUNs in a cluster

    In the original release of Data ONTAP 6.3,which included SAN support for Fibre Channel, cfmode standby was the implieddefault. There was not a setting for cfmode in that release, and it was notcalled cfmode standby. However, when Data ONTAP 6.5 was released, four cfmodeswere introduced. One of these modes was standby. The others were partner, mixedand dual fabric. In Data ONTAP 7.1, a new cfmode called single system image(SSI) became available. SSI is the default cfmode for new installations withData ONTAP 7.2. The availability of standby, partner, mixed and dual fabricmodes is dependant on the storage controller model, Data ONTAP version, and/orthe use of 2Gb or 4Gb FC ports.

    Five cfmodes

     

    There are five possible cfmodes on the storage controller. Only one cfmode can be set per each storage controller, and in a cluster situation the cfmode must be the same for both systems.

    Standby

    The standby mode is supported on all systems except the FAS270c. It supports only Windows and Solaris operating systems. In addition, this mode requires additional switch ports.

    Partner

    The partner mode is supported on all systems except the FAS270c and the FAS6000 series. All switches and host operating systems are supported.

    Mixed

    The mixed mode is supported on all systems except the FAS270c and the FAS6000 series. Mixed mode supports all host operating systems, but requires a switch that supports a public loop.

    Dual Fabric

    The dual fabric mode is only supported on a FAS270c. All host operating systems are supported by this mode. This mode requires a switch that supports a public loop. It requires fewer switch ports.

    Single Image

    The single image mode is supported on all systems, switches, and host operating systems. This mode makes all LUNs available on all target ports.

    These modes can be viewed with the fcp show cfmode command.

     

    FCP Proxy and Multi-ID Support

     

    FCP ProxyFCP proxy allows Data ONTAP? tosupport hosts that have active/passive multipathing software. Theactive/passive connection is maintained using an adapter between storagecontrollers to proxy commands. Specifically, FCP Proxy is needed to supportHP-UX and AIX hosts that bind the physical switch port address to the targetdevice. HP-UX and AIX must have static source identifier (S_ID) and destinationidentifier (D_ID) addresses in a SAN FCP environment.

    Multi-ID SupportMulti-ID support is required for controllerssuch as the FAS270c, which have only one physical FC port per controller.

    Solaris , Windows , AIX HP-UX , and Linux FC Multipathing

     

    The following multipathing solutions arecurrently supported by NetApp:

     

    Solaris

     

    VERITAS Dynamic Multipathing(DMP) – VERITAS DMP providesmultiple paths to a single LUN. This allows for greater throughput and highavailability.

    VERITAS DMP supports active/active and active/passive modes. Inactive/active mode, multiple paths are used simultaneously. VERITAS uses a loadbalancing policy that balances the I/O across all available paths.

    In active/passive mode, a primary path is owned by one controller and asecondary path by another controller. If a path that was passive becomesactive, the ownership of the LUN switches to the controller on the active path.

     

    Windows

     

    NTAP DSM – NetApp uses the sameDSM for FC- and iSCSI-connected LUNs. This DSM provides the active/passive failover-onlyload balancing policy on a per-LUN basis. For a single LUN, all I/Os will bedone across a single path until a failure of the active path occurs. However,the LUNs are assigned paths based on the round robin algorithm.

     

    VERITAS DSM

     

    VERITAS DMP

     

    AIX

     

    Dot Hill? SANpath? – SANpath isthe tool that manages multiple paths for AIX. This provides high availabilityand improved performance. SANpath allows you direct I/O requests throughmultiple HBAs and to assign logical drives to an HBA.

     

    Native MPIO

     

    VERITAS DMP

     

    HP-UX

     

    HP-UX PVLinks – PVLinks (PhysicalVolume Links) are HP-UX’s multipathing solution for virtual disks. Duringfailover, PVLinks uses a simple algorithm: it tries the first known path,followed by the second known path, and so on, until it has tried all paths. Ifall paths are unavailable, the LUN (Virtual Disk) goes offline. PVLinks doesnot perform any load balancing functions.

     

    VERITAS DMP

     

    Linux

     

    QLogic? SANsurfer forLinux – SANsurfer is used to manage QLogic HBAs. It provides multipathingoptions for Linux.

    Depending onyour cfmode, multipathing software may be required. This course focuses onSolaris and Windows multipathing solutions.

     

     

     

    DMP看来是个通用的选择,EMC的POWERPATH配合其自己的产品是非常不错的,对别的厂家看说明是有支持的,但没用过.

    NETAPP不知道为什么不开发一个自己的多通道管理软件.OEM也可以么.哈哈.

     

    The single image, partner, and dual fabriccfmodes always require multipathing software on the host operating system. Themixed cfmode requires multipathing software on AIX and HP-UX only. The standbycfmode is not supported on AIX or HP-UX.

    iSCSI Multipathing for Windows

     

    Multipathing Input/Output (MPIO) – The “classic” way to do multipathing is to insert a separate multipathing layer into the storage stack. This method is not specific to iSCSI or to any underlying transport, and is the standard way to achieve multipathing access to Fibre Channel and even parallel SCSI targets. There are multiple implementations of this type of multipathing on the various operating systems. The MPIO infrastructure offered by Microsoft is the standard way to do this on Windows Server technologies. With the Microsoft MPIO, each storage vendor supplies a device-specific module device specific module for its storage array.

    NTAP (NetApp) DSM – NTAP DSM is a part of SnapDrive 3.2 and higher. It requires a software initiator. NTAP DSM is not currently supported in configurations that include VERITAS Storage Foundation.

     

    Note: NTAP or NetApp DSM is for use with SnapDrive 4.1 and earlier. In future releases (subject to change), the DSM will be called Data ONTAP? DSM for Windows? MPIO. The Data ONTAP DSM for Windows will be compatible with SnapDrive 4.2 and later.

    Microsoft iSCSI DSM – Microsoft iSCSI DSM is supported in active/passive and active/active modes. It requires a software initiator.

    VERITAS? DSM – Veritas DSM is supported beginning with Windows Server 2003.

     

    好不容易找到个SNAPDRIVE,结果安装的时候还要LIC的,没有不给装,做个实验都不行。

     

    Multiple Connections per Session (MCS) – MCS creates the multiple paths starting at the iSCSI session layer of the storage stack. Both the iSCSI initiator (host) and the iSCSI target (controller) need to support multiconnection sessions in order to configure sessions with multiple connections. MCS requires a software initiator. MCS should not be confused with the Microsoft Cluster Service

    iSCSI Sessions

     

    A session is established when the host initiator logs into the iSCSI target. This session is similar to a pipe or conduit. Within the session you can have one or more connections. The session is the pipe and the connections run inside the pipe or session.

    In most instances of iSCSI, each session has a single connection. When using a multipathing option like TCP/IP link aggregation and Multiple Connections per Session (MCS), the number of connections within a session can be increased. Having multiple connections within a session provides these benefits:

    There is no single point of failure on the iSCSI-type SAN network

    There may be increased throughput

    There is session persistence in the event of a connection failure.

    Sessions and Connections

     

    The diagram provides two examples illustrating sessions and connections, and how they relate to each other.

    The top example shows four sessions each with one connection each. Microsoft Multipath I/O (MPIO) would use a session and connection configuration like this. The MPIO software manages how the sessions are used to move data between the host and the target LUNs.

     

    这些东西和NETAPP就没有直接关系了,对所有使用ISCSI的都一样。

    Microsoft Multipathing Input/Output (MPIO)

     

    There are multiple implementations of multipathing on the various operating systems. The Windows MPIO infrastructure offered by Microsoft is the standard way to do this on Windows server technologies. With the Microsoft MPIO, each storage vendor supplies a device-specific module (DSM) for its storage array. NetApp currently supports three DSMs, NTAP DSM (Data ONTAP 6.5.4 or higher), Microsoft iSCSI DSM (Data ONTAP 6.5.6 or higher), and VERITAS DSM (Data ONTAP 7.0? or higher). NetApp does not support the installation of the Microsoft iSCSi DSM on the same host as the NTAP DSM, even if the Microsoft iSCSI DSM is not being used. The same applies to the VERITAS DSM. It cannot be installed on the same host as the NTAP DSM at this time.

    Note: VERITAS DSM and Microsoft iSCSI DSM are not currently compatible with SnapDrive or SnapManager?.

    NTAP DSM

    The 图 shows a host running the NetApp NTAP DSM with two iSCSI HBA’s, or two NICs with the Microsoft iSCSI Software Initiator, or one iSCSI HBA and one NIC with the Microsoft iSCSI Software Initiator attached to the IP SAN network. The MPIO drivers and the DSM layer manage the “paths” and shield the upper layers of the operating system from the underlying multiple paths to each LUN and the management of those paths. The two black arrows represent two sessions that have been established between the host and the target. The blue arrows represent two paths (over the two sessions) managed by the DSM that the host can use to access a LUN on the target. The DSM presents a single LUN to the upper layers of the OS. Without the DSM to manage the paths, the host would be presented with the same LUN multiple times.

    Multiple Connections per Session (MCS)

     

    Multiple Connections per Session (MCS) is an optional part of the iSCSI specification. These contain multiple connections starting at the iSCSI session layer of the storage stack. Both the iSCSI initiator (host) and the iSCSI target (storage controller) need to support MCS in order to configure sessions with multiple connections.

    MCS is currently implemented on the Windows host side using the Microsoft iSCSI Software Initiator 2.X and on the target side using Data ONTAP 7.1 or higher. There is no additional MPIO layer required on the host. Refer to the NetApp iSCSI support matrix for the most up-to-date information regarding supported Data ONTAP and initiator releases. MCS does not currently support the use of iSCSI HBAs, only network interface cards. Given that the iSCSI initiator portion of the stack resides on the HBA itself, implementing MCS across iSCSI HBAs will have its challenges.

    In the diagram, two connections are present in one session. With Microsoft iSCSI Software Initiator 2.0 and higher, up to four connections are allowed per session. Data ONTAP allows up to 16 connections per session.

    The following points apply to MCS:

    MCS requires Data ONTAP 7.1 or higher.

    iSCSI must be licensed and enabled.

    igroups and LUNs must be created.

    A Session between the host and target must be established. This process also establishes a single connection within the session.

    Additional connections can be added within the session.

    In order to allow Data ONTAP to support MCS an administrator must set the options iscsi.max_connections_per_session option to specify the number of connections per session that Data ONTAP will accept. The value can range from 1 to 16 connections per session. Four is the recommended value if MCS is to be employed. The default value is use_system_default. (In Data ONTAP 7.1 and 7.2, the system default is one connection per session)

    All LUNs accessed over a session are affected by that session’s load balancing policy. Active/active round robin is the only load balancing policy supported by NetApp at this time.

    All interfaces on the NetApp Storage System that are involved in a particular MCS session must all belong in the same Target Portal Group

    iSCSI Multipathing for UNIX? and Linux?

     

    放这里供大家备查

     

    Linux

    IP Trunking, Portal Failover, and Device-Mapper are Linux multipathing options. The options that are supported will vary Linux distribution (Red Hat or SuSE) and version.

    IP Trunking

    Aggregating bandwidth across multiple physical links to a switch is referred to as teaming, trunking, port trunking, or link aggregation. This allows a machine (frequently a server) to treat multiple physical connections to switch units as a single logical link.

    NIC Teaming is not supported by Microsoft with its iSCSI Software Initiator. NIC Teaming is supported on the public (no iSCSI traffic) Windows network. NIC Teaming is also currently supported by NetApp on all other operating systems on the NetApp iSCSI Support Matrix.

    Portal Failover

    When an existing path to the target fails on Linux, the iSCSI driver attempts to connect through the next available IP address. You may select a preferred portal in the case of failure. The portal failover feature is turned on by default and the process of failover is automatic. You may turn off portal failover by disabling the portal failover parameter in /etc/iscsi.conf.

    Device-Mapper

    Device-Mapper is a Linux kernel component that supports Logical Volume Management (LVM). If a path fails, Device-Mapper will reroute requests over available paths.

    Device-Mapper recognizes each path as a separate device. It creates another device on top of these multiple paths and uses the new device to reroute requests to the underlying devices.

    For more information on Device-Mapper, refer to Red Hat’s Multipath-usage.txt file at http://www.redhat.com/docs/manua … ipath-usagetxt.html

    For more information on accessing LUNs using Linux multipathing solutions, refer to the iSCSI Red Hat Enterprise Linux Initiator Support Kit Setup Guide.

    HP-UX

    IP Trunking and HP-UX PVLinks are NetApp supported HP-UX multipathing options.

    HP-UX PVLinks

    PVLinks (Physical Volume Links) are HP-UX’s LVM integrated multipathing solution for disks. During failover, PVLinks uses a simple algorithm: it tries the first known path, followed by the second known path, and so on, until it has tried all paths and will continue to try all the paths. PVLinks does not perform any load balancing functions.

    Solaris

    IP multipathing (IPMP) and Solaris MPxIO are NetApp supported Solaris multipathing options.

    IP/MP

    In Solaris IP multipathing (IP/MP) two identical network cards are grouped together with one IP address. The cards automatically fail over from one card to the other with no loss of service. You may create multiple logical IP addresses in order to force load balancing. IPMP provides failover across switches, providing an additional layer of redundancy. IPMP works across various network adapters to ensure that the switch is not a single point of failure.

    MPxIO

    MPxIO is the standard Solaris multipathing solution. To use MPxIO for iSCSI, you should have at least two Ethernet interfaces on the storage system enabled for iSCSI traffic. Each iSCSI interface must be in a different iSCSI target portal group. Data ONTAP does this by default.

    For more information on enabling MPxIO on Solaris, refer to the iSCSI Solaris Initiator Support Kit Setup Guide.

    FlexShare

     

    FlexShare is a Data ONTAP software feature that provides workload prioritization for a storage system. It prioritizes processing resources for key services when the system is under heavy load. FlexShare does not provide guarantees on the availability of resources or how long particular operations will take to complete. FlexShare provides a priority mechanism to give preferential treatment to higher-priority tasks.

    How FlexShare Works: Basics

     

    FlexShare allows you to assign priorities to different volumes. In addition, it provides the ability to configure certain per-volume attributes, including user versus system priority and cache policies.

    WAFL Operations

    A read or write request initiated from any data protocol is translated to individual read or write WAFL operations by the file system. Similarly, a system request is translated into individual WAFL operations. Data ONTAP classifies each WAFL operation as a user or system operation based on its origin. For example, a client read request is classified as a user operation; a SnapMirror? request is classified as a system operation.

    Processing Buckets

    FlexShare maintains different processing buckets for each volume that has a configured priority setting. FlexShare populates the processing buckets for each volume with WAFL operations as they are submitted for execution. The processing buckets are only used when the FlexShare service is on; when the FlexShare service is off, all WAFL operations are bypassed from processing buckets and sent directly to WAFL.

    Data ONTAP maintains a default processing bucket. When the FlexShare service is on, all WAFL operations associated with volumes that do not have a FlexShare priority configuration are populated in the default processing bucket; all WAFL operations for a volume that have a FlexShare priority configuration are populated into a dedicated bucket.

    How FlexShare Works: Example

     

    The “FlexShare off” figure depicts the order in which tasks arrive for processing and the order in which they are processed by the storage system. The order of tasks processed is exactly the same as the order in which the tasks arrive.

    The “FlexShare on” figure depicts a possible ordering of tasks when the FlexShare service is enabled. The order in which tasks arrive is different from the order in which they are processed by the storage system. FlexShare orders tasks for processing by taking into account the priority configuration. In this example, Vol1 has a higher priority configuration than the other volumes. This means that the WAFL operations from Vol1 are preferentially processed.

    FlexShare Benefits

     

    FlexShare provides storage systems with these key features:

    Multiple, disparate workloads can share a single storage system. This allows for more storage consolidation.

    Business-critical applications receive I/O priority

    Control of system and client workloads allows you to easily adjust workloads when priorities change.

    These features allow storage administrators to tune how the system should prioritize system resources in the event that the system is overloaded

     

    SnapMirror

     

    There are two types of SnapMirror solutions:

    Asynchronous SnapMirror – Asynchronous SnapMirror is an automated file system or qtree replication for disaster recovery or data distribution. Updates of new and changed data from the source to the destination occur on a schedule defined by the storage administrator. Updates can be as frequent as per minute or as infrequent as per weekly or even monthly, depends on the need of the user.

    Synchronous SnapMirror – Synchronous SnapMirror replicates writes from a source volume to a partner destination volume at the same time it is written to the source volume. Updates from source to destination are performed in real time interval.

     

    异步是要丢部分数据的,这个无法避免。总比全丢好。

    What is Snapshot?

     

    The WAFL file system can copy itself (snapshot) at any point in time, and make the copied versions of the file system available via “special” subdirectories that appear in the current (active) file system. Each copied version of the file system is called a snapshot. Up to 255 concurrent snapshots per volume are upported by the current Data ONTAP operating system.

     

    A storage snapshot creates a separate set of pointers to data that can be mounted as a volume or file system to another host and treated as though it were a duplicate of the original data. Creating snapshots is very quick, because it is essentially an index. Data blocks are not duplicated.

     

    A snapshot can be scheduled to occur automatically or taken manually. Automatic schedules can be created on an hourly, nightly, or weekly basis.

     

    When you install Data ONTAP on a storage appliance, it creates a default snapshot schedule.The default snapshot schedule automatically creates one nightly snapshot Monday through Saturday at midnight, and four hourly snapshots at 8 a.m., noon, 4 p.m., and 8 p.m. Data ONTAP retains the two most recent nightly snapshots and the six most recent hourly snapshots, and deletes the oldest nightly and hourly snapshots when new snapshots are created.

    看一个老外的BLOG说,aggrsnapshot可以关掉,又多5%的空间可以用。




    SnapRestore

     

    Instantaneous recovery – any size volume (up to 16TB)

    File-level SnapRestore capability – crucial for large files/LUNs

    Instantly recover from data corruptions

    Critical tool when integrated into change control processes (e.g., SnapRestore as a backout plan for software upgrades)

     

    SnapRestore is an optional feature 意思应该是要用就要买许可,哈哈。NETAPP的特点,功能很多,都包含在DOT中,只要买了许可加入就可以用。和容量无关,系统大了划得来。

    Recover 比较
    磁带和SnapRestore

     

    Recover 比较
    磁带和SnapRestore

     

    SnapRestore 的就是把指针的指向改一下,当然很快。图中写的磁带60G/H,实际现在的情况应该不止。


    同步?异步?半同步?

     

    SnapMirror can be configured into 3 different replications mode. All are available with a single license.

     

     

     

    The first is Synchronous mirroring. In this solution the data at the DR site exactly matches the data at the primary site.

    This is achieved by replicating every data write to the remote location and not acknowledging to the host the write occurred until the remote systems confirms the data was written.

    This solution provides the least data loss but there is a limit of 50-100Km before latency becomes too great because the host application must wait for an acknowledgement from the remote NetApp devices.

     

     

     

    The second is semi-synchronous SnapMirror. This configuration allows you to achieve a near zero data loss DR solution without the performance impact on the host application.

    This solution also allows you to do synchronous type replication over longer distances.

    How this works is that when data is written to the primary storage an acknowledgement is immediately sent back eliminating the latency impact on the host. In the background SnapMirror tries to maintain as close to synchronous communication as possible with the remote system. SnapMirror has user defined thresholds on how far out of sync are allowed for the source and remote copy data sets.

     

     

     

    Finally asynchronous SnapMirror.

    Asynchronous SnapMirror allows you to replicate data at adjustable frequencies. You can do this type of point in time replication as frequently as 1 minute or as infrequent as days.

    There is no distance limitation and is frequently used to replicate long distances to protect against regional disasters. And only the blocks that have changed between each replication are sent, minimizing network usage.

     

    通过存储设备来做的数据复制,把主机的CPU,MEMORY资源节约出来处理应用需求.在同一个城市里见的多的是同步模式.半同步模式沒见过实际使用.

    SRDF的时候为了不把系统的性能一下拉下来,还是要把模式设为AC,等数据基本同步完的时候再改成同步.

    snapmirror中的几个名词定义

     

    Source: a storage appliance system whose data is to be replicated.

    Destination: a storage appliance system which contains data replica.

    源和目标,不要搞反了.反了就不是数据保护是毁灭了.

     

    VSMQSM

    Volume SnapMirror (VSM) is the replication process from a source volume to a destination

    volume.

    Qtree SnapMirror (QSM) is the replication process from a source qtree to a destination qtree.

    这个比较特殊,別的厂家都是VSM.Q的沒见过.

     

    SnapMirror Components

    Source volumes or qtrees: SnapMirror source volumes and qtrees are writable data objects

    whose data is to be replicated. The source volumes and qtrees are the objects normally visible,

    accessible, and writable by the clients.

    就是说这边是生产端,数据按照常规模式在读写.

    Destination volumes or qtrees: the SnapMirror destination volumes and qtrees are read-only

    objects, usually on a separate system, to which the source volumes and qtrees are replicated. The

    destination volumes and qtrees are normally accessed by users only when a disaster takes down

    the source system and the administrator uses SnapMirror commands to make the replicated data

    at the destination accessible and writable.

    目标端对于一般的主机来说在snapmirror时刻是不可写的,这个是当然.要不源端的数据也写过来,这边的主机也写,那不乱套了.

     

    读是可以的,但往往做了这种保护的VOL都是操作系统中VG的一部分,在目标端也沒办法读.因为这个时候LVM去激活VG的时候需要写些东西到盘上,但盘是只读的,VG挂不起来.也无意义.

    要使用目标端容灾的数据多是在这些容灾的VOL上创建快照之类的东西来用.

    VSM can be synchronous or asynchronous, while QSM is available with asynchronous

    mode only.

    VSM is a block-for-block replication; it transfers the file system verbatim. QSM is a

    logical replication; all of the files and directories in the source file system are created in

    the destination file system.

    VSM can occur only with volumes of the same type; that is, both volumes are traditional

    volumes or both volumes are flexible volumes. QSM occurs between qtrees regardless of

    the type of volume.

    With VSM, the destination volume is always a replica of a single source volume and it is

    read-only. With QSM, only the destination qtree is read-only, while the volume remains

    writable and may contain replicated qtrees from multiple source volumes and qtrees or

    non-tree data not managed by SnapMirror.

    VSM replicates all snapshot on the source volume to the destination volume. QSM

    replicates only one snapshot of the source qtree to the destination qtree.

    VSM can be initialized using a tape device (SnapMirror to tape) whereas QSM does not

    support this feature. As well, VSM updates are delayed during a dump of the destination

    volume to tape. QSM updates are not affected by backup operations.

    Cascading of mirrors is supported only for VSM.

     

    NETAPP比较有意思,有基于BLOCK的还有基于文件的增量模式.也算是比较灵活吧.还能直接同步到磁带,这个也够有意思.

    baseline

     

    The first step involved in SnapMirror replication, is to perform an initial baseline transfer from

    the selected source volume or qtree to the destination volume or qtree. This operation is

    commonly named an “initialization” and creates the SnapMirror relationship between the source

    qtree or volume and the destination qtree or volume.

     

    The baseline transfer is initiated and driven by the destination by establishing TCP connection

    with the source system. For VSM, upon this transfer request, the source system takes a snapshot

    of the file system to be replicated and sends all of the snapshot’s data to the mirror system. For

    QSM, no snapshots are sent from the source to the destination. This version of the mirror acts as

    the baseline for the next incremental update. After the baseline transfer has completed, the

    destination volume or qtree on the target system is consistent and identical to the base snapshot

    read only copy on the source system. The replica can be accessed as a read-only file system by

    authorized network clients.

     

    同步的初始化,自己指定源和目标的对应关系.EMCSRDF在做BIN的时候直接对应好,想改就麻烦了.还是这种方式灵活一些.

    我看资料里还说SNAPMIRROR中间可以走FC,或者和以太网混着走,沒明白.

    不过阵列走FC做容灾的多,走以太网的不多见,倒是成本低很多,关键要看网络的情况如何了.

    Scheduling incremental updates

     

    After the source and destination file systems are synchronized for the first time, you can schedule

    incremental updates using the snapmirror.conf file. This file must be created in the destination

    root volume, under the /etc directory. Based on this schedule, the destination will request an

    incremental updates to the source system.

    异步模式下的数据更新问题.这个很好理解,定时系统自动把增量数据传送到目标端.

     

    两个重要的系统配置文件

    snapmirror.allow

    The snapmirror.allow file specifies the hostnames of destination systems that are allowed to

    copy data directly from the source system. You can generate this file on the root volume of the

    source system, under the /etc directory.

    snapmirror.conf

    The snapmirror.conf file defines the relationship between the source and the destination, the

    schedule used by the destination to copy data, and the arguments that control SnapMirror when

    copying data. The file must be created on the root volume of the destination system, under the

    /etc directory, then ONTAP will read the file every minute.

    Note: you can disable replication for a specific relationship by commenting out the entry in the

    snapmirror.conf file by preceding the entry with a pound sign (#).

    When you edit the file, each entry is made of 4 fields separated by a space and within a field,

    values are devided by a comma.

     

    同步前的一切准备工作(异步方式)

     

    Make sure the source volume or qtree is online

    For VSM

    Create a non-root restricted destination volume

    Destination volume capacity > or = to source

    这个比较奇怪,对于LUN来说都是把一样大小的做同步,LVM才能行,目标盘大的情况还真沒做过.不知道同步过去VG起的来不.

    Disks checksum type must be identical

    Quota cannot be enabled on destination volume

    For QSM

    Destination qtree must not exist and cannot be /etc

    Destination volume must have 5% extra space

    TCP port range 10565-10569 must be open

     




     

    3050:



    6000:

     

    模拟拔盘:

    v7-2-1P1> aggr status -d aggr0
    Device          HA    SHELF BAY CHAN    Disk Vital Product Information
    ———-      ————— —–   ——————————
    v4.16   v4    1   0     FC:B    72355100
    v4.17   v4    1   1     FC:B    72355101
    v4.18   v4    1   2     FC:B    72355102

    [root@DOT ,disks]# pwd
    /sim/,disks
    [root@DOT ,disks]# ls -l
    total 3687300
    -rwxr-xr-x    1 root     root          908 Jan 19  2007 ,reservations
    -rwxr-xr-x    1 root     root         1344 Jan 19  2007 ShelfiskShelf14
    -rwxr-xr-x    1 root     root     134784000 Sep 19 02:20 v0.16:NETAPP__:VD-100MB-FZ-520_:72355100:259200
    -rwxr-xr-x    1 root     root     134784000 Sep 19 02:20 v0.17:NETAPP__:VD-100MB-FZ-520_:72355101:259200
    -rwxr-xr-x    1 root     root     134784000 Sep 19 02:20 v0.18:NETAPP__:VD-100MB-FZ-520_:72355102:259200
    -rw-r–r–    1 root     root     134784000 Jan 19  2007 v0.19:NETAPP__:VD-100MB-FZ-520_:89570503:259200
    -rw-r–r–    1 root     root     134784000 Jan 19  2007 v0.20:NETAPP__:VD-100MB-FZ-520_:89570504:259200

    v7-2-1P1> disk simpull v4.16
    PANIC: aggr aggr0: raid volfsm, fatal disk error in RAID group with no parity disk. raid type raid0
    Group name plex0/rg0 state NORMAL 1 disk failed in the group.
    Disk v4.16 Shelf 1 Bay 0 [NETAPP   VD-100MB-FZ-520  0042] S/N [72355100] error disk does not exist.
    version: Data ONTAP Release 7.2.1P1: Thu Jan  4 02:52:47 PST 2007 (IBM)
    cc flags: L
    [root@DOT root]#

    系统崩溃了,看来这个raid0是真的.
    [root@DOT ,disks]# ls
    ,pulled                                          v0.29:NETAPP__:VD-100MB-FZ-520_:89570513:259200
    ,reservations                                    v0.32:NETAPP__:VD-100MB-FZ-520_:89570514:259200
    ShelfiskShelf14                                v0.33:NETAPP__:VD-100MB-FZ-520_:89570515:259200
    v0.17:NETAPP__:VD-100MB-FZ-520_:72355101:259200  v0.34:NETAPP__:VD-100MB-FZ-520_:89570516:259200
    v0.18:NETAPP__:VD-100MB-FZ-520_:72355102:259200  v0.35:NETAPP__:VD-100MB-FZ-520_:89570517:259200
    v0.19:NETAPP__:VD-100MB-FZ-520_:89570503:259200  v0.36:NETAPP__:VD-100MB-FZ-520_:89570518:259200

    模拟拔出去的盘在系统中被改了名字而已.
    [root@DOT ,disks]# mv ,pulled  v0.16:NETAPP__:VD-100MB-FZ-520_:72355100:259200
    [root@DOT ,disks]# /sim/runsim.sh
    runsim.sh script version Script version 19 (24/Nov/2005)
    This session is logged in /sim/sessionlogs/log

    PANIC: vha_disk_add_device: could not open file ,disks/v0.16:NETAPP__:VD-100MB-FZ-520_:72355100:259200 error (-2)
    version: Data ONTAP Release 7.2.1P1: Thu Jan  4 02:52:47 PST 2007 (IBM)
    cc flags: L
    dumpcore: vha_disk_add_device: could not open file ,disks/v0.16:NETAPP__:VD-100MB-FZ-520_:72355100:259200 error (-2)
    [root@DOT ,disks]#

    还好,仔细看了下,pull是个目录,应该是把这个下面的硬盘文件移动回去就对了.

    [root@DOT ,disks]# mv v0.16:NETAPP__:VD-100MB-FZ-520_:72355100:259200 ,pulled
    [root@DOT ,disks]# cd ,pulled
    [root@DOT ,pulled]# ls
    v0.16:NETAPP__:VD-100MB-FZ-520_:72355100:259200
    [root@DOT ,pulled]# mv v0.16\:NETAPP__\:VD-100MB-FZ-520_\:72355100\:259200  ../
    [root@DOT ,pulled]# /sim/runsim.sh
    runsim.sh script version Script version 19 (24/Nov/2005)
    This session is logged in /sim/sessionlogs/log

    Data ONTAP Release 7.2.1P1: Thu Jan  4 02:52:47 PST 2007 (IBM)
    Copyright (c) 1992-2006 Network Appliance, Inc.
    Starting boot on Fri Sep 19 06:31:17 GMT 2008

     

    NDMP

     

    Network Data Management Protocol (NDMP) is an open standard for centralized control of

    enterprise-wide data management. NDMP enables backup software vendors to provide support

    for Network Appliance storage systems without having to port client code.

    An NDMP-compliant solution separates the flow of backup/restore control information from the

    flow of data to and from backup media. These solutions invoke Data ONTAP software’s native

    dump and restore to back up data from and restore data to a NetApp storage system.

    NDMP also provides low-level control of tape devices and medium changers.

     

    Accessing these data protection services through backup applications that support NDMP offers

    a number of advantages:

    ? NDMP backup applications provide sophisticated scheduling of data protection

    operations across multiple storage systems. They also provide media management and

    tape inventory management services to eliminate or minimize manual tape handling

    during data protection operations.

    ? NDMP backup applications support data cataloging services that simplify the process of

    locating specific recovery data. Direct Access Recovery (DAR) optimizes the access of

    specific data from large backup tape sets.

    ? NDMP supports multiple topology configurations, allowing efficient sharing of

    secondary storage (tape library) resources through the use of three-way network data

    connections.

     

    NDMP-based solutions can centrally manage and control backup and recovery of highly distributed data

    while minimizing network traffic. These products can direct a NetApp storage system to back itself up to a

    locally attached tape drive, without sending the backup data over the network. NDMP-based solutions are

    designed to assure data protection and efficient restoration in the event of data loss and include many

    control and management features not available with a NetApp storage system’s native dump and restore

    commands, such as discovery, configuration, scheduling, media management, tape library control, and

    user interface.

    In 1996, Network Appliance partnered with Intelliguard, to create NDMP. The two companies have

    promoted its standardization in the industry. Key backup vendors and their NDMP-based solutions are

    also listed in the above NDMP compatibility Matrix. Refer to the online documentation on the NOW site to

    obtain a complete list of 3rd party vendor NDMP backup application and software version.

    NDMP-based third-party solutions provide:

    ? Central management and control of highly distributed data

    ? Local backup of NetApp storage systems without sending data over the network

    ? Control of robotics in tape libraries

    ? Data protection in a mixed server environment of UNIX, Windows NT, and NetApp

    storage systems

    ? Investment protection in established backup strategies

     

    Network Appliance delivers both certified Fibre Channel Fabric Tape SAN backup solutions and

    Gigabit Ethernet Tape SAN solutions. These solutions are made possible through our joint

    partnerships with industry leaders in the fields of tape automation, fabric switches, and backup

    software. They offer significant benefits for enterprise customers over tape devices attached

    directly to NetApp storage systems via SCSI. Specifically, both solutions offer the following

    benefits:

    ? Tape sharing and amortization of tape resources

    ? Extended distances from data to centralized tape backup libraries

    ? Minimized impact from backups on servers on the network

    ? Tape drive hot-swapping

    ? Dynamic tape configuration changes without shutting down the NetApp storage system

    Gigabit Ethernet Channel Tape SAN Solutions

    The Gigabit Ethernet Tape SAN configurations allow multiple Network Appliance storage

    systems to concurrently transfer data over Gigabit Ethernet to one or more tape libraries that

    support NDMP. This architecture allows each drive inside the tape library to be seen as a shared

    resource and as an NDMP server. One clear advantage of this configuration is the demonstrated

    interoperability of Ethernet-based components.

     

    NDMP client (DMA)

     

    – Backup application is the NDMP client

    – Also referred to as the DMA (Data Management

    Application)

    – NDMP clients submit request to NDMP servers, and

    receive replies and status back from the NDMP

    server

     

    后面找NBU来做个试验

     

    NDMP server

     

    – A process or service that runs on the NetApp

    storage appliance

    – Processes requests from NDMP clients, and returns

    reply and status information back to the NDMP

    client

     

    host2>

    host2>

    host2> options ndmp

    ndmpd.access all

    ndmpd.authtype challenge

    ndmpd.connectlog.enabled off

    ndmpd.enable off

    ndmpd.ignore_ctime.enabled off

    ndmpd.offset_map.enable on

    ndmpd.password_length 16

    ndmpd.preferred_interface disable (value might be overwritten in takeover)

    ndmpd.tcpnodelay.enable off

    host2>

    NDMP Tape Backup Topologies

    Storage system to local tape (Direct attached)
    Storage system to network attached
     tape library
    Storage system to storage system to
     tape
    Storage system to server to tape
    Server to storage system to tape

    这样看来功能很强大哦

     

    host1*> ndmpcopy -da root:root123 host1:/vol/vol0/ndmptest 192.168.222.200:/vol/vol0/
    Ndmpcopy: Starting copy [ 7 ] …
    Ndmpcopy: host1: Notify: Connection established
    Ndmpcopy: 192.168.222.200: Notify: Connection established
    Ndmpcopy: host1: Connect: Authentication successful
    Ndmpcopy: 192.168.222.200: Connect: Authentication successful
    Ndmpcopy: Failed to start dump on source
    Ndmpcopy: Done

     

    Note: 连接成功了,发现Ubuntu SecureCRT对汉字显示乱码。google了下,好像说ubuntu 2.6 kernel的ssh默认编码是utf8,而Ubuntu SecureCRT的默认显示编码好像不是utf8,那么只要把Ubuntu SecureCRT的显示编码修改成utf8就可以正确显示了。在Ubuntu SecureCRT中,选择option(选项)->session option(会话)->apperance(外观)里把字符编码改成utf8。

2 Thoughts on “Netapp test cases

  1. Wow Your site is of the chain

  2. Your post is like a walk in the park! You’re an inspiration to me and i’m sure many others!

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Post Navigation