Home » RDBMS Server » Server Administration » Setting up ASM using ASMLib On Native Linux Multipath Mapper (RHEL5 )
| Setting up ASM using ASMLib On Native Linux Multipath Mapper [message #481967] |
Mon, 08 November 2010 12:34  |
asangapradeep
Messages: 128
Registered: October 2005
Location: UK
|
Senior Member |
|
|
Hi,
I've been trying to setup ASM disks for two node cluster using ASMLib. It's a multipath environment Quote:multipath -ll
mpath2 (36842b2b000528f8d0000029b4ccead44) dm-1 DELL,MD32xx
[size=280M][features=3 queue_if_no_path pg_init_retries 50][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=100][active]
\_ 1:0:0:1 sdc 8:32 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 1:0:1:1 sdl 8:176 [active][ghost]
mpath1 (36842b2b0005524f8000002a24ccead18) dm-0 DELL,MD32xx
[size=280M][features=3 queue_if_no_path pg_init_retries 50][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=100][active]
\_ 1:0:1:0 sdk 8:160 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 1:0:0:0 sdb 8:16 [active][ghost]
mpath8 (36842b2b000528f8d000002a14cceae08) dm-7 DELL,MD32xx
[size=418G][features=3 queue_if_no_path pg_init_retries 50][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=100][active]
\_ 1:0:0:7 sdi 8:128 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 1:0:1:7 sdr 65:16 [active][ghost]
mpath7 (36842b2b0005524f8000002a94cceadda) dm-6 DELL,MD32xx
[size=417G][features=3 queue_if_no_path pg_init_retries 50][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=100][active]
\_ 1:0:1:6 sdq 65:0 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 1:0:0:6 sdh 8:112 [active][ghost]
mpath6 (36842b2b000528f8d0000029f4cceadad) dm-5 DELL,MD32xx
[size=100M][features=3 queue_if_no_path pg_init_retries 50][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=100][active]
\_ 1:0:0:5 sdg 8:96 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 1:0:1:5 sdp 8:240 [active][ghost]
mpath5 (36842b2b0005524f8000002a74ccead87) dm-4 DELL,MD32xx
[size=280M][features=3 queue_if_no_path pg_init_retries 50][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=100][active]
\_ 1:0:1:4 sdo 8:224 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 1:0:0:4 sdf 8:80 [active][ghost]
mpath4 (36842b2b000528f8d0000029d4ccead63) dm-3 DELL,MD32xx
[size=280M][features=3 queue_if_no_path pg_init_retries 50][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=100][active]
\_ 1:0:0:3 sde 8:64 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 1:0:1:3 sdn 8:208 [active][ghost]
mpath3 (36842b2b0005524f8000002a54ccead3e) dm-2 DELL,MD32xx
[size=280M][features=3 queue_if_no_path pg_init_retries 50][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=100][active]
\_ 1:0:1:2 sdm 8:192 [active][ready]
\_ round-robin 0 [prio=0][enabled]
I've been following Metalink note 602952.1, 309815.1 and 605828.1
metalink note 602952.1 states Quote:Note: Keep in mind for ASMLIB disks you will need to use the /dev/dm-* disks instead of the /dev/mapper/mpath* devices since they are present in the partition table ==)> /proc/partitions and ASMLIB will look at that file during the disk discovering phase. If you do not use the disks located at /dev/dm-* then ASM will not be able to discover them. and to create ASM disks with Quote:/etc/init.d/oracleasm createdisk DISK1 /dev/dm-<disk><partition> Also further stating Quote:NOTE: When scanning, only the device names known by the kernel are scanned. With device-mapper, the kernel sees the devices as /dev/dm-XX. The /dev/mapper/XXX names are created by udev for human readability. Any configuration of ORACLEASM_SCANORDER or ORACLEASM_SCANEXCLUDE must use the dm prefix.
My environment is RHEL5 and towards the end of the note it says Quote:Note: On release RH 5 the /dev/dm-N devices are not persistent, starting, these devices are no longer created by udev. So release RedHat 5 you can create the ASMLIB disks on the /dev/mpath/<partition> or /dev/mapper/<partition>
Example:
# /etc/init.d/oracleasm createdisk VOL1 /dev/mpath/sata01p1
Or
# /etc/init.d/oracleasm createdisk VOL1 /dev/mapper/mpath2p1
Therefore I created a ASM disk using /dev/mapper path and it is only visible on the node it was created. Even after scandisks on other node it is still not visible when listdisks/oracleasm-discover
I've also tried creating the asm disks with dm-12 option and the same, disk get created but not visible on the other node.
Moreover if I restart the oracleasm restart the disk disappears from the node that it was visible Quote:[root@db2 ~]# /etc/init.d/oracleasm createdisk vol1 /dev/mapper/mpath6p1
Marking disk "vol1" as an ASM disk: [ OK ]
[root@db2 ~]# ls -l /dev/oracleasm/disks/*
brw-rw---- 1 oracle dba 253, 12 Nov 8 10:22 /dev/oracleasm/disks/VOL1
[root@db2 ~]# /etc/init.d/oracleasm scandisks
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@db2 ~]# /etc/init.d/oracleasm listdisks
VOL1
[root@db2 ~]# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks: [ OK ]
Shutting down the Oracle ASMLib driver: [ OK ]
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@db2 ~]# /etc/init.d/oracleasm scandisks
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@db2 ~]# /etc/init.d/oracleasm listdisks
[root@db2 ~]# ls -l /dev/oracleasm/disks/*
ls: /dev/oracleasm/disks/*: No such file or directory
But disk lable is there and could be seen from both nodes on the same partition Quote:head /dev/mapper/mpath6p1
VOL1ORCLDISKVOL1
My suspicion is on multipath configuration. I've tried both (not at the same time) combinations given below.
Quote:ORACLEASM_SCANORDER="dm"
ORACLEASM_SCANEXCLUDE="mpath mapper sd sda sdb"
and Quote:ORACLEASM_SCANORDER="mapper"
ORACLEASM_SCANEXCLUDE="mpath dm sd sda sdb" But still the problem is there, (disks disappears when asm is restarted and never visible on the "other" node).
I'm obviously missing something fundamental here, but for some reason I can't spot it. So any help, advice, pointers is appriciated.
Thank you.
|
|
|
|
| Re: Setting up ASM using ASMLib On Native Linux Multipath Mapper [message #481968 is a reply to message #481967] |
Mon, 08 November 2010 13:22  |
mkounalis
Messages: 147
Registered: October 2009
Location: Dallas, TX
|
Senior Member |
|
|
Can you please define this --> "disks disappears when asm is restarted and never visible on the "other" node"?? How do you see the disk 'dissapear'. Does it disappear from the operating system, or does it become a disk that ASM can't use? If you start Oracle services on the other node, do the disks then 'dissapear' on the other? i.e. - does it matter what node you start on - only the first one has access??
What SAN are you using?
|
|
|
|
| Re: Setting up ASM using ASMLib On Native Linux Multipath Mapper [message #481969 is a reply to message #481968] |
Mon, 08 November 2010 14:04 |
asangapradeep
Messages: 128
Registered: October 2005
Location: UK
|
Senior Member |
|
|
Hi,
disks disappears when asm is restarted :- I've put the output of this on the original post as well.
Quote: ls -l /dev/oracleasm/disks/*
brw-rw---- 1 oracle dba 253, 12 Nov 8 10:22 /dev/oracleasm/disks/VOL1 <--- ASM disk present
/etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks: [ OK ]
Shutting down the Oracle ASMLib driver: [ OK ]
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@db2 ~]# /etc/init.d/oracleasm scandisks
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@db2 ~]# /etc/init.d/oracleasm listdisks
[root@db2 ~]# ls -l /dev/oracleasm/disks/*
ls: /dev/oracleasm/disks/*: No such file or directory <-- no longer there
never visible on the "other" node" :- if I run the createdisk statement on db1 then listdisks will only show that disks if I run it on db1, running listdisks on db2 will not show it (db1 and db2 are hostnames). If I create another disk on db2 then that one will only be shown on db2 not on db1. ( even after running scandisks)
Does it disappear from the operating system :- No
does it become a disk that ASM can't use :- It is the ASM disk that disappear.
does it matter what node you start on - only the first one has access :- No, only the node you created the disk will list it.
It's a Dell SAN, I can paste the exact number bit later.
Hope this helps.
thank you.
|
|
|
|
| Re: Setting up ASM using ASMLib On Native Linux Multipath Mapper [message #481977 is a reply to message #481969] |
Mon, 08 November 2010 15:41 |
mkounalis
Messages: 147
Registered: October 2009
Location: Dallas, TX
|
Senior Member |
|
|
I think this might be required reading for you --> http://www.dell.com/downloads/global/products/pvaul/en/powervault-md32x0-md32x0i-linux-dm-installation-en.pdf
The document mentions a setting that is required for Oracle RAC, suggesting that Dell as at least tested Oracle on this SAN. I would give this a read - which also describes in detail how to configure linux native dm/multipathing with the 3200 PowerVault, as well as tell you the parameter change in your multipath.conf. I have to say that I am a little puzzled - as you are - as to why this issue is happening. When in doubt, go back to the basics. Use 'dd' and dump a text file to the device just like it's a file destination and see if you can read the file on the other node. Then do the same thing the other direction. This will ensure that you are able to both read and write the same physical volumes on both nodes.
Post more when you get a chance. Good luck!
Mark
|
|
|
|
|
|
| Re: Setting up ASM using ASMLib On Native Linux Multipath Mapper [message #482062 is a reply to message #481967] |
Tue, 09 November 2010 05:24 |
asangapradeep
Messages: 128
Registered: October 2005
Location: UK
|
Senior Member |
|
|
Hi,
Thanks for the Dell pdf and your pointers. Most of the things said on it under RHEL 5.5 were setup by dell with the exception of failback parameter which the pdf said Quote:please set "failback" parameter, in MD32xx/I device section to "manual" in /etc/multipath.conf So I changed that paramter and the SAN is a Dell MD3200x (so you're pdf does apply to this model).
I've done what you said with dd and could read the text from both nodes no problem. It's not like on each node the mapper files are pointing to different partitions. Like I said on my first post when I examin the head of the partition I could see the same ASM lable from both nodes. So disks do get marked as ASM disks when I create them. But the subsequent discovery of them is the problem.
I don't think you need to change the permission of the underlying physical disks when you use ASMLibs but I did tried that as well. Still no change in the behavior.
Found this multipath doc from oracle which says Quote:NOTE: When scanning, only the device names known by the kernel are scanned. With device-mapper, the kernel sees the devices as /dev/dm-XX. The /dev/mapper/XXX names are created by udev for human readability. Any configuration of ORACLEASM_SCANORDER or ORACLEASM_SCANEXCLUDE must use the dm prefix.
So I've changed my /etc/sysconfig/oracleasm to Quote:# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER="dm"
# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
ORACLEASM_SCANEXCLUDE="sd sda sdb sdc sdd sde sdf sdg sdh sdi sdj sdk sdl sdm sdn sdo sdp sdq sdr sds" And going to stick with this. But I'll be creating ASM disks using the mapper files as the metalink note 602952.1 suggests.
So now the problem is down to why the ASM disks aren't rediscovered after the creation on the same node and why they aren't discovered in the "other" node at all!
|
|
|
|
|
|
|
|
| Re: Setting up ASM using ASMLib On Native Linux Multipath Mapper [message #482189 is a reply to message #482113] |
Wed, 10 November 2010 04:48 |
asangapradeep
Messages: 128
Registered: October 2005
Location: UK
|
Senior Member |
|
|
Hi,
Well I did a strace of the scandisks. That's the first thing I did when things weren't working as expected but I was looking for wrong thing. Searched for devices /dev/dm and /dev/mapper but it didn't turn out anything.
Then went line by line and found the following 6575 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
6575 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
6575 stat("/etc/sysconfig/oracleasm-_dev_oracleasm", {st_mode=S_IFREG|0644, st_size=791, ...}) = 0
6575 stat("/etc/sysconfig/oracleasm", {st_mode=S_IFREG|0644, st_size=854, ...}) = 0
6575 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
6575 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
6575 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
6575 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
6575 stat("/etc/sysconfig/oracleasm-_dev_oracleasm", {st_mode=S_IFREG|0644, st_size=791, ...}) = 0
6575 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
6575 stat("/etc/sysconfig/oracleasm-_dev_oracleasm", {st_mode=S_IFREG|0644, st_size=791, ...}) = 0
6575 access("/etc/sysconfig/oracleasm-_dev_oracleasm", X_OK) = -1 EACCES (Permission denied)
6575 access("/etc/sysconfig/oracleasm-_dev_oracleasm", R_OK) = 0
6575 open("/etc/sysconfig/oracleasm-_dev_oracleasm", O_RDONLY) = 4
6575 fstat(4, {st_mode=S_IFREG|0644, st_size=791, ...}) = 0
6575 read(4, "#\n# This is a configuration file"..., 791) = 791
Permission denied seems interesting so when into the file /etc/sysconfig/oracleasm-_dev_oracleasm and saw that it didn't had any of the changes I made, so opened /etc/sysconfig/oracleasm which had the changes and came to the conclusion these are two different files.
|
|
|
|
Goto Forum:
Current Time: Wed May 15 15:09:11 CDT 2024
|
] 
Blog
Search
Help
Members
Register
Login
Home
