Wednesday, May 8, 2013

Failed connect to mongod with the following java exception: java.io.IOException - message: couldn't connect to [host:port] bc:java.net.SocketException: Too m any open files





15:46:04,067 WARNING [com.mongodb.ReplicaSetStatus] (ReplicaSetStatus:Updater) Server seen down: hostname.somecompany.com/172.20.X.X:27021 
- java.io.IOException - message: couldn't connect to [hostname.somecompany.com/172.20.X.X:27021] bc:java.net.SocketException: Too many open files


This issue was caused by ulimit setting in mongod user is not correct. 

To fix it. 
1. Modify ulimit according to installation spec for correct version. 
sudo vi /etc/security/limits.conf
...
#add for mongod group
@mongod         soft    nofile          65000
@mongod         soft    nproc           65000
@mongod         hard    nofile          65000
@mongod         hard    nproc           65000
# 


2. Restart mongod and mongos
sudo service mongod restart


Thursday, May 2, 2013

cell_offload_processing parameter caused high CPU usage for No Exadata Oracle

Encountered issue with cell_offload_processing parameter in 11.2.0.2. This parameter is set to TRUE by default and caused high CPU usage. According to metalink note "High 'ksv master wait' And 'ASM File Metadata Operation' Waits In Non-Exadata 11g [ID 1308282.1]". This was caused by "Event 'KSV master wait' indicates the process on the RDBMS side is waiting for a reply from a process on the ASM side." 

To resolve this issue do "alter system set cell_offload_processing = false;" 

Wednesday, May 1, 2013

MTU setting in Oracle database appliance


In Oracle database appliance. Interconnect uses jumbo frame, all others NIC use regular 1500 frame. 

There is a very interesting testing article in Oracle blog regarding use jumbo frame to improve proceeding speed as well as reduce server load. "Jumbo Frames for RAC Interconnect by Rene Kundersma".

This is the beauty of packaged product. There is no worry about the none standard frame causing compatibility issues. 

ifconfig -a                                        
bond0     UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1           
...                                                                                                                                                       
bond1     UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1           
                                                                              
bond2     UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1           
                                                                              
eth0      Link encap:Ethernet  HWaddr 00:21:28:F9:19:3E
          inet addr:192.168.16.25  Bcast:192.168.16.255  Mask:255.255.255.0
          inet6 addr: fe80::221:28ff:fef9:193e/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:83864468 errors:0 dropped:0 overruns:0 frame:0
          TX packets:88488696 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:100590633610 (93.6 GiB)  TX bytes:109000919319 (101.5 GiB)
          Interrupt:28 Memory:dee80000-deea0000
                       
                                                                              
eth0:1    Link encap:Ethernet  HWaddr 00:21:28:F9:19:3E                       
          inet addr:169.254.74.231  Bcast:169.254.127.255  Mask:255.255.128.0 
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1                  
          Interrupt:28 Memory:dee80000-deea0000                               
                                                                              
eth1      Link encap:Ethernet  HWaddr 00:21:28:F9:19:3F                       
          inet addr:192.168.17.25  Bcast:192.168.17.255  Mask:255.255.255.0   
          inet6 addr: fe80::221:28ff:fef9:193f/64 Scope:Link                  
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1                  
          RX packets:72255555 errors:0 dropped:0 overruns:0 frame:0           
          TX packets:76183594 errors:0 dropped:0 overruns:0 carrier:0         
          collisions:0 txqueuelen:1000                                        
          RX bytes:55355355888 (51.5 GiB)  TX bytes:78343835583 (72.9 GiB)    
          Interrupt:40 Memory:deee0000-def00000                               
                                                                              
eth1:1    Link encap:Ethernet  HWaddr 00:21:28:F9:19:3F                       
          inet addr:169.254.211.232  Bcast:169.254.255.255  Mask:255.255.128.0
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1                  
          Interrupt:40 Memory:deee0000-def00000                               
                                                                              
eth2      
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1            
          
                                                                              
eth3      UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1            
                                                                              
eth4      UP BROADCAST SLAVE MULTICAST  MTU:1500  Metric:1                    
                                                                              
eth5      UP BROADCAST SLAVE MULTICAST  MTU:1500  Metric:1                    
                                                                              
eth6      UP BROADCAST SLAVE MULTICAST  MTU:1500  Metric:1                    
                                                                              
eth7      UP BROADCAST SLAVE MULTICAST  MTU:1500  Metric:1                    
                                                                                        
eth8      UP BROADCAST SLAVE MULTICAST  MTU:1500  Metric:1                             
                                                                              
eth9      UP BROADCAST SLAVE MULTICAST  MTU:1500  Metric:1                    
                                                                                        
lo        Link encap:Local Loopback                                           
          inet addr:127.0.0.1  Mask:255.0.0.0                                 
          inet6 addr: ::1/128 Scope:Host                                      
          UP LOOPBACK RUNNING  MTU:16436  Metric:1                            
          RX packets:196742468 errors:0 dropped:0 overruns:0 frame:0          
          TX packets:196742468 errors:0 dropped:0 overruns:0 carrier:0        
          collisions:0 txqueuelen:0                                           
          RX bytes:87192154769 (81.2 GiB)  TX bytes:87192154769 (81.2 GiB)    
                                                                              
xbond0    Link encap:Ethernet  HWaddr 90:E2:BA:1A:A7:58                       
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1           
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0                  
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0                
          collisions:0 txqueuelen:0                                           
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)      

Tuesday, April 30, 2013

Oracle database appliance 500G internal disk



Checked our ODA box today and found there is 251G free in the internal disk waiting to be allocated. Sweet.

[root@n1 ~]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/md1
  VG Name               VolGroupSys
  PV Size               465.66 GB / not usable 3.44 MB
  Allocatable           yes
  PE Size (KByte)       32768
  Total PE              14901
  Free PE               8053
  Allocated PE          6848
  PV UUID               qta79y-SYoP-T24F-aNoq-aTpS-eG1z-O443E4

[root@n1 ~]# lvdisplay
  --- Logical volume ---
  LV Name                /dev/VolGroupSys/LogVolRoot
  VG Name                VolGroupSys
  LV UUID                jLUfrQ-EbOg-xae0-ihaQ-2sSM-0qsu-dcKXH8
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                30.00 GB
  Current LE             960
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

  --- Logical volume ---
  LV Name                /dev/VolGroupSys/LogVolU01
  VG Name                VolGroupSys
  LV UUID                fqBmOA-SEP5-qziV-f98F-bJdw-jLdW-NOtTYA
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                100.00 GB
  Current LE             3200
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1

  --- Logical volume ---
  LV Name                /dev/VolGroupSys/LogVolOpt
  VG Name                VolGroupSys
  LV UUID                w7zOfV-MVDh-l44f-9S5V-vwzl-FW5P-zoQKpY
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                60.00 GB
  Current LE             1920
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

  --- Logical volume ---
  LV Name                /dev/VolGroupSys/LogVolSwap
  VG Name                VolGroupSys
  LV UUID                hSnWvz-HRyf-cycE-MMdS-3jPd-RXUr-hMFUPQ
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                24.00 GB
  Current LE             768
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:3

Thursday, April 25, 2013

Oracle data appliance validate failure on network bond2 does not have salve


Got the following error :


/opt/oracle/oak/bin/oakcli validate -a -f /tmp/validateAll.t 


...

ERROR: Bond interface bond2 has 0 slave interfaces, expected 2 interface
WARNING: Bond interface bond2 has the following current status:down
RESULT: Bond interface bond2 is down configured in mode:fault-tolerance (active-backup) with current active interface as None
...

Not sure what happened. I made some change during a testing to bring up bond2 with IP assign to it. Only /etc/sysconfig/network-scirpts/ifcfg-bond2 got changed with IP/Subnet/Mask information. ifcfg-eth6 and eth7 were not touched and associating with bond2. 

After checking, found the eth6 and eth7 are down. which is the reason I got this error. 

ifconfig -a

...

bond2     Link encap:Ethernet  HWaddr 00:00:00:00:00:00
          BROADCAST MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)


...

eth6      Link encap:Ethernet  HWaddr A0:36:9F:08:E3:9F
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Interrupt:25 Memory:df2a0000-df2c0000

eth7      Link encap:Ethernet  HWaddr A0:36:9F:08:E3:9E
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:8512319 errors:0 dropped:0 overruns:0 frame:0
          TX packets:867912785 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1389865582 (1.2 GiB)  TX bytes:388587192156 (361.9 GiB)
          Interrupt:26 Memory:df2e0000-df300000


Bring up eth6 and eth7 


ifdown eth6
ifdown eth7

ifup eth6
ifup eth7
ifconfig -a
...
bond2     Link encap:Ethernet  HWaddr A0:36:9F:08:E3:9F
          BROADCAST MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:8512319 errors:0 dropped:0 overruns:0 frame:0
          TX packets:867912785 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1389865582 (1.2 GiB)  TX bytes:388587192156 (361.9 GiB)

...
eth6      Link encap:Ethernet  HWaddr A0:36:9F:08:E3:9F
          UP BROADCAST SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Interrupt:25 Memory:df2a0000-df2c0000

eth7      Link encap:Ethernet  HWaddr A0:36:9F:08:E3:9F
          UP BROADCAST SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:8512319 errors:0 dropped:0 overruns:0 frame:0
          TX packets:867912785 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1389865582 (1.2 GiB)  TX bytes:388587192156 (361.9 GiB)
          Interrupt:26 Memory:df2e0000-df300000


Validate agin. with oakcli. All good.
...

WARNING: Bond interface bond2 has the following current status:down
RESULT: Bond interface bond2 is down configured in mode:fault-tolerance (active-backup) with current active interface as None
                Slave1 interface is eth6 with status:down Link fail count=0 Maccaddr:a0:36:9f:08:e3:9f
                Slave2 interface is eth7 with status:down Link fail count=0 Maccaddr:a0:36:9f:08:e3:9e
...

Friday, April 12, 2013

/etc/resolv.conf got overwritten in Oracle EL6


To resolve this problem. There are two options.

1. Use network manager interface to change the interface setting.
System-> Perferences-> Network Connections -> Wired tab -> [InterfaceName]-> Edit-> IPV4 Settings Tab->
Change DNS Servers
Change search domains

2. Use the the following manual change
In the file: /etc/sysconfig/network-scripts/ifcfg-<iface>

You need to add all of your specific resolv.conf entries, such as:

DNS1="216.239.32.10"
DNS2="216.83.130.2"
DNS3="216.83.130.7"
DOMAIN="mydomain.com"
SEARCH="mydomain.com. yourdomain.com. otherdomain.com.

Wednesday, April 10, 2013

Steps to setup apache zookeeper

Setup a 2 node zookeeper cluster

1. Download zookeeper-3.4.5.tar.gz. tar xvf this file. 

2. Go to zookeeper-3.4.5/conf directory

vi zoo.cfg

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
# Change to your repository location.
dataDir=/u01/hadoop/zookeeperCluster
# the port at which the clients will connect
# Changed from default port 2181, My port is used other program
clientPort=12181
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
#My 12888 and 13888 port are used other program
server.1=host1:12888:13888
server.2=host2:12888:13888

3. Depend on your log4j.properties setting. Find the logfile location. 

4. Edit myid in the dataDir (dataDir=/u01/hadoop/zookeeperCluster)
In host1 vi myid put "1" in it.
In host2 vi myid put "2" in it.

5. Start server 
host1 :
cd ${zooKeeperHome}
 ./bin/zkServer.sh start

host2:
cd ${zooKeeperHome}
 ./bin/zkServer.sh start

6. Test client connection 
 bin/zkCli.sh -server host1:12181
ls

[zk: host1:12181(CONNECTED) 0] ls /
[zookeeper]