Oracle database appliance 500G internal disk

Checked our ODA box today and found there is 251G free in the internal disk waiting to be allocated. Sweet.

[root@n1 ~]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/md1
  VG Name               VolGroupSys
  PV Size               465.66 GB / not usable 3.44 MB
  Allocatable           yes
  PE Size (KByte)       32768
  Total PE              14901
  Free PE               8053
  Allocated PE          6848
  PV UUID               qta79y-SYoP-T24F-aNoq-aTpS-eG1z-O443E4

[root@n1 ~]# lvdisplay
  --- Logical volume ---
  LV Name                /dev/VolGroupSys/LogVolRoot
  VG Name                VolGroupSys
  LV UUID                jLUfrQ-EbOg-xae0-ihaQ-2sSM-0qsu-dcKXH8
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                30.00 GB
  Current LE             960
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

  --- Logical volume ---
  LV Name                /dev/VolGroupSys/LogVolU01
  VG Name                VolGroupSys
  LV UUID                fqBmOA-SEP5-qziV-f98F-bJdw-jLdW-NOtTYA
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                100.00 GB
  Current LE             3200
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1

  --- Logical volume ---
  LV Name                /dev/VolGroupSys/LogVolOpt
  VG Name                VolGroupSys
  LV UUID                w7zOfV-MVDh-l44f-9S5V-vwzl-FW5P-zoQKpY
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                60.00 GB
  Current LE             1920
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

  --- Logical volume ---
  LV Name                /dev/VolGroupSys/LogVolSwap
  VG Name                VolGroupSys
  LV UUID                hSnWvz-HRyf-cycE-MMdS-3jPd-RXUr-hMFUPQ
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                24.00 GB
  Current LE             768
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:3

Oracle data appliance validate failure on network bond2 does not have salve

Got the following error :

/opt/oracle/oak/bin/oakcli validate -a -f /tmp/validateAll.t 

...

ERROR: Bond interface bond2 has 0 slave interfaces, expected 2 interface
WARNING: Bond interface bond2 has the following current status:down
RESULT: Bond interface bond2 is down configured in mode:fault-tolerance (active-backup) with current active interface as None

...

Not sure what happened. I made some change during a testing to bring up bond2 with IP assign to it. Only /etc/sysconfig/network-scirpts/ifcfg-bond2 got changed with IP/Subnet/Mask information. ifcfg-eth6 and eth7 were not touched and associating with bond2. 

After checking, found the eth6 and eth7 are down. which is the reason I got this error. 

ifconfig -a

...

bond2     Link encap:Ethernet  HWaddr 00:00:00:00:00:00
          BROADCAST MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

...

eth6      Link encap:Ethernet  HWaddr A0:36:9F:08:E3:9F
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Interrupt:25 Memory:df2a0000-df2c0000

eth7      Link encap:Ethernet  HWaddr A0:36:9F:08:E3:9E
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:8512319 errors:0 dropped:0 overruns:0 frame:0
          TX packets:867912785 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1389865582 (1.2 GiB)  TX bytes:388587192156 (361.9 GiB)
          Interrupt:26 Memory:df2e0000-df300000

Bring up eth6 and eth7 

ifdown eth6

ifdown eth7

ifup eth6

ifup eth7

ifconfig -a

...

bond2     Link encap:Ethernet  HWaddr A0:36:9F:08:E3:9F

          BROADCAST MASTER MULTICAST  MTU:1500  Metric:1

          RX packets:8512319 errors:0 dropped:0 overruns:0 frame:0

          TX packets:867912785 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:0

          RX bytes:1389865582 (1.2 GiB)  TX bytes:388587192156 (361.9 GiB)

...

eth6      Link encap:Ethernet  HWaddr A0:36:9F:08:E3:9F

          UP BROADCAST SLAVE MULTICAST  MTU:1500  Metric:1

          RX packets:0 errors:0 dropped:0 overruns:0 frame:0

          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

          Interrupt:25 Memory:df2a0000-df2c0000

eth7      Link encap:Ethernet  HWaddr A0:36:9F:08:E3:9F

          UP BROADCAST SLAVE MULTICAST  MTU:1500  Metric:1

          RX packets:8512319 errors:0 dropped:0 overruns:0 frame:0

          TX packets:867912785 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:1389865582 (1.2 GiB)  TX bytes:388587192156 (361.9 GiB)

          Interrupt:26 Memory:df2e0000-df300000

Validate agin. with oakcli. All good.
...

WARNING: Bond interface bond2 has the following current status:down
RESULT: Bond interface bond2 is down configured in mode:fault-tolerance (active-backup) with current active interface as None
                Slave1 interface is eth6 with status:down Link fail count=0 Maccaddr:a0:36:9f:08:e3:9f
                Slave2 interface is eth7 with status:down Link fail count=0 Maccaddr:a0:36:9f:08:e3:9e

...

/etc/resolv.conf got overwritten in Oracle EL6

To resolve this problem. There are two options.

1. Use network manager interface to change the interface setting.
System-> Perferences-> Network Connections -> Wired tab -> [InterfaceName]-> Edit-> IPV4 Settings Tab->
Change DNS Servers
Change search domains

2. Use the the following manual change
In the file: /etc/sysconfig/network-scripts/ifcfg-<iface>

You need to add all of your specific resolv.conf entries, such as:

DNS1="216.239.32.10"
DNS2="216.83.130.2"
DNS3="216.83.130.7"
DOMAIN="mydomain.com"
SEARCH="mydomain.com. yourdomain.com. otherdomain.com.

Steps to setup apache zookeeper

Setup a 2 node zookeeper cluster

1. Download zookeeper-3.4.5.tar.gz. tar xvf this file. 

2. Go to zookeeper-3.4.5/conf directory

vi zoo.cfg

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
# Change to your repository location.
dataDir=/u01/hadoop/zookeeperCluster
# the port at which the clients will connect
# Changed from default port 2181, My port is used other program
clientPort=12181
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
#My 12888 and 13888 port are used other program
server.1=host1:12888:13888
server.2=host2:12888:13888

3. Depend on your log4j.properties setting. Find the logfile location. 

4. Edit myid in the dataDir (dataDir=/u01/hadoop/zookeeperCluster)
In host1 vi myid put "1" in it.
In host2 vi myid put "2" in it.

5. Start server 
host1 :
cd ${zooKeeperHome}
 ./bin/zkServer.sh start

host2:
cd ${zooKeeperHome}
 ./bin/zkServer.sh start

6. Test client connection 
 bin/zkCli.sh -server host1:12181
ls

[zk: host1:12181(CONNECTED) 0] ls /
[zookeeper]

flume java.lang.ClassNotFoundException: org.apache.hadoop.io.SequenceFile$CompressionType

To resolve the following error
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.SequenceFile$CompressionType
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)

You need to copy hadoop-core-x.x.x.jar to ${flume_home}/lib directory.

This file can be found under your hadoop home directory. 

e.g. 

hadoop 1.0.4 installed in "/home/hadoop/hadoop-1.0.4"

hadoop-core-1.0.4.jar is in this directory. 

cp /home/hadoop/hadoop-1.0.4/hadoop-core-1.0.4.ja ${flume_home}/lib should be able to resolve this problem. 

setup mongo sharding, a sample configuration and testing sharding behavior.

Terminal 1
mongod --port 27022 --dbpath /config --configsvr

Terminal 2
mongos --configdb localhost:27022 --port 27034 --chunkSize 1

Termail 3
mongod --port 27023 --dbpath /app/mongo/shards/shard0/data --shardsvr

Termail 4
mongod --port 27024 --dbpath /app/monngo/shards/shard1/data -shardsvr

Connect to mongos
mongo localhost:27034


mongos>  db.runCommand({addshard : "localhost:27023",allowLocal: true})
{ "shardAdded" : "shard0000", "ok" : 1 }
mongos> db.runCommand({addshard : "localhost:27024",allowLocal: true})
{ "shardAdded" : "shard0001", "ok" : 1 }

mongos> testdb = db.getSisterDB("testdb");

testdb

mongos> db.runCommand({ enablesharding:"testd"})

{ "ok" : 1 }

mongos> db.runCommand({ enablesharding:"testdb"})

{ "ok" : 1 }

mongos> db.runCommand({ enablesharding:"testdb"})

{ "ok" : 0, "errmsg" : "already enabled" }

mongos> db.runCommand({shardcollection : "testdb.testcollection", key : {testkey : 1}})
{ "collectionsharded" : "testdb.testcollection", "ok" : 1 }

Use the following java to insert 200000+ rows into new db 

package dbHelper;  

import java.util.Random;

import com.mongodb.*;

import java.net.UnknownHostException;

public class InsertTester {

MongoHelper mh; 

public InsertTester (String p_hostname, int p_portnumber , String p_username, String p_password , String p_dbname) {

try {

mh = new MongoHelper(p_hostname,p_portnumber,p_username,p_password,p_dbname);

}

catch ( Exception e ) {

e.printStackTrace();

}

}

public static void main(String[] args) {

// TODO Auto-generated method stub

InsertTester tester=new InsertTester ("cvlqmongo1",27034,null,null,"testdb");

int testKey=0;

Random rd = new Random();

BasicDBObject myobj = null;

for ( int i=0; i < 10000 ; i++ ) {

testKey = rd.nextInt();

myobj=new BasicDBObject("testkey",testKey);

myobj.append("Content", "WhatEver");

   tester.mh.addContents("testcollection", myobj);

   

}

}

}

Row counts 

mongos> db.testcollection.count();

20342

Numbers in shr0 and shr1 are on pair. Almost 10000 on each node. 

Add another shard

Termial 5

mongod --port 27025 --dbpath /app/mongo/shards/shards/shard2 --shardsvr 

mogos localhost:27034

db.runCommand( { addshard : "localhost:27025", allowLocal:true});

mongos> use admin

switched to db admin

mongos> db.runCommand({listShards:1});

{

        "shards" : [

                {

                        "_id" : "shard0000",

                        "host" : "localhost:27023"

                },

                {

                        "_id" : "shard0001",

                        "host" : "localhost:27024"

                },

                {

                        "_id" : "shard0002",

                        "host" : "localhost:27025"

                }

        ],

        "ok" : 1

}

Execute testing program again.  insert 20000+ more rows. Node 3 only has 2 rows populated. 

mongos> use testdb

switched to db testdb

mongos> db.testcollection.count();

40342

Node 1: 

mongo localhost:27023

> use testdb

switched to db testdb

> db.testcollection.count();

19957

Node 2: 

 mongo localhost:27024

> use testdb

switched to db testdb

> db.testcollection.count();

20383

Node 3: 

 mongo localhost:27025

> use testdb

switched to db testdb

>  db.testcollection.count();

2

DYNAMIC_REGISTRATION_LISTENER = OFF cause listener not work correctly

     Several my Oracle databases suddenly has invalid objects due to db link not longer working. The error message is like "ORA-12514: TNS:listener does not currently know of service requested in connect descriptor".

     It turned out, someone implemented  DYNAMIC_REGISTRATION_LISTENER = OFF on my source database without putting descriptions regarding what this listener should be handling. As the result, databases services and instances are no longer registered with default port listener as default. The listener ended up running without doing anything. 

     To fix it. The following description need to be added in order to tell the listener to whom it services. 

SID_LIST_LISTENER =

  (SID_LIST =

    (SID_DESC =

      (GLOBAL_DBNAME= service.company.com)

      (ORACLE_HOME=$ORACLE_HOME)

      (SID_NAME =somesid )

    )

  )

resolv.conf get overwritten in Oracle Enterprise Linux Server 6 EL6

/etc/resolv.conf get overwritten in EL6


1. Use network manager interface to change the interface setting.
System-> Perferences-> Network Connections -> Wired tab -> [InterfaceName]-> Edit-> IPV4 Settings Tab->
Change DNS Servers
Change search domains

2. Use the the following manual change
In the file: /etc/sysconfig/network-scripts/ifcfg-<iface>

You need to add all of your specific resolv.conf entries, such as:

DNS1="216.239.32.10"
DNS2="216.83.130.2"
DNS3="216.83.130.7"
DOMAIN="mydomain.com"
SEARCH="mydomain.com. yourdomain.com. otherdomain.com."

Steps to setup apache sqoop

Copy and untar sqoop-1.99.1-bin-hadoop100.tar.gz

Since default port 8080 is running my mongo Query. The listener ports are changed 

Go to /home/hadoop/sqoop-1.99.1-bin-hadoop100
[list]cd server/conf
vi server.xml
Change tomcat's default port to some other values.
[hadoop@someHost-n2 conf]$ diff server.xml server.xml.old
22c22
< <Server port="18005" shutdown="SHUTDOWN">
---
> <Server port="8005" shutdown="SHUTDOWN">
67c67
<          Define a non-SSL HTTP/1.1 Connector on port 18080
---
>          Define a non-SSL HTTP/1.1 Connector on port 8080
69c69
<     <Connector port="18080" protocol="HTTP/1.1"
---
>     <Connector port="8080" protocol="HTTP/1.1"
75c75
<                port="18080" protocol="HTTP/1.1"
---
>                port="8080" protocol="HTTP/1.1"
90c90
<     <Connector port="18009" protocol="AJP/1.3" redirectPort="8443" />
---
>     <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
[/list]

Change configuration to reflect the correct hadoop installation
vi $SQOOPBASE/server/conf/sqoop.properties
[hadoop@someHost-n2 conf]$ diff sqoop.properties sqoop.properties.old  Change the following.
110,111c110
< #org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/etc/hadoop/conf/
< org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/home/hadoop/hadoop-1.0.4/conf
---
> org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/etc/hadoop/conf/


Start the server
[list]cd /home/hadoop/sqoop-1.99.1-bin-hadoop100/
./bin/sqoop.sh server start
[/list]

Watch the log file, make sure no exception
[list]cd /home/hadoop/sqoop-1.99.1-bin-hadoop100/server/logs
tail -f catalina.2013-04-07.log  <--- File name will be different depend on date.[/list]

Start client
[list][hadoop@someHost-n2 sqoop-1.99.1-bin-hadoop100]$ bin/sqoop.sh client
..
sqoop:000> set server --host someHost-n2 --port 18080
Server is set successfully
[/list]

Show server version 
sqoop:000> show version -s
server version:
  Sqoop 1.99.1 revision Unknown
  Compiled by jarcec on Fri Dec 21 12:11:06 PST 2012

org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.initialize(MapreduceSubmissionEngine.java:93) Error

Setting up sqoop on my testing environment and got this error

"org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.initialize(MapreduceSubmissionEngine.java:93)"

No idea at first, very hard to find this was the cause. But clean $SQOOPBASE/server/logs/localhost.[date].log helps to clarify the problem.

In the newly generated file. I see the following error. It is clear the sqoop apps failed to find mapreduce setting. Checked the $SQOOPBASE/server/conf/sqoop.properties file and found the parameter "org.apache.sqoop.submission.engine.mapreduce.configuration.directory" is set to default."/etc/hadoop/conf". Replaced this with the correct hadoop installation directory then restart server. The error is gone.


Apr 07, 2013 2:43:15 PM org.apache.catalina.core.StandardContext listenerStart
SEVERE: Exception sending context initialized event to listener instance of class org.apache.sqoop.server.ServerInitializer
java.lang.RuntimeException: Failure in server initialization
        at org.apache.sqoop.server.ServerInitializer.contextInitialized(ServerInitializer.java:57)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
        at org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
        at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
        at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
        at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
        at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
        at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
        at org.apache.catalina.core.StandardService.start(StandardService.java:525)
        at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: java.lang.NullPointerException
        at org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.initialize(MapreduceSubmissionEngine.java:93)
        at org.apache.sqoop.framework.FrameworkManager.initialize(FrameworkManager.java:200)
        at org.apache.sqoop.server.ServerInitializer.contextInitialized(ServerInitializer.java:53)
        ... 24 more