Failed connect to mongod with the following java exception: java.io.IOException - message: couldn't connect to [host:port] bc:java.net.SocketException: Too m any open files

15:46:04,067 WARNING [com.mongodb.ReplicaSetStatus] (ReplicaSetStatus:Updater) Server seen down: hostname.somecompany.com/172.20.X.X:27021 

- java.io.IOException - message: couldn't connect to [hostname.somecompany.com/172.20.X.X:27021] bc:java.net.SocketException: Too many open files

This issue was caused by ulimit setting in mongod user is not correct. 

To fix it. 

1. Modify ulimit according to installation spec for correct version. 

sudo vi /etc/security/limits.conf

...

#add for mongod group

@mongod         soft    nofile          65000

@mongod         soft    nproc           65000

@mongod         hard    nofile          65000

@mongod         hard    nproc           65000

# 

2. Restart mongod and mongos

sudo service mongod restart