(selinux) context is important!

May 8th, 2013

Just found this in the .bashrc of a rhev hypervisor. Good to know…

# aliases used for the temporary
function mod_vi() {
/bin/vi $@
restorecon -v $@ >/dev/null 2>&1
}
alias vi="mod_vi"

sshd…elay? Troubleshoot with real weapons!

May 7th, 2013

When my first ssh connection to a new server had a 30secs delay, I thought I was cool enough to speed it up with
#ssh -o GSSAPIAuthentication=off
but this time it didn’t work.

I didn’t waste time and simply strace and connected
#strace -qfp $(pgrep -f /sshd)
...
[pid 15756] socket(PF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 7
[pid 15756] connect(7, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.0.0.254")}, 16) = 0
...
[pid 15756] poll([{fd=7, events=POLLIN}], 1, 5000) = 0 (Timeout)
....

That was something involving DNS and ssh…
# grep -ri dns /etc/ssh/
...
/etc/ssh/sshd_config:#UseDNS no

So you may consider to disable it, as it’s enabled by default.

A 30 second delay should have been considered a DNS issue even without strace, but nobody is perfect and that way worked fine ;)

vconfig off-by-one?

May 7th, 2013

Vlan tagging (aka RFC 802.1q) is a Layer2 protocol enabling more VLAN on a single ethernet connection. It works tagging ethernet frames so that each port accepts only the configured frames.

The vlan id is a 12bit value (from 0 to 4095), and usually network devices use the 4095 value (0xFFF) for the management network.
But the following command gave me an error
# vconfig add eth1 4095

And I discovered that – actually – the last valid VLAN ID is 4094. Here’s a brief discussion on the subject

r(h)e(v)set password

April 24th, 2013

RHEV uses free-ipa services to authenticate users to its portal. If password expired, the administrator can only set an one-time-password via the IPA portal eg. https://freeipa.rhevlab.local/ipa/ui/ and the user must change it *before* logging in.

The user – eg. u01@rhevlab.local – needs only to
#kinit u01;

The kinit executable reads kerberos configuration from /etc/krb5.conf and sets the default realm (eg. rhevlab.local) thus associating the user with the ldap entry

[libdefaults]
default_realm = RHEVLAB.LOCAL
dns_lookup_realm = false
dns_lookup_kdc = false
rdns = false
ticket_lifetime = 24h
forwardable = yes

You can otherwise specify the UPPERCASE domain in your kinit request
#kinit u01@RHEVLAB.LOCAL

While all your *lowercase* requests are doomed to fail ;
#kinit u01@rhevlab.local
kinit: Cannot find KDC for requested realm while getting initial credentials

IO Monitoring: pgrep thread support!

April 24th, 2013

pgrep is a cool tool that shows all pids matching a given pattern, like
# pgrep -fl mysql

You can use it with #top to monitor memory usage and other stuff. Modern tools like #iotop are task/thread based and pgrep is not very useful in this case, because you are not tracking the IO made by subtasks.

We patched pgrep to print out all task/thread id, and we’re glad that our patch was merged in the trunk (available in Fedora 19)!
Just use the -w flag to list all threads (lightweight processes)
# pgrep -flw mysql

You can otherwise take that infos with the following
egrep -rl mysql /proc/[0-9]*/task/[0-9]*/cmdline | cut -d\/ -f 5

Associate private keys and certificates (requests)

April 15th, 2013

openssl can simply extract modulus (aka the PubKey) from certificates and rsa keys with:
openssl rsa -modulus < rsa.key
openss req -modulus < cert.csr

How to manage duplicated UUID on RHEV 3.1

April 12th, 2013

Do you remember the duplicated UID issue on RHEV 3.0? With v3.0 you can register multiple hosts with the same UUID but you can’t perform a live migration.

On RHEV 3.1 you cannot register more than one host with a specific UUID.

This means that you need to change it on each host before registering it to your RHEV Manager.

How to solve this situation? You can simply generate a new UUID by executing this command:

# uuidgen > /etc/vdsm/vdsm.id

then restart the vdsmd service…

# service vdsmd restart

…or reboot your hypervisor.

As you can see it is no longer necessary to edit the file libvirtd.conf on each hypervisor.


The limits inference…

April 5th, 2013

I was enjoying the /etc/security/limits.d/ directory for splitting user configuration limits when something didn’t seem to work.

I added in limits.d/apache.conf the lines

root - nofile 65536
apache - nofile 8192

but the following still returned 1024

# service httpd restart
# grep files /proc/$(pidof httpd | cut -d\ -f1)/limits
1024

While I restarted httpd from the same bash process used to edit the limits.d – in which I had the following

# ulimit -n
1024

apache was ignoring the limits.d statement. This could have happened only if httpd was invoked without a `su` procedure (the one reading limits and setrlimit) – and that was actually the case on Red Hat 6.3.

So when the bash process fork-exec’d to httpd, the ulimit was still 1024 and no step were done to “raise” the limits; moreover after re-logging in and restarting httpd, the new limits were 65536 – the root’s one.

The solution on RHEL6.3 – for me – was to statically put apache ulimits in /etc/sysconfig/httpd.

bashing ipython

March 28th, 2013

iPython is a wonderful tool that avoids continuosly switching from bash to other utilities like bc, perl & co.

One of its limitation is the I/O redirection – at which bash is really good of. As iPython py-shell profile uses /bin/sh by default – thru os.system, I implemented a quick and dirty system replacement that diverts it into bash.

I added the following line here .ipython/profile_pysh/ipython_config.py

import os
def system2(cmd):
  pid = os.fork()
  if pid == 0:
    args = ['/bin/bash', '-c', cmd ]
    return os.execvp("/bin/bash", args)
  else:
    c_pid, status = os.waitpid(pid, 0)
    return status

print("Overriding os.system with bash")
os.system = system2

# or you can simply use subprocess if available
os.system = lambda cmd: subprocess.call(cmd, shell=True,executable='/bin/bash')

While py-shell profile runs commands for every call outside python globals(), other profiles use the `bang` syntax.
Eg. ! ls -l

To work it out too, I just changed the following line in
/usr/lib/python2.7/dist-packages/IPython/utils/_process_common.py
71 p = subprocess.Popen(cmd, shell=True,
72 executable='/bin/bash',
73 stdin=subprocess.PIPE,

oracle doesn’t find its client

March 20th, 2013

If an oracle client isn’t able to resolve client hostname, throws an
ORA-21561: OID generation failed

On log/listener.logĀ I found
20-MAR-2013 16:15:30 * (CONNECT_DATA=(SID=test)(CID=(PROGRAM=perl@foo.example.com)(HOST=foo.example.com)(USER=bar))) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.0.139)(PORT=48363))

The client hostname was managed via DHCP, so the hostname wasn’t in /etc/hosts.

The root cause of the issue was a slow DNS response – so adding the client hostname on the client machine fixed the issue.