Upgrade CentOS 4.8 to 5.x (32bit)

Traditionally, the dist upgrade path that many were familiar with from the RH8/9->Fedora or similarly Fedora dist upgrades, have applied more or less to RHEL/CentOS but with the release of 4.5 and early releases of 5.0 the actual dist upgrade path was messy or nearly impossible. The early versions of 5.0 (up to 5.2) had excessive dependency issues with versions later than 4.4 for straight dist upgrades that would often result in a box blowing up on you or forcing a messy downgrade attempt of 4.5+ to 4.4 to try get things to dist upgrade. With more recent release updates the gap has closed and now dist upgrades on are far more reasonable to complete with little in the way of problems.

If you are currently running a version of RHEL/CentOS earlier than 4.8 (cat /etc/redhat-release) then please do a proper ‘yum update’ and get yourself on 4.8. Although this is intended for CentOS it “should” (read: at own risk) work on RHEL systems as well, in the unfortunate situation that something does blow up please post a comment and I will try to assist.

The first thing we must do is make sure none of our core binaries, libraries or other content is set immutable as this will cause a package to fail on installation. If you are running an earlier version of LES or you use immutable bits on system paths (sbin/bin/share/include/libexec/etc) then you should run the following:

wget http://www.rfxn.com/downloads/disable.les.rpmpkg
sh disable.les.rpmpkg

Once that is done we should go ahead and have a quick run through of cleaning up yum cache, double check that any pending updates are installed and rebuild the rpmdb:

rpm --rebuilddb
yum clean all
yum update

If for some reason the rpm rebuild hangs for more than a few minutes then you may need to manually clear the rpmdb files:

rm -f /var/lib/rpm/__db.00*
rpm --rebuilddb

If you run into any minor dependency issues for packages that are not essential, such as syslinux and lftp then you can either exclude them or better yet remove them. If you are not sure what a package does, then you should query it for description details and make an educated choice (rpm -qi PACKAGE):

rpm -e lftp syslinux mkbootdisk

OR (but not recommended)

yum update --exclude=syslinux --exclude=lftp --exclude=mkboot

At this point you should be able to run a ‘yum update’ command with optional exclude and receive no errors (again, I recommend you remove conflicts items instead of using exclusions).

# yum update –exclude=nagios-plugins
Setting up Update Process
Setting up repositories
Reading repository metadata in from local files
Excluding Packages in global exclude list
Finished
No Packages marked for Update/Obsoletion

Now we are ready to get going, I have put together a small package that contains the needed packages for this upgrade in addition to a few that you might require to resolve dependency conflicts:

wget http://www.rfxn.com/downloads/CentOS-5up.tar.gz
tar xvfz CentOS-5up.tar.gz
cd CentOS-5up

We need to go ahead and setup the centos-release package as follows:

rpm -Uhv centos-release-*

If you see that CentOS-Base.repo was created as /etc/yum.repos.d/CentOS-Base.repo.rpmnew then go ahead and move it into the proper place:

mv /etc/yum.repos.d/CentOS-Base.repo.rpmnew /etc/yum.repos.d/CentOS-Base.repo

Now we are ready to go with kernel changes, this is an important part so pay attention. The key to successful upgrade is that you remove ALL OLD KERNELS as many packages will fail to install during the upgrade if they detect a release 4.x kernel due to minimum kernel version dependency checks. We will start with first installing the new kernel so it preserves grub templating:

rpm -ivh kernel-2.6.18-164.el5.i686.rpm kernel-devel-2.6.18-164.el5.i686.rpm --nodeps

NOTE: release 5.x has smp support integrated into the standard kernel, so no -smp version is required for mp systems

If you are running an older system the chances are you got allot of older kernel packages installed so make sure you get them all out of the way:

rpm -e $(rpm -qa | grep kernel | grep -v 2.6.18 | tr '\n' ' ')

You may end up with a few dependencies coming up such as lm_sensors and net-snmp if the list is fairly small and packages you do not recognize as critical (if unsure always query the package for info ‘rpm -qi PACKAGE’, remember you can reinstall them later):

# rpm -e $(rpm -qa | grep kernel | grep -v 2.6.18)
error: Failed dependencies:
kernel-utils is needed by (installed) lm_sensors-2.8.7-2.40.5.i386

The command the ended up being required on most of my servers to get the kernel packages and related dependencies came out to the following:

rpm -e $(rpm -qa | grep kernel | grep -v 2.6.18 | tr '\n' ' ') lm_sensors net-snmp net-snmp-devel net-snmp-utils

That said and done you should now only have 2 kernel packages installed which are the 2.6.18 release 5.x kernels, DO NOT under any circumstance continue if you still got 2.6.9 release 4.x kernels packages still installed, remove them!

# rpm -qa | grep -i kernel
kernel-2.6.18-164.el5
kernel-devel-2.6.18-164.el5

A cleanup of /etc/grub.conf may be required, though if all went as planned then the rpm command should have done this up for us but review it anyways for good measure. You should find that 2.6.18-164.el5 is the only kernel in the file, if it is not go ahead and clean it by removing all older entries for 2.6.9 kernels.

There is a known bug with python-elementtree package versions which cause yum/rpm to think the release 4.x version is newer than the 5.x version, to get around this without blowing up the entire python installation we need to remove the package from just the rpmdb as follows:

rpm -e --justdb python-elementtree --nodeps

We can now go ahead and use yum to start the upgrade process, this is a dry run and will take a few minutes to compile list of available packages and associated dependency checks. You should carry the exclude options, if any, that you used during the ‘yum update’ process as so to avoid unresolvable dependencies:

yum clean all
yum upgrade --exclude=nagios-plugins

You will end up with a small list of dependency errors, these should be resolved by again evaluating a packages need as a critical system component and either removing it with ‘rpm -e’ or excluding it with ‘–exclude’ (remember to query description with ‘rpm -qi PACKAGE’ if you are unsure what something does). In my case the packages that threw up red flags were stuff I had manually installed over time such as iftop and mrtg in addition to default installed samba, these can all safely be removed or excluded as you prefer (removal always safest to prevent dependency chain issues).

Error: Missing Dependency: libpcap.so.0.8.3 is needed by package iftop
Error: Missing Dependency: perl(Convert::ASN1) is needed by package samba
Error: Missing Dependency: libevent-1.1a.so.1 is needed bypackage nfs-utils
Error: Missing Dependency: perl-Socket6 is needed by package mrtg
Error: Missing Dependency: perl-IO-Socket-INET6 is needed by package mrtg


rpm -e iftop samba nfs-utils mrtg system-config-samba

At this point we should be ready to do a final dry run of with yum and see where we stand on dependencies, rerun the earlier ‘yum upgrade’ while making sure to carry over any exclude options you are using.

yum upgrade --exclude=nagios-plugins

You should now end up with a summary of actions that yum needs to perform, go ahead and kick it off… this will take a bit to complete so go grab some coffee/jolt/redbull and maybe a small snack cause it could be a long night if this blows up on you.

Transaction Summary
=============================================================================
Install 183 Package(s)
Update 327 Package(s)
Remove 0 Package(s)
Total download size: 299 M
Is this ok [y/N]:

Once yum has completed (hopefully without major errors) we need to fix a few things, the first is the rpmdb needs a rebuild due to version changes that will cause any rpm commands to fail:

# rpm -qa
rpmdb: Program version 4.3 doesn’t match environment version
error: db4 error(-30974) from dbenv->open: DB_VERSION_MISMATCH: Database environment version mismatch
error: cannot open Packages index using db3 – (-30974)
error: cannot open Packages database in /var/lib/rpm

This can be fixed by running the following to manually rebuild the rpmdb:

rm -f /var/lib/rpm/__db.00*
rpm --rebuilddb
yum clean all

The next issue on the list is python-elementtree and python-sqlite, one or both of these may have ended up in a broken state that will cause all yum commands to break, so we will go ahead and reinstall both of these for good measure:

rpm -e --justdb python-elementtree --nodeps
rpm -ivh python-elementtree-1.2.6-5.el5.i386.rpm
rpm -ivh python-sqlite-1.1.7-1.2.1.i386.rpm --nodeps --force

The yum command should now work, go ahead and run it with no options, if you do not get any errors you are all sorted.

Hopefully the install went well for you, the only thing left to do is go ahead and reboot the system; this is the last point at which you have to make backups (but we all maintain backups right?). For the sake of avoiding a heart attack if the system goes into an fsck, we will reboot with the -f option to skip fsck:

shutdown -rf now

That’s a wrap, I hope you found this HowTo useful, if you did run into any issues then go ahead and post them into the comments field and I will try to assist but when in doubt typically google is the fastest alternative.

Snorting the Web Farm

Here are some rules for you snort freaks to chew on that I have found useful in web heavy environments.

alert tcp $HTTP_SERVERS $HTTP_PORTS -> any any (msg:"ET ATTACK RESPONSE x2300 phpshell detected"; content:"Locus7Shell"; nocase; classtype:web-application-activity; reference:url,www.rfxn.com; sid:300010; rev:1;)
alert tcp $HTTP_SERVERS $HTTP_PORTS -> any any (msg:"ET ATTACK RESPONSE RFI Scanner detected"; content:"RFI Scanner"; classtype:web-application-activity; reference:url,www.rfxn.com; sid:300020; rev:2;)
alert tcp $HTTP_SERVERS $HTTP_PORTS -> any any (msg:"ET ATTACK RESPONSE lila.jpg phpshell detected"; content:"CMD PHP"; classtype:web-application-activity; reference:url,www.rfxn.com; sid:300030; rev:2;)
alert tcp $HTTP_SERVERS $HTTP_PORTS -> any any (msg:"ET ATTACK RESPONSE ALBANIA id.php detected"; content:"UNITED ALBANIANS aka ALBOSS PARADISE"; classtype:web-application-activity; reference:url,www.rfxn.com; sid:300040; rev:2;)

alert tcp $HTTP_SERVERS $HTTP_PORTS -> any any (msg:"ET ATTACK RESPONSE Data Cha0s Backdoor"; content:"Data Cha0s Connect Back Backdoor"; classtype:web-application-activity; reference:url,www.rfxn.com; sid:300050; rev:1;)
alert tcp $HTTP_SERVERS $HTTP_PORTS -> any any (msg:"ET ATTACK RESPONSE Data Cha0s Backdoor"; content:"Spawning Shell"; classtype:web-application-activity; reference:url,www.rfxn.com; sid:300051; rev:1;)

alert tcp $HTTP_SERVERS $HTTP_PORTS -> any any (msg:"ET ATTACK RESPONSE c99 phpshell variant"; content:"c999shell v."; classtype:web-application-activity; reference:url,www.rfxn.com; sid:300060; rev:1;)
alert tcp $HTTP_SERVERS $HTTP_PORTS -> any any (msg:"ET ATTACK RESPONSE c99 phpshell variant"; content:"C99 Modified"; classtype:web-application-activity; reference:url,www.rfxn.com; sid:300061; rev:1;)
alert tcp $HTTP_SERVERS $HTTP_PORTS -> any any (msg:"ET ATTACK RESPONSE c99 phpshell variant"; content:"RootShell Security Group"; classtype:web-application-activity; reference:url,www.rfxn.com; sid:300062; rev:1;)
alert tcp $HTTP_SERVERS $HTTP_PORTS -> any any (msg:"ET ATTACK RESPONSE c99 phpshell variant"; content:"Modded by Shadow & Preddy"; classtype:web-application-activity; reference:url,www.rfxn.com; sid:300063; rev:1;)

alert tcp $HTTP_SERVERS $HTTP_PORTS -> any any (msg:"ET ATTACK RESPONSE js insert redirect"; content:"META HTTP-EQUIV='Refresh' CONTENT='0"; nocase; content:"text/javascript"; nocase; pcre:"/daobrains\.info|engaolika\.info|globalsecurityscans\.com|goscanpark\.com|goscansome\.com|goscansoon\.com|goslimscan\.com|neborin\.info|safetyshareonline\.com|securityexamineonline\.com/i"; classtype:web-application-activity; reference:url,blog.unmaskparasites.com/2009/07/23/goscanpark-13-facts-about-malicious-server-wide-meta-redirects/; sid:300070; rev:3;)

alert tcp $HTTP_SERVERS $HTTP_PORTS -> any any (msg:"ET ATTACK RESPONSE DarkMailer"; uricontent:"cgi"; nocase; uricontent:"m="; nocase;  pcre:"/dark\.cgi|dm\.cgi|da\.cgi/iU"; classtype: web-application-attack; sid: 300080; rev:1;)
alert tcp $HTTP_SERVERS $HTTP_PORTS -> any any (msg:"ET ATTACK RESPONSE DarkMailer"; content:"yellsoft"; nocase; content:"DirectMailer v"; nocase; classtype: web-application-attack; sid: 300081; rev:1;)

alert tcp $HTTP_SERVERS $HTTP_PORTS -> any any (msg:"ET ATTACK RESPONSE r57 phpshell footer detected"; content:"http-shell by RST/GHC"; classtype:web-application-activity; reference:url,www.pestpatrol.com/spywarecenter/pest.aspx?id=453096755; sid:2003535; rev:4;)
alert tcp any any -> $HTTP_SERVERS $HTTP_PORTS (msg:"ET ATTACK RESPONSE r57 phpshell source being uploaded"; content:"by 1dt.w0lf"; content:"GHC http"; distance:0; classtype:web-application-activity; reference:url,www.pestpatrol.com/spywarecenter/pest.aspx?id=453096755; sid:2003536; rev:5;)

alert tcp any any -> any any (msg:"ET ATTACK RESPONSE devil_ shellbot: conback request"; content:"Conectando-se em"; classtype:attempted-admin; reference:url,www.rfxn.com; sid:300090; rev:1;)
alert tcp any any -> any any (msg:"ET ATTACK RESPONSE devil_ shellbot: udp flood"; content:"pacotando"; content:"temp"; classtype:attempted-dos; reference:url,www.rfxn.com; sid:300091; rev:1;)
alert tcp any any -> any any (msg:"ET ATTACK RESPONSE devil_ shellbot: udp flood"; content:"de envio"; content:"Total pacotes"; content:"Total bytes"; content:"Tempo"; classtype:attempted-dos; reference:url,www.rfxn.com; sid:300092; rev:1;)

alert tcp any any -> any any (msg:"ET ATTACK RESPONSE /bin/sh shell spawn"; content:"no job control in this shell"; classtype:attempted-admin; reference:url,www.rfxn.com; sid:300100; rev:1;)

BOGON Filtering, Update It

One of the features used by APF to prevent address spoofing is that it filters reserved IP address space, also known as BOGON filtering. This is an otherwise very reliable method to keep out random unallocated spoofed addresses from injecting traffic towards your server, assuming, of course, the list is updated regularly.

We decided a few months ago that we would disable by default all remote features in APF, including the auto-updating of the reserved networks file (BOGON filter list). However, this was done with one ill-fated consequence… That if you do not turn on the update feature yourself you may inadvertently start filtering newly allocated address space. This was a change that in hindsight was probably not the best of ideas and as such has now been turned back on by default in the current APF release.

That is then the purpose of this post, every feature even in its best intentions has the capacity when it goes unmanaged to cause more harm than good and that is exactly what the BOGON filtering will do over time if left enabled and it goes unmanaged. The quick fix to this issue is that you enable the reserved network update feature or disable the use of the reserved network filtering in conf.apf altogether; below are details on conducting either task:

1) This change will make APF download and update to the latest reserved network file daily at 4 am (recommended):
edit /etc/apf/conf.apf and set the option DLIST_RESERVED=”1″
restart APF with /etc/init.d/apf restart

OR

2) This change will disable the use of reserved network filtering from APF altogether (not recommended!):
edit /etc/apf/conf.apf and set the option BLK_RESNET=”0″
restart APF with /etc/init.d/apf restart

I hope this assists anyone who is experiencing random connection issues from clients or to better decide if this feature is right for you. The BOGON list APF uses is maintained at http://www.rfxn.com/downloads/reserved.networks , which is updated twice daily with the latest data from ARIN.