Tag: projects

Posted on

Linux Malware Detect v1.3.6: Loose Ends

In LMD 1.3.3 there was allot of changes, 29 to be exact, that made LMD much more robust and especially the monitoring component, much more usable. If that release was about making good things better, then this release is about bringing loose ends together. I spent a couple of days running LMD through its paces along with having many people help me test it and during that process, we brought allot of little things to the surface that needed fixing or revising.

In total, there has been 31 changes, fixes or new additions to LMD since that 1.3.3 release on the 15th, most of these changes were completed days ago but I wanted to take the time to make sure they were working as intended and that no more bugs/issues came to the surface. At the moment, since releasing LMD on the 11th, there has been a total of 1349 downloads, so to say that there is plenty of opportunity for bug reports would be understated. I am comfortable in saying that the changes from 1.3.3 to 1.3.6 are stable, reliable and working as intended.

The version changes aside for the moment, there has also been a mountain of user submitted files with the –checkout feature, I processed many of those yesterday and earlier last week for a total of 71 new signatures for the week. Those signatures will have automatically been updated to your install through the cron.daily run of –update, or you can run it yourself if you do not use the default cronjob.

So, what of significance has changed since 1.3.3? The biggest changes are that there is now a -d|–update-ver feature that performs a version update check and if a new version of LMD is available, it will install it. This feature does both a version number check and hashes the main LMD files checking for differences with the server side files, when one of the two checks fails, an update is forced. The version update is not automatically run for a number of reasons that I am to lazy to explain, just think about it a bit. All session and quarantine data is migrated on update.

Most of the other changes are fixes and improvements on existing features, especially the monitoring component which of the 31 changes since 1.3.3, 17 of them are all within the monitoring component. There has also been a few changes to the README file to reflect some minor usage changes, to clarify better some usage of the monitoring service and to explain some new ignore options.

That is all from me, changelog is below, enjoy.

Project Page: http://www.rfxn.com/projects/linux-malware-detect/

Change Log v1.3.3 => v1.3.6:
[Fix] session data gets recreated if it disappears during scan
[Fix] tlog now handles data that logged between 0bytes and first wake cycle
[Fix] monitor_check now properly handles CREATE,ISDIR events
[Change] –alert-daily|weekly alerts have been changed similar to manual alerts
[Fix] cleaner was not properly running on monitor_check calls to scan files
[Fix] quar_suspend was not properly running on monitor_check calls to quar()
[Change] monitor tracker files now pass through trim_log to avoid oversizing
[Fix] monitor_check now properly handles path names with spaces
[Fix] monitor_check was throwing nx file/directory error for monitor.pid
[Fix] older bash versions were having trouble with the [[ =~ ]] regexp search
[Change] set all script files from shebang/bin/sh to shebang/bin/bash
[Change] –alert-daily|weekly will now only send alerts if hits were found
[New] -d|–update-ver now compares file hashes to determine update status
[Fix] suspend events were not properly being added to monitor alerts
[Change] all alerts have had spacing changes to make them more readable
[Fix] signature names now properly list for daily|weekly alerts hit list
[Fix] monitor_check will now recursive monitor newly created directories
[New] monitor daily|weekly alerts now save as a pseudo scan report with SCANID
[Fix] monitor reports now generate properly when quar_hits=0
[Fix] cleaner function was not properly executing under certain conditions
[Change] additional error checking/output added to the cleaner function
[Change] default status output of scans changed for better performance
[New] added ignore_intofiy for ignoring paths from the monitor service
[Change] updated ignore section of README
[Fix] backreference errors kicking from scan_stage1 function
[New] -d|–update-ver option added to update installed version from rfxn.com
[Change] updated short and long usage output for update-ver usage
[Fix] -k|–kill-monitor now properly kills only the inotifywait/monitor pid’s
[Fix] monitor_cycle function now correctly stores its pid in the pidfile
[Fix] files with multiple events in the same waking cycle are only scanned once
[Change] install.sh now symlinks maldet executable to /usr/local/sbin/lmd

Posted on

Let The Rewrites Begin: New Life For PRM

In my last post, I reflected on the last 7-8 years of projects here at rfxn.com, in doing so I also dug up some statistics on project downloads. I not only did this for my own curiosity but to prioritize the mile long to do list I have for the projects, based on downloads. One of the revealing things was just exactly what people are downloading, in particular that projects like LES , PRM & SIM are still very popular download destinations on the site.

Although a new incarnation of APF & BFD are on the agenda, I thought I would work up to those by first knocking off rewrites of some of the smaller projects, starting this off is PRM. This is a project originally written in December of 2003 and although it has stood the test of time by doing exactly what it was intended for and doing it reliably, it was starting to show its age in a number of ways, especially the not-so-intuitive logic and less-than-appealing documentation.

Today I have put out PRM v1.0.6, a ground-up rewrite of just about everything in the project, simplified logic, oodles of new features and one of the biggest problem areas over the years, far better ignore options to control exactly what PRM does along with detailed documentation.

Enough said, check the changelog for for summary of changes and the README for details on the new usage.

Project Page: http://www.rfxn.com/projects/process-resource-monitor/
Current Release:
http://www.rfxn.com/downloads/prm-current.tar.gz
http://www.rfxn.com/appdocs/README.prm
http://www.rfxn.com/appdocs/CHANGELOG.prm

Posted on

The Test Of Time: 7 Years & Counting…

Today I woke up and was in a weird mood, I started to take stock of some thing while at the same time cleaning out the rfxn.com projects and downloads repo (thats a whole other story in itself). In doing so, I realized just how long I have been doing this, it sometimes gets past me just how much time has gone by since my first projects went up.

In November of 2002 I put out the first public version of System Integrity Monitor over at the then rackshack community forums, at a time when Cobalt Raq’s and bargain basement Ensim servers were still the cool thing and ProFTPd and Apache crashing every other morning was also the norm. A short time later, in March of 2003, I put up the first release of Advanced Policy Firewall, without a doubt my most popular project so far.

Here we are, a little over 7 years later and the projects are by any standard still going strong, certainly not as strong as they always were but then again 7 years ago — let alone a couple of years ago — alternatives were few and far between and now there are many projects that have derived some form of inspiration from my own and it is certainly satisfying to know people continue to find value in my work or that I have helped inspire the creations of others.

Over the years, I have moved servers many times, either because of changing employers (often my hosting is provided by my employer) or because I am just A.D.D. like that and am forever breaking things / moving things around. This has always caused a bit of an issue to grasp the actual amount of downloads the projects receive, the last time I took stock of any tangible stats was nearly 3 years ago and the projects had a yearly download rate of about 140k.

This morning I compiled some stats on the last year of project downloads as I finally do have a full year of workable stats again. I will end this post with the stats below while saying that the projects are very much alive and not going anywhere, I have some exciting things planed for the future of the projects and hope everyone can join me while I work towards getting there. Thank you to new and old users alike for always being supportive with simply downloading my work, offering feedback and most of all to everyone that has and continues to donate.

Download Stats (May 2009 – April 2010)
APF 41,374
BFD 13,643
LES 5,662
SIM 4,074
OTHER 6,073 (prm, spri etc…)
Total 70,826

Posted on

Linux Malware Detect v1.3.3: Making good things better

This morning I have put out LMD v1.3.3, this is on the back of two other successive releases in recent days that improved LMD in many areas, along with correcting some bugs that were graciously reported by those helping to break-in the project. I have also listened to feedback and revised a number of features along with completely redoing how the inotify monitoring operates, to provide a much more robust model for real-time file monitoring.

I am also happy to say that people are embracing the use of the -c|–checkout option to send me malware that is not currently detected, which is being processed daily with my regular signature maintenance tasks. I have today added 24 new signatures, all of them created from user submissions.

There are a few big changes in this release…

First and foremost is that the configuration file conf.maldet has been completely revised with more granular options provided for quarantine, scan and monitoring, along with better commenting. Adding to the configuration convenience is that the install.sh script will now import config settings from previous install along with migrating session data.

Next up and something I am excited about, is a rule driven – signature based – cleaner function that can remove string based malware injections from files. The cleaner has two default rules created by me that will clean files of base64 and gzinflate injected strings very accurately. Through the next couple of days/weeks, I will be adding more cleaner rules that will allow for a much broader base of signatures that we can clean files for.

Finally, inotify monitoring got some loving with a top-down review of things and I came up with a less-invasive way of spawning the inotifywait processes that no longer requires a process for each path/user monitored. There is now a single master process that will monitor all configured paths, with better dynamic scaling of the sysctl hooks for inotify based on system resources. In addition, I added an option to pass the monitor service a comma spaced paths list or file containing line spaced paths, from the command line. This is in addition to preserving the users monitoring feature which has also been improved but is no longer the default, you must now call -m|–monitor with one of the USERS|FILE|PATHS options, see –help or the README file for more details.

Please be mindful that although LMD is considered stable it is still a relatively new project and as such your mileage may be a little bumpy, if you run into any issues please post comments on the project page, in this post or send me an email to ryan rfxn.com.

Home: http://www.rfxn.com/projects/linux-malware-detect/
Current Release:
http://www.rfxn.com/downloads/maldetect-current.tar.gz
http://www.rfxn.com/appdocs/README.maldetect
http://www.rfxn.com/appdocs/CHANGELOG.maldetect

v1.3.3 | May 15th 2010:
[Fix] quarantined files were not properly dropping owner
[New] signature based, rule driven, cleaner component added
[New] base64.inject cleaner rule
[New] gzbase64.inject cleaner rule
[New] -n|--clean SCANID option added to batch clean scan all files from a scan
[Fix] made default install file/path permissions more strict (750/640)
[New] install.sh now preserves conf.maldet settings
[New] install.sh now links backups of old installation to INSTALL_PATH.last
[Fix] install.sh now properly imports session data from previous install
[New] -s|--restore can now take a SCANID to batch restore all files from a scan
[Change] improved the layout of conf.maldet; more scan options and commenting
[New] added quar_susp_minuid option for suspend user minimum user id
[Fix] inotify monitor now properly acts on MODIFY,MOVE_TO,MOVE_FROM states
[Change] inotify monitor now can take a list of paths or file for path input
[Change] inotify monitor now has no default use, must specifiy USER|FILE|PATHS
[Change] revised short and long usage output for new options/usage changes
[Change] inotify monitor now spawns only one process for all monitored paths
[Change] inotify monitor sets max_user_instances to processors*2
[Change] inotify monitor sets max_user_watches to inotify_base_watches*users
[Change] migrated all inotify options from internals.conf to conf.maldet
[New] added inotify_base_watches to conf.maldet for max file wathces multiplier
[New] added inotify_nice to conf.maldet for run-time prio of inotifywait
[New] added inotify_webdir to conf.maldet for html/web root only monitoring
[Change] extensive format change to README
[Change] rewrote inotify section of README to reflect the many changes
[Change] -q|--quarantine now calls cleaner if quar_clean=1
[Change] -n|--clean can now do in place cleaning without quarantine
Posted on

Better Late Than Never: Linux Malware Detect 1.3

Today I have released Linux Malware Detect (LMD) 1.3, the first public stable release of my malware detection tool. The documentation is a little thin but the details are on the project page and the README file should fill you in on anything you need to know, otherwise you can post a comment on the bottom of the project page and I will assist where possible. Input on feature ideas, bugs and malware data is always welcome, see the –help options on LMD for the checkout feature to upload malware data to rfxn.com.

In October I detailed the concepts behind the then to-be-released LMD in a post, though allot has changed since then in how LMD operates, the jist of the post is still on point.

To those (unfortunate?) enough to ride in on the closed testing, it certainly was a long road and I thank everyone that over time submitted new malware data, bug reports and feature ideas. To say this is the most banged-in release of one of my projects would be understated and I hope it shows in the end product.

So, What has changed since the first incarnations of LMD? Well first is that I ditched the whole “chunked hash” concept for a simpler HEX based pattern matching feature to find malware variants which has proved far more accurate and easier to manage. Though I can see some scaling issues with the current implementation of the HEX scanner as the signature set grows, this is something I do expect to resolve in a future release. The basic MD5 hashed scanning is still the stage-1 scanning component and then the HEX scanner picks up as a stage-2 scanner if no MD5 hit was found.

The kernel based inotify real-time file creation/modification monitoring has been reworked and now more gracefully handles users of any type in addition to monitoring the /dev/shm, /var/tmp and /tmp paths on execution of the monitoring component. Also changed is that the scanner will now batch through new/changed files every 30 seconds for the sake of efficiency but this can easily be modified in the internals.conf down to as low as a 1 second iteration on the scanning of new/changed files.

The quarantine queue now stores files original path, owner and mode to facilitate a –restore feature that allows any file to be restored to its original path with owner and file modes restored as well. This can be used to recover false-positive hits or to restore files after you have cleaned malware from within its contents (default quarantining of malware is now also disabled, see conf.maldet).

The final notable change is that there is now a quarantine suspend account feature, the owning user account (UID>=500) can optionally be Cpanel suspended or have its shell set to /bin/false on non-cpanel systems (configurable in conf.maldet). When Cpanel users are suspended, they will have a comment attached to it with the ‘maldet –report SCANID’ value so you can easily call up the report that suspended the user.

There has been many more changes to LMD but I certainly can not list them all, give it a spin and let me know how it goes, happy malware hunting!