When I first introduced you all to the Aggregate Threat Feed back in May, it was a much smaller feed with very simple ambitions — pulling together threat data at work from our network edge and host based firewalls and aggregating the data into a usable feed. The actual intention being that as an attacker exposes themselves more on the network through invasive scans and attacks, they would quickly climb up the threat feed and end up banned proactively. Though this did and still does happen in a way, a problem was introduced when more and more data started to come in from the network edge and it quickly outweighed data from the hosts.
The old way the threat feed was sorted was by number of events. For the network edge IPS the events correlated with actual signature events on the network edge, so these could number from 50 events for an SNMP community scan to thousands of events for an SSH scan. Then you have the host based firewall events (mostly brute force attacks), these events are correlated into the feed by the occurrence of an attackers address across unique servers, so if 1.2.1.2 made a brute force attempt against 11 servers it would show in the feed as 11 events.
The problem that developed here is that the network edge IPS is far more noisy on an exposure level than the host based firewalls, so you would end up with hundreds of IP’s from the network edge with thousands of events each, while the host based firewalls, even though they represent hundreds of attacking IP’s also, the actual event counts relative to unique servers those IP’s attacked, was FAR lower. This meant that often the top 50 or 100 items in the threat feed were all IPS events, though quite valid events the actual host based events had more of a threat significance than some of the IPS events. The host events were simply being washed out of the top 100 on the list from the sheer volume of IPS events (who really wants to import 300 addresses from a threat feed? let alone even 100).
So, what I decided on doing was adding a weighted field into the database that is based on unique targets for each attacking IP. This weighted field is the new sort method for the feed and it works something like this. If the IPS picks up an attacker hammering five servers with an SQL injection exploit, that attacking IP ends up in the threat feed with a weight of 5, if we then have an attacker that runs brute force attacks on 30 servers, that attacking IP ends up in the threat feed with a weight of 30. The end result is that the threat feed gets better populated with the highest weighted attackers at the top, so those attackers who are more aggressive across unique targets, quickly end up at the top of the list. This allows the feed to better protect the devices/hosts it is being used on from a developing attack before the attacker reaches that device/host on the network.
Drop Format:
http://asonoc.com/api/atf.php?top=50
List Format (fields: IP | SERVICE | EVENTS | WEIGHT):
http://asonoc.com/api/atf.php?top=50&fmt=list