Tools And Methods We Used To Research This SotM
Tools Used / Misused | • | Unix commands: cat, grep, cut, wc, sort, uniq .... |
|
| • | RubyParse (_parse.rb_) (Brennan) - Extracts summary metadata from logfiles |
|
| • | RubyGraph (_graph.rb_) (Brennan) - ruby+imagemagick - to plot those 'HeatCharts' with 10minutes interval shown! |
|
| • | logsnorter (modified by BESA to let it parse the custom honeywall's --log-prefixes. BTW, also BESA Rocks!) |
|
| • | QwikiWiki (modified to auto-link to IP addresses - just one extra regexp) |
|
Methods Of Collaboration In order to keep communicating, share the discoveries, and address versioning of documents and ideas contributed to the research, we took advantage of some facilities we set up ourselves:
|
| • | a Central website with a WIKI for the Team members to post their discoveries to a central location |
|
| • | a Central ACID Installation with the imported logs to run queries and the simpler graphs in a couple of clicks |
|
| • | a bunch of static pages with logs grep'd for EACH address appearing as inbound traffic (created with a batch at the beginning of the effort) - just for convenience |
|
We also tried to keep a less formal way of explaining things, and to leave a bit of the narration to the original way the discoveries happened for us: you can now read us chatting together and discussing things seen together on the central repository. And of course you can laugh at us for the many silly things and mistakes we happened to make :)
|
But we LEARN together and we SHARE the lessons learned.
|
Each one of us also learns alot from his own mistakes.
|
Some of this chat excerpts have been translated, or they were already in English. Some others (just a few of them) are left in Italian.
|
(21:42:19) BESA: Do you know what I like? do everything like a chat!
|
(21:42:29) BESA: it's like if we're chatting between us and tring to discover the answers
|
(21:42:33) Dani3l3: infact we start in this manner
|
(21:42:39) Dani3l3: it's how it's done; I'm working on it, do you have noticed?
|
(21:43:03) Dani3l3: something like a story/role plaing game between geek
|
(21:43:12) Dani3l3: at the end this is the true
|
(21:43:18) BESA: yes, some piece are so and they're spectacular
|
Aside from the technical stuff, we need to trace back many of these IP addrs, start to group information and identify interesting sessions; mark-up the dataset. I like using web based tracert.cgi or whois.cgi scripts to spread things around, and can automate large batch scans if we want to do something like get a whois and traceroute for every IP we saw from Israel, Saudi Arabia, Iran, etc; break it down by country, and attach meta-data such as whois or traceroute information with a summary of the session (why that traffic is interesting). We can just do these lookups selectively and report the interesting findings back to the list or into a "workspace".
|
I was thinking it would be cool if we could figure out the real IP addrs. For example, if we knew when a certain worm was scanning a certain range, we could at least make an educated guess. I am not sure if we will be able to find that kind of information, but it could be done. Say we know that scans for port 8866 by WORM_BAGLE.B started at ??:?? CEST; the worm started by scanning (??...??), and the first scan in our log was at ??:?? CEST, hence we can guess that the real IPs for this log are in the range of (??...??). Then we could do a little recon of our own maybe, based on the direction that gives us, and scan for sufficiently complex combinations of known hosts for the class C we have masked in the log data; scan for x.x.x.0/24 where x.x.x.67 and x.x.x.95 and (...) exist, but not (...), and with x.x.x.??? port blocked, etc.
|
In any case, just noting things like when that network first started seeing (...) scan from (...) will help get the report going, and that can be as simple as a flat text file or a Wiki style "working document" that we all plug information into when we find things.
|
How should we do that ? I am thinking a Wiki would be best. (and in fact here we are ;-))
|
This is where I am thinking about directing my efforts.
|
Feedback is always appreciated.
|
| — | Timeline: I will have better timeline graphics that should help identify any other smaller gaps we may have missed. |
|
| — | Time: cat | grep; whois; traceroute; ACID - I need to look at it more; I will try to identify more things. |
|
| — | Scripts: I need to make the scripts nicer if they are to be published, but a couple short scripts.rb would be cool to include. Then we can also keep some things sort of secret. For example, I would like to advance our graphing and visualization tools, and we would not need to publish the code itself, but could publish the output & talk about how it works. |
|
| — | Graphing & Visualization: I am thinking about a dshield style graphic with a map and a couple pie charts. This is actually really easy to do, and would be cool. We of course can already make the pie charts, and then it is just a few lines of ImageMagick to make a composite image; put the pie charts on the map image, and scale the diameter of the pie chart relative to alerts. The other graphic I really need to do, is the more detailed timeline. Also, I should get everything to run in a more user friendly way, rather than all the little script.rb files I use now. |
|
Keep In Mind That In several questions/answers we quote the command we gave performing the analysis, and their output. In most cases there are examples of GREPing the original logfile for some patterns. When we use the ORIGINAL log file, it is usually called with the original name "honeynet-Feb1_FebXX.log". In some other cases it is just called shortly "honeynet.log" for on some of our PCs has been renamed to save us some typing (Dani@NL can be VERY lazy at times!).
|
GPL License It is important to note that most of the software used are release under the GNU GPL. This includes qwikiwiki and logsnorter which have been modified.
|
Who Made It
Extra Info
|
