26 04 2012
Splunk and Malware Fun
So I was playing with Splunk at home and was curious what it looks like when you run some malware while pulling real time data into it.
The environment I set up for this test was a simple install of Windows 2008 R2 server, which has no special services running. I had it running via VMWare. As for Splunk, it’s the free edition, and I have it installed locally on the machine, and have it pulling System, Application, and Security logs only.
Let’s get started. First ill grab an md5 hash of the malware:
And here is a quick check on Virus Total. As you can see 36/43 find this to be malicious. I believe it’s a Zeus variant. I forgot where I downloaded it from. I believe I downloaded it from MalwareBlacklist.com
So now that we know we are working with a piece of malware let’s execute it on our system. One note: prior to executing the piece of malware I fired up Splunk and set it to monitor real-time so I could see events come across as they happened. This obviously wouldn’t be as clean in “real-life”, but it should allow you to identify some areas that you can go back and search for. Once I felt like the malicious program ran its course I ended the search so I had a self contained time period that I could work with, and I didn’t have to keep fighting normal system activity.
One last thing. Here is the Windows 2008 R2 auditing configuration I have set. I set it all so I could capture a whole picture. I also wasn’t concerned with space, etc. I don’t know how you all have your environments setup. In either case, it is simply for testing purposes.
“In God we Trust…all others we Splunk”
As you can see our first hit is event id 4688. Event id 4688 documents each program that is executed, who the program ran as. and the process that started the process.  You see here that melt.exe was executed and was given a process ID of 660. This makes sense given that I personally clicked on the exe file.
Next we see another process created via the same event id 4688. You can also see that PID 660 created it, which we see in the picture below belonged to the malicious executable file. So now we have PID 660 creating PID 1820 (0x71c). It also looks like the file dropped another exe into AppData\Roaming[random_folder]\[random_name.exe], which is a signature of Zeus. Zeus randomizes the folder name and .exe file.
Let’s verify that’s indeed what happened. In this picture I navigated to the AppData location and it does appear a file was created.
So within two hits in Splunk we already have an idea (in real-time) that something bad might have just happened. Let’s keep moving forward so we can see how this entire process played out. It wouldn’t be fun if we stopped now.
Now this next Splunk entry had me guessing so I would appreciate some feedback. I’m going to assume that this event id is the malicious program gaining SYSTEM level privledges, or maybe it just happened to execute at the same time. It seems unlikley, and I lean more towards it being malware related, but please correct me if I am wrong.
Next we see NT AUTHORITY\SYSTEM being assigned special privileges based off its new logon from the previous picture.
In this screen shot we see that PID 660, which was our original PID for melt.exe. It creates another PID 1506, which is a cmd.exe.
I thought about this for a few minuets and realized when I had originally analyzed Zeus, I came across a cmd.exe prefetch. When looking at the prefetch file it appeared that the cmd.exe file executed a bat file in order to delete itself, which makes sense given that it has already moved itself over to AppData.
Without looking into more details on the system I can’t be certain this is the cmd.exe that is spawned to delete itself via the batch file, but it’s highly possible it is. I’ve included the .bat file from my original analysis so you can see what that looked like 
What also leads me to believe that it is indeed the cmd file used to delete the original file is that right after the cmd.exe process is created the original PID 660 is exited. You can see that here:
So if you remember from above we now have: PID 660 Creating PID 1506 (cmd.exe) and also spawning PID 1820 (owoz.exe).
Here you will see PID 1820, which is the owoz.exe file located in the AppData directory bind an IP address to a local port. The source port is 27006 and the address is 0.0.0.0.
So right after PID 1820 (owoz.exe) binds a port to itself you get event id 5031 triggered, which occurs when windows firewall blocks an application from accepting incoming connections on the network.
Just an FYI – I was in an isolated lab. In either case, here is the image:
So awesome….our firewall blocked it!! Well….not so fast… It made about 3 attempts and the firewall blocked it all three times, but then here came event id 5447 to rain on our parade. As you can see PID 984 (svchost.exe, which happens to be svchost.exe appears to have added a rule in our windows firewall to allow connections via port 27006. If you remember, port 27006 was the port that PID 1820 (owoz.exe) bound to 0.0.0.0. Not looking good for us…
And this all but confirms it. Here you will see that a change was made to the windows firewall exception list. The rule to allow 27006 was added.
So here we see another event id 5447, where owoz.exe binds itself again via port 12224.
I wanted to take a closer look at the svchost.exe process that’s created and did all of those modifications to the firewall. I will be using Process Hacker to look at the process.
So here you can see svchost.exe running on the machine. It’s PID 984.
As you can see in red I selected the, “Handles” tab. This handle view shows which object handles belong to the process. Per the Windows SysInternals book; “Object handles are what programs use to manipulate system objects managed by kernel-mode code, such as files, registry keys, synchronization objects, memory sections, window stations, and desktops.” 
You can see here that we see a process named “owoz.exe”, which is PID 1820. This malicious program appears to have injected itself into svchost.exe.
Next I decided to run strings against the process via the “Memory” tab. Here are some of the results:
These strings look interesting because of the firewall changes this specific process made:
Here are some more strings that seal the deal. We see here the location of our favorite owoz.exe (PID 1820) smack dab in the middle of our strings search.
To bring it full circle here are the ports that were punched through our windows firewall. I viewed this via Process Hackers Network tab.
I’ve been thinking about a way to run a query in Splunk to capture this data. Obviously this wouldn’t be very easy if you’re indexing gigs each day, but the information is there, it’s only a matter of pulling it out. I also understand that I was pulling the process information logs, which in a large environment would be a ton of information. In either case, maybe only on the more critical systems? In the end it’s about risk management I guess.
I hope you liked this post. Please feel free to post comments. And for the Splunk Ninja’s out there, free beer to the first one that makes a nice search query to make this much easier to pull out. I’ll be trying myself. In the mean time, just some simple event id searches along with some time restraints might pull something useful out.
 Windows Sysinternals Administrator’s Reference