Threat Hunting: Experts Not Required

Apologies to anyone that may have periodically checked this blog in the past few years only to find the same two posts (I'm betting that includes no one lol). I have found that while the Advisory side of Incident Response offers many more opportunities to analyze and deal with new and interesting threats, it also comes with an operational tempo and travel schedule that leaves very opportunity to write anything more than my required reports. As such, I am deciding to not over think it and just try to get something out on a more regular basis. 

One of my duties besides Threat Hunting, Malware Analysis, and Incident Response is to train client security personnel to do all of these. I have always enjoyed teaching, training, and hopefully motivating people to be successful in DFIR, ever since I was a Research Assistant at Mississippi State under Dr. Dave Dampier. One of the main reasons that I enjoy it is that I still remember being that goofy teenager living in Kosciusko, Mississippi, desperate to find any possible resource to learn just a little bit more about anything dealing with computers. Fortunately, I met, and bugged the crap out of, the most knowledgeable people I could find along the way, and finding some amazing mentors along the way. If I can pay any of that forward, I'm all for it. That, and anyone in DFIR who truly loves it will take any excuse to sit and talk shop. :)

So, in the interest of contributing whatever I can, I will be talking about some useful methods and skill sets for Threat Hunting that I have seen be successful at even the most junior analyst levels. I often start with these when training my clients, and hope someone finds them useful. 

"What is Threat Hunting?"

Threat Hunting is a defined as "the process of proactively and iteratively searching through networks to detect and isolate advanced threats that evade existing security solutions."  Yeah, I thought it was a little wordy as well.  Basically, Threat Hunting is, at the basest level, investigation without cause.  You have no reason or indication to believe that your environment is currently compromised, but you go looking for it anyway.  It can get significantly more involved the more it is integrated into your security practices and posture, but this is where it starts.

That being said, and many of the IR practitioners I have spoken with have agreed with me on this, you do not need to be a DFIR analyst ninja to hunt for threats.  However, in my experience with training client analysts, I have found that hunting for threats can help you become a much better DFIR analyst.  Just so this point isn't lost, as I believe it is very important, I'll repeat:

You do not have to be a DFIR ninja to hunt for threats, but hunting for threats will help you become a DFIR ninja.

This is because it helps exercise the primary five things I believe you must have to be a good DFIR analyst:

1. Solid Network and Host Fundamentals
2. Critical Thinking
3. Investigative Mindset and Methods
4. Understanding of Attacks
5. Self-Starting Motivation

The rest is just practice, exposure, and continued study, all of which can be done rather cheaply and easily today via virtual labs, Internet resources, and community events.  Additionally, Threat Hunting is something that you can practice the skills for at home (via forensic challenges, attacking your own lab machines, and online education materials) or at work (if you are currently in an InfoSec or SysAdmin role, though maybe not in DFIR).  Just take whatever host or network data you have available or can create, and contextualize it to look for signs of attack or compromise. 

How to Get Started: Methodology 

 

Two main things to remember when getting started with Threat Hunting are: 

1. Focus on the Basics
2. Think About What You Are Doing
3. Start Big and Go Small 
4. Keep It Simple

We'll discuss each of these in turn.

Number 1: Focus on the Basics

 

The more I deal with various incidents and threats, the more I find that a solid foundation of network and operating system basics to be critically important.  Often times I find, even with very careful attackers who are mindful to leave little forensic trace of their actions, that they are still able to be detected and tracked using knowledge learned when I was just getting started in IT 20 years ago.  Things like understanding of the OSI Model, TCP/UDP/IP headers, HTTP, and Windows/Linux Administration basics have served me better than any advanced forensic, malware, or attack method I have ever learned.  This is because, even with the advances in the last 25 years, everything is still built off of those building blocks.  And since we have placed SO much on top of these technologies, they are extremely slow and resistant to change. Using this fact to an analyst's advantage is one of the best force multipliers you will ever have as a DFIR analyst, as these fundamentals are tool and vendor agnostic, traverse almost every environment worldwide, and act as a primer in understanding almost any advanced situation you may encounter.  So, here is a short, abridged list of fundamentals that you would want to become intimately familiar with to get you started:

• Networking
• OSI 7 Layer Model
• TCP/UDP/IP Protocols and Headers
• HTTP/HTTPS Protocols and Webservers
• SSH/RDP Protocols and Commonly Used Tools
• Windows
• Windows Registry (SYSTEM, SOFTWARE, NTUSER.DAT, USRCLASS.DAT, AMCACHE.HVE, etc.)
• Windows Event Logging
• Windows Processess and DLLs
• Windows Services
• Windows Kernel Drivers 
• Windows Host Network Tables
• NTFS File System
• Linux
• EXT3/4 File Systems
• Linux Kernel vs. User Space
• Linux Logging (audit, syslog, apache/tomcat/nginx, last/lastb, wtmp/utmp/btmp, bash_history, etc)
• Linux Host Networking
• Linux User Authentication
• Common Linux Locations and Paths

With a solid understanding of the topics in any one of these categories, you can begin Threat Hunting with some level of efficacy. 

Number 2: Think About What You Are Doing

 

This often trips up analysts that have worked or are working in alert-shop operational security.  Many of the analysts that I work with are so used to requiring an alert to tell them where to start that they begin to rely on alerts as starting points.  Without them, they can get a little lost as to where to start their investigation (remember, Threat Hunting is just investigation without cause).  In Threat Hunting, there is no alert to tell you where to start so you have to come up with that on your own.  So the first question that you should ask yourself is: "What attack or activity to I want to start hunting for?"  For this you will need to have some understanding of what you are looking for in order to get a starting place.  However, this isn't as difficult as it may seem.  Let's take webshells for example, as they are simplistic, effective, used by all kinds of attackers, and aren't going away anytime soon.  So, we will try to hunt for webshell activity on the network.

So for a webshell to work, you have to have a webserver.  We'll assume for this exercise that you only have one, with and IP address of 10.0.0.1.  Okay, so we already know that we will be looking at traffic associated with our webserver.  We've narrowed our data set significantly already!  Now, if someone is talking to your webserver and based on how webservers work, where would the attacker likely be?  Probably outside of your environment in the Internet.  So the connection will most likely start from the Internet with our webservers as the destination, meaning we are looking at inbound connections.  Now we are getting something to work with!  Now, what protocol do webservers use?  Yes, they use TCP/IP, but so do a whole lot of other things.  What protocol is most specific to webservers?  That's correct, it's HTTP.  So right now, we are querying whatever network tool we have with the following query (using Wireshark notation as example):

ip.dst = 10.0.0.1 && http && (ip.src not net 10.0.0.0/8 || ip.src not net 172.16.0.0/12 || ip.src not net 192.168.0.0/16)

This query says "Show me all connections to my webserver that use the HTTP protocol and the source IP address is publicly routable (not an RFC 1918 IP address)".  Now you are hunting through only inbound web traffic to your webserver, which is where webshells will live. :) 

 Number 3: Start Big and Go Small

 

In the previous section, we discussed reducing our data set to only the data that could possibly contain evidence of the attack we were hunting for.  Since we were hunting for webshells, we started with all of the network traffic data that we had and reduced it down to only inbound HTTP traffic to a single webserver.  This "breadth-first, depth-last" approach is effective for two primary reasons.  First, it helps you to frame your investigation based on your dataset and the nature of the attack you are hunting for.  This methodical approach allows for better understanding of your investigative methods, which leads to more effectively communicating any investigative findings you may have.  It also keeps you as an analyst structured and focused, with analytical structure that can be checked and reviewed to identify errors if you get wonky results.  Second, it helps you to more effectively increase or decrease granularity in your investigation.  This is a critical part of the effectiveness of this methodology, as increasing granularity can assist in root-cause analysis, while decreasing granularity based on specific aspects allow an analyst to quickly and effectively scope findings to determine additional attack activity.  Both scoping and root-cause analysis are cornerstones of every investigation, and will have to occur to the farthest extent possible for an investigation to be effective and complete.  To illustrate this, we will continue with our webshell example.

Given the query that we constructed in the last section, will add to this query to zero in on possible webshell traffic.  To do this, we keep asking ourselves questions and thinking about what we are looking for.  The purpose of a webshell is to allow an attacker to:

• Send commands to the webshell to be executed on the host operating system
• Send files to be stored on the webserver
• Retrieve files from the host operating system's file system

Looking at this, if every webshell were used for each purpose equally that would mean that at least 66% of webshell activity would be an attacker sending data to the webserver.  In reality and practice, the majority of webshell communication is attackers sending commands to the webserver, but we'll ignore that knowledge for the moment and stick to the 66%.  So, most of the time with webshell traffic, the attacker will be transmitting data to the webserver and the webserver "speaks" the HTTP protocol.  A quick Google search of the HTTP protocol, and a few minutes reading, shows us that the primary HTTP function (called "method") that allows for sending data to the web server is the POST Request method.  So, we will add this to our query from last section:

ip.dst = 10.0.0.1 && http && (ip.src not net 10.0.0.0/8 || ip.src not net 172.16.0.0/12 || ip.src not net 192.168.0.0/16) && http.request.method == "POST"

 Now we are only reviewing inbound HTTP POST Request traffic to our webserver. Given that we also learned in our Google reading that the majority of HTTP traffic involves the GET Request method, we have again significantly reduced our hunting data set.  Now we can start reviewing this data set for additional indications of webshell activity such as:

• Long Base64 Strings
• Cookies Not in Name/Value Pair Format
• File Listings, Command-Line Output, or Common Windows/Linux Administrative Commands
• PHP/JavaScript/ASP Code and Functions (such as eval, base64_decode, charCode, and WScript.Shell)

At this point, you are Threat Hunting for webshells, you have a better idea of the type of traffic your webserver gets, and you are becoming a better DFIR analyst!

Number 4: Keep It Simple

 

Now, at this point I'm often asked about super-advanced stuff, super-technical hunting, or whatever "What If" question based on the latest threat report or Mr. Robot episode.  And yes, sometimes dissecting and determining what is actually occurring, and the associating context around the data, can get slightly difficult to extremely complex.  And attacker methods often range from commonly used everyday tools to highly complex malware, and even to actions that you would never in a thousand years think they would do.  However, even with the most complex malware, the most advanced attackers, and the most sophisticated methods, the vast majority still has to run on the same basic infrastructure as the rest of the Internet.  The trick is to use that to your advantage.  A coworker of mine often says "If it sounds like Hollywood, then it's Hollywood."  Experience has shown him to be right.  So start simple, the complex will come with time and practice.  And when the complex stuff comes, look at it in terms of the basics, keep it simple, and follow the evidence.  Until then, just identifying that something unusual is occurring, even if you aren't sure what it is, is often the difference between a successful Response and the 9 o'clock news.

In Conclusion

 

I hope that this may have helped someone that has wanted to begin hunting, but maybe didn't know where to start, how it works, what it is, or why they would want to do it.  If it helps someone, then it was useful.  Either way, like I told my buddy @jared703, blogs are a good notebook if nothing else. So just remember, while Threat Hunting isn't without its complexities, a simple starting approach is still effective, the basics are always important, think about what you are doing, and just knowing that something is wrong can make all the difference.

Good luck and good hunting. :)