Perched | Security Education, Consulting, and Support
Security Solutions

Perched Blog

DNS Tunneling & Other Hunts w/ RockNSM (Zeek [Bro] & ELK)

In this post, I am expanding on my DNS typosquatting detection post as well as (re-)introducing DNS tunneling detection.

In order to make this blog post worthwhile and applicable to defenders and threat hunters alike, we will cover the following:

  1. Setup of RockNSM ELK instance with Bro.

  2. Clone GitHub repository for the configurations for the remaining steps.

  3. Elasticsearch (template) mappings for additional fields added.

  4. Logstash configuration for additional data to make DNS tunneling detection more efficient. Also, the DNS enrichment added, such as IDN (ie: Punycode/emoji) or if the domain is an IP address, will aide in additional threat hunting.

  5. Kibana visualizations and dashboards for the detection.

  6. Test detection method.

  7. Additional hunts/detection with the added Logstash configuration.

Before I begin, I must recognize a blog I stumbled upon last week while performing my due diligence.

While the blog I mentioned above has some great information, I am still writing this post because I am expanding on it a bit, providing a full walkthrough on the setup, Logstash configurations, and some Kibana visualizations/dashboards.

You can skip straight to the technical section and bypass the introduction and problems with other detection methods:

An Intro to DNS Tunneling/Exfiltration

Although there are many vehicles for implant command and control (C2) and exfiltration, the use of DNS is an interesting method. Generally speaking, it is not properly controlled, not monitored, allowed on most networks, and is a massive amount of logs making for great cover-traffic. Therefore, DNS tunneling is a great option to use as a C2.

To level-set on the technology, DNS requests can contain upwards of 255 characters (and more if an attacker is controlling the DNS server). The data being exfilled is encoded into the remaining free characters of the domain.

Example:
Here I am performing a lookup for neu5ron.com but adding I are secret data... as an encoded string.

JEQGC4TFEBZWKY3SMV2CAZDBORQS4LRO.neu5ron.com

Similar to how DNS exfiltration works, using DNS for C2 uses the free characters to communicate back and forth (C2). Also, many times tunneling will use TXT records because of the ability to use Base 64 encoding.

Why Everyone Should Monitor/Control DNS

First I must preface that even when controlling DNS, DNS tunneling is still very much possible.

DNS is often overlooked as a critical priority for a company/network. However, its use cases are far and wide - security/prevention, human resources, threat hunting, and as a historical database that can aid in network forensics (passive DNS).

Before we dive into the detection process…it is important to underscore that prevention is not bulletproof. As with anything in security you have to weigh the Pro’s and Con’s and communicate to your leadership that your security implementation is cost beneficial.

For example, if it takes 4 hours to properly set up your DNS infrastructure (monitoring and control), and you can prevent 100’s of hours of incident response work or customer/user downtime, then you have effectively provided a financial benefit to your company/organization.

We’ve talked a bit about the monitoring aspect, but control is another critical DNS process. If an adversary is able to use their own DNS server, they can effectively use any domain they would like. For example, here is a scenario where “adobe.com” was used as the C2. Just think about how many whitelists that would bypass…

Preventative Steps

Here are a few preventative steps you can consider:

Positive Impacts

  • Reduces user downtime by not having to re-image their machine from infections

  • Having passive DNS reduces cost (time) of incident response, help desk, and anyone else involved during cleanup of compromises

  • Monitoring and controlling DNS can prevent certain HR violations/issues that may arise from users accessing and using certain content while at work.. ie: Adult Content or Abusive material

Considerations

  • Managing and controlling DNS can require additional support for the management of infrastructure

  • If legitimate domains/services are blocked this could cause user downtime or business impact

Despite DoH

Even with DNS over HTTPs (DoH), DNS tunneling detection is relevant for management networks because many times DNS is allowed out of the management networks, but outbound HTTPS is not.

Also, it may be possible that the JA3 + JA3S hashes and some additional data analytics, like frequency/count, could be used to detect DoH.

I bring this up because recently a few major DNS providers, and even browser implementations, have provided the capability to perform DNS requests over TLS/HTTPs.

If you have any additional detections for DoH that you would like to add then I would love to hear from you and will update the post accordingly.

The Problem With Some Detection Methods

Briefly, I would like to bring up a few pitfalls with current methods of DNS tunneling detection. Many “out of the box” detection methods are a lot more involved than led to believe.

Length of domain

In one weeks time, I observed domains with the following length followed by the number of times seen:

  • 69 characters — 36,926,289

  • 72 characters — 20,965,981

  • 75 characters — 24,475,506

  • 78 characters — 694,677

  • 93 characters — 12,678

  • 174 characters — 1,309

Because I have seen certain SIEMs or analytics built around the length of a domain lookup, some even specifically saying above 70, this would result in millions of alerts…

Passive DNS database

I highly recommend a passive DNS database in addition to your logging infrastructure, assuming you can afford the infrastructure and (human) resources to set up and manage. If that’s not an option, you can use your existing SIEM/logging infrastructure!

Machine Learning

Using a detection method of frequency and unique count of subdomains will lead to many false positives as well as thousands of advertisement domains, content delivery networks, mail providers, and other legitimate websites that use hundreds, if not thousands, of subdomains.

Also, if based upon frequency (time) it’s just a matter of slowly communicating over the tunnel in order to bypass this detection mechanism.

Show Us Something Useful

1. ELK + Bro using RockNSM

For this setup, we will use RockNSM. Here are some detailed walkthroughs(skip to video 2 if you are already familiar with spans and taps).

As many of you know, your monitoring infrastructure is a target by attackers, which is why I have chosen RockNSM over other ELK+Bro deployments because of its secure-by-design implementation — which uses SELinux.

2. RockNSM Enrichment

git clone https://github.com/neu5ron/rocknsm-add-enrichment

Now upload the repository to your RockNSM server (if you did not perform the git clone from the server itself).

3. Additional Elasticsearch Configurations

Before we start ingesting new data, with the added enrichment, we need to make sure that these new configurations are applied to the Elasticsearch database.

Add the Elasticsearch mapping template by performing the following:

curl -H 'Content-Type: application/json' -XPUT "http://localhost:9200/_template/bro-domain-names" -d @rocknsm-add-enrichment/elasticsearch/index-mappings/92-bro-domain-names.json;

4. Additional Logstash Configuration

In order to detect the patterns in DNS tunneling/exfil we need to add specific fields for the 1st level domain (aka TLD), and 1st + 2nd level domain (ie: google.com) and 1st + 2nd + 3rd level domain (ie: www.google.com).

The reason for this is because, as shown earlier, DNS tunneling/exfil will create a high volume of unique subdomains. Therefore, we will be performing aggregations on the 1st + 2nd level of the domain.

Also, we want to add other metadata such as length and total levels (ie: counting each “.”).

# Copy logstash file to rocknsm
sudo cp rocknsm-add-enrichment/logstash/conf.d/logstash-816-domain-enrichment-filter.conf /etc/logstash/conf.d/

# Give logstash permissions for the file
sudo chown logstash:logstash /etc/logstash/conf.d/logstash-816-domain-enrichment-filter.conf

# Change permissions
sudo chmod 640 /etc/logstash/conf.d/logstash-816-domain-enrichment-filter.conf

# Restart logstash to implement new configuration
sudo systemctl restart logstash

5. Kibana Visualizations/Dashboards

I have created and shared all the necessary Kibana visualizations/dashboards in order to see and test the detection method for yourself.

First, we need to import the visualizations/dashboards file:
This can be performed in Kibana by, after logging in, going to “Management” then click “Saved Objects” then clicking “Import” and selecting the “rocknsm-add-enrichment/kibana/dns.json” file from the previously collected repository.

Second, we need to refresh the index:
This can be performed in Kibana by, after logging in, going to “Management” then click “Index Patterns” then selecting the “bro-*” index and then clicking the recycle button (“Refresh field list”) in the top right.

Management > “Index Patterns” refresh “bro-*” index.

6. Detection Method

We will aggregate on the 1st + 2nd level domains and look for a high count of unique subdomains.

The test is performed with an injected DNS tunneling PCAP into two of my environments. I will show you how you can do the same on your own network/setup.

Sub aggregations of the unique count of hosts that performed the lookup may aid in detection. For example, 1 host performing a lookup on 4,000 subdomains is a lot more suspicious than 500 hosts performing lookups on 4,000 unique subdomains. Also, sub aggregations on max total length and max number of levels in the domain allow us to make a more informed decision.

PCAP Injection into Live Network

You’ll notice, in this network, that the tunneled domain “chickenkiller[.]com” is nowhere near the domain with the most unique subdomains. In fact, it has 37,424 fewer subdomains!

Combined with only 1 UniqHosts performing the lookup and the ratio of unique subdomains (Unique Domain Names) compared to total lookups (Count) 4,837 / 4,858 (in this case more subdomains than lookups?) we can begin to see some outliers in this domain.

PCAP Injection Into (My) Home Network

Tunneling Highlighted

Clearly, the same tunneled domain stands out far above the rest of the domains.

Also, note in this dashboard that there is a (data table) visualization for domain_1n2n3_name, this is used for domains that would exist for tunneling/exfil living off of something such as co.uk.

That way you could filter out co.uk in the unique 1n2 subdomains but something such as somedomain.co.uk would still appear in this visualization.

How to Inject the PCAP Into Your Network

# Find interface to replay the PCAP to
sudo tcpreplay --listnics

# Make sure to use one of the NICs that is listed that is the same as one of the listening NICs in /etc/rocknsm/config.yml under "rock_monifs"
grep -A5 "rock_monifs" /etc/rocknsm/config.yml
# Convert PCAP GZ to regular PCAP for replay
editcap -F pcap rocknsm-add-enrichment/testing/dnscat2.pcap.gz rocknsm-add-enrichment/testing/dnscat2.pcap
# Replay the PCAP, replacing "enp0s8" with your interface determined above
sudo tcpreplay -i enp0s8 -p 1250 rocknsm-add-enrichment/testing/dnscat2.pcap

# You should then see output similar to this
Actual: 10000 packets (1465823 bytes) sent in 7.99 seconds
Rated: 183245.6 Bps, 1.46 Mbps, 1250.12 pps
Flows: 7 flows, 0.87 fps, 9682 flow packets, 318 non-flow
Statistics for network device: enp0s8
        Successful packets:        10000
        Failed packets:            0
        Truncated packets:         0
        Retried packets (ENOBUFS): 0
        Retried packets (EAGAIN):  0

Once you have injected the data into your network, wait a minute for the data to refresh and then you can view the detection method by logging into Kibana then going to Dashboard and select the dashboard “DNS Tunneling Detection”.

Dashboard Selection

How the Dashboard should look

When you are done you can delete the “injected” data from your Elasticsearch/ELK (so we won’t continuously have a false positive from the test data). This will NOT delete any other data you have.

You can either perform this from the CLI or from Kibana “Dev Tools” as shown below:

Delete injected data

POST /bro-*/_delete_by_query
{
    "query": {
        "bool": {
            "must": [
                 { "match": { "domain_1n2n3_name": "sirknightthe.chickenkiller.com" } }
            ]
        }
    }
}

7. Additional Detection/Hunts

Because we added additional metadata/enrichment to our Logstash pipeline, we can detect additional low hanging fruit in DNS, HTTP, and SSL logs.

Detect IDN/Punycode/Emoji Domains

IDN homograph attacks made some news when they were used to trick users when visiting domains they thought were owned by Apple.

domain.is_idn:true OR domain.has_non_ascii:true

IDN/Punycode domains

Detect Domains that are an IP Address in SSL or HTTP

domain.ends_with_int:true AND domain_type:(ssl OR http)

Detect Domains that would not be a valid IP or Domain in SSL or HTTP

domain.has_dot:false AND domain_type:(ssl OR http)

After you try the dashboard and visualizations I would love to hear any feedback and recommendations you may have.