grep, awk, wc, and such

[Uturn]

I have a firewall sending logs to an RPi syslog.

2020-09-26T21:37:10.723358-04:00 172.18.1.10 date=2020-09-26 time=21:37:10 devname="FWF60FTK20001328" devid="FWF60FTK20001328" logid="0000000020" type="traffic" subtype="forward" level="notice" vd="root" eventtime=1601170630 srcip=172.18.1.45 srcport=11471 srcintf="lan" srcintfrole="lan" dstip=54.86.131.217 dstport=443 dstintf="wan1" dstintfrole="wan" poluuid="b8f7ffdc-fdce-51ea-ebb2-c3396d04084e" sessionid=103496 proto=6 action="accept" policyid=3 policytype="policy" service="HTTPS" dstcountry="United States" srccountry="Reserved" trandisp="snat" transip=174.99.24.180 transport=11471 duration=108186 sentbyte=578200 rcvdbyte=851567 sentpkt=10222 rcvdpkt=5945 appcat="unscanned" sentdelta=590 rcvddelta=360

I am trying to find a way to get a query of srcip, dstip, action=!allow and a count of dstport

[bacchi]

egrep will let you use regex. -o will look for only the match.

grep -E -o "srcip=([0-9]{1,3}[\.]){3}[0-9]{1,3}" foo.log

You can then use a | for multiple regex.

grep -E -o "srcip=([0-9]{1,3}[\.]){3}[0-9]{1,3}|dstip=([0-9]{1,3}[\.]){3}[0-9]{1,3}" foo.log

For the count in the same regex? Got me. Make a shell script and add a 2nd line, "grep -c dstport foo.txt"?

[PDXTabs]

For stuff like that I usually just use python (or back in the day, perl).

[sherr]

Lol, StackOverflow is probably a better forum for questions like these.

So you want the report to include unique (srcip, dstip) and a count for disallowed actions? Agreed that something like a python script is going to be the easiest thing. It's probably possible to do it in awk, but I certainly couldn't tell you how off the top of my head. Perhaps something like this (untested):

Code: [Select]

from collections import defaultdict
disallowed_count = defaultdict(int)  # new entries in the dicitonary initialize the count at 0
with open('input.log', 'r') as f:
    for line in f:
        src, dst, action = None, None, None
        for keyvalue in line.split():
            key, _, value = keyvalue.partition('=')
            if key == 'srcip':
                src = value
            elif key == 'dstip':
                dst = value
            elif key == 'action':
                action = value
        if src and dst and action and action == '!allow':
            disallowed_count[(src, dst)] += 1

for (src, dst), count in disallowed_count.items():
    print(src, dst, count)


[Uturn]

I messed with it a bit more yesterday, then just decided to export to csv and create a pivot table in Excel.