GoAccess custom forwarded log parsing - goaccess

I am currently using goaccess-1.0.2. I have installed it on an Amazon Linux box. The box which it resides has customized logs that were forwarded from an Apache WebApp Server. What I have tried to accomplish but can't seem to figure out is how to get GoAccess to parse our customized log.
Here is an example of the custom forwarded WebApp Log entry:
Jun 24 00:00:41 directory1 httpd-access: 55.117.170.95 www.URLaddress.com - [24/Jun/2016:00:00:41 -0700] "GET /sites/all/themes/somthing_on_demand/js/fancybox/jquery.fancybox-1.3.4.css HTTP/1.1" 304 - "ht
tps://www.IPaddress.com/my_account/yum" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" "SESSb9948a0b21e4d377a7d82f6adbf86c91=l
on7pgjlikml7q4tq954ejiao1; cookie_js=1; __utma=23285183.1119616966.1452095139.1468883973.1468963151.39; __utmb=23285183.500.10.1468963151; __utmc=23285183; __utmz=23285183.1468963151.39.39.utmcsr=fyi.URLaddress.com|utm
ccn=(r/INFOSEC-MAXLEN-256" "-" 57630
Here are a few log-formats I have tried:
log-format %^ %^ %^ "%h %^ %u %t \"%r\" %>s %b \"%R\" \"%u\" \"%^\" \"%^\" %D"
log-format "%h %{Host}i %{SSL_CLIENT_S_DN_CN}x %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" \"%{SHORT_COOKIE}e\" \"%{X-Forwarded-For}i\" %D"
log-format "%h %{Host}i %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" \"%{SHORT_COOKIE}e\" \"%{X-Forwarded-For}i\" %D"
I thought I would ignore the date and time format using %^ then use date format %m %d and time format %T .
I am very new at this and could really use help. Thank you for your feedback in advance.

Please try this, it works for me:
goaccess -f access.log --log-format='%^:%^:%^: %h %v %^[%d:%t %^] "%r" %s %b "%R" "%u" "%^" "%^" %D' --date-format='%d/%b/%Y' --time-format='%T'

Related

Show subdomains in apache logs

I would like the subdomains to show up in the my apache logs.
At the moment a request for:
abc.website.com/doc1.html
def.website.com/doc2.html
show up in my logs only as:
/doc1.html
/doc2.html
I was looking for an option at:
http://httpd.apache.org/docs/2.2/mod/mod_log_config.html#formats
but i found nothing related to the subdomains.
(Pleas excuse my English.)
Put a %V in your LogFormat line, for example:
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" %V"

understanding apache logs request/response time and size

I need to get the request size, response size, and response time from apache logs. Here is the one sample entry
2018-05-21T23:58:49-0700 112.135.80.119 10.143.112.99 10.143.112.99 GET "anb.com" "/abc/metadata" "?indexversion=4&token=MTM1ODlfNjQzXzY0NTM0Nw%3d%3d&countrycode=USA&version=26.0.4.12" "-" "-" 200 1059801 3103300 - 204 1060812
From the apache conf it looks we are using combined format which is
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
But the log entry does not seem to be matching the format. Can someone help me understanding the log entry?

log apache2 ssl session id and unique client values

I want to realize a more detailed apache2 server log with the unique ssl_session_id of each user for forensic reasons. I adapted the LogFormat in my apache2.conf but it does not log the session id. using apache 2.2.22 - mod_ssl is enabled.
current LogFormat:
LogFormat "%{%a %m/%d/%Y # %I:%M:%S.}t%{msec_frac}t %{%p %Z}t %h \"%{SSL_SESSION_ID}e\" (%{X-Forwarded-For}i) > %v:%p \"%r\" %I %D %>s %b %k \"%{Referer}i\" \"%{User-Agent}i\" %u %{User}C %{SessionTracker}C" forensic
also tried:
LogFormat "%{%a %m/%d/%Y # %I:%M:%S.}t%{msec_frac}t %{%p %Z}t %h \"%{SSL_SESSION_ID}x\" (%{X-Forwarded-For}i) > %v:%p \"%r\" %I %D %>s %b %k \"%{Referer}i\" \"%{User-Agent}i\" %u %{User}C %{SessionTracker}C" forensic
A log line looks like:
Fri 05/20/2016 # 09:40:33.msec_frac AM CEST 0.0.0.0 "-" (-) > example.com:443 "GET /path/to/the/file.svg HTTP/1.1" 837 440 304 - 35 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0" - - -
(IP addr, URL, ref have been adapted)
How can i retrieve the ssl_session_id. Is there any other way to gather (any other) unique client-information for forensic analysis?
well it seemend that apache v.2.2.22 wasn't able to deactivate ssl_ticket_session. So i had to do this manually to be able to log the ssl_ID which wasn't part of the client_header each time if you had the ticket_session activated. with the ticket_session deactivated the client had to "handshake" each time.

Log a call's execution time using wamp

Is there a way to log how long a call takes, as currently it only logs the time the call was made? I need the actual duration.
Thanks
Well one way to do it is using the already existing Apache logging feature.
So first check that you have this module activated in httpd.conf LoadModule log_config_module modules/mod_log_config.so it probably is.
Then change your LogFormat parameter to add these 2 new options
So if you start with
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
Then change it to
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" $$%T/%D$$" combined
Then make sure that the CustomLog parameter is using the newly changed combined output.
CustomLog "c:/wamp/logs/access.log" combined
All the directories in this example assume you are using WAMPServer, if not dont use the actual directories I have in this example.
%T shows The time taken to serve the request, in seconds.
%D shows The time taken to serve the request, in microseconds
This will give you the extra parameters at the end of each line in the access.log like so
mypc - - [24/Feb/2015:11:10:58 +0000] "GET /index.php?img=favicon HTTP/1.1" 200 1429 "-" "Mozilla/5.0 (Windows NT 6.1; rv:35.0) Gecko/20100101 Firefox/35.0" $$0/5000$$
You can change the $$ symbols I used here to whatever you like.

goaccess parsing not working

I'm trying to get goaccess working.
Already take some time on docs and examples but without success.
Basically i have log lines like:
10031488_71334 xpto.domain.com 88.103.999.999 - - [16/Jun/2013:15:03:26 +0000] "GET / HTTP/1.1" 500 5624 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0" -
and already compile goaccess and create a conf file:
cat ~/.goaccessrc
date_format %d/%b/%Y:%T %z
log_format %^ %^ %h - - [$d] "%r" %s %b "%R" "%u" %^
Then i got:
./goaccess -a -f miniout.log
GoAccess - version 0.6 - Aug 5 2013 20:20:16
An error has occurred
Error occured at: goaccess.c - render_screens - 358
Message: Nothing valid to process.
Any help?
Thanks.
If you don't want to use the global options use --no-global-config option with goaccess.
I am using this to process my log files on a day by day basis:
grep --color=auto `date +"%d/%b"` /var/log/nginx/sitename.access.log | goaccess --no-global-config > report.html
goaccess version : 0.8
Solved by log format change to:
log_format %^ %^ %h %^[%d:%^] "%r" %s %b "%R" "%u"
date_format %d/%b/%Y
First, review you log configuration in httpd.conf or into the sites configuration files for vhost.
then follow the next logics.
combinedvhost apache log format configuration:
%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\
to goaccess:
apache - what is? - how in goaccess
%v - vhost - %^ ignore it
%h - host - %h
%l - hyphen - %^ ignore it
%u - user - %^ ignore it
%t - timestamp - [%d:t% %^] ignore the zone from apache log
%r - request - %r
%s - status - %s
%b - size - %b
%{referer}i - request header - %R
%{UAgent}i - user agent - %u
goaccess.conf
log-format %^ %h %^ %^[%d:%t %^] "%r" %s %b "%R" "%u"
Note that in %r, %R and %u you will need "", because those are text fields (i think)
Note that between %u and date, there is not space (i do not know why...)
with:
time-format %H:%M:%S
date-format %d/%b/%Y
Reference:
https://httpd.apache.org/docs/2.2/logs.html
http://goaccess.io/man

Resources