Structured JSON Logging in Nginx
by Tykling
27. apr 2020 21:33 UTC
Webserver log analysis has been a sore point in the opensource world for years. People just use Google Analytics or whatever so noone is working on tools for this, and I got properly fed up with the old perl monsters like AWStats. So I haven't really done any serious attempts at analyzing the logfiles from my webservers for some years, but recently I started playing around with goaccess which is pretty nice.
However, analyzing logfiles requires that you have a log format which can be parsed of course, regardless of the software you use for it. I currently use nginx for both frontend and backend webserver, although I am mainly interested in analyzing the logs from the frontend servers (which terminate TLS and forward requests to upstream/backend servers in different jails).
Nginx has a default logformat named combined which looks like this:
log_format combined '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent"';
The format I've been using for my Nginx proxies has more information than this, and it has also changed over the years. This means that any log analysis I want to do has to parse more than one format, and some of them are not as easy to parse as others. For example I started logging the HTTP client portnumber back in April 2017, but I didn't add brackets around the IP until recently, meaning it is not trivial to regex what is the IP and what is the port, in case of an IPv6 client.
The log format might also change in the future, the next time I think of something extra that would be nice to have in the logs. Then I'd have to adjust the regex again, and on and on it goes. Having to parse logs with regular expressions is just not the correct way to do this. A structured format would be a much better choice, as adding fields and parsing would be trivial.
Nginx JSON Logging
nginx doesn't really support JSON logging where you just tell it to output json and which variables you'd like in it. But it does support escaping values to be contained in JSON, with the escape=json parameter for the log_format config line. My logformat has been called timing until now, and the most recent iteration of it looks like this:
log_format timing '$time_local - $http_host - [$remote_addr]:$remote_port - $request - HTTP $status - upstream_response_time $upstream_response_time - total_request_time $request_time - tls: $ssl_protocol/$ssl_cipher - "$http_user_agent" - request_id: $request_id - upstream: $upstream_addr - cache: $upstream_cache_status - request_length: $request_length - bytes_sent: $bytes_sent';
Since the switch to JSON logging gives me a golden opportunity to add some more fields without breaking anything I did so and ended up with this new log_format:
log_format timingjson escape=json '{'
'"time_iso8601": "$time_iso8601", ' # local time in the ISO 8601 standard format
'"msec": "$msec", ' # request unixtime in seconds with a milliseconds resolution
'"connection": "$connection", ' # connection serial number
'"connection_requests": "$connection_requests", ' # number of requests made in connection
'"request_id": "$request_id", ' # the unique request id
'"request_length": "$request_length", ' # request length (including headers and body)
'"request_time": "$request_time", ' # request processing time in seconds with msec resolution
'"remote_addr": "$remote_addr", ' # client IP
'"remote_port": "$remote_port", ' # client port
'"remote_user": "$remote_user", ' # client HTTP username
'"ssl_protocol": "$ssl_protocol", ' # TLS protocol
'"ssl_cipher": "$ssl_cipher", ' # TLS cipher
'"http_user_agent": "$http_user_agent", ' # user agent
'"http_referer": "$http_referer", ' # HTTP referer
'"http_host": "$http_host", ' # the request Host: header
'"server_name": "$server_name", ' # the name of the vhost serving the request
'"scheme": "$scheme", ' # http or https
'"request_method": "$request_method", ' # request method
'"request_uri": "$request_uri", ' # full path and arguments if the request
'"server_protocol": "$server_protocol", ' # request protocol, like HTTP/1.1 or HTTP/2.0
'"bytes_sent": "$bytes_sent", ' # the number of bytes sent to a client
'"status": "$status", ' # response status code
'"pipe": "$pipe", ' # “p” if request was pipelined, “.” otherwise
'"upstream": "$upstream_addr", ' # upstream backend server for proxied requests
'"upstream_connect_time": "$upstream_connect_time", ' # upstream handshake time incl. TLS
'"upstream_header_time": "$upstream_header_time", ' # time spent receiving upstream headers
'"upstream_response_time": "$upstream_response_time", ' # time spend receiving upstream body
'"upstream_cache_status": "$upstream_cache_status"' # cache HIT/MISS where applicable
'}';
access_log /usr/local/www/logs/proxytiming.json timingjson;
So while nginx doesn't support outputting JSON natively, it is possible to construct a log line which is valid JSON, and because of escape=json it will stay valid even if something like a useragent contains a " or whatever. I also changed the name of the log from proxytiming.log to proxytiming.json so I can easily tell them apart.
I also remembered to update my newsyslog.conf.d file for Nginx so it rotates the new JSON logs correctly:
[tykling@webproxy2 ~]$ cat /usr/local/etc/newsyslog.conf.d/nginx
# path mode copies size time flags pidfile signal
/usr/local/www/logs/proxytiming.log 644 1000 * @T00 JX /var/run/nginx.pid SIGUSR1
/usr/local/www/logs/proxytiming.json 644 1000 * @T00 JX /var/run/nginx.pid SIGUSR1
/usr/local/www/logs/error.log 644 1000 * @T00 JX /var/run/nginx.pid SIGUSR1
[tykling@webproxy2 ~]$
Note that I kept the old filename in there so newsyslog will keep rotating them. In 1000 days I can remove that bit of config after the last plaintext log has been rotated into oblivion :)
Result
Before making the change a typical logline (a hit from blackbox_exporter checking certificate lifetime) looked like this:
26/Apr/2020:00:01:15 +0000 - bornhack.dk - [2a01:3a0:1:1900:10::30]:30406 - GET / HTTP/1.1 - HTTP 302 - upstream_response_time 0.042 - total_request_time 0.043 - tls: TLSv1.2/ECDHE-RSA-AES256-GCM-SHA384 - "Go-http-client/1.1" - request_id: 606497b37973d62c0aba9acbf574dc97 - upstream: [2a01:3a0:1:1900:10::27]:80 - cache: - - request_length: 129 - bytes_sent: 530
After making the change to JSON logging a similar request looks like this:
{"time_iso8601": "2020-04-27T21:31:00+00:00", "msec": "1588023060.552", "connection": "268042", "connection_requests": "1", "request_id": "e56341b31fac51da3af458dd74988531", "request_length": "129", "request_time": "0.044", "remote_addr": "2a01:3a0:1:1900:10::30", "remote_port": "48897", "remote_user": "", "ssl_protocol": "TLSv1.2", "ssl_cipher": "ECDHE-RSA-AES256-GCM-SHA384", "http_user_agent": "Go-http-client/1.1", "http_referer": "http://[2a01:3a0:1:1900:81::1]", "http_host": "bornhack.dk", "server_name": "bornhack.dk", "scheme": "https", "request_method": "GET", "request_uri": "/", "server_protocol": "HTTP/1.1", "bytes_sent": "530", "status": "302", "pipe": ".", "upstream": "[2a01:3a0:1:1900:10::27]:80", "upstream_connect_time": "0.000", "upstream_header_time": "0.043", "upstream_response_time": "0.043", "upstream_cache_status": ""}
If we run it through a pretty formatter it becomes a lot more readable:
{
"time_iso8601": "2020-04-27T21:31:00+00:00",
"msec": "1588023060.552",
"connection": "268042",
"connection_requests": "1",
"request_id": "e56341b31fac51da3af458dd74988531",
"request_length": "129",
"request_time": "0.044",
"remote_addr": "2a01:3a0:1:1900:10::30",
"remote_port": "48897",
"remote_user": "",
"ssl_protocol": "TLSv1.2",
"ssl_cipher": "ECDHE-RSA-AES256-GCM-SHA384",
"http_user_agent": "Go-http-client/1.1",
"http_referer": "http://[2a01:3a0:1:1900:81::1]",
"http_host": "bornhack.dk",
"server_name": "bornhack.dk",
"scheme": "https",
"request_method": "GET",
"request_uri": "/",
"server_protocol": "HTTP/1.1",
"bytes_sent": "530",
"status": "302",
"pipe": ".",
"upstream": "[2a01:3a0:1:1900:10::27]:80",
"upstream_connect_time": "0.000",
"upstream_header_time": "0.043",
"upstream_response_time": "0.043",
"upstream_cache_status": ""
}
Lovely. It can be parsed by any language worth its salt, and if my log analysis tool of choice doesn't support JSON it will be easy to convert to something it does understand. Adding new fields will not break anything, and I already have much more information available than I had before.
Conclusion
Structured logs are great. I am considering doing the same for Postfix and a few others. When logs are structured it makes it trivial to analyze them, monitor them, put some values into a database, or whatever. Observe how easy it is to get some value out with the the great jq tool:
[tykling@webproxy2 ~]$ cat /usr/local/www/logs/proxytiming.json | grep -v "logfile turned over" | jq '[inputs] | .[].ssl_cipher' | sort | uniq -c | sort -rn
25506 "ECDHE-RSA-AES256-GCM-SHA384"
11092 ""
187 "ECDHE-RSA-AES128-GCM-SHA256"
104 "ECDHE-RSA-AES256-SHA384"
[tykling@webproxy2 ~]$
Or maybe check how many HTTP vs HTTPS requests today:
[tykling@webproxy2 ~]$ cat /usr/local/www/logs/proxytiming.json | grep -v "logfile turned over" | jq '[inputs] | .[].scheme' | sort | uniq -c | sort -rn
25817 "https"
11102 "http"
[tykling@webproxy2 ~]$
Or even useragents (the big one is Go-http-client/1.1 which as mentioned is my blackbox_exporter doing monitoring):
[tykling@webproxy2 ~]$ cat /usr/local/www/logs/proxytiming.json | grep -v "logfile turned over" | jq '[inputs] | .[].http_user_agent' | sort | uniq -c | sort -rn
32089 "Go-http-client/1.1"
611 "Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)"
377 ""
268 "NextCloud-News/1.0"
266 "Xymon xymonnet/4.3.17"
214 "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0"
213 "Mozilla/5.0 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)"
168 "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:28.0) Gecko/20100101 Firefox/28.0"
165 "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
131 "Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)"
128 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"
113 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36"
108 "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0"
106 "Mozilla/5.0 (X11; Linux x86_64; rv:74.0) Gecko/20100101 Firefox/74.0"
101 "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
99 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:47.0) Gecko/20100101 Firefox/47.0"
90 "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
88 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36"
78 "Wget/1.14 (linux-gnu)"
78 "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0"
71 "python-requests/2.22.0"
65 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36"
59 "ZoominfoBot (zoominfobot at zoominfo dot com)"
54 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.162 Safari/537.36"
50 "Mozilla/5.0 (X11; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0"
49 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:75.0) Gecko/20100101 Firefox/75.0"
45 "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36"
45 "Feedly/1.0 (+http://www.feedly.com/fetcher.html; 4 subscribers; like FeedFetcher-Google)"
43 "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)"
39 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/80.0.3987.163 Chrome/80.0.3987.163 Safari/537.36"
38 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:75.0) Gecko/20100101 Firefox/75.0"
33 "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.92 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
32 "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
31 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18363"
30 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36"
27 "Mozilla/5.0 (iPhone; CPU iPhone OS 13_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.5 Mobile/15E148 Safari/604.1"
27 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36 OPR/67.0.3575.137"
27 "Mozilla/5.0 (Linux; Android 7.0; TECNO IN5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.162 Mobile Safari/537.36"
26 "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:75.0) Gecko/20100101 Firefox/75.0"
26 "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.162 Safari/537.36"
26 "Mozilla/5.0 (Linux; Android 8.1.0; BBB100-2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.116 Mobile Safari/537.36"
24 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36"
23 "Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)"
23 "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) LinkCheck by Siteimprove.com"
22 "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
22 "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
21 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"
18 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36"
17 "Mozilla/5.0 zgrab/0.x"
17 "Mozilla/4.0 (compatible; MSIE 6.0; Windows XP)"
16 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36"
15 "Twitterbot/1.0"
15 "Mozilla/5.0 (X11; FreeBSD amd64; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36"
14 "Mozilla/5.0 (Windows NT 5.1; rv:9.0.1) Gecko/20100101 Firefox/9.0.1"
12 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 Edge/16.16299"
12 "Liferea/1.12.6 (Linux; https://lzone.de/liferea/) AppleWebKit (KHTML, like Gecko)"
11 "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
11 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)"
10 "python-requests/2.18.1"
10 "Mozilla/5.0 (Linux; Android 9; SHT-W09) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.111 Mobile Safari/537.36"
9 "Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0"
8 "Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://napoveda.seznam.cz/en/seznambot-intro/)"
8 "Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)"
8 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon"
8 "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.85 Safari/537.36"
8 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.4.2661.102 Safari/537.36; 360Spider"
8 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.93 Safari/537.36"
8 "Google-Calendar-Importer"
7 "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
7 "Mozilla/5.0 (X11; Linux x86_64; rv:75.0) Gecko/20100101 Firefox/75.0"
7 "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.7.0 Lightning/68.7.0"
7 "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.117 Safari/537.36"
7 "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; AspiegelBot)"
7 "Mozilla/2.0 (compatible; MSIE 2.1; Mac_PowerPC)"
6 "masscan/1.0 (https://github.com/robertdavidgraham/masscan)"
6 "Mozilla/5.0 (compatible; coccocbot-image/1.0; +http://help.coccoc.com/searchengine)"
6 "Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com)"
5 "python-requests/2.23.0"
5 "python-requests/2.10.0"
5 "Mozilla/5.0 zgrab/0.x (compatible; Researchscan/t13rl; +http://researchscan.comsys.rwth-aachen.de)"
5 "Cyberdog/2.0 (Macintosh; 68k)"
4 "polaris botnet"
4 "Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)"
4 "Mozilla/5.0 (compatible, MSIE 10.0, Windows NT, DigExt)"
4 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.137 Safari/537.36"
4 "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.117 Safari/537.36"
4 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.170 Safari/537.36 OPR/53.0.2907.99"
4 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11"
4 "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.170 Safari/537.36"
4 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"
4 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36"
4 "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.26 Safari/537.36 Core/1.63.5478.400 QQBrowser/10.1.1550.400"
4 "Lynx/2.6 libwww-FM/2.14"
4 "Feeder / 1.9.3(73)"
3 "Opera/9.0 (Windows NT 5.1; U; en)"
3 "Opera/7.60 (Windows NT 5.2; U) [en] (IBM EVV/3.0/EAK01AG9/LE)"
3 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b"
3 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36"
3 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36"
3 "Mozilla/4.7 (compatible; OffByOne; Windows 98) Webster Pro V3.2"
3 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts; .NET CLR 1.1.4322; PeoplePal 6.2)"
3 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322; Alexa Toolbar; (R1 1.5))"
3 "MoonBrowser (version 0.41 Beta4)"
3 "Evolution/3.28.5"
2 "newsbeuter/2.9 (Linux i686)"
2 "Wget/1.18 (linux-gnu)"
2 "Wget/1.17.1 (linux-gnu)"
2 "RSSOwl/1.2.4 Preview Release 2007-04-15 (Windows; U; zhtw)"
2 "Python-urllib/3.7"
2 "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0"
2 "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2b) Gecko/20021007 Phoenix/0.3"
2 "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 /68.8.0"
2 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36"
2 "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36"
2 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19041"
2 "Mozilla/5.0 ( compatible ; Veooz/1.0 ; +http://www.veooz.com/veoozbot.html )"
2 "Mozilla/4.77C-SGI [en] (X11; U; IRIX 6.5 IP32)"
2 "Mozilla/4.5b1 [en] (X11; I; Linux 2.0.35 i586)"
2 "Mozilla/4.0 (compatible; SiteKiosk 4.0; MSIE 5.0; Windows 98; SiteCoach 1.0)"
2 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
2 "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; AIRF)"
2 "Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; MSN Companion 2.0; 800x600; Compaq)"
2 "Mozilla/3.04 (compatible;QNX Voyager 2.03B ;Photon)"
2 "Mozilla/2.0 (compatible; MSIE 3.0; Windows 3.1)"
2 "Internet-structure-research-project-bot"
2 "Googlebot-Image/1.0"
2 "BlogBridge 2.13 (http://www.blogbridge.com/)"
1 "netEstate NE Crawler (+http://www.website-datenbank.de/)"
1 "lwp-trivial/1.35"
1 "iCab/2.5.2 (Macintosh; I; PPC)"
1 "annotate_google; http://ponderer.org/download/annotate_google.user.js"
1 "WinWAP/3.x (3.x.x.xx; Win32) (Google WAP Proxy/1.0)"
1 "Wget/1.19.4 (linux-gnu)"
1 "UP.Browser/3.01-IG01 UP.Link/3.2.3.4"
1 "Sylera/1.2.x"
1 "Sleipnir"
1 "Sleipnir Version2.x"
1 "Sleipnir Version 1.xx"
1 "SimpleFavPanel/1.2"
1 "Ruby"
1 "Pleroma 2.0.2; https://pleroma.gidikroon.eu "
1 "Orca Browser (http://www.orcabrowser.com)"
1 "Opera/9.00 (Windows NT 5.1; U; de)"
1 "NewsGatorOnline/2.0 (http://www.newsgator.com; 1 subscribers)"
1 "NewsGator FetchLinks extension/0.2.0 (http://graemef.com)"
1 "NSPlayer/10.0.0.xxxx WMFSDK/10.0"
1 "Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)"
1 "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0"
1 "Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.7.6) Gecko/20050405 Epiphany/1.6.1 (Ubuntu) (Ubuntu package 1.0.2)"
1 "Mozilla/5.0 (X11; U; Linux i686; de-AT; rv:1.8.0.2) Gecko/20060309 SeaMonkey/1.0"
1 "Mozilla/5.0 (X11; Linux x86_64; rv:77.0) Gecko/20100101 /77.0a1"
1 "Mozilla/5.0 (X11; Linux x86_64; rv:76.0) Gecko/20100101 /76.0"
1 "Mozilla/5.0 (X11; Linux x86_64; rv:39.0) Gecko/20100101 Firefox/39.0"
1 "Mozilla/5.0 (X11; Linux x86_64)"
1 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3602.2 Safari/537.36"
1 "Mozilla/5.0 (X11; CrOS x86_64 12871.76.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.103 Safari/537.36"
1 "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:52.9) Gecko/20100101 Goanna/3.4 Firefox/52.9 PaleMoon/27.8.1"
1 "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0"
1 "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)"
1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0"
1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36"
1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36"
1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"
1 "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"
1 "Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.0.1) Gecko/20021219 Chimera/0.6"
1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/603.1.30 (KHTML, like Gecko) Version/10.1 Safari/603.1.30"
1 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/601.7.7 (KHTML, like Gecko) Version/9.1.2 Safari/601.7.7"
1 "Mozilla/4.76 [en] (X11; U; FreeBSD 4.4-STABLE i386)"
1 "Mozilla/4.05 (Macintosh; I; PPC Nav)"
1 "Mozilla/4.02 [en] (X11; I; SunOS 5.6 sun4u)"
1 "Mozilla/4.0 (compatible; ibisBrowser)"
1 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; iOpus-I-M)"
1 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 8.50"
1 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Embedded Web Browser from: http://bsalsa.com/; MSIECrawler)"
1 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
1 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; KKman3.0)"
1 "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; N_o_k_i_a)"
1 "Mozilla/4.0 (compatible; MSIE 5.0; Mac_PowerPC; AtHome021)"
1 "Mozilla/4.0 (compatible; MSIE 4.01; Mac_PowerPC)"
1 "Mozilla/3.04 (compatible; NCBrowser/2.35; ANTFresco/2.17; RISC OS-NC 5.13 Laz1UK1309)"
1 "Mozilla/3.01SGoldC-SGI (X11; I; IRIX 6.3 IP32)"
1 "Mozilla/3.01Gold (X11; I; Linux 2.0.32 i486)"
1 "Mozilla/3.01-C-MACOS8 (Macintosh; I; PPC)"
1 "Mozilla/3.0 (Planetweb/2.100 JS SSL US; Dreamcast US)"
1 "Mozilla/2.0 (compatible; MS FrontPage x.0)"
1 "Lunascape"
1 "Klondike/1.50 (WSP Win32) (Google WAP Proxy/1.0)"
1 "Kazehakase/0.x.x.[x]"
1 "Jakarta Commons-HttpClient/3.0-rcx"
1 "Jakarta Commons-HttpClient/2.0xxx"
1 "FeedZcollector v1.x (Platinum) http://www.feeds4all.com/feedzcollector"
1 "ExactSearch"
1 "Evolution/3.28; Evolution-RSS/0.3.95"
1 "Commerce Browser Center"
1 "Aplix HTTP/1.0.1"
1 "AU-MIC/2.0 MMP/2.0"
1 "ANTFresco/x.xx"
[tykling@webproxy2 ~]$
Just lovely. Until next time: wash your hands and stay safe!