Defense
Knowing where to look is a real important piece of the incident response puzzle. With a large number of incidents involving web servers, I figured it was a good idea to talk about some of the common log files, their locations and some gotchas. We are going to dive into some tech 101 then follow up with how this ties into the Incident Response process (so hopefully this helps if you re more PowerPoint than Bash).
Why do we care about where the default paths are? Well hopefully if you have planned ahead and got a security monitoring solution you won’t have to. But all things start from acorns. A good way to start to understand how logs and incident response tie together are to understand what is needed under the hood. This isn’t a deep dive but more a glimpse. When we visit a web page the webserver should be configured to capture the access logs. These logs are really helpful in an incident involving web services, so where can we find them?
Web Server Default Log Paths
Different platforms store logs in different locaitons, you can also customise this. However by default we have the following for the top web platforms in use today:
Apache
/var/log/apache2/access
Nginx
/var/log/nginx/
Internet Information Services (IIS)
%SystemDrive%\inetpub\logs\LogFiles
Cloud Services
Now in today’s age we need to also be mindful of PaaS logs e.g., Amazon S3 Static Web Services Logs.
Ensuring logging is enabled before an incident is key. It is common for people to forget this part when building cloud solutions by assuming of all the logs will be there, they will not unless you enable them.
Access Log Visualisation
Now just trawling through log files in raw format is fun and all, but thwat we really want to be able to to is spot patterns and trends. We also need to be able to help explain to people who aren’t quite as “cyber” as others. Now for UNIX style web access logs there’s logstalgia.
This tool is really tool. It works great with access logs for apache and nginx. It is not so great for IIS logs (you can however modify the logs for IIS to fit)
Thanks to Cary (@figjam) for this!
Log File Analysis
Analysing log files is not an easy task, however there are some tools to help you along your way. Ideally you do not want to be deploying tools mid incident. You should have tools deployed as part of your security operations monitoring capabilities. If you are however responding and do not have tools in place, there’s a few tools which can really help. My favourite for rapid deployment is Splunk. https://www.splunk.com/
There are however other tools such as:
Graylog
Elasticsearch, Logstash and Kibana (ELK)
HTTP response codes
A key part of inspecting web services is knowing what the response codes mean:
| Code | Description |
| 1xx | Informational |
| 2xx | Success |
| 3xx | Redirects |
| 4xx | Client Errors |
| 5xx | Server Errors |
https://developer.mozilla.org/en-US/docs/Web/HTTP/Status
Application and Database Events
Outside of web server platform logs you are going to want to look for custom application logs. Think here about:
- Logon Data (e.g., time/date of last logon, last password change, last logon source IP address)
- User Activity Logs (date of record changes)
Incident Response Considerations
Some key considerations you should make for incident response:
- Are adequate logs available (remember sometimes you will not know this straight away)?
- Do the systems record the correct data to be able to piece together an end-to-end story?
- Are timestamps in a universal standard format?
- Do the responders have access to the data?
- How do we ensure the data is immutable (think two copies)?
- How long will it take to copy the log volume to the analysis environment (this could be significant)?
- How long will it take to analyse the log files (again this could take quite a bit of time)?
- Are you being realistic with expectations between incident discovery and analysis, people often think magic can be performed (it cannot!)?
- Are you clear in your priorities between service restoration and investigation? Remember you do not want to repeat the same incident twice so understanding the vectors, causes and impacts are key to enabling the incident to return value in terms of future improvements.
Getting Started
There’s a ton of resources online, I put together an incident response planning workbook (https://www.pwndefend.com/2020/08/16/have-you-planned-to-fail/) but there’s loads of resources from NCSC, NIST, SANS etc.
If you don’t have an centralised logging there’s a cool solution accelerator from NCSC called “Logging Made Easy” – https://www.ncsc.gov.uk/blog-post/logging-made-easy
Also if you need a kick start for IR planning from a tabletop perspective there’s the NCSC excercise in a box:
https://www.ncsc.gov.uk/information/exercise-in-a-box
Remember thought tools are tools, making this stuff work requires leadership, effort, commitment and practise.
Summary
It’s important to plan ahead for an incident, we too often find organisations have not done this part, they then invoke an incident response team and then are sometimes surprised that they can’t magic data from nowhere, or that they can analyse the scenario in minutes. It’s key to remember you need a range of logs including:
- Network
- Operating System
- Platform
- Cloud Services
- Applications and Database Logs
Some of these may require custom development for your application and use cases. Whichever course you take, having the data is not only going to help you enable operational efficiency and reduce risk, but also to help you respond effectively if you have tried and tested plans in place.
