Best Practices for Log Collection and Management in the UK

In this article, we will delve into best practices concerning log management, storage, and safeguarding. Grasping these best practices is crucial for enhancing log searches and bolstering your efforts in pinpointing and addressing security incidents.

Logs

Logs are records of events occurring within an organisation’s systems, generated by data sources such as devices. Initially, logs were primarily used for troubleshooting technology issues. For example, system error logs could reveal the causes of malfunctions, aiding in their resolution. Nowadays, logs from almost all computing devices offer valuable insights beyond mere troubleshooting.

Security teams utilise log receivers like SIEM (Security Information and Event Management) tools to access logs. These tools consolidate logs, forming a centralised repository. Logs are instrumental in log analysis, aiding in uncovering the details about the 5 W’s of incident investigation: who, what, when, where, and why an incident occurred.

Types of Logs

Logs can vary depending on the data source. Common types include:

• Network Logs: Generated by network devices such as firewalls and routers.
• System Logs: Produced by operating systems like Windows or Linux.
• Application Logs: Emanate from software applications, detailing events within them, like in a mobile app.
• Security Logs: Originating from various devices or systems, these logs contain security-related data, like file deletions.
• Authentication Logs: Created with each authentication event, like a computer login.

Log Details

Logs typically include a date, time, location, action, and action’s author. For instance, an authentication log might read:

“Login Event [06:30:25] User123 Successfully Logged In”

Verbose logging captures more detailed information than standard logs. A verbose authentication log might include the exact time, location, and method of authentication.

Log Management

With the vast number of logs generated by devices, organisations need effective log management, which includes collecting, storing, analysing, and disposing of log data.

What to Log

Deciding what to log is a pivotal aspect of log management. Organisations must balance the need for information with the risks of overlogging. Logging excessive data, like personal details (e.g., phone numbers, names), can breach data protection laws like the UK’s GDPR.

The Problem of Overlogging

Overlogging can be counterproductive, leading to increased storage costs and system overloads, making it difficult to identify critical events.

Log Retention

Different sectors have specific log retention requirements. For instance, the public sector may adhere to regulations like the UK’s Data Protection Act, while healthcare organisations might need to follow the NHS’s data retention guidelines.

Log Protection

Protecting logs is essential to ensure their integrity. Storing logs on a centralised server, separate from local machines, can reduce the risk of tampering.

Conclusion

Effective log management is key to successful incident investigations. A well-structured log management strategy enhances the usefulness of logs and optimises resource efficiency.

File formats

Understanding various log file formats is vital. Logs are critical in providing insights into activities across an organisation, like login events in applications. Proficiency in log analysis, which involves examining logs to spot significant events, requires familiarity with different log formats. This guide reviews several key log formats:

• JSON
• Syslog
• XML
• CSV
• CEF

JavaScript Object Notation (JSON)
JSON is a lightweight, readable file format for data storage and transmission, often used in web technologies and cloud environments. Its syntax, derived from JavaScript, includes:

• Key-Value Pairs: A key-value pair is a data set linking two items: a key and its corresponding value. For example, "User_Type": "Admin".
• Commas: Used to separate data, e.g., "User_Type": "Admin", "Access_Level": 5.
• Double Quotes: Enclose text data, e.g., "Error_Message": "Invalid login".
• Curly Brackets: Enclose an object, which stores data in a list of key-value pairs, e.g., { "User": { "ID": "001", "Name": "Alice" } }.
• Square Brackets: Enclose an array, a list of ordered data, e.g., ["Admin", "User", "Guest"].

Syslog
Syslog is a versatile standard for logging and transmitting data, utilised in three main ways:

• Protocol: Transports logs to a centralized server using specific ports.
• Service: Acts as a log forwarding service.
• Log Format: Commonly used in Unix systems, consisting of a header, structured-data, and a message.

An example syslog entry is:

<34>1 2023-06-15T10:30:00Z myserver app1 - ID123 [authResult@45678 result="Success" userID="A123"] User A123 logged in successfully.

• Header: Includes timestamp, hostname, application name, and message ID.
• Structured-Data: Additional logging info in key-value pairs.
• Message: Detailed log message about the event.

XML (eXtensible Markup Language)
XML, a native format in Windows systems, is used for data storage and transmission. It uses:

• Tags: To store and identify data, e.g., <UserName>John</UserName>.
• Elements: Both the data inside a tag and the tags themselves.
• Attributes: Provide additional information about elements, e.g., <File name="log.txt" size="14KB"></File>.

CSV (Comma Separated Value)
CSV logs use commas to separate data values. An example is:

"2023-06-15", "Login Attempt", "User123", "Failed"

CEF (Common Event Format)
CEF structures data using key-value pairs. A typical CEF log entry looks like:

CEF:0|AcmeCorp|Firewall|5.4|100|Attempted Breach Detected|High|src=192.168.1.1 dst=10.2.3.4

• Syslog Prefix: Includes timestamp and hostname (optional).
• Version: Indicates the CEF format version.
• Device Details: Vendor, product, and version.
• Signature ID and Name: ID and name of the event.
• Severity: Event urgency level.
• Extension: Additional data in key-value format.