Menu

RHSA: Analyzing and Storing Logs

No Comments

Linux Logging and System Logs

What Are Logs in Linux?

  • Logs = System’s “diary” recording events like:
    • User logins, service start/stop, crashes, security checks.
  • Stored mainly in /var/log directory as plain text files.

Why Logs Matter

  • Troubleshooting:
    • Identify what happened, when, and why (e.g., failed login, system crash).
  • Auditing & Security:
    • Track user activity, file access, configuration changes.
  • Compliance & Monitoring:
    • Essential for enterprise environments and centralized logging.

Common Log Files

  • /var/log/messages: General system messages.
  • /var/log/secure: Authentication events (logins, SSH, sudo).
  • /var/log/cron: Scheduled tasks (Cron jobs).
  • /var/log/boot.log: Boot process messages.
  • /var/log/dmesg: Kernel and hardware-related messages.

Log Management Services

  • rsyslog: Traditional logging daemon.
  • journald: Modern systemd-based logging.
  • journalctl: Command to view logs managed by journald.

Real-Life Use Cases

  • Failed login → Check /var/log/secure for “failed password”.
  • System crash → Check /var/log/messages for repeated warnings.
  • Security audit → Identify login times, IP addresses, commands executed.

Best Practices

  • Monitor regularly: Don’t wait for issues; review logs proactively.
  • Protect logs: Set proper permissions (contain sensitive info).
  • Rotate logs: Use logrotate to manage size and archive old logs.
  • Search efficiently:
    • Use grep, less, and journalctl filters for quick navigation.
  • Centralized logging:
    • Collect logs from multiple systems for security and redundancy.

rsyslog Architecture

What is rsyslog?

  • Definition: A fast, flexible logging service used in most Linux systems.
  • Purpose:
    • Collects log messages from kernel, services, daemons, and applications.
    • Stores logs in appropriate files (e.g., /var/log/messages, /var/log/secure).
    • Can filter, forward, and send logs to remote servers.

How rsyslog Works

  • Input:
    • Receives messages from system processes, apps, or network.
  • Parsing & Filtering:
    • Classifies messages (error, warning, info).
  • Rules & Configuration:
    • Determines destination based on config rules.
  • Output:
    • Writes to local files or forwards to remote servers.

Configuration

  • Main file: /etc/rsyslog.conf
  • Additional configs: /etc/rsyslog.d/
  • Rule format examples:
    • *.info /var/log/messages → All facilities, level info or higher → /var/log/messages.
    • authpriv.* /var/log/secure → Authentication logs → /var/log/secure.

Log Collection Mechanism

  • Socket: /dev/log → Receives messages from services.
  • Kernel messages: /proc/kmsg.
  • Network logging: Can receive logs from other servers for centralized logging.

Advanced Features

  • Remote forwarding (centralized logging).
  • Database logging.
  • Message encryption.
  • Templates for custom log formats.
  • Alerts for specific messages.

Why rsyslog is Important

  • Handles thousands of messages silently.
  • Essential for troubleshooting, monitoring, and auditing.
  • Critical for RHCSA: Know how it works, config file locations, and basic rule syntax.

Log File Location

Important Log Files in RHEL

  • /var/log/messages
    • General system activity, services, and kernel messages.
  • /var/log/secure
    • Authentication logs: login attempts, sudo usage, SSH.
  • /var/log/cron
    • Scheduled tasks (Cron jobs): execution time, user, command.
  • dmesg
    • Kernel and hardware messages, especially during boot.

Commands Demonstrated

  • Navigate to log directory:
    • cd /var/log
  • List files in human-readable format:
    • ls -lh
  • View logs with scrollable viewer:
    • less /var/log/messages
  • Follow live logs:
    • tail -f /var/log/messages
    • tail -f /var/log/secure
  • View Cron logs:
    • tail /var/log/cron
  • View kernel/boot messages:
    • sudo dmesg | less

Key RHCSA Concepts

  • Know log file locations and purposes.
  • Use less, tail, and tail -f for log viewing.
  • Understand dmesg for hardware and boot troubleshooting.
  • Live monitoring with tail -f is essential for real-time analysis.

journalctl Demonstration

journalctl Overview

  • Purpose: View and filter systemd-based logs in RHEL.
  • Default behavior: journalctl shows the entire system journal.

Key journalctl Filters & Options

  1. Basic View:
    • journalctl
  2. Filter by Time:
    • Logs from last 2 minutes:
      • journalctl –since “2 minutes ago”
    • Logs between two dates:
      • journalctl –since “2025-07-25” –until “2025-07-26”
  3. Filter by Unit (Service):
    • journalctl -u sshd
  4. Filter by Priority:
    • journalctl -p err
      • Displays error-level and higher severity logs.
  5. Filter by Boot Session:
    • List boots:
      • journalctl –list-boots
    • Current boot:
      • journalctl -b 0
    • Previous boot:
      • journalctl -b -1
  6. Real-Time Log Streaming:
    • journalctl -f
  7. Combine Filters:
    • Example: Errors for SSH service in current boot:
      • journalctl -b -p err -u sshd

RHCSA Exam Tips

  • Know journalctl syntax for filtering by:
    • Time (--since, --until)
    • Unit (-u)
    • Priority (-p)
    • Boot sessions (-b, --list-boots)
  • Practice real-time monitoring with -f.
  • Understand priority levels: emerg, alert, crit, err, warning, notice, info, debug.

Log rotation and Management

What is logrotate?

  • Purpose: Manages log file rotation to prevent disk space issues and keep logs organized.
  • Default behavior:
    • Rotates logs weekly.
    • Retains 4 weeks of backups.
  • Config files:
    • Main: /etc/logrotate.conf
    • Service-specific: /etc/logrotate.d/

Key Commands

  1. View main config:
    • cat /etc/logrotate.conf
  2. List service-specific configs:
    • ls /etc/logrotate.d/
  3. Force manual rotation:
    • logrotate -f /etc/logrotate.conf
  4. Verify rotated logs:
    • ls -lh /var/log/secure*

Custom Log Rotation Example

  • Create config file:
    • nano /etc/logrotate.d/customlog
# weekly rotate 4 compress missingok notifempty
/var/log/custom.log {
    weekly
    rotate 4
    compress
    missingok
    notifempty
}
  • Meaning:
    • weekly: Rotate every week.
    • rotate 4: Keep 4 old logs.
    • compress: Save space.
    • missingok: Skip if file missing.
    • notifempty: Rotate only if not empty.
  • Save & exit: Ctrl+X, Y, Enter.
  1. Create log file:
    • touch /var/log/custom.log
  2. Force rotation using custom config:
    • logrotate -f /etc/logrotate.d/customlog

RHCSA Exam Tips

  • Know default rotation settings and config file locations.
  • Understand custom rotation options (weekly, rotate, compress, etc.).
  • Practice manual rotation and verification.
  • Be familiar with logrotate.d for service-specific rules.

Persistent Journal Storage

Goal

  • Make systemd journal logs persistent across reboots (default is volatile).

Steps

  1. Check current disk usage:
    • sudo journalctl –disk-usage
      • Shows space used by journal logs.
  2. Verify current storage location:
    • ls -ld /run/log/journal
      • Logs stored in /run/log/journal → temporary (RAM).
  3. Create persistent log directory:
    • sudo mkdir -p /var/log/journal
      • /var/log/journal is the path for persistent logs.
  4. Prepare directory for systemd:
    • sudo systemd-tmpfiles –create –prefix /var/log/journal
  5. Restart journald service:
    • sudo systemctl restart systemd-journald
  6. Verify disk usage again:
    • journalctl –disk-usage
  7. Confirm persistent directory:
    • ls /var/log/journal/
      • If empty, reboot system to activate persistent logging.

Key RHCSA Concepts

  • Default journal logs are volatile → stored in /run/log/journal.
  • Persistent logs require /var/log/journal directory.
  • Use systemd-tmpfiles and restart systemd-journald.
  • Persistent logging is crucial for auditing and troubleshooting.

Implementation Log Analysis

Goal

Analyze system logs to detect:

  • Suspicious activity
  • Failures
  • Performance bottlenecks

Key Commands & Use Cases

1. Scan for Errors

grep -i error /var/log/messages
  • Searches for “error” (case-insensitive) in main system log.

2. Check Failed Authentications

grep -i fail /var/log/secure
  • Finds failed login attempts or service failures.

3. Audit Password Changes

sudo ausearch -x passwd
  • Shows all executions of passwd command.

4. Audit Current User Activity

ausearch -ua $(whoami)
  • Displays login attempts for the current user.

5. View Login History

last
  • Lists recent user login sessions.

6. View Failed Logins

lastb
  • Shows bad login attempts.

7. Kernel Errors

journalctl -k | grep -i error
  • Filters kernel messages for errors (e.g., disk or hardware issues).

8. Performance Monitoring

top
  • Live view of CPU, memory usage, and running processes.

RHCSA Exam Tips

  • Know grep for filtering logs.
  • Understand audit tools (ausearch) for security checks.
  • Use journalctl for kernel and systemd logs.
  • Practice last/lastb for login history.
  • Monitor performance with top.

Configuring Forwarding Logs

Goal

Configure rsyslog to forward logs to a remote log server for centralized logging.

Steps

1. Enable and Start rsyslog

sudo systemctl enable --now rsyslog
  • Ensures rsyslog starts immediately and on every boot.

2. Configure Remote Logging

  • Edit main config: (Server)
sudo nano /etc/rsyslog.conf
  • Add at the bottom:
# Provides TCP syslog reception
# for parameters see http://www.rsyslog.com/doc/imtcp.html
module(load="imtcp") # needs to be done just once
input(type="imtcp" port="514")
*.* @192.168.0.100:514
  • Meaning:
    • *.* → All facilities and priorities.
    • @192.168.0.100:514 → Remote log server IP and UDP port.
  • Save and exit: Ctrl+X, Y, Enter.

3. Restart rsyslog

sudo systemctl restart rsyslog

4. Configure Firewall

  • Allow UDP port 514:
sudo firewall-cmd --add-port=514/udp --permanent
  • Reload firewall:
sudo firewall-cmd --reload
  • On CLient:
    • sudo rsyslogd -b OR systemctl status rsyslog.service
      • to check if syslog in installed and running
    • Modify the default logging rules:
      • sudo vi /etc/rsyslog.d/50-default.conf
        • add this line to Default rules: *.* @@central-log-ser-ip:514
      • restart service: systemctl restart rsyslog.service
    • Testing, run command: logger “Hi from client”
    • check if log entry is in syslog file: tail /var/log/syslog
    • On server check in /var/log if it has created a new folder
      • run command to see the files created: ls /var/log/client-vm

Key RHCSA Concepts

  • rsyslog can forward logs to remote servers for centralized monitoring.
  • Config file: /etc/rsyslog.conf.
  • Syntax for forwarding: *.* @<remote-IP>:514.
  • Ensure firewall rules allow traffic on port 514 (UDP).
  • Restart rsyslog after changes.

Leave a Reply

Your email address will not be published. Required fields are marked *

About

Enter your site info here.

Recent Posts