Linux Troubleshooting

https://linuxwheel.com/using-log-and-journal-files/

journalctl

journalctl is a logging service similar to a syslog. The command journalctl can be used to display failures or errors from specific services.

Logs collected by systemd can be viewed by using journalctl. The journal is implemented with the journald daemon and it retrieves messages from the kernel, systemd services, and other sources. These logs are gathered in a central location, which makes it easy to review. The log records in the journal are structured and indexed. As a result, journalctl is able to present your log information in various useful formats.

Configure journal

By default, journal logs are enabled and stores log data at /run/log/journal/. But, since logs are deleted automatically after a system reboot, you will need to configure Journal to store all logs permanently.

mkdir /var/log/journal      # create directory to store permanent journal log
chown -R root:systemd-journal /var/log/journal      # set proper permission
vi /etc/systemd/journald.conf       # update configuration file
    Storage=persistent

systemctl restart systemd-journald      # restart journald service

journalctl commands

# View latest systemd log information
    journalctl

# View boot messages
    journalctl -b
    journalct --list-boots  # Find out how many boots have happened
Note:
    0   current boot
        journalctl -b
    -1  previous boot
        journalctl -b -1
    -2  other previous boot
        journalctl -b -2

# View service logs
    journalctl -u <service>
        Example: journalctl -u ssh.service
                 journalctl -u nginx.service

# View logs with date range
    journalctl --since "1 hour ago"
    journalctl --since yesterday
    journalctl --since 06:00 --until "1 hour ago"
    journalctl --since "2022-08-01 09:00:00" --until "2022-08-01 14:00:00"
    journalctl -u <service> --since "2022-08-01 09:00:00" --until "2022-08-01 14:00:00"

# View output in different format
    journalctl -o json
    journalctl -o verbose
---------------------------------------------------------
Format      Description
-----------------------
json        json format
json-pretty     in easy to read json format
verbose     detailed information for each entry
cat         in very short form
shortis     defaut, syslog, output format
short-monotonic     similar to short, but include time stamp value
---------------------------------------------------------

# view most recent log entries
    journalctl --lines 10       # view last 10 entries
    journalctl -n 10

# To print the log continously
    journalctl --follow
    journalctl -f
    # read live tail logs of multiple services
    journalctl --follow _SYSTEMD_UNIT=docker.service + _SYSTEMD_UNIT=apache2.service

# View kernel related logs
    journalctl -k

# Display logs which contain error or critical
    jorunalctl -p err -b

# Display the amount of space used by the journal
    journalctl --disk-usage

# Display only the last few logs
    journalctl -xe

How to change system IP address

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/sec-configuring_ip_networking_with_nmcli

https://www.golinuxcloud.com/nmcli-command-examples-cheatsheet-centos-rhel/

# verify connection
nmcli connection show       # verify all connections
nmcli connection --active   # verify only currently active connections
nmcli device status         # view only devices recognized by NetworkManager and their state

# verify ip address configuration
nmcli
ip a
nmcli
ip addr

# change IP address
nmtui       # change from the text GUI

# Reapply after changes
nmcli device reapply ens192     # <ens192> is the example, maybe eth0

# check routing table
ip route list
cat /etc/sysconfig/network-scripts/route-ens192     # verify manually configured static route

# Restart NetworkManager service
systemctl restart NetworkManager
nmcli device reapply <network-name>

Verify dns servers

grep "nameserver" /etc/resolv.conf
nslookup host-fqdn      # query dns record
dig host-fqdn           # more detail information

Check Top Processes sorted by RAM or CPU Usage in Linux

ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem | head

Check the remote port is open

# nc required to have netcat package install
    dnf install nc   or yum install nc
# nc [-options] [HostName or IP] [PortNumber]
nc -zvw10 192.168.0.1 22

    z: zero-I/O mode which is used for scanning
    v: for verbose output
    w10: timeout wait seconds

# $ nmap [-options] [HostName or IP] [-p] [PortNumber]
nmap 192.168.0.1 -p 22

# $ telnet [HostName or IP] [PortNumber]
telnet 192.168.0.1 22

# Python module
Python -c “import socket; s = socket.socket(); s.settimeout(10); s.connect((‘192.168.0.1’, 22)); ”

# curl
curl -v telnet://192.168.0.1:22

#********** Very handy way *************
There is another way to check for open ports. In Linux, everything is a file, including the host status and its port availability. 
This can come handy in cases where no commands are working on the remote host.
echo > /dev/tcp/[host]/[port] && echo "Port is open"
    echo > /dev/tcp/192.168.0.10/22 && echo "Port is open"
        # Should receive return as "Port is open", otherwise
        # connection refuse and port is not open
echo > /dev/udp/[host]/[port] && echo "Port is open"

# To check the listening ports by run the command locally on the remote system
watch -n0.2 'netstat -tuplen'   # watch every 0.2 second
                                # tcp, udp for listening port
                                # list and include state, and

netstat -nr     # verify routing table
netstat -i      # display interface statistics, watch for packet loss
netstat -ta     # display all connections

Helpful Linux Commands

Understanding the Linux passwd file fields

https://www.cyberciti.biz/faq/understanding-etcpasswd-file-format/

# View local users
    cat  /etc/passwd

# List all local users
sed 's/:.*//' < /etc/passwd
awk -F':' '{print $1}' < /etc/passwd

You can red /etc/passwd file using while loop and IFS separator

#!/bin/bash
# seven fields from /etc/passwd stored in $f1,f2...,$f7
# 
while IFS=: read -r f1 f2 f3 f4 f5 f6 f7
do 
 echo "User $f1 use $f7 shell and stores files in $f6 directory."
done < /etc/passwd

List directory usage and file sizing when troubleshooting directory or partition full

# Find directory size
du -h /home/testuser        # It will list all directories and subdirectoris usage
du -mh /home/testuser       # List all directories and subdirectories usage in MB unit
du -ch /home/testuser       # -c will give a granted total usage disk space at the last line
du -sch /tmp                # find the total size of a directory
du -ah  /tmp                # display the disk usage of all the files and directories

# Display disk usage based on modification of time,  -time  flag
du -ha --time /home/testuser   

# List file size in reverse order
ls -IS -r /var/log          # S   is uppercase 

# Find all files and their size
find . -name "test*.gz" | xargs du -sch
    # "*" wildcard to find all match
    # xargs     execute the command

du -ah | sort -hr
du -ah | sort -n
du -h -d 1 | sort -n    # list directory usage

Journalctl commands

https://www.debugpoint.com/systemd-journalctl/

# verify journalctl entries
journalctl -xe
journalctl -p 0     # View only errors, warnings in journal logs, or view emergency system messages
        0   emergency
        1   alerts
        2   critical
        3   errors
        4   warning
        5   notice
        6   info
        7   debug

journalctl --list-boots     # view journal logs for boot
journalctl -b  -<bootnumber>    # view specific boot information
journalctl -k       # view kernel specific journal logs

# View journal logs for a specific time, date duration
journalctl --since "2020-12-04 06:00:00"
journalctl --since "2020-12-03" --until "2020-12-05 03:00:00"
journalctl --since yesterday
journalctl --since 09:00 --until "1 hour ago"

# View journal log for a service, or PID
journalctl -u <servicename>
    Example:    journalctl -u NetworkManager.service

If you do not know the service name, list the systemd services in the system
    systemctl list-units --type=service

# How to view journal log for a user, group
id -u <userlogin name>  # find the user login uid
journalctl _UID=<User UID identifed> --since today      # View the journal for the user activity today
    # _GID      for group

# Viwe journal logs for an executable
journalctl /usr/bin/<executable name> --since today
    Example:    journalctl /user/bin/gonme-shell --since today

Boot into Single User Mode

In single-user mode, your computer boots to runlevel 1

Your local file systems are mounted, but your network is not activated. You have a usable system maintenance shell. Unlike rescue mode, single-user mode automatically tries to mount your file system.

Note:
Do not use single-user mode if your file system cannot be mounted successfully. 
You cannot use single-user mode if the runlevel 1 configuration on your system is corrupted.

# How to boot to single user mode
1. Access the server console, via iLo, iDRAC, or Hitachi BMC, or from vSphere VM console
2. Reboot the server, at the GRUB splash screen at boot time, press any key to enter the GRUB interactive menu.
3. Select Red Hat Enterprise Linux with the version of the kernel that you wish to boot, and type a to append the line.
      "a"   Type a to append the line
4. Go to the end of the line starts with linux, and type single as a separate word (press the Spacebar and then type single)
      Note
      you could type    1     at the end of the line instead of  "single"
5. Press Enter to exit edit mode.
6. Once in single user mode, you could carry out disk partition resizing maintenance
7. Finally reboot the server after finishing the maintenance tasks.

How to boot RHEL into Maintenance Mode

In the event that the root password is forgotten, or the fstab mount has issue mounting the mount points, it is necessary to boot the system into maintenance mode.

Note
Sometimes, people refer the maintenance mode as single user mode, but they are different

1. On RHEL or Centos sytem, reboot the server, wait for GRUB boot menu to appear
Note:
At the bottom of the screen shows
      Press `e` to edit the select item, or 'c' for a command prompt
2. Select the kernel version from the GRUB menu, and press "e" key to edit the first boot option
3. Using the Down arrow key to find the kernel line starts with "linux16", then press END to go the end of the line, and enter
    rd.break
4. Once the update has been done, press Ctrl+X  or F10 to reboot into emergency shell (Maintenance mode, sometime called single user mode)
5. To make changes to the "sysroot" file system, need to remount it into READ and WRITE (rw)
    mount -o remount, rw /sysroot
5. Run the command to change the environment, commnly known as "jailed directory" or "chroot jail"
      chroot  /sysroot/
6. Finally, the single user mode is ready to use, carry out maintenance tasks required
      passwd root       # Reset root password
      vi /etc/fstab     # Fix the /etc/fstab mount points
7. Finally, recreate the hidden file
      touch /.autorelabel       # There is no space between "/" and "."
7. Reboot the system
      Type exit command twice, or type "reboot -f", or "exec init 6"

How to reset forgotten root password

https://www.redhat.com/sysadmin/recover-root-passwd

On a RHEL/CentOS version 7 or later system, thanks to the Grub bootloader it’s actually pretty simple.
1. Begin by starting a kettle of water to boil (Optional, but recommended).
2. First, you need console access: you will need to see and interact with the bootloader
    Either at a keyboard and monitor locally, or 
    via Virtual Machine remote console

3. Reboot the machine: 
    As soon as the bootloader comes up with the selection screen, 
    quickly tap the up and down arrows up and down to pause the countdown.

4. Select the kernel you want to boot into, and 
    hit 'e': This will take you into a screen where you can edit the grub bootloader script.

5. Find the line that refers to the kernel: There will be a series of 'boot parameters' 
    here: these are instructions passed during the loading of the kernel.

For RHEL/CentOS 7, the line starts with 'linux16'.
For RHEL/Centos 8x, and Fedora the line starts with 'linux'.

6. Add 'rd.break' at the end of that line 
    (There are other things you can do here, but for now, this is all you need) 
    [ Note: This change is temporary ].

7. Now hit Ctrl-x to run the edited bootloader script.
8. You’ll boot to a 'rescue' prompt that looks like this: 
        switch_root:/#.

9. Remount the root partition in read-write mode so that you can run commands. 
    Enter the following, and then hit ENTER
    mount -o remount rw /sysroot

10. Now type 
        chroot /sysroot and hit enter
Note: This will change you into the sysroot (/) directory, and make that your path for executing commands. 

11. Now you can simply change the password for root using the passwd command.
    passwd

12. Next, before you reboot, you will need to make sure that SELinux allows the file changes. 
At the prompt, enter: 
    touch /.autorelabel

This will signal SELinux on the next reboot that the filesystem has changed (the changed password) and allow the change to be loaded. 
This will cause the whole filesystem to be 'relabeled' which might take a while, 
depending on the size of the filesystem and the speed of the machine, so be aware of this possibility.

13. Type exit to leave the chroot environment and enter reboot

How to troubleshooting ssh connection issue

ssh <usernane>:<rmote-host-ip> -vvv     # -vvv for verbose mode
ssh -c <ciphers type> <user>:<remote-ip> -vvv   # specify ciphers type

# Cipher types
ssh-ed25519
ecdsa-sha2-nistp256, ecdsa-sha2-nistp384, ecdsa-sha2-nistp521
rsa-sha2-512, rsa-sha2-256, ssh-rsa-sha256@ssh.com

chacha20-poly1305@openssh.com ciphers
aes128-gcm@openssh.com, aes256-gcm@openssh.com ciphers
aes128-ctr, aes192-ctr, aes256-ctr ciphers when combined with hmac-sha2-512-etm@openssh.com, hmac-sha2-256-etm@openssh.com MACs
aes128-ctr, aes192-ctr, aes256-ctr ciphers when combined with umac-128-etm@openssh.com MACs