Published on

Red Hat Troubleshooting

Authors
  • Name
    Jackson Chen

Red Hat Enterprise Linux 8 Documentation

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/

Red Hat Enterprise Linux technology capabilities and limits

https://access.redhat.com/articles/rhel-limits

Red Hat Customer Portal

Login to Red Hat customer portal for support and download https://access.redhat.com/

Common administrative commands in Red Hat Enterprise Linux 5, 6, 7, and 8

https://access.redhat.com/articles/1189123

Red Hat Satellite 6.9 (Latest Version)

https://access.redhat.com/documentation/en-US/Red_Hat_Satellite/6.9/

Red Hat Satellite Server has life cylcle for 16 months.

Red Hat Satellite Life-Cycle Dates
Version   General availability  Full support ends   Maintenance support ends    End of Life
Full Support
6.9         21 April 2021       Estimated Oct 2021  Estimated Apr 2022      Estimated Oct 2022

Maintenance support
6.8         27 October 2020     30 April 2021       Estimated Oct 2021      Estimated Apr 2022
6.7         14 April 2020       31 October 2020     30 April 2021           Estimated Oct 2021

Important Red Hat Enterprise Linux Directories

LOCATION    PURPOSE
/usr        Installed software, shared libraries, include files, and read-only program data.
            Important subdirectories include:
            • /usr/bin: User commands.
            • /usr/sbin: System administration commands.
            • /usr/local: Locally customized software.
/etc        Configuration files specific to this system.
/var        Variable data specific to this system that should persist between boots. Files
            that dynamically change, such as databases, cache directories, log files,
            printer-spooled documents, and website content may be found under /var.
/run        Runtime data for processes started since the last boot. This includes process
            ID files and lock files, among other things. The contents of this directory are
            recreated on reboot. This directory consolidates /var/run and /var/lock
            from earlier versions of Red Hat Enterprise Linux.
/home       Home directories are where regular users store their personal data and
            configuration files.
/root       Home directory for the administrative superuser, root.
/tmp        A world-writable space for temporary files. Files which have not been
            accessed, changed, or modified for 10 days are deleted from this directory
            automatically. Another temporary directory exists, /var/tmp, in which files
            that have not been accessed, changed, or modified in more than 30 days are
            deleted automatically.
/boot       Files needed in order to start the boot process.
/dev        Contains special device files that are used by the system to access hardware.

List available block devices

For each listed block device, the blkid command displays available attributes such as its volume label (LABEL), universally unique identifier (UUID), file system type (TYPE)

blkid                       # List all block device
blkid /dev/vda1             # List a particular device
blkid -po udev  /dev/vda1   # List more detailed information

Hardening Red Hat

Red Hat 8 Hardening Guide Hardening Red Hat

System Log Files

M any systems record logs of events in text files which are kept in the /var/log directory. These logs can be inspected using normal text utilities such as less and tail.

System Log Files
LOG FILE            TYPE OF MESSAGES STORED
/var/log/messages   Most syslog messages are logged here. Exceptions include
                    messages related to authentication and email processing,
                    scheduled job execution, and those which are purely debuggingrelated.
/var/log/secure     Syslog messages related to security and authentication events.
/var/log/maillog    Syslog messages related to the mail server.
/var/log/cron       Syslog messages related to scheduled job execution.
/var/log/boot.log   Non-syslog console messages related to system startup.

Maintaing System Time

The timedatectl command shows an overview of the current time-related system settings, including current time, time zone, and NTP synchronization settings of the system.

timedatectl                 # Show the overview ofthe current time related system settings
timedatectl list-timezones  # List all timezones
timedatectl set-timezone <timezone>
timedatectl set-time <time>

chronyd

The chronyd service keeps the usually-inaccurate local hardware clock (RTC) on track by synchronizing it to the configured NTP servers. If no network connectivity is available, chronyd calculates the RTC clock drift, which is recorded in the driftfile specified in the /etc/chrony.conf configuration file.

The first argument of the server line is the IP address or DNS name of the NTP server. Following the server IP address or name, a series of options for the server can be listed. It is recommended to use the iburst option, because after the service starts, four measurements are taken in a short time period for a more accurate initial clock synchronization.

chrony.conf File

server ntp.redhat.com iburst

chronyd commands

systemctl rstart chronyd

The chronyc command acts as a client to the chronyd service. To verify the sources

chronyc sources -v
chronyc tracking
chronyc sourcestats

Other chrony commands

chrony sources      # Verify the sources
tzselect            # time zone select utility
datetimectl         

Networking

Identify network interfaces

ip link show
ip addr show ens1       # Show IP address
ip -s link show ens1    # Display performance statistics
tracepath ntp.redhat.com    # Traceroute
ss                      # Display socket statistics, which replace netstat
nmcli                   # network utility

Network configuration files are stored at /etc/sysconfig/network-scripts/ifcfg-name

nmcli utility update the ifcfg-name files

Configure Host Names and Name Resolution

hostname                            # display hostname
hostnamectl set-hostname <hostname> # set hostname in file  /etc/hostname
hostnamectl status                  # display hostname information
cat /etc/hosts                      # display hosts entries
getent hosts <verifyhost>           # command to test entry in /etc/hosts file

If an entry is found in /etc/hosts file, the system then using DNS nameserver, /etc/resolv.conf file manage the DNS lookup

# Entry in /etc/hots file (example)
# Generated by NetworkManager
domain testing.net
search testing.net
nameserver 192.168.1.254

Lookup host name from DNS

host dc.testing.net     # Verify DNS entry
host 192.168.1.254      # Reverse DNS lookup

File Transfer

The secure copy command "scp" is part of the OpenSSH, it is used for file transfer.

scp /<sourceDir>/<file1>  user1@remotesystem:/tmp/<dirDest>
scp user1@remotesystem:/tmp/<sourcefile> /<localDir>/<dir1>
scp -r user1@remotesystem:/tmp/<sourcefile> /<localDir>/<dir1>  # copy all files and folders recursively

rsync Synchronizing files between systems securely

The most common options when synchronizing are rynsc -v and rynsc -a

"-a" is --archive optoin. This option recursively copying and synchronzing the files and folders.

rsync -av /<sourceDir>/<Dir1>  /<destDir>
rsync -av remotesystem:/<sourceDir>/<Dir1> /<localDir>

Installing and Updating Software Packages

Create the repo file with the content location

# /etc/yum.repos.d/update.repo
[Update]
name=Red Hat Updates
baseurl=http://update.test.net/contents/
enabled=1
gpgcheck=0

To install software package

yum list    # list the software packages
yum install <packagename>

File Systems and Devices

Useful commands

lsblk       # identify blcok device
lsblk -fp   # lisgt full path of device, including UUID and mount point
mount /dev/sr0  /mnt/cdrom      # mount dvd to /mnt/cdroom

Search File

"locate" - search a pre-generated index for file name or file path "find" - search for file in real time

locate -i <filename>    # -i option, search case sensitive
locate -n 10 <filename> # -n option, limit the search result
find / -name <filename>
find / -name '*.txt'

How to recover or reset forgotten RHEL root password

# How to recover RHEL 8.x root password
1. Reboot the VM
2. Press 'e' before the menu time out and boots normally
3. Look for line starts "linux....", at the end of the line, find and replace
    rhgb quiet   # parametre with the following parameter
    rd.break enforcing=0    # set SELINUX enforcement to zero/not enforce
4. Press Ctrl+x   to start the boot with the new parameters.
5. After the system boot up, enter the following command to remount the sysroot filesystem as read/write: 
        mount -o remount,rw /sysroot
    switch_root:/#  mount -o remount,rw /sysroot
6. We chroot into the sysroot, using the following command: chroot /sysroot
    switch_root:/#  chroot /sysroot
7. Use the passwd command to change the root password
    sh-4.2#  passwd
8. Issue the following command to bring us back to the switch_root:/# prompt: exit
    sh-4.2# exit
9. Enter the following command to remount the sysroot filesystem as read-only once again: 
        mount -o remount,ro /sysroot
    switch_root:/#  mount -o remount,ro /sysroot
10. We can exit the session and allow the system to reboot using the following command: exit
    switch_root:/#  exit
11. Reboot and login with new password
12. Cleanup
    We must clean things up a bit before rebooting again or doing anything else with the system. 
    First, let’s update the /etc/shadow file by issuing the following command:  restorecon /etc/shadow
    root@..]#  restorecon /etc/shadow
13. Set SELINUX
    Finally we set our SELINUX back to enforcing mode by issuing the following command: setenforce 1
    root@..]#  setenforce 1
14. Reboot again

How to set RHEL static ip address

# Identity the network device is use
    nmcli con       # Note down "DEVICE", normally for virtual NIC is ens192

# Update TCP/IP configuration
    vim /etc/sysconfig/network-scripts/ifcfg-<device>
        vim /etc/sysconfig/network-scripts/ifcfg-ens192     # Example

# Restart network service
    systemctl restart network

How to Verify network adapter with multiple IP addresses

Quite often when run ifconfig, we may only see IP address assocation with the system, but when run ip a show ens192, for example, we may see othe IP addresses that are associated with the system.

# Verify IP address
    ifconfig

# Verify physical network adapter
    nmcli -p device     # show network interface name

# From the output of nmcli -p device, verify the IP address association
    ip a show ens192    # Example show ens192 association with the interface

# To delete the IP address association
    ip addr del  192.168.10.5/<24> dev ens192
        # Example delete IP address 192.168.10.5/24 assication with interface ens192
    
# To add the IP address association
    ip addr add 1.2.3.4/24 dev ens192

# Verify BOOTPROTO
# Check /etc/sysconfig/network-scripts/ifcfg-<network-name>
    such as
        /etc/sysconfig/network-scripts/ifcfg-ens192
            BOOTPROTO=none      or
            BOOTPROTO="static"
        If
            BOOTPROTO=dhcp      # then when run "ip a" will show another IP address obtained from DHCP server

        # If BOOTPROTO=dhcp  has been configured, need to change to none or static

# Check network adapter TCP/IP configuration
    nmtui

# After change the network interface IP address or configuration, run
    nmcli con reload <interface-name>       # To take effect
    ip addr
    ip a            # check IP address again to ensure there is no multiple IP addresses set for the network interface

Network Bonding

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/configuring-network-bonding_configuring-and-managing-networking

https://www.kernel.org/doc/Documentation/networking/bonding.txt

Network bonding is a method to combine or aggregate network interfaces to provide a logical interface with higher throughput or redundancy.

The active-backup, balance-tlb, and balance-alb modes do not require any specific configuration of the network switch. However, other bonding modes require configuring the switch to aggregate the links. For example, Cisco switches requires EtherChannel for modes 0, 2, and 3, but for mode 4, the Link Aggregation Control Protocol (LACP) and EtherChannel are required.

Important

Network teaming is deprecated in Red Hat Enterprise Linux 9. If you plan to upgrade your server to a future version of RHEL, consider using the kernel bonding driver as an alternative. For details, see Configuring network bonding.

  1. Configuring a network bond using nmcli commands
  2. Configuring a network bond using nm-connection-editor
# Open a terminal, and enter nm-connection-editor:
    nm-connection-editor

How to create network bonding in RHEL

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/configuring-network-bonding_configuring-and-managing-networking

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/managing_systems_using_the_rhel_8_web_console/configuring-network-bonds-using-the-web-console_system-management-using-the-rhel-8-web-console

Verify network connections or connection profiles
# Verify network connections or connection profiles
nmcli con show
    # Check Name, UUID, Type, DEVICE
    # Note down "UUID", as will need to delete the UUID after bond creation

# Other commands
nmcli device status
nmclic device show
Install and enable lldpd
# mount RHEL repository, such as CD/ISO
mount /dev/sr0 /media/cdrom
dnf install -y lldpd
systemctl enable --now lldpd

# Verify lacp network neighbors
lldpcli show neighbors
Create network connection or interface name - bond0
# Create bond interface
# Configure bond option "4" - 802.3ad   Requires LACP-negotiated Etherchannel enabled in network switch
nmcli connection add type bond con-name bond0 interface bond0 bond.loptions "mode=4,miion=1000"

# Add physical network adapter to bond0
nmcli con add type ethernet slave-type bond con-name bond0-port1 ifname eno12399np0 master bond0
nmcli con add type ethernet slave-type bond con-name bond0-port1 ifname eno12409np1 master bond0

# Verify connection interface
nmcli con show
Configure bond0 TCP IP configuration

Need to create the bond TCP IP configuration, and delete eno12399np0 & eno12409np1, aslo bring up bond0 at the same time

Otherwise, will lost connection connection, and need to login to console

# configure bond0 TCP/IP & delete UUID, also bring up bond0
nmcli con modify bond0 ipv4.addr '10.10.10.11/24'
nmcli con modify bond0 ipv4.gateway '10.10.10.1'
nmcli con modify bond0 ipv4.dns '10.10.1.1, 10.10.1.2'
nmcli con modify bond0 ipv4.dns-search 'test.lab'
nmcli con modify bond0 ipv4.method manual
(nmcli con del eno12399np0-UUID) && (nmcli con del eno12409np1-UUID) && (nmcli con up bond0)    
    Note: Ensure run all the above commands at the sametime from ssh

# verify connection
nmcli con show
Configure jumpbo frame
# Configure MTU 9000
echo 'MTU=9000' >> /etc/sysconfig/network-scripts/ifcfg-bond0

cat /etc/sysconfig/network-scripts/ifcfg-bond0
Verify network connection packets
tcpdump --nnpi bond0
How to view linux kernel logs live
# print everything appended to /var/log/messages
    tail -F /var/log/messages

# execute dmesg every second
    while true; do dmesg -c; sleep 1; done

# verify kernel log
    cat /proc/kmsg

    journalctl --system -f