- Published on
Red Hat Troubleshooting
- Authors
- Name
- Jackson Chen
Red Hat Enterprise Linux 8 Documentation
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/
Red Hat Enterprise Linux technology capabilities and limits
https://access.redhat.com/articles/rhel-limits
Red Hat Customer Portal
Login to Red Hat customer portal for support and download https://access.redhat.com/
Common administrative commands in Red Hat Enterprise Linux 5, 6, 7, and 8
https://access.redhat.com/articles/1189123
Red Hat Satellite 6.9 (Latest Version)
https://access.redhat.com/documentation/en-US/Red_Hat_Satellite/6.9/
Red Hat Satellite Server has life cylcle for 16 months.
Red Hat Satellite Life-Cycle Dates
Version General availability Full support ends Maintenance support ends End of Life
Full Support
6.9 21 April 2021 Estimated Oct 2021 Estimated Apr 2022 Estimated Oct 2022
Maintenance support
6.8 27 October 2020 30 April 2021 Estimated Oct 2021 Estimated Apr 2022
6.7 14 April 2020 31 October 2020 30 April 2021 Estimated Oct 2021
Important Red Hat Enterprise Linux Directories
LOCATION PURPOSE
/usr Installed software, shared libraries, include files, and read-only program data.
Important subdirectories include:
• /usr/bin: User commands.
• /usr/sbin: System administration commands.
• /usr/local: Locally customized software.
/etc Configuration files specific to this system.
/var Variable data specific to this system that should persist between boots. Files
that dynamically change, such as databases, cache directories, log files,
printer-spooled documents, and website content may be found under /var.
/run Runtime data for processes started since the last boot. This includes process
ID files and lock files, among other things. The contents of this directory are
recreated on reboot. This directory consolidates /var/run and /var/lock
from earlier versions of Red Hat Enterprise Linux.
/home Home directories are where regular users store their personal data and
configuration files.
/root Home directory for the administrative superuser, root.
/tmp A world-writable space for temporary files. Files which have not been
accessed, changed, or modified for 10 days are deleted from this directory
automatically. Another temporary directory exists, /var/tmp, in which files
that have not been accessed, changed, or modified in more than 30 days are
deleted automatically.
/boot Files needed in order to start the boot process.
/dev Contains special device files that are used by the system to access hardware.
List available block devices
For each listed block device, the blkid command displays available attributes such as its volume label (LABEL), universally unique identifier (UUID), file system type (TYPE)
blkid # List all block device
blkid /dev/vda1 # List a particular device
blkid -po udev /dev/vda1 # List more detailed information
Hardening Red Hat
Red Hat 8 Hardening Guide Hardening Red Hat
System Log Files
M any systems record logs of events in text files which are kept in the /var/log directory. These logs can be inspected using normal text utilities such as less and tail.
System Log Files
LOG FILE TYPE OF MESSAGES STORED
/var/log/messages Most syslog messages are logged here. Exceptions include
messages related to authentication and email processing,
scheduled job execution, and those which are purely debuggingrelated.
/var/log/secure Syslog messages related to security and authentication events.
/var/log/maillog Syslog messages related to the mail server.
/var/log/cron Syslog messages related to scheduled job execution.
/var/log/boot.log Non-syslog console messages related to system startup.
Maintaing System Time
The timedatectl command shows an overview of the current time-related system settings, including current time, time zone, and NTP synchronization settings of the system.
timedatectl # Show the overview ofthe current time related system settings
timedatectl list-timezones # List all timezones
timedatectl set-timezone <timezone>
timedatectl set-time <time>
chronyd
The chronyd service keeps the usually-inaccurate local hardware clock (RTC) on track by synchronizing it to the configured NTP servers. If no network connectivity is available, chronyd calculates the RTC clock drift, which is recorded in the driftfile specified in the /etc/chrony.conf configuration file.
The first argument of the server line is the IP address or DNS name of the NTP server. Following the server IP address or name, a series of options for the server can be listed. It is recommended to use the iburst option, because after the service starts, four measurements are taken in a short time period for a more accurate initial clock synchronization.
chrony.conf File
server ntp.redhat.com iburst
chronyd commands
systemctl rstart chronyd
The chronyc command acts as a client to the chronyd service. To verify the sources
chronyc sources -v
chronyc tracking
chronyc sourcestats
Other chrony commands
chrony sources # Verify the sources
tzselect # time zone select utility
datetimectl
Networking
Identify network interfaces
ip link show
ip addr show ens1 # Show IP address
ip -s link show ens1 # Display performance statistics
tracepath ntp.redhat.com # Traceroute
ss # Display socket statistics, which replace netstat
nmcli # network utility
Network configuration files are stored at /etc/sysconfig/network-scripts/ifcfg-name
nmcli utility update the ifcfg-name files
Configure Host Names and Name Resolution
hostname # display hostname
hostnamectl set-hostname <hostname> # set hostname in file /etc/hostname
hostnamectl status # display hostname information
cat /etc/hosts # display hosts entries
getent hosts <verifyhost> # command to test entry in /etc/hosts file
If an entry is found in /etc/hosts file, the system then using DNS nameserver, /etc/resolv.conf file manage the DNS lookup
# Entry in /etc/hots file (example)
# Generated by NetworkManager
domain testing.net
search testing.net
nameserver 192.168.1.254
Lookup host name from DNS
host dc.testing.net # Verify DNS entry
host 192.168.1.254 # Reverse DNS lookup
File Transfer
The secure copy command "scp" is part of the OpenSSH, it is used for file transfer.
scp /<sourceDir>/<file1> user1@remotesystem:/tmp/<dirDest>
scp user1@remotesystem:/tmp/<sourcefile> /<localDir>/<dir1>
scp -r user1@remotesystem:/tmp/<sourcefile> /<localDir>/<dir1> # copy all files and folders recursively
rsync Synchronizing files between systems securely
The most common options when synchronizing are rynsc -v and rynsc -a
"-a" is --archive optoin. This option recursively copying and synchronzing the files and folders.
rsync -av /<sourceDir>/<Dir1> /<destDir>
rsync -av remotesystem:/<sourceDir>/<Dir1> /<localDir>
Installing and Updating Software Packages
Create the repo file with the content location
# /etc/yum.repos.d/update.repo
[Update]
name=Red Hat Updates
baseurl=http://update.test.net/contents/
enabled=1
gpgcheck=0
To install software package
yum list # list the software packages
yum install <packagename>
File Systems and Devices
Useful commands
lsblk # identify blcok device
lsblk -fp # lisgt full path of device, including UUID and mount point
mount /dev/sr0 /mnt/cdrom # mount dvd to /mnt/cdroom
Search File
"locate" - search a pre-generated index for file name or file path "find" - search for file in real time
locate -i <filename> # -i option, search case sensitive
locate -n 10 <filename> # -n option, limit the search result
find / -name <filename>
find / -name '*.txt'
How to recover or reset forgotten RHEL root password
# How to recover RHEL 8.x root password
1. Reboot the VM
2. Press 'e' before the menu time out and boots normally
3. Look for line starts "linux....", at the end of the line, find and replace
rhgb quiet # parametre with the following parameter
rd.break enforcing=0 # set SELINUX enforcement to zero/not enforce
4. Press Ctrl+x to start the boot with the new parameters.
5. After the system boot up, enter the following command to remount the sysroot filesystem as read/write:
mount -o remount,rw /sysroot
switch_root:/# mount -o remount,rw /sysroot
6. We chroot into the sysroot, using the following command: chroot /sysroot
switch_root:/# chroot /sysroot
7. Use the passwd command to change the root password
sh-4.2# passwd
8. Issue the following command to bring us back to the switch_root:/# prompt: exit
sh-4.2# exit
9. Enter the following command to remount the sysroot filesystem as read-only once again:
mount -o remount,ro /sysroot
switch_root:/# mount -o remount,ro /sysroot
10. We can exit the session and allow the system to reboot using the following command: exit
switch_root:/# exit
11. Reboot and login with new password
12. Cleanup
We must clean things up a bit before rebooting again or doing anything else with the system.
First, let’s update the /etc/shadow file by issuing the following command: restorecon /etc/shadow
root@..]# restorecon /etc/shadow
13. Set SELINUX
Finally we set our SELINUX back to enforcing mode by issuing the following command: setenforce 1
root@..]# setenforce 1
14. Reboot again
How to set RHEL static ip address
# Identity the network device is use
nmcli con # Note down "DEVICE", normally for virtual NIC is ens192
# Update TCP/IP configuration
vim /etc/sysconfig/network-scripts/ifcfg-<device>
vim /etc/sysconfig/network-scripts/ifcfg-ens192 # Example
# Restart network service
systemctl restart network
How to Verify network adapter with multiple IP addresses
Quite often when run ifconfig, we may only see IP address assocation with the system, but when run ip a show ens192, for example, we may see othe IP addresses that are associated with the system.
# Verify IP address
ifconfig
# Verify physical network adapter
nmcli -p device # show network interface name
# From the output of nmcli -p device, verify the IP address association
ip a show ens192 # Example show ens192 association with the interface
# To delete the IP address association
ip addr del 192.168.10.5/<24> dev ens192
# Example delete IP address 192.168.10.5/24 assication with interface ens192
# To add the IP address association
ip addr add 1.2.3.4/24 dev ens192
# Verify BOOTPROTO
# Check /etc/sysconfig/network-scripts/ifcfg-<network-name>
such as
/etc/sysconfig/network-scripts/ifcfg-ens192
BOOTPROTO=none or
BOOTPROTO="static"
If
BOOTPROTO=dhcp # then when run "ip a" will show another IP address obtained from DHCP server
# If BOOTPROTO=dhcp has been configured, need to change to none or static
# Check network adapter TCP/IP configuration
nmtui
# After change the network interface IP address or configuration, run
nmcli con reload <interface-name> # To take effect
ip addr
ip a # check IP address again to ensure there is no multiple IP addresses set for the network interface
Network Bonding
https://www.kernel.org/doc/Documentation/networking/bonding.txt
Network bonding is a method to combine or aggregate network interfaces to provide a logical interface with higher throughput or redundancy.
The active-backup, balance-tlb, and balance-alb modes do not require any specific configuration of the network switch. However, other bonding modes require configuring the switch to aggregate the links. For example, Cisco switches requires EtherChannel for modes 0, 2, and 3, but for mode 4, the Link Aggregation Control Protocol (LACP) and EtherChannel are required.
Important
Network teaming is deprecated in Red Hat Enterprise Linux 9. If you plan to upgrade your server to a future version of RHEL, consider using the kernel bonding driver as an alternative. For details, see Configuring network bonding.
- Configuring a network bond using nmcli commands
- Configuring a network bond using nm-connection-editor
# Open a terminal, and enter nm-connection-editor:
nm-connection-editor
How to create network bonding in RHEL
Verify network connections or connection profiles
# Verify network connections or connection profiles
nmcli con show
# Check Name, UUID, Type, DEVICE
# Note down "UUID", as will need to delete the UUID after bond creation
# Other commands
nmcli device status
nmclic device show
Install and enable lldpd
# mount RHEL repository, such as CD/ISO
mount /dev/sr0 /media/cdrom
dnf install -y lldpd
systemctl enable --now lldpd
# Verify lacp network neighbors
lldpcli show neighbors
Create network connection or interface name - bond0
# Create bond interface
# Configure bond option "4" - 802.3ad Requires LACP-negotiated Etherchannel enabled in network switch
nmcli connection add type bond con-name bond0 interface bond0 bond.loptions "mode=4,miion=1000"
# Add physical network adapter to bond0
nmcli con add type ethernet slave-type bond con-name bond0-port1 ifname eno12399np0 master bond0
nmcli con add type ethernet slave-type bond con-name bond0-port1 ifname eno12409np1 master bond0
# Verify connection interface
nmcli con show
Configure bond0 TCP IP configuration
Need to create the bond TCP IP configuration, and delete eno12399np0 & eno12409np1, aslo bring up bond0 at the same time
Otherwise, will lost connection connection, and need to login to console
# configure bond0 TCP/IP & delete UUID, also bring up bond0
nmcli con modify bond0 ipv4.addr '10.10.10.11/24'
nmcli con modify bond0 ipv4.gateway '10.10.10.1'
nmcli con modify bond0 ipv4.dns '10.10.1.1, 10.10.1.2'
nmcli con modify bond0 ipv4.dns-search 'test.lab'
nmcli con modify bond0 ipv4.method manual
(nmcli con del eno12399np0-UUID) && (nmcli con del eno12409np1-UUID) && (nmcli con up bond0)
Note: Ensure run all the above commands at the sametime from ssh
# verify connection
nmcli con show
Configure jumpbo frame
# Configure MTU 9000
echo 'MTU=9000' >> /etc/sysconfig/network-scripts/ifcfg-bond0
cat /etc/sysconfig/network-scripts/ifcfg-bond0
Verify network connection packets
tcpdump --nnpi bond0
How to view linux kernel logs live
# print everything appended to /var/log/messages
tail -F /var/log/messages
# execute dmesg every second
while true; do dmesg -c; sleep 1; done
# verify kernel log
cat /proc/kmsg
journalctl --system -f