fabrice/scripts-fabq
fabrice/scripts-fabq
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: split and clean drive/storage notes into three focused files

Browse Source 
* notes/btrfs.md : remove generic drive commands, genericize all UUIDs/paths/hostnames, add Placeholders section, add command shorthands table, collapse backup procedures into a single generic template
* notes/linux.md : remove all storage and drive-related content, genericize all hardcoded usernames/hostnames/paths
* notes/storage.md : new file consolidating all storage tooling (smartctl, badblocks, dd, hdparm, fsck, findmnt, fstab) from both btrfs.md and linux.md
This commit is contained in:
Fabrice Quenneville 2026-04-05 18:22:20 -04:00
parent d1767e7409
commit 13c0acd990
3 changed files with 629 additions and 546 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

565
notes/btrfs.md
View File

@ -4,542 +4,475 @@
- [BTRFS](#btrfs)
- [Table of Contents](#table-of-contents)
- [Information on Drives](#information-on-drives)
- [Placeholders](#placeholders)
- [BTRFS Command Shorthands](#btrfs-command-shorthands)
- [Information on Filesystem](#information-on-filesystem)
- [Backup Procedures](#backup-procedures)
- [Recovery](#recovery)
- [Drive Manipulation](#drive-manipulation)
- [Replace Drives](#replace-drives)
- [Filesystem Manipulation](#filesystem-manipulation)
- [Upgrading Btrfs block group cache to V2](#upgrading-btrfs-block-group-cache-to-v2)
- [Upgrading Btrfs Block Group Cache to V2](#upgrading-btrfs-block-group-cache-to-v2)
- [Defrag](#defrag)
- [Balances](#balances)
- [Scrub](#scrub)
- [Snapshots](#snapshots)
- [Create Snapshots](#create-snapshots)
- [Delete Snapshots](#delete-snapshots)
- [Backup Procedures](#backup-procedures)
- [Recovery](#recovery)
## Information on Drives
## Placeholders
**List of Drives and Mountpoints**
Replace the placeholders below with the appropriate values for your setup:
To check all attached drives:
- **Devices**
- Block device: `<device>` (e.g., /dev/sda)
- Source device to replace: `<source-device>` (e.g., /dev/sdb)
- Target device: `<target-device>` (e.g., /dev/sdc)
- UUID: `<uuid>` (e.g., a1b2c3d4-e5f6-7890-abcd-ef1234567890)
```bash
ls /dev/sd*
ls /dev/nv*
```
- **Paths**
- Mount point: `<mountpoint>` (e.g., /mnt/media)
- Subvolume name: `<subvolume>` (e.g., root, home, backups, snapshots)
- Subvolume ID: `<subvolume-id>` (e.g., 257)
- Snapshot label: `<snapshot-label>` (e.g., 2024-09-15)
To view mountpoints and drive details such as names, sizes, and mountpoints:
## BTRFS Command Shorthands
```bash
lsblk
df -h
cat /etc/fstab
```
Most `btrfs` subcommands accept shortened aliases. The table below lists the common ones used throughout this document.
**Drive Information**
To get detailed information and serial number of a specific drive:
```bash
smartctl -i /dev/sdc
```
**Find the Device Path from UUID**
Using lsblk:
```bash
lsblk -o NAME,UUID,MOUNTPOINT
```
Using blkid:
```bash
blkid | grep <UUID>
```
| Long form | Short form |
| ----------------------------- | ----------------------- |
| `btrfs filesystem` | `btrfs fi` |
| `btrfs filesystem show` | `btrfs fi show` |
| `btrfs filesystem usage` | `btrfs fi usage` |
| `btrfs filesystem df` | `btrfs fi df` |
| `btrfs filesystem resize` | `btrfs fi resize` |
| `btrfs filesystem defragment` | `btrfs fi defrag` |
| `btrfs subvolume` | `btrfs sub` |
| `btrfs subvolume list` | `btrfs sub list` |
| `btrfs subvolume snapshot` | `btrfs sub snap` |
| `btrfs subvolume delete` | `btrfs sub del` |
| `btrfs subvolume get-default` | `btrfs sub get-default` |
| `btrfs subvolume set-default` | `btrfs sub set-default` |
| `btrfs device` | `btrfs dev` |
| `btrfs device add` | `btrfs dev add` |
| `btrfs device usage` | `btrfs dev usage` |
| `btrfs device stats` | `btrfs dev stats` |
| `btrfs balance start` | `btrfs bal start` |
| `btrfs balance status` | `btrfs bal status` |
| `btrfs balance cancel` | `btrfs bal cancel` |
| `btrfs inspect-internal` | `btrfs insp` |
## Information on Filesystem
**Show Basic Filesystem Information**
To display basic information (size, IDs, paths, etc.) for the specified mountpoint:
Display basic information (size, IDs, paths, etc.) for the specified mountpoint:
```bash
btrfs fi show /mnt/media/
btrfs fi show <mountpoint>
```
**Display Detailed Usage Information**
**Display detailed usage information**
To show detailed usage information (allocated, unallocated, etc.) for the specified mountpoint:
To show detailed usage information (allocated, unallocated, free, used, etc.) for the specified mountpoint:
```bash
btrfs fi usage /mnt/media
btrfs fi usage <mountpoint>
```
**Display detailed usage information as a table**
```bash
btrfs fi usage -T <mountpoint>
```
- `-T` The tabular flag gives you a nice grid that shows exactly how much Data, Metadata, and System space is allocated per device.
**Display Detailed Allocation Information**
To view detailed allocation information (block groups, used space) for the specified mountpoint:
View block groups and used space:
```bash
btrfs fi df /mnt/media
btrfs fi df <mountpoint>
```
**Get Detailed Device Usage Statistics**
To get detailed device usage statistics (physical size, unallocated space, RAID levels, etc.) for a BTRFS filesystem:
Physical size, unallocated space, RAID levels, etc.:
```bash
btrfs device usage /mnt/media
btrfs device usage <mountpoint>
```
**Scan and Display BTRFS Information**
To scan and display BTRFS information for all devices or a specific drive:
Scan all devices or a specific drive:
```bash
btrfs device scan /dev/sda/
btrfs device scan <device>
```
**Retrieve Statistics and Error Information**
To get statistics and error information (read errors, write errors, flush errors, etc.) for the specified mountpoint:
Read errors, write errors, flush errors, etc.:
```bash
btrfs device stats /mnt/media
btrfs device stats <mountpoint>
```
**List BTRFS Subvolumes**
To list BTRFS subvolumes:
```bash
btrfs subvolume list /
btrfs subvolume list /home/fabrice
btrfs subvolume list /mnt/workbench
btrfs subvolume list <mountpoint>
```
**Default Subvolume**
To check if a non-standard subvolume is set as the default:
Check if a non-standard subvolume is set as the default:
```bash
btrfs subvol get-default /mnt/tmp/
btrfs subvol list /mnt/tmp/
btrfs subvol get-default <mountpoint>
btrfs subvol list <mountpoint>
```
To change the default subvolume if a non-standard one is set:
Change the default subvolume:
```bash
btrfs subvol set-default 257 /mnt/tmp/
btrfs subvol set-default <subvolume-id> <mountpoint>
```
**Verify Current Cache Version**
To check if your filesystem is using cache V1 by device:
Check if your filesystem is using cache V1 by device:
```bash
btrfs inspect-internal dump-super -f /dev/<device> | grep cache_generation
btrfs inspect-internal dump-super -f <device> | grep cache_generation
```
To check if your filesystem is using cache V1 by UUID:
By UUID:
```bash
btrfs inspect-internal dump-super -f $(blkid -U <UUID>) | grep cache_generation
btrfs inspect-internal dump-super -f $(blkid -U <uuid>) | grep cache_generation
```
- If cache_generation is present, it indicates cache V1 is in use. If it's absent, the filesystem is already using V2.
## Backup Procedures
**Desktop Backup: Root and Home**
1. Mount snapshot location:
```bash
mount UUID=394decca-4780-47c9-9ae3-e4d03681a791 -o subvol=snapshots /mnt/snapshots
```
2. Create snapshots for root and home:
```bash
btrfs subvolume snapshot / "/mnt/snapshots/root/2021-05-23 - Fedora 34 upgrade"
btrfs subvolume snapshot /home "/mnt/snapshots/home/2021-05-23 - Fedora 34 upgrade"
```
3. Unmount after creating snapshots:
```bash
umount /mnt/snapshots
```
**Data Backup: Workbench, Documents, Education**
1. Mount snapshot location:
```bash
mount UUID=72e1770a-9fc0-461e-88d3-db640ff53dd9 -o subvol=snapshots /mnt/snapshots
```
2. Create snapshots for multiple directories:
```bash
btrfs subvolume snapshot /mnt/workbench "/mnt/snapshots/workbench/2021-05-23 - Fedora 34 upgrade"
btrfs subvolume snapshot /home/fabrice/Documents "/mnt/snapshots/Documents/2021-05-23 - Fedora 34 upgrade"
btrfs subvolume snapshot /home/fabrice/Education "/mnt/snapshots/Education/2021-05-23 - Fedora 34 upgrade"
```
3. Unmount after creating snapshots:
```bash
umount /mnt/snapshots
```
**STOR1 Fedora Backup**
1. Mount snapshot location:
```bash
mount UUID=e4fd608e-cfe8-4c10-b6d0-03b05bae8aa6 -o subvol=snapshots /mnt/snapshots
```
2. Create snapshot:
```bash
btrfs subvolume snapshot / "/mnt/snapshots/root/2021-06-06"
```
3. Unmount after creating snapshot:
```bash
umount /mnt/snapshots
```
**STOR1 Debian Backup**
1. Mount snapshot location:
```bash
mount UUID=c9a77f3c-626f-47bd-b4e3-9a094bea287f -o subvol=snapshots /mnt/snapshots
```
2. Create snapshot:
```bash
btrfs subvolume snapshot / "/mnt/snapshots/root/2021-07-12 - post mostly setup"
```
3. Unmount after creating snapshot:
```bash
umount /mnt/snapshots
```
**STOR2 Backup**
1. Mount snapshot location:
```bash
mount UUID=30bd5e0e-e781-4e87-8fb8-ea5606403b15 -o subvol=snapshots /mnt/snapshots
```
2. Create snapshot:
```bash
btrfs subvolume snapshot / "/mnt/snapshots/root/2021-06-06 - Fedora 34"
```
3. Unmount after creating snapshot:
```bash
umount /mnt/snapshots
```
## Recovery
**Mount a Subvolume with Recovery Options**
```bash
mount -o recovery,subvol=backups UUID=aa5c1d34-ecba-42a9-9339-8f7879d47536 /mnt/tmp
```
**Clear Cache During Mount**
```bash
mount -o clear_cache,subvol=backups UUID=aa5c1d34-ecba-42a9-9339-8f7879d47536 /mnt/tmp
```
**Data Restoration**
To restore data using `btrfs restore`:
```bash
btrfs restore -D /dev/sdb
```
- If `cache_generation` is present, cache V1 is in use. If absent, the filesystem is already using V2.
## Drive Manipulation
**Mount Whole Drive**
```bash
mount UUID=c9a77f3c-626f-47bd-b4e3-9a094bea287f /mnt/tmp
mount UUID=<uuid> <mountpoint>
```
**Mount Subvolume by Name**
```bash
mount UUID=<uuid> -o subvol=<subvolume> <mountpoint>
```
**Mount Subvolume by ID**
```bash
btrfs subvol list /
mount -o subvolid=5 /dev/disk/by-uuid/7a22514b-594a-43a3-8fdd-4df1530b5465 /mnt
mount -o subvolid=<subvolume-id> /dev/disk/by-uuid/<uuid> <mountpoint>
```
**Add a New Drive**
To add a new drive to an existing BTRFS setup:
```bash
btrfs device add /dev/sdf /mnt/media/
btrfs device add <device> <mountpoint>
```
**Resize Filesystem**
Grow the filesystem on a specific device to its maximum:
```bash
btrfs filesystem resize 1:max /mnt/media/
btrfs filesystem resize 1:max <mountpoint>
```
**Create Subvolumes**
```bash
btrfs subvol create /mnt/tmp/root
btrfs subvol create /mnt/tmp/snapshots
btrfs subvol create <mountpoint>/<subvolume>
```
### Replace Drives
**Replace the source drive with the target drive:**
**Start the replacement process:**
This command will start the replacement process where the data from the old drive (`/dev/sdb`) is copied over to the new drive (`/dev/sdj`).
Copies data from the old drive to the new drive while the filesystem remains mounted:
```bash
btrfs replace start /dev/sdb /dev/sdj /mnt/media
btrfs replace start <source-device> <target-device> <mountpoint>
```
- `/dev/sdb`: Source drive to be replaced.
- `/dev/sdj`: Target drive to replace the source drive.
- `/mnt/media`: Mount point of the BTRFS filesystem.
- `<source-device>`: Drive to be replaced.
- `<target-device>`: Drive to replace it with.
- `<mountpoint>`: Mount point of the BTRFS filesystem.
**Monitor the progress of the replacement:**
Once the replacement process has started, you can monitor its progress with the following command:
**Monitor the progress:**
```bash
btrfs replace status /mnt/media
btrfs replace status <mountpoint>
```
- This will print the current status of the drive replacement operation, showing how much data has been migrated.
**Monitor progress interactively:**
For a more detailed, interactive status view of the replacement process, use the `-i` option:
```bash
btrfs replace status -i /mnt/media
btrfs replace status -i <mountpoint>
```
- `-i`: This flag provides an interactive mode where the progress is updated in real time.
- `-i`: Updates progress in real time.
**Notes:**
- The `btrfs replace` command allows you to replace a faulty or underperforming drive without unmounting the filesystem, making it ideal for live systems.
- It can be used for upgrading storage by replacing smaller drives with larger ones, or for replacing failing drives.
- Ensure that the target drive has enough space to accommodate the data from the source drive.
- `btrfs replace` works on a live mounted filesystem — no unmounting required.
- Useful for both failing drive replacement and capacity upgrades.
- Ensure the target drive has enough space to accommodate the source data.
## Filesystem Manipulation
### Upgrading Btrfs block group cache to V2
### Upgrading Btrfs Block Group Cache to V2
**From a running system non-root filesystems**
**Non-root filesystems (running system):**
```bash
mount -o remount,clear_cache,space_cache=v2 /mnt/<mount-point>
mount -o remount,clear_cache,space_cache=v2 <mountpoint>
```
**From a running system on root**
**Root filesystem (running system):**
Check if your filesystem is using cache V1:
1. Check if using cache V1:
```bash
btrfs inspect-internal dump-super -f /dev/<device> | grep cache_generation
```
```bash
btrfs inspect-internal dump-super -f <device> | grep cache_generation
```
Enable Cache V2
2. Enable cache V2 via GRUB:
```bash
nano /etc/default/grub
# Locate the line starting with GRUB_CMDLINE_LINUX_DEFAULT or GRUB_CMDLINE_LINUX and add the following options:
rootflags=clear_cache,space_cache=v2
```
```bash
nano /etc/default/grub
# Add to GRUB_CMDLINE_LINUX_DEFAULT or GRUB_CMDLINE_LINUX:
# rootflags=clear_cache,space_cache=v2
```
Example:
Example:
```bash
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash rootflags=clear_cache,space_cache=v2"
```
```bash
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash rootflags=clear_cache,space_cache=v2"
```
```bash
update-grub
reboot
```
```bash
update-grub
reboot
```
Verify the Change
3. Verify the change:
```bash
btrfs inspect-internal dump-super -f /dev/<device> | grep cache_generation
```
```bash
btrfs inspect-internal dump-super -f <device> | grep cache_generation
```
Remove `clear_cache` Option
4. Remove `clear_cache` from GRUB after confirming:
```bash
nano /etc/default/grub
# Remove clear_cache from the rootflags.
update-grub
```
```bash
nano /etc/default/grub
# Remove clear_cache from rootflags, then:
update-grub
```
**From a live system**
**From a live system:**
```bash
apt update
apt install btrfs-progs
lsblk -o NAME,UUID
blkid
mount -o clear_cache,space_cache=v2 /dev/disk/by-uuid/<UUID> /mnt
btrfs inspect-internal dump-super -f /dev/disk/by-uuid/<UUID> | grep cache_generation
umount /mnt
mount -o clear_cache,space_cache=v2 /dev/disk/by-uuid/<uuid> <mountpoint>
btrfs inspect-internal dump-super -f /dev/disk/by-uuid/<uuid> | grep cache_generation
umount <mountpoint>
```
### Defrag
```bash
btrfs filesystem defrag -r -v -clzo <mountpoint>
```
- `-r`: Recursive.
- `-v`: Verbose.
- `-clzo`: Optional LZO compression to save space.
## Balances
**Perform a Full Balance with Minimal Usage**
**Full balance on nearly empty block groups:**
```bash
btrfs balance start --full-balance -dusage=0 -musage=0 /mnt/media/
btrfs balance start --full-balance -dusage=0 -musage=0 <mountpoint>
```
- `--full-balance` is default but with a warning if not specified.
- `-dusage=0` means only data block groups that are nearly empty (0% full) will be balanced.
- `-musage=0` means only metadata block groups that are nearly empty (0% full) will be balanced.
- `--full-balance`: Default but with a warning if not specified.
- `-dusage=0`: Only balance data block groups that are ~0% full.
- `-musage=0`: Only balance metadata block groups that are ~0% full.
**Perform a Full Balance on Partially Used Blocks**
**Full balance on partially used block groups:**
```bash
btrfs balance start --full-balance -dusage=50 -musage=50 /mnt/media/
btrfs balance start --full-balance -dusage=50 -musage=50 <mountpoint>
```
- `-dusage=50` means data block groups that are less than 50% full will be included in the balance process.
- `-musage=50` means metadata block groups that are less than 50% full will also be balanced.
- `-dusage=50`: Include data block groups less than 50% full.
- `-musage=50`: Include metadata block groups less than 50% full.
**Balance data in the background**
**Balance data in the background:**
```bash
btrfs balance start --bg -d /mnt/media
btrfs balance start --bg -d <mountpoint>
```
**Balance metadata in the background**
**Balance metadata in the background:**
```bash
btrfs balance start --bg -m /mnt/media
btrfs balance start --bg -m <mountpoint>
```
**Balance data and metadata in the background**
**Balance data and metadata in the background:**
```bash
btrfs balance start --bg --full-balance -dusage=0 -musage=0 /mnt/media/
btrfs balance start --bg --full-balance -dusage=0 -musage=0 <mountpoint>
```
**To balance 100 chunks of data**
**Balance a limited number of chunks:**
```bash
btrfs balance start --bg -dlimit=100 /mnt/media/
btrfs balance start --bg -dlimit=100 <mountpoint>
```
**Cancel Balance Operation**
**Cancel a balance:**
```bash
btrfs balance cancel /mnt/media/
btrfs balance cancel <mountpoint>
```
**Monitor Balance Status**
**Monitor balance status:**
```bash
btrfs balance status /mnt/media/
btrfs balance status <mountpoint>
```
## Scrub
**Start a Scrub Operation**
**Start a scrub**
To start a scrub operation to verify data integrity:
The scrub operation verifies data integrity against checksums
```bash
btrfs scrub start /mnt/media/
btrfs scrub start <mountpoint>
```
**Check Scrub Status**
To check the progress and status of the ongoing scrub:
**Check scrub status:**
```bash
btrfs scrub status /mnt/media/
btrfs scrub status <mountpoint>
```
**Cancel a Scrub Operation**
**Cancel a scrub:**
```bash
btrfs scrub cancel /mnt/media/
btrfs scrub cancel <mountpoint>
```
## Snapshots
**Create Snapshots**
### Create Snapshots
1. **Mount snapshot subvolume**
```bash
mount UUID=c9a77f3c-626f-47bd-b4e3-9a094bea287f -o subvol=snapshots /mnt/snapshots
```
2. **Create a new snapshot**
```bash
btrfs subvolume snapshot / "/mnt/snapshots/root/2021-06-26 - Debian install"
```
3. **Unmount after creating snapshots**
```bash
umount /mnt/snapshots
```
**Delete Snapshots**
1. **Mount subvolume containing snapshots**
1. **Mount the snapshots subvolume:**
```bash
mount -o subvol=snapshots /dev/disk/by-uuid/7a22514b-594a-43a3-8fdd-4df1530b5465 /mnt/snapshots/
mount UUID=<uuid> -o subvol=snapshots <mountpoint>
```
2. **List available snapshots**
2. **Create a snapshot:**
```bash
btrfs subvol list /mnt/snapshots/
btrfs subvolume snapshot <source-subvolume> "<mountpoint>/<snapshot-label>"
```
3. **Delete the desired snapshot**
3. **Unmount after creating:**
```bash
btrfs subvolume delete /mnt/snapshots/@rootfs/2024-09-15
umount <mountpoint>
```
4. **Unmount after deleting snapshots**
### Delete Snapshots
1. **Mount the snapshots subvolume:**
```bash
umount /mnt/snapshots/
mount -o subvol=snapshots /dev/disk/by-uuid/<uuid> <mountpoint>
```
2. **List available snapshots:**
```bash
btrfs subvol list <mountpoint>
```
3. **Delete the desired snapshot:**
```bash
btrfs subvolume delete <mountpoint>/<snapshot-label>
```
4. **Unmount after deleting:**
```bash
umount <mountpoint>
```
## Backup Procedures
**Snapshot backup procedure:**
1. Mount snapshot location:
```bash
mount UUID=<uuid> -o subvol=snapshots <mountpoint>
```
2. Create snapshots for the desired subvolumes:
```bash
btrfs subvolume snapshot / "<mountpoint>/root/<snapshot-label>"
btrfs subvolume snapshot /home "<mountpoint>/home/<snapshot-label>"
btrfs subvolume snapshot <source-subvolume> "<mountpoint>/<subvolume>/<snapshot-label>"
```
3. Unmount after creating snapshots:
```bash
umount <mountpoint>
```
## Recovery
1. **Mount a Subvolume with Recovery Options:**
```bash
mount -o recovery,subvol=<subvolume> UUID=<uuid> <mountpoint>
```
2. **Clear Cache During Mount:**
```bash
mount -o clear_cache,subvol=<subvolume> UUID=<uuid> <mountpoint>
```
3. **Data Restoration with btrfs restore:**
```bash
btrfs restore -D <device>
```

296
notes/linux.md
View File

@ -7,7 +7,6 @@
- [System Information](#system-information)
- [Hardware Information](#hardware-information)
- [Software Information](#software-information)
- [Commands to Get Information About Linux Version, Kernel Version, and Release](#commands-to-get-information-about-linux-version-kernel-version-and-release)
- [User Management](#user-management)
- [User Information](#user-information)
- [Super User Management](#super-user-management)
@ -15,8 +14,6 @@
- [System Management](#system-management)
- [Change password of a tar/openssl archive](#change-password-of-a-taropenssl-archive)
- [Verify two possibly identical folders recursively](#verify-two-possibly-identical-folders-recursively)
- [USB Devices](#usb-devices)
- [Test USB Key](#test-usb-key)
- [Diagnosis](#diagnosis)
- [Debian Upgrade Issues](#debian-upgrade-issues)
- [Wayland Issues](#wayland-issues)
@ -33,9 +30,6 @@ To gather detailed information about your hardware, use the following commands:
- **`hwinfo`**: Offers detailed information about hardware components and can be more verbose than `lshw`.
- **`lsscsi`**: Lists SCSI devices, including disks and other SCSI-attached hardware.
- **`lsusb`**: Shows information about USB devices connected to your system.
- **`lsblk`**: Lists block devices such as hard drives and their partitions.
- **`df -H`**: Displays disk space usage in a human-readable format.
- **`fdisk -l`**: Lists all partitions on the system.
- **`dmidecode`**: Retrieves hardware information from the BIOS. Use:
- `dmidecode -t processor` for CPU details
- `dmidecode -t memory` for RAM details
@ -45,13 +39,11 @@ To gather detailed information about your hardware, use the following commands:
**Finding information on the Linux distribution**
# Commands to Get Information About Linux Version, Kernel Version, and Release
- **`lsb_release -a`**: Displays detailed information about the Linux distribution, including the distributor ID, description, release number, and codename.
- **`cat /etc/debian_version`**: Displays the version of the Debian distribution if you're running a Debian-based system (like Ubuntu).
- **`cat /etc/os-release`**: Displays information about the operating system, such as the name, version, and ID of the distribution.
- **`cat /etc/*release`**: Searches for any files in the `/etc/` directory that contain the word `release` and displays their contents. This typically includes more detailed distribution information.
- **`cat /etc/*version`**: Similar to `cat /etc/*release`, but it looks for files containing the word `version`. It can provide additional version-related details.
- **`cat /etc/*version`**: Similar to `cat /etc/*release`, but looks for files containing the word `version`. It can provide additional version-related details.
- **`hostnamectl`**: Displays system information related to the hostname and other metadata about the system. This may include the operating system, kernel version, and architecture.
**Finding Path to Binary**
@ -59,17 +51,17 @@ To gather detailed information about your hardware, use the following commands:
To find the location of an executable binary, use:
```bash
type composer
type <binary-name>
```
This command will show the path to the `composer` executable if it's available in your `PATH`.
This command will show the path to a binary executable, ex `composer`, if it's available in your `PATH`.
**Number of Words in a File**
To count the number of words in a file, use:
```bash
wc filepath
wc <filepath>
```
This command will show the number of words along with other details like lines and characters.
@ -79,11 +71,9 @@ This command will show the number of words along with other details like lines a
To count the number of lines in a file, use:
```bash
wc -l filepath
wc -l <filepath>
```
This command will display the number of lines in the specified file.
## User Management
### User Information
@ -93,25 +83,23 @@ This command will display the number of lines in the specified file.
This variation of the adduser command uses the --gecos option to pre-fill the user's information (Full name, Room number, Work Phone, Home Phone, and Email) non-interactively, allowing you to automate user creation with predefined details.
```bash
adduser --gecos "Fabrice Quenneville,,,fabrice@fabq.ca" fabrice
adduser --gecos "<full-name>,,,<email>" <username>
```
This variation of the adduser command creates a system user named "aptly" with a Bash shell, no password login (--disabled-password), a specified home directory (/home/aptly), and adds the user to a new group, while using the --gecos option to set the full name as "Aptly repository".
This variation creates a system user with a Bash shell, no password login (--disabled-password), a specified home directory (`/home/<username>`), and adds the user to a new group, while using the --gecos option to set the full name as `<service-description>`.
```bash
adduser --system --shell /bin/bash --gecos 'Aptly repository' --group --disabled-password --home /home/aptly aptly
adduser --system --shell /bin/bash --gecos '<service-description>' --group --disabled-password --home /home/<username> <username>
```
**List Users**
To list all users from the `/etc/passwd` file, use:
To list all users from the `/etc/passwd` file in alphabetical order, use:
```bash
awk -F':' '{ print $1}' /etc/passwd | sort
```
This command extracts the usernames from the `/etc/passwd` file and sorts them in alphabetical order.
### Super User Management
**Disable Root Login**
@ -124,7 +112,7 @@ To disable root login via SSH, perform the following steps:
nano /etc/ssh/sshd_config
```
Comment out the line containing `PermitRootLogin` by adding a `#` at the beginning of the line.
Comment out the line containing `PermitRootLogin`.
2. **Change Shell for Root User:**
@ -132,34 +120,36 @@ To disable root login via SSH, perform the following steps:
nano /etc/passwd
```
Find the line starting with `root` and change `/bin/bash` to `/sbin/nologin` to disable login for the root user.
Save and close the file. Restart the SSH service for changes to take effect:
Find the line starting with `root` and change `/bin/bash` to `/sbin/nologin`.
```bash
systemctl restart ssh
```
**Sudo Management**
**Add User to Sudo Group**
```bash
adduser fabrice sudo
adduser <username> sudo
```
**Update Sudoers File to Remove Password Requirement**
Edit the sudoers file:
Edit the sudoers file with the default editor:
```bash
visudo
```
Edit the sudoers file with `nano`:
```bash
EDITOR=nano visudo
```
Add the following line to allow the user to execute commands without a password:
```bash
fabrice ALL=(ALL) NOPASSWD:ALL
<username> ALL=(ALL) NOPASSWD:ALL
```
### Switch User
@ -167,82 +157,74 @@ fabrice ALL=(ALL) NOPASSWD:ALL
**Switch to Another User as Sudoer**
```bash
sudo -i -u postgres
sudo -i -u <username>
```
This command switches to the `postgres` user with sudo privileges.
**Switch to Another User as Root**
```bash
su - postgres
su - <username>
```
This command switches to the `postgres` user with root privileges.
**Run command as specific user**
```bash
sudo -u www-data somecommand and arguments
sudo -u <username> <command>
```
**Change shell of a user**
```bash
chsh -s /bin/bash www-data
chsh -s /usr/sbin/nologin www-data
chsh -s /bin/bash <username>
chsh -s /usr/sbin/nologin <username>
```
**Change user with specific shell**
```bash
sudo -u www-data bash
sudo -u <username> bash
```
## System Management
**Ensure hostname or add alias**
Set or update the hostname for your server.
```bash
nano /etc/hosts
# Add the hostname alias:
# 127.0.1.1 local.servername.domain.com
# 127.0.1.1 <hostname-intranet>
nano /etc/hostname
# Set the main hostname:
# 127.0.1.1 servername.domain.com servername
# 127.0.1.1 <hostname-intranet> <hostname-short>
hostnamectl set-hostname servername.domain.com
hostnamectl set-hostname <hostname-intranet>
```
**Tar backup for a large number of small files**
These commands create backups using `tar` and transfer them securely over SSH.
Create a tar archive and transfer it to a remote server:
```bash
tar -c /path/to/dir | ssh fabrice@servername.domain.com 'tar -xvf - -C /absolute/path/to/remotedir'
tar -c /path/to/dir | ssh <username>@<hostname-intranet> 'tar -xvf - -C /absolute/path/to/remotedir'
```
Compress and transfer a folder, then store it as a .tar.gz file:
Compress and transfer a folder, then store it as a `.tar.gz` file:
```bash
tar zcvf - /folder | ssh fabrice@servername.domain.com "cat > /backup/folder.tar.gz"
tar zcvf - /folder | ssh <username>@<hostname-intranet> "cat > /backup/folder.tar.gz"
```
Transfer a compressed .tar.gz file and extract it on the remote server:
Transfer a compressed `.tar.gz` file and extract it on the remote server:
```bash
cat folder.tar.gz | ssh fabrice@servername.domain.com "tar zxvf -"
cat folder.tar.gz | ssh <username>@<hostname-intranet> "tar zxvf -"
```
Alternative method: change directory on the remote server before extracting:
Alternative: change directory on the remote server before extracting:
```bash
cat folder.tar.gz | ssh fabrice@servername.domain.com "cd /path/to/dest/; tar zxvf -"
cat folder.tar.gz | ssh <username>@<hostname-intranet> "cd /path/to/dest/; tar zxvf -"
```
**List time zones**
@ -266,15 +248,13 @@ Alternatively, manually set the time zone by linking the correct file:
```bash
mv /etc/localtime /etc/localtime-old
ln -s /usr/share/zoneinfo/America/Toronto /etc/localtime
ln -s /usr/share/zoneinfo/<timezone> /etc/localtime
```
**Find a specific service**
Search for a specific service running on your system.
```bash
systemctl list-units --type=service | grep php
systemctl list-units --type=service | grep <service-name>
```
### Change password of a tar/openssl archive
@ -283,66 +263,61 @@ systemctl list-units --type=service | grep php
To decrypt an `openssl`-encrypted archive using a password stored in a file:
1. **Store your password in this file.**
```bash
nano $HOME/xyz001.txt
```
2. **Decrypt the archive**
Decrypt the archive using the password stored in xyz001.txt.
1. **Store your password in a temporary file:**
```bash
openssl aes-256-cbc -d -pbkdf2 -in servername-backup.tar.gz -out servername-backup.tar -pass file:$HOME/xyz001.txt
nano $HOME/<filename>
```
3. **Re-encrypt the archive with a new password**
2. **Decrypt the archive:**
```bash
nano $HOME/xyz001.txt
openssl aes-256-cbc -e -pbkdf2 -in servername-backup.tar -out servername-backup-new.tar.gz -pass file:$HOME/xyz001.txt
rm $HOME/xyz001.txt
openssl aes-256-cbc -d -pbkdf2 -in <archive>.tar.gz -out <archive>.tar -pass file:$HOME/<filename>
```
3. **Re-encrypt the archive with a new password:**
```bash
nano $HOME/<filename>
openssl aes-256-cbc -e -pbkdf2 -in <archive>.tar -out <archive>-new.tar.gz -pass file:$HOME/<filename>
rm $HOME/<filename>
```
**Decode / Extract**
To decrypt and extract the contents of an encrypted archive directly into a directory:
Decrypt and extract the contents directly into a directory:
```bash
nano $HOME/xyz001.txt
openssl aes-256-cbc -d -pbkdf2 -in servername-backup.tar.gz -pass file:xyz001.txt | tar xz -C .
rm $HOME/xyz001.txt
nano $HOME/<filename>
openssl aes-256-cbc -d -pbkdf2 -in <archive>.tar.gz -pass file:<filename> | tar xz -C .
rm $HOME/<filename>
```
### Verify two possibly identical folders recursively
**With `diff`**
Check for differences between two directories, comparing all files recursively:
```bash
diff -r servername-files/data/servername-repositories/ servername-repositories/
diff -r <dir1>/ <dir2>/
```
Outputs any differences found between the two directories.
**With `rsync`**
Use `rsync` to show differences without copying any data:
Dry run — shows differences without copying any data:
```bash
rsync -avn servername-files/data/servername-repositories/ servername-repositories/
rsync -avn <dir1>/ <dir2>/
```
- The `-n` flag means this is a dry run, which won’t make any changes.
- `-n`: dry run, no changes made.
**With `cmp`**
This script compares files in two directories and identifies any differences between matching file names.
```bash
#!/bin/bash
dir1="servername-files/data/servername-repositories/"
dir2="servername-repositories/"
dir1="<dir1>/"
dir2="<dir2>/"
# Check if both directories exist before proceeding.
if [ ! -d "$dir1" ] || [ ! -d "$dir2" ]; then
@ -361,149 +336,12 @@ for file1 in $(find "$dir1" -type f); do
done
```
## USB Devices
### Test USB Key
**Device Information**
Check if the system recognizes the device and show the latest system messages related to USB devices being connected:
```bash
lsusb
dmesg | tail -n 20
```
**Find Mount Points and Device Information**
Identify mount points, partitions, and other relevant details of mounted devices:
```bash
lsblk -f
df -h | grep /dev/sdc
findmnt /dev/sdc1
mount | grep /dev/sd
```
**Print Detailed Information About the USB Key**
View detailed partition and disk information:
```bash
fdisk -l /dev/sdc
```
**Test the File System**
Check and repair the filesystem on the USB key:
```bash
fsck /dev/sdc1
```
**Test Data Integrity**
Perform read/write tests to ensure the integrity of the USB key:
1. **Unmount the USB Key** (if mounted):
```bash
umount /media/fabrice/BD48-F8BB
```
2. **Write Test**:
```bash
dd if=/dev/zero of=/dev/sdc bs=4M count=256 status=progress
```
3. **Read Test**:
```bash
dd if=/dev/sdc of=/dev/null bs=4M count=256 status=progress
```
**Check for Bad Blocks**
Identify any bad sectors on the USB key:
- **Read-only test**:
```bash
badblocks -v /dev/sdc
```
- **Non-destructive read-write test**:
```bash
badblocks -nsv /dev/sdc
```
- The `-n` option performs a non-destructive read-write test.
- The `-s` option shows progress.
- The `-v` option is for verbose output.
**Perform a SMART Test**
Run SMART diagnostics to test the health of the USB key:
1. **Start a short SMART test**:
```bash
smartctl -t short /dev/sdc
```
2. **View test results**:
```bash
smartctl -a /dev/sdc
```
**Benchmark the Speed**
Measure the read speed of the USB key:
```bash
hdparm -t /dev/sdc
```
**Unmount and Safely Remove**
Unmount the USB key and safely remove it from the system:
```bash
umount /mnt/usb
eject /dev/sdc
```
**Switching two USB keys**
The following commands copy data between two USB drives, format one of them, and restore the data.
```bash
cp -r /media/fabrice/465A-759B "/tmp/Michael Allison"
umount /dev/sdc1
mkfs.vfat /dev/sdc1
umount /dev/sdc1
dd if=/dev/sdc of=/tmp/usb_image.img bs=4M status=progress
mkfs.vfat /dev/sdc1
cp -r "/tmp/Michael Allison" /media/fabrice/D67D-ADF8
umount /dev/sdc1
dd if=/tmp/usb_image.img of=/dev/sdc bs=4M status=progress
sync
```
## Diagnosis
### Debian Upgrade Issues
**Apt Logs**
View the APT logs to check for package installation and updates history:
```bash
less /var/log/apt/history.log
```
@ -533,19 +371,19 @@ journalctl -b | grep -i "gnome-shell"
**Download and Install Fonts**
1. **Download the Font Archive**:
1. **Download the Font Archive:**
```bash
wget https://path/to/font/archive.tar.gz
wget https://<font-archive-url>
```
2. **Extract the Font Files**:
2. **Extract the Font Files:**
```bash
tar -xzvf font-archive.tar.gz
tar -xzvf <font-archive>.tar.gz
```
3. **Copy the Font Files to the Local Fonts Directory**:
3. **Copy the Font Files:**
**Local font directory**
@ -567,14 +405,12 @@ journalctl -b | grep -i "gnome-shell"
**Update the Font Cache**
**Force a Reload of the Installed Font Cache**:
```bash
sudo su -
fc-cache -fv
fc-cache -frv
```
- **`-f`**: Force re-generation of apparently up-to-date cache files, overriding the timestamp checking.
- **`-f`**: Force re-generation of cache files, overriding timestamp checking.
- **`-r`**: Erase all existing cache files and rescan.
- **`-v`**: Display status information while busy.

314
notes/storage.md Normal file
View File

@ -0,0 +1,314 @@
# Storage
## Table of Contents
- [Storage](#storage)
- [Table of Contents](#table-of-contents)
- [Placeholders](#placeholders)
- [Drive Information](#drive-information)
- [List Block Devices](#list-block-devices)
- [List Mountpoints and Usage](#list-mountpoints-and-usage)
- [Inspect fstab](#inspect-fstab)
- [Find Device Path from UUID](#find-device-path-from-uuid)
- [Power On Hours](#power-on-hours)
- [Partitions and Filesystems](#partitions-and-filesystems)
- [Mounting](#mounting)
- [SMART Diagnostics](#smart-diagnostics)
- [Badblocks](#badblocks)
- [Cloning drives and images with dd](#cloning-drives-and-images-with-dd)
- [Benchmarking](#benchmarking)
- [USB Devices](#usb-devices)
- [Device Recognition](#device-recognition)
- [Test Device Integrity](#test-device-integrity)
- [Switching Two USB Keys](#switching-two-usb-keys)
## Placeholders
Replace the placeholders below with the appropriate values for your setup:
- **Devices**
- Block device: `<device>` (e.g., /dev/sda)
- Partition: `<partition>` (e.g., /dev/sda1)
- UUID: `<uuid>` (e.g., a1b2c3d4-e5f6-7890-abcd-ef1234567890)
- **Paths**
- Mount point: `<mountpoint>` (e.g., /mnt/media)
- Directory path: `<dirpath>` (e.g., /mnt/data)
## Drive Information
### List Block Devices
Check all attached block devices by type:
```bash
ls /dev/sd*
ls /dev/nv*
```
### List Mountpoints and Usage
**Tree view of all block devices with sizes and mountpoints**
```bash
lsblk
lsblk -f
```
- `-f`: Add filesystem type, label, and UUID to the tree.
**Disk space usage for all mounted filesystems**
```bash
df -h
```
- `-h`: Human-readable.
**Disk space usage for a specific device**
```bash
df -h | grep <device>
```
### Inspect fstab
**View all configured mount entries**
```bash
cat /etc/fstab
```
### Find Device Path from UUID
Using `lsblk`:
```bash
lsblk -o NAME,UUID,MOUNTPOINT
```
Using `blkid`:
```bash
blkid | grep <uuid>
blkid -U <uuid>
```
### Power On Hours
Check power-on hours across multiple drives at once:
```bash
for dev in /dev/sd{a,b,c,d}; do echo -n "$dev: "; smartctl -a $dev | grep "Power_On_Hours"; done
```
## Partitions and Filesystems
**View partition table and disk details:**
```bash
fdisk -l <device>
```
**Check and repair a filesystem:**
```bash
fsck <partition>
```
## Mounting
**Validate all `fstab` entries**
```bash
findmnt --verify --verbose
```
**Dry-run mount of all fstab entries, reports what would succeed or fail**
```bash
mount -fav
```
- `-f`: Fake mount — goes through the motions without making the actual syscall.
- `-a`: Process all entries in `/etc/fstab` (excluding `noauto`).
- `-v`: Verbose output.
**Find mount point details:**
```bash
findmnt <device>
mount | grep <device>
```
**Apply fstab changes without rebooting**
```bash
mount -o remount -a
```
**Unmount and Safely Remove**
```bash
umount <device>
eject <device>
```
## SMART Diagnostics
**Get full SMART information for a drive:**
```bash
smartctl -a <device>
```
**Get drive identity and model info only:**
```bash
smartctl -i <device>
```
**Run a short SMART test:**
```bash
smartctl -t short <device>
```
**Run a long SMART test:**
```bash
smartctl -t long <device>
```
**View test results:**
```bash
smartctl -a <device>
```
**Check SMART device statistics (error counters, etc.):**
```bash
smartctl -A <device>
```
## Badblocks
**Read-only test:**
```bash
badblocks -v <device>
```
**Non-destructive read-write test:**
`Save to RAM -> Write Pattern -> Verify -> Restore`
```bash
badblocks -nsv <device>
```
- `-n`: Non-destructive read-write test.
- `-s`: Show progress.
- `-v`: Verbose output.
- Warning: If the computer loses power or the kernel panics after Step 2 but before Step 4, that specific block of data will be lost
**Destructive write test**
Overwrites all data — use only on blank drives or drives to be deleted.
```bash
badblocks -wsv <device>
```
## Cloning drives and images with dd
**Clone a drive or create an image:**
```bash
dd if=<source-device> of=<destination-device> bs=4M status=progress
dd if=<source-device> of=<image-file>.img bs=4M status=progress
```
**Restore from image:**
```bash
dd if=<image-file>.img of=<destination-device> bs=4M status=progress
sync
```
## Benchmarking
**Measure read speed with `hdparm`:**
```bash
hdparm -t <device>
```
## USB Devices
### Device Recognition
Check if the system recognizes the device and show recent USB-related kernel messages:
```bash
lsusb
dmesg | tail -n 20
```
### Test Device Integrity
Warning: The write test is destructive. All data on <device> will be permanently overwritten. Verify the target device with lsblk before running.
1. **Unmount the device** (if mounted):
```bash
umount <mountpoint>
```
2. **Write test:**
```bash
dd if=/dev/zero of=<device> bs=4M count=256 status=progress
```
3. **Read test:**
```bash
dd if=<device> of=/dev/null bs=4M count=256 status=progress
```
4. **Check for bad blocks:**
```bash
badblocks -wsv <device>
```
5. **Run a SMART test:**
```bash
smartctl -t short <device>
smartctl -a <device>
```
### Switching Two USB Keys
Copy data off, reformat, and restore:
```bash
cp -r /media/<username>/<volume-label> /tmp/<backup-folder>
umount <device-partition>
mkfs.vfat <device-partition>
cp -r /tmp/<backup-folder> /media/<username>/<new-volume-label>
umount <device-partition>
```
Or clone with `dd`:
```bash
dd if=<source-device> of=/tmp/usb_image.img bs=4M status=progress
mkfs.vfat <device-partition>
dd if=/tmp/usb_image.img of=<destination-device> bs=4M status=progress
sync
```