Compare commits
5 Commits
main
...
setups-dra
| Author | SHA1 | Date | |
|---|---|---|---|
| 497bbb6324 | |||
| 385eee64cb | |||
| b1ee30dd80 | |||
| efccada19b | |||
| ad1fefc02a |
32
README.md
32
README.md
@ -7,33 +7,24 @@
|
|||||||
This repository is structured into several key directories:
|
This repository is structured into several key directories:
|
||||||
|
|
||||||
- **scripts/**: Contains individual scripts for various tasks. Currently, it includes:
|
- **scripts/**: Contains individual scripts for various tasks. Currently, it includes:
|
||||||
- `scripts/library/`: Libraries used by Python scripts.
|
|
||||||
- `venv_utils.py`: Utility functions for creating, activating, and managing Python virtual environments.
|
- `video_remove_audio.py`: A script for removing audio from video files.
|
||||||
- `change_case.py`: A script for renaming files and directories by changing their case.
|
|
||||||
- `efi-update-mirror.sh`: Script for syncing a two drive EFI mirror.
|
|
||||||
- `video_autoreduce.py`: A script for automatic resolution reduction of video files.
|
|
||||||
- `video_autoreduce_rename.py`: A script for automated renaming of video files post resolution reduction.
|
|
||||||
- `video_manage_audio.py`: A script for removing audio from video files.
|
|
||||||
- `video_manage_subtitles.py`: A script for removing subtitles from video files.
|
|
||||||
- `video_repackage_mkv.py`: Recursively repackage non-MKV video files into Matroska containers.
|
|
||||||
|
|
||||||
- **notes/**: A collection of markdown files containing notes on various topics, including:
|
- **notes/**: A collection of markdown files containing notes on various topics, including:
|
||||||
- `brother.md`: Brother printer setup and troubleshooting on Linux.
|
|
||||||
- `btrfs.md`: Btrfs filesystem management, balances, scrubs, snapshots, and drive replacement.
|
- `brother.md`: Information about Brother printers on Linux.
|
||||||
- `chrome-driver.md`: ChromeDriver setup and usage.
|
- `btrfs.md`: Btrfs filesystem configuration and tips.
|
||||||
- `debian-packaging.md`: Building and maintaining Debian packages.
|
- `chrome-driver.md`: Notes on ChromeDriver setup and usage on Linux.
|
||||||
|
- `debian packaging.md`: Debian packaging guidelines.
|
||||||
- `dns.md`: DNS configuration and troubleshooting.
|
- `dns.md`: DNS configuration and troubleshooting.
|
||||||
- `efi-repair.md`: EFI partition repair guide.
|
- `linux.md`: General Linux tips.
|
||||||
- `efi-update-mirror.md`: Redundant root with EFI mirror setup guide.
|
|
||||||
- `linux.md`: General Linux commands — users, system management, archiving, and diagnostics.
|
|
||||||
- `pdf.md`: PDF manipulation with Linux command line.
|
- `pdf.md`: PDF manipulation with Linux command line.
|
||||||
- `pdftk.md`: PDF Toolkit usage.
|
- `pdftk.md`: PDF Toolkit usage.
|
||||||
- `pip-packaging.md`: Packaging and publishing Python projects with pip.
|
- `pip packaging.md`: Packaging Python projects with pip.
|
||||||
- `ssh.md`: SSH configuration, key management, and tunneling tips.
|
- `ssh.md`: Secure Shell (SSH) configuration and tips.
|
||||||
- `storage.md`: Storage tooling — smartctl, badblocks, dd, hdparm, fdisk, and fstab/mount management.
|
|
||||||
- `wordpress.md`: WordPress debugging and tips.
|
|
||||||
|
|
||||||
- **pages/other/**: Templates for other pages, such as the homepage of my Debian package repository. These are provided as inspiration and should not be used as-is.
|
- **pages/other/**: Templates for other pages, such as the homepage of my Debian package repository. These are provided as inspiration and should not be used as-is.
|
||||||
|
|
||||||
- `debrepo.fabq.ca.html`: This file serves as the homepage for my Debian package repository. It provides the instructions to add the repository to user's system's sources list and install packages securely.
|
- `debrepo.fabq.ca.html`: This file serves as the homepage for my Debian package repository. It provides the instructions to add the repository to user's system's sources list and install packages securely.
|
||||||
|
|
||||||
- **pages/errors/**: Templates for error pages. These are provided as inspiration and should not be used as-is.
|
- **pages/errors/**: Templates for error pages. These are provided as inspiration and should not be used as-is.
|
||||||
@ -44,7 +35,6 @@ This repository is structured into several key directories:
|
|||||||
|
|
||||||
- **setups/**: A collection of markdown files containing notes on configuring servers, including:
|
- **setups/**: A collection of markdown files containing notes on configuring servers, including:
|
||||||
- `debian_setup_aptly.md`: Comprehensive guide for installing Aptly on Debian.
|
- `debian_setup_aptly.md`: Comprehensive guide for installing Aptly on Debian.
|
||||||
- `debian_setup_dolibarr.md`: Comprehensive guide for installing Dolibarr on Debian.
|
|
||||||
- `debian_setup_gitea.md`: Comprehensive guide for installing Gitea on Debian.
|
- `debian_setup_gitea.md`: Comprehensive guide for installing Gitea on Debian.
|
||||||
|
|
||||||
## 📖 Documentation
|
## 📖 Documentation
|
||||||
|
|||||||
725
notes/btrfs.md
725
notes/btrfs.md
@ -4,635 +4,542 @@
|
|||||||
|
|
||||||
- [BTRFS](#btrfs)
|
- [BTRFS](#btrfs)
|
||||||
- [Table of Contents](#table-of-contents)
|
- [Table of Contents](#table-of-contents)
|
||||||
- [Placeholders](#placeholders)
|
- [Information on Drives](#information-on-drives)
|
||||||
- [BTRFS Command Shorthands](#btrfs-command-shorthands)
|
|
||||||
- [Information on Filesystem](#information-on-filesystem)
|
- [Information on Filesystem](#information-on-filesystem)
|
||||||
|
- [Backup Procedures](#backup-procedures)
|
||||||
|
- [Recovery](#recovery)
|
||||||
- [Drive Manipulation](#drive-manipulation)
|
- [Drive Manipulation](#drive-manipulation)
|
||||||
- [Replace Drives](#replace-drives)
|
- [Replace Drives](#replace-drives)
|
||||||
- [Degraded Mount and Missing Device Removal](#degraded-mount-and-missing-device-removal)
|
|
||||||
- [Filesystem Manipulation](#filesystem-manipulation)
|
- [Filesystem Manipulation](#filesystem-manipulation)
|
||||||
- [Upgrading Btrfs Block Group Cache to V2](#upgrading-btrfs-block-group-cache-to-v2)
|
- [Upgrading Btrfs block group cache to V2](#upgrading-btrfs-block-group-cache-to-v2)
|
||||||
- [Defrag](#defrag)
|
|
||||||
- [Balances](#balances)
|
- [Balances](#balances)
|
||||||
- [Scrub](#scrub)
|
- [Scrub](#scrub)
|
||||||
- [Snapshots](#snapshots)
|
- [Snapshots](#snapshots)
|
||||||
- [Create Snapshots](#create-snapshots)
|
|
||||||
- [Delete Snapshots](#delete-snapshots)
|
|
||||||
- [Backup Procedures](#backup-procedures)
|
|
||||||
- [Recovery](#recovery)
|
|
||||||
- [Filesystem Check](#filesystem-check)
|
|
||||||
- [Diagnosis](#diagnosis)
|
|
||||||
|
|
||||||
## Placeholders
|
## Information on Drives
|
||||||
|
|
||||||
Replace the placeholders below with the appropriate values for your setup:
|
**List of Drives and Mountpoints**
|
||||||
|
|
||||||
- **Devices**
|
To check all attached drives:
|
||||||
- Block device: `<device>` (e.g., /dev/sda)
|
|
||||||
- Source device to replace: `<source-device>` (e.g., /dev/sdb)
|
|
||||||
- Target device: `<target-device>` (e.g., /dev/sdc)
|
|
||||||
- UUID: `<uuid>` (e.g., a1b2c3d4-e5f6-7890-abcd-ef1234567890)
|
|
||||||
|
|
||||||
- **Paths**
|
```bash
|
||||||
- Mount point: `<mountpoint>` (e.g., /mnt/media)
|
ls /dev/sd*
|
||||||
- Subvolume name: `<subvolume>` (e.g., root, home, backups, snapshots)
|
ls /dev/nv*
|
||||||
- Subvolume ID: `<subvolume-id>` (e.g., 257)
|
```
|
||||||
- Snapshot label: `<snapshot-label>` (e.g., 2024-09-15)
|
|
||||||
|
|
||||||
## BTRFS Command Shorthands
|
To view mountpoints and drive details such as names, sizes, and mountpoints:
|
||||||
|
|
||||||
Most `btrfs` subcommands accept shortened aliases. The table below lists the common ones used throughout this document.
|
```bash
|
||||||
|
lsblk
|
||||||
|
df -h
|
||||||
|
cat /etc/fstab
|
||||||
|
```
|
||||||
|
|
||||||
| Long form | Short form |
|
**Drive Information**
|
||||||
| ----------------------------- | ----------------------- |
|
|
||||||
| `btrfs filesystem` | `btrfs fi` |
|
To get detailed information and serial number of a specific drive:
|
||||||
| `btrfs filesystem show` | `btrfs fi show` |
|
|
||||||
| `btrfs filesystem usage` | `btrfs fi usage` |
|
```bash
|
||||||
| `btrfs filesystem df` | `btrfs fi df` |
|
smartctl -i /dev/sdc
|
||||||
| `btrfs filesystem resize` | `btrfs fi resize` |
|
```
|
||||||
| `btrfs filesystem defragment` | `btrfs fi defrag` |
|
|
||||||
| `btrfs subvolume` | `btrfs sub` |
|
**Find the Device Path from UUID**
|
||||||
| `btrfs subvolume list` | `btrfs sub list` |
|
|
||||||
| `btrfs subvolume snapshot` | `btrfs sub snap` |
|
Using lsblk:
|
||||||
| `btrfs subvolume delete` | `btrfs sub del` |
|
|
||||||
| `btrfs subvolume get-default` | `btrfs sub get-default` |
|
```bash
|
||||||
| `btrfs subvolume set-default` | `btrfs sub set-default` |
|
lsblk -o NAME,UUID,MOUNTPOINT
|
||||||
| `btrfs device` | `btrfs dev` |
|
```
|
||||||
| `btrfs device add` | `btrfs dev add` |
|
|
||||||
| `btrfs device usage` | `btrfs dev usage` |
|
Using blkid:
|
||||||
| `btrfs device stats` | `btrfs dev stats` |
|
|
||||||
| `btrfs balance start` | `btrfs bal start` |
|
```bash
|
||||||
| `btrfs balance status` | `btrfs bal status` |
|
blkid | grep <UUID>
|
||||||
| `btrfs balance cancel` | `btrfs bal cancel` |
|
```
|
||||||
| `btrfs inspect-internal` | `btrfs insp` |
|
|
||||||
|
|
||||||
## Information on Filesystem
|
## Information on Filesystem
|
||||||
|
|
||||||
**Show Basic Filesystem Information**
|
**Show Basic Filesystem Information**
|
||||||
|
|
||||||
Display basic information (size, IDs, paths, etc.) for the specified mountpoint:
|
To display basic information (size, IDs, paths, etc.) for the specified mountpoint:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs fi show <mountpoint>
|
btrfs fi show /mnt/media/
|
||||||
```
|
```
|
||||||
|
|
||||||
**Display detailed usage information**
|
**Display Detailed Usage Information**
|
||||||
|
|
||||||
To show detailed usage information (allocated, unallocated, free, used, etc.) for the specified mountpoint:
|
To show detailed usage information (allocated, unallocated, etc.) for the specified mountpoint:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs fi usage <mountpoint>
|
btrfs fi usage /mnt/media
|
||||||
```
|
```
|
||||||
|
|
||||||
**Display detailed usage information as a table**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
btrfs fi usage -T <mountpoint>
|
|
||||||
```
|
|
||||||
|
|
||||||
- `-T` The tabular flag gives you a nice grid that shows exactly how much Data, Metadata, and System space is allocated per device.
|
|
||||||
|
|
||||||
**Display Detailed Allocation Information**
|
**Display Detailed Allocation Information**
|
||||||
|
|
||||||
View block groups and used space:
|
To view detailed allocation information (block groups, used space) for the specified mountpoint:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs fi df <mountpoint>
|
btrfs fi df /mnt/media
|
||||||
```
|
```
|
||||||
|
|
||||||
**Get Detailed Device Usage Statistics**
|
**Get Detailed Device Usage Statistics**
|
||||||
|
|
||||||
Physical size, unallocated space, RAID levels, etc.:
|
To get detailed device usage statistics (physical size, unallocated space, RAID levels, etc.) for a BTRFS filesystem:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs device usage <mountpoint>
|
btrfs device usage /mnt/media
|
||||||
```
|
```
|
||||||
|
|
||||||
**Scan and Display BTRFS Information**
|
**Scan and Display BTRFS Information**
|
||||||
|
|
||||||
Scan all devices or a specific drive:
|
To scan and display BTRFS information for all devices or a specific drive:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs device scan
|
btrfs device scan /dev/sda/
|
||||||
btrfs device scan <device>
|
|
||||||
```
|
```
|
||||||
|
|
||||||
**Retrieve Statistics and Error Information**
|
**Retrieve Statistics and Error Information**
|
||||||
|
|
||||||
Read errors, write errors, flush errors, etc.:
|
To get statistics and error information (read errors, write errors, flush errors, etc.) for the specified mountpoint:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs device stats <mountpoint>
|
btrfs device stats /mnt/media
|
||||||
```
|
```
|
||||||
|
|
||||||
**Reset Device Error Counters**
|
|
||||||
|
|
||||||
Reset all per-device error counters to zero after acknowledging them:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
btrfs device stats --reset <mountpoint>
|
|
||||||
btrfs device stats -z <mountpoint>
|
|
||||||
```
|
|
||||||
|
|
||||||
- `-z` / `--reset`: Zeroes the counters after printing. Useful after a known event you've already investigated.
|
|
||||||
|
|
||||||
**List BTRFS Subvolumes**
|
**List BTRFS Subvolumes**
|
||||||
|
|
||||||
|
To list BTRFS subvolumes:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs subvolume list <mountpoint>
|
btrfs subvolume list /
|
||||||
|
btrfs subvolume list /home/fabrice
|
||||||
|
btrfs subvolume list /mnt/workbench
|
||||||
```
|
```
|
||||||
|
|
||||||
**Default Subvolume**
|
**Default Subvolume**
|
||||||
|
|
||||||
Check if a non-standard subvolume is set as the default:
|
To check if a non-standard subvolume is set as the default:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs subvol get-default <mountpoint>
|
btrfs subvol get-default /mnt/tmp/
|
||||||
btrfs subvol list <mountpoint>
|
btrfs subvol list /mnt/tmp/
|
||||||
```
|
```
|
||||||
|
|
||||||
Change the default subvolume:
|
To change the default subvolume if a non-standard one is set:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs subvol set-default <subvolume-id> <mountpoint>
|
btrfs subvol set-default 257 /mnt/tmp/
|
||||||
```
|
```
|
||||||
|
|
||||||
**Verify Current Cache Version**
|
**Verify Current Cache Version**
|
||||||
|
|
||||||
Check if your filesystem is using cache V1 by device:
|
To check if your filesystem is using cache V1 by device:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs inspect-internal dump-super -f <device> | grep cache_generation
|
btrfs inspect-internal dump-super -f /dev/<device> | grep cache_generation
|
||||||
```
|
```
|
||||||
|
|
||||||
By UUID:
|
To check if your filesystem is using cache V1 by UUID:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs inspect-internal dump-super -f $(blkid -U <uuid>) | grep cache_generation
|
btrfs inspect-internal dump-super -f $(blkid -U <UUID>) | grep cache_generation
|
||||||
```
|
```
|
||||||
|
|
||||||
- If `cache_generation` is present, cache V1 is in use. If absent, the filesystem is already using V2.
|
- If cache_generation is present, it indicates cache V1 is in use. If it's absent, the filesystem is already using V2.
|
||||||
|
|
||||||
|
## Backup Procedures
|
||||||
|
|
||||||
|
**Desktop Backup: Root and Home**
|
||||||
|
|
||||||
|
1. Mount snapshot location:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mount UUID=394decca-4780-47c9-9ae3-e4d03681a791 -o subvol=snapshots /mnt/snapshots
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Create snapshots for root and home:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
btrfs subvolume snapshot / "/mnt/snapshots/root/2021-05-23 - Fedora 34 upgrade"
|
||||||
|
btrfs subvolume snapshot /home "/mnt/snapshots/home/2021-05-23 - Fedora 34 upgrade"
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Unmount after creating snapshots:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
umount /mnt/snapshots
|
||||||
|
```
|
||||||
|
|
||||||
|
**Data Backup: Workbench, Documents, Education**
|
||||||
|
|
||||||
|
1. Mount snapshot location:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mount UUID=72e1770a-9fc0-461e-88d3-db640ff53dd9 -o subvol=snapshots /mnt/snapshots
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Create snapshots for multiple directories:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
btrfs subvolume snapshot /mnt/workbench "/mnt/snapshots/workbench/2021-05-23 - Fedora 34 upgrade"
|
||||||
|
btrfs subvolume snapshot /home/fabrice/Documents "/mnt/snapshots/Documents/2021-05-23 - Fedora 34 upgrade"
|
||||||
|
btrfs subvolume snapshot /home/fabrice/Education "/mnt/snapshots/Education/2021-05-23 - Fedora 34 upgrade"
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Unmount after creating snapshots:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
umount /mnt/snapshots
|
||||||
|
```
|
||||||
|
|
||||||
|
**STOR1 Fedora Backup**
|
||||||
|
|
||||||
|
1. Mount snapshot location:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mount UUID=e4fd608e-cfe8-4c10-b6d0-03b05bae8aa6 -o subvol=snapshots /mnt/snapshots
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Create snapshot:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
btrfs subvolume snapshot / "/mnt/snapshots/root/2021-06-06"
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Unmount after creating snapshot:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
umount /mnt/snapshots
|
||||||
|
```
|
||||||
|
|
||||||
|
**STOR1 Debian Backup**
|
||||||
|
|
||||||
|
1. Mount snapshot location:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mount UUID=c9a77f3c-626f-47bd-b4e3-9a094bea287f -o subvol=snapshots /mnt/snapshots
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Create snapshot:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
btrfs subvolume snapshot / "/mnt/snapshots/root/2021-07-12 - post mostly setup"
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Unmount after creating snapshot:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
umount /mnt/snapshots
|
||||||
|
```
|
||||||
|
|
||||||
|
**STOR2 Backup**
|
||||||
|
|
||||||
|
1. Mount snapshot location:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mount UUID=30bd5e0e-e781-4e87-8fb8-ea5606403b15 -o subvol=snapshots /mnt/snapshots
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Create snapshot:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
btrfs subvolume snapshot / "/mnt/snapshots/root/2021-06-06 - Fedora 34"
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Unmount after creating snapshot:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
umount /mnt/snapshots
|
||||||
|
```
|
||||||
|
|
||||||
|
## Recovery
|
||||||
|
|
||||||
|
**Mount a Subvolume with Recovery Options**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mount -o recovery,subvol=backups UUID=aa5c1d34-ecba-42a9-9339-8f7879d47536 /mnt/tmp
|
||||||
|
```
|
||||||
|
|
||||||
|
**Clear Cache During Mount**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mount -o clear_cache,subvol=backups UUID=aa5c1d34-ecba-42a9-9339-8f7879d47536 /mnt/tmp
|
||||||
|
```
|
||||||
|
|
||||||
|
**Data Restoration**
|
||||||
|
|
||||||
|
To restore data using `btrfs restore`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
btrfs restore -D /dev/sdb
|
||||||
|
```
|
||||||
|
|
||||||
## Drive Manipulation
|
## Drive Manipulation
|
||||||
|
|
||||||
**Mount Whole Drive**
|
**Mount Whole Drive**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
mount UUID=<uuid> <mountpoint>
|
mount UUID=c9a77f3c-626f-47bd-b4e3-9a094bea287f /mnt/tmp
|
||||||
```
|
|
||||||
|
|
||||||
**Mount Subvolume by Name**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mount UUID=<uuid> -o subvol=<subvolume> <mountpoint>
|
|
||||||
```
|
```
|
||||||
|
|
||||||
**Mount Subvolume by ID**
|
**Mount Subvolume by ID**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs subvol list /
|
btrfs subvol list /
|
||||||
mount -o subvolid=<subvolume-id> /dev/disk/by-uuid/<uuid> <mountpoint>
|
mount -o subvolid=5 /dev/disk/by-uuid/7a22514b-594a-43a3-8fdd-4df1530b5465 /mnt
|
||||||
```
|
|
||||||
|
|
||||||
**Mount Read-Only**
|
|
||||||
|
|
||||||
Mount a partition in read-only mode, useful for forensics or recovery without risking further writes:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mount -r <device> <mountpoint>
|
|
||||||
```
|
|
||||||
|
|
||||||
**Remount with Performance Options**
|
|
||||||
|
|
||||||
Apply common performance mount options to a live filesystem without unmounting:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mount -o remount,noatime,compress=zstd:3,autodefrag,space_cache=v2 <mountpoint>
|
|
||||||
```
|
|
||||||
|
|
||||||
**Remount with Default Options**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mount -o remount,defaults,noatime,compress=zstd:3 <mountpoint>
|
|
||||||
```
|
```
|
||||||
|
|
||||||
**Add a New Drive**
|
**Add a New Drive**
|
||||||
|
|
||||||
|
To add a new drive to an existing BTRFS setup:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs device add <device> <mountpoint>
|
btrfs device add /dev/sdf /mnt/media/
|
||||||
```
|
```
|
||||||
|
|
||||||
**Resize Filesystem**
|
**Resize Filesystem**
|
||||||
|
|
||||||
Grow the filesystem on a specific device to its maximum:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs filesystem resize 1:max <mountpoint>
|
btrfs filesystem resize 1:max /mnt/media/
|
||||||
```
|
```
|
||||||
|
|
||||||
**Create Subvolumes**
|
**Create Subvolumes**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs subvol create <mountpoint>/<subvolume>
|
btrfs subvol create /mnt/tmp/root
|
||||||
|
btrfs subvol create /mnt/tmp/snapshots
|
||||||
```
|
```
|
||||||
|
|
||||||
### Replace Drives
|
### Replace Drives
|
||||||
|
|
||||||
**Start the replacement process:**
|
**Replace the source drive with the target drive:**
|
||||||
|
|
||||||
Copies data from the old drive to the new drive while the filesystem remains mounted:
|
This command will start the replacement process where the data from the old drive (`/dev/sdb`) is copied over to the new drive (`/dev/sdj`).
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs replace start <source-device> <target-device> <mountpoint>
|
btrfs replace start /dev/sdb /dev/sdj /mnt/media
|
||||||
```
|
```
|
||||||
|
|
||||||
- `<source-device>`: Drive to be replaced.
|
- `/dev/sdb`: Source drive to be replaced.
|
||||||
- `<target-device>`: Drive to replace it with.
|
- `/dev/sdj`: Target drive to replace the source drive.
|
||||||
- `<mountpoint>`: Mount point of the BTRFS filesystem.
|
- `/mnt/media`: Mount point of the BTRFS filesystem.
|
||||||
|
|
||||||
**Monitor the progress:**
|
**Monitor the progress of the replacement:**
|
||||||
|
|
||||||
|
Once the replacement process has started, you can monitor its progress with the following command:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs replace status <mountpoint>
|
btrfs replace status /mnt/media
|
||||||
```
|
```
|
||||||
|
|
||||||
|
- This will print the current status of the drive replacement operation, showing how much data has been migrated.
|
||||||
|
|
||||||
**Monitor progress interactively:**
|
**Monitor progress interactively:**
|
||||||
|
|
||||||
|
For a more detailed, interactive status view of the replacement process, use the `-i` option:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs replace status -i <mountpoint>
|
btrfs replace status -i /mnt/media
|
||||||
```
|
```
|
||||||
|
|
||||||
- `-i`: Updates progress in real time.
|
- `-i`: This flag provides an interactive mode where the progress is updated in real time.
|
||||||
|
|
||||||
**Notes:**
|
**Notes:**
|
||||||
|
|
||||||
- `btrfs replace` works on a live mounted filesystem — no unmounting required.
|
- The `btrfs replace` command allows you to replace a faulty or underperforming drive without unmounting the filesystem, making it ideal for live systems.
|
||||||
- Useful for both failing drive replacement and capacity upgrades.
|
- It can be used for upgrading storage by replacing smaller drives with larger ones, or for replacing failing drives.
|
||||||
- Ensure the target drive has enough space to accommodate the source data.
|
- Ensure that the target drive has enough space to accommodate the data from the source drive.
|
||||||
|
|
||||||
### Degraded Mount and Missing Device Removal
|
|
||||||
|
|
||||||
Use when a drive has failed and you need to access the filesystem with the remaining devices.
|
|
||||||
|
|
||||||
**Mount in degraded mode:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mount -o ro,degraded <device> <mountpoint>
|
|
||||||
```
|
|
||||||
|
|
||||||
**Mount a specific subvolume in degraded mode:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mount -t btrfs -o degraded,subvol=<subvolume>,noatime,compress=zstd:3 UUID=<uuid> <mountpoint>
|
|
||||||
```
|
|
||||||
|
|
||||||
**Remove the missing device from the filesystem:**
|
|
||||||
|
|
||||||
Once mounted degraded, remove the placeholder for the missing drive:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
btrfs device remove missing <mountpoint>
|
|
||||||
```
|
|
||||||
|
|
||||||
- This cleans up the missing device slot so the filesystem no longer expects it.
|
|
||||||
- Only safe to run if data is intact on the remaining devices (e.g., RAID1 with one drive).
|
|
||||||
|
|
||||||
## Filesystem Manipulation
|
## Filesystem Manipulation
|
||||||
|
|
||||||
### Upgrading Btrfs Block Group Cache to V2
|
### Upgrading Btrfs block group cache to V2
|
||||||
|
|
||||||
**Non-root filesystems (running system):**
|
**From a running system non-root filesystems**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
mount -o remount,clear_cache,space_cache=v2 <mountpoint>
|
mount -o remount,clear_cache,space_cache=v2 /mnt/<mount-point>
|
||||||
```
|
```
|
||||||
|
|
||||||
**Root filesystem (running system):**
|
**From a running system on root**
|
||||||
|
|
||||||
1. Check if using cache V1:
|
Check if your filesystem is using cache V1:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs inspect-internal dump-super -f <device> | grep cache_generation
|
btrfs inspect-internal dump-super -f /dev/<device> | grep cache_generation
|
||||||
```
|
```
|
||||||
|
|
||||||
2. Enable cache V2 via GRUB:
|
Enable Cache V2
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
nano /etc/default/grub
|
nano /etc/default/grub
|
||||||
# Add to GRUB_CMDLINE_LINUX_DEFAULT or GRUB_CMDLINE_LINUX:
|
# Locate the line starting with GRUB_CMDLINE_LINUX_DEFAULT or GRUB_CMDLINE_LINUX and add the following options:
|
||||||
# rootflags=clear_cache,space_cache=v2
|
rootflags=clear_cache,space_cache=v2
|
||||||
```
|
```
|
||||||
|
|
||||||
Example:
|
Example:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash rootflags=clear_cache,space_cache=v2"
|
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash rootflags=clear_cache,space_cache=v2"
|
||||||
```
|
```
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
update-grub
|
update-grub
|
||||||
reboot
|
reboot
|
||||||
```
|
```
|
||||||
|
|
||||||
3. Verify the change:
|
Verify the Change
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs inspect-internal dump-super -f <device> | grep cache_generation
|
btrfs inspect-internal dump-super -f /dev/<device> | grep cache_generation
|
||||||
```
|
```
|
||||||
|
|
||||||
4. Remove `clear_cache` from GRUB after confirming:
|
Remove `clear_cache` Option
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
nano /etc/default/grub
|
nano /etc/default/grub
|
||||||
# Remove clear_cache from rootflags, then:
|
# Remove clear_cache from the rootflags.
|
||||||
update-grub
|
update-grub
|
||||||
```
|
```
|
||||||
|
|
||||||
**From a live system:**
|
**From a live system**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
apt update
|
apt update
|
||||||
apt install btrfs-progs
|
apt install btrfs-progs
|
||||||
lsblk -o NAME,UUID
|
lsblk -o NAME,UUID
|
||||||
blkid
|
blkid
|
||||||
mount -o clear_cache,space_cache=v2 /dev/disk/by-uuid/<uuid> <mountpoint>
|
mount -o clear_cache,space_cache=v2 /dev/disk/by-uuid/<UUID> /mnt
|
||||||
btrfs inspect-internal dump-super -f /dev/disk/by-uuid/<uuid> | grep cache_generation
|
btrfs inspect-internal dump-super -f /dev/disk/by-uuid/<UUID> | grep cache_generation
|
||||||
umount <mountpoint>
|
umount /mnt
|
||||||
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Defrag
|
|
||||||
|
|
||||||
**Standard recursive defrag with LZO compression:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
btrfs filesystem defrag -r -v -clzo <mountpoint>
|
|
||||||
```
|
|
||||||
|
|
||||||
- `-r`: Recursive.
|
|
||||||
- `-v`: Verbose.
|
|
||||||
- `-clzo`: Optional LZO compression to save space.
|
|
||||||
|
|
||||||
**Recursive defrag with Zstd compression, logged to file:**
|
|
||||||
|
|
||||||
Runs in the background with unbuffered output so the log file updates in real time:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
stdbuf -oL btrfs filesystem defrag -r -v -czstd <mountpoint> > /root/<date>-defrag.log 2>&1 &
|
|
||||||
```
|
|
||||||
|
|
||||||
- `stdbuf -oL`: Forces line-buffered stdout so log entries appear immediately.
|
|
||||||
- `-czstd`: Zstd compression (better ratio than LZO, available since kernel 5.1).
|
|
||||||
- `&`: Runs in the background; use `tail -f /root/<date>-defrag.log` to monitor.
|
|
||||||
|
|
||||||
## Balances
|
## Balances
|
||||||
|
|
||||||
**Full balance on nearly empty block groups:**
|
**Perform a Full Balance with Minimal Usage**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs balance start --full-balance -dusage=0 -musage=0 <mountpoint>
|
btrfs balance start --full-balance -dusage=0 -musage=0 /mnt/media/
|
||||||
```
|
```
|
||||||
|
|
||||||
- `--full-balance`: Default but with a warning if not specified.
|
- `--full-balance` is default but with a warning if not specified.
|
||||||
- `-dusage=0`: Only balance data block groups that are ~0% full.
|
- `-dusage=0` means only data block groups that are nearly empty (0% full) will be balanced.
|
||||||
- `-musage=0`: Only balance metadata block groups that are ~0% full.
|
- `-musage=0` means only metadata block groups that are nearly empty (0% full) will be balanced.
|
||||||
|
|
||||||
**Full balance on partially used block groups:**
|
**Perform a Full Balance on Partially Used Blocks**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs balance start --full-balance -dusage=50 -musage=50 <mountpoint>
|
btrfs balance start --full-balance -dusage=50 -musage=50 /mnt/media/
|
||||||
```
|
```
|
||||||
|
|
||||||
- `-dusage=50`: Include data block groups less than 50% full.
|
- `-dusage=50` means data block groups that are less than 50% full will be included in the balance process.
|
||||||
- `-musage=50`: Include metadata block groups less than 50% full.
|
- `-musage=50` means metadata block groups that are less than 50% full will also be balanced.
|
||||||
|
|
||||||
**Balance data in the background:**
|
**Balance data in the background**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs balance start --bg -d <mountpoint>
|
btrfs balance start --bg -d /mnt/media
|
||||||
```
|
```
|
||||||
|
|
||||||
**Balance metadata in the background:**
|
**Balance metadata in the background**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs balance start --bg -m <mountpoint>
|
btrfs balance start --bg -m /mnt/media
|
||||||
```
|
```
|
||||||
|
|
||||||
**Balance data and metadata in the background:**
|
**Balance data and metadata in the background**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs balance start --bg --full-balance -dusage=0 -musage=0 <mountpoint>
|
btrfs balance start --bg --full-balance -dusage=0 -musage=0 /mnt/media/
|
||||||
```
|
```
|
||||||
|
|
||||||
**Balance a limited number of chunks:**
|
**To balance 100 chunks of data**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs balance start --bg -dlimit=100 <mountpoint>
|
btrfs balance start --bg -dlimit=100 /mnt/media/
|
||||||
```
|
```
|
||||||
|
|
||||||
**Convert to RAID1:**
|
**Cancel Balance Operation**
|
||||||
|
|
||||||
Rebalances data and metadata to RAID1 profile. Use after adding a second drive or to switch from single to mirrored:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs balance start -mconvert=raid1 -dconvert=raid1 <mountpoint>
|
btrfs balance cancel /mnt/media/
|
||||||
```
|
```
|
||||||
|
|
||||||
**Cancel a balance:**
|
**Monitor Balance Status**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs balance cancel <mountpoint>
|
btrfs balance status /mnt/media/
|
||||||
```
|
|
||||||
|
|
||||||
**Monitor balance status:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
btrfs balance status <mountpoint>
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Scrub
|
## Scrub
|
||||||
|
|
||||||
**Start a scrub**
|
**Start a Scrub Operation**
|
||||||
|
|
||||||
The scrub operation verifies data integrity against checksums:
|
To start a scrub operation to verify data integrity:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs scrub start <mountpoint>
|
btrfs scrub start /mnt/media/
|
||||||
```
|
```
|
||||||
|
|
||||||
**Check scrub status:**
|
**Check Scrub Status**
|
||||||
|
|
||||||
|
To check the progress and status of the ongoing scrub:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs scrub status <mountpoint>
|
btrfs scrub status /mnt/media/
|
||||||
```
|
```
|
||||||
|
|
||||||
**Cancel a scrub:**
|
**Cancel a Scrub Operation**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs scrub cancel <mountpoint>
|
btrfs scrub cancel /mnt/media/
|
||||||
```
|
|
||||||
|
|
||||||
**Lower scrub I/O priority:**
|
|
||||||
|
|
||||||
Reduce the impact of a running scrub on system I/O by setting it to idle class:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ionice -c 3 -p $(pgrep btrfs-scrub)
|
|
||||||
```
|
|
||||||
|
|
||||||
- `-c 3`: Idle class — only uses I/O when no other process needs it.
|
|
||||||
|
|
||||||
**Watch scrub status and device stats:**
|
|
||||||
|
|
||||||
Continuously display scrub progress and per-device error counters:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
watch -n 10 "btrfs scrub status <mountpoint>; echo ''; btrfs device stats <mountpoint>"
|
|
||||||
```
|
|
||||||
|
|
||||||
**Watch scrub status and all drive temperatures:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
watch -n 5 "btrfs scrub status <mountpoint> && echo '' && \
|
|
||||||
smartctl --scan | awk '{print \$1}' | while read dev; do \
|
|
||||||
echo -n \"\$dev: \"; \
|
|
||||||
smartctl -A \$dev | grep -iE 'Temperature|Airflow_Temp' | awk '\
|
|
||||||
/Temperature_Celsius/ {print \$10 \"°C\"} \
|
|
||||||
/Airflow_Temperature_Cel/ {print \$10 \"°C\"} \
|
|
||||||
/Temperature:/ {print \$2 \"°C\"}' | head -n 1; \
|
|
||||||
done && echo '' && btrfs device stats <mountpoint>"
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Snapshots
|
## Snapshots
|
||||||
|
|
||||||
### Create Snapshots
|
**Create Snapshots**
|
||||||
|
|
||||||
1. **Mount the snapshots subvolume:**
|
1. **Mount snapshot subvolume**
|
||||||
|
|
||||||
```bash
|
|
||||||
mount UUID=<uuid> -o subvol=snapshots <mountpoint>
|
|
||||||
```
|
|
||||||
|
|
||||||
2. **Create a snapshot:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
btrfs subvolume snapshot <source-subvolume> "<mountpoint>/<snapshot-label>"
|
|
||||||
```
|
|
||||||
|
|
||||||
3. **Unmount after creating:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
umount <mountpoint>
|
|
||||||
```
|
|
||||||
|
|
||||||
### Delete Snapshots
|
|
||||||
|
|
||||||
1. **Mount the snapshots subvolume:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mount -o subvol=snapshots /dev/disk/by-uuid/<uuid> <mountpoint>
|
|
||||||
```
|
|
||||||
|
|
||||||
2. **List available snapshots:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
btrfs subvol list <mountpoint>
|
|
||||||
```
|
|
||||||
|
|
||||||
3. **Delete the desired snapshot:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
btrfs subvolume delete <mountpoint>/<snapshot-label>
|
|
||||||
```
|
|
||||||
|
|
||||||
4. **Unmount after deleting:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
umount <mountpoint>
|
|
||||||
```
|
|
||||||
|
|
||||||
## Backup Procedures
|
|
||||||
|
|
||||||
**Snapshot backup procedure:**
|
|
||||||
|
|
||||||
1. Mount snapshot location:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mount UUID=<uuid> -o subvol=snapshots <mountpoint>
|
|
||||||
```
|
|
||||||
|
|
||||||
2. Create snapshots for the desired subvolumes:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
btrfs subvolume snapshot / "<mountpoint>/root/<snapshot-label>"
|
|
||||||
btrfs subvolume snapshot /home "<mountpoint>/home/<snapshot-label>"
|
|
||||||
btrfs subvolume snapshot <source-subvolume> "<mountpoint>/<subvolume>/<snapshot-label>"
|
|
||||||
```
|
|
||||||
|
|
||||||
3. Unmount after creating snapshots:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
umount <mountpoint>
|
|
||||||
```
|
|
||||||
|
|
||||||
## Recovery
|
|
||||||
|
|
||||||
1. **Mount a Subvolume with Recovery Options:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mount -o recovery,subvol=<subvolume> UUID=<uuid> <mountpoint>
|
|
||||||
```
|
|
||||||
|
|
||||||
2. **Clear Cache During Mount:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mount -o clear_cache,subvol=<subvolume> UUID=<uuid> <mountpoint>
|
|
||||||
```
|
|
||||||
|
|
||||||
3. **Data Restoration with btrfs restore:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
btrfs restore -D <device>
|
|
||||||
```
|
|
||||||
|
|
||||||
### Filesystem Check
|
|
||||||
|
|
||||||
Run offline consistency checks on an unmounted BTRFS filesystem.
|
|
||||||
|
|
||||||
**Check an unmounted filesystem:**
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs check <device>
|
mount UUID=c9a77f3c-626f-47bd-b4e3-9a094bea287f -o subvol=snapshots /mnt/snapshots
|
||||||
```
|
```
|
||||||
|
|
||||||
- Must be run on an **unmounted** device. Running on a mounted filesystem risks corruption.
|
2. **Create a new snapshot**
|
||||||
- Use the UUID path if needed: `/dev/disk/by-uuid/<uuid>`
|
|
||||||
|
|
||||||
**Force check (use with caution):**
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
btrfs check --force <device>
|
btrfs subvolume snapshot / "/mnt/snapshots/root/2021-06-26 - Debian install"
|
||||||
```
|
```
|
||||||
|
|
||||||
- `--force`: Bypasses the mount check. Only use this if you are certain the filesystem is not mounted and understand the risks.
|
3. **Unmount after creating snapshots**
|
||||||
|
|
||||||
### Diagnosis
|
|
||||||
|
|
||||||
Filter system logs and kernel messages to diagnose BTRFS-related events.
|
|
||||||
|
|
||||||
**Search journal logs by date range:**
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
journalctl --since "<date>" --until "<date>" | grep -i btrfs
|
umount /mnt/snapshots
|
||||||
```
|
```
|
||||||
|
|
||||||
Example:
|
**Delete Snapshots**
|
||||||
|
|
||||||
```bash
|
1. **Mount subvolume containing snapshots**
|
||||||
journalctl --since "2026-01-01" --until "2026-01-02" | grep -i btrfs
|
|
||||||
```
|
|
||||||
|
|
||||||
**Search kernel ring buffer for BTRFS events:**
|
```bash
|
||||||
|
mount -o subvol=snapshots /dev/disk/by-uuid/7a22514b-594a-43a3-8fdd-4df1530b5465 /mnt/snapshots/
|
||||||
|
```
|
||||||
|
|
||||||
```bash
|
2. **List available snapshots**
|
||||||
dmesg | grep -i btrfs
|
|
||||||
```
|
```bash
|
||||||
|
btrfs subvol list /mnt/snapshots/
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Delete the desired snapshot**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
btrfs subvolume delete /mnt/snapshots/@rootfs/2024-09-15
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Unmount after deleting snapshots**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
umount /mnt/snapshots/
|
||||||
|
```
|
||||||
|
|||||||
@ -1,152 +0,0 @@
|
|||||||
# EFI Partition Repair
|
|
||||||
|
|
||||||
## Table of Contents
|
|
||||||
|
|
||||||
- [EFI Partition Repair](#efi-partition-repair)
|
|
||||||
- [Table of Contents](#table-of-contents)
|
|
||||||
- [Overview](#overview)
|
|
||||||
- [Placeholders](#placeholders)
|
|
||||||
- [Repair Procedure](#repair-procedure)
|
|
||||||
- [1. Verify What You're Working With](#1-verify-what-youre-working-with)
|
|
||||||
- [2. Mount Everything for Chroot](#2-mount-everything-for-chroot)
|
|
||||||
- [3. Chroot In](#3-chroot-in)
|
|
||||||
- [4. Reinstall GRUB](#4-reinstall-grub)
|
|
||||||
- [5. Regenerate GRUB Config](#5-regenerate-grub-config)
|
|
||||||
- [6. Verify fstab](#6-verify-fstab)
|
|
||||||
- [7. Exit and Reboot](#7-exit-and-reboot)
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
Use this procedure when a Debian machine fails to boot but the root filesystem appears intact — for example after a failed kernel upgrade, a corrupted EFI partition, or GRUB being wiped by another OS. Boot from a live environment, chroot into the installed system, and reinstall GRUB.
|
|
||||||
|
|
||||||
## Placeholders
|
|
||||||
|
|
||||||
Replace the placeholders below with the appropriate values for your setup:
|
|
||||||
|
|
||||||
- **Devices**
|
|
||||||
- Disk: `<disk>` (e.g., /dev/sda, /dev/nvme0n1)
|
|
||||||
- EFI partition: `<efi-part>` (e.g., /dev/sda1, /dev/nvme0n1p1)
|
|
||||||
- Root partition: `<root-part>` (e.g., /dev/sda2, /dev/nvme0n1p2)
|
|
||||||
|
|
||||||
- **Filesystem**
|
|
||||||
- Root subvolume name: `<root-subvol>` (e.g., @rootfs)
|
|
||||||
|
|
||||||
## Repair Procedure
|
|
||||||
|
|
||||||
### 1. Verify What You're Working With
|
|
||||||
|
|
||||||
Identify the disk layout, partition types, UUIDs, and current mount points:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
lsblk -o NAME,SIZE,FSTYPE,PARTTYPE,UUID,MOUNTPOINT
|
|
||||||
```
|
|
||||||
|
|
||||||
Confirm which partition is EFI and which is root before proceeding.
|
|
||||||
|
|
||||||
### 2. Mount Everything for Chroot
|
|
||||||
|
|
||||||
**Mount the root filesystem**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mkdir -p /mnt
|
|
||||||
mount -o subvol=<root-subvol> <root-part> /mnt
|
|
||||||
```
|
|
||||||
|
|
||||||
**Mount the EFI partition**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mkdir -p /mnt/boot/efi
|
|
||||||
mount <efi-part> /mnt/boot/efi
|
|
||||||
```
|
|
||||||
|
|
||||||
**Bind mount the pseudo-filesystems**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mkdir -p /mnt/{dev,proc,sys,run}
|
|
||||||
mount --bind /dev /mnt/dev
|
|
||||||
mount --bind /proc /mnt/proc
|
|
||||||
mount --bind /sys /mnt/sys
|
|
||||||
mount --bind /run /mnt/run
|
|
||||||
```
|
|
||||||
|
|
||||||
**Bind mount EFI vars**
|
|
||||||
|
|
||||||
Required for `grub-install` to write boot entries to the firmware:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mount --bind /sys/firmware/efi/efivars /mnt/sys/firmware/efi/efivars
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3. Chroot In
|
|
||||||
|
|
||||||
```bash
|
|
||||||
chroot /mnt /bin/bash
|
|
||||||
source /etc/profile
|
|
||||||
export PS1="(chroot) $PS1"
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4. Reinstall GRUB
|
|
||||||
|
|
||||||
**Run grub-install on the disk**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=debian --recheck <disk>
|
|
||||||
```
|
|
||||||
|
|
||||||
If you get an error about EFI vars not being writable:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mount -o remount,rw /sys/firmware/efi/efivars
|
|
||||||
```
|
|
||||||
|
|
||||||
Then re-run `grub-install`.
|
|
||||||
|
|
||||||
### 5. Regenerate GRUB Config
|
|
||||||
|
|
||||||
```bash
|
|
||||||
update-grub
|
|
||||||
```
|
|
||||||
|
|
||||||
Check the output — it should find your kernel on the btrfs volume. If it complains about not finding a kernel, you may need to also reinstall it:
|
|
||||||
|
|
||||||
**Check what kernel is installed**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
dpkg -l | grep linux-image
|
|
||||||
```
|
|
||||||
|
|
||||||
**Reinstall the kernel if missing or broken**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
apt-get install --reinstall linux-image-amd64
|
|
||||||
```
|
|
||||||
|
|
||||||
Then re-run `update-grub`.
|
|
||||||
|
|
||||||
### 6. Verify fstab
|
|
||||||
|
|
||||||
Confirm the EFI and root entries look correct before rebooting:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cat /etc/fstab
|
|
||||||
```
|
|
||||||
|
|
||||||
### 7. Exit and Reboot
|
|
||||||
|
|
||||||
**Exit the chroot**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
exit
|
|
||||||
```
|
|
||||||
|
|
||||||
**Unmount everything**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
umount -R /mnt
|
|
||||||
```
|
|
||||||
|
|
||||||
**Reboot**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
reboot
|
|
||||||
```
|
|
||||||
@ -1,334 +0,0 @@
|
|||||||
# Redundant Root with EFI Mirror
|
|
||||||
|
|
||||||
## Table of Contents
|
|
||||||
|
|
||||||
- [Redundant Root with EFI Mirror](#redundant-root-with-efi-mirror)
|
|
||||||
- [Table of Contents](#table-of-contents)
|
|
||||||
- [Overview](#overview)
|
|
||||||
- [Placeholders](#placeholders)
|
|
||||||
- [How It Works](#how-it-works)
|
|
||||||
- [Paths](#paths)
|
|
||||||
- [Setup Procedure](#setup-procedure)
|
|
||||||
- [1. Format the Disks](#1-format-the-disks)
|
|
||||||
- [2. Convert the Root Filesystem to BTRFS RAID1](#2-convert-the-root-filesystem-to-btrfs-raid1)
|
|
||||||
- [3. Configure fstab](#3-configure-fstab)
|
|
||||||
- [4. Install the Sync Script](#4-install-the-sync-script)
|
|
||||||
- [5. Configure apt to Run the Script Post-Install](#5-configure-apt-to-run-the-script-post-install)
|
|
||||||
- [6. Test](#6-test)
|
|
||||||
- [Moving to a New Disk](#moving-to-a-new-disk)
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
This setup provides a redundant Debian root drive using BTRFS RAID1 across two disks, with a synced EFI partition on both so the machine can boot from either. The script `update-efi-mirror` keeps the backup EFI in sync and re-installs GRUB on both physical disks. It is triggered automatically after every `apt` operation.
|
|
||||||
|
|
||||||
## Placeholders
|
|
||||||
|
|
||||||
Replace the placeholders below with the appropriate values for your setup:
|
|
||||||
|
|
||||||
- **Devices**
|
|
||||||
- Primary disk: `<primary-disk>` (e.g., /dev/sda, /dev/nvme0n1)
|
|
||||||
- Backup disk: `<backup-disk>` (e.g., /dev/sdb, /dev/nvme1n1)
|
|
||||||
- Primary EFI partition: `<primary-efi-part>` (e.g., /dev/sda1, /dev/nvme0n1p1)
|
|
||||||
- Backup EFI partition: `<backup-efi-part>` (e.g., /dev/sdb1, /dev/nvme1n1p1)
|
|
||||||
- Primary root partition: `<primary-root-part>` (e.g., /dev/sda2, /dev/nvme0n1p2)
|
|
||||||
- Backup root partition: `<backup-root-part>` (e.g., /dev/sdb2, /dev/nvme1n1p2)
|
|
||||||
- Primary swap partition: `<primary-swap-part>` (e.g., /dev/sda3, /dev/nvme0n1p3)
|
|
||||||
- Backup swap partition: `<backup-swap-part>` (e.g., /dev/sdb3, /dev/nvme1n1p3)
|
|
||||||
|
|
||||||
- **UUIDs**
|
|
||||||
- Primary EFI UUID: `<primary-efi-uuid>` (e.g., D167-0F46)
|
|
||||||
- Backup EFI UUID: `<backup-efi-uuid>` (e.g., 08E8-A87C)
|
|
||||||
- Root BTRFS UUID: `<root-uuid>` (e.g., 387526ec-f3bd-4fa8-a17e-e985121ada0b)
|
|
||||||
- Primary swap UUID: `<primary-swap-uuid>`
|
|
||||||
- Backup swap UUID: `<backup-swap-uuid>`
|
|
||||||
|
|
||||||
## How It Works
|
|
||||||
|
|
||||||
- The root filesystem is BTRFS RAID1 across two partitions. Either disk alone can serve reads and writes.
|
|
||||||
- Each disk has its own EFI partition. GRUB is installed on both physical disks.
|
|
||||||
- After every `apt` install or upgrade, an apt hook calls `update-efi-mirror`, which:
|
|
||||||
1. Rsyncs the live `/boot/efi` to the backup EFI partition.
|
|
||||||
2. Runs `grub-install` on both physical disks.
|
|
||||||
- Either disk can be removed and the system will still boot and run.
|
|
||||||
|
|
||||||
## Paths
|
|
||||||
|
|
||||||
- **Sync script**: `/usr/local/sbin/update-efi-mirror`
|
|
||||||
- **Apt hook**: `/etc/apt/apt.conf.d/99-update-efi-mirror`
|
|
||||||
- **fstab**: `/etc/fstab`
|
|
||||||
|
|
||||||
## Setup Procedure
|
|
||||||
|
|
||||||
### 1. Format the Disks
|
|
||||||
|
|
||||||
**Verify which disks are present before touching anything**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
lsblk
|
|
||||||
lsblk -f
|
|
||||||
```
|
|
||||||
|
|
||||||
**Open `fdisk` on the backup disk**
|
|
||||||
|
|
||||||
If the disk was previously used at a smaller capacity, the GPT backup table may be misaligned — `fdisk` will correct it on write.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
fdisk <backup-disk>
|
|
||||||
```
|
|
||||||
|
|
||||||
Create a GPT partition table if not already present:
|
|
||||||
|
|
||||||
```
|
|
||||||
g
|
|
||||||
```
|
|
||||||
|
|
||||||
**Check the primary disk partition layout and note the exact sector boundaries**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
fdisk -l <primary-disk>
|
|
||||||
```
|
|
||||||
|
|
||||||
This shows the `Start` and `End` sectors for each partition. Use these values when partitioning the backup disk so the layouts match exactly.
|
|
||||||
|
|
||||||
**Create the three partitions to match the layout of the primary disk**
|
|
||||||
|
|
||||||
| # | Size | Type |
|
|
||||||
| --- | --------- | ---------- |
|
|
||||||
| 1 | 830M | EFI System |
|
|
||||||
| 2 | ~12G | Linux fs |
|
|
||||||
| 3 | Remainder | Linux swap |
|
|
||||||
|
|
||||||
Inside `fdisk`, the sequence is:
|
|
||||||
|
|
||||||
```
|
|
||||||
n # new partition
|
|
||||||
# enter the exact End sector from the primary disk to match partition sizes
|
|
||||||
t # change type
|
|
||||||
1 # select partition 1
|
|
||||||
EFI System # set type to EFI
|
|
||||||
t # change type
|
|
||||||
3 # select partition 3
|
|
||||||
Linux swap # set type to swap
|
|
||||||
w # write and exit
|
|
||||||
```
|
|
||||||
|
|
||||||
**Partition sizes must be at least as large as the corresponding primary partition** — the BTRFS replace will fail if the target partition is smaller than the source.
|
|
||||||
|
|
||||||
**Format the new EFI partition**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
apt install dosfstools # if not installed
|
|
||||||
mkfs.fat -F32 <backup-efi-part>
|
|
||||||
```
|
|
||||||
|
|
||||||
**Format the new swap partition**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mkswap <backup-swap-part>
|
|
||||||
```
|
|
||||||
|
|
||||||
**Check UUIDs of the new partitions**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
lsblk -f <backup-disk>
|
|
||||||
```
|
|
||||||
|
|
||||||
### 2. Convert the Root Filesystem to BTRFS RAID1
|
|
||||||
|
|
||||||
**Add the backup partition to the BTRFS volume**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
btrfs device add <backup-root-part> /
|
|
||||||
```
|
|
||||||
|
|
||||||
**Convert to RAID1**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
btrfs balance start -mconvert=raid1 -dconvert=raid1 /
|
|
||||||
```
|
|
||||||
|
|
||||||
Monitor the balance:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
btrfs balance status /
|
|
||||||
```
|
|
||||||
|
|
||||||
**Verify the final layout**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
btrfs filesystem show /
|
|
||||||
btrfs fi usage /
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3. Configure fstab
|
|
||||||
|
|
||||||
Edit `/etc/fstab` to mount the primary EFI partition normally and document the backup EFI UUID in a commented-out entry. The script reads the backup UUID from this comment.
|
|
||||||
|
|
||||||
**Edit fstab**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nano /etc/fstab
|
|
||||||
```
|
|
||||||
|
|
||||||
The fstab should contain:
|
|
||||||
|
|
||||||
```
|
|
||||||
# Primary EFI — mounted at boot
|
|
||||||
UUID=<primary-efi-uuid> /boot/efi vfat umask=0077 0 1
|
|
||||||
|
|
||||||
# Backup EFI — not auto-mounted, used by update-efi-mirror
|
|
||||||
#UUID=<backup-efi-uuid> /boot/efi vfat umask=0077 0 1
|
|
||||||
|
|
||||||
# Root — BTRFS RAID1 (both partitions share the same UUID)
|
|
||||||
UUID=<root-uuid> / btrfs defaults,noatime 0 1
|
|
||||||
|
|
||||||
# Swap — both disks
|
|
||||||
UUID=<primary-swap-uuid> none swap sw 0 0
|
|
||||||
UUID=<backup-swap-uuid> none swap sw 0 0
|
|
||||||
```
|
|
||||||
|
|
||||||
The backup EFI line must start with `#UUID=` — this is the format the script greps for:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
grep "^#UUID=" /etc/fstab | grep "/boot/efi"
|
|
||||||
```
|
|
||||||
|
|
||||||
**Reload systemd after editing fstab**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
systemctl daemon-reload
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4. Install the Sync Script
|
|
||||||
|
|
||||||
**Create the script**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nano /usr/local/sbin/update-efi-mirror
|
|
||||||
```
|
|
||||||
|
|
||||||
Paste the contents of `update-efi-mirror.sh`.
|
|
||||||
|
|
||||||
**Set ownership and permissions**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
chown root:root /usr/local/sbin/update-efi-mirror
|
|
||||||
chmod 755 /usr/local/sbin/update-efi-mirror
|
|
||||||
```
|
|
||||||
|
|
||||||
The script will:
|
|
||||||
|
|
||||||
1. Detect the currently mounted EFI partition and its disk.
|
|
||||||
2. Read the backup EFI UUID from the `#UUID=` comment in `/etc/fstab`.
|
|
||||||
3. Mount the backup EFI to a temp directory, rsync all files from the live EFI, then unmount.
|
|
||||||
4. Run `grub-install --recheck` on both physical disks.
|
|
||||||
|
|
||||||
### 5. Configure apt to Run the Script Post-Install
|
|
||||||
|
|
||||||
**Create the apt hook**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nano /etc/apt/apt.conf.d/99-update-efi-mirror
|
|
||||||
```
|
|
||||||
|
|
||||||
Add:
|
|
||||||
|
|
||||||
```
|
|
||||||
DPkg::Post-Invoke {"if [ -x /usr/local/sbin/update-efi-mirror ]; then /usr/local/sbin/update-efi-mirror; fi";};
|
|
||||||
```
|
|
||||||
|
|
||||||
This runs the sync script after every `apt` operation that invokes `dpkg` — including kernel upgrades, which update `/boot/efi` with new initramfs and GRUB config.
|
|
||||||
|
|
||||||
### 6. Test
|
|
||||||
|
|
||||||
**Run the script manually**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
/usr/local/sbin/update-efi-mirror
|
|
||||||
```
|
|
||||||
|
|
||||||
Expected output:
|
|
||||||
|
|
||||||
```
|
|
||||||
Syncing EFI files: <primary-efi-uuid> -> <backup-efi-uuid>
|
|
||||||
EFI Files Sync Complete.
|
|
||||||
Updating GRUB bootloader on both physical disks...
|
|
||||||
Redundancy Complete: Files synced and GRUB updated on <primary-disk> and <backup-disk>.
|
|
||||||
```
|
|
||||||
|
|
||||||
**Test the apt hook**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
apt update
|
|
||||||
apt upgrade -y
|
|
||||||
```
|
|
||||||
|
|
||||||
The sync output should appear at the end of the upgrade, after GRUB regenerates its config.
|
|
||||||
|
|
||||||
**Verify the backup EFI contents match**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
TMPMNT=$(mktemp -d)
|
|
||||||
mount -U <backup-efi-uuid> "$TMPMNT"
|
|
||||||
diff -r /boot/efi/ "$TMPMNT/"
|
|
||||||
umount "$TMPMNT"
|
|
||||||
rmdir "$TMPMNT"
|
|
||||||
```
|
|
||||||
|
|
||||||
**Verify GRUB is installed on both disks**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
fdisk -l <primary-disk> | grep -i efi
|
|
||||||
fdisk -l <backup-disk> | grep -i efi
|
|
||||||
```
|
|
||||||
|
|
||||||
**`Optional` Boot from the backup disk**
|
|
||||||
|
|
||||||
Swap the boot order in BIOS/UEFI to confirm the machine boots cleanly from `<backup-disk>`.
|
|
||||||
|
|
||||||
## Moving to a New Disk
|
|
||||||
|
|
||||||
Use this procedure when replacing one of the two disks in the BTRFS RAID1 — for example when upgrading to a larger drive.
|
|
||||||
|
|
||||||
**1. Partition the new disk** following [Step 1](#1-format-the-disks).
|
|
||||||
|
|
||||||
**2. Find the device ID of the partition to be replaced:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
btrfs filesystem show /
|
|
||||||
```
|
|
||||||
|
|
||||||
**3. Shrink the filesystem on the disk being replaced to fit the new partition:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
btrfs filesystem resize <device-id>:<size> /
|
|
||||||
```
|
|
||||||
|
|
||||||
**4. Replace the old partition with the new one:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
btrfs replace start <device-id> <new-root-part> /
|
|
||||||
btrfs replace status /
|
|
||||||
```
|
|
||||||
|
|
||||||
**5. Grow the new partition to max:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
btrfs filesystem resize <device-id>:max /
|
|
||||||
```
|
|
||||||
|
|
||||||
**6. Update fstab** if the backup EFI UUID has changed (new disk = new EFI UUID):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
lsblk -f <new-disk>
|
|
||||||
nano /etc/fstab
|
|
||||||
systemctl daemon-reload
|
|
||||||
```
|
|
||||||
|
|
||||||
**7. Run the sync script** to install GRUB on the new disk and populate its EFI partition:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
/usr/local/sbin/update-efi-mirror
|
|
||||||
```
|
|
||||||
|
|
||||||
**8. Verify** following [Step 6](#6-test).
|
|
||||||
398
notes/linux.md
398
notes/linux.md
@ -7,6 +7,7 @@
|
|||||||
- [System Information](#system-information)
|
- [System Information](#system-information)
|
||||||
- [Hardware Information](#hardware-information)
|
- [Hardware Information](#hardware-information)
|
||||||
- [Software Information](#software-information)
|
- [Software Information](#software-information)
|
||||||
|
- [Commands to Get Information About Linux Version, Kernel Version, and Release](#commands-to-get-information-about-linux-version-kernel-version-and-release)
|
||||||
- [User Management](#user-management)
|
- [User Management](#user-management)
|
||||||
- [User Information](#user-information)
|
- [User Information](#user-information)
|
||||||
- [Super User Management](#super-user-management)
|
- [Super User Management](#super-user-management)
|
||||||
@ -14,8 +15,8 @@
|
|||||||
- [System Management](#system-management)
|
- [System Management](#system-management)
|
||||||
- [Change password of a tar/openssl archive](#change-password-of-a-taropenssl-archive)
|
- [Change password of a tar/openssl archive](#change-password-of-a-taropenssl-archive)
|
||||||
- [Verify two possibly identical folders recursively](#verify-two-possibly-identical-folders-recursively)
|
- [Verify two possibly identical folders recursively](#verify-two-possibly-identical-folders-recursively)
|
||||||
- [NFS](#nfs)
|
- [USB Devices](#usb-devices)
|
||||||
- [Network Diagnostics](#network-diagnostics)
|
- [Test USB Key](#test-usb-key)
|
||||||
- [Diagnosis](#diagnosis)
|
- [Diagnosis](#diagnosis)
|
||||||
- [Debian Upgrade Issues](#debian-upgrade-issues)
|
- [Debian Upgrade Issues](#debian-upgrade-issues)
|
||||||
- [Wayland Issues](#wayland-issues)
|
- [Wayland Issues](#wayland-issues)
|
||||||
@ -32,46 +33,25 @@ To gather detailed information about your hardware, use the following commands:
|
|||||||
- **`hwinfo`**: Offers detailed information about hardware components and can be more verbose than `lshw`.
|
- **`hwinfo`**: Offers detailed information about hardware components and can be more verbose than `lshw`.
|
||||||
- **`lsscsi`**: Lists SCSI devices, including disks and other SCSI-attached hardware.
|
- **`lsscsi`**: Lists SCSI devices, including disks and other SCSI-attached hardware.
|
||||||
- **`lsusb`**: Shows information about USB devices connected to your system.
|
- **`lsusb`**: Shows information about USB devices connected to your system.
|
||||||
|
- **`lsblk`**: Lists block devices such as hard drives and their partitions.
|
||||||
|
- **`df -H`**: Displays disk space usage in a human-readable format.
|
||||||
|
- **`fdisk -l`**: Lists all partitions on the system.
|
||||||
- **`dmidecode`**: Retrieves hardware information from the BIOS. Use:
|
- **`dmidecode`**: Retrieves hardware information from the BIOS. Use:
|
||||||
- `dmidecode -t processor` for CPU details
|
- `dmidecode -t processor` for CPU details
|
||||||
- `dmidecode -t memory` for RAM details
|
- `dmidecode -t memory` for RAM details
|
||||||
- `dmidecode -t bios` for BIOS information
|
- `dmidecode -t bios` for BIOS information
|
||||||
|
|
||||||
**CPU information**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
lscpu
|
|
||||||
cat /proc/cpuinfo
|
|
||||||
grep -c 'model name' /proc/cpuinfo
|
|
||||||
```
|
|
||||||
|
|
||||||
- `lscpu`: Structured summary of CPU architecture, cores, threads, and NUMA topology.
|
|
||||||
- `cat /proc/cpuinfo`: Raw per-core details including model name, flags, and frequencies.
|
|
||||||
- `grep -c 'model name'`: Quick count of logical CPU cores.
|
|
||||||
|
|
||||||
**GPU information**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
lspci | grep -i vga
|
|
||||||
```
|
|
||||||
|
|
||||||
**CPU frequency scaling driver**
|
|
||||||
|
|
||||||
Check which driver is managing CPU frequency scaling (e.g., `intel_pstate`, `acpi-cpufreq`):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver
|
|
||||||
```
|
|
||||||
|
|
||||||
### Software Information
|
### Software Information
|
||||||
|
|
||||||
**Finding information on the Linux distribution**
|
**Finding information on the Linux distribution**
|
||||||
|
|
||||||
|
# Commands to Get Information About Linux Version, Kernel Version, and Release
|
||||||
|
|
||||||
- **`lsb_release -a`**: Displays detailed information about the Linux distribution, including the distributor ID, description, release number, and codename.
|
- **`lsb_release -a`**: Displays detailed information about the Linux distribution, including the distributor ID, description, release number, and codename.
|
||||||
- **`cat /etc/debian_version`**: Displays the version of the Debian distribution if you're running a Debian-based system (like Ubuntu).
|
- **`cat /etc/debian_version`**: Displays the version of the Debian distribution if you're running a Debian-based system (like Ubuntu).
|
||||||
- **`cat /etc/os-release`**: Displays information about the operating system, such as the name, version, and ID of the distribution.
|
- **`cat /etc/os-release`**: Displays information about the operating system, such as the name, version, and ID of the distribution.
|
||||||
- **`cat /etc/*release`**: Searches for any files in the `/etc/` directory that contain the word `release` and displays their contents. This typically includes more detailed distribution information.
|
- **`cat /etc/*release`**: Searches for any files in the `/etc/` directory that contain the word `release` and displays their contents. This typically includes more detailed distribution information.
|
||||||
- **`cat /etc/*version`**: Similar to `cat /etc/*release`, but looks for files containing the word `version`. It can provide additional version-related details.
|
- **`cat /etc/*version`**: Similar to `cat /etc/*release`, but it looks for files containing the word `version`. It can provide additional version-related details.
|
||||||
- **`hostnamectl`**: Displays system information related to the hostname and other metadata about the system. This may include the operating system, kernel version, and architecture.
|
- **`hostnamectl`**: Displays system information related to the hostname and other metadata about the system. This may include the operating system, kernel version, and architecture.
|
||||||
|
|
||||||
**Finding Path to Binary**
|
**Finding Path to Binary**
|
||||||
@ -79,17 +59,17 @@ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver
|
|||||||
To find the location of an executable binary, use:
|
To find the location of an executable binary, use:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
type <binary-name>
|
type composer
|
||||||
```
|
```
|
||||||
|
|
||||||
This command will show the path to a binary executable, ex `composer`, if it's available in your `PATH`.
|
This command will show the path to the `composer` executable if it's available in your `PATH`.
|
||||||
|
|
||||||
**Number of Words in a File**
|
**Number of Words in a File**
|
||||||
|
|
||||||
To count the number of words in a file, use:
|
To count the number of words in a file, use:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
wc <filepath>
|
wc filepath
|
||||||
```
|
```
|
||||||
|
|
||||||
This command will show the number of words along with other details like lines and characters.
|
This command will show the number of words along with other details like lines and characters.
|
||||||
@ -99,9 +79,11 @@ This command will show the number of words along with other details like lines a
|
|||||||
To count the number of lines in a file, use:
|
To count the number of lines in a file, use:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
wc -l <filepath>
|
wc -l filepath
|
||||||
```
|
```
|
||||||
|
|
||||||
|
This command will display the number of lines in the specified file.
|
||||||
|
|
||||||
## User Management
|
## User Management
|
||||||
|
|
||||||
### User Information
|
### User Information
|
||||||
@ -111,23 +93,25 @@ wc -l <filepath>
|
|||||||
This variation of the adduser command uses the --gecos option to pre-fill the user's information (Full name, Room number, Work Phone, Home Phone, and Email) non-interactively, allowing you to automate user creation with predefined details.
|
This variation of the adduser command uses the --gecos option to pre-fill the user's information (Full name, Room number, Work Phone, Home Phone, and Email) non-interactively, allowing you to automate user creation with predefined details.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
adduser --gecos "<full-name>,,,<email>" <username>
|
adduser --gecos "Fabrice Quenneville,,,fabrice@fabq.ca" fabrice
|
||||||
```
|
```
|
||||||
|
|
||||||
This variation creates a system user with a Bash shell, no password login (--disabled-password), a specified home directory (`/home/<username>`), and adds the user to a new group, while using the --gecos option to set the full name as `<service-description>`.
|
This variation of the adduser command creates a system user named "aptly" with a Bash shell, no password login (--disabled-password), a specified home directory (/home/aptly), and adds the user to a new group, while using the --gecos option to set the full name as "Aptly repository".
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
adduser --system --shell /bin/bash --gecos '<service-description>' --group --disabled-password --home /home/<username> <username>
|
adduser --system --shell /bin/bash --gecos 'Aptly repository' --group --disabled-password --home /home/aptly aptly
|
||||||
```
|
```
|
||||||
|
|
||||||
**List Users**
|
**List Users**
|
||||||
|
|
||||||
To list all users from the `/etc/passwd` file in alphabetical order, use:
|
To list all users from the `/etc/passwd` file, use:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
awk -F':' '{ print $1}' /etc/passwd | sort
|
awk -F':' '{ print $1}' /etc/passwd | sort
|
||||||
```
|
```
|
||||||
|
|
||||||
|
This command extracts the usernames from the `/etc/passwd` file and sorts them in alphabetical order.
|
||||||
|
|
||||||
### Super User Management
|
### Super User Management
|
||||||
|
|
||||||
**Disable Root Login**
|
**Disable Root Login**
|
||||||
@ -140,7 +124,7 @@ To disable root login via SSH, perform the following steps:
|
|||||||
nano /etc/ssh/sshd_config
|
nano /etc/ssh/sshd_config
|
||||||
```
|
```
|
||||||
|
|
||||||
Comment out the line containing `PermitRootLogin`.
|
Comment out the line containing `PermitRootLogin` by adding a `#` at the beginning of the line.
|
||||||
|
|
||||||
2. **Change Shell for Root User:**
|
2. **Change Shell for Root User:**
|
||||||
|
|
||||||
@ -148,36 +132,34 @@ To disable root login via SSH, perform the following steps:
|
|||||||
nano /etc/passwd
|
nano /etc/passwd
|
||||||
```
|
```
|
||||||
|
|
||||||
Find the line starting with `root` and change `/bin/bash` to `/sbin/nologin`.
|
Find the line starting with `root` and change `/bin/bash` to `/sbin/nologin` to disable login for the root user.
|
||||||
|
|
||||||
|
Save and close the file. Restart the SSH service for changes to take effect:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
systemctl restart ssh
|
systemctl restart ssh
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Sudo Management**
|
||||||
|
|
||||||
**Add User to Sudo Group**
|
**Add User to Sudo Group**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
adduser <username> sudo
|
adduser fabrice sudo
|
||||||
```
|
```
|
||||||
|
|
||||||
**Update Sudoers File to Remove Password Requirement**
|
**Update Sudoers File to Remove Password Requirement**
|
||||||
|
|
||||||
Edit the sudoers file with the default editor:
|
Edit the sudoers file:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
visudo
|
visudo
|
||||||
```
|
```
|
||||||
|
|
||||||
Edit the sudoers file with `nano`:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
EDITOR=nano visudo
|
|
||||||
```
|
|
||||||
|
|
||||||
Add the following line to allow the user to execute commands without a password:
|
Add the following line to allow the user to execute commands without a password:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
<username> ALL=(ALL) NOPASSWD:ALL
|
fabrice ALL=(ALL) NOPASSWD:ALL
|
||||||
```
|
```
|
||||||
|
|
||||||
### Switch User
|
### Switch User
|
||||||
@ -185,74 +167,82 @@ Add the following line to allow the user to execute commands without a password:
|
|||||||
**Switch to Another User as Sudoer**
|
**Switch to Another User as Sudoer**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
sudo -i -u <username>
|
sudo -i -u postgres
|
||||||
```
|
```
|
||||||
|
|
||||||
|
This command switches to the `postgres` user with sudo privileges.
|
||||||
|
|
||||||
**Switch to Another User as Root**
|
**Switch to Another User as Root**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
su - <username>
|
su - postgres
|
||||||
```
|
```
|
||||||
|
|
||||||
|
This command switches to the `postgres` user with root privileges.
|
||||||
|
|
||||||
**Run command as specific user**
|
**Run command as specific user**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
sudo -u <username> <command>
|
sudo -u www-data somecommand and arguments
|
||||||
```
|
```
|
||||||
|
|
||||||
**Change shell of a user**
|
**Change shell of a user**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
chsh -s /bin/bash <username>
|
chsh -s /bin/bash www-data
|
||||||
chsh -s /usr/sbin/nologin <username>
|
chsh -s /usr/sbin/nologin www-data
|
||||||
```
|
```
|
||||||
|
|
||||||
**Change user with specific shell**
|
**Change user with specific shell**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
sudo -u <username> bash
|
sudo -u www-data bash
|
||||||
```
|
```
|
||||||
|
|
||||||
## System Management
|
## System Management
|
||||||
|
|
||||||
**Ensure hostname or add alias**
|
**Ensure hostname or add alias**
|
||||||
|
|
||||||
|
Set or update the hostname for your server.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
nano /etc/hosts
|
nano /etc/hosts
|
||||||
# Add the hostname alias:
|
# Add the hostname alias:
|
||||||
# 127.0.1.1 <hostname-intranet>
|
# 127.0.1.1 local.servername.domain.com
|
||||||
|
|
||||||
nano /etc/hostname
|
nano /etc/hostname
|
||||||
# Set the main hostname:
|
# Set the main hostname:
|
||||||
# 127.0.1.1 <hostname-intranet> <hostname-short>
|
# 127.0.1.1 servername.domain.com servername
|
||||||
|
|
||||||
hostnamectl set-hostname <hostname-intranet>
|
hostnamectl set-hostname servername.domain.com
|
||||||
```
|
```
|
||||||
|
|
||||||
**Tar backup for a large number of small files**
|
**Tar backup for a large number of small files**
|
||||||
|
|
||||||
|
These commands create backups using `tar` and transfer them securely over SSH.
|
||||||
|
|
||||||
Create a tar archive and transfer it to a remote server:
|
Create a tar archive and transfer it to a remote server:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
tar -c /path/to/dir | ssh <username>@<hostname-intranet> 'tar -xvf - -C /absolute/path/to/remotedir'
|
tar -c /path/to/dir | ssh fabrice@servername.domain.com 'tar -xvf - -C /absolute/path/to/remotedir'
|
||||||
```
|
```
|
||||||
|
|
||||||
Compress and transfer a folder, then store it as a `.tar.gz` file:
|
Compress and transfer a folder, then store it as a .tar.gz file:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
tar zcvf - /folder | ssh <username>@<hostname-intranet> "cat > /backup/folder.tar.gz"
|
tar zcvf - /folder | ssh fabrice@servername.domain.com "cat > /backup/folder.tar.gz"
|
||||||
```
|
```
|
||||||
|
|
||||||
Transfer a compressed `.tar.gz` file and extract it on the remote server:
|
Transfer a compressed .tar.gz file and extract it on the remote server:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cat folder.tar.gz | ssh <username>@<hostname-intranet> "tar zxvf -"
|
cat folder.tar.gz | ssh fabrice@servername.domain.com "tar zxvf -"
|
||||||
```
|
```
|
||||||
|
|
||||||
Alternative: change directory on the remote server before extracting:
|
Alternative method: change directory on the remote server before extracting:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cat folder.tar.gz | ssh <username>@<hostname-intranet> "cd /path/to/dest/; tar zxvf -"
|
cat folder.tar.gz | ssh fabrice@servername.domain.com "cd /path/to/dest/; tar zxvf -"
|
||||||
```
|
```
|
||||||
|
|
||||||
**List time zones**
|
**List time zones**
|
||||||
@ -276,130 +266,83 @@ Alternatively, manually set the time zone by linking the correct file:
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
mv /etc/localtime /etc/localtime-old
|
mv /etc/localtime /etc/localtime-old
|
||||||
ln -s /usr/share/zoneinfo/<timezone> /etc/localtime
|
ln -s /usr/share/zoneinfo/America/Toronto /etc/localtime
|
||||||
```
|
```
|
||||||
|
|
||||||
**Find a specific service**
|
**Find a specific service**
|
||||||
|
|
||||||
```bash
|
Search for a specific service running on your system.
|
||||||
systemctl list-units --type=service | grep <service-name>
|
|
||||||
```
|
|
||||||
|
|
||||||
**Bind mount a directory**
|
|
||||||
|
|
||||||
Make a directory available at another path, useful during chroot recovery or container setup:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
mount --bind /dev /mnt/<newroot>/dev
|
systemctl list-units --type=service | grep php
|
||||||
```
|
```
|
||||||
|
|
||||||
**Chroot into another system**
|
|
||||||
|
|
||||||
Enter a mounted system's root as if it were the running OS. Useful for recovery, initramfs rebuilds, or bootloader fixes:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
chroot /mnt/<newroot>
|
|
||||||
```
|
|
||||||
|
|
||||||
Typically preceded by binding the required pseudo-filesystems:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mount --bind /dev /mnt/<newroot>/dev
|
|
||||||
mount --bind /proc /mnt/<newroot>/proc
|
|
||||||
mount --bind /sys /mnt/<newroot>/sys
|
|
||||||
chroot /mnt/<newroot>
|
|
||||||
```
|
|
||||||
|
|
||||||
**Rebuild initramfs**
|
|
||||||
|
|
||||||
After kernel or driver changes, rebuild the initramfs and refresh the GRUB configuration:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
update-initramfs -u
|
|
||||||
update-initramfs -u -k all
|
|
||||||
```
|
|
||||||
|
|
||||||
- `update-initramfs -u`: Rebuilds the initramfs for the currently running kernel.
|
|
||||||
- `-k all`: Rebuilds for all installed kernels.
|
|
||||||
|
|
||||||
**Rebuild initramfs for a specific kernel version:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
update-initramfs -c -k $(uname -r)
|
|
||||||
```
|
|
||||||
|
|
||||||
- `-c`: Create a new initramfs (instead of updating).
|
|
||||||
- `-k $(uname -r)`: Targets the currently running kernel version.
|
|
||||||
|
|
||||||
**Update GRUB:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
update-grub
|
|
||||||
```
|
|
||||||
|
|
||||||
- Scans for kernels and regenerates `/boot/grub/grub.cfg`.
|
|
||||||
|
|
||||||
### Change password of a tar/openssl archive
|
### Change password of a tar/openssl archive
|
||||||
|
|
||||||
**Decrypt the archive**
|
**Decrypt the archive**
|
||||||
|
|
||||||
To decrypt an `openssl`-encrypted archive using a password stored in a file:
|
To decrypt an `openssl`-encrypted archive using a password stored in a file:
|
||||||
|
|
||||||
1. **Store your password in a temporary file:**
|
1. **Store your password in this file.**
|
||||||
|
```bash
|
||||||
|
nano $HOME/xyz001.txt
|
||||||
|
```
|
||||||
|
2. **Decrypt the archive**
|
||||||
|
Decrypt the archive using the password stored in xyz001.txt.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
nano $HOME/<filename>
|
openssl aes-256-cbc -d -pbkdf2 -in servername-backup.tar.gz -out servername-backup.tar -pass file:$HOME/xyz001.txt
|
||||||
```
|
```
|
||||||
|
|
||||||
2. **Decrypt the archive:**
|
3. **Re-encrypt the archive with a new password**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
openssl aes-256-cbc -d -pbkdf2 -in <archive>.tar.gz -out <archive>.tar -pass file:$HOME/<filename>
|
nano $HOME/xyz001.txt
|
||||||
```
|
openssl aes-256-cbc -e -pbkdf2 -in servername-backup.tar -out servername-backup-new.tar.gz -pass file:$HOME/xyz001.txt
|
||||||
|
rm $HOME/xyz001.txt
|
||||||
3. **Re-encrypt the archive with a new password:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nano $HOME/<filename>
|
|
||||||
openssl aes-256-cbc -e -pbkdf2 -in <archive>.tar -out <archive>-new.tar.gz -pass file:$HOME/<filename>
|
|
||||||
rm $HOME/<filename>
|
|
||||||
```
|
```
|
||||||
|
|
||||||
**Decode / Extract**
|
**Decode / Extract**
|
||||||
|
|
||||||
Decrypt and extract the contents directly into a directory:
|
To decrypt and extract the contents of an encrypted archive directly into a directory:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
nano $HOME/<filename>
|
nano $HOME/xyz001.txt
|
||||||
openssl aes-256-cbc -d -pbkdf2 -in <archive>.tar.gz -pass file:<filename> | tar xz -C .
|
openssl aes-256-cbc -d -pbkdf2 -in servername-backup.tar.gz -pass file:xyz001.txt | tar xz -C .
|
||||||
rm $HOME/<filename>
|
rm $HOME/xyz001.txt
|
||||||
```
|
```
|
||||||
|
|
||||||
### Verify two possibly identical folders recursively
|
### Verify two possibly identical folders recursively
|
||||||
|
|
||||||
**With `diff`**
|
**With `diff`**
|
||||||
|
|
||||||
|
Check for differences between two directories, comparing all files recursively:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
diff -r <dir1>/ <dir2>/
|
diff -r servername-files/data/servername-repositories/ servername-repositories/
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Outputs any differences found between the two directories.
|
||||||
|
|
||||||
**With `rsync`**
|
**With `rsync`**
|
||||||
|
|
||||||
Dry run — shows differences without copying any data:
|
Use `rsync` to show differences without copying any data:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
rsync -avn <dir1>/ <dir2>/
|
rsync -avn servername-files/data/servername-repositories/ servername-repositories/
|
||||||
```
|
```
|
||||||
|
|
||||||
- `-n`: dry run, no changes made.
|
- The `-n` flag means this is a dry run, which won’t make any changes.
|
||||||
|
|
||||||
**With `cmp`**
|
**With `cmp`**
|
||||||
|
|
||||||
|
This script compares files in two directories and identifies any differences between matching file names.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
|
|
||||||
dir1="<dir1>/"
|
dir1="servername-files/data/servername-repositories/"
|
||||||
dir2="<dir2>/"
|
dir2="servername-repositories/"
|
||||||
|
|
||||||
# Check if both directories exist before proceeding.
|
# Check if both directories exist before proceeding.
|
||||||
if [ ! -d "$dir1" ] || [ ! -d "$dir2" ]; then
|
if [ ! -d "$dir1" ] || [ ! -d "$dir2" ]; then
|
||||||
@ -418,71 +361,140 @@ for file1 in $(find "$dir1" -type f); do
|
|||||||
done
|
done
|
||||||
```
|
```
|
||||||
|
|
||||||
## NFS
|
## USB Devices
|
||||||
|
|
||||||
**Show NFS exports from a server:**
|
### Test USB Key
|
||||||
|
|
||||||
|
**Device Information**
|
||||||
|
|
||||||
|
Check if the system recognizes the device and show the latest system messages related to USB devices being connected:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
showmount -e <hostname>
|
lsusb
|
||||||
showmount -e localhost
|
dmesg | tail -n 20
|
||||||
```
|
```
|
||||||
|
|
||||||
**List active exports and their options on the server:**
|
**Find Mount Points and Device Information**
|
||||||
|
|
||||||
|
Identify mount points, partitions, and other relevant details of mounted devices:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
exportfs -v
|
lsblk -f
|
||||||
|
df -h | grep /dev/sdc
|
||||||
|
findmnt /dev/sdc1
|
||||||
|
mount | grep /dev/sd
|
||||||
```
|
```
|
||||||
|
|
||||||
## Network Diagnostics
|
**Print Detailed Information About the USB Key**
|
||||||
|
|
||||||
**List network interfaces**
|
View detailed partition and disk information:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ip link show
|
fdisk -l /dev/sdc
|
||||||
```
|
```
|
||||||
|
|
||||||
**List network bridges and their attached interfaces**
|
**Test the File System**
|
||||||
|
|
||||||
|
Check and repair the filesystem on the USB key:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
brctl show
|
fsck /dev/sdc1
|
||||||
```
|
```
|
||||||
|
|
||||||
**Measure HTTP response timing:**
|
**Test Data Integrity**
|
||||||
|
|
||||||
Breaks down the full request lifecycle — useful for diagnosing DNS, TLS, or TTFB issues:
|
Perform read/write tests to ensure the integrity of the USB key:
|
||||||
|
|
||||||
|
1. **Unmount the USB Key** (if mounted):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
umount /media/fabrice/BD48-F8BB
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Write Test**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dd if=/dev/zero of=/dev/sdc bs=4M count=256 status=progress
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Read Test**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dd if=/dev/sdc of=/dev/null bs=4M count=256 status=progress
|
||||||
|
```
|
||||||
|
|
||||||
|
**Check for Bad Blocks**
|
||||||
|
|
||||||
|
Identify any bad sectors on the USB key:
|
||||||
|
|
||||||
|
- **Read-only test**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
badblocks -v /dev/sdc
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Non-destructive read-write test**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
badblocks -nsv /dev/sdc
|
||||||
|
```
|
||||||
|
|
||||||
|
- The `-n` option performs a non-destructive read-write test.
|
||||||
|
- The `-s` option shows progress.
|
||||||
|
- The `-v` option is for verbose output.
|
||||||
|
|
||||||
|
**Perform a SMART Test**
|
||||||
|
|
||||||
|
Run SMART diagnostics to test the health of the USB key:
|
||||||
|
|
||||||
|
1. **Start a short SMART test**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
smartctl -t short /dev/sdc
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **View test results**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
smartctl -a /dev/sdc
|
||||||
|
```
|
||||||
|
|
||||||
|
**Benchmark the Speed**
|
||||||
|
|
||||||
|
Measure the read speed of the USB key:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
curl -o /dev/null -s -w \
|
hdparm -t /dev/sdc
|
||||||
'Lookup: %{time_namelookup}s\nConnect: %{time_connect}s\nAppConnect: %{time_appconnect}s\nTTFB: %{time_starttransfer}s\nTotal: %{time_total}s\n' \
|
|
||||||
https://<hostname>
|
|
||||||
```
|
```
|
||||||
|
|
||||||
- `time_namelookup`: DNS resolution time.
|
**Unmount and Safely Remove**
|
||||||
- `time_connect`: TCP connection time.
|
|
||||||
- `time_appconnect`: TLS handshake time.
|
|
||||||
- `time_starttransfer`: Time to first byte (TTFB).
|
|
||||||
- `-o /dev/null`: Discards the response body.
|
|
||||||
|
|
||||||
**High-frequency ping:**
|
Unmount the USB key and safely remove it from the system:
|
||||||
|
|
||||||
Flood-style ping to stress-test latency or detect intermittent packet loss:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ping -i 0.002 <host>
|
umount /mnt/usb
|
||||||
|
eject /dev/sdc
|
||||||
```
|
```
|
||||||
|
|
||||||
- `-i 0.002`: Send a packet every 2ms. Requires root.
|
**Switching two USB keys**
|
||||||
|
|
||||||
**Jumbo frame ping:**
|
The following commands copy data between two USB drives, format one of them, and restore the data.
|
||||||
|
|
||||||
Test whether the network path supports large MTU frames (useful for diagnosing MTU mismatches):
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ping -s 1472 -i 0.01 <host>
|
cp -r /media/fabrice/465A-759B "/tmp/Michael Allison"
|
||||||
```
|
umount /dev/sdc1
|
||||||
|
mkfs.vfat /dev/sdc1
|
||||||
|
|
||||||
- `-s 1472`: Payload size of 1472 bytes (1472 + 28-byte IP/ICMP header = 1500-byte MTU).
|
umount /dev/sdc1
|
||||||
- Increase `-s` to test jumbo frames (e.g., `-s 8972` for 9000-byte MTU).
|
dd if=/dev/sdc of=/tmp/usb_image.img bs=4M status=progress
|
||||||
|
mkfs.vfat /dev/sdc1
|
||||||
|
|
||||||
|
cp -r "/tmp/Michael Allison" /media/fabrice/D67D-ADF8
|
||||||
|
umount /dev/sdc1
|
||||||
|
|
||||||
|
dd if=/tmp/usb_image.img of=/dev/sdc bs=4M status=progress
|
||||||
|
sync
|
||||||
|
```
|
||||||
|
|
||||||
## Diagnosis
|
## Diagnosis
|
||||||
|
|
||||||
@ -490,6 +502,8 @@ ping -s 1472 -i 0.01 <host>
|
|||||||
|
|
||||||
**Apt Logs**
|
**Apt Logs**
|
||||||
|
|
||||||
|
View the APT logs to check for package installation and updates history:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
less /var/log/apt/history.log
|
less /var/log/apt/history.log
|
||||||
```
|
```
|
||||||
@ -515,45 +529,23 @@ journalctl -b | grep -i "drm\|gpu\|display\|wayland\|monitor"
|
|||||||
journalctl -b | grep -i "gnome-shell"
|
journalctl -b | grep -i "gnome-shell"
|
||||||
```
|
```
|
||||||
|
|
||||||
**Journal Filtering by Date and Keyword**
|
|
||||||
|
|
||||||
Search logs within a specific time window:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
journalctl --since "<date>" --until "<date>" | grep -i <keyword>
|
|
||||||
```
|
|
||||||
|
|
||||||
Example:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
journalctl --since "2026-01-01" --until "2026-01-02" | grep -i btrfs
|
|
||||||
```
|
|
||||||
|
|
||||||
**Kernel microcode events:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
journalctl -k | grep -i "microcode"
|
|
||||||
```
|
|
||||||
|
|
||||||
- `-k`: Show only kernel messages (equivalent to `dmesg` output via the journal).
|
|
||||||
|
|
||||||
## Fonts
|
## Fonts
|
||||||
|
|
||||||
**Download and Install Fonts**
|
**Download and Install Fonts**
|
||||||
|
|
||||||
1. **Download the Font Archive:**
|
1. **Download the Font Archive**:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
wget https://<font-archive-url>
|
wget https://path/to/font/archive.tar.gz
|
||||||
```
|
```
|
||||||
|
|
||||||
2. **Extract the Font Files:**
|
2. **Extract the Font Files**:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
tar -xzvf <font-archive>.tar.gz
|
tar -xzvf font-archive.tar.gz
|
||||||
```
|
```
|
||||||
|
|
||||||
3. **Copy the Font Files:**
|
3. **Copy the Font Files to the Local Fonts Directory**:
|
||||||
|
|
||||||
**Local font directory**
|
**Local font directory**
|
||||||
|
|
||||||
@ -575,12 +567,14 @@ journalctl -k | grep -i "microcode"
|
|||||||
|
|
||||||
**Update the Font Cache**
|
**Update the Font Cache**
|
||||||
|
|
||||||
|
**Force a Reload of the Installed Font Cache**:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
sudo su -
|
sudo su -
|
||||||
fc-cache -fv
|
fc-cache -fv
|
||||||
fc-cache -frv
|
fc-cache -frv
|
||||||
```
|
```
|
||||||
|
|
||||||
- **`-f`**: Force re-generation of cache files, overriding timestamp checking.
|
- **`-f`**: Force re-generation of apparently up-to-date cache files, overriding the timestamp checking.
|
||||||
- **`-r`**: Erase all existing cache files and rescan.
|
- **`-r`**: Erase all existing cache files and rescan.
|
||||||
- **`-v`**: Display status information while busy.
|
- **`-v`**: Display status information while busy.
|
||||||
|
|||||||
324
notes/ssh.md
324
notes/ssh.md
@ -4,9 +4,7 @@
|
|||||||
|
|
||||||
- [SSH](#ssh)
|
- [SSH](#ssh)
|
||||||
- [Table of Contents](#table-of-contents)
|
- [Table of Contents](#table-of-contents)
|
||||||
- [Placeholders](#placeholders)
|
|
||||||
- [Connect with specific key](#connect-with-specific-key)
|
- [Connect with specific key](#connect-with-specific-key)
|
||||||
- [Skip Host Key Verification](#skip-host-key-verification)
|
|
||||||
- [SSH Key Management](#ssh-key-management)
|
- [SSH Key Management](#ssh-key-management)
|
||||||
- [Verbose](#verbose)
|
- [Verbose](#verbose)
|
||||||
- [Enable root login](#enable-root-login)
|
- [Enable root login](#enable-root-login)
|
||||||
@ -18,124 +16,93 @@
|
|||||||
- [Change SSH Port](#change-ssh-port)
|
- [Change SSH Port](#change-ssh-port)
|
||||||
- [Restart ssh](#restart-ssh)
|
- [Restart ssh](#restart-ssh)
|
||||||
|
|
||||||
## Placeholders
|
|
||||||
|
|
||||||
Replace the placeholders below with the appropriate values for your setup:
|
|
||||||
|
|
||||||
- **Connection**
|
|
||||||
- Username: `<username>` (e.g., john)
|
|
||||||
- Hostname: `<hostname>` (e.g., server.example.com)
|
|
||||||
- IP address: `<ip>` (e.g., 192.168.1.100)
|
|
||||||
- SSH port: `<port>` (e.g., 2222)
|
|
||||||
- SSH key: `<keyfile>` (e.g., ~/.ssh/id_rsa)
|
|
||||||
- Key comment: `<key-comment>` (e.g., john@example.com)
|
|
||||||
- Host alias: `<alias>` (e.g., myserver)
|
|
||||||
|
|
||||||
- **Paths**
|
|
||||||
- Local file: `<local-path>` (e.g., /home/user/file.txt)
|
|
||||||
- Remote path: `<remote-path>` (e.g., /home/user/file.txt)
|
|
||||||
- Local script: `<script-path>` (e.g., /home/user/script.sh)
|
|
||||||
- Project name: `<project>` (e.g., myapp)
|
|
||||||
|
|
||||||
## Connect with specific key
|
## Connect with specific key
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh -i <keyfile> root@<hostname>
|
ssh -i /home/fabrice/.ssh/fabquenneville root@servername.fabq.ca
|
||||||
ssh -i <keyfile> <username>@<hostname>
|
ssh -i /home/fabrice/.ssh/fabquenneville fabrice@servername.fabq.ca
|
||||||
```
|
```
|
||||||
|
|
||||||
## Skip Host Key Verification
|
|
||||||
|
|
||||||
Useful for ephemeral machines, VMs, or hosts that are frequently rebuilt where saved known_hosts entries would cause conflicts:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null <username>@<hostname>
|
|
||||||
```
|
|
||||||
|
|
||||||
- `StrictHostKeyChecking=no`: Automatically accepts new or changed host keys without prompting.
|
|
||||||
- `UserKnownHostsFile=/dev/null`: Discards the host key entirely — nothing is saved to `~/.ssh/known_hosts`.
|
|
||||||
- ⚠️ Do not use on untrusted networks — this disables MITM protection.
|
|
||||||
|
|
||||||
## SSH Key Management
|
## SSH Key Management
|
||||||
|
|
||||||
**Generate a new RSA SSH key pair with a 4096-bit key length**
|
**Generate a new RSA SSH key pair with a 4096-bit key length**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh-keygen -t rsa -b 4096 -C "<key-comment>" -f <keyfile>
|
ssh-keygen -t rsa -b 4096 -C "fabrice@fabq.ca" -f ~/.ssh/fabrice@fabq.ca
|
||||||
```
|
```
|
||||||
|
|
||||||
- `ssh-keygen -t rsa -b 4096`: Generates a new RSA SSH key pair with a key size of 4096 bits for enhanced security.
|
- `ssh-keygen -t rsa -b 4096`: This command generates a new RSA SSH key pair with a key size of 4096 bits for enhanced security.
|
||||||
- `-C "<key-comment>"`: Adds a comment to the key, usually the email address of the key owner.
|
- `-C "fabrice@fabq.ca"`: This option adds a comment to the key, usually the email address of the key owner.
|
||||||
- `-f <keyfile>`: Specifies the filename for the private key; the public key will be saved with the same name but with a `.pub` extension.
|
- `-f ~/.ssh/fabrice@fabq.ca`: This specifies the filename for the private key; the public key will be saved with the same name but with a `.pub` extension.
|
||||||
|
|
||||||
**Copy the generated SSH keys to the remote server**
|
**Copy the generated SSH keys to the remote server**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
scp <keyfile>* <username>@<hostname>:~/.ssh/
|
scp ~/.ssh/fabrice@fabq.ca* fabrice@servername.fabq.ca:~/.ssh/
|
||||||
```
|
```
|
||||||
|
|
||||||
- `scp <keyfile>*`: Securely copies both the private and public keys to the remote server.
|
- `scp ~/.ssh/fabrice@fabq.ca*`: This command securely copies both the private and public keys to the remote server.
|
||||||
- `<username>@<hostname>:~/.ssh/`: Specifies the destination path on the remote server where the keys will be copied.
|
- `fabrice@servername.fabq.ca:~/.ssh/`: Specifies the destination path on the remote server where the keys will be copied.
|
||||||
|
|
||||||
**Install the public key on the remote server for passwordless authentication**
|
**Install the public key on the remote server for passwordless authentication**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh-copy-id <username>@<ip>
|
ssh-copy-id fabrice@192.168.1.100
|
||||||
ssh-copy-id <username>@<hostname>
|
ssh-copy-id fabrice@servername.fabq.ca
|
||||||
```
|
```
|
||||||
|
|
||||||
- `ssh-copy-id`: Installs the public key on the specified remote server, allowing for passwordless SSH login.
|
- `ssh-copy-id "fabrice@servername.fabq.ca"`: This command installs the public key on the specified remote server, allowing for passwordless SSH login.
|
||||||
|
|
||||||
**Install the public key on multiple servers using a specific private key**
|
**Install the public key on multiple servers using specific private key**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh-copy-id -i <keyfile> root@<ip>
|
ssh-copy-id -i /home/fabrice/.ssh/fabquenneville root@192.168.1.100
|
||||||
ssh-copy-id -i <keyfile> <username>@<hostname>
|
ssh-copy-id -i /home/fabrice/.ssh/fabquenneville fabrice@servername.fabq.ca
|
||||||
```
|
```
|
||||||
|
|
||||||
- `-i <keyfile>`: Specifies which private key to use for authentication when copying the public key.
|
- `ssh-copy-id -i /home/fabrice/.ssh/fabquenneville`: This specifies which private key to use for authentication when copying the public key.
|
||||||
|
- `root@192.168.1.100` and `fabrice@servername.fabq.ca`: These commands install the public key on the respective remote servers, allowing for secure, passwordless access.
|
||||||
|
|
||||||
**Install the public key on the remote server for passwordless authentication manually**
|
**Install the public key on the remote server for passwordless authentication manually**
|
||||||
|
This process is useful when ssh-copy-id is unavailable, or when you want more granular control over the manual setup of passwordless SSH authentication. The ssh-copy-id tool automatically installs your public key on the remote machine, but if you prefer or need to do it manually, these are the steps:
|
||||||
|
|
||||||
This process is useful when `ssh-copy-id` is unavailable, or when you want more granular control over the setup. Follow these steps on the remote server:
|
1. Create the .ssh directory if it doesn't exist and set proper permissions
|
||||||
|
|
||||||
1. Create the `.ssh` directory if it doesn't exist and set proper permissions:
|
```bash
|
||||||
|
mkdir -p /home/fabrice/.ssh
|
||||||
|
chmod 700 /home/fabrice/.ssh
|
||||||
|
```
|
||||||
|
|
||||||
```bash
|
2. Open the authorized_keys file in an editor and paste the public key (usually from ~/.ssh/id_rsa.pub on the local machine)
|
||||||
mkdir -p /home/<username>/.ssh
|
|
||||||
chmod 700 /home/<username>/.ssh
|
|
||||||
```
|
|
||||||
|
|
||||||
2. Open the `authorized_keys` file and paste the public key (usually from `~/.ssh/id_rsa.pub` on the local machine):
|
```bash
|
||||||
|
nano /home/fabrice/.ssh/authorized_keys
|
||||||
|
```
|
||||||
|
|
||||||
```bash
|
3. Set the correct permissions for the authorized_keys file
|
||||||
nano /home/<username>/.ssh/authorized_keys
|
|
||||||
```
|
|
||||||
|
|
||||||
3. Set the correct permissions for the `authorized_keys` file:
|
```bash
|
||||||
|
chmod 600 /home/fabrice/.ssh/authorized_keys
|
||||||
|
```
|
||||||
|
|
||||||
```bash
|
4. Ensure the ownership of the .ssh directory and its contents is set to the correct user
|
||||||
chmod 600 /home/<username>/.ssh/authorized_keys
|
|
||||||
```
|
|
||||||
|
|
||||||
4. Ensure the ownership of the `.ssh` directory and its contents is set to the correct user:
|
```bash
|
||||||
|
chown -R fabrice:fabrice /home/fabrice/.ssh
|
||||||
```bash
|
```
|
||||||
chown -R <username>:<username> /home/<username>/.ssh
|
|
||||||
```
|
|
||||||
|
|
||||||
## Verbose
|
## Verbose
|
||||||
|
|
||||||
Use the `-v` option to enable verbose mode, which provides detailed information about the connection process:
|
- Use the 'ssh' command with the '-v' option to enable verbose mode, which provides detailed information about the connection process.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh -i <keyfile> -v root@<hostname>
|
ssh -i /home/fabrice/.ssh/fabquenneville -v root@servername.fabq.ca
|
||||||
ssh -i <keyfile> -v <username>@<hostname>
|
ssh -i /home/fabrice/.ssh/fabquenneville -v fabrice@servername.fabq.ca
|
||||||
```
|
```
|
||||||
|
|
||||||
## Enable root login
|
## Enable root login
|
||||||
|
|
||||||
Modify the SSH configuration file to allow root login:
|
- Modify the SSH configuration file to allow root login.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
nano /etc/ssh/sshd_config
|
nano /etc/ssh/sshd_config
|
||||||
@ -156,203 +123,284 @@ PermitRootLogin yes
|
|||||||
firewall-cmd --permanent --zone=public --add-service=ssh
|
firewall-cmd --permanent --zone=public --add-service=ssh
|
||||||
```
|
```
|
||||||
|
|
||||||
- `firewall-cmd`: The command-line tool used to manage `firewalld`.
|
- `firewall-cmd`: This is the command-line tool used to manage `firewalld`.
|
||||||
- `--permanent`: Ensures that the change persists across reboots.
|
- `--permanent`: This option ensures that the change persists across reboots.
|
||||||
- `--zone=public`: Specifies the zone to which the rule applies. The "public" zone is typically used for untrusted networks.
|
- `--zone=public`: Specifies the zone to which the rule applies. The "public" zone is typically used for untrusted networks.
|
||||||
- `--add-service=ssh`: Adds the SSH service to the specified zone, allowing incoming SSH connections.
|
- `--add-service=ssh`: This adds the SSH service to the specified zone, allowing incoming SSH connections.
|
||||||
|
|
||||||
**Examples of configuring other Linux firewalls**
|
**Examples of configuring other Linux firewalls**
|
||||||
|
|
||||||
1. **Using UFW (Uncomplicated Firewall)**
|
1. **Using UFW (Uncomplicated Firewall)**
|
||||||
|
|
||||||
|
**Allow SSH traffic**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ufw allow ssh
|
ufw allow ssh
|
||||||
```
|
```
|
||||||
|
|
||||||
|
- This command allows incoming SSH traffic through the firewall. UFW is designed to simplify the process of managing a firewall.
|
||||||
|
|
||||||
2. **Using iptables**
|
2. **Using iptables**
|
||||||
|
|
||||||
|
**Allow SSH traffic**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
|
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
|
||||||
```
|
```
|
||||||
|
|
||||||
|
- `iptables`: This is a low-level tool for managing Linux firewalls.
|
||||||
- `-A INPUT`: Appends the rule to the INPUT chain.
|
- `-A INPUT`: Appends the rule to the INPUT chain.
|
||||||
- `-p tcp --dport 22`: Matches TCP traffic on port 22.
|
- `-p tcp`: Specifies that this rule applies to TCP packets.
|
||||||
- `-j ACCEPT`: Accepts the specified traffic.
|
- `--dport 22`: Indicates that this rule applies to traffic on port 22 (the default SSH port).
|
||||||
|
- `-j ACCEPT`: Instructs the firewall to accept the specified traffic.
|
||||||
|
|
||||||
3. **Using nftables**
|
3. **Using nftables**
|
||||||
|
|
||||||
|
**Allow SSH traffic**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
nft add rule ip filter input tcp dport 22 accept
|
nft add rule ip filter input tcp dport 22 accept
|
||||||
```
|
```
|
||||||
|
|
||||||
|
- `nft`: The command-line tool for interacting with the nftables framework.
|
||||||
- `add rule ip filter input`: Adds a new rule to the input chain of the filter table.
|
- `add rule ip filter input`: Adds a new rule to the input chain of the filter table.
|
||||||
- `tcp dport 22`: Matches TCP packets directed to port 22.
|
- `tcp dport 22`: Matches TCP packets directed to port 22.
|
||||||
- `accept`: Accepts the matching packets.
|
- `accept`: Specifies that the matching packets should be accepted.
|
||||||
|
|
||||||
**Note:** Reload or restart the firewall service after making changes to apply the new rules.
|
**Note:** Be sure to reload or restart the firewall service after making changes to apply the new rules effectively.
|
||||||
|
|
||||||
## SCP (Secure Copy Protocol)
|
## SCP (Secure Copy Protocol)
|
||||||
|
|
||||||
The `scp` command securely transfers files and directories between local and remote systems over SSH.
|
- The `scp` command is used to securely transfer files and directories between local and remote systems over SSH.
|
||||||
|
|
||||||
**Copy a local file to a remote server:**
|
**Copy Local File to Remote Server**
|
||||||
|
|
||||||
|
To copy a file from your local machine to a remote server, use the following syntax:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
scp <local-path> <username>@<hostname>:<remote-path>
|
scp /local/file/path fabrice@servername.fabq.ca:/remote/file/path
|
||||||
```
|
```
|
||||||
|
|
||||||
**Copy a directory recursively:**
|
- `/local/file/path`: Specify the path to the local file you want to copy.
|
||||||
|
- `fabrice@servername.fabq.ca`: This is the user and remote server where the file will be copied.
|
||||||
|
- `/remote/file/path`: This is the destination path on the remote server.
|
||||||
|
|
||||||
|
**Copy a Directory**
|
||||||
|
|
||||||
|
To copy an entire directory, use the `-r` option, which stands for "recursive":
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
scp -r <local-path>/ <username>@<hostname>:<remote-path>/
|
scp -r /home/fabrice/foldername/ root@servername.fabq.ca:/remote/parent/
|
||||||
```
|
```
|
||||||
|
|
||||||
- `-r`: Enables recursive copying of directories and their contents.
|
- `-r`: This option enables recursive copying of directories and their contents.
|
||||||
|
- `/home/fabrice/foldername/`: The path to the local directory you wish to copy.
|
||||||
|
- `root@servername.fabq.ca:/remote/parent/`: The destination path on the remote server where the directory will be copied.
|
||||||
|
|
||||||
**Copy a file using a specific SSH key:**
|
**Copy a Configuration File Using a Specific SSH Key**
|
||||||
|
|
||||||
|
To copy a configuration file while specifying a particular SSH key for authentication, use the following command:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
scp -i <keyfile> <local-path> <username>@<hostname>:<remote-path>
|
scp -i /home/fabrice/.ssh/fabquenneville /mnt/workbench/webserver/projectname/config.ini fabrice@servername.fabq.ca:/mnt/workbench/projectname/
|
||||||
```
|
```
|
||||||
|
|
||||||
- `-i <keyfile>`: Specifies the identity file (private key) for authentication.
|
- `-i /home/fabrice/.ssh/fabquenneville`: This option specifies the identity file (private key) for authentication.
|
||||||
|
- `/mnt/workbench/webserver/projectname/config.ini`: The path to the local configuration file being transferred.
|
||||||
|
- `fabrice@servername.fabq.ca`: The user and server to which the file is being copied.
|
||||||
|
- `/mnt/workbench/projectname/`: The destination path on the remote server where the file will be stored.
|
||||||
|
|
||||||
## Send Remote Commands
|
## Send Remote Commands
|
||||||
|
|
||||||
**Run a local script on a remote server:**
|
**Execute Commands Directly on a Remote Server**
|
||||||
|
|
||||||
|
You can use the `ssh` command to execute various commands on a remote server. Here are some examples:
|
||||||
|
|
||||||
|
**Run a Local Script on a Remote Server**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh <username>@<hostname> 'bash -s' < <script-path>
|
ssh fabrice@servername.fabq.ca 'bash -s' < /local/path/to/script.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
|
- This command will execute the local script located at `/local/path/to/script.sh` on the remote server.
|
||||||
|
|
||||||
**Remove a file:**
|
**Remove a file:**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh <username>@<ip> "rm <remote-path>"
|
ssh fabrice@192.168.1.100 "rm /home/fabrice/filename.log"
|
||||||
```
|
```
|
||||||
|
|
||||||
**Mount all filesystems:**
|
**Mount all filesystems:**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh root@<hostname> "mount -a"
|
ssh root@servername.fabq.ca "mount -a"
|
||||||
```
|
```
|
||||||
|
|
||||||
**Reboot the remote server:**
|
**Reboot the remote server:**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh root@<hostname> "reboot -h now"
|
ssh root@servername.fabq.ca "reboot -h now"
|
||||||
```
|
```
|
||||||
|
|
||||||
**Connect using a host key alias:**
|
**Connect using a host key alias:**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh -o "HostKeyAlias=<alias>" root@<ip>
|
ssh -o "HostKeyAlias=servername" root@192.168.1.100
|
||||||
```
|
```
|
||||||
|
|
||||||
## Test Connection, Add Alias, and Update Known Hosts
|
## Test Connection, Add Alias, and Update Known Hosts
|
||||||
|
|
||||||
**Test connection with host key alias:**
|
**Test Connection with Host Key Alias**
|
||||||
|
|
||||||
Commands to establish an SSH connection while specifying a host key alias. This helps avoid conflicts with existing entries in the `known_hosts` file.
|
Use the following commands to establish an SSH connection while specifying a host key alias. This helps avoid conflicts with existing entries in the `known_hosts` file.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh -o 'HostKeyAlias=<hostname>' <username>@<ip>
|
ssh -o 'HostKeyAlias=servername.fabq.ca' fabrice@192.168.1.100
|
||||||
ssh -o 'HostKeyAlias=<alias>' <username>@<ip>
|
ssh -o 'HostKeyAlias=servername' fabrice@192.168.1.100
|
||||||
```
|
```
|
||||||
|
|
||||||
**Test host identity without authenticating:**
|
**Test Host Identity without Authenticating**
|
||||||
|
|
||||||
|
To test the identity of a remote server without fully authenticating and to check connectivity, use the following command:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=<alias>' <username>@<ip> /bin/true
|
ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=servername' fabrice@192.168.1.100 /bin/true
|
||||||
```
|
```
|
||||||
|
|
||||||
- `-e none`: Disables escape character processing.
|
- `-e none`: Disables encryption for this command, which is useful in specific testing scenarios.
|
||||||
- `-o 'BatchMode=yes'`: Suppresses all prompts, suitable for scripts.
|
- `-o 'BatchMode=yes'`: Ensures that SSH does not prompt for user interaction, making it suitable for scripts.
|
||||||
- `/bin/true`: Simple command that always returns success, confirming the connection without further actions.
|
- `/bin/true`: Executes a simple command that always returns success, confirming the connection without further actions.
|
||||||
|
|
||||||
**Retrieve public SSH keys from a remote server:**
|
This command allows you to verify that you can connect to the server while avoiding any authentication prompts.
|
||||||
|
|
||||||
```bash
|
**Retrieve Public SSH Keys**
|
||||||
ssh-keyscan -H <hostname>
|
|
||||||
|
To retrieve the public SSH keys from a remote server, use the following command:
|
||||||
|
|
||||||
|
```
|
||||||
|
ssh-keyscan -H servername.fabq.ca
|
||||||
```
|
```
|
||||||
|
|
||||||
- Fetches the server's public SSH keys without establishing a full session. Used to pre-populate `known_hosts`.
|
- This command fetches the public SSH keys from the specified server, allowing you to add them to your `known_hosts` file.
|
||||||
|
- It does not establish a full SSH session and is primarily used for key retrieval, which helps ensure secure connections in future interactions.
|
||||||
|
|
||||||
**Add an alias to SSH config for easy access:**
|
By using both commands, you can test connectivity to a remote server and gather its public SSH keys for secure authentication later.
|
||||||
|
|
||||||
|
**Add Alias to SSH Config for Easy Access**
|
||||||
|
|
||||||
|
To simplify your SSH connections, you can create an alias for your SSH connections by editing the `~/.ssh/config` file:
|
||||||
|
|
||||||
```ini
|
```ini
|
||||||
Host <alias>
|
Host servername
|
||||||
HostName <hostname>
|
HostName servername.fabq.ca
|
||||||
User <username>
|
User fabrice
|
||||||
IdentityFile <keyfile>
|
IdentityFile ~/.ssh/fabquenneville
|
||||||
```
|
```
|
||||||
|
|
||||||
|
- `Host servername`: This defines the alias you will use for the SSH connection.
|
||||||
|
- `HostName servername.fabq.ca`: This is the actual hostname of the remote server.
|
||||||
|
- `User fabrice`: This specifies the user to log in as.
|
||||||
|
- `IdentityFile ~/.ssh/fabquenneville`: This indicates the SSH key file to use for authentication.
|
||||||
|
|
||||||
## Remove Offending SSH Keys
|
## Remove Offending SSH Keys
|
||||||
|
|
||||||
When a server's host key has changed, remove the old entry from `known_hosts`.
|
When you encounter an "offending key" warning when connecting to a server, you can remove the old key from the `known_hosts` file. This is necessary if the server's host key has changed.
|
||||||
|
|
||||||
**View known hosts:**
|
**View Known Hosts**
|
||||||
|
|
||||||
|
To view the contents of your `known_hosts` file, use:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cat ~/.ssh/known_hosts
|
cat ~/.ssh/known_hosts
|
||||||
```
|
```
|
||||||
|
|
||||||
**Edit known hosts manually:**
|
**Edit Known Hosts Manually (Optional)**
|
||||||
|
|
||||||
|
You can edit the `known_hosts` file manually if you prefer:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
nano ~/.ssh/known_hosts
|
nano ~/.ssh/known_hosts
|
||||||
```
|
```
|
||||||
|
|
||||||
**Update known hosts with current server key:**
|
**Update Known Hosts File with SSH Key**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh-keyscan -H <hostname> >> ~/.ssh/known_hosts
|
ssh-keyscan -H servername.fabq.ca >> ~/.ssh/known_hosts
|
||||||
```
|
```
|
||||||
|
|
||||||
**Remove offending key by hostname or IP:**
|
- This command retrieves the public key of the specified server and appends it to your `known_hosts` file, allowing SSH to recognize the server during subsequent connections.
|
||||||
|
|
||||||
|
**Remove Offending Key by Hostname**
|
||||||
|
|
||||||
|
You can use the `ssh-keygen` command to remove specific keys from your `known_hosts` file. Here are examples for different scenarios:
|
||||||
|
|
||||||
|
- To remove the offending key for a specific server:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh-keygen -R "<hostname>"
|
ssh-keygen -R "servername.fabq.ca"
|
||||||
ssh-keygen -R "<ip>"
|
ssh-keygen -R "192.168.1.100"
|
||||||
```
|
```
|
||||||
|
|
||||||
**Remove offending key specifying the known_hosts file:**
|
- To specify the `known_hosts` file directly:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh-keygen -f "/home/<username>/.ssh/known_hosts" -R "<hostname>"
|
ssh-keygen -f "/home/fabrice/.ssh/known_hosts" -R "servername.fabq.ca"
|
||||||
ssh-keygen -f "/root/.ssh/known_hosts" -R "<ip>"
|
ssh-keygen -f "/root/.ssh/known_hosts" -R "192.168.1.100"
|
||||||
ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R "<hostname>"
|
ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R "servername.fabq.ca"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Summary of Key Removal**
|
||||||
|
|
||||||
|
You can also use a shorthand command to remove the offending key without specifying the file:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh-keygen -R servername.fabq.ca
|
||||||
|
```
|
||||||
|
|
||||||
|
This will automatically target the correct `known_hosts` file based on your user and system configuration.
|
||||||
|
|
||||||
## Change SSH Port
|
## Change SSH Port
|
||||||
|
|
||||||
**1. Edit the SSH configuration file:**
|
To enhance security, you may want to change the default SSH port (22) to a custom port. Follow these steps:
|
||||||
|
|
||||||
|
**1. Edit the SSH Configuration File**
|
||||||
|
|
||||||
|
Open the SSH daemon configuration file using a text editor:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
nano /etc/ssh/sshd_config
|
nano /etc/ssh/sshd_config
|
||||||
```
|
```
|
||||||
|
|
||||||
Set the desired port:
|
Edit the following line to set a new port (e.g., port 2222):
|
||||||
|
|
||||||
```ini
|
```ini
|
||||||
Port <port>
|
Port 2222
|
||||||
```
|
```
|
||||||
|
|
||||||
**2. Create directory for systemd override:**
|
- Locate the line that specifies the port (usually `#Port 22`) and change it to your desired port number (e.g., `Port 2222`).
|
||||||
|
- Make sure to uncomment the line by removing the `#`.
|
||||||
|
|
||||||
|
**2. Create Directory for Systemd Override**
|
||||||
|
|
||||||
|
If you're using systemd, create a directory for the SSH socket override:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
mkdir -p /etc/systemd/system/ssh.socket.d
|
mkdir -p /etc/systemd/system/ssh.socket.d
|
||||||
```
|
```
|
||||||
|
|
||||||
**3. Create the override configuration file:**
|
**3. Create an Override Configuration File**
|
||||||
|
|
||||||
|
Create or edit the override configuration file for the SSH socket:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
nano /etc/systemd/system/ssh.socket.d/override.conf
|
nano /etc/systemd/system/ssh.socket.d/override.conf
|
||||||
```
|
```
|
||||||
|
|
||||||
|
- Add the following lines to specify the custom port:
|
||||||
|
|
||||||
```ini
|
```ini
|
||||||
[Socket]
|
[Socket]
|
||||||
ListenPort=<port>
|
ListenPort=2222 # Replace with your desired port number
|
||||||
```
|
```
|
||||||
|
|
||||||
**4. (Optional) Edit the Sockets Target Configuration**
|
**4. (Optional) Edit the Sockets Target Configuration**
|
||||||
@ -363,20 +411,32 @@ You may also want to edit the sockets target configuration to ensure it points t
|
|||||||
nano /etc/systemd/system/sockets.target.wants/ssh.socket
|
nano /etc/systemd/system/sockets.target.wants/ssh.socket
|
||||||
```
|
```
|
||||||
|
|
||||||
**5. Restart the SSH service to apply the changes:**
|
- Make any necessary adjustments based on your custom port.
|
||||||
|
|
||||||
|
**5. Restart the SSH service to apply the changes**
|
||||||
|
|
||||||
|
After making changes, restart the SSH service to apply the new configuration:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
systemctl restart sshd
|
systemctl restart sshd
|
||||||
```
|
```
|
||||||
|
|
||||||
**6. Verify the new port:**
|
**6. (Optional) Verify the New Port**
|
||||||
|
|
||||||
|
To verify that SSH is listening on the new port, you can use:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
netstat -tuln | grep LISTEN
|
netstat -tuln | grep LISTEN
|
||||||
```
|
```
|
||||||
|
|
||||||
|
This will display the ports currently being listened to, allowing you to confirm that your changes were successful.
|
||||||
|
|
||||||
## Restart ssh
|
## Restart ssh
|
||||||
|
|
||||||
|
**Restart the SSH service to apply changes**
|
||||||
|
|
||||||
|
To restart the SSH service, use the following command:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
systemctl restart sshd
|
systemctl restart sshd
|
||||||
```
|
```
|
||||||
|
|||||||
530
notes/storage.md
530
notes/storage.md
@ -1,530 +0,0 @@
|
|||||||
# Storage
|
|
||||||
|
|
||||||
## Table of Contents
|
|
||||||
|
|
||||||
- [Storage](#storage)
|
|
||||||
- [Table of Contents](#table-of-contents)
|
|
||||||
- [Placeholders](#placeholders)
|
|
||||||
- [Drive Information](#drive-information)
|
|
||||||
- [List Block Devices](#list-block-devices)
|
|
||||||
- [List Mountpoints and Usage](#list-mountpoints-and-usage)
|
|
||||||
- [Inspect fstab](#inspect-fstab)
|
|
||||||
- [Find Device Path from UUID](#find-device-path-from-uuid)
|
|
||||||
- [Power On Hours](#power-on-hours)
|
|
||||||
- [Swap](#swap)
|
|
||||||
- [Partitions and Filesystems](#partitions-and-filesystems)
|
|
||||||
- [TRIM](#trim)
|
|
||||||
- [Mounting](#mounting)
|
|
||||||
- [SMART Diagnostics](#smart-diagnostics)
|
|
||||||
- [Hardware Monitoring](#hardware-monitoring)
|
|
||||||
- [Kernel Messages](#kernel-messages)
|
|
||||||
- [Badblocks](#badblocks)
|
|
||||||
- [Hex Dump](#hex-dump)
|
|
||||||
- [Cloning drives and images with dd](#cloning-drives-and-images-with-dd)
|
|
||||||
- [Benchmarking](#benchmarking)
|
|
||||||
- [USB Devices](#usb-devices)
|
|
||||||
- [Device Recognition](#device-recognition)
|
|
||||||
- [Test Device Integrity](#test-device-integrity)
|
|
||||||
- [Switching Two USB Keys](#switching-two-usb-keys)
|
|
||||||
|
|
||||||
## Placeholders
|
|
||||||
|
|
||||||
Replace the placeholders below with the appropriate values for your setup:
|
|
||||||
|
|
||||||
- **Devices**
|
|
||||||
- Block device: `<device>` (e.g., /dev/sda)
|
|
||||||
- Partition: `<partition>` (e.g., /dev/sda1)
|
|
||||||
- UUID: `<uuid>` (e.g., a1b2c3d4-e5f6-7890-abcd-ef1234567890)
|
|
||||||
|
|
||||||
- **Paths**
|
|
||||||
- Mount point: `<mountpoint>` (e.g., /mnt/media)
|
|
||||||
- Directory path: `<dirpath>` (e.g., /mnt/data)
|
|
||||||
|
|
||||||
## Drive Information
|
|
||||||
|
|
||||||
### List Block Devices
|
|
||||||
|
|
||||||
Check all attached block devices by type:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ls /dev/sd*
|
|
||||||
ls /dev/nv*
|
|
||||||
```
|
|
||||||
|
|
||||||
### List Mountpoints and Usage
|
|
||||||
|
|
||||||
**Tree view of all block devices with sizes and mountpoints**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
lsblk
|
|
||||||
lsblk -f
|
|
||||||
```
|
|
||||||
|
|
||||||
- `-f`: Add filesystem type, label, and UUID to the tree.
|
|
||||||
|
|
||||||
**Exclude loop devices from the listing**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
lsblk -e 7
|
|
||||||
```
|
|
||||||
|
|
||||||
- `-e 7`: Excludes device major number 7 (loop devices), keeping the output clean on systems with many snaps or loop mounts.
|
|
||||||
|
|
||||||
**Disk space usage for all mounted filesystems**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
df -h
|
|
||||||
```
|
|
||||||
|
|
||||||
- `-h`: Human-readable.
|
|
||||||
|
|
||||||
**Disk space usage for a specific device**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
df -h | grep <device>
|
|
||||||
```
|
|
||||||
|
|
||||||
### Inspect fstab
|
|
||||||
|
|
||||||
**View all configured mount entries**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cat /etc/fstab
|
|
||||||
```
|
|
||||||
|
|
||||||
### Find Device Path from UUID
|
|
||||||
|
|
||||||
Using `lsblk`:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
lsblk -o NAME,UUID,MOUNTPOINT
|
|
||||||
```
|
|
||||||
|
|
||||||
Using `blkid`:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
blkid | grep <uuid>
|
|
||||||
blkid -U <uuid>
|
|
||||||
blkid <partition>
|
|
||||||
```
|
|
||||||
|
|
||||||
### Power On Hours
|
|
||||||
|
|
||||||
Check power-on hours across multiple drives at once:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
for dev in /dev/sd{a,b,c,d}; do echo -n "$dev: "; smartctl -a $dev | grep "Power_On_Hours"; done
|
|
||||||
for dev in /dev/sd{a..d}; do echo -n "$dev: "; smartctl -a $dev | grep "Power_On_Hours"; done
|
|
||||||
```
|
|
||||||
|
|
||||||
### Swap
|
|
||||||
|
|
||||||
Check Swap currently used by the system:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
swapon --show
|
|
||||||
```
|
|
||||||
|
|
||||||
## Partitions and Filesystems
|
|
||||||
|
|
||||||
**View partition table and disk details:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
fdisk -l <device>
|
|
||||||
```
|
|
||||||
|
|
||||||
**Check and repair a filesystem:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
fsck <partition>
|
|
||||||
```
|
|
||||||
|
|
||||||
## TRIM
|
|
||||||
|
|
||||||
TRIM allows the OS to inform the drive which blocks are no longer in use, maintaining performance on SSDs and NVMe drives over time.
|
|
||||||
|
|
||||||
**Run TRIM once manually across all mounted filesystems:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
fstrim -av
|
|
||||||
```
|
|
||||||
|
|
||||||
- `-a`: Run on all mounted filesystems that support TRIM.
|
|
||||||
- `-v`: Verbose — reports how much space was freed per filesystem.
|
|
||||||
|
|
||||||
**Enable the weekly TRIM timer:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
systemctl enable --now fstrim.timer
|
|
||||||
```
|
|
||||||
|
|
||||||
- Debian/Ubuntu run this weekly by default once enabled.
|
|
||||||
- Check timer status with `systemctl status fstrim.timer`.
|
|
||||||
|
|
||||||
## Mounting
|
|
||||||
|
|
||||||
**Validate all `fstab` entries**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
findmnt --verify --verbose
|
|
||||||
```
|
|
||||||
|
|
||||||
**Dry-run mount of all fstab entries, reports what would succeed or fail**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mount -fav
|
|
||||||
```
|
|
||||||
|
|
||||||
- `-f`: Fake mount — goes through the motions without making the actual syscall.
|
|
||||||
- `-a`: Process all entries in `/etc/fstab` (excluding `noauto`).
|
|
||||||
- `-v`: Verbose output.
|
|
||||||
|
|
||||||
**Find mount point details:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
findmnt <device>
|
|
||||||
mount | grep <device>
|
|
||||||
```
|
|
||||||
|
|
||||||
**Check active mounts for a specific mountpoint:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cat /proc/mounts | grep <mountpoint>
|
|
||||||
```
|
|
||||||
|
|
||||||
**Apply fstab changes without rebooting**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mount -o remount -a
|
|
||||||
```
|
|
||||||
|
|
||||||
**Unmount and Safely Remove**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
umount <device>
|
|
||||||
eject <device>
|
|
||||||
```
|
|
||||||
|
|
||||||
## SMART Diagnostics
|
|
||||||
|
|
||||||
**Get full SMART information for a drive:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
smartctl -a <device>
|
|
||||||
```
|
|
||||||
|
|
||||||
**Get drive identity and model info only:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
smartctl -i <device>
|
|
||||||
```
|
|
||||||
|
|
||||||
```bash
|
|
||||||
for dev in /dev/sd[a-z] /dev/nvme[0-9]n[0-9]; do
|
|
||||||
echo "--- $dev ---"
|
|
||||||
smartctl -i $dev | grep -Ei "Model|Serial Number|Capacity"
|
|
||||||
done
|
|
||||||
```
|
|
||||||
|
|
||||||
**Run a short SMART test:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
smartctl -t short <device>
|
|
||||||
```
|
|
||||||
|
|
||||||
**Run a long SMART test:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
smartctl -t long <device>
|
|
||||||
```
|
|
||||||
|
|
||||||
**View test results:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
smartctl -a <device>
|
|
||||||
```
|
|
||||||
|
|
||||||
**Check SMART device statistics (error counters, etc.):**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
smartctl -A <device>
|
|
||||||
```
|
|
||||||
|
|
||||||
**Filter for key health attributes:**
|
|
||||||
|
|
||||||
Check the most important failure indicators in a single line:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
smartctl -A <device> | grep -E "Reallocated|Pending|UDMA_CRC"
|
|
||||||
```
|
|
||||||
|
|
||||||
**Check multiple health attributes at once:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
smartctl -a <device> | grep -E "Power_On_Hours|Load_Cycle_Count|Reallocated_Sector_Ct"
|
|
||||||
```
|
|
||||||
|
|
||||||
**Check temperatures across all drives:**
|
|
||||||
|
|
||||||
Scans all SMART-capable devices and prints their temperature:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
smartctl --scan | awk '{print $1}' | while read dev; do \
|
|
||||||
echo -n "$dev: "; \
|
|
||||||
smartctl -A $dev | grep -iE 'Temperature|Airflow_Temp' | awk '
|
|
||||||
/Temperature_Celsius/ {print $10 "°C"}
|
|
||||||
/Airflow_Temperature_Cel/ {print $10 "°C"}
|
|
||||||
/Temperature:/ {print $2 "°C"}
|
|
||||||
' | head -n 1; \
|
|
||||||
done
|
|
||||||
```
|
|
||||||
|
|
||||||
**Watch drive temperatures continuously:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
watch -n 5 "smartctl --scan | awk '{print \$1}' | while read dev; do \
|
|
||||||
echo -n \"\$dev: \"; \
|
|
||||||
smartctl -A \$dev | grep -iE 'Temperature|Airflow_Temp' | awk '\
|
|
||||||
/Temperature_Celsius/ {print \$10 \"°C\"} \
|
|
||||||
/Airflow_Temperature_Cel/ {print \$10 \"°C\"} \
|
|
||||||
/Temperature:/ {print \$2 \"°C\"}' | head -n 1; \
|
|
||||||
done"
|
|
||||||
```
|
|
||||||
|
|
||||||
## Hardware Monitoring
|
|
||||||
|
|
||||||
**Install lm-sensors:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
apt install lm-sensors
|
|
||||||
```
|
|
||||||
|
|
||||||
**Detect available sensor chips:**
|
|
||||||
|
|
||||||
Run once after installation to probe for hardware sensors:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sensors-detect
|
|
||||||
```
|
|
||||||
|
|
||||||
**Display current sensor readings:**
|
|
||||||
|
|
||||||
Shows CPU, GPU, and motherboard temperatures, fan speeds, and voltages:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sensors
|
|
||||||
```
|
|
||||||
|
|
||||||
## Kernel Messages
|
|
||||||
|
|
||||||
**Tail the most recent kernel messages:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
dmesg | tail -n 25
|
|
||||||
```
|
|
||||||
|
|
||||||
**Show only errors and warnings:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
dmesg --level=err,warn
|
|
||||||
```
|
|
||||||
|
|
||||||
**Show kernel messages with human-readable timestamps:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
dmesg -T
|
|
||||||
```
|
|
||||||
|
|
||||||
**Filter for NVMe events:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
dmesg | grep -i nvme
|
|
||||||
dmesg -w | grep -i nvme
|
|
||||||
```
|
|
||||||
|
|
||||||
- `-w`: Follow — print new messages as they arrive (like `tail -f`).
|
|
||||||
|
|
||||||
**Filter for ATA/SCSI/SATA/NVMe device events:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
dmesg | grep -i -E 'scsi|ata|nvme|sata'
|
|
||||||
```
|
|
||||||
|
|
||||||
**Filter for I/O errors:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
dmesg | grep -i "I/O error"
|
|
||||||
```
|
|
||||||
|
|
||||||
**Filter for ATA/SCSI/SATA/NVMe device errors:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
dmesg | grep -i -E 'scsi|ata|nvme|sata'
|
|
||||||
```
|
|
||||||
|
|
||||||
**Map ATA port number to block device name:**
|
|
||||||
|
|
||||||
When `dmesg` reports an error on e.g. `ata7` and you need to identify which physical drive that is:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ls -l /sys/class/block/ | grep ata<port-number>
|
|
||||||
dmesg -T | grep -iE "ata"
|
|
||||||
```
|
|
||||||
|
|
||||||
**Filter for BTRFS events:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
dmesg | grep -i btrfs
|
|
||||||
```
|
|
||||||
|
|
||||||
## Badblocks
|
|
||||||
|
|
||||||
**Read-only test:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
badblocks -v <device>
|
|
||||||
```
|
|
||||||
|
|
||||||
**Non-destructive read-write test:**
|
|
||||||
|
|
||||||
`Save to RAM -> Write Pattern -> Verify -> Restore`
|
|
||||||
|
|
||||||
```bash
|
|
||||||
badblocks -nsv <device>
|
|
||||||
```
|
|
||||||
|
|
||||||
- `-n`: Non-destructive read-write test.
|
|
||||||
- `-s`: Show progress.
|
|
||||||
- `-v`: Verbose output.
|
|
||||||
- Warning: If the computer loses power or the kernel panics after Step 2 but before Step 4, that specific block of data will be lost
|
|
||||||
|
|
||||||
**Destructive write test**
|
|
||||||
|
|
||||||
Overwrites all data — use only on blank drives or drives to be deleted.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
badblocks -wsv <device>
|
|
||||||
```
|
|
||||||
|
|
||||||
## Hex Dump
|
|
||||||
|
|
||||||
Inspect raw bytes on a device directly, useful for verifying partition tables, boot sectors, or investigating corruption.
|
|
||||||
|
|
||||||
**View the first 512 bytes (MBR / partition table):**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
hexdump -C -n 512 <device>
|
|
||||||
```
|
|
||||||
|
|
||||||
**View 1 MB of data starting at a specific offset:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
hexdump -C -s 1G -n 1M <device>
|
|
||||||
```
|
|
||||||
|
|
||||||
- `-C`: Canonical format — hex on the left, ASCII on the right.
|
|
||||||
- `-n <length>`: Number of bytes to read.
|
|
||||||
- `-s <offset>`: Skip to this offset before reading.
|
|
||||||
|
|
||||||
**Extract readable strings from a raw device:**
|
|
||||||
|
|
||||||
Useful for locating file paths, filenames, or metadata remnants directly on a block device:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
strings <device> | grep -C 200 "<search-term>" > <output-file>.txt
|
|
||||||
```
|
|
||||||
|
|
||||||
- `-C 200`: Show 200 lines of context around each match.
|
|
||||||
- Redirect to a file — output can be very large on multi-TB drives.
|
|
||||||
|
|
||||||
## Cloning drives and images with dd
|
|
||||||
|
|
||||||
**Clone a drive or create an image:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
dd if=<source-device> of=<destination-device> bs=4M status=progress
|
|
||||||
dd if=<source-device> of=<image-file>.img bs=4M status=progress
|
|
||||||
```
|
|
||||||
|
|
||||||
**Restore from image:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
dd if=<image-file>.img of=<destination-device> bs=4M status=progress
|
|
||||||
sync
|
|
||||||
```
|
|
||||||
|
|
||||||
## Benchmarking
|
|
||||||
|
|
||||||
**Measure read speed with `hdparm`:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
hdparm -t <device>
|
|
||||||
```
|
|
||||||
|
|
||||||
## USB Devices
|
|
||||||
|
|
||||||
### Device Recognition
|
|
||||||
|
|
||||||
Check if the system recognizes the device and show recent USB-related kernel messages:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
lsusb
|
|
||||||
dmesg | tail -n 20
|
|
||||||
```
|
|
||||||
|
|
||||||
### Test Device Integrity
|
|
||||||
|
|
||||||
Warning: The write test is destructive. All data on <device> will be permanently overwritten. Verify the target device with lsblk before running.
|
|
||||||
|
|
||||||
1. **Unmount the device** (if mounted):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
umount <mountpoint>
|
|
||||||
```
|
|
||||||
|
|
||||||
2. **Write test:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
dd if=/dev/zero of=<device> bs=4M count=256 status=progress
|
|
||||||
```
|
|
||||||
|
|
||||||
3. **Read test:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
dd if=<device> of=/dev/null bs=4M count=256 status=progress
|
|
||||||
```
|
|
||||||
|
|
||||||
4. **Check for bad blocks:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
badblocks -wsv <device>
|
|
||||||
```
|
|
||||||
|
|
||||||
5. **Run a SMART test:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
smartctl -t short <device>
|
|
||||||
smartctl -a <device>
|
|
||||||
```
|
|
||||||
|
|
||||||
### Switching Two USB Keys
|
|
||||||
|
|
||||||
Copy data off, reformat, and restore:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cp -r /media/<username>/<volume-label> /tmp/<backup-folder>
|
|
||||||
umount <device-partition>
|
|
||||||
mkfs.vfat <device-partition>
|
|
||||||
|
|
||||||
cp -r /tmp/<backup-folder> /media/<username>/<new-volume-label>
|
|
||||||
umount <device-partition>
|
|
||||||
```
|
|
||||||
|
|
||||||
Or clone with `dd`:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
dd if=<source-device> of=/tmp/usb_image.img bs=4M status=progress
|
|
||||||
mkfs.vfat <device-partition>
|
|
||||||
dd if=/tmp/usb_image.img of=<destination-device> bs=4M status=progress
|
|
||||||
sync
|
|
||||||
```
|
|
||||||
@ -1,163 +0,0 @@
|
|||||||
# WordPress
|
|
||||||
|
|
||||||
## Table of Contents
|
|
||||||
|
|
||||||
- [WordPress](#wordpress)
|
|
||||||
- [Table of Contents](#table-of-contents)
|
|
||||||
- [Documentation](#documentation)
|
|
||||||
- [Common WordPress Errors](#common-wordpress-errors)
|
|
||||||
- [Debugging Steps](#debugging-steps)
|
|
||||||
- [Debugging Tools](#debugging-tools)
|
|
||||||
|
|
||||||
## Documentation
|
|
||||||
|
|
||||||
- [wp-cli](https://developer.wordpress.org/cli/commands/)
|
|
||||||
- [Documentation Overview](https://www.wordpress.info/doc/overview/)
|
|
||||||
- [Tutorials](https://wordpress.com/learn/)
|
|
||||||
|
|
||||||
## Common WordPress Errors
|
|
||||||
|
|
||||||
1. **HTTP Errors**
|
|
||||||
|
|
||||||
- **404 Not Found**: Page or resource missing.
|
|
||||||
- **403 Forbidden**: Insufficient permissions to access the resource.
|
|
||||||
- **500 Internal Server Error**: Generic error, often caused by server misconfiguration or PHP errors.
|
|
||||||
- **503 Service Unavailable**: Server is overloaded or in maintenance mode.
|
|
||||||
|
|
||||||
2. **Database Errors**
|
|
||||||
|
|
||||||
- **Error Establishing a Database Connection**: Database credentials incorrect or database server is down.
|
|
||||||
- **Table Prefix Issues**: Wrong `$table_prefix` in `wp-config.php`.
|
|
||||||
- **Corrupt Database**: Can be fixed using `wp db repair`.
|
|
||||||
|
|
||||||
3. **PHP Errors**
|
|
||||||
|
|
||||||
- **Parse Error**: Syntax error in PHP files.
|
|
||||||
- **Fatal Error**: Missing function or class.
|
|
||||||
- **Deprecated Function Warnings**: Old functions being used.
|
|
||||||
|
|
||||||
## Debugging Steps
|
|
||||||
|
|
||||||
1. **Enable Debug Mode**
|
|
||||||
|
|
||||||
Edit `wp-config.php`:
|
|
||||||
|
|
||||||
```php
|
|
||||||
// Enable Debug Mode
|
|
||||||
define('WP_DEBUG', true);
|
|
||||||
// Enable Debug logging to the /wp-content/debug.log file
|
|
||||||
define('WP_DEBUG_LOG', true);
|
|
||||||
// Disable display of WordPress errors and warnings
|
|
||||||
define('WP_DEBUG_DISPLAY', false);
|
|
||||||
// Disable display of PHP errors and warnings
|
|
||||||
@ini_set('display_errors', 0);
|
|
||||||
```
|
|
||||||
|
|
||||||
Logs will be saved in `wp-content/debug.log`.
|
|
||||||
|
|
||||||
2. **Check `.htaccess`**
|
|
||||||
|
|
||||||
Ensure WordPress rules exist:
|
|
||||||
|
|
||||||
```apache
|
|
||||||
# BEGIN WordPress
|
|
||||||
|
|
||||||
<IfModule mod_rewrite.c>
|
|
||||||
RewriteEngine On
|
|
||||||
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
|
|
||||||
RewriteBase /
|
|
||||||
RewriteRule ^index\.php$ - [L]
|
|
||||||
RewriteCond %{REQUEST_FILENAME} !-f
|
|
||||||
RewriteCond %{REQUEST_FILENAME} !-d
|
|
||||||
RewriteRule . /index.php [L]
|
|
||||||
</IfModule>
|
|
||||||
|
|
||||||
# END WordPress
|
|
||||||
```
|
|
||||||
|
|
||||||
3. **Check Apache and VirtualHost config**
|
|
||||||
|
|
||||||
**Redhat and derivatives**
|
|
||||||
|
|
||||||
- `/etc/httpd/conf/httpd.conf`
|
|
||||||
- `/etc/httpd/conf.d/welcome.conf`
|
|
||||||
|
|
||||||
**Debian and derivatives**
|
|
||||||
|
|
||||||
- `/etc/apache2/apache2.conf`
|
|
||||||
- `/etc/apache2/sites-available/000-default.conf`
|
|
||||||
|
|
||||||
```apache
|
|
||||||
<Directory "/var/www">
|
|
||||||
AllowOverride All
|
|
||||||
# Allow open access:
|
|
||||||
Require all granted
|
|
||||||
</Directory>
|
|
||||||
<Directory "/var/www/html">
|
|
||||||
Options Indexes FollowSymLinks
|
|
||||||
AllowOverride All
|
|
||||||
Require all granted
|
|
||||||
</Directory>
|
|
||||||
```
|
|
||||||
|
|
||||||
4. **Reset File Permissions**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
find /var/www/html -type d -exec chmod 755 {} \;
|
|
||||||
find /var/www/html -type f -exec chmod 644 {} \;
|
|
||||||
```
|
|
||||||
|
|
||||||
4. **Disable Plugins & Themes**
|
|
||||||
|
|
||||||
Hide the plugin directory:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mv wp-content/plugins wp-content/.plugins
|
|
||||||
```
|
|
||||||
|
|
||||||
Switch to a default theme:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
wp theme install twentytwentyfive --activate
|
|
||||||
wp theme activate twentytwentyfive
|
|
||||||
```
|
|
||||||
|
|
||||||
5. **Check Server Logs**
|
|
||||||
|
|
||||||
For Apache:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
tail -f /var/log/httpd/error_log
|
|
||||||
```
|
|
||||||
|
|
||||||
For Nginx:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
tail -f /var/log/nginx/error.log
|
|
||||||
```
|
|
||||||
|
|
||||||
6. **Verify Database Connection**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
wp db check
|
|
||||||
```
|
|
||||||
|
|
||||||
If corrupt:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
wp db repair
|
|
||||||
```
|
|
||||||
|
|
||||||
7. **Increase Memory Limit**
|
|
||||||
|
|
||||||
Edit `wp-config.php`:
|
|
||||||
|
|
||||||
```php
|
|
||||||
define('WP_MEMORY_LIMIT', '256M');
|
|
||||||
```
|
|
||||||
|
|
||||||
## Debugging Tools
|
|
||||||
|
|
||||||
- **WP-CLI**: Command-line interface for WordPress.
|
|
||||||
- **Query Monitor**: Plugin for analyzing database queries and errors.
|
|
||||||
- **Health Check & Troubleshooting**: Plugin for diagnosing issues.
|
|
||||||
@ -1 +0,0 @@
|
|||||||
argcomplete
|
|
||||||
@ -1,125 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
|
|
||||||
# === Standard library ===
|
|
||||||
import sys
|
|
||||||
import os
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Optional
|
|
||||||
from argparse import ArgumentParser, Namespace
|
|
||||||
|
|
||||||
# Allow importing from scripts/library even when run directly
|
|
||||||
project_root = str(Path(__file__).resolve().parent.parent)
|
|
||||||
if project_root not in sys.path:
|
|
||||||
sys.path.append(project_root)
|
|
||||||
|
|
||||||
# === Local import ===
|
|
||||||
from scripts.library import parse_verbose, run_in_venv
|
|
||||||
|
|
||||||
|
|
||||||
def rename_by_case(root_dir: str, case: str, recursive: bool = False) -> None:
|
|
||||||
"""
|
|
||||||
Renames all files and directories in a given directory by changing their case.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
root_dir (str): Root path to start renaming.
|
|
||||||
case (str): One of "lower", "upper", "capitalize".
|
|
||||||
recursive (bool): Whether to apply renaming recursively.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
None
|
|
||||||
"""
|
|
||||||
|
|
||||||
def apply_case(name: str) -> str:
|
|
||||||
if case == "lower":
|
|
||||||
return name.lower()
|
|
||||||
elif case == "upper":
|
|
||||||
return name.upper()
|
|
||||||
elif case == "capitalize":
|
|
||||||
return name.capitalize()
|
|
||||||
else:
|
|
||||||
raise ValueError(f"Unsupported case transformation: {case}")
|
|
||||||
|
|
||||||
for current_dir, dirs, files in os.walk(root_dir, topdown=False):
|
|
||||||
# Skip descending into subdirectories if not recursive
|
|
||||||
if not recursive and current_dir != root_dir:
|
|
||||||
continue
|
|
||||||
|
|
||||||
# Rename files
|
|
||||||
for name in files:
|
|
||||||
old_path = os.path.join(current_dir, name)
|
|
||||||
new_name = apply_case(name)
|
|
||||||
new_path = os.path.join(current_dir, new_name)
|
|
||||||
if old_path != new_path:
|
|
||||||
if not os.path.exists(new_path):
|
|
||||||
os.rename(old_path, new_path)
|
|
||||||
else:
|
|
||||||
print(
|
|
||||||
f"Cannot rename {old_path} to {new_path}: target already exists."
|
|
||||||
)
|
|
||||||
|
|
||||||
# Rename directories
|
|
||||||
for name in dirs:
|
|
||||||
old_path = os.path.join(current_dir, name)
|
|
||||||
new_name = apply_case(name)
|
|
||||||
new_path = os.path.join(current_dir, new_name)
|
|
||||||
if old_path != new_path:
|
|
||||||
if not os.path.exists(new_path):
|
|
||||||
os.rename(old_path, new_path)
|
|
||||||
else:
|
|
||||||
print(
|
|
||||||
f"Cannot rename {old_path} to {new_path}: target already exists."
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def parse_args() -> Namespace:
|
|
||||||
"""
|
|
||||||
Parse and validate command-line arguments.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Namespace: Parsed arguments with 'case' and 'recursive' options.
|
|
||||||
"""
|
|
||||||
import argcomplete
|
|
||||||
|
|
||||||
parser = ArgumentParser(
|
|
||||||
description="Rename files and directories by changing case (lower, upper, capitalize)."
|
|
||||||
)
|
|
||||||
|
|
||||||
parser.add_argument(
|
|
||||||
"--case",
|
|
||||||
"-c",
|
|
||||||
type=str,
|
|
||||||
choices=["lower", "upper", "capitalize"],
|
|
||||||
default="lower",
|
|
||||||
help="Case transformation to apply: 'lower', 'upper', or 'capitalize'. Default is 'lower'.",
|
|
||||||
)
|
|
||||||
|
|
||||||
parser.add_argument(
|
|
||||||
"--recursive",
|
|
||||||
"-r",
|
|
||||||
action="store_true",
|
|
||||||
help="Apply the transformation recursively in all subdirectories.",
|
|
||||||
)
|
|
||||||
|
|
||||||
parser.add_argument(
|
|
||||||
"-v",
|
|
||||||
"--verbose",
|
|
||||||
action="store_true",
|
|
||||||
help="Enable verbose output (debug information, warnings)",
|
|
||||||
)
|
|
||||||
|
|
||||||
argcomplete.autocomplete(parser)
|
|
||||||
return parser.parse_args()
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
|
||||||
"""
|
|
||||||
Main entry point. Parses arguments and runs case renaming in current working directory.
|
|
||||||
"""
|
|
||||||
args = parse_args()
|
|
||||||
cwd: str = os.getcwd()
|
|
||||||
rename_by_case(cwd, case=args.case, recursive=args.recursive)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
args = parse_verbose()
|
|
||||||
run_in_venv(main, verbose=args.verbose)
|
|
||||||
@ -1,50 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
|
|
||||||
# Identify the currently mounted EFI partition and its device
|
|
||||||
SOURCE_MOUNT="/boot/efi"
|
|
||||||
SOURCE_PART=$(findmnt -no SOURCE "$SOURCE_MOUNT")
|
|
||||||
SOURCE_UUID=$(findmnt -no UUID "$SOURCE_MOUNT")
|
|
||||||
SOURCE_DISK=$(lsblk -no PKNAME "$SOURCE_PART")
|
|
||||||
SOURCE_DISK_PATH="/dev/$SOURCE_DISK"
|
|
||||||
|
|
||||||
# Extract the backup UUID from fstab
|
|
||||||
BACKUP_UUID=$(grep "^#UUID=" /etc/fstab | grep "/boot/efi" | awk -F'[= ]' '{print $2}')
|
|
||||||
|
|
||||||
if [ -z "$BACKUP_UUID" ]; then
|
|
||||||
echo "Error: Backup EFI UUID not found in /etc/fstab"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [ "$SOURCE_UUID" == "$BACKUP_UUID" ]; then
|
|
||||||
echo "Error: Source and Backup UUIDs are identical. Check /etc/fstab."
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Identify the physical disk for the backup UUID
|
|
||||||
BACKUP_PART=$(blkid -U "$BACKUP_UUID")
|
|
||||||
BACKUP_DISK=$(lsblk -no PKNAME "$BACKUP_PART")
|
|
||||||
BACKUP_DISK_PATH="/dev/$BACKUP_DISK"
|
|
||||||
|
|
||||||
# Define temporary mount point
|
|
||||||
TARGET_MOUNT=$(mktemp -d -t efi-sync.XXXXXXXXXX)
|
|
||||||
|
|
||||||
# Trap to ensure the directory is removed even if the script fails
|
|
||||||
trap 'rm -rf "$TARGET_MOUNT"' EXIT
|
|
||||||
|
|
||||||
# Mount, Sync, and Cleanup
|
|
||||||
if mount -U "$BACKUP_UUID" "$TARGET_MOUNT"; then
|
|
||||||
echo "Syncing EFI files: $SOURCE_UUID -> $BACKUP_UUID"
|
|
||||||
rsync -aqAX --delete "$SOURCE_MOUNT/" "$TARGET_MOUNT/"
|
|
||||||
umount "$TARGET_MOUNT"
|
|
||||||
echo "EFI Files Sync Complete."
|
|
||||||
else
|
|
||||||
echo "Error: Could not mount backup EFI ($BACKUP_UUID)"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Ensure GRUB binary is installed on BOTH physical disks
|
|
||||||
echo "Updating GRUB bootloader on both physical disks..."
|
|
||||||
grub-install "$SOURCE_DISK_PATH" --recheck > /dev/null 2>&1
|
|
||||||
grub-install "$BACKUP_DISK_PATH" --recheck > /dev/null 2>&1
|
|
||||||
|
|
||||||
echo "Redundancy Complete: Files synced and GRUB updated on $SOURCE_DISK_PATH and $BACKUP_DISK_PATH."
|
|
||||||
@ -1,20 +0,0 @@
|
|||||||
# scripts/library/__init__.py
|
|
||||||
|
|
||||||
from .tools import (
|
|
||||||
apply_resolution_rename,
|
|
||||||
deletefile,
|
|
||||||
findfreename,
|
|
||||||
get_intermediate_dirs,
|
|
||||||
get_spacer,
|
|
||||||
)
|
|
||||||
from .venv_utils import parse_verbose, run_in_venv
|
|
||||||
|
|
||||||
__all__ = [
|
|
||||||
"apply_resolution_rename",
|
|
||||||
"deletefile",
|
|
||||||
"findfreename",
|
|
||||||
"get_intermediate_dirs",
|
|
||||||
"get_spacer",
|
|
||||||
"parse_verbose",
|
|
||||||
"run_in_venv",
|
|
||||||
]
|
|
||||||
@ -1,134 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
|
|
||||||
import colorama
|
|
||||||
import os
|
|
||||||
import re
|
|
||||||
|
|
||||||
colorama.init()
|
|
||||||
|
|
||||||
creset = colorama.Fore.RESET
|
|
||||||
cgreen = colorama.Fore.GREEN
|
|
||||||
cred = colorama.Fore.RED
|
|
||||||
|
|
||||||
# Define patterns to search for various resolutions
|
|
||||||
resolution_patterns = {
|
|
||||||
2160: r"(4096x2160|3840x2160|2880p|2160p|2160|1440p|4k)",
|
|
||||||
1080: r"(1920x1080|1080p|1080)",
|
|
||||||
720: r"(1280x720|720p|720)",
|
|
||||||
}
|
|
||||||
|
|
||||||
# Wrap each pattern to only match when isolated by non-alphanumeric chars or string boundaries
|
|
||||||
resolution_patterns = {
|
|
||||||
res: rf"(?<![a-zA-Z0-9]){pat}(?![a-zA-Z0-9])"
|
|
||||||
for res, pat in resolution_patterns.items()
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def apply_resolution_rename(name, max_height):
|
|
||||||
"""
|
|
||||||
Apply resolution substitution to a filename or directory name if an isolated
|
|
||||||
resolution pattern greater than max_height is found.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
name (str): The filename or directory name to process (without path).
|
|
||||||
max_height (int): The target resolution height to substitute to.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
str: The renamed string if a match was found, or the original string unchanged.
|
|
||||||
"""
|
|
||||||
for resolution, pattern in resolution_patterns.items():
|
|
||||||
if resolution > max_height and re.search(pattern, name, re.IGNORECASE):
|
|
||||||
return re.sub(pattern, f"{max_height}p", name, flags=re.IGNORECASE)
|
|
||||||
return name
|
|
||||||
|
|
||||||
|
|
||||||
def deletefile(filepath):
|
|
||||||
"""Delete a file, Returns a boolean
|
|
||||||
|
|
||||||
Args:
|
|
||||||
filepath : A string containing the full filepath
|
|
||||||
Returns:
|
|
||||||
Bool : The success of the operation
|
|
||||||
"""
|
|
||||||
try:
|
|
||||||
os.remove(filepath)
|
|
||||||
except OSError:
|
|
||||||
print(f"{cred}Error deleting {filepath}{creset}")
|
|
||||||
return False
|
|
||||||
|
|
||||||
print(f"{cgreen}Successfully deleted {filepath}{creset}")
|
|
||||||
return True
|
|
||||||
|
|
||||||
|
|
||||||
def get_intermediate_dirs(input_path, file_path):
|
|
||||||
"""
|
|
||||||
Return the list of directory paths between input_path (exclusive) and
|
|
||||||
the directory containing file_path (inclusive), ordered deepest first.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
input_path (str): The root path (exclusive upper bound).
|
|
||||||
file_path (str): The full path to a file.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
list: List of directory paths, deepest first.
|
|
||||||
|
|
||||||
Raises:
|
|
||||||
ValueError: If file_path not under input_path.
|
|
||||||
"""
|
|
||||||
input_path = os.path.realpath(input_path)
|
|
||||||
current = os.path.realpath(os.path.dirname(file_path))
|
|
||||||
|
|
||||||
if current != input_path and not current.startswith(input_path + os.sep):
|
|
||||||
raise ValueError(
|
|
||||||
f"file_path '{file_path}' is not under input_path '{input_path}'"
|
|
||||||
)
|
|
||||||
|
|
||||||
dirs = []
|
|
||||||
|
|
||||||
while current != input_path and current != os.path.dirname(current):
|
|
||||||
dirs.append(current)
|
|
||||||
current = os.path.dirname(current)
|
|
||||||
|
|
||||||
return dirs
|
|
||||||
|
|
||||||
|
|
||||||
def findfreename(filepath, attempt=0):
|
|
||||||
"""
|
|
||||||
Given a filepath, it will try to find a free filename by appending a number to the name.
|
|
||||||
First, it tries using the filepath passed in the argument, then adds a number to the end. If all fail, it appends (#).
|
|
||||||
|
|
||||||
Args:
|
|
||||||
filepath (str): A string containing the full filepath.
|
|
||||||
attempt (int): The number of attempts already made to find a free filename.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
str: The first free filepath found.
|
|
||||||
"""
|
|
||||||
attempt += 1
|
|
||||||
filename = str(filepath)[: str(filepath).rindex(".")]
|
|
||||||
extension = str(filepath)[str(filepath).rindex(".") :]
|
|
||||||
copynumpath = filename + f"({attempt})" + extension
|
|
||||||
|
|
||||||
if not os.path.exists(filepath) and attempt <= 2:
|
|
||||||
return filepath
|
|
||||||
elif not os.path.exists(copynumpath):
|
|
||||||
return copynumpath
|
|
||||||
return findfreename(filepath, attempt)
|
|
||||||
|
|
||||||
|
|
||||||
def get_spacer(name: str) -> str:
|
|
||||||
"""
|
|
||||||
Determine the spacer character ('-' or '_') to use for naming a virtual environment
|
|
||||||
based on the given project name. Default to underscore.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
name (str): The project name to analyze.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
str: The detected spacer ('-' or '_'), or '_' if none found.
|
|
||||||
"""
|
|
||||||
if "-" in name:
|
|
||||||
return "-"
|
|
||||||
elif "_" in name:
|
|
||||||
return "_"
|
|
||||||
return "_"
|
|
||||||
@ -1,149 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
|
|
||||||
# === Standard library ===
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
import subprocess
|
|
||||||
import venv
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Callable
|
|
||||||
from argparse import ArgumentParser, Namespace
|
|
||||||
|
|
||||||
# === Configuration ===
|
|
||||||
# Define the virtual environment directory in the user's home folder
|
|
||||||
VENV_DIR: Path = Path.home() / ".scripts_fabq_venv"
|
|
||||||
|
|
||||||
# Define the path to the requirements.txt file in the project root
|
|
||||||
REQUIREMENTS_FILE: Path = Path(__file__).resolve().parents[2] / "requirements.txt"
|
|
||||||
|
|
||||||
|
|
||||||
def parse_verbose() -> Namespace:
|
|
||||||
"""
|
|
||||||
Parse the --verbose flag before virtual environment setup.
|
|
||||||
|
|
||||||
This is useful when argument parsing requires packages that may
|
|
||||||
only be available after the virtual environment is initialized.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Namespace: Contains a single attribute 'verbose' (bool).
|
|
||||||
"""
|
|
||||||
parser = ArgumentParser(add_help=False)
|
|
||||||
|
|
||||||
parser.add_argument(
|
|
||||||
"-v",
|
|
||||||
"--verbose",
|
|
||||||
action="count",
|
|
||||||
default=0,
|
|
||||||
help="Increase verbosity: -v = verbose, -vv = debug, -vvv = trace",
|
|
||||||
)
|
|
||||||
|
|
||||||
parser.add_argument(
|
|
||||||
"--debug",
|
|
||||||
dest="verbose",
|
|
||||||
action="store_const",
|
|
||||||
const=2,
|
|
||||||
help="Enable debug output (equivalent to -vv)",
|
|
||||||
)
|
|
||||||
|
|
||||||
parser.add_argument(
|
|
||||||
"--trace",
|
|
||||||
dest="verbose",
|
|
||||||
action="store_const",
|
|
||||||
const=3,
|
|
||||||
help="Enable full trace output (equivalent to -vvv)",
|
|
||||||
)
|
|
||||||
|
|
||||||
return parser.parse_known_args()[0]
|
|
||||||
|
|
||||||
|
|
||||||
def is_in_venv() -> bool:
|
|
||||||
"""
|
|
||||||
Determine whether the script is currently running inside a virtual environment.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
bool: True if running inside a virtual environment, False otherwise.
|
|
||||||
"""
|
|
||||||
return hasattr(sys, "real_prefix") or sys.prefix != getattr(
|
|
||||||
sys, "base_prefix", sys.prefix
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def create_venv(verbose: bool = False) -> None:
|
|
||||||
"""
|
|
||||||
Create a virtual environment in the predefined VENV_DIR location.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
verbose (bool): If True, prints progress messages.
|
|
||||||
"""
|
|
||||||
if verbose >= 2:
|
|
||||||
print(f"Creating virtual environment in {VENV_DIR}...")
|
|
||||||
venv.create(VENV_DIR, with_pip=True)
|
|
||||||
|
|
||||||
|
|
||||||
def install_requirements(verbose: bool = False) -> None:
|
|
||||||
"""
|
|
||||||
Install Python dependencies from requirements.txt into the venv.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
verbose (bool): If True, prints progress and pip output.
|
|
||||||
"""
|
|
||||||
if not REQUIREMENTS_FILE.exists():
|
|
||||||
print(f"Error: requirements.txt not found at {REQUIREMENTS_FILE}")
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
if verbose >= 2:
|
|
||||||
print(f"Installing/updating requirements from {REQUIREMENTS_FILE}...")
|
|
||||||
|
|
||||||
pip_cmd = [str(VENV_DIR / "bin" / "pip")]
|
|
||||||
|
|
||||||
try:
|
|
||||||
subprocess.check_call(
|
|
||||||
pip_cmd + ["install", "-r", str(REQUIREMENTS_FILE)],
|
|
||||||
stdout=None if verbose >= 2 else subprocess.DEVNULL,
|
|
||||||
stderr=None if verbose >= 2 else subprocess.DEVNULL,
|
|
||||||
)
|
|
||||||
except subprocess.CalledProcessError as e:
|
|
||||||
print(f"Error: Failed to install requirements: {e}")
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
try:
|
|
||||||
subprocess.check_call(
|
|
||||||
pip_cmd + ["check"],
|
|
||||||
stdout=None if verbose >= 2 else subprocess.DEVNULL,
|
|
||||||
stderr=None if verbose >= 2 else subprocess.DEVNULL,
|
|
||||||
)
|
|
||||||
except subprocess.CalledProcessError:
|
|
||||||
print("Error: Dependency conflict detected — your environment may be broken.")
|
|
||||||
raise
|
|
||||||
|
|
||||||
|
|
||||||
def run_in_venv(main_func: Callable[[], None], verbose: bool = False) -> None:
|
|
||||||
"""
|
|
||||||
Ensure the script is running inside the expected virtual environment.
|
|
||||||
|
|
||||||
If not already in the venv:
|
|
||||||
- Create it if needed
|
|
||||||
- Install requirements
|
|
||||||
- Relaunch the script inside the venv
|
|
||||||
|
|
||||||
If already in the venv:
|
|
||||||
- Ensure requirements are installed
|
|
||||||
- Execute the given main function
|
|
||||||
|
|
||||||
Args:
|
|
||||||
main_func (Callable[[], None]): The function to execute after setup.
|
|
||||||
verbose (bool): If True, enables progress messages and pip output.
|
|
||||||
"""
|
|
||||||
if not is_in_venv():
|
|
||||||
if not VENV_DIR.exists():
|
|
||||||
create_venv(verbose)
|
|
||||||
install_requirements(verbose)
|
|
||||||
|
|
||||||
# Relaunch the script inside the virtual environment
|
|
||||||
os.execv(
|
|
||||||
str(VENV_DIR / "bin" / "python"),
|
|
||||||
[str(VENV_DIR / "bin" / "python")] + sys.argv,
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
install_requirements(verbose)
|
|
||||||
main_func()
|
|
||||||
@ -1,662 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
# video_autoreduce.py
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import argcomplete
|
|
||||||
import colorama
|
|
||||||
import os
|
|
||||||
import re
|
|
||||||
import subprocess
|
|
||||||
import sys
|
|
||||||
import json
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
# Allow importing from scripts/library even when run directly
|
|
||||||
project_root = str(Path(__file__).resolve().parent.parent)
|
|
||||||
if project_root not in sys.path:
|
|
||||||
sys.path.append(project_root)
|
|
||||||
|
|
||||||
# === Local import ===
|
|
||||||
from scripts.library import (
|
|
||||||
deletefile,
|
|
||||||
findfreename,
|
|
||||||
apply_resolution_rename,
|
|
||||||
get_intermediate_dirs,
|
|
||||||
)
|
|
||||||
|
|
||||||
colorama.init()
|
|
||||||
|
|
||||||
creset = colorama.Fore.RESET
|
|
||||||
ccyan = colorama.Fore.CYAN
|
|
||||||
cyellow = colorama.Fore.YELLOW
|
|
||||||
cgreen = colorama.Fore.GREEN
|
|
||||||
cred = colorama.Fore.RED
|
|
||||||
|
|
||||||
VIDEO_EXTENSIONS = (".mp4", ".avi", ".mkv", ".mov", ".wmv", ".flv", ".divx")
|
|
||||||
|
|
||||||
|
|
||||||
def get_ffmpeg_codecs(codec_type="S"):
|
|
||||||
"""
|
|
||||||
Retrieve a list of supported codecs for a specific type from ffmpeg.
|
|
||||||
|
|
||||||
This function runs the `ffmpeg -codecs` command and parses its output to extract
|
|
||||||
the list of supported codecs based on the provided `codec_type` (e.g., 'S' for subtitles,
|
|
||||||
'V' for video, 'A' for audio). The list of codecs is returned as a set for efficient lookup.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
codec_type (str): The type of codecs to retrieve.
|
|
||||||
- "S" for subtitles (default)
|
|
||||||
- "V" for video
|
|
||||||
- "A" for audio
|
|
||||||
- "D" for data
|
|
||||||
- "T" for attachment
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
set: A set of codec names of the specified type.
|
|
||||||
If an error occurs or no codecs are found, an empty set is returned.
|
|
||||||
"""
|
|
||||||
try:
|
|
||||||
# Run ffmpeg -codecs
|
|
||||||
result = subprocess.run(
|
|
||||||
["ffmpeg", "-codecs"], capture_output=True, text=True, check=True
|
|
||||||
)
|
|
||||||
output = result.stdout
|
|
||||||
|
|
||||||
# Extract codecs using regex
|
|
||||||
codec_pattern = re.compile(rf"^[ D.E]+{codec_type}[ A-Z.]+ (\S+)", re.MULTILINE)
|
|
||||||
codecs = codec_pattern.findall(output)
|
|
||||||
|
|
||||||
return set(codecs) # Return as a set for easy lookup
|
|
||||||
|
|
||||||
except subprocess.CalledProcessError as e:
|
|
||||||
print("Error running ffmpeg:", e)
|
|
||||||
return set()
|
|
||||||
|
|
||||||
|
|
||||||
def user_confirm(message, color="yellow"):
|
|
||||||
"""
|
|
||||||
Prompt the user for a yes/no confirmation.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
message (str): The confirmation message to display.
|
|
||||||
color (str): Color for the prompt ('yellow', 'red', 'green'). Default is 'yellow'.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
bool: True if the user confirms with 'Y', False otherwise.
|
|
||||||
"""
|
|
||||||
color_map = {
|
|
||||||
"yellow": cyellow,
|
|
||||||
"red": cred,
|
|
||||||
"green": cgreen,
|
|
||||||
}
|
|
||||||
c = color_map.get(color, cyellow)
|
|
||||||
response = input(f"{c}{message}{creset} ").strip().upper()
|
|
||||||
return response == "Y"
|
|
||||||
|
|
||||||
|
|
||||||
def ensure_output_path(output_path):
|
|
||||||
"""
|
|
||||||
Ensure that the output directory exists. If it doesn't, attempt to create it.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
output_path (str): The path of the output directory to be ensured.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
None
|
|
||||||
"""
|
|
||||||
try:
|
|
||||||
os.makedirs(output_path, exist_ok=True)
|
|
||||||
# Attempts to create the output directory if it doesn't exist.
|
|
||||||
# exist_ok=True ensures that an OSError is not raised if the directory already exists.
|
|
||||||
except OSError as e:
|
|
||||||
# If an OSError occurs during directory creation, print an error message and exit the program.
|
|
||||||
print(f"{cred}Failed to create output directory:{creset} {e}", file=sys.stderr)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
|
|
||||||
def has_supported_subs(video_file, supported_subtitle_codecs, debug=False):
|
|
||||||
"""
|
|
||||||
Check if the given video file contains subtitles in a supported codec.
|
|
||||||
|
|
||||||
This function uses `ffprobe` to extract subtitle stream information from the
|
|
||||||
video file and checks whether any subtitle stream has a codec that matches
|
|
||||||
one of the supported codecs provided in the `supported_subtitle_codecs` list.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
video_file (str): Path to the video file to be checked.
|
|
||||||
supported_subtitle_codecs (set): Set of subtitle codec names that are considered supported.
|
|
||||||
debug (bool): If True, prints debug information for errors. Defaults to False.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
bool: True if at least one subtitle stream is found with a supported codec,
|
|
||||||
False otherwise or in case of an error.
|
|
||||||
"""
|
|
||||||
try:
|
|
||||||
result = subprocess.run(
|
|
||||||
[
|
|
||||||
"ffprobe",
|
|
||||||
"-v",
|
|
||||||
"error",
|
|
||||||
"-select_streams",
|
|
||||||
"s",
|
|
||||||
"-show_entries",
|
|
||||||
"stream=index,codec_name",
|
|
||||||
"-of",
|
|
||||||
"json",
|
|
||||||
video_file,
|
|
||||||
],
|
|
||||||
stdout=subprocess.PIPE,
|
|
||||||
stderr=subprocess.PIPE,
|
|
||||||
text=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
if result.returncode != 0:
|
|
||||||
if debug:
|
|
||||||
print(f"ffprobe error: {result.stderr.strip()}")
|
|
||||||
return False # Assume no supported subtitles on failure
|
|
||||||
|
|
||||||
try:
|
|
||||||
streams = json.loads(result.stdout).get("streams", [])
|
|
||||||
except json.JSONDecodeError as e:
|
|
||||||
if debug:
|
|
||||||
print(f"JSON parsing error: {e}")
|
|
||||||
return False # Assume no supported subtitles if JSON is malformed
|
|
||||||
|
|
||||||
return any(
|
|
||||||
stream.get("codec_name") in supported_subtitle_codecs for stream in streams
|
|
||||||
)
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
if debug:
|
|
||||||
print(f"Unexpected error: {e}")
|
|
||||||
return False # Assume no supported subtitles
|
|
||||||
|
|
||||||
|
|
||||||
def find_videos_to_convert(input_path, max_height=720, debug=False):
|
|
||||||
"""
|
|
||||||
Find video files in the specified directory tree that meet conversion criteria.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
input_path (str): The path of the directory to search for video files.
|
|
||||||
max_height (int, optional): The maximum height (in pixels) of the video to consider for conversion. Default is 720.
|
|
||||||
debug (bool, optional): If True, print debug messages. Default is False.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
list: A list of paths to video files that meet the conversion criteria.
|
|
||||||
"""
|
|
||||||
# Print a message indicating the start of the scanning process
|
|
||||||
print(f"{cgreen}Scanning {input_path} for video files to convert.{creset}\n")
|
|
||||||
|
|
||||||
# Initialize lists to store all video files and valid video files
|
|
||||||
video_files = []
|
|
||||||
valid_videos_files = []
|
|
||||||
|
|
||||||
# Find all video files in the directory tree
|
|
||||||
for root, dirs, files in os.walk(input_path):
|
|
||||||
for file in files:
|
|
||||||
fullpath = os.path.join(root, file)
|
|
||||||
# Check if the file is a video file with one of the specified extensions
|
|
||||||
if file.lower().endswith(VIDEO_EXTENSIONS) and os.path.isfile(fullpath):
|
|
||||||
video_files.append(fullpath)
|
|
||||||
|
|
||||||
# List of encodings to try for decoding output from ffprobe
|
|
||||||
ENCODINGS_TO_TRY = ["utf-8", "latin-1", "iso-8859-1"]
|
|
||||||
|
|
||||||
# Iterate through each video file
|
|
||||||
for video_file in video_files:
|
|
||||||
try:
|
|
||||||
# Run ffprobe command to get video resolution
|
|
||||||
resolution_output = subprocess.check_output(
|
|
||||||
[
|
|
||||||
"ffprobe",
|
|
||||||
"-v",
|
|
||||||
"error",
|
|
||||||
"-select_streams",
|
|
||||||
"v:0",
|
|
||||||
"-show_entries",
|
|
||||||
"stream=width,height",
|
|
||||||
"-of",
|
|
||||||
"csv=p=0",
|
|
||||||
video_file,
|
|
||||||
],
|
|
||||||
stderr=subprocess.STDOUT,
|
|
||||||
)
|
|
||||||
|
|
||||||
# Attempt to decode the output using different encodings
|
|
||||||
dimensions_str = None
|
|
||||||
width = height = None
|
|
||||||
for encoding in ENCODINGS_TO_TRY:
|
|
||||||
try:
|
|
||||||
dimensions_str = (
|
|
||||||
resolution_output.decode(encoding).strip().rstrip(",")
|
|
||||||
)
|
|
||||||
|
|
||||||
dimensions = [
|
|
||||||
int(d) for d in dimensions_str.split(",") if d.strip().isdigit()
|
|
||||||
]
|
|
||||||
if len(dimensions) >= 2:
|
|
||||||
width, height = dimensions[:2]
|
|
||||||
else:
|
|
||||||
raise ValueError(
|
|
||||||
f"Unexpected dimensions format: {dimensions_str}"
|
|
||||||
)
|
|
||||||
break # Stop trying encodings once successful decoding and conversion
|
|
||||||
except (UnicodeDecodeError, ValueError):
|
|
||||||
continue # Try the next encoding
|
|
||||||
|
|
||||||
# If decoding was unsuccessful with all encodings, skip this video
|
|
||||||
if width is None or height is None:
|
|
||||||
if debug:
|
|
||||||
print(
|
|
||||||
f"{cred}Error processing {video_file}: Unable to decode dimensions from ffprobe output.{creset}",
|
|
||||||
file=sys.stderr,
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
|
|
||||||
# If decoding was unsuccessful with all encodings, skip this video
|
|
||||||
if dimensions_str is None:
|
|
||||||
if debug:
|
|
||||||
print(
|
|
||||||
f"{cred}Error processing {video_file}: Unable to decode output from ffprobe using any of the specified encodings.{creset}",
|
|
||||||
file=sys.stderr,
|
|
||||||
)
|
|
||||||
continue # Skip this video and continue with the next one
|
|
||||||
|
|
||||||
# Ignore vertical videos
|
|
||||||
if height > width:
|
|
||||||
continue # Skip this video and continue with the next one
|
|
||||||
|
|
||||||
# Check if the video height exceeds the maximum allowed height
|
|
||||||
if height > max_height:
|
|
||||||
valid_videos_files.append(video_file)
|
|
||||||
|
|
||||||
except subprocess.CalledProcessError as e:
|
|
||||||
if debug:
|
|
||||||
# Print error message if subprocess call fails
|
|
||||||
if e.output is not None:
|
|
||||||
decoded_error = e.output.decode("utf-8")
|
|
||||||
print(
|
|
||||||
f"{cred}Error processing {video_file}: {decoded_error}{creset}\n",
|
|
||||||
file=sys.stderr,
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
print(
|
|
||||||
f"{cred}Error processing {video_file}: No output from subprocess{creset}\n",
|
|
||||||
file=sys.stderr,
|
|
||||||
)
|
|
||||||
except ValueError as e:
|
|
||||||
if debug:
|
|
||||||
# Print error message if value error occurs during decoding
|
|
||||||
print(
|
|
||||||
f"{cred}Error processing {video_file}: ValueError decoding the file{creset}\n",
|
|
||||||
file=sys.stderr,
|
|
||||||
)
|
|
||||||
|
|
||||||
valid_videos_files.sort()
|
|
||||||
|
|
||||||
return valid_videos_files
|
|
||||||
|
|
||||||
|
|
||||||
def convert_videos(
|
|
||||||
input_path,
|
|
||||||
output_path=None,
|
|
||||||
max_height=720,
|
|
||||||
delete=False,
|
|
||||||
rename=False,
|
|
||||||
debug=False,
|
|
||||||
):
|
|
||||||
"""
|
|
||||||
Convert videos taller than a specified height to 720p resolution with x265 encoding and MKV container.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
input_path (str): The path of the directory containing input video files.
|
|
||||||
output_path (str, optional): The path of the directory where converted video files will be saved.
|
|
||||||
If None, output files are written alongside the originals (in-place).
|
|
||||||
max_height (int, optional): The maximum height (in pixels) of the video to consider for conversion. Default is 720.
|
|
||||||
delete (bool, optional): If True, delete the original file after successful conversion. Default is False.
|
|
||||||
rename (bool, optional): If True, rename files and intermediate folders to reflect the target resolution. Default is False.
|
|
||||||
debug (bool, optional): If True, print debug messages. Default is False.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
bool: True if conversion succeeds, False otherwise.
|
|
||||||
"""
|
|
||||||
|
|
||||||
# Get supported subtitle codecs
|
|
||||||
supported_subtitle_codecs = get_ffmpeg_codecs("S")
|
|
||||||
|
|
||||||
# Find video files to convert in the input directory
|
|
||||||
video_files = find_videos_to_convert(input_path, max_height, debug)
|
|
||||||
counter = 0
|
|
||||||
|
|
||||||
# If no video files found, print a message and return False
|
|
||||||
if len(video_files) == 0:
|
|
||||||
print(f"No video files to convert found.\n")
|
|
||||||
return False
|
|
||||||
|
|
||||||
# Ensure the fixed output directory exists (only when output_path is set)
|
|
||||||
if output_path:
|
|
||||||
ensure_output_path(output_path)
|
|
||||||
|
|
||||||
# Print a message indicating the start of the conversion process
|
|
||||||
print(
|
|
||||||
f"Converting {len(video_files)} videos taller than {max_height} pixels to {max_height}p resolution with x265 encoding and MKV container...\n"
|
|
||||||
)
|
|
||||||
|
|
||||||
# Variable to keep a track of the current_file in case of failure
|
|
||||||
current_file = None
|
|
||||||
|
|
||||||
# Track successfully converted original paths (used for folder rename pass)
|
|
||||||
converted_files = set()
|
|
||||||
|
|
||||||
# Iterate through each video file for conversion
|
|
||||||
for video_file in video_files:
|
|
||||||
counter += 1
|
|
||||||
|
|
||||||
# Print conversion progress information
|
|
||||||
print(
|
|
||||||
f"{cgreen}****** Starting conversion {counter} of {len(video_files)}: '{os.path.basename(video_file)}'...{creset}"
|
|
||||||
)
|
|
||||||
print(f"{ccyan}Original file:{creset}")
|
|
||||||
print(video_file)
|
|
||||||
|
|
||||||
# Determine output directory: fixed path, or same dir as the source file
|
|
||||||
effective_output_path = (
|
|
||||||
output_path if output_path else os.path.dirname(video_file)
|
|
||||||
)
|
|
||||||
|
|
||||||
# Compute base output filename, applying resolution rename if requested
|
|
||||||
base_name = os.path.splitext(os.path.basename(video_file))[0]
|
|
||||||
if rename:
|
|
||||||
renamed_base = apply_resolution_rename(base_name, max_height)
|
|
||||||
if renamed_base == base_name and f"{max_height}p" not in renamed_base:
|
|
||||||
# No resolution found in filename: append .{max_height}p before extension
|
|
||||||
renamed_base = f"{base_name}.{max_height}p"
|
|
||||||
base_name = renamed_base
|
|
||||||
|
|
||||||
# Generate output file path, using findfreename to avoid conflicts
|
|
||||||
output_file = findfreename(
|
|
||||||
os.path.join(effective_output_path, base_name + ".mkv")
|
|
||||||
)
|
|
||||||
|
|
||||||
try:
|
|
||||||
# Get original video width and height
|
|
||||||
video_info = subprocess.run(
|
|
||||||
[
|
|
||||||
"ffprobe",
|
|
||||||
"-v",
|
|
||||||
"error",
|
|
||||||
"-select_streams",
|
|
||||||
"v:0",
|
|
||||||
"-show_entries",
|
|
||||||
"stream=width,height",
|
|
||||||
"-of",
|
|
||||||
"csv=p=0",
|
|
||||||
video_file,
|
|
||||||
],
|
|
||||||
stdout=subprocess.PIPE,
|
|
||||||
stderr=subprocess.PIPE,
|
|
||||||
text=True,
|
|
||||||
)
|
|
||||||
width, height = map(int, video_info.stdout.strip().rstrip(",").split(","))
|
|
||||||
|
|
||||||
# Calculate the scaled width maintaining aspect ratio
|
|
||||||
scaled_width = int(max_height * (width / height))
|
|
||||||
# Ensure the width is even
|
|
||||||
if scaled_width % 2 != 0:
|
|
||||||
scaled_width += 1
|
|
||||||
|
|
||||||
# Keep a track of the current_file in case of failure
|
|
||||||
current_file = output_file
|
|
||||||
|
|
||||||
# Check if the video has supported subtitles
|
|
||||||
include_subs = has_supported_subs(
|
|
||||||
video_file, supported_subtitle_codecs, debug
|
|
||||||
)
|
|
||||||
|
|
||||||
# Construct the FFmpeg command dynamically
|
|
||||||
ffmpeg_command = [
|
|
||||||
"ffmpeg",
|
|
||||||
"-n",
|
|
||||||
"-i",
|
|
||||||
video_file,
|
|
||||||
"-map",
|
|
||||||
"0:v",
|
|
||||||
"-c:v",
|
|
||||||
"libx265",
|
|
||||||
"-vf",
|
|
||||||
f"scale={scaled_width}:{max_height}",
|
|
||||||
"-preset",
|
|
||||||
"medium",
|
|
||||||
"-crf",
|
|
||||||
"28",
|
|
||||||
"-map",
|
|
||||||
"0:a?",
|
|
||||||
"-c:a",
|
|
||||||
"copy",
|
|
||||||
]
|
|
||||||
|
|
||||||
# Only include subtitles if they are supported
|
|
||||||
if include_subs:
|
|
||||||
ffmpeg_command.extend(["-map", "0:s?", "-c:s", "copy"])
|
|
||||||
|
|
||||||
ffmpeg_command.append(output_file)
|
|
||||||
|
|
||||||
# Print the FFmpeg command as a single string
|
|
||||||
if debug:
|
|
||||||
print("FFmpeg command: " + " ".join(ffmpeg_command))
|
|
||||||
|
|
||||||
# Run the FFmpeg command
|
|
||||||
process = subprocess.run(
|
|
||||||
ffmpeg_command,
|
|
||||||
check=True,
|
|
||||||
stdout=subprocess.DEVNULL,
|
|
||||||
stderr=subprocess.STDOUT,
|
|
||||||
)
|
|
||||||
|
|
||||||
# Check if ffmpeg process returned successfully
|
|
||||||
if process.returncode != 0:
|
|
||||||
raise subprocess.CalledProcessError(
|
|
||||||
process.returncode, process.args, output=process.stderr
|
|
||||||
)
|
|
||||||
|
|
||||||
except subprocess.CalledProcessError as e:
|
|
||||||
# If we have a partial video converted, delete it due to the failure
|
|
||||||
if current_file:
|
|
||||||
deletefile(current_file)
|
|
||||||
|
|
||||||
# Handle subprocess errors
|
|
||||||
if debug:
|
|
||||||
if e.output is not None:
|
|
||||||
decoded_error = e.output.decode("utf-8")
|
|
||||||
print(
|
|
||||||
f"{cred}Error processing {video_file}: {decoded_error}{creset}\n",
|
|
||||||
file=sys.stderr,
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
print(
|
|
||||||
f"{cred}Error processing {video_file}: No output from subprocess{creset}\n",
|
|
||||||
file=sys.stderr,
|
|
||||||
)
|
|
||||||
|
|
||||||
except KeyboardInterrupt:
|
|
||||||
print(f"{cyellow}Conversions cancelled, cleaning up...{creset}")
|
|
||||||
|
|
||||||
# If we have a partial video converted, delete it due to the failure
|
|
||||||
if current_file:
|
|
||||||
deletefile(current_file)
|
|
||||||
current_file = None
|
|
||||||
|
|
||||||
exit()
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
# If we have a partial video converted, delete it due to the failure
|
|
||||||
if current_file:
|
|
||||||
deletefile(current_file)
|
|
||||||
|
|
||||||
# Handle other exceptions
|
|
||||||
if debug:
|
|
||||||
print(
|
|
||||||
f"{cred}Error processing {video_file}: {str(e)}{creset}\n",
|
|
||||||
file=sys.stderr,
|
|
||||||
)
|
|
||||||
|
|
||||||
else:
|
|
||||||
# Print success message if conversion is successful
|
|
||||||
print(f"{ccyan}Successfully converted video to output file:{creset}")
|
|
||||||
print(f"{output_file}\n")
|
|
||||||
|
|
||||||
try:
|
|
||||||
os.chmod(output_file, 0o777)
|
|
||||||
except PermissionError:
|
|
||||||
print(f"{cred}PermissionError on: '{output_file}'{creset}")
|
|
||||||
|
|
||||||
# Delete the original file if requested
|
|
||||||
if delete:
|
|
||||||
deletefile(video_file)
|
|
||||||
|
|
||||||
# Track this original and its converted output as successfully converted
|
|
||||||
converted_files.add(os.path.realpath(video_file))
|
|
||||||
converted_files.add(os.path.realpath(output_file))
|
|
||||||
|
|
||||||
# Remove the now successfully converted filepath
|
|
||||||
current_file = None
|
|
||||||
|
|
||||||
# Folder rename second pass
|
|
||||||
if rename and converted_files:
|
|
||||||
# Collect all intermediate dirs touched by successful conversions, deepest first
|
|
||||||
dirs_to_consider = {}
|
|
||||||
for original_file in converted_files:
|
|
||||||
for d in get_intermediate_dirs(input_path, original_file):
|
|
||||||
dirs_to_consider[d] = True
|
|
||||||
|
|
||||||
# For each candidate dir, check that ALL video files under it were converted
|
|
||||||
dirs_to_rename = []
|
|
||||||
for d in sorted(dirs_to_consider, key=len, reverse=True):
|
|
||||||
all_converted = True
|
|
||||||
for root, _, files in os.walk(d):
|
|
||||||
for f in files:
|
|
||||||
if f.lower().endswith(VIDEO_EXTENSIONS):
|
|
||||||
fpath = os.path.realpath(os.path.join(root, f))
|
|
||||||
if fpath not in converted_files:
|
|
||||||
all_converted = False
|
|
||||||
break
|
|
||||||
if not all_converted:
|
|
||||||
break
|
|
||||||
|
|
||||||
if all_converted:
|
|
||||||
new_dirname = apply_resolution_rename(os.path.basename(d), max_height)
|
|
||||||
if new_dirname != os.path.basename(d):
|
|
||||||
new_dir = os.path.join(os.path.dirname(d), new_dirname)
|
|
||||||
dirs_to_rename.append((d, new_dir))
|
|
||||||
|
|
||||||
# Rename dirs deepest first to avoid breaking paths
|
|
||||||
for old_dir, new_dir in dirs_to_rename:
|
|
||||||
if os.path.exists(new_dir):
|
|
||||||
print(
|
|
||||||
f"{cred}Error: Target directory {new_dir} already exists, skipping rename.{creset}"
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
os.rename(old_dir, new_dir)
|
|
||||||
if debug:
|
|
||||||
print(f"Renamed directory: {old_dir} -> {new_dir}")
|
|
||||||
|
|
||||||
return True
|
|
||||||
|
|
||||||
|
|
||||||
def parse_args():
|
|
||||||
"""
|
|
||||||
Parse command line arguments.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
argparse.Namespace: Parsed arguments.
|
|
||||||
"""
|
|
||||||
|
|
||||||
parser = argparse.ArgumentParser(
|
|
||||||
description="Convert videos taller than a certain height to 720p resolution with x265 encoding and MKV container."
|
|
||||||
)
|
|
||||||
|
|
||||||
parser.add_argument(
|
|
||||||
"input_path",
|
|
||||||
nargs="?",
|
|
||||||
default=os.getcwd(),
|
|
||||||
help="directory path to search for video files (default: current directory)",
|
|
||||||
)
|
|
||||||
parser.add_argument(
|
|
||||||
"-o",
|
|
||||||
"--output-path",
|
|
||||||
default=None,
|
|
||||||
help="directory path to save converted videos (default: ./converted, or alongside originals with --delete)",
|
|
||||||
)
|
|
||||||
parser.add_argument(
|
|
||||||
"-mh",
|
|
||||||
"--max-height",
|
|
||||||
type=int,
|
|
||||||
default=720,
|
|
||||||
help="maximum height of videos to be converted (default: 720)",
|
|
||||||
)
|
|
||||||
parser.add_argument(
|
|
||||||
"-del",
|
|
||||||
"--delete",
|
|
||||||
action="store_true",
|
|
||||||
help="delete original files after successful conversion",
|
|
||||||
)
|
|
||||||
parser.add_argument(
|
|
||||||
"-y",
|
|
||||||
"--yes",
|
|
||||||
action="store_true",
|
|
||||||
help="skip confirmation prompts",
|
|
||||||
)
|
|
||||||
parser.add_argument(
|
|
||||||
"-r",
|
|
||||||
"--rename",
|
|
||||||
action="store_true",
|
|
||||||
help="rename converted files and intermediate folders to reflect the target resolution",
|
|
||||||
)
|
|
||||||
parser.add_argument(
|
|
||||||
"--debug",
|
|
||||||
action="store_true",
|
|
||||||
help="enable debug mode for printing additional error messages",
|
|
||||||
)
|
|
||||||
|
|
||||||
# Enable autocomplete for argparse
|
|
||||||
argcomplete.autocomplete(parser)
|
|
||||||
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
if args.output_path is None and not args.delete:
|
|
||||||
args.output_path = os.path.join(os.getcwd(), "converted")
|
|
||||||
|
|
||||||
# Warn and confirm if --delete is active
|
|
||||||
if args.delete:
|
|
||||||
print(f"{cyellow}WARNING: Delete option selected!{creset}")
|
|
||||||
if not args.yes and not user_confirm(
|
|
||||||
"Are you sure you wish to delete original files after successful conversion? [Y/N] ?",
|
|
||||||
color="yellow",
|
|
||||||
):
|
|
||||||
print(f"{cgreen}Exiting!{creset}")
|
|
||||||
exit()
|
|
||||||
|
|
||||||
return args
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
"""
|
|
||||||
Main function to parse command line arguments and initiate video conversion.
|
|
||||||
"""
|
|
||||||
|
|
||||||
args = parse_args()
|
|
||||||
|
|
||||||
# Convert videos
|
|
||||||
convert_videos(
|
|
||||||
args.input_path,
|
|
||||||
args.output_path,
|
|
||||||
args.max_height,
|
|
||||||
args.delete,
|
|
||||||
args.rename,
|
|
||||||
args.debug,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
# Execute main function when the script is run directly
|
|
||||||
main()
|
|
||||||
@ -1,120 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
# video_autoreduce_rename.py
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import argcomplete
|
|
||||||
import colorama
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
# Allow importing from scripts/library even when run directly
|
|
||||||
project_root = str(Path(__file__).resolve().parent.parent)
|
|
||||||
if project_root not in sys.path:
|
|
||||||
sys.path.append(project_root)
|
|
||||||
|
|
||||||
# === Local import ===
|
|
||||||
from scripts.library import apply_resolution_rename
|
|
||||||
|
|
||||||
colorama.init()
|
|
||||||
|
|
||||||
creset = colorama.Fore.RESET
|
|
||||||
ccyan = colorama.Fore.CYAN
|
|
||||||
cyellow = colorama.Fore.YELLOW
|
|
||||||
cgreen = colorama.Fore.GREEN
|
|
||||||
cred = colorama.Fore.RED
|
|
||||||
|
|
||||||
|
|
||||||
def autorename(input_path, max_height=720, debug=False):
|
|
||||||
"""
|
|
||||||
Rename video files and folders in the specified directory tree that contain a resolution
|
|
||||||
in their name to the specified max_height resolution.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
input_path (str): The path of the directory to search for video files and folders.
|
|
||||||
max_height (int, optional): The maximum height (in pixels) of the video to consider for conversion. Default is 720.
|
|
||||||
debug (bool, optional): If True, print debug messages. Default is False.
|
|
||||||
"""
|
|
||||||
|
|
||||||
files_to_rename = []
|
|
||||||
dirs_to_rename = []
|
|
||||||
|
|
||||||
for dirpath, dirnames, filenames in os.walk(input_path, topdown=True):
|
|
||||||
# Collect files to rename
|
|
||||||
for filename in filenames:
|
|
||||||
new_filename = apply_resolution_rename(filename, max_height)
|
|
||||||
if new_filename != filename:
|
|
||||||
old_file = os.path.join(dirpath, filename)
|
|
||||||
new_file = os.path.join(dirpath, new_filename)
|
|
||||||
files_to_rename.append((old_file, new_file))
|
|
||||||
|
|
||||||
# Collect directories to rename
|
|
||||||
for dirname in dirnames:
|
|
||||||
new_dirname = apply_resolution_rename(dirname, max_height)
|
|
||||||
if new_dirname != dirname:
|
|
||||||
old_dir = os.path.join(dirpath, dirname)
|
|
||||||
new_dir = os.path.join(dirpath, new_dirname)
|
|
||||||
dirs_to_rename.append((old_dir, new_dir))
|
|
||||||
|
|
||||||
# Rename files first
|
|
||||||
for old_file, new_file in files_to_rename:
|
|
||||||
if os.path.exists(new_file):
|
|
||||||
print(f"Error: Target filename {new_file} already exists.")
|
|
||||||
continue
|
|
||||||
os.rename(old_file, new_file)
|
|
||||||
if debug:
|
|
||||||
print(f"Renamed file: {old_file} -> {new_file}")
|
|
||||||
|
|
||||||
# Rename directories after
|
|
||||||
for old_dir, new_dir in sorted(dirs_to_rename, key=lambda x: -len(x[0])):
|
|
||||||
if os.path.exists(new_dir):
|
|
||||||
print(f"Error: Target directory name {new_dir} already exists.")
|
|
||||||
continue
|
|
||||||
os.rename(old_dir, new_dir)
|
|
||||||
if debug:
|
|
||||||
print(f"Renamed directory: {old_dir} -> {new_dir}")
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
"""
|
|
||||||
Main function to parse command line arguments and initiate file renamings.
|
|
||||||
"""
|
|
||||||
|
|
||||||
# Create argument parser
|
|
||||||
parser = argparse.ArgumentParser(
|
|
||||||
description="Rename video files containing resolutions in their filenames to a specified max_height resolution."
|
|
||||||
)
|
|
||||||
|
|
||||||
# Define command line arguments
|
|
||||||
parser.add_argument(
|
|
||||||
"input_path",
|
|
||||||
nargs="?",
|
|
||||||
default=os.getcwd(),
|
|
||||||
help="directory path to search for video files (default: current directory)",
|
|
||||||
)
|
|
||||||
parser.add_argument(
|
|
||||||
"-mh",
|
|
||||||
"--max-height",
|
|
||||||
type=int,
|
|
||||||
default=720,
|
|
||||||
help="maximum height of videos to be converted (default: 720)",
|
|
||||||
)
|
|
||||||
parser.add_argument(
|
|
||||||
"--debug",
|
|
||||||
action="store_true",
|
|
||||||
help="enable debug mode for printing additional messages",
|
|
||||||
)
|
|
||||||
|
|
||||||
# Enable autocomplete for argparse
|
|
||||||
argcomplete.autocomplete(parser)
|
|
||||||
|
|
||||||
# Parse command line arguments
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
# Rename videos
|
|
||||||
autorename(args.input_path, args.max_height, args.debug)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
# Execute main function when the script is run directly
|
|
||||||
main()
|
|
||||||
@ -1,266 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
# video_manage_audio.py
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import argcomplete
|
|
||||||
import colorama
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import subprocess
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
# Allow importing from scripts/library even when run directly
|
|
||||||
project_root = str(Path(__file__).resolve().parent.parent)
|
|
||||||
if project_root not in sys.path:
|
|
||||||
sys.path.append(project_root)
|
|
||||||
|
|
||||||
# === Local import ===
|
|
||||||
from scripts.library import deletefile
|
|
||||||
|
|
||||||
# Initialize colorama for colored output in the terminal
|
|
||||||
colorama.init()
|
|
||||||
|
|
||||||
creset = colorama.Fore.RESET
|
|
||||||
ccyan = colorama.Fore.CYAN
|
|
||||||
cyellow = colorama.Fore.YELLOW
|
|
||||||
cgreen = colorama.Fore.GREEN
|
|
||||||
cred = colorama.Fore.RED
|
|
||||||
|
|
||||||
|
|
||||||
def get_audio_metadata(file_path):
|
|
||||||
"""
|
|
||||||
Uses ffprobe to extract audio stream information.
|
|
||||||
Returns a list of dicts with track details.
|
|
||||||
"""
|
|
||||||
abs_file_path = os.path.abspath(file_path)
|
|
||||||
ffprobe_command = [
|
|
||||||
"ffprobe",
|
|
||||||
"-v",
|
|
||||||
"error",
|
|
||||||
"-select_streams",
|
|
||||||
"a",
|
|
||||||
"-show_entries",
|
|
||||||
"stream=index,codec_name:stream_tags=language,title",
|
|
||||||
"-of",
|
|
||||||
"json",
|
|
||||||
abs_file_path,
|
|
||||||
]
|
|
||||||
|
|
||||||
try:
|
|
||||||
result = subprocess.run(
|
|
||||||
ffprobe_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True
|
|
||||||
)
|
|
||||||
|
|
||||||
if result.returncode != 0:
|
|
||||||
return None
|
|
||||||
metadata = json.loads(result.stdout)
|
|
||||||
return metadata.get("streams", [])
|
|
||||||
except Exception:
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def list_audio_tracks(file_path):
|
|
||||||
"""Displays all audio tracks in a video file."""
|
|
||||||
print(f"{ccyan}Analyzing tracks for: {file_path}{creset}")
|
|
||||||
streams = get_audio_metadata(file_path)
|
|
||||||
|
|
||||||
if streams is None:
|
|
||||||
print(f"{cred} Failed to read file metadata.{creset}\n")
|
|
||||||
return
|
|
||||||
if not streams:
|
|
||||||
print(f"{cyellow} No audio tracks found in this file.{creset}\n")
|
|
||||||
return
|
|
||||||
|
|
||||||
for audio_idx, stream in enumerate(streams):
|
|
||||||
abs_idx = stream.get("index")
|
|
||||||
codec = stream.get("codec_name", "unknown")
|
|
||||||
tags = stream.get("tags", {})
|
|
||||||
lang = tags.get("language", "und")
|
|
||||||
title = tags.get("title", "No Title")
|
|
||||||
|
|
||||||
print(
|
|
||||||
f" {cgreen}[Audio Track {audio_idx}]{creset} "
|
|
||||||
f"Absolute Stream Index: {abs_idx} | "
|
|
||||||
f"Codec: {codec} | Lang: {lang} | Title: {title}"
|
|
||||||
)
|
|
||||||
print()
|
|
||||||
|
|
||||||
|
|
||||||
def resolve_track_target(streams, target):
|
|
||||||
"""
|
|
||||||
Resolves a track target (either an integer index string like '1'
|
|
||||||
or a language code string like 'eng') to a relative audio track index.
|
|
||||||
Returns None if no match is found.
|
|
||||||
"""
|
|
||||||
# Case 1: Target is an explicit number (e.g., "0", "1")
|
|
||||||
if target.isdigit():
|
|
||||||
idx = int(target)
|
|
||||||
if 0 <= idx < len(streams):
|
|
||||||
return idx
|
|
||||||
return None
|
|
||||||
|
|
||||||
# Case 2: Target is a language code (e.g., "eng", "rus")
|
|
||||||
for audio_idx, stream in enumerate(streams):
|
|
||||||
lang = stream.get("tags", {}).get("language", "").lower()
|
|
||||||
if lang == target.lower():
|
|
||||||
return audio_idx
|
|
||||||
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def process_audio(file_path, track_target, command):
|
|
||||||
"""Modify or list audio tracks of a video file."""
|
|
||||||
if command == "list":
|
|
||||||
list_audio_tracks(file_path)
|
|
||||||
return
|
|
||||||
|
|
||||||
streams = get_audio_metadata(file_path)
|
|
||||||
if not streams:
|
|
||||||
print(f"{cred}Skipping {file_path}: Could not parse audio streams.{creset}\n")
|
|
||||||
return
|
|
||||||
|
|
||||||
# Dynamic target resolution (converts 'eng' or '1' to the proper track index)
|
|
||||||
resolved_track = resolve_track_target(streams, track_target)
|
|
||||||
|
|
||||||
if resolved_track is None:
|
|
||||||
print(
|
|
||||||
f"{cyellow}Skipping {file_path}: Target audio track/language '{track_target}' not found.{creset}\n"
|
|
||||||
)
|
|
||||||
return
|
|
||||||
|
|
||||||
print(f"{cgreen}Processing file: {file_path}{creset}")
|
|
||||||
print(
|
|
||||||
f"-> Target resolved to Audio Track {resolved_track} based on '{track_target}'"
|
|
||||||
)
|
|
||||||
|
|
||||||
output_file = f"{os.path.splitext(file_path)[0]}_{command}_audio{os.path.splitext(file_path)[1]}"
|
|
||||||
|
|
||||||
if command == "remove":
|
|
||||||
# Remove only the specified audio track while keeping all other streams
|
|
||||||
ffmpeg_command = [
|
|
||||||
"ffmpeg",
|
|
||||||
"-i",
|
|
||||||
file_path,
|
|
||||||
"-map",
|
|
||||||
"0",
|
|
||||||
"-map",
|
|
||||||
f"-0:a:{resolved_track}",
|
|
||||||
"-c",
|
|
||||||
"copy",
|
|
||||||
output_file,
|
|
||||||
]
|
|
||||||
elif command == "keep":
|
|
||||||
# Keep only the specified audio track while preserving video, subtitles, and metadata
|
|
||||||
ffmpeg_command = [
|
|
||||||
"ffmpeg",
|
|
||||||
"-i",
|
|
||||||
file_path,
|
|
||||||
"-map",
|
|
||||||
"0:v",
|
|
||||||
"-map",
|
|
||||||
f"0:a:{resolved_track}",
|
|
||||||
"-map",
|
|
||||||
"0:s?",
|
|
||||||
"-map",
|
|
||||||
"0:t?",
|
|
||||||
"-c",
|
|
||||||
"copy",
|
|
||||||
output_file,
|
|
||||||
]
|
|
||||||
else:
|
|
||||||
return
|
|
||||||
|
|
||||||
try:
|
|
||||||
# Keep a track of the current_file in case of failure
|
|
||||||
current_file = output_file
|
|
||||||
|
|
||||||
result = subprocess.run(
|
|
||||||
ffmpeg_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True
|
|
||||||
)
|
|
||||||
|
|
||||||
if result.returncode == 0:
|
|
||||||
print(
|
|
||||||
f"{ccyan}Audio processing complete. Output saved to {output_file}{creset}\n"
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
print(
|
|
||||||
f"{cred}Command failed with return code {result.returncode}{creset}\n"
|
|
||||||
)
|
|
||||||
print(f"{cred}Error output:\n{result.stderr}{creset}\n")
|
|
||||||
|
|
||||||
except subprocess.CalledProcessError as e:
|
|
||||||
# If we have a partial video converted, delete it due to the failure
|
|
||||||
if current_file:
|
|
||||||
deletefile(current_file)
|
|
||||||
|
|
||||||
except KeyboardInterrupt:
|
|
||||||
print(f"{cyellow}Conversions cancelled, cleaning up...{creset}")
|
|
||||||
|
|
||||||
# If we have a partial video converted, delete it due to the failure
|
|
||||||
if current_file:
|
|
||||||
deletefile(current_file)
|
|
||||||
current_file = None
|
|
||||||
|
|
||||||
exit()
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
print(f"{cred}Error processing audio: {e}{creset}\n")
|
|
||||||
|
|
||||||
|
|
||||||
def process_directory(dir_path, track_target, command):
|
|
||||||
"""Recursively processes all video files in the specified directory."""
|
|
||||||
for root, _, files in os.walk(dir_path):
|
|
||||||
for file in files:
|
|
||||||
if file.endswith((".mp4", ".mkv", ".avi", ".mov")):
|
|
||||||
file_path = os.path.join(root, file)
|
|
||||||
process_audio(file_path, track_target, command)
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
"""
|
|
||||||
Main function to parse command-line arguments and initiate the audio processing.
|
|
||||||
"""
|
|
||||||
|
|
||||||
parser = argparse.ArgumentParser(
|
|
||||||
description="Manage or list audio tracks in video files dynamically."
|
|
||||||
)
|
|
||||||
|
|
||||||
parser.add_argument(
|
|
||||||
"command", choices=["remove", "keep", "list"], help="Command to run"
|
|
||||||
)
|
|
||||||
|
|
||||||
parser.add_argument(
|
|
||||||
"-t",
|
|
||||||
"--track",
|
|
||||||
type=str,
|
|
||||||
default="0",
|
|
||||||
help="Audio track index (e.g., 0, 1) OR language ISO code (e.g., eng, rus). Default is '0'.",
|
|
||||||
)
|
|
||||||
parser.add_argument("-f", "--file", type=str, help="Path to a specific video file.")
|
|
||||||
parser.add_argument(
|
|
||||||
"-d",
|
|
||||||
"--dir",
|
|
||||||
type=str,
|
|
||||||
default=os.getcwd(),
|
|
||||||
help="Directory to process (default is current directory).",
|
|
||||||
)
|
|
||||||
|
|
||||||
argcomplete.autocomplete(parser)
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
# Process a single file if provided, otherwise process a directory
|
|
||||||
if args.file:
|
|
||||||
if os.path.isfile(args.file):
|
|
||||||
process_audio(args.file, args.track, args.command)
|
|
||||||
else:
|
|
||||||
print(f"{cred}File {args.file} does not exist.{creset}")
|
|
||||||
else:
|
|
||||||
if os.path.isdir(args.dir):
|
|
||||||
process_directory(args.dir, args.track, args.command)
|
|
||||||
else:
|
|
||||||
print(f"{cred}Directory {args.dir} does not exist.{creset}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@ -1,169 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
# video_manage_subtitles.py
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import argcomplete
|
|
||||||
import colorama
|
|
||||||
import os
|
|
||||||
import subprocess
|
|
||||||
|
|
||||||
# Initialize colorama for colored output in the terminal
|
|
||||||
colorama.init()
|
|
||||||
|
|
||||||
creset = colorama.Fore.RESET
|
|
||||||
ccyan = colorama.Fore.CYAN
|
|
||||||
cyellow = colorama.Fore.YELLOW
|
|
||||||
cgreen = colorama.Fore.GREEN
|
|
||||||
cred = colorama.Fore.RED
|
|
||||||
|
|
||||||
|
|
||||||
def process_subtitles(file_path, track, command):
|
|
||||||
"""
|
|
||||||
Modify subtitles tracks of a video file based on the given command.
|
|
||||||
- 'remove': Remove the specified subtitle track while keeping everything else.
|
|
||||||
- 'keep': Keep only the specified subtitle track and remove all others.
|
|
||||||
- 'none': Remove all subtitle tracks.
|
|
||||||
The function preserves video, audio, and metadata.
|
|
||||||
"""
|
|
||||||
print(f"{cgreen}Processing file: {file_path}{creset}")
|
|
||||||
output_file = f"{os.path.splitext(file_path)[0]}_{command}_subtitles{os.path.splitext(file_path)[1]}"
|
|
||||||
|
|
||||||
# Construct ffmpeg command based on user choice
|
|
||||||
if command == "remove":
|
|
||||||
# Remove only the specified subtitle track while keeping all other streams
|
|
||||||
ffmpeg_command = [
|
|
||||||
"ffmpeg",
|
|
||||||
"-i",
|
|
||||||
file_path,
|
|
||||||
"-map",
|
|
||||||
"0",
|
|
||||||
"-map",
|
|
||||||
f"-0:s:{track}",
|
|
||||||
"-c",
|
|
||||||
"copy",
|
|
||||||
output_file,
|
|
||||||
]
|
|
||||||
elif command == "keep":
|
|
||||||
# Keep only the specified subtitle track while preserving video, audio, and metadata
|
|
||||||
ffmpeg_command = [
|
|
||||||
"ffmpeg",
|
|
||||||
"-i",
|
|
||||||
file_path,
|
|
||||||
"-map",
|
|
||||||
"0:v",
|
|
||||||
"-map",
|
|
||||||
"0:a",
|
|
||||||
"-map",
|
|
||||||
f"0:s:{track}",
|
|
||||||
"-map",
|
|
||||||
"0:t?",
|
|
||||||
"-c",
|
|
||||||
"copy",
|
|
||||||
output_file,
|
|
||||||
]
|
|
||||||
elif command == "none":
|
|
||||||
# Remove all subtitle tracks
|
|
||||||
ffmpeg_command = [
|
|
||||||
"ffmpeg",
|
|
||||||
"-i",
|
|
||||||
file_path,
|
|
||||||
"-map",
|
|
||||||
"0",
|
|
||||||
"-map",
|
|
||||||
"-0:s",
|
|
||||||
"-c",
|
|
||||||
"copy",
|
|
||||||
output_file,
|
|
||||||
]
|
|
||||||
else:
|
|
||||||
print(f"{cred}Invalid command: {command}{creset}")
|
|
||||||
return
|
|
||||||
|
|
||||||
try:
|
|
||||||
# Execute the ffmpeg command and capture output
|
|
||||||
result = subprocess.run(
|
|
||||||
ffmpeg_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True
|
|
||||||
)
|
|
||||||
|
|
||||||
if result.returncode == 0:
|
|
||||||
print(
|
|
||||||
f"{ccyan}Subtitles processing complete. Output saved to {output_file}{creset}\n"
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
print(
|
|
||||||
f"{cred}Command failed with return code {result.returncode}{creset}\n"
|
|
||||||
)
|
|
||||||
print(f"{cred}Error output:\n{result.stderr}{creset}\n")
|
|
||||||
|
|
||||||
except subprocess.CalledProcessError as e:
|
|
||||||
print(f"{cred}Error processing subtitles: {e}{creset}\n")
|
|
||||||
|
|
||||||
|
|
||||||
def process_directory(dir_path, track, command):
|
|
||||||
"""
|
|
||||||
Recursively processes all video files in the specified directory.
|
|
||||||
Applies the chosen subtitle modification (remove, keep, none) to each file.
|
|
||||||
"""
|
|
||||||
for root, _, files in os.walk(dir_path):
|
|
||||||
for file in files:
|
|
||||||
if file.endswith((".mp4", ".mkv", ".avi", ".mov")):
|
|
||||||
file_path = os.path.join(root, file)
|
|
||||||
process_subtitles(file_path, track, command)
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
"""
|
|
||||||
Main function to parse command-line arguments and initiate the subtitle processing.
|
|
||||||
"""
|
|
||||||
|
|
||||||
# Create argument parser
|
|
||||||
parser = argparse.ArgumentParser(
|
|
||||||
description="Manage subtitle tracks in video files."
|
|
||||||
)
|
|
||||||
|
|
||||||
# Define command line arguments
|
|
||||||
# Add a positional argument for the command
|
|
||||||
parser.add_argument(
|
|
||||||
"command",
|
|
||||||
choices=["remove", "keep", "none"],
|
|
||||||
help="Command to run (remove, keep, or none)",
|
|
||||||
)
|
|
||||||
|
|
||||||
# Add other arguments with both short and long options, including defaults
|
|
||||||
parser.add_argument(
|
|
||||||
"-t",
|
|
||||||
"--track",
|
|
||||||
type=int,
|
|
||||||
default=0,
|
|
||||||
help="Subtitle track index (default is 0). Use 'none' to remove all subtitles.",
|
|
||||||
)
|
|
||||||
parser.add_argument("-f", "--file", type=str, help="Path to a specific video file.")
|
|
||||||
parser.add_argument(
|
|
||||||
"-d",
|
|
||||||
"--dir",
|
|
||||||
type=str,
|
|
||||||
default=os.getcwd(),
|
|
||||||
help="Directory to process (default is current directory).",
|
|
||||||
)
|
|
||||||
|
|
||||||
# Enable autocomplete for command-line arguments
|
|
||||||
argcomplete.autocomplete(parser)
|
|
||||||
|
|
||||||
# Parse command line arguments
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
# Process a single file if provided, otherwise process a directory
|
|
||||||
if args.file:
|
|
||||||
if os.path.isfile(args.file):
|
|
||||||
process_subtitles(args.file, args.track, args.command)
|
|
||||||
else:
|
|
||||||
print(f"{cred}File {args.file} does not exist.{creset}")
|
|
||||||
else:
|
|
||||||
if os.path.isdir(args.dir):
|
|
||||||
process_directory(args.dir, args.track, args.command)
|
|
||||||
else:
|
|
||||||
print(f"{cred}Directory {args.dir} does not exist.{creset}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
95
scripts/video_remove_audio.py
Executable file
95
scripts/video_remove_audio.py
Executable file
@ -0,0 +1,95 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
|
||||||
|
import os
|
||||||
|
import argparse
|
||||||
|
import argcomplete
|
||||||
|
import colorama
|
||||||
|
import subprocess
|
||||||
|
|
||||||
|
colorama.init()
|
||||||
|
|
||||||
|
creset = colorama.Fore.RESET
|
||||||
|
ccyan = colorama.Fore.CYAN
|
||||||
|
cyellow = colorama.Fore.YELLOW
|
||||||
|
cgreen = colorama.Fore.GREEN
|
||||||
|
cred = colorama.Fore.RED
|
||||||
|
|
||||||
|
|
||||||
|
# Function to remove audio track from a video using ffmpeg
|
||||||
|
def remove_audio_track(file_path, track):
|
||||||
|
print(f"{cgreen}Processing file: {file_path}{creset}")
|
||||||
|
output_file = f"{os.path.splitext(file_path)[0]}_no_audio{os.path.splitext(file_path)[1]}"
|
||||||
|
# ffmpeg command to remove the specified audio track and keep other streams
|
||||||
|
command = [
|
||||||
|
"ffmpeg", "-i", file_path, "-map", "0", "-map", f"-0:a:{track}", "-c",
|
||||||
|
"copy", output_file
|
||||||
|
]
|
||||||
|
|
||||||
|
try:
|
||||||
|
result = subprocess.run(command,
|
||||||
|
stdout=subprocess.PIPE,
|
||||||
|
stderr=subprocess.PIPE,
|
||||||
|
text=True)
|
||||||
|
|
||||||
|
if result.returncode == 0:
|
||||||
|
print(
|
||||||
|
f"{ccyan}Audio track {track} removed. Output saved to {output_file}{creset}\n"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
print(
|
||||||
|
f"{cred}Command failed with return code {result.returncode}{creset}\n"
|
||||||
|
)
|
||||||
|
print(f"{cred}Error output:", result.stderr, f"{creset}\n")
|
||||||
|
|
||||||
|
except subprocess.CalledProcessError as e:
|
||||||
|
print(f"{cred}Error removing audio track: {e}{creset}\n")
|
||||||
|
|
||||||
|
|
||||||
|
# Function to recursively process videos in a directory
|
||||||
|
def process_directory(dir_path, track):
|
||||||
|
for root, _, files in os.walk(dir_path):
|
||||||
|
for file in files:
|
||||||
|
if file.endswith((".mp4", ".mkv", ".avi",
|
||||||
|
".mov")): # Add more formats as needed
|
||||||
|
file_path = os.path.join(root, file)
|
||||||
|
remove_audio_track(file_path, track)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
# Set up argument parser
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description="Remove audio track from video files.")
|
||||||
|
parser.add_argument("--track",
|
||||||
|
type=int,
|
||||||
|
default=0,
|
||||||
|
help="Audio track index to remove (default is 0).")
|
||||||
|
parser.add_argument("--file",
|
||||||
|
type=str,
|
||||||
|
help="Path to a specific video file.")
|
||||||
|
parser.add_argument(
|
||||||
|
"--dir",
|
||||||
|
type=str,
|
||||||
|
default=os.getcwd(),
|
||||||
|
help="Directory to process (default is current directory).")
|
||||||
|
|
||||||
|
# Enable autocomplete for argparse
|
||||||
|
argcomplete.autocomplete(parser)
|
||||||
|
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
# Process single file if provided
|
||||||
|
if args.file:
|
||||||
|
if os.path.isfile(args.file):
|
||||||
|
remove_audio_track(args.file, args.track)
|
||||||
|
else:
|
||||||
|
print(f"File {args.file} does not exist.")
|
||||||
|
# Otherwise, process all files in the specified directory
|
||||||
|
else:
|
||||||
|
if os.path.isdir(args.dir):
|
||||||
|
process_directory(args.dir, args.track)
|
||||||
|
else:
|
||||||
|
print(f"Directory {args.dir} does not exist.")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@ -1,139 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""A utility script to repackage video files into MKV containers using FFmpeg.
|
|
||||||
|
|
||||||
This script recursively scans a specified directory for non-MKV video files
|
|
||||||
and uses FFmpeg to repackage them into '.mkv' containers. It explicitly maps
|
|
||||||
all streams (video, audio, subtitles, data) and preserves all metadata without
|
|
||||||
re-encoding. It includes options to automatically delete the source files upon
|
|
||||||
successful conversion.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import subprocess
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Sequence
|
|
||||||
|
|
||||||
# Define the targeted non-MKV extensions (case-insensitive handling below)
|
|
||||||
TARGET_EXTENSIONS = {".mp4", ".m4v", ".flv", ".avi", ".mov", ".wmv"}
|
|
||||||
|
|
||||||
|
|
||||||
def parse_args(args: Sequence[str] | None = None) -> argparse.Namespace:
|
|
||||||
"""Parse command-line arguments for the repackaging utility.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
args: Optional sequence of strings to parse. If None, sys.argv is used.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
An argparse.Namespace object containing the parsed arguments.
|
|
||||||
"""
|
|
||||||
parser = argparse.ArgumentParser(
|
|
||||||
description="Recursively repackage non-MKV video files into MKV containers, preserving all streams and metadata."
|
|
||||||
)
|
|
||||||
|
|
||||||
parser.add_argument(
|
|
||||||
"-i",
|
|
||||||
"--input",
|
|
||||||
type=Path,
|
|
||||||
default=Path.cwd(),
|
|
||||||
help="Path to the root directory to scan for video files (default: current working directory).",
|
|
||||||
)
|
|
||||||
|
|
||||||
parser.add_argument(
|
|
||||||
"-d",
|
|
||||||
"--delete",
|
|
||||||
action="store_true",
|
|
||||||
default=False,
|
|
||||||
help="Delete the original video files upon successful conversion.",
|
|
||||||
)
|
|
||||||
|
|
||||||
# Enable argcomplete support if the module is available
|
|
||||||
try:
|
|
||||||
import argcomplete
|
|
||||||
|
|
||||||
argcomplete.autocomplete(parser)
|
|
||||||
except ImportError:
|
|
||||||
pass
|
|
||||||
|
|
||||||
return parser.parse_args(args)
|
|
||||||
|
|
||||||
|
|
||||||
def repackage_to_mkv(input_dir: Path, delete_source: bool) -> None:
|
|
||||||
"""Recursively scan the input directory and repackage non-MKV videos to MKV.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
input_dir: The root Path directory to scan.
|
|
||||||
delete_source: If True, deletes the original video file after a
|
|
||||||
successful FFmpeg operation.
|
|
||||||
"""
|
|
||||||
if not input_dir.is_dir():
|
|
||||||
print(f"Error: The directory '{input_dir}' does not exist.")
|
|
||||||
return
|
|
||||||
|
|
||||||
print(f"Scanning '{input_dir}' recursively for video files...")
|
|
||||||
|
|
||||||
video_files = [
|
|
||||||
p
|
|
||||||
for p in input_dir.rglob("*")
|
|
||||||
if p.is_file() and p.suffix.lower() in TARGET_EXTENSIONS
|
|
||||||
]
|
|
||||||
|
|
||||||
if not video_files:
|
|
||||||
print("No matching video files found.")
|
|
||||||
return
|
|
||||||
|
|
||||||
print(f"Found {len(video_files)} files to process.\n")
|
|
||||||
|
|
||||||
for file_path in video_files:
|
|
||||||
output_path = file_path.with_suffix(".mkv")
|
|
||||||
|
|
||||||
if output_path.exists():
|
|
||||||
print(
|
|
||||||
f"Warning: Destination {output_path.name} already exists. Skipping to avoid accidental overwrite."
|
|
||||||
)
|
|
||||||
print("-" * 40)
|
|
||||||
continue
|
|
||||||
|
|
||||||
cmd = [
|
|
||||||
"ffmpeg",
|
|
||||||
"-i",
|
|
||||||
str(file_path),
|
|
||||||
"-map",
|
|
||||||
"0",
|
|
||||||
"-c",
|
|
||||||
"copy",
|
|
||||||
"-map_metadata",
|
|
||||||
"0",
|
|
||||||
str(output_path),
|
|
||||||
"-loglevel",
|
|
||||||
"error",
|
|
||||||
"-y",
|
|
||||||
]
|
|
||||||
|
|
||||||
# Display the relative path from the input directory so it's easier to track progress
|
|
||||||
relative_display = file_path.relative_to(input_dir)
|
|
||||||
print(f"Processing: {relative_display} -> {output_path.name}...")
|
|
||||||
|
|
||||||
try:
|
|
||||||
subprocess.run(cmd, check=True)
|
|
||||||
print(f"Successfully converted: {file_path.name}")
|
|
||||||
|
|
||||||
if delete_source:
|
|
||||||
file_path.unlink()
|
|
||||||
print(f"Deleted original file: {file_path.name}")
|
|
||||||
|
|
||||||
except subprocess.CalledProcessError:
|
|
||||||
print(f"Failed to convert: {file_path.name}")
|
|
||||||
except OSError as e:
|
|
||||||
print(f"System error processing {file_path.name}: {e}")
|
|
||||||
|
|
||||||
print("-" * 40)
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
|
||||||
"""Main execution entry point."""
|
|
||||||
args = parse_args()
|
|
||||||
repackage_to_mkv(input_dir=args.input, delete_source=args.delete)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
520
setups-drafts/almalinux_setup_wordpress.md
Normal file
520
setups-drafts/almalinux_setup_wordpress.md
Normal file
@ -0,0 +1,520 @@
|
|||||||
|
# Installing Wordpress on AlmaLinux
|
||||||
|
|
||||||
|
## Table of Contents
|
||||||
|
|
||||||
|
- [Installing Wordpress on AlmaLinux](#installing-wordpress-on-almalinux)
|
||||||
|
- [Table of Contents](#table-of-contents)
|
||||||
|
- [Disclaimer: Incomplete Guide](#disclaimer-incomplete-guide)
|
||||||
|
- [Introduction](#introduction)
|
||||||
|
- [Why WordPress?](#why-wordpress)
|
||||||
|
- [Prerequisites](#prerequisites)
|
||||||
|
- [What This Guide Covers](#what-this-guide-covers)
|
||||||
|
- [Placeholders](#placeholders)
|
||||||
|
- [Important Warnings and Security Practices](#important-warnings-and-security-practices)
|
||||||
|
- [Useful Commands and Information](#useful-commands-and-information)
|
||||||
|
- [Documentation](#documentation)
|
||||||
|
- [Links](#links)
|
||||||
|
- [Software on the Machine](#software-on-the-machine)
|
||||||
|
- [Paths](#paths)
|
||||||
|
- [Proxmox Commands](#proxmox-commands)
|
||||||
|
- [SSH Connection](#ssh-connection)
|
||||||
|
- [Installation Procedure](#installation-procedure)
|
||||||
|
|
||||||
|
## Disclaimer: Incomplete Guide
|
||||||
|
|
||||||
|
This document is a draft and may contain incomplete, untested, or outdated information. It is a work in progress and has not been verified for accuracy or usability. Use this guide at your own discretion, and consider it as a reference for further development or exploration. Updates may follow in the future, but no guarantees are made.
|
||||||
|
|
||||||
|
## Introduction
|
||||||
|
|
||||||
|
Welcome to the installation guide for WordPress on AlmaLinux in a Proxmox LXC container. WordPress is a powerful content management system (CMS) that allows you to create and manage websites efficiently.
|
||||||
|
|
||||||
|
## Why WordPress?
|
||||||
|
|
||||||
|
WordPress is one of the most popular content management systems (CMS) in the world, powering over 40% of websites on the internet. It offers a flexible and user-friendly platform for building anything from simple blogs to complex e-commerce sites.
|
||||||
|
|
||||||
|
**Key Benefits:**
|
||||||
|
|
||||||
|
- **Open-Source & Free** – No licensing fees, with a large community contributing to its continuous development.
|
||||||
|
- **Extensive Plugin Ecosystem** – Thousands of plugins to add features like SEO, security, performance optimization, and more.
|
||||||
|
- **Customizable Themes** – A wide variety of free and premium themes allow you to tailor your website's design.
|
||||||
|
- **SEO-Friendly** – Built-in SEO features and plugins like Yoast SEO help improve search engine rankings.
|
||||||
|
- **Scalability** – Suitable for small personal blogs to large enterprise websites with high traffic.
|
||||||
|
- **Active Community & Support** – Large developer and user communities provide extensive documentation, forums, and professional support options.
|
||||||
|
|
||||||
|
Whether you're launching a blog, a portfolio, or a business website, WordPress provides the flexibility and power to meet your needs.
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
Before you begin the installation process, ensure that your AlmaLinux system meets the following requirements:
|
||||||
|
|
||||||
|
- AlmaLinux GNU/Linux 9 or later
|
||||||
|
- Access to a terminal with sudo privileges
|
||||||
|
- Basic familiarity with the command line interface
|
||||||
|
- Stable internet connection to download necessary packages
|
||||||
|
|
||||||
|
## What This Guide Covers
|
||||||
|
|
||||||
|
This guide covers the installation and configuration of Wordpress on an AlmaLinux server, along with additional setup tasks such as SSH connection management and Proxmox commands. It covers:
|
||||||
|
|
||||||
|
1. **Installation**: Installing Wordpress from the official site.
|
||||||
|
2. **Configuration**: Configuring Wordpress to suit your environment and preferences.
|
||||||
|
3. **Setup**: Setting up Wordpress as a service and accessing it.
|
||||||
|
|
||||||
|
## Placeholders
|
||||||
|
|
||||||
|
Replace the placeholders below with the appropriate values for your setup:
|
||||||
|
|
||||||
|
- **User Details**
|
||||||
|
|
||||||
|
- Username: `<username>` (e.g., admin)
|
||||||
|
- Username - Hypervisor: `<username-hypervisor>` (e.g., admin)
|
||||||
|
|
||||||
|
- **Server Configuration**
|
||||||
|
|
||||||
|
- Server IP address: `<server-ip-address>` (e.g., 192.168.1.100)
|
||||||
|
- Hostname - Intranet: `<hostname-intranet>` (e.g., wordpress-server.domain.com)
|
||||||
|
- Hostname - Internet: `<hostname-internet>` (e.g., wordpress.domain.com)
|
||||||
|
- Hostname - Hypervisor: `<hostname-hypervisor>` (e.g., proxmox-hypervisor.domain.com)
|
||||||
|
- Hostname - Hypervisor NAS: `<hostname-hypervisor-nas>` (e.g., nas-server.domain.com)
|
||||||
|
- Name - Hypervisor NAS: `<name-hypervisor-nas>` (e.g., nas-server)
|
||||||
|
- Container ID: `<container-id>` (e.g., 100)
|
||||||
|
|
||||||
|
- **SSH Keys**
|
||||||
|
|
||||||
|
- SSH key - Proxmox: `<ssh-key-proxmox>` (e.g., /home/user/.ssh/id_rsa.pub)
|
||||||
|
- SSH key - Client: `<ssh-key-client>` (e.g., /home/user/.ssh/client_id_rsa.pub)
|
||||||
|
|
||||||
|
- **Networking**
|
||||||
|
|
||||||
|
- Wireguard port: `<wireguard-port>` (e.g., 51820)
|
||||||
|
|
||||||
|
- **Database**
|
||||||
|
|
||||||
|
- Database password : `<database-password>` (e.g., 15GbGnOn3Vjy9RQ4G9TfUF95wPcoKAy5)
|
||||||
|
|
||||||
|
- **Paths**
|
||||||
|
|
||||||
|
- Path index: `<path-index>` (e.g., /var/www/html)
|
||||||
|
|
||||||
|
## Important Warnings and Security Practices
|
||||||
|
|
||||||
|
Before executing any commands in this documentation, please adhere to the following guidelines to ensure the security and integrity of the system:
|
||||||
|
|
||||||
|
1. **Execute Commands with Caution**: Always review and understand a command before executing it. Misuse of commands can lead to data loss or system instability.
|
||||||
|
2. **Backup Command Execution**: The backup command must be executed only by authorized users. Ensure that proper permissions are set to prevent unauthorized access to backup files.
|
||||||
|
3. **Regular Backups**: Maintain regular backups of all critical data. It is advisable to use automated backup solutions and verify backup integrity periodically.
|
||||||
|
4. **System Updates**: Regularly update the system and all installed packages to protect against vulnerabilities. Use the package manager responsibly to avoid potential conflicts.
|
||||||
|
5. **Monitor System Logs**: Continuously monitor system logs for any unusual activity. Use logging tools to help identify potential security breaches or system failures.
|
||||||
|
6. **User Permissions**: Ensure that user permissions are strictly managed. Limit access to sensitive commands and data to only those who need it to perform their job functions.
|
||||||
|
7. **Network Security**: Implement proper network security measures, such as firewalls and intrusion detection systems, to protect against external threats.
|
||||||
|
8. **Data Encryption**: Encrypt sensitive data at rest and in transit to prevent unauthorized access.
|
||||||
|
|
||||||
|
By following these practices, you will help maintain the security and stability of the system while minimizing the risk of data loss or compromise.
|
||||||
|
|
||||||
|
## Useful Commands and Information
|
||||||
|
|
||||||
|
### Documentation
|
||||||
|
|
||||||
|
- [AlmaLinux Wiki](https://wiki.almalinux.org/)
|
||||||
|
- [Documentation Overview](https://www.wordpress.info/doc/overview/)
|
||||||
|
- [Tutorials](https://wordpress.com/learn/)
|
||||||
|
|
||||||
|
### Links
|
||||||
|
|
||||||
|
- [Wordpress appliance](https://<hostname-internet>/)
|
||||||
|
|
||||||
|
### Software on the Machine
|
||||||
|
|
||||||
|
- **Operating System**: AlmaLinux
|
||||||
|
- **Web Server**: Apache
|
||||||
|
- **Database**: Mariadb
|
||||||
|
- **Security**: GnuPG, WireGuard, firewalld
|
||||||
|
- **Other**: Git, sudo
|
||||||
|
|
||||||
|
### Paths
|
||||||
|
|
||||||
|
- **Apache AlmaLinux Default Configuration**: `/etc/httpd/conf.d/welcome.conf`
|
||||||
|
- **Wordpress Configuration**: `/var/www/html/wp-config.php`
|
||||||
|
- **Wordpress Work Path**: `/var/www/html`
|
||||||
|
|
||||||
|
### Proxmox Commands
|
||||||
|
|
||||||
|
**List available Proxmox templates**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh <username>@<hostname-hypervisor-nas> "ls /mnt/proxmox/template/cache/"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Create the container**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "pct create <container-id> <name-hypervisor-nas>:vztmpl/almalinux-9-default_20240911_amd64.tar.xz --hostname <hostname-intranet> --cores 2 --memory 4096 --swap 2048 --net0 name=net0,bridge=vmbr0,ip=dhcp,firewall=1 --rootfs <name-hypervisor-nas>:100 --unprivileged 1 --features nesting=1 --ssh-public-keys <ssh-key-proxmox>"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Backup**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump <container-id> --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
|
||||||
|
```
|
||||||
|
|
||||||
|
**Set the state of the Proxmox HA Manager for Container <container-id>**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager add ct:<container-id>"
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager remove ct:<container-id>"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Set the state and limits of the Proxmox Container <container-id> in the HA Manager**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id> --state started --max_relocate 3 --max_restart 3"
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id> --state stopped"
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "pct start <container-id>"
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "pct stop <container-id>"
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "pct reboot <container-id>"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Destroy the Proxmox Container <container-id> forcefully**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "pct destroy <container-id> --force --purge"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Move the Proxmox Container <container-id> to another host**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "pct migrate <container-id> hv2"
|
||||||
|
```
|
||||||
|
|
||||||
|
### SSH Connection
|
||||||
|
|
||||||
|
**Connection with specific keys**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -i <ssh-key-client> root@<hostname-intranet>
|
||||||
|
ssh -i <ssh-key-client> root@<server-ip-address>
|
||||||
|
ssh -i <ssh-key-client> <username>@<hostname-intranet>
|
||||||
|
ssh -i <ssh-key-client> <username>@<server-ip-address>
|
||||||
|
```
|
||||||
|
|
||||||
|
**Remove offending keys from known_hosts**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh-keygen -f "/home/<username>/.ssh/known_hosts" -R "<hostname-intranet>"
|
||||||
|
ssh-keygen -f "/home/<username>/.ssh/known_hosts" -R "<server-ip-address>"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Copy SSH public key to remote host**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh-copy-id -i <ssh-key-client> root@<server-ip-address>
|
||||||
|
ssh-copy-id -i <ssh-key-client> root@<hostname-intranet>
|
||||||
|
ssh-copy-id -i <ssh-key-client> <username>@<server-ip-address>
|
||||||
|
ssh-copy-id -i <ssh-key-client> <username>@<hostname-intranet>
|
||||||
|
```
|
||||||
|
|
||||||
|
## Installation Procedure
|
||||||
|
|
||||||
|
1. **Fresh AlmaLinux Installation**
|
||||||
|
|
||||||
|
- Install a fresh AlmaLinux operating system on your new server following the standard installation procedure.
|
||||||
|
|
||||||
|
2. **Backup before starting**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump <container-id> --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Install Required Dependencies**
|
||||||
|
|
||||||
|
**Upgrade the base system**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dnf update
|
||||||
|
```
|
||||||
|
|
||||||
|
**Enable EPEL repository**
|
||||||
|
Extra Package for Enterprise Linux repository has packages like Apache and Nginx
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dnf install -y epel-release
|
||||||
|
dnf makecache
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Install LAMP Stack**
|
||||||
|
|
||||||
|
**Install dependencies**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dnf install -y sudo nano firewalld firewall-config tar wget curl unzip
|
||||||
|
dnf install -y gnupg nginx git wireguard-tools
|
||||||
|
|
||||||
|
dnf install -y httpd httpd-tools mariadb-server mariadb
|
||||||
|
|
||||||
|
# Basic command
|
||||||
|
dnf install -y php php-mysqlnd php-fpm php-json php-mbstring php-xml php-curl php-zip php-gd php-intl php-bcmath php-soap php-exif
|
||||||
|
# More tooling and security
|
||||||
|
dnf install -y httpd mod_ssl php php-cli php-common php-fpm php-gd php-intl php-json php-mbstring php-mysqlnd php-opcache php-pdo php-pecl-imagick php-xml php-zip policycoreutils-python-utils
|
||||||
|
# Other xml php modules
|
||||||
|
dnf install -y php-dom php-simplexml php-xmlreader php-iconv php-posix php-sockets php-tokenizer
|
||||||
|
|
||||||
|
# Necessary for wp-cli
|
||||||
|
dnf install -y php-cli php-mbstring unzip curl
|
||||||
|
```
|
||||||
|
|
||||||
|
**WordPress Dependencies on AlmaLinux (MariaDB)**
|
||||||
|
|
||||||
|
**Web Server**
|
||||||
|
|
||||||
|
- `httpd` (Apache)
|
||||||
|
- `mod_ssl` (For HTTPS support)
|
||||||
|
- OR `nginx` (If using Nginx instead of Apache)
|
||||||
|
|
||||||
|
**PHP**
|
||||||
|
|
||||||
|
- `php` (Main PHP package)
|
||||||
|
- `php-cli` (Command-line interface for PHP)
|
||||||
|
- `php-common` (Common PHP files)
|
||||||
|
- `php-fpm` (FastCGI Process Manager for PHP, required for Nginx)
|
||||||
|
- `php-gd` (Image processing)
|
||||||
|
- `php-intl` (Internationalization)
|
||||||
|
- `php-json` (JSON support)
|
||||||
|
- `php-mbstring` (Multibyte string functions)
|
||||||
|
- `php-mysqlnd` (MySQL/MariaDB support)
|
||||||
|
- `php-opcache` (Performance optimization)
|
||||||
|
- `php-pdo` (PHP Data Objects)
|
||||||
|
- `php-pecl-imagick` (ImageMagick extension, recommended for media handling)
|
||||||
|
- `php-xml` (XML parsing)
|
||||||
|
- `php-zip` (ZIP file support)
|
||||||
|
|
||||||
|
**Database (MariaDB)**
|
||||||
|
|
||||||
|
- `mariadb-server` (MariaDB database server)
|
||||||
|
- `mariadb` (MariaDB client)
|
||||||
|
|
||||||
|
**Additional System Packages**
|
||||||
|
|
||||||
|
- `tar` (Required for extracting WordPress archives)
|
||||||
|
- `wget` (To fetch files from the web)
|
||||||
|
- `curl` (For network requests)
|
||||||
|
- `unzip` (Extracting ZIP files)
|
||||||
|
- `policycoreutils-python-utils` (SELinux tools, required if SELinux is enabled)
|
||||||
|
- `firewalld` (For firewall management, if needed)
|
||||||
|
|
||||||
|
**Optional Debugging & Performance Tools**
|
||||||
|
|
||||||
|
- `php-pecl-apcu` (APC User Cache for PHP)
|
||||||
|
- `php-pecl-memcached` (Memcached support)
|
||||||
|
- `php-pecl-redis` (Redis support)
|
||||||
|
|
||||||
|
5. **Ensure Hostname**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nano /etc/hosts
|
||||||
|
# Add line: 127.0.1.1 <hostname-intranet>
|
||||||
|
nano /etc/hostname
|
||||||
|
# Set to: <hostname-intranet>
|
||||||
|
hostnamectl set-hostname <hostname-intranet>
|
||||||
|
```
|
||||||
|
|
||||||
|
6. **Add Users and set Credentials**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
passwd root
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
adduser <username>
|
||||||
|
passwd <username>
|
||||||
|
|
||||||
|
groupadd sudo
|
||||||
|
usermod -aG sudo <username>
|
||||||
|
nano /etc/sudoers
|
||||||
|
```
|
||||||
|
|
||||||
|
```
|
||||||
|
## Allows people in group sudo to run all commands
|
||||||
|
%sudo ALL=(ALL) ALL
|
||||||
|
```
|
||||||
|
|
||||||
|
7. **Setup SSH Connectors**
|
||||||
|
|
||||||
|
- Configure SSH connectors as per your setup script to establish secure connections to the server.
|
||||||
|
|
||||||
|
8. **Test users, SSH, and sudo**
|
||||||
|
|
||||||
|
1. **Transfer SSH keys for User**
|
||||||
|
2. **Connect as User with SSH key**
|
||||||
|
3. **Test sudo**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo su -
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Disconnect as root**
|
||||||
|
|
||||||
|
9. **Secure SSH**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nano /etc/ssh/sshd_config
|
||||||
|
```
|
||||||
|
|
||||||
|
```ini
|
||||||
|
PermitRootLogin no
|
||||||
|
PasswordAuthentication no
|
||||||
|
ChallengeResponseAuthentication no
|
||||||
|
```
|
||||||
|
|
||||||
|
**Restart SSH**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl restart sshd
|
||||||
|
```
|
||||||
|
|
||||||
|
10. **Configure Firewall**
|
||||||
|
|
||||||
|
**Open ports**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
firewall-cmd --permanent --zone=public --add-service=ssh
|
||||||
|
firewall-cmd --permanent --zone=public --add-service=http
|
||||||
|
firewall-cmd --permanent --zone=public --add-service=https
|
||||||
|
firewall-cmd --permanent --zone=public --add-port=<wireguard-port>/udp
|
||||||
|
```
|
||||||
|
|
||||||
|
**Reload firewall to apply changes**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
firewall-cmd --reload
|
||||||
|
```
|
||||||
|
|
||||||
|
**Enable and start firewall**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl enable firewalld
|
||||||
|
systemctl start firewalld
|
||||||
|
```
|
||||||
|
|
||||||
|
11. **Start Apache And MariaDB**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl enable httpd --now
|
||||||
|
systemctl enable mariadb --now
|
||||||
|
```
|
||||||
|
|
||||||
|
Now, your web server is available at `http://<hostname-intranet>`.
|
||||||
|
|
||||||
|
12. **Create PHP test page**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
echo "<?php phpinfo() ?>" > /var/www/html/info.php
|
||||||
|
```
|
||||||
|
|
||||||
|
Now, your web server php information is available at `http://<hostname-intranet>/info.php`.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rm /var/www/html/info.php
|
||||||
|
```
|
||||||
|
|
||||||
|
13. **Secure MariaDB Installation**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mysql_secure_installation
|
||||||
|
```
|
||||||
|
|
||||||
|
14. **Creating the new Database**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mariadb
|
||||||
|
```
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE DATABASE wordpress;
|
||||||
|
CREATE USER `admin`@`localhost` IDENTIFIED BY '<database-password>';
|
||||||
|
GRANT ALL ON wordpress.* TO `admin`@`localhost`;
|
||||||
|
FLUSH PRIVILEGES;
|
||||||
|
EXIT;
|
||||||
|
```
|
||||||
|
|
||||||
|
15. **Download and Extract WordPress**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl https://wordpress.org/latest.tar.gz --output wordpress.tar.gz
|
||||||
|
tar xf wordpress.tar.gz
|
||||||
|
cp -r wordpress/* /var/www/html/
|
||||||
|
```
|
||||||
|
|
||||||
|
16. **Download and Install WordPress CLI**
|
||||||
|
|
||||||
|
As root
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -O https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar
|
||||||
|
php wp-cli.phar --info
|
||||||
|
chmod +x wp-cli.phar
|
||||||
|
mv wp-cli.phar /usr/local/bin/wp
|
||||||
|
wp --info
|
||||||
|
```
|
||||||
|
|
||||||
|
As user **Enable Tab Completion**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
su -s /bin/bash -l apache
|
||||||
|
mkdir -p ~/.wp-cli
|
||||||
|
wp cli completions --shell=bash > ~/.wp-cli/wp-completion.bash
|
||||||
|
echo 'source ~/.wp-cli/wp-completion.bash' >> ~/.bashrc
|
||||||
|
source ~/.bashrc
|
||||||
|
```
|
||||||
|
|
||||||
|
17. **Modify Permissions**
|
||||||
|
|
||||||
|
Set appropriate ownership and adjust the SELinux security context for WordPress files:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
chown -R apache:apache /var/www/html
|
||||||
|
chmod -R 755 /var/www/html
|
||||||
|
```
|
||||||
|
|
||||||
|
Enable Apache's ability to establish network connections, allowing WordPress to download updates and plugins:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
setsebool -P httpd_can_network_connect true
|
||||||
|
```
|
||||||
|
|
||||||
|
18. **Allow Override**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nano /etc/httpd/conf/httpd.conf
|
||||||
|
```
|
||||||
|
|
||||||
|
Allow Overrides on `/var/www` and `/var/www/html`:
|
||||||
|
|
||||||
|
```apache
|
||||||
|
AllowOverride All
|
||||||
|
```
|
||||||
|
|
||||||
|
19. **Configure Wordpress**
|
||||||
|
|
||||||
|
Now, visit `http://<hostname-intranet>` to follow the wordpress configuration.
|
||||||
|
|
||||||
|
20. **Configure WireGuard**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nano /etc/wireguard/proxy-lan.conf
|
||||||
|
systemctl enable wg-quick@proxy-lan --now
|
||||||
|
wg show
|
||||||
|
```
|
||||||
|
|
||||||
|
21. **Back-up post installation**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id> --state stopped"
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump <container-id> --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup post installation\""
|
||||||
|
```
|
||||||
|
|
||||||
|
22. **Start the server**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "pct start <container-id>"
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id> --state started --max_relocate 3 --max_restart 3"
|
||||||
|
```
|
||||||
481
setups-drafts/oracle_setup_oro.md
Normal file
481
setups-drafts/oracle_setup_oro.md
Normal file
@ -0,0 +1,481 @@
|
|||||||
|
# Installing Oro on Oracle Linux
|
||||||
|
|
||||||
|
## Table of Contents
|
||||||
|
|
||||||
|
- [Installing Oro on Oracle Linux](#installing-oro-on-oracle-linux)
|
||||||
|
- [Table of Contents](#table-of-contents)
|
||||||
|
- [Disclaimer: Incomplete Guide](#disclaimer-incomplete-guide)
|
||||||
|
- [Introduction](#introduction)
|
||||||
|
- [Prerequisites](#prerequisites)
|
||||||
|
- [What This Guide Covers](#what-this-guide-covers)
|
||||||
|
- [Placeholders](#placeholders)
|
||||||
|
- [Important Warnings and Security Practices](#important-warnings-and-security-practices)
|
||||||
|
- [Useful Commands and Information](#useful-commands-and-information)
|
||||||
|
- [Documentation](#documentation)
|
||||||
|
- [Links](#links)
|
||||||
|
- [Software on the Machine](#software-on-the-machine)
|
||||||
|
- [Paths](#paths)
|
||||||
|
- [Proxmox Commands](#proxmox-commands)
|
||||||
|
- [SSH Connection](#ssh-connection)
|
||||||
|
- [Installation Procedure](#installation-procedure)
|
||||||
|
|
||||||
|
## Disclaimer: Incomplete Guide
|
||||||
|
|
||||||
|
This document is a draft and may contain incomplete, untested, or outdated information. It is a work in progress and has not been verified for accuracy or usability. Use this guide at your own discretion, and consider it as a reference for further development or exploration. Updates may follow in the future, but no guarantees are made.
|
||||||
|
|
||||||
|
## Introduction
|
||||||
|
|
||||||
|
This guide provides step-by-step instructions for installing and configuring Oro on a Oracle Linux server.
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
Before you begin the installation process, ensure that your Oracle Linux system meets the following requirements:
|
||||||
|
|
||||||
|
- Oracle Linux GNU/Linux 9 or later
|
||||||
|
- Access to a terminal with sudo privileges
|
||||||
|
- Basic familiarity with the command line interface
|
||||||
|
- Stable internet connection to download necessary packages
|
||||||
|
|
||||||
|
## What This Guide Covers
|
||||||
|
|
||||||
|
This guide covers the installation and configuration of Oro on a Oracle Linux server, along with additional setup tasks such as SSH connection management and Proxmox commands. It covers:
|
||||||
|
|
||||||
|
1. **Installation**: Installing Oro from the official repository.
|
||||||
|
2. **Configuration**: Configuring Oro to suit your environment and preferences.
|
||||||
|
3. **Setup**: Setting up Oro as a service and accessing it.
|
||||||
|
|
||||||
|
## Placeholders
|
||||||
|
|
||||||
|
Replace the placeholders below with the appropriate values for your setup:
|
||||||
|
|
||||||
|
- **User Details**
|
||||||
|
|
||||||
|
- Username: `<username>` (e.g., admin)
|
||||||
|
- Username - Hypervisor: `<username-hypervisor>` (e.g., admin)
|
||||||
|
|
||||||
|
- **Server Configuration**
|
||||||
|
|
||||||
|
- Server IP address: `<server-ip-address>` (e.g., 192.168.1.100)
|
||||||
|
- Hostname - Intranet: `<hostname-intranet>` (e.g., oro-server.domain.com)
|
||||||
|
- Hostname - Internet: `<hostname-internet>` (e.g., oro.domain.com)
|
||||||
|
- Hostname - Hypervisor: `<hostname-hypervisor>` (e.g., proxmox-hypervisor.domain.com)
|
||||||
|
- Hostname - Hypervisor NAS: `<hostname-hypervisor-nas>` (e.g., nas-server.domain.com)
|
||||||
|
- Name - Hypervisor NAS: `<name-hypervisor-nas>` (e.g., nas-server)
|
||||||
|
|
||||||
|
- **SSH Keys**
|
||||||
|
|
||||||
|
- SSH key - Proxmox: `<ssh-key-proxmox>` (e.g., /home/user/.ssh/id_rsa.pub)
|
||||||
|
- SSH key - Client: `<ssh-key-client>` (e.g., /home/user/.ssh/client_id_rsa.pub)
|
||||||
|
|
||||||
|
- **Networking**
|
||||||
|
|
||||||
|
- Wireguard port: `<wireguard-port>` (e.g., 51820)
|
||||||
|
|
||||||
|
- **Paths**
|
||||||
|
|
||||||
|
## Important Warnings and Security Practices
|
||||||
|
|
||||||
|
Before executing any commands in this documentation, please adhere to the following guidelines to ensure the security and integrity of the system:
|
||||||
|
|
||||||
|
1. **Execute Commands with Caution**: Always review and understand a command before executing it. Misuse of commands can lead to data loss or system instability.
|
||||||
|
2. **Backup Command Execution**: The backup command must be executed only by authorized users. Ensure that proper permissions are set to prevent unauthorized access to backup files.
|
||||||
|
3. **Regular Backups**: Maintain regular backups of all critical data. It is advisable to use automated backup solutions and verify backup integrity periodically.
|
||||||
|
4. **System Updates**: Regularly update the system and all installed packages to protect against vulnerabilities. Use the package manager responsibly to avoid potential conflicts.
|
||||||
|
5. **Monitor System Logs**: Continuously monitor system logs for any unusual activity. Use logging tools to help identify potential security breaches or system failures.
|
||||||
|
6. **User Permissions**: Ensure that user permissions are strictly managed. Limit access to sensitive commands and data to only those who need it to perform their job functions.
|
||||||
|
7. **Network Security**: Implement proper network security measures, such as firewalls and intrusion detection systems, to protect against external threats.
|
||||||
|
8. **Data Encryption**: Encrypt sensitive data at rest and in transit to prevent unauthorized access.
|
||||||
|
|
||||||
|
By following these practices, you will help maintain the security and stability of the system while minimizing the risk of data loss or compromise.
|
||||||
|
|
||||||
|
## Useful Commands and Information
|
||||||
|
|
||||||
|
### Documentation
|
||||||
|
|
||||||
|
- [Documentation Overview](https://doc.oroinc.com/)
|
||||||
|
- [Documentation community edition](https://doc.oroinc.com/backend/setup/dev-environment/community-edition/)
|
||||||
|
- [Installation](https://doc.oroinc.com/backend/setup/installation/)
|
||||||
|
|
||||||
|
### Links
|
||||||
|
|
||||||
|
- [Oro appliance](https://<hostname-internet>/)
|
||||||
|
|
||||||
|
### Software on the Machine
|
||||||
|
|
||||||
|
- **Operating System**: Oracle Linux
|
||||||
|
- **Web Server**:
|
||||||
|
- **Security**: GnuPG, WireGuard, firewalld
|
||||||
|
- **Other**: Git, sudo
|
||||||
|
|
||||||
|
### Paths
|
||||||
|
|
||||||
|
- **Oro Configuration**:
|
||||||
|
- **Oro Work Path**:
|
||||||
|
|
||||||
|
### Proxmox Commands
|
||||||
|
|
||||||
|
**List available Proxmox templates**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh <username>@<hostname-hypervisor-nas> "ls /mnt/proxmox/template/cache/"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Create the container**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "pct create 100 <name-hypervisor-nas>:vztmpl/oracle-9-sshnano_20240603_amd64.tar.zst --hostname <hostname-intranet> --cores 2 --memory 4096 --swap 2048 --net0 name=net0,bridge=vmbr0,ip=dhcp,firewall=1 --rootfs <name-hypervisor-nas>:100 --unprivileged 1 --features nesting=1 --ssh-public-keys <ssh-key-proxmox>"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Backup**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump 100 --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
|
||||||
|
```
|
||||||
|
|
||||||
|
**Set the state of the Proxmox HA Manager for Container 100**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager add ct:100"
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager remove ct:100"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Set the state and limits of the Proxmox Container 100 in the HA Manager**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:100 --state started --max_relocate 3 --max_restart 3"
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:100 --state stopped"
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "pct start 100"
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "pct stop 100"
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "pct reboot 100"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Destroy the Proxmox Container 100 forcefully**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "pct destroy 100 --force --purge"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Move the Proxmox Container 100 to another host**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "pct migrate 100 hv2"
|
||||||
|
```
|
||||||
|
|
||||||
|
### SSH Connection
|
||||||
|
|
||||||
|
**Connection with specific keys**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh -i <ssh-key-client> root@<hostname-intranet>
|
||||||
|
ssh -i <ssh-key-client> root@<server-ip-address>
|
||||||
|
ssh -i <ssh-key-client> <username>@<hostname-intranet>
|
||||||
|
ssh -i <ssh-key-client> <username>@<server-ip-address>
|
||||||
|
```
|
||||||
|
|
||||||
|
**Remove offending keys from known_hosts**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh-keygen -f "/home/<username>/.ssh/known_hosts" -R "<hostname-intranet>"
|
||||||
|
ssh-keygen -f "/home/<username>/.ssh/known_hosts" -R "<server-ip-address>"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Copy SSH public key to remote host**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh-copy-id -i <ssh-key-client> root@<server-ip-address>
|
||||||
|
ssh-copy-id -i <ssh-key-client> root@<hostname-intranet>
|
||||||
|
ssh-copy-id -i <ssh-key-client> <username>@<server-ip-address>
|
||||||
|
ssh-copy-id -i <ssh-key-client> <username>@<hostname-intranet>
|
||||||
|
```
|
||||||
|
|
||||||
|
**Transfer SSH keys and files**
|
||||||
|
|
||||||
|
## Installation Procedure
|
||||||
|
|
||||||
|
1. **Fresh Oracle Linux Installation**
|
||||||
|
|
||||||
|
- Install a fresh Oracle Linux operating system on your new server following the standard installation procedure.
|
||||||
|
|
||||||
|
2. **Backup before starting**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump 100 --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Install Required Dependencies**
|
||||||
|
|
||||||
|
**Upgrade the base system**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dnf update
|
||||||
|
```
|
||||||
|
|
||||||
|
**Enable EPEL repository**
|
||||||
|
Extra Package for Enterprise Linux repository has packages like Apache and Nginx
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dnf install epel-release
|
||||||
|
```
|
||||||
|
|
||||||
|
**Enable Postgres repository**
|
||||||
|
|
||||||
|
Get instructions and urls in their [documentation](https://www.postgresql.org/download/linux/redhat/)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Install the repository RPM:
|
||||||
|
dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-x86_64/pgdg-redhat-repo-latest.noarch.rpm
|
||||||
|
|
||||||
|
# Disable the built-in PostgreSQL module:
|
||||||
|
dnf -qy module disable postgresql
|
||||||
|
|
||||||
|
# Install PostgreSQL:
|
||||||
|
dnf install -y postgresql15-server
|
||||||
|
|
||||||
|
# Optionally initialize the database and enable automatic start:
|
||||||
|
/usr/pgsql-15/bin/postgresql-15-setup initdb
|
||||||
|
systemctl enable postgresql-15
|
||||||
|
systemctl start postgresql-15
|
||||||
|
```
|
||||||
|
|
||||||
|
**Enable Remi repository**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cat >"/etc/yum.repos.d/oropublic.repo" <<__EOF__
|
||||||
|
[oropublic]
|
||||||
|
name=OroPublic
|
||||||
|
baseurl=https://nexus.oro.cloud/repository/oropublic/8/x86_64/
|
||||||
|
enabled=1
|
||||||
|
gpgcheck=0
|
||||||
|
module_hotfixes=1
|
||||||
|
__EOF__
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dnf config-manager --set-enabled remi
|
||||||
|
```
|
||||||
|
|
||||||
|
**Enable oro repository**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dnf install https://rpms.remirepo.net/enterprise/remi-release-9.rpm
|
||||||
|
```
|
||||||
|
|
||||||
|
**Enable DNF streams**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dnf module list nginx
|
||||||
|
dnf module list nodejs
|
||||||
|
dnf module list php
|
||||||
|
|
||||||
|
dnf -y module enable nginx:1.24 nodejs:20 php:remi-8.3
|
||||||
|
dnf -y upgrade
|
||||||
|
```
|
||||||
|
|
||||||
|
**Install dependencies**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dnf install -y sudo nano findutils rsync psmisc wget glibc-langpack-en bzip2 unzip p7zip p7zip-plugins parallel patch nodejs npm git-core jq bc postgresql postgresql-server postgresql-contrib
|
||||||
|
dnf install -y gnupg wireguard-tools firewalld firewall-config
|
||||||
|
```
|
||||||
|
|
||||||
|
**Install Apache or Nginx**
|
||||||
|
Uncomment one of the following lines depending on the web server you prefer to use
|
||||||
|
|
||||||
|
**Apache**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dnf install -y httpd
|
||||||
|
```
|
||||||
|
|
||||||
|
**Nginx Configuration**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dnf install -y nginx
|
||||||
|
```
|
||||||
|
|
||||||
|
**PHP and modules**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dnf install -y php-common php-cli php-fpm php-opcache php-mbstring php-mysqlnd php-pgsql php-pdo php-json php-process php-ldap php-gd php-ctype php-curl php-fileinfo php-intl php-bcmath php-xml php-soap php-sodium php-openssl php-pcre php-simplexml php-tokenizer php-zip php-tidy php-imap php-pecl-zip php-pecl-mongodb
|
||||||
|
```
|
||||||
|
|
||||||
|
**nodejs**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dnf install -y nodejs
|
||||||
|
```
|
||||||
|
|
||||||
|
Verify Node.js and NPM versions
|
||||||
|
|
||||||
|
```bash
|
||||||
|
node -v
|
||||||
|
npm -v
|
||||||
|
```
|
||||||
|
|
||||||
|
**Supervisor for process control**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dnf install -y supervisor
|
||||||
|
```
|
||||||
|
|
||||||
|
Enable and start Supervisor service
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl enable supervisord
|
||||||
|
systemctl start supervisord
|
||||||
|
```
|
||||||
|
|
||||||
|
**Redis**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dnf install -y redis
|
||||||
|
```
|
||||||
|
|
||||||
|
Enable and start Redis service
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl enable redis
|
||||||
|
systemctl start redis
|
||||||
|
```
|
||||||
|
|
||||||
|
**pngquant and jpegoptim**
|
||||||
|
dnf install -y pngquant jpegoptim
|
||||||
|
|
||||||
|
4. **Ensure Hostname**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nano /etc/hosts
|
||||||
|
# Add line: 127.0.1.1 <hostname-intranet>
|
||||||
|
nano /etc/hostname
|
||||||
|
# Set to: <hostname-intranet>
|
||||||
|
hostnamectl set-hostname <hostname-intranet>
|
||||||
|
```
|
||||||
|
|
||||||
|
5. **Add Users and set Credentials**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
passwd -f root
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
adduser <username>
|
||||||
|
passwd -f <username>
|
||||||
|
groupadd sudo
|
||||||
|
usermod -aG sudo <username>
|
||||||
|
nano /etc/sudoers
|
||||||
|
```
|
||||||
|
|
||||||
|
```
|
||||||
|
## Allows people in group sudo to run all commands
|
||||||
|
%sudo ALL=(ALL) ALL
|
||||||
|
```
|
||||||
|
|
||||||
|
6. **Setup SSH Connectors**
|
||||||
|
|
||||||
|
- Configure SSH connectors as per your setup script to establish secure connections to the server.
|
||||||
|
|
||||||
|
7. **Test users, SSH, and sudo**
|
||||||
|
|
||||||
|
1. **Transfer SSH keys for User**
|
||||||
|
2. **Connect as User with SSH key**
|
||||||
|
3. **Test sudo**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo su -
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Disconnect as root**
|
||||||
|
|
||||||
|
8. **Secure SSH**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nano /etc/ssh/sshd_config
|
||||||
|
```
|
||||||
|
|
||||||
|
```ini
|
||||||
|
PermitRootLogin no
|
||||||
|
PasswordAuthentication no
|
||||||
|
ChallengeResponseAuthentication no
|
||||||
|
```
|
||||||
|
|
||||||
|
**Restart SSH**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl restart sshd
|
||||||
|
```
|
||||||
|
|
||||||
|
9. **Configure Firewall**
|
||||||
|
|
||||||
|
**Open ports**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
firewall-cmd --permanent --add-service=ssh
|
||||||
|
firewall-cmd --permanent --add-service=http
|
||||||
|
firewall-cmd --permanent --add-service=https
|
||||||
|
firewall-cmd --permanent --add-port=<wireguard-port>/udp
|
||||||
|
```
|
||||||
|
|
||||||
|
**Reload firewall to apply changes**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
firewall-cmd --reload
|
||||||
|
```
|
||||||
|
|
||||||
|
**Enable and start firewall**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl enable firewalld
|
||||||
|
systemctl start firewalld
|
||||||
|
```
|
||||||
|
|
||||||
|
10. **Configure PHP**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
php --ini
|
||||||
|
nano /etc/php.ini
|
||||||
|
```
|
||||||
|
|
||||||
|
**Add or update the following settings**
|
||||||
|
|
||||||
|
````ini
|
||||||
|
date.timezone = America/Toronto
|
||||||
|
detect_unicode = Off
|
||||||
|
memory_limit = 1G
|
||||||
|
max_execution_time = <appropriate-time>
|
||||||
|
|
||||||
|
If xdebug is installed, update or add these settings
|
||||||
|
```ini
|
||||||
|
xdebug.scream = Off
|
||||||
|
xdebug.show_exception_trace = Off
|
||||||
|
xdebug.max_nesting_level = 100
|
||||||
|
````
|
||||||
|
|
||||||
|
11. **Install Oro**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dnf -y --setopt=install_weak_deps=False --best --nogpgcheck install oro-nginx oro-nginx-mod-http-cache_purge oro-nginx-mod-http-cookie_flag oro-nginx-mod-http-geoip oro-nginx-mod-http-gridfs oro-nginx-mod-http-headers_more oro-nginx-mod-http-naxsi oro-nginx-mod-http-njs oro-nginx-mod-http-pagespeed oro-nginx-mod-http-sorted_querystring oro-nginx-mod-http-testcookie_access oro-nginx-mod-http-xslt-filter
|
||||||
|
```
|
||||||
|
|
||||||
|
12. **Enable Oro service**
|
||||||
|
|
||||||
|
13. **Setup nginx proxy**
|
||||||
|
|
||||||
|
14. **Configure SSL**
|
||||||
|
|
||||||
|
15. **Correct permissions**
|
||||||
|
|
||||||
|
16. **Run the installer**
|
||||||
|
|
||||||
|
17. **Verify installation**
|
||||||
|
|
||||||
|
18. **Back-up post installation**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:100 --state stopped"
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump 100 --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup post installation\""
|
||||||
|
```
|
||||||
|
|
||||||
|
19. **Start the server**
|
||||||
|
```bash
|
||||||
|
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:100 --state started --max_relocate 3 --max_restart 3"
|
||||||
|
```
|
||||||
@ -68,7 +68,6 @@ Replace the placeholders below with the appropriate values for your setup:
|
|||||||
- Hostname - Hypervisor: `<hostname-hypervisor>` (e.g., proxmox-hypervisor.domain.com)
|
- Hostname - Hypervisor: `<hostname-hypervisor>` (e.g., proxmox-hypervisor.domain.com)
|
||||||
- Hostname - Hypervisor NAS: `<hostname-hypervisor-nas>` (e.g., nas-server.domain.com)
|
- Hostname - Hypervisor NAS: `<hostname-hypervisor-nas>` (e.g., nas-server.domain.com)
|
||||||
- Name - Hypervisor NAS: `<name-hypervisor-nas>` (e.g., nas-server)
|
- Name - Hypervisor NAS: `<name-hypervisor-nas>` (e.g., nas-server)
|
||||||
- Container ID - Hypervisor: `<container-id-hypervisor>` (e.g., 100)
|
|
||||||
|
|
||||||
- **SSH Keys**
|
- **SSH Keys**
|
||||||
|
|
||||||
@ -141,40 +140,40 @@ ssh <username>@<hostname-hypervisor-nas> "ls /mnt/proxmox/template/cache/"
|
|||||||
**Create the container**
|
**Create the container**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "pct create <container-id-hypervisor> <name-hypervisor-nas>:vztmpl/debian-12-upgraded_12.5_amd64.tar.zst --hostname <hostname-intranet> --cores 2 --memory 2048 --swap 2048 --net0 name=net0,bridge=vmbr0,ip=dhcp,firewall=1 --rootfs <name-hypervisor-nas>:250 --unprivileged 1 --features nesting=1 --ssh-public-keys <ssh-key-proxmox> --start 1"
|
ssh <username-hypervisor>@<hostname-hypervisor> "pct create 100 <name-hypervisor-nas>:vztmpl/debian-12-upgraded_12.5_amd64.tar.zst --hostname <hostname-intranet> --cores 2 --memory 2048 --swap 2048 --net0 name=net0,bridge=vmbr0,ip=dhcp,firewall=1 --rootfs <name-hypervisor-nas>:250 --unprivileged 1 --features nesting=1 --ssh-public-keys <ssh-key-proxmox> --start 1"
|
||||||
```
|
```
|
||||||
|
|
||||||
**Backup**
|
**Backup**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump <container-id-hypervisor> --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
|
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump 100 --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
|
||||||
```
|
```
|
||||||
|
|
||||||
**Set the state of the Proxmox HA Manager for Container <container-id-hypervisor>**
|
**Set the state of the Proxmox HA Manager for Container 100**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager add ct:<container-id-hypervisor>"
|
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager add ct:100"
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager remove ct:<container-id-hypervisor>"
|
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager remove ct:100"
|
||||||
```
|
```
|
||||||
|
|
||||||
**Set the state and limits of the Proxmox Container <container-id-hypervisor> in the HA Manager**
|
**Set the state and limits of the Proxmox Container 100 in the HA Manager**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id-hypervisor> --state started --max_relocate 3 --max_restart 3"
|
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:100 --state started --max_relocate 3 --max_restart 3"
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id-hypervisor> --state stopped"
|
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:100 --state stopped"
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "pct reboot <container-id-hypervisor>"
|
ssh <username-hypervisor>@<hostname-hypervisor> "pct reboot 100"
|
||||||
```
|
```
|
||||||
|
|
||||||
**Destroy the Proxmox Container <container-id-hypervisor> forcefully**
|
**Destroy the Proxmox Container 100 forcefully**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "pct destroy <container-id-hypervisor> --force --purge"
|
ssh <username-hypervisor>@<hostname-hypervisor> "pct destroy 100 --force --purge"
|
||||||
```
|
```
|
||||||
|
|
||||||
**Move the Proxmox Container <container-id-hypervisor> to another host**
|
**Move the Proxmox Container 100 to another host**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "pct migrate <container-id-hypervisor> hv2"
|
ssh <username-hypervisor>@<hostname-hypervisor> "pct migrate 100 hv2"
|
||||||
```
|
```
|
||||||
|
|
||||||
### SSH Connection
|
### SSH Connection
|
||||||
@ -221,7 +220,7 @@ chown -R aptly:aptly /home/aptly/.ssh/
|
|||||||
2. **Backup before starting**
|
2. **Backup before starting**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump <container-id-hypervisor> --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
|
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump 100 --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
|
||||||
```
|
```
|
||||||
|
|
||||||
3. **Install Required Dependencies**
|
3. **Install Required Dependencies**
|
||||||
@ -624,11 +623,11 @@ chown -R aptly:aptly /home/aptly/.ssh/
|
|||||||
14. **Back-up post installation**
|
14. **Back-up post installation**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id-hypervisor> --state stopped"
|
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:100 --state stopped"
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump <container-id-hypervisor> --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup post installation\""
|
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump 100 --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup post installation\""
|
||||||
```
|
```
|
||||||
|
|
||||||
15. **Start the server**
|
15. **Start the server**
|
||||||
```bash
|
```bash
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id-hypervisor> --state started --max_relocate 3 --max_restart 3"
|
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:100 --state started --max_relocate 3 --max_restart 3"
|
||||||
```
|
```
|
||||||
|
|||||||
@ -1,626 +0,0 @@
|
|||||||
# Installing Dolibarr on Debian
|
|
||||||
|
|
||||||
## Table of Contents
|
|
||||||
|
|
||||||
- [Installing Dolibarr on Debian](#installing-dolibarr-on-debian)
|
|
||||||
- [Table of Contents](#table-of-contents)
|
|
||||||
- [Introduction](#introduction)
|
|
||||||
- [Why Dolibarr?](#why-dolibarr)
|
|
||||||
- [Prerequisites](#prerequisites)
|
|
||||||
- [What This Guide Covers](#what-this-guide-covers)
|
|
||||||
- [Placeholders](#placeholders)
|
|
||||||
- [Important Warnings and Security Practices](#important-warnings-and-security-practices)
|
|
||||||
- [Useful Commands and Information](#useful-commands-and-information)
|
|
||||||
- [Documentation](#documentation)
|
|
||||||
- [Links](#links)
|
|
||||||
- [Software on the Machine](#software-on-the-machine)
|
|
||||||
- [Paths](#paths)
|
|
||||||
- [Proxmox Commands](#proxmox-commands)
|
|
||||||
- [SSH Connection](#ssh-connection)
|
|
||||||
- [Installation Procedure](#installation-procedure)
|
|
||||||
|
|
||||||
## Introduction
|
|
||||||
|
|
||||||
Welcome to the installation guide for Dolibarr on Debian. Dolibarr is an open-source ERP and CRM platform designed for small and medium-sized businesses, offering features such as invoicing, accounting, inventory management, and customer relationship management through a clean, web-based interface. By following this guide, you’ll be able to install and configure Dolibarr on your Debian system efficiently.
|
|
||||||
|
|
||||||
## Why Dolibarr?
|
|
||||||
|
|
||||||
Dolibarr is a flexible and open-source ERP/CRM solution that allows you to keep full control over your business data and infrastructure. It covers a wide range of needs for small and medium-sized organizations, including invoicing, accounting, customer management, inventory, and project tracking, all from a single web interface. Being self-hosted, Dolibarr gives you data ownership, transparency, and the ability to customize or extend the platform to fit your workflows without relying on third-party SaaS providers.
|
|
||||||
|
|
||||||
## Prerequisites
|
|
||||||
|
|
||||||
Before you begin the installation process, ensure that your Debian system meets the following requirements:
|
|
||||||
|
|
||||||
- Debian GNU/Linux 9 (Stretch) or later
|
|
||||||
- Access to a terminal with sudo privileges
|
|
||||||
- Basic familiarity with the command line interface
|
|
||||||
- Stable internet connection to download necessary packages
|
|
||||||
|
|
||||||
## What This Guide Covers
|
|
||||||
|
|
||||||
This guide will walk you through the installation of Dolibarr on Debian step by step, covering:
|
|
||||||
|
|
||||||
1. **Installation**: Installing Dolibarr from the official Git repository.
|
|
||||||
2. **Configuration**: Adjusting Dolibarr and system settings to match your environment.
|
|
||||||
3. **Web Setup**: Configuring Dolibarr with Apache and accessing the web installer.
|
|
||||||
4. **Security**: Hardening Debian, PHP, and Dolibarr for a production setup.
|
|
||||||
|
|
||||||
## Placeholders
|
|
||||||
|
|
||||||
Replace the placeholders below with the appropriate values for your setup:
|
|
||||||
|
|
||||||
- **User Details**
|
|
||||||
|
|
||||||
- Username: `<username>` (e.g., admin)
|
|
||||||
- Username - Hypervisor: `<username-hypervisor>` (e.g., admin)
|
|
||||||
|
|
||||||
- **Server Configuration**
|
|
||||||
|
|
||||||
- Server IP address: `<server-ip-address>` (e.g., 192.168.1.100)
|
|
||||||
- Hostname - Intranet: `<hostname-intranet>` (e.g., dolibarr-server.domain.com)
|
|
||||||
- Hostname - Internet: `<hostname-internet>` (e.g., dolibarr.domain.com)
|
|
||||||
- Hostname - Hypervisor: `<hostname-hypervisor>` (e.g., proxmox-hypervisor.domain.com)
|
|
||||||
- Hostname - Hypervisor NAS: `<hostname-hypervisor-nas>` (e.g., nas-server.domain.com)
|
|
||||||
- Name - Hypervisor NAS: `<name-hypervisor-nas>` (e.g., nas-server)
|
|
||||||
- Container ID - Hypervisor: `<container-id-hypervisor>` (e.g., 100)
|
|
||||||
|
|
||||||
- **SSH Keys**
|
|
||||||
|
|
||||||
- SSH key - Proxmox: `<ssh-key-proxmox>` (e.g., /home/user/.ssh/id_rsa.pub)
|
|
||||||
- SSH key - Client: `<ssh-key-client>` (e.g., /home/user/.ssh/client_id_rsa.pub)
|
|
||||||
|
|
||||||
- **Networking**
|
|
||||||
|
|
||||||
- Wireguard port: `<wireguard-port>` (e.g., 51820)
|
|
||||||
|
|
||||||
- **Dolibarr specifics post-install**
|
|
||||||
|
|
||||||
- Contract paths: `<dolibarr-contract-paths-source>` (e.g., /home/username/contracts/)
|
|
||||||
- Fonts paths: `<dolibarr-fonts-paths-source>` (e.g., /home/username/fonts/)
|
|
||||||
|
|
||||||
## Important Warnings and Security Practices
|
|
||||||
|
|
||||||
Before executing any commands in this documentation, please adhere to the following guidelines to ensure the security and integrity of the system:
|
|
||||||
|
|
||||||
1. **Execute Commands with Caution**: Always review and understand a command before executing it. Misuse of commands can lead to data loss or system instability.
|
|
||||||
2. **Backup Command Execution**: The backup command must be executed only by authorized users. Ensure that proper permissions are set to prevent unauthorized access to backup files.
|
|
||||||
3. **Regular Backups**: Maintain regular backups of all critical data. It is advisable to use automated backup solutions and verify backup integrity periodically.
|
|
||||||
4. **System Updates**: Regularly update the system and all installed packages to protect against vulnerabilities. Use the package manager responsibly to avoid potential conflicts.
|
|
||||||
5. **Monitor System Logs**: Continuously monitor system logs for any unusual activity. Use logging tools to help identify potential security breaches or system failures.
|
|
||||||
6. **User Permissions**: Ensure that user permissions are strictly managed. Limit access to sensitive commands and data to only those who need it to perform their job functions.
|
|
||||||
7. **Network Security**: Implement proper network security measures, such as firewalls and intrusion detection systems, to protect against external threats.
|
|
||||||
8. **Data Encryption**: Encrypt sensitive data at rest and in transit to prevent unauthorized access.
|
|
||||||
|
|
||||||
By following these practices, you will help maintain the security and stability of the system while minimizing the risk of data loss or compromise.
|
|
||||||
|
|
||||||
## Useful Commands and Information
|
|
||||||
|
|
||||||
### Documentation
|
|
||||||
|
|
||||||
- [Setup other](https://wiki.dolibarr.org/index.php?title=Setup_Other)
|
|
||||||
- [Create an ODT or ODS document template](https://wiki.dolibarr.org/index.php?title=Create_an_ODT_or_ODS_document_template)
|
|
||||||
- [Setup Other](https://wiki.dolibarr.org/index.php?title=Setup_Other)
|
|
||||||
- [List of releases, change log and compatibilities](https://wiki.dolibarr.org/index.php?title=List_of_releases,_change_log_and_compatibilities)
|
|
||||||
- [GitHub](https://github.com/Dolibarr/dolibarr)
|
|
||||||
|
|
||||||
### Links
|
|
||||||
|
|
||||||
- [https](https://<hostname-intranet>/)
|
|
||||||
- [http](http://<hostname-intranet>/)
|
|
||||||
- [local https](https://<hostname-intranet>/)
|
|
||||||
- [local http](http://<hostname-intranet>/)
|
|
||||||
- [<server-ip-address>](http://<server-ip-address>/)
|
|
||||||
|
|
||||||
### Software on the Machine
|
|
||||||
|
|
||||||
- **Operating System**: Debian
|
|
||||||
- **Database**: MariaDB (configured as MySQL)
|
|
||||||
- **Web Server**: Apache
|
|
||||||
- **Security**: GnuPG, WireGuard, UFW
|
|
||||||
- **Other**: Git, sudo, certbot (for SSL certificate management)
|
|
||||||
|
|
||||||
### Paths
|
|
||||||
|
|
||||||
- **Dolibarr Configuration**: `/etc/dolibarr/app.ini`
|
|
||||||
- **Dolibarr Work Path**: `/var/lib/dolibarr`
|
|
||||||
|
|
||||||
### Proxmox Commands
|
|
||||||
|
|
||||||
**List available Proxmox templates**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ssh <username>@<hostname-hypervisor-nas> "ls /mnt/proxmox/template/cache/"
|
|
||||||
```
|
|
||||||
|
|
||||||
**Create the container**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "pct create <container-id-hypervisor> <name-hypervisor-nas>:vztmpl/debian-12-upgraded_12.5_amd64.tar.zst --hostname <hostname-intranet> --cores 4 --memory 4096 --swap 2048 --net0 name=net0,bridge=vmbr0,ip=dhcp,firewall=1 --rootfs <name-hypervisor-nas>:250 --unprivileged 1 --features nesting=1 --ssh-public-keys <ssh-key-proxmox>"
|
|
||||||
```
|
|
||||||
|
|
||||||
**Backup**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump <container-id-hypervisor> --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
|
|
||||||
```
|
|
||||||
|
|
||||||
**Set the state of the Proxmox HA Manager for Container <container-id-hypervisor>**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager add ct:<container-id-hypervisor>"
|
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager remove ct:<container-id-hypervisor>"
|
|
||||||
```
|
|
||||||
|
|
||||||
**Set the state and limits of the Proxmox Container <container-id-hypervisor> in the HA Manager**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id-hypervisor> --state started --max_relocate 3 --max_restart 3"
|
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id-hypervisor> --state stopped"
|
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "pct start <container-id-hypervisor>"
|
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "pct stop <container-id-hypervisor>"
|
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "pct reboot <container-id-hypervisor>"
|
|
||||||
```
|
|
||||||
|
|
||||||
**Destroy the Proxmox Container <container-id-hypervisor> forcefully**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "pct destroy <container-id-hypervisor> --force --purge"
|
|
||||||
```
|
|
||||||
|
|
||||||
**Move the Proxmox Container <container-id-hypervisor> to another host**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "pct migrate <container-id-hypervisor> hv2"
|
|
||||||
```
|
|
||||||
|
|
||||||
### SSH Connection
|
|
||||||
|
|
||||||
**Connection with specific keys**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ssh -i <ssh-key-client> root@<hostname-intranet>
|
|
||||||
ssh -i <ssh-key-client> root@<server-ip-address>
|
|
||||||
ssh -i <ssh-key-client> <username>@<hostname-intranet>
|
|
||||||
ssh -i <ssh-key-client> <username>@<server-ip-address>
|
|
||||||
```
|
|
||||||
|
|
||||||
**Remove offending keys from known_hosts**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ssh-keygen -f "/home/<username>/.ssh/known_hosts" -R "<hostname-intranet>"
|
|
||||||
ssh-keygen -f "/home/<username>/.ssh/known_hosts" -R "<server-ip-address>"
|
|
||||||
```
|
|
||||||
|
|
||||||
**Copy SSH public key to remote host**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ssh-copy-id -i <ssh-key-client> root@<server-ip-address>
|
|
||||||
ssh-copy-id -i <ssh-key-client> root@<hostname-intranet>
|
|
||||||
ssh-copy-id -i <ssh-key-client> <username>@<server-ip-address>
|
|
||||||
ssh-copy-id -i <ssh-key-client> <username>@<hostname-intranet>
|
|
||||||
```
|
|
||||||
|
|
||||||
**Transfer SSH keys and files**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
scp /home/<username>/.ssh/<username>* <username>@<hostname-intranet>:/home/<username>/.ssh/
|
|
||||||
mkdir -p /home/<username>/.ssh/
|
|
||||||
mv /home/<username>/.ssh/<username>* /home/<username>/.ssh/
|
|
||||||
chown -R <username>:<username> /home/<username>/.ssh/
|
|
||||||
cat /home/<username>/.ssh/<username>.pub >> /home/<username>/.ssh/authorized_keys
|
|
||||||
```
|
|
||||||
|
|
||||||
## Installation Procedure
|
|
||||||
|
|
||||||
1. **Fresh Debian Installation**
|
|
||||||
|
|
||||||
- Install a fresh Debian operating system on your new server following the standard installation procedure.
|
|
||||||
|
|
||||||
2. **Backup before starting**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump <container-id-hypervisor> --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
|
|
||||||
```
|
|
||||||
|
|
||||||
3. **Install Required Dependencies**
|
|
||||||
|
|
||||||
**Upgrade the base system**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
apt update
|
|
||||||
apt upgrade -y
|
|
||||||
```
|
|
||||||
|
|
||||||
**Install dependencies**
|
|
||||||
|
|
||||||
- [Prerequisites](https://wiki.dolibarr.org/index.php?title=Prerequisites)
|
|
||||||
- [Dolibarr downloads](https://www.dolibarr.org/downloads.php)
|
|
||||||
- [Dolibarr Sourceforge Debian packages](https://sourceforge.net/projects/dolibarr/files/Dolibarr%20installer%20for%20Debian-Ubuntu%20%28DoliDeb%29/20.0.3/)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
apt install -y git apache2 mariadb-server php php-mysql php-mbstring php-gd php-curl php-xml php-intl php-imap php-zip libapache2-mod-php sudo
|
|
||||||
apt install -y libreoffice-common libreoffice-writer --no-install-recommends
|
|
||||||
apt install -y fonts-dejavu fonts-liberation ttf-mscorefonts-installer
|
|
||||||
|
|
||||||
apt install -y gpg
|
|
||||||
```
|
|
||||||
|
|
||||||
4. **Ensure Hostname**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nano /etc/hosts
|
|
||||||
# Add line: 127.0.1.1 <hostname-intranet>
|
|
||||||
nano /etc/hostname
|
|
||||||
# Set to: <hostname-intranet>
|
|
||||||
hostnamectl set-hostname <hostname-intranet>
|
|
||||||
```
|
|
||||||
|
|
||||||
5. **Add Users**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
adduser <username>
|
|
||||||
usermod -aG sudo <username>
|
|
||||||
```
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mkdir -p /home/www-data
|
|
||||||
chown www-data:www-data /home/www-data
|
|
||||||
usermod -d /home/www-data www-data
|
|
||||||
usermod -s /bin/bash www-data
|
|
||||||
```
|
|
||||||
|
|
||||||
```bash
|
|
||||||
adduser --system --shell /bin/bash --gecos 'Dolibarr CRM' --group --disabled-password --home /home/www-data www-data
|
|
||||||
```
|
|
||||||
|
|
||||||
6. **Setup SSH Connectors**
|
|
||||||
|
|
||||||
- Configure SSH connectors as per your setup script to establish secure connections to the server.
|
|
||||||
|
|
||||||
7. **Test users, SSH, and sudo**
|
|
||||||
|
|
||||||
1. **Transfer SSH keys for User**
|
|
||||||
2. **Connect as User with SSH key**
|
|
||||||
3. **Test sudo**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sudo su -
|
|
||||||
```
|
|
||||||
|
|
||||||
4. **Disconnect as root**
|
|
||||||
|
|
||||||
8. **Secure SSH**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nano /etc/ssh/sshd_config
|
|
||||||
```
|
|
||||||
|
|
||||||
```ini
|
|
||||||
PermitRootLogin no
|
|
||||||
PasswordAuthentication no
|
|
||||||
ChallengeResponseAuthentication no
|
|
||||||
```
|
|
||||||
|
|
||||||
**Restart SSH**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
systemctl restart ssh
|
|
||||||
```
|
|
||||||
|
|
||||||
9. **Configure Firewall**
|
|
||||||
|
|
||||||
**Open ports**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ufw allow OpenSSH
|
|
||||||
ufw allow <wireguard-port>/udp
|
|
||||||
```
|
|
||||||
|
|
||||||
**Enable firewall**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ufw enable
|
|
||||||
```
|
|
||||||
|
|
||||||
10. **Configure Mariadb**
|
|
||||||
|
|
||||||
1. **Security**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mariadb-secure-installation
|
|
||||||
mariadb
|
|
||||||
```
|
|
||||||
|
|
||||||
2. **Mariadb Users**
|
|
||||||
|
|
||||||
```sql
|
|
||||||
SET old_passwords=0;
|
|
||||||
CREATE USER 'dolibarr' IDENTIFIED BY '';
|
|
||||||
```
|
|
||||||
|
|
||||||
```sql
|
|
||||||
CREATE USER IF NOT EXISTS 'dolibarr'@'192.168.1.%' IDENTIFIED BY '';
|
|
||||||
GRANT ALL PRIVILEGES ON *.* TO 'dolibarr'@'192.168.1.%' IDENTIFIED BY '' WITH GRANT OPTION;
|
|
||||||
GRANT ALL PRIVILEGES ON *.* TO 'dolibarr'@'localhost' IDENTIFIED BY '' WITH GRANT OPTION;
|
|
||||||
SET PASSWORD FOR 'dolibarr'@'localhost' = PASSWORD('');
|
|
||||||
SET PASSWORD FOR 'dolibarr'@'192.168.1.%' = PASSWORD('');
|
|
||||||
FLUSH PRIVILEGES;
|
|
||||||
```
|
|
||||||
|
|
||||||
3. **Create Dolibarr Database**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mariadb -u root -p
|
|
||||||
```
|
|
||||||
|
|
||||||
Inside the MariaDB shell:
|
|
||||||
|
|
||||||
```sql
|
|
||||||
CREATE DATABASE dolibarr;
|
|
||||||
CREATE USER 'dolibarruser'@'localhost' IDENTIFIED BY 'strongpassword';
|
|
||||||
GRANT ALL PRIVILEGES ON dolibarr.* TO 'dolibarruser'@'localhost';
|
|
||||||
FLUSH PRIVILEGES;
|
|
||||||
EXIT;
|
|
||||||
```
|
|
||||||
|
|
||||||
11. **Install Dolibarr**
|
|
||||||
|
|
||||||
[Versions](https://wiki.dolibarr.org/index.php?title=List_of_releases,_change_log_and_compatibilities)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sudo su -
|
|
||||||
mv /var/www/html /var/www/.html
|
|
||||||
|
|
||||||
cd /var/www
|
|
||||||
git clone --depth 1 -b 20.0.3 https://github.com/Dolibarr/dolibarr.git dolibarr
|
|
||||||
mv dolibarr-20.0.3 dolibarr
|
|
||||||
```
|
|
||||||
|
|
||||||
12. **Create required directory structure**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
chmod -R 755 /var/www/dolibarr
|
|
||||||
chown -R www-data:www-data /var/www/dolibarr
|
|
||||||
cd dolibarr ; touch htdocs/conf/conf.php ; chown www-data htdocs/conf/conf.php
|
|
||||||
mkdir -p /var/lib/dolibarr/sessions ; chown www-data /var/lib/dolibarr/sessions
|
|
||||||
mkdir -p /var/lib/dolibarr/documents ; chown www-data /var/lib/dolibarr/documents
|
|
||||||
```
|
|
||||||
|
|
||||||
13. **Apache Configuration**
|
|
||||||
|
|
||||||
1. **Allow http https**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ufw allow https
|
|
||||||
ufw allow http
|
|
||||||
```
|
|
||||||
|
|
||||||
2. **Backup Default Files**
|
|
||||||
Backup the default index.html and configuration files.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mv /var/www/html /var/www/.html
|
|
||||||
mv /etc/apache2/sites-available/000-default.conf /etc/apache2/sites-available/.000-default.conf
|
|
||||||
rm /etc/apache2/sites-enabled/000-default.conf
|
|
||||||
```
|
|
||||||
|
|
||||||
3. **Create Default Server Block for Unauthorized Access**
|
|
||||||
|
|
||||||
4. **Site configuration**
|
|
||||||
|
|
||||||
Edit the Apache configuration file to proxy to the Dolibarr CRM.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nano /etc/apache2/sites-available/<hostname-intranet>.conf
|
|
||||||
```
|
|
||||||
|
|
||||||
Add the following configuration:
|
|
||||||
|
|
||||||
```apache
|
|
||||||
<VirtualHost *:80>
|
|
||||||
LogLevel info
|
|
||||||
ServerName <hostname-intranet>
|
|
||||||
ServerAdmin admin@fabq.ca
|
|
||||||
DocumentRoot /var/www/dolibarr
|
|
||||||
|
|
||||||
# Alias for Dolibarr
|
|
||||||
Alias / /var/www/dolibarr/htdocs/
|
|
||||||
|
|
||||||
<Directory /var/www/dolibarr/htdocs>
|
|
||||||
Options +FollowSymLinks
|
|
||||||
AllowOverride All
|
|
||||||
Require all granted
|
|
||||||
</Directory>
|
|
||||||
|
|
||||||
ErrorLog ${APACHE_LOG_DIR}/error.dolibarr.log
|
|
||||||
CustomLog ${APACHE_LOG_DIR}/access.dolibarr.log combined
|
|
||||||
</VirtualHost>
|
|
||||||
```
|
|
||||||
|
|
||||||
5. **Enabling the new VirtualHost**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
a2ensite <hostname-intranet>.conf
|
|
||||||
a2enmod rewrite
|
|
||||||
```
|
|
||||||
|
|
||||||
6. **Managing Apache**
|
|
||||||
|
|
||||||
**Enable Apache**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
systemctl enable apache2
|
|
||||||
```
|
|
||||||
|
|
||||||
**Apache reload**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
systemctl reload apache2
|
|
||||||
```
|
|
||||||
|
|
||||||
**`Optional` Apache restart**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
systemctl restart apache2
|
|
||||||
```
|
|
||||||
|
|
||||||
14. **Create folders**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mkdir -p /var/tmp
|
|
||||||
mkdir -p /var/upload_tmp
|
|
||||||
```
|
|
||||||
|
|
||||||
15. **Set temporary file paths**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nano /etc/php/8.2/apache2/php.ini
|
|
||||||
```
|
|
||||||
|
|
||||||
```ini
|
|
||||||
sys_temp_dir = "/var/tmp"
|
|
||||||
upload_tmp_dir = "/var/upload_tmp"
|
|
||||||
open_basedir = "/var/www/dolibarr:/var/lib/dolibarr/documents:/var/lib/dolibarr/sessions:/var/tmp:/var/upload_tmp"
|
|
||||||
```
|
|
||||||
|
|
||||||
16. **Fix permissions**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
chown -R www-data:www-data /var/lib/php/sessions
|
|
||||||
|
|
||||||
chown -R www-data:www-data /var/www/dolibarr/htdocs/conf/conf.php
|
|
||||||
chmod 640 /var/www/dolibarr/htdocs/conf/conf.php
|
|
||||||
|
|
||||||
find /var/www/dolibarr -type d -exec chmod 755 {} \;
|
|
||||||
find /var/www/dolibarr -type f -exec chmod 644 {} \;
|
|
||||||
|
|
||||||
chmod go-w /var/lib/dolibarr/documents;
|
|
||||||
chmod go-w /var/lib/dolibarr/sessions;
|
|
||||||
|
|
||||||
chown -R www-data:www-data /var/tmp
|
|
||||||
chown -R www-data:www-data /var/upload_tmp
|
|
||||||
chmod -R 750 /var/tmp
|
|
||||||
chmod -R 750 /var/upload_tmp
|
|
||||||
```
|
|
||||||
|
|
||||||
17. **Secure PHP**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nano /etc/php/8.2/apache2/php.ini
|
|
||||||
```
|
|
||||||
|
|
||||||
```ini
|
|
||||||
session.use_strict_mode = 1
|
|
||||||
|
|
||||||
allow_url_fopen = No
|
|
||||||
|
|
||||||
disable_functions = dl, apache_note, apache_setenv, pcntl_alarm, pcntl_fork, pcntl_waitpid, pcntl_wait, pcntl_wifexited, pcntl_wifstopped, pcntl_wifsignaled, pcntl_wifcontinued, pcntl_wexitstatus, pcntl_wtermsig, pcntl_wstopsig, pcntl_signal, pcntl_signal_get_handler, pcntl_signal_dispatch, pcntl_get_last_error, pcntl_strerror, pcntl_sigprocmask, pcntl_sigwaitinfo, pcntl_sigtimedwait, pcntl_exec, pcntl_getpriority, pcntl_setpriority, pcntl_async_signals, show_source, virtual, passthru, shell_exec, system, proc_open, popen
|
|
||||||
|
|
||||||
session.gc_maxlifetime = 604800
|
|
||||||
```
|
|
||||||
|
|
||||||
18. **Complete the Installation through Web Interface**
|
|
||||||
|
|
||||||
- Open your web browser and navigate to `http://<hostname-intranet>/install/`
|
|
||||||
- Follow the on-screen instructions to complete the installation.
|
|
||||||
|
|
||||||
19. **Lock installation**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
chmod -w /var/www/dolibarr/htdocs/conf/conf.php
|
|
||||||
touch /var/lib/dolibarr/documents/install.lock;
|
|
||||||
chmod go-w /var/lib/dolibarr/documents;
|
|
||||||
```
|
|
||||||
|
|
||||||
20. **Secure installation**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
chmod -R -w /var/www/dolibarr/htdocs
|
|
||||||
cp /etc/php/8.4/apache2/php.ini /etc/php/8.4/apache2/php.ini.back
|
|
||||||
nano /etc/php/8.4/apache2/php.ini
|
|
||||||
```
|
|
||||||
|
|
||||||
```ini
|
|
||||||
session.save_path = /var/lib/dolibarr/sessions
|
|
||||||
session.use_strict_mode = 1
|
|
||||||
session.use_only_cookies = 1
|
|
||||||
session.cookie_httponly = 1
|
|
||||||
session.cookie_samesite = Lax
|
|
||||||
session.gc_maxlifetime = 604800
|
|
||||||
open_basedir = "/var/www/dolibarr:/var/lib/dolibarr/documents:/var/lib/dolibarr/sessions:/var/tmp:/var/upload_tmp"
|
|
||||||
short_open_tag = Off
|
|
||||||
allow_url_fopen = Off
|
|
||||||
allow_url_include = Off
|
|
||||||
disable_functions = dl, apache_note, apache_setenv, pcntl_alarm, pcntl_fork, pcntl_waitpid, pcntl_wait, pcntl_wifexited, pcntl_wifstopped, pcntl_wifsignaled, pcntl_wifcontinued, pcntl_wexitstatus, pcntl_wtermsig, pcntl_wstopsig, pcntl_signal, pcntl_signal_get_handler, pcntl_signal_dispatch, pcntl_get_last_error, pcntl_strerror, pcntl_sigprocmask, pcntl_sigwaitinfo, pcntl_sigtimedwait, pcntl_exec, pcntl_getpriority, pcntl_setpriority, pcntl_async_signals, show_source, virtual, passthru, shell_exec, system, proc_open, popen
|
|
||||||
```
|
|
||||||
|
|
||||||
```bash
|
|
||||||
nano /var/www/dolibarr/htdocs/conf/conf.php
|
|
||||||
chmod o-r /var/www/dolibarr/htdocs/conf/conf.php
|
|
||||||
```
|
|
||||||
|
|
||||||
```php
|
|
||||||
$dolibarr_main_prod='1';
|
|
||||||
```
|
|
||||||
|
|
||||||
21. **Configure SSL with Certbot (Optional, If directly accessed)**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
certbot --apache -d <hostname-internet>
|
|
||||||
systemctl status certbot.timer
|
|
||||||
certbot renew --dry-run
|
|
||||||
```
|
|
||||||
|
|
||||||
22. **Verify installation**
|
|
||||||
|
|
||||||
- Verify that the Dolibarr instance is installed restored by accessing it through a web browser. Ensure that all repositories, users, and configurations are intact.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
systemctl status apache2
|
|
||||||
```
|
|
||||||
|
|
||||||
Visit the server at [http://<hostname-internet>](http://<hostname-internet>) or [https://<hostname-internet>](https://<hostname-internet>).
|
|
||||||
|
|
||||||
23. **Backup post installation**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id-hypervisor> --state stopped"
|
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump <container-id-hypervisor> --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup post installation\""
|
|
||||||
```
|
|
||||||
|
|
||||||
24. **Start the server**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id-hypervisor> --state started --max_relocate 3 --max_restart 3"
|
|
||||||
```
|
|
||||||
|
|
||||||
25. **Move contracts**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
scp '<dolibarr-contract-paths-source>/contract.odt' <username>@<hostname-intranet>:~/
|
|
||||||
|
|
||||||
sudo su -
|
|
||||||
mv /home/<username>/*.odt /var/lib/dolibarr/documents/doctemplates/contracts/
|
|
||||||
chown -R www-data:www-data /var/lib/dolibarr/documents/doctemplates/contracts/
|
|
||||||
```
|
|
||||||
|
|
||||||
26. **ODT to PDF**
|
|
||||||
|
|
||||||
[See forum 1](https://www.dolibarr.org/forum/t/solved-odt-to-pdf/16931)
|
|
||||||
[See forum 2](https://www.dolibarr.org/forum/t/setup-dolibarr-to-generate-odt-to-pdf/22112/9)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
MAIN_ODT_AS_PDF
|
|
||||||
```
|
|
||||||
|
|
||||||
27. **Install fonts**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
scp -r <dolibarr-fonts-paths-source>/Lato <username>@<hostname-intranet>:~/
|
|
||||||
```
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mv /home/<username>/Lato /usr/local/share/fonts/
|
|
||||||
chmod -R 755 /usr/local/share/fonts/
|
|
||||||
chown -R root:root /usr/local/share/fonts/
|
|
||||||
```
|
|
||||||
@ -63,7 +63,6 @@ Replace the placeholders below with the appropriate values for your setup:
|
|||||||
- Hostname - Hypervisor: `<hostname-hypervisor>` (e.g., proxmox-hypervisor.domain.com)
|
- Hostname - Hypervisor: `<hostname-hypervisor>` (e.g., proxmox-hypervisor.domain.com)
|
||||||
- Hostname - Hypervisor NAS: `<hostname-hypervisor-nas>` (e.g., nas-server.domain.com)
|
- Hostname - Hypervisor NAS: `<hostname-hypervisor-nas>` (e.g., nas-server.domain.com)
|
||||||
- Name - Hypervisor NAS: `<name-hypervisor-nas>` (e.g., nas-server)
|
- Name - Hypervisor NAS: `<name-hypervisor-nas>` (e.g., nas-server)
|
||||||
- Container ID - Hypervisor: `<container-id-hypervisor>` (e.g., 100)
|
|
||||||
|
|
||||||
- **SSH Keys**
|
- **SSH Keys**
|
||||||
|
|
||||||
@ -127,40 +126,40 @@ ssh <username>@<hostname-hypervisor-nas> "ls /mnt/proxmox/template/cache/"
|
|||||||
**Create the container**
|
**Create the container**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "pct create <container-id-hypervisor> <name-hypervisor-nas>:vztmpl/debian-12-upgraded_12.5_amd64.tar.zst --hostname <hostname-intranet> --cores 2 --memory 2048 --swap 2048 --net0 name=net0,bridge=vmbr0,ip=dhcp,firewall=1 --rootfs <name-hypervisor-nas>:250 --unprivileged 1 --features nesting=1 --ssh-public-keys <ssh-key-proxmox> --start 1"
|
ssh <username-hypervisor>@<hostname-hypervisor> "pct create 100 <name-hypervisor-nas>:vztmpl/debian-12-upgraded_12.5_amd64.tar.zst --hostname <hostname-intranet> --cores 2 --memory 2048 --swap 2048 --net0 name=net0,bridge=vmbr0,ip=dhcp,firewall=1 --rootfs <name-hypervisor-nas>:250 --unprivileged 1 --features nesting=1 --ssh-public-keys <ssh-key-proxmox> --start 1"
|
||||||
```
|
```
|
||||||
|
|
||||||
**Backup**
|
**Backup**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump <container-id-hypervisor> --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
|
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump 100 --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
|
||||||
```
|
```
|
||||||
|
|
||||||
**Set the state of the Proxmox HA Manager for Container <container-id-hypervisor>**
|
**Set the state of the Proxmox HA Manager for Container 100**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager add ct:<container-id-hypervisor>"
|
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager add ct:100"
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager remove ct:<container-id-hypervisor>"
|
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager remove ct:100"
|
||||||
```
|
```
|
||||||
|
|
||||||
**Set the state and limits of the Proxmox Container <container-id-hypervisor> in the HA Manager**
|
**Set the state and limits of the Proxmox Container 100 in the HA Manager**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id-hypervisor> --state started --max_relocate 3 --max_restart 3"
|
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:100 --state started --max_relocate 3 --max_restart 3"
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id-hypervisor> --state stopped"
|
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:100 --state stopped"
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "pct reboot <container-id-hypervisor>"
|
ssh <username-hypervisor>@<hostname-hypervisor> "pct reboot 100"
|
||||||
```
|
```
|
||||||
|
|
||||||
**Destroy the Proxmox Container <container-id-hypervisor> forcefully**
|
**Destroy the Proxmox Container 100 forcefully**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "pct destroy <container-id-hypervisor> --force --purge"
|
ssh <username-hypervisor>@<hostname-hypervisor> "pct destroy 100 --force --purge"
|
||||||
```
|
```
|
||||||
|
|
||||||
**Move the Proxmox Container <container-id-hypervisor> to another host**
|
**Move the Proxmox Container 100 to another host**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "pct migrate <container-id-hypervisor> hv2"
|
ssh <username-hypervisor>@<hostname-hypervisor> "pct migrate 100 hv2"
|
||||||
```
|
```
|
||||||
|
|
||||||
### SSH Connection
|
### SSH Connection
|
||||||
@ -209,7 +208,7 @@ cat /home/<username>/.ssh/<username>.pub >> /home/<username>/.ssh/authorized_key
|
|||||||
2. **Backup before starting**
|
2. **Backup before starting**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump <container-id-hypervisor> --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
|
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump 100 --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
|
||||||
```
|
```
|
||||||
|
|
||||||
3. **Install Required Dependencies**
|
3. **Install Required Dependencies**
|
||||||
@ -504,11 +503,11 @@ cat /home/<username>/.ssh/<username>.pub >> /home/<username>/.ssh/authorized_key
|
|||||||
19. **Backup post installation**
|
19. **Backup post installation**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id-hypervisor> --state stopped"
|
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:100 --state stopped"
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump <container-id-hypervisor> --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup post installation\""
|
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump 100 --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup post installation\""
|
||||||
```
|
```
|
||||||
|
|
||||||
20. **Start the server**
|
20. **Start the server**
|
||||||
```bash
|
```bash
|
||||||
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id-hypervisor> --state started --max_relocate 3 --max_restart 3"
|
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:100 --state started --max_relocate 3 --max_restart 3"
|
||||||
```
|
```
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user