fabrice/scripts-fabq
fabrice/scripts-fabq
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: fabrice:main
Branches Tags
fabrice:main
fabrice:setups-drafts
..
compare: fabrice:setups-drafts
Branches Tags
fabrice:main
fabrice:setups-drafts

5 Commits
main ... setups-dra

Author SHA1 Message Date
Fabrice Quenneville
497bbb6324 Added information on installing and setting up wp-cli. 2025-02-03 10:16:59 -05:00
Fabrice Quenneville
385eee64cb Added missing note on allowing overrides. 2025-01-31 09:09:07 -05:00
Fabrice Quenneville
b1ee30dd80 Added draft notes on setting up a wordpress site 
on AlmaLinux on proxmox behind wireguard and firewalld.

The draft is missing ssl notes.
 2025-01-31 09:01:51 -05:00
Fabrice Quenneville
efccada19b Merge branch 'main' into setups-drafts 2025-01-30 12:42:49 -05:00
Fabrice Quenneville
ad1fefc02a Create 'setups-drafts' branch and 'setups-drafts' folder for storing incomplete guides 
- Created the 'setups-drafts' branch for organizing drafts and works in progress.
- Added the 'setups-drafts/oracle_setup_oro.md' guide to the 'setups-drafts' folder for storage.
- The guide provides instructions for setting up Oro on Oracle Linux.
- This guide is a work in progress and not yet fully tested or production-ready.
 2024-12-13 19:43:19 -05:00
27 changed files with 1845 additions and 4440 deletions
Show Stats Expand all files Collapse all files

32
README.md
View File

@ -7,33 +7,24 @@
This repository is structured into several key directories:
- **scripts/**: Contains individual scripts for various tasks. Currently, it includes:
- `scripts/library/`: Libraries used by Python scripts.
- `venv_utils.py`: Utility functions for creating, activating, and managing Python virtual environments.
- `change_case.py`: A script for renaming files and directories by changing their case.
- `efi-update-mirror.sh`: Script for syncing a two drive EFI mirror.
- `video_autoreduce.py`: A script for automatic resolution reduction of video files.
- `video_autoreduce_rename.py`: A script for automated renaming of video files post resolution reduction.
- `video_manage_audio.py`: A script for removing audio from video files.
- `video_manage_subtitles.py`: A script for removing subtitles from video files.
- `video_repackage_mkv.py`: Recursively repackage non-MKV video files into Matroska containers.
- `video_remove_audio.py`: A script for removing audio from video files.
- **notes/**: A collection of markdown files containing notes on various topics, including:
- `brother.md`: Brother printer setup and troubleshooting on Linux.
- `btrfs.md`: Btrfs filesystem management, balances, scrubs, snapshots, and drive replacement.
- `chrome-driver.md`: ChromeDriver setup and usage.
- `debian-packaging.md`: Building and maintaining Debian packages.
- `brother.md`: Information about Brother printers on Linux.
- `btrfs.md`: Btrfs filesystem configuration and tips.
- `chrome-driver.md`: Notes on ChromeDriver setup and usage on Linux.
- `debian packaging.md`: Debian packaging guidelines.
- `dns.md`: DNS configuration and troubleshooting.
- `efi-repair.md`: EFI partition repair guide.
- `efi-update-mirror.md`: Redundant root with EFI mirror setup guide.
- `linux.md`: General Linux commands — users, system management, archiving, and diagnostics.
- `linux.md`: General Linux tips.
- `pdf.md`: PDF manipulation with Linux command line.
- `pdftk.md`: PDF Toolkit usage.
- `pip-packaging.md`: Packaging and publishing Python projects with pip.
- `ssh.md`: SSH configuration, key management, and tunneling tips.
- `storage.md`: Storage tooling — smartctl, badblocks, dd, hdparm, fdisk, and fstab/mount management.
- `wordpress.md`: WordPress debugging and tips.
- `pip packaging.md`: Packaging Python projects with pip.
- `ssh.md`: Secure Shell (SSH) configuration and tips.
- **pages/other/**: Templates for other pages, such as the homepage of my Debian package repository. These are provided as inspiration and should not be used as-is.
- `debrepo.fabq.ca.html`: This file serves as the homepage for my Debian package repository. It provides the instructions to add the repository to user's system's sources list and install packages securely.
- **pages/errors/**: Templates for error pages. These are provided as inspiration and should not be used as-is.
@ -44,7 +35,6 @@ This repository is structured into several key directories:
- **setups/**: A collection of markdown files containing notes on configuring servers, including:
- `debian_setup_aptly.md`: Comprehensive guide for installing Aptly on Debian.
- `debian_setup_dolibarr.md`: Comprehensive guide for installing Dolibarr on Debian.
- `debian_setup_gitea.md`: Comprehensive guide for installing Gitea on Debian.
## 📖 Documentation

661
notes/btrfs.md
View File

@ -4,323 +4,377 @@
- [BTRFS](#btrfs)
- [Table of Contents](#table-of-contents)
- [Placeholders](#placeholders)
- [BTRFS Command Shorthands](#btrfs-command-shorthands)
- [Information on Drives](#information-on-drives)
- [Information on Filesystem](#information-on-filesystem)
- [Backup Procedures](#backup-procedures)
- [Recovery](#recovery)
- [Drive Manipulation](#drive-manipulation)
- [Replace Drives](#replace-drives)
- [Degraded Mount and Missing Device Removal](#degraded-mount-and-missing-device-removal)
- [Filesystem Manipulation](#filesystem-manipulation)
- [Upgrading Btrfs Block Group Cache to V2](#upgrading-btrfs-block-group-cache-to-v2)
- [Defrag](#defrag)
- [Upgrading Btrfs block group cache to V2](#upgrading-btrfs-block-group-cache-to-v2)
- [Balances](#balances)
- [Scrub](#scrub)
- [Snapshots](#snapshots)
- [Create Snapshots](#create-snapshots)
- [Delete Snapshots](#delete-snapshots)
- [Backup Procedures](#backup-procedures)
- [Recovery](#recovery)
- [Filesystem Check](#filesystem-check)
- [Diagnosis](#diagnosis)
## Placeholders
## Information on Drives
Replace the placeholders below with the appropriate values for your setup:
**List of Drives and Mountpoints**
- **Devices**
- Block device: `<device>` (e.g., /dev/sda)
- Source device to replace: `<source-device>` (e.g., /dev/sdb)
- Target device: `<target-device>` (e.g., /dev/sdc)
- UUID: `<uuid>` (e.g., a1b2c3d4-e5f6-7890-abcd-ef1234567890)
To check all attached drives:
- **Paths**
- Mount point: `<mountpoint>` (e.g., /mnt/media)
- Subvolume name: `<subvolume>` (e.g., root, home, backups, snapshots)
- Subvolume ID: `<subvolume-id>` (e.g., 257)
- Snapshot label: `<snapshot-label>` (e.g., 2024-09-15)
```bash
ls /dev/sd*
ls /dev/nv*
```
## BTRFS Command Shorthands
To view mountpoints and drive details such as names, sizes, and mountpoints:
Most `btrfs` subcommands accept shortened aliases. The table below lists the common ones used throughout this document.
```bash
lsblk
df -h
cat /etc/fstab
```
| Long form | Short form |
| ----------------------------- | ----------------------- |
| `btrfs filesystem` | `btrfs fi` |
| `btrfs filesystem show` | `btrfs fi show` |
| `btrfs filesystem usage` | `btrfs fi usage` |
| `btrfs filesystem df` | `btrfs fi df` |
| `btrfs filesystem resize` | `btrfs fi resize` |
| `btrfs filesystem defragment` | `btrfs fi defrag` |
| `btrfs subvolume` | `btrfs sub` |
| `btrfs subvolume list` | `btrfs sub list` |
| `btrfs subvolume snapshot` | `btrfs sub snap` |
| `btrfs subvolume delete` | `btrfs sub del` |
| `btrfs subvolume get-default` | `btrfs sub get-default` |
| `btrfs subvolume set-default` | `btrfs sub set-default` |
| `btrfs device` | `btrfs dev` |
| `btrfs device add` | `btrfs dev add` |
| `btrfs device usage` | `btrfs dev usage` |
| `btrfs device stats` | `btrfs dev stats` |
| `btrfs balance start` | `btrfs bal start` |
| `btrfs balance status` | `btrfs bal status` |
| `btrfs balance cancel` | `btrfs bal cancel` |
| `btrfs inspect-internal` | `btrfs insp` |
**Drive Information**
To get detailed information and serial number of a specific drive:
```bash
smartctl -i /dev/sdc
```
**Find the Device Path from UUID**
Using lsblk:
```bash
lsblk -o NAME,UUID,MOUNTPOINT
```
Using blkid:
```bash
blkid | grep <UUID>
```
## Information on Filesystem
**Show Basic Filesystem Information**
Display basic information (size, IDs, paths, etc.) for the specified mountpoint:
To display basic information (size, IDs, paths, etc.) for the specified mountpoint:
```bash
btrfs fi show <mountpoint>
btrfs fi show /mnt/media/
```
**Display detailed usage information**
**Display Detailed Usage Information**
To show detailed usage information (allocated, unallocated, free, used, etc.) for the specified mountpoint:
To show detailed usage information (allocated, unallocated, etc.) for the specified mountpoint:
```bash
btrfs fi usage <mountpoint>
btrfs fi usage /mnt/media
```
**Display detailed usage information as a table**
```bash
btrfs fi usage -T <mountpoint>
```
- `-T` The tabular flag gives you a nice grid that shows exactly how much Data, Metadata, and System space is allocated per device.
**Display Detailed Allocation Information**
View block groups and used space:
To view detailed allocation information (block groups, used space) for the specified mountpoint:
```bash
btrfs fi df <mountpoint>
btrfs fi df /mnt/media
```
**Get Detailed Device Usage Statistics**
Physical size, unallocated space, RAID levels, etc.:
To get detailed device usage statistics (physical size, unallocated space, RAID levels, etc.) for a BTRFS filesystem:
```bash
btrfs device usage <mountpoint>
btrfs device usage /mnt/media
```
**Scan and Display BTRFS Information**
Scan all devices or a specific drive:
To scan and display BTRFS information for all devices or a specific drive:
```bash
btrfs device scan
btrfs device scan <device>
btrfs device scan /dev/sda/
```
**Retrieve Statistics and Error Information**
Read errors, write errors, flush errors, etc.:
To get statistics and error information (read errors, write errors, flush errors, etc.) for the specified mountpoint:
```bash
btrfs device stats <mountpoint>
btrfs device stats /mnt/media
```
**Reset Device Error Counters**
Reset all per-device error counters to zero after acknowledging them:
```bash
btrfs device stats --reset <mountpoint>
btrfs device stats -z <mountpoint>
```
- `-z` / `--reset`: Zeroes the counters after printing. Useful after a known event you've already investigated.
**List BTRFS Subvolumes**
To list BTRFS subvolumes:
```bash
btrfs subvolume list <mountpoint>
btrfs subvolume list /
btrfs subvolume list /home/fabrice
btrfs subvolume list /mnt/workbench
```
**Default Subvolume**
Check if a non-standard subvolume is set as the default:
To check if a non-standard subvolume is set as the default:
```bash
btrfs subvol get-default <mountpoint>
btrfs subvol list <mountpoint>
btrfs subvol get-default /mnt/tmp/
btrfs subvol list /mnt/tmp/
```
Change the default subvolume:
To change the default subvolume if a non-standard one is set:
```bash
btrfs subvol set-default <subvolume-id> <mountpoint>
btrfs subvol set-default 257 /mnt/tmp/
```
**Verify Current Cache Version**
Check if your filesystem is using cache V1 by device:
To check if your filesystem is using cache V1 by device:
```bash
btrfs inspect-internal dump-super -f <device> | grep cache_generation
btrfs inspect-internal dump-super -f /dev/<device> | grep cache_generation
```
By UUID:
To check if your filesystem is using cache V1 by UUID:
```bash
btrfs inspect-internal dump-super -f $(blkid -U <uuid>) | grep cache_generation
btrfs inspect-internal dump-super -f $(blkid -U <UUID>) | grep cache_generation
```
- If `cache_generation` is present, cache V1 is in use. If absent, the filesystem is already using V2.
- If cache_generation is present, it indicates cache V1 is in use. If it's absent, the filesystem is already using V2.
## Backup Procedures
**Desktop Backup: Root and Home**
1. Mount snapshot location:
```bash
mount UUID=394decca-4780-47c9-9ae3-e4d03681a791 -o subvol=snapshots /mnt/snapshots
```
2. Create snapshots for root and home:
```bash
btrfs subvolume snapshot / "/mnt/snapshots/root/2021-05-23 - Fedora 34 upgrade"
btrfs subvolume snapshot /home "/mnt/snapshots/home/2021-05-23 - Fedora 34 upgrade"
```
3. Unmount after creating snapshots:
```bash
umount /mnt/snapshots
```
**Data Backup: Workbench, Documents, Education**
1. Mount snapshot location:
```bash
mount UUID=72e1770a-9fc0-461e-88d3-db640ff53dd9 -o subvol=snapshots /mnt/snapshots
```
2. Create snapshots for multiple directories:
```bash
btrfs subvolume snapshot /mnt/workbench "/mnt/snapshots/workbench/2021-05-23 - Fedora 34 upgrade"
btrfs subvolume snapshot /home/fabrice/Documents "/mnt/snapshots/Documents/2021-05-23 - Fedora 34 upgrade"
btrfs subvolume snapshot /home/fabrice/Education "/mnt/snapshots/Education/2021-05-23 - Fedora 34 upgrade"
```
3. Unmount after creating snapshots:
```bash
umount /mnt/snapshots
```
**STOR1 Fedora Backup**
1. Mount snapshot location:
```bash
mount UUID=e4fd608e-cfe8-4c10-b6d0-03b05bae8aa6 -o subvol=snapshots /mnt/snapshots
```
2. Create snapshot:
```bash
btrfs subvolume snapshot / "/mnt/snapshots/root/2021-06-06"
```
3. Unmount after creating snapshot:
```bash
umount /mnt/snapshots
```
**STOR1 Debian Backup**
1. Mount snapshot location:
```bash
mount UUID=c9a77f3c-626f-47bd-b4e3-9a094bea287f -o subvol=snapshots /mnt/snapshots
```
2. Create snapshot:
```bash
btrfs subvolume snapshot / "/mnt/snapshots/root/2021-07-12 - post mostly setup"
```
3. Unmount after creating snapshot:
```bash
umount /mnt/snapshots
```
**STOR2 Backup**
1. Mount snapshot location:
```bash
mount UUID=30bd5e0e-e781-4e87-8fb8-ea5606403b15 -o subvol=snapshots /mnt/snapshots
```
2. Create snapshot:
```bash
btrfs subvolume snapshot / "/mnt/snapshots/root/2021-06-06 - Fedora 34"
```
3. Unmount after creating snapshot:
```bash
umount /mnt/snapshots
```
## Recovery
**Mount a Subvolume with Recovery Options**
```bash
mount -o recovery,subvol=backups UUID=aa5c1d34-ecba-42a9-9339-8f7879d47536 /mnt/tmp
```
**Clear Cache During Mount**
```bash
mount -o clear_cache,subvol=backups UUID=aa5c1d34-ecba-42a9-9339-8f7879d47536 /mnt/tmp
```
**Data Restoration**
To restore data using `btrfs restore`:
```bash
btrfs restore -D /dev/sdb
```
## Drive Manipulation
**Mount Whole Drive**
```bash
mount UUID=<uuid> <mountpoint>
```
**Mount Subvolume by Name**
```bash
mount UUID=<uuid> -o subvol=<subvolume> <mountpoint>
mount UUID=c9a77f3c-626f-47bd-b4e3-9a094bea287f /mnt/tmp
```
**Mount Subvolume by ID**
```bash
btrfs subvol list /
mount -o subvolid=<subvolume-id> /dev/disk/by-uuid/<uuid> <mountpoint>
```
**Mount Read-Only**
Mount a partition in read-only mode, useful for forensics or recovery without risking further writes:
```bash
mount -r <device> <mountpoint>
```
**Remount with Performance Options**
Apply common performance mount options to a live filesystem without unmounting:
```bash
mount -o remount,noatime,compress=zstd:3,autodefrag,space_cache=v2 <mountpoint>
```
**Remount with Default Options**
```bash
mount -o remount,defaults,noatime,compress=zstd:3 <mountpoint>
mount -o subvolid=5 /dev/disk/by-uuid/7a22514b-594a-43a3-8fdd-4df1530b5465 /mnt
```
**Add a New Drive**
To add a new drive to an existing BTRFS setup:
```bash
btrfs device add <device> <mountpoint>
btrfs device add /dev/sdf /mnt/media/
```
**Resize Filesystem**
Grow the filesystem on a specific device to its maximum:
```bash
btrfs filesystem resize 1:max <mountpoint>
btrfs filesystem resize 1:max /mnt/media/
```
**Create Subvolumes**
```bash
btrfs subvol create <mountpoint>/<subvolume>
btrfs subvol create /mnt/tmp/root
btrfs subvol create /mnt/tmp/snapshots
```
### Replace Drives
**Start the replacement process:**
**Replace the source drive with the target drive:**
Copies data from the old drive to the new drive while the filesystem remains mounted:
This command will start the replacement process where the data from the old drive (`/dev/sdb`) is copied over to the new drive (`/dev/sdj`).
```bash
btrfs replace start <source-device> <target-device> <mountpoint>
btrfs replace start /dev/sdb /dev/sdj /mnt/media
```
- `<source-device>`: Drive to be replaced.
- `<target-device>`: Drive to replace it with.
- `<mountpoint>`: Mount point of the BTRFS filesystem.
- `/dev/sdb`: Source drive to be replaced.
- `/dev/sdj`: Target drive to replace the source drive.
- `/mnt/media`: Mount point of the BTRFS filesystem.
**Monitor the progress:**
**Monitor the progress of the replacement:**
Once the replacement process has started, you can monitor its progress with the following command:
```bash
btrfs replace status <mountpoint>
btrfs replace status /mnt/media
```
- This will print the current status of the drive replacement operation, showing how much data has been migrated.
**Monitor progress interactively:**
For a more detailed, interactive status view of the replacement process, use the `-i` option:
```bash
btrfs replace status -i <mountpoint>
btrfs replace status -i /mnt/media
```
- `-i`: Updates progress in real time.
- `-i`: This flag provides an interactive mode where the progress is updated in real time.
**Notes:**
- `btrfs replace` works on a live mounted filesystem — no unmounting required.
- Useful for both failing drive replacement and capacity upgrades.
- Ensure the target drive has enough space to accommodate the source data.
### Degraded Mount and Missing Device Removal
Use when a drive has failed and you need to access the filesystem with the remaining devices.
**Mount in degraded mode:**
```bash
mount -o ro,degraded <device> <mountpoint>
```
**Mount a specific subvolume in degraded mode:**
```bash
mount -t btrfs -o degraded,subvol=<subvolume>,noatime,compress=zstd:3 UUID=<uuid> <mountpoint>
```
**Remove the missing device from the filesystem:**
Once mounted degraded, remove the placeholder for the missing drive:
```bash
btrfs device remove missing <mountpoint>
```
- This cleans up the missing device slot so the filesystem no longer expects it.
- Only safe to run if data is intact on the remaining devices (e.g., RAID1 with one drive).
- The `btrfs replace` command allows you to replace a faulty or underperforming drive without unmounting the filesystem, making it ideal for live systems.
- It can be used for upgrading storage by replacing smaller drives with larger ones, or for replacing failing drives.
- Ensure that the target drive has enough space to accommodate the data from the source drive.
## Filesystem Manipulation
### Upgrading Btrfs Block Group Cache to V2
### Upgrading Btrfs block group cache to V2
**Non-root filesystems (running system):**
**From a running system non-root filesystems**
```bash
mount -o remount,clear_cache,space_cache=v2 <mountpoint>
mount -o remount,clear_cache,space_cache=v2 /mnt/<mount-point>
```
**Root filesystem (running system):**
**From a running system on root**
1. Check if using cache V1:
Check if your filesystem is using cache V1:
```bash
btrfs inspect-internal dump-super -f <device> | grep cache_generation
btrfs inspect-internal dump-super -f /dev/<device> | grep cache_generation
```
2. Enable cache V2 via GRUB:
Enable Cache V2
```bash
nano /etc/default/grub
# Add to GRUB_CMDLINE_LINUX_DEFAULT or GRUB_CMDLINE_LINUX:
# rootflags=clear_cache,space_cache=v2
# Locate the line starting with GRUB_CMDLINE_LINUX_DEFAULT or GRUB_CMDLINE_LINUX and add the following options:
rootflags=clear_cache,space_cache=v2
```
Example:
@ -334,305 +388,158 @@ mount -o remount,clear_cache,space_cache=v2 <mountpoint>
reboot
```
3. Verify the change:
Verify the Change
```bash
btrfs inspect-internal dump-super -f <device> | grep cache_generation
btrfs inspect-internal dump-super -f /dev/<device> | grep cache_generation
```
4. Remove `clear_cache` from GRUB after confirming:
Remove `clear_cache` Option
```bash
nano /etc/default/grub
# Remove clear_cache from rootflags, then:
# Remove clear_cache from the rootflags.
update-grub
```
**From a live system:**
**From a live system**
```bash
apt update
apt install btrfs-progs
lsblk -o NAME,UUID
blkid
mount -o clear_cache,space_cache=v2 /dev/disk/by-uuid/<uuid> <mountpoint>
btrfs inspect-internal dump-super -f /dev/disk/by-uuid/<uuid> | grep cache_generation
umount <mountpoint>
mount -o clear_cache,space_cache=v2 /dev/disk/by-uuid/<UUID> /mnt
btrfs inspect-internal dump-super -f /dev/disk/by-uuid/<UUID> | grep cache_generation
umount /mnt
```
### Defrag
**Standard recursive defrag with LZO compression:**
```bash
btrfs filesystem defrag -r -v -clzo <mountpoint>
```
- `-r`: Recursive.
- `-v`: Verbose.
- `-clzo`: Optional LZO compression to save space.
**Recursive defrag with Zstd compression, logged to file:**
Runs in the background with unbuffered output so the log file updates in real time:
```bash
stdbuf -oL btrfs filesystem defrag -r -v -czstd <mountpoint> > /root/<date>-defrag.log 2>&1 &
```
- `stdbuf -oL`: Forces line-buffered stdout so log entries appear immediately.
- `-czstd`: Zstd compression (better ratio than LZO, available since kernel 5.1).
- `&`: Runs in the background; use `tail -f /root/<date>-defrag.log` to monitor.
## Balances
**Full balance on nearly empty block groups:**
**Perform a Full Balance with Minimal Usage**
```bash
btrfs balance start --full-balance -dusage=0 -musage=0 <mountpoint>
btrfs balance start --full-balance -dusage=0 -musage=0 /mnt/media/
```
- `--full-balance`: Default but with a warning if not specified.
- `-dusage=0`: Only balance data block groups that are ~0% full.
- `-musage=0`: Only balance metadata block groups that are ~0% full.
- `--full-balance` is default but with a warning if not specified.
- `-dusage=0` means only data block groups that are nearly empty (0% full) will be balanced.
- `-musage=0` means only metadata block groups that are nearly empty (0% full) will be balanced.
**Full balance on partially used block groups:**
**Perform a Full Balance on Partially Used Blocks**
```bash
btrfs balance start --full-balance -dusage=50 -musage=50 <mountpoint>
btrfs balance start --full-balance -dusage=50 -musage=50 /mnt/media/
```
- `-dusage=50`: Include data block groups less than 50% full.
- `-musage=50`: Include metadata block groups less than 50% full.
- `-dusage=50` means data block groups that are less than 50% full will be included in the balance process.
- `-musage=50` means metadata block groups that are less than 50% full will also be balanced.
**Balance data in the background:**
**Balance data in the background**
```bash
btrfs balance start --bg -d <mountpoint>
btrfs balance start --bg -d /mnt/media
```
**Balance metadata in the background:**
**Balance metadata in the background**
```bash
btrfs balance start --bg -m <mountpoint>
btrfs balance start --bg -m /mnt/media
```
**Balance data and metadata in the background:**
**Balance data and metadata in the background**
```bash
btrfs balance start --bg --full-balance -dusage=0 -musage=0 <mountpoint>
btrfs balance start --bg --full-balance -dusage=0 -musage=0 /mnt/media/
```
**Balance a limited number of chunks:**
**To balance 100 chunks of data**
```bash
btrfs balance start --bg -dlimit=100 <mountpoint>
btrfs balance start --bg -dlimit=100 /mnt/media/
```
**Convert to RAID1:**
Rebalances data and metadata to RAID1 profile. Use after adding a second drive or to switch from single to mirrored:
**Cancel Balance Operation**
```bash
btrfs balance start -mconvert=raid1 -dconvert=raid1 <mountpoint>
btrfs balance cancel /mnt/media/
```
**Cancel a balance:**
**Monitor Balance Status**
```bash
btrfs balance cancel <mountpoint>
```
**Monitor balance status:**
```bash
btrfs balance status <mountpoint>
btrfs balance status /mnt/media/
```
## Scrub
**Start a scrub**
**Start a Scrub Operation**
The scrub operation verifies data integrity against checksums:
To start a scrub operation to verify data integrity:
```bash
btrfs scrub start <mountpoint>
btrfs scrub start /mnt/media/
```
**Check scrub status:**
**Check Scrub Status**
To check the progress and status of the ongoing scrub:
```bash
btrfs scrub status <mountpoint>
btrfs scrub status /mnt/media/
```
**Cancel a scrub:**
**Cancel a Scrub Operation**
```bash
btrfs scrub cancel <mountpoint>
```
**Lower scrub I/O priority:**
Reduce the impact of a running scrub on system I/O by setting it to idle class:
```bash
ionice -c 3 -p $(pgrep btrfs-scrub)
```
- `-c 3`: Idle class — only uses I/O when no other process needs it.
**Watch scrub status and device stats:**
Continuously display scrub progress and per-device error counters:
```bash
watch -n 10 "btrfs scrub status <mountpoint>; echo ''; btrfs device stats <mountpoint>"
```
**Watch scrub status and all drive temperatures:**
```bash
watch -n 5 "btrfs scrub status <mountpoint> && echo '' && \
smartctl --scan | awk '{print \$1}' | while read dev; do \
echo -n \"\$dev: \"; \
smartctl -A \$dev | grep -iE 'Temperature|Airflow_Temp' | awk '\
/Temperature_Celsius/ {print \$10 \"°C\"} \
/Airflow_Temperature_Cel/ {print \$10 \"°C\"} \
/Temperature:/ {print \$2 \"°C\"}' | head -n 1; \
done && echo '' && btrfs device stats <mountpoint>"
btrfs scrub cancel /mnt/media/
```
## Snapshots
### Create Snapshots
**Create Snapshots**
1. **Mount the snapshots subvolume:**
1. **Mount snapshot subvolume**
```bash
mount UUID=<uuid> -o subvol=snapshots <mountpoint>
mount UUID=c9a77f3c-626f-47bd-b4e3-9a094bea287f -o subvol=snapshots /mnt/snapshots
```
2. **Create a snapshot:**
2. **Create a new snapshot**
```bash
btrfs subvolume snapshot <source-subvolume> "<mountpoint>/<snapshot-label>"
btrfs subvolume snapshot / "/mnt/snapshots/root/2021-06-26 - Debian install"
```
3. **Unmount after creating:**
3. **Unmount after creating snapshots**
```bash
umount <mountpoint>
umount /mnt/snapshots
```
### Delete Snapshots
**Delete Snapshots**
1. **Mount the snapshots subvolume:**
1. **Mount subvolume containing snapshots**
```bash
mount -o subvol=snapshots /dev/disk/by-uuid/<uuid> <mountpoint>
mount -o subvol=snapshots /dev/disk/by-uuid/7a22514b-594a-43a3-8fdd-4df1530b5465 /mnt/snapshots/
```
2. **List available snapshots:**
2. **List available snapshots**
```bash
btrfs subvol list <mountpoint>
btrfs subvol list /mnt/snapshots/
```
3. **Delete the desired snapshot:**
3. **Delete the desired snapshot**
```bash
btrfs subvolume delete <mountpoint>/<snapshot-label>
btrfs subvolume delete /mnt/snapshots/@rootfs/2024-09-15
```
4. **Unmount after deleting:**
4. **Unmount after deleting snapshots**
```bash
umount <mountpoint>
```
## Backup Procedures
**Snapshot backup procedure:**
1. Mount snapshot location:
```bash
mount UUID=<uuid> -o subvol=snapshots <mountpoint>
```
2. Create snapshots for the desired subvolumes:
```bash
btrfs subvolume snapshot / "<mountpoint>/root/<snapshot-label>"
btrfs subvolume snapshot /home "<mountpoint>/home/<snapshot-label>"
btrfs subvolume snapshot <source-subvolume> "<mountpoint>/<subvolume>/<snapshot-label>"
```
3. Unmount after creating snapshots:
```bash
umount <mountpoint>
```
## Recovery
1. **Mount a Subvolume with Recovery Options:**
```bash
mount -o recovery,subvol=<subvolume> UUID=<uuid> <mountpoint>
```
2. **Clear Cache During Mount:**
```bash
mount -o clear_cache,subvol=<subvolume> UUID=<uuid> <mountpoint>
```
3. **Data Restoration with btrfs restore:**
```bash
btrfs restore -D <device>
```
### Filesystem Check
Run offline consistency checks on an unmounted BTRFS filesystem.
**Check an unmounted filesystem:**
```bash
btrfs check <device>
```
- Must be run on an **unmounted** device. Running on a mounted filesystem risks corruption.
- Use the UUID path if needed: `/dev/disk/by-uuid/<uuid>`
**Force check (use with caution):**
```bash
btrfs check --force <device>
```
- `--force`: Bypasses the mount check. Only use this if you are certain the filesystem is not mounted and understand the risks.
### Diagnosis
Filter system logs and kernel messages to diagnose BTRFS-related events.
**Search journal logs by date range:**
```bash
journalctl --since "<date>" --until "<date>" | grep -i btrfs
```
Example:
```bash
journalctl --since "2026-01-01" --until "2026-01-02" | grep -i btrfs
```
**Search kernel ring buffer for BTRFS events:**
```bash
dmesg | grep -i btrfs
umount /mnt/snapshots/
```

0
notes/debian-packaging.md → notes/debian packaging.md
View File

152
notes/efi-repair.md
View File

@ -1,152 +0,0 @@
# EFI Partition Repair
## Table of Contents
- [EFI Partition Repair](#efi-partition-repair)
- [Table of Contents](#table-of-contents)
- [Overview](#overview)
- [Placeholders](#placeholders)
- [Repair Procedure](#repair-procedure)
- [1. Verify What You're Working With](#1-verify-what-youre-working-with)
- [2. Mount Everything for Chroot](#2-mount-everything-for-chroot)
- [3. Chroot In](#3-chroot-in)
- [4. Reinstall GRUB](#4-reinstall-grub)
- [5. Regenerate GRUB Config](#5-regenerate-grub-config)
- [6. Verify fstab](#6-verify-fstab)
- [7. Exit and Reboot](#7-exit-and-reboot)
## Overview
Use this procedure when a Debian machine fails to boot but the root filesystem appears intact — for example after a failed kernel upgrade, a corrupted EFI partition, or GRUB being wiped by another OS. Boot from a live environment, chroot into the installed system, and reinstall GRUB.
## Placeholders
Replace the placeholders below with the appropriate values for your setup:
- **Devices**
- Disk: `<disk>` (e.g., /dev/sda, /dev/nvme0n1)
- EFI partition: `<efi-part>` (e.g., /dev/sda1, /dev/nvme0n1p1)
- Root partition: `<root-part>` (e.g., /dev/sda2, /dev/nvme0n1p2)
- **Filesystem**
- Root subvolume name: `<root-subvol>` (e.g., @rootfs)
## Repair Procedure
### 1. Verify What You're Working With
Identify the disk layout, partition types, UUIDs, and current mount points:
```bash
lsblk -o NAME,SIZE,FSTYPE,PARTTYPE,UUID,MOUNTPOINT
```
Confirm which partition is EFI and which is root before proceeding.
### 2. Mount Everything for Chroot
**Mount the root filesystem**
```bash
mkdir -p /mnt
mount -o subvol=<root-subvol> <root-part> /mnt
```
**Mount the EFI partition**
```bash
mkdir -p /mnt/boot/efi
mount <efi-part> /mnt/boot/efi
```
**Bind mount the pseudo-filesystems**
```bash
mkdir -p /mnt/{dev,proc,sys,run}
mount --bind /dev /mnt/dev
mount --bind /proc /mnt/proc
mount --bind /sys /mnt/sys
mount --bind /run /mnt/run
```
**Bind mount EFI vars**
Required for `grub-install` to write boot entries to the firmware:
```bash
mount --bind /sys/firmware/efi/efivars /mnt/sys/firmware/efi/efivars
```
### 3. Chroot In
```bash
chroot /mnt /bin/bash
source /etc/profile
export PS1="(chroot) $PS1"
```
### 4. Reinstall GRUB
**Run grub-install on the disk**
```bash
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=debian --recheck <disk>
```
If you get an error about EFI vars not being writable:
```bash
mount -o remount,rw /sys/firmware/efi/efivars
```
Then re-run `grub-install`.
### 5. Regenerate GRUB Config
```bash
update-grub
```
Check the output — it should find your kernel on the btrfs volume. If it complains about not finding a kernel, you may need to also reinstall it:
**Check what kernel is installed**
```bash
dpkg -l | grep linux-image
```
**Reinstall the kernel if missing or broken**
```bash
apt-get install --reinstall linux-image-amd64
```
Then re-run `update-grub`.
### 6. Verify fstab
Confirm the EFI and root entries look correct before rebooting:
```bash
cat /etc/fstab
```
### 7. Exit and Reboot
**Exit the chroot**
```bash
exit
```
**Unmount everything**
```bash
umount -R /mnt
```
**Reboot**
```bash
reboot
```

334
notes/efi-update-mirror.md
View File

@ -1,334 +0,0 @@
# Redundant Root with EFI Mirror
## Table of Contents
- [Redundant Root with EFI Mirror](#redundant-root-with-efi-mirror)
- [Table of Contents](#table-of-contents)
- [Overview](#overview)
- [Placeholders](#placeholders)
- [How It Works](#how-it-works)
- [Paths](#paths)
- [Setup Procedure](#setup-procedure)
- [1. Format the Disks](#1-format-the-disks)
- [2. Convert the Root Filesystem to BTRFS RAID1](#2-convert-the-root-filesystem-to-btrfs-raid1)
- [3. Configure fstab](#3-configure-fstab)
- [4. Install the Sync Script](#4-install-the-sync-script)
- [5. Configure apt to Run the Script Post-Install](#5-configure-apt-to-run-the-script-post-install)
- [6. Test](#6-test)
- [Moving to a New Disk](#moving-to-a-new-disk)
## Overview
This setup provides a redundant Debian root drive using BTRFS RAID1 across two disks, with a synced EFI partition on both so the machine can boot from either. The script `update-efi-mirror` keeps the backup EFI in sync and re-installs GRUB on both physical disks. It is triggered automatically after every `apt` operation.
## Placeholders
Replace the placeholders below with the appropriate values for your setup:
- **Devices**
- Primary disk: `<primary-disk>` (e.g., /dev/sda, /dev/nvme0n1)
- Backup disk: `<backup-disk>` (e.g., /dev/sdb, /dev/nvme1n1)
- Primary EFI partition: `<primary-efi-part>` (e.g., /dev/sda1, /dev/nvme0n1p1)
- Backup EFI partition: `<backup-efi-part>` (e.g., /dev/sdb1, /dev/nvme1n1p1)
- Primary root partition: `<primary-root-part>` (e.g., /dev/sda2, /dev/nvme0n1p2)
- Backup root partition: `<backup-root-part>` (e.g., /dev/sdb2, /dev/nvme1n1p2)
- Primary swap partition: `<primary-swap-part>` (e.g., /dev/sda3, /dev/nvme0n1p3)
- Backup swap partition: `<backup-swap-part>` (e.g., /dev/sdb3, /dev/nvme1n1p3)
- **UUIDs**
- Primary EFI UUID: `<primary-efi-uuid>` (e.g., D167-0F46)
- Backup EFI UUID: `<backup-efi-uuid>` (e.g., 08E8-A87C)
- Root BTRFS UUID: `<root-uuid>` (e.g., 387526ec-f3bd-4fa8-a17e-e985121ada0b)
- Primary swap UUID: `<primary-swap-uuid>`
- Backup swap UUID: `<backup-swap-uuid>`
## How It Works
- The root filesystem is BTRFS RAID1 across two partitions. Either disk alone can serve reads and writes.
- Each disk has its own EFI partition. GRUB is installed on both physical disks.
- After every `apt` install or upgrade, an apt hook calls `update-efi-mirror`, which:
1. Rsyncs the live `/boot/efi` to the backup EFI partition.
2. Runs `grub-install` on both physical disks.
- Either disk can be removed and the system will still boot and run.
## Paths
- **Sync script**: `/usr/local/sbin/update-efi-mirror`
- **Apt hook**: `/etc/apt/apt.conf.d/99-update-efi-mirror`
- **fstab**: `/etc/fstab`
## Setup Procedure
### 1. Format the Disks
**Verify which disks are present before touching anything**
```bash
lsblk
lsblk -f
```
**Open `fdisk` on the backup disk**
If the disk was previously used at a smaller capacity, the GPT backup table may be misaligned — `fdisk` will correct it on write.
```bash
fdisk <backup-disk>
```
Create a GPT partition table if not already present:
```
g
```
**Check the primary disk partition layout and note the exact sector boundaries**
```bash
fdisk -l <primary-disk>
```
This shows the `Start` and `End` sectors for each partition. Use these values when partitioning the backup disk so the layouts match exactly.
**Create the three partitions to match the layout of the primary disk**
| # | Size | Type |
| --- | --------- | ---------- |
| 1 | 830M | EFI System |
| 2 | ~12G | Linux fs |
| 3 | Remainder | Linux swap |
Inside `fdisk`, the sequence is:
```
n # new partition
# enter the exact End sector from the primary disk to match partition sizes
t # change type
1 # select partition 1
EFI System # set type to EFI
t # change type
3 # select partition 3
Linux swap # set type to swap
w # write and exit
```
**Partition sizes must be at least as large as the corresponding primary partition** — the BTRFS replace will fail if the target partition is smaller than the source.
**Format the new EFI partition**
```bash
apt install dosfstools # if not installed
mkfs.fat -F32 <backup-efi-part>
```
**Format the new swap partition**
```bash
mkswap <backup-swap-part>
```
**Check UUIDs of the new partitions**
```bash
lsblk -f <backup-disk>
```
### 2. Convert the Root Filesystem to BTRFS RAID1
**Add the backup partition to the BTRFS volume**
```bash
btrfs device add <backup-root-part> /
```
**Convert to RAID1**
```bash
btrfs balance start -mconvert=raid1 -dconvert=raid1 /
```
Monitor the balance:
```bash
btrfs balance status /
```
**Verify the final layout**
```bash
btrfs filesystem show /
btrfs fi usage /
```
### 3. Configure fstab
Edit `/etc/fstab` to mount the primary EFI partition normally and document the backup EFI UUID in a commented-out entry. The script reads the backup UUID from this comment.
**Edit fstab**
```bash
nano /etc/fstab
```
The fstab should contain:
```
# Primary EFI — mounted at boot
UUID=<primary-efi-uuid> /boot/efi vfat umask=0077 0 1
# Backup EFI — not auto-mounted, used by update-efi-mirror
#UUID=<backup-efi-uuid> /boot/efi vfat umask=0077 0 1
# Root — BTRFS RAID1 (both partitions share the same UUID)
UUID=<root-uuid> / btrfs defaults,noatime 0 1
# Swap — both disks
UUID=<primary-swap-uuid> none swap sw 0 0
UUID=<backup-swap-uuid> none swap sw 0 0
```
The backup EFI line must start with `#UUID=` — this is the format the script greps for:
```bash
grep "^#UUID=" /etc/fstab | grep "/boot/efi"
```
**Reload systemd after editing fstab**
```bash
systemctl daemon-reload
```
### 4. Install the Sync Script
**Create the script**
```bash
nano /usr/local/sbin/update-efi-mirror
```
Paste the contents of `update-efi-mirror.sh`.
**Set ownership and permissions**
```bash
chown root:root /usr/local/sbin/update-efi-mirror
chmod 755 /usr/local/sbin/update-efi-mirror
```
The script will:
1. Detect the currently mounted EFI partition and its disk.
2. Read the backup EFI UUID from the `#UUID=` comment in `/etc/fstab`.
3. Mount the backup EFI to a temp directory, rsync all files from the live EFI, then unmount.
4. Run `grub-install --recheck` on both physical disks.
### 5. Configure apt to Run the Script Post-Install
**Create the apt hook**
```bash
nano /etc/apt/apt.conf.d/99-update-efi-mirror
```
Add:
```
DPkg::Post-Invoke {"if [ -x /usr/local/sbin/update-efi-mirror ]; then /usr/local/sbin/update-efi-mirror; fi";};
```
This runs the sync script after every `apt` operation that invokes `dpkg` — including kernel upgrades, which update `/boot/efi` with new initramfs and GRUB config.
### 6. Test
**Run the script manually**
```bash
/usr/local/sbin/update-efi-mirror
```
Expected output:
```
Syncing EFI files: <primary-efi-uuid> -> <backup-efi-uuid>
EFI Files Sync Complete.
Updating GRUB bootloader on both physical disks...
Redundancy Complete: Files synced and GRUB updated on <primary-disk> and <backup-disk>.
```
**Test the apt hook**
```bash
apt update
apt upgrade -y
```
The sync output should appear at the end of the upgrade, after GRUB regenerates its config.
**Verify the backup EFI contents match**
```bash
TMPMNT=$(mktemp -d)
mount -U <backup-efi-uuid> "$TMPMNT"
diff -r /boot/efi/ "$TMPMNT/"
umount "$TMPMNT"
rmdir "$TMPMNT"
```
**Verify GRUB is installed on both disks**
```bash
fdisk -l <primary-disk> | grep -i efi
fdisk -l <backup-disk> | grep -i efi
```
**`Optional` Boot from the backup disk**
Swap the boot order in BIOS/UEFI to confirm the machine boots cleanly from `<backup-disk>`.
## Moving to a New Disk
Use this procedure when replacing one of the two disks in the BTRFS RAID1 — for example when upgrading to a larger drive.
**1. Partition the new disk** following [Step 1](#1-format-the-disks).
**2. Find the device ID of the partition to be replaced:**
```bash
btrfs filesystem show /
```
**3. Shrink the filesystem on the disk being replaced to fit the new partition:**
```bash
btrfs filesystem resize <device-id>:<size> /
```
**4. Replace the old partition with the new one:**
```bash
btrfs replace start <device-id> <new-root-part> /
btrfs replace status /
```
**5. Grow the new partition to max:**
```bash
btrfs filesystem resize <device-id>:max /
```
**6. Update fstab** if the backup EFI UUID has changed (new disk = new EFI UUID):
```bash
lsblk -f <new-disk>
nano /etc/fstab
systemctl daemon-reload
```
**7. Run the sync script** to install GRUB on the new disk and populate its EFI partition:
```bash
/usr/local/sbin/update-efi-mirror
```
**8. Verify** following [Step 6](#6-test).

400
notes/linux.md
View File

@ -7,6 +7,7 @@
- [System Information](#system-information)
- [Hardware Information](#hardware-information)
- [Software Information](#software-information)
- [Commands to Get Information About Linux Version, Kernel Version, and Release](#commands-to-get-information-about-linux-version-kernel-version-and-release)
- [User Management](#user-management)
- [User Information](#user-information)
- [Super User Management](#super-user-management)
@ -14,8 +15,8 @@
- [System Management](#system-management)
- [Change password of a tar/openssl archive](#change-password-of-a-taropenssl-archive)
- [Verify two possibly identical folders recursively](#verify-two-possibly-identical-folders-recursively)
- [NFS](#nfs)
- [Network Diagnostics](#network-diagnostics)
- [USB Devices](#usb-devices)
- [Test USB Key](#test-usb-key)
- [Diagnosis](#diagnosis)
- [Debian Upgrade Issues](#debian-upgrade-issues)
- [Wayland Issues](#wayland-issues)
@ -32,46 +33,25 @@ To gather detailed information about your hardware, use the following commands:
- **`hwinfo`**: Offers detailed information about hardware components and can be more verbose than `lshw`.
- **`lsscsi`**: Lists SCSI devices, including disks and other SCSI-attached hardware.
- **`lsusb`**: Shows information about USB devices connected to your system.
- **`lsblk`**: Lists block devices such as hard drives and their partitions.
- **`df -H`**: Displays disk space usage in a human-readable format.
- **`fdisk -l`**: Lists all partitions on the system.
- **`dmidecode`**: Retrieves hardware information from the BIOS. Use:
- `dmidecode -t processor` for CPU details
- `dmidecode -t memory` for RAM details
- `dmidecode -t bios` for BIOS information
**CPU information**
```bash
lscpu
cat /proc/cpuinfo
grep -c 'model name' /proc/cpuinfo
```
- `lscpu`: Structured summary of CPU architecture, cores, threads, and NUMA topology.
- `cat /proc/cpuinfo`: Raw per-core details including model name, flags, and frequencies.
- `grep -c 'model name'`: Quick count of logical CPU cores.
**GPU information**
```bash
lspci | grep -i vga
```
**CPU frequency scaling driver**
Check which driver is managing CPU frequency scaling (e.g., `intel_pstate`, `acpi-cpufreq`):
```bash
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver
```
### Software Information
**Finding information on the Linux distribution**
# Commands to Get Information About Linux Version, Kernel Version, and Release
- **`lsb_release -a`**: Displays detailed information about the Linux distribution, including the distributor ID, description, release number, and codename.
- **`cat /etc/debian_version`**: Displays the version of the Debian distribution if you're running a Debian-based system (like Ubuntu).
- **`cat /etc/os-release`**: Displays information about the operating system, such as the name, version, and ID of the distribution.
- **`cat /etc/*release`**: Searches for any files in the `/etc/` directory that contain the word `release` and displays their contents. This typically includes more detailed distribution information.
- **`cat /etc/*version`**: Similar to `cat /etc/*release`, but looks for files containing the word `version`. It can provide additional version-related details.
- **`cat /etc/*version`**: Similar to `cat /etc/*release`, but it looks for files containing the word `version`. It can provide additional version-related details.
- **`hostnamectl`**: Displays system information related to the hostname and other metadata about the system. This may include the operating system, kernel version, and architecture.
**Finding Path to Binary**
@ -79,17 +59,17 @@ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver
To find the location of an executable binary, use:
```bash
type <binary-name>
type composer
```
This command will show the path to a binary executable, ex `composer`, if it's available in your `PATH`.
This command will show the path to the `composer` executable if it's available in your `PATH`.
**Number of Words in a File**
To count the number of words in a file, use:
```bash
wc <filepath>
wc filepath
```
This command will show the number of words along with other details like lines and characters.
@ -99,9 +79,11 @@ This command will show the number of words along with other details like lines a
To count the number of lines in a file, use:
```bash
wc -l <filepath>
wc -l filepath
```
This command will display the number of lines in the specified file.
## User Management
### User Information
@ -111,23 +93,25 @@ wc -l <filepath>
This variation of the adduser command uses the --gecos option to pre-fill the user's information (Full name, Room number, Work Phone, Home Phone, and Email) non-interactively, allowing you to automate user creation with predefined details.
```bash
adduser --gecos "<full-name>,,,<email>" <username>
adduser --gecos "Fabrice Quenneville,,,fabrice@fabq.ca" fabrice
```
This variation creates a system user with a Bash shell, no password login (--disabled-password), a specified home directory (`/home/<username>`), and adds the user to a new group, while using the --gecos option to set the full name as `<service-description>`.
This variation of the adduser command creates a system user named "aptly" with a Bash shell, no password login (--disabled-password), a specified home directory (/home/aptly), and adds the user to a new group, while using the --gecos option to set the full name as "Aptly repository".
```bash
adduser --system --shell /bin/bash --gecos '<service-description>' --group --disabled-password --home /home/<username> <username>
adduser --system --shell /bin/bash --gecos 'Aptly repository' --group --disabled-password --home /home/aptly aptly
```
**List Users**
To list all users from the `/etc/passwd` file in alphabetical order, use:
To list all users from the `/etc/passwd` file, use:
```bash
awk -F':' '{ print $1}' /etc/passwd | sort
```
This command extracts the usernames from the `/etc/passwd` file and sorts them in alphabetical order.
### Super User Management
**Disable Root Login**
@ -140,7 +124,7 @@ To disable root login via SSH, perform the following steps:
nano /etc/ssh/sshd_config
```
Comment out the line containing `PermitRootLogin`.
Comment out the line containing `PermitRootLogin` by adding a `#` at the beginning of the line.
2. **Change Shell for Root User:**
@ -148,36 +132,34 @@ To disable root login via SSH, perform the following steps:
nano /etc/passwd
```
Find the line starting with `root` and change `/bin/bash` to `/sbin/nologin`.
Find the line starting with `root` and change `/bin/bash` to `/sbin/nologin` to disable login for the root user.
Save and close the file. Restart the SSH service for changes to take effect:
```bash
systemctl restart ssh
```
**Sudo Management**
**Add User to Sudo Group**
```bash
adduser <username> sudo
adduser fabrice sudo
```
**Update Sudoers File to Remove Password Requirement**
Edit the sudoers file with the default editor:
Edit the sudoers file:
```bash
visudo
```
Edit the sudoers file with `nano`:
```bash
EDITOR=nano visudo
```
Add the following line to allow the user to execute commands without a password:
```bash
<username> ALL=(ALL) NOPASSWD:ALL
fabrice ALL=(ALL) NOPASSWD:ALL
```
### Switch User
@ -185,74 +167,82 @@ Add the following line to allow the user to execute commands without a password:
**Switch to Another User as Sudoer**
```bash
sudo -i -u <username>
sudo -i -u postgres
```
This command switches to the `postgres` user with sudo privileges.
**Switch to Another User as Root**
```bash
su - <username>
su - postgres
```
This command switches to the `postgres` user with root privileges.
**Run command as specific user**
```bash
sudo -u <username> <command>
sudo -u www-data somecommand and arguments
```
**Change shell of a user**
```bash
chsh -s /bin/bash <username>
chsh -s /usr/sbin/nologin <username>
chsh -s /bin/bash www-data
chsh -s /usr/sbin/nologin www-data
```
**Change user with specific shell**
```bash
sudo -u <username> bash
sudo -u www-data bash
```
## System Management
**Ensure hostname or add alias**
Set or update the hostname for your server.
```bash
nano /etc/hosts
# Add the hostname alias:
# 127.0.1.1 <hostname-intranet>
# 127.0.1.1 local.servername.domain.com
nano /etc/hostname
# Set the main hostname:
# 127.0.1.1 <hostname-intranet> <hostname-short>
# 127.0.1.1 servername.domain.com servername
hostnamectl set-hostname <hostname-intranet>
hostnamectl set-hostname servername.domain.com
```
**Tar backup for a large number of small files**
These commands create backups using `tar` and transfer them securely over SSH.
Create a tar archive and transfer it to a remote server:
```bash
tar -c /path/to/dir | ssh <username>@<hostname-intranet> 'tar -xvf - -C /absolute/path/to/remotedir'
tar -c /path/to/dir | ssh fabrice@servername.domain.com 'tar -xvf - -C /absolute/path/to/remotedir'
```
Compress and transfer a folder, then store it as a `.tar.gz` file:
Compress and transfer a folder, then store it as a .tar.gz file:
```bash
tar zcvf - /folder | ssh <username>@<hostname-intranet> "cat > /backup/folder.tar.gz"
tar zcvf - /folder | ssh fabrice@servername.domain.com "cat > /backup/folder.tar.gz"
```
Transfer a compressed `.tar.gz` file and extract it on the remote server:
Transfer a compressed .tar.gz file and extract it on the remote server:
```bash
cat folder.tar.gz | ssh <username>@<hostname-intranet> "tar zxvf -"
cat folder.tar.gz | ssh fabrice@servername.domain.com "tar zxvf -"
```
Alternative: change directory on the remote server before extracting:
Alternative method: change directory on the remote server before extracting:
```bash
cat folder.tar.gz | ssh <username>@<hostname-intranet> "cd /path/to/dest/; tar zxvf -"
cat folder.tar.gz | ssh fabrice@servername.domain.com "cd /path/to/dest/; tar zxvf -"
```
**List time zones**
@ -276,130 +266,83 @@ Alternatively, manually set the time zone by linking the correct file:
```bash
mv /etc/localtime /etc/localtime-old
ln -s /usr/share/zoneinfo/<timezone> /etc/localtime
ln -s /usr/share/zoneinfo/America/Toronto /etc/localtime
```
**Find a specific service**
```bash
systemctl list-units --type=service | grep <service-name>
```
**Bind mount a directory**
Make a directory available at another path, useful during chroot recovery or container setup:
Search for a specific service running on your system.
```bash
mount --bind /dev /mnt/<newroot>/dev
systemctl list-units --type=service | grep php
```
**Chroot into another system**
Enter a mounted system's root as if it were the running OS. Useful for recovery, initramfs rebuilds, or bootloader fixes:
```bash
chroot /mnt/<newroot>
```
Typically preceded by binding the required pseudo-filesystems:
```bash
mount --bind /dev /mnt/<newroot>/dev
mount --bind /proc /mnt/<newroot>/proc
mount --bind /sys /mnt/<newroot>/sys
chroot /mnt/<newroot>
```
**Rebuild initramfs**
After kernel or driver changes, rebuild the initramfs and refresh the GRUB configuration:
```bash
update-initramfs -u
update-initramfs -u -k all
```
- `update-initramfs -u`: Rebuilds the initramfs for the currently running kernel.
- `-k all`: Rebuilds for all installed kernels.
**Rebuild initramfs for a specific kernel version:**
```bash
update-initramfs -c -k $(uname -r)
```
- `-c`: Create a new initramfs (instead of updating).
- `-k $(uname -r)`: Targets the currently running kernel version.
**Update GRUB:**
```bash
update-grub
```
- Scans for kernels and regenerates `/boot/grub/grub.cfg`.
### Change password of a tar/openssl archive
**Decrypt the archive**
To decrypt an `openssl`-encrypted archive using a password stored in a file:
1. **Store your password in a temporary file:**
1. **Store your password in this file.**
```bash
nano $HOME/xyz001.txt
```
2. **Decrypt the archive**
Decrypt the archive using the password stored in xyz001.txt.
```bash
nano $HOME/<filename>
openssl aes-256-cbc -d -pbkdf2 -in servername-backup.tar.gz -out servername-backup.tar -pass file:$HOME/xyz001.txt
```
2. **Decrypt the archive:**
3. **Re-encrypt the archive with a new password**
```bash
openssl aes-256-cbc -d -pbkdf2 -in <archive>.tar.gz -out <archive>.tar -pass file:$HOME/<filename>
```
3. **Re-encrypt the archive with a new password:**
```bash
nano $HOME/<filename>
openssl aes-256-cbc -e -pbkdf2 -in <archive>.tar -out <archive>-new.tar.gz -pass file:$HOME/<filename>
rm $HOME/<filename>
nano $HOME/xyz001.txt
openssl aes-256-cbc -e -pbkdf2 -in servername-backup.tar -out servername-backup-new.tar.gz -pass file:$HOME/xyz001.txt
rm $HOME/xyz001.txt
```
**Decode / Extract**
Decrypt and extract the contents directly into a directory:
To decrypt and extract the contents of an encrypted archive directly into a directory:
```bash
nano $HOME/<filename>
openssl aes-256-cbc -d -pbkdf2 -in <archive>.tar.gz -pass file:<filename> | tar xz -C .
rm $HOME/<filename>
nano $HOME/xyz001.txt
openssl aes-256-cbc -d -pbkdf2 -in servername-backup.tar.gz -pass file:xyz001.txt | tar xz -C .
rm $HOME/xyz001.txt
```
### Verify two possibly identical folders recursively
**With `diff`**
Check for differences between two directories, comparing all files recursively:
```bash
diff -r <dir1>/ <dir2>/
diff -r servername-files/data/servername-repositories/ servername-repositories/
```
Outputs any differences found between the two directories.
**With `rsync`**
Dry run — shows differences without copying any data:
Use `rsync` to show differences without copying any data:
```bash
rsync -avn <dir1>/ <dir2>/
rsync -avn servername-files/data/servername-repositories/ servername-repositories/
```
- `-n`: dry run, no changes made.
- The `-n` flag means this is a dry run, which won’t make any changes.
**With `cmp`**
This script compares files in two directories and identifies any differences between matching file names.
```bash
#!/bin/bash
dir1="<dir1>/"
dir2="<dir2>/"
dir1="servername-files/data/servername-repositories/"
dir2="servername-repositories/"
# Check if both directories exist before proceeding.
if [ ! -d "$dir1" ] || [ ! -d "$dir2" ]; then
@ -418,71 +361,140 @@ for file1 in $(find "$dir1" -type f); do
done
```
## NFS
## USB Devices
**Show NFS exports from a server:**
### Test USB Key
**Device Information**
Check if the system recognizes the device and show the latest system messages related to USB devices being connected:
```bash
showmount -e <hostname>
showmount -e localhost
lsusb
dmesg | tail -n 20
```
**List active exports and their options on the server:**
**Find Mount Points and Device Information**
Identify mount points, partitions, and other relevant details of mounted devices:
```bash
exportfs -v
lsblk -f
df -h | grep /dev/sdc
findmnt /dev/sdc1
mount | grep /dev/sd
```
## Network Diagnostics
**Print Detailed Information About the USB Key**
**List network interfaces**
View detailed partition and disk information:
```bash
ip link show
fdisk -l /dev/sdc
```
**List network bridges and their attached interfaces**
**Test the File System**
Check and repair the filesystem on the USB key:
```bash
brctl show
fsck /dev/sdc1
```
**Measure HTTP response timing:**
**Test Data Integrity**
Breaks down the full request lifecycle — useful for diagnosing DNS, TLS, or TTFB issues:
Perform read/write tests to ensure the integrity of the USB key:
1. **Unmount the USB Key** (if mounted):
```bash
curl -o /dev/null -s -w \
'Lookup: %{time_namelookup}s\nConnect: %{time_connect}s\nAppConnect: %{time_appconnect}s\nTTFB: %{time_starttransfer}s\nTotal: %{time_total}s\n' \
https://<hostname>
umount /media/fabrice/BD48-F8BB
```
- `time_namelookup`: DNS resolution time.
- `time_connect`: TCP connection time.
- `time_appconnect`: TLS handshake time.
- `time_starttransfer`: Time to first byte (TTFB).
- `-o /dev/null`: Discards the response body.
**High-frequency ping:**
Flood-style ping to stress-test latency or detect intermittent packet loss:
2. **Write Test**:
```bash
ping -i 0.002 <host>
dd if=/dev/zero of=/dev/sdc bs=4M count=256 status=progress
```
- `-i 0.002`: Send a packet every 2ms. Requires root.
**Jumbo frame ping:**
Test whether the network path supports large MTU frames (useful for diagnosing MTU mismatches):
3. **Read Test**:
```bash
ping -s 1472 -i 0.01 <host>
dd if=/dev/sdc of=/dev/null bs=4M count=256 status=progress
```
- `-s 1472`: Payload size of 1472 bytes (1472 + 28-byte IP/ICMP header = 1500-byte MTU).
- Increase `-s` to test jumbo frames (e.g., `-s 8972` for 9000-byte MTU).
**Check for Bad Blocks**
Identify any bad sectors on the USB key:
- **Read-only test**:
```bash
badblocks -v /dev/sdc
```
- **Non-destructive read-write test**:
```bash
badblocks -nsv /dev/sdc
```
- The `-n` option performs a non-destructive read-write test.
- The `-s` option shows progress.
- The `-v` option is for verbose output.
**Perform a SMART Test**
Run SMART diagnostics to test the health of the USB key:
1. **Start a short SMART test**:
```bash
smartctl -t short /dev/sdc
```
2. **View test results**:
```bash
smartctl -a /dev/sdc
```
**Benchmark the Speed**
Measure the read speed of the USB key:
```bash
hdparm -t /dev/sdc
```
**Unmount and Safely Remove**
Unmount the USB key and safely remove it from the system:
```bash
umount /mnt/usb
eject /dev/sdc
```
**Switching two USB keys**
The following commands copy data between two USB drives, format one of them, and restore the data.
```bash
cp -r /media/fabrice/465A-759B "/tmp/Michael Allison"
umount /dev/sdc1
mkfs.vfat /dev/sdc1
umount /dev/sdc1
dd if=/dev/sdc of=/tmp/usb_image.img bs=4M status=progress
mkfs.vfat /dev/sdc1
cp -r "/tmp/Michael Allison" /media/fabrice/D67D-ADF8
umount /dev/sdc1
dd if=/tmp/usb_image.img of=/dev/sdc bs=4M status=progress
sync
```
## Diagnosis
@ -490,6 +502,8 @@ ping -s 1472 -i 0.01 <host>
**Apt Logs**
View the APT logs to check for package installation and updates history:
```bash
less /var/log/apt/history.log
```
@ -515,45 +529,23 @@ journalctl -b | grep -i "drm\|gpu\|display\|wayland\|monitor"
journalctl -b | grep -i "gnome-shell"
```
**Journal Filtering by Date and Keyword**
Search logs within a specific time window:
```bash
journalctl --since "<date>" --until "<date>" | grep -i <keyword>
```
Example:
```bash
journalctl --since "2026-01-01" --until "2026-01-02" | grep -i btrfs
```
**Kernel microcode events:**
```bash
journalctl -k | grep -i "microcode"
```
- `-k`: Show only kernel messages (equivalent to `dmesg` output via the journal).
## Fonts
**Download and Install Fonts**
1. **Download the Font Archive:**
1. **Download the Font Archive**:
```bash
wget https://<font-archive-url>
wget https://path/to/font/archive.tar.gz
```
2. **Extract the Font Files:**
2. **Extract the Font Files**:
```bash
tar -xzvf <font-archive>.tar.gz
tar -xzvf font-archive.tar.gz
```
3. **Copy the Font Files:**
3. **Copy the Font Files to the Local Fonts Directory**:
**Local font directory**
@ -575,12 +567,14 @@ journalctl -k | grep -i "microcode"
**Update the Font Cache**
**Force a Reload of the Installed Font Cache**:
```bash
sudo su -
fc-cache -fv
fc-cache -frv
```
- **`-f`**: Force re-generation of cache files, overriding timestamp checking.
- **`-f`**: Force re-generation of apparently up-to-date cache files, overriding the timestamp checking.
- **`-r`**: Erase all existing cache files and rescan.
- **`-v`**: Display status information while busy.

0
notes/pip-packaging.md → notes/pip packaging.md
View File

308
notes/ssh.md
View File

@ -4,9 +4,7 @@
- [SSH](#ssh)
- [Table of Contents](#table-of-contents)
- [Placeholders](#placeholders)
- [Connect with specific key](#connect-with-specific-key)
- [Skip Host Key Verification](#skip-host-key-verification)
- [SSH Key Management](#ssh-key-management)
- [Verbose](#verbose)
- [Enable root login](#enable-root-login)
@ -18,124 +16,93 @@
- [Change SSH Port](#change-ssh-port)
- [Restart ssh](#restart-ssh)
## Placeholders
Replace the placeholders below with the appropriate values for your setup:
- **Connection**
- Username: `<username>` (e.g., john)
- Hostname: `<hostname>` (e.g., server.example.com)
- IP address: `<ip>` (e.g., 192.168.1.100)
- SSH port: `<port>` (e.g., 2222)
- SSH key: `<keyfile>` (e.g., ~/.ssh/id_rsa)
- Key comment: `<key-comment>` (e.g., john@example.com)
- Host alias: `<alias>` (e.g., myserver)
- **Paths**
- Local file: `<local-path>` (e.g., /home/user/file.txt)
- Remote path: `<remote-path>` (e.g., /home/user/file.txt)
- Local script: `<script-path>` (e.g., /home/user/script.sh)
- Project name: `<project>` (e.g., myapp)
## Connect with specific key
```bash
ssh -i <keyfile> root@<hostname>
ssh -i <keyfile> <username>@<hostname>
ssh -i /home/fabrice/.ssh/fabquenneville root@servername.fabq.ca
ssh -i /home/fabrice/.ssh/fabquenneville fabrice@servername.fabq.ca
```
## Skip Host Key Verification
Useful for ephemeral machines, VMs, or hosts that are frequently rebuilt where saved known_hosts entries would cause conflicts:
```bash
ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null <username>@<hostname>
```
- `StrictHostKeyChecking=no`: Automatically accepts new or changed host keys without prompting.
- `UserKnownHostsFile=/dev/null`: Discards the host key entirely — nothing is saved to `~/.ssh/known_hosts`.
- ⚠️ Do not use on untrusted networks — this disables MITM protection.
## SSH Key Management
**Generate a new RSA SSH key pair with a 4096-bit key length**
```bash
ssh-keygen -t rsa -b 4096 -C "<key-comment>" -f <keyfile>
ssh-keygen -t rsa -b 4096 -C "fabrice@fabq.ca" -f ~/.ssh/fabrice@fabq.ca
```
- `ssh-keygen -t rsa -b 4096`: Generates a new RSA SSH key pair with a key size of 4096 bits for enhanced security.
- `-C "<key-comment>"`: Adds a comment to the key, usually the email address of the key owner.
- `-f <keyfile>`: Specifies the filename for the private key; the public key will be saved with the same name but with a `.pub` extension.
- `ssh-keygen -t rsa -b 4096`: This command generates a new RSA SSH key pair with a key size of 4096 bits for enhanced security.
- `-C "fabrice@fabq.ca"`: This option adds a comment to the key, usually the email address of the key owner.
- `-f ~/.ssh/fabrice@fabq.ca`: This specifies the filename for the private key; the public key will be saved with the same name but with a `.pub` extension.
**Copy the generated SSH keys to the remote server**
```bash
scp <keyfile>* <username>@<hostname>:~/.ssh/
scp ~/.ssh/fabrice@fabq.ca* fabrice@servername.fabq.ca:~/.ssh/
```
- `scp <keyfile>*`: Securely copies both the private and public keys to the remote server.
- `<username>@<hostname>:~/.ssh/`: Specifies the destination path on the remote server where the keys will be copied.
- `scp ~/.ssh/fabrice@fabq.ca*`: This command securely copies both the private and public keys to the remote server.
- `fabrice@servername.fabq.ca:~/.ssh/`: Specifies the destination path on the remote server where the keys will be copied.
**Install the public key on the remote server for passwordless authentication**
```bash
ssh-copy-id <username>@<ip>
ssh-copy-id <username>@<hostname>
ssh-copy-id fabrice@192.168.1.100
ssh-copy-id fabrice@servername.fabq.ca
```
- `ssh-copy-id`: Installs the public key on the specified remote server, allowing for passwordless SSH login.
- `ssh-copy-id "fabrice@servername.fabq.ca"`: This command installs the public key on the specified remote server, allowing for passwordless SSH login.
**Install the public key on multiple servers using a specific private key**
**Install the public key on multiple servers using specific private key**
```bash
ssh-copy-id -i <keyfile> root@<ip>
ssh-copy-id -i <keyfile> <username>@<hostname>
ssh-copy-id -i /home/fabrice/.ssh/fabquenneville root@192.168.1.100
ssh-copy-id -i /home/fabrice/.ssh/fabquenneville fabrice@servername.fabq.ca
```
- `-i <keyfile>`: Specifies which private key to use for authentication when copying the public key.
- `ssh-copy-id -i /home/fabrice/.ssh/fabquenneville`: This specifies which private key to use for authentication when copying the public key.
- `root@192.168.1.100` and `fabrice@servername.fabq.ca`: These commands install the public key on the respective remote servers, allowing for secure, passwordless access.
**Install the public key on the remote server for passwordless authentication manually**
This process is useful when ssh-copy-id is unavailable, or when you want more granular control over the manual setup of passwordless SSH authentication. The ssh-copy-id tool automatically installs your public key on the remote machine, but if you prefer or need to do it manually, these are the steps:
This process is useful when `ssh-copy-id` is unavailable, or when you want more granular control over the setup. Follow these steps on the remote server:
1. Create the `.ssh` directory if it doesn't exist and set proper permissions:
1. Create the .ssh directory if it doesn't exist and set proper permissions
```bash
mkdir -p /home/<username>/.ssh
chmod 700 /home/<username>/.ssh
mkdir -p /home/fabrice/.ssh
chmod 700 /home/fabrice/.ssh
```
2. Open the `authorized_keys` file and paste the public key (usually from `~/.ssh/id_rsa.pub` on the local machine):
2. Open the authorized_keys file in an editor and paste the public key (usually from ~/.ssh/id_rsa.pub on the local machine)
```bash
nano /home/<username>/.ssh/authorized_keys
nano /home/fabrice/.ssh/authorized_keys
```
3. Set the correct permissions for the `authorized_keys` file:
3. Set the correct permissions for the authorized_keys file
```bash
chmod 600 /home/<username>/.ssh/authorized_keys
chmod 600 /home/fabrice/.ssh/authorized_keys
```
4. Ensure the ownership of the `.ssh` directory and its contents is set to the correct user:
4. Ensure the ownership of the .ssh directory and its contents is set to the correct user
```bash
chown -R <username>:<username> /home/<username>/.ssh
chown -R fabrice:fabrice /home/fabrice/.ssh
```
## Verbose
Use the `-v` option to enable verbose mode, which provides detailed information about the connection process:
- Use the 'ssh' command with the '-v' option to enable verbose mode, which provides detailed information about the connection process.
```bash
ssh -i <keyfile> -v root@<hostname>
ssh -i <keyfile> -v <username>@<hostname>
ssh -i /home/fabrice/.ssh/fabquenneville -v root@servername.fabq.ca
ssh -i /home/fabrice/.ssh/fabquenneville -v fabrice@servername.fabq.ca
```
## Enable root login
Modify the SSH configuration file to allow root login:
- Modify the SSH configuration file to allow root login.
```bash
nano /etc/ssh/sshd_config
@ -156,203 +123,284 @@ PermitRootLogin yes
firewall-cmd --permanent --zone=public --add-service=ssh
```
- `firewall-cmd`: The command-line tool used to manage `firewalld`.
- `--permanent`: Ensures that the change persists across reboots.
- `firewall-cmd`: This is the command-line tool used to manage `firewalld`.
- `--permanent`: This option ensures that the change persists across reboots.
- `--zone=public`: Specifies the zone to which the rule applies. The "public" zone is typically used for untrusted networks.
- `--add-service=ssh`: Adds the SSH service to the specified zone, allowing incoming SSH connections.
- `--add-service=ssh`: This adds the SSH service to the specified zone, allowing incoming SSH connections.
**Examples of configuring other Linux firewalls**
1. **Using UFW (Uncomplicated Firewall)**
**Allow SSH traffic**
```bash
ufw allow ssh
```
- This command allows incoming SSH traffic through the firewall. UFW is designed to simplify the process of managing a firewall.
2. **Using iptables**
**Allow SSH traffic**
```bash
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
```
- `iptables`: This is a low-level tool for managing Linux firewalls.
- `-A INPUT`: Appends the rule to the INPUT chain.
- `-p tcp --dport 22`: Matches TCP traffic on port 22.
- `-j ACCEPT`: Accepts the specified traffic.
- `-p tcp`: Specifies that this rule applies to TCP packets.
- `--dport 22`: Indicates that this rule applies to traffic on port 22 (the default SSH port).
- `-j ACCEPT`: Instructs the firewall to accept the specified traffic.
3. **Using nftables**
**Allow SSH traffic**
```bash
nft add rule ip filter input tcp dport 22 accept
```
- `nft`: The command-line tool for interacting with the nftables framework.
- `add rule ip filter input`: Adds a new rule to the input chain of the filter table.
- `tcp dport 22`: Matches TCP packets directed to port 22.
- `accept`: Accepts the matching packets.
- `accept`: Specifies that the matching packets should be accepted.
**Note:** Reload or restart the firewall service after making changes to apply the new rules.
**Note:** Be sure to reload or restart the firewall service after making changes to apply the new rules effectively.
## SCP (Secure Copy Protocol)
The `scp` command securely transfers files and directories between local and remote systems over SSH.
- The `scp` command is used to securely transfer files and directories between local and remote systems over SSH.
**Copy a local file to a remote server:**
**Copy Local File to Remote Server**
To copy a file from your local machine to a remote server, use the following syntax:
```bash
scp <local-path> <username>@<hostname>:<remote-path>
scp /local/file/path fabrice@servername.fabq.ca:/remote/file/path
```
**Copy a directory recursively:**
- `/local/file/path`: Specify the path to the local file you want to copy.
- `fabrice@servername.fabq.ca`: This is the user and remote server where the file will be copied.
- `/remote/file/path`: This is the destination path on the remote server.
**Copy a Directory**
To copy an entire directory, use the `-r` option, which stands for "recursive":
```bash
scp -r <local-path>/ <username>@<hostname>:<remote-path>/
scp -r /home/fabrice/foldername/ root@servername.fabq.ca:/remote/parent/
```
- `-r`: Enables recursive copying of directories and their contents.
- `-r`: This option enables recursive copying of directories and their contents.
- `/home/fabrice/foldername/`: The path to the local directory you wish to copy.
- `root@servername.fabq.ca:/remote/parent/`: The destination path on the remote server where the directory will be copied.
**Copy a file using a specific SSH key:**
**Copy a Configuration File Using a Specific SSH Key**
To copy a configuration file while specifying a particular SSH key for authentication, use the following command:
```bash
scp -i <keyfile> <local-path> <username>@<hostname>:<remote-path>
scp -i /home/fabrice/.ssh/fabquenneville /mnt/workbench/webserver/projectname/config.ini fabrice@servername.fabq.ca:/mnt/workbench/projectname/
```
- `-i <keyfile>`: Specifies the identity file (private key) for authentication.
- `-i /home/fabrice/.ssh/fabquenneville`: This option specifies the identity file (private key) for authentication.
- `/mnt/workbench/webserver/projectname/config.ini`: The path to the local configuration file being transferred.
- `fabrice@servername.fabq.ca`: The user and server to which the file is being copied.
- `/mnt/workbench/projectname/`: The destination path on the remote server where the file will be stored.
## Send Remote Commands
**Run a local script on a remote server:**
**Execute Commands Directly on a Remote Server**
You can use the `ssh` command to execute various commands on a remote server. Here are some examples:
**Run a Local Script on a Remote Server**
```bash
ssh <username>@<hostname> 'bash -s' < <script-path>
ssh fabrice@servername.fabq.ca 'bash -s' < /local/path/to/script.sh
```
- This command will execute the local script located at `/local/path/to/script.sh` on the remote server.
**Remove a file:**
```bash
ssh <username>@<ip> "rm <remote-path>"
ssh fabrice@192.168.1.100 "rm /home/fabrice/filename.log"
```
**Mount all filesystems:**
```bash
ssh root@<hostname> "mount -a"
ssh root@servername.fabq.ca "mount -a"
```
**Reboot the remote server:**
```bash
ssh root@<hostname> "reboot -h now"
ssh root@servername.fabq.ca "reboot -h now"
```
**Connect using a host key alias:**
```bash
ssh -o "HostKeyAlias=<alias>" root@<ip>
ssh -o "HostKeyAlias=servername" root@192.168.1.100
```
## Test Connection, Add Alias, and Update Known Hosts
**Test connection with host key alias:**
**Test Connection with Host Key Alias**
Commands to establish an SSH connection while specifying a host key alias. This helps avoid conflicts with existing entries in the `known_hosts` file.
Use the following commands to establish an SSH connection while specifying a host key alias. This helps avoid conflicts with existing entries in the `known_hosts` file.
```bash
ssh -o 'HostKeyAlias=<hostname>' <username>@<ip>
ssh -o 'HostKeyAlias=<alias>' <username>@<ip>
ssh -o 'HostKeyAlias=servername.fabq.ca' fabrice@192.168.1.100
ssh -o 'HostKeyAlias=servername' fabrice@192.168.1.100
```
**Test host identity without authenticating:**
**Test Host Identity without Authenticating**
To test the identity of a remote server without fully authenticating and to check connectivity, use the following command:
```bash
ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=<alias>' <username>@<ip> /bin/true
ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=servername' fabrice@192.168.1.100 /bin/true
```
- `-e none`: Disables escape character processing.
- `-o 'BatchMode=yes'`: Suppresses all prompts, suitable for scripts.
- `/bin/true`: Simple command that always returns success, confirming the connection without further actions.
- `-e none`: Disables encryption for this command, which is useful in specific testing scenarios.
- `-o 'BatchMode=yes'`: Ensures that SSH does not prompt for user interaction, making it suitable for scripts.
- `/bin/true`: Executes a simple command that always returns success, confirming the connection without further actions.
**Retrieve public SSH keys from a remote server:**
This command allows you to verify that you can connect to the server while avoiding any authentication prompts.
```bash
ssh-keyscan -H <hostname>
**Retrieve Public SSH Keys**
To retrieve the public SSH keys from a remote server, use the following command:
```
ssh-keyscan -H servername.fabq.ca
```
- Fetches the server's public SSH keys without establishing a full session. Used to pre-populate `known_hosts`.
- This command fetches the public SSH keys from the specified server, allowing you to add them to your `known_hosts` file.
- It does not establish a full SSH session and is primarily used for key retrieval, which helps ensure secure connections in future interactions.
**Add an alias to SSH config for easy access:**
By using both commands, you can test connectivity to a remote server and gather its public SSH keys for secure authentication later.
**Add Alias to SSH Config for Easy Access**
To simplify your SSH connections, you can create an alias for your SSH connections by editing the `~/.ssh/config` file:
```ini
Host <alias>
HostName <hostname>
User <username>
IdentityFile <keyfile>
Host servername
HostName servername.fabq.ca
User fabrice
IdentityFile ~/.ssh/fabquenneville
```
- `Host servername`: This defines the alias you will use for the SSH connection.
- `HostName servername.fabq.ca`: This is the actual hostname of the remote server.
- `User fabrice`: This specifies the user to log in as.
- `IdentityFile ~/.ssh/fabquenneville`: This indicates the SSH key file to use for authentication.
## Remove Offending SSH Keys
When a server's host key has changed, remove the old entry from `known_hosts`.
When you encounter an "offending key" warning when connecting to a server, you can remove the old key from the `known_hosts` file. This is necessary if the server's host key has changed.
**View known hosts:**
**View Known Hosts**
To view the contents of your `known_hosts` file, use:
```bash
cat ~/.ssh/known_hosts
```
**Edit known hosts manually:**
**Edit Known Hosts Manually (Optional)**
You can edit the `known_hosts` file manually if you prefer:
```bash
nano ~/.ssh/known_hosts
```
**Update known hosts with current server key:**
**Update Known Hosts File with SSH Key**
```bash
ssh-keyscan -H <hostname> >> ~/.ssh/known_hosts
ssh-keyscan -H servername.fabq.ca >> ~/.ssh/known_hosts
```
**Remove offending key by hostname or IP:**
- This command retrieves the public key of the specified server and appends it to your `known_hosts` file, allowing SSH to recognize the server during subsequent connections.
**Remove Offending Key by Hostname**
You can use the `ssh-keygen` command to remove specific keys from your `known_hosts` file. Here are examples for different scenarios:
- To remove the offending key for a specific server:
```bash
ssh-keygen -R "<hostname>"
ssh-keygen -R "<ip>"
ssh-keygen -R "servername.fabq.ca"
ssh-keygen -R "192.168.1.100"
```
**Remove offending key specifying the known_hosts file:**
- To specify the `known_hosts` file directly:
```bash
ssh-keygen -f "/home/<username>/.ssh/known_hosts" -R "<hostname>"
ssh-keygen -f "/root/.ssh/known_hosts" -R "<ip>"
ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R "<hostname>"
ssh-keygen -f "/home/fabrice/.ssh/known_hosts" -R "servername.fabq.ca"
ssh-keygen -f "/root/.ssh/known_hosts" -R "192.168.1.100"
ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R "servername.fabq.ca"
```
**Summary of Key Removal**
You can also use a shorthand command to remove the offending key without specifying the file:
```bash
ssh-keygen -R servername.fabq.ca
```
This will automatically target the correct `known_hosts` file based on your user and system configuration.
## Change SSH Port
**1. Edit the SSH configuration file:**
To enhance security, you may want to change the default SSH port (22) to a custom port. Follow these steps:
**1. Edit the SSH Configuration File**
Open the SSH daemon configuration file using a text editor:
```bash
nano /etc/ssh/sshd_config
```
Set the desired port:
Edit the following line to set a new port (e.g., port 2222):
```ini
Port <port>
Port 2222
```
**2. Create directory for systemd override:**
- Locate the line that specifies the port (usually `#Port 22`) and change it to your desired port number (e.g., `Port 2222`).
- Make sure to uncomment the line by removing the `#`.
**2. Create Directory for Systemd Override**
If you're using systemd, create a directory for the SSH socket override:
```bash
mkdir -p /etc/systemd/system/ssh.socket.d
```
**3. Create the override configuration file:**
**3. Create an Override Configuration File**
Create or edit the override configuration file for the SSH socket:
```bash
nano /etc/systemd/system/ssh.socket.d/override.conf
```
- Add the following lines to specify the custom port:
```ini
[Socket]
ListenPort=<port>
ListenPort=2222 # Replace with your desired port number
```
**4. (Optional) Edit the Sockets Target Configuration**
@ -363,20 +411,32 @@ You may also want to edit the sockets target configuration to ensure it points t
nano /etc/systemd/system/sockets.target.wants/ssh.socket
```
**5. Restart the SSH service to apply the changes:**
- Make any necessary adjustments based on your custom port.
**5. Restart the SSH service to apply the changes**
After making changes, restart the SSH service to apply the new configuration:
```bash
systemctl restart sshd
```
**6. Verify the new port:**
**6. (Optional) Verify the New Port**
To verify that SSH is listening on the new port, you can use:
```bash
netstat -tuln | grep LISTEN
```
This will display the ports currently being listened to, allowing you to confirm that your changes were successful.
## Restart ssh
**Restart the SSH service to apply changes**
To restart the SSH service, use the following command:
```bash
systemctl restart sshd
```

530
notes/storage.md
View File

@ -1,530 +0,0 @@
# Storage
## Table of Contents
- [Storage](#storage)
- [Table of Contents](#table-of-contents)
- [Placeholders](#placeholders)
- [Drive Information](#drive-information)
- [List Block Devices](#list-block-devices)
- [List Mountpoints and Usage](#list-mountpoints-and-usage)
- [Inspect fstab](#inspect-fstab)
- [Find Device Path from UUID](#find-device-path-from-uuid)
- [Power On Hours](#power-on-hours)
- [Swap](#swap)
- [Partitions and Filesystems](#partitions-and-filesystems)
- [TRIM](#trim)
- [Mounting](#mounting)
- [SMART Diagnostics](#smart-diagnostics)
- [Hardware Monitoring](#hardware-monitoring)
- [Kernel Messages](#kernel-messages)
- [Badblocks](#badblocks)
- [Hex Dump](#hex-dump)
- [Cloning drives and images with dd](#cloning-drives-and-images-with-dd)
- [Benchmarking](#benchmarking)
- [USB Devices](#usb-devices)
- [Device Recognition](#device-recognition)
- [Test Device Integrity](#test-device-integrity)
- [Switching Two USB Keys](#switching-two-usb-keys)
## Placeholders
Replace the placeholders below with the appropriate values for your setup:
- **Devices**
- Block device: `<device>` (e.g., /dev/sda)
- Partition: `<partition>` (e.g., /dev/sda1)
- UUID: `<uuid>` (e.g., a1b2c3d4-e5f6-7890-abcd-ef1234567890)
- **Paths**
- Mount point: `<mountpoint>` (e.g., /mnt/media)
- Directory path: `<dirpath>` (e.g., /mnt/data)
## Drive Information
### List Block Devices
Check all attached block devices by type:
```bash
ls /dev/sd*
ls /dev/nv*
```
### List Mountpoints and Usage
**Tree view of all block devices with sizes and mountpoints**
```bash
lsblk
lsblk -f
```
- `-f`: Add filesystem type, label, and UUID to the tree.
**Exclude loop devices from the listing**
```bash
lsblk -e 7
```
- `-e 7`: Excludes device major number 7 (loop devices), keeping the output clean on systems with many snaps or loop mounts.
**Disk space usage for all mounted filesystems**
```bash
df -h
```
- `-h`: Human-readable.
**Disk space usage for a specific device**
```bash
df -h | grep <device>
```
### Inspect fstab
**View all configured mount entries**
```bash
cat /etc/fstab
```
### Find Device Path from UUID
Using `lsblk`:
```bash
lsblk -o NAME,UUID,MOUNTPOINT
```
Using `blkid`:
```bash
blkid | grep <uuid>
blkid -U <uuid>
blkid <partition>
```
### Power On Hours
Check power-on hours across multiple drives at once:
```bash
for dev in /dev/sd{a,b,c,d}; do echo -n "$dev: "; smartctl -a $dev | grep "Power_On_Hours"; done
for dev in /dev/sd{a..d}; do echo -n "$dev: "; smartctl -a $dev | grep "Power_On_Hours"; done
```
### Swap
Check Swap currently used by the system:
```bash
swapon --show
```
## Partitions and Filesystems
**View partition table and disk details:**
```bash
fdisk -l <device>
```
**Check and repair a filesystem:**
```bash
fsck <partition>
```
## TRIM
TRIM allows the OS to inform the drive which blocks are no longer in use, maintaining performance on SSDs and NVMe drives over time.
**Run TRIM once manually across all mounted filesystems:**
```bash
fstrim -av
```
- `-a`: Run on all mounted filesystems that support TRIM.
- `-v`: Verbose — reports how much space was freed per filesystem.
**Enable the weekly TRIM timer:**
```bash
systemctl enable --now fstrim.timer
```
- Debian/Ubuntu run this weekly by default once enabled.
- Check timer status with `systemctl status fstrim.timer`.
## Mounting
**Validate all `fstab` entries**
```bash
findmnt --verify --verbose
```
**Dry-run mount of all fstab entries, reports what would succeed or fail**
```bash
mount -fav
```
- `-f`: Fake mount — goes through the motions without making the actual syscall.
- `-a`: Process all entries in `/etc/fstab` (excluding `noauto`).
- `-v`: Verbose output.
**Find mount point details:**
```bash
findmnt <device>
mount | grep <device>
```
**Check active mounts for a specific mountpoint:**
```bash
cat /proc/mounts | grep <mountpoint>
```
**Apply fstab changes without rebooting**
```bash
mount -o remount -a
```
**Unmount and Safely Remove**
```bash
umount <device>
eject <device>
```
## SMART Diagnostics
**Get full SMART information for a drive:**
```bash
smartctl -a <device>
```
**Get drive identity and model info only:**
```bash
smartctl -i <device>
```
```bash
for dev in /dev/sd[a-z] /dev/nvme[0-9]n[0-9]; do
echo "--- $dev ---"
smartctl -i $dev | grep -Ei "Model|Serial Number|Capacity"
done
```
**Run a short SMART test:**
```bash
smartctl -t short <device>
```
**Run a long SMART test:**
```bash
smartctl -t long <device>
```
**View test results:**
```bash
smartctl -a <device>
```
**Check SMART device statistics (error counters, etc.):**
```bash
smartctl -A <device>
```
**Filter for key health attributes:**
Check the most important failure indicators in a single line:
```bash
smartctl -A <device> | grep -E "Reallocated|Pending|UDMA_CRC"
```
**Check multiple health attributes at once:**
```bash
smartctl -a <device> | grep -E "Power_On_Hours|Load_Cycle_Count|Reallocated_Sector_Ct"
```
**Check temperatures across all drives:**
Scans all SMART-capable devices and prints their temperature:
```bash
smartctl --scan | awk '{print $1}' | while read dev; do \
echo -n "$dev: "; \
smartctl -A $dev | grep -iE 'Temperature|Airflow_Temp' | awk '
/Temperature_Celsius/ {print $10 "°C"}
/Airflow_Temperature_Cel/ {print $10 "°C"}
/Temperature:/ {print $2 "°C"}
' | head -n 1; \
done
```
**Watch drive temperatures continuously:**
```bash
watch -n 5 "smartctl --scan | awk '{print \$1}' | while read dev; do \
echo -n \"\$dev: \"; \
smartctl -A \$dev | grep -iE 'Temperature|Airflow_Temp' | awk '\
/Temperature_Celsius/ {print \$10 \"°C\"} \
/Airflow_Temperature_Cel/ {print \$10 \"°C\"} \
/Temperature:/ {print \$2 \"°C\"}' | head -n 1; \
done"
```
## Hardware Monitoring
**Install lm-sensors:**
```bash
apt install lm-sensors
```
**Detect available sensor chips:**
Run once after installation to probe for hardware sensors:
```bash
sensors-detect
```
**Display current sensor readings:**
Shows CPU, GPU, and motherboard temperatures, fan speeds, and voltages:
```bash
sensors
```
## Kernel Messages
**Tail the most recent kernel messages:**
```bash
dmesg | tail -n 25
```
**Show only errors and warnings:**
```bash
dmesg --level=err,warn
```
**Show kernel messages with human-readable timestamps:**
```bash
dmesg -T
```
**Filter for NVMe events:**
```bash
dmesg | grep -i nvme
dmesg -w | grep -i nvme
```
- `-w`: Follow — print new messages as they arrive (like `tail -f`).
**Filter for ATA/SCSI/SATA/NVMe device events:**
```bash
dmesg | grep -i -E 'scsi|ata|nvme|sata'
```
**Filter for I/O errors:**
```bash
dmesg | grep -i "I/O error"
```
**Filter for ATA/SCSI/SATA/NVMe device errors:**
```bash
dmesg | grep -i -E 'scsi|ata|nvme|sata'
```
**Map ATA port number to block device name:**
When `dmesg` reports an error on e.g. `ata7` and you need to identify which physical drive that is:
```bash
ls -l /sys/class/block/ | grep ata<port-number>
dmesg -T | grep -iE "ata"
```
**Filter for BTRFS events:**
```bash
dmesg | grep -i btrfs
```
## Badblocks
**Read-only test:**
```bash
badblocks -v <device>
```
**Non-destructive read-write test:**
`Save to RAM -> Write Pattern -> Verify -> Restore`
```bash
badblocks -nsv <device>
```
- `-n`: Non-destructive read-write test.
- `-s`: Show progress.
- `-v`: Verbose output.
- Warning: If the computer loses power or the kernel panics after Step 2 but before Step 4, that specific block of data will be lost
**Destructive write test**
Overwrites all data — use only on blank drives or drives to be deleted.
```bash
badblocks -wsv <device>
```
## Hex Dump
Inspect raw bytes on a device directly, useful for verifying partition tables, boot sectors, or investigating corruption.
**View the first 512 bytes (MBR / partition table):**
```bash
hexdump -C -n 512 <device>
```
**View 1 MB of data starting at a specific offset:**
```bash
hexdump -C -s 1G -n 1M <device>
```
- `-C`: Canonical format — hex on the left, ASCII on the right.
- `-n <length>`: Number of bytes to read.
- `-s <offset>`: Skip to this offset before reading.
**Extract readable strings from a raw device:**
Useful for locating file paths, filenames, or metadata remnants directly on a block device:
```bash
strings <device> | grep -C 200 "<search-term>" > <output-file>.txt
```
- `-C 200`: Show 200 lines of context around each match.
- Redirect to a file — output can be very large on multi-TB drives.
## Cloning drives and images with dd
**Clone a drive or create an image:**
```bash
dd if=<source-device> of=<destination-device> bs=4M status=progress
dd if=<source-device> of=<image-file>.img bs=4M status=progress
```
**Restore from image:**
```bash
dd if=<image-file>.img of=<destination-device> bs=4M status=progress
sync
```
## Benchmarking
**Measure read speed with `hdparm`:**
```bash
hdparm -t <device>
```
## USB Devices
### Device Recognition
Check if the system recognizes the device and show recent USB-related kernel messages:
```bash
lsusb
dmesg | tail -n 20
```
### Test Device Integrity
Warning: The write test is destructive. All data on <device> will be permanently overwritten. Verify the target device with lsblk before running.
1. **Unmount the device** (if mounted):
```bash
umount <mountpoint>
```
2. **Write test:**
```bash
dd if=/dev/zero of=<device> bs=4M count=256 status=progress
```
3. **Read test:**
```bash
dd if=<device> of=/dev/null bs=4M count=256 status=progress
```
4. **Check for bad blocks:**
```bash
badblocks -wsv <device>
```
5. **Run a SMART test:**
```bash
smartctl -t short <device>
smartctl -a <device>
```
### Switching Two USB Keys
Copy data off, reformat, and restore:
```bash
cp -r /media/<username>/<volume-label> /tmp/<backup-folder>
umount <device-partition>
mkfs.vfat <device-partition>
cp -r /tmp/<backup-folder> /media/<username>/<new-volume-label>
umount <device-partition>
```
Or clone with `dd`:
```bash
dd if=<source-device> of=/tmp/usb_image.img bs=4M status=progress
mkfs.vfat <device-partition>
dd if=/tmp/usb_image.img of=<destination-device> bs=4M status=progress
sync
```

163
notes/wordpress.md
View File

@ -1,163 +0,0 @@
# WordPress
## Table of Contents
- [WordPress](#wordpress)
- [Table of Contents](#table-of-contents)
- [Documentation](#documentation)
- [Common WordPress Errors](#common-wordpress-errors)
- [Debugging Steps](#debugging-steps)
- [Debugging Tools](#debugging-tools)
## Documentation
- [wp-cli](https://developer.wordpress.org/cli/commands/)
- [Documentation Overview](https://www.wordpress.info/doc/overview/)
- [Tutorials](https://wordpress.com/learn/)
## Common WordPress Errors
1. **HTTP Errors**
- **404 Not Found**: Page or resource missing.
- **403 Forbidden**: Insufficient permissions to access the resource.
- **500 Internal Server Error**: Generic error, often caused by server misconfiguration or PHP errors.
- **503 Service Unavailable**: Server is overloaded or in maintenance mode.
2. **Database Errors**
- **Error Establishing a Database Connection**: Database credentials incorrect or database server is down.
- **Table Prefix Issues**: Wrong `$table_prefix` in `wp-config.php`.
- **Corrupt Database**: Can be fixed using `wp db repair`.
3. **PHP Errors**
- **Parse Error**: Syntax error in PHP files.
- **Fatal Error**: Missing function or class.
- **Deprecated Function Warnings**: Old functions being used.
## Debugging Steps
1. **Enable Debug Mode**
Edit `wp-config.php`:
```php
// Enable Debug Mode
define('WP_DEBUG', true);
// Enable Debug logging to the /wp-content/debug.log file
define('WP_DEBUG_LOG', true);
// Disable display of WordPress errors and warnings
define('WP_DEBUG_DISPLAY', false);
// Disable display of PHP errors and warnings
@ini_set('display_errors', 0);
```
Logs will be saved in `wp-content/debug.log`.
2. **Check `.htaccess`**
Ensure WordPress rules exist:
```apache
# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPress
```
3. **Check Apache and VirtualHost config**
**Redhat and derivatives**
- `/etc/httpd/conf/httpd.conf`
- `/etc/httpd/conf.d/welcome.conf`
**Debian and derivatives**
- `/etc/apache2/apache2.conf`
- `/etc/apache2/sites-available/000-default.conf`
```apache
<Directory "/var/www">
AllowOverride All
# Allow open access:
Require all granted
</Directory>
<Directory "/var/www/html">
Options Indexes FollowSymLinks
AllowOverride All
Require all granted
</Directory>
```
4. **Reset File Permissions**
```bash
find /var/www/html -type d -exec chmod 755 {} \;
find /var/www/html -type f -exec chmod 644 {} \;
```
4. **Disable Plugins & Themes**
Hide the plugin directory:
```bash
mv wp-content/plugins wp-content/.plugins
```
Switch to a default theme:
```bash
wp theme install twentytwentyfive --activate
wp theme activate twentytwentyfive
```
5. **Check Server Logs**
For Apache:
```bash
tail -f /var/log/httpd/error_log
```
For Nginx:
```bash
tail -f /var/log/nginx/error.log
```
6. **Verify Database Connection**
```bash
wp db check
```
If corrupt:
```bash
wp db repair
```
7. **Increase Memory Limit**
Edit `wp-config.php`:
```php
define('WP_MEMORY_LIMIT', '256M');
```
## Debugging Tools
- **WP-CLI**: Command-line interface for WordPress.
- **Query Monitor**: Plugin for analyzing database queries and errors.
- **Health Check & Troubleshooting**: Plugin for diagnosing issues.

1
requirements.txt
View File

@ -1 +0,0 @@
argcomplete

125
scripts/change_case.py
View File

@ -1,125 +0,0 @@
#!/usr/bin/env python3
# === Standard library ===
import sys
import os
from pathlib import Path
from typing import Optional
from argparse import ArgumentParser, Namespace
# Allow importing from scripts/library even when run directly
project_root = str(Path(__file__).resolve().parent.parent)
if project_root not in sys.path:
sys.path.append(project_root)
# === Local import ===
from scripts.library import parse_verbose, run_in_venv
def rename_by_case(root_dir: str, case: str, recursive: bool = False) -> None:
"""
Renames all files and directories in a given directory by changing their case.
Args:
root_dir (str): Root path to start renaming.
case (str): One of "lower", "upper", "capitalize".
recursive (bool): Whether to apply renaming recursively.
Returns:
None
"""
def apply_case(name: str) -> str:
if case == "lower":
return name.lower()
elif case == "upper":
return name.upper()
elif case == "capitalize":
return name.capitalize()
else:
raise ValueError(f"Unsupported case transformation: {case}")
for current_dir, dirs, files in os.walk(root_dir, topdown=False):
# Skip descending into subdirectories if not recursive
if not recursive and current_dir != root_dir:
continue
# Rename files
for name in files:
old_path = os.path.join(current_dir, name)
new_name = apply_case(name)
new_path = os.path.join(current_dir, new_name)
if old_path != new_path:
if not os.path.exists(new_path):
os.rename(old_path, new_path)
else:
print(
f"Cannot rename {old_path} to {new_path}: target already exists."
)
# Rename directories
for name in dirs:
old_path = os.path.join(current_dir, name)
new_name = apply_case(name)
new_path = os.path.join(current_dir, new_name)
if old_path != new_path:
if not os.path.exists(new_path):
os.rename(old_path, new_path)
else:
print(
f"Cannot rename {old_path} to {new_path}: target already exists."
)
def parse_args() -> Namespace:
"""
Parse and validate command-line arguments.
Returns:
Namespace: Parsed arguments with 'case' and 'recursive' options.
"""
import argcomplete
parser = ArgumentParser(
description="Rename files and directories by changing case (lower, upper, capitalize)."
)
parser.add_argument(
"--case",
"-c",
type=str,
choices=["lower", "upper", "capitalize"],
default="lower",
help="Case transformation to apply: 'lower', 'upper', or 'capitalize'. Default is 'lower'.",
)
parser.add_argument(
"--recursive",
"-r",
action="store_true",
help="Apply the transformation recursively in all subdirectories.",
)
parser.add_argument(
"-v",
"--verbose",
action="store_true",
help="Enable verbose output (debug information, warnings)",
)
argcomplete.autocomplete(parser)
return parser.parse_args()
def main() -> None:
"""
Main entry point. Parses arguments and runs case renaming in current working directory.
"""
args = parse_args()
cwd: str = os.getcwd()
rename_by_case(cwd, case=args.case, recursive=args.recursive)
if __name__ == "__main__":
args = parse_verbose()
run_in_venv(main, verbose=args.verbose)

50
scripts/efi-update-mirror.sh
View File

@ -1,50 +0,0 @@
#!/bin/bash
# Identify the currently mounted EFI partition and its device
SOURCE_MOUNT="/boot/efi"
SOURCE_PART=$(findmnt -no SOURCE "$SOURCE_MOUNT")
SOURCE_UUID=$(findmnt -no UUID "$SOURCE_MOUNT")
SOURCE_DISK=$(lsblk -no PKNAME "$SOURCE_PART")
SOURCE_DISK_PATH="/dev/$SOURCE_DISK"
# Extract the backup UUID from fstab
BACKUP_UUID=$(grep "^#UUID=" /etc/fstab | grep "/boot/efi" | awk -F'[= ]' '{print $2}')
if [ -z "$BACKUP_UUID" ]; then
echo "Error: Backup EFI UUID not found in /etc/fstab"
exit 1
fi
if [ "$SOURCE_UUID" == "$BACKUP_UUID" ]; then
echo "Error: Source and Backup UUIDs are identical. Check /etc/fstab."
exit 1
fi
# Identify the physical disk for the backup UUID
BACKUP_PART=$(blkid -U "$BACKUP_UUID")
BACKUP_DISK=$(lsblk -no PKNAME "$BACKUP_PART")
BACKUP_DISK_PATH="/dev/$BACKUP_DISK"
# Define temporary mount point
TARGET_MOUNT=$(mktemp -d -t efi-sync.XXXXXXXXXX)
# Trap to ensure the directory is removed even if the script fails
trap 'rm -rf "$TARGET_MOUNT"' EXIT
# Mount, Sync, and Cleanup
if mount -U "$BACKUP_UUID" "$TARGET_MOUNT"; then
echo "Syncing EFI files: $SOURCE_UUID -> $BACKUP_UUID"
rsync -aqAX --delete "$SOURCE_MOUNT/" "$TARGET_MOUNT/"
umount "$TARGET_MOUNT"
echo "EFI Files Sync Complete."
else
echo "Error: Could not mount backup EFI ($BACKUP_UUID)"
exit 1
fi
# Ensure GRUB binary is installed on BOTH physical disks
echo "Updating GRUB bootloader on both physical disks..."
grub-install "$SOURCE_DISK_PATH" --recheck > /dev/null 2>&1
grub-install "$BACKUP_DISK_PATH" --recheck > /dev/null 2>&1
echo "Redundancy Complete: Files synced and GRUB updated on $SOURCE_DISK_PATH and $BACKUP_DISK_PATH."

20
scripts/library/__init__.py
View File

@ -1,20 +0,0 @@
# scripts/library/__init__.py
from .tools import (
apply_resolution_rename,
deletefile,
findfreename,
get_intermediate_dirs,
get_spacer,
)
from .venv_utils import parse_verbose, run_in_venv
__all__ = [
"apply_resolution_rename",
"deletefile",
"findfreename",
"get_intermediate_dirs",
"get_spacer",
"parse_verbose",
"run_in_venv",
]

134
scripts/library/tools.py
View File

@ -1,134 +0,0 @@
#!/usr/bin/env python3
import colorama
import os
import re
colorama.init()
creset = colorama.Fore.RESET
cgreen = colorama.Fore.GREEN
cred = colorama.Fore.RED
# Define patterns to search for various resolutions
resolution_patterns = {
2160: r"(4096x2160|3840x2160|2880p|2160p|2160|1440p|4k)",
1080: r"(1920x1080|1080p|1080)",
720: r"(1280x720|720p|720)",
}
# Wrap each pattern to only match when isolated by non-alphanumeric chars or string boundaries
resolution_patterns = {
res: rf"(?<![a-zA-Z0-9]){pat}(?![a-zA-Z0-9])"
for res, pat in resolution_patterns.items()
}
def apply_resolution_rename(name, max_height):
"""
Apply resolution substitution to a filename or directory name if an isolated
resolution pattern greater than max_height is found.
Args:
name (str): The filename or directory name to process (without path).
max_height (int): The target resolution height to substitute to.
Returns:
str: The renamed string if a match was found, or the original string unchanged.
"""
for resolution, pattern in resolution_patterns.items():
if resolution > max_height and re.search(pattern, name, re.IGNORECASE):
return re.sub(pattern, f"{max_height}p", name, flags=re.IGNORECASE)
return name
def deletefile(filepath):
"""Delete a file, Returns a boolean
Args:
filepath : A string containing the full filepath
Returns:
Bool : The success of the operation
"""
try:
os.remove(filepath)
except OSError:
print(f"{cred}Error deleting {filepath}{creset}")
return False
print(f"{cgreen}Successfully deleted {filepath}{creset}")
return True
def get_intermediate_dirs(input_path, file_path):
"""
Return the list of directory paths between input_path (exclusive) and
the directory containing file_path (inclusive), ordered deepest first.
Args:
input_path (str): The root path (exclusive upper bound).
file_path (str): The full path to a file.
Returns:
list: List of directory paths, deepest first.
Raises:
ValueError: If file_path not under input_path.
"""
input_path = os.path.realpath(input_path)
current = os.path.realpath(os.path.dirname(file_path))
if current != input_path and not current.startswith(input_path + os.sep):
raise ValueError(
f"file_path '{file_path}' is not under input_path '{input_path}'"
)
dirs = []
while current != input_path and current != os.path.dirname(current):
dirs.append(current)
current = os.path.dirname(current)
return dirs
def findfreename(filepath, attempt=0):
"""
Given a filepath, it will try to find a free filename by appending a number to the name.
First, it tries using the filepath passed in the argument, then adds a number to the end. If all fail, it appends (#).
Args:
filepath (str): A string containing the full filepath.
attempt (int): The number of attempts already made to find a free filename.
Returns:
str: The first free filepath found.
"""
attempt += 1
filename = str(filepath)[: str(filepath).rindex(".")]
extension = str(filepath)[str(filepath).rindex(".") :]
copynumpath = filename + f"({attempt})" + extension
if not os.path.exists(filepath) and attempt <= 2:
return filepath
elif not os.path.exists(copynumpath):
return copynumpath
return findfreename(filepath, attempt)
def get_spacer(name: str) -> str:
"""
Determine the spacer character ('-' or '_') to use for naming a virtual environment
based on the given project name. Default to underscore.
Args:
name (str): The project name to analyze.
Returns:
str: The detected spacer ('-' or '_'), or '_' if none found.
"""
if "-" in name:
return "-"
elif "_" in name:
return "_"
return "_"

149
scripts/library/venv_utils.py
View File

@ -1,149 +0,0 @@
#!/usr/bin/env python3
# === Standard library ===
import os
import sys
import subprocess
import venv
from pathlib import Path
from typing import Callable
from argparse import ArgumentParser, Namespace
# === Configuration ===
# Define the virtual environment directory in the user's home folder
VENV_DIR: Path = Path.home() / ".scripts_fabq_venv"
# Define the path to the requirements.txt file in the project root
REQUIREMENTS_FILE: Path = Path(__file__).resolve().parents[2] / "requirements.txt"
def parse_verbose() -> Namespace:
"""
Parse the --verbose flag before virtual environment setup.
This is useful when argument parsing requires packages that may
only be available after the virtual environment is initialized.
Returns:
Namespace: Contains a single attribute 'verbose' (bool).
"""
parser = ArgumentParser(add_help=False)
parser.add_argument(
"-v",
"--verbose",
action="count",
default=0,
help="Increase verbosity: -v = verbose, -vv = debug, -vvv = trace",
)
parser.add_argument(
"--debug",
dest="verbose",
action="store_const",
const=2,
help="Enable debug output (equivalent to -vv)",
)
parser.add_argument(
"--trace",
dest="verbose",
action="store_const",
const=3,
help="Enable full trace output (equivalent to -vvv)",
)
return parser.parse_known_args()[0]
def is_in_venv() -> bool:
"""
Determine whether the script is currently running inside a virtual environment.
Returns:
bool: True if running inside a virtual environment, False otherwise.
"""
return hasattr(sys, "real_prefix") or sys.prefix != getattr(
sys, "base_prefix", sys.prefix
)
def create_venv(verbose: bool = False) -> None:
"""
Create a virtual environment in the predefined VENV_DIR location.
Args:
verbose (bool): If True, prints progress messages.
"""
if verbose >= 2:
print(f"Creating virtual environment in {VENV_DIR}...")
venv.create(VENV_DIR, with_pip=True)
def install_requirements(verbose: bool = False) -> None:
"""
Install Python dependencies from requirements.txt into the venv.
Args:
verbose (bool): If True, prints progress and pip output.
"""
if not REQUIREMENTS_FILE.exists():
print(f"Error: requirements.txt not found at {REQUIREMENTS_FILE}")
sys.exit(1)
if verbose >= 2:
print(f"Installing/updating requirements from {REQUIREMENTS_FILE}...")
pip_cmd = [str(VENV_DIR / "bin" / "pip")]
try:
subprocess.check_call(
pip_cmd + ["install", "-r", str(REQUIREMENTS_FILE)],
stdout=None if verbose >= 2 else subprocess.DEVNULL,
stderr=None if verbose >= 2 else subprocess.DEVNULL,
)
except subprocess.CalledProcessError as e:
print(f"Error: Failed to install requirements: {e}")
sys.exit(1)
try:
subprocess.check_call(
pip_cmd + ["check"],
stdout=None if verbose >= 2 else subprocess.DEVNULL,
stderr=None if verbose >= 2 else subprocess.DEVNULL,
)
except subprocess.CalledProcessError:
print("Error: Dependency conflict detected — your environment may be broken.")
raise
def run_in_venv(main_func: Callable[[], None], verbose: bool = False) -> None:
"""
Ensure the script is running inside the expected virtual environment.
If not already in the venv:
- Create it if needed
- Install requirements
- Relaunch the script inside the venv
If already in the venv:
- Ensure requirements are installed
- Execute the given main function
Args:
main_func (Callable[[], None]): The function to execute after setup.
verbose (bool): If True, enables progress messages and pip output.
"""
if not is_in_venv():
if not VENV_DIR.exists():
create_venv(verbose)
install_requirements(verbose)
# Relaunch the script inside the virtual environment
os.execv(
str(VENV_DIR / "bin" / "python"),
[str(VENV_DIR / "bin" / "python")] + sys.argv,
)
else:
install_requirements(verbose)
main_func()

662
scripts/video_autoreduce.py
View File

@ -1,662 +0,0 @@
#!/usr/bin/env python3
# video_autoreduce.py
import argparse
import argcomplete
import colorama
import os
import re
import subprocess
import sys
import json
from pathlib import Path
# Allow importing from scripts/library even when run directly
project_root = str(Path(__file__).resolve().parent.parent)
if project_root not in sys.path:
sys.path.append(project_root)
# === Local import ===
from scripts.library import (
deletefile,
findfreename,
apply_resolution_rename,
get_intermediate_dirs,
)
colorama.init()
creset = colorama.Fore.RESET
ccyan = colorama.Fore.CYAN
cyellow = colorama.Fore.YELLOW
cgreen = colorama.Fore.GREEN
cred = colorama.Fore.RED
VIDEO_EXTENSIONS = (".mp4", ".avi", ".mkv", ".mov", ".wmv", ".flv", ".divx")
def get_ffmpeg_codecs(codec_type="S"):
"""
Retrieve a list of supported codecs for a specific type from ffmpeg.
This function runs the `ffmpeg -codecs` command and parses its output to extract
the list of supported codecs based on the provided `codec_type` (e.g., 'S' for subtitles,
'V' for video, 'A' for audio). The list of codecs is returned as a set for efficient lookup.
Args:
codec_type (str): The type of codecs to retrieve.
- "S" for subtitles (default)
- "V" for video
- "A" for audio
- "D" for data
- "T" for attachment
Returns:
set: A set of codec names of the specified type.
If an error occurs or no codecs are found, an empty set is returned.
"""
try:
# Run ffmpeg -codecs
result = subprocess.run(
["ffmpeg", "-codecs"], capture_output=True, text=True, check=True
)
output = result.stdout
# Extract codecs using regex
codec_pattern = re.compile(rf"^[ D.E]+{codec_type}[ A-Z.]+ (\S+)", re.MULTILINE)
codecs = codec_pattern.findall(output)
return set(codecs) # Return as a set for easy lookup
except subprocess.CalledProcessError as e:
print("Error running ffmpeg:", e)
return set()
def user_confirm(message, color="yellow"):
"""
Prompt the user for a yes/no confirmation.
Args:
message (str): The confirmation message to display.
color (str): Color for the prompt ('yellow', 'red', 'green'). Default is 'yellow'.
Returns:
bool: True if the user confirms with 'Y', False otherwise.
"""
color_map = {
"yellow": cyellow,
"red": cred,
"green": cgreen,
}
c = color_map.get(color, cyellow)
response = input(f"{c}{message}{creset} ").strip().upper()
return response == "Y"
def ensure_output_path(output_path):
"""
Ensure that the output directory exists. If it doesn't, attempt to create it.
Args:
output_path (str): The path of the output directory to be ensured.
Returns:
None
"""
try:
os.makedirs(output_path, exist_ok=True)
# Attempts to create the output directory if it doesn't exist.
# exist_ok=True ensures that an OSError is not raised if the directory already exists.
except OSError as e:
# If an OSError occurs during directory creation, print an error message and exit the program.
print(f"{cred}Failed to create output directory:{creset} {e}", file=sys.stderr)
sys.exit(1)
def has_supported_subs(video_file, supported_subtitle_codecs, debug=False):
"""
Check if the given video file contains subtitles in a supported codec.
This function uses `ffprobe` to extract subtitle stream information from the
video file and checks whether any subtitle stream has a codec that matches
one of the supported codecs provided in the `supported_subtitle_codecs` list.
Args:
video_file (str): Path to the video file to be checked.
supported_subtitle_codecs (set): Set of subtitle codec names that are considered supported.
debug (bool): If True, prints debug information for errors. Defaults to False.
Returns:
bool: True if at least one subtitle stream is found with a supported codec,
False otherwise or in case of an error.
"""
try:
result = subprocess.run(
[
"ffprobe",
"-v",
"error",
"-select_streams",
"s",
"-show_entries",
"stream=index,codec_name",
"-of",
"json",
video_file,
],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
)
if result.returncode != 0:
if debug:
print(f"ffprobe error: {result.stderr.strip()}")
return False # Assume no supported subtitles on failure
try:
streams = json.loads(result.stdout).get("streams", [])
except json.JSONDecodeError as e:
if debug:
print(f"JSON parsing error: {e}")
return False # Assume no supported subtitles if JSON is malformed
return any(
stream.get("codec_name") in supported_subtitle_codecs for stream in streams
)
except Exception as e:
if debug:
print(f"Unexpected error: {e}")
return False # Assume no supported subtitles
def find_videos_to_convert(input_path, max_height=720, debug=False):
"""
Find video files in the specified directory tree that meet conversion criteria.
Args:
input_path (str): The path of the directory to search for video files.
max_height (int, optional): The maximum height (in pixels) of the video to consider for conversion. Default is 720.
debug (bool, optional): If True, print debug messages. Default is False.
Returns:
list: A list of paths to video files that meet the conversion criteria.
"""
# Print a message indicating the start of the scanning process
print(f"{cgreen}Scanning {input_path} for video files to convert.{creset}\n")
# Initialize lists to store all video files and valid video files
video_files = []
valid_videos_files = []
# Find all video files in the directory tree
for root, dirs, files in os.walk(input_path):
for file in files:
fullpath = os.path.join(root, file)
# Check if the file is a video file with one of the specified extensions
if file.lower().endswith(VIDEO_EXTENSIONS) and os.path.isfile(fullpath):
video_files.append(fullpath)
# List of encodings to try for decoding output from ffprobe
ENCODINGS_TO_TRY = ["utf-8", "latin-1", "iso-8859-1"]
# Iterate through each video file
for video_file in video_files:
try:
# Run ffprobe command to get video resolution
resolution_output = subprocess.check_output(
[
"ffprobe",
"-v",
"error",
"-select_streams",
"v:0",
"-show_entries",
"stream=width,height",
"-of",
"csv=p=0",
video_file,
],
stderr=subprocess.STDOUT,
)
# Attempt to decode the output using different encodings
dimensions_str = None
width = height = None
for encoding in ENCODINGS_TO_TRY:
try:
dimensions_str = (
resolution_output.decode(encoding).strip().rstrip(",")
)
dimensions = [
int(d) for d in dimensions_str.split(",") if d.strip().isdigit()
]
if len(dimensions) >= 2:
width, height = dimensions[:2]
else:
raise ValueError(
f"Unexpected dimensions format: {dimensions_str}"
)
break # Stop trying encodings once successful decoding and conversion
except (UnicodeDecodeError, ValueError):
continue # Try the next encoding
# If decoding was unsuccessful with all encodings, skip this video
if width is None or height is None:
if debug:
print(
f"{cred}Error processing {video_file}: Unable to decode dimensions from ffprobe output.{creset}",
file=sys.stderr,
)
continue
# If decoding was unsuccessful with all encodings, skip this video
if dimensions_str is None:
if debug:
print(
f"{cred}Error processing {video_file}: Unable to decode output from ffprobe using any of the specified encodings.{creset}",
file=sys.stderr,
)
continue # Skip this video and continue with the next one
# Ignore vertical videos
if height > width:
continue # Skip this video and continue with the next one
# Check if the video height exceeds the maximum allowed height
if height > max_height:
valid_videos_files.append(video_file)
except subprocess.CalledProcessError as e:
if debug:
# Print error message if subprocess call fails
if e.output is not None:
decoded_error = e.output.decode("utf-8")
print(
f"{cred}Error processing {video_file}: {decoded_error}{creset}\n",
file=sys.stderr,
)
else:
print(
f"{cred}Error processing {video_file}: No output from subprocess{creset}\n",
file=sys.stderr,
)
except ValueError as e:
if debug:
# Print error message if value error occurs during decoding
print(
f"{cred}Error processing {video_file}: ValueError decoding the file{creset}\n",
file=sys.stderr,
)
valid_videos_files.sort()
return valid_videos_files
def convert_videos(
input_path,
output_path=None,
max_height=720,
delete=False,
rename=False,
debug=False,
):
"""
Convert videos taller than a specified height to 720p resolution with x265 encoding and MKV container.
Args:
input_path (str): The path of the directory containing input video files.
output_path (str, optional): The path of the directory where converted video files will be saved.
If None, output files are written alongside the originals (in-place).
max_height (int, optional): The maximum height (in pixels) of the video to consider for conversion. Default is 720.
delete (bool, optional): If True, delete the original file after successful conversion. Default is False.
rename (bool, optional): If True, rename files and intermediate folders to reflect the target resolution. Default is False.
debug (bool, optional): If True, print debug messages. Default is False.
Returns:
bool: True if conversion succeeds, False otherwise.
"""
# Get supported subtitle codecs
supported_subtitle_codecs = get_ffmpeg_codecs("S")
# Find video files to convert in the input directory
video_files = find_videos_to_convert(input_path, max_height, debug)
counter = 0
# If no video files found, print a message and return False
if len(video_files) == 0:
print(f"No video files to convert found.\n")
return False
# Ensure the fixed output directory exists (only when output_path is set)
if output_path:
ensure_output_path(output_path)
# Print a message indicating the start of the conversion process
print(
f"Converting {len(video_files)} videos taller than {max_height} pixels to {max_height}p resolution with x265 encoding and MKV container...\n"
)
# Variable to keep a track of the current_file in case of failure
current_file = None
# Track successfully converted original paths (used for folder rename pass)
converted_files = set()
# Iterate through each video file for conversion
for video_file in video_files:
counter += 1
# Print conversion progress information
print(
f"{cgreen}****** Starting conversion {counter} of {len(video_files)}: '{os.path.basename(video_file)}'...{creset}"
)
print(f"{ccyan}Original file:{creset}")
print(video_file)
# Determine output directory: fixed path, or same dir as the source file
effective_output_path = (
output_path if output_path else os.path.dirname(video_file)
)
# Compute base output filename, applying resolution rename if requested
base_name = os.path.splitext(os.path.basename(video_file))[0]
if rename:
renamed_base = apply_resolution_rename(base_name, max_height)
if renamed_base == base_name and f"{max_height}p" not in renamed_base:
# No resolution found in filename: append .{max_height}p before extension
renamed_base = f"{base_name}.{max_height}p"
base_name = renamed_base
# Generate output file path, using findfreename to avoid conflicts
output_file = findfreename(
os.path.join(effective_output_path, base_name + ".mkv")
)
try:
# Get original video width and height
video_info = subprocess.run(
[
"ffprobe",
"-v",
"error",
"-select_streams",
"v:0",
"-show_entries",
"stream=width,height",
"-of",
"csv=p=0",
video_file,
],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
)
width, height = map(int, video_info.stdout.strip().rstrip(",").split(","))
# Calculate the scaled width maintaining aspect ratio
scaled_width = int(max_height * (width / height))
# Ensure the width is even
if scaled_width % 2 != 0:
scaled_width += 1
# Keep a track of the current_file in case of failure
current_file = output_file
# Check if the video has supported subtitles
include_subs = has_supported_subs(
video_file, supported_subtitle_codecs, debug
)
# Construct the FFmpeg command dynamically
ffmpeg_command = [
"ffmpeg",
"-n",
"-i",
video_file,
"-map",
"0:v",
"-c:v",
"libx265",
"-vf",
f"scale={scaled_width}:{max_height}",
"-preset",
"medium",
"-crf",
"28",
"-map",
"0:a?",
"-c:a",
"copy",
]
# Only include subtitles if they are supported
if include_subs:
ffmpeg_command.extend(["-map", "0:s?", "-c:s", "copy"])
ffmpeg_command.append(output_file)
# Print the FFmpeg command as a single string
if debug:
print("FFmpeg command: " + " ".join(ffmpeg_command))
# Run the FFmpeg command
process = subprocess.run(
ffmpeg_command,
check=True,
stdout=subprocess.DEVNULL,
stderr=subprocess.STDOUT,
)
# Check if ffmpeg process returned successfully
if process.returncode != 0:
raise subprocess.CalledProcessError(
process.returncode, process.args, output=process.stderr
)
except subprocess.CalledProcessError as e:
# If we have a partial video converted, delete it due to the failure
if current_file:
deletefile(current_file)
# Handle subprocess errors
if debug:
if e.output is not None:
decoded_error = e.output.decode("utf-8")
print(
f"{cred}Error processing {video_file}: {decoded_error}{creset}\n",
file=sys.stderr,
)
else:
print(
f"{cred}Error processing {video_file}: No output from subprocess{creset}\n",
file=sys.stderr,
)
except KeyboardInterrupt:
print(f"{cyellow}Conversions cancelled, cleaning up...{creset}")
# If we have a partial video converted, delete it due to the failure
if current_file:
deletefile(current_file)
current_file = None
exit()
except Exception as e:
# If we have a partial video converted, delete it due to the failure
if current_file:
deletefile(current_file)
# Handle other exceptions
if debug:
print(
f"{cred}Error processing {video_file}: {str(e)}{creset}\n",
file=sys.stderr,
)
else:
# Print success message if conversion is successful
print(f"{ccyan}Successfully converted video to output file:{creset}")
print(f"{output_file}\n")
try:
os.chmod(output_file, 0o777)
except PermissionError:
print(f"{cred}PermissionError on: '{output_file}'{creset}")
# Delete the original file if requested
if delete:
deletefile(video_file)
# Track this original and its converted output as successfully converted
converted_files.add(os.path.realpath(video_file))
converted_files.add(os.path.realpath(output_file))
# Remove the now successfully converted filepath
current_file = None
# Folder rename second pass
if rename and converted_files:
# Collect all intermediate dirs touched by successful conversions, deepest first
dirs_to_consider = {}
for original_file in converted_files:
for d in get_intermediate_dirs(input_path, original_file):
dirs_to_consider[d] = True
# For each candidate dir, check that ALL video files under it were converted
dirs_to_rename = []
for d in sorted(dirs_to_consider, key=len, reverse=True):
all_converted = True
for root, _, files in os.walk(d):
for f in files:
if f.lower().endswith(VIDEO_EXTENSIONS):
fpath = os.path.realpath(os.path.join(root, f))
if fpath not in converted_files:
all_converted = False
break
if not all_converted:
break
if all_converted:
new_dirname = apply_resolution_rename(os.path.basename(d), max_height)
if new_dirname != os.path.basename(d):
new_dir = os.path.join(os.path.dirname(d), new_dirname)
dirs_to_rename.append((d, new_dir))
# Rename dirs deepest first to avoid breaking paths
for old_dir, new_dir in dirs_to_rename:
if os.path.exists(new_dir):
print(
f"{cred}Error: Target directory {new_dir} already exists, skipping rename.{creset}"
)
continue
os.rename(old_dir, new_dir)
if debug:
print(f"Renamed directory: {old_dir} -> {new_dir}")
return True
def parse_args():
"""
Parse command line arguments.
Returns:
argparse.Namespace: Parsed arguments.
"""
parser = argparse.ArgumentParser(
description="Convert videos taller than a certain height to 720p resolution with x265 encoding and MKV container."
)
parser.add_argument(
"input_path",
nargs="?",
default=os.getcwd(),
help="directory path to search for video files (default: current directory)",
)
parser.add_argument(
"-o",
"--output-path",
default=None,
help="directory path to save converted videos (default: ./converted, or alongside originals with --delete)",
)
parser.add_argument(
"-mh",
"--max-height",
type=int,
default=720,
help="maximum height of videos to be converted (default: 720)",
)
parser.add_argument(
"-del",
"--delete",
action="store_true",
help="delete original files after successful conversion",
)
parser.add_argument(
"-y",
"--yes",
action="store_true",
help="skip confirmation prompts",
)
parser.add_argument(
"-r",
"--rename",
action="store_true",
help="rename converted files and intermediate folders to reflect the target resolution",
)
parser.add_argument(
"--debug",
action="store_true",
help="enable debug mode for printing additional error messages",
)
# Enable autocomplete for argparse
argcomplete.autocomplete(parser)
args = parser.parse_args()
if args.output_path is None and not args.delete:
args.output_path = os.path.join(os.getcwd(), "converted")
# Warn and confirm if --delete is active
if args.delete:
print(f"{cyellow}WARNING: Delete option selected!{creset}")
if not args.yes and not user_confirm(
"Are you sure you wish to delete original files after successful conversion? [Y/N] ?",
color="yellow",
):
print(f"{cgreen}Exiting!{creset}")
exit()
return args
def main():
"""
Main function to parse command line arguments and initiate video conversion.
"""
args = parse_args()
# Convert videos
convert_videos(
args.input_path,
args.output_path,
args.max_height,
args.delete,
args.rename,
args.debug,
)
if __name__ == "__main__":
# Execute main function when the script is run directly
main()

120
scripts/video_autoreduce_rename.py
View File

@ -1,120 +0,0 @@
#!/usr/bin/env python3
# video_autoreduce_rename.py
import argparse
import argcomplete
import colorama
import os
import sys
from pathlib import Path
# Allow importing from scripts/library even when run directly
project_root = str(Path(__file__).resolve().parent.parent)
if project_root not in sys.path:
sys.path.append(project_root)
# === Local import ===
from scripts.library import apply_resolution_rename
colorama.init()
creset = colorama.Fore.RESET
ccyan = colorama.Fore.CYAN
cyellow = colorama.Fore.YELLOW
cgreen = colorama.Fore.GREEN
cred = colorama.Fore.RED
def autorename(input_path, max_height=720, debug=False):
"""
Rename video files and folders in the specified directory tree that contain a resolution
in their name to the specified max_height resolution.
Args:
input_path (str): The path of the directory to search for video files and folders.
max_height (int, optional): The maximum height (in pixels) of the video to consider for conversion. Default is 720.
debug (bool, optional): If True, print debug messages. Default is False.
"""
files_to_rename = []
dirs_to_rename = []
for dirpath, dirnames, filenames in os.walk(input_path, topdown=True):
# Collect files to rename
for filename in filenames:
new_filename = apply_resolution_rename(filename, max_height)
if new_filename != filename:
old_file = os.path.join(dirpath, filename)
new_file = os.path.join(dirpath, new_filename)
files_to_rename.append((old_file, new_file))
# Collect directories to rename
for dirname in dirnames:
new_dirname = apply_resolution_rename(dirname, max_height)
if new_dirname != dirname:
old_dir = os.path.join(dirpath, dirname)
new_dir = os.path.join(dirpath, new_dirname)
dirs_to_rename.append((old_dir, new_dir))
# Rename files first
for old_file, new_file in files_to_rename:
if os.path.exists(new_file):
print(f"Error: Target filename {new_file} already exists.")
continue
os.rename(old_file, new_file)
if debug:
print(f"Renamed file: {old_file} -> {new_file}")
# Rename directories after
for old_dir, new_dir in sorted(dirs_to_rename, key=lambda x: -len(x[0])):
if os.path.exists(new_dir):
print(f"Error: Target directory name {new_dir} already exists.")
continue
os.rename(old_dir, new_dir)
if debug:
print(f"Renamed directory: {old_dir} -> {new_dir}")
def main():
"""
Main function to parse command line arguments and initiate file renamings.
"""
# Create argument parser
parser = argparse.ArgumentParser(
description="Rename video files containing resolutions in their filenames to a specified max_height resolution."
)
# Define command line arguments
parser.add_argument(
"input_path",
nargs="?",
default=os.getcwd(),
help="directory path to search for video files (default: current directory)",
)
parser.add_argument(
"-mh",
"--max-height",
type=int,
default=720,
help="maximum height of videos to be converted (default: 720)",
)
parser.add_argument(
"--debug",
action="store_true",
help="enable debug mode for printing additional messages",
)
# Enable autocomplete for argparse
argcomplete.autocomplete(parser)
# Parse command line arguments
args = parser.parse_args()
# Rename videos
autorename(args.input_path, args.max_height, args.debug)
if __name__ == "__main__":
# Execute main function when the script is run directly
main()

266
scripts/video_manage_audio.py
View File

@ -1,266 +0,0 @@
#!/usr/bin/env python3
# video_manage_audio.py
import argparse
import argcomplete
import colorama
import json
import os
import subprocess
import sys
from pathlib import Path
# Allow importing from scripts/library even when run directly
project_root = str(Path(__file__).resolve().parent.parent)
if project_root not in sys.path:
sys.path.append(project_root)
# === Local import ===
from scripts.library import deletefile
# Initialize colorama for colored output in the terminal
colorama.init()
creset = colorama.Fore.RESET
ccyan = colorama.Fore.CYAN
cyellow = colorama.Fore.YELLOW
cgreen = colorama.Fore.GREEN
cred = colorama.Fore.RED
def get_audio_metadata(file_path):
"""
Uses ffprobe to extract audio stream information.
Returns a list of dicts with track details.
"""
abs_file_path = os.path.abspath(file_path)
ffprobe_command = [
"ffprobe",
"-v",
"error",
"-select_streams",
"a",
"-show_entries",
"stream=index,codec_name:stream_tags=language,title",
"-of",
"json",
abs_file_path,
]
try:
result = subprocess.run(
ffprobe_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True
)
if result.returncode != 0:
return None
metadata = json.loads(result.stdout)
return metadata.get("streams", [])
except Exception:
return None
def list_audio_tracks(file_path):
"""Displays all audio tracks in a video file."""
print(f"{ccyan}Analyzing tracks for: {file_path}{creset}")
streams = get_audio_metadata(file_path)
if streams is None:
print(f"{cred} Failed to read file metadata.{creset}\n")
return
if not streams:
print(f"{cyellow} No audio tracks found in this file.{creset}\n")
return
for audio_idx, stream in enumerate(streams):
abs_idx = stream.get("index")
codec = stream.get("codec_name", "unknown")
tags = stream.get("tags", {})
lang = tags.get("language", "und")
title = tags.get("title", "No Title")
print(
f" {cgreen}[Audio Track {audio_idx}]{creset} "
f"Absolute Stream Index: {abs_idx} | "
f"Codec: {codec} | Lang: {lang} | Title: {title}"
)
print()
def resolve_track_target(streams, target):
"""
Resolves a track target (either an integer index string like '1'
or a language code string like 'eng') to a relative audio track index.
Returns None if no match is found.
"""
# Case 1: Target is an explicit number (e.g., "0", "1")
if target.isdigit():
idx = int(target)
if 0 <= idx < len(streams):
return idx
return None
# Case 2: Target is a language code (e.g., "eng", "rus")
for audio_idx, stream in enumerate(streams):
lang = stream.get("tags", {}).get("language", "").lower()
if lang == target.lower():
return audio_idx
return None
def process_audio(file_path, track_target, command):
"""Modify or list audio tracks of a video file."""
if command == "list":
list_audio_tracks(file_path)
return
streams = get_audio_metadata(file_path)
if not streams:
print(f"{cred}Skipping {file_path}: Could not parse audio streams.{creset}\n")
return
# Dynamic target resolution (converts 'eng' or '1' to the proper track index)
resolved_track = resolve_track_target(streams, track_target)
if resolved_track is None:
print(
f"{cyellow}Skipping {file_path}: Target audio track/language '{track_target}' not found.{creset}\n"
)
return
print(f"{cgreen}Processing file: {file_path}{creset}")
print(
f"-> Target resolved to Audio Track {resolved_track} based on '{track_target}'"
)
output_file = f"{os.path.splitext(file_path)[0]}_{command}_audio{os.path.splitext(file_path)[1]}"
if command == "remove":
# Remove only the specified audio track while keeping all other streams
ffmpeg_command = [
"ffmpeg",
"-i",
file_path,
"-map",
"0",
"-map",
f"-0:a:{resolved_track}",
"-c",
"copy",
output_file,
]
elif command == "keep":
# Keep only the specified audio track while preserving video, subtitles, and metadata
ffmpeg_command = [
"ffmpeg",
"-i",
file_path,
"-map",
"0:v",
"-map",
f"0:a:{resolved_track}",
"-map",
"0:s?",
"-map",
"0:t?",
"-c",
"copy",
output_file,
]
else:
return
try:
# Keep a track of the current_file in case of failure
current_file = output_file
result = subprocess.run(
ffmpeg_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True
)
if result.returncode == 0:
print(
f"{ccyan}Audio processing complete. Output saved to {output_file}{creset}\n"
)
else:
print(
f"{cred}Command failed with return code {result.returncode}{creset}\n"
)
print(f"{cred}Error output:\n{result.stderr}{creset}\n")
except subprocess.CalledProcessError as e:
# If we have a partial video converted, delete it due to the failure
if current_file:
deletefile(current_file)
except KeyboardInterrupt:
print(f"{cyellow}Conversions cancelled, cleaning up...{creset}")
# If we have a partial video converted, delete it due to the failure
if current_file:
deletefile(current_file)
current_file = None
exit()
except Exception as e:
print(f"{cred}Error processing audio: {e}{creset}\n")
def process_directory(dir_path, track_target, command):
"""Recursively processes all video files in the specified directory."""
for root, _, files in os.walk(dir_path):
for file in files:
if file.endswith((".mp4", ".mkv", ".avi", ".mov")):
file_path = os.path.join(root, file)
process_audio(file_path, track_target, command)
def main():
"""
Main function to parse command-line arguments and initiate the audio processing.
"""
parser = argparse.ArgumentParser(
description="Manage or list audio tracks in video files dynamically."
)
parser.add_argument(
"command", choices=["remove", "keep", "list"], help="Command to run"
)
parser.add_argument(
"-t",
"--track",
type=str,
default="0",
help="Audio track index (e.g., 0, 1) OR language ISO code (e.g., eng, rus). Default is '0'.",
)
parser.add_argument("-f", "--file", type=str, help="Path to a specific video file.")
parser.add_argument(
"-d",
"--dir",
type=str,
default=os.getcwd(),
help="Directory to process (default is current directory).",
)
argcomplete.autocomplete(parser)
args = parser.parse_args()
# Process a single file if provided, otherwise process a directory
if args.file:
if os.path.isfile(args.file):
process_audio(args.file, args.track, args.command)
else:
print(f"{cred}File {args.file} does not exist.{creset}")
else:
if os.path.isdir(args.dir):
process_directory(args.dir, args.track, args.command)
else:
print(f"{cred}Directory {args.dir} does not exist.{creset}")
if __name__ == "__main__":
main()

169
scripts/video_manage_subtitles.py
View File

@ -1,169 +0,0 @@
#!/usr/bin/env python3
# video_manage_subtitles.py
import argparse
import argcomplete
import colorama
import os
import subprocess
# Initialize colorama for colored output in the terminal
colorama.init()
creset = colorama.Fore.RESET
ccyan = colorama.Fore.CYAN
cyellow = colorama.Fore.YELLOW
cgreen = colorama.Fore.GREEN
cred = colorama.Fore.RED
def process_subtitles(file_path, track, command):
"""
Modify subtitles tracks of a video file based on the given command.
- 'remove': Remove the specified subtitle track while keeping everything else.
- 'keep': Keep only the specified subtitle track and remove all others.
- 'none': Remove all subtitle tracks.
The function preserves video, audio, and metadata.
"""
print(f"{cgreen}Processing file: {file_path}{creset}")
output_file = f"{os.path.splitext(file_path)[0]}_{command}_subtitles{os.path.splitext(file_path)[1]}"
# Construct ffmpeg command based on user choice
if command == "remove":
# Remove only the specified subtitle track while keeping all other streams
ffmpeg_command = [
"ffmpeg",
"-i",
file_path,
"-map",
"0",
"-map",
f"-0:s:{track}",
"-c",
"copy",
output_file,
]
elif command == "keep":
# Keep only the specified subtitle track while preserving video, audio, and metadata
ffmpeg_command = [
"ffmpeg",
"-i",
file_path,
"-map",
"0:v",
"-map",
"0:a",
"-map",
f"0:s:{track}",
"-map",
"0:t?",
"-c",
"copy",
output_file,
]
elif command == "none":
# Remove all subtitle tracks
ffmpeg_command = [
"ffmpeg",
"-i",
file_path,
"-map",
"0",
"-map",
"-0:s",
"-c",
"copy",
output_file,
]
else:
print(f"{cred}Invalid command: {command}{creset}")
return
try:
# Execute the ffmpeg command and capture output
result = subprocess.run(
ffmpeg_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True
)
if result.returncode == 0:
print(
f"{ccyan}Subtitles processing complete. Output saved to {output_file}{creset}\n"
)
else:
print(
f"{cred}Command failed with return code {result.returncode}{creset}\n"
)
print(f"{cred}Error output:\n{result.stderr}{creset}\n")
except subprocess.CalledProcessError as e:
print(f"{cred}Error processing subtitles: {e}{creset}\n")
def process_directory(dir_path, track, command):
"""
Recursively processes all video files in the specified directory.
Applies the chosen subtitle modification (remove, keep, none) to each file.
"""
for root, _, files in os.walk(dir_path):
for file in files:
if file.endswith((".mp4", ".mkv", ".avi", ".mov")):
file_path = os.path.join(root, file)
process_subtitles(file_path, track, command)
def main():
"""
Main function to parse command-line arguments and initiate the subtitle processing.
"""
# Create argument parser
parser = argparse.ArgumentParser(
description="Manage subtitle tracks in video files."
)
# Define command line arguments
# Add a positional argument for the command
parser.add_argument(
"command",
choices=["remove", "keep", "none"],
help="Command to run (remove, keep, or none)",
)
# Add other arguments with both short and long options, including defaults
parser.add_argument(
"-t",
"--track",
type=int,
default=0,
help="Subtitle track index (default is 0). Use 'none' to remove all subtitles.",
)
parser.add_argument("-f", "--file", type=str, help="Path to a specific video file.")
parser.add_argument(
"-d",
"--dir",
type=str,
default=os.getcwd(),
help="Directory to process (default is current directory).",
)
# Enable autocomplete for command-line arguments
argcomplete.autocomplete(parser)
# Parse command line arguments
args = parser.parse_args()
# Process a single file if provided, otherwise process a directory
if args.file:
if os.path.isfile(args.file):
process_subtitles(args.file, args.track, args.command)
else:
print(f"{cred}File {args.file} does not exist.{creset}")
else:
if os.path.isdir(args.dir):
process_directory(args.dir, args.track, args.command)
else:
print(f"{cred}Directory {args.dir} does not exist.{creset}")
if __name__ == "__main__":
main()

95
scripts/video_remove_audio.py Executable file
View File

@ -0,0 +1,95 @@
#!/usr/bin/env python3
import os
import argparse
import argcomplete
import colorama
import subprocess
colorama.init()
creset = colorama.Fore.RESET
ccyan = colorama.Fore.CYAN
cyellow = colorama.Fore.YELLOW
cgreen = colorama.Fore.GREEN
cred = colorama.Fore.RED
# Function to remove audio track from a video using ffmpeg
def remove_audio_track(file_path, track):
print(f"{cgreen}Processing file: {file_path}{creset}")
output_file = f"{os.path.splitext(file_path)[0]}_no_audio{os.path.splitext(file_path)[1]}"
# ffmpeg command to remove the specified audio track and keep other streams
command = [
"ffmpeg", "-i", file_path, "-map", "0", "-map", f"-0:a:{track}", "-c",
"copy", output_file
]
try:
result = subprocess.run(command,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True)
if result.returncode == 0:
print(
f"{ccyan}Audio track {track} removed. Output saved to {output_file}{creset}\n"
)
else:
print(
f"{cred}Command failed with return code {result.returncode}{creset}\n"
)
print(f"{cred}Error output:", result.stderr, f"{creset}\n")
except subprocess.CalledProcessError as e:
print(f"{cred}Error removing audio track: {e}{creset}\n")
# Function to recursively process videos in a directory
def process_directory(dir_path, track):
for root, _, files in os.walk(dir_path):
for file in files:
if file.endswith((".mp4", ".mkv", ".avi",
".mov")): # Add more formats as needed
file_path = os.path.join(root, file)
remove_audio_track(file_path, track)
def main():
# Set up argument parser
parser = argparse.ArgumentParser(
description="Remove audio track from video files.")
parser.add_argument("--track",
type=int,
default=0,
help="Audio track index to remove (default is 0).")
parser.add_argument("--file",
type=str,
help="Path to a specific video file.")
parser.add_argument(
"--dir",
type=str,
default=os.getcwd(),
help="Directory to process (default is current directory).")
# Enable autocomplete for argparse
argcomplete.autocomplete(parser)
args = parser.parse_args()
# Process single file if provided
if args.file:
if os.path.isfile(args.file):
remove_audio_track(args.file, args.track)
else:
print(f"File {args.file} does not exist.")
# Otherwise, process all files in the specified directory
else:
if os.path.isdir(args.dir):
process_directory(args.dir, args.track)
else:
print(f"Directory {args.dir} does not exist.")
if __name__ == "__main__":
main()

139
scripts/video_repackage_mkv.py
View File

@ -1,139 +0,0 @@
#!/usr/bin/env python3
"""A utility script to repackage video files into MKV containers using FFmpeg.
This script recursively scans a specified directory for non-MKV video files
and uses FFmpeg to repackage them into '.mkv' containers. It explicitly maps
all streams (video, audio, subtitles, data) and preserves all metadata without
re-encoding. It includes options to automatically delete the source files upon
successful conversion.
"""
import argparse
import subprocess
from pathlib import Path
from typing import Sequence
# Define the targeted non-MKV extensions (case-insensitive handling below)
TARGET_EXTENSIONS = {".mp4", ".m4v", ".flv", ".avi", ".mov", ".wmv"}
def parse_args(args: Sequence[str] | None = None) -> argparse.Namespace:
"""Parse command-line arguments for the repackaging utility.
Args:
args: Optional sequence of strings to parse. If None, sys.argv is used.
Returns:
An argparse.Namespace object containing the parsed arguments.
"""
parser = argparse.ArgumentParser(
description="Recursively repackage non-MKV video files into MKV containers, preserving all streams and metadata."
)
parser.add_argument(
"-i",
"--input",
type=Path,
default=Path.cwd(),
help="Path to the root directory to scan for video files (default: current working directory).",
)
parser.add_argument(
"-d",
"--delete",
action="store_true",
default=False,
help="Delete the original video files upon successful conversion.",
)
# Enable argcomplete support if the module is available
try:
import argcomplete
argcomplete.autocomplete(parser)
except ImportError:
pass
return parser.parse_args(args)
def repackage_to_mkv(input_dir: Path, delete_source: bool) -> None:
"""Recursively scan the input directory and repackage non-MKV videos to MKV.
Args:
input_dir: The root Path directory to scan.
delete_source: If True, deletes the original video file after a
successful FFmpeg operation.
"""
if not input_dir.is_dir():
print(f"Error: The directory '{input_dir}' does not exist.")
return
print(f"Scanning '{input_dir}' recursively for video files...")
video_files = [
p
for p in input_dir.rglob("*")
if p.is_file() and p.suffix.lower() in TARGET_EXTENSIONS
]
if not video_files:
print("No matching video files found.")
return
print(f"Found {len(video_files)} files to process.\n")
for file_path in video_files:
output_path = file_path.with_suffix(".mkv")
if output_path.exists():
print(
f"Warning: Destination {output_path.name} already exists. Skipping to avoid accidental overwrite."
)
print("-" * 40)
continue
cmd = [
"ffmpeg",
"-i",
str(file_path),
"-map",
"0",
"-c",
"copy",
"-map_metadata",
"0",
str(output_path),
"-loglevel",
"error",
"-y",
]
# Display the relative path from the input directory so it's easier to track progress
relative_display = file_path.relative_to(input_dir)
print(f"Processing: {relative_display} -> {output_path.name}...")
try:
subprocess.run(cmd, check=True)
print(f"Successfully converted: {file_path.name}")
if delete_source:
file_path.unlink()
print(f"Deleted original file: {file_path.name}")
except subprocess.CalledProcessError:
print(f"Failed to convert: {file_path.name}")
except OSError as e:
print(f"System error processing {file_path.name}: {e}")
print("-" * 40)
def main() -> None:
"""Main execution entry point."""
args = parse_args()
repackage_to_mkv(input_dir=args.input, delete_source=args.delete)
if __name__ == "__main__":
main()

520
setups-drafts/almalinux_setup_wordpress.md Normal file
View File

@ -0,0 +1,520 @@
# Installing Wordpress on AlmaLinux
## Table of Contents
- [Installing Wordpress on AlmaLinux](#installing-wordpress-on-almalinux)
- [Table of Contents](#table-of-contents)
- [Disclaimer: Incomplete Guide](#disclaimer-incomplete-guide)
- [Introduction](#introduction)
- [Why WordPress?](#why-wordpress)
- [Prerequisites](#prerequisites)
- [What This Guide Covers](#what-this-guide-covers)
- [Placeholders](#placeholders)
- [Important Warnings and Security Practices](#important-warnings-and-security-practices)
- [Useful Commands and Information](#useful-commands-and-information)
- [Documentation](#documentation)
- [Links](#links)
- [Software on the Machine](#software-on-the-machine)
- [Paths](#paths)
- [Proxmox Commands](#proxmox-commands)
- [SSH Connection](#ssh-connection)
- [Installation Procedure](#installation-procedure)
## Disclaimer: Incomplete Guide
This document is a draft and may contain incomplete, untested, or outdated information. It is a work in progress and has not been verified for accuracy or usability. Use this guide at your own discretion, and consider it as a reference for further development or exploration. Updates may follow in the future, but no guarantees are made.
## Introduction
Welcome to the installation guide for WordPress on AlmaLinux in a Proxmox LXC container. WordPress is a powerful content management system (CMS) that allows you to create and manage websites efficiently.
## Why WordPress?
WordPress is one of the most popular content management systems (CMS) in the world, powering over 40% of websites on the internet. It offers a flexible and user-friendly platform for building anything from simple blogs to complex e-commerce sites.
**Key Benefits:**
- **Open-Source & Free** – No licensing fees, with a large community contributing to its continuous development.
- **Extensive Plugin Ecosystem** – Thousands of plugins to add features like SEO, security, performance optimization, and more.
- **Customizable Themes** – A wide variety of free and premium themes allow you to tailor your website's design.
- **SEO-Friendly** – Built-in SEO features and plugins like Yoast SEO help improve search engine rankings.
- **Scalability** – Suitable for small personal blogs to large enterprise websites with high traffic.
- **Active Community & Support** – Large developer and user communities provide extensive documentation, forums, and professional support options.
Whether you're launching a blog, a portfolio, or a business website, WordPress provides the flexibility and power to meet your needs.
## Prerequisites
Before you begin the installation process, ensure that your AlmaLinux system meets the following requirements:
- AlmaLinux GNU/Linux 9 or later
- Access to a terminal with sudo privileges
- Basic familiarity with the command line interface
- Stable internet connection to download necessary packages
## What This Guide Covers
This guide covers the installation and configuration of Wordpress on an AlmaLinux server, along with additional setup tasks such as SSH connection management and Proxmox commands. It covers:
1. **Installation**: Installing Wordpress from the official site.
2. **Configuration**: Configuring Wordpress to suit your environment and preferences.
3. **Setup**: Setting up Wordpress as a service and accessing it.
## Placeholders
Replace the placeholders below with the appropriate values for your setup:
- **User Details**
- Username: `<username>` (e.g., admin)
- Username - Hypervisor: `<username-hypervisor>` (e.g., admin)
- **Server Configuration**
- Server IP address: `<server-ip-address>` (e.g., 192.168.1.100)
- Hostname - Intranet: `<hostname-intranet>` (e.g., wordpress-server.domain.com)
- Hostname - Internet: `<hostname-internet>` (e.g., wordpress.domain.com)
- Hostname - Hypervisor: `<hostname-hypervisor>` (e.g., proxmox-hypervisor.domain.com)
- Hostname - Hypervisor NAS: `<hostname-hypervisor-nas>` (e.g., nas-server.domain.com)
- Name - Hypervisor NAS: `<name-hypervisor-nas>` (e.g., nas-server)
- Container ID: `<container-id>` (e.g., 100)
- **SSH Keys**
- SSH key - Proxmox: `<ssh-key-proxmox>` (e.g., /home/user/.ssh/id_rsa.pub)
- SSH key - Client: `<ssh-key-client>` (e.g., /home/user/.ssh/client_id_rsa.pub)
- **Networking**
- Wireguard port: `<wireguard-port>` (e.g., 51820)
- **Database**
- Database password : `<database-password>` (e.g., 15GbGnOn3Vjy9RQ4G9TfUF95wPcoKAy5)
- **Paths**
- Path index: `<path-index>` (e.g., /var/www/html)
## Important Warnings and Security Practices
Before executing any commands in this documentation, please adhere to the following guidelines to ensure the security and integrity of the system:
1. **Execute Commands with Caution**: Always review and understand a command before executing it. Misuse of commands can lead to data loss or system instability.
2. **Backup Command Execution**: The backup command must be executed only by authorized users. Ensure that proper permissions are set to prevent unauthorized access to backup files.
3. **Regular Backups**: Maintain regular backups of all critical data. It is advisable to use automated backup solutions and verify backup integrity periodically.
4. **System Updates**: Regularly update the system and all installed packages to protect against vulnerabilities. Use the package manager responsibly to avoid potential conflicts.
5. **Monitor System Logs**: Continuously monitor system logs for any unusual activity. Use logging tools to help identify potential security breaches or system failures.
6. **User Permissions**: Ensure that user permissions are strictly managed. Limit access to sensitive commands and data to only those who need it to perform their job functions.
7. **Network Security**: Implement proper network security measures, such as firewalls and intrusion detection systems, to protect against external threats.
8. **Data Encryption**: Encrypt sensitive data at rest and in transit to prevent unauthorized access.
By following these practices, you will help maintain the security and stability of the system while minimizing the risk of data loss or compromise.
## Useful Commands and Information
### Documentation
- [AlmaLinux Wiki](https://wiki.almalinux.org/)
- [Documentation Overview](https://www.wordpress.info/doc/overview/)
- [Tutorials](https://wordpress.com/learn/)
### Links
- [Wordpress appliance](https://<hostname-internet>/)
### Software on the Machine
- **Operating System**: AlmaLinux
- **Web Server**: Apache
- **Database**: Mariadb
- **Security**: GnuPG, WireGuard, firewalld
- **Other**: Git, sudo
### Paths
- **Apache AlmaLinux Default Configuration**: `/etc/httpd/conf.d/welcome.conf`
- **Wordpress Configuration**: `/var/www/html/wp-config.php`
- **Wordpress Work Path**: `/var/www/html`
### Proxmox Commands
**List available Proxmox templates**
```bash
ssh <username>@<hostname-hypervisor-nas> "ls /mnt/proxmox/template/cache/"
```
**Create the container**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "pct create <container-id> <name-hypervisor-nas>:vztmpl/almalinux-9-default_20240911_amd64.tar.xz --hostname <hostname-intranet> --cores 2 --memory 4096 --swap 2048 --net0 name=net0,bridge=vmbr0,ip=dhcp,firewall=1 --rootfs <name-hypervisor-nas>:100 --unprivileged 1 --features nesting=1 --ssh-public-keys <ssh-key-proxmox>"
```
**Backup**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump <container-id> --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
```
**Set the state of the Proxmox HA Manager for Container <container-id>**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager add ct:<container-id>"
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager remove ct:<container-id>"
```
**Set the state and limits of the Proxmox Container <container-id> in the HA Manager**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id> --state started --max_relocate 3 --max_restart 3"
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id> --state stopped"
ssh <username-hypervisor>@<hostname-hypervisor> "pct start <container-id>"
ssh <username-hypervisor>@<hostname-hypervisor> "pct stop <container-id>"
ssh <username-hypervisor>@<hostname-hypervisor> "pct reboot <container-id>"
```
**Destroy the Proxmox Container <container-id> forcefully**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "pct destroy <container-id> --force --purge"
```
**Move the Proxmox Container <container-id> to another host**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "pct migrate <container-id> hv2"
```
### SSH Connection
**Connection with specific keys**
```bash
ssh -i <ssh-key-client> root@<hostname-intranet>
ssh -i <ssh-key-client> root@<server-ip-address>
ssh -i <ssh-key-client> <username>@<hostname-intranet>
ssh -i <ssh-key-client> <username>@<server-ip-address>
```
**Remove offending keys from known_hosts**
```bash
ssh-keygen -f "/home/<username>/.ssh/known_hosts" -R "<hostname-intranet>"
ssh-keygen -f "/home/<username>/.ssh/known_hosts" -R "<server-ip-address>"
```
**Copy SSH public key to remote host**
```bash
ssh-copy-id -i <ssh-key-client> root@<server-ip-address>
ssh-copy-id -i <ssh-key-client> root@<hostname-intranet>
ssh-copy-id -i <ssh-key-client> <username>@<server-ip-address>
ssh-copy-id -i <ssh-key-client> <username>@<hostname-intranet>
```
## Installation Procedure
1. **Fresh AlmaLinux Installation**
- Install a fresh AlmaLinux operating system on your new server following the standard installation procedure.
2. **Backup before starting**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump <container-id> --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
```
3. **Install Required Dependencies**
**Upgrade the base system**
```bash
dnf update
```
**Enable EPEL repository**
Extra Package for Enterprise Linux repository has packages like Apache and Nginx
```bash
dnf install -y epel-release
dnf makecache
```
4. **Install LAMP Stack**
**Install dependencies**
```bash
dnf install -y sudo nano firewalld firewall-config tar wget curl unzip
dnf install -y gnupg nginx git wireguard-tools
dnf install -y httpd httpd-tools mariadb-server mariadb
# Basic command
dnf install -y php php-mysqlnd php-fpm php-json php-mbstring php-xml php-curl php-zip php-gd php-intl php-bcmath php-soap php-exif
# More tooling and security
dnf install -y httpd mod_ssl php php-cli php-common php-fpm php-gd php-intl php-json php-mbstring php-mysqlnd php-opcache php-pdo php-pecl-imagick php-xml php-zip policycoreutils-python-utils
# Other xml php modules
dnf install -y php-dom php-simplexml php-xmlreader php-iconv php-posix php-sockets php-tokenizer
# Necessary for wp-cli
dnf install -y php-cli php-mbstring unzip curl
```
**WordPress Dependencies on AlmaLinux (MariaDB)**
**Web Server**
- `httpd` (Apache)
- `mod_ssl` (For HTTPS support)
- OR `nginx` (If using Nginx instead of Apache)
**PHP**
- `php` (Main PHP package)
- `php-cli` (Command-line interface for PHP)
- `php-common` (Common PHP files)
- `php-fpm` (FastCGI Process Manager for PHP, required for Nginx)
- `php-gd` (Image processing)
- `php-intl` (Internationalization)
- `php-json` (JSON support)
- `php-mbstring` (Multibyte string functions)
- `php-mysqlnd` (MySQL/MariaDB support)
- `php-opcache` (Performance optimization)
- `php-pdo` (PHP Data Objects)
- `php-pecl-imagick` (ImageMagick extension, recommended for media handling)
- `php-xml` (XML parsing)
- `php-zip` (ZIP file support)
**Database (MariaDB)**
- `mariadb-server` (MariaDB database server)
- `mariadb` (MariaDB client)
**Additional System Packages**
- `tar` (Required for extracting WordPress archives)
- `wget` (To fetch files from the web)
- `curl` (For network requests)
- `unzip` (Extracting ZIP files)
- `policycoreutils-python-utils` (SELinux tools, required if SELinux is enabled)
- `firewalld` (For firewall management, if needed)
**Optional Debugging & Performance Tools**
- `php-pecl-apcu` (APC User Cache for PHP)
- `php-pecl-memcached` (Memcached support)
- `php-pecl-redis` (Redis support)
5. **Ensure Hostname**
```bash
nano /etc/hosts
# Add line: 127.0.1.1 <hostname-intranet>
nano /etc/hostname
# Set to: <hostname-intranet>
hostnamectl set-hostname <hostname-intranet>
```
6. **Add Users and set Credentials**
```bash
passwd root
```
```bash
adduser <username>
passwd <username>
groupadd sudo
usermod -aG sudo <username>
nano /etc/sudoers
```
```
## Allows people in group sudo to run all commands
%sudo ALL=(ALL) ALL
```
7. **Setup SSH Connectors**
- Configure SSH connectors as per your setup script to establish secure connections to the server.
8. **Test users, SSH, and sudo**
1. **Transfer SSH keys for User**
2. **Connect as User with SSH key**
3. **Test sudo**
```bash
sudo su -
```
4. **Disconnect as root**
9. **Secure SSH**
```bash
nano /etc/ssh/sshd_config
```
```ini
PermitRootLogin no
PasswordAuthentication no
ChallengeResponseAuthentication no
```
**Restart SSH**
```bash
systemctl restart sshd
```
10. **Configure Firewall**
**Open ports**
```bash
firewall-cmd --permanent --zone=public --add-service=ssh
firewall-cmd --permanent --zone=public --add-service=http
firewall-cmd --permanent --zone=public --add-service=https
firewall-cmd --permanent --zone=public --add-port=<wireguard-port>/udp
```
**Reload firewall to apply changes**
```bash
firewall-cmd --reload
```
**Enable and start firewall**
```bash
systemctl enable firewalld
systemctl start firewalld
```
11. **Start Apache And MariaDB**
```bash
systemctl enable httpd --now
systemctl enable mariadb --now
```
Now, your web server is available at `http://<hostname-intranet>`.
12. **Create PHP test page**
```bash
echo "<?php phpinfo() ?>" > /var/www/html/info.php
```
Now, your web server php information is available at `http://<hostname-intranet>/info.php`.
```bash
rm /var/www/html/info.php
```
13. **Secure MariaDB Installation**
```bash
mysql_secure_installation
```
14. **Creating the new Database**
```bash
mariadb
```
```sql
CREATE DATABASE wordpress;
CREATE USER `admin`@`localhost` IDENTIFIED BY '<database-password>';
GRANT ALL ON wordpress.* TO `admin`@`localhost`;
FLUSH PRIVILEGES;
EXIT;
```
15. **Download and Extract WordPress**
```bash
curl https://wordpress.org/latest.tar.gz --output wordpress.tar.gz
tar xf wordpress.tar.gz
cp -r wordpress/* /var/www/html/
```
16. **Download and Install WordPress CLI**
As root
```bash
curl -O https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar
php wp-cli.phar --info
chmod +x wp-cli.phar
mv wp-cli.phar /usr/local/bin/wp
wp --info
```
As user **Enable Tab Completion**
```bash
su -s /bin/bash -l apache
mkdir -p ~/.wp-cli
wp cli completions --shell=bash > ~/.wp-cli/wp-completion.bash
echo 'source ~/.wp-cli/wp-completion.bash' >> ~/.bashrc
source ~/.bashrc
```
17. **Modify Permissions**
Set appropriate ownership and adjust the SELinux security context for WordPress files:
```bash
chown -R apache:apache /var/www/html
chmod -R 755 /var/www/html
```
Enable Apache's ability to establish network connections, allowing WordPress to download updates and plugins:
```bash
setsebool -P httpd_can_network_connect true
```
18. **Allow Override**
```bash
nano /etc/httpd/conf/httpd.conf
```
Allow Overrides on `/var/www` and `/var/www/html`:
```apache
AllowOverride All
```
19. **Configure Wordpress**
Now, visit `http://<hostname-intranet>` to follow the wordpress configuration.
20. **Configure WireGuard**
```bash
nano /etc/wireguard/proxy-lan.conf
systemctl enable wg-quick@proxy-lan --now
wg show
```
21. **Back-up post installation**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id> --state stopped"
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump <container-id> --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup post installation\""
```
22. **Start the server**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "pct start <container-id>"
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id> --state started --max_relocate 3 --max_restart 3"
```

481
setups-drafts/oracle_setup_oro.md Normal file
View File

@ -0,0 +1,481 @@
# Installing Oro on Oracle Linux
## Table of Contents
- [Installing Oro on Oracle Linux](#installing-oro-on-oracle-linux)
- [Table of Contents](#table-of-contents)
- [Disclaimer: Incomplete Guide](#disclaimer-incomplete-guide)
- [Introduction](#introduction)
- [Prerequisites](#prerequisites)
- [What This Guide Covers](#what-this-guide-covers)
- [Placeholders](#placeholders)
- [Important Warnings and Security Practices](#important-warnings-and-security-practices)
- [Useful Commands and Information](#useful-commands-and-information)
- [Documentation](#documentation)
- [Links](#links)
- [Software on the Machine](#software-on-the-machine)
- [Paths](#paths)
- [Proxmox Commands](#proxmox-commands)
- [SSH Connection](#ssh-connection)
- [Installation Procedure](#installation-procedure)
## Disclaimer: Incomplete Guide
This document is a draft and may contain incomplete, untested, or outdated information. It is a work in progress and has not been verified for accuracy or usability. Use this guide at your own discretion, and consider it as a reference for further development or exploration. Updates may follow in the future, but no guarantees are made.
## Introduction
This guide provides step-by-step instructions for installing and configuring Oro on a Oracle Linux server.
## Prerequisites
Before you begin the installation process, ensure that your Oracle Linux system meets the following requirements:
- Oracle Linux GNU/Linux 9 or later
- Access to a terminal with sudo privileges
- Basic familiarity with the command line interface
- Stable internet connection to download necessary packages
## What This Guide Covers
This guide covers the installation and configuration of Oro on a Oracle Linux server, along with additional setup tasks such as SSH connection management and Proxmox commands. It covers:
1. **Installation**: Installing Oro from the official repository.
2. **Configuration**: Configuring Oro to suit your environment and preferences.
3. **Setup**: Setting up Oro as a service and accessing it.
## Placeholders
Replace the placeholders below with the appropriate values for your setup:
- **User Details**
- Username: `<username>` (e.g., admin)
- Username - Hypervisor: `<username-hypervisor>` (e.g., admin)
- **Server Configuration**
- Server IP address: `<server-ip-address>` (e.g., 192.168.1.100)
- Hostname - Intranet: `<hostname-intranet>` (e.g., oro-server.domain.com)
- Hostname - Internet: `<hostname-internet>` (e.g., oro.domain.com)
- Hostname - Hypervisor: `<hostname-hypervisor>` (e.g., proxmox-hypervisor.domain.com)
- Hostname - Hypervisor NAS: `<hostname-hypervisor-nas>` (e.g., nas-server.domain.com)
- Name - Hypervisor NAS: `<name-hypervisor-nas>` (e.g., nas-server)
- **SSH Keys**
- SSH key - Proxmox: `<ssh-key-proxmox>` (e.g., /home/user/.ssh/id_rsa.pub)
- SSH key - Client: `<ssh-key-client>` (e.g., /home/user/.ssh/client_id_rsa.pub)
- **Networking**
- Wireguard port: `<wireguard-port>` (e.g., 51820)
- **Paths**
## Important Warnings and Security Practices
Before executing any commands in this documentation, please adhere to the following guidelines to ensure the security and integrity of the system:
1. **Execute Commands with Caution**: Always review and understand a command before executing it. Misuse of commands can lead to data loss or system instability.
2. **Backup Command Execution**: The backup command must be executed only by authorized users. Ensure that proper permissions are set to prevent unauthorized access to backup files.
3. **Regular Backups**: Maintain regular backups of all critical data. It is advisable to use automated backup solutions and verify backup integrity periodically.
4. **System Updates**: Regularly update the system and all installed packages to protect against vulnerabilities. Use the package manager responsibly to avoid potential conflicts.
5. **Monitor System Logs**: Continuously monitor system logs for any unusual activity. Use logging tools to help identify potential security breaches or system failures.
6. **User Permissions**: Ensure that user permissions are strictly managed. Limit access to sensitive commands and data to only those who need it to perform their job functions.
7. **Network Security**: Implement proper network security measures, such as firewalls and intrusion detection systems, to protect against external threats.
8. **Data Encryption**: Encrypt sensitive data at rest and in transit to prevent unauthorized access.
By following these practices, you will help maintain the security and stability of the system while minimizing the risk of data loss or compromise.
## Useful Commands and Information
### Documentation
- [Documentation Overview](https://doc.oroinc.com/)
- [Documentation community edition](https://doc.oroinc.com/backend/setup/dev-environment/community-edition/)
- [Installation](https://doc.oroinc.com/backend/setup/installation/)
### Links
- [Oro appliance](https://<hostname-internet>/)
### Software on the Machine
- **Operating System**: Oracle Linux
- **Web Server**:
- **Security**: GnuPG, WireGuard, firewalld
- **Other**: Git, sudo
### Paths
- **Oro Configuration**:
- **Oro Work Path**:
### Proxmox Commands
**List available Proxmox templates**
```bash
ssh <username>@<hostname-hypervisor-nas> "ls /mnt/proxmox/template/cache/"
```
**Create the container**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "pct create 100 <name-hypervisor-nas>:vztmpl/oracle-9-sshnano_20240603_amd64.tar.zst --hostname <hostname-intranet> --cores 2 --memory 4096 --swap 2048 --net0 name=net0,bridge=vmbr0,ip=dhcp,firewall=1 --rootfs <name-hypervisor-nas>:100 --unprivileged 1 --features nesting=1 --ssh-public-keys <ssh-key-proxmox>"
```
**Backup**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump 100 --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
```
**Set the state of the Proxmox HA Manager for Container 100**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager add ct:100"
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager remove ct:100"
```
**Set the state and limits of the Proxmox Container 100 in the HA Manager**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:100 --state started --max_relocate 3 --max_restart 3"
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:100 --state stopped"
ssh <username-hypervisor>@<hostname-hypervisor> "pct start 100"
ssh <username-hypervisor>@<hostname-hypervisor> "pct stop 100"
ssh <username-hypervisor>@<hostname-hypervisor> "pct reboot 100"
```
**Destroy the Proxmox Container 100 forcefully**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "pct destroy 100 --force --purge"
```
**Move the Proxmox Container 100 to another host**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "pct migrate 100 hv2"
```
### SSH Connection
**Connection with specific keys**
```bash
ssh -i <ssh-key-client> root@<hostname-intranet>
ssh -i <ssh-key-client> root@<server-ip-address>
ssh -i <ssh-key-client> <username>@<hostname-intranet>
ssh -i <ssh-key-client> <username>@<server-ip-address>
```
**Remove offending keys from known_hosts**
```bash
ssh-keygen -f "/home/<username>/.ssh/known_hosts" -R "<hostname-intranet>"
ssh-keygen -f "/home/<username>/.ssh/known_hosts" -R "<server-ip-address>"
```
**Copy SSH public key to remote host**
```bash
ssh-copy-id -i <ssh-key-client> root@<server-ip-address>
ssh-copy-id -i <ssh-key-client> root@<hostname-intranet>
ssh-copy-id -i <ssh-key-client> <username>@<server-ip-address>
ssh-copy-id -i <ssh-key-client> <username>@<hostname-intranet>
```
**Transfer SSH keys and files**
## Installation Procedure
1. **Fresh Oracle Linux Installation**
- Install a fresh Oracle Linux operating system on your new server following the standard installation procedure.
2. **Backup before starting**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump 100 --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
```
3. **Install Required Dependencies**
**Upgrade the base system**
```bash
dnf update
```
**Enable EPEL repository**
Extra Package for Enterprise Linux repository has packages like Apache and Nginx
```bash
dnf install epel-release
```
**Enable Postgres repository**
Get instructions and urls in their [documentation](https://www.postgresql.org/download/linux/redhat/)
```bash
# Install the repository RPM:
dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-x86_64/pgdg-redhat-repo-latest.noarch.rpm
# Disable the built-in PostgreSQL module:
dnf -qy module disable postgresql
# Install PostgreSQL:
dnf install -y postgresql15-server
# Optionally initialize the database and enable automatic start:
/usr/pgsql-15/bin/postgresql-15-setup initdb
systemctl enable postgresql-15
systemctl start postgresql-15
```
**Enable Remi repository**
```bash
cat >"/etc/yum.repos.d/oropublic.repo" <<__EOF__
[oropublic]
name=OroPublic
baseurl=https://nexus.oro.cloud/repository/oropublic/8/x86_64/
enabled=1
gpgcheck=0
module_hotfixes=1
__EOF__
```
```bash
dnf config-manager --set-enabled remi
```
**Enable oro repository**
```bash
dnf install https://rpms.remirepo.net/enterprise/remi-release-9.rpm
```
**Enable DNF streams**
```bash
dnf module list nginx
dnf module list nodejs
dnf module list php
dnf -y module enable nginx:1.24 nodejs:20 php:remi-8.3
dnf -y upgrade
```
**Install dependencies**
```bash
dnf install -y sudo nano findutils rsync psmisc wget glibc-langpack-en bzip2 unzip p7zip p7zip-plugins parallel patch nodejs npm git-core jq bc postgresql postgresql-server postgresql-contrib
dnf install -y gnupg wireguard-tools firewalld firewall-config
```
**Install Apache or Nginx**
Uncomment one of the following lines depending on the web server you prefer to use
**Apache**
```bash
dnf install -y httpd
```
**Nginx Configuration**
```bash
dnf install -y nginx
```
**PHP and modules**
```bash
dnf install -y php-common php-cli php-fpm php-opcache php-mbstring php-mysqlnd php-pgsql php-pdo php-json php-process php-ldap php-gd php-ctype php-curl php-fileinfo php-intl php-bcmath php-xml php-soap php-sodium php-openssl php-pcre php-simplexml php-tokenizer php-zip php-tidy php-imap php-pecl-zip php-pecl-mongodb
```
**nodejs**
```bash
dnf install -y nodejs
```
Verify Node.js and NPM versions
```bash
node -v
npm -v
```
**Supervisor for process control**
```bash
dnf install -y supervisor
```
Enable and start Supervisor service
```bash
systemctl enable supervisord
systemctl start supervisord
```
**Redis**
```bash
dnf install -y redis
```
Enable and start Redis service
```bash
systemctl enable redis
systemctl start redis
```
**pngquant and jpegoptim**
dnf install -y pngquant jpegoptim
4. **Ensure Hostname**
```bash
nano /etc/hosts
# Add line: 127.0.1.1 <hostname-intranet>
nano /etc/hostname
# Set to: <hostname-intranet>
hostnamectl set-hostname <hostname-intranet>
```
5. **Add Users and set Credentials**
```bash
passwd -f root
```
```bash
adduser <username>
passwd -f <username>
groupadd sudo
usermod -aG sudo <username>
nano /etc/sudoers
```
```
## Allows people in group sudo to run all commands
%sudo ALL=(ALL) ALL
```
6. **Setup SSH Connectors**
- Configure SSH connectors as per your setup script to establish secure connections to the server.
7. **Test users, SSH, and sudo**
1. **Transfer SSH keys for User**
2. **Connect as User with SSH key**
3. **Test sudo**
```bash
sudo su -
```
4. **Disconnect as root**
8. **Secure SSH**
```bash
nano /etc/ssh/sshd_config
```
```ini
PermitRootLogin no
PasswordAuthentication no
ChallengeResponseAuthentication no
```
**Restart SSH**
```bash
systemctl restart sshd
```
9. **Configure Firewall**
**Open ports**
```bash
firewall-cmd --permanent --add-service=ssh
firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-service=https
firewall-cmd --permanent --add-port=<wireguard-port>/udp
```
**Reload firewall to apply changes**
```bash
firewall-cmd --reload
```
**Enable and start firewall**
```bash
systemctl enable firewalld
systemctl start firewalld
```
10. **Configure PHP**
```bash
php --ini
nano /etc/php.ini
```
**Add or update the following settings**
````ini
date.timezone = America/Toronto
detect_unicode = Off
memory_limit = 1G
max_execution_time = <appropriate-time>
If xdebug is installed, update or add these settings
```ini
xdebug.scream = Off
xdebug.show_exception_trace = Off
xdebug.max_nesting_level = 100
````
11. **Install Oro**
```bash
dnf -y --setopt=install_weak_deps=False --best --nogpgcheck install oro-nginx oro-nginx-mod-http-cache_purge oro-nginx-mod-http-cookie_flag oro-nginx-mod-http-geoip oro-nginx-mod-http-gridfs oro-nginx-mod-http-headers_more oro-nginx-mod-http-naxsi oro-nginx-mod-http-njs oro-nginx-mod-http-pagespeed oro-nginx-mod-http-sorted_querystring oro-nginx-mod-http-testcookie_access oro-nginx-mod-http-xslt-filter
```
12. **Enable Oro service**
13. **Setup nginx proxy**
14. **Configure SSL**
15. **Correct permissions**
16. **Run the installer**
17. **Verify installation**
18. **Back-up post installation**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:100 --state stopped"
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump 100 --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup post installation\""
```
19. **Start the server**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:100 --state started --max_relocate 3 --max_restart 3"
```

35
setups/debian_setup_aptly.md
View File

@ -68,7 +68,6 @@ Replace the placeholders below with the appropriate values for your setup:
- Hostname - Hypervisor: `<hostname-hypervisor>` (e.g., proxmox-hypervisor.domain.com)
- Hostname - Hypervisor NAS: `<hostname-hypervisor-nas>` (e.g., nas-server.domain.com)
- Name - Hypervisor NAS: `<name-hypervisor-nas>` (e.g., nas-server)
- Container ID - Hypervisor: `<container-id-hypervisor>` (e.g., 100)
- **SSH Keys**
@ -141,40 +140,40 @@ ssh <username>@<hostname-hypervisor-nas> "ls /mnt/proxmox/template/cache/"
**Create the container**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "pct create <container-id-hypervisor> <name-hypervisor-nas>:vztmpl/debian-12-upgraded_12.5_amd64.tar.zst --hostname <hostname-intranet> --cores 2 --memory 2048 --swap 2048 --net0 name=net0,bridge=vmbr0,ip=dhcp,firewall=1 --rootfs <name-hypervisor-nas>:250 --unprivileged 1 --features nesting=1 --ssh-public-keys <ssh-key-proxmox> --start 1"
ssh <username-hypervisor>@<hostname-hypervisor> "pct create 100 <name-hypervisor-nas>:vztmpl/debian-12-upgraded_12.5_amd64.tar.zst --hostname <hostname-intranet> --cores 2 --memory 2048 --swap 2048 --net0 name=net0,bridge=vmbr0,ip=dhcp,firewall=1 --rootfs <name-hypervisor-nas>:250 --unprivileged 1 --features nesting=1 --ssh-public-keys <ssh-key-proxmox> --start 1"
```
**Backup**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump <container-id-hypervisor> --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump 100 --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
```
**Set the state of the Proxmox HA Manager for Container <container-id-hypervisor>**
**Set the state of the Proxmox HA Manager for Container 100**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager add ct:<container-id-hypervisor>"
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager remove ct:<container-id-hypervisor>"
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager add ct:100"
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager remove ct:100"
```
**Set the state and limits of the Proxmox Container <container-id-hypervisor> in the HA Manager**
**Set the state and limits of the Proxmox Container 100 in the HA Manager**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id-hypervisor> --state started --max_relocate 3 --max_restart 3"
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id-hypervisor> --state stopped"
ssh <username-hypervisor>@<hostname-hypervisor> "pct reboot <container-id-hypervisor>"
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:100 --state started --max_relocate 3 --max_restart 3"
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:100 --state stopped"
ssh <username-hypervisor>@<hostname-hypervisor> "pct reboot 100"
```
**Destroy the Proxmox Container <container-id-hypervisor> forcefully**
**Destroy the Proxmox Container 100 forcefully**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "pct destroy <container-id-hypervisor> --force --purge"
ssh <username-hypervisor>@<hostname-hypervisor> "pct destroy 100 --force --purge"
```
**Move the Proxmox Container <container-id-hypervisor> to another host**
**Move the Proxmox Container 100 to another host**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "pct migrate <container-id-hypervisor> hv2"
ssh <username-hypervisor>@<hostname-hypervisor> "pct migrate 100 hv2"
```
### SSH Connection
@ -221,7 +220,7 @@ chown -R aptly:aptly /home/aptly/.ssh/
2. **Backup before starting**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump <container-id-hypervisor> --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump 100 --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
```
3. **Install Required Dependencies**
@ -624,11 +623,11 @@ chown -R aptly:aptly /home/aptly/.ssh/
14. **Back-up post installation**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id-hypervisor> --state stopped"
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump <container-id-hypervisor> --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup post installation\""
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:100 --state stopped"
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump 100 --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup post installation\""
```
15. **Start the server**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id-hypervisor> --state started --max_relocate 3 --max_restart 3"
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:100 --state started --max_relocate 3 --max_restart 3"
```

626
setups/debian_setup_dolibarr.md
View File

@ -1,626 +0,0 @@
# Installing Dolibarr on Debian
## Table of Contents
- [Installing Dolibarr on Debian](#installing-dolibarr-on-debian)
- [Table of Contents](#table-of-contents)
- [Introduction](#introduction)
- [Why Dolibarr?](#why-dolibarr)
- [Prerequisites](#prerequisites)
- [What This Guide Covers](#what-this-guide-covers)
- [Placeholders](#placeholders)
- [Important Warnings and Security Practices](#important-warnings-and-security-practices)
- [Useful Commands and Information](#useful-commands-and-information)
- [Documentation](#documentation)
- [Links](#links)
- [Software on the Machine](#software-on-the-machine)
- [Paths](#paths)
- [Proxmox Commands](#proxmox-commands)
- [SSH Connection](#ssh-connection)
- [Installation Procedure](#installation-procedure)
## Introduction
Welcome to the installation guide for Dolibarr on Debian. Dolibarr is an open-source ERP and CRM platform designed for small and medium-sized businesses, offering features such as invoicing, accounting, inventory management, and customer relationship management through a clean, web-based interface. By following this guide, you’ll be able to install and configure Dolibarr on your Debian system efficiently.
## Why Dolibarr?
Dolibarr is a flexible and open-source ERP/CRM solution that allows you to keep full control over your business data and infrastructure. It covers a wide range of needs for small and medium-sized organizations, including invoicing, accounting, customer management, inventory, and project tracking, all from a single web interface. Being self-hosted, Dolibarr gives you data ownership, transparency, and the ability to customize or extend the platform to fit your workflows without relying on third-party SaaS providers.
## Prerequisites
Before you begin the installation process, ensure that your Debian system meets the following requirements:
- Debian GNU/Linux 9 (Stretch) or later
- Access to a terminal with sudo privileges
- Basic familiarity with the command line interface
- Stable internet connection to download necessary packages
## What This Guide Covers
This guide will walk you through the installation of Dolibarr on Debian step by step, covering:
1. **Installation**: Installing Dolibarr from the official Git repository.
2. **Configuration**: Adjusting Dolibarr and system settings to match your environment.
3. **Web Setup**: Configuring Dolibarr with Apache and accessing the web installer.
4. **Security**: Hardening Debian, PHP, and Dolibarr for a production setup.
## Placeholders
Replace the placeholders below with the appropriate values for your setup:
- **User Details**
- Username: `<username>` (e.g., admin)
- Username - Hypervisor: `<username-hypervisor>` (e.g., admin)
- **Server Configuration**
- Server IP address: `<server-ip-address>` (e.g., 192.168.1.100)
- Hostname - Intranet: `<hostname-intranet>` (e.g., dolibarr-server.domain.com)
- Hostname - Internet: `<hostname-internet>` (e.g., dolibarr.domain.com)
- Hostname - Hypervisor: `<hostname-hypervisor>` (e.g., proxmox-hypervisor.domain.com)
- Hostname - Hypervisor NAS: `<hostname-hypervisor-nas>` (e.g., nas-server.domain.com)
- Name - Hypervisor NAS: `<name-hypervisor-nas>` (e.g., nas-server)
- Container ID - Hypervisor: `<container-id-hypervisor>` (e.g., 100)
- **SSH Keys**
- SSH key - Proxmox: `<ssh-key-proxmox>` (e.g., /home/user/.ssh/id_rsa.pub)
- SSH key - Client: `<ssh-key-client>` (e.g., /home/user/.ssh/client_id_rsa.pub)
- **Networking**
- Wireguard port: `<wireguard-port>` (e.g., 51820)
- **Dolibarr specifics post-install**
- Contract paths: `<dolibarr-contract-paths-source>` (e.g., /home/username/contracts/)
- Fonts paths: `<dolibarr-fonts-paths-source>` (e.g., /home/username/fonts/)
## Important Warnings and Security Practices
Before executing any commands in this documentation, please adhere to the following guidelines to ensure the security and integrity of the system:
1. **Execute Commands with Caution**: Always review and understand a command before executing it. Misuse of commands can lead to data loss or system instability.
2. **Backup Command Execution**: The backup command must be executed only by authorized users. Ensure that proper permissions are set to prevent unauthorized access to backup files.
3. **Regular Backups**: Maintain regular backups of all critical data. It is advisable to use automated backup solutions and verify backup integrity periodically.
4. **System Updates**: Regularly update the system and all installed packages to protect against vulnerabilities. Use the package manager responsibly to avoid potential conflicts.
5. **Monitor System Logs**: Continuously monitor system logs for any unusual activity. Use logging tools to help identify potential security breaches or system failures.
6. **User Permissions**: Ensure that user permissions are strictly managed. Limit access to sensitive commands and data to only those who need it to perform their job functions.
7. **Network Security**: Implement proper network security measures, such as firewalls and intrusion detection systems, to protect against external threats.
8. **Data Encryption**: Encrypt sensitive data at rest and in transit to prevent unauthorized access.
By following these practices, you will help maintain the security and stability of the system while minimizing the risk of data loss or compromise.
## Useful Commands and Information
### Documentation
- [Setup other](https://wiki.dolibarr.org/index.php?title=Setup_Other)
- [Create an ODT or ODS document template](https://wiki.dolibarr.org/index.php?title=Create_an_ODT_or_ODS_document_template)
- [Setup Other](https://wiki.dolibarr.org/index.php?title=Setup_Other)
- [List of releases, change log and compatibilities](https://wiki.dolibarr.org/index.php?title=List_of_releases,_change_log_and_compatibilities)
- [GitHub](https://github.com/Dolibarr/dolibarr)
### Links
- [https](https://<hostname-intranet>/)
- [http](http://<hostname-intranet>/)
- [local https](https://<hostname-intranet>/)
- [local http](http://<hostname-intranet>/)
- [<server-ip-address>](http://<server-ip-address>/)
### Software on the Machine
- **Operating System**: Debian
- **Database**: MariaDB (configured as MySQL)
- **Web Server**: Apache
- **Security**: GnuPG, WireGuard, UFW
- **Other**: Git, sudo, certbot (for SSL certificate management)
### Paths
- **Dolibarr Configuration**: `/etc/dolibarr/app.ini`
- **Dolibarr Work Path**: `/var/lib/dolibarr`
### Proxmox Commands
**List available Proxmox templates**
```bash
ssh <username>@<hostname-hypervisor-nas> "ls /mnt/proxmox/template/cache/"
```
**Create the container**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "pct create <container-id-hypervisor> <name-hypervisor-nas>:vztmpl/debian-12-upgraded_12.5_amd64.tar.zst --hostname <hostname-intranet> --cores 4 --memory 4096 --swap 2048 --net0 name=net0,bridge=vmbr0,ip=dhcp,firewall=1 --rootfs <name-hypervisor-nas>:250 --unprivileged 1 --features nesting=1 --ssh-public-keys <ssh-key-proxmox>"
```
**Backup**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump <container-id-hypervisor> --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
```
**Set the state of the Proxmox HA Manager for Container <container-id-hypervisor>**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager add ct:<container-id-hypervisor>"
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager remove ct:<container-id-hypervisor>"
```
**Set the state and limits of the Proxmox Container <container-id-hypervisor> in the HA Manager**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id-hypervisor> --state started --max_relocate 3 --max_restart 3"
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id-hypervisor> --state stopped"
ssh <username-hypervisor>@<hostname-hypervisor> "pct start <container-id-hypervisor>"
ssh <username-hypervisor>@<hostname-hypervisor> "pct stop <container-id-hypervisor>"
ssh <username-hypervisor>@<hostname-hypervisor> "pct reboot <container-id-hypervisor>"
```
**Destroy the Proxmox Container <container-id-hypervisor> forcefully**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "pct destroy <container-id-hypervisor> --force --purge"
```
**Move the Proxmox Container <container-id-hypervisor> to another host**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "pct migrate <container-id-hypervisor> hv2"
```
### SSH Connection
**Connection with specific keys**
```bash
ssh -i <ssh-key-client> root@<hostname-intranet>
ssh -i <ssh-key-client> root@<server-ip-address>
ssh -i <ssh-key-client> <username>@<hostname-intranet>
ssh -i <ssh-key-client> <username>@<server-ip-address>
```
**Remove offending keys from known_hosts**
```bash
ssh-keygen -f "/home/<username>/.ssh/known_hosts" -R "<hostname-intranet>"
ssh-keygen -f "/home/<username>/.ssh/known_hosts" -R "<server-ip-address>"
```
**Copy SSH public key to remote host**
```bash
ssh-copy-id -i <ssh-key-client> root@<server-ip-address>
ssh-copy-id -i <ssh-key-client> root@<hostname-intranet>
ssh-copy-id -i <ssh-key-client> <username>@<server-ip-address>
ssh-copy-id -i <ssh-key-client> <username>@<hostname-intranet>
```
**Transfer SSH keys and files**
```bash
scp /home/<username>/.ssh/<username>* <username>@<hostname-intranet>:/home/<username>/.ssh/
mkdir -p /home/<username>/.ssh/
mv /home/<username>/.ssh/<username>* /home/<username>/.ssh/
chown -R <username>:<username> /home/<username>/.ssh/
cat /home/<username>/.ssh/<username>.pub >> /home/<username>/.ssh/authorized_keys
```
## Installation Procedure
1. **Fresh Debian Installation**
- Install a fresh Debian operating system on your new server following the standard installation procedure.
2. **Backup before starting**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump <container-id-hypervisor> --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
```
3. **Install Required Dependencies**
**Upgrade the base system**
```bash
apt update
apt upgrade -y
```
**Install dependencies**
- [Prerequisites](https://wiki.dolibarr.org/index.php?title=Prerequisites)
- [Dolibarr downloads](https://www.dolibarr.org/downloads.php)
- [Dolibarr Sourceforge Debian packages](https://sourceforge.net/projects/dolibarr/files/Dolibarr%20installer%20for%20Debian-Ubuntu%20%28DoliDeb%29/20.0.3/)
```bash
apt install -y git apache2 mariadb-server php php-mysql php-mbstring php-gd php-curl php-xml php-intl php-imap php-zip libapache2-mod-php sudo
apt install -y libreoffice-common libreoffice-writer --no-install-recommends
apt install -y fonts-dejavu fonts-liberation ttf-mscorefonts-installer
apt install -y gpg
```
4. **Ensure Hostname**
```bash
nano /etc/hosts
# Add line: 127.0.1.1 <hostname-intranet>
nano /etc/hostname
# Set to: <hostname-intranet>
hostnamectl set-hostname <hostname-intranet>
```
5. **Add Users**
```bash
adduser <username>
usermod -aG sudo <username>
```
```bash
mkdir -p /home/www-data
chown www-data:www-data /home/www-data
usermod -d /home/www-data www-data
usermod -s /bin/bash www-data
```
```bash
adduser --system --shell /bin/bash --gecos 'Dolibarr CRM' --group --disabled-password --home /home/www-data www-data
```
6. **Setup SSH Connectors**
- Configure SSH connectors as per your setup script to establish secure connections to the server.
7. **Test users, SSH, and sudo**
1. **Transfer SSH keys for User**
2. **Connect as User with SSH key**
3. **Test sudo**
```bash
sudo su -
```
4. **Disconnect as root**
8. **Secure SSH**
```bash
nano /etc/ssh/sshd_config
```
```ini
PermitRootLogin no
PasswordAuthentication no
ChallengeResponseAuthentication no
```
**Restart SSH**
```bash
systemctl restart ssh
```
9. **Configure Firewall**
**Open ports**
```bash
ufw allow OpenSSH
ufw allow <wireguard-port>/udp
```
**Enable firewall**
```bash
ufw enable
```
10. **Configure Mariadb**
1. **Security**
```bash
mariadb-secure-installation
mariadb
```
2. **Mariadb Users**
```sql
SET old_passwords=0;
CREATE USER 'dolibarr' IDENTIFIED BY '';
```
```sql
CREATE USER IF NOT EXISTS 'dolibarr'@'192.168.1.%' IDENTIFIED BY '';
GRANT ALL PRIVILEGES ON *.* TO 'dolibarr'@'192.168.1.%' IDENTIFIED BY '' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON *.* TO 'dolibarr'@'localhost' IDENTIFIED BY '' WITH GRANT OPTION;
SET PASSWORD FOR 'dolibarr'@'localhost' = PASSWORD('');
SET PASSWORD FOR 'dolibarr'@'192.168.1.%' = PASSWORD('');
FLUSH PRIVILEGES;
```
3. **Create Dolibarr Database**
```bash
mariadb -u root -p
```
Inside the MariaDB shell:
```sql
CREATE DATABASE dolibarr;
CREATE USER 'dolibarruser'@'localhost' IDENTIFIED BY 'strongpassword';
GRANT ALL PRIVILEGES ON dolibarr.* TO 'dolibarruser'@'localhost';
FLUSH PRIVILEGES;
EXIT;
```
11. **Install Dolibarr**
[Versions](https://wiki.dolibarr.org/index.php?title=List_of_releases,_change_log_and_compatibilities)
```bash
sudo su -
mv /var/www/html /var/www/.html
cd /var/www
git clone --depth 1 -b 20.0.3 https://github.com/Dolibarr/dolibarr.git dolibarr
mv dolibarr-20.0.3 dolibarr
```
12. **Create required directory structure**
```bash
chmod -R 755 /var/www/dolibarr
chown -R www-data:www-data /var/www/dolibarr
cd dolibarr ; touch htdocs/conf/conf.php ; chown www-data htdocs/conf/conf.php
mkdir -p /var/lib/dolibarr/sessions ; chown www-data /var/lib/dolibarr/sessions
mkdir -p /var/lib/dolibarr/documents ; chown www-data /var/lib/dolibarr/documents
```
13. **Apache Configuration**
1. **Allow http https**
```bash
ufw allow https
ufw allow http
```
2. **Backup Default Files**
Backup the default index.html and configuration files.
```bash
mv /var/www/html /var/www/.html
mv /etc/apache2/sites-available/000-default.conf /etc/apache2/sites-available/.000-default.conf
rm /etc/apache2/sites-enabled/000-default.conf
```
3. **Create Default Server Block for Unauthorized Access**
4. **Site configuration**
Edit the Apache configuration file to proxy to the Dolibarr CRM.
```bash
nano /etc/apache2/sites-available/<hostname-intranet>.conf
```
Add the following configuration:
```apache
<VirtualHost *:80>
LogLevel info
ServerName <hostname-intranet>
ServerAdmin admin@fabq.ca
DocumentRoot /var/www/dolibarr
# Alias for Dolibarr
Alias / /var/www/dolibarr/htdocs/
<Directory /var/www/dolibarr/htdocs>
Options +FollowSymLinks
AllowOverride All
Require all granted
</Directory>
ErrorLog ${APACHE_LOG_DIR}/error.dolibarr.log
CustomLog ${APACHE_LOG_DIR}/access.dolibarr.log combined
</VirtualHost>
```
5. **Enabling the new VirtualHost**
```bash
a2ensite <hostname-intranet>.conf
a2enmod rewrite
```
6. **Managing Apache**
**Enable Apache**
```bash
systemctl enable apache2
```
**Apache reload**
```bash
systemctl reload apache2
```
**`Optional` Apache restart**
```bash
systemctl restart apache2
```
14. **Create folders**
```bash
mkdir -p /var/tmp
mkdir -p /var/upload_tmp
```
15. **Set temporary file paths**
```bash
nano /etc/php/8.2/apache2/php.ini
```
```ini
sys_temp_dir = "/var/tmp"
upload_tmp_dir = "/var/upload_tmp"
open_basedir = "/var/www/dolibarr:/var/lib/dolibarr/documents:/var/lib/dolibarr/sessions:/var/tmp:/var/upload_tmp"
```
16. **Fix permissions**
```bash
chown -R www-data:www-data /var/lib/php/sessions
chown -R www-data:www-data /var/www/dolibarr/htdocs/conf/conf.php
chmod 640 /var/www/dolibarr/htdocs/conf/conf.php
find /var/www/dolibarr -type d -exec chmod 755 {} \;
find /var/www/dolibarr -type f -exec chmod 644 {} \;
chmod go-w /var/lib/dolibarr/documents;
chmod go-w /var/lib/dolibarr/sessions;
chown -R www-data:www-data /var/tmp
chown -R www-data:www-data /var/upload_tmp
chmod -R 750 /var/tmp
chmod -R 750 /var/upload_tmp
```
17. **Secure PHP**
```bash
nano /etc/php/8.2/apache2/php.ini
```
```ini
session.use_strict_mode = 1
allow_url_fopen = No
disable_functions = dl, apache_note, apache_setenv, pcntl_alarm, pcntl_fork, pcntl_waitpid, pcntl_wait, pcntl_wifexited, pcntl_wifstopped, pcntl_wifsignaled, pcntl_wifcontinued, pcntl_wexitstatus, pcntl_wtermsig, pcntl_wstopsig, pcntl_signal, pcntl_signal_get_handler, pcntl_signal_dispatch, pcntl_get_last_error, pcntl_strerror, pcntl_sigprocmask, pcntl_sigwaitinfo, pcntl_sigtimedwait, pcntl_exec, pcntl_getpriority, pcntl_setpriority, pcntl_async_signals, show_source, virtual, passthru, shell_exec, system, proc_open, popen
session.gc_maxlifetime = 604800
```
18. **Complete the Installation through Web Interface**
- Open your web browser and navigate to `http://<hostname-intranet>/install/`
- Follow the on-screen instructions to complete the installation.
19. **Lock installation**
```bash
chmod -w /var/www/dolibarr/htdocs/conf/conf.php
touch /var/lib/dolibarr/documents/install.lock;
chmod go-w /var/lib/dolibarr/documents;
```
20. **Secure installation**
```bash
chmod -R -w /var/www/dolibarr/htdocs
cp /etc/php/8.4/apache2/php.ini /etc/php/8.4/apache2/php.ini.back
nano /etc/php/8.4/apache2/php.ini
```
```ini
session.save_path = /var/lib/dolibarr/sessions
session.use_strict_mode = 1
session.use_only_cookies = 1
session.cookie_httponly = 1
session.cookie_samesite = Lax
session.gc_maxlifetime = 604800
open_basedir = "/var/www/dolibarr:/var/lib/dolibarr/documents:/var/lib/dolibarr/sessions:/var/tmp:/var/upload_tmp"
short_open_tag = Off
allow_url_fopen = Off
allow_url_include = Off
disable_functions = dl, apache_note, apache_setenv, pcntl_alarm, pcntl_fork, pcntl_waitpid, pcntl_wait, pcntl_wifexited, pcntl_wifstopped, pcntl_wifsignaled, pcntl_wifcontinued, pcntl_wexitstatus, pcntl_wtermsig, pcntl_wstopsig, pcntl_signal, pcntl_signal_get_handler, pcntl_signal_dispatch, pcntl_get_last_error, pcntl_strerror, pcntl_sigprocmask, pcntl_sigwaitinfo, pcntl_sigtimedwait, pcntl_exec, pcntl_getpriority, pcntl_setpriority, pcntl_async_signals, show_source, virtual, passthru, shell_exec, system, proc_open, popen
```
```bash
nano /var/www/dolibarr/htdocs/conf/conf.php
chmod o-r /var/www/dolibarr/htdocs/conf/conf.php
```
```php
$dolibarr_main_prod='1';
```
21. **Configure SSL with Certbot (Optional, If directly accessed)**
```bash
certbot --apache -d <hostname-internet>
systemctl status certbot.timer
certbot renew --dry-run
```
22. **Verify installation**
- Verify that the Dolibarr instance is installed restored by accessing it through a web browser. Ensure that all repositories, users, and configurations are intact.
```bash
systemctl status apache2
```
Visit the server at [http://<hostname-internet>](http://<hostname-internet>) or [https://<hostname-internet>](https://<hostname-internet>).
23. **Backup post installation**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id-hypervisor> --state stopped"
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump <container-id-hypervisor> --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup post installation\""
```
24. **Start the server**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id-hypervisor> --state started --max_relocate 3 --max_restart 3"
```
25. **Move contracts**
```bash
scp '<dolibarr-contract-paths-source>/contract.odt' <username>@<hostname-intranet>:~/
sudo su -
mv /home/<username>/*.odt /var/lib/dolibarr/documents/doctemplates/contracts/
chown -R www-data:www-data /var/lib/dolibarr/documents/doctemplates/contracts/
```
26. **ODT to PDF**
[See forum 1](https://www.dolibarr.org/forum/t/solved-odt-to-pdf/16931)
[See forum 2](https://www.dolibarr.org/forum/t/setup-dolibarr-to-generate-odt-to-pdf/22112/9)
```bash
MAIN_ODT_AS_PDF
```
27. **Install fonts**
```bash
scp -r <dolibarr-fonts-paths-source>/Lato <username>@<hostname-intranet>:~/
```
```bash
mv /home/<username>/Lato /usr/local/share/fonts/
chmod -R 755 /usr/local/share/fonts/
chown -R root:root /usr/local/share/fonts/
```

35
setups/debian_setup_gitea.md
View File

@ -63,7 +63,6 @@ Replace the placeholders below with the appropriate values for your setup:
- Hostname - Hypervisor: `<hostname-hypervisor>` (e.g., proxmox-hypervisor.domain.com)
- Hostname - Hypervisor NAS: `<hostname-hypervisor-nas>` (e.g., nas-server.domain.com)
- Name - Hypervisor NAS: `<name-hypervisor-nas>` (e.g., nas-server)
- Container ID - Hypervisor: `<container-id-hypervisor>` (e.g., 100)
- **SSH Keys**
@ -127,40 +126,40 @@ ssh <username>@<hostname-hypervisor-nas> "ls /mnt/proxmox/template/cache/"
**Create the container**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "pct create <container-id-hypervisor> <name-hypervisor-nas>:vztmpl/debian-12-upgraded_12.5_amd64.tar.zst --hostname <hostname-intranet> --cores 2 --memory 2048 --swap 2048 --net0 name=net0,bridge=vmbr0,ip=dhcp,firewall=1 --rootfs <name-hypervisor-nas>:250 --unprivileged 1 --features nesting=1 --ssh-public-keys <ssh-key-proxmox> --start 1"
ssh <username-hypervisor>@<hostname-hypervisor> "pct create 100 <name-hypervisor-nas>:vztmpl/debian-12-upgraded_12.5_amd64.tar.zst --hostname <hostname-intranet> --cores 2 --memory 2048 --swap 2048 --net0 name=net0,bridge=vmbr0,ip=dhcp,firewall=1 --rootfs <name-hypervisor-nas>:250 --unprivileged 1 --features nesting=1 --ssh-public-keys <ssh-key-proxmox> --start 1"
```
**Backup**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump <container-id-hypervisor> --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump 100 --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
```
**Set the state of the Proxmox HA Manager for Container <container-id-hypervisor>**
**Set the state of the Proxmox HA Manager for Container 100**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager add ct:<container-id-hypervisor>"
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager remove ct:<container-id-hypervisor>"
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager add ct:100"
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager remove ct:100"
```
**Set the state and limits of the Proxmox Container <container-id-hypervisor> in the HA Manager**
**Set the state and limits of the Proxmox Container 100 in the HA Manager**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id-hypervisor> --state started --max_relocate 3 --max_restart 3"
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id-hypervisor> --state stopped"
ssh <username-hypervisor>@<hostname-hypervisor> "pct reboot <container-id-hypervisor>"
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:100 --state started --max_relocate 3 --max_restart 3"
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:100 --state stopped"
ssh <username-hypervisor>@<hostname-hypervisor> "pct reboot 100"
```
**Destroy the Proxmox Container <container-id-hypervisor> forcefully**
**Destroy the Proxmox Container 100 forcefully**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "pct destroy <container-id-hypervisor> --force --purge"
ssh <username-hypervisor>@<hostname-hypervisor> "pct destroy 100 --force --purge"
```
**Move the Proxmox Container <container-id-hypervisor> to another host**
**Move the Proxmox Container 100 to another host**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "pct migrate <container-id-hypervisor> hv2"
ssh <username-hypervisor>@<hostname-hypervisor> "pct migrate 100 hv2"
```
### SSH Connection
@ -209,7 +208,7 @@ cat /home/<username>/.ssh/<username>.pub >> /home/<username>/.ssh/authorized_key
2. **Backup before starting**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump <container-id-hypervisor> --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump 100 --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup fresh install\""
```
3. **Install Required Dependencies**
@ -504,11 +503,11 @@ cat /home/<username>/.ssh/<username>.pub >> /home/<username>/.ssh/authorized_key
19. **Backup post installation**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id-hypervisor> --state stopped"
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump <container-id-hypervisor> --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup post installation\""
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:100 --state stopped"
ssh <username-hypervisor>@<hostname-hypervisor> "vzdump 100 --compress zstd --mode stop --storage <name-hypervisor-nas> --note \"$(date +'%Y-%m-%d %H:%M') Backup post installation\""
```
20. **Start the server**
```bash
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:<container-id-hypervisor> --state started --max_relocate 3 --max_restart 3"
ssh <username-hypervisor>@<hostname-hypervisor> "ha-manager set ct:100 --state started --max_relocate 3 --max_restart 3"
```