From 20dfae7ddcc819e63ef20fb4e008abf2e7a2d287 Mon Sep 17 00:00:00 2001 From: Richy Date: Thu, 12 Dec 2024 15:58:15 +0100 Subject: [PATCH] =?UTF-8?q?=E2=9C=A8=20New=20tutorial=20-=20"Object=20Stor?= =?UTF-8?q?age=20Based=20Filesystem=20with=20s3fs"=20=20(#1007)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * added s3fs tutorial * added License * Spelling and formatting updates --------- Co-authored-by: svenja.michal <84835304+svenja11@users.noreply.github.com> --- .../object-storage-based-filesystem/01.de.md | 263 ++++++++++++++++++ .../object-storage-based-filesystem/01.en.md | 263 ++++++++++++++++++ 2 files changed, 526 insertions(+) create mode 100644 tutorials/object-storage-based-filesystem/01.de.md create mode 100644 tutorials/object-storage-based-filesystem/01.en.md diff --git a/tutorials/object-storage-based-filesystem/01.de.md b/tutorials/object-storage-based-filesystem/01.de.md new file mode 100644 index 000000000..7d7790ff5 --- /dev/null +++ b/tutorials/object-storage-based-filesystem/01.de.md @@ -0,0 +1,263 @@ +--- +SPDX-License-Identifier: MIT +path: "/tutorials/object-storage-based-filesystem/de" +slug: "object-storage-based-filesystem" +date: "2024-12-12" +title: "Objektspeicher als Dateisystem mit s3fs" +short_description: "Lerne, wie man Object Storage als lokales Dateisystem mit s3fs einbindet. Inklusive Installation, Konfiguration und Leistungsoptimierung." +tags: ["Development", "Object Storage", "Filesystem". "s3fs"] +author: "Richy" +author_link: "https://github.com/DasCanard" +author_img: "https://avatars.githubusercontent.com/u/17070204" +author_description: "" +language: "de" +available_languages: ["de", "en"] +header_img: "header-5" +cta: "cloud" +--- + +## Einführung + +In diesem Tutorial wird erklärt, wie man einen S3 Bucket mittels [s3fs](https://github.com/s3fs-fuse/s3fs-fuse) in das lokale Dateisystem eines Servers einhängt. + +> **Empfehlung**: Diese Anleitung verwendet Hetzner Object Storage, der S3-kompatiblen Speicher mit ausgezeichnetem Preis-Leistungs-Verhältnis und europäischen Rechenzentren bietet. Weitere Informationen: [Hetzner Object Storage Dokumentation](https://docs.hetzner.com/storage/object-storage/overview/) + +## Schritt 1 - Installation + +* Für Ubuntu/Debian: + ```bash + sudo apt-get update + sudo apt-get install s3fs + ``` + +* Für CentOS/RHEL: + ```bash + sudo yum install s3fs-fuse + ``` + +* Aus dem Quellcode (falls nötig): + ```bash + sudo apt-get install automake autotools-dev g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config + git clone https://github.com/s3fs-fuse/s3fs-fuse.git + cd s3fs-fuse + ./autogen.sh + ./configure + make + sudo make install + ``` + +## Schritt 2 - Zugangsdaten konfigurieren und Mountpunkt erstellen +> Für Hetzner Object Storage, erstelle die Zugangsdaten in der Hetzner Cloud Console + +1. Zugangsdatei erstellen: + ```bash + echo "ACCESS_KEY_ID:SECRET_ACCESS_KEY" | sudo tee /root/.passwd-s3fs > /dev/null + ``` + +2. Korrekte Berechtigungen setzen: + ```bash + sudo chmod 600 /root/.passwd-s3fs + ``` + +3. Mountpunkt erstellen + ```bash + sudo mkdir /mnt/s3storage + ``` + +## Schritt 3 - Mount-Optionen + +Ersetze in den Mount-Befehlen die Platzhalter `` und `https://nbg1.your-objectstorage.com/` mit dem tatsächlichen Bucket-Namen und Endpunkt. + +Wenn du Hetzner Object Storage verwendest, muss die URL entsprechend der Region gewählt werden, in welcher der Bucket erstellt wurde. Die Region kannst du in der Hetzner Cloud Console unter "Object Storage" einsehen. Im [Object Storage Überblick](https://docs.hetzner.com/de/storage/object-storage/overview#verfugbare-endpunkte) findest du eine Liste verfügbarer Endpunkte. Beispiel: Wenn dein Bucket in Nürnberg erstellt wurde, verwende in den Mount-Befehlen und der fstab die URL `https://nbg1.your-objectstorage.com`. Die Region kann nach der Bucket-Erstellung nicht mehr geändert werden. + +> **Hinweis**: Eine falsche URL führt zu Verbindungsfehlern. + +* Manueller Mount-Befehl + ```bash + sudo s3fs /mnt/s3storage \ + -o url=https://nbg1.your-objectstorage.com/ \ + -o allow_other \ + -o use_path_request_style \ + -o use_cache=/tmp/s3fs \ + -o multipart_size=100 \ + -o parallel_count=8 \ + -o big_writes \ + -o kernel_cache \ + -o umask=0022 \ + -o enable_noobj_cache \ + -o retries=5 \ + -o ensure_diskfree=20000 \ + -o connect_timeout=180 \ + -o max_dirty_data=1024 \ + -o max_stat_cache_size=100000 \ + -o passwd_file=/root/.passwd-s3fs + ``` + +* Automatischer Mount über fstab + + Füge diese Zeile zu `/etc/fstab` hinzu: + ```bash + s3fs# /mnt/s3storage fuse _netdev,allow_other,use_path_request_style,url=https://nbg1.your-objectstorage.com/,use_cache=/tmp/s3fs,multipart_size=100,parallel_count=8,big_writes,kernel_cache,umask=0022,enable_noobj_cache,retries=5,ensure_diskfree=20000,connect_timeout=180,max_dirty_data=1024,max_stat_cache_size=100000,passwd_file=/root/.passwd-s3fs 0 0 + ``` + +Erklärung der Konfigurationsparameter: + +Netzwerkeinstellungen + +| Flag | Beschreibung | +| ----------- | ------------ | +| url= | Object Storage Endpunkt-URL | +| connect_timeout=180 | Verbindungs-Timeout in Sekunden | +| retries=5 | Anzahl der Wiederholungsversuche | +| use_path_request_style | Verwendet Pfad-Stil S3-URLs | + +Cache-Konfiguration + +| Flag | Beschreibung | +| ----------- | ------------ | +| use_cache=/tmp/s3fs | Lokales Cache-Verzeichnis | +| max_stat_cache_size=100000 | Maximale Anzahl der Stat-Cache-Einträge | +| enable_noobj_cache | Speichert nicht existierende Objekte im Cache | +| max_dirty_data=1024 | Maximale Menge an unsauberen Cache-Daten (MB) | + +Leistungsoptionen + +| Flag | Beschreibung | +| ----------- | ------------ | +| multipart_size=100 | Größe für Multipart-Uploads (MB) | +| parallel_count=8 | Anzahl paralleler Verbindungen | +| big_writes | Aktiviert größere Schreiboperationen | +| kernel_cache | Aktiviert Kernel-Caching | +| ensure_diskfree=20000 | Minimaler freier Speicherplatz (MB) | + +Berechtigungseinstellungen + +| Flag | Beschreibung | +| ----------- | ------------ | +| allow_other | Erlaubt Zugriff durch andere Benutzer | +| umask=0022 | Standard-Unix-Berechtigungen | + +## Schritt 4 - Test und Überprüfung + +* Mount-Befehl testen + ```bash + # Manueller Mount + sudo s3fs bucketname /mnt/s3storage [Optionen wie oben] + + # Mount überprüfen + df -h + mount | grep s3fs + ``` + +* fstab-Eintrag testen + ```bash + # fstab-Eintrag ohne Neustart testen + sudo mount -a + + # Mount überprüfen + df -h + mount | grep s3fs + ``` + +## Schritt 5 - Fehlerbehebung + +* Debug-Modus + ```bash + # Diese Optionen für Debugging hinzufügen + -o dbglevel=info -f -o curldbg + ``` + +* Häufige Probleme + + Berechtigungsprobleme: + ```bash + # Dateiberechtigungen prüfen + sudo ls -la /root/.passwd-s3fs + ls -la /mnt/s3storage + ``` + + Cache-Probleme: + ```bash + # Cache leeren + sudo rm -rf /tmp/s3fs/* + ``` + +## Schritt 6 - Wartung + +* Aushängen + ```bash + # Manuelles Aushängen + sudo umount /mnt/s3storage + + # Erzwungenes Aushängen falls nötig + sudo umount -f /mnt/s3storage + ``` + +* Cache-Verwaltung + ```bash + # Cache leeren + sudo rm -rf /tmp/s3fs/* + + # Neues Cache-Verzeichnis erstellen + sudo mkdir -p /tmp/s3fs + sudo chmod 777 /tmp/s3fs + ``` + +## Schritt 7 - Sicherheitsempfehlungen + +1. Immer HTTPS-Endpunkte verwenden +2. Zugangsdatei sichern: + ```bash + sudo chmod 600 /root/.passwd-s3fs + ``` +3. Regelmäßige Berechtigungsprüfungen +4. Zugriffsprotokolle überwachen +5. Backup-Strategie implementieren + +## Schritt 8 - Leistungsoptimierung für Object Storage + +Für optimale Leistung mit Object Storage: + +- Wähle den nächstgelegenen Endpunkt zu deinem Server +- Verwende eine angemessene `multipart_size` (100MB ist für die meisten Fälle gut) +- Passe `parallel_count` basierend auf deiner Bandbreite an (8-16 wird empfohlen) +- Aktiviere `kernel_cache` für bessere Leseleistung +- Nutze `big_writes` für verbesserte Schreibleistung +- Erwäge die Nutzung eines Servern in der gleichen Region wie dein Object Storage + +Denken daran, nach allen Konfigurationsänderungen gründlich zu testen. + +## Ergebnis + +In `/mnt/s3storage` solltest du nun die Inhalte deines S3-Buckets sehen. Über diesen Pfad kannst du jetzt Daten in deinem Bucket hinzufügen oder löschen, genau wie du es in jedem anderen Ordner auf dem Server würdest. + +##### License: MIT + + \ No newline at end of file diff --git a/tutorials/object-storage-based-filesystem/01.en.md b/tutorials/object-storage-based-filesystem/01.en.md new file mode 100644 index 000000000..2784b9ea4 --- /dev/null +++ b/tutorials/object-storage-based-filesystem/01.en.md @@ -0,0 +1,263 @@ +--- +SPDX-License-Identifier: MIT +path: "/tutorials/object-storage-based-filesystem" +slug: "object-storage-based-filesystem" +date: "2024-12-12" +title: "Object Storage Based Filesystem with s3fs" +short_description: "Learn how to mount Object Storage as a local filesystem using s3fs. Includes installation, configuration, and performance optimization." +tags: ["Development", "Object Storage", "Filesystem". "s3fs"] +author: "Richy" +author_link: "https://github.com/DasCanard" +author_img: "https://avatars.githubusercontent.com/u/17070204" +author_description: "" +language: "en" +available_languages: ["en", "de"] +header_img: "header-5" +cta: "cloud" +--- + +## Introduction + +This tutorial explains how to mount an S3 bucket onto the local filesystem of a server using [s3fs](https://github.com/s3fs-fuse/s3fs-fuse). + +> **Recommendation**: This guide uses Hetzner Object Storage, which offers S3-compatible storage with excellent price-performance ratio and European data centers. More information: [Hetzner Object Storage Documentation](https://docs.hetzner.com/storage/object-storage/overview/) + +## Step 1 - Installation + +* For Ubuntu/Debian: + ```bash + sudo apt-get update + sudo apt-get install s3fs + ``` + +* For CentOS/RHEL: + ```bash + sudo yum install s3fs-fuse + ``` + +* From source (if needed): + ```bash + sudo apt-get install automake autotools-dev g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config + git clone https://github.com/s3fs-fuse/s3fs-fuse.git + cd s3fs-fuse + ./autogen.sh + ./configure + make + sudo make install + ``` + +## Step 2 - Configure Access Credentials and mount point +> For Hetzner Object Storage, create access credentials in the Hetzner Cloud Console + +1. Create a credentials file: + ```bash + echo "ACCESS_KEY_ID:SECRET_ACCESS_KEY" | sudo tee /root/.passwd-s3fs > /dev/null + ``` + +2. Set proper permissions: + ```bash + sudo chmod 600 /root/.passwd-s3fs + ``` + +3. Create the mount point + ```bash + sudo mkdir /mnt/s3storage + ``` + +## Step 3 - Mounting Options + +In the mount commands, replace `` and `https://nbg1.your-objectstorage.com/` with your actual bucket name and endpoint. + +If you use Hetzner Object Storage, the URL must be chosen according to the region where your Bucket was created. You can find your region in the Hetzner Cloud Console under "Object Storage". You can find a list of available regions and endpoints in their [Object Storage overview](https://docs.hetzner.com/storage/object-storage/overview#available-endpoints). Example: If your Bucket was created in Nuremberg, use the URL `https://nbg1.your-objectstorage.com` in your mount commands and fstab. You cannot change the region after the Bucket was created. + +Note: An incorrect endpoint URL will result in connection errors. + +* Manual Mount Command + ```bash + sudo s3fs /mnt/s3storage \ + -o url=https://nbg1.your-objectstorage.com/ \ + -o allow_other \ + -o use_path_request_style \ + -o use_cache=/tmp/s3fs \ + -o multipart_size=100 \ + -o parallel_count=8 \ + -o big_writes \ + -o kernel_cache \ + -o umask=0022 \ + -o enable_noobj_cache \ + -o retries=5 \ + -o ensure_diskfree=20000 \ + -o connect_timeout=180 \ + -o max_dirty_data=1024 \ + -o max_stat_cache_size=100000 \ + -o passwd_file=/root/.passwd-s3fs + ``` + +* Automatic Mount via fstab + + Add this line to `/etc/fstab`: + ```bash + s3fs# /mnt/s3storage fuse _netdev,allow_other,use_path_request_style,url=https://nbg1.your-objectstorage.com/,use_cache=/tmp/s3fs,multipart_size=100,parallel_count=8,big_writes,kernel_cache,umask=0022,enable_noobj_cache,retries=5,ensure_diskfree=20000,connect_timeout=180,max_dirty_data=1024,max_stat_cache_size=100000,passwd_file=/root/.passwd-s3fs 0 0 + ``` + +Configuration Parameters Explained: + +Network Settings + +| Flag | Description | +| ----------- | ----------- | +| url= | Object storage endpoint URL | +| connect_timeout=180 | Connection timeout in seconds | +| retries=5 | Number of retry attempts | +| use_path_request_style | Uses path-style S3 URLs | + +Cache Configuration + +| Flag | Description | +| ----------- | ----------- | +| use_cache=/tmp/s3fs | Local cache directory | +| max_stat_cache_size=100000 | Maximum stat cache entries | +| enable_noobj_cache | Caches non-existent objects | +| max_dirty_data=1024 | Maximum dirty cache data (MB) | + +Performance Options + +| Flag | Description | +| ----------- | ----------- | +| multipart_size=100 | Multipart upload size (MB) | +| parallel_count=8 | Parallel connection count | +| big_writes | Enables larger write operations | +| kernel_cache | Enables kernel caching | +| ensure_diskfree=20000 | Minimum free space (MB) | + +Permission Settings + +| Flag | Description | +| ----------- | ----------- | +| allow_other | Allows access by other users | +| umask=0022 | Standard Unix permissions | + +## Step 4 - Testing and Verification + +* Test Mount Command + ```bash + # Manual mount + sudo s3fs your-bucket /mnt/s3storage [options as above] + + # Verify mount + df -h + mount | grep s3fs + ``` + +* Test fstab Entry + ```bash + # Test fstab entry without reboot + sudo mount -a + + # Verify mount + df -h + mount | grep s3fs + ``` + +## Step 5 - Troubleshooting + +* Debug Mode + ```bash + # Add these options for debugging + -o dbglevel=info -f -o curldbg + ``` + +* Common Issues + + Permission Problems: + ```bash + # Check file permissions + sudo ls -la /root/.passwd-s3fs + ls -la /mnt/s3storage + ``` + + Cache Issues: + ```bash + # Clear cache + sudo rm -rf /tmp/s3fs/* + ``` + +## Step 6 - Maintenance + +* Unmounting + ```bash + # Manual unmount + sudo umount /mnt/s3storage + + # Force unmount if needed + sudo umount -f /mnt/s3storage + ``` + +* Cache Management + ```bash + # Clear cache + sudo rm -rf /tmp/s3fs/* + + # Create new cache directory + sudo mkdir -p /tmp/s3fs + sudo chmod 777 /tmp/s3fs + ``` + +## Step 7 - Security Best Practices + +1. Always use HTTPS endpoints +2. Secure credentials file: + ```bash + sudo chmod 600 /root/.passwd-s3fs + ``` +3. Regular permission audits +4. Monitor access logs +5. Implement backup strategy + +## Step 8 - Performance Optimization for Object Storage + +For optimal performance with object storage: + +- Choose the closest endpoint to your server +- Use appropriate `multipart_size` (100MB is good for most cases) +- Adjust `parallel_count` based on your bandwidth (8-16 is recommended) +- Enable `kernel_cache` for better read performance +- Use `big_writes` for improved write performance +- Consider using a server that is in the same region as your object storage + +Remember to test thoroughly after any configuration changes. + +## Conclusion + +In `/mnt/s3storage`, you should see the contents of you S3 bucket. Via this path, you can now add or remove data in your bucket just as you would in any other directory on the server. + +##### License: MIT + + \ No newline at end of file