feat: new version, auto GIF conversion, meta improvements

vas3k · May 13, 2024 · a51c656 · a51c656
1 parent 0741b92
commit a51c656
Show file tree

Hide file tree

Showing 21 changed files with 287 additions and 158 deletions.
diff --git a/README.md b/README.md
@@ -4,63 +4,84 @@
   <h1>PEPIC</h1>
 </div>
 
-Pepic is a small app that helps me to upload, store, convert and serve pictures (or videos) on my dedicated servers.
+Pepic is a small self-hosted media proxy that helps me to upload, store, serve and convert pictures and videos on my own servers locally.
 
-I use it as media proxy for my [pet-projects](https://github.com/vas3k/vas3k.club) and on [my blog](https://vas3k.com). 
-Pepic can convert, resize and optimize media files in-flight to save you monies and bandwidth. 
-Internally it uses [ffmpeg](https://ffmpeg.org/download.html) for videos and [vips](https://libvips.github.io/libvips/install.html) for images,
-which means it's quite fast and supports **JPG, PNG, GIF, WEBP, SVG, HEIF, TIFF** and wide range of video formats.  
+Currently, I use it as a main storage for media files in my [pet-projects](https://github.com/vas3k/vas3k.club) and on [my blog](https://vas3k.blog).
 
-It's not meant to be used by anyone else except me. Use it only if you're brave. Scroll down this README for better alternatives.
+Pepic can upload and optimize media files in-flight to save you money and bandwidth. It's highly recommended to use it in combination with Cloudflare CDN for better caching.
 
-## 🤖 How to run it locally
+Internally, Pepic it uses [ffmpeg](https://ffmpeg.org/download.html) for videos and [vips](https://libvips.github.io/libvips/install.html) for images, which makes it quite fast and supports many media file formats.
 
-This command starts a local server on [localhost:8118](http://localhost:8118). Useful for development.
+Images: **JPG, PNG, GIF, WEBP, SVG, HEIF, TIFF, AVIF, etc**
+Video: **basically everything ffmpeg supports**
 
+Pepic is open source, however it's not meant to be used by anyone. Only if you're brave. Scroll down this README for better alternatives.
+
+
+## Main Features
+
+- **Local files upload**: Accept files in multipart/form-data or bytes and store them to a local directory.
+- **Automatic GIF to video conversion**: Convert GIFs to videos because GIFs suck, slow down web pages, and don't support hardware acceleration.
+- **Media format conversion and optimization**: Convert and optimize media files on-the-fly.
+- **Dynamic resizing**: Easily resize images in real-time by modifying the URL, which helps in reducing bandwidth and storage space on devices.
+- **High performance**: Pepic uses native libraries like `ffmpeg` and `vips` for video and image processing to ensure high performance and fast processing times.
+- **Local and containerized environments**: Designed to run smoothly in both local environments and within Docker containers, making it versatile for development and deployment.
+- **Custom configuration**: Flexible configuration options through `config.yml`, allowing adjustments to image size, quality, automatic conversion, templates, etc.
+
+![](static/images/screenshot1.png)
+
+## 🤖 How to Run
+
+1. Install `vips` and `ffmpeg` first, as they are external dependencies.
+
+```bash
+brew install vips ffmpeg
 ```
+
+2. Use the following command to start a local server on [localhost:8118](http://localhost:8118).
+
+```bash
 go run main.go serve --config ./etc/pepic/config.yml
 ```
 
-> ⚠️ If you're getting `invalid flag in pkg-config` error, run `export CGO_CFLAGS_ALLOW="-Xpreprocessor"` in advance
+> ⚠️ If you're getting `invalid flag in pkg-config` error, run `brew install pkg-config` and `export CGO_CFLAGS_ALLOW="-Xpreprocessor"`. Then try `go run` again.
+
+3. Enjoy!
 
 ## 🐳 Running in Docker
 
 1. Get [Docker](https://www.docker.com/get-started)
 
 2. Clone the repo
 
-```
+```bash
 git clone [email protected]:vas3k/pepic.git
 cd pepic
 ```
 
 3. Build and run the app
 
-```
+```bash
 docker build .
 docker run -p 8118:8118 -v ${PWD}/uploads:/app/uploads $(docker build -q .)
 ```
 
-4. Go to [http://localhost:8118](http://localhost:8118) and try uploading something. 
-You should see uploaded images or videos in the data directory (`./uploads`) after that.
+4. Go to [http://localhost:8118](http://localhost:8118) and try uploading something. You should see uploaded images or videos in the data directory (`./uploads`) after that.
 
 5. Try to resize an image by adding a number of pixels to its URL. For example: `https://localhost:8118/file.jpg -> https://localhost:8118/500/file.jpg`
 
-6. Check out the [etc/pepic/config.yml](etc/pepic/config.yml) file. Some stuff is turned off by default.
-You can tweak them for yourself and rebuild the docker again (step 3) to apply them.
+6. Check out the [etc/pepic/config.yml](etc/pepic/config.yml) file. Some stuff is turned off by default. You can tweak them for yourself and rebuild the Docker again (step 3) to apply them.
 
-![](static/images/screenshot1.png)
 
-## 🚢 Production Usage
+## 🚢 Production Deployment
 
-> ⚠️ If you plan to host anything bigger than a blog, always put it behind CDN. 
-> CloudFlare offers a free one if you don't hate big corporations :D
+> ⚠️ If you plan to host anything bigger than a blog, always put Pepic behind a CDN. CloudFlare offers a free one if you don't hate big corporations :D
 
 Let's say, you want to host it on `https://media.mydomain.org`
 
-#### 1. Modify `etc/pepic/config.yml` to your taste
+1. Modify `etc/pepic/config.yml` to your taste
 
-```
+```yaml
 global:
   host: 0.0.0.0 
   port: 8118  # internal host and port, leave it as it is
@@ -69,31 +90,44 @@ global:
   max_upload_size: "500M"
 ```
 
-#### 2. Build and run production docker
+2. Build and run production Docker
 
-Don't forget to mount upload volume to store files on host (or you can lose those files when container will be killed).
+Don't forget to mount upload volume to store files on host (or you can lose those files when the container is killed).
 
-```
+```bash
 docker run -p 8118:8118 -v /host/dir/uploads:/app/uploads --restart=always $(docker build -q .)
 ```
 
 If you prefer docker-compose, you can use it too. Check out the included [docker-compose.example.yml](docker-compose.example.yml). 
 You can easily transform it into your favourite k8s config or whatever is fashionable this summer. 
 
-> 👍 Don't forger to periodically backup the `/host/dir/uploads` directory just in case :)
+> 👍 Don't forget to periodically backup the `/host/dir/uploads` directory just in case :)
 
-#### 3. Use nginx or your other favourite proxy
+3. Use nginx or your other favourite proxy
 
-Just proxy all calls from the domain (media.mydomain.org) to pepic backend (0.0.0.0:8118). It can handle static files too.
+Just proxy all calls from the domain (media.mydomain.org) to Pepic backend (0.0.0.0:8118). Don't forget to set `max file size` and `proxy timeot` directives to avoid gateway errors on big files (especially videos).
 
-```
+```nginx
 server {
     listen 80;
     server_name media.mydomain.org;
 
+    client_max_body_size 500M;
+    real_ip_header X-Real-IP;
+
     location / {
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_read_timeout 300;
+        proxy_connect_timeout 300;
+        proxy_send_timeout 300;
+        send_timeout 300;
+
         proxy_set_header Host $http_host;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_redirect off;
+        proxy_buffering off;
+
         proxy_pass http://0.0.0.0:8118;
     }
 }
@@ -103,15 +137,16 @@ server {
 
 Contributions are welcome.  
 
-Open an [Issue](https://github.com/vas3k/vas3k.club/issues) if you want to report a bug and propose an idea.
+Open an [Issue](https://github.com/vas3k/vas3k.club/issues) if you want to report a bug or propose an idea.
 
 ## ✅ TODO
 
 - [ ] Tests :D
 - [ ] Upload by URL
 - [ ] Crop, rotate and other useful transformations (face blur? pre-loader generator?)
 - [ ] Live conversion by changing file's extension 
-- [ ] Set format and quality during the upload (using GET/POST params?)
+- [ ] Set format and media quality during the upload (using GET/POST params?)
+
 
 ## 🤔 Alternatives
 
@@ -121,6 +156,7 @@ After reading all this, you probably realized how bad it is and looking for othe
 - [imaginary](https://github.com/h2non/imaginary)
 - [flyimg](https://github.com/flyimg/flyimg)
 
+
 ## 👩‍💼 License 
 
 It's [MIT](LICENSE).

diff --git a/etc/pepic/config.yml b/etc/pepic/config.yml
@@ -17,6 +17,7 @@ images:
   live_resize: true # enables special URLs that return resized images (increases storage usage)
   jpeg_quality: 95 # default quality of any saved jpeg
   png_compression: 0  # 0 - default, -1 - no compression, -2 - best speed, -3 - best compression
+  gif_convert: "video/mp4"  # video format for auto-converting gifs (turned off then store_originals=true)
 
 videos:
   store_originals: false # use "true" if you want to store original files (browser compatibility is on you)
@@ -37,7 +38,7 @@ videos:
     pix_fmt: "yuv420p"
 
 meta:
-  blocks:  # you can add your custom blocks to make it easier to copy-paste
+  image_templates:  # you can add your custom blocks to make it easier to copy-paste
     - title: "URL"
       template: "{{ file.Url }}"
     - title: "Simple Markdown"
@@ -50,11 +51,17 @@ meta:
       template: "{% verbatim %}{{{{% endverbatim %}.block-side.block-side__right  ![]({{ file.Url }}) {% verbatim %}}}}{% endverbatim %}"
     - title: "Left"
       template: "{% verbatim %}{{{{% endverbatim %}.block-side.block-side__left  ![]({{ file.Url }}) {% verbatim %}}}}{% endverbatim %}"
-    - title: "2 in a row"
-      template: "{% verbatim %}{{{{% endverbatim %}.block-media.block-media__2  ![]({{ file.Url }})  {% verbatim %}}}}{% endverbatim %}"
-    - title: "3 in a row"
-      template: "{% verbatim %}{{{{% endverbatim %}.block-media.block-media__3-full  ![]({{ file.Url }})  {% verbatim %}}}}{% endverbatim %}"
     - title: "75% center"
-      template: "{% verbatim %}{{{{% endverbatim %}.block-media.block-media__body.width-75 ![]({{ file.Url }}) {% verbatim %}}}}{% endverbatim %}"
+      template: "{% verbatim %}{{{{% endverbatim %}.block-media.block-media__body.width-75 {% for file in files %}![]({{ file.Url }}) {% endfor %} {% verbatim %}}}}{% endverbatim %}"
     - title: "50% center"
-      template: "{% verbatim %}{{{{% endverbatim %}.block-media.block-media__body.width-50 ![]({{ file.Url }}) {% verbatim %}}}}{% endverbatim %}"
+      template: "{% verbatim %}{{{{% endverbatim %}.block-media.block-media__body.width-50 {% for file in files %}![]({{ file.Url }}) {% endfor %} {% verbatim %}}}}{% endverbatim %}"
+  video_templates:
+    - title: "URL"
+      template: "{{ file.Url }}"
+    - title: "Simple Markdown"
+      template: "![]({{ file.Url }})"
+  multi_templates:
+    - title: "2 in a row"
+      template: "{% verbatim %}{{{{% endverbatim %}.block-media.block-media__2 {% for file in files %}![]({{ file.Url }}) {% endfor %}  {% verbatim %}}}}{% endverbatim %}"
+    - title: "3 in a row"
+      template: "{% verbatim %}{{{{% endverbatim %}.block-media.block-media__3-full  {% for file in files %}![]({{ file.Url }}) {% endfor %} {% verbatim %}}}}{% endverbatim %}"
diff --git a/go.mod b/go.mod
@@ -1,12 +1,32 @@
 module github.com/vas3k/pepic
 
-go 1.14
+go 1.22
 
 require (
 	github.com/flosch/pongo2 v0.0.0-20200529170236-5abacdfa4915
-	github.com/h2non/bimg v1.1.5
+	github.com/h2non/bimg v1.1.9
 	github.com/ilyakaznacheev/cleanenv v1.2.3
-	github.com/labstack/echo/v4 v4.1.16
+	github.com/labstack/echo/v4 v4.12.0
 	github.com/spf13/cobra v1.1.1
-	github.com/xfrr/goffmpeg v0.0.0-20200403115021-c3a1545b29f4
+	github.com/xfrr/goffmpeg v1.0.0
+)
+
+require (
+	github.com/BurntSushi/toml v0.3.1 // indirect
+	github.com/golang-jwt/jwt v3.2.2+incompatible // indirect
+	github.com/inconshreveable/mousetrap v1.0.0 // indirect
+	github.com/joho/godotenv v1.3.0 // indirect
+	github.com/labstack/gommon v0.4.2 // indirect
+	github.com/mattn/go-colorable v0.1.13 // indirect
+	github.com/mattn/go-isatty v0.0.20 // indirect
+	github.com/spf13/pflag v1.0.5 // indirect
+	github.com/valyala/bytebufferpool v1.0.0 // indirect
+	github.com/valyala/fasttemplate v1.2.2 // indirect
+	golang.org/x/crypto v0.22.0 // indirect
+	golang.org/x/net v0.24.0 // indirect
+	golang.org/x/sys v0.19.0 // indirect
+	golang.org/x/text v0.14.0 // indirect
+	golang.org/x/time v0.5.0 // indirect
+	gopkg.in/yaml.v2 v2.2.8 // indirect
+	olympos.io/encoding/edn v0.0.0-20200308123125-93e3b8dd0e24 // indirect
 )