Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DietPi-Software | Lemmy: A link aggregator for the fediverse #6434

Draft
wants to merge 15 commits into
base: dev
Choose a base branch
from
Draft

DietPi-Software | Lemmy: A link aggregator for the fediverse #6434

Changes from 13 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
2131aea
initial lemmy skeleton, based on
mtekman Jun 15, 2023
f4aacea
initial cleanup, untested
mtekman Jun 15, 2023
cb1cd21
testing installation
mtekman Jun 15, 2023
768781b
second test
mtekman Jun 16, 2023
efc0b42
running on vmachine
mtekman Jun 16, 2023
636bdbc
Run and build lemmy as own user.
mtekman Jun 16, 2023
4935e8c
fixed postgresql issue
mtekman Jun 17, 2023
e8c29b7
updated the update script
mtekman Jun 17, 2023
0afb53e
Added uninstall script
mtekman Jun 17, 2023
915c98a
namefix
mtekman Jun 17, 2023
e02dec2
shellcheck fixes
mtekman Jun 17, 2023
33d5a83
tabify changes
mtekman Jun 19, 2023
99c3510
simplified lemmy update script
mtekman Jun 21, 2023
12f7053
Update dietpi/dietpi-software
mtekman Jun 22, 2023
1848bc1
Update dietpi/dietpi-software
mtekman Jun 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
242 changes: 242 additions & 0 deletions dietpi/dietpi-software
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1042,6 +1042,13 @@ Available commands:
aSOFTWARE_CATX[$software_id]=6
aSOFTWARE_DEPS[$software_id]='17'
aSOFTWARE_DOCS[$software_id]='https://dietpi.com/docs/software/social/#microblogpub'
#-----------------
software_id=210
aSOFTWARE_NAME[$software_id]='lemmy'
aSOFTWARE_DESC[$software_id]='A link aggregator for the fediverse'
aSOFTWARE_CATX[$software_id]=6
aSOFTWARE_DEPS[$software_id]='9 17 194'
aSOFTWARE_DOCS[$software_id]='https://dietpi.com/docs/software/social/#lemmy'

# Camera & Surveillance
#--------------------------------------------------------------------------------
Expand Down Expand Up @@ -4332,6 +4339,223 @@ _EOF_
G_DIETPI-NOTIFY 0 'microblog.pub installation finished. In order to complete the setup, please run "microblog-pub configure" in a new bash session. Append the port ":8007" to the domain when being asked for it if you do not use a reverse proxy.'
fi

if To_Install 210 # lemmy
then
aDEPS=('make' 'gcc' 'libc6-dev' 'pkg-config' 'libssl-dev' 'protobuf-compiler' 'libprotobuf-dev' 'libpq-dev')

# This script creates the lemmy user, a dedicated postgres
# db in the dietpi userspace, the lemmy backend, the lemmy
# frontend, and apparently the pict-rs image server too
# (that last one is NOT clear).

## Install up-to-date rustup (cargo version in debian is too old)
local lemm_user='lemmy'
local lemm_home="/home/$lemm_user"
local lemm_server_path=$lemm_home/.cargo/bin
# User
##Create_User -G dialout,gpio,i2c -d "$lemm_home" "$lemm_user"
G_EXEC mkdir -p "$lemm_home"
Create_User -d "$lemm_home" "$lemm_user"
Comment on lines +4357 to +4358
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this user require a home directory in /home? For service users it is common to use the software data directory instead, i.e. /mnt/dietpi_userdata/lemmy.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is where I become unsure. I thought userdata was purely for data and configs generated by the user, but not the binaries themselves.

The lemm_server_path seems to install by default to /home/lemmy/.cargo/bin/, so I figured I would leave it there since it's not data. I did initially try installing to /usr/bin and then setting a sudoers file for lemmy to run it with no password, but I encountered some problems with that (something to do with cargo and rust environment issues).

What's the recommended course of action here?

Copy link
Owner

@MichaIng MichaIng Jun 24, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, yeah I see. I'd prefer /opt/lemmy for binaries. /home should at best only be used for actual login users, not service users.

In this case, I actually wonder whether Rust really needs to be/should be installed as lemmy user or better just as root and installed to /opt/lemmy? Since we install Rust for root in some other cases, it might reduce some overhead, at least some disk usage. And usually it is not needed and for security reasons even better if the service user has no write access to the binary/executable, but only to a dedicated working directory for configs and data.

I did initially try installing to /usr/bin and then setting a sudoers file for lemmy to run it with no password

Executables are usually word-executable, so every user can execute every command without sudo, or do I misunderstand?

G_EXEC chown "$lemm_user":"$lemm_user" -R "$lemm_home"

##G_EXEC sudo su -u $lemm_user mkdir -p $lemm_bin

## Install Rust, cargo
## We assume users will want to upload images, so we enable pictrs
G_DIETPI-NOTIFY 0 '[ INFO ] lemmy backend installation started'
mtekman marked this conversation as resolved.
Show resolved Hide resolved
G_EXEC cd "$lemm_home"
G_EXEC sudo -u lemmy dash -c '\
curl -sSf "https://sh.rustup.rs/" -o rustup-init.sh && \
chmod +x rustup-init.sh && \
./rustup-init.sh -y --profile minimal && \
rm rustup-init.sh; \
. /home/lemmy/.cargo/env
cargo install lemmy_server --locked --features embed-pictrs'
G_DIETPI-NOTIFY 0 '[ INFO ] lemmy backend installation finished'

## Install lemmy front-end server
## Deps
local url='https://dl.yarnpkg.com/debian/pubkey.gpg'
G_CHECK_URL "$url"
G_EXEC eval "curl -sSfL '$url' | gpg --dearmor -o /etc/apt/trusted.gpg.d/dietpi-lemmy.gpg --yes"
G_EXEC eval "echo 'deb https://dl.yarnpkg.com/debian/ stable main' > /etc/apt/sources.list.d/dietpi-yarn.list"
G_AGUP
G_AGI yarn nodejs
# ## Installation
if ! [[ -e /var/lib/lemmy-ui ]]; then
G_EXEC mkdir -p /var/lib/lemmy-ui
G_EXEC chown "$lemm_user":"$lemm_user" -R /var/lib/lemmy-ui
# dont compile as admin
G_EXEC cd /var/lib/lemmy-ui
G_DIETPI-NOTIFY 0 '[ INFO ] lemmy UI installation started'
G_EXEC sudo -u "$lemm_user" dash -c 'git clone https://github.com/LemmyNet/lemmy-ui.git --recursive .'
G_EXEC sudo -u "$lemm_user" dash -c '\
git checkout 0.17.3 && \
yarn install --pure-lockfile && \
yarn build:prod'
fi
G_DIETPI-NOTIFY 0 '[ INFO ] lemmy UI installation finished'

# Config Section
local host_bind=127.0.0.1
local host_broadcast=0.0.0.0
local host_domain=dietpi.lemmy.com
local pass_database="dblemmypassword"
local port_database=5432
local port_backend=5433 ## default:8536
local port_pictrs=5434
local port_frontend=5435 ## default:1234

local lemmy_userdata=/mnt/dietpi_userdata/lemmy

G_DIETPI-NOTIFY 0 "[ INFO ] lemmy will be up and running at https://$host_broadcast:$port_frontend"

# Prepare our new config + data directory if not yet present
if [[ ! -f "$lemmy_userdata/lemmy.hjson" ]]
then
[[ -d "$lemmy_userdata" ]] || G_EXEC mkdir "$lemmy_userdata"
fi

cat << _EOF_ > "$lemmy_userdata/lemmy.hjson"
# https://join-lemmy.org/docs/en/administration/configuration.html#full-config-with-default-values
{
database: {
# For some reason, host needs to be blank.
host: ""
password: "$pass_database"
port: $port_database
}
# replace with your domain
hostname: "$host_domain"
bind: "$host_bind"
port: $port_backend
federation: {
enabled: true
}
# remove this block if you don't require image hosting
pictrs: {
url: "https://$host_bind:$port_pictrs/"
api_key: "$pass_database"
}
setup: {
admin_username: "dietpi"
# Password for the admin user. It must be at least 10 characters.
admin_password: "dietpi-lemmy"
site_name: "My DietPi Lemmy Instance"
}
## Disable this if not a federated instance
tls_enabled: true
}
_EOF_
## Backend Service
# Pictrs options: https://git.asonix.dog/asonix/pict-rs/
G_EXEC mkdir -p "$lemmy_userdata/pictrs/sled-repo"
G_EXEC mkdir -p "$lemmy_userdata/pictrs/store"
cat << _EOF_ > /etc/systemd/system/lemmy.service
[Unit]
Description=Lemmy - A link aggregator for the fediverse
After=network.target

[Service]
User=lemmy
ExecStart=$lemm_server_path/lemmy_server
Environment=LEMMY_CONFIG_LOCATION="$lemmy_userdata/lemmy.hjson"
Environment=PICTRS_PATH=$lemmy_userdata/pictrs
Environment=PICTRS__REPO__PATH=$lemmy_userdata/pictrs/sled-repo
Environment=PICTRS__STORE__PATH=$lemmy_userdata/pictrs/store
Environment=PICTRS_ADDR=$host_bind:$port_pictrs
Restart=on-failure

# Hardening
ProtectSystem=yes
PrivateTmp=true
MemoryDenyWriteExecute=true
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target
_EOF_
## UI Service - https://github.com/LemmyNet/lemmy-ui#configuration
cat << _EOF_ > /etc/systemd/system/lemmy-ui.service
[Unit]
Description=Lemmy UI - Web frontend for Lemmy
After=lemmy.service
##Before=nginx.service

[Service]
User=lemmy
WorkingDirectory=/var/lib/lemmy-ui
ExecStart=/usr/bin/node dist/js/server.js
Environment=LEMMY_UI_HOST=$host_broadcast:$port_frontend
Environment=LEMMY_UI_LEMMY_INTERNAL_HOST=$host_bind:$port_backend
Environment=LEMMY_UI_LEMMY_EXTERNAL_HOST="$host_domain"
Environment=LEMMY_UI_EXTRA_THEMES_FOLDER="$lemmy_userdata/extra-themes"
Environment=LEMMY_UI_HTTPS=true ## Hey, should this be enforced?
Restart=on-failure

# Hardening
ProtectSystem=full
PrivateTmp=true
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target
_EOF_
## TODO: Upgrade Scripts. FIX THIS SO THAT IT WORKS WITH THE GLOBAL CARGO ENV
cat << _EOF_ > /etc/bashrc.d/lemmy.sh
#!/bin/dash
lemmy-update()
{
case "\$1" in
l*) ## Lemmy
sudo -u '$lemm_user' dash -c '. "$lemm_home/.cargo/env"; cargo install lemmy_server --locked --features embed-pictrs'
sudo systemctl restart lemmy
;;
ui) ## UI
sudo -u '$lemm_user' dash -c '\
. "$lemm_home/.cargo/env"; \
cd /var/lib/lemmy-ui; \
git checkout main && \
git pull --tags && \
git checkout main && \
git submodule update && \
yarn install --pure-lockfile && \
yarn build:prod;';
sudo systemctl restart lemmy-ui
;;
*) echo 'lemmy-update helper script

usage: lemmy-update <lemmy|ui>

where:
lemmy update and rebuild the backend server (via cargo)
ui update and rebuild the ui (via yarn)

branch defaults to 'main'
return 1;;
esac
}
_EOF_

# Postgres
## We need to tell postgres that we're doing password auth (md5) and not peer.
G_EXEC sed -i -r '/^local\s+all\s+all\s+peer/i local lemmy lemmy md5' /etc/postgresql/15/main/pg_hba.conf
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. The PostgreSQL version depends on the Debian version: https://packages.debian.org/postgresql
  2. However, it shouldn't be required to alter any config file for this. Something like this should create a user with password authentication:
if [[ $(sudo -u lemmy psql -tAc "SELECT 1 FROM pg_roles WHERE rolname='lemmy'") != 1 ]]
then
	G_EXEC sudo -u lemmy psql -c "CREATE ROLE synapse WITH LOGIN PASSWORD '$pass_database';"
else
	G_EXEC sudo -u lemmy psql -c "ALTER ROLE synapse WITH PASSWORD '$pass_database';"
fi

I think/hope MD5 is deprecated and not the default how PostgreSQL is hashing passwords. MD5 is okay for integrity checks only, not secure at all since decades 😉.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm! I tried this at first, but could not login to the database. I'll try it again just in case I overlooked something

G_EXEC systemctl restart postgresql

local db_files="$lemmy_userdata/postgres"
G_EXEC mkdir -p "$db_files"
Comment on lines +4545 to +4546
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd just leave the database at the default PostgreSQL database directory. This is what what most admins will expect. SQLite is the only database engine where it is common to store databases within the client software data directory.

G_EXEC chown postgres:postgres -R "$db_files"
G_EXEC_OUTPUT=1 G_EXEC sudo -iu postgres psql -c "DROP DATABASE IF EXISTS lemmy;"
G_EXEC_OUTPUT=1 G_EXEC sudo -iu postgres psql -c "DROP TABLESPACE IF EXISTS lemmydata;"
G_EXEC_OUTPUT=1 G_EXEC sudo -iu postgres psql -c "DROP ROLE IF EXISTS lemmy;"
Comment on lines +4548 to +4550
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will drop all info on reinstalls, which is not intended, is it? In addition to above conditional database user creation, here is how we do in case of Synapse for the database itself:

if [[ $(sudo -u lemmy psql -tAc "SELECT 1 FROM pg_database WHERE datname='lemmy'") != 1 ]]
then
	G_EXEC sudo -u lemmy createdb --encoding=UTF8 --locale=C --template=template0 --owner=lemmy lemmy
fi

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep sorry, leftover from testing

G_EXEC_OUTPUT=1 G_EXEC sudo -iu postgres psql -c "CREATE USER lemmy WITH PASSWORD '""$pass_database""';"
G_EXEC_OUTPUT=1 G_EXEC sudo -iu postgres psql -c "CREATE TABLESPACE lemmydata OWNER lemmy LOCATION '""$db_files""';"
G_EXEC_OUTPUT=1 G_EXEC sudo -iu postgres psql -c "CREATE DATABASE lemmy WITH OWNER lemmy ENCODING 'UTF8' TEMPLATE template0 LC_COLLATE 'C' LC_CTYPE 'C' TABLESPACE='lemmydata';"
G_EXEC_OUTPUT=1 G_EXEC sudo -iu postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE lemmy TO lemmy;"
G_EXEC_OUTPUT=1 G_EXEC sudo -iu postgres psql -c "ALTER USER lemmy WITH SUPERUSER;"
G_EXEC systemctl restart postgresql
fi

if To_Install 2 fahclient # Folding@Home
then
G_DIETPI-NOTIFY 2 'Pre-configuring FAHClient DEB package'
Expand Down Expand Up @@ -13041,6 +13265,24 @@ If no WireGuard (auto)start is included, but you require it, please do the follo
G_DIETPI-NOTIFY 2 'All microblog.pub data and settings are still retained.\n - You can remove these manually: rm -R /mnt/dietpi_userdata/microblog-pub'
fi

if To_Uninstall 210 # lemmy
then
Remove_Service lemmy-ui 1 1
Remove_Service lemmy 1 1
## Remove_Service postgres ??
G_EXEC userdel lemmy
mtekman marked this conversation as resolved.
Show resolved Hide resolved
G_EXEC_OUTPUT=1 G_EXEC sudo -iu postgres psql -c "DROP DATABASE IF EXISTS lemmy;"
G_EXEC_OUTPUT=1 G_EXEC sudo -iu postgres psql -c "DROP TABLESPACE IF EXISTS lemmydata;"
G_EXEC_OUTPUT=1 G_EXEC sudo -iu postgres psql -c "DROP ROLE IF EXISTS lemmy;"
## Remove Lemmy config line
[[ -f '/etc/postgresql/15/main/pg_hba.conf' ]] && G_EXEC sed -i -r '/^local\s+lemmy\s+lemmy\s+md5/d' /etc/postgresql/15/main/pg_hba.conf
[[ -d '/var/lib/lemmy-ui' ]] && G_EXEC rm -rf /var/lib/lemmy-ui
[[ -d '/home/lemmy' ]] && G_EXEC rm -rf /home/lemmy
[[ -f '/etc/systemd/system/lemmy.service' ]] && G_EXEC rm /etc/systemd/system/lemmy.service
[[ -f '/etc/systemd/system/lemmy-ui.service' ]] && G_EXEC rm /etc/systemd/system/lemmy-ui.service
[[ -f '/etc/bashrc.d/lemmy.sh' ]] && G_EXEC rm /etc/bashrc.d/lemmy.sh
fi

if To_Uninstall 98 # HAProxy
then
Remove_Service haproxy
Expand Down