Merge pull request #37 from Podcastindex-org/feature/1.1/#36_default_…

…medium_reason_for_server

Feature/1.1/#36 default medium reason for server

CLI.md

@@ -8,6 +8,8 @@ $ podping [OPTIONS] COMMAND [ARGS]...
 
 **Options**:
 
+* `--medium TEXT`: The medium of the feed being updated. If used in combination with the 'server', this sets the default medium only. Must be one of the following: newsletter film audiobook podcast music blog video  [env var: PODPING_MEDIUM; default: podcast]
+* `--reason TEXT`: The reason the feed is being updated. If used in combination with the 'server', this sets the default reason only. Must be one of the following: update live  [env var: PODPING_REASON; default: update]
 * `--hive-account TEXT`: Hive account used to post  [env var: PODPING_HIVE_ACCOUNT, HIVE_ACCOUNT, HIVE_SERVER_ACCOUNT; required]
 * `--hive-posting-key TEXT`: Hive account used to post  [env var: PODPING_HIVE_POSTING_KEY, HIVE_POSTING_KEY; required]
 * `--sanity-check / --no-sanity-check`: By default, podping will test for available resources and the ability to post to the Hive chain on the given hive account at startup by posting startup information. Disabling this will result in a faster startup, time, but may result in unexpected errors.  [env var: PODPING_SANITY_CHECK; default: True]
@@ -108,6 +110,4 @@ $ podping write [OPTIONS] IRI...
 
 **Options**:
 
-* `--medium TEXT`: The medium of the feed being updated. Must be one of the following: audiobook music podcast video newsletter blog film  [env var: PODPING_MEDIUM; default: podcast]
-* `--reason TEXT`: The reason the feed is being updated. Must be one of the following: update live  [env var: PODPING_REASON; default: update]
 * `--help`: Show this message and exit.

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "podping-hivewriter"
-version = "1.1.0-beta.2"
+version = "1.1.0-beta.3"
 license = "MIT"
 authors = ["Alecks Gates <[email protected]>", "Brian of London <[email protected]>"]
 maintainers = ["Alecks Gates <[email protected]>", "Brian of London <[email protected]>"]

setup.py

@@ -29,7 +29,7 @@
 
 setup_kwargs = {
     "name": "podping-hivewriter",
-    "version": "1.1.0-beta.2",
+    "version": "1.1.0-beta.3",
     "description": "This is a tool used to submit RFC 3987-compliant International Resource Identifiers as a Podping notification on the Hive blockchain.",
     "long_description": "# podping-hivewriter\nThe Hive writer component of podping. You will need a Hive account, see section [Hive account and Authorization](#hive-account) below.\n\n## CLI Install\n\nThe following have been tested on Linux and macOS.  However, Windows should work also.  If you have issues on Windows we highly recommend the [Windows Subsystem for Linux](https://docs.microsoft.com/en-us/windows/wsl/) and/or Docker.\n\n### Using [pipx](https://pypa.github.io/pipx/) (preferred over pip)\n```shell\npipx install podping-hivewriter\n```\n\n### Using pip\n```shell\npip install --user podping-hivewriter\n```\n\n### Installing the server\n\nIf you'd like to install the server component, it's hidden behind the extra flag `server`.  This is to make it easier to install only the `write` CLI component `podping-hivewriter` on non-standard systems without a configured development enviornment.\n\n```shell\npipx install podping-hivewriter[server]\n```\n\nMake sure you have `~/.local/bin/` on your `PATH`.\n\nSee the dedicated [CLI docs](CLI.md) for more information.\n\n## Container\n\nThe container images are hosted on [Docker Hub](https://hub.docker.com/r/podcastindexorg/podping-hivewriter).  Images are currently based on Debian bullseye-based Python 3.9 with the following architectures: `amd64`, `i386`, `arm64`, `armv7`, `armv6`\n\n### docker-compose\n\n```yaml\nversion: '2.0'\nservices:\n  podping-hivewriter:\n    image: podcastindexorg/podping-hivewriter\n    restart: always\n    ports:\n      - \"9999:9999\"\n    environment:\n      - PODPING_HIVE_ACCOUNT=<account>\n      - PODPING_HIVE_POSTING_KEY=<posting-key>\n      - PODPING_LISTEN_IP=0.0.0.0\n      - PODPING_LISTEN_PORT=9999\n      - PODPING_LIVETEST=false\n      - PODPING_DRY_RUN=false\n      - PODPING_STATUS=true\n      - PODPING_IGNORE_CONFIG_UPDATES=false\n      - PODPING_I_KNOW_WHAT_IM_DOING=false\n      - PODPING_DEBUG=false\n```\n\nAssuming you just copy-pasted without reading, the above will fail at first.  As noted in the [server command documentation](https://github.com/Podcastindex-org/podping-hivewriter/blob/main/CLI.md#podping-server):\n\n>WARNING: DO NOT run this on a publicly accessible host. There currently is NO authentication required to submit to the server. Set to * or 0.0.0.0 for all interfaces.\n\nAs all Docker installations vary, we set `0.0.0.0` as the listen IP for connectivity.  This doesn't affect the IP address docker listens on when we tell it to pass port `9999` through to the container.  If you understand the consequences of this, set `PODPING_I_KNOW_WHAT_IM_DOING` to `true`.\n\n### Building the image with Docker\n\nLocally build the podping-hivewriter container with a \"develop\" tag\n\n```shell\ndocker build -t podping-hivewriter:develop .\n```\n\n\n### Running the image\n\nRun the locally built image in a container, passing local port 9999 to port 9999 in the container.\nENV variables can be passed to docker with `--env-file` option after modifying the `.env.EXAMPLE` file and renaming it to `.env`\n\n```shell\ndocker run --rm -p 9999:9999 --env-file .env --name podping podping-hivewriter:develop\n```\n\nRunning with command line options, like `--dry-run` for example, add them with the full podping command.\nSettings can also be passed with the `-e` option for Docker.  Note, we leave out `-p 9999:9999` here because we're not running the server.\n\n```shell\ndocker run --rm \\\n    -e PODPING_HIVE_ACCOUNT=<account> \\\n    -e PODPING_HIVE_POSTING_KEY=<posting-key> \\\n    podping-hivewriter:develop \\\n    podping --dry-run write https://www.example.com/feed.xml\n```\n\nAs another example for running a server, to run in *detached* mode, note the `-d` in the `docker run` options. Also note that `client` or `server` must come *after* the command line options for `podping`:\n```shell\ndocker run --rm -d \\\n    -p 9999:9999 --env-file .env \\\n    --name podping podping-hivewriter:develop \\\n    podping --livetest server\n```\n\nOne running you can view and follow the live output with:\n```shell\ndocker logs podping -f\n```\n\nSee the [CLI docs](https://github.com/Podcastindex-org/podping-hivewriter/blob/main/CLI.md) for default values.\n\n## Development\n\nYou'll need a few extras:\n\n1. [capnproto](https://capnproto.org/). On a Mac: `brew instal capnp`\n2. [Poetry](https://python-poetry.org/docs/)\n\n\nWe use [poetry](https://python-poetry.org/) for dependency management.  Once you have it, clone this repo and run:\n\n```shell\npoetry install\n```\n\nThen to switch to the virtual environment, use:\n\n```shell\npoetry shell\n```\nMake sure you have a the environment variables `PODPING_HIVE_ACCOUNT` and `PODPING_HIVE_POSTING_KEY` set.\n\nAfter that you should be able to run the `podping` command or run the tests:\n\n```shell\npytest\n```\n\nTo run all tests, make sure to set the necessary environment variables for your Hive account.  This can take many minutes:\n\n```shell\npytest --runslow\n```\n\n## Hive account\n\nIf you need a Hive account, please download the [Hive Keychain extension for your browser](https://hive-keychain.com/) then use this link to get your account from [https://HiveOnboard.com?ref=podping](https://hiveonboard.com?ref=podping). You will need at least 20 Hive Power \"powered up\" to get started (worth around $10). Please contact [@brianoflondon](https://peakd.com/@brianoflondon) [email protected] if you need assistance getting set up.\n\nIf you use the [Hiveonboard]((https://hiveonboard.com?ref=podping)) link `podping` will **delegate** enough Hive Power to get you started.\n\n### Permissions and Authorization\n\nYou don't need permission, but you do need to tell `podping` that you want to send valid `podpings`:\n\n- Hive is a so-called \"permissionless\" blockchain.  Once you have a Hive Account and a minimal amount of Hive Power, that account can post to Hive, including sending `podpings`.\n\n- Nobody can block any valid Hive Account from sending and nobody can help you if you lose your keys.\n\n- Whilst anyone can post `podpings` to Hive, there is a need to register your Hive Accountname for those `podpings` to be recognized by all clients.  This is merely a spam-prevention measure and clients may choose to ignore it.\n\n- Please contact [email protected] or send a Hive Transfer to [@podping](https://peakd.com/@podping) to have your account validated.\n\n- Side note on keys: `podping` uses the `posting-key` which is the lowest value of the four Hive keys (`owner`, `active`, `memo`, `posting` and there is usually a `master password` which can generate all the keys). That is not to say that losing control of it is a good idea, but that key is not authorized to make financially important transfers. It can, however, post public information so should be treated carefully and kept secure.\n\nFor a [comprehensive explanation of Hive and Podping, please see this post](https://peakd.com/podping/@brianoflondon/podping-and-podcasting-20-funding-to-put-hive-at-the-center-of-global-podcasting-infrastructure).",
     "author": "Alecks Gates",

src/podping_hivewriter/cli/podping.py

@@ -71,6 +71,8 @@ def version_callback(value: bool):
 class Config:
     hive_account: str
     hive_posting_key: str
+    medium: Medium
+    reason: Reason
     sanity_check: bool
     livetest: bool
     dry_run: bool
@@ -88,20 +90,6 @@ def exit_cli(_):
 
 @app.command()
 def write(
-    medium: str = typer.Option(
-        str(Medium.podcast),
-        envvar=["PODPING_MEDIUM"],
-        callback=medium_callback,
-        autocompletion=lambda: list(mediums),
-        help=f"The medium of the feed being updated. Must be one of the following: {str(' '.join(mediums))}",
-    ),
-    reason: str = typer.Option(
-        str(Reason.update),
-        envvar=["PODPING_REASON"],
-        callback=reason_callback,
-        autocompletion=lambda: list(reasons),
-        help=f"The reason the feed is being updated. Must be one of the following: {str(' '.join(reasons))}",
-    ),
     iris: List[str] = typer.Argument(
         ...,
         metavar="IRI...",
@@ -147,13 +135,15 @@ def write(
         Config.hive_account,
         [Config.hive_posting_key],
         settings_manager,
+        medium=Config.medium,
+        reason=Config.reason,
         operation_id=Config.operation_id,
         resource_test=Config.sanity_check,
         daemon=False,
         dry_run=Config.dry_run,
     ) as podping_hivewriter:
         coro = podping_hivewriter.failure_retry(
-            set(iris), medium=str_medium_map[medium], reason=str_reason_map[reason]
+            set(iris), medium=Config.medium, reason=Config.reason
         )
         try:
             # Try to get an existing loop in case of running from other program
@@ -238,6 +228,8 @@ def server(
         Config.hive_account,
         [Config.hive_posting_key],
         settings_manager,
+        medium=Config.medium,
+        reason=Config.reason,
         listen_ip=listen_ip,
         listen_port=listen_port,
         operation_id=Config.operation_id,
@@ -261,6 +253,22 @@ def server(
 
 @app.callback()
 def callback(
+    medium: str = typer.Option(
+        str(Medium.podcast),
+        envvar=["PODPING_MEDIUM"],
+        callback=medium_callback,
+        autocompletion=lambda: list(mediums),
+        help=f"The medium of the feed being updated. If used in combination with the 'server', this sets the default "
+        f"medium only. Must be one of the following: {str(' '.join(mediums))}",
+    ),
+    reason: str = typer.Option(
+        str(Reason.update),
+        envvar=["PODPING_REASON"],
+        callback=reason_callback,
+        autocompletion=lambda: list(reasons),
+        help=f"The reason the feed is being updated. If used in combination with the 'server', this sets the default "
+        f"reason only. Must be one of the following: {str(' '.join(reasons))}",
+    ),
     hive_account: str = typer.Option(
         ...,
         envvar=["PODPING_HIVE_ACCOUNT", "HIVE_ACCOUNT", "HIVE_SERVER_ACCOUNT"],
@@ -325,6 +333,8 @@ def callback(
 ):
     Config.hive_account = hive_account
     Config.hive_posting_key = hive_posting_key
+    Config.medium = str_medium_map[medium]
+    Config.reason = str_reason_map[reason]
     Config.sanity_check = sanity_check
     Config.livetest = livetest
     Config.dry_run = dry_run

src/podping_hivewriter/podping_hivewriter.py

@@ -7,7 +7,7 @@
 from datetime import datetime, timedelta
 from itertools import cycle
 from timeit import default_timer as timer
-from typing import List, Set, Tuple, Union
+from typing import List, Set, Tuple, Union, Optional
 
 import rfc3987
 from lighthive.datastructures import Operation
@@ -42,6 +42,8 @@ def __init__(
         server_account: str,
         posting_keys: List[str],
         settings_manager: PodpingSettingsManager,
+        medium: Medium = Medium.podcast,
+        reason: Reason = Reason.update,
         listen_ip: str = "127.0.0.1",
         listen_port: int = 9999,
         operation_id="pp",
@@ -55,6 +57,8 @@ def __init__(
         self.server_account: str = server_account
         self.required_posting_auths = [self.server_account]
         self.settings_manager = settings_manager
+        self.medium = medium
+        self.reason = reason
         self.listen_ip = listen_ip
         self.listen_port = listen_port
         self.posting_keys: List[str] = posting_keys
@@ -222,7 +226,9 @@ async def _iri_batch_handler_loop(self):
                 iri_batch = await self.iri_batch_queue.get()
 
                 start = timer()
-                failure_count = await self.failure_retry(iri_batch.iri_set)
+                failure_count = await self.failure_retry(
+                    iri_batch.iri_set, medium=self.medium, reason=self.reason
+                )
                 duration = timer() - start
 
                 self.iri_batch_queue.task_done()
@@ -413,10 +419,12 @@ async def send_notification(
     async def send_notification_iri(
         self,
         iri: str,
-        medium: Medium = Medium.podcast,
-        reason: Reason = Reason.update,
+        medium: Optional[Medium],
+        reason: Optional[Reason],
     ) -> None:
-        payload = Podping(medium=medium, reason=reason, iris=[iri])
+        payload = Podping(
+            medium=medium or self.medium, reason=reason or self.reason, iris=[iri]
+        )
 
         hive_operation_id = HiveOperationId(self.operation_id, medium, reason)
 
@@ -427,11 +435,13 @@ async def send_notification_iri(
     async def send_notification_iris(
         self,
         iris: Set[str],
-        medium: Medium = Medium.podcast,
-        reason: Reason = Reason.update,
+        medium: Optional[Medium],
+        reason: Optional[Reason],
     ) -> None:
         num_iris = len(iris)
-        payload = Podping(medium=medium, reason=reason, iris=list(iris))
+        payload = Podping(
+            medium=medium or self.medium, reason=reason or self.reason, iris=list(iris)
+        )
 
         hive_operation_id = HiveOperationId(self.operation_id, medium, reason)
 
@@ -442,8 +452,8 @@ async def send_notification_iris(
     async def failure_retry(
         self,
         iri_set: Set[str],
-        medium: Medium = Medium.podcast,
-        reason: Reason = Reason.update,
+        medium: Optional[Medium],
+        reason: Optional[Reason],
     ) -> int:
         await self.wait_startup()
         failure_count = 0
@@ -462,7 +472,9 @@ async def failure_retry(
 
             try:
                 await self.send_notification_iris(
-                    iris=iri_set, medium=medium, reason=reason
+                    iris=iri_set,
+                    medium=medium or self.medium,
+                    reason=reason or self.reason,
                 )
                 if failure_count > 0:
                     logging.info(

tests/integration/test_write_cli_multiple.py

@@ -1,5 +1,6 @@
 import asyncio
 import json
+import random
 import uuid
 from random import randint
 from platform import python_version as pv
@@ -13,8 +14,8 @@
 from podping_hivewriter.cli.podping import app
 from podping_hivewriter.constants import LIVETEST_OPERATION_ID
 from podping_hivewriter.models.hive_operation_id import HiveOperationId
-from podping_hivewriter.models.medium import Medium
-from podping_hivewriter.models.reason import Reason
+from podping_hivewriter.models.medium import Medium, str_medium_map, mediums
+from podping_hivewriter.models.reason import Reason, str_reason_map, reasons
 from podping_hivewriter.podping_settings_manager import PodpingSettingsManager
 
 
@@ -39,9 +40,10 @@ async def test_write_cli_multiple():
         for i in range(num_iris)
     }
 
-    default_hive_operation_id = HiveOperationId(
-        LIVETEST_OPERATION_ID, Medium.podcast, Reason.update
-    )
+    medium = str_medium_map[random.sample(mediums, 1)[0]]
+    reason = str_reason_map[random.sample(reasons, 1)[0]]
+
+    default_hive_operation_id = HiveOperationId(LIVETEST_OPERATION_ID, medium, reason)
     default_hive_operation_id_str = str(default_hive_operation_id)
 
     async def get_iri_from_blockchain(start_block: int):
@@ -58,6 +60,10 @@ async def get_iri_from_blockchain(start_block: int):
                         yield iri
 
     args = [
+        "--medium",
+        str(medium),
+        "--reason",
+        str(reason),
         "--livetest",
         "--no-sanity-check",
         "--ignore-config-updates",

tests/integration/test_write_cli_single.py

@@ -1,5 +1,6 @@
 import asyncio
 import json
+import random
 import uuid
 from platform import python_version as pv
 
@@ -12,8 +13,8 @@
 from podping_hivewriter.cli.podping import app
 from podping_hivewriter.constants import LIVETEST_OPERATION_ID
 from podping_hivewriter.models.hive_operation_id import HiveOperationId
-from podping_hivewriter.models.medium import Medium
-from podping_hivewriter.models.reason import Reason
+from podping_hivewriter.models.medium import Medium, str_medium_map, mediums
+from podping_hivewriter.models.reason import Reason, str_reason_map, reasons
 from podping_hivewriter.podping_settings_manager import PodpingSettingsManager
 
 
@@ -33,9 +34,10 @@ async def test_write_cli_single():
     test_name = "cli_single"
     iri = f"https://example.com?t={test_name}&v={pv()}&s={session_uuid_str}"
 
-    default_hive_operation_id = HiveOperationId(
-        LIVETEST_OPERATION_ID, Medium.podcast, Reason.update
-    )
+    medium = str_medium_map[random.sample(mediums, 1)[0]]
+    reason = str_reason_map[random.sample(reasons, 1)[0]]
+
+    default_hive_operation_id = HiveOperationId(LIVETEST_OPERATION_ID, medium, reason)
     default_hive_operation_id_str = str(default_hive_operation_id)
 
     async def get_iri_from_blockchain(start_block: int):
@@ -46,9 +48,22 @@ async def get_iri_from_blockchain(start_block: int):
         ):
             data = json.loads(post["op"][1]["json"])
             if "iris" in data and len(data["iris"]) == 1:
-                yield data["iris"][0]
-
-    args = ["--livetest", "--no-sanity-check", "--ignore-config-updates", "write", iri]
+                iri = data["iris"][0]
+                # Only look for IRIs from current session
+                if iri.endswith(session_uuid_str):
+                    yield data["iris"][0]
+
+    args = [
+        "--medium",
+        str(medium),
+        "--reason",
+        str(reason),
+        "--livetest",
+        "--no-sanity-check",
+        "--ignore-config-updates",
+        "write",
+        iri,
+    ]
 
     current_block = client.get_dynamic_global_properties()["head_block_number"]