Improved `deno run --watch` support, watch support inside Docker #79

daniel-chambers · 2023-11-28T03:04:54Z

Description

This PR improves support for running the connector using deno run --watch where the connector gets restarted by Deno whenever the function files change. It adds support for running the connector in watch mode using Docker.

docker build . -t "typescript-deno:latest" # only because we don't publish a docker container anywhere
docker run -it --rm -p 8080:8080 -e WATCH=1 -v ./functions:/functions/src typescript-deno:latest

Part of the improvements to make watch mode work better include

rearranging when we infer the functions schema to reduce work done, but also reduce watch-induced reloads caused by vendoring during inference
rearranging the dockerfile and entrypoint script so that the generated vendor, node_modules and schema.json artifacts are stored outside of the functions source directory, so that we don't pollute the user's volume mount

Various version updates have also been made:

Version of deno used in Docker has been updated
Deno github action has been updated
the deno std library version has been updated and made consistent across the entire project

Solution Design

We now have a difference between RawConfiguration and Configuration. We perform our inference only in validate_raw_configuration and capture the results in the Configuration. This then powers every usage of the inference, including the schema endpoint and actual queries. It also provides a central place for configuration defaulting (such as defaulting to "INFER").

The entrypoint script now looks for a WATCH env var and if it is "1" or "true" it generates a different configuration that enables INFER mode and turns on deno's --watch flag.

Fixes: #26

Watch mode restarts in a loop

sordina · 2023-11-28T03:53:07Z

sordina

Awesome work and great cleanup with the validate phase.

I'd say the only change I'd like is to split the entrypoint script, since we're adding more logic into here with watch.

Also have a few comments and questions inline.

sordina · 2023-11-28T03:56:20Z

README.md

@@ -123,7 +123,6 @@ Limitations:
 * All numbers are exported as `Float`s
 * Unrecognised types will become opaque scalars, for example: union types.
 * Optional object fields are not currently supported
-* Complex input types are supported by the connector, but are not supported in "commands" in Hasura3 projects


sordina · 2023-11-28T03:57:00Z

README.md

-* (Optionally) If you want your development vendor and inference resources to be used to speed up deployment, add the following to your `./config.json`:
-  ```json
-  {
-    "functions": "./functions/index.ts",
-    "vendor": "./functions/vendor",
-    "preVendor": true,
-    "schemaLocation": "./functions/schema.json",
-    "schemaMode": "INFER"
-  }
-  ```
-  * Make sure to .gitignore your computed `vendor` and `schema.json` files.


sordina · 2023-11-28T04:09:49Z

README.md

-  functions      (string): Location of your functions entrypoint (default: ./functions/index.ts)
+  functions      (string): Location of your functions entrypoint


Why no documented default for functions, but present for preVendor, schemaMode?

You can't default functions any more, you must specify it as something yourself, so there is no default. You will get an error if you leave it out or put in a blank string. In all the places where we control the locations of the functions (ie. in docker) we specify it. We specify it in all the documentation. We also generate a value in a blank configuration object.

Is there a reason for this? I thought it was an advantage minimise boilerplate in the config.

sordina · 2023-11-28T04:28:23Z

src/connector.ts

-    const defaults = {
-      preVendor: true,
+  validate_raw_configuration(configuration: RawConfiguration): Promise<Configuration> {
+    if (configuration.functions.trim() === "") {


This seems like an odd check.

You can't leave out functions any more. You must specify it.

sordina · 2023-11-28T04:32:11Z

Dockerfile


 # Pre-cache inference results and dependencies
-RUN EARLY_ENTRYPOINT_EXIT=true sh /app/entrypoint.sh
+RUN PRECACHE_ONLY=true /app/entrypoint.sh


We should probably just split the initial actions into a seperate script.

What, and call it from this script? It still needs to happen on container start for those volume-mounting their functions.

Call the build actions from the dockerfile directly or via a build script, then remove them from the entrypoint script

Oh right... yeah this would work on connector create, but not for people doing local docker... bummer

sordina · 2023-11-28T04:32:23Z

CHANGELOG.md

-Changes to be included in the next upcoming releaase.
+Changes to be included in the next upcoming release.


…re-vendoring anything

sordina · 2023-11-29T04:59:13Z

Should probably have a better error if the config is missing the functions:

> deno run -A --watch --check ~/hasura/ndc-typescript-deno/src/mod.ts serve --configuration config.json
Watcher Process started.
Check file:///Users/lyndon/hasura/ndc-typescript-deno/src/mod.ts
TypeError: Cannot read properties of undefined (reading 'trim')

daniel-chambers · 2023-11-29T05:01:13Z

Weird. I thought the SDK would prevent it from getting that far in the code :/ The types are a lie.

sordina · 2023-11-29T05:02:20Z

Fatal errors seem to kill the watch command in some instances:

Watcher File change detected! Restarting!
Inferring schema with map location /Users/lyndon/temp/temp-2023-11-29_14-33-24-rtYNSYAuhX/vendor
Vendoring dependencies: /Users/lyndon/bin/binaries/deno vendor --node-modules-dir --output /Users/lyndon/temp/temp-2023-11-29_14-33-24-rtYNSYAuhX/vendor --force /Users/lyndon/temp/temp-2023-11-29_14-33-24-rtYNSYAuhX/functions/index.ts
Couldn't find import map: /Users/lyndon/temp/temp-2023-11-29_14-33-24-rtYNSYAuhX/vendor/import_map.json
There were 1 diagnostic errors.
FATAL: /Users/lyndon/temp/temp-2023-11-29_14-33-24-rtYNSYAuhX/functions/index.ts (10,30): The return type of an async function or method must be the global Promise<T> type. Did you mean to write 'Promise<string>'?
Fatal errors: 1
Previous comand exited with error status: 1

sordina · 2023-11-29T13:32:11Z

There's not much point having the two versions:

programInfo
programInfoException

after making programInfo just propagate exceptions rather than exit.

sordina

Let's goooo!

Follow up items:

test direct invocation off deno.com
non-null coalescense
test Docker image from registry
test with new CLI tools
Tag new release
Add changelog entry for new tag
Notify channels of new release

daniel-chambers added 8 commits November 28, 2023 12:22

WIP

05e19ac

Watch mode restarts in a loop

Fix watch mode restarting in a loop

e11d256

Update versions, fix tests

077fa6f

Rename entrypoint env var

cd88128

Don't do inference twice during startup

bc9710e

Disallow blank functions setting

8c2953c

Update readme

d93b0af

Add port expose to dockerfile

365dce0

daniel-chambers self-assigned this Nov 28, 2023

daniel-chambers requested a review from sordina November 28, 2023 03:35

sordina reviewed Nov 28, 2023

View reviewed changes

daniel-chambers added 2 commits November 28, 2023 19:15

Don't include any functions in the docker image by default to avoid p…

60c66de

…re-vendoring anything

Move connector config to non-standard location

d80e412

daniel-chambers mentioned this pull request Nov 29, 2023

Updates for DDN Alpha and TypeScript Deno connector compatibility hasura/ndc-sendgrid-deno#1

Merged

Fix watch getting cancelled when inference fails

987a1ad

daniel-chambers added 2 commits November 30, 2023 10:04

Update SDK

791d594

Inline programInfo function

c9fdea1

sordina approved these changes Nov 30, 2023

View reviewed changes

daniel-chambers merged commit a686cf7 into main Nov 30, 2023
1 check passed

daniel-chambers deleted the daniel/deno-run-watch branch November 30, 2023 01:19

sordina mentioned this pull request Dec 15, 2023

Support nested field selection [+Engine] #26

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improved `deno run --watch` support, watch support inside Docker #79

Improved `deno run --watch` support, watch support inside Docker #79

daniel-chambers commented Nov 28, 2023 •

edited by sordina

Loading

sordina commented Nov 28, 2023 •

edited

Loading

sordina left a comment

sordina Nov 28, 2023

sordina Nov 28, 2023

sordina Nov 28, 2023

daniel-chambers Nov 28, 2023

sordina Nov 28, 2023

sordina Nov 28, 2023

daniel-chambers Nov 28, 2023

sordina Nov 28, 2023

daniel-chambers Nov 28, 2023

sordina Nov 28, 2023

sordina Nov 28, 2023

sordina Nov 28, 2023

sordina commented Nov 29, 2023

daniel-chambers commented Nov 29, 2023

sordina commented Nov 29, 2023

sordina commented Nov 29, 2023

sordina left a comment •

edited

Loading

		functions (string): Location of your functions entrypoint (default: ./functions/index.ts)
		functions (string): Location of your functions entrypoint

		Changes to be included in the next upcoming releaase.
		Changes to be included in the next upcoming release.

Improved deno run --watch support, watch support inside Docker #79

Improved deno run --watch support, watch support inside Docker #79

Conversation

daniel-chambers commented Nov 28, 2023 • edited by sordina Loading

Description

Solution Design

sordina commented Nov 28, 2023 • edited Loading

sordina left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sordina commented Nov 29, 2023

daniel-chambers commented Nov 29, 2023

sordina commented Nov 29, 2023

sordina commented Nov 29, 2023

sordina left a comment • edited Loading

Choose a reason for hiding this comment

Improved `deno run --watch` support, watch support inside Docker #79

Improved `deno run --watch` support, watch support inside Docker #79

daniel-chambers commented Nov 28, 2023 •

edited by sordina

Loading

sordina commented Nov 28, 2023 •

edited

Loading

sordina left a comment •

edited

Loading