-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Media contributor guide: add Gold drawing #122
base: main
Are you sure you want to change the base?
Changes from 1 commit
2bcd52c
2fefa89
f7ef194
9ff3739
3cbc37b
04a10b2
da6e9c1
c963663
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,178 @@ | ||
# Media Contributor Guide | ||
|
||
## How to add a new Gold Drawing | ||
|
||
In this tutorial, you will learn how to add a new Gold Drawing image to the library. | ||
|
||
You can watch the whole screen capture of this tutorial on [YouTube](https://youtu.be/Nf-ePEdA5xc). | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe I would include a two lines description of what a Gold Image is ("A HQ Image generated from the raw acquired, with no further processing but cropping, aligning and some small resizing") There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, in fact, we should add that descripción here and link this tutorial with that section. |
||
### 1. Generate filename | ||
|
||
We start with the original artwork: `00IWO.tif` (276,3 MB). It's a `tif` image. | ||
|
||
The first thing we need to do is to give it the right name: `000000-32.600.2.tif` | ||
|
||
Please read the [File Name Convention](./File_Naming_Convention.md) documention for more information. | ||
|
||
- `000001` is the artwork id. It is a sequential number manually assigned by the media contributor. | ||
- `32` is the `purpose code`. It means the image is a Gold image. | ||
- `600` is the code for `original` images. | ||
- `2` is the suffix for image files. We could also upload other file types like metadata (in `json` format). | ||
|
||
### 2. Place the image in the correct folder | ||
|
||
We copy the image into the data folder: `data/000001/32/`. We have a folder for all artifacts related to the same artwork (`data/000001`) and another one inside the first one for all `Gold` images (`32/`). | ||
|
||
### 3. Create a new issue and branch | ||
|
||
We follow a pull request process to add new images to the library. That means the media contributor has to create an issue and a new branch for the issue. Then, the new image has to be added to that branch (only the `dvc` files, not the binary file) and the image has to be uploaded to the `dvc` storage. Once we have all the information and data on GitHub and DVC, the media contributor has to create a pull request. | ||
|
||
The process to create a new issue is just the standard process for GitHub. You can use this template: | ||
|
||
- Issue title: "New Gold image: 000000-32.600.2.tif" | ||
- Issue description: whatever the media contributor considers relevant. | ||
- Label: `media` | ||
|
||
![Add Gold image - issue template](./images/media-contributor-guide/new-issue-template.png) | ||
|
||
We also need to create a new branch. The command to do it using git from the console is: | ||
|
||
```shell | ||
git checkout -b issue-106-add-gold-image-000000-32.600.2.tif | ||
``` | ||
|
||
Make sure you are in the `main` branch and you have the latest version. | ||
|
||
### 4. Add the new image locally | ||
|
||
We are using `dvc` to store the image. That means the actual image will be stored on the `dvc` storage and we will only add two files into the git repo. | ||
|
||
First, we use the `dvc` client to add the new image. We have already added the image in the rigth folder. The `dcv` command is: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Here the second time you mention dvc you have misspelled it. |
||
|
||
```shell | ||
dvc add data/000000/32/000000-32.600.2.tif | ||
``` | ||
|
||
That command will copy the image in the local `dvc` cache folder and it will generate two new files: | ||
|
||
```shell | ||
data/000000/32/000000-32.600.2.tif.dvc | ||
data/000000/32/.gitignore | ||
``` | ||
|
||
The first file is what we call the `dvc` pointer. It's just an small file that is used by `dvc` to track the file with git. This is the content for our example: | ||
|
||
```text | ||
outs: | ||
- md5: 32b0d05acaadf5e1d5f6b67641678b1c | ||
size: 273627048 | ||
path: 000000-32.600.2.tif | ||
``` | ||
|
||
The second file is only a `.gitignore` file to avoid git tracking the actual binary file. It only contains the name of the image file (`/000000-32.600.2.tif`). | ||
|
||
We need to commit the changes. We have to add them: | ||
|
||
```shell | ||
git add data/000000/32/000000-32.600.2.tif.dvc data/000000/32/.gitignore | ||
``` | ||
|
||
and then, commit: | ||
|
||
```shell | ||
git commit -m "[#106] Add Gold image: 000000-32.600.2.tif" | ||
``` | ||
|
||
The template for the commit message is: | ||
|
||
```text | ||
[#ISSUE_NUMBER] Add Gold image: GOLD_IMAGE_FILENAME | ||
``` | ||
|
||
At this point we have the image locally in our branch. We need to apply our changes remotelly: | ||
|
||
- Push our new git commit. | ||
- Copy the image to the remote `dcv` storage. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same misspelling of dvc. |
||
|
||
> TIP: you can list all the files handled with `dvc` in the library with this command: `dvc list --dvc-only -R data/`. | ||
|
||
### 5. Share the new image remotelly | ||
|
||
Before pushing our new branch we should push the image into the remote `dvc` storage, otherwise other people is going to have access to the new `dvc` pointer in the git repo but they will not be able to download the image. | ||
|
||
The way to share our local image is by sugin the `dvc` command [`push`](https://dvc.org/doc/command-reference/push): | ||
|
||
```shell | ||
dvc push | ||
``` | ||
|
||
Make sure you have set the credentials before writing on the remote storage. We are using Azure Blob Storage for the first proof of concept. You can authenticate the `dvc` client with a SAS token using environment variables: | ||
|
||
```shell | ||
export AZURE_STORAGE_ACCOUNT='nautiluscyberneering' | ||
export AZURE_STORAGE_SAS_TOKEN='?sv=2020-08-04&ss=xxx=sco&sp=xxx&se=2021-12-31T19:15:33Z&st=2021-10-01T10:15:33Z&spr=https&sig=xxx' | ||
``` | ||
|
||
You can find those variables in the [template file for environment varaibles](../.secrets.ci). | ||
|
||
After pushing the image you also need to push your git changes with: | ||
|
||
```shell | ||
git push --set-upstream origin issue-106-add-gold-image-000000-32.600.2.tif | ||
``` | ||
|
||
Now we have the new branch with the new image ready to create the pull request. | ||
|
||
### 6. Create the new pull request | ||
|
||
You can use the GitHub interface or you can use the link provied after pushing using the git from console. The pull request should look like: | ||
|
||
![Add Gold image - create pull request](./images/media-contributor-guide/new-pull-request.png) | ||
|
||
You can also use the `media` label. | ||
|
||
The PR is going to trigger the [workflow to process the Gold images](../.github/workflows/gold-drawings-processing.yml). | ||
|
||
What this worflow does is: | ||
|
||
- Use `dvc diff` command to detect the new Gold image. | ||
- Validate the filename, folder and size of the image. | ||
- Generate a Base image from the Gold. It's an smaller version of the image. | ||
|
||
When the workflow finishes you should see two new commits in your pull request branch. | ||
|
||
![Add Gold image - commits after Gold images workflow](./images/media-contributor-guide/commits-after-gold-proccesing-workflow.png) | ||
|
||
One commit should be your own commit adding the new Gold image. And the other commit should be the automatic commit generated by the workflow adding the corresponding Base image. | ||
|
||
You can check if the workflow has generated the correct Base image by downloading it to you local machine. | ||
|
||
```shell | ||
git pull && dvc pull | ||
``` | ||
|
||
After running that command you should see the new Base image along with the `dvc` files (pointer and gitignore) for the Base image: | ||
|
||
![Add Gold image - new Base image](./images/media-contributor-guide/new-base-image.png) | ||
|
||
### 6. Validate and merge the pull requests | ||
|
||
If there were not any error, a maintainer can merge the pull request. This role should be a media maintainer in charge of the image collection. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "If there were not any error" -- The verb tense seems off. |
||
|
||
The merge event is going to trigger another [workflow to update the website project](../.github/workflows/update-website.yml). | ||
|
||
### 7. New version of the website deployed | ||
|
||
If there were no errors in the previous stepts you should see the new Gold drawing on the website: https://nautilus-cyberneering.github.io/chinese-ideographs-website. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same as the previous. |
||
|
||
The website contains another workflow which is triggered by the [update-website workflow](../.github/workflows/update-website.yml). That workflow obtains the image from the `dvc` storage and convert it to `JPG` format. It also copies the image inside the git repo (it's not using git LFS). You can get more information about that process on the [website repository](https://nautilus-cyberneering.github.io/chinese-ideographs-website/). | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thin link to the "website repository", links directly to the website instead. |
||
|
||
## How to update a Gold Drawing | ||
|
||
In this tutorial, you will learn how to update a new Gold Drawing image to the library. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Whe you say update it will lead to a modification. |
||
|
||
NOTE: The intention is to update an image. If you want to add the artist revision or edition, that's a different artwork, so you should add a new image. The "update" case should only be used when you want to fix the wrong image (invalid, file, corrupted file, wrong content, etcetera). | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "If you want to add the artist revision or edition" --- ... artist's ... |
||
|
||
You can watch the whole screen capture of this tutorial on [YouTube](https://youtu.be/CB5hZGWcs8o). | ||
|
||
TODO. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe add an index for the document. With a brief overview of the steps.