Skip to content

AllAboutProjects

Jump to bottom
Max G edited this page Nov 22, 2019 · 27 revisions

All about Projects!

Remember: if you don't find what you need, or if you'd like to ask a question, then please email [email protected] at any time. We'd love to hear from you! Please include a link (the URL address in your browser) to any relevant project or document, as part of your email.

List of Questions:

Question: What is a project?

Projects are the fundamental way in which your files on CoCalc will be organized. A project is your own private computational work space that you can share with others and upgrade. This means, even files in free projects aren't publicly available unless you explicitly share/publish files.

For example, all of your files will live in a project (though it is easy to copy files between projects).

Each project has a set of collaborators, who have the ability to read, write, delete, or modify anything in the project (except backups). (See also, What is a collaborator and what can they do?.)

Sometimes, you might rather give someone read-only access. In CoCalc, this is called "sharing" with non-collaborators. Click here for details on how to share a file.

Your resource limits (RAM, Disk Storage, CPUs, and the "idle timeout") will be set project-by-project, and can be elevated with upgrades by any collaborator. (See Why upgrade a project? for details.)

Because projects can have folders in them (including folders-within-folders, folders-within-folders-within-folders, and so forth), the files in your project can be as organized or disorganized as you might like.

Technical note: the folders are actually directories in the UNIX/Linux world.

Question: How do I add a new project?

When logged into CoCalc, look in the far upper-left corner, and click on "Projects." Then click on the green button "+ Create new project."

You'll be invited to set a project title and a project description, but don't worry too much about the wording. You can very easily change the title and description at any time later. (See also, How do I change the Title and/or Description of a Project?.)

A project can be with or without upgrades. The upgrades are only available to paid subscribers. Free users must click "Create Project without Upgrades."

In the next question, we'll discuss why you might want to upgrade a project.

Question: Why upgrade a project?

There are many reasons that you might want to have an upgraded project, and paid subscriptions start at $14 per month.

  • To give your project access to the internet. For example,
    • To download a software package from the internet.
    • To use Github/Bitbucket/Gitlab with your project.
    • To download datasets into your project.
    • To connect to your project with SSH.
  • To get extra storage space (both RAM and disk space).
  • To get more compute resources.
  • To have a higher "idle timeout" threshold. (See also, What is an "idle timeout?".)

You can share upgrades with any project that you are a collaborator on. You must be a collaborator (or owner) to update a project.

For details on paid subscriptions, see Subscription and Pricing Information.

You will not be charged until you select a specific subscription then click "Add Subscription".

If you have any questions at all, email [email protected] immediately.

Question: What is a collaborator and what can they do?

Collaborators can read, write, delete, and modify anything in the project, except backups. They can add and remove other collaborators, but cannot remove owners.

Sometimes, you'd rather give someone read-only access. In CoCalc, this is called "sharing" with non-collaborators. Click here for details on how to share a file.

Question: How do I add collaborators?

  1. With the project open, you can see a wrench icon, toward the upper-left corner of the screen. Click that.
  2. In the upper-right part of the settings screen, under "Collaborators," you will see a box labeled: "Add Collaborators."
  3. You can type in a person's name, or their email address. Then CoCalc will search its database of known users and give you possible matches.
  4. After you select a name, don't forget to click "Invite User."
  5. The user has to accept the invitation.

Pragmatically, it is better to use an email address. The reason is that some CoCalc users have multiple accounts. Also, if someone has a several month haitus from CoCalc, they might forget which email address they had used when they created their CoCalc account. This is often a point of confusion.

If there are no matches for an email address, then you can send an invitation for the user to start using CoCalc. You can modify the standard email. It has useful links to make it easier for the other person to start using CoCalc.

Question: How do I change the Title and/or Description of a Project?

  1. With the project open, you can see a wrench icon, toward the upper-left corner of the screen. Click that.
  2. In the upper-left part of the settings screen you will see boxes for the project title and the project description.
  3. When you make a change, a green button marked "Save" will appear. Be certain to click it, otherwise your changes will not take effect.

Question: How can I copy a file from one project to another?

  1. In the upper-left corner of the screen, you'll see a folder icon. Click that.
  2. Click the check-box next to the file or files that you wish to copy.
  3. The "destination" pull-down menu is restricted to the current project... however...
  4. To the left, and slightly upward, you will see "Copy to a folder or a different project." Click on "a different project."
  5. Now you have two pulldown menus, one for the project, and one for the destination within the project.
  6. The button to make this actually happen is on the left, slightly downward, and says "Copy 1 item" or "Copy 3 items", etc...

Question: How can I move a file from one project to another?

This is currently done with the "copy" command, not the "move" command. That's for the safety of your files. If you like, you can copy the file from one project to another, and then delete the original.

Question: How do I switch between projects?

Look in the far upper-left corner, and click on "Projects." Then you will see a list of all of your projects (those that you own and those that you are a collaborator on).

They are sorted by time. In other words, how long it has been since any collaborator made any change in the project. However, there are other ways to organize your projects, using hashtags.

Question: How can I create public projects on CoCalc?

You can make individual files and folders public. For more information, see Sharing Files.

Question: If I remove myself as a collaborator on a project that I've upgraded, what happens to the upgrade?

The upgrade is also removed.

Question: How Do I Restart a project?

First, when the project is open, you should see an icon with a wrench marked "settings." This is near the top of the window, towards the left about 1/4th of the way across your screen. Click on that wrench.

Second, there is a box with several items in it, called "Project Control." There is an icon with some gears next to that. Often you have to scroll-down to find that box.

Third, inside that box, there is a button marked "Restart Project..." with a lightening bolt. Click that. By the way, the reason a lightening bolt was used is that this is not a minor operation.

  • All computations will be stopped (in UNIX-language, that means "all processes will be killed").
  • Good News: You don't lose unsaved files.
  • You do lose any information (state of variables/processes) in RAM.
  • However, anything in files, as long as it's moved from the browser to the web servers (in most cases, at most a few seconds of information), is actually permanently saved to disk already in the database, and will not be lost.
  • When the project starts back up, even if the files on disk are in an older state, the files you see yourself editing in your browser are new with nothing lost. Those files will then be updated on disc very shortly.
  • On the other hand, it will update the project code, and start the project running again.

Fourth, after you click on the "Restart Project..." icon, a warning pops up. Read the warning, and then click on "Restart Project Server" if you want to restart the project. Otherwise, hit "cancel."

Fifth, give the project a good 30 seconds to restart. A lot has to happen, and it might take some time.

Question: I can access some of my files/projects in my CoCalc account but not others in that same account. What's wrong?

Sometimes this can happen if you are using multiple CoCalc accounts in the same browser. This can happen even if you are not using two different accounts simultaneously.

We have a short help page, Tips for using more than one CoCalc account which discusses this. The suggestions there will make it work very smoothly for you.

Question: What can I store in a project?

The available webspace is a courtesy to be able to run your computational projects online. It's not for downloading arbitrary stuff from the internet. For more details about our policies, please read https://cocalc.com/policies/terms.html

Question: In my project list I see some projects archived. Is it done automatically, or can I archive a project myself?

Projects are archived automatically to save disk space. Normally projects get archived if they aren't used for a few weeks. We recently significantly revamped and upgraded our backend infrastructure, and then we ended up archiving everything, then restoring it as people open projects.

CoCalc projects have actually been archived automatically for a long time (years), but until recently (summer, 2018) the UI didn't explicitly tell you that they were archived, even though they were. It was just that they would take longer to start. We have changed the UI so that it's clearer when a project is archived, so you can anticipate that it will take longer to start.

We also revamped things so that starting projects that are archived should be much faster (and more robust). However, this only applies to projects that were recently archived.

This tiered storage for projects (archived, closed, opened, etc.) makes it so we can store a lot more projects a lot longer now. Archived projects themselves sit on nearline Google Cloud Storage, with a copy both of the ZFS streams that define the project, and also a bup archive with a bunch of snapshots, in case there is any issue with the ZFS streams.

Question: What is the difference between Hiding and Deleting a project?

If you delete a project, then you delete it for everyone---for all your collaborators. The good news is that this can be undone.

Alternatively, if you hide a project, then you will not see it in your projects listing, but your collaborators are unaffected.

Question: How do I Hide or Delete a Project?

Note, the previous question describes the difference between hiding a project and deleting a project.

When the project is open, you can see a wrench icon named "Settings" in the tab bar for the project. Click that and scroll all the way down. In the lower-left part of the settings screen, you will see "Hide or Delete project", with a button for hiding the project and a button for deleting the project.

Actually, if you have one or more hidden projects, then in your projects listing (found by clicking on projects in the far upper-left corner of the screen), there will be a check-box marked "Hidden." Clicking that check-box will show the hidden projects (and no other projects). Similarly, if you have one or more deleted projects, then in your projects listing, there will be a check-box marked "Deleted."

Question: How do I unhide a hidden project?

  1. Click on projects, in the far upper-left hand corner of the screen.
  2. If you have one or more hidden projects, there will be a check-box labeled "hidden" near the center of the top of the screen. Click that.
  3. Now you should see all of your hidden projects. Click on the project that you want to unhide.
  4. Click on the wrench icon to get the settings menu.
  5. Scroll all the way to the bottom. In the lower-left corner, you will see an area called "Hide or Delete project."
  6. There is a button marked "unhide project." Click that button.

Note: A project can be both hidden and deleted. In that case, during Step 2 above, be sure both the "hidden" and "deleted" check boxes are both checked.

Question: How do I undelete a deleted project?

  1. Click on projects, in the far upper-left hand corner of the screen.
  2. If you have one or more deleted projects, there will be a check-box labeled "deleted" near the center of the top of the screen. Click that.
  3. Now you should see all of your deleted projects. Click on the project that you want to undelete.
  4. Click on the wrench icon to get the settings menu.
  5. Scroll all the way to the bottom. In the lower-left corner, you will see an area called "Hide or Delete project."
  6. There is a button marked "undelete project." Click that button.

Note: A project can be both hidden and deleted. In that case, during Step 2 above, be sure both the "hidden" and "deleted" check boxes are both checked.

Question: How do I connect via SSH into one of my CoCalc projects?

The next few questions are about the Secure SHell (SSH), a tool for connecting securely between two computers. If you're unfamiliar with SSH, then you probably don't want to attempt this.

Here is a screenshot showing you what this looks like. The terminal application from your local desktop is then able to directly access your project. This can be useful in various ways and also facilitates bulk up and download of files (via scp or rsync).

To SSH into your CoCalc project you need to create public and private keys. That's covered by the next question. A key pair can be configured for use with a single project or for all projects which you can normally log into.

Question: How do I create public/private keys, so that I can SSH into my project?

SSH authentication uses a pair of keys, a private key and a public key. Each key is stored in a separate file. For example, a private key might be in the file id_ed25519 and the matching public key in id_ed25519.pub. In general, private keys are not distributed, while public keys are uploaded to remote systems.

You must have owner or collaborator status on a project for SSH access to be permitted.

When logging into a project with ssh, make sure the project is running. If the project is stopped, or has been restarted within the last 20 seconds or so, you may get a message of 'Permission denied'.

For Microsoft Windows users, a free SSH client with key generator is Putty; see instructions for generating keys using Putty here.

On OS X, and Linux, key pairs are stored in ~/.ssh, where ~ indicates your user's home directory. Use the ssh-keygen command to generate a key pair. (You can do man ssh-keygen for details on the command.)

Example (OS X or Linux). On your local computer, do

ssh-keygen -t ed25519

You may optionally provide a nontrivial passphrase when prompted, which would make it so that even if somebody has a copy of your private key, then they can't use it without knowing that passphrase. After the command completes, you should have these two files in your ~/.ssh folder:

id_ed25519
id_ed25519.pub

NOTE: CoCalc no longer supports manual editing of the authorized_keys file on the remote server for SSH authentication. Instead, use one of the procedures below to install a public key into CoCalc.

Question: How do I configure a pair of ssh keys to access a single project?

This section assumes you have created an SSH key pair as described above.

  1. Open the project Settings tab (wrench icon) for the project you want to access.
  2. Look for the section SSH Keys at lower left.
  3. Click Add an SSH Key.
  4. Enter a title for the key in the Title field. Specify a title that is meaningful to you for the key pair you are using, for example Jane's CoCalc Key.
  5. Copy the public key into the Key field. To do this, open the file for your public key on your local computer. For example, if you are using macOS or Ubuntu, you could open a terminal and type something like the following, depending on the name of your public key file.
cat ~/.ssh/id_ed25519.pub

Use your mouse to highlight the contents of the key file, then copy and paste it into the Key area.

  1. Click Add SSH Key. Your key is now saved for that project.
  2. The user for the SSH connection is the project id without the hyphens (why? because the project id is not a valid Linux username). The hostname is ssh.cocalc.com. Look for Use the following username@host: in the 'SSH Keys' section of project status for a string you can copy and paste. For example, if the Project id is
2507078b-6e5b-43da-809a-0073f1196181

then the SSH username@host will be

[email protected]
  1. To login from your local computer, use a command equivalent to the following:
ssh [email protected]
  1. On macOS or Linux, you can specify a host alias in ~/.ssh/config to avoid having to explicitly pass the project id as above. For example, the following lines in ~/.ssh/config
Host CCPROJ
    Hostname ssh.cocalc.com
    User 2507078b6e5b43da809a0073f1196181
    IdentityFile ~/.ssh/id_ed25519

will allow you to log into the your project from your local computer with the command

ssh CCPROJ

Question: How do I use a single pair of ssh keys for all my projects?

This section assumes you have created an SSH key pair as described above.

  1. Click the gear icon next to your name at upper right to open Account Settings.
  2. Choose the tab "SSH Keys" and note the form for adding a key at right.
  3. Enter a title for the key in the Title field. Specify a title that is meaningful to you for the key pair you are using, for example John's CoCalc Key.
  4. Copy the public key into the Key field. To do this, open the file for your public key on your local computer. For example, if you are using macOS or Ubuntu, you could open a terminal and type something like the following, depending on the name of your public key file.
cat ~/.ssh/id_ed25519.pub

Use your mouse to highlight the contents of the key file, then copy and paste it into the Key area. saving the entry.

  1. Click Add SSH Key. Your key is now saved for that account and will work for all projects for which that account has owner or collaborator status.
  2. As with the previous section, the user@hostname string needed for the ssh command consists of the project id with hyphens removed for the user, and 'ssh.cocalc.com' for the hostname, and can be found just below the caption Use the following username@host: in the 'SSH Keys' section of project status tab.

Question: How do I use mosh?

YOU CAN'T: Mosh is NOT supported with CoCalc.

Question: "I would like to use hashtags to describe my projects" or "I want to categorize my projects"

You can put #foo in the title or description of any project. When you view your list of projects a button will appear for each hashtag, which you can click to show only projects with that tag. There is also excellent support for hashtags in task lists (CoCalc's todo list functionality).

Question: I want to start long-running numerically intensive computations on CoCalc. What are the current limitations?

Open your project and click on Settings. The default limitations are listed under "Quotas" in the lower left. These can be raised, as mentioned there. Notes:

  1. Projects on free non-members only Virtual Machines will get restarted regularly (these are hosted on Google preemtible instances). You can check if a VM rebooted by typing "uptime". crontab files are persistent.

  2. If a project isn't used (via the web-based UI) for the idle timeout (as listed in quotas), then all processes in that project are terminated and the user is removed (so ssh into the project also is not possible). You can pay to raise the idle timeout. See also, What is an "idle timeout?".

Question: I want some scratch space

Use /tmp. Files in /tmp may be deleted at any time, and aren't backed up.

Question: Will my code keep running if I disconnect? Even if my computer is put to sleep? Or do I need to have a machine open in order for the process to run?

You definitely do not need to have your computer awake, or a window open, for your project to keep working. However, this is controlled by something called an "idle timeout," described in the next question.

Question: What is an "idle timeout?"

Under project settings (that's the wrench icon) there is an entry under "Project Usage and Quotas" (left-hand side), which will tell you how long the process will run "in the background." There is an idle timeout for each project, and it will be completely stopped (the technical term in UNIX is "killed") if you don't actively edit a file for that amount of time.

The default for free projects is 1/2 hour. You can increase this to 24 hours for only $14 per month. This means that if you use your project a little bit once per day, then it will never timeout.

However, free projects have another limitation. A free project can be "killed" (stopped) at any time, whatsoever. This will happen at least once per day. You have to keep this in mind when designing your project. (For example, use checkpointing.) In contrast, all paid projects are immune to this issue. See also Subscription and Pricing Information.

The next question will discuss the output of your processes.

Question: If I have code that has been running for a while, and it times out or is otherwise "killed" (see previous question), what happens to the output?

If you are using the "classical" Jupyter notebook, then all output that is printed will be lost if no browser is viewing it. This is a major design flaw in Jupyter.

In contrast, CoCalc's Jupyter Notebook and Sage worksheets will capture output even if no browser is observing them.

You can also (of course) write to a file on disk, which might be preferable in some cases.

Question: I want to see all processes running in my project

Type exactly the following in a full terminal (+New--> Terminal) to see all processing running in a project:

htop

You can kill things, etc. See http://linux.die.net/man/1/htop.

Question: I want to know how much memory I am using

Type exactly the following in a full terminal (+New--> Terminal):

smem -tk

It lists all processes and the bottom line shows the total sum. The last RSS column is probably the most interesting one, for more consult man smem. The total used memory is also listed under 'Project usage and quotas" in project settings.

Question: How do I raise the limit on the number of output messages per cell in a Sage worksheet?

import sage_server
sage_server.MAX_OUTPUT_MESSAGES=100000

See this published worksheet for more details.

Also, type sage_server.[tab key] to see information about other limitations.

Question: I want to XXX, but I don't see XXX above.

Do not hesitate to email THE LINK TO YOUR PROJECT TO [email protected] or https://groups.google.com/forum/?fromgroups#!forum/sage-cloud and ask a question.

Analytics

This Wiki is for CoCalc.com.

A more structured documentation is the CoCalc User Manual.

For further questions, please contact us.

Clone this wiki locally