making some space on pollux

[jbenet]

• making some space on pollux for @diasdavid
• @lgierth started the move (hasnt finished as of now, i dont think)
• @diasdavid is running out space faster than expected
• @davidar jumped into action and is deleting some stuff that has been moved.

it would be good to:

• figure out what is being moved
• what can be deleted as it moves

if you tell me what im able to delete and by when i can take care of it

[jbenet]

@diasdavid how much space total do you need? (overshoot)

[daviddias]

@jbenet about 1.6Tb (if progress bar is telling the truth)

[jbenet]

@diasdavid that doesn't include adding it again though (for ipfs tar, etc)-- how much space for everything for your demo.

[daviddias]

02:36 daviddias: ok, make it like 4Tb (figured out why it is creating so much overhead, it is because the new ipfs add -r, uses an improved files API and files API doesn't gc old dir MerkleDAG nodes)

[davidar]

@diasdavid Um, the disk on pollux is only 2.7TB in total :/

[davidar]

My PMCOA mirror from ipfs-inactive/archives#4 is consuming 1.1TB, so we can either:

• transfer it to castor
• delete it and re-mirror from FTP somewhere else
• wait until IA mirrors it for us (I believe someone's working on that for @OpenJournal)

[ghost]

SCP in progress since 3:49 UTC, in a root screen on pollux:

• root@pollux ~ # screen scp -r -B /home/davidar/sirius/ [email protected]:/home/davidar/

  ~320 GB, already moved a fair chunk yesterday (which is not included in the 320 number)

So far it's averaging at 24.5 MB/s, not sure whether it's bounded by network, or disk (ipfs add running concurrently to SCP). Yesterday, without a concurrent ipfs add, it made 50 to 60 MB/s.

Next:

• pick the next thing to move from pollux:/home/davidar/
• put everything you move to castor into a subdirectory like castor:/home/davidar/pollux/

[jbenet]

@lgierth ok thanks!

[ghost]

Throughput metrics: http://metrics.i.ipfs.io/dashboard/db/hosts (see https://github.com/ipfs/infrastructure/tree/master/solarnet#monitoring)

[ghost]

If you wanna go delete stuff before that first SCP finishes, a simple way is to list the files from castor which have already arrived, and delete these on pollux. Omit the last few (file timestamp) so that you don't delete ones which are still being transferred. (That's what I did yesterday after the partial transfer.)

At the moment it's gonna be a bit more than 1 hour before pollux is out of space.

[ghost]

Just saying this because SCP doesn't seem to be copying in any particular order

[ghost]

Diskspace metrics: http://metrics.i.ipfs.io/dashboard/db/meta?panelId=3&fullscreen

[jbenet]

finished:

mv pollux/home/davidar/pmcoa castor/home/davidar/pollux/pmcoa

1.1 TB free in pollux now.

[ghost]

Two new 4x6TB hosts with Hetzner should be ready to deploy any moment

[daviddias]

❤️❤️❤️❤️❤️❤️❤️❤️❤️❤️

what will be their CPU?

[ghost]

same i7 3770 as pollux

[daviddias]

sweet! :D