| Time |
Nickname |
Message |
|
00:31
🔗
|
|
mariusz has joined #internetarchive.bak |
|
00:32
🔗
|
mariusz |
fatal: Out of memory? mmap failed: Cannot allocate memory |
|
00:32
🔗
|
mariusz |
git-annex-shell: git-shell failed |
|
00:33
🔗
|
mariusz |
any ideas? |
|
00:39
🔗
|
|
Apathy_ has quit IRC (Quit: OOOOoooooooooo................) |
|
00:41
🔗
|
|
Apathy has joined #internetarchive.bak |
|
01:01
🔗
|
tpw_rules |
mariusz: which repo? |
|
01:01
🔗
|
tpw_rules |
shard |
|
01:17
🔗
|
|
wp494 has quit IRC (Ping timeout: 306 seconds) |
|
01:19
🔗
|
mariusz |
8 |
|
01:20
🔗
|
mariusz |
but this also happend when i ran cron job |
|
01:26
🔗
|
|
primus104 has quit IRC (Leaving.) |
|
01:32
🔗
|
tpw_rules |
there was a server problem that broke 3. perhaps 8 is now suffering the same affliction |
|
01:39
🔗
|
Senji |
That's the same symptoms 3 had |
|
01:41
🔗
|
Senji |
I think we're finding exciting new bugs |
|
02:13
🔗
|
Senji |
If so we might start seeing it on other shards too :-( |
|
02:14
🔗
|
mariusz |
oh well, nobody expected this to be bug free |
|
02:15
🔗
|
Senji |
Someone should tell db-whatever-his-name-is next time they see him |
|
02:20
🔗
|
tpw_rules |
dee bee fourty eight ex? |
|
02:21
🔗
|
|
hatseflat has quit IRC (Read error: Operation timed out) |
|
02:23
🔗
|
|
hatseflat has joined #internetarchive.bak |
|
05:22
🔗
|
|
wickedpla has joined #internetarchive.bak |
|
05:22
🔗
|
|
wickedpla is now known as wp494 |
|
05:23
🔗
|
|
wp494 is now known as wp494_ |
|
05:23
🔗
|
|
wp494_ is now known as wp494 |
|
08:08
🔗
|
|
primus104 has joined #internetarchive.bak |
|
08:18
🔗
|
|
db48x has joined #internetarchive.bak |
|
08:31
🔗
|
|
primus104 has quit IRC (Leaving.) |
|
08:57
🔗
|
iabak-reg |
03registrar 05master 242b49f 06other 10SHARD9/pubkeys registration of sunny on SHARD9 |
|
09:04
🔗
|
|
primus104 has joined #internetarchive.bak |
|
10:12
🔗
|
|
primus104 has quit IRC (Leaving.) |
|
11:39
🔗
|
|
VADemon has joined #internetarchive.bak |
|
12:19
🔗
|
|
zz_CyberJ is now known as CyberJaco |
|
12:52
🔗
|
iabak-reg |
03registrar 05master 29b3bae 06other 10SHARD7/pubkeys registration of sunny on SHARD7 |
|
13:06
🔗
|
iabak-reg |
03registrar 05master 8018d79 06other 10SHARD8/pubkeys registration of sunny on SHARD8 |
|
13:08
🔗
|
|
mariusz_ has joined #internetarchive.bak |
|
13:32
🔗
|
iabak-reg |
03registrar 05master a3b8962 06other 10SHARD9/pubkeys registration of sunny on SHARD9 |
|
13:33
🔗
|
|
Apathy has quit IRC (Excess Flood) |
|
13:33
🔗
|
|
Apathy has joined #internetarchive.bak |
|
13:39
🔗
|
mariusz_ |
shard9?? what's the point if we can't have current 8 complete? |
|
13:41
🔗
|
Senji |
Bandwidth is an issue. Might as well get on with downloading *something*? |
|
13:42
🔗
|
Senji |
I'm downloading on 2,5,and 6 and would be on 3 and 8 if they were available |
|
13:44
🔗
|
mariusz_ |
ok.. any idea if/when 3 and 8 will be back? |
|
13:45
🔗
|
Senji |
When the bug gets fixed? Might have to wait for closure to get back from holiday? |
|
14:19
🔗
|
tpw_rules |
was it figured out? |
|
14:19
🔗
|
tpw_rules |
i'm pretty much out of space tho :( |
|
14:30
🔗
|
|
primus104 has joined #internetarchive.bak |
|
14:32
🔗
|
mariusz_ |
hm. i'm starting to get out of mem on shard6 |
|
14:32
🔗
|
tpw_rules |
hmmm |
|
14:42
🔗
|
mariusz_ |
yeap, it's fked |
|
14:45
🔗
|
|
toad1 has joined #internetarchive.bak |
|
14:46
🔗
|
Senji |
I'm surprised it's happening on all the shards about at onceish. Maybe it's a recent bug in git-annex? |
|
14:47
🔗
|
|
toad2 has quit IRC (Read error: Operation timed out) |
|
14:55
🔗
|
mariusz_ |
the only question is how hard it will be to recover from that |
|
15:08
🔗
|
|
mariusz_ has quit IRC (Read error: Operation timed out) |
|
17:44
🔗
|
SketchCow |
So, just to answer the question. |
|
17:45
🔗
|
SketchCow |
We have SHARD9 set up "just in case" for people who register enough and have enough space that they flush out their copy of the needed space. |
|
17:45
🔗
|
SketchCow |
I absolutely think we have a round of bugfixes awaiting db48x and closure |
|
19:25
🔗
|
iabak-reg |
03registrar 05master f6f1110 06other 10SHARD7/pubkeys registration of chris on SHARD7 |
|
19:30
🔗
|
|
RX14 has joined #internetarchive.bak |
|
19:31
🔗
|
RX14 |
i'm having an issue running the internetarchive.bak script: https://i.imgur.com/9R0B5lq.png |
|
19:33
🔗
|
tpw_rules |
RX14: that's normal. some files are no longer available publicly to be backed up. you can ignore them |
|
19:33
🔗
|
RX14 |
it looks to be happening for every file, unless it doesn't log every file |
|
19:34
🔗
|
RX14 |
and if it is normal, you shouls probably put a note on the wiki |
|
19:34
🔗
|
tpw_rules |
you probably downloaded a bunch of working ones and now it's retrying all the broken ones |
|
19:34
🔗
|
tpw_rules |
give it time |
|
19:35
🔗
|
RX14 |
i just started it actually |
|
19:35
🔗
|
tpw_rules |
ah, you must not have shuf |
|
19:35
🔗
|
tpw_rules |
but it will do it eventually |
|
19:35
🔗
|
RX14 |
aha |
|
19:35
🔗
|
tpw_rules |
that partucular one has a bunch at the beginning |
|
19:35
🔗
|
RX14 |
this looks better |
|
19:36
🔗
|
tpw_rules |
progress bars and wget? |
|
19:36
🔗
|
RX14 |
after many attempts |
|
19:36
🔗
|
RX14 |
yup |
|
19:36
🔗
|
RX14 |
now i need more concurrent downloads |
|
19:36
🔗
|
RX14 |
there |
|
19:37
🔗
|
tpw_rules |
it won't print anything if it downloads concurrently. i like to run a bunch at once so i get a screen of progress bars |
|
19:37
🔗
|
tpw_rules |
looks more important that way |
|
19:37
🔗
|
RX14 |
that's annoying, is that a git-annex thing |
|
19:37
🔗
|
tpw_rules |
yeah |
|
19:37
🔗
|
tpw_rules |
what os do you use? |
|
19:37
🔗
|
RX14 |
i can view nethogs |
|
19:37
🔗
|
RX14 |
linux |
|
19:37
🔗
|
RX14 |
my dedi |
|
19:37
🔗
|
tpw_rules |
i like to use iftop |
|
19:37
🔗
|
RX14 |
never heard of that |
|
19:37
🔗
|
tpw_rules |
it's like top but for internet |
|
19:38
🔗
|
RX14 |
why is apt so sloooow |
|
19:38
🔗
|
RX14 |
yeah, iftoop looks neat |
|
19:39
🔗
|
tpw_rules |
if you pass -B -m 50M |
|
19:39
🔗
|
tpw_rules |
then it will show in bytes/sec and keep the scale fixed at 50Mbits/sec |
|
19:40
🔗
|
RX14 |
why would you want to fix the scale? |
|
19:40
🔗
|
tpw_rules |
i liked it to see how much of my internet i'm using total |
|
19:40
🔗
|
RX14 |
lets fix it at 1gbps then |
|
19:40
🔗
|
tpw_rules |
how much disk do you have |
|
19:40
🔗
|
RX14 |
900GB |
|
19:40
🔗
|
tpw_rules |
ahh |
|
19:41
🔗
|
tpw_rules |
-f 'net 207.241' |
|
19:41
🔗
|
RX14 |
? |
|
19:41
🔗
|
tpw_rules |
will make it only count packets coming/going to archive'org domains |
|
19:41
🔗
|
RX14 |
ok |
|
19:41
🔗
|
tpw_rules |
(not quite but enough for the purposes) |
|
19:41
🔗
|
RX14 |
and, ANNEXGETOPTS doesn't seem to work |
|
19:41
🔗
|
RX14 |
where do you put ANNEXGETOPTS |
|
19:41
🔗
|
tpw_rules |
yeah it doe |
|
19:41
🔗
|
tpw_rules |
s |
|
19:42
🔗
|
tpw_rules |
in a file in the IA.BAK directory |
|
19:42
🔗
|
tpw_rules |
like echo -J100 > ANNEXGETOPTS |
|
19:42
🔗
|
RX14 |
i did that |
|
19:42
🔗
|
RX14 |
i only see 1 connection in iftop |
|
19:42
🔗
|
tpw_rules |
did you restart the client? |
|
19:42
🔗
|
RX14 |
and i see progress bars |
|
19:42
🔗
|
tpw_rules |
also install shuf |
|
19:42
🔗
|
RX14 |
tpw_rules, i did |
|
19:42
🔗
|
tpw_rules |
what script are you running? |
|
19:42
🔗
|
tpw_rules |
./iabak |
|
19:42
🔗
|
RX14 |
yup |
|
19:42
🔗
|
RX14 |
i ctrl-c |
|
19:42
🔗
|
tpw_rules |
hm |
|
19:43
🔗
|
RX14 |
doesn't seem to have stopped every process though |
|
19:43
🔗
|
tpw_rules |
do you have git annex installed on your system though |
|
19:43
🔗
|
tpw_rules |
yeah it doesn't do that |
|
19:43
🔗
|
RX14 |
i don't |
|
19:43
🔗
|
tpw_rules |
ok |
|
19:43
🔗
|
tpw_rules |
can you paste the whole log from you doing ./iabak to it downloading? |
|
19:45
🔗
|
RX14 |
http://hastebin.com/ijicexized |
|
19:46
🔗
|
tpw_rules |
how many concurrent downloads did you say? |
|
19:46
🔗
|
RX14 |
7 |
|
19:46
🔗
|
RX14 |
it's just -J7 in the file |
|
19:46
🔗
|
tpw_rules |
so cat ANNEXGETOPTS is "-J7" |
|
19:46
🔗
|
tpw_rules |
hm |
|
19:46
🔗
|
RX14 |
yup |
|
19:46
🔗
|
RX14 |
i think... |
|
19:46
🔗
|
RX14 |
it might have been 5 |
|
19:46
🔗
|
tpw_rules |
no newline? |
|
19:47
🔗
|
RX14 |
well echo prints a newline |
|
19:47
🔗
|
RX14 |
so i guess it does |
|
19:47
🔗
|
tpw_rules |
hm. it should work tho |
|
19:48
🔗
|
tpw_rules |
echo -n '-J5' > ANNEXGETOPTS |
|
19:48
🔗
|
tpw_rules |
then cat ANNEXGETOPTS should be -J5<prompt> |
|
19:48
🔗
|
tpw_rules |
also install shuf |
|
19:48
🔗
|
RX14 |
i have suf |
|
19:49
🔗
|
tpw_rules |
it's not detecting it |
|
19:49
🔗
|
RX14 |
i just typed shuf |
|
19:49
🔗
|
RX14 |
it's a command |
|
19:49
🔗
|
tpw_rules |
hm |
|
19:49
🔗
|
tpw_rules |
do you use a different shell? |
|
19:49
🔗
|
RX14 |
zsh |
|
19:49
🔗
|
tpw_rules |
oh, it may hate you |
|
19:49
🔗
|
RX14 |
i tried without the newline |
|
19:49
🔗
|
RX14 |
it worked |
|
19:50
🔗
|
RX14 |
i think |
|
19:50
🔗
|
RX14 |
it has the progress bar |
|
19:50
🔗
|
tpw_rules |
shuf may be a builtin |
|
19:50
🔗
|
tpw_rules |
in zsh |
|
19:50
🔗
|
RX14 |
but it has 5 connections |
|
19:50
🔗
|
RX14 |
oh and it's back down to 1 |
|
19:50
🔗
|
tpw_rules |
that could be because you don't have shuf. also it may just be downloading five files from one domain |
|
19:50
🔗
|
RX14 |
500KB/s |
|
19:50
🔗
|
tpw_rules |
does /usr/bin/shuf exist? |
|
19:51
🔗
|
tpw_rules |
oh |
|
19:51
🔗
|
RX14 |
one archive.org server is being really slow |
|
19:51
🔗
|
tpw_rules |
it's probably finishing the file before you ctrl+c it |
|
19:51
🔗
|
tpw_rules |
what does 'which shuf' give you? |
|
19:51
🔗
|
RX14 |
/usr/bin/shuf |
|
19:52
🔗
|
tpw_rules |
is /bin/sh a link to zsh somehow? |
|
19:52
🔗
|
RX14 |
let me try running it in bash |
|
19:52
🔗
|
RX14 |
sh seems to be a link to dash |
|
19:52
🔗
|
RX14 |
wth |
|
19:53
🔗
|
tpw_rules |
it may not work quite right wiht that |
|
19:54
🔗
|
RX14 |
i ran it with bash iabak |
|
19:54
🔗
|
RX14 |
and it's still not in parallel |
|
19:54
🔗
|
RX14 |
maybe |
|
19:54
🔗
|
tpw_rules |
has it downloaded more than one file? |
|
19:54
🔗
|
tpw_rules |
try commenting out line 100 in iabak-helper |
|
19:54
🔗
|
RX14 |
htop says one wget binary |
|
19:55
🔗
|
tpw_rules |
is it at least detecting shuf? |
|
19:56
🔗
|
RX14 |
it's downloading |
|
19:56
🔗
|
RX14 |
but only at 140Mbps |
|
19:56
🔗
|
tpw_rules |
awwww |
|
19:56
🔗
|
RX14 |
only lol |
|
19:56
🔗
|
tpw_rules |
my home connection peaks at about 120 |
|
19:56
🔗
|
tpw_rules |
but i have no more hard drives to shove in to my pc |
|
19:56
🔗
|
RX14 |
i should run it from home too... |
|
19:56
🔗
|
tpw_rules |
about 40TB as of now |
|
19:57
🔗
|
RX14 |
i'm runnign this on my dedi |
|
19:58
🔗
|
tpw_rules |
ah, i lied a little |
|
19:58
🔗
|
tpw_rules |
25TB used |
|
19:58
🔗
|
tpw_rules |
28TB free |
|
19:58
🔗
|
tpw_rules |
total* |
|
19:58
🔗
|
RX14 |
i don't have that much storage |
|
19:58
🔗
|
tpw_rules |
i do <3 |
|
19:59
🔗
|
tpw_rules |
but now i am out |
|
20:00
🔗
|
RX14 |
wget seems to be in parallel now |
|
20:00
🔗
|
tpw_rules |
i'm using a program called mhddfs to mount a bunch of filesystems as one, kind of like raid but not |
|
20:00
🔗
|
RX14 |
after i commented out your line |
|
20:00
🔗
|
tpw_rules |
so i'm up to /dev/sdn |
|
20:01
🔗
|
RX14 |
nice |
|
20:02
🔗
|
RX14 |
i should put this on my nas |
|
20:02
🔗
|
RX14 |
but actually I shouldn't |
|
20:02
🔗
|
tpw_rules |
i had a shit-ton of hard drives laying around so i just bought really cheap enclosures on ebay and i have like 15 usb drives hanging off my poor nas |
|
20:02
🔗
|
RX14 |
i considered the same |
|
20:02
🔗
|
RX14 |
i have a ghetto nas |
|
20:03
🔗
|
RX14 |
pentium 4 |
|
20:03
🔗
|
RX14 |
got some shitty PCI (not e) raid cards |
|
20:03
🔗
|
RX14 |
no raid |
|
20:03
🔗
|
RX14 |
used them to add sata support |
|
20:03
🔗
|
tpw_rules |
ah, mine's real |
|
20:03
🔗
|
RX14 |
ran btrfs on the random collection of disks |
|
20:03
🔗
|
RX14 |
only 3 disks in there right noew |
|
20:04
🔗
|
tpw_rules |
zfs master race here (at least for the real nas part) |
|
20:04
🔗
|
RX14 |
i should have gone for zfs |
|
20:04
🔗
|
tpw_rules |
the neat thing of mhddfs is that if one of the disks implodes, i'll only lose the junk on that disk |
|
20:04
🔗
|
RX14 |
btrfs crashes all the time |
|
20:04
🔗
|
tpw_rules |
because each disk has an ext4 filesystem and it just combines the files together |
|
20:04
🔗
|
RX14 |
tpw_rules, btrfs is the same |
|
20:05
🔗
|
tpw_rules |
is it? |
|
20:05
🔗
|
RX14 |
are there archive servers outside the us? |
|
20:05
🔗
|
tpw_rules |
no |
|
20:05
🔗
|
tpw_rules |
i think they're all in one datacenter |
|
20:05
🔗
|
RX14 |
because my dedi is in france |
|
20:05
🔗
|
tpw_rules |
which is why we are trying to back them up |
|
20:05
🔗
|
RX14 |
online.net have really good dedi plans |
|
20:05
🔗
|
tpw_rules |
a given file is only one one server anyway |
|
20:06
🔗
|
tpw_rules |
nah they got the it down pretty well |
|
20:08
🔗
|
RX14 |
i wish i had a load of disks lying around |
|
20:08
🔗
|
RX14 |
well i do have a load of disks lying around |
|
20:09
🔗
|
RX14 |
except they are all unles 150GB |
|
20:09
🔗
|
RX14 |
so not that useful |
|
20:13
🔗
|
RX14 |
tpw_rules, so can you just run iabak many times? |
|
20:14
🔗
|
tpw_rules |
yeah |
|
20:14
🔗
|
RX14 |
and they don't conflict |
|
20:14
🔗
|
tpw_rules |
yeah |
|
20:19
🔗
|
RX14 |
http://i.imgur.com/EqpSmQr.png |
|
20:19
🔗
|
RX14 |
this is fun |
|
20:19
🔗
|
tpw_rules |
that's what i did |
|
20:20
🔗
|
RX14 |
it's strange how one is 16MB/s and the rest are flagging |
|
20:40
🔗
|
Senji |
dash> /bin/sh is usually dash nowadays on linux; or posh, or another small shell |
|
20:41
🔗
|
RX14 |
well i'm up and running so... it wroks now? |
|
23:23
🔗
|
|
RX14 has quit IRC (Remote host closed the connection) |