| Time |
Nickname |
Message |
|
00:00
🔗
|
|
kyan has quit IRC (Read error: Connection reset by peer) |
|
00:04
🔗
|
closure |
how about https://archive.org/details/ephemera |
|
00:05
🔗
|
closure |
that+78rpm will be 2 tb |
|
00:07
🔗
|
closure |
moved shard generation to the iabak server from fos, and it runs about 10x as fast no |
|
00:15
🔗
|
closure |
hmm, I can toss in oldtimeradio too, for a 2.9 tb shard |
|
00:37
🔗
|
db48x |
just when I'm out of disk space |
|
00:40
🔗
|
db48x |
closure: do you have a second for a generic git-annex question? |
|
00:40
🔗
|
closure |
sure |
|
00:40
🔗
|
db48x |
is there a way I can annex a url and have it transformed after downloading? |
|
00:41
🔗
|
db48x |
I want to annex a gzipped file and have it end up ungzipped automatically |
|
00:42
🔗
|
closure |
ah.. this is possible to do using the external special remote interface, but that may be overkill or not suited to what you're trying to do |
|
00:42
🔗
|
db48x |
git annex addurl https://archive.org/download/emularity_engine_jsmess/messa2600.js.gz --file emulators/jsmess/messa2600.js.gz |
|
00:43
🔗
|
closure |
there's a example one for bittorrent (before it got built into git-annex); for *.torrent urls, it makes addurl add not the torrent file, but download the contents and add those |
|
00:44
🔗
|
db48x |
hmm |
|
00:44
🔗
|
closure |
http://git-annex.branchable.com/special_remotes/external/git-annex-remote-torrent |
|
00:44
🔗
|
db48x |
if I did that, how would another user of the same repository get the file? |
|
00:44
🔗
|
closure |
they have to install the program and enable the special remote, and then they can get files using it |
|
00:45
🔗
|
db48x |
figures |
|
00:45
🔗
|
db48x |
that's a bit unwieldy |
|
00:45
🔗
|
closure |
for gz, yes.. it might be useful for tars or zips |
|
00:45
🔗
|
db48x |
should be able to do git annex addurl http://example.com/foo.gz --pipe gunzip --file foo |
|
00:46
🔗
|
db48x |
or something |
|
00:47
🔗
|
|
niyaje4 has quit IRC (Ping timeout: 600 seconds) |
|
00:48
🔗
|
db48x |
could meticulously translate all shell syntax into command-line arguments and let us build arbitrarily complex scripts which get stored in-line... |
|
00:48
🔗
|
closure |
or not :P |
|
00:49
🔗
|
db48x |
heh |
|
00:51
🔗
|
db48x |
maybe a smudge filter? |
|
01:08
🔗
|
closure |
sounds more like it |
|
01:43
🔗
|
beardicus |
ooh. so i'm "expired", at least on shard1. |
|
01:43
🔗
|
beardicus |
but i'm running an iabak |
|
01:44
🔗
|
closure |
those expires haven't happened yet |
|
01:44
🔗
|
closure |
but, which repo is it? |
|
01:45
🔗
|
beardicus |
oh, nm. i think both my iabaks were hung. |
|
01:45
🔗
|
beardicus |
restarted |
|
01:45
🔗
|
beardicus |
bert@storage is me |
|
01:46
🔗
|
closure |
see if the uuid there matches your repo |
|
01:46
🔗
|
closure |
it might be an old repo you had, if you deleted it.. the one that it wants to expire is not recorded as containing any files |
|
01:48
🔗
|
beardicus |
oh ok. that would be old then. |
|
01:48
🔗
|
beardicus |
what's the incantation for getting into the shell where i can run git annex info or what-have-you? |
|
01:49
🔗
|
closure |
cd shard1; git config annex.uuid |
|
01:49
🔗
|
beardicus |
runshell is what i was thinking of... but clearly that's not what i actually needed :) |
|
01:50
🔗
|
closure |
ah, git-annex.linux/runshell |
|
01:50
🔗
|
beardicus |
ok. that indeed does not match my current uuid. |
|
01:50
🔗
|
closure |
great, expiry working as intended |
|
01:53
🔗
|
tpw_rules |
https://archive.org/download/Ttscribe/Ttscribe_files.xml is still darked |
|
01:54
🔗
|
tpw_rules |
i can't complete my IAdex unless that gets removed from shard1 |
|
01:55
🔗
|
tpw_rules |
closure: can you fix that? i just synced |
|
01:58
🔗
|
tpw_rules |
or amn i doing something wrong? |
|
02:03
🔗
|
tpw_rules |
closure: ^ |
|
02:10
🔗
|
|
kyan has joined #internetarchive.bak |
|
02:20
🔗
|
|
cloudmons has quit IRC (Read error: Operation timed out) |
|
02:21
🔗
|
|
cloudmons has joined #internetarchive.bak |
|
02:57
🔗
|
|
SN4T14__ has quit IRC (Read error: Connection reset by peer) |
|
02:58
🔗
|
|
cloudmons has quit IRC (Read error: Operation timed out) |
|
03:00
🔗
|
|
SN4T14 has joined #internetarchive.bak |
|
03:00
🔗
|
|
cloudmons has joined #internetarchive.bak |
|
03:01
🔗
|
|
chfoo has quit IRC (Read error: Connection reset by peer) |
|
03:14
🔗
|
|
beardicus has quit IRC (Quit: Sleep.) |
|
03:16
🔗
|
|
chfoo has joined #internetarchive.bak |
|
03:16
🔗
|
|
svchfoo1 sets mode: +o chfoo |
|
04:08
🔗
|
|
berndj has quit IRC (Read error: Operation timed out) |
|
04:36
🔗
|
|
zottelbey has joined #internetarchive.bak |
|
05:03
🔗
|
|
zottelbey has quit IRC (Remote host closed the connection) |
|
06:10
🔗
|
|
cloudmons has quit IRC (Read error: Operation timed out) |
|
06:10
🔗
|
|
chfoo has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
iten has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
espes___ has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
Quile has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
SketchCow has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
closure has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
ersi has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
bpye_ has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
serapeum has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
svchfoo3 has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
destrudo has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
Cameron_D has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
balrog has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
marvinw has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
GLaDOS has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
Lord_Nigh has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
ppiixx has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
hatseflat has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
Muad-Dib has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
Vito` has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
lhobas has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
mrfoo has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
jbenet_ has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
ryang has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
antomatic has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
Kazzy has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
edsu_ has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
chfoo- has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
hater has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
garyrh has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
Senji has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
raylee has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
svchfoo2 has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
Atluxity has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
kyan has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
db48x has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
dirt has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
midas has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
patrickod has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
Kenshin has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
trs80 has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
mhazinsk has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
swebb has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
fenn has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
SN4T14 has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
arkiver has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
yipdw has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
realeyes has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
svchfoo1 has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:10
🔗
|
|
chazchaz has quit IRC (ircd.choopa.net hub.efnet.us) |
|
06:12
🔗
|
|
cloudmons has joined #internetarchive.bak |
|
06:35
🔗
|
|
berndj has joined #internetarchive.bak |
|
06:35
🔗
|
|
chfoo has joined #internetarchive.bak |
|
06:35
🔗
|
|
SN4T14 has joined #internetarchive.bak |
|
06:35
🔗
|
|
kyan has joined #internetarchive.bak |
|
06:35
🔗
|
|
garyrh has joined #internetarchive.bak |
|
06:35
🔗
|
|
raylee has joined #internetarchive.bak |
|
06:35
🔗
|
|
svchfoo2 has joined #internetarchive.bak |
|
06:35
🔗
|
|
Atluxity has joined #internetarchive.bak |
|
06:35
🔗
|
|
balrog has joined #internetarchive.bak |
|
06:35
🔗
|
|
db48x has joined #internetarchive.bak |
|
06:35
🔗
|
|
ny.us.hub sets mode: +oooo chfoo garyrh svchfoo2 db48x |
|
06:35
🔗
|
|
iten has joined #internetarchive.bak |
|
06:35
🔗
|
|
Vito` has joined #internetarchive.bak |
|
06:35
🔗
|
|
marvinw has joined #internetarchive.bak |
|
06:35
🔗
|
|
GLaDOS has joined #internetarchive.bak |
|
06:35
🔗
|
|
arkiver has joined #internetarchive.bak |
|
06:35
🔗
|
|
Lord_Nigh has joined #internetarchive.bak |
|
06:35
🔗
|
|
dirt has joined #internetarchive.bak |
|
06:35
🔗
|
|
mhazinsk has joined #internetarchive.bak |
|
06:35
🔗
|
|
trs80 has joined #internetarchive.bak |
|
06:35
🔗
|
|
chazchaz has joined #internetarchive.bak |
|
06:35
🔗
|
|
fenn has joined #internetarchive.bak |
|
06:35
🔗
|
|
swebb has joined #internetarchive.bak |
|
06:35
🔗
|
|
svchfoo1 has joined #internetarchive.bak |
|
06:35
🔗
|
|
ny.us.hub sets mode: +oooo GLaDOS arkiver mhazinsk svchfoo1 |
|
06:35
🔗
|
|
realeyes has joined #internetarchive.bak |
|
06:35
🔗
|
|
Kenshin has joined #internetarchive.bak |
|
06:35
🔗
|
|
yipdw has joined #internetarchive.bak |
|
06:35
🔗
|
|
patrickod has joined #internetarchive.bak |
|
06:35
🔗
|
|
midas has joined #internetarchive.bak |
|
06:35
🔗
|
|
chfoo- has joined #internetarchive.bak |
|
06:35
🔗
|
|
ppiixx has joined #internetarchive.bak |
|
06:35
🔗
|
|
Senji has joined #internetarchive.bak |
|
06:35
🔗
|
|
ersi has joined #internetarchive.bak |
|
06:35
🔗
|
|
hatseflat has joined #internetarchive.bak |
|
06:35
🔗
|
|
Muad-Dib has joined #internetarchive.bak |
|
06:35
🔗
|
|
edsu_ has joined #internetarchive.bak |
|
06:35
🔗
|
|
jbenet_ has joined #internetarchive.bak |
|
06:35
🔗
|
|
mrfoo has joined #internetarchive.bak |
|
06:35
🔗
|
|
lhobas has joined #internetarchive.bak |
|
06:35
🔗
|
|
bpye_ has joined #internetarchive.bak |
|
06:35
🔗
|
|
ryang has joined #internetarchive.bak |
|
06:35
🔗
|
|
serapeum has joined #internetarchive.bak |
|
06:35
🔗
|
|
antomatic has joined #internetarchive.bak |
|
06:35
🔗
|
|
svchfoo3 has joined #internetarchive.bak |
|
06:35
🔗
|
|
ny.us.hub sets mode: +oooo Kenshin yipdw ersi svchfoo3 |
|
06:35
🔗
|
|
destrudo has joined #internetarchive.bak |
|
06:35
🔗
|
|
Cameron_D has joined #internetarchive.bak |
|
06:35
🔗
|
|
espes___ has joined #internetarchive.bak |
|
06:35
🔗
|
|
closure has joined #internetarchive.bak |
|
06:35
🔗
|
|
SketchCow has joined #internetarchive.bak |
|
06:35
🔗
|
|
Quile has joined #internetarchive.bak |
|
06:35
🔗
|
|
hater has joined #internetarchive.bak |
|
06:35
🔗
|
|
Kazzy has joined #internetarchive.bak |
|
06:35
🔗
|
|
ny.us.hub sets mode: +ooo closure SketchCow Kazzy |
|
07:31
🔗
|
|
atomotic has joined #internetarchive.bak |
|
09:23
🔗
|
|
niyaje4 has joined #internetarchive.bak |
|
10:13
🔗
|
|
VADemon has joined #internetarchive.bak |
|
10:18
🔗
|
|
niyaje4 has quit IRC (Ping timeout: 600 seconds) |
|
10:43
🔗
|
|
atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) |
|
11:19
🔗
|
|
beardicus has joined #internetarchive.bak |
|
11:40
🔗
|
|
atomotic has joined #internetarchive.bak |
|
11:44
🔗
|
|
zottelbey has joined #internetarchive.bak |
|
12:48
🔗
|
|
sankin has joined #internetarchive.bak |
|
13:02
🔗
|
|
VADemon has quit IRC (Read error: Connection reset by peer) |
|
13:14
🔗
|
|
sankin has quit IRC (Leaving.) |
|
13:26
🔗
|
|
sankin has joined #internetarchive.bak |
|
13:26
🔗
|
trs80 |
freed up some space, so starting multiple iabaks. closure, could you maybe skip the fsck if it's been done recently? |
|
13:26
🔗
|
trs80 |
touch a stamp file or something |
|
13:56
🔗
|
|
Start has quit IRC (Disconnected.) |
|
14:33
🔗
|
|
Start has joined #internetarchive.bak |
|
14:44
🔗
|
db48x |
fast fsck should be really fast now |
|
14:44
🔗
|
Senji |
Oh? |
|
14:45
🔗
|
db48x |
yea, seconds |
|
14:47
🔗
|
db48x |
do a git pull in IA.BAK to make sure you have the latest version of the scripts |
|
14:52
🔗
|
SketchCow |
The new shard begins |
|
14:57
🔗
|
|
atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) |
|
14:58
🔗
|
|
Start has quit IRC (Disconnected.) |
|
15:02
🔗
|
|
Start has joined #internetarchive.bak |
|
15:39
🔗
|
midas |
woop woop |
|
15:51
🔗
|
|
Start has quit IRC (Disconnected.) |
|
15:58
🔗
|
|
Start has joined #internetarchive.bak |
|
16:38
🔗
|
|
Start has quit IRC (Disconnected.) |
|
16:50
🔗
|
|
Start has joined #internetarchive.bak |
|
17:42
🔗
|
|
Start has quit IRC (Disconnected.) |
|
18:08
🔗
|
SketchCow |
What's the amount of files per shard? |
|
18:09
🔗
|
|
zottelbey has quit IRC (Remote host closed the connection) |
|
18:11
🔗
|
db48x |
SketchCow: 100kish |
|
18:11
🔗
|
|
zottelbey has joined #internetarchive.bak |
|
18:16
🔗
|
SketchCow |
Thanks, that helps. |
|
18:17
🔗
|
db48x |
yw |
|
19:29
🔗
|
|
Start has joined #internetarchive.bak |
|
19:49
🔗
|
|
SN4T14_ has joined #internetarchive.bak |
|
19:55
🔗
|
|
SN4T14 has quit IRC (Ping timeout: 369 seconds) |
|
20:19
🔗
|
|
Start has quit IRC (Disconnected.) |
|
20:53
🔗
|
|
sankin has quit IRC (Leaving.) |
|
22:01
🔗
|
|
niyaje4 has joined #internetarchive.bak |
|
22:04
🔗
|
|
zottelbey has quit IRC (Remote host closed the connection) |
|
22:08
🔗
|
|
garyrh has quit IRC (Ping timeout: 506 seconds) |
|
22:17
🔗
|
|
Start has joined #internetarchive.bak |
|
22:17
🔗
|
|
svchfoo1 sets mode: +o Start |
|
22:30
🔗
|
|
ersi has quit IRC (Ping timeout: 512 seconds) |
|
22:31
🔗
|
|
ohhdemgir has joined #internetarchive.bak |
|
22:49
🔗
|
trs80 |
db48x: I have 1dc6c53578949c922116decb24c6af417f323da6 switch fast fask to be a truely fast expiry-preventing ping and the shard1 "This shard is in maintenance mode; checking it." has taken 3 minutes so far |
|
22:50
🔗
|
trs80 |
looking at maint(), it doesn't even call fastfsck |
|
22:51
🔗
|
trs80 |
it does take a lock at least, so if I'd started them one after another, the second one would have skipped it |
|
22:52
🔗
|
|
niyaje4 has quit IRC (Ping timeout: 600 seconds) |
|
22:55
🔗
|
db48x |
trs80: ah, indeed. that's a normal fsck |
|
22:59
🔗
|
trs80 |
at least it took less than 10 minutes |
|
23:00
🔗
|
trs80 |
Checking for any files that still need to be downloaded... is a bit slow too |
|
23:01
🔗
|
|
Start has quit IRC (Read error: Connection reset by peer) |
|
23:01
🔗
|
|
Start_ has joined #internetarchive.bak |
|
23:05
🔗
|
trs80 |
again, about 10 minutes. and now for the shuf delay |
|
23:06
🔗
|
trs80 |
I guess the real answer is a long running/parallel process to amortise these startup costs |
|
23:48
🔗
|
|
Start_ has quit IRC (Read error: Connection reset by peer) |
|
23:48
🔗
|
|
Start has joined #internetarchive.bak |
|
23:49
🔗
|
|
svchfoo1 sets mode: +o Start |