| Time |
Nickname |
Message |
|
00:49
🔗
|
|
DogsRNice has quit IRC (Ping timeout: 252 seconds) |
|
01:20
🔗
|
|
Raccoon` has joined #archiveteam-bs |
|
01:24
🔗
|
|
Raccoon has quit IRC (Ping timeout: 360 seconds) |
|
01:24
🔗
|
|
Raccoon` is now known as Raccoon |
|
01:35
🔗
|
|
Sokar has joined #archiveteam-bs |
|
03:06
🔗
|
|
odemgi_ has joined #archiveteam-bs |
|
03:09
🔗
|
|
odemgi has quit IRC (Ping timeout: 252 seconds) |
|
03:16
🔗
|
|
qw3rty has joined #archiveteam-bs |
|
03:25
🔗
|
|
qw3rty2 has quit IRC (Ping timeout: 745 seconds) |
|
03:25
🔗
|
|
HashbangI has quit IRC (Read error: Connection reset by peer) |
|
03:26
🔗
|
|
HashbangI has joined #archiveteam-bs |
|
04:43
🔗
|
|
Raccoon` has joined #archiveteam-bs |
|
04:45
🔗
|
|
Raccoon has quit IRC (Ping timeout: 258 seconds) |
|
04:45
🔗
|
|
Raccoon` is now known as Raccoon |
|
04:59
🔗
|
|
BlueMaxim has joined #archiveteam-bs |
|
04:59
🔗
|
|
BlueMax has quit IRC (Read error: Connection reset by peer) |
|
05:36
🔗
|
|
Atom__ has joined #archiveteam-bs |
|
05:37
🔗
|
|
Atom-- has quit IRC (Ping timeout: 252 seconds) |
|
05:42
🔗
|
|
d5f4a3622 has quit IRC (Read error: Operation timed out) |
|
05:45
🔗
|
|
Stilettoo has joined #archiveteam-bs |
|
05:49
🔗
|
|
Stiletto has quit IRC (Ping timeout: 604 seconds) |
|
05:51
🔗
|
|
ScruffyB has quit IRC (Remote host closed the connection) |
|
05:51
🔗
|
|
Mayonaise has quit IRC (Read error: Operation timed out) |
|
05:52
🔗
|
|
HashbangI has quit IRC (Read error: Operation timed out) |
|
05:53
🔗
|
|
ndiddy has quit IRC (Read error: Operation timed out) |
|
05:53
🔗
|
|
Mayonaise has joined #archiveteam-bs |
|
05:53
🔗
|
|
ndiddy has joined #archiveteam-bs |
|
05:54
🔗
|
|
luckcolor has quit IRC (Read error: Operation timed out) |
|
05:54
🔗
|
|
luckcolor has joined #archiveteam-bs |
|
05:55
🔗
|
|
phillipsj has joined #archiveteam-bs |
|
05:57
🔗
|
|
c4rc4s has quit IRC (Read error: Operation timed out) |
|
05:57
🔗
|
|
ShellyRol has quit IRC (Read error: Operation timed out) |
|
05:57
🔗
|
|
ShellyRol has joined #archiveteam-bs |
|
05:58
🔗
|
|
Fusl has quit IRC (Excess Flood) |
|
05:58
🔗
|
|
Dragnog2 has joined #archiveteam-bs |
|
05:58
🔗
|
|
Lord_Nigh has quit IRC (Read error: Operation timed out) |
|
05:59
🔗
|
|
Dj-Wawa has joined #archiveteam-bs |
|
05:59
🔗
|
|
TC01 has quit IRC (Read error: Operation timed out) |
|
05:59
🔗
|
|
kiskabak has quit IRC (Read error: Operation timed out) |
|
06:00
🔗
|
|
Lord_Nigh has joined #archiveteam-bs |
|
06:00
🔗
|
|
Fusl has joined #archiveteam-bs |
|
06:00
🔗
|
|
Fusl____ sets mode: +o Fusl |
|
06:00
🔗
|
|
Fusl_ sets mode: +o Fusl |
|
06:01
🔗
|
|
twigfoot has quit IRC (Ping timeout: 612 seconds) |
|
06:01
🔗
|
|
twigfoot has joined #archiveteam-bs |
|
06:01
🔗
|
|
Dj-Wawa_ has quit IRC (Read error: Operation timed out) |
|
06:02
🔗
|
|
TC01 has joined #archiveteam-bs |
|
06:03
🔗
|
|
anarcat has quit IRC (Read error: Connection reset by peer) |
|
06:03
🔗
|
|
systwi has quit IRC (Ping timeout: 612 seconds) |
|
06:04
🔗
|
|
anarcat has joined #archiveteam-bs |
|
06:04
🔗
|
|
Coderjo has joined #archiveteam-bs |
|
06:05
🔗
|
|
Coderjo has quit IRC (Handshake flooding) |
|
06:07
🔗
|
|
jspiros has quit IRC (Read error: Operation timed out) |
|
06:09
🔗
|
|
Coderjo_ has quit IRC (Ping timeout: 612 seconds) |
|
06:09
🔗
|
|
Coderjo has joined #archiveteam-bs |
|
06:11
🔗
|
|
jspiros has joined #archiveteam-bs |
|
06:14
🔗
|
|
pikami_ has quit IRC (Ping timeout: 612 seconds) |
|
06:14
🔗
|
|
pikami has joined #archiveteam-bs |
|
06:14
🔗
|
|
deevious has joined #archiveteam-bs |
|
06:22
🔗
|
|
jspiros has quit IRC (Read error: Operation timed out) |
|
06:23
🔗
|
|
d5f4a3622 has joined #archiveteam-bs |
|
06:30
🔗
|
|
jspiros has joined #archiveteam-bs |
|
06:32
🔗
|
|
HashbangI has joined #archiveteam-bs |
|
06:32
🔗
|
|
c4rc4s has joined #archiveteam-bs |
|
06:32
🔗
|
|
systwi has joined #archiveteam-bs |
|
07:07
🔗
|
|
Yurume has quit IRC (No Ping reply in 180 seconds.) |
|
07:10
🔗
|
|
Yurume has joined #archiveteam-bs |
|
07:10
🔗
|
|
ShellyRol has quit IRC (Ping timeout: 604 seconds) |
|
07:20
🔗
|
|
ShellyRol has joined #archiveteam-bs |
|
07:23
🔗
|
|
m007a83 has joined #archiveteam-bs |
|
07:31
🔗
|
|
godane has joined #archiveteam-bs |
|
08:02
🔗
|
godane |
so i'm going be upload Aerospace America |
|
08:03
🔗
|
godane |
turns out that there website has pdfs going back to 2009 |
|
08:08
🔗
|
|
Dragnog2 has quit IRC (Quit: Connection closed for inactivity) |
|
08:19
🔗
|
|
Maylay_ has quit IRC (Ping timeout: 252 seconds) |
|
08:19
🔗
|
|
ScruffyB has joined #archiveteam-bs |
|
08:20
🔗
|
|
yuitimoth has quit IRC (Ping timeout: 252 seconds) |
|
08:20
🔗
|
|
yuitimoth has joined #archiveteam-bs |
|
08:21
🔗
|
|
JH8813269 has quit IRC (Ping timeout: 252 seconds) |
|
08:23
🔗
|
|
Maylay has joined #archiveteam-bs |
|
08:23
🔗
|
|
phillipsj has quit IRC (Ping timeout: 252 seconds) |
|
08:24
🔗
|
|
anarchat has joined #archiveteam-bs |
|
08:25
🔗
|
|
LeG0ax has joined #archiveteam-bs |
|
08:27
🔗
|
|
bluefoo_ has joined #archiveteam-bs |
|
08:29
🔗
|
|
bluefoo has quit IRC (Read error: Operation timed out) |
|
08:30
🔗
|
|
m007a83 has quit IRC (se.hub irc.underworld.no) |
|
08:30
🔗
|
|
deevious has quit IRC (se.hub irc.underworld.no) |
|
08:30
🔗
|
|
anarcat has quit IRC (se.hub irc.underworld.no) |
|
08:30
🔗
|
|
odemgi_ has quit IRC (se.hub irc.underworld.no) |
|
08:30
🔗
|
|
Ing3b0rg has quit IRC (se.hub irc.underworld.no) |
|
08:30
🔗
|
|
Ganonmast has quit IRC (se.hub irc.underworld.no) |
|
08:30
🔗
|
|
purplebot has quit IRC (se.hub irc.underworld.no) |
|
08:30
🔗
|
|
foureyes has quit IRC (se.hub irc.underworld.no) |
|
08:30
🔗
|
|
yano_ has quit IRC (se.hub irc.underworld.no) |
|
08:30
🔗
|
|
tuluu has quit IRC (se.hub irc.underworld.no) |
|
08:30
🔗
|
|
coderobe has quit IRC (se.hub irc.underworld.no) |
|
08:30
🔗
|
|
kiska has quit IRC (se.hub irc.underworld.no) |
|
08:30
🔗
|
|
i0npulse has quit IRC (se.hub irc.underworld.no) |
|
08:30
🔗
|
|
pew has quit IRC (se.hub irc.underworld.no) |
|
08:30
🔗
|
|
ranma has quit IRC (se.hub irc.underworld.no) |
|
08:30
🔗
|
|
Frogging has quit IRC (se.hub irc.underworld.no) |
|
08:31
🔗
|
|
godane has quit IRC (Ping timeout: 360 seconds) |
|
08:40
🔗
|
|
Yurume has quit IRC (No Ping reply in 180 seconds.) |
|
08:41
🔗
|
|
Yurume has joined #archiveteam-bs |
|
08:43
🔗
|
|
godane has joined #archiveteam-bs |
|
08:46
🔗
|
|
LeG0ax is now known as Ing3b0rg |
|
08:48
🔗
|
|
Ganonmast has joined #archiveteam-bs |
|
08:55
🔗
|
|
m007a83 has joined #archiveteam-bs |
|
08:55
🔗
|
|
odemgi_ has joined #archiveteam-bs |
|
08:55
🔗
|
|
purplebot has joined #archiveteam-bs |
|
08:55
🔗
|
|
foureyes has joined #archiveteam-bs |
|
08:55
🔗
|
|
yano_ has joined #archiveteam-bs |
|
08:55
🔗
|
|
tuluu has joined #archiveteam-bs |
|
08:55
🔗
|
|
coderobe has joined #archiveteam-bs |
|
08:55
🔗
|
|
i0npulse has joined #archiveteam-bs |
|
08:55
🔗
|
|
pew has joined #archiveteam-bs |
|
08:55
🔗
|
|
ranma has joined #archiveteam-bs |
|
08:55
🔗
|
|
Frogging has joined #archiveteam-bs |
|
08:56
🔗
|
|
JH8813269 has joined #archiveteam-bs |
|
09:05
🔗
|
|
deevious has joined #archiveteam-bs |
|
10:15
🔗
|
|
ShellyRol has quit IRC (Read error: Connection reset by peer) |
|
10:18
🔗
|
|
ShellyRol has joined #archiveteam-bs |
|
10:37
🔗
|
|
kiska has joined #archiveteam-bs |
|
10:37
🔗
|
|
Fusl____ sets mode: +o kiska |
|
10:37
🔗
|
|
Fusl sets mode: +o kiska |
|
10:37
🔗
|
|
Fusl_ sets mode: +o kiska |
|
10:43
🔗
|
|
Maylay has quit IRC (Read error: Operation timed out) |
|
11:32
🔗
|
|
Maylay has joined #archiveteam-bs |
|
11:55
🔗
|
|
killsushi has quit IRC (Quit: Leaving) |
|
12:17
🔗
|
JAA |
Soo, turns out that apparently 'ia upload' doesn't have a non-zero exit status when the upload fails. Ugh. The intermediate machine I use for uploading to IA filled up, and there's now a backlog on both that machine and the crawler machine. Still fine for now, and the crawl's unaffected. But as an FYI, never expect the 'ia' tool to behave as you think it should, because it might not. |
|
12:18
🔗
|
JAA |
the picosong crawl* |
|
12:19
🔗
|
JAA |
(No data lost either because fortunately rsync behaves correctly.) |
|
12:34
🔗
|
|
ScruffyB has quit IRC (Remote host closed the connection) |
|
12:34
🔗
|
|
Smiley has quit IRC (Remote host closed the connection) |
|
12:34
🔗
|
|
Smiley has joined #archiveteam-bs |
|
12:34
🔗
|
|
ScruffyB has joined #archiveteam-bs |
|
12:34
🔗
|
|
antomati_ has joined #archiveteam-bs |
|
12:35
🔗
|
|
antomatic has quit IRC (Read error: Operation timed out) |
|
13:47
🔗
|
|
Sokar has quit IRC (Ping timeout: 745 seconds) |
|
14:13
🔗
|
SketchCow |
I'm losing this fight |
|
14:15
🔗
|
SketchCow |
Can archivebot traffic be temporarily away from FOS |
|
14:17
🔗
|
SketchCow |
Too late, I killed rsync |
|
14:17
🔗
|
SketchCow |
Until this machine heals up across a few hours, it's a smoke test for lost materials |
|
14:21
🔗
|
SketchCow |
Oh man, this thing is PEGGED |
|
14:26
🔗
|
kiska |
Oh dear.... |
|
14:33
🔗
|
Igloo |
SketchCow: we can move it away from FOS. We can direct upload from the new place into the collection. |
|
15:04
🔗
|
SketchCow |
300gb free on FOS for now |
|
15:04
🔗
|
SketchCow |
Going to try to get it to a tb, then will turn on RSYNC again |
|
15:04
🔗
|
JAA |
We're draining some data to another machine now as well. |
|
15:04
🔗
|
JAA |
From the AB pipelines, that is. |
|
15:05
🔗
|
SketchCow |
Something makes archivebot stuff choke and then it lives on the machine |
|
15:05
🔗
|
SketchCow |
I don't have a routine to go through and re-run the ones that died, I should |
|
15:05
🔗
|
SketchCow |
So over time, it just slowly builds. |
|
15:05
🔗
|
SketchCow |
I wrote a "surgical" one-off uploader that finishes dead jobs, but I only run it once in a while. |
|
15:06
🔗
|
SketchCow |
But, like, for the record, 62 are sitting in the outbox. |
|
15:06
🔗
|
SketchCow |
I see a lot are 150gb |
|
15:06
🔗
|
SketchCow |
Is that the upper limit we put on archivebot stuff now? |
|
15:07
🔗
|
Igloo |
https://github.com/ArchiveTeam/ArchiveBot/blob/master/pipeline/pipeline.py#L58 I believe they should be around 5G each |
|
15:08
🔗
|
JAA |
150 GB WARCs or directories/items? |
|
15:08
🔗
|
JAA |
Such WARCs should be extremely rare. |
|
15:08
🔗
|
SketchCow |
I mean the cutoff being 150gb |
|
15:09
🔗
|
SketchCow |
We probably made that decision and I signed off on it, years ago. |
|
15:09
🔗
|
JAA |
Yeah, probably. |
|
15:19
🔗
|
SketchCow |
I told "Surgical" to do the untouched directories. |
|
15:20
🔗
|
SketchCow |
https://archive.org/details/archivebot looks sweet now, by the way - aiming the screenshotting at "most viewed" really prioritized the right ones. |
|
15:22
🔗
|
SketchCow |
350gb |
|
15:22
🔗
|
SketchCow |
Let's not call it out of the woods (I don't have rsync on yet) but at least the trend is upwards. |
|
15:22
🔗
|
SketchCow |
I'm wondering if the massive amount of chrome-headless I've been causing caused this |
|
15:43
🔗
|
|
ReimuHaku has quit IRC (Quit: http://quassel-irc.org - Chat comfortably. Anywhere.) |
|
15:59
🔗
|
|
ReimuHaku has joined #archiveteam-bs |
|
16:23
🔗
|
|
Hoolootwo is now known as Hooloovoo |
|
16:37
🔗
|
|
ReimuHaku has quit IRC (Ping timeout: 492 seconds) |
|
16:37
🔗
|
|
ReimuHaku has joined #archiveteam-bs |
|
16:37
🔗
|
|
ReimuHaku has quit IRC (Handshake flooding) |
|
16:37
🔗
|
|
ReimuHaku has joined #archiveteam-bs |
|
16:37
🔗
|
|
ReimuHaku has quit IRC (Handshake flooding) |
|
16:37
🔗
|
|
ReimuHaku has joined #archiveteam-bs |
|
16:37
🔗
|
|
ReimuHaku has quit IRC (Handshake flooding) |
|
16:37
🔗
|
|
ReimuHaku has joined #archiveteam-bs |
|
16:37
🔗
|
|
ReimuHaku has quit IRC (Handshake flooding) |
|
16:57
🔗
|
|
Dragnog2 has joined #archiveteam-bs |
|
17:13
🔗
|
|
BlueMaxim has quit IRC (Quit: Leaving) |
|
17:58
🔗
|
|
d5f4a3622 has quit IRC (Read error: Connection reset by peer) |
|
17:58
🔗
|
|
af10b3e5e has joined #archiveteam-bs |
|
18:03
🔗
|
SketchCow |
989b |
|
18:03
🔗
|
SketchCow |
989gb |
|
18:13
🔗
|
|
Stilettoo has quit IRC (Read error: Operation timed out) |
|
18:21
🔗
|
|
Stiletto has joined #archiveteam-bs |
|
18:56
🔗
|
Raccoon |
* |
|
18:58
🔗
|
JAA |
Raccoon: I'm not sure really. There's the warrior of course, but that won't use much bandwidth most of the time. Maybe if someone could revive the FTP archival project (#effteepee), that might be a good option. It's been dead for years now though. |
|
18:58
🔗
|
JAA |
Also, any of that would be more or less symmetric traffic of course. |
|
18:58
🔗
|
Raccoon |
what about torrent seeds |
|
18:58
🔗
|
JAA |
Yeah, you could seed some IA items. |
|
18:58
🔗
|
JAA |
I have no idea which ones are popular though. |
|
18:59
🔗
|
Raccoon |
is there a top-down step by step handhold guide |
|
18:59
🔗
|
Raccoon |
where to get torrent/magnet/rss links |
|
18:59
🔗
|
JAA |
Also, IA torrents are broken for large items, and we no longer generate them on new uploads. |
|
18:59
🔗
|
JAA |
The .torrent file is within each item. |
|
18:59
🔗
|
JAA |
(If it exists, that is.) |
|
19:00
🔗
|
Raccoon |
right. but i mean for a 'set it and forget it' torrent client that accepts all modes of tracker/http/rss monitor links. |
|
19:00
🔗
|
Raccoon |
IA would need to have a news feed it updates on their end |
|
19:00
🔗
|
JAA |
Not sure if that exists. I remember something that showed IA items most in need of seeds, but I don't know where that was. |
|
19:01
🔗
|
Raccoon |
if you know who to poke, i'll take orders |
|
19:02
🔗
|
JAA |
I think it used to be here, but that seems to be broken now: https://bt1.archive.org/hotlist.php |
|
19:03
🔗
|
JAA |
No idea who to ask. Maybe there's something on the IA forums. |
|
19:03
🔗
|
Raccoon |
do no clouted IA reps hang out here? |
|
19:05
🔗
|
JAA |
You could certainly try emailing info@archive.org and ask whether there's a replacement for that hotlist.php or something else you could use like that. |
|
19:05
🔗
|
Raccoon |
and do you know if IA does any sort of bandwith redistribution DNS round robining or some other fancy peer-based CDN |
|
19:06
🔗
|
JAA |
The hot list is still mentioned in the help at https://help.archive.org/hc/en-us/articles/360004715251-Archive-BitTorrents by the way (but not linked). |
|
19:06
🔗
|
Raccoon |
mirrors* |
|
19:06
🔗
|
JAA |
For item downloads? Nope, they just go directly to the servers that store the data as far as I know. |
|
19:07
🔗
|
Raccoon |
thanks |
|
19:49
🔗
|
|
schbirid has quit IRC (Remote host closed the connection) |
|
20:15
🔗
|
kiska |
I was going to suggest rsync target... but then I realised you didn't want saturation |
|
20:15
🔗
|
kiska |
You could probably do ia.bak I guess |
|
20:15
🔗
|
Raccoon |
I would only dedicate 4 to 10 TB, also |
|
20:16
🔗
|
Raccoon |
it would be nice if I could specifically support things I believe in, like old time radio etc |
|
20:24
🔗
|
|
Sokar has joined #archiveteam-bs |
|
20:46
🔗
|
|
Dragnog2 has quit IRC (Quit: Connection closed for inactivity) |
|
21:05
🔗
|
|
DogsRNice has joined #archiveteam-bs |
|
21:08
🔗
|
JAA |
picosong update after about 52 hours: 191k of 481k items done. Running quite smoothly still. I can't provide any numbers on the size anymore because of the upload problems earlier. (By the way, 'ia upload' does exit with status 1 if something goes wrong. There was another problem.) |
|
21:14
🔗
|
JAA |
Nevermind, fixed my size calculation command. I just passed 1 TiB of data a few minutes ago. |
|
23:16
🔗
|
|
Atom-- has joined #archiveteam-bs |
|
23:19
🔗
|
|
Fusl has quit IRC (Ping timeout: 264 seconds) |
|
23:19
🔗
|
|
BlueMax has joined #archiveteam-bs |
|
23:19
🔗
|
|
Fusl has joined #archiveteam-bs |
|
23:19
🔗
|
|
Fusl____ sets mode: +o Fusl |
|
23:19
🔗
|
|
Fusl_ sets mode: +o Fusl |
|
23:20
🔗
|
|
Atom__ has quit IRC (Read error: Operation timed out) |
|
23:21
🔗
|
|
Pixi` has joined #archiveteam-bs |
|
23:22
🔗
|
|
Pixi has quit IRC (Read error: Operation timed out) |
|
23:22
🔗
|
|
chfoo has quit IRC (Ping timeout: 360 seconds) |
|
23:23
🔗
|
|
chfoo has joined #archiveteam-bs |
|
23:23
🔗
|
|
svchfoo1 sets mode: +o chfoo |
|
23:23
🔗
|
|
Fusl____ sets mode: +o chfoo |
|
23:23
🔗
|
|
Fusl sets mode: +o chfoo |
|
23:23
🔗
|
|
Fusl_ sets mode: +o chfoo |
|
23:23
🔗
|
|
svchfoo3 sets mode: +o chfoo |
|
23:27
🔗
|
|
twigfoot has quit IRC (Ping timeout: 360 seconds) |
|
23:27
🔗
|
|
voltagex has quit IRC (Ping timeout: 360 seconds) |
|
23:29
🔗
|
|
voltagex has joined #archiveteam-bs |
|
23:30
🔗
|
|
twigfoot has joined #archiveteam-bs |
|
23:39
🔗
|
|
Pixi has joined #archiveteam-bs |
|
23:40
🔗
|
|
zino_ has quit IRC (Ping timeout: 360 seconds) |
|
23:42
🔗
|
|
zino_ has joined #archiveteam-bs |
|
23:43
🔗
|
|
superkuh_ has quit IRC (Excess Flood) |
|
23:44
🔗
|
|
superkuh_ has joined #archiveteam-bs |
|
23:47
🔗
|
|
jrwr has quit IRC (Ping timeout: 264 seconds) |
|
23:48
🔗
|
|
jrwr has joined #archiveteam-bs |
|
23:48
🔗
|
|
Fusl sets mode: +o jrwr |
|
23:48
🔗
|
|
Fusl____ sets mode: +o jrwr |
|
23:48
🔗
|
|
Fusl_ sets mode: +o jrwr |
|
23:50
🔗
|
|
Pixi` has quit IRC (Read error: Operation timed out) |
|
23:56
🔗
|
|
VADemon has joined #archiveteam-bs |
|
23:58
🔗
|
|
superkuh_ has quit IRC (Excess Flood) |
|
23:58
🔗
|
|
superkuh_ has joined #archiveteam-bs |