| Time |
Nickname |
Message |
|
00:05
🔗
|
|
maseck has joined #archiveteam |
|
00:10
🔗
|
SketchCow |
I see 1.7tb of Gamefront |
|
00:10
🔗
|
|
maseck_ has quit IRC (Read error: Operation timed out) |
|
00:47
🔗
|
|
philpem has quit IRC (Ping timeout: 260 seconds) |
|
01:34
🔗
|
|
WinterFox has joined #archiveteam |
|
02:02
🔗
|
|
JesseW has joined #archiveteam |
|
02:56
🔗
|
|
tomwsmf-a has quit IRC (Ping timeout: 258 seconds) |
|
03:22
🔗
|
|
bwn has quit IRC (Ping timeout: 492 seconds) |
|
03:25
🔗
|
|
brayden_ has quit IRC (Read error: Connection reset by peer) |
|
03:25
🔗
|
|
brayden_ has joined #archiveteam |
|
03:25
🔗
|
|
swebb sets mode: +o brayden_ |
|
03:46
🔗
|
|
vitzli has joined #archiveteam |
|
03:47
🔗
|
|
morbus_ has quit IRC (Read error: Operation timed out) |
|
03:49
🔗
|
|
JKa has joined #archiveteam |
|
03:50
🔗
|
|
JKa has quit IRC (Client Quit) |
|
04:22
🔗
|
|
vitzli has quit IRC (Quit: Leaving) |
|
04:36
🔗
|
|
BlueMaxim has quit IRC (Read error: Operation timed out) |
|
04:37
🔗
|
|
BlueMaxim has joined #archiveteam |
|
04:45
🔗
|
|
Sk1d has quit IRC (Ping timeout: 194 seconds) |
|
04:50
🔗
|
|
Sk1d has joined #archiveteam |
|
04:58
🔗
|
|
JesseW has quit IRC (Ping timeout: 370 seconds) |
|
05:07
🔗
|
|
BartoCH has quit IRC (Ping timeout: 260 seconds) |
|
05:08
🔗
|
|
Honno has joined #archiveteam |
|
05:14
🔗
|
|
Emcy has quit IRC (Read error: Connection reset by peer) |
|
05:14
🔗
|
|
BartoCH has joined #archiveteam |
|
05:16
🔗
|
|
Emcy has joined #archiveteam |
|
06:09
🔗
|
|
bwn has joined #archiveteam |
|
06:33
🔗
|
|
fie has quit IRC (Read error: Connection reset by peer) |
|
06:33
🔗
|
|
fie has joined #archiveteam |
|
06:34
🔗
|
|
vitzli has joined #archiveteam |
|
06:51
🔗
|
|
schbirid has joined #archiveteam |
|
06:54
🔗
|
|
Honno has quit IRC (Read error: Operation timed out) |
|
06:58
🔗
|
|
Medowar has joined #archiveteam |
|
07:09
🔗
|
|
Emcy has quit IRC (Read error: Operation timed out) |
|
07:51
🔗
|
|
atomotic has joined #archiveteam |
|
07:57
🔗
|
|
vitzli has quit IRC (Leaving) |
|
07:59
🔗
|
|
metalcamp has joined #archiveteam |
|
08:19
🔗
|
|
Emcy has joined #archiveteam |
|
09:09
🔗
|
|
espes__ has quit IRC (Ping timeout: 244 seconds) |
|
09:33
🔗
|
|
espes__ has joined #archiveteam |
|
09:41
🔗
|
|
toad2 has joined #archiveteam |
|
09:42
🔗
|
|
toad1 has quit IRC (Read error: Operation timed out) |
|
09:44
🔗
|
|
Peetz0r has joined #archiveteam |
|
09:45
🔗
|
|
bwn has quit IRC (Read error: Operation timed out) |
|
09:59
🔗
|
|
Medowar has quit IRC (Quit: Connection closed for inactivity) |
|
10:05
🔗
|
|
bwn has joined #archiveteam |
|
10:20
🔗
|
|
atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) |
|
10:28
🔗
|
|
bwn has quit IRC (ny.us.hub west.us.hub) |
|
10:28
🔗
|
|
Emcy has quit IRC (ny.us.hub west.us.hub) |
|
10:28
🔗
|
|
fmope has quit IRC (ny.us.hub west.us.hub) |
|
10:28
🔗
|
|
mr-b has quit IRC (ny.us.hub west.us.hub) |
|
10:28
🔗
|
|
Atom__ has quit IRC (ny.us.hub west.us.hub) |
|
10:28
🔗
|
|
Frogging has quit IRC (ny.us.hub west.us.hub) |
|
10:28
🔗
|
|
jspiros has quit IRC (ny.us.hub west.us.hub) |
|
10:28
🔗
|
|
acridAxid has quit IRC (ny.us.hub west.us.hub) |
|
10:28
🔗
|
|
yakfish has quit IRC (ny.us.hub west.us.hub) |
|
10:28
🔗
|
|
SirCmpwn has quit IRC (ny.us.hub west.us.hub) |
|
10:28
🔗
|
|
SadDM has quit IRC (ny.us.hub west.us.hub) |
|
10:28
🔗
|
|
matthusby has quit IRC (ny.us.hub west.us.hub) |
|
10:28
🔗
|
|
zenguy has quit IRC (ny.us.hub west.us.hub) |
|
10:28
🔗
|
|
Peetz0r has quit IRC (ny.us.hub west.us.hub) |
|
10:28
🔗
|
|
goekesmi has quit IRC (ny.us.hub west.us.hub) |
|
10:28
🔗
|
|
Stiletto has quit IRC (ny.us.hub west.us.hub) |
|
10:28
🔗
|
|
d_rebel has quit IRC (ny.us.hub west.us.hub) |
|
10:30
🔗
|
|
toad2 has quit IRC (Read error: Operation timed out) |
|
10:31
🔗
|
|
toad1 has joined #archiveteam |
|
10:37
🔗
|
|
HCross has quit IRC (Read error: Connection reset by peer) |
|
10:43
🔗
|
|
bwn has joined #archiveteam |
|
10:43
🔗
|
|
Peetz0r has joined #archiveteam |
|
10:43
🔗
|
|
Emcy has joined #archiveteam |
|
10:43
🔗
|
|
goekesmi has joined #archiveteam |
|
10:43
🔗
|
|
fmope has joined #archiveteam |
|
10:43
🔗
|
|
Stiletto has joined #archiveteam |
|
10:43
🔗
|
|
d_rebel has joined #archiveteam |
|
10:43
🔗
|
|
mr-b has joined #archiveteam |
|
10:43
🔗
|
|
Atom__ has joined #archiveteam |
|
10:43
🔗
|
|
Frogging has joined #archiveteam |
|
10:43
🔗
|
|
jspiros has joined #archiveteam |
|
10:43
🔗
|
|
acridAxid has joined #archiveteam |
|
10:43
🔗
|
|
yakfish has joined #archiveteam |
|
10:43
🔗
|
|
SirCmpwn has joined #archiveteam |
|
10:43
🔗
|
|
SadDM has joined #archiveteam |
|
10:43
🔗
|
|
matthusby has joined #archiveteam |
|
10:43
🔗
|
|
zenguy has joined #archiveteam |
|
10:43
🔗
|
|
west.us.hub sets mode: +o SadDM |
|
10:43
🔗
|
|
swebb sets mode: +o SadDM |
|
10:43
🔗
|
|
atomotic has joined #archiveteam |
|
10:54
🔗
|
|
Morbus has joined #archiveteam |
|
10:59
🔗
|
|
hawc145 has joined #archiveteam |
|
10:59
🔗
|
|
metal_cam has joined #archiveteam |
|
11:01
🔗
|
|
metalcamp has quit IRC (Ping timeout: 244 seconds) |
|
11:04
🔗
|
|
BlueMaxim has quit IRC (Quit: Leaving) |
|
11:05
🔗
|
|
hawc145 is now known as HCross |
|
11:07
🔗
|
|
metal_cam is now known as metalcamp |
|
11:15
🔗
|
|
fie has quit IRC (Read error: Connection reset by peer) |
|
11:15
🔗
|
|
fie has joined #archiveteam |
|
11:16
🔗
|
|
jspiros has quit IRC (leaving) |
|
11:22
🔗
|
|
toad1 has quit IRC (Read error: Operation timed out) |
|
11:26
🔗
|
|
Medowar has joined #archiveteam |
|
11:38
🔗
|
|
toad1 has joined #archiveteam |
|
11:45
🔗
|
|
jspiros has joined #archiveteam |
|
11:58
🔗
|
|
jspiros has quit IRC (Remote host closed the connection) |
|
12:05
🔗
|
|
jspiros has joined #archiveteam |
|
12:09
🔗
|
|
jspiros has quit IRC (Client Quit) |
|
12:15
🔗
|
|
jspiros has joined #archiveteam |
|
12:26
🔗
|
|
VADemon has joined #archiveteam |
|
13:11
🔗
|
arkiver |
We need a lot more grabbers on GameFront. |
|
13:11
🔗
|
|
tomwsmf-a has joined #archiveteam |
|
13:12
🔗
|
arkiver |
If you can, please start something up |
|
13:12
🔗
|
HCross |
let me see what I can do |
|
13:12
🔗
|
arkiver |
Great! |
|
13:12
🔗
|
arkiver |
These items are single files, so they should not be very big |
|
13:13
🔗
|
arkiver |
And I haven't seen any limiting from GameFront's side |
|
13:13
🔗
|
|
yipdw_ has quit IRC (Read error: Operation timed out) |
|
13:14
🔗
|
|
Infreq has quit IRC (Read error: Operation timed out) |
|
13:14
🔗
|
|
Infreq has joined #archiveteam |
|
13:15
🔗
|
atrocity |
arkiver: all i'm getting are these: 'Tracker rate limiting is active. We don't want to overload the site we're archiving, so we've limited the number of downloads per minute. Retrying after 300 seconds...' |
|
13:16
🔗
|
atrocity |
for game front |
|
13:16
🔗
|
arkiver |
I guess that's on the forums grab? |
|
13:16
🔗
|
HCross |
GameFront or the forums arkiver? |
|
13:16
🔗
|
arkiver |
Normal GameFront |
|
13:16
🔗
|
atrocity |
whatever the recommended project is |
|
13:16
🔗
|
|
yipdw has joined #archiveteam |
|
13:16
🔗
|
atrocity |
has 570k+ items in it |
|
13:16
🔗
|
arkiver |
I just set it to normal gamefront |
|
13:16
🔗
|
atrocity |
both show the normal gamefrom logo, lol |
|
13:17
🔗
|
arkiver |
yeah |
|
13:17
🔗
|
atrocity |
should i manually switch to gamefront then? |
|
13:18
🔗
|
arkiver |
chfoo: can you please create these 3 targets on FOS: bayimg, corbisimages, experienceproject |
|
13:18
🔗
|
arkiver |
atrocity: I just switched default to the normal gamefront, so it should automatically update the project soon |
|
13:19
🔗
|
atrocity |
i switched it, lol. i set ti to 6 concurrent, too |
|
13:19
🔗
|
phuzion |
arkiver: Want 30-40 DO instances for GameFront? |
|
13:20
🔗
|
arkiver |
phuzion: would be great! |
|
13:20
🔗
|
phuzion |
on it, gimme 5-10 |
|
13:20
🔗
|
arkiver |
yes! Thanks! |
|
13:20
🔗
|
arkiver |
phuzion: you might want to make sure they have access first |
|
13:20
🔗
|
phuzion |
Spin one and try it? |
|
13:20
🔗
|
arkiver |
yeah |
|
13:20
🔗
|
arkiver |
the script will let you know if you're banned |
|
13:20
🔗
|
phuzion |
Sure. Are they banning IPs? |
|
13:21
🔗
|
arkiver |
Yeah, not individial IPS, but countries |
|
13:21
🔗
|
arkiver |
individual* |
|
13:22
🔗
|
phuzion |
Recommend a region? NYC? |
|
13:22
🔗
|
HCross |
London works |
|
13:22
🔗
|
phuzion |
lon1 it is |
|
13:22
🔗
|
arkiver |
This page has a list of countries that aren't banned http://www.gamefront.com/file-hosting-faq/ |
|
13:26
🔗
|
phuzion |
Deploying 1 droplet now. |
|
13:26
🔗
|
arkiver |
ok! |
|
13:26
🔗
|
arkiver |
Make sure you start it with the 'normal' gamefront, and not the gamefrontforums |
|
13:27
🔗
|
phuzion |
I'm running the gamefront-grab project, so yeah |
|
13:27
🔗
|
arkiver |
ok |
|
13:29
🔗
|
|
tomwsmf-a has quit IRC (Read error: Operation timed out) |
|
13:35
🔗
|
HCross |
20 new instances have just appeared |
|
13:37
🔗
|
HCross |
No HTTP response received from tracker. The tracker is probably overloaded. Retrying after 150 seconds... oops |
|
13:37
🔗
|
arkiver |
hmm |
|
13:38
🔗
|
arkiver |
I'll have a look at that |
|
13:38
🔗
|
arkiver |
RuTracker is restarted. New pages are in |
|
13:41
🔗
|
|
atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) |
|
13:47
🔗
|
arkiver |
virginmedia scripts updated and items requeued for final run. |
|
13:47
🔗
|
HCross |
Ill warm that up as well |
|
13:47
🔗
|
arkiver |
it's only 100 items or so |
|
13:47
🔗
|
Medowar |
35 script units + 18 warrior units running for gf |
|
13:48
🔗
|
arkiver |
I saw that, awesome! :) |
|
13:48
🔗
|
Medowar |
Uploading with Rsync to rsync://fos.textfiles.com/chfoo/warrior/gamefront/Medowar/ |
|
13:48
🔗
|
Medowar |
Starting RsyncUpload for Item singlefile:6037760 |
|
13:48
🔗
|
Medowar |
rsync: failed to connect to fos.textfiles.com (208.70.31.74): Connection refused (111) |
|
13:48
🔗
|
Medowar |
is that me? |
|
13:49
🔗
|
HCross |
more like FOS cant cope |
|
13:49
🔗
|
MrRadar |
FOS could be hitting its max connection limit |
|
13:49
🔗
|
MrRadar |
If it doesn't clear up after a while then it might be you |
|
13:49
🔗
|
arkiver |
Medowar: is it on everything or random items? |
|
13:50
🔗
|
Medowar |
not sure, the warrior just startet |
|
13:50
🔗
|
Medowar |
yep, it is me |
|
13:50
🔗
|
Medowar |
what port is rsync using from the warrior? |
|
13:52
🔗
|
HCross |
arkiver, just pointed 20 concurrent at Virgin from within their network |
|
13:55
🔗
|
|
Honno has joined #archiveteam |
|
13:56
🔗
|
Medowar |
fixed, now up and running |
|
13:57
🔗
|
HCross |
40 Virgin concurrent from me now |
|
13:58
🔗
|
Medowar |
overkill... |
|
14:00
🔗
|
|
VADemon has quit IRC (Quit: left4dead) |
|
14:01
🔗
|
|
brayden_ has quit IRC (Read error: Connection reset by peer) |
|
14:02
🔗
|
|
brayden has joined #archiveteam |
|
14:02
🔗
|
|
swebb sets mode: +o brayden |
|
14:03
🔗
|
|
Honno has quit IRC (Quit: Leaving) |
|
14:23
🔗
|
atrocity |
uh |
|
14:23
🔗
|
atrocity |
the gamefront one is asking me if i'm banned. this normal? |
|
14:24
🔗
|
atrocity |
The link to the download did not return status code 200. Are you banned? (This error might also be due to a problem on GameFront's side.) |
|
14:28
🔗
|
MrRadar |
atrocity: If the CheckIP step fails then you are banned |
|
14:28
🔗
|
midas |
^ that |
|
14:28
🔗
|
MrRadar |
Otherwise the download is most likely broken on GameFront's end |
|
14:28
🔗
|
phuzion |
My droplet in lon1 is banned |
|
14:31
🔗
|
phuzion |
Trying again in nyc1 |
|
14:34
🔗
|
HCross |
ivan`, Mark just replied on LET to you, like a complete idiot |
|
14:35
🔗
|
arkiver |
Yeah, some downloads are broken on gamefront's end |
|
14:35
🔗
|
arkiver |
In this case for example a 404 on a download URL can mean two thing |
|
14:35
🔗
|
arkiver |
The file really gone, or the file hasn't loaded yet for the download |
|
14:43
🔗
|
phuzion |
Bringing up 40 instances for gamefront |
|
14:45
🔗
|
|
SilSte has quit IRC (Ping timeout: 633 seconds) |
|
14:46
🔗
|
atrocity |
ahh, kk. yeah, my other ones are donwloading fine, so doubt it's banned. thanks for the clarification |
|
14:47
🔗
|
arkiver |
As soon as we have a rsync target on FOS the BayImg grab will be started. |
|
14:47
🔗
|
arkiver |
Scripts are on github |
|
14:48
🔗
|
phuzion |
ok |
|
14:49
🔗
|
phuzion |
Any idea on how hard we're gonna be hitting that one? |
|
14:50
🔗
|
arkiver |
No idea |
|
14:50
🔗
|
arkiver |
Put some resources on it and I'll raise the limit slowly |
|
14:50
🔗
|
phuzion |
I'll get my script started then with 2 concurrent at first, let me know if we should bump concurrency |
|
14:52
🔗
|
arkiver |
ok! |
|
14:57
🔗
|
|
WinterFox has quit IRC (Remote host closed the connection) |
|
15:00
🔗
|
HCross |
arkiver, can you check my PR on gamefront please |
|
15:00
🔗
|
arkiver |
yes |
|
15:00
🔗
|
arkiver |
merged! |
|
15:02
🔗
|
HCross |
Thanks |
|
15:04
🔗
|
* |
phuzion checks PR to make sure it won't require a redeploy of the droplets |
|
15:05
🔗
|
phuzion |
Nope, looks good. |
|
15:05
🔗
|
HCross |
just a documentation change |
|
15:05
🔗
|
phuzion |
Oh jeez, looks like I'm still doing the initial droplet yum update |
|
15:06
🔗
|
phuzion |
If Gamefront bans this /24, almost my entire collection of droplets is screwed, lol |
|
15:07
🔗
|
phuzion |
34/40 of the droplets I deployed ended up in the same /24 |
|
15:09
🔗
|
|
tomwsmf-a has joined #archiveteam |
|
15:19
🔗
|
midas |
lol |
|
15:24
🔗
|
|
atomotic has joined #archiveteam |
|
15:34
🔗
|
Rotab |
for some reason im banned without even participating in this :) |
|
15:36
🔗
|
phuzion |
Rotab: Gamefront bans arbitrary IP blocks and entire countries for no really apparent reason |
|
15:36
🔗
|
Rotab |
lol |
|
15:41
🔗
|
|
vitzli has joined #archiveteam |
|
15:46
🔗
|
|
JesseW has joined #archiveteam |
|
16:24
🔗
|
|
JesseW has quit IRC (Ping timeout: 370 seconds) |
|
16:30
🔗
|
HCross |
ivan`, can you let me know when you see my PM please? |
|
16:40
🔗
|
|
philpem has joined #archiveteam |
|
16:48
🔗
|
chfoo |
arkiver, ok done |
|
16:51
🔗
|
|
VADemon has joined #archiveteam |
|
16:53
🔗
|
SketchCow |
I'll TRY and stay top of the space usage. |
|
16:55
🔗
|
ivan` |
HCross: got it, thanks |
|
17:03
🔗
|
|
atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) |
|
17:09
🔗
|
|
tomwsmf-a has quit IRC (Read error: Operation timed out) |
|
17:24
🔗
|
|
Honno has joined #archiveteam |
|
17:24
🔗
|
arkiver |
chfoo: thank you! |
|
17:25
🔗
|
arkiver |
SketchCow: do you think we can get more connections on FOS? |
|
17:26
🔗
|
SketchCow |
Machine is at 41% full. I've seen it wavering between 37% and 41% |
|
17:26
🔗
|
SketchCow |
I can add connections but the machine will slow down |
|
17:26
🔗
|
SketchCow |
Load average is at 97 |
|
17:27
🔗
|
arkiver |
It might be better to use an other target in that case |
|
17:27
🔗
|
arkiver |
zino, can we use your target for gamefront for a while? |
|
17:27
🔗
|
SketchCow |
Why are we grabbing bayimg and corbis, anyway. |
|
17:27
🔗
|
arkiver |
for bayimg: https://torrentfreak.com/pirate-bays-image-hosting-site-bayimg-returns-for-a-bit-160425/ |
|
17:28
🔗
|
arkiver |
And corbisimages is shutting down too. It's not as high priority to grab as other sites, but I'd like to get it anyway |
|
17:30
🔗
|
SketchCow |
Archivebot channel is fine - 29gb there. |
|
17:34
🔗
|
zino |
arkiver: Sure. Doesn't look like I will have time to set up a new machine for at least two weeks. One moment and I'll set up a target. |
|
17:34
🔗
|
arkiver |
thanks! |
|
17:34
🔗
|
arkiver |
That'll be on the machine we used before right? |
|
17:34
🔗
|
zino |
Yea. |
|
17:38
🔗
|
arkiver |
FOS seems to be recovering a bit, no more number of connection problems at the moment |
|
17:39
🔗
|
arkiver |
SketchCow: bayimg is now incoming |
|
17:40
🔗
|
zino |
arkiver: Try now. Same host as before (eldrimner), but rsyncd module GameFront |
|
17:41
🔗
|
arkiver |
zino: files should now be coming in |
|
17:42
🔗
|
zino |
Not yet at least. Did you double check that you could write first. I have messed that up before. :-P |
|
17:42
🔗
|
SketchCow |
If it's significant, it will flood FOS, is all I'm saying. |
|
17:42
🔗
|
MrRadar |
arkiver: A bayimg item just failed on me with an rsync error. (Max connections reached (10)) |
|
17:42
🔗
|
arkiver |
yeah |
|
17:42
🔗
|
arkiver |
gamefront is now off of FOS |
|
17:42
🔗
|
arkiver |
bayimg is on FOS |
|
17:43
🔗
|
SketchCow |
2.2tb of stuff in the outgoing |
|
17:43
🔗
|
SketchCow |
The big thing is if I turn around and suddenly there's new stuff hitting FOS and I don't have a pump going for it. That's when we really lose it. |
|
17:44
🔗
|
MrRadar |
Two more bayimg items just failed due to rsync. I'm going to stop my scripts for now |
|
17:44
🔗
|
arkiver |
MrRadar: I don't think you have to |
|
17:44
🔗
|
arkiver |
It will recover |
|
17:44
🔗
|
MrRadar |
Are the items supposed to abort when rsync fails? |
|
17:44
🔗
|
arkiver |
And I can pause the tracker if it needs to be paused |
|
17:45
🔗
|
MrRadar |
Without retrying the upload |
|
17:45
🔗
|
MrRadar |
That's what I'm seeing |
|
17:45
🔗
|
SketchCow |
So, here's a problem I'd like addressed, maybe |
|
17:45
🔗
|
SketchCow |
Sometimes people, in their incredible fucking brilliance, go "huh, FOS is slow" |
|
17:46
🔗
|
SketchCow |
And then go "well, the thing to do is open 145,203,845 simultaneous connections" |
|
17:46
🔗
|
arkiver |
MrRadar: I see, grab is paused. |
|
17:46
🔗
|
SketchCow |
I don't have any way to tell who is who, they all just upload. |
|
17:47
🔗
|
MrRadar |
Yeah, short of upgrading FOS to SSDs I don't see how that issue could be solved |
|
17:48
🔗
|
arkiver |
That's why gamefront is now moved off of FOS |
|
17:48
🔗
|
SketchCow |
The thing to do is move FOS's functionality to a more powerful machine to relate the fact that it's veering into critical infrastructure at the archive. |
|
17:48
🔗
|
arkiver |
It would be great if that can be done! |
|
17:49
🔗
|
SketchCow |
I agree! |
|
17:49
🔗
|
SketchCow |
And what comes out of my mouth next would not be a joy to listen to! |
|
17:49
🔗
|
SketchCow |
So I'm not going to do that! |
|
17:49
🔗
|
SketchCow |
And instead get lunch and figure out how I will pay my bills this month. |
|
17:50
🔗
|
zino |
Sounds like the correct prioritisation. |
|
17:50
🔗
|
MrRadar |
What kind of hardware would a proper FOS replacement need? |
|
17:51
🔗
|
arkiver |
zino: did you get the PM? |
|
17:51
🔗
|
zino |
arkiver: Yep, switching to PM tab now. :) |
|
17:51
🔗
|
arkiver |
:) |
|
17:52
🔗
|
HCross |
Umm. Unknown module 'gamefront' |
|
17:52
🔗
|
arkiver |
Yeah, that's what that PM is about |
|
17:53
🔗
|
arkiver |
GameFront is back up and running. |
|
17:55
🔗
|
zino |
Note: eldrimner only has 9T free. Another 8T or so should clear up in a few days when the ftpgrab has finished uploading to IA. |
|
17:57
🔗
|
|
n00b111 has joined #archiveteam |
|
17:57
🔗
|
arkiver |
I think 9TB should be enough for what's remaining of gamefront |
|
17:58
🔗
|
n00b111 |
Can anyone point me to a way to access old material on archive.org that is now blocked by the dreaded robots.txt? |
|
17:59
🔗
|
SketchCow |
You're going to need a time machine and a ham sandwich |
|
17:59
🔗
|
MrRadar |
If it was scraped by us you can download the raw scrape data (check if the site is on our wiki or on the ArchiveBot viewer http://archive.fart.website/archivebot/viewer/ ) |
|
17:59
🔗
|
MrRadar |
Otherwise what SketchCow said |
|
18:00
🔗
|
n00b111 |
I can do the ham sandwich easy enough. |
|
18:00
🔗
|
SketchCow |
Exactly |
|
18:00
🔗
|
SketchCow |
Eat the ham sandwich to make yourself feel better about not having a time machine |
|
18:01
🔗
|
n00b111 |
I'm on the west coast, so I'm going to need to wait an hour before I get lunch. |
|
18:01
🔗
|
n00b111 |
it's for an artcile I am writing about tech and police. I'm looking for any old cached versions of "studiooccupy.org" |
|
18:02
🔗
|
n00b111 |
damn, nothing in the fart |
|
18:03
🔗
|
MrRadar |
Yeah, sorry |
|
18:03
🔗
|
MrRadar |
You might also try checking archive.is or WebCite |
|
18:04
🔗
|
n00b111 |
These things happen. |
|
18:06
🔗
|
n00b111 |
I should have taken it when I had the chance, but foolishly, I did not know robots were retroactive. |
|
18:07
🔗
|
MrRadar |
That's the IA's policy. If someone removes the robots.txt entry in the future, though, the site will becoem available again from archive.org |
|
18:07
🔗
|
SketchCow |
http://www.studiooccupy.org/robots.txt |
|
18:07
🔗
|
SketchCow |
It doesn't have one. |
|
18:08
🔗
|
SketchCow |
http://web.archive.org/web/*/http://www.studiooccupy.org/ |
|
18:08
🔗
|
arkiver |
In the old version of the site some other website is embedded, which has a bad robots.txt |
|
18:08
🔗
|
arkiver |
https://web.archive.org/web/20120221032357/http://studiooccupy.org/ |
|
18:08
🔗
|
arkiver |
https://apps.sendus.com/robots.txt |
|
18:09
🔗
|
SketchCow |
Welcome to StudioOccupy, a movie studio in the cloud that is entirely dedicated to the Occupy movement. This free application can turn any Occupy location into a film studio. The Occupy movement is on the verge of discovering a new way forward for our society. This new way is an alternative to what has until now been the dominant worldview of conflict, division, and fear. Fundamentally, Occupy is ab |
|
18:09
🔗
|
SketchCow |
out bringing forth a world of compassion, interconnection, and... let us come right out and say it... it is love that ultimately must work for the 100%. So the intention behind StudioOccupy is to give people everywhere the power to tell their stories of the LOVE and possibility already happening in Occupy. |
|
18:10
🔗
|
n00b111 |
How did you find that text? |
|
18:10
🔗
|
|
VADemon has quit IRC (Quit: left4dead) |
|
18:13
🔗
|
|
Stiletto has quit IRC (Read error: Operation timed out) |
|
18:18
🔗
|
JW_work |
n00b111: http://archive.is/*.citizenglobal.com may be useful. The text, btw, is in the meta tag on the wayback page for studiooccupy.org |
|
18:20
🔗
|
JW_work |
n00b111: also this: https://www.sendus.com/together |
|
18:23
🔗
|
n00b111 |
This may be interesting to some of you... the app that was developed by citizen global for Occupy protesters has been rebranded and sold to the LA county sheriff's department as LEEDIR, an app for citizen crime reporting. |
|
18:23
🔗
|
|
mismatch has joined #archiveteam |
|
18:31
🔗
|
* |
n00b111 slaps n00b111 around a bit with a large fishbot |
|
18:32
🔗
|
n00b111 |
@jw_work thank you! |
|
18:32
🔗
|
JW_work |
n00b111: glad to help |
|
18:42
🔗
|
SketchCow |
29G ARCHIVEBOT |
|
18:42
🔗
|
SketchCow |
2.2T ARCHIVETEAM |
|
18:42
🔗
|
SketchCow |
517G CHFOO |
|
18:43
🔗
|
SketchCow |
Now we're going to play "Which of these projects is dead." |
|
18:48
🔗
|
SketchCow |
telenor. |
|
18:48
🔗
|
SketchCow |
yuku. |
|
18:48
🔗
|
SketchCow |
wikis. |
|
18:48
🔗
|
SketchCow |
virginmedia. |
|
18:48
🔗
|
SketchCow |
googlecode. fotolog. gametrailers. |
|
18:48
🔗
|
HCross |
virginmedia is ending/ended |
|
18:49
🔗
|
HCross |
wait though |
|
18:49
🔗
|
HCross |
telenor is done |
|
18:56
🔗
|
SketchCow |
telenor gone |
|
19:00
🔗
|
|
pfallenop has quit IRC (Read error: Connection reset by peer) |
|
19:07
🔗
|
|
bwn has quit IRC (Ping timeout: 246 seconds) |
|
19:12
🔗
|
zino |
A bunch of fotolog is sitting on my server |
|
19:13
🔗
|
zino |
next in line to be uploaded after ftpgrab |
|
19:13
🔗
|
|
zino has quit IRC (Remote host closed the connection) |
|
19:18
🔗
|
|
schbirid has quit IRC (Quit: Leaving) |
|
19:19
🔗
|
arkiver |
telenor and gametrailers are done. |
|
19:19
🔗
|
arkiver |
virginmedia is done tomorrow |
|
19:19
🔗
|
arkiver |
leave the others up |
|
19:26
🔗
|
MrRadar |
arkiver: I've been seeing some more rsync failures on the bayimg grab. Did you mean to set max_tries=None for the rsync step in the pipeline script? |
|
19:26
🔗
|
|
powerKitt has joined #archiveteam |
|
19:26
🔗
|
MrRadar |
It looks like it defaults to 1 which would cause the behavior I'm seeing (the task aborts if that step fails) |
|
19:27
🔗
|
powerKitt |
Can I suggest projects here? |
|
19:27
🔗
|
arkiver |
yes |
|
19:27
🔗
|
SketchCow |
Gametrailers has 168gb of rsync-tmp files |
|
19:28
🔗
|
arkiver |
MrRadar: We don't want to have a too big pile of uploads for FOS, I guess some delay'd retrying would be good though |
|
19:28
🔗
|
arkiver |
SketchCow: all partial WARCs? |
|
19:28
🔗
|
SketchCow |
Yeah. |
|
19:28
🔗
|
SketchCow |
And I'm doing an -sb to see if they're floating in. |
|
19:28
🔗
|
SketchCow |
No. |
|
19:28
🔗
|
MrRadar |
It's not the upload to FOS where it's failing, it's the rsync to storage.harrycross.me that is |
|
19:28
🔗
|
arkiver |
oh really |
|
19:29
🔗
|
arkiver |
^HCross, there seems to be a small problem there |
|
19:29
🔗
|
MrRadar |
Yeah, the ExternalProcess("rsync", ... line |
|
19:29
🔗
|
arkiver |
SketchCow: maybe they can be uploaded to an item that will not be indexed in the wayback machine? |
|
19:29
🔗
|
arkiver |
So we can have a look at them or fix them with scripts in the future |
|
19:30
🔗
|
SketchCow |
My question is if they're out there already. |
|
19:30
🔗
|
SketchCow |
No easy way to check, I think. |
|
19:30
🔗
|
HCross |
Hang on |
|
19:30
🔗
|
SketchCow |
Like I have a list now. |
|
19:30
🔗
|
SketchCow |
gametrailers-10videos_173984-20160210-195322.warc.gz gametrailers-10videos_295394-20160211-065929.warc.gz |
|
19:30
🔗
|
SketchCow |
gametrailers-10videos_294425-20160211-195722.warc.gz gametrailers-10videos_295399-20160210-214839.warc.gz |
|
19:30
🔗
|
SketchCow |
Etc. |
|
19:30
🔗
|
SketchCow |
Any easy way to see if it got reassigned? |
|
19:30
🔗
|
HCross |
MrRadar, what seems to be the issue? |
|
19:31
🔗
|
MrRadar |
The step of the pipeline script is failing due to your server hitting the max connection limit of 10 |
|
19:31
🔗
|
arkiver |
SketchCow: I can have a look at the logs of gametrailers and check which of those items returned as done to the tracker |
|
19:31
🔗
|
MrRadar |
And that's causing it to abort because the pipeline doesn't have any retries configured for that step |
|
19:31
🔗
|
MrRadar |
I think allowing retries would fix the issue |
|
19:32
🔗
|
|
bwn has joined #archiveteam |
|
19:32
🔗
|
arkiver |
yep |
|
19:32
🔗
|
HCross |
MrRadar, opened up a few slots |
|
19:32
🔗
|
arkiver |
will fix that later today (retries) |
|
19:32
🔗
|
* |
arkiver is afk for a bit |
|
19:32
🔗
|
MrRadar |
OK |
|
19:32
🔗
|
MrRadar |
Thanks HCross, hopefully that will be enough until arkiver can fix the script |
|
19:32
🔗
|
|
RedType has joined #archiveteam |
|
19:32
🔗
|
HCross |
Ive upped it to 50 |
|
19:33
🔗
|
|
RedType has quit IRC (Client Quit) |
|
19:34
🔗
|
|
RedType has joined #archiveteam |
|
19:34
🔗
|
|
zino has joined #archiveteam |
|
19:36
🔗
|
|
SimpBrain has joined #archiveteam |
|
19:37
🔗
|
balrog |
is it safe to use parallel tasks when uploading to fos? |
|
19:37
🔗
|
|
SimpBrain has quit IRC (Client Quit) |
|
19:37
🔗
|
|
SimpBrain has joined #archiveteam |
|
19:43
🔗
|
|
RedType has left |
|
19:45
🔗
|
|
powerKitt has quit IRC (Quit: Page closed) |
|
20:03
🔗
|
|
vitzli has quit IRC (Ping timeout: 246 seconds) |
|
20:04
🔗
|
|
pfallenop has joined #archiveteam |
|
20:13
🔗
|
|
pfallenop has quit IRC (Remote host closed the connection) |
|
20:14
🔗
|
|
pfallenop has joined #archiveteam |
|
20:16
🔗
|
|
Mayonaise has quit IRC (Read error: Operation timed out) |
|
20:17
🔗
|
|
balrog has quit IRC (Read error: Operation timed out) |
|
20:17
🔗
|
|
balrog has joined #archiveteam |
|
20:17
🔗
|
|
swebb sets mode: +o balrog |
|
20:17
🔗
|
|
Zebranky has quit IRC (Read error: Operation timed out) |
|
20:18
🔗
|
|
aMunster has quit IRC (Read error: Operation timed out) |
|
20:18
🔗
|
|
Froggypwn has quit IRC (Read error: Operation timed out) |
|
20:18
🔗
|
|
beardicus has quit IRC (Read error: Operation timed out) |
|
20:19
🔗
|
|
Froggypwn has joined #archiveteam |
|
20:19
🔗
|
|
RKenshin has joined #archiveteam |
|
20:19
🔗
|
|
Kenshin has quit IRC (Read error: Operation timed out) |
|
20:19
🔗
|
|
RKenshin is now known as Kenshin |
|
20:20
🔗
|
|
MMovie2 has quit IRC (Read error: Operation timed out) |
|
20:21
🔗
|
|
beardicus has joined #archiveteam |
|
20:21
🔗
|
|
swebb sets mode: +o beardicus |
|
20:22
🔗
|
|
Mayonaise has joined #archiveteam |
|
20:24
🔗
|
|
Starholme has quit IRC (Read error: Operation timed out) |
|
20:25
🔗
|
|
BartoCH has quit IRC (Ping timeout: 260 seconds) |
|
20:26
🔗
|
|
Starholme has joined #archiveteam |
|
20:28
🔗
|
|
BartoCH has joined #archiveteam |
|
20:33
🔗
|
|
wp494_ has joined #archiveteam |
|
20:33
🔗
|
|
Mayonaise has quit IRC (Read error: Operation timed out) |
|
20:34
🔗
|
|
chfoo has quit IRC (Read error: Operation timed out) |
|
20:34
🔗
|
|
mhazinsk has quit IRC (Read error: Operation timed out) |
|
20:35
🔗
|
|
vtyl has joined #archiveteam |
|
20:35
🔗
|
|
chfoo has joined #archiveteam |
|
20:35
🔗
|
|
BartoCH has quit IRC (Ping timeout: 260 seconds) |
|
20:37
🔗
|
|
beardicus has quit IRC (Read error: Operation timed out) |
|
20:37
🔗
|
|
lytv has quit IRC (Read error: Operation timed out) |
|
20:37
🔗
|
|
K4k has quit IRC (Read error: Operation timed out) |
|
20:39
🔗
|
|
K4k has joined #archiveteam |
|
20:39
🔗
|
|
Zebranky has joined #archiveteam |
|
20:40
🔗
|
|
wp494 has quit IRC (Read error: Operation timed out) |
|
20:55
🔗
|
|
VADemon has joined #archiveteam |
|
20:57
🔗
|
|
n00b111 has quit IRC (Quit: Page closed) |
|
21:08
🔗
|
|
BartoCH has joined #archiveteam |
|
21:09
🔗
|
|
Fusl has quit IRC (Read error: Operation timed out) |
|
21:09
🔗
|
|
Smiley has quit IRC (Remote host closed the connection) |
|
21:09
🔗
|
|
Stiletto has joined #archiveteam |
|
21:10
🔗
|
|
RichardG has quit IRC (Ping timeout: 272 seconds) |
|
21:10
🔗
|
|
RichardG has joined #archiveteam |
|
21:11
🔗
|
|
tomwsmf-a has joined #archiveteam |
|
21:21
🔗
|
|
Fusl has joined #archiveteam |
|
21:23
🔗
|
MrRadar |
arkiver: New issue with the bayimg grab. It randomly freezes after the "rsync" step before the main upload. In the web UI the "rsync" task count goes negative: https://i.imgur.com/0y4Vnag.png |
|
21:25
🔗
|
MrRadar |
Looking at the console output it says the data directory doesn't exist: http://pastebin.com/xQMrMand |
|
21:27
🔗
|
MrRadar |
Strange, that item matches neither of the active items for this pipeline. Let me check the console scrollback.... |
|
21:28
🔗
|
MrRadar |
Looks like a problem in Seesaw: http://pastebin.com/EBA8vfQK |
|
21:29
🔗
|
MrRadar |
It's failing to catch an exception which causes the pipelien to get into a bad state |
|
21:29
🔗
|
JW_work |
sorry |
|
21:29
🔗
|
JW_work |
wrong channel |
|
21:30
🔗
|
|
Honno has quit IRC (Read error: Operation timed out) |
|
21:30
🔗
|
|
Smiley has joined #archiveteam |
|
21:30
🔗
|
MrRadar |
chfoo, do you have any insights? |
|
21:33
🔗
|
|
tomwsmf-a has quit IRC (Ping timeout: 258 seconds) |
|
21:38
🔗
|
chfoo |
MrRadar, yeah, that's a known issue |
|
21:38
🔗
|
chfoo |
post the info at https://github.com/ArchiveTeam/seesaw-kit/issues/48 |
|
21:38
🔗
|
MrRadar |
OK |
|
21:40
🔗
|
chfoo |
i could never find the proper cause and fix for it. the only thing i can suggest is to reboot it |
|
21:47
🔗
|
yipdw |
chfoo: FWIW, that happens (rarely) on ArchiveBot also |
|
21:47
🔗
|
yipdw |
well the result is the same, I don't know what the root cause is |
|
21:50
🔗
|
MrRadar |
I posted all the details on the github issue |
|
21:50
🔗
|
MrRadar |
Let me know if you want any more information |
|
21:52
🔗
|
|
metalcamp has quit IRC (Quit: Bye) |
|
22:04
🔗
|
|
xhdr has quit IRC (Ping timeout: 194 seconds) |
|
22:05
🔗
|
|
vOYtEC has quit IRC (Ping timeout: 258 seconds) |
|
22:05
🔗
|
|
vOYtEC has joined #archiveteam |
|
22:07
🔗
|
|
fie_ has joined #archiveteam |
|
22:08
🔗
|
|
Starholme has quit IRC (Read error: Connection reset by peer) |
|
22:08
🔗
|
|
vtyl has quit IRC (Read error: Connection reset by peer) |
|
22:08
🔗
|
|
fie has quit IRC (Read error: Connection reset by peer) |
|
22:08
🔗
|
|
xhdr has joined #archiveteam |
|
22:08
🔗
|
|
Starholme has joined #archiveteam |
|
22:09
🔗
|
|
Atluxity has quit IRC (Ping timeout: 260 seconds) |
|
22:10
🔗
|
|
vtyl has joined #archiveteam |
|
22:13
🔗
|
|
Atluxity has joined #archiveteam |
|
22:13
🔗
|
|
swebb sets mode: +o Atluxity |
|
22:18
🔗
|
|
beardicus has joined #archiveteam |
|
22:18
🔗
|
|
swebb sets mode: +o beardicus |
|
22:19
🔗
|
|
aMunster has joined #archiveteam |
|
22:25
🔗
|
|
MMovie has joined #archiveteam |
|
22:27
🔗
|
|
antonizoo has joined #archiveteam |
|
22:37
🔗
|
|
Stiletto has quit IRC (Ping timeout: 260 seconds) |
|
22:37
🔗
|
|
Simpbrai_ has quit IRC (Ping timeout: 260 seconds) |
|
22:37
🔗
|
|
szalwia has quit IRC (Ping timeout: 260 seconds) |
|
22:38
🔗
|
|
szalwia has joined #archiveteam |
|
22:39
🔗
|
|
Mayonaise has joined #archiveteam |
|
22:42
🔗
|
|
Simpbrai_ has joined #archiveteam |
|
22:42
🔗
|
|
tomwsmf-a has joined #archiveteam |
|
22:44
🔗
|
|
mhazinsk has joined #archiveteam |
|
22:44
🔗
|
|
closure has quit IRC (Read error: Operation timed out) |
|
22:46
🔗
|
|
r3c0d3x has joined #archiveteam |
|
22:46
🔗
|
r3c0d3x |
shout out to whoever beat me to the bayimg scraper, was working on something using node.js for that |
|
22:48
🔗
|
|
WinterFox has joined #archiveteam |
|
22:50
🔗
|
MrRadar |
That would be arkiver (with help from joepie91, VADemon, and Phuzion in reverse-engineering the URL short codes) |
|
22:50
🔗
|
MrRadar |
http://archive.fart.website/bin/irclogger_log/archiveteam?date=2016-04-26,Tue |
|
22:52
🔗
|
r3c0d3x |
thanks! |
|
22:53
🔗
|
|
dashcloud has quit IRC (Read error: Operation timed out) |
|
22:56
🔗
|
|
dashcloud has joined #archiveteam |
|
22:58
🔗
|
|
tomwsmf-a has quit IRC (Read error: Operation timed out) |
|
23:23
🔗
|
|
BlueMaxim has joined #archiveteam |
|
23:29
🔗
|
|
nertzy has joined #archiveteam |
|
23:49
🔗
|
arkiver |
r3c0d3x: hi |
|
23:49
🔗
|
arkiver |
What is your question regarding the bayimg grab? |
|
23:50
🔗
|
MrRadar |
I don't think r3c0d3x had a question, just appreciation |
|
23:51
🔗
|
|
Stiletto has joined #archiveteam |