| Time |
Nickname |
Message |
|
02:35
🔗
|
Somebody2 |
Fixing tracker again; minu-me got stuck |
|
02:44
🔗
|
Somebody2 |
Turning off the queue for minu-me for a bit; let's see if it drains or not |
|
03:09
🔗
|
Somebody2 |
down to 40 rather than 70... we'll see how it goes |
|
03:55
🔗
|
Somebody2 |
ok, the queue finally drained; now re-doing the 7 I removed to clear the blockage |
|
03:55
🔗
|
Somebody2 |
we'll see if any of them consistently fail |
|
03:56
🔗
|
Somebody2 |
nope, they all went through very nicely |
|
03:56
🔗
|
Somebody2 |
strange |
|
03:57
🔗
|
Somebody2 |
ok, turning the queue back on at 60 this time |
|
04:17
🔗
|
|
Sk1d has joined #urlteam |
|
04:18
🔗
|
|
jornane has quit IRC (Ping timeout: 268 seconds) |
|
04:22
🔗
|
|
jornane has joined #urlteam |
|
04:26
🔗
|
Somebody2 |
lower queue to 40 |
|
05:48
🔗
|
Somebody2 |
reducing queue even further |
|
05:49
🔗
|
Somebody2 |
increase time between requests to 0.8 |
|
05:59
🔗
|
Somebody2 |
Frogging: very good point |
|
05:59
🔗
|
Somebody2 |
What do we have that handles resuming better? Warrior jobs? |
|
05:59
🔗
|
Frogging |
yes, something more custom |
|
06:00
🔗
|
Frogging |
warrior jobs don't have the resuming problem because they're split up into units anyway |
|
06:00
🔗
|
Frogging |
but throwing recursive crawlers at something like metafilter is asking for trouble, especially if it's archivebot |
|
06:01
🔗
|
Somebody2 |
If we're doing this on request of the site owner, I'm not sure why they can't do it themselves... |
|
06:01
🔗
|
Somebody2 |
Or mail IA a hard drive of the data, and we can convert it into web pages locally at our leasure. |
|
06:02
🔗
|
Frogging |
I don't think they'd get into wayback that way |
|
06:09
🔗
|
Somebody2 |
They could get into wayback eventually... |
|
06:10
🔗
|
Somebody2 |
If the site owner sent us the full data, we could spin up a virtual machine that thought it was the original, talk to it with a browser, and thereby get the same as the real site, but without the network delay (and cost) |
|
06:10
🔗
|
Somebody2 |
But I'm pretty sure IA isn't (yet) set up to do that. |
|
06:11
🔗
|
Somebody2 |
Anyway, this isn't actually #urlteam -- we both just mistakenly switched channels halfway through this conversation! |
|
06:23
🔗
|
Somebody2 |
turning off the queue on minu-me for the evening, to avoid it getting the tracker stuck |
|
06:38
🔗
|
Somebody2 |
trying the queue at 5 items, with 10 urls in each |
|
06:41
🔗
|
Somebody2 |
OK, that seems to be working; I'll leave that for the night |
|
09:42
🔗
|
|
Jonison has joined #urlteam |
|
10:34
🔗
|
|
JAA has joined #urlteam |
|
13:09
🔗
|
|
JAA has quit IRC (Quit: Page closed) |
|
15:50
🔗
|
chfoo |
i'll merge that pull request later today |
|
21:28
🔗
|
|
Jonison has quit IRC (Read error: Connection reset by peer) |