| Time |
Nickname |
Message |
|
00:36
🔗
|
arkiver |
I think I can update the current urlteam project to create WARCs of simple GETs |
|
00:37
🔗
|
arkiver |
not sure if we want HEADs and POSTs in WARCs too (and in the Wayback Machine) |
|
00:38
🔗
|
|
jornbaer has quit IRC (Read error: Connection reset by peer) |
|
00:39
🔗
|
|
jornane has joined #urlteam |
|
00:39
🔗
|
JAA |
Not sure regarding the WBM, but we currently only have shortener projects active that use HEAD. |
|
00:40
🔗
|
|
hook54321 has quit IRC (Read error: Connection reset by peer) |
|
00:41
🔗
|
|
hook54321 has joined #urlteam |
|
00:42
🔗
|
JAA |
I guess we could switch everything to GET (except POST ones obviously). |
|
00:43
🔗
|
JAA |
Would mean much higher network traffic for the workers. Relatively speaking, that is (probably an increase by several thousand per cent). The absolute traffic would still be very small due to the rate limits etc. |
|
00:43
🔗
|
astrid |
i wouldn't worry about it much |
|
00:43
🔗
|
JAA |
But even if we don't do that, I still think recording the entire requests and responses as WARCs is useful, whether they end up in the WBM or not. |
|
00:44
🔗
|
JAA |
Given that the WBM doesn't actually show the contents of a 30x anyway though, I don't really see why HEAD requests couldn't be included. |
|
00:44
🔗
|
JAA |
For those status codes at least. |
|
00:45
🔗
|
JAA |
astrid: Yeah, it would still only be on the order of kB/s, so not even noticeable. |
|
00:45
🔗
|
astrid |
might as well ask IA if HEADs are acceptable before we go changing up our methods |
|
01:39
🔗
|
Somebody2 |
Finished another scrape of shortdoi.org ; it's only up to c--- |
|
01:39
🔗
|
Somebody2 |
turned it back off |
|
01:45
🔗
|
Somebody2 |
delighted with the energy to switch to capturing WARCs; I'm fully in favor, just haven't gotten around to it |
|
04:24
🔗
|
|
mtntmnky has quit IRC (Read error: Operation timed out) |
|
04:25
🔗
|
|
Mayonaise has quit IRC (Read error: Operation timed out) |
|
04:27
🔗
|
|
Mayonaise has joined #urlteam |
|
04:30
🔗
|
|
kiska1 has quit IRC (Read error: Connection reset by peer) |
|
04:30
🔗
|
|
TigerbotH has quit IRC (Read error: Connection reset by peer) |
|
04:30
🔗
|
|
kiska1 has joined #urlteam |
|
04:30
🔗
|
|
wmvhater has quit IRC (Ping timeout: 600 seconds) |
|
04:31
🔗
|
|
wmvhater has joined #urlteam |
|
04:32
🔗
|
|
TigerbotH has joined #urlteam |
|
04:33
🔗
|
|
mtntmnky has joined #urlteam |
|
04:41
🔗
|
|
odemg has quit IRC (Ping timeout: 265 seconds) |
|
04:53
🔗
|
|
odemg has joined #urlteam |
|
07:29
🔗
|
|
mtntmnky has quit IRC (Remote host closed the connection) |
|
07:32
🔗
|
|
mtntmnky has joined #urlteam |
|
07:34
🔗
|
|
jornbaer has joined #urlteam |
|
07:35
🔗
|
|
jornane has quit IRC (Read error: Connection reset by peer) |
|
07:50
🔗
|
|
mtntmnky has quit IRC (Remote host closed the connection) |
|
08:15
🔗
|
|
svchfoo3 has quit IRC (Read error: Operation timed out) |
|
08:15
🔗
|
|
wmvhater has quit IRC (Read error: Operation timed out) |
|
08:15
🔗
|
|
kiska1 has quit IRC (Read error: Operation timed out) |
|
08:15
🔗
|
|
wmvhater has joined #urlteam |
|
08:16
🔗
|
|
kiska1 has joined #urlteam |
|
08:23
🔗
|
|
svchfoo3 has joined #urlteam |
|
08:23
🔗
|
|
svchfoo1 sets mode: +o svchfoo3 |
|
10:33
🔗
|
|
hook54321 has quit IRC (Quit: Connection closed for inactivity) |
|
11:26
🔗
|
|
chazchaz has quit IRC (Ping timeout: 360 seconds) |
|
13:23
🔗
|
|
hook54321 has joined #urlteam |
|
16:41
🔗
|
|
MrRadar has joined #urlteam |
|
18:32
🔗
|
|
hook54321 has quit IRC (Quit: Connection closed for inactivity) |
|
18:43
🔗
|
|
hook54321 has joined #urlteam |
|
19:22
🔗
|
|
mtntmnky has joined #urlteam |
|
19:29
🔗
|
|
mtntmnky has quit IRC (Remote host closed the connection) |
|
19:30
🔗
|
|
mtntmnky has joined #urlteam |
|
19:50
🔗
|
|
mtntmnky_ has joined #urlteam |
|
19:55
🔗
|
|
mtntmnky has quit IRC (Ping timeout: 600 seconds) |
|
21:27
🔗
|
|
chfoo has quit IRC (Read error: Operation timed out) |
|
21:34
🔗
|
|
chfoo has joined #urlteam |
|
23:02
🔗
|
|
Hani has quit IRC (Read error: Connection reset by peer) |