#urlteam 2019-01-08,Tue

↑back Search

Time Nickname Message
00:36 🔗 arkiver I think I can update the current urlteam project to create WARCs of simple GETs
00:37 🔗 arkiver not sure if we want HEADs and POSTs in WARCs too (and in the Wayback Machine)
00:38 🔗 jornbaer has quit IRC (Read error: Connection reset by peer)
00:39 🔗 jornane has joined #urlteam
00:39 🔗 JAA Not sure regarding the WBM, but we currently only have shortener projects active that use HEAD.
00:40 🔗 hook54321 has quit IRC (Read error: Connection reset by peer)
00:41 🔗 hook54321 has joined #urlteam
00:42 🔗 JAA I guess we could switch everything to GET (except POST ones obviously).
00:43 🔗 JAA Would mean much higher network traffic for the workers. Relatively speaking, that is (probably an increase by several thousand per cent). The absolute traffic would still be very small due to the rate limits etc.
00:43 🔗 astrid i wouldn't worry about it much
00:43 🔗 JAA But even if we don't do that, I still think recording the entire requests and responses as WARCs is useful, whether they end up in the WBM or not.
00:44 🔗 JAA Given that the WBM doesn't actually show the contents of a 30x anyway though, I don't really see why HEAD requests couldn't be included.
00:44 🔗 JAA For those status codes at least.
00:45 🔗 JAA astrid: Yeah, it would still only be on the order of kB/s, so not even noticeable.
00:45 🔗 astrid might as well ask IA if HEADs are acceptable before we go changing up our methods
01:39 🔗 Somebody2 Finished another scrape of shortdoi.org ; it's only up to c---
01:39 🔗 Somebody2 turned it back off
01:45 🔗 Somebody2 delighted with the energy to switch to capturing WARCs; I'm fully in favor, just haven't gotten around to it
04:24 🔗 mtntmnky has quit IRC (Read error: Operation timed out)
04:25 🔗 Mayonaise has quit IRC (Read error: Operation timed out)
04:27 🔗 Mayonaise has joined #urlteam
04:30 🔗 kiska1 has quit IRC (Read error: Connection reset by peer)
04:30 🔗 TigerbotH has quit IRC (Read error: Connection reset by peer)
04:30 🔗 kiska1 has joined #urlteam
04:30 🔗 wmvhater has quit IRC (Ping timeout: 600 seconds)
04:31 🔗 wmvhater has joined #urlteam
04:32 🔗 TigerbotH has joined #urlteam
04:33 🔗 mtntmnky has joined #urlteam
04:41 🔗 odemg has quit IRC (Ping timeout: 265 seconds)
04:53 🔗 odemg has joined #urlteam
07:29 🔗 mtntmnky has quit IRC (Remote host closed the connection)
07:32 🔗 mtntmnky has joined #urlteam
07:34 🔗 jornbaer has joined #urlteam
07:35 🔗 jornane has quit IRC (Read error: Connection reset by peer)
07:50 🔗 mtntmnky has quit IRC (Remote host closed the connection)
08:15 🔗 svchfoo3 has quit IRC (Read error: Operation timed out)
08:15 🔗 wmvhater has quit IRC (Read error: Operation timed out)
08:15 🔗 kiska1 has quit IRC (Read error: Operation timed out)
08:15 🔗 wmvhater has joined #urlteam
08:16 🔗 kiska1 has joined #urlteam
08:23 🔗 svchfoo3 has joined #urlteam
08:23 🔗 svchfoo1 sets mode: +o svchfoo3
10:33 🔗 hook54321 has quit IRC (Quit: Connection closed for inactivity)
11:26 🔗 chazchaz has quit IRC (Ping timeout: 360 seconds)
13:23 🔗 hook54321 has joined #urlteam
16:41 🔗 MrRadar has joined #urlteam
18:32 🔗 hook54321 has quit IRC (Quit: Connection closed for inactivity)
18:43 🔗 hook54321 has joined #urlteam
19:22 🔗 mtntmnky has joined #urlteam
19:29 🔗 mtntmnky has quit IRC (Remote host closed the connection)
19:30 🔗 mtntmnky has joined #urlteam
19:50 🔗 mtntmnky_ has joined #urlteam
19:55 🔗 mtntmnky has quit IRC (Ping timeout: 600 seconds)
21:27 🔗 chfoo has quit IRC (Read error: Operation timed out)
21:34 🔗 chfoo has joined #urlteam
23:02 🔗 Hani has quit IRC (Read error: Connection reset by peer)

irclogger-viewer