#archiveteam-bs 2017-08-15,Tue

↑back Search

Time Nickname Message
00:02 πŸ”— SN4T14 has quit IRC (Quit: ZNC 1.6.3 - http://znc.in)
00:03 πŸ”— SN4T14 has joined #archiveteam-bs
00:11 πŸ”— nightpool has quit IRC (Read error: Operation timed out)
00:13 πŸ”— nightpool has joined #archiveteam-bs
01:01 πŸ”— TheLovina has quit IRC (Read error: Operation timed out)
01:03 πŸ”— TheLovina has joined #archiveteam-bs
01:16 πŸ”— hook54321 !a http://82.221.129.208/ --useragent firefox
01:28 πŸ”— username1 has joined #archiveteam-bs
01:31 πŸ”— schbirid2 has quit IRC (Read error: Operation timed out)
01:51 πŸ”— pizzaiolo has quit IRC (Quit: pizzaiolo)
02:41 πŸ”— Odd0002 has quit IRC (Remote host closed the connection)
03:00 πŸ”— hook54321 So I just saw this: https://github.com/chfoo/wpull/issues/356
03:00 πŸ”— hook54321 Would it be possible to incentivize sites to not disallow ia_archiver in their robots.txt file by respecting delay specified in robots.txt?
03:01 πŸ”— SketchCow We don't negotiate with terrorists
03:01 πŸ”— hook54321 lol.
03:02 πŸ”— Frogging :p
03:06 πŸ”— hook54321 but like if we were going to do that as the issue suggests, i don't see why we would want to cooperate with people that disallow the wayback machine.
03:07 πŸ”— hook54321 i think that it's stupid that some sites try to tell people to use a crawl delay of 10 seconds though
03:27 πŸ”— hook54321 Brendan Eich appears to be supporting this: https://github.com/EdOverflow/security-txt
03:32 πŸ”— qw3rty119 has joined #archiveteam-bs
03:38 πŸ”— qw3rty118 has quit IRC (Read error: Operation timed out)
03:51 πŸ”— Stilett0 has joined #archiveteam-bs
04:09 πŸ”— hook54321 Wiki is acting kinda funny
04:24 πŸ”— hook54321 JAA: Daily Stormer is moving to the TOR Network
04:26 πŸ”— Sk1d has quit IRC (Ping timeout: 250 seconds)
04:33 πŸ”— Sk1d has joined #archiveteam-bs
04:37 πŸ”— hook54321 Apparently Google froze their domain, so they can't move it now.
04:46 πŸ”— robink has quit IRC (Read error: Connection reset by peer)
04:51 πŸ”— robink has joined #archiveteam-bs
04:55 πŸ”— dashcloud has quit IRC (Read error: Operation timed out)
05:01 πŸ”— dashcloud has joined #archiveteam-bs
05:05 πŸ”— kimmer22 has joined #archiveteam-bs
05:14 πŸ”— kimmer2 has quit IRC (Ping timeout: 633 seconds)
05:20 πŸ”— Stilett0 is now known as Stiletto
05:25 πŸ”— kimmer2 has joined #archiveteam-bs
05:33 πŸ”— kimmer22 has quit IRC (Ping timeout: 633 seconds)
05:53 πŸ”— zino hook54321: Something that might be more fruitful is checking what the support for HTTP error 429 is in wpull. I've seen logs where we get a lot of 429s followed by a 200 followed by a lot of 429s again. RFC6585. Either:
05:53 πŸ”— zino 1) wpull does not handle the Retry-After header
05:53 πŸ”— zino 2) The site is still not prepared to answer requests after timeout
05:53 πŸ”— zino 3) The site does not send a Rety-After header
05:53 πŸ”— zino If it's 2 or 3, then it's not much we can do, if it's 1 we would probably save all sides trouble by implementing it, and minimize chances to get IP-banned. Then add a pipeline override if there is reason to ignore requests from the server to back off.
05:53 πŸ”— HCross has quit IRC (Read error: Connection reset by peer)
05:54 πŸ”— HCross has joined #archiveteam-bs
05:55 πŸ”— robogoat has quit IRC (Read error: Operation timed out)
05:56 πŸ”— robogoat has joined #archiveteam-bs
06:19 πŸ”— kimmer22 has joined #archiveteam-bs
06:19 πŸ”— godane has quit IRC (Quit: Leaving.)
06:26 πŸ”— kimmer2 has quit IRC (Ping timeout: 633 seconds)
06:49 πŸ”— j08nY has joined #archiveteam-bs
06:59 πŸ”— dashcloud has quit IRC (Read error: Operation timed out)
07:14 πŸ”— BlueMaxim has joined #archiveteam-bs
07:15 πŸ”— kimmer2 has joined #archiveteam-bs
07:15 πŸ”— TheLovina has quit IRC (Ping timeout: 370 seconds)
07:15 πŸ”— TheLovina has joined #archiveteam-bs
07:20 πŸ”— kimmer22 has quit IRC (Ping timeout: 633 seconds)
07:20 πŸ”— dashcloud has joined #archiveteam-bs
07:28 πŸ”— Boppen has quit IRC (Ping timeout: 194 seconds)
07:41 πŸ”— BlueMaxim has quit IRC (Read error: Operation timed out)
07:41 πŸ”— BlueMaxim has joined #archiveteam-bs
07:48 πŸ”— Honno has joined #archiveteam-bs
07:49 πŸ”— HCross has quit IRC (Remote host closed the connection)
07:49 πŸ”— HCross has joined #archiveteam-bs
08:32 πŸ”— j08nY has quit IRC (Read error: Operation timed out)
08:34 πŸ”— kimmer22 has joined #archiveteam-bs
08:38 πŸ”— kimmer2 has quit IRC (Ping timeout: 633 seconds)
08:40 πŸ”— kimmer2 has joined #archiveteam-bs
08:40 πŸ”— Boppen has joined #archiveteam-bs
08:45 πŸ”— kimmer22 has quit IRC (Ping timeout: 633 seconds)
08:45 πŸ”— kimmer22 has joined #archiveteam-bs
08:50 πŸ”— kimmer2 has quit IRC (Ping timeout: 632 seconds)
08:51 πŸ”— hook54321 JAA: Onion address for Daily Stormer: http://dstormer6em3i4km.onion/
08:51 πŸ”— BlueMaxim has quit IRC (Quit: Leaving)
09:25 πŸ”— kimmer2 has joined #archiveteam-bs
09:30 πŸ”— kimmer22 has quit IRC (Ping timeout: 633 seconds)
09:32 πŸ”— kimmer1 has joined #archiveteam-bs
09:36 πŸ”— godane has joined #archiveteam-bs
09:37 πŸ”— godane looks like IA is down again
09:48 πŸ”— hook54321 yup
09:49 πŸ”— hook54321 nothing on their twitter yet.
10:19 πŸ”— Honno has quit IRC (Read error: Operation timed out)
10:30 πŸ”— Mateon1 has quit IRC (Ping timeout: 268 seconds)
10:30 πŸ”— Mateon1 has joined #archiveteam-bs
10:48 πŸ”— j08nY has joined #archiveteam-bs
10:56 πŸ”— ivan has quit IRC (Leaving)
11:18 πŸ”— marvinw has joined #archiveteam-bs
11:21 πŸ”— JAA Very interesting court decision: https://www.reuters.com/article/us-microsoft-linkedin-ruling-idUSKCN1AU2BV
11:44 πŸ”— atluxity1 has joined #archiveteam-bs
11:46 πŸ”— atluxity has quit IRC (Ping timeout: 506 seconds)
11:50 πŸ”— JAA We should start archiving whois information.
11:50 πŸ”— JAA And DNS records
12:43 πŸ”— joepie91 holy shit
12:43 πŸ”— joepie91 that is actually a Very Big Deal
13:17 πŸ”— s2e has joined #archiveteam-bs
13:27 πŸ”— s2e Is there guidance on how to best submit dozens of websites to the internet archive in a way that is respectful of their infrastructure? I work in the internet freedom sector focusing on educational content and many of the resources that get created dissapear in months or a few years. I currently use a simple script to spider and submit new ones to the archive. I'd like to do this in a more automated fashion.
13:27 πŸ”— s2e But, I want to make sure I am doing it as respectfully as possible.
13:29 πŸ”— Sanqui to IA's infrastructure?
13:29 πŸ”— j08nY has quit IRC (Read error: Operation timed out)
13:29 πŸ”— Sanqui I mean, respectful of IA's infrastructure?
13:29 πŸ”— Sanqui you probably want archivebot
13:29 πŸ”— s2e Yeah, if possible. I've seen other efforts try to archive seperately, but they are largely unavailable to others
13:30 πŸ”— Sanqui join #archivebot, check out how it works, submit a website with !a, watch the dashboard, it'll get absorbed into wayback
13:30 πŸ”— s2e awesome
13:32 πŸ”— Frogging joepie91: eli5?
13:33 πŸ”— s2e Since archivebot is a volunteer service is the method it uses the best method for doing this without a drain on others resources? Is it something I could run on my own to do the archiving and supply the WARC files in the same way?
13:33 πŸ”— Sanqui Frogging: my understanding is - it is legal to scrape public personal information on websites for commercial purposes
13:34 πŸ”— Sanqui s2e: you could provide a pipeline, but I'm not sure if we're accepting right now; or you can run something like grab-site yourself, but you'd have to find some avenue to get the warcs into wayback.
13:35 πŸ”— Sanqui Frogging: not only it is legal, you cannot put measures in place against it
13:35 πŸ”— Frogging I see.
13:35 πŸ”— Sanqui IANAL
13:35 πŸ”— s2e Sanqui: Thanks! I'll start with archivebot and bother IA about WARC inclusion.
13:35 πŸ”— Frogging the applications they mentioned on the page don't instill confidence
13:36 πŸ”— Frogging using "publicly available data and artificial intelligence to help companies identify potential customers"
13:36 πŸ”— Frogging building "algorithms capable of predicting employee behaviors, such as when they might quit"
13:37 πŸ”— omglolbah "If LinkedIn is going to allow profiles to be indexed by search engines to benefit their platform then why shouldn't the rest of the internet benefit from that as well?"
13:40 πŸ”— Mateon1 has quit IRC (Remote host closed the connection)
13:40 πŸ”— kimmer22 has joined #archiveteam-bs
13:41 πŸ”— Mateon1 has joined #archiveteam-bs
13:43 πŸ”— s2e has left WeeChat 1.6
13:47 πŸ”— kimmer2 has quit IRC (Ping timeout: 633 seconds)
14:15 πŸ”— j08nY has joined #archiveteam-bs
15:01 πŸ”— pizzaiolo has joined #archiveteam-bs
16:04 πŸ”— wabu has quit IRC (Read error: Operation timed out)
16:09 πŸ”— kimmer2 has joined #archiveteam-bs
16:13 πŸ”— username1 is now known as schbirid
16:14 πŸ”— wabu has joined #archiveteam-bs
16:17 πŸ”— kimmer22 has quit IRC (Ping timeout: 633 seconds)
17:07 πŸ”— pizzaiolo has quit IRC (pizzaiolo)
17:08 πŸ”— xmc JAA, joepie91: i was talking with FalconK the other day, and he mentioned the idea of running a recursive resolver that archives all results, and having archivebot and the warrior use it as their default resolvers
17:08 πŸ”— xmc i really like this idea
17:09 πŸ”— xmc i'm not sure what the proper archival format for DNS would be
17:09 πŸ”— xmc I suppose you could cram it into a warc
17:12 πŸ”— schbirid i thought warc is http
17:12 πŸ”— schbirid *think
17:12 πŸ”— PurpleSym It is not limited to HTTP, there’s a generic β€œresource” record.
17:13 πŸ”— schbirid oh nice
17:16 πŸ”— godane this looks like a torrent of the IA 911 videos: http://torrentproject.se/2d64409b6f179bc999159284156b3534711447a1/
17:16 πŸ”— PurpleSym Also, DNS perfectly fits into the request/response scheme WARC is using for HTTP.
17:19 πŸ”— JAA That's a nice idea, apart from the fact that it introduces a single point of failure. If the resolver is down, *everything* crashes and burns.
17:22 πŸ”— xmc yes, also that
17:33 πŸ”— joepie91 xmc: schbirid: heritrix stores DNS records in WARCs
17:33 πŸ”— joepie91 or well, DNS requests and responses
17:33 πŸ”— xmc hmmmmm
17:36 πŸ”— kristian_ has joined #archiveteam-bs
18:14 πŸ”— kristian_ has quit IRC (Quit: Leaving)
18:23 πŸ”— godane so my birthday is tomorrow
18:36 πŸ”— Aoede happy birthday godane (I would forget to say this tomorrow :p)
18:53 πŸ”— fie_ has quit IRC (Ping timeout: 246 seconds)
19:11 πŸ”— fie has joined #archiveteam-bs
19:26 πŸ”— hook54321 godane: happy birthday
19:44 πŸ”— kimmer2 has quit IRC (Ping timeout: 633 seconds)
20:16 πŸ”— kimmer1 has quit IRC (Quit: Going offline, see ya! (www.adiirc.com))
20:56 πŸ”— hook54321 Anyone know if there's something like this for Firefox? https://github.com/kissarat/never-lose
21:08 πŸ”— bwn has quit IRC (Ping timeout: 268 seconds)
21:13 πŸ”— bwn has joined #archiveteam-bs
21:56 πŸ”— Honno has joined #archiveteam-bs
22:03 πŸ”— arkiver it's 00:03 here now, happy birthday godane :D
22:16 πŸ”— DFJustin has quit IRC (Read error: Connection reset by peer)
22:17 πŸ”— DFJustin has joined #archiveteam-bs
22:18 πŸ”— dashcloud has quit IRC (Read error: Operation timed out)
22:18 πŸ”— dashcloud has joined #archiveteam-bs
22:23 πŸ”— pikhq has quit IRC (Read error: Operation timed out)
22:23 πŸ”— Frogging that repo's list of porn sites seems to have a disproportionate amount of gay porn
22:24 πŸ”— Frogging and random tumblrs. interesting. I wonder where they got it from
22:38 πŸ”— Igloo has quit IRC (Read error: Operation timed out)
22:38 πŸ”— j08nY has quit IRC (Read error: Operation timed out)
22:42 πŸ”— pikhq has joined #archiveteam-bs
22:43 πŸ”— godane has quit IRC (Ping timeout: 250 seconds)
22:43 πŸ”— Jonimus has quit IRC (Ping timeout: 268 seconds)
22:45 πŸ”— j08nY has joined #archiveteam-bs
22:47 πŸ”— godane has joined #archiveteam-bs
22:47 πŸ”— Igloo has joined #archiveteam-bs
22:56 πŸ”— * hook54321 shrugs
23:08 πŸ”— qw3rty111 has joined #archiveteam-bs
23:10 πŸ”— Jonimus has joined #archiveteam-bs
23:10 πŸ”— swebb sets mode: +o Jonimus
23:11 πŸ”— qw3rty112 has joined #archiveteam-bs
23:11 πŸ”— qw3rty119 has quit IRC (Ping timeout: 600 seconds)
23:18 πŸ”— qw3rty111 has quit IRC (Read error: Operation timed out)
23:30 πŸ”— j08nY has quit IRC (Quit: Leaving)

irclogger-viewer