[00:18] *** Meta has left [00:45] *** octarine has joined #archiveteam [01:22] *** brayden_ has joined #archiveteam [01:22] *** swebb sets mode: +o brayden_ [01:26] *** brayden has quit IRC (Read error: Operation timed out) [02:14] *** Ctrl has joined #archiveteam [03:13] *** marshall has joined #archiveteam [03:18] *** marshall is now known as sun_rise [03:39] *** pizzaiolo has quit IRC (pizzaiolo) [03:42] *** iguana has quit IRC (Read error: Operation timed out) [04:17] *** Sk1d has quit IRC (Ping timeout: 250 seconds) [04:24] *** Sk1d has joined #archiveteam [04:24] *** Sk1d has quit IRC (Connection Closed) [04:31] is there some kind of framework for automated mass downloading over http according to some kind of ruleset provided by the user? [04:32] like, imagine someone has a file upload site where files are all under https://site.com/files/ and are named the date they're uploaded and an incremental integer [04:33] ploop: your best bet is probably wpull with some scripting [04:33] wpull? [04:33] wpull is similar to wget, but better in various ways [04:33] written by one of the people associated with ArchiveTeam [04:34] http://wpull.readthedocs.org/ [04:35] If you want to talk about this in more detail, come on over to #archiveteam-bs , which is the channel for discussion (rather than this one, which is more for announcements/initial questions). [04:35] ah, ok [04:38] *** BlueMaxim has joined #archiveteam [04:47] *** Aranje has quit IRC (Quit: Three sheets to the wind) [05:06] *** Morbus has quit IRC (Read error: Operation timed out) [06:21] *** Ctrl has quit IRC (Ping timeout: 244 seconds) [07:44] *** Jonison has joined #archiveteam [07:53] *** schbirid has joined #archiveteam [07:56] *** anhedonis has quit IRC (Ping timeout: 244 seconds) [07:58] *** anhedonis has joined #archiveteam [08:00] *** Deewiant has quit IRC (Ping timeout: 250 seconds) [08:01] *** Deewiant has joined #archiveteam [08:04] *** gibigian1 has joined #archiveteam [08:05] *** espes___ has joined #archiveteam [08:06] *** will has quit IRC (Ping timeout: 250 seconds) [08:07] *** luckcolor has quit IRC (Remote host closed the connection) [08:08] *** anhedonis has quit IRC (hub.se irc.underworld.no) [08:08] *** gibigiana has quit IRC (hub.se irc.underworld.no) [08:08] *** midas has quit IRC (hub.se irc.underworld.no) [08:08] *** Marc has quit IRC (hub.se irc.underworld.no) [08:08] *** Jonimus has quit IRC (hub.se irc.underworld.no) [08:08] *** JensRex has quit IRC (hub.se irc.underworld.no) [08:08] *** Lord_Nigh has quit IRC (hub.se irc.underworld.no) [08:08] *** andersju has quit IRC (hub.se irc.underworld.no) [08:08] *** alfiepate has quit IRC (hub.se irc.underworld.no) [08:08] *** Riviera has quit IRC (hub.se irc.underworld.no) [08:08] *** espes__ has quit IRC (hub.se irc.underworld.no) [08:08] *** tammy_ has quit IRC (hub.se irc.underworld.no) [08:08] *** i0npulse has quit IRC (hub.se irc.underworld.no) [08:08] *** Nemo_bis has quit IRC (hub.se irc.underworld.no) [08:08] *** Rai-chan has quit IRC (hub.se irc.underworld.no) [08:08] *** medowar has quit IRC (hub.se irc.underworld.no) [08:08] *** Hecatz has quit IRC (hub.se irc.underworld.no) [08:09] *** LordNigh2 has joined #archiveteam [08:09] *** luckcolor has joined #archiveteam [08:09] *** will has joined #archiveteam [08:10] *** alfie has joined #archiveteam [08:13] *** sun_rise has quit IRC (Read error: Connection reset by peer) [08:17] *** midas1 has joined #archiveteam [08:17] *** swebb sets mode: +o midas1 [08:17] *** Jonimoose has joined #archiveteam [08:17] *** swebb sets mode: +o Jonimoose [08:23] *** LordNigh2 is now known as Lord_Nigh [08:24] *** anhedonis has joined #archiveteam [08:57] *** atomotic has joined #archiveteam [09:12] *** Jonison has quit IRC (Read error: Connection reset by peer) [09:18] *** Jonison has joined #archiveteam [09:19] *** Somebody2 has quit IRC (Read error: Operation timed out) [09:20] *** Jonimoose has quit IRC (west.us.hub irc.Prison.NET) [09:20] *** db48x has quit IRC (west.us.hub irc.Prison.NET) [09:21] *** xmc has quit IRC (Read error: Operation timed out) [09:21] *** Somebody2 has joined #archiveteam [09:24] *** midas1 is now known as midas [09:26] *** xmc has joined #archiveteam [09:26] *** swebb sets mode: +o xmc [09:43] *** flyingzum has quit IRC (Remote host closed the connection) [09:43] *** deathy has quit IRC (Remote host closed the connection) [09:43] *** HCross2 has quit IRC (Remote host closed the connection) [09:47] *** JAA has joined #archiveteam [09:52] *** deathy has joined #archiveteam [10:02] *** atomotic has quit IRC (Remote host closed the connection) [10:04] *** atomotic has joined #archiveteam [10:15] *** flyingzum has joined #archiveteam [10:20] *** HCross2 has joined #archiveteam [10:23] *** iguana has joined #archiveteam [10:28] *** JAA has quit IRC (Quit: Page closed) [10:34] *** Jonimoose has joined #archiveteam [10:34] *** db48x has joined #archiveteam [10:34] *** irc.Prison.NET sets mode: +o Jonimoose [10:34] *** swebb sets mode: +o Jonimoose [10:36] *** Rai-chan has joined #archiveteam [10:36] *** medowar has joined #archiveteam [10:36] *** Hecatz has joined #archiveteam [10:38] *** Marc has joined #archiveteam [10:38] *** Nemo_bis has joined #archiveteam [10:39] *** i0npulse has joined #archiveteam [10:39] *** tammy_ has joined #archiveteam [10:41] *** andersju has joined #archiveteam [10:50] *** iguana has quit IRC (Ping timeout: 260 seconds) [11:03] *** JensRex has joined #archiveteam [11:03] *** dashcloud has quit IRC (Read error: Connection reset by peer) [11:04] *** dashcloud has joined #archiveteam [11:12] *** iguana has joined #archiveteam [11:12] *** hive-mind has quit IRC (Ping timeout: 260 seconds) [11:13] *** iguana has quit IRC (Client Quit) [11:14] *** hive-mind has joined #archiveteam [11:49] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [12:16] *** Morbus has joined #archiveteam [12:30] *** pizzaiolo has joined #archiveteam [12:39] *** atomotic has joined #archiveteam [13:05] *** BlueMaxim has quit IRC (Quit: Leaving) [13:23] *** topdownji has joined #archiveteam [14:02] *** JensRex has quit IRC (Remote host closed the connection) [14:03] *** JensRex has joined #archiveteam [14:20] *** Yurume has quit IRC (Remote host closed the connection) [14:20] *** antomati_ is now known as antomatic [14:31] *** Yurume has joined #archiveteam [14:42] *** octarine has quit IRC (Ping timeout: 260 seconds) [14:42] *** maxogden has quit IRC (Ping timeout: 260 seconds) [14:44] *** Dark_Star has quit IRC (Read error: Operation timed out) [14:44] *** maxogden has joined #archiveteam [14:44] *** hook54321 has quit IRC (Ping timeout: 250 seconds) [14:44] *** godane has quit IRC (Ping timeout: 250 seconds) [14:44] *** kanzure has quit IRC (Ping timeout: 250 seconds) [14:44] *** kanzure has joined #archiveteam [14:44] *** alembic has quit IRC (Ping timeout: 260 seconds) [14:44] *** Rondom has quit IRC (Remote host closed the connection) [14:44] *** QBcrusher has quit IRC (Ping timeout: 250 seconds) [14:46] *** Rondom has joined #archiveteam [14:46] *** dimeshake has joined #archiveteam [14:46] *** QBcrusher has joined #archiveteam [14:46] *** FalconK has quit IRC (Ping timeout: 260 seconds) [14:46] *** cheez has quit IRC (Ping timeout: 250 seconds) [14:46] *** gibigian1 has quit IRC (Ping timeout: 250 seconds) [14:47] *** gibigiana has joined #archiveteam [14:47] *** godane has joined #archiveteam [14:47] *** cheez has joined #archiveteam [14:58] *** logchfoo0 starts logging #archiveteam at Tue May 02 14:58:53 2017 [14:58] *** logchfoo0 has joined #archiveteam [14:59] *** hook54321 has joined #archiveteam [14:59] *** hive-mind has joined #archiveteam [15:00] *** alembic has joined #archiveteam [15:07] *** Ctrl-S___ has joined #archiveteam [15:13] *** nightpool has joined #archiveteam [15:26] *** icedice has joined #archiveteam [15:26] *** icedice2 has joined #archiveteam [15:30] *** khaoohs_ has joined #archiveteam [15:31] *** yipdw has quit IRC (Read error: Operation timed out) [15:32] *** atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) [15:33] *** me_ has joined #archiveteam [15:36] *** icedice2 has quit IRC (Quit: Leaving) [15:43] *** khaoohs has quit IRC (Read error: Operation timed out) [16:10] *** kris33 has joined #archiveteam [17:15] Hi [17:21] *** bRick5772 has joined #archiveteam [17:25] *** kris33 has quit IRC (Remote host closed the connection) [17:30] Nyaa was the biggest anime torrent site in the world, but the admin shut it down a few days ago since an ECJ ruling freaked him out: https://cdn.discordapp.com/attachments/290540858149961728/308992085518843905/image.png [17:30] I tried archiving a mirror site, but I don't think I got anything past the first page [17:30] https://nyaa.pantsu.cat/ [17:30] https://sukeibei.pantsu.cat/ [17:59] * namespace isn't sure if archiveteam really does actual piracy sites [18:00] is it part of our culture? [18:00] * pizzaiolo works with the assumption of don't ask don't tell [18:00] in my mind archiveteam is about preserving the full variety of culture that we see around ourselves [18:00] but this is maybe a -bs topic [18:01] xmc: I meant more for legal risk reasons than anything else. [18:01] But yeah this might be. [18:01] please join to #archiveteam-bs [18:12] *** me_ is now known as yipdw [18:42] Don't ask don't watch the new season of Orange is the New Black [19:06] *** ndiddy has quit IRC () [19:39] *** Hobart has joined #archiveteam [19:41] Are there any projects/toolsets that help unify archiving a deceased individuals work? A friend passed away on Saturday, and I can manually spider his various sites but if someone knows of a tool that makes it easier that'd be awesome. [19:42] (He had a few FB pages, two Twitter feeds, website, blogger, etc etc.) [19:51] Hobart: Archivebot can handle most websites and blogs. It can try Twitter, but sometimes it times out loading new data so it doesn't get a complete grab. Facebook requires an account for a full scrape so our usual tools don't handle it very well [19:52] *** bRick5772 has quit IRC (Quit: Leaving.) [20:30] *** schbirid has quit IRC (Quit: Leaving) [20:51] *** pfallenop has quit IRC (Read error: Operation timed out) [20:51] *** pfallenop has joined #archiveteam [21:04] *** tuluu_ has joined #archiveteam [21:04] *** tuluu has quit IRC (Ping timeout: 250 seconds) [21:07] *** Jonison has quit IRC (Read error: Connection reset by peer) [21:10] *** ndiddy has joined #archiveteam [21:13] archivebot crawls will become part of the public record on web.archive.org and can be downloaded as warc files [21:14] usually one file per crawl job (it splits files at, by default, 2 gigabytes) [21:28] I thought it was 5 gigabytes [21:29] (Unless it changed) [21:32] depends on how people have their pipelines configured. i know FalconK and i have ours at 2G [21:32] it's configurable. some pipelines are doing 2GB and some 5. [21:32] I lowered mine to 2GB because cloud disk is very expensive and it's helpful to clear them out a bit more smoothly. [21:58] *** espes__ has joined #archiveteam [21:59] *** espes___ has quit IRC (Ping timeout: 250 seconds) [21:59] Hobart: In addition to what MrRadar said, Facebook usually ends up giving up captchas [21:59] *us [22:02] *** iguana has joined #archiveteam [22:02] *** xhdr has quit IRC (Ping timeout: 250 seconds) [22:02] *** Deewiant has quit IRC (Ping timeout: 250 seconds) [22:02] *** midas has quit IRC (Ping timeout: 250 seconds) [22:02] *** Gfy has quit IRC (Ping timeout: 250 seconds) [22:02] *** sHATNER has quit IRC (Ping timeout: 250 seconds) [22:03] *** Deewiant has joined #archiveteam [22:03] *** mls has quit IRC (Ping timeout: 250 seconds) [22:03] *** iguana has quit IRC (Client Quit) [22:03] *** midas has joined #archiveteam [22:03] *** swebb sets mode: +o midas [22:04] *** tsr has quit IRC (Ping timeout: 250 seconds) [22:05] *** Gfy has joined #archiveteam [22:05] *** lesderid has quit IRC (Ping timeout: 250 seconds) [22:06] *** andai has quit IRC (Ping timeout: 250 seconds) [22:08] *** Kaz has quit IRC (Ping timeout: 250 seconds) [22:11] *** Aoede has quit IRC (Ping timeout: 250 seconds) [22:11] *** hook54321 has quit IRC (Ping timeout: 250 seconds) [22:11] *** C4K3 has quit IRC (Ping timeout: 250 seconds) [22:13] *** tsr has joined #archiveteam [22:13] *** HP_ has joined #archiveteam [22:13] *** C4K3 has joined #archiveteam [22:14] *** hook54321 has joined #archiveteam [22:14] *** andai has joined #archiveteam [22:14] *** xhdr has joined #archiveteam [22:14] *** HP has quit IRC (Ping timeout: 250 seconds) [22:14] *** nightpool has quit IRC (Ping timeout: 250 seconds) [22:14] *** lesderid has joined #archiveteam [22:15] *** Kaz has joined #archiveteam [22:16] *** mls has joined #archiveteam [22:17] *** andai has quit IRC (Ping timeout: 250 seconds) [22:17] *** SN4T14 has quit IRC (Ping timeout: 250 seconds) [22:17] *** SN4T14 has joined #archiveteam [22:17] *** xhdr has quit IRC (Ping timeout: 250 seconds) [22:21] *** mls has quit IRC (Ping timeout: 250 seconds) [22:21] *** xhdr has joined #archiveteam [22:21] *** mls has joined #archiveteam [22:21] *** xhdr has quit IRC (Excess Flood) [22:22] *** Aoede has joined #archiveteam [22:22] *** xhdr has joined #archiveteam [22:22] *** andai has joined #archiveteam [22:27] *** nightpool has joined #archiveteam [22:46] *** Aoede has quit IRC (Ping timeout: 250 seconds) [22:48] *** Aoede has joined #archiveteam [22:57] *** andai has quit IRC (Ping timeout: 250 seconds) [22:58] *** andai has joined #archiveteam [23:28] *** dashcloud has quit IRC (Remote host closed the connection) [23:30] *** BlueMaxim has joined #archiveteam [23:33] *** dashcloud has joined #archiveteam