#archiveteam 2017-05-02,Tue

↑back Search

Time Nickname Message
00:18 🔗 Meta has left
00:45 🔗 octarine has joined #archiveteam
01:22 🔗 brayden_ has joined #archiveteam
01:22 🔗 swebb sets mode: +o brayden_
01:26 🔗 brayden has quit IRC (Read error: Operation timed out)
02:14 🔗 Ctrl has joined #archiveteam
03:13 🔗 marshall has joined #archiveteam
03:18 🔗 marshall is now known as sun_rise
03:39 🔗 pizzaiolo has quit IRC (pizzaiolo)
03:42 🔗 iguana has quit IRC (Read error: Operation timed out)
04:17 🔗 Sk1d has quit IRC (Ping timeout: 250 seconds)
04:24 🔗 Sk1d has joined #archiveteam
04:24 🔗 Sk1d has quit IRC (Connection Closed)
04:31 🔗 ploop is there some kind of framework for automated mass downloading over http according to some kind of ruleset provided by the user?
04:32 🔗 ploop like, imagine someone has a file upload site where files are all under https://site.com/files/ and are named the date they're uploaded and an incremental integer
04:33 🔗 Somebody2 ploop: your best bet is probably wpull with some scripting
04:33 🔗 ploop wpull?
04:33 🔗 Somebody2 wpull is similar to wget, but better in various ways
04:33 🔗 Somebody2 written by one of the people associated with ArchiveTeam
04:34 🔗 Somebody2 http://wpull.readthedocs.org/
04:35 🔗 Somebody2 If you want to talk about this in more detail, come on over to #archiveteam-bs , which is the channel for discussion (rather than this one, which is more for announcements/initial questions).
04:35 🔗 ploop ah, ok
04:38 🔗 BlueMaxim has joined #archiveteam
04:47 🔗 Aranje has quit IRC (Quit: Three sheets to the wind)
05:06 🔗 Morbus has quit IRC (Read error: Operation timed out)
06:21 🔗 Ctrl has quit IRC (Ping timeout: 244 seconds)
07:44 🔗 Jonison has joined #archiveteam
07:53 🔗 schbirid has joined #archiveteam
07:56 🔗 anhedonis has quit IRC (Ping timeout: 244 seconds)
07:58 🔗 anhedonis has joined #archiveteam
08:00 🔗 Deewiant has quit IRC (Ping timeout: 250 seconds)
08:01 🔗 Deewiant has joined #archiveteam
08:04 🔗 gibigian1 has joined #archiveteam
08:05 🔗 espes___ has joined #archiveteam
08:06 🔗 will has quit IRC (Ping timeout: 250 seconds)
08:07 🔗 luckcolor has quit IRC (Remote host closed the connection)
08:08 🔗 anhedonis has quit IRC (hub.se irc.underworld.no)
08:08 🔗 gibigiana has quit IRC (hub.se irc.underworld.no)
08:08 🔗 midas has quit IRC (hub.se irc.underworld.no)
08:08 🔗 Marc has quit IRC (hub.se irc.underworld.no)
08:08 🔗 Jonimus has quit IRC (hub.se irc.underworld.no)
08:08 🔗 JensRex has quit IRC (hub.se irc.underworld.no)
08:08 🔗 Lord_Nigh has quit IRC (hub.se irc.underworld.no)
08:08 🔗 andersju has quit IRC (hub.se irc.underworld.no)
08:08 🔗 alfiepate has quit IRC (hub.se irc.underworld.no)
08:08 🔗 Riviera has quit IRC (hub.se irc.underworld.no)
08:08 🔗 espes__ has quit IRC (hub.se irc.underworld.no)
08:08 🔗 tammy_ has quit IRC (hub.se irc.underworld.no)
08:08 🔗 i0npulse has quit IRC (hub.se irc.underworld.no)
08:08 🔗 Nemo_bis has quit IRC (hub.se irc.underworld.no)
08:08 🔗 Rai-chan has quit IRC (hub.se irc.underworld.no)
08:08 🔗 medowar has quit IRC (hub.se irc.underworld.no)
08:08 🔗 Hecatz has quit IRC (hub.se irc.underworld.no)
08:09 🔗 LordNigh2 has joined #archiveteam
08:09 🔗 luckcolor has joined #archiveteam
08:09 🔗 will has joined #archiveteam
08:10 🔗 alfie has joined #archiveteam
08:13 🔗 sun_rise has quit IRC (Read error: Connection reset by peer)
08:17 🔗 midas1 has joined #archiveteam
08:17 🔗 swebb sets mode: +o midas1
08:17 🔗 Jonimoose has joined #archiveteam
08:17 🔗 swebb sets mode: +o Jonimoose
08:23 🔗 LordNigh2 is now known as Lord_Nigh
08:24 🔗 anhedonis has joined #archiveteam
08:57 🔗 atomotic has joined #archiveteam
09:12 🔗 Jonison has quit IRC (Read error: Connection reset by peer)
09:18 🔗 Jonison has joined #archiveteam
09:19 🔗 Somebody2 has quit IRC (Read error: Operation timed out)
09:20 🔗 Jonimoose has quit IRC (west.us.hub irc.Prison.NET)
09:20 🔗 db48x has quit IRC (west.us.hub irc.Prison.NET)
09:21 🔗 xmc has quit IRC (Read error: Operation timed out)
09:21 🔗 Somebody2 has joined #archiveteam
09:24 🔗 midas1 is now known as midas
09:26 🔗 xmc has joined #archiveteam
09:26 🔗 swebb sets mode: +o xmc
09:43 🔗 flyingzum has quit IRC (Remote host closed the connection)
09:43 🔗 deathy has quit IRC (Remote host closed the connection)
09:43 🔗 HCross2 has quit IRC (Remote host closed the connection)
09:47 🔗 JAA has joined #archiveteam
09:52 🔗 deathy has joined #archiveteam
10:02 🔗 atomotic has quit IRC (Remote host closed the connection)
10:04 🔗 atomotic has joined #archiveteam
10:15 🔗 flyingzum has joined #archiveteam
10:20 🔗 HCross2 has joined #archiveteam
10:23 🔗 iguana has joined #archiveteam
10:28 🔗 JAA has quit IRC (Quit: Page closed)
10:34 🔗 Jonimoose has joined #archiveteam
10:34 🔗 db48x has joined #archiveteam
10:34 🔗 irc.Prison.NET sets mode: +o Jonimoose
10:34 🔗 swebb sets mode: +o Jonimoose
10:36 🔗 Rai-chan has joined #archiveteam
10:36 🔗 medowar has joined #archiveteam
10:36 🔗 Hecatz has joined #archiveteam
10:38 🔗 Marc has joined #archiveteam
10:38 🔗 Nemo_bis has joined #archiveteam
10:39 🔗 i0npulse has joined #archiveteam
10:39 🔗 tammy_ has joined #archiveteam
10:41 🔗 andersju has joined #archiveteam
10:50 🔗 iguana has quit IRC (Ping timeout: 260 seconds)
11:03 🔗 JensRex has joined #archiveteam
11:03 🔗 dashcloud has quit IRC (Read error: Connection reset by peer)
11:04 🔗 dashcloud has joined #archiveteam
11:12 🔗 iguana has joined #archiveteam
11:12 🔗 hive-mind has quit IRC (Ping timeout: 260 seconds)
11:13 🔗 iguana has quit IRC (Client Quit)
11:14 🔗 hive-mind has joined #archiveteam
11:49 🔗 atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
12:16 🔗 Morbus has joined #archiveteam
12:30 🔗 pizzaiolo has joined #archiveteam
12:39 🔗 atomotic has joined #archiveteam
13:05 🔗 BlueMaxim has quit IRC (Quit: Leaving)
13:23 🔗 topdownji has joined #archiveteam
14:02 🔗 JensRex has quit IRC (Remote host closed the connection)
14:03 🔗 JensRex has joined #archiveteam
14:20 🔗 Yurume has quit IRC (Remote host closed the connection)
14:20 🔗 antomati_ is now known as antomatic
14:31 🔗 Yurume has joined #archiveteam
14:42 🔗 octarine has quit IRC (Ping timeout: 260 seconds)
14:42 🔗 maxogden has quit IRC (Ping timeout: 260 seconds)
14:44 🔗 Dark_Star has quit IRC (Read error: Operation timed out)
14:44 🔗 maxogden has joined #archiveteam
14:44 🔗 hook54321 has quit IRC (Ping timeout: 250 seconds)
14:44 🔗 godane has quit IRC (Ping timeout: 250 seconds)
14:44 🔗 kanzure has quit IRC (Ping timeout: 250 seconds)
14:44 🔗 kanzure has joined #archiveteam
14:44 🔗 alembic has quit IRC (Ping timeout: 260 seconds)
14:44 🔗 Rondom has quit IRC (Remote host closed the connection)
14:44 🔗 QBcrusher has quit IRC (Ping timeout: 250 seconds)
14:46 🔗 Rondom has joined #archiveteam
14:46 🔗 dimeshake has joined #archiveteam
14:46 🔗 QBcrusher has joined #archiveteam
14:46 🔗 FalconK has quit IRC (Ping timeout: 260 seconds)
14:46 🔗 cheez has quit IRC (Ping timeout: 250 seconds)
14:46 🔗 gibigian1 has quit IRC (Ping timeout: 250 seconds)
14:47 🔗 gibigiana has joined #archiveteam
14:47 🔗 godane has joined #archiveteam
14:47 🔗 cheez has joined #archiveteam
14:58 🔗 logchfoo0 starts logging #archiveteam at Tue May 02 14:58:53 2017
14:58 🔗 logchfoo0 has joined #archiveteam
14:59 🔗 hook54321 has joined #archiveteam
14:59 🔗 hive-mind has joined #archiveteam
15:00 🔗 alembic has joined #archiveteam
15:07 🔗 Ctrl-S___ has joined #archiveteam
15:13 🔗 nightpool has joined #archiveteam
15:26 🔗 icedice has joined #archiveteam
15:26 🔗 icedice2 has joined #archiveteam
15:30 🔗 khaoohs_ has joined #archiveteam
15:31 🔗 yipdw has quit IRC (Read error: Operation timed out)
15:32 🔗 atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
15:33 🔗 me_ has joined #archiveteam
15:36 🔗 icedice2 has quit IRC (Quit: Leaving)
15:43 🔗 khaoohs has quit IRC (Read error: Operation timed out)
16:10 🔗 kris33 has joined #archiveteam
17:15 🔗 icedice Hi
17:21 🔗 bRick5772 has joined #archiveteam
17:25 🔗 kris33 has quit IRC (Remote host closed the connection)
17:30 🔗 icedice Nyaa was the biggest anime torrent site in the world, but the admin shut it down a few days ago since an ECJ ruling freaked him out: https://cdn.discordapp.com/attachments/290540858149961728/308992085518843905/image.png
17:30 🔗 icedice I tried archiving a mirror site, but I don't think I got anything past the first page
17:30 🔗 icedice https://nyaa.pantsu.cat/
17:30 🔗 icedice https://sukeibei.pantsu.cat/
17:59 🔗 * namespace isn't sure if archiveteam really does actual piracy sites
18:00 🔗 xmc is it part of our culture?
18:00 🔗 * pizzaiolo works with the assumption of don't ask don't tell
18:00 🔗 xmc in my mind archiveteam is about preserving the full variety of culture that we see around ourselves
18:00 🔗 xmc but this is maybe a -bs topic
18:01 🔗 namespace xmc: I meant more for legal risk reasons than anything else.
18:01 🔗 namespace But yeah this might be.
18:01 🔗 xmc please join to #archiveteam-bs
18:12 🔗 me_ is now known as yipdw
18:42 🔗 SketchCow Don't ask don't watch the new season of Orange is the New Black
19:06 🔗 ndiddy has quit IRC ()
19:39 🔗 Hobart has joined #archiveteam
19:41 🔗 Hobart Are there any projects/toolsets that help unify archiving a deceased individuals work? A friend passed away on Saturday, and I can manually spider his various sites but if someone knows of a tool that makes it easier that'd be awesome.
19:42 🔗 Hobart (He had a few FB pages, two Twitter feeds, website, blogger, etc etc.)
19:51 🔗 MrRadar Hobart: Archivebot can handle most websites and blogs. It can try Twitter, but sometimes it times out loading new data so it doesn't get a complete grab. Facebook requires an account for a full scrape so our usual tools don't handle it very well
19:52 🔗 bRick5772 has quit IRC (Quit: Leaving.)
20:30 🔗 schbirid has quit IRC (Quit: Leaving)
20:51 🔗 pfallenop has quit IRC (Read error: Operation timed out)
20:51 🔗 pfallenop has joined #archiveteam
21:04 🔗 tuluu_ has joined #archiveteam
21:04 🔗 tuluu has quit IRC (Ping timeout: 250 seconds)
21:07 🔗 Jonison has quit IRC (Read error: Connection reset by peer)
21:10 🔗 ndiddy has joined #archiveteam
21:13 🔗 xmc archivebot crawls will become part of the public record on web.archive.org and can be downloaded as warc files
21:14 🔗 xmc usually one file per crawl job (it splits files at, by default, 2 gigabytes)
21:28 🔗 Frogging I thought it was 5 gigabytes
21:29 🔗 Frogging (Unless it changed)
21:32 🔗 xmc depends on how people have their pipelines configured. i know FalconK and i have ours at 2G
21:32 🔗 FalconK it's configurable. some pipelines are doing 2GB and some 5.
21:32 🔗 FalconK I lowered mine to 2GB because cloud disk is very expensive and it's helpful to clear them out a bit more smoothly.
21:58 🔗 espes__ has joined #archiveteam
21:59 🔗 espes___ has quit IRC (Ping timeout: 250 seconds)
21:59 🔗 hook54321 Hobart: In addition to what MrRadar said, Facebook usually ends up giving up captchas
21:59 🔗 hook54321 *us
22:02 🔗 iguana has joined #archiveteam
22:02 🔗 xhdr has quit IRC (Ping timeout: 250 seconds)
22:02 🔗 Deewiant has quit IRC (Ping timeout: 250 seconds)
22:02 🔗 midas has quit IRC (Ping timeout: 250 seconds)
22:02 🔗 Gfy has quit IRC (Ping timeout: 250 seconds)
22:02 🔗 sHATNER has quit IRC (Ping timeout: 250 seconds)
22:03 🔗 Deewiant has joined #archiveteam
22:03 🔗 mls has quit IRC (Ping timeout: 250 seconds)
22:03 🔗 iguana has quit IRC (Client Quit)
22:03 🔗 midas has joined #archiveteam
22:03 🔗 swebb sets mode: +o midas
22:04 🔗 tsr has quit IRC (Ping timeout: 250 seconds)
22:05 🔗 Gfy has joined #archiveteam
22:05 🔗 lesderid has quit IRC (Ping timeout: 250 seconds)
22:06 🔗 andai has quit IRC (Ping timeout: 250 seconds)
22:08 🔗 Kaz has quit IRC (Ping timeout: 250 seconds)
22:11 🔗 Aoede has quit IRC (Ping timeout: 250 seconds)
22:11 🔗 hook54321 has quit IRC (Ping timeout: 250 seconds)
22:11 🔗 C4K3 has quit IRC (Ping timeout: 250 seconds)
22:13 🔗 tsr has joined #archiveteam
22:13 🔗 HP_ has joined #archiveteam
22:13 🔗 C4K3 has joined #archiveteam
22:14 🔗 hook54321 has joined #archiveteam
22:14 🔗 andai has joined #archiveteam
22:14 🔗 xhdr has joined #archiveteam
22:14 🔗 HP has quit IRC (Ping timeout: 250 seconds)
22:14 🔗 nightpool has quit IRC (Ping timeout: 250 seconds)
22:14 🔗 lesderid has joined #archiveteam
22:15 🔗 Kaz has joined #archiveteam
22:16 🔗 mls has joined #archiveteam
22:17 🔗 andai has quit IRC (Ping timeout: 250 seconds)
22:17 🔗 SN4T14 has quit IRC (Ping timeout: 250 seconds)
22:17 🔗 SN4T14 has joined #archiveteam
22:17 🔗 xhdr has quit IRC (Ping timeout: 250 seconds)
22:21 🔗 mls has quit IRC (Ping timeout: 250 seconds)
22:21 🔗 xhdr has joined #archiveteam
22:21 🔗 mls has joined #archiveteam
22:21 🔗 xhdr has quit IRC (Excess Flood)
22:22 🔗 Aoede has joined #archiveteam
22:22 🔗 xhdr has joined #archiveteam
22:22 🔗 andai has joined #archiveteam
22:27 🔗 nightpool has joined #archiveteam
22:46 🔗 Aoede has quit IRC (Ping timeout: 250 seconds)
22:48 🔗 Aoede has joined #archiveteam
22:57 🔗 andai has quit IRC (Ping timeout: 250 seconds)
22:58 🔗 andai has joined #archiveteam
23:28 🔗 dashcloud has quit IRC (Remote host closed the connection)
23:30 🔗 BlueMaxim has joined #archiveteam
23:33 🔗 dashcloud has joined #archiveteam

irclogger-viewer