| Time |
Nickname |
Message |
|
00:18
🔗
|
|
Meta has left |
|
00:45
🔗
|
|
octarine has joined #archiveteam |
|
01:22
🔗
|
|
brayden_ has joined #archiveteam |
|
01:22
🔗
|
|
swebb sets mode: +o brayden_ |
|
01:26
🔗
|
|
brayden has quit IRC (Read error: Operation timed out) |
|
02:14
🔗
|
|
Ctrl has joined #archiveteam |
|
03:13
🔗
|
|
marshall has joined #archiveteam |
|
03:18
🔗
|
|
marshall is now known as sun_rise |
|
03:39
🔗
|
|
pizzaiolo has quit IRC (pizzaiolo) |
|
03:42
🔗
|
|
iguana has quit IRC (Read error: Operation timed out) |
|
04:17
🔗
|
|
Sk1d has quit IRC (Ping timeout: 250 seconds) |
|
04:24
🔗
|
|
Sk1d has joined #archiveteam |
|
04:24
🔗
|
|
Sk1d has quit IRC (Connection Closed) |
|
04:31
🔗
|
ploop |
is there some kind of framework for automated mass downloading over http according to some kind of ruleset provided by the user? |
|
04:32
🔗
|
ploop |
like, imagine someone has a file upload site where files are all under https://site.com/files/ and are named the date they're uploaded and an incremental integer |
|
04:33
🔗
|
Somebody2 |
ploop: your best bet is probably wpull with some scripting |
|
04:33
🔗
|
ploop |
wpull? |
|
04:33
🔗
|
Somebody2 |
wpull is similar to wget, but better in various ways |
|
04:33
🔗
|
Somebody2 |
written by one of the people associated with ArchiveTeam |
|
04:34
🔗
|
Somebody2 |
http://wpull.readthedocs.org/ |
|
04:35
🔗
|
Somebody2 |
If you want to talk about this in more detail, come on over to #archiveteam-bs , which is the channel for discussion (rather than this one, which is more for announcements/initial questions). |
|
04:35
🔗
|
ploop |
ah, ok |
|
04:38
🔗
|
|
BlueMaxim has joined #archiveteam |
|
04:47
🔗
|
|
Aranje has quit IRC (Quit: Three sheets to the wind) |
|
05:06
🔗
|
|
Morbus has quit IRC (Read error: Operation timed out) |
|
06:21
🔗
|
|
Ctrl has quit IRC (Ping timeout: 244 seconds) |
|
07:44
🔗
|
|
Jonison has joined #archiveteam |
|
07:53
🔗
|
|
schbirid has joined #archiveteam |
|
07:56
🔗
|
|
anhedonis has quit IRC (Ping timeout: 244 seconds) |
|
07:58
🔗
|
|
anhedonis has joined #archiveteam |
|
08:00
🔗
|
|
Deewiant has quit IRC (Ping timeout: 250 seconds) |
|
08:01
🔗
|
|
Deewiant has joined #archiveteam |
|
08:04
🔗
|
|
gibigian1 has joined #archiveteam |
|
08:05
🔗
|
|
espes___ has joined #archiveteam |
|
08:06
🔗
|
|
will has quit IRC (Ping timeout: 250 seconds) |
|
08:07
🔗
|
|
luckcolor has quit IRC (Remote host closed the connection) |
|
08:08
🔗
|
|
anhedonis has quit IRC (hub.se irc.underworld.no) |
|
08:08
🔗
|
|
gibigiana has quit IRC (hub.se irc.underworld.no) |
|
08:08
🔗
|
|
midas has quit IRC (hub.se irc.underworld.no) |
|
08:08
🔗
|
|
Marc has quit IRC (hub.se irc.underworld.no) |
|
08:08
🔗
|
|
Jonimus has quit IRC (hub.se irc.underworld.no) |
|
08:08
🔗
|
|
JensRex has quit IRC (hub.se irc.underworld.no) |
|
08:08
🔗
|
|
Lord_Nigh has quit IRC (hub.se irc.underworld.no) |
|
08:08
🔗
|
|
andersju has quit IRC (hub.se irc.underworld.no) |
|
08:08
🔗
|
|
alfiepate has quit IRC (hub.se irc.underworld.no) |
|
08:08
🔗
|
|
Riviera has quit IRC (hub.se irc.underworld.no) |
|
08:08
🔗
|
|
espes__ has quit IRC (hub.se irc.underworld.no) |
|
08:08
🔗
|
|
tammy_ has quit IRC (hub.se irc.underworld.no) |
|
08:08
🔗
|
|
i0npulse has quit IRC (hub.se irc.underworld.no) |
|
08:08
🔗
|
|
Nemo_bis has quit IRC (hub.se irc.underworld.no) |
|
08:08
🔗
|
|
Rai-chan has quit IRC (hub.se irc.underworld.no) |
|
08:08
🔗
|
|
medowar has quit IRC (hub.se irc.underworld.no) |
|
08:08
🔗
|
|
Hecatz has quit IRC (hub.se irc.underworld.no) |
|
08:09
🔗
|
|
LordNigh2 has joined #archiveteam |
|
08:09
🔗
|
|
luckcolor has joined #archiveteam |
|
08:09
🔗
|
|
will has joined #archiveteam |
|
08:10
🔗
|
|
alfie has joined #archiveteam |
|
08:13
🔗
|
|
sun_rise has quit IRC (Read error: Connection reset by peer) |
|
08:17
🔗
|
|
midas1 has joined #archiveteam |
|
08:17
🔗
|
|
swebb sets mode: +o midas1 |
|
08:17
🔗
|
|
Jonimoose has joined #archiveteam |
|
08:17
🔗
|
|
swebb sets mode: +o Jonimoose |
|
08:23
🔗
|
|
LordNigh2 is now known as Lord_Nigh |
|
08:24
🔗
|
|
anhedonis has joined #archiveteam |
|
08:57
🔗
|
|
atomotic has joined #archiveteam |
|
09:12
🔗
|
|
Jonison has quit IRC (Read error: Connection reset by peer) |
|
09:18
🔗
|
|
Jonison has joined #archiveteam |
|
09:19
🔗
|
|
Somebody2 has quit IRC (Read error: Operation timed out) |
|
09:20
🔗
|
|
Jonimoose has quit IRC (west.us.hub irc.Prison.NET) |
|
09:20
🔗
|
|
db48x has quit IRC (west.us.hub irc.Prison.NET) |
|
09:21
🔗
|
|
xmc has quit IRC (Read error: Operation timed out) |
|
09:21
🔗
|
|
Somebody2 has joined #archiveteam |
|
09:24
🔗
|
|
midas1 is now known as midas |
|
09:26
🔗
|
|
xmc has joined #archiveteam |
|
09:26
🔗
|
|
swebb sets mode: +o xmc |
|
09:43
🔗
|
|
flyingzum has quit IRC (Remote host closed the connection) |
|
09:43
🔗
|
|
deathy has quit IRC (Remote host closed the connection) |
|
09:43
🔗
|
|
HCross2 has quit IRC (Remote host closed the connection) |
|
09:47
🔗
|
|
JAA has joined #archiveteam |
|
09:52
🔗
|
|
deathy has joined #archiveteam |
|
10:02
🔗
|
|
atomotic has quit IRC (Remote host closed the connection) |
|
10:04
🔗
|
|
atomotic has joined #archiveteam |
|
10:15
🔗
|
|
flyingzum has joined #archiveteam |
|
10:20
🔗
|
|
HCross2 has joined #archiveteam |
|
10:23
🔗
|
|
iguana has joined #archiveteam |
|
10:28
🔗
|
|
JAA has quit IRC (Quit: Page closed) |
|
10:34
🔗
|
|
Jonimoose has joined #archiveteam |
|
10:34
🔗
|
|
db48x has joined #archiveteam |
|
10:34
🔗
|
|
irc.Prison.NET sets mode: +o Jonimoose |
|
10:34
🔗
|
|
swebb sets mode: +o Jonimoose |
|
10:36
🔗
|
|
Rai-chan has joined #archiveteam |
|
10:36
🔗
|
|
medowar has joined #archiveteam |
|
10:36
🔗
|
|
Hecatz has joined #archiveteam |
|
10:38
🔗
|
|
Marc has joined #archiveteam |
|
10:38
🔗
|
|
Nemo_bis has joined #archiveteam |
|
10:39
🔗
|
|
i0npulse has joined #archiveteam |
|
10:39
🔗
|
|
tammy_ has joined #archiveteam |
|
10:41
🔗
|
|
andersju has joined #archiveteam |
|
10:50
🔗
|
|
iguana has quit IRC (Ping timeout: 260 seconds) |
|
11:03
🔗
|
|
JensRex has joined #archiveteam |
|
11:03
🔗
|
|
dashcloud has quit IRC (Read error: Connection reset by peer) |
|
11:04
🔗
|
|
dashcloud has joined #archiveteam |
|
11:12
🔗
|
|
iguana has joined #archiveteam |
|
11:12
🔗
|
|
hive-mind has quit IRC (Ping timeout: 260 seconds) |
|
11:13
🔗
|
|
iguana has quit IRC (Client Quit) |
|
11:14
🔗
|
|
hive-mind has joined #archiveteam |
|
11:49
🔗
|
|
atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) |
|
12:16
🔗
|
|
Morbus has joined #archiveteam |
|
12:30
🔗
|
|
pizzaiolo has joined #archiveteam |
|
12:39
🔗
|
|
atomotic has joined #archiveteam |
|
13:05
🔗
|
|
BlueMaxim has quit IRC (Quit: Leaving) |
|
13:23
🔗
|
|
topdownji has joined #archiveteam |
|
14:02
🔗
|
|
JensRex has quit IRC (Remote host closed the connection) |
|
14:03
🔗
|
|
JensRex has joined #archiveteam |
|
14:20
🔗
|
|
Yurume has quit IRC (Remote host closed the connection) |
|
14:20
🔗
|
|
antomati_ is now known as antomatic |
|
14:31
🔗
|
|
Yurume has joined #archiveteam |
|
14:42
🔗
|
|
octarine has quit IRC (Ping timeout: 260 seconds) |
|
14:42
🔗
|
|
maxogden has quit IRC (Ping timeout: 260 seconds) |
|
14:44
🔗
|
|
Dark_Star has quit IRC (Read error: Operation timed out) |
|
14:44
🔗
|
|
maxogden has joined #archiveteam |
|
14:44
🔗
|
|
hook54321 has quit IRC (Ping timeout: 250 seconds) |
|
14:44
🔗
|
|
godane has quit IRC (Ping timeout: 250 seconds) |
|
14:44
🔗
|
|
kanzure has quit IRC (Ping timeout: 250 seconds) |
|
14:44
🔗
|
|
kanzure has joined #archiveteam |
|
14:44
🔗
|
|
alembic has quit IRC (Ping timeout: 260 seconds) |
|
14:44
🔗
|
|
Rondom has quit IRC (Remote host closed the connection) |
|
14:44
🔗
|
|
QBcrusher has quit IRC (Ping timeout: 250 seconds) |
|
14:46
🔗
|
|
Rondom has joined #archiveteam |
|
14:46
🔗
|
|
dimeshake has joined #archiveteam |
|
14:46
🔗
|
|
QBcrusher has joined #archiveteam |
|
14:46
🔗
|
|
FalconK has quit IRC (Ping timeout: 260 seconds) |
|
14:46
🔗
|
|
cheez has quit IRC (Ping timeout: 250 seconds) |
|
14:46
🔗
|
|
gibigian1 has quit IRC (Ping timeout: 250 seconds) |
|
14:47
🔗
|
|
gibigiana has joined #archiveteam |
|
14:47
🔗
|
|
godane has joined #archiveteam |
|
14:47
🔗
|
|
cheez has joined #archiveteam |
|
14:58
🔗
|
|
logchfoo0 starts logging #archiveteam at Tue May 02 14:58:53 2017 |
|
14:58
🔗
|
|
logchfoo0 has joined #archiveteam |
|
14:59
🔗
|
|
hook54321 has joined #archiveteam |
|
14:59
🔗
|
|
hive-mind has joined #archiveteam |
|
15:00
🔗
|
|
alembic has joined #archiveteam |
|
15:07
🔗
|
|
Ctrl-S___ has joined #archiveteam |
|
15:13
🔗
|
|
nightpool has joined #archiveteam |
|
15:26
🔗
|
|
icedice has joined #archiveteam |
|
15:26
🔗
|
|
icedice2 has joined #archiveteam |
|
15:30
🔗
|
|
khaoohs_ has joined #archiveteam |
|
15:31
🔗
|
|
yipdw has quit IRC (Read error: Operation timed out) |
|
15:32
🔗
|
|
atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) |
|
15:33
🔗
|
|
me_ has joined #archiveteam |
|
15:36
🔗
|
|
icedice2 has quit IRC (Quit: Leaving) |
|
15:43
🔗
|
|
khaoohs has quit IRC (Read error: Operation timed out) |
|
16:10
🔗
|
|
kris33 has joined #archiveteam |
|
17:15
🔗
|
icedice |
Hi |
|
17:21
🔗
|
|
bRick5772 has joined #archiveteam |
|
17:25
🔗
|
|
kris33 has quit IRC (Remote host closed the connection) |
|
17:30
🔗
|
icedice |
Nyaa was the biggest anime torrent site in the world, but the admin shut it down a few days ago since an ECJ ruling freaked him out: https://cdn.discordapp.com/attachments/290540858149961728/308992085518843905/image.png |
|
17:30
🔗
|
icedice |
I tried archiving a mirror site, but I don't think I got anything past the first page |
|
17:30
🔗
|
icedice |
https://nyaa.pantsu.cat/ |
|
17:30
🔗
|
icedice |
https://sukeibei.pantsu.cat/ |
|
17:59
🔗
|
* |
namespace isn't sure if archiveteam really does actual piracy sites |
|
18:00
🔗
|
xmc |
is it part of our culture? |
|
18:00
🔗
|
* |
pizzaiolo works with the assumption of don't ask don't tell |
|
18:00
🔗
|
xmc |
in my mind archiveteam is about preserving the full variety of culture that we see around ourselves |
|
18:00
🔗
|
xmc |
but this is maybe a -bs topic |
|
18:01
🔗
|
namespace |
xmc: I meant more for legal risk reasons than anything else. |
|
18:01
🔗
|
namespace |
But yeah this might be. |
|
18:01
🔗
|
xmc |
please join to #archiveteam-bs |
|
18:12
🔗
|
|
me_ is now known as yipdw |
|
18:42
🔗
|
SketchCow |
Don't ask don't watch the new season of Orange is the New Black |
|
19:06
🔗
|
|
ndiddy has quit IRC () |
|
19:39
🔗
|
|
Hobart has joined #archiveteam |
|
19:41
🔗
|
Hobart |
Are there any projects/toolsets that help unify archiving a deceased individuals work? A friend passed away on Saturday, and I can manually spider his various sites but if someone knows of a tool that makes it easier that'd be awesome. |
|
19:42
🔗
|
Hobart |
(He had a few FB pages, two Twitter feeds, website, blogger, etc etc.) |
|
19:51
🔗
|
MrRadar |
Hobart: Archivebot can handle most websites and blogs. It can try Twitter, but sometimes it times out loading new data so it doesn't get a complete grab. Facebook requires an account for a full scrape so our usual tools don't handle it very well |
|
19:52
🔗
|
|
bRick5772 has quit IRC (Quit: Leaving.) |
|
20:30
🔗
|
|
schbirid has quit IRC (Quit: Leaving) |
|
20:51
🔗
|
|
pfallenop has quit IRC (Read error: Operation timed out) |
|
20:51
🔗
|
|
pfallenop has joined #archiveteam |
|
21:04
🔗
|
|
tuluu_ has joined #archiveteam |
|
21:04
🔗
|
|
tuluu has quit IRC (Ping timeout: 250 seconds) |
|
21:07
🔗
|
|
Jonison has quit IRC (Read error: Connection reset by peer) |
|
21:10
🔗
|
|
ndiddy has joined #archiveteam |
|
21:13
🔗
|
xmc |
archivebot crawls will become part of the public record on web.archive.org and can be downloaded as warc files |
|
21:14
🔗
|
xmc |
usually one file per crawl job (it splits files at, by default, 2 gigabytes) |
|
21:28
🔗
|
Frogging |
I thought it was 5 gigabytes |
|
21:29
🔗
|
Frogging |
(Unless it changed) |
|
21:32
🔗
|
xmc |
depends on how people have their pipelines configured. i know FalconK and i have ours at 2G |
|
21:32
🔗
|
FalconK |
it's configurable. some pipelines are doing 2GB and some 5. |
|
21:32
🔗
|
FalconK |
I lowered mine to 2GB because cloud disk is very expensive and it's helpful to clear them out a bit more smoothly. |
|
21:58
🔗
|
|
espes__ has joined #archiveteam |
|
21:59
🔗
|
|
espes___ has quit IRC (Ping timeout: 250 seconds) |
|
21:59
🔗
|
hook54321 |
Hobart: In addition to what MrRadar said, Facebook usually ends up giving up captchas |
|
21:59
🔗
|
hook54321 |
*us |
|
22:02
🔗
|
|
iguana has joined #archiveteam |
|
22:02
🔗
|
|
xhdr has quit IRC (Ping timeout: 250 seconds) |
|
22:02
🔗
|
|
Deewiant has quit IRC (Ping timeout: 250 seconds) |
|
22:02
🔗
|
|
midas has quit IRC (Ping timeout: 250 seconds) |
|
22:02
🔗
|
|
Gfy has quit IRC (Ping timeout: 250 seconds) |
|
22:02
🔗
|
|
sHATNER has quit IRC (Ping timeout: 250 seconds) |
|
22:03
🔗
|
|
Deewiant has joined #archiveteam |
|
22:03
🔗
|
|
mls has quit IRC (Ping timeout: 250 seconds) |
|
22:03
🔗
|
|
iguana has quit IRC (Client Quit) |
|
22:03
🔗
|
|
midas has joined #archiveteam |
|
22:03
🔗
|
|
swebb sets mode: +o midas |
|
22:04
🔗
|
|
tsr has quit IRC (Ping timeout: 250 seconds) |
|
22:05
🔗
|
|
Gfy has joined #archiveteam |
|
22:05
🔗
|
|
lesderid has quit IRC (Ping timeout: 250 seconds) |
|
22:06
🔗
|
|
andai has quit IRC (Ping timeout: 250 seconds) |
|
22:08
🔗
|
|
Kaz has quit IRC (Ping timeout: 250 seconds) |
|
22:11
🔗
|
|
Aoede has quit IRC (Ping timeout: 250 seconds) |
|
22:11
🔗
|
|
hook54321 has quit IRC (Ping timeout: 250 seconds) |
|
22:11
🔗
|
|
C4K3 has quit IRC (Ping timeout: 250 seconds) |
|
22:13
🔗
|
|
tsr has joined #archiveteam |
|
22:13
🔗
|
|
HP_ has joined #archiveteam |
|
22:13
🔗
|
|
C4K3 has joined #archiveteam |
|
22:14
🔗
|
|
hook54321 has joined #archiveteam |
|
22:14
🔗
|
|
andai has joined #archiveteam |
|
22:14
🔗
|
|
xhdr has joined #archiveteam |
|
22:14
🔗
|
|
HP has quit IRC (Ping timeout: 250 seconds) |
|
22:14
🔗
|
|
nightpool has quit IRC (Ping timeout: 250 seconds) |
|
22:14
🔗
|
|
lesderid has joined #archiveteam |
|
22:15
🔗
|
|
Kaz has joined #archiveteam |
|
22:16
🔗
|
|
mls has joined #archiveteam |
|
22:17
🔗
|
|
andai has quit IRC (Ping timeout: 250 seconds) |
|
22:17
🔗
|
|
SN4T14 has quit IRC (Ping timeout: 250 seconds) |
|
22:17
🔗
|
|
SN4T14 has joined #archiveteam |
|
22:17
🔗
|
|
xhdr has quit IRC (Ping timeout: 250 seconds) |
|
22:21
🔗
|
|
mls has quit IRC (Ping timeout: 250 seconds) |
|
22:21
🔗
|
|
xhdr has joined #archiveteam |
|
22:21
🔗
|
|
mls has joined #archiveteam |
|
22:21
🔗
|
|
xhdr has quit IRC (Excess Flood) |
|
22:22
🔗
|
|
Aoede has joined #archiveteam |
|
22:22
🔗
|
|
xhdr has joined #archiveteam |
|
22:22
🔗
|
|
andai has joined #archiveteam |
|
22:27
🔗
|
|
nightpool has joined #archiveteam |
|
22:46
🔗
|
|
Aoede has quit IRC (Ping timeout: 250 seconds) |
|
22:48
🔗
|
|
Aoede has joined #archiveteam |
|
22:57
🔗
|
|
andai has quit IRC (Ping timeout: 250 seconds) |
|
22:58
🔗
|
|
andai has joined #archiveteam |
|
23:28
🔗
|
|
dashcloud has quit IRC (Remote host closed the connection) |
|
23:30
🔗
|
|
BlueMaxim has joined #archiveteam |
|
23:33
🔗
|
|
dashcloud has joined #archiveteam |