[00:24] *** Ryz has joined #archiveteam-ot [00:35] *** m007a83 has joined #archiveteam-ot [01:49] *** Ryz has quit IRC (Quit: ChatZilla 0.9.92-rdmsoft [XULRunner 35.0.1/20150122214805]) [02:47] *** moufu has quit IRC (Read error: Operation timed out) [02:47] *** moufu has joined #archiveteam-ot [02:50] *** Mateon1 has quit IRC (Ping timeout: 268 seconds) [02:50] *** Mateon1 has joined #archiveteam-ot [02:54] *** Tenebrae has quit IRC (Ping timeout: 365 seconds) [02:54] *** MrRadar2 has quit IRC (Ping timeout: 268 seconds) [02:54] *** BnAboyZ has quit IRC (Ping timeout: 268 seconds) [02:55] *** scifi_ has quit IRC (Ping timeout: 268 seconds) [02:55] *** VoynichCr has quit IRC (Ping timeout: 268 seconds) [02:56] *** Tenebrae has joined #archiveteam-ot [02:57] *** VoynichCr has joined #archiveteam-ot [02:57] *** scifi_ has joined #archiveteam-ot [02:58] *** BnAboyZ has joined #archiveteam-ot [02:59] *** MrRadar2 has joined #archiveteam-ot [04:06] *** godane has joined #archiveteam-ot [04:18] *** BlueMax has quit IRC (Quit: Leaving) [04:22] *** S1mpbrain has joined #archiveteam-ot [04:23] *** SimpBrain has quit IRC (Read error: Operation timed out) [04:59] *** odemg has quit IRC (Ping timeout: 246 seconds) [05:03] *** BlueMax has joined #archiveteam-ot [05:12] *** odemg has joined #archiveteam-ot [05:42] JAA: Are the changes ivan made going to be merged into the main snscrape repo? [06:42] *** Ryz has joined #archiveteam-ot [07:55] *** odemg has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** bithippo has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** jspiros has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** dashcloud has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** svchfoo1 has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** sknebel_ has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** JAA has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** svchfoo3 has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** MrRadar2 has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** BnAboyZ has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** VoynichCr has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** Tenebrae has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** Jusque has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** ivan has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** mal has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** Odd0002 has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** sep332 has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** arkiver has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** Albardin has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** dxrt_ has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** wmvhater has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** kiska1 has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** godane has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** Mateon1 has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** tuluu has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** w0rmybak has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** kiskabak has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** robogoat has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** jut has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** Kaz has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** BlueMax has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** scifi_ has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** jodizzle has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** Sanqui has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** vectr0n has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** ggus has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** jeekl has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** Meroje has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** eientei95 has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** voltagex_ has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** Muad-Dib has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** apache2 has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** nightpool has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** eLbot has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** DrasticAc has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** chr1sm has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** bitspill has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** HCross has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** xarph_ has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** Vito` has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** Ctrl-S_ has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** Hecatz has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** S1mpbrain has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** anarcat has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** betamax has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** Polylith has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** dxrt has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** hook54321 has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** ColdIce has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** w0rmhole has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** kiska has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** Flashfire has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** bakJAA has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** mgrytbak_ has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** Aoede has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** VerifiedJ has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** mr_archiv has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** chirlu has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** zino has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** Igloo has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** MrRadar has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** swebb has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** Frogging has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** erin has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** m007a83 has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** Martle has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** Stiletto has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** t2t2 has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** adinbied has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** chfoo has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** JensRex has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** phuzion has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** Jon has quit IRC (irc.Prison.NET west.us.hub) [07:55] *** jrwr has quit IRC (irc.Prison.NET west.us.hub) [08:01] *** odemg has joined #archiveteam-ot [08:01] *** BlueMax has joined #archiveteam-ot [08:01] *** S1mpbrain has joined #archiveteam-ot [08:01] *** godane has joined #archiveteam-ot [08:01] *** MrRadar2 has joined #archiveteam-ot [08:01] *** BnAboyZ has joined #archiveteam-ot [08:01] *** scifi_ has joined #archiveteam-ot [08:01] *** VoynichCr has joined #archiveteam-ot [08:01] *** Tenebrae has joined #archiveteam-ot [08:01] *** Mateon1 has joined #archiveteam-ot [08:01] *** m007a83 has joined #archiveteam-ot [08:01] *** Martle has joined #archiveteam-ot [08:01] *** svchfoo3 has joined #archiveteam-ot [08:01] *** bithippo has joined #archiveteam-ot [08:01] *** jspiros has joined #archiveteam-ot [08:01] *** Jusque has joined #archiveteam-ot [08:01] *** dashcloud has joined #archiveteam-ot [08:01] *** ivan has joined #archiveteam-ot [08:01] *** VerifiedJ has joined #archiveteam-ot [08:01] *** mr_archiv has joined #archiveteam-ot [08:01] *** Stiletto has joined #archiveteam-ot [08:01] *** mal has joined #archiveteam-ot [08:01] *** Aoede has joined #archiveteam-ot [08:01] *** anarcat has joined #archiveteam-ot [08:01] *** Odd0002 has joined #archiveteam-ot [08:01] *** bakJAA has joined #archiveteam-ot [08:01] *** west.us.hub sets mode: +oooo svchfoo3 ivan Aoede bakJAA [08:01] *** betamax has joined #archiveteam-ot [08:01] *** t2t2 has joined #archiveteam-ot [08:01] *** jodizzle has joined #archiveteam-ot [08:01] *** sep332 has joined #archiveteam-ot [08:01] *** adinbied has joined #archiveteam-ot [08:01] *** chirlu has joined #archiveteam-ot [08:01] *** svchfoo1 has joined #archiveteam-ot [08:01] *** tuluu has joined #archiveteam-ot [08:01] *** JAA has joined #archiveteam-ot [08:01] *** sknebel_ has joined #archiveteam-ot [08:01] *** arkiver has joined #archiveteam-ot [08:01] *** Polylith has joined #archiveteam-ot [08:01] *** dxrt has joined #archiveteam-ot [08:01] *** west.us.hub sets mode: +oooo svchfoo1 JAA arkiver dxrt [08:01] *** w0rmybak has joined #archiveteam-ot [08:01] *** kiskabak has joined #archiveteam-ot [08:01] *** Albardin has joined #archiveteam-ot [08:01] *** dxrt_ has joined #archiveteam-ot [08:01] *** wmvhater has joined #archiveteam-ot [08:01] *** kiska1 has joined #archiveteam-ot [08:01] *** west.us.hub sets mode: +o dxrt_ [08:01] *** Sanqui has joined #archiveteam-ot [08:01] *** hook54321 has joined #archiveteam-ot [08:01] *** robogoat has joined #archiveteam-ot [08:01] *** erin has joined #archiveteam-ot [08:01] *** Frogging has joined #archiveteam-ot [08:01] *** mgrytbak_ has joined #archiveteam-ot [08:01] *** apache2 has joined #archiveteam-ot [08:01] *** nightpool has joined #archiveteam-ot [08:01] *** eLbot has joined #archiveteam-ot [08:01] *** DrasticAc has joined #archiveteam-ot [08:01] *** chr1sm has joined #archiveteam-ot [08:01] *** bitspill has joined #archiveteam-ot [08:01] *** HCross has joined #archiveteam-ot [08:01] *** xarph_ has joined #archiveteam-ot [08:01] *** Vito` has joined #archiveteam-ot [08:01] *** Ctrl-S_ has joined #archiveteam-ot [08:01] *** Hecatz has joined #archiveteam-ot [08:01] *** swebb has joined #archiveteam-ot [08:01] *** jrwr has joined #archiveteam-ot [08:01] *** MrRadar has joined #archiveteam-ot [08:01] *** Muad-Dib has joined #archiveteam-ot [08:01] *** Jon has joined #archiveteam-ot [08:01] *** voltagex_ has joined #archiveteam-ot [08:01] *** kiska has joined #archiveteam-ot [08:01] *** Flashfire has joined #archiveteam-ot [08:01] *** w0rmhole has joined #archiveteam-ot [08:01] *** ColdIce has joined #archiveteam-ot [08:01] *** eientei95 has joined #archiveteam-ot [08:01] *** Igloo has joined #archiveteam-ot [08:01] *** Meroje has joined #archiveteam-ot [08:01] *** jeekl has joined #archiveteam-ot [08:01] *** Kaz has joined #archiveteam-ot [08:01] *** phuzion has joined #archiveteam-ot [08:01] *** JensRex has joined #archiveteam-ot [08:01] *** ggus has joined #archiveteam-ot [08:01] *** jut has joined #archiveteam-ot [08:01] *** chfoo has joined #archiveteam-ot [08:01] *** zino has joined #archiveteam-ot [08:01] *** west.us.hub sets mode: +oooo Sanqui Frogging HCross zino [08:01] *** vectr0n has joined #archiveteam-ot [09:09] ivan: I'm going a grab of ~6,000 campaign sites [09:09] to save space I'm deleting the downloaded files and only keeping the warcs [09:10] but it should be possible to extract info like youtube / social media links from these gzipped warcs [09:10] I tried tried the following command: [09:11] cat *.warc.gz | zcat | sed 's/href=/\nhref=/g' | grep href=\" | sed 's/.*href="//g;s/".*//g' | egrep 'twitter.com|facebook.com|instagram.com|yt.be|youtube.com' > ../links.txt [09:11] but it seemed to get stuck after the first warc (each individual campaign site has its own warc right now) [09:13] if you can give me a command that works , I'm more than happy to run it [09:14] (currently got ~4,000 campaign sites) [09:17] betamax: why not `zcat *.warc.gz` instead of `cat *.warc.gz | zcat` [10:56] betamax: Don't forget youtu.be [11:26] *** BlueMax has quit IRC (Quit: Leaving) [12:37] *** Ryz has quit IRC (Quit: ChatZilla 0.9.92-rdmsoft [XULRunner 35.0.1/20150122214805]) [12:48] *** svchfoo3 has quit IRC (Read error: Operation timed out) [13:03] *** ggus has quit IRC (Quit: Changing server) [13:03] *** user has joined #archiveteam-ot [13:04] *** user is now known as ggus [13:15] *** Stilett0 has joined #archiveteam-ot [13:15] joedizzle: about to try that command, but before I do, apart from youtu.be, are there any other url patterns I should look out for? [13:19] *** Stiletto has quit IRC (Read error: Operation timed out) [13:25] *** svchfoo3 has joined #archiveteam-ot [13:25] *** svchfoo1 sets mode: +o svchfoo3 [14:10] *** wp494 has quit IRC (Ping timeout: 255 seconds) [14:10] *** wp494 has joined #archiveteam-ot [14:18] *** odemg has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** bithippo has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** jspiros has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** dashcloud has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** svchfoo1 has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** sknebel_ has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** JAA has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** svchfoo3 has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** MrRadar2 has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** BnAboyZ has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** VoynichCr has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** Tenebrae has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** Jusque has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** ivan has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** mal has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** Odd0002 has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** sep332 has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** arkiver has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** Albardin has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** dxrt_ has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** wmvhater has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** kiska1 has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** ggus has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** godane has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** Mateon1 has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** tuluu has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** w0rmybak has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** kiskabak has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** robogoat has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** jut has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** Kaz has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** wp494 has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** scifi_ has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** jodizzle has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** Sanqui has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** vectr0n has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** jeekl has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** Meroje has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** eientei95 has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** voltagex_ has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** Muad-Dib has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** apache2 has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** nightpool has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** eLbot has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** DrasticAc has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** chr1sm has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** bitspill has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** HCross has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** xarph_ has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** Vito` has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** Ctrl-S_ has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** Hecatz has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** S1mpbrain has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** anarcat has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** betamax has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** Polylith has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** dxrt has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** hook54321 has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** ColdIce has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** w0rmhole has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** kiska has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** Flashfire has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** bakJAA has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** mgrytbak_ has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** Aoede has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** VerifiedJ has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** mr_archiv has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** chirlu has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** zino has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** Igloo has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** MrRadar has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** swebb has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** Frogging has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** erin has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** Stilett0 has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** m007a83 has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** Martle has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** t2t2 has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** adinbied has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** chfoo has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** JensRex has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** phuzion has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** Jon has quit IRC (irc.Prison.NET west.us.hub) [14:18] *** jrwr has quit IRC (irc.Prison.NET west.us.hub) [14:24] *** wp494 has joined #archiveteam-ot [14:24] *** svchfoo3 has joined #archiveteam-ot [14:24] *** Stilett0 has joined #archiveteam-ot [14:24] *** ggus has joined #archiveteam-ot [14:24] *** odemg has joined #archiveteam-ot [14:24] *** S1mpbrain has joined #archiveteam-ot [14:24] *** godane has joined #archiveteam-ot [14:24] *** MrRadar2 has joined #archiveteam-ot [14:24] *** BnAboyZ has joined #archiveteam-ot [14:24] *** scifi_ has joined #archiveteam-ot [14:24] *** VoynichCr has joined #archiveteam-ot [14:24] *** Tenebrae has joined #archiveteam-ot [14:24] *** Mateon1 has joined #archiveteam-ot [14:24] *** m007a83 has joined #archiveteam-ot [14:24] *** Martle has joined #archiveteam-ot [14:24] *** bithippo has joined #archiveteam-ot [14:24] *** jspiros has joined #archiveteam-ot [14:24] *** Jusque has joined #archiveteam-ot [14:24] *** dashcloud has joined #archiveteam-ot [14:24] *** ivan has joined #archiveteam-ot [14:24] *** VerifiedJ has joined #archiveteam-ot [14:24] *** mr_archiv has joined #archiveteam-ot [14:24] *** mal has joined #archiveteam-ot [14:24] *** Aoede has joined #archiveteam-ot [14:24] *** anarcat has joined #archiveteam-ot [14:24] *** Odd0002 has joined #archiveteam-ot [14:24] *** bakJAA has joined #archiveteam-ot [14:24] *** west.us.hub sets mode: +oooo svchfoo3 ivan Aoede bakJAA [14:24] *** betamax has joined #archiveteam-ot [14:24] *** t2t2 has joined #archiveteam-ot [14:24] *** jodizzle has joined #archiveteam-ot [14:24] *** sep332 has joined #archiveteam-ot [14:24] *** adinbied has joined #archiveteam-ot [14:24] *** chirlu has joined #archiveteam-ot [14:24] *** svchfoo1 has joined #archiveteam-ot [14:24] *** tuluu has joined #archiveteam-ot [14:24] *** JAA has joined #archiveteam-ot [14:24] *** sknebel_ has joined #archiveteam-ot [14:24] *** arkiver has joined #archiveteam-ot [14:24] *** Polylith has joined #archiveteam-ot [14:24] *** dxrt has joined #archiveteam-ot [14:24] *** west.us.hub sets mode: +oooo svchfoo1 JAA arkiver dxrt [14:24] *** w0rmybak has joined #archiveteam-ot [14:24] *** kiskabak has joined #archiveteam-ot [14:24] *** Albardin has joined #archiveteam-ot [14:24] *** dxrt_ has joined #archiveteam-ot [14:24] *** wmvhater has joined #archiveteam-ot [14:24] *** west.us.hub sets mode: +o dxrt_ [14:24] *** kiska1 has joined #archiveteam-ot [14:24] *** Sanqui has joined #archiveteam-ot [14:24] *** hook54321 has joined #archiveteam-ot [14:24] *** robogoat has joined #archiveteam-ot [14:24] *** erin has joined #archiveteam-ot [14:24] *** Frogging has joined #archiveteam-ot [14:24] *** mgrytbak_ has joined #archiveteam-ot [14:24] *** apache2 has joined #archiveteam-ot [14:24] *** nightpool has joined #archiveteam-ot [14:24] *** eLbot has joined #archiveteam-ot [14:24] *** DrasticAc has joined #archiveteam-ot [14:24] *** chr1sm has joined #archiveteam-ot [14:24] *** bitspill has joined #archiveteam-ot [14:24] *** HCross has joined #archiveteam-ot [14:24] *** xarph_ has joined #archiveteam-ot [14:24] *** Vito` has joined #archiveteam-ot [14:24] *** Ctrl-S_ has joined #archiveteam-ot [14:24] *** Hecatz has joined #archiveteam-ot [14:24] *** swebb has joined #archiveteam-ot [14:24] *** jrwr has joined #archiveteam-ot [14:24] *** MrRadar has joined #archiveteam-ot [14:24] *** Muad-Dib has joined #archiveteam-ot [14:24] *** Jon has joined #archiveteam-ot [14:24] *** voltagex_ has joined #archiveteam-ot [14:24] *** kiska has joined #archiveteam-ot [14:24] *** Flashfire has joined #archiveteam-ot [14:24] *** w0rmhole has joined #archiveteam-ot [14:24] *** ColdIce has joined #archiveteam-ot [14:24] *** eientei95 has joined #archiveteam-ot [14:24] *** Igloo has joined #archiveteam-ot [14:24] *** Meroje has joined #archiveteam-ot [14:24] *** jeekl has joined #archiveteam-ot [14:24] *** Kaz has joined #archiveteam-ot [14:24] *** phuzion has joined #archiveteam-ot [14:24] *** JensRex has joined #archiveteam-ot [14:24] *** jut has joined #archiveteam-ot [14:24] *** chfoo has joined #archiveteam-ot [14:24] *** zino has joined #archiveteam-ot [14:24] *** west.us.hub sets mode: +oooo Sanqui Frogging HCross zino [14:24] *** vectr0n has joined #archiveteam-ot [14:32] betamax: If you haven't run it yet, only other things I think of just from reading it are "fb.com", "fb.me" and adding an `-i` (ignore-case) to your `egrep` for safety. [14:39] hook54321: If ivan sends a PR, sure. [16:30] *** schbirid has joined #archiveteam-ot [16:46] *** bithippo has quit IRC (My MacBook Air has gone to sleep. ZZZzzz…) [17:00] jodizzle: ran out of RAM so I never got the chance to run it yet [17:01] but I've since found one website that uses Vimeo so I'm adding 'vimeo' as well [17:01] (for reference, it's: https://troutmanpa.com/) [17:01] with https://vimeo.com/293586571 [17:18] *** Martle has quit IRC (Read error: Operation timed out) [17:34] *** Martle has joined #archiveteam-ot [17:37] *** Stiletto has joined #archiveteam-ot [17:40] *** Stilett0 has quit IRC (Read error: Operation timed out) [17:42] jodizzle: I've decided to extract ALL urls to a file, as then its much easier to get another type of URL if we think of it later [18:01] *** Martle_ has joined #archiveteam-ot [18:01] *** Martle has quit IRC (Read error: Connection reset by peer) [18:02] *** Martle_ has quit IRC (Client Quit) [18:02] *** Martle has joined #archiveteam-ot [18:39] *** bithippo has joined #archiveteam-ot [19:03] *** S1mpbrain has quit IRC (Remote host closed the connection) [19:06] *** SimpBrain has joined #archiveteam-ot [19:22] *** m007a83_ has joined #archiveteam-ot [19:25] *** m007a83 has quit IRC (Read error: Operation timed out) [19:25] *** adinbied has quit IRC (Read error: Operation timed out) [19:29] *** adinbied has joined #archiveteam-ot [19:29] *** JensRex has quit IRC (Ping timeout: 633 seconds) [19:34] *** SimpBrain has quit IRC (Ping timeout: 633 seconds) [19:45] *** SimpBrain has joined #archiveteam-ot [19:47] *** bithippo has quit IRC (Textual IRC Client: www.textualapp.com) [19:52] *** odemg has quit IRC (west.us.hub hub.efnet.us) [19:52] *** jspiros has quit IRC (west.us.hub hub.efnet.us) [19:52] *** dashcloud has quit IRC (west.us.hub hub.efnet.us) [19:52] *** svchfoo1 has quit IRC (west.us.hub hub.efnet.us) [19:52] *** sknebel_ has quit IRC (west.us.hub hub.efnet.us) [19:52] *** JAA has quit IRC (west.us.hub hub.efnet.us) [19:52] *** svchfoo3 has quit IRC (west.us.hub hub.efnet.us) [19:52] *** adinbied has quit IRC (west.us.hub hub.efnet.us) [19:52] *** MrRadar2 has quit IRC (west.us.hub hub.efnet.us) [19:52] *** BnAboyZ has quit IRC (west.us.hub hub.efnet.us) [19:52] *** VoynichCr has quit IRC (west.us.hub hub.efnet.us) [19:52] *** Tenebrae has quit IRC (west.us.hub hub.efnet.us) [19:52] *** m007a83_ has quit IRC (west.us.hub hub.efnet.us) [19:52] *** Jusque has quit IRC (west.us.hub hub.efnet.us) [19:52] *** ivan has quit IRC (west.us.hub hub.efnet.us) [19:52] *** mal has quit IRC (west.us.hub hub.efnet.us) [19:52] *** Odd0002 has quit IRC (west.us.hub hub.efnet.us) [19:52] *** sep332 has quit IRC (west.us.hub hub.efnet.us) [19:52] *** arkiver has quit IRC (west.us.hub hub.efnet.us) [19:52] *** Albardin has quit IRC (west.us.hub hub.efnet.us) [19:52] *** dxrt_ has quit IRC (west.us.hub hub.efnet.us) [19:52] *** wmvhater has quit IRC (west.us.hub hub.efnet.us) [19:52] *** kiska1 has quit IRC (west.us.hub hub.efnet.us) [19:52] *** ggus has quit IRC (west.us.hub hub.efnet.us) [19:52] *** godane has quit IRC (west.us.hub hub.efnet.us) [19:52] *** Mateon1 has quit IRC (west.us.hub hub.efnet.us) [19:52] *** tuluu has quit IRC (west.us.hub hub.efnet.us) [19:52] *** w0rmybak has quit IRC (west.us.hub hub.efnet.us) [19:52] *** kiskabak has quit IRC (west.us.hub hub.efnet.us) [19:52] *** robogoat has quit IRC (west.us.hub hub.efnet.us) [19:52] *** jut has quit IRC (west.us.hub hub.efnet.us) [19:52] *** Kaz has quit IRC (west.us.hub hub.efnet.us) [19:52] *** wp494 has quit IRC (west.us.hub hub.efnet.us) [19:52] *** scifi_ has quit IRC (west.us.hub hub.efnet.us) [19:52] *** jodizzle has quit IRC (west.us.hub hub.efnet.us) [19:52] *** Sanqui has quit IRC (west.us.hub hub.efnet.us) [19:52] *** vectr0n has quit IRC (west.us.hub hub.efnet.us) [19:52] *** jeekl has quit IRC (west.us.hub hub.efnet.us) [19:52] *** Meroje has quit IRC (west.us.hub hub.efnet.us) [19:52] *** eientei95 has quit IRC (west.us.hub hub.efnet.us) [19:52] *** voltagex_ has quit IRC (west.us.hub hub.efnet.us) [19:52] *** Muad-Dib has quit IRC (west.us.hub hub.efnet.us) [19:52] *** apache2 has quit IRC (west.us.hub hub.efnet.us) [19:52] *** nightpool has quit IRC (west.us.hub hub.efnet.us) [19:52] *** eLbot has quit IRC (west.us.hub hub.efnet.us) [19:52] *** DrasticAc has quit IRC (west.us.hub hub.efnet.us) [19:52] *** chr1sm has quit IRC (west.us.hub hub.efnet.us) [19:52] *** bitspill has quit IRC (west.us.hub hub.efnet.us) [19:52] *** HCross has quit IRC (west.us.hub hub.efnet.us) [19:52] *** xarph_ has quit IRC (west.us.hub hub.efnet.us) [19:52] *** Vito` has quit IRC (west.us.hub hub.efnet.us) [19:52] *** Ctrl-S_ has quit IRC (west.us.hub hub.efnet.us) [19:52] *** Hecatz has quit IRC (west.us.hub hub.efnet.us) [19:52] *** SimpBrain has quit IRC (west.us.hub hub.efnet.us) [19:52] *** Martle has quit IRC (west.us.hub hub.efnet.us) [19:52] *** anarcat has quit IRC (west.us.hub hub.efnet.us) [19:52] *** betamax has quit IRC (west.us.hub hub.efnet.us) [19:52] *** Polylith has quit IRC (west.us.hub hub.efnet.us) [19:52] *** dxrt has quit IRC (west.us.hub hub.efnet.us) [19:52] *** hook54321 has quit IRC (west.us.hub hub.efnet.us) [19:52] *** ColdIce has quit IRC (west.us.hub hub.efnet.us) [19:52] *** w0rmhole has quit IRC (west.us.hub hub.efnet.us) [19:52] *** kiska has quit IRC (west.us.hub hub.efnet.us) [19:52] *** Flashfire has quit IRC (west.us.hub hub.efnet.us) [19:52] *** bakJAA has quit IRC (west.us.hub hub.efnet.us) [19:52] *** mgrytbak_ has quit IRC (west.us.hub hub.efnet.us) [19:52] *** Aoede has quit IRC (west.us.hub hub.efnet.us) [19:52] *** schbirid has quit IRC (west.us.hub hub.efnet.us) [19:52] *** VerifiedJ has quit IRC (west.us.hub hub.efnet.us) [19:52] *** mr_archiv has quit IRC (west.us.hub hub.efnet.us) [19:52] *** chirlu has quit IRC (west.us.hub hub.efnet.us) [19:52] *** zino has quit IRC (west.us.hub hub.efnet.us) [19:52] *** Igloo has quit IRC (west.us.hub hub.efnet.us) [19:52] *** MrRadar has quit IRC (west.us.hub hub.efnet.us) [19:52] *** swebb has quit IRC (west.us.hub hub.efnet.us) [19:52] *** Frogging has quit IRC (west.us.hub hub.efnet.us) [19:52] *** erin has quit IRC (west.us.hub hub.efnet.us) [19:52] *** Stiletto has quit IRC (west.us.hub hub.efnet.us) [19:52] *** t2t2 has quit IRC (west.us.hub hub.efnet.us) [19:52] *** chfoo has quit IRC (west.us.hub hub.efnet.us) [19:52] *** phuzion has quit IRC (west.us.hub hub.efnet.us) [19:52] *** Jon has quit IRC (west.us.hub hub.efnet.us) [19:52] *** jrwr has quit IRC (west.us.hub hub.efnet.us) [19:54] *** SimpBrain has joined #archiveteam-ot [19:54] *** adinbied has joined #archiveteam-ot [19:54] *** m007a83_ has joined #archiveteam-ot [19:54] *** Martle has joined #archiveteam-ot [19:54] *** Stiletto has joined #archiveteam-ot [19:54] *** schbirid has joined #archiveteam-ot [19:54] *** wp494 has joined #archiveteam-ot [19:54] *** svchfoo3 has joined #archiveteam-ot [19:54] *** ggus has joined #archiveteam-ot [19:54] *** odemg has joined #archiveteam-ot [19:54] *** godane has joined #archiveteam-ot [19:54] *** MrRadar2 has joined #archiveteam-ot [19:54] *** BnAboyZ has joined #archiveteam-ot [19:54] *** scifi_ has joined #archiveteam-ot [19:54] *** VoynichCr has joined #archiveteam-ot [19:54] *** Tenebrae has joined #archiveteam-ot [19:54] *** Mateon1 has joined #archiveteam-ot [19:54] *** jspiros has joined #archiveteam-ot [19:54] *** Jusque has joined #archiveteam-ot [19:54] *** dashcloud has joined #archiveteam-ot [19:54] *** ivan has joined #archiveteam-ot [19:54] *** VerifiedJ has joined #archiveteam-ot [19:54] *** mr_archiv has joined #archiveteam-ot [19:54] *** mal has joined #archiveteam-ot [19:54] *** Aoede has joined #archiveteam-ot [19:54] *** anarcat has joined #archiveteam-ot [19:54] *** Odd0002 has joined #archiveteam-ot [19:54] *** bakJAA has joined #archiveteam-ot [19:54] *** hub.efnet.us sets mode: +oooo svchfoo3 ivan Aoede bakJAA [19:54] *** betamax has joined #archiveteam-ot [19:54] *** t2t2 has joined #archiveteam-ot [19:54] *** jodizzle has joined #archiveteam-ot [19:54] *** sep332 has joined #archiveteam-ot [19:54] *** chirlu has joined #archiveteam-ot [19:54] *** svchfoo1 has joined #archiveteam-ot [19:54] *** tuluu has joined #archiveteam-ot [19:54] *** JAA has joined #archiveteam-ot [19:54] *** sknebel_ has joined #archiveteam-ot [19:54] *** arkiver has joined #archiveteam-ot [19:54] *** Polylith has joined #archiveteam-ot [19:54] *** dxrt has joined #archiveteam-ot [19:54] *** hub.efnet.us sets mode: +oooo svchfoo1 JAA arkiver dxrt [19:54] *** w0rmybak has joined #archiveteam-ot [19:54] *** kiskabak has joined #archiveteam-ot [19:54] *** Albardin has joined #archiveteam-ot [19:54] *** dxrt_ has joined #archiveteam-ot [19:54] *** wmvhater has joined #archiveteam-ot [19:54] *** hub.efnet.us sets mode: +o dxrt_ [19:54] *** kiska1 has joined #archiveteam-ot [19:54] *** Sanqui has joined #archiveteam-ot [19:54] *** hook54321 has joined #archiveteam-ot [19:54] *** robogoat has joined #archiveteam-ot [19:54] *** vectr0n has joined #archiveteam-ot [19:54] *** zino has joined #archiveteam-ot [19:54] *** chfoo has joined #archiveteam-ot [19:54] *** jut has joined #archiveteam-ot [19:54] *** phuzion has joined #archiveteam-ot [19:54] *** Kaz has joined #archiveteam-ot [19:54] *** jeekl has joined #archiveteam-ot [19:54] *** Meroje has joined #archiveteam-ot [19:54] *** Igloo has joined #archiveteam-ot [19:54] *** eientei95 has joined #archiveteam-ot [19:54] *** ColdIce has joined #archiveteam-ot [19:54] *** w0rmhole has joined #archiveteam-ot [19:54] *** Flashfire has joined #archiveteam-ot [19:54] *** kiska has joined #archiveteam-ot [19:54] *** voltagex_ has joined #archiveteam-ot [19:54] *** Jon has joined #archiveteam-ot [19:54] *** Muad-Dib has joined #archiveteam-ot [19:54] *** MrRadar has joined #archiveteam-ot [19:54] *** jrwr has joined #archiveteam-ot [19:54] *** swebb has joined #archiveteam-ot [19:54] *** Hecatz has joined #archiveteam-ot [19:54] *** Ctrl-S_ has joined #archiveteam-ot [19:54] *** Vito` has joined #archiveteam-ot [19:54] *** xarph_ has joined #archiveteam-ot [19:54] *** HCross has joined #archiveteam-ot [19:54] *** bitspill has joined #archiveteam-ot [19:54] *** chr1sm has joined #archiveteam-ot [19:54] *** DrasticAc has joined #archiveteam-ot [19:54] *** eLbot has joined #archiveteam-ot [19:54] *** nightpool has joined #archiveteam-ot [19:54] *** apache2 has joined #archiveteam-ot [19:54] *** mgrytbak_ has joined #archiveteam-ot [19:54] *** Frogging has joined #archiveteam-ot [19:54] *** hub.efnet.us sets mode: +oooo Sanqui zino HCross Frogging [19:54] *** erin has joined #archiveteam-ot [19:57] *** Ryz has joined #archiveteam-ot [20:05] *** Jens has joined #archiveteam-ot [20:21] *** SimpBrain has quit IRC (Read error: Connection reset by peer) [20:23] *** SimpBrain has joined #archiveteam-ot [21:03] *** schbirid has quit IRC (Remote host closed the connection) [21:27] *** bithippo has joined #archiveteam-ot [21:28] *** VerifiedJ has quit IRC (Read error: Operation timed out) [21:48] *** VerifiedJ has joined #archiveteam-ot [22:01] *** twoTBHetz has joined #archiveteam-ot [22:08] *** bithippo has quit IRC (My MacBook Air has gone to sleep. ZZZzzz…) [22:16] *** BlueMax has joined #archiveteam-ot [22:38] *** Martle_ has joined #archiveteam-ot [22:41] *** Martle has quit IRC (Ping timeout: 252 seconds) [22:46] *** Martle__ has joined #archiveteam-ot [22:49] *** Martle_ has quit IRC (Ping timeout: 252 seconds) [22:55] *** odemg has quit IRC (irc.Prison.NET west.us.hub) [22:56] *** jspiros has quit IRC (irc.Prison.NET west.us.hub) [22:56] *** dashcloud has quit IRC (irc.Prison.NET west.us.hub) [22:56] *** svchfoo1 has quit IRC (irc.Prison.NET west.us.hub) [22:56] *** sknebel_ has quit IRC (irc.Prison.NET west.us.hub) [22:56] *** JAA has quit IRC (irc.Prison.NET west.us.hub) [22:58] *** logchfoo1 starts logging #archiveteam-ot at Wed Nov 07 22:58:41 2018 [22:58] *** logchfoo1 has joined #archiveteam-ot [22:58] *** moufu_ has joined #archiveteam-ot [23:31] *** twoTBHetz has quit IRC (Ping timeout: 260 seconds) [23:32] *** bithippo has joined #archiveteam-ot [23:33] *** Ryz has joined #archiveteam-ot [23:43] *** bithippo has quit IRC (Textual IRC Client: www.textualapp.com) [23:47] *** m007a83_ has quit IRC (Read error: Connection reset by peer)