[00:27] wow, i actually managed to download 11 items with gamefront warrior, lol [00:34] *** PurpleSym has quit IRC (Ping timeout: 506 seconds) [00:34] *** coretx_ has joined #archiveteam-bs [00:35] *** altlabel has quit IRC (Ping timeout: 506 seconds) [00:35] *** PotcFdk has quit IRC (Ping timeout: 506 seconds) [00:36] *** Darkstar has quit IRC (Ping timeout: 506 seconds) [00:36] *** coretx has quit IRC (Ping timeout: 506 seconds) [00:37] *** altlabel has joined #archiveteam-bs [00:39] *** Darkstar has joined #archiveteam-bs [00:41] *** PurpleSym has joined #archiveteam-bs [00:41] *** PotcFdk has joined #archiveteam-bs [01:13] *** VADemon has quit IRC (Quit: left4dead) [01:16] *** Stiletto has joined #archiveteam-bs [01:48] *** acridAxid has quit IRC (Quit: marauder) [01:49] *** acridAxid has joined #archiveteam-bs [02:00] *** Yoshimura has quit IRC (http://www.kiwiirc.com/ - A hand crafted IRC client) [02:00] *** Yoshimura has joined #archiveteam-bs [02:02] *** JesseW has quit IRC (Quit: Leaving.) [02:02] *** JesseW has joined #archiveteam-bs [02:29] *** JesseW has quit IRC (Ping timeout: 370 seconds) [03:00] *** hook54321 has joined #archiveteam-bs [03:00] Is their a way to save all data (posts, comments, likes, etc) from a Facebook group? [03:24] *** dashcloud has quit IRC (Read error: Operation timed out) [03:27] *** dashcloud has joined #archiveteam-bs [03:28] *** robink has quit IRC (Ping timeout: 633 seconds) [03:31] *** bwn_ has joined #archiveteam-bs [03:35] *** robink has joined #archiveteam-bs [03:46] *** bwn has quit IRC (Read error: Operation timed out) [03:57] *** Start has quit IRC (Quit: Disconnected.) [03:58] *** Start has joined #archiveteam-bs [04:24] *** PotcFdk has quit IRC (hub.dk irc.homelien.no) [04:24] *** PurpleSym has quit IRC (hub.dk irc.homelien.no) [04:24] *** i0npulse has quit IRC (hub.dk irc.homelien.no) [04:47] *** Yoshimura has quit IRC (http://www.kiwiirc.com/ - A hand crafted IRC client) [04:53] *** Honno has joined #archiveteam-bs [04:53] *** PotcFdk has joined #archiveteam-bs [04:53] *** PurpleSym has joined #archiveteam-bs [04:53] *** i0npulse has joined #archiveteam-bs [04:54] *** Sk1d has quit IRC (Ping timeout: 194 seconds) [04:57] *** bwn has joined #archiveteam-bs [04:58] *** bwn_ has quit IRC (Quit: Quit) [05:00] *** Sk1d has joined #archiveteam-bs [05:08] *** JesseW has joined #archiveteam-bs [05:36] *** DopefishJ is now known as DFJustin [06:18] *** Medowar has joined #archiveteam-bs [06:20] *** brayden_ has quit IRC (Read error: Connection reset by peer) [06:20] *** hook54321 has quit IRC (Quit: Connection closed for inactivity) [06:20] *** brayden_ has joined #archiveteam-bs [06:20] *** swebb sets mode: +o brayden_ [06:31] *** hook54321 has joined #archiveteam-bs [06:41] *** metalcamp has joined #archiveteam-bs [06:52] *** Honno has quit IRC (Read error: Operation timed out) [07:16] *** bwn has quit IRC (Read error: Operation timed out) [07:22] *** RichardG_ has joined #archiveteam-bs [07:22] *** Kenshin has quit IRC (Ping timeout: 244 seconds) [07:22] *** Kenshin has joined #archiveteam-bs [07:22] *** Atros has joined #archiveteam-bs [07:23] *** RichardG has quit IRC (Ping timeout: 244 seconds) [07:23] *** Fletcher has quit IRC (Ping timeout: 244 seconds) [07:23] *** BnA-Rob1n has quit IRC (Ping timeout: 244 seconds) [07:23] *** atrocity has quit IRC (Ping timeout: 244 seconds) [07:23] *** Fusl has quit IRC (Ping timeout: 244 seconds) [07:24] *** hook54321 has quit IRC (Ping timeout: 244 seconds) [07:24] *** zerkalo has quit IRC (Ping timeout: 244 seconds) [07:24] *** metalcamp has quit IRC (Ping timeout: 244 seconds) [07:26] *** zerkalo has joined #archiveteam-bs [07:26] *** schbirid has joined #archiveteam-bs [07:27] *** hook54321 has joined #archiveteam-bs [07:27] *** metalcamp has joined #archiveteam-bs [07:29] *** BnA-Rob1n has joined #archiveteam-bs [07:38] *** bwn has joined #archiveteam-bs [07:42] *** bwn_ has joined #archiveteam-bs [07:43] *** Fusl has joined #archiveteam-bs [07:47] *** Fletcher has joined #archiveteam-bs [07:47] *** Fletcher_ sets mode: +o Fletcher [07:55] *** bwn has quit IRC (Read error: Operation timed out) [08:15] *** JesseW has quit IRC (Ping timeout: 370 seconds) [08:40] *** hook54321 has quit IRC (Quit: Connection closed for inactivity) [09:01] *** toad1 has joined #archiveteam-bs [09:07] *** toad2 has quit IRC (Read error: Operation timed out) [09:24] *** toad2 has joined #archiveteam-bs [09:26] *** toad1 has quit IRC (Read error: Operation timed out) [09:42] *** bwn__ has joined #archiveteam-bs [09:44] *** toad2 has quit IRC (Read error: Operation timed out) [09:51] *** Yoshimura has joined #archiveteam-bs [09:54] *** toad1 has joined #archiveteam-bs [09:56] *** bwn_ has quit IRC (Read error: Operation timed out) [11:08] *** Medowar has quit IRC (Quit: Connection closed for inactivity) [11:52] *** fdo54ss has quit IRC (Remote host closed the connection) [11:52] *** fdo54ss has joined #archiveteam-bs [12:25] blah, netsplit [12:41] phuzion: Hey, what is your preferred method to update passenger? I mean Ruby, node.js version, etc. .. you are the author of the docker package, right? [12:41] I am not. [12:43] Oh, sorry then. [12:49] *** Medowar has joined #archiveteam-bs [12:58] *** BlueMaxim has quit IRC (Quit: Leaving) [13:08] *** slyphic has joined #archiveteam-bs [13:40] *** hook54321 has joined #archiveteam-bs [13:55] *** vitzli has joined #archiveteam-bs [14:37] *** RichardG_ has quit IRC (Ping timeout: 260 seconds) [14:49] *** RichardG has joined #archiveteam-bs [15:03] *** Honno has joined #archiveteam-bs [15:10] *** Start has quit IRC (Quit: Disconnected.) [15:17] *** Atros is now known as atrocity [15:17] blah x2 [15:38] *** JesseW has joined #archiveteam-bs [15:50] *** hook54321 has quit IRC (Quit: Connection closed for inactivity) [15:56] should i just out of gamefront now that there's like nothing left? [15:57] atrocity: eh, might as well leave it on until arkiver confirms there won't be any new items coming down the pipe [15:58] arkiver? anything planned? if not, i'm going to swap off to help a cause again, haha [16:00] *** Jonimus has quit IRC (Ping timeout: 633 seconds) [16:01] *** Jonimus has joined #archiveteam-bs [16:01] *** swebb sets mode: +o Jonimus [16:02] *** chazchaz has quit IRC (Read error: Operation timed out) [16:02] ALL HAIL THE CAUSE [16:03] *** chazchaz has joined #archiveteam-bs [16:04] *** JesseW has quit IRC (Ping timeout: 370 seconds) [16:20] *** Jonimus has quit IRC (Ping timeout: 633 seconds) [16:23] *** Jonimus has joined #archiveteam-bs [16:23] *** swebb sets mode: +o Jonimus [16:26] well, better than the VM sitting around doing nothing, lol [16:27] ALL HAIL THE VM SITTING AROUND DOING NOTHING [16:28] woo [16:42] I kinda want to see Jon St. John vs. Alex St. John [16:54] *** vitzli has quit IRC (Leaving) [17:17] *** Jonimus has quit IRC (Read error: Operation timed out) [17:30] *** Honno has quit IRC (Quit: Leaving) [17:31] *** Jonimus has joined #archiveteam-bs [17:31] *** swebb sets mode: +o Jonimus [17:31] *** JW_work1 has joined #archiveteam-bs [17:31] *** JW_work has quit IRC (Read error: Operation timed out) [17:32] *** Jonimus has quit IRC (Read error: Operation timed out) [17:33] *** JW_work1 has quit IRC (Client Quit) [17:34] *** JW_work has joined #archiveteam-bs [17:40] atrocity: What OS you got? [17:42] To anyone knowing about IPFS: I wonder in which cases a node would receive Add for the same hash more then twice. And also why I did not get today any Get just once, everything several times. [17:42] Which now seems to be odd as yesterday I seen a flood of stuff requested only once. [17:45] Either something is wrong or my node gets no requests kinda. Except few objects very often [17:46] *** JW_work has quit IRC (Quit: Leaving.) [17:47] Yoshimura: windows [17:47] *** Jonimus has joined #archiveteam-bs [17:47] *** swebb sets mode: +o Jonimus [17:47] atrocity: Oh. Well, then depending on how fast net you have or if you want to do more work, I would have a trick. [17:48] i have 50/50 that's settled to around 60/79 [17:48] And you are running Warrior, not a pipe, right? [17:48] what's a pipe? [17:49] Yes, you are xD [17:49] i'm running the warrior vm, lol [17:49] If so, making Linux host in VM with bare minimum, ssh, you can use docker containers. [17:49] yeah, i was considering that [17:49] That way you can run multiple warriors easily, all sharing mem and space as needed. [17:49] Running manually is a hassle you do not want. [17:50] Running multiple warriors in docker is easy though. [17:50] *** Jonimus has quit IRC (WeeChat 1.4) [17:50] That way you can run one for each project xD [17:51] yeah, haha [17:51] *** JW_work has joined #archiveteam-bs [17:52] I was working (took a day break) on new Warrior VM, tried to do something that yipdw came with, Packer, the only good thing is it replays keys over VNC. But another method of building VM might be better. [17:52] Had a lot of trouble and when I got it to run, somehow the automated partitioning does not work. Not sure if I have the definition wrong or if its something else, but essentially I copy pasetd. [17:53] what did you mean about me running a pipe? [17:53] *** VADemon has joined #archiveteam-bs [17:54] Would like to fix some stuff about the UI of the Warrior afterwards, maybe add detection of recursive loops and avoid them, report them, etc. Maybe in future a "hidden" way to run all projects at once. [17:54] atrocity: That is for the high bandwidth or high cpu stuff, run usually on a dedicated machines on backbone, etc. [17:54] ahh [17:54] but yeah, a hidden way to run all projects would rock, haha [17:55] or even just an easy way to join multiple projects at once. have it based on the # of concurrent connections [17:57] Yes, it would be wiser to do multiple projects with low number of concurrency, so tickboxes to select projects or boolean buttons of same functionality, so one has a choice. [17:57] what [17:58] Yoshimura: heh dude Packer was just a suggestion, I have no financial involvement with HashiCorp or anything [17:58] Yeah, I know and it sucked. [17:58] ok [17:58] good to know [17:58] It would work great if : a) normal disk, b) end user. [17:59] I just find it weird that every time you mention it you mention me [17:59] like you're insinuiating somethin [17:59] With additional commands during install to ssh inside, that would make packer a good choice. [17:59] brb [17:59] *** atrocity has quit IRC () [17:59] Do not take it badly, but you kinda dismissed what I was working on for Packer, so that was it. [18:00] apologie for making you feel that way [18:00] What I did is remove the drive and do it manually again. So that works, also dynamic dirs for the vnc typing or http server is problem. [18:01] I fixed everything, now stuck at partitioning, but I just copy pasted it from the one on github plus added second drive, so unless the second drive definition is messing it up, I do not know. [18:03] If it works, it would be ok to use it. Other way would be to just use bootstrap, make a directory, do everything, make a fake disk, format, mount, copy over, install bootloader, unmount, convert to vmdk. [18:03] Or if one could write vmdk directly, then that would be better, not having to create 8GB loop. Though I guess one could create a sparse file if the FS supports it. [18:05] Also one thing I did notice in instructions is booting into the machine in rescue mode and removing stuff. Unless I do not see something, one can do that during the post-install automagically. [18:06] That way, using custom scripts or packer (to monkey type the boot) one could fully automate. Another idea I had is to just use docker inside the VM, and by that being able to just ship one thing as warrior: docker container. Plus the machine as a docker host. [18:07] Not sure if brilliant idea or not. Other option is a compressed overlay that autoupdates. (squashfs + aufs/overlayfs). That would contain the warrior itself. (This one lacks though at this point) [18:08] To sum my point of work on this one thing: Stuck at partitioning, creating a host, for the docker host, not much different from warrior itself. [18:25] *** bwn has joined #archiveteam-bs [18:33] https://www.youtube.com/channel/UCZM0q3tj_RstUAUM4ox3R3g/live [18:38] *** bwn__ has quit IRC (Read error: Operation timed out) [18:40] *** atrocity has joined #archiveteam-bs [18:43] Maybe figured out why I do not get a lot of hits on IPFS for get anymore. [18:44] For they were all the geoip stuff, which I did pin (recursively) of course, so I got them all locally. [18:44] But I think those still should show? [19:00] *** SimpBrain has quit IRC (Ping timeout: 633 seconds) [19:02] *** bwn__ has joined #archiveteam-bs [19:02] *** bwn has quit IRC (Quit: Quit) [19:55] *** mismatch_ has joined #archiveteam-bs [19:58] *** mismatch has quit IRC (Ping timeout: 499 seconds) [20:19] I have documents (HTML responses). Want to parse them for links. Which methods/tools are advised? [20:21] all urls? something like egrep -o 'https?://[-_A-Za-z0-9$.+!*'"'"'():/]*' [20:23] s#.*#:/#g [20:24] ... ? [20:24] :D [20:24] but i used -o [20:25] oh i forgot % and a few others [20:25] ugh [20:25] urls are complicated [20:25] you can have XN encoded, %, special chars and stuff. [20:26] XN encoded? [20:26] diacritics and stuff. [20:26] anyway, you get the picture, use a regex like that one [20:26] if that fits your needs, use it [20:26] simplest tools that do the job, etc [20:26] Yeah, thanks, will have to figure out the syntax :( [20:27] Well, the project does use something usually for parsing, except wget I thought, so I asked. [20:27] which project? [20:29] I should have said AT. [20:29] *** Honno has joined #archiveteam-bs [20:38] *** schbirid has quit IRC (Quit: Leaving) [21:10] *** hook54321 has joined #archiveteam-bs [21:37] i'm up to 2008-06-30 with funny or die [21:55] *** tomwsmf-a has joined #archiveteam-bs [22:06] *** VADemon has quit IRC (Quit: left4dead) [22:07] *** VADemon has joined #archiveteam-bs [22:07] i really hate the fact that windows doesn't allow passing of an equal sign as a parameter in a batch file [22:07] trying to setup youtube-dl into a script and it's failing with %1 :/ [22:08] poor jason [22:08] *** mistym has quit IRC (Read error: Connection reset by peer) [22:08] also, sad thing about that IA van going missing, but haha what a van to steal [22:10] atrocity: try ^= [22:10] ^ should work as an escape character [22:14] no go [22:14] even just trying set download=%1 then echo %download% just shows the URL up to ?v then ends [22:15] a "feature" that's existed since the DOS days [22:15] up to ?v : & is also a batch character [22:15] and has to be escaped [22:17] yeah, but the url i'm trying doesn't have a & in it [22:17] *** mistym has joined #archiveteam-bs [22:17] i'll read more into the youtube-dl docs. maybe i can set a system-wide config that will keep all the entries i want active [22:18] *** metalcamp has quit IRC (Ping timeout: 244 seconds) [22:19] looks like %APPDATA%/youtube-dl/config.txt [22:19] *** Honno has quit IRC (Quit: Leaving) [22:24] atrocity, have you considered using the batch download function to bypass passing urls on the command line? [22:25] like dump all the urls i want into a text file and pass that instead? [22:25] i have, but i only download normally one video at a time [22:28] *** Honno has joined #archiveteam-bs [22:28] If you want to use it as a browser extension, I've done some nifty stuff with ydl and Open With [22:29] like Add Video, Remove last video etc [22:29] Download Added videos [22:29] well, all this time i've been using jdownloader2 which adds to it's queue to download on add, and it auto-adds when i copy to clipboard [22:30] but you guys have been saying youtube-dl is better since it gets all the extra information, not just the .mp4 file [22:31] https://imgur.com/gallery/QehyF6u here's what it looks like for me [22:34] oh shit, that's nice [22:34] openwith an extension then? [22:35] yes [22:35] not finding it on the addon store thingy [22:37] Chrome or Firefox? https://addons.mozilla.org/en-US/firefox/addon/open-with/ [22:37] ahh, that's the one i saw, figured it only let you choose web-browsers, lol [22:38] it's the main purpose and it looks like *many* web designers use this [22:38] *** Medowar has quit IRC (Quit: Connection closed for inactivity) [22:39] can i see your best video+audio batch? [22:39] I'm preparing a "how to" on this. hold on for a few minutes [22:43] kk [22:46] i could see this easily being extended to package a site you're looking at into a warc or something, too [22:46] Anyone know what the barriers/roadblocks there are for creating a browser plugin that exposes a single "record" toggle button, and while it is on, all requests/responses are captured to WARC? [22:46] Sort of like webrecorder.io, but built into the browser. [22:48] So like a built in warcproxy [22:50] I think so, yeah [22:51] or a plugin that adds a one-click button to turn on/off warcproxy [22:54] atrocity: http://pastebin.com/zKkXEgs6 [22:56] My "bestvideo+bestaudio" options actually download not the best quality, but the one set in the config [22:56] So I'd add the extra archiving options to the config file [23:02] shouldn't it be X: && cd "path"? [23:02] why switch to the drive after you switched folders? [23:03] doesn't matter in which order you do that [23:04] basically, each partition has it's own working dir. as soon you switch drives, you have the same directory as set before for this drive [23:05] gotcha [23:05] so with this, you have 1200 set, guessing that's the height... does that mean it won't handle 4k? [23:07] if you must use Windows, try cygwin and zsh [23:07] My config forbids 4K [23:08] else, if you have ffmpeg.exe it will try to get the best possible quality by default [23:11] -f bestvideo+bestaudio/best will get me the best of both, even if 4k then? [23:12] that's the default, yes. if the first fails for whatever reason (it shouldn't) then it will fallback to "best" which is 720p at best (hehe) [23:12] youtube-dl's default behaviour* [23:13] actually in my experience bestvideo+bestaudio has downloaded VP9, which is lower quality than the MP4 version [23:13] not always. and sometimes the highest-res formats are only in VP9 [23:13] hm [23:14] VP9 can be pretty screwed-up on low-res videos [23:14] Yea, another note: bestvideo+bestaudio will download any format that it thinks is the best quality. But WEBM (VP9 Codec) is basically unplayable on mobile devices at the moment [23:14] VP9 on low-res videos: http://video.dailymail.co.uk/video/1418450360/2012/08/1418450360_1808428649001_55c52114930ba41b65cccc464f3037ad-503b9131.jpg [23:14] i use vlc for all [23:15] *** Yoshimura has quit IRC (http://www.kiwiirc.com/ - A hand crafted IRC client) [23:15] my archiving operation prefers VP9 because I Just Don't Care to figure out when I should prefer h.264 instead and use 40% more space [23:17] it also prefers 60fps over higher resolution. my format priority list is all inverted [23:19] doesn't youtube transcode all video to the same fps, though? [23:19] i know they used to not, but now, yes? [23:19] nope [23:19] I think I've seen 18, 24, 25, 30, 60 [23:19] so there're high resolution videos with less than 60fps even when the original had 60fps? [23:19] VADemon: yep [23:20] 60fps on YouTube is a recent thing AFAIK [23:20] *** hook54321 has quit IRC (Quit: Connection closed for inactivity) [23:20] FWIW if the person *uploaded* 60fps in the first place I think you should get it in 60fps now. [23:21] I think "recent" has turned 1 year old or so [23:21] If I remember correctly they keep the originals and re-transcode for newer formats as needed. [23:21] VADemon: that's pretty recent [23:21] youtube-dl will prefer to get the higher-resolution < 60fps format instead of the 60fps format [23:21] grr, lol [23:21] I'd prefer that too, really [23:21] i know the vids i upload all stay at 60fps [23:22] VADemon: btw, that worked perfectly, ty! [23:22] i think i'm going to add one more statement to make a new folder named the video file each time it's run in my download folder [23:22] since i have 5 files being saved, lol [23:24] nm, lol [23:24] Cool! And you should be able to set the output folder with -o but it doesn't seem to be easy [23:27] yeah, the issue is that i'm passing it a url... windows doesn't like to name folders stuff like "http://www.youtube.com/watch?v=LKJ$#JALJdfLKAjdla" etc. lol [23:27] just dump it all in the folder, i don't care [23:27] i only go back and watch all my downloaded content usually every few months before i archive it into m ycollection anyway, lol [23:27] if i don't want the info later, i'll delete, if i do, it's all there [23:30] wow, okay, just keeping this script. works wonderfully for 4k video as well [23:30] thanks again :D [23:33] atrocity: if you want to have a brainf* for a few hours but get the folders working: https://github.com/rg3/youtube-dl#output-template [23:33] e.g. /VIDEO-ID/name.format [23:35] *** dashcloud has quit IRC (Read error: Operation timed out) [23:36] *** dashcloud has joined #archiveteam-bs [23:54] *** Yoshimura has joined #archiveteam-bs [23:55] * Yoshimura notes: I might disappear at some point of time for some time (health, pc problems), would likely eventualy be back [23:56] Before you leave, I wish you best health, luck etc etc :)