[07:03] *** trs80 has quit IRC (ny.us.hub irc.colosolutions.net) [07:04] *** trs80 has joined #warrior [07:08] *** Jonison has joined #warrior [10:22] *** Jonison2 has joined #warrior [10:25] *** Jonison has quit IRC (Ping timeout: 260 seconds) [10:46] *** Jonison2 has quit IRC (Quit: Leaving) [17:54] *** SmileyG has joined #warrior [17:54] *** Smiley has quit IRC (Read error: Connection reset by peer) [20:02] *** Sanqui has joined #warrior [20:02] *** jrwr has joined #warrior [20:02] *** alembic has joined #warrior [20:09] from #msgbored: [20:09] 13:01 <@Sanqui> what's the State of Warrior anyway? i heard it's a system past LTS, so probably security issues [20:09] 13:01 <@xmc> eep [20:09] 13:02 <@jrwr> We could update it to Ubuntu 16.04 LTS [20:09] 13:02 <@jrwr> that should work for a long time [20:09] 13:02 <@MrRadar> Yeah, I was looking into replacing it but I'm waiting on the next Debian release to go Stable [20:09] 13:02 <@jrwr> Ya [20:09] 13:02 <@MrRadar> Which should be any time now [20:09] 13:02 <@jrwr> Ya [20:11] You could start on it now, since its feature complete in debian [20:11] Upgrading the Warrior VM is not just a good idea for security, but also for robustness of scraping since as much as 5% of HTTPS sites today are TLS 1.2-only and the current warrior doesn't support above 1.0 [20:12] Yep [20:12] and Keeping the Certs up to date as well [20:12] IIRC we run all our scrapes with the certificate ignored [20:12] Ya [20:12] Probably in part because they're so out of date [20:12] *the roots on the VM are so out of date [20:13] Also, a better update path [20:14] Keeping the VM Packages up to date and overall all the software on the VMs [20:14] since most are set and forget for most people [20:15] like cron-job auto-update? [20:15] Ya [20:15] Have it phone home a little for checking on updates, could be a setting on the web panel [20:15] I would stage the updates of course to make sure nothing broke [20:17] You could even on the VMs have a X server boot as a option so a webbrowser could interface with the warrior without having to port forward [20:17] You can get that down to really only taking about 30MB to do that [20:18] we should also consider like [20:18] idk, docker [20:18] seems to be a popular thing these days [20:18] or vagrant [20:19] Part of the appeal of the VM is you just tell people "download VirtualBox and run this file" and it just works [20:20] Docker et. al is a great option for people running VPSes [20:20] But to get "average" people (who may want to help save their favorite site that's closing) I think a VM is a must [20:20] Yes [20:21] warrior doesn't require long-term uptime? [20:21] Hell, with Win10 having Hyper-V pretty much included [20:21] we should have that being easier [20:21] I could write a tool that would make that easy mode [20:21] hyper-v is still kinda a pain to get going [20:21] because otherwise i think the number of people running warrior form their windows desktop is pretty low [20:22] it requires like two reboots and does scary-sounding things [20:22] and it only works on windows whatever pro [20:22] We don't assume Warrior instances are reliable [20:22] ^^ Correct [20:22] If a warrior instance fails to return its work we just requeue it [20:22] If only Qemu could be faster on windows [20:22] that would be legit method [20:24] Hrm, Needs Intel HAXM to run [20:24] haxm? [20:24] Its Intel's Usermode KVM thingie in windows [20:24] chipset instruction, I believe [20:24] ah [20:24] Only works on Intels [20:25] Since its a wrapper for VT [20:25] AMD doesn't have a compatibility layer? [20:25] Not for Qemu [20:25] :( [20:26] Virtual Box is kind of a pain to get running since its not all done for you [20:27] I work tech support I've seen what dumb of the dumb can do [20:27] vmware player ? [20:27] Most VMs still need a install [20:27] but VMWare Player can take OVAs [20:28] i've been using vmware workstation for a little while, it is pretty darn fooproof [20:28] hrm [20:28] What about flat out a portable cygwin [20:28] all the warriors are just python+wget-lua anyway [20:28] the point of the vm is to isolate the warrior from system weirdness and provide a stable platform for captures [20:28] Right [20:29] i guess stuff like antivirus [20:29] which, again, is only relevant on windows lol [20:29] Im more or less thinking of getting it down to almost no though needed to run one [20:30] i'd rather ensure we have something lightweight and stable for experienced users, so we don't have to tell them "better go run the script directly if you can figure it out" [20:30] s/experienced users/linux users/ [20:30] and then we can try to wrap that thing for windows users [20:30] Ya [20:31] so i'd honestly say we need a container expert [20:31] The containers are easy on everything else but windows [20:31] I've used cygwin as a close second [20:32] on windows in the past, but A/V can be a PITA [20:32] worst case we end up with a vm that starts the container again, but if we can make it more pleasant for people who can Just Spin Up a VPS that'd be ideal [20:33] vm with a container in it sounds like a least-hassle solution [20:33] in two ways [20:33] vm for people who want a vm [20:33] container comes from the same build toolchain as the vm [20:33] Ya [20:33] so whoever maintains the warrior only has to deal with one output [20:33] since LXC is supported in everything [20:33] everything new, you mean [20:33] i have a bunch of installs that predate it [20:34] Oh... what is this [20:34] LinuxKit [20:34] Custom Linux Subsystems for Windows [20:34] I'm not sure install a docker application is simpler than running the scripts directly from a user-experience level.. if that's what we're arguing [20:35] True, making the warriors very self contained is key [20:35] but that might just be my inexperience with containers shining-through ;) [20:35] but we don't want users to run scripts directly anyway [20:35] even now the scripts are a little wonky with the data dir [20:35] and containers are easy if there are instructions [20:35] well running docker for me is pretty easy, 1-2 commands and then it's installed [20:36] then I just run a command line thingy for each container I want to run (always the same command) and then the web interface goes up after 30s and I don't have to worry about it [20:36] I used the commands out of the dockerfile to spin one up on my LXD box I have [20:37] For windows users, We could abuse Windows Subsystem for Linux [20:38] This bat https://github.com/alwsl/alwsl/blob/master/alwsl.bat [20:38] handles the whole shebang [20:38] for getting Arch on Windows [20:38] Possibly OT, but on the "less technical user" side, I've been toying with an idea for a web-portal where users can just paypal $5 to automagically spin-up a VPS (via DO/OVH API) for month and run their ArchiveTeam Warrior project of choice... [20:39] unsure if that would be helpful [20:39] but it be easy to do "call-to-action" calls for it [20:39] for the linux newbies [20:39] yah [20:39] a one liner would be nice [20:40] tell people to get a DO VPS Running Ubuntu [20:40] and have the one liner do the rest [20:40] I would suggest base LXC for that since docker is a little heavy [20:45] I think I will make that [20:46] A Page to pay X and it spins up a DO Droplet with everything, configure it and email the end user the creds [20:46] Ill make it only the cost of the VM [20:48] i'm not a lawyer, but you have to be careful if you're emailing root creds b/c that might be reselling. Depending on the host, that might or might not be allowed. Also, don't know if that increases your liability [20:48] but I also get why you'd want to forward creds. [20:48] The creds for the webpanel [20:48] I'm not going to just leave it there unconfigured :3 [20:49] is a $5 VPS enough to run warrior, and are two $5 instances better than one $10 one? [20:50] stupid question i know but if we can avoid it i'd not waste resources [20:50] more ip addresses are usually better than more cpu [20:50] ^^ [20:50] I run the scripts directly on 2GB OVH servers which are $5/mo [20:50] What kind of diskspace are we looking at [20:50] 50GB Enough? [20:50] ok, just wanted to make sure we're clear on that [20:51] jrwr, I would think so [20:51] jrwr: It depends on the project. Some require gigabytes per item, others only megabytes [20:52] I might use scaleway as the backend on their x64 VPS [20:52] the specs match and are cheaper, reselling is allowed [20:54] 2.99 Euro for 2GB RAm, 50GB Disk, 200Mbit connection [20:54] thats not bad [20:55] Yeah, I've been using them for projects that don't require a lot of disk and I've *mostly* been happy [20:55] I've used them in the past as well [20:55] When I first signed up they put all my VPSes on a machine with a flaky netowrk (as in silently corrupting packets) [20:55] Nice [20:55] And they use Atom CPUs that are always sluggish [20:56] Ill work on that today then, get some framework up, research payments and such [20:56] (In contrast my VPSes on Vultr are snappier than the old laptop I use as a Linux server at home) [20:57] I could use the baremetal arm VPS [20:58] brb seting up a tower of raspberry pis ;) [20:58] lol [21:06] so you're saying a VPS is better than a pi? [21:09] spec-wise [21:09] Unless you've got fiber at home a VPS (even an overloaded one) probably has better Internet access [21:13] well the pi only has 1 GB of RAM and usually 8, 16, or 32 GB of storage [21:18] Ill use nginx to protect the web interface a little [21:18] with Auth Basic [21:18] I just use an SSH tunnel whenever I want to access the web interface (I only bind it to localhost and use port forwarding through SSH) [21:19] Ya, but I'm thinking of the end users [21:19] So they can get to the web interface, and it not being a issue that its on the public internet [22:14] So, If I made this service, could I get a endorsement from AT? [23:28] So, I've made this page https://jrwr.io/doku.php?id=projects:atvps [23:29] Made up some script to deploy the image and do all the setup