#warrior 2017-06-02,Fri

↑back Search

Time Nickname Message
07:03 🔗 trs80 has quit IRC (ny.us.hub irc.colosolutions.net)
07:04 🔗 trs80 has joined #warrior
07:08 🔗 Jonison has joined #warrior
10:22 🔗 Jonison2 has joined #warrior
10:25 🔗 Jonison has quit IRC (Ping timeout: 260 seconds)
10:46 🔗 Jonison2 has quit IRC (Quit: Leaving)
17:54 🔗 SmileyG has joined #warrior
17:54 🔗 Smiley has quit IRC (Read error: Connection reset by peer)
20:02 🔗 Sanqui has joined #warrior
20:02 🔗 jrwr has joined #warrior
20:02 🔗 alembic has joined #warrior
20:09 🔗 xmc from #msgbored:
20:09 🔗 xmc 13:01 <@Sanqui> what's the State of Warrior anyway? i heard it's a system past LTS, so probably security issues
20:09 🔗 xmc 13:01 <@xmc> eep
20:09 🔗 xmc 13:02 <@jrwr> We could update it to Ubuntu 16.04 LTS
20:09 🔗 xmc 13:02 <@jrwr> that should work for a long time
20:09 🔗 xmc 13:02 <@MrRadar> Yeah, I was looking into replacing it but I'm waiting on the next Debian release to go Stable
20:09 🔗 xmc 13:02 <@jrwr> Ya
20:09 🔗 xmc 13:02 <@MrRadar> Which should be any time now
20:09 🔗 xmc 13:02 <@jrwr> Ya
20:11 🔗 jrwr You could start on it now, since its feature complete in debian
20:11 🔗 MrRadar Upgrading the Warrior VM is not just a good idea for security, but also for robustness of scraping since as much as 5% of HTTPS sites today are TLS 1.2-only and the current warrior doesn't support above 1.0
20:12 🔗 jrwr Yep
20:12 🔗 jrwr and Keeping the Certs up to date as well
20:12 🔗 MrRadar IIRC we run all our scrapes with the certificate ignored
20:12 🔗 jrwr Ya
20:12 🔗 MrRadar Probably in part because they're so out of date
20:12 🔗 MrRadar *the roots on the VM are so out of date
20:13 🔗 jrwr Also, a better update path
20:14 🔗 jrwr Keeping the VM Packages up to date and overall all the software on the VMs
20:14 🔗 jrwr since most are set and forget for most people
20:15 🔗 alembic like cron-job auto-update?
20:15 🔗 jrwr Ya
20:15 🔗 jrwr Have it phone home a little for checking on updates, could be a setting on the web panel
20:15 🔗 jrwr I would stage the updates of course to make sure nothing broke
20:17 🔗 jrwr You could even on the VMs have a X server boot as a option so a webbrowser could interface with the warrior without having to port forward
20:17 🔗 jrwr You can get that down to really only taking about 30MB to do that
20:18 🔗 Sanqui we should also consider like
20:18 🔗 Sanqui idk, docker
20:18 🔗 Sanqui seems to be a popular thing these days
20:18 🔗 Sanqui or vagrant
20:19 🔗 MrRadar Part of the appeal of the VM is you just tell people "download VirtualBox and run this file" and it just works
20:20 🔗 MrRadar Docker et. al is a great option for people running VPSes
20:20 🔗 MrRadar But to get "average" people (who may want to help save their favorite site that's closing) I think a VM is a must
20:20 🔗 jrwr Yes
20:21 🔗 Sanqui warrior doesn't require long-term uptime?
20:21 🔗 jrwr Hell, with Win10 having Hyper-V pretty much included
20:21 🔗 jrwr we should have that being easier
20:21 🔗 jrwr I could write a tool that would make that easy mode
20:21 🔗 xmc hyper-v is still kinda a pain to get going
20:21 🔗 Sanqui because otherwise i think the number of people running warrior form their windows desktop is pretty low
20:22 🔗 xmc it requires like two reboots and does scary-sounding things
20:22 🔗 xmc and it only works on windows whatever pro
20:22 🔗 MrRadar We don't assume Warrior instances are reliable
20:22 🔗 jrwr ^^ Correct
20:22 🔗 MrRadar If a warrior instance fails to return its work we just requeue it
20:22 🔗 jrwr If only Qemu could be faster on windows
20:22 🔗 jrwr that would be legit method
20:24 🔗 jrwr Hrm, Needs Intel HAXM to run
20:24 🔗 xmc haxm?
20:24 🔗 jrwr Its Intel's Usermode KVM thingie in windows
20:24 🔗 alembic chipset instruction, I believe
20:24 🔗 xmc ah
20:24 🔗 jrwr Only works on Intels
20:25 🔗 jrwr Since its a wrapper for VT
20:25 🔗 alembic AMD doesn't have a compatibility layer?
20:25 🔗 jrwr Not for Qemu
20:25 🔗 alembic :(
20:26 🔗 jrwr Virtual Box is kind of a pain to get running since its not all done for you
20:27 🔗 jrwr I work tech support I've seen what dumb of the dumb can do
20:27 🔗 xmc vmware player ?
20:27 🔗 jrwr Most VMs still need a install
20:27 🔗 jrwr but VMWare Player can take OVAs
20:28 🔗 xmc i've been using vmware workstation for a little while, it is pretty darn fooproof
20:28 🔗 jrwr hrm
20:28 🔗 jrwr What about flat out a portable cygwin
20:28 🔗 jrwr all the warriors are just python+wget-lua anyway
20:28 🔗 xmc the point of the vm is to isolate the warrior from system weirdness and provide a stable platform for captures
20:28 🔗 jrwr Right
20:29 🔗 Sanqui i guess stuff like antivirus
20:29 🔗 Sanqui which, again, is only relevant on windows lol
20:29 🔗 jrwr Im more or less thinking of getting it down to almost no though needed to run one
20:30 🔗 Sanqui i'd rather ensure we have something lightweight and stable for experienced users, so we don't have to tell them "better go run the script directly if you can figure it out"
20:30 🔗 Sanqui s/experienced users/linux users/
20:30 🔗 Sanqui and then we can try to wrap that thing for windows users
20:30 🔗 jrwr Ya
20:31 🔗 Sanqui so i'd honestly say we need a container expert
20:31 🔗 jrwr The containers are easy on everything else but windows
20:31 🔗 jrwr I've used cygwin as a close second
20:32 🔗 jrwr on windows in the past, but A/V can be a PITA
20:32 🔗 Sanqui worst case we end up with a vm that starts the container again, but if we can make it more pleasant for people who can Just Spin Up a VPS that'd be ideal
20:33 🔗 xmc vm with a container in it sounds like a least-hassle solution
20:33 🔗 xmc in two ways
20:33 🔗 xmc vm for people who want a vm
20:33 🔗 xmc container comes from the same build toolchain as the vm
20:33 🔗 jrwr Ya
20:33 🔗 xmc so whoever maintains the warrior only has to deal with one output
20:33 🔗 jrwr since LXC is supported in everything
20:33 🔗 xmc everything new, you mean
20:33 🔗 xmc i have a bunch of installs that predate it
20:34 🔗 jrwr Oh... what is this
20:34 🔗 jrwr LinuxKit
20:34 🔗 jrwr Custom Linux Subsystems for Windows
20:34 🔗 alembic I'm not sure install a docker application is simpler than running the scripts directly from a user-experience level.. if that's what we're arguing
20:35 🔗 jrwr True, making the warriors very self contained is key
20:35 🔗 alembic but that might just be my inexperience with containers shining-through ;)
20:35 🔗 Sanqui but we don't want users to run scripts directly anyway
20:35 🔗 jrwr even now the scripts are a little wonky with the data dir
20:35 🔗 Sanqui and containers are easy if there are instructions
20:35 🔗 Odd0002 well running docker for me is pretty easy, 1-2 commands and then it's installed
20:36 🔗 Odd0002 then I just run a command line thingy for each container I want to run (always the same command) and then the web interface goes up after 30s and I don't have to worry about it
20:36 🔗 jrwr I used the commands out of the dockerfile to spin one up on my LXD box I have
20:37 🔗 jrwr For windows users, We could abuse Windows Subsystem for Linux
20:38 🔗 jrwr This bat https://github.com/alwsl/alwsl/blob/master/alwsl.bat
20:38 🔗 jrwr handles the whole shebang
20:38 🔗 jrwr for getting Arch on Windows
20:38 🔗 alembic Possibly OT, but on the "less technical user" side, I've been toying with an idea for a web-portal where users can just paypal $5 to automagically spin-up a VPS (via DO/OVH API) for month and run their ArchiveTeam Warrior project of choice...
20:39 🔗 alembic unsure if that would be helpful
20:39 🔗 alembic but it be easy to do "call-to-action" calls for it
20:39 🔗 jrwr for the linux newbies
20:39 🔗 alembic yah
20:39 🔗 jrwr a one liner would be nice
20:40 🔗 jrwr tell people to get a DO VPS Running Ubuntu
20:40 🔗 jrwr and have the one liner do the rest
20:40 🔗 jrwr I would suggest base LXC for that since docker is a little heavy
20:45 🔗 jrwr I think I will make that
20:46 🔗 jrwr A Page to pay X and it spins up a DO Droplet with everything, configure it and email the end user the creds
20:46 🔗 jrwr Ill make it only the cost of the VM
20:48 🔗 alembic i'm not a lawyer, but you have to be careful if you're emailing root creds b/c that might be reselling. Depending on the host, that might or might not be allowed. Also, don't know if that increases your liability
20:48 🔗 alembic but I also get why you'd want to forward creds.
20:48 🔗 jrwr The creds for the webpanel
20:48 🔗 jrwr I'm not going to just leave it there unconfigured :3
20:49 🔗 Sanqui is a $5 VPS enough to run warrior, and are two $5 instances better than one $10 one?
20:50 🔗 Sanqui stupid question i know but if we can avoid it i'd not waste resources
20:50 🔗 xmc more ip addresses are usually better than more cpu
20:50 🔗 jrwr ^^
20:50 🔗 alembic I run the scripts directly on 2GB OVH servers which are $5/mo
20:50 🔗 jrwr What kind of diskspace are we looking at
20:50 🔗 jrwr 50GB Enough?
20:50 🔗 Sanqui ok, just wanted to make sure we're clear on that
20:51 🔗 alembic jrwr, I would think so
20:51 🔗 MrRadar jrwr: It depends on the project. Some require gigabytes per item, others only megabytes
20:52 🔗 jrwr I might use scaleway as the backend on their x64 VPS
20:52 🔗 jrwr the specs match and are cheaper, reselling is allowed
20:54 🔗 jrwr 2.99 Euro for 2GB RAm, 50GB Disk, 200Mbit connection
20:54 🔗 jrwr thats not bad
20:55 🔗 MrRadar Yeah, I've been using them for projects that don't require a lot of disk and I've *mostly* been happy
20:55 🔗 jrwr I've used them in the past as well
20:55 🔗 MrRadar When I first signed up they put all my VPSes on a machine with a flaky netowrk (as in silently corrupting packets)
20:55 🔗 jrwr Nice
20:55 🔗 MrRadar And they use Atom CPUs that are always sluggish
20:56 🔗 jrwr Ill work on that today then, get some framework up, research payments and such
20:56 🔗 MrRadar (In contrast my VPSes on Vultr are snappier than the old laptop I use as a Linux server at home)
20:57 🔗 jrwr I could use the baremetal arm VPS
20:58 🔗 alembic brb seting up a tower of raspberry pis ;)
20:58 🔗 jrwr lol
21:06 🔗 Odd0002 so you're saying a VPS is better than a pi?
21:09 🔗 Odd0002 spec-wise
21:09 🔗 MrRadar Unless you've got fiber at home a VPS (even an overloaded one) probably has better Internet access
21:13 🔗 Odd0002 well the pi only has 1 GB of RAM and usually 8, 16, or 32 GB of storage
21:18 🔗 jrwr Ill use nginx to protect the web interface a little
21:18 🔗 jrwr with Auth Basic
21:18 🔗 MrRadar I just use an SSH tunnel whenever I want to access the web interface (I only bind it to localhost and use port forwarding through SSH)
21:19 🔗 jrwr Ya, but I'm thinking of the end users
21:19 🔗 jrwr So they can get to the web interface, and it not being a issue that its on the public internet
22:14 🔗 jrwr So, If I made this service, could I get a endorsement from AT?
23:28 🔗 jrwr So, I've made this page https://jrwr.io/doku.php?id=projects:atvps
23:29 🔗 jrwr Made up some script to deploy the image and do all the setup

irclogger-viewer