| Time |
Nickname |
Message |
|
07:03
🔗
|
|
trs80 has quit IRC (ny.us.hub irc.colosolutions.net) |
|
07:04
🔗
|
|
trs80 has joined #warrior |
|
07:08
🔗
|
|
Jonison has joined #warrior |
|
10:22
🔗
|
|
Jonison2 has joined #warrior |
|
10:25
🔗
|
|
Jonison has quit IRC (Ping timeout: 260 seconds) |
|
10:46
🔗
|
|
Jonison2 has quit IRC (Quit: Leaving) |
|
17:54
🔗
|
|
SmileyG has joined #warrior |
|
17:54
🔗
|
|
Smiley has quit IRC (Read error: Connection reset by peer) |
|
20:02
🔗
|
|
Sanqui has joined #warrior |
|
20:02
🔗
|
|
jrwr has joined #warrior |
|
20:02
🔗
|
|
alembic has joined #warrior |
|
20:09
🔗
|
xmc |
from #msgbored: |
|
20:09
🔗
|
xmc |
13:01 <@Sanqui> what's the State of Warrior anyway? i heard it's a system past LTS, so probably security issues |
|
20:09
🔗
|
xmc |
13:01 <@xmc> eep |
|
20:09
🔗
|
xmc |
13:02 <@jrwr> We could update it to Ubuntu 16.04 LTS |
|
20:09
🔗
|
xmc |
13:02 <@jrwr> that should work for a long time |
|
20:09
🔗
|
xmc |
13:02 <@MrRadar> Yeah, I was looking into replacing it but I'm waiting on the next Debian release to go Stable |
|
20:09
🔗
|
xmc |
13:02 <@jrwr> Ya |
|
20:09
🔗
|
xmc |
13:02 <@MrRadar> Which should be any time now |
|
20:09
🔗
|
xmc |
13:02 <@jrwr> Ya |
|
20:11
🔗
|
jrwr |
You could start on it now, since its feature complete in debian |
|
20:11
🔗
|
MrRadar |
Upgrading the Warrior VM is not just a good idea for security, but also for robustness of scraping since as much as 5% of HTTPS sites today are TLS 1.2-only and the current warrior doesn't support above 1.0 |
|
20:12
🔗
|
jrwr |
Yep |
|
20:12
🔗
|
jrwr |
and Keeping the Certs up to date as well |
|
20:12
🔗
|
MrRadar |
IIRC we run all our scrapes with the certificate ignored |
|
20:12
🔗
|
jrwr |
Ya |
|
20:12
🔗
|
MrRadar |
Probably in part because they're so out of date |
|
20:12
🔗
|
MrRadar |
*the roots on the VM are so out of date |
|
20:13
🔗
|
jrwr |
Also, a better update path |
|
20:14
🔗
|
jrwr |
Keeping the VM Packages up to date and overall all the software on the VMs |
|
20:14
🔗
|
jrwr |
since most are set and forget for most people |
|
20:15
🔗
|
alembic |
like cron-job auto-update? |
|
20:15
🔗
|
jrwr |
Ya |
|
20:15
🔗
|
jrwr |
Have it phone home a little for checking on updates, could be a setting on the web panel |
|
20:15
🔗
|
jrwr |
I would stage the updates of course to make sure nothing broke |
|
20:17
🔗
|
jrwr |
You could even on the VMs have a X server boot as a option so a webbrowser could interface with the warrior without having to port forward |
|
20:17
🔗
|
jrwr |
You can get that down to really only taking about 30MB to do that |
|
20:18
🔗
|
Sanqui |
we should also consider like |
|
20:18
🔗
|
Sanqui |
idk, docker |
|
20:18
🔗
|
Sanqui |
seems to be a popular thing these days |
|
20:18
🔗
|
Sanqui |
or vagrant |
|
20:19
🔗
|
MrRadar |
Part of the appeal of the VM is you just tell people "download VirtualBox and run this file" and it just works |
|
20:20
🔗
|
MrRadar |
Docker et. al is a great option for people running VPSes |
|
20:20
🔗
|
MrRadar |
But to get "average" people (who may want to help save their favorite site that's closing) I think a VM is a must |
|
20:20
🔗
|
jrwr |
Yes |
|
20:21
🔗
|
Sanqui |
warrior doesn't require long-term uptime? |
|
20:21
🔗
|
jrwr |
Hell, with Win10 having Hyper-V pretty much included |
|
20:21
🔗
|
jrwr |
we should have that being easier |
|
20:21
🔗
|
jrwr |
I could write a tool that would make that easy mode |
|
20:21
🔗
|
xmc |
hyper-v is still kinda a pain to get going |
|
20:21
🔗
|
Sanqui |
because otherwise i think the number of people running warrior form their windows desktop is pretty low |
|
20:22
🔗
|
xmc |
it requires like two reboots and does scary-sounding things |
|
20:22
🔗
|
xmc |
and it only works on windows whatever pro |
|
20:22
🔗
|
MrRadar |
We don't assume Warrior instances are reliable |
|
20:22
🔗
|
jrwr |
^^ Correct |
|
20:22
🔗
|
MrRadar |
If a warrior instance fails to return its work we just requeue it |
|
20:22
🔗
|
jrwr |
If only Qemu could be faster on windows |
|
20:22
🔗
|
jrwr |
that would be legit method |
|
20:24
🔗
|
jrwr |
Hrm, Needs Intel HAXM to run |
|
20:24
🔗
|
xmc |
haxm? |
|
20:24
🔗
|
jrwr |
Its Intel's Usermode KVM thingie in windows |
|
20:24
🔗
|
alembic |
chipset instruction, I believe |
|
20:24
🔗
|
xmc |
ah |
|
20:24
🔗
|
jrwr |
Only works on Intels |
|
20:25
🔗
|
jrwr |
Since its a wrapper for VT |
|
20:25
🔗
|
alembic |
AMD doesn't have a compatibility layer? |
|
20:25
🔗
|
jrwr |
Not for Qemu |
|
20:25
🔗
|
alembic |
:( |
|
20:26
🔗
|
jrwr |
Virtual Box is kind of a pain to get running since its not all done for you |
|
20:27
🔗
|
jrwr |
I work tech support I've seen what dumb of the dumb can do |
|
20:27
🔗
|
xmc |
vmware player ? |
|
20:27
🔗
|
jrwr |
Most VMs still need a install |
|
20:27
🔗
|
jrwr |
but VMWare Player can take OVAs |
|
20:28
🔗
|
xmc |
i've been using vmware workstation for a little while, it is pretty darn fooproof |
|
20:28
🔗
|
jrwr |
hrm |
|
20:28
🔗
|
jrwr |
What about flat out a portable cygwin |
|
20:28
🔗
|
jrwr |
all the warriors are just python+wget-lua anyway |
|
20:28
🔗
|
xmc |
the point of the vm is to isolate the warrior from system weirdness and provide a stable platform for captures |
|
20:28
🔗
|
jrwr |
Right |
|
20:29
🔗
|
Sanqui |
i guess stuff like antivirus |
|
20:29
🔗
|
Sanqui |
which, again, is only relevant on windows lol |
|
20:29
🔗
|
jrwr |
Im more or less thinking of getting it down to almost no though needed to run one |
|
20:30
🔗
|
Sanqui |
i'd rather ensure we have something lightweight and stable for experienced users, so we don't have to tell them "better go run the script directly if you can figure it out" |
|
20:30
🔗
|
Sanqui |
s/experienced users/linux users/ |
|
20:30
🔗
|
Sanqui |
and then we can try to wrap that thing for windows users |
|
20:30
🔗
|
jrwr |
Ya |
|
20:31
🔗
|
Sanqui |
so i'd honestly say we need a container expert |
|
20:31
🔗
|
jrwr |
The containers are easy on everything else but windows |
|
20:31
🔗
|
jrwr |
I've used cygwin as a close second |
|
20:32
🔗
|
jrwr |
on windows in the past, but A/V can be a PITA |
|
20:32
🔗
|
Sanqui |
worst case we end up with a vm that starts the container again, but if we can make it more pleasant for people who can Just Spin Up a VPS that'd be ideal |
|
20:33
🔗
|
xmc |
vm with a container in it sounds like a least-hassle solution |
|
20:33
🔗
|
xmc |
in two ways |
|
20:33
🔗
|
xmc |
vm for people who want a vm |
|
20:33
🔗
|
xmc |
container comes from the same build toolchain as the vm |
|
20:33
🔗
|
jrwr |
Ya |
|
20:33
🔗
|
xmc |
so whoever maintains the warrior only has to deal with one output |
|
20:33
🔗
|
jrwr |
since LXC is supported in everything |
|
20:33
🔗
|
xmc |
everything new, you mean |
|
20:33
🔗
|
xmc |
i have a bunch of installs that predate it |
|
20:34
🔗
|
jrwr |
Oh... what is this |
|
20:34
🔗
|
jrwr |
LinuxKit |
|
20:34
🔗
|
jrwr |
Custom Linux Subsystems for Windows |
|
20:34
🔗
|
alembic |
I'm not sure install a docker application is simpler than running the scripts directly from a user-experience level.. if that's what we're arguing |
|
20:35
🔗
|
jrwr |
True, making the warriors very self contained is key |
|
20:35
🔗
|
alembic |
but that might just be my inexperience with containers shining-through ;) |
|
20:35
🔗
|
Sanqui |
but we don't want users to run scripts directly anyway |
|
20:35
🔗
|
jrwr |
even now the scripts are a little wonky with the data dir |
|
20:35
🔗
|
Sanqui |
and containers are easy if there are instructions |
|
20:35
🔗
|
Odd0002 |
well running docker for me is pretty easy, 1-2 commands and then it's installed |
|
20:36
🔗
|
Odd0002 |
then I just run a command line thingy for each container I want to run (always the same command) and then the web interface goes up after 30s and I don't have to worry about it |
|
20:36
🔗
|
jrwr |
I used the commands out of the dockerfile to spin one up on my LXD box I have |
|
20:37
🔗
|
jrwr |
For windows users, We could abuse Windows Subsystem for Linux |
|
20:38
🔗
|
jrwr |
This bat https://github.com/alwsl/alwsl/blob/master/alwsl.bat |
|
20:38
🔗
|
jrwr |
handles the whole shebang |
|
20:38
🔗
|
jrwr |
for getting Arch on Windows |
|
20:38
🔗
|
alembic |
Possibly OT, but on the "less technical user" side, I've been toying with an idea for a web-portal where users can just paypal $5 to automagically spin-up a VPS (via DO/OVH API) for month and run their ArchiveTeam Warrior project of choice... |
|
20:39
🔗
|
alembic |
unsure if that would be helpful |
|
20:39
🔗
|
alembic |
but it be easy to do "call-to-action" calls for it |
|
20:39
🔗
|
jrwr |
for the linux newbies |
|
20:39
🔗
|
alembic |
yah |
|
20:39
🔗
|
jrwr |
a one liner would be nice |
|
20:40
🔗
|
jrwr |
tell people to get a DO VPS Running Ubuntu |
|
20:40
🔗
|
jrwr |
and have the one liner do the rest |
|
20:40
🔗
|
jrwr |
I would suggest base LXC for that since docker is a little heavy |
|
20:45
🔗
|
jrwr |
I think I will make that |
|
20:46
🔗
|
jrwr |
A Page to pay X and it spins up a DO Droplet with everything, configure it and email the end user the creds |
|
20:46
🔗
|
jrwr |
Ill make it only the cost of the VM |
|
20:48
🔗
|
alembic |
i'm not a lawyer, but you have to be careful if you're emailing root creds b/c that might be reselling. Depending on the host, that might or might not be allowed. Also, don't know if that increases your liability |
|
20:48
🔗
|
alembic |
but I also get why you'd want to forward creds. |
|
20:48
🔗
|
jrwr |
The creds for the webpanel |
|
20:48
🔗
|
jrwr |
I'm not going to just leave it there unconfigured :3 |
|
20:49
🔗
|
Sanqui |
is a $5 VPS enough to run warrior, and are two $5 instances better than one $10 one? |
|
20:50
🔗
|
Sanqui |
stupid question i know but if we can avoid it i'd not waste resources |
|
20:50
🔗
|
xmc |
more ip addresses are usually better than more cpu |
|
20:50
🔗
|
jrwr |
^^ |
|
20:50
🔗
|
alembic |
I run the scripts directly on 2GB OVH servers which are $5/mo |
|
20:50
🔗
|
jrwr |
What kind of diskspace are we looking at |
|
20:50
🔗
|
jrwr |
50GB Enough? |
|
20:50
🔗
|
Sanqui |
ok, just wanted to make sure we're clear on that |
|
20:51
🔗
|
alembic |
jrwr, I would think so |
|
20:51
🔗
|
MrRadar |
jrwr: It depends on the project. Some require gigabytes per item, others only megabytes |
|
20:52
🔗
|
jrwr |
I might use scaleway as the backend on their x64 VPS |
|
20:52
🔗
|
jrwr |
the specs match and are cheaper, reselling is allowed |
|
20:54
🔗
|
jrwr |
2.99 Euro for 2GB RAm, 50GB Disk, 200Mbit connection |
|
20:54
🔗
|
jrwr |
thats not bad |
|
20:55
🔗
|
MrRadar |
Yeah, I've been using them for projects that don't require a lot of disk and I've *mostly* been happy |
|
20:55
🔗
|
jrwr |
I've used them in the past as well |
|
20:55
🔗
|
MrRadar |
When I first signed up they put all my VPSes on a machine with a flaky netowrk (as in silently corrupting packets) |
|
20:55
🔗
|
jrwr |
Nice |
|
20:55
🔗
|
MrRadar |
And they use Atom CPUs that are always sluggish |
|
20:56
🔗
|
jrwr |
Ill work on that today then, get some framework up, research payments and such |
|
20:56
🔗
|
MrRadar |
(In contrast my VPSes on Vultr are snappier than the old laptop I use as a Linux server at home) |
|
20:57
🔗
|
jrwr |
I could use the baremetal arm VPS |
|
20:58
🔗
|
alembic |
brb seting up a tower of raspberry pis ;) |
|
20:58
🔗
|
jrwr |
lol |
|
21:06
🔗
|
Odd0002 |
so you're saying a VPS is better than a pi? |
|
21:09
🔗
|
Odd0002 |
spec-wise |
|
21:09
🔗
|
MrRadar |
Unless you've got fiber at home a VPS (even an overloaded one) probably has better Internet access |
|
21:13
🔗
|
Odd0002 |
well the pi only has 1 GB of RAM and usually 8, 16, or 32 GB of storage |
|
21:18
🔗
|
jrwr |
Ill use nginx to protect the web interface a little |
|
21:18
🔗
|
jrwr |
with Auth Basic |
|
21:18
🔗
|
MrRadar |
I just use an SSH tunnel whenever I want to access the web interface (I only bind it to localhost and use port forwarding through SSH) |
|
21:19
🔗
|
jrwr |
Ya, but I'm thinking of the end users |
|
21:19
🔗
|
jrwr |
So they can get to the web interface, and it not being a issue that its on the public internet |
|
22:14
🔗
|
jrwr |
So, If I made this service, could I get a endorsement from AT? |
|
23:28
🔗
|
jrwr |
So, I've made this page https://jrwr.io/doku.php?id=projects:atvps |
|
23:29
🔗
|
jrwr |
Made up some script to deploy the image and do all the setup |