| Time |
Nickname |
Message |
|
02:17
🔗
|
|
klg has joined #urlteam |
|
04:11
🔗
|
|
dashcloud has quit IRC (Ping timeout: 633 seconds) |
|
04:15
🔗
|
|
dashcloud has joined #urlteam |
|
04:17
🔗
|
|
odemg has quit IRC (Ping timeout: 265 seconds) |
|
04:29
🔗
|
|
odemg has joined #urlteam |
|
06:05
🔗
|
|
dashcloud has quit IRC (No Ping reply in 210 seconds.) |
|
06:09
🔗
|
|
dashcloud has joined #urlteam |
|
06:28
🔗
|
|
dashcloud has quit IRC (Read error: Connection reset by peer) |
|
06:31
🔗
|
|
dashcloud has joined #urlteam |
|
06:32
🔗
|
|
bacchusm has joined #urlteam |
|
06:33
🔗
|
bacchusm |
Hi, I have some extra Bandwidth, lots of Disk Space and a extra Raspberry Pi 3+. What would be the best team to join with that? |
|
06:34
🔗
|
teej_ |
bacchusm: Team? |
|
06:35
🔗
|
bacchusm |
ftp team, Newsgrabber, URL etc... |
|
06:35
🔗
|
bacchusm |
Sorry Projects |
|
06:39
🔗
|
teej_ |
Oh. Well you can get started with URLTeam's TerrorOfTinyTown project. It uses very little bandwidth and space. |
|
06:41
🔗
|
teej_ |
That's an easy one to get started with. I haven't done the others yet. |
|
06:41
🔗
|
teej_ |
bacchusm: Firstly, you need to have Python installed. Use `pip install --upgrade seesaw requests`. |
|
06:42
🔗
|
bacchusm |
'okay That is easy |
|
06:42
🔗
|
teej_ |
Then you go to a directory... for example `cd Downloads`. |
|
06:44
🔗
|
teej_ |
Then you want to get the official git repo by typing `git clone --recurse-submodules https://github.com/ArchiveTeam/terroroftinytown-client-grab.git` |
|
06:44
🔗
|
teej_ |
That will get the URLTeam's code. |
|
06:45
🔗
|
teej_ |
Then you go into the directory: `cd terroroftinytown-client-grab/` |
|
06:47
🔗
|
teej_ |
Whenever you want to run the code, just type `run-pipeline --concurrent 6 --auto-update pipeline.py USERNAME` and replace the USERNAME with anything you like. |
|
06:49
🔗
|
teej_ |
You can set the concurrent flag to 3 if you want to save resources. I guess it doesn't matter much with bitly and tinyurl queries. |
|
06:49
🔗
|
bacchusm |
I put a new SD card I will follow directions shortly. |
|
06:49
🔗
|
bacchusm |
I did a screen grabe |
|
06:49
🔗
|
|
dashcloud has quit IRC (Quit: No Ping reply in 180 seconds.) |
|
06:51
🔗
|
teej_ |
You can the go to https://tracker.archiveteam.org:1338 to see your progress. |
|
06:51
🔗
|
bacchusm |
ok install some now. |
|
06:51
🔗
|
teej_ |
I have mine running 24/7. It's pretty easy. |
|
06:53
🔗
|
bacchusm |
IT says bash: run-pipeline command not found |
|
06:54
🔗
|
teej_ |
Did you do `pip install --upgrade seesaw requests`? |
|
06:54
🔗
|
|
dashcloud has joined #urlteam |
|
06:55
🔗
|
bacchusm |
Yes I did |
|
06:56
🔗
|
teej_ |
Type `python --version` what does it say? |
|
06:56
🔗
|
bacchusm |
2.7.13 |
|
06:57
🔗
|
bacchusm |
Does it need 3.7? |
|
06:59
🔗
|
teej_ |
Hmm... well you can use python3 as well... |
|
07:00
🔗
|
teej_ |
You can do `pip3 install --upgrade seesaw requests` and then `run-pipeline3 --concurrent 6 --auto-update pipeline.py USERNAME`. |
|
07:01
🔗
|
teej_ |
I'm not sure why your `run-pipeline` command isn't there. |
|
07:02
🔗
|
bacchusm |
Do I need the TinyTown grab using Python 3? |
|
07:03
🔗
|
bacchusm |
Where does 'Pipeline' get installed is that same as Seesaw? |
|
07:06
🔗
|
teej_ |
So when you use the pip install thing, I think it would install the run-pipeline command as well... I remember having the same issue as you did when I first set it up. |
|
07:07
🔗
|
teej_ |
You can run the bash script, `bash warrior-install.sh`. |
|
07:07
🔗
|
teej_ |
Mayve try closing the terminal and opening it again... |
|
07:08
🔗
|
bacchusm |
I will try closing and opening. |
|
07:08
🔗
|
bacchusm |
Bash warrior didn't work |
|
07:11
🔗
|
teej_ |
Hmmm... So when you do `pip install --upgrade seesaw requests`, what does it say? |
|
07:14
🔗
|
bacchusm |
Collecting seesaw , Collecting requests... |
|
07:15
🔗
|
bacchusm |
then a whole bunch of packagse.. but I don't see pipeline. weird but it might have scrolld by |
|
07:16
🔗
|
teej_ |
Oh... if it's collecting, doesn't that mean it just finished installing? |
|
07:16
🔗
|
teej_ |
If you type run-pip and then press tab, does it autocomplete to run-pipeline? |
|
07:17
🔗
|
bacchusm |
nope. It installed six,singledispatch, certifi, backports-abc, tornado, socksjs, sesaw, urllib3, idna, chardet , requests |
|
07:17
🔗
|
bacchusm |
Thank you for helping me really apprecaited it |
|
07:18
🔗
|
bacchusm |
run and tab... it has runcon, runleve, run-parts, run-with-aspell, run-erl, run-mailcap,runuser |
|
07:20
🔗
|
teej_ |
Hmm... |
|
07:21
🔗
|
teej_ |
I'm trying to remember what I did to fix the same issue. |
|
07:22
🔗
|
bacchusm |
I used this and it started to work |
|
07:22
🔗
|
bacchusm |
~/.local/bin/run-pipeline pipeline.py --concurrent 2 YOURNICKHERE |
|
07:22
🔗
|
bacchusm |
I believe it is working at least |
|
07:25
🔗
|
teej_ |
Oh! That means pip installed run-pipeline into your local user directory! |
|
07:26
🔗
|
teej_ |
So you can probably try `sudo pip install --upgrade seesaw requests`. |
|
07:26
🔗
|
bacchusm |
So stop it and use the sudo? |
|
07:27
🔗
|
teej_ |
Try `pip install --global --upgrade seesaw requests` first. |
|
07:29
🔗
|
teej_ |
Actually... I think you're just using your standard python. So everything is pretty much perfectly fine. My system uses pyenv for python, so that's probably why it's a little different. |
|
07:30
🔗
|
bacchusm |
oh. okay. So I can just let it run. I saw my name flash by in the stats so it seems to be working. |
|
07:30
🔗
|
bacchusm |
Is there another project to run on a Pi or is this the only one you tried? |
|
07:31
🔗
|
teej_ |
So... recap: whenever you want to run the URLTeam code, just `cd` to the directory and type `~/.local/bin/run-pipeline pipeline.py --concurrent 2 YOURNICKHERE` and just let it run. |
|
07:31
🔗
|
teej_ |
Well... this is the only one I tried so far. |
|
07:32
🔗
|
bacchusm |
Yeah. I am going to set it up at my office tomorrow and let it continually run. |
|
07:32
🔗
|
bacchusm |
Cool! Once again, thank you for your help! |
|
07:32
🔗
|
teej_ |
You could also set up the ArchiveBot which can archive thousands of pages, but if you do, your IP might get accidentally banned on some sites. |
|
07:33
🔗
|
teej_ |
That's why I haven't done that. |
|
07:33
🔗
|
bacchusm |
Does that work on a PI? I didn't see any install. |
|
07:33
🔗
|
bacchusm |
They all need VirtualBox or Vmware, which the PI doesn't run. At least not that I know of |
|
07:34
🔗
|
bacchusm |
I'm okay with getting banned on some sites. |
|
07:34
🔗
|
teej_ |
I don't think it will run on Pi. You can see what's being archived here: http://dashboard.at.ninjawedding.org/ |
|
07:36
🔗
|
teej_ |
Some jobs lasts for months. They eventually get uploaded to the Internet Archive so people can view websites using the Wayback Machine. |
|
07:37
🔗
|
bacchusm |
Very Cool. Do you know what type of machine do I need for an Archivebot? I have a couple old ones sitting around and plenty of HD space |
|
07:38
🔗
|
teej_ |
https://www.archiveteam.org/index.php?title=ArchiveBot#Volunteer_to_run_a_Pipeline |
|
07:39
🔗
|
bacchusm |
Is that the same pipeline I am running now? |
|
07:40
🔗
|
teej_ |
So you join the #archivebot channel and ask one of the Ops for setting things up and getting things working. |
|
07:40
🔗
|
bacchusm |
Okay cool. Thank You again. Will bounce over there now. |
|
07:40
🔗
|
teej_ |
No, the pipeline you're running is for URLTeam's TerrorOfTinyTown. |
|
07:40
🔗
|
bacchusm |
okay got Ya |
|
07:41
🔗
|
|
bacchusm_ has joined #urlteam |
|
07:42
🔗
|
|
qwebirc57 has joined #urlteam |
|
07:43
🔗
|
qwebirc57 |
sorry I click on your link and it exited me |
|
07:43
🔗
|
|
bacchusm_ has quit IRC (Client Quit) |
|
07:43
🔗
|
teej_ |
Who's link? |
|
07:43
🔗
|
|
bacchusm_ has joined #urlteam |
|
07:45
🔗
|
|
bacchusm has quit IRC (Ping timeout: 260 seconds) |
|
07:45
🔗
|
|
bacchusm has joined #urlteam |
|
07:47
🔗
|
|
qwebirc57 has quit IRC (Ping timeout: 260 seconds) |
|
07:48
🔗
|
bacchusm |
Sorry Teej, I click on your link and got trasnfer to other IRC channel. I thought another window would open |
|
07:48
🔗
|
|
bacchusm_ has quit IRC (Ping timeout: 260 seconds) |
|
07:56
🔗
|
teej_ |
bacchusm: Oh. That's fine. Don't worry. |
|
07:57
🔗
|
teej_ |
For IRC stuff, I use IRCCloud. It makes things easier. |
|
12:12
🔗
|
|
zerkalo has quit IRC (Ping timeout: 260 seconds) |
|
12:23
🔗
|
|
zerkalo has joined #urlteam |
|
13:01
🔗
|
|
Kagee has quit IRC (WeeChat 2.3) |
|
13:03
🔗
|
|
Kagee has joined #urlteam |
|
21:55
🔗
|
|
coldon2dr has quit IRC (Read error: Operation timed out) |
|
22:43
🔗
|
|
bacchusm has quit IRC (Ping timeout: 260 seconds) |
|
22:55
🔗
|
|
hook54321 has quit IRC (Quit: Connection closed for inactivity) |