[02:17] *** klg has joined #urlteam [04:11] *** dashcloud has quit IRC (Ping timeout: 633 seconds) [04:15] *** dashcloud has joined #urlteam [04:17] *** odemg has quit IRC (Ping timeout: 265 seconds) [04:29] *** odemg has joined #urlteam [06:05] *** dashcloud has quit IRC (No Ping reply in 210 seconds.) [06:09] *** dashcloud has joined #urlteam [06:28] *** dashcloud has quit IRC (Read error: Connection reset by peer) [06:31] *** dashcloud has joined #urlteam [06:32] *** bacchusm has joined #urlteam [06:33] Hi, I have some extra Bandwidth, lots of Disk Space and a extra Raspberry Pi 3+. What would be the best team to join with that? [06:34] bacchusm: Team? [06:35] ftp team, Newsgrabber, URL etc... [06:35] Sorry Projects [06:39] Oh. Well you can get started with URLTeam's TerrorOfTinyTown project. It uses very little bandwidth and space. [06:41] That's an easy one to get started with. I haven't done the others yet. [06:41] bacchusm: Firstly, you need to have Python installed. Use `pip install --upgrade seesaw requests`. [06:42] 'okay That is easy [06:42] Then you go to a directory... for example `cd Downloads`. [06:44] Then you want to get the official git repo by typing `git clone --recurse-submodules https://github.com/ArchiveTeam/terroroftinytown-client-grab.git` [06:44] That will get the URLTeam's code. [06:45] Then you go into the directory: `cd terroroftinytown-client-grab/` [06:47] Whenever you want to run the code, just type `run-pipeline --concurrent 6 --auto-update pipeline.py USERNAME` and replace the USERNAME with anything you like. [06:49] You can set the concurrent flag to 3 if you want to save resources. I guess it doesn't matter much with bitly and tinyurl queries. [06:49] I put a new SD card I will follow directions shortly. [06:49] I did a screen grabe [06:49] *** dashcloud has quit IRC (Quit: No Ping reply in 180 seconds.) [06:51] You can the go to https://tracker.archiveteam.org:1338 to see your progress. [06:51] ok install some now. [06:51] I have mine running 24/7. It's pretty easy. [06:53] IT says bash: run-pipeline command not found [06:54] Did you do `pip install --upgrade seesaw requests`? [06:54] *** dashcloud has joined #urlteam [06:55] Yes I did [06:56] Type `python --version` what does it say? [06:56] 2.7.13 [06:57] Does it need 3.7? [06:59] Hmm... well you can use python3 as well... [07:00] You can do `pip3 install --upgrade seesaw requests` and then `run-pipeline3 --concurrent 6 --auto-update pipeline.py USERNAME`. [07:01] I'm not sure why your `run-pipeline` command isn't there. [07:02] Do I need the TinyTown grab using Python 3? [07:03] Where does 'Pipeline' get installed is that same as Seesaw? [07:06] So when you use the pip install thing, I think it would install the run-pipeline command as well... I remember having the same issue as you did when I first set it up. [07:07] You can run the bash script, `bash warrior-install.sh`. [07:07] Mayve try closing the terminal and opening it again... [07:08] I will try closing and opening. [07:08] Bash warrior didn't work [07:11] Hmmm... So when you do `pip install --upgrade seesaw requests`, what does it say? [07:14] Collecting seesaw , Collecting requests... [07:15] then a whole bunch of packagse.. but I don't see pipeline. weird but it might have scrolld by [07:16] Oh... if it's collecting, doesn't that mean it just finished installing? [07:16] If you type run-pip and then press tab, does it autocomplete to run-pipeline? [07:17] nope. It installed six,singledispatch, certifi, backports-abc, tornado, socksjs, sesaw, urllib3, idna, chardet , requests [07:17] Thank you for helping me really apprecaited it [07:18] run and tab... it has runcon, runleve, run-parts, run-with-aspell, run-erl, run-mailcap,runuser [07:20] Hmm... [07:21] I'm trying to remember what I did to fix the same issue. [07:22] I used this and it started to work [07:22] ~/.local/bin/run-pipeline pipeline.py --concurrent 2 YOURNICKHERE [07:22] I believe it is working at least [07:25] Oh! That means pip installed run-pipeline into your local user directory! [07:26] So you can probably try `sudo pip install --upgrade seesaw requests`. [07:26] So stop it and use the sudo? [07:27] Try `pip install --global --upgrade seesaw requests` first. [07:29] Actually... I think you're just using your standard python. So everything is pretty much perfectly fine. My system uses pyenv for python, so that's probably why it's a little different. [07:30] oh. okay. So I can just let it run. I saw my name flash by in the stats so it seems to be working. [07:30] Is there another project to run on a Pi or is this the only one you tried? [07:31] So... recap: whenever you want to run the URLTeam code, just `cd` to the directory and type `~/.local/bin/run-pipeline pipeline.py --concurrent 2 YOURNICKHERE` and just let it run. [07:31] Well... this is the only one I tried so far. [07:32] Yeah. I am going to set it up at my office tomorrow and let it continually run. [07:32] Cool! Once again, thank you for your help! [07:32] You could also set up the ArchiveBot which can archive thousands of pages, but if you do, your IP might get accidentally banned on some sites. [07:33] That's why I haven't done that. [07:33] Does that work on a PI? I didn't see any install. [07:33] They all need VirtualBox or Vmware, which the PI doesn't run. At least not that I know of [07:34] I'm okay with getting banned on some sites. [07:34] I don't think it will run on Pi. You can see what's being archived here: http://dashboard.at.ninjawedding.org/ [07:36] Some jobs lasts for months. They eventually get uploaded to the Internet Archive so people can view websites using the Wayback Machine. [07:37] Very Cool. Do you know what type of machine do I need for an Archivebot? I have a couple old ones sitting around and plenty of HD space [07:38] https://www.archiveteam.org/index.php?title=ArchiveBot#Volunteer_to_run_a_Pipeline [07:39] Is that the same pipeline I am running now? [07:40] So you join the #archivebot channel and ask one of the Ops for setting things up and getting things working. [07:40] Okay cool. Thank You again. Will bounce over there now. [07:40] No, the pipeline you're running is for URLTeam's TerrorOfTinyTown. [07:40] okay got Ya [07:41] *** bacchusm_ has joined #urlteam [07:42] *** qwebirc57 has joined #urlteam [07:43] sorry I click on your link and it exited me [07:43] *** bacchusm_ has quit IRC (Client Quit) [07:43] Who's link? [07:43] *** bacchusm_ has joined #urlteam [07:45] *** bacchusm has quit IRC (Ping timeout: 260 seconds) [07:45] *** bacchusm has joined #urlteam [07:47] *** qwebirc57 has quit IRC (Ping timeout: 260 seconds) [07:48] Sorry Teej, I click on your link and got trasnfer to other IRC channel. I thought another window would open [07:48] *** bacchusm_ has quit IRC (Ping timeout: 260 seconds) [07:56] bacchusm: Oh. That's fine. Don't worry. [07:57] For IRC stuff, I use IRCCloud. It makes things easier. [12:12] *** zerkalo has quit IRC (Ping timeout: 260 seconds) [12:23] *** zerkalo has joined #urlteam [13:01] *** Kagee has quit IRC (WeeChat 2.3) [13:03] *** Kagee has joined #urlteam [21:55] *** coldon2dr has quit IRC (Read error: Operation timed out) [22:43] *** bacchusm has quit IRC (Ping timeout: 260 seconds) [22:55] *** hook54321 has quit IRC (Quit: Connection closed for inactivity)