#archiveteam 2015-09-21,Mon

↑back Search

Time Nickname Message
00:01 🔗 zenguy_pc has joined #archiveteam
00:05 🔗 zenguy_pc has quit IRC (Read error: Connection reset by peer)
00:15 🔗 segundo2 has joined #archiveteam
00:16 🔗 primus104 has quit IRC (Leaving.)
00:20 🔗 segundo2 has quit IRC (Ping timeout: 240 seconds)
00:22 🔗 zenguy_pc has joined #archiveteam
00:25 🔗 rduser has quit IRC (Read error: Operation timed out)
00:32 🔗 rduser has joined #archiveteam
00:38 🔗 Wyatts has quit IRC (Remote host closed the connection)
00:39 🔗 Wyatts has joined #archiveteam
00:41 🔗 BlueMaxim has joined #archiveteam
00:49 🔗 superkuh_ is now known as superkuh
01:11 🔗 Fusl has quit IRC (Ping timeout: 240 seconds)
01:20 🔗 Aranje has quit IRC (Quit: Three sheets to the wind)
01:32 🔗 aaaaaaaaa has quit IRC (Leaving)
01:51 🔗 dashcloud has quit IRC (Read error: Connection reset by peer)
01:52 🔗 dashcloud has joined #archiveteam
02:26 🔗 SketchCow boop
02:26 🔗 SketchCow lotta questions
02:28 🔗 SketchCow hi, that musictube item is spam bullshit
02:29 🔗 SketchCow Don't upload spam, problwm solved.
02:31 🔗 godane so i mistakenly deleted 1995-12-01 to 1995-12-16 of mbc newsdesk before uploading
02:32 🔗 godane i'm redownloading them now
02:32 🔗 godane it will take over a hour
03:44 🔗 Froggypwn has quit IRC (Ping timeout: 252 seconds)
03:45 🔗 Froggypwn has joined #archiveteam
04:21 🔗 JesseW has joined #archiveteam
04:28 🔗 chfoo0 has joined #archiveteam
04:30 🔗 chfoo has quit IRC (Ping timeout: 306 seconds)
04:31 🔗 zenguy_pc has quit IRC (Read error: Connection reset by peer)
04:41 🔗 chfoo0 is now known as chfoo
04:55 🔗 Fletcher has quit IRC (Ping timeout: 252 seconds)
04:56 🔗 diacope has quit IRC (Ping timeout: 252 seconds)
05:00 🔗 scyther has joined #archiveteam
05:07 🔗 zenguy_pc has joined #archiveteam
05:08 🔗 zenguy_pc has quit IRC (Read error: Connection reset by peer)
05:13 🔗 scyther has quit IRC (Quit: Leaving)
05:14 🔗 diacope has joined #archiveteam
05:25 🔗 zenguy_pc has joined #archiveteam
05:27 🔗 Fletcher has joined #archiveteam
06:09 🔗 Fusl has joined #archiveteam
06:29 🔗 Fusl has quit IRC (Contact: http://hallowe.lt/)
06:30 🔗 JesseW has quit IRC (Read error: Operation timed out)
06:30 🔗 Fusl has joined #archiveteam
06:31 🔗 phuzion has quit IRC (Read error: Operation timed out)
06:40 🔗 phuzion has joined #archiveteam
07:16 🔗 JesseW has joined #archiveteam
07:31 🔗 schbirid has joined #archiveteam
08:13 🔗 atomotic has joined #archiveteam
08:13 🔗 primus104 has joined #archiveteam
08:14 🔗 JesseW has quit IRC (Read error: Operation timed out)
08:45 🔗 primus104 has quit IRC (Leaving.)
08:53 🔗 russss__ has quit IRC (Remote host closed the connection)
08:53 🔗 JSharp has quit IRC (Remote host closed the connection)
08:53 🔗 zyphlar has quit IRC (Remote host closed the connection)
08:53 🔗 Boltsie has quit IRC (Remote host closed the connection)
08:56 🔗 arkiver2 has joined #archiveteam
08:57 🔗 russss__ has joined #archiveteam
09:01 🔗 JSharp has joined #archiveteam
09:05 🔗 arkiver2 has quit IRC (Ping timeout: 252 seconds)
09:08 🔗 arkiver2 has joined #archiveteam
09:14 🔗 zyphlar has joined #archiveteam
09:48 🔗 Boltsie has joined #archiveteam
10:29 🔗 primus104 has joined #archiveteam
10:33 🔗 RedType has quit IRC (Ping timeout: 483 seconds)
10:42 🔗 arkiver2 Blingee has started!
10:53 🔗 arkiver2 has quit IRC (Ping timeout: 252 seconds)
10:54 🔗 atomotic has quit IRC (Quit: My Mac has gone to sleep. ZZZzzz…)
10:59 🔗 primus104 has quit IRC (Leaving.)
11:00 🔗 RedType has joined #archiveteam
11:01 🔗 arkiver2 has joined #archiveteam
11:03 🔗 arkiver2 chfoo: I think you can move project of choice to URLTeam or Blingee
11:56 🔗 PurpleSym has joined #archiveteam
12:00 🔗 scyther has joined #archiveteam
12:13 🔗 no2pencil has joined #archiveteam
12:17 🔗 primus104 has joined #archiveteam
12:28 🔗 zenguy_pc has quit IRC (Read error: Connection reset by peer)
12:44 🔗 atomotic has joined #archiveteam
12:45 🔗 zenguy_pc has joined #archiveteam
12:49 🔗 arkiver2 has quit IRC (Ping timeout: 252 seconds)
13:27 🔗 oli has joined #archiveteam
13:43 🔗 primus104 has quit IRC (Leaving.)
13:49 🔗 kris33 has joined #archiveteam
13:50 🔗 kris33 has quit IRC (Client Quit)
13:52 🔗 RichardG has quit IRC (Ping timeout: 252 seconds)
13:53 🔗 SketchCow Upload to archive.org of Skillfeed continues its long, lonesome journey
13:56 🔗 HCross Out of curiosity, what kinda speeds and file sizes are we talking?
14:16 🔗 SketchCow FOS lives inside the network. Everything's huge and fast.
14:17 🔗 SketchCow Especially with 30-40 simultaneous cconnections aren't ruining its upstream.
14:17 🔗 SketchCow Are you new?
14:17 🔗 SketchCow I feel like we have about six people who are new.
14:23 🔗 HCross Moderatly new. I was here during the Sourceforge thing. It was joepie91 who got me here from LET
14:23 🔗 * joepie91 waves at HCross
14:31 🔗 * HCross waves back
14:40 🔗 scyther has quit IRC (Quit: Leaving)
14:42 🔗 BlueMaxim has quit IRC (Quit: Leaving)
14:48 🔗 SketchCow Well, welcome then.
14:50 🔗 HCross Thank you
14:50 🔗 atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
15:49 🔗 primus104 has joined #archiveteam
15:50 🔗 JesseW has joined #archiveteam
15:57 🔗 lytv has quit IRC (Ping timeout: 240 seconds)
15:57 🔗 lytv has joined #archiveteam
16:07 🔗 xk_id_ has quit IRC (Remote host closed the connection)
16:19 🔗 JesseW has quit IRC (Read error: Operation timed out)
16:27 🔗 primus104 has quit IRC (Leaving.)
16:46 🔗 lbft has quit IRC (Read error: Operation timed out)
16:52 🔗 scyther has joined #archiveteam
16:58 🔗 lbft has joined #archiveteam
17:16 🔗 primus104 has joined #archiveteam
17:36 🔗 Lord_Nigh has quit IRC (ZNC - http://znc.in)
17:40 🔗 RichardG has joined #archiveteam
17:57 🔗 ete has joined #archiveteam
18:04 🔗 atomotic has joined #archiveteam
18:15 🔗 habi has joined #archiveteam
18:15 🔗 habi has left
18:16 🔗 xk_id has joined #archiveteam
18:22 🔗 limebyte has joined #archiveteam
18:22 🔗 nertzy has joined #archiveteam
18:22 🔗 limebyte Hey, gus i would like to help ya with indexing and getting stuff on Linux
18:23 🔗 limebyte how do i do that?
18:30 🔗 joepie91 limebyte: ohai - there's lots of ways to get involved, really (also, did you come from vpsboard/lowendtalk by any chance? I think I recognize your nick...)
18:30 🔗 limebyte booth
18:30 🔗 joepie91 limebyte: are you looking to contribute resources or time/work? (or both?)
18:30 🔗 joepie91 heh, right :P
18:30 🔗 limebyte resources like storage/bandwidth cpu
18:30 🔗 limebyte etc
18:31 🔗 joepie91 limebyte: on a server?
18:31 🔗 limebyte sure
18:31 🔗 limebyte mostly wheezy
18:32 🔗 joepie91 limebyte: alright, so we have the archiveteam warrior, which automatically picks projects and does tasks for them... but you probably don't want to run that on a server, because it requires virtualbox
18:32 🔗 joepie91 limebyte: so there's a few options, generally:
18:32 🔗 joepie91 1) manually running warrior scripts - when a project appears that you want to contribute to, follow the setup instructions to get it running, and it'll go off and do tasks for that project
18:33 🔗 joepie91 2) supplying an archivebot node (archivebot is an on-demand archival IRC bot, basically), but this requires a bunch of RAM (probably 2GB+?) and longer-term guaranteed availability (several months, iirc)
18:34 🔗 joepie91 3) being an rsync target server - this is occasionally needed, not really centrally announced anywhere... they function as collection servers, and occasionally we'll run out of space and need more rsync target servers; this requires a lot of disk space, ability to handle a lot of connections at once, and a lot of bandwidth (eg. for Blip we had it basically bursting a 1gbps connection for an entire weekend)
18:34 🔗 joepie91 option 1 doesn't require any availability - you can just shut it down whenever needed, and it also doesn't require very many resources
18:34 🔗 joepie91 usually only like 5GB of disk _at most_, a bit of RAM, and a bit of bandwidth
18:35 🔗 limebyte 2 and 3
18:35 🔗 limebyte actually i dont know if 10TB on Gbit
18:35 🔗 limebyte and 2x2 TB is enought
18:35 🔗 joepie91 for an archivebot node, if it's going to be available for a while, it certainly should be
18:35 🔗 limebyte thats not the issue
18:35 🔗 joepie91 for an rsync target, it depends on the project (but I don't think we have any projects needing rsync servers right now)
18:36 🔗 joepie91 paging yipdw
18:36 🔗 limebyte okay how do i setup the warrior then?
18:36 🔗 limebyte it needs virtualbox?
18:36 🔗 joepie91 limebyte: the warrior, yes. if you run the scripts manually, then no - it's just a Python script
18:36 🔗 limebyte confused
18:36 🔗 joepie91 limebyte: the warrior is just a Debian image that downloads and runs the scripts for you, basically :P
18:36 🔗 limebyte ah yeah, i am okay with python, should be good on a non root user
18:36 🔗 joepie91 yeah
18:37 🔗 joepie91 limebyte: okay, so for the warrior scripts, just pick any project from the archiveteam site frontpage that you want to contribute to: http://archiveteam.org/index.php?title=Main_Page
18:37 🔗 joepie91 (they may not all be actively running)
18:37 🔗 joepie91 and in the sidebar there should be a repository link
18:37 🔗 joepie91 for each project
18:37 🔗 joepie91 just go there, and follow the README - it has instructions for most major distros
18:37 🔗 joepie91 that will set it up under an unprivileged 'archiveteam' user
18:37 🔗 joepie91 so you can pretty much just copypaste them and enter your own nickname
18:37 🔗 limebyte hmmm
18:38 🔗 limebyte manual setup each project?
18:38 🔗 limebyte nice would be something like automated
18:38 🔗 joepie91 limebyte: yeah - that's what the warrior normally does, but we don't have a non-virtualbox version of that (and it only runs one at a time)
18:38 🔗 joepie91 I'm working on it
18:38 🔗 joepie91 :p
18:38 🔗 limebyte why virtualbox based?
18:38 🔗 joepie91 limebyte: okay, looks like Blingee is still running
18:38 🔗 joepie91 as is urlteam
18:39 🔗 joepie91 limebyte: the idea of the Warrior is to make it pretty much zero-setup for random people
18:39 🔗 joepie91 to run on their desktop, etc
18:39 🔗 joepie91 download image, load into virtualbox, start it, forget about it
18:39 🔗 limebyte but they do have like ppm limits and bw limits?
18:39 🔗 joepie91 Windows doesn't have a very nice filesystem, so a Linux system is needed for it
18:39 🔗 joepie91 how do you mean?
18:39 🔗 limebyte pages per minute
18:39 🔗 limebyte not like its pooling 500 per sec
18:40 🔗 joepie91 that depends on the site - the scripts usually account for that
18:40 🔗 limebyte i guess does need some setup like tor?
18:40 🔗 joepie91 each project has its own scripts
18:40 🔗 limebyte ah
18:40 🔗 joepie91 with its own rate limits, etc
18:40 🔗 joepie91 as an end user, you just need to run it :P
18:40 🔗 joepie91 think of it as a voluntary botnet
18:40 🔗 joepie91 a bit like SETI@Home
18:41 🔗 limebyte joepie91, you getting to deep into it
18:41 🔗 limebyte :D
18:41 🔗 limebyte lemme install python and lets see
18:41 🔗 joepie91 limebyte: have a look at the instructions here: https://github.com/ArchiveTeam/blingee-grab#distribution-specific-setup
18:41 🔗 joepie91 blingee is a currently active project
18:42 🔗 joepie91 limebyte: and if you want to run multiple projects, then you can set up the urlteam project also: https://github.com/ArchiveTeam/terroroftinytown-client-grab#for-debianubuntu
18:42 🔗 HCross limebyte, I have a feeling you may know who I am
18:42 🔗 joepie91 _however_, for the second script, you'd have to change the address in the commands to 127.0.0.2 instead of 127.0.0.1, or they'll conflict
18:42 🔗 Lord_Nigh has joined #archiveteam
18:43 🔗 joepie91 limebyte: and don't forget to set a nickname
18:43 🔗 joepie91 :p
18:43 🔗 limebyte i will run a lot
18:43 🔗 balrog sets mode: +o Lord_Nigh
18:43 🔗 limebyte until HDD or CPU under load
18:43 🔗 joepie91 limebyte: let me know whether you can get everything up and running
18:43 🔗 limebyte Yacy actually burning the shit out the one node
18:43 🔗 joepie91 limebyte: probably won't happen with the warrior projects, at all :P
18:43 🔗 joepie91 they're usually pretty light-weight
18:43 🔗 HCross limebyte, when projects start compressing stuff then you may notice CPU
18:43 🔗 limebyte old HDD
18:43 🔗 limebyte so
18:43 🔗 limebyte yacy is killing it
18:44 🔗 limebyte okay then
18:45 🔗 joepie91 limebyte: after you're done with the warrior script setup, if you can have it available for a few months at least, you'll want to talk to yipdw about setting up an archivebot node - probably best to just PM him directly
18:45 🔗 limebyte okay
18:45 🔗 limebyte he can see the uptime etc?
18:45 🔗 HCross joepie91, I asked him the other day and he said he didnt need more
18:45 🔗 limebyte lol
18:46 🔗 HCross limebyte, you would have to send him an SSH cert
18:46 🔗 joepie91 HCross: huh? we had queue issues recently
18:46 🔗 HCross It was recently I asked
18:46 🔗 limebyte root user or what?
18:46 🔗 joepie91 limebyte: iirc archivebot still uses an SSH tunnel for talking to the master server
18:46 🔗 joepie91 doesn't require access to your server though, afaik
18:49 🔗 limebyte fuck
18:49 🔗 limebyte aaaaaaa
18:49 🔗 limebyte dang
18:53 🔗 joepie91 limebyte: what broke?
18:54 🔗 limebyte should run bro
18:55 🔗 limebyte i can install multiple over the same user ya?
18:56 🔗 limebyte which port does it listen?
18:56 🔗 limebyte on localhost?
18:56 🔗 joepie91 limebyte: I forgot :P but you can track progress on the leaderboard
18:56 🔗 joepie91 limebyte: you can set up multiple, but they will need to listen on different addresses
18:56 🔗 joepie91 limebyte: http://tracker.archiveteam.org/blingee/ <- that's where the progress for blingee goes
18:57 🔗 joepie91 limebyte: also, you may want to join #archiveteam-bs - this channel is usually just for short "OMG SOMETHING IS GOING DOWN" discussions, longer discussions go to -bs
18:57 🔗 limebyte just have one ip
18:57 🔗 limebyte sadly
18:57 🔗 limebyte datashitty
18:57 🔗 joepie91 limebyte: local IP I mean
18:57 🔗 joepie91 127.0.0.x
18:57 🔗 joepie91 the --address flag
18:57 🔗 limebyte why lol?
18:57 🔗 joepie91 ?
18:57 🔗 HCross or you do --port 8080 then --port 8081
18:58 🔗 limebyte easy
18:58 🔗 limebyte ya
18:58 🔗 limebyte change port not ip
18:58 🔗 HCross Ive done that on all the others
18:58 🔗 joepie91 HCross: then i get to explain to people why it's conflicting with their custom daemon on 8089... ;)
18:58 🔗 limebyte --concurent?
18:59 🔗 limebyte does that mean the crawlers?
18:59 🔗 HCross yes, dont do too many
18:59 🔗 joepie91 limebyte: that's how many jobs it'll work on at the same time
18:59 🔗 limebyte hmmm
18:59 🔗 limebyte quad core idle
18:59 🔗 limebyte any suggestions?
18:59 🔗 * joepie91 just runs it at 20 on servers
18:59 🔗 limebyte but i have only 1 external ip maybe get blocked for to much jobs
18:59 🔗 limebyte good
18:59 🔗 joepie91 not sure if blingee does IP throttling
19:00 🔗 limebyte eh
19:00 🔗 limebyte now it dies instant
19:00 🔗 limebyte if i try to restart after i killed the process
19:01 🔗 HCross limebyte, take screen out of the command then you can see why it died
19:01 🔗 joepie91 limebyte: oh - don't just kill the process, please. use `touch STOP` in the repository to let it stop gracefully
19:01 🔗 joepie91 and what HCross said
19:01 🔗 joepie91 (if you kill it, jobs will get stuck)
19:01 🔗 limebyte bad
19:01 🔗 limebyte seems like its broken now
19:02 🔗 limebyte eh
19:02 🔗 limebyte works again
19:03 🔗 limebyte weired
19:03 🔗 limebyte however its up and running now i guess
19:03 🔗 joepie91 limebyte: join #archiveteam-bs? :P
19:03 🔗 joepie91 right
19:03 🔗 limebyte bs = bullshit?
19:03 🔗 joepie91 yeah :P
19:03 🔗 limebyte takes ages
19:04 🔗 limebyte to get up 20 workez
19:04 🔗 limebyte sucks
19:04 🔗 HCross yep, we dont want to DDoS blingee, although I have a feeling that uni isnt liking that archivebot job ive got
19:05 🔗 nertzy has quit IRC (Quit: This computer has gone to sleep)
19:05 🔗 ersi Yeah, please move over to #archiveteam-bs
19:05 🔗 ersi for general conversations and long-running talks that aren't strictly now-archiving-this-project-right-here-things
19:11 🔗 aaaaaaaaa has joined #archiveteam
19:15 🔗 no2pencil What projects can I currently help with?
19:17 🔗 joepie91 no2pencil: blingee and urlteam are currently active
19:17 🔗 joepie91 no2pencil: see -bs :)
19:23 🔗 atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
19:24 🔗 arkiver chfoo: if skillfeed doesn't undo the changes they made to their website we're done
19:24 🔗 arkiver so for now skillfeed can be removed from the json
19:28 🔗 SketchCow Skillfeed uploads on FOS going swimmingly now.
19:34 🔗 arkiver Nice!
19:35 🔗 arkiver blingee has sarted again, if we're going too fast for FOS to keep up I'll slow it down
19:44 🔗 nertzy has joined #archiveteam
19:50 🔗 SketchCow OK.
19:51 🔗 SketchCow I've got blingee sorting and uploading now.
19:56 🔗 nertzy has quit IRC (Quit: This computer has gone to sleep)
20:06 🔗 schbirid has quit IRC (Quit: Leaving)
20:23 🔗 useretail can anyone check why this item was blocked: http://archive.org/details/GamingAnywhere
20:23 🔗 xmc virustotal flagged it apparently
20:24 🔗 xmc https://www.virustotal.com/file/996d221e7ac08550ce8f8c7b5d693699223b611fe6faa36a98fd06811d283da9/analysis/1370911134/
20:25 🔗 useretail is there a way to free-up that id? i'm the uploader
20:25 🔗 xmc email info@archive.org
20:27 🔗 joepie91 useretail: https://www.virustotal.com/en/file/996d221e7ac08550ce8f8c7b5d693699223b611fe6faa36a98fd06811d283da9/analysis/1370911134/
20:28 🔗 joepie91 the scan results
20:28 🔗 useretail fresh results: https://www.virustotal.com/en/file/996d221e7ac08550ce8f8c7b5d693699223b611fe6faa36a98fd06811d283da9/analysis/1442867247/
20:29 🔗 useretail that's ridiculous
20:29 🔗 joepie91 triggering some heuristics it seems
20:31 🔗 useretail i can't upload stuff because smartest av thinks there's a virus
20:31 🔗 useretail great
20:31 🔗 joepie91 useretail: it's uploaded and stored, just not publicly accessible
20:32 🔗 useretail uploading wasn't complete
20:32 🔗 useretail *upload
20:34 🔗 useretail SketchCow: how do you deal with situations like this one?
20:38 🔗 Jon has quit IRC (Read error: Operation timed out)
20:41 🔗 jmtd has joined #archiveteam
20:42 🔗 edsu_ has quit IRC (Read error: Operation timed out)
20:47 🔗 SketchCow What.
20:48 🔗 scyther has quit IRC (Quit: Leaving)
20:50 🔗 useretail SketchCow: uploading a file that is false-positively reported by virustotal
20:51 🔗 edsu has joined #archiveteam
20:53 🔗 SketchCow undarked
20:56 🔗 useretail thanks
20:56 🔗 PurpleSym has quit IRC (Remote host closed the connection)
20:56 🔗 SketchCow -------------------------------------------
20:56 🔗 SketchCow Anybody here familiar with setting up a testing harness so we can do some needed JSMESS work?
20:56 🔗 SketchCow -------------------------------------------
21:14 🔗 nertzy has joined #archiveteam
21:48 🔗 nertzy has quit IRC (Quit: This computer has gone to sleep)
23:00 🔗 balrog http://www.publishersweekly.com/pw/by-topic/digital/content-and-e-books/article/68130-oyster-is-shutting-down-operations.html
23:01 🔗 arkiver Scripts for GameFront are ready, will test abit more tomorrow but we'll start the GameFront project very soon
23:04 🔗 arkiver balrog: will get on it
23:04 🔗 arkiver we can discovery the books through https://www.oysterbooks.com/explore
23:04 🔗 balrog btw are there any ongoing activities for google code?
23:04 🔗 arkiver yes
23:05 🔗 arkiver code was last updated a day ago, need to add support for some more json files and then we're done
23:05 🔗 arkiver or well, ready to start
23:13 🔗 arkiver I'm not sure if we already have a channel for gamefront, but we need one
23:13 🔗 arkiver we also need a wiki page, so if someone would like to create a page for gamefront, please do :)
23:43 🔗 SketchCow useretail: Our security guy does not agree with you that it's false positives.
23:44 🔗 SketchCow So it's darked until further notice.
23:53 🔗 c_b has joined #archiveteam
23:53 🔗 balrog SketchCow: reanalysis of the offending file component of that zip shows a detection rate of 0/56

irclogger-viewer