| Time |
Nickname |
Message |
|
00:15
🔗
|
yipdw| |
heh |
|
00:15
🔗
|
yipdw| |
running ffgrab.rb on AT&T's 3G network is an exercise in omg |
|
00:19
🔗
|
Coderjoe |
eek |
|
01:21
🔗
|
bsmith093 |
/home/ben/.rvm/gems/ruby-1.9.3-p0/gems/girl_friday-0.9.7/lib/girl_friday/actor.rb:212:in `block in initialize' |
|
01:21
🔗
|
bsmith093 |
/home/ben/.rvm/gems/ruby-1.9.3-p0/gems/girl_friday-0.9.7/lib/girl_friday/actor.rb:212:in `initialize' |
|
01:21
🔗
|
bsmith093 |
/home/ben/.rvm/gems/ruby-1.9.3-p0/gems/girl_friday-0.9.7/lib/girl_friday/actor.rb:377:in `watchdog' |
|
01:21
🔗
|
bsmith093 |
/home/ben/.rvm/gems/ruby-1.9.3-p0/gems/girl_friday-0.9.7/lib/girl_friday/actor.rb:66:in `new' |
|
01:21
🔗
|
bsmith093 |
yipdw im getting a whole mess of these/home/ben/.rvm/gems/ruby-1.9.3-p0/gems/girl_friday-0.9.7/lib/girl_friday/actor.rb:69:in `block (2 levels) in spawn' |
|
01:21
🔗
|
bsmith093 |
/home/ben/.rvm/gems/ruby-1.9.3-p0/gems/girl_friday-0.9.7/lib/girl_friday/actor.rb:66:in `block in spawn' |
|
01:21
🔗
|
bsmith093 |
E, [2011-12-09T20:21:18.680575 #18455] ERROR -- : Exception Errno::ECONNREFUSED (Connection refused - Unable to connect to Redis on 127.0.0.1:6379) raised while scraping /cartoon/; requeuing. |
|
01:26
🔗
|
yipdw| |
yeah |
|
01:26
🔗
|
yipdw| |
I said you need a Redis instance |
|
01:31
🔗
|
bsmith093 |
i figured out how to run reedis, now where is it saving these links, to, or isnt it? |
|
01:32
🔗
|
yipdw| |
all data is being saved in the Redis database |
|
01:32
🔗
|
yipdw| |
the list of story IDs is present in the "stories" key |
|
01:32
🔗
|
yipdw| |
all other keys are discovery state |
|
01:32
🔗
|
bsmith093 |
which is where? |
|
01:33
🔗
|
yipdw| |
the stories key is in the Redis database |
|
01:33
🔗
|
yipdw| |
redis-cli is the Redis CLI interface |
|
01:33
🔗
|
yipdw| |
to query, launch it and run e.g. scard stories, smembers stories |
|
01:33
🔗
|
yipdw| |
more info available at http://redis.io |
|
01:34
🔗
|
yipdw| |
also, you should pull again, because I fixed a problem with usage of the If-Modified-Since header |
|
01:34
🔗
|
bsmith093 |
wow, 5k stories in 45sec thats fast |
|
01:34
🔗
|
yipdw| |
it's not actually pulling story data |
|
01:35
🔗
|
yipdw| |
but, yeah, it's decently quick |
|
01:35
🔗
|
yipdw| |
it can be faster without the random waits, but |
|
01:35
🔗
|
bsmith093 |
i know its just saving valid ids right |
|
01:35
🔗
|
yipdw| |
I feel bad about taking that out |
|
01:35
🔗
|
yipdw| |
if you run it on JRuby 1.6+, it is possible to scale it to a very high number of threads |
|
01:36
🔗
|
bsmith093 |
do i need to run anything else before running ffgrab agian after apull |
|
01:36
🔗
|
yipdw| |
but again, I don't feel good about doing that, because I don't know fanfiction.net's capabilities |
|
01:36
🔗
|
yipdw| |
no, terminate it |
|
01:36
🔗
|
bsmith093 |
i did |
|
01:36
🔗
|
yipdw| |
ok |
|
01:36
🔗
|
yipdw| |
then just re-run it |
|
01:36
🔗
|
bsmith093 |
and i git pull, and now im running again |
|
01:36
🔗
|
yipdw| |
the scraper will pick up from where it left off |
|
01:37
🔗
|
yipdw| |
anyway, brb |
|
01:37
🔗
|
yipdw| |
well bbl more like, heh |
|
01:37
🔗
|
bsmith093 |
thanks bye |
|
06:55
🔗
|
Coderjoe |
http://retropc.net/ |
|
06:57
🔗
|
Coderjoe |
the japanese counterpart to SketchCow? |
|
07:01
🔗
|
Coderjoe |
man... why didn't I buy another HR-S9911U or two before they disappeared? |
|
07:38
🔗
|
DFJustin |
the japanese guys are like negative sketchcow, they collect shitloads of stuff and then never digitize any of it ever |
|
07:40
🔗
|
DFJustin |
there are a few exceptions |
|
08:47
🔗
|
Coderjoe |
not sure if there was a link from the main page of the site, so: http://retropc.net/alice/ |
|
09:49
🔗
|
yipdw |
well this is awesome |
|
09:49
🔗
|
yipdw |
9) "/Wrestling_and_CSI_Miami_Crossovers/230/1686/_cache_control" |
|
09:50
🔗
|
chronomex |
woooo |
|
09:50
🔗
|
chronomex |
the best |
|
09:50
🔗
|
Coderjoe |
O_O |
|
09:51
🔗
|
Coderjoe |
fan fiction gets weeeeeird |
|
09:51
🔗
|
yipdw |
47) "/Frasier_and_Megami_Tensei_Crossovers/381/1074/_cache_control" |
|
09:51
🔗
|
Coderjoe |
i wonder how many ff.net stories have self-insertion |
|
09:51
🔗
|
dnova |
fan fiction starts weird, remains weird, and ends weird. |
|
09:51
🔗
|
chronomex |
is _cache_control some sort of strange fanfic thing? |
|
09:52
🔗
|
yipdw |
no, that's my Cache-Control observance mechanism |
|
09:52
🔗
|
chronomex |
:P |
|
09:52
🔗
|
yipdw |
it's kind of hacky |
|
09:52
🔗
|
yipdw |
but it works |
|
09:54
🔗
|
Schbirid |
argh, what is the linux tool to compare two textfiles where you can specify eg "only show entries that appear in a but not b". not diff, something simpler |
|
09:55
🔗
|
chronomex |
grep -v -f b a |
|
09:56
🔗
|
chronomex |
works best when B is small and A is large |
|
09:56
🔗
|
Schbirid |
it was something with the flags -1 , -2, -3 |
|
09:56
🔗
|
Schbirid |
but that sounds good too :) |
|
09:56
🔗
|
chronomex |
it's not quite there, but that's where I would start. |
|
09:57
🔗
|
chronomex |
you also want to tell grep to match whole lines only (dunno the option), and interpret the lines as fixed strings rather than patterns (-F, I think) |
|
10:00
🔗
|
Schbirid |
chronomex: comm it is :) |
|
10:00
🔗
|
chronomex |
ah |
|
10:08
🔗
|
DFJustin |
ah this alice soft archive is nice |
|
13:36
🔗
|
Schbirid |
http://dtrace.org/blogs/brendan/2011/12/08/2000x-performance-win/ |
|
21:05
🔗
|
chronomex |
Schbirid: I bet it would have really flown with fgrep. |
|
21:58
🔗
|
bsmith093 |
gsc-game.com wget-warc appears to be done, 1.5gb total, including cdx and warc fle |
|
21:58
🔗
|
bsmith093 |
44.185 items, at 1.8gb |
|
23:14
🔗
|
SketchCow |
Yes, once again, archiveteam's site has spam on it. |
|
23:18
🔗
|
SketchCow |
You know, for being all twitchy about the fact I've been porting their digitized magazines to archive.org, this site has done a spectacularly shitty job getting consistent spams. |
|
23:18
🔗
|
SketchCow |
scans. |
|
23:19
🔗
|
SketchCow |
Some of these are literally mish-mashes where issues 1 2 3 8 and 9 are collections of JPG files, and then 4 5 6 are pdfs and then 7 is a set of jpg files in two file directories. |
|
23:47
🔗
|
underscor |
SketchCow: hah |
|
23:58
🔗
|
chronomex |
scanning is boring, let's get high first |