#archiveteam 2016-04-26,Tue

↑back Search

Time Nickname Message
00:06 πŸ”— arkiver http://tracker.archiveteam.org/gamefrontforums/ is now active!
00:43 πŸ”— BlueMaxim has quit IRC (Read error: Operation timed out)
00:44 πŸ”— WinterFox has joined #archiveteam
00:46 πŸ”— sigkell has quit IRC (Ping timeout: 260 seconds)
00:48 πŸ”— philpem has quit IRC (Ping timeout: 260 seconds)
00:57 πŸ”— sigkell has joined #archiveteam
01:17 πŸ”— JesseW has joined #archiveteam
01:43 πŸ”— Honno has quit IRC (Read error: Operation timed out)
01:53 πŸ”— xmc has quit IRC (Read error: Operation timed out)
01:53 πŸ”— mismatch has joined #archiveteam
01:54 πŸ”— Fletcher has quit IRC (Read error: Operation timed out)
01:54 πŸ”— mismatch_ has quit IRC (Read error: Operation timed out)
01:54 πŸ”— Famicoma1 has quit IRC (Read error: Operation timed out)
01:55 πŸ”— xmc has joined #archiveteam
01:55 πŸ”— swebb sets mode: +o xmc
01:55 πŸ”— robink has quit IRC (Read error: Connection reset by peer)
01:56 πŸ”— Fletcher has joined #archiveteam
01:59 πŸ”— robink has joined #archiveteam
02:03 πŸ”— MrRadar arkiver: In the gamefrontforums grab did you mean to include the geo-IP block check for downloading GameFront files?
02:03 πŸ”— MrRadar I don't think the forums have the same block
02:04 πŸ”— dashcloud has quit IRC (Read error: Operation timed out)
02:08 πŸ”— JesseW has quit IRC (Ping timeout: 370 seconds)
02:13 πŸ”— dashcloud has joined #archiveteam
02:48 πŸ”— Famicoma1 has joined #archiveteam
03:55 πŸ”— JesseW has joined #archiveteam
03:59 πŸ”— JesseW arkiver: since I'm not banned from gamefront, shall I stay with that one, rather than switching over to forums?
04:28 πŸ”— JesseW up to 6 concurrency on gamefront
04:47 πŸ”— Sk1d has quit IRC (Ping timeout: 194 seconds)
04:53 πŸ”— Sk1d has joined #archiveteam
05:24 πŸ”— metalcamp has joined #archiveteam
05:39 πŸ”— Honno has joined #archiveteam
05:43 πŸ”— BlueMaxim has joined #archiveteam
06:21 πŸ”— signius has joined #archiveteam
06:22 πŸ”— mismatch has quit IRC (Ping timeout: 633 seconds)
06:33 πŸ”— metalcamp has quit IRC (Ping timeout: 244 seconds)
07:24 πŸ”— Honno has quit IRC (Read error: Operation timed out)
07:30 πŸ”— schbirid has joined #archiveteam
07:35 πŸ”— morbus_ has joined #archiveteam
07:41 πŸ”— Morbus has quit IRC (Read error: Operation timed out)
07:57 πŸ”— JesseW has quit IRC (Ping timeout: 370 seconds)
08:01 πŸ”— redlob has quit IRC (Read error: Operation timed out)
08:02 πŸ”— redlob has joined #archiveteam
08:09 πŸ”— metalcamp has joined #archiveteam
08:37 πŸ”— Wuked has joined #archiveteam
09:14 πŸ”— Smiley has joined #archiveteam
09:25 πŸ”— atomotic has joined #archiveteam
09:34 πŸ”— metalcamp has quit IRC (Ping timeout: 244 seconds)
09:35 πŸ”— bwn has quit IRC (Ping timeout: 492 seconds)
09:45 πŸ”— metalcamp has joined #archiveteam
09:50 πŸ”— bwn has joined #archiveteam
10:15 πŸ”— metalcamp has quit IRC (Ping timeout: 244 seconds)
10:25 πŸ”— Wuked has quit IRC (My Mac has gone to sleep. ZZZzzz…)
10:35 πŸ”— metalcamp has joined #archiveteam
10:35 πŸ”— Wuked has joined #archiveteam
10:41 πŸ”— Wuked has quit IRC (Quit: My Mac has gone to sleep. ZZZzzz…)
10:53 πŸ”— Wuked has joined #archiveteam
11:07 πŸ”— Wuked has quit IRC (My Mac has gone to sleep. ZZZzzz…)
11:26 πŸ”— Lord_Nigh has quit IRC (Ping timeout: 244 seconds)
11:29 πŸ”— Crocatowa has quit IRC (Read error: Operation timed out)
11:30 πŸ”— Crocatowa has joined #archiveteam
11:35 πŸ”— Medowar has joined #archiveteam
11:37 πŸ”— atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
11:41 πŸ”— vitzli has joined #archiveteam
11:45 πŸ”— Lord_Nigh has joined #archiveteam
11:54 πŸ”— kris33 has joined #archiveteam
12:02 πŸ”— metalcamp has quit IRC (Ping timeout: 244 seconds)
12:25 πŸ”— RichardG has quit IRC (Ping timeout: 260 seconds)
12:25 πŸ”— atomotic has joined #archiveteam
12:32 πŸ”— RichardG has joined #archiveteam
12:45 πŸ”— Honno has joined #archiveteam
12:47 πŸ”— Wuked has joined #archiveteam
12:59 πŸ”— kris33 has quit IRC (Textual IRC Client: www.textualapp.com)
13:06 πŸ”— Honno has quit IRC (Ping timeout: 1208 seconds)
13:23 πŸ”— suggestio has joined #archiveteam
13:26 πŸ”— suggestio Image hosting site run by ThePirateBay crew has been temporarily revived after sudden shutdown in 2014. Old images now accessible again. http://bayimg.com/
13:28 πŸ”— BlueMaxim has quit IRC (Read error: Operation timed out)
13:33 πŸ”— phuzion suggestio: Know anything about how their image URLs are generated?
13:33 πŸ”— suggestio has quit IRC (Ping timeout: 268 seconds)
13:53 πŸ”— VADemon has joined #archiveteam
13:53 πŸ”— scyther has joined #archiveteam
13:53 πŸ”— scyther has quit IRC (Connection closed)
14:26 πŸ”— metalcamp has joined #archiveteam
14:43 πŸ”— joepie91 phuzion: potential vector for mapping stuff out: http://bayimg.com/album/
14:43 πŸ”— joepie91 it seems to try to load everything
14:44 πŸ”— phuzion I can't imagine that it's more than a few TB, think it might be worth trying to archive?
14:44 πŸ”— joepie91 everything lives on image.bayimg.com
14:44 πŸ”— joepie91 seemingly using hashes of files
14:44 πŸ”— joepie91 http://image.bayimg.com/d3099f010b848bd079b53d0c985e409f67914928.jpg
14:44 πŸ”— phuzion gross
14:44 πŸ”— joepie91 the 'view' pages are easier
14:44 πŸ”— joepie91 http://bayimg.com/PaiLPAAgH
14:44 πŸ”— phuzion lol Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 32 bytes) in /var/data/bayimg.com/www/ajax_album.php on line 70
14:44 πŸ”— joepie91 album sample: http://bayimg.com/album/MAANGaaaC
14:45 πŸ”— joepie91 phuzion: yes, I think it tries to list everything
14:45 πŸ”— joepie91 note the ajax_ prefix
14:45 πŸ”— phuzion yeah
14:45 πŸ”— joepie91 I think it's a sort of API as well
14:45 πŸ”— atomotic has quit IRC (Ping timeout: 260 seconds)
14:45 πŸ”— joepie91 might be able to enumerate it with certain params
14:45 πŸ”— joepie91 ah, hold on
14:46 πŸ”— joepie91 http://bayimg.com/album/- also fails
14:47 πŸ”— joepie91 it seems to always fail..
14:48 πŸ”— joepie91 strange...
14:50 πŸ”— joepie91 Google knows a lot of them anyway
14:51 πŸ”— joepie91 interesting
14:51 πŸ”— joepie91 phuzion: the image IDs are NOT randomly generated
14:52 πŸ”— joepie91 phuzion: first two pages of Google: https://gist.github.com/joepie91/ac014769a3446e074d62c2792b9c05b2
14:52 πŸ”— joepie91 entirely too consistent
14:52 πŸ”— joepie91 always a lowercase or uppercase A in the 2nd 6th position
14:53 πŸ”— joepie91 2nd and 6th*
14:53 πŸ”— joepie91 various other apparent patterns
14:53 πŸ”— joepie91 always a lowercase or uppercase A in the 7th posuition to, it seems
14:53 πŸ”— joepie91 position too*
14:54 πŸ”— PurpleSym bing results: http://paste.nerds.io/edixocitog.txt
14:55 πŸ”— joepie91 yeah, very not random
14:55 πŸ”— joepie91 lol
14:58 πŸ”— joepie91 PurpleSym: phuzion: combined and sorted: http://sprunge.us/ceTE
14:58 πŸ”— joepie91 time to find patterns :P
15:04 πŸ”— Wuked has quit IRC (Quit: My Mac has gone to sleep. ZZZzzz…)
15:06 πŸ”— PurpleSym Ids seem to be case-insensitive.
15:06 πŸ”— joepie91 whoa, fucking seriously?
15:07 πŸ”— joepie91 wow
15:07 πŸ”— joepie91 that is great
15:10 πŸ”— joepie91 PurpleSym: updated list: http://sprunge.us/UiPM
15:10 πŸ”— joepie91 seems the 8th position is just a-f?
15:12 πŸ”— PurpleSym Could be a coincidence. The sample size is quite small.
15:12 πŸ”— joepie91 seems unlikely
15:12 πŸ”— joepie91 PurpleSym: can you get a larger list?
15:12 πŸ”— joepie91 via Google or w/e
15:13 πŸ”— PurpleSym I could try the common crawl index.
15:14 πŸ”— joepie91 please do :P
15:15 πŸ”— VADemon What's the status of maxfile.ro? Has anyone more detailed information?
15:16 πŸ”— atomotic has joined #archiveteam
15:16 πŸ”— VADemon PurpleSym: I've grabbed yandex, duckduckgo and bing for maxfile.ro. Turns out 50 of your results were still unique to mine
15:17 πŸ”— PurpleSym Better get them all, VADemon.
15:17 πŸ”— PurpleSym joepie91: Nothing in the Common Crawl Index, as far as I see.
15:17 πŸ”— atomotic has quit IRC (Client Quit)
15:17 πŸ”— joepie91 PurpleSym: try Google then?
15:17 πŸ”— atomotic has joined #archiveteam
15:18 πŸ”— PurpleSym I don’t have scripts for that.
15:18 πŸ”— joepie91 PurpleSym: if you use Chrome, "Link Grabber" is greatly useful for this
15:18 πŸ”— joepie91 lets you ignore internal stuff
15:19 πŸ”— joepie91 and only extract actual search results
15:19 πŸ”— joepie91 makes it a semi-automated process
15:22 πŸ”— PurpleSym Well, I don’t.
15:31 πŸ”— Rotab has joined #archiveteam
15:31 πŸ”— joepie91 PurpleSym: char frequency counts: http://storage2.static.itmages.com/i/16/0426/h_1461684754_5932817_5afad23c2c.png
15:32 πŸ”— Wuked has joined #archiveteam
15:34 πŸ”— joepie91 http://bayimg.com/fajkkaadd and http://bayimg.com/fajkkaaddd are the same image
15:34 πŸ”— joepie91 so it ignores everything beyond 8 chars
15:34 πŸ”— joepie91 also doesn't seem to go beyond P anywhere
15:34 πŸ”— Rotab are you murdering gamefront (filefront?) forums?
15:35 πŸ”— joepie91 so...
15:35 πŸ”— * joepie91 makes permutation calc
15:35 πŸ”— MrRadar Rotab: we did start archiving them last night, so probably
15:35 πŸ”— MrRadar Ping arkiver ^^^^
15:35 πŸ”— joepie91 phuzion: PurpleSym: 6291456 permutations
15:35 πŸ”— joepie91 I think we can pull that off
15:35 πŸ”— joepie91 (for bayimg)
15:36 πŸ”— phuzion 6.2m, that's not bad
15:36 πŸ”— PurpleSym Sure, that’s not too bad.
15:36 πŸ”— Rotab i cant even join in on gamefront, the ip check fails :S
15:37 πŸ”— PurpleSym joepie91: Wrt frequency counts: Could be an increasing 32 bit counter with nibbles shuffled around.
15:37 πŸ”— PurpleSym *shifted
15:37 πŸ”— VADemon Rotab: doesn't seem to be caused by us, the file downloading from their servers still works for me
15:38 πŸ”— MrRadar The issue is not the file hosting, it's the forums
15:38 πŸ”— MrRadar I can access them but it takes about 10 seconds for each page to load
15:38 πŸ”— luckcolor has joined #archiveteam
15:38 πŸ”— luckcolor Hello
15:39 πŸ”— MrRadar Hello
15:39 πŸ”— WinterFox has quit IRC (Remote host closed the connection)
15:39 πŸ”— Rotab yeah, it is very slow
15:39 πŸ”— joepie91 PurpleSym: that goes beyond my capabilities :)
15:39 πŸ”— joepie91 the distribution is a bit odd but I think we can just treat it as randomized
15:39 πŸ”— joepie91 with the given ranges
15:39 πŸ”— Rotab although the forumgrab needlessly checks if you can download files
15:39 πŸ”— joepie91 and have it be Good Enough
15:39 πŸ”— MrRadar Yeah, I mentioned that last night but arkiver is AFK
15:40 πŸ”— PurpleSym Yeah, that should work fine, joepie91.
15:44 πŸ”— luckcolor has quit IRC (Quit: Page closed)
15:48 πŸ”— JesseW has joined #archiveteam
15:49 πŸ”— atomotic has quit IRC (Quit: My Mac has gone to sleep. ZZZzzz…)
15:53 πŸ”— atomotic has joined #archiveteam
15:56 πŸ”— atomotic has quit IRC (Client Quit)
15:58 πŸ”— VADemon - Started grabbing maxfile.ro, with 660 links from search engines -
15:59 πŸ”— PurpleSym joepie91: Three pictures I just uploaded in this order: http://bayimg.com/aAimFAAgH http://bayimg.com/aaiMgaagH http://bayimg.com/aAiMHaagh
16:02 πŸ”— joepie91 incremental? excellent :)
16:03 πŸ”— joepie91 huh
16:03 πŸ”— joepie91 that's odd
16:03 πŸ”— joepie91 those are 9 chars
16:19 πŸ”— JesseW has quit IRC (Ping timeout: 370 seconds)
16:40 πŸ”— arkiver hi
16:40 πŸ”— arkiver I limited filefront
16:40 πŸ”— arkiver What's up with bayimg and maxfile.ro?
16:43 πŸ”— HCross its come back up for a bit
16:44 πŸ”— VADemon arkiver: Started mirroring with 660 links off search engines
16:44 πŸ”— arkiver Ah yeah, it was shutting down
16:44 πŸ”— arkiver Great!
16:44 πŸ”— vitzli has quit IRC (Quit: Leaving)
16:48 πŸ”— joepie91 arkiver: can you write something for bayimg given the above information?
16:48 πŸ”— joepie91 may be a warrior project thing
16:48 πŸ”— arkiver sure
16:48 πŸ”— joepie91 I'm not sure about the 9th char though
16:48 πŸ”— arkiver what is wrong with the website
16:48 πŸ”— arkiver I haven't read it all
16:48 πŸ”— joepie91 it ignores everything after the 8th char but we did get a 9th char
16:48 πŸ”— joepie91 arkiver: https://torrentfreak.com/pirate-bays-image-hosting-site-bayimg-returns-for-a-bit-160425/
16:48 πŸ”— joepie91 "The site will remain online for a week or so. This allows people to secure their files, if needed, but in a few days the site will close its doors again. Apparently, the TPB team prefers to focus exclusively on the torrent site."
16:49 πŸ”— joepie91 that sounds like an invitation ;)
16:51 πŸ”— joepie91 so it seems like they've simply shifted it by an A
16:51 πŸ”— joepie91 for the newer uploads
16:52 πŸ”— joepie91 maybe they ran out of keyspace?
16:52 πŸ”— joepie91 hm, maybe not
16:52 πŸ”— joepie91 wow, nevermind
16:52 πŸ”— joepie91 I'm blind
16:52 πŸ”— joepie91 it has always been 9 chars, not 8
16:52 πŸ”— joepie91 ignore everything I just said
16:52 πŸ”— joepie91 kik
16:53 πŸ”— joepie91 lol*
16:57 πŸ”— arkiver I'll have a look at the site this evening
17:08 πŸ”— Honno has joined #archiveteam
17:12 πŸ”— philpem has joined #archiveteam
17:16 πŸ”— joepie91 arkiver: alright.
17:16 πŸ”— joepie91 arkiver: the essential information:
17:16 πŸ”— joepie91 1) char frequency information: http://storage2.static.itmages.com/i/16/0426/h_1461684754_5932817_5afad23c2c.png
17:16 πŸ”— joepie91 2) everything after 9 chars is ignored (so you can ignore `position 9` there)
17:16 πŸ”— joepie91 3) image IDs are case-insensitive
17:17 πŸ”— joepie91 4) album URLs are linked from the pages of the images that belong to them, so you can discover albums just by scraping the "Album" buttons (eg. http://bayimg.com/PaiLPAAgH )
17:18 πŸ”— joepie91 5) nothing ever goes beyond P in the image IDs
17:19 πŸ”— MrRadar arkiver: For the FileFront Forums did you mean to include the GameFront geo-IP block check in the scripts? I don't think the forums have any geoblocking
17:23 πŸ”— joepie91 arkiver: oh and 6) it's gone in a week :)
17:24 πŸ”— joepie91 okay, so some quick calculation work
17:24 πŸ”— joepie91 6 million permutations and change
17:24 πŸ”— joepie91 in a week's time
17:25 πŸ”— joepie91 say 1 million permutations a day
17:25 πŸ”— joepie91 works out to ~12 requests per second
17:25 πŸ”— joepie91 not sure they're going to be able to handle that
17:25 πŸ”— joepie91 and that's just for the images, not the discovered albums
17:25 πŸ”— joepie91 they're already slow, so chances are they will start complaining at us
18:19 πŸ”— Medowar has quit IRC (Quit: Connection closed for inactivity)
18:27 πŸ”— SketchCow https://archive.org/details/roiocollection
18:37 πŸ”— hictooth has joined #archiveteam
18:42 πŸ”— hictooth has quit IRC (Quit: Bye!)
18:44 πŸ”— Peetz0r has quit IRC (Read error: Operation timed out)
18:48 πŸ”— hictooth has joined #archiveteam
19:03 πŸ”— BartoCH has joined #archiveteam
19:18 πŸ”— SketchCow If gangsta art and music is your thing, you're in luck with https://archive.org/details/@sketch_the_cow?and[]=mediatype%3A%22audio%22&and[]=collection:audio
19:18 πŸ”— SketchCow https://www.flickr.com/photos/textfiles/sets/72157594265759470 is getting all the papers I'm now scanning.
19:19 πŸ”— SketchCow https://www.flickr.com/photos/textfiles/albums/72157663634874672 is getting all CD-ROM faces I'm scanning (ISOs will go up on hard drives mailed to IA)
19:22 πŸ”— SketchCow I'm also describing Negativland items, but that's once every 5-8 hours, that's hardly worth noting.
19:27 πŸ”— bwn has quit IRC (Ping timeout: 246 seconds)
19:30 πŸ”— dashcloud has quit IRC (Read error: Operation timed out)
19:33 πŸ”— dashcloud has joined #archiveteam
19:56 πŸ”— sivoais has quit IRC (Read error: Operation timed out)
20:03 πŸ”— Wuked has quit IRC (Read error: Connection reset by peer)
20:03 πŸ”— bwn has joined #archiveteam
20:07 πŸ”— sivoais has joined #archiveteam
20:07 πŸ”— VADemon joepie91, bayimg: if my bruteforce string generator is correct and we exclude strings which has "a" in positions 2,6,7 then our search space is totalling at 1,048,600 URLs
20:08 πŸ”— Wuked has joined #archiveteam
20:09 πŸ”— VADemon arkiver: basically I've generated items for warrior here: https://github.com/VADemon/bayimg-brute/blob/40f3bfd130e3e405ec83dd874ee8990b9c0bc192/bayimg-portion-list.txt portion;<id>;<starting string>;<ending string>;<endString length>;<total strings in this item>
20:11 πŸ”— VADemon each individual string would be generated on the fly by lua and given to wget-lua, that's how I imagine this to work
20:11 πŸ”— schbirid can anyone recommend a feed reader that stores warc files of each post automatically?
20:13 πŸ”— Wuked has quit IRC (Read error: Connection reset by peer)
20:13 πŸ”— joepie91 VADemon: huh. hold on.
20:14 πŸ”— Wuked has joined #archiveteam
20:14 πŸ”— joepie91 VADemon:
20:14 πŸ”— joepie91 > 16 * 1 * 16 * 16 * 16 * 1 * 1 * 6 * 16
20:14 πŸ”— joepie91 6291456
20:14 πŸ”— joepie91 what am I missing?
20:18 πŸ”— Sanqui has quit IRC (Remote host closed the connection)
20:19 πŸ”— Sanqui has joined #archiveteam
20:31 πŸ”— Medowar has joined #archiveteam
20:37 πŸ”— Wuked_ has joined #archiveteam
20:37 πŸ”— Wuked has quit IRC (Read error: Connection reset by peer)
20:38 πŸ”— Ravenloft has joined #archiveteam
20:46 πŸ”— Wuked_ has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
20:51 πŸ”— schbirid has quit IRC (Quit: Leaving)
20:54 πŸ”— Lord_Nigh has quit IRC (Ping timeout: 250 seconds)
20:54 πŸ”— metalcamp has quit IRC (Ping timeout: 244 seconds)
20:54 πŸ”— Lord_Nigh has joined #archiveteam
21:13 πŸ”— Emcy_ has joined #archiveteam
21:15 πŸ”— Emcy has quit IRC (Ping timeout: 246 seconds)
21:24 πŸ”— Emcy_ has quit IRC (Ping timeout: 246 seconds)
21:25 πŸ”— Emcy has joined #archiveteam
21:37 πŸ”— Emcy has quit IRC (Ping timeout: 370 seconds)
22:25 πŸ”— arkiver bayimg is not case sensitive. It seems to randomly use some case
22:25 πŸ”— Honno has quit IRC (Read error: Operation timed out)
22:29 πŸ”— arkiver we'll get it
22:30 πŸ”— arkiver they also have albums a tags
22:30 πŸ”— arkiver will have to add some discovery for that
22:30 πŸ”— arkiver Who has some rsync space for the discovery part?
22:33 πŸ”— joepie91 arkiver: see above, albums can be derived from images
22:35 πŸ”— arkiver Yeah, http://bayimg.com/cAIMfAaGH the album button
22:52 πŸ”— arkiver will be 458752 items, 16 images/item
23:20 πŸ”— JW_work has quit IRC (Read error: Operation timed out)
23:29 πŸ”— JW_work has joined #archiveteam
23:29 πŸ”— joepie91 ack
23:31 πŸ”— dashcloud has quit IRC (Read error: Operation timed out)
23:43 πŸ”— Ymgve__ has joined #archiveteam
23:46 πŸ”— Ymgve has quit IRC (Ping timeout: 506 seconds)
23:47 πŸ”— dashcloud has joined #archiveteam
23:50 πŸ”— Ravenloft has quit IRC (Ping timeout: 260 seconds)
23:52 πŸ”— Rye has quit IRC (Ping timeout: 244 seconds)
23:52 πŸ”— Ravenloft has joined #archiveteam
23:58 πŸ”— Rye has joined #archiveteam

irclogger-viewer