[00:08] *** dashcloud has quit IRC (Ping timeout: 260 seconds) [00:08] *** j08nY has quit IRC (Quit: Leaving) [00:10] *** dashcloud has joined #archiveteam-bs [00:12] Lord_Nigh: for what it's worth, I use an SVHS vcr and an AVT-8710 (which has TBC) for VHS capture [00:13] Nobody complains about anyone [00:16] I figured SketchCow, we are using a nginx proxy to reduce connections to the API [00:16] since we where maxing it out [00:29] *** dashcloud has quit IRC (Quit: No Ping reply in 180 seconds.) [00:29] *** dashcloud has joined #archiveteam-bs [00:52] *** BlueMaxim has joined #archiveteam-bs [00:57] are there any projects worth going for at the moment other than urlteam? [00:57] sorry, haven't been involved lately. [01:00] *** dashcloud has quit IRC (Read error: Operation timed out) [01:07] *** dashcloud has joined #archiveteam-bs [01:08] 2/6 [01:09] *** bitBaron has quit IRC (Quit: My computer has gone to sleep. ZZZzzz…) [01:09] *** kristian_ has quit IRC (Quit: Leaving) [01:10] nyany: NewsGrabber [01:11] https://twitter.com/_jared/status/878036763805048832 Basho probably dead [01:13] http://basho.com/ [01:13] check out the link near the bottom middle that says Basho’s Riak® is a distributed NoSQL database that delivers: [01:13] its Broken [01:13] how in the hell do you have a professional website with a broken link [01:15] Their tech blog has some good posts, e.g. http://basho.com/posts/technical/vector-clocks-revisited-part-2-dotted-version-vectors/ [01:17] I know someone who used to work there. [01:21] *** Stilett0 has quit IRC (Ping timeout: 268 seconds) [01:30] jrwr: got it [01:41] hey SketchCow [01:41] i'm closer to been done with my tape captures [01:41] i got the 2 full selfs of tapes gone [01:42] i'm just down to a cupboard [01:42] of tapes [01:42] Audio/Video/Data [01:46] *** Soni has joined #archiveteam-bs [01:46] If your system is stationary a better solution might be a cantenna or equivalent [01:46] Ya [01:46] Most APs are about 100-200mW [01:46] that Mikrotik is rated for outdoor use :) [01:46] why not add grounded tinfoil padding to my house and a cell booster? [01:47] :) [01:47] Its what I use, and have been so far impressed as hell by it [02:04] *** Aranje has joined #archiveteam-bs [02:13] *** dashcloud has quit IRC (Read error: Operation timed out) [02:15] Isn't a 1000mW AP a good way to have the FCC / CRTC knocking at your door? [02:16] *** Asparagir has joined #archiveteam-bs [02:16] not if you have an amateur radio license and are using the right frequency :) [02:17] (part of one of the ham bands intersects wifi :p) [02:19] Crusher: https://routerboard.com/RB951Ui-2HnD [02:23] Hue [02:23] *** zenguy has quit IRC (Read error: Operation timed out) [02:23] *** Stilett0 has joined #archiveteam-bs [02:23] That's naughty :P [02:24] Hehehe [02:24] And I can set it to non-us channels [02:24] CH18 HERE I COME [02:25] *** zenguy has joined #archiveteam-bs [02:26] Uh oh... Why did my server stop contributing to imzy... [02:26] That's not good [02:30] I use this little guy Crusher mounted to a pith helmet https://routerboard.com/RBmAP2nD [02:30] https://jrwr.io/doku.php?id=projects:hatnas [02:32] *** dashcloud has joined #archiveteam-bs [02:33] that is neat [02:34] Combine that with the guy who made the solar charging hat Kickstarter [02:34] And the runtime becomes forever :P [02:35] Right now its 29 Hours [02:35] (yes, he used LiFePo4 cells) [02:35] with that battery pack [02:35] That one is using LiFe cells right? [02:35] Not straight Lithium ion? [02:36] Li-ion [02:36] its a 27k mWh Battery [02:36] fits in my pocket very well [02:36] Oh ok [02:37] Pocket mounted battery [02:37] Long 6ft cables [02:37] I'm not putting a lion battery on my head xD [02:37] yay for docker [02:37] never used it before but it seems to have worked [02:37] Brb, switching to WiFi [02:37] Crusher: its a Public File Sharing system [02:37] *** Crusher has quit IRC (Bye) [02:38] *** Crusher has joined #archiveteam-bs [02:40] So its a public file sharing system [02:40] I take it to Conventions and things [02:40] and wear it around walmart sometimes [02:40] hehe [02:40] it has a PHP based interface [02:40] a little menu on the pi it self to do admin things [02:41] (uses 2FA OTC to unlock) [02:41] and is pretty secure, even has SSL + HTTP/2 [02:41] I can hear my wifi and it's annoying [02:41] You can hear 2.4Ghz?! [02:41] can you hear radio stations [02:41] that'd be super cool [02:41] my speakers kinda pick up wifi and basically any radio signals [02:42] Oh [02:42] so I hear annoying noise [02:42] You need a magnet puck [02:42] but that's how I know it's working [02:42] need ferrites [02:42] I need a faraday cage tbh [02:42] or uh [02:42] shielded cables [02:44] Anyway, I call it HatNAS [02:44] good name [02:47] I figured [02:47] Im kind of broke, or would of gotten more drives for it [02:48] Need a way to hack a voltagereg on the pi so I can talk USB Charger and get more volts [02:48] if I can get 14 @ 2A (what QC2 is rated for) [02:48] and down convert to 5v 1.5A Stable [02:48] that would be legit [02:51] jrwr: you're going to defcon? [02:51] Yes [02:51] * robogoat is jealous. [02:51] Im going to go super broke doing it [02:51] Bunch of buddies are headed out for that week. [02:51] Whole paycheck [02:51] *** Crusher has quit IRC (Read error: Connection reset by peer) [02:51] *** Crusher has joined #archiveteam-bs [02:52] *** Crusher has quit IRC (Client Quit) [02:52] I hate vegas, but seeing some folks that have come distances would be fun. [02:52] I am going to a CTF this weekend [02:52] my first one [02:53] Cool, have fun! [02:53] Im with a local CitySec style group [02:53] so I have backup, we meetup once a month, damn good fellows [02:54] Nice, yeah, [02:54] there is a local defcon group and 2600 meetup in my area. [02:54] And I host a sparsely attended cryptoparty. [02:55] *** Crusher has joined #archiveteam-bs [02:55] We have about 10 guys show up in a very nice tavern and talk shit for about 8 hours [02:55] Nice. [02:56] We hang out in a slack all day as well, about 200 of us [02:56] Whoa, that's quite a crew. [02:56] There's no local infosec slack here AFAIK. [02:56] We have a general technology slack people hang out in, though. [03:01] Ya, Ours is called STLSec [03:04] Hey, imzy is really speeding up [03:04] Half way done [03:21] *** th1x has joined #archiveteam-bs [03:28] aw shit *plugs holes* [03:29] Mundus, stop making us look bad! [03:29] :P [03:30] holes plugged [03:30] idk why I operate my firewall in blacklist mode .-. [03:58] *** qw3rty2 has joined #archiveteam-bs [04:04] *** qw3rty has quit IRC (Read error: Operation timed out) [04:39] *** Crusher_ has joined #archiveteam-bs [04:39] *** Crusher has quit IRC (Read error: Connection reset by peer) [04:40] *** Sk1d has quit IRC (Ping timeout: 194 seconds) [04:46] *** Sk1d has joined #archiveteam-bs [05:12] *** Crusher has joined #archiveteam-bs [05:12] *** Crusher_ has quit IRC (Read error: Connection reset by peer) [05:33] Is there any reason for the checkip() function to not be run only once when the warrior starts? [05:34] Currently it seems to run during every thread, which seems inefficient given that "am I behind a proxy / firewall" is not going to change between threads [05:34] *** pizzaiolo has quit IRC (Quit: pizzaiolo) [05:38] I'm sure the sites used as examples would appreciate not having our packets hit them every time we try to scrape a file [05:54] *** Famicoman has quit IRC (Ping timeout: 260 seconds) [06:01] *** Famicoman has joined #archiveteam-bs [06:29] *** j08nY has joined #archiveteam-bs [06:49] Crusher: The IP check only looks up the IPs for those hosts, it doesn't actually contact them (as far as I know). [06:50] It's really just "Do these hosts resolve to different IPs?", not a thorough check whether those pages return specific content or something like that. [07:11] *** th1x_ has joined #archiveteam-bs [07:16] *** th1x has quit IRC (Read error: Operation timed out) [07:16] *** fie has quit IRC (Ping timeout: 600 seconds) [07:42] *** fie has joined #archiveteam-bs [08:06] *** bsmith093 has quit IRC (Ping timeout: 250 seconds) [08:08] *** j08nY has quit IRC (Quit: Leaving) [08:11] *** SilSte has quit IRC (hub.efnet.us hub.dk) [08:11] *** dan- has quit IRC (hub.efnet.us hub.dk) [08:11] *** robinak has quit IRC (hub.efnet.us hub.dk) [08:11] *** eprillios has quit IRC (hub.efnet.us hub.dk) [08:11] *** Lord_Nigh has quit IRC (hub.efnet.us hub.dk) [08:11] *** ReimuHaku has quit IRC (hub.efnet.us hub.dk) [08:11] *** RedType has quit IRC (hub.efnet.us hub.dk) [08:11] *** pikhq has quit IRC (hub.efnet.us hub.dk) [08:11] *** mgrytbak has quit IRC (hub.efnet.us hub.dk) [08:11] *** acridAxid has quit IRC (hub.efnet.us hub.dk) [08:11] *** joepie91 has quit IRC (hub.efnet.us hub.dk) [08:11] *** cf has quit IRC (hub.efnet.us hub.dk) [08:11] *** chfoo has quit IRC (hub.efnet.us hub.dk) [08:11] *** tapedrive has quit IRC (hub.efnet.us hub.dk) [08:11] *** underscor has quit IRC (hub.efnet.us hub.dk) [08:11] *** espes__ has quit IRC (hub.efnet.us hub.dk) [08:11] *** kvieta has quit IRC (hub.efnet.us hub.dk) [08:11] *** klg has quit IRC (hub.efnet.us hub.dk) [08:17] *** schbirid has joined #archiveteam-bs [08:21] *** bsmith093 has joined #archiveteam-bs [08:29] *** bsmith093 has quit IRC (Ping timeout: 250 seconds) [08:36] *** bsmith093 has joined #archiveteam-bs [08:41] *** BlueMaxim has quit IRC (Quit: Leaving) [08:47] *** Lord_Nigh has joined #archiveteam-bs [08:57] *** werethles has joined #archiveteam-bs [08:57] So. [08:57] I ran the "pip install --upgrade seesaw" But what directory do I cd to? [08:58] The repository. I.e. git clone https://github.com/ArchiveTeam/imzy-grab.git && cd imzy-grab [09:00] You'll probably run into issues when you run get-wget-lua.sh though. Coincidentally, I just helped someone with that yesterday. See here: http://archive.fart.website/bin/irclogger_log/archiveteam-bs?date=2017-06-22,Thu&sel=347#l343 [09:01] (Obviously, you can ignore the OpenSSL parts if you use GnuTLS.) [09:04] So I can't have the script auto choose a project? [09:05] No, that's not possible. [09:05] The scripts will also not update automatically. If you're getting an "out of date" error, you'll have to stop them, run git pull, and restart. [09:05] Ohhhh [09:05] Okay. [09:06] The warrior is more of a "set up and forget"-type of thing, while the scripts offer more control. [09:06] That seems super easy to automate with chron though [09:06] Yeah. [09:06] Yeah, it can certainly be done. [09:06] All righty. I'll just have chron auto re-pull every 24hrs or so [09:07] Thanks a bunch! [09:07] You'll want something like `touch STOP && wait && git pull && run-pipeline pipeline.py ...`. [09:08] `touch STOP` initiates the stop, but the scripts will finish the current items first, so you need to wait for the process. Then update and restart. [09:09] Good call! [09:09] Or you could set something up which first checks for updates and only runs this if there are changes. [09:09] Something something git fetch && git diff master origin/master [09:20] *** SHODAN_UI has joined #archiveteam-bs [09:24] *** werethles has quit IRC (Quit: Page closed) [09:42] *** thuban4 has joined #archiveteam-bs [09:47] *** zenguy has quit IRC (Ping timeout: 245 seconds) [09:48] *** thuban3 has quit IRC (Read error: Operation timed out) [09:52] *** zenguy has joined #archiveteam-bs [09:52] *** sun_shine has joined #archiveteam-bs [09:52] Hi. There's a long-running "ex-mormon" forum that will close July located lifeaftermormonism.net . Can someone help point archivebot at it? [10:00] *** dan- has joined #archiveteam-bs [10:35] *** SHODAN_UI has quit IRC (Remote host closed the connection) [10:35] The countries around Qatar have demanded (again, I believe) that Al-Jazeera be closed in 10 days: http://www.aljazeera.com/news/2017/06/arab-states-issue-list-demands-qatar-crisis-170623022133024.html [10:38] There was an ArchiveBot job a bit over two weeks ago. :-) [10:40] ... but only for the English version. I'll queue the others. [11:00] *** C4K3 has quit IRC (Read error: Connection reset by peer) [11:01] *** C4K3 has joined #archiveteam-bs [11:05] *** icedice has joined #archiveteam-bs [11:22] *** icedice has quit IRC (Ping timeout: 268 seconds) [11:38] *** icedice has joined #archiveteam-bs [11:42] TIL: the referrer matters in wpull. If you recursively grab https://example.com/a and /b, and /a contains a link to (or in another way adds) /b/foo to the queue, /b/foo will not be retrieved, but if it's added from /b, it will. [11:45] good to know, thanks [11:46] I'm not sure what happens if both /a and /b link to /b/foo. I assume that whichever discovers the link first will win. [11:52] *** kvieta has joined #archiveteam-bs [11:52] *** espes__ has joined #archiveteam-bs [11:52] *** tapedrive has joined #archiveteam-bs [11:52] *** chfoo has joined #archiveteam-bs [11:52] *** cf has joined #archiveteam-bs [11:52] *** joepie91 has joined #archiveteam-bs [11:52] *** acridAxid has joined #archiveteam-bs [11:52] *** mgrytbak has joined #archiveteam-bs [11:52] *** pikhq has joined #archiveteam-bs [11:52] *** RedType has joined #archiveteam-bs [11:52] *** klg has joined #archiveteam-bs [11:52] *** underscor has joined #archiveteam-bs [11:52] *** ReimuHaku has joined #archiveteam-bs [11:52] *** eprillios has joined #archiveteam-bs [11:52] *** robinak has joined #archiveteam-bs [11:52] *** SilSte has joined #archiveteam-bs [11:52] *** hub.dk sets mode: +o underscor [11:52] *** swebb sets mode: +o underscor [11:55] *** pizzaiolo has joined #archiveteam-bs [12:12] *** Crusher_ has joined #archiveteam-bs [12:17] *** Crusher has quit IRC (Read error: Operation timed out) [12:21] Haha, I was wondering why my wpull tests were so slow. Turns out, I had a time.sleep(5) in the get_urls hook. [12:22] *** Jonison has joined #archiveteam-bs [12:22] *** Hiccup has joined #archiveteam-bs [12:22] moved to -bs [12:28] Unless my connection to the tracker died [12:29] It looks like the tracker died [12:29] Imzy is still up [12:29] *** Hiccup has quit IRC (Quit: Page closed) [12:30] Hmm. There's still a few uploads trickling in, but something is borked [12:31] And it's back... That was odd [12:36] Fhew, For a second there it looked like that was the end of the downloads [12:40] *** icedice has quit IRC (Quit: Leaving) [12:41] *** SHODAN_UI has joined #archiveteam-bs [12:42] Crusher_: That's been happening from time to time at least since yesterday evening (UTC). [12:42] Huh. Any idea what's going on? [12:43] A hiccup at imzy or the tracker? [12:43] *** icedice has joined #archiveteam-bs [12:43] Imzy [12:44] Maybe it's some sort of routinely run backup that's pushing it over the edge with our scraping [12:44] I have to say though that I'm a bit surprised they're still online, given how firmly greenie said yesterday that they're probably going to shut down around 00:00 EST (05:00 UTC, 7h 45m ago). [12:46] So for anyone actually in that region, 01:00 ;P [12:47] But yeah, that is a welcome time extension [12:47] Right. (He did say EST though.) [12:47] Just a few more hours is all we need [12:47] Well, we'll have to requeue lots of items as well. [12:47] But yeah, I won't complain. :-) [12:48] how many items are out right now? [12:48] Oh wow [12:48] That's bad xD [12:48] Make that more than a few hours [12:48] I wonder how many of those are mundus's. ;-) [12:49] To be fair, I think a good number of them are failing [12:49] Right before I left, I noticed my logs were returning a good number a dead ones [12:50] Yeah, I know. [12:50] *** REiN^ has quit IRC (Max SendQ exceeded) [12:50] I see about as many fails as successes on my machines. [12:51] I wonder if they *are* in the middle of shutting down [12:51] *** REiN has joined #archiveteam-bs [12:51] Awfully slow shutdown, but possible [12:52] Or we're overloading their database servers again. [12:52] Eh, the site is still responding snappily [12:53] Much of the site is probably cached. [12:53] True... [12:53] you're busting through all the caches though [12:54] Yeah, obviously [12:54] *** icedice has quit IRC (Read error: Connection reset by peer) [12:54] I was trying to say: the site working fine is not a surprise and doesn't contradict the errors we see on the grab. [12:54] ah, yeah [12:56] So what's the difference between the site and our grabs, are we grabbing the raw posts? [12:56] We're iterating over *all posts*. [12:56] Ah. [12:56] *** Jonison has quit IRC (Read error: Connection reset by peer) [12:57] The site will just retrieve a post once, then serve it from cache (until the cache expires), so refreshing will be very fast. [12:58] Right. Alright, well off to a compsci exam. [12:58] Good luck! [12:58] *** Crusher has joined #archiveteam-bs [13:00] have fun! [13:01] *** user0815 has quit IRC (Ping timeout: 268 seconds) [13:02] *** Crusher_ has quit IRC (Ping timeout: 246 seconds) [13:17] *** godane has quit IRC (Ping timeout: 492 seconds) [13:22] JAA: The shutdown was planned for last night, but it'll happen tonight instead since ArchiveTeam is pretty close. [13:23] Or, well, that's the plan as I hear it. If the DB goes down or something, it probably won't be brought back up. [13:25] What is the "out" number on the tracker? [13:27] Items currently claimed by a worker. This includes items that have failed. Once the todo counter is down to zero, we'll requeue all items which are "out". [13:28] So yeah, we still have quite a bit to do. [13:28] interesting, got it [13:29] *** th1x_ has quit IRC (Leaving) [14:34] *** icedice has joined #archiveteam-bs [15:23] *** j08nY has joined #archiveteam-bs [15:30] *** icedice has quit IRC (Quit: Leaving) [16:02] *** C4K3_ has joined #archiveteam-bs [16:05] *** C4K3 has quit IRC (Read error: Operation timed out) [16:16] *** C4K3 has joined #archiveteam-bs [16:18] *** C4K3_ has quit IRC (Read error: Operation timed out) [16:27] *** bitBaron has joined #archiveteam-bs [16:29] *** bitBaron_ has joined #archiveteam-bs [16:30] *** bitBaron has quit IRC (Read error: Operation timed out) [16:40] Oh god, I'm about to use a Pentium 4 to backup all my files that are stored on my school's network drive [16:40] I should have thought about this one sooner... [16:43] *** kristian_ has joined #archiveteam-bs [16:59] *** bitBaron_ has quit IRC (Read error: Operation timed out) [17:02] *** bitBaron has joined #archiveteam-bs [17:03] Imzy has *at most* 11 hours of life left, but looking at the tracker's graphs, at the fastest rate yet it would still take 15 hours to get everything. [17:03] Are the URLs shuffled before being distributed to warriors? [17:13] What is the port that's taxing their server the most, the sheer number of lookups? Or the actual downloading? [17:13] Part* [17:21] *** ZexaronS has quit IRC (Quit: Leaving) [17:30] *** kristian_ has quit IRC (Quit: Leaving) [17:43] Not entirely sure, but I think it's DB load. [17:43] I wonder if walking the URLs in chronological order would help. :-P [17:57] haha [18:00] (Presumably that would help with cache/disk locality and whatnot.) [18:11] Hypothetical, has anyone considered the option of buying the disks off of companies closing down? [18:12] Obviously wouldn't work for an aquihire [18:24] *** odemg has quit IRC (Quit: Leaving) [18:28] *** Crusher_ has joined #archiveteam-bs [18:31] *** Crusher has quit IRC (Ping timeout: 246 seconds) [18:41] *** ZexaronS has joined #archiveteam-bs [18:54] timmc: Yes, as far as I know, jobs are handed out randomly. And I don't think it's possible to prioritise certain jobs (e.g. do users before posts). [19:33] *** Crusher has joined #archiveteam-bs [19:33] *** Crusher_ has quit IRC (Read error: Connection reset by peer) [19:37] *** Crusher has quit IRC (Read error: Connection reset by peer) [19:46] *** bwn has quit IRC (Max SendQ exceeded) [19:48] *** bwn has joined #archiveteam-bs [20:16] *** kristian_ has joined #archiveteam-bs [20:39] oh god, i made a 30MB torrent file [20:40] at 64mb piece size [20:40] not sure i can improve on that :\ [20:41] Wow, how much data do you have in there? [20:42] ~400GB with 400k files. i guess i should zip some directories to reduce the num of files [20:43] haha [20:43] I know deluge falls over at a couple of thousand files, i guess rtorrent is your only option at that sort of size [20:43] oh it loaded it rtorrent just fine [20:44] *** robinak has quit IRC (Read error: Connection reset by peer) [20:49] yeah, imzy returning 0 now, they hating on us [20:49] i'm guessing it's dead, jim [20:50] *** RedType has left [20:51] nevermind, it's back now, at least returning non-zero status codes on some pages [20:52] *** BartoCH has quit IRC (Ping timeout: 260 seconds) [20:52] *** dashcloud has quit IRC (Read error: Operation timed out) [20:54] site is slow Kalroth [20:54] *** dashcloud has joined #archiveteam-bs [20:55] yeah, i'm guessing they're preparing to flip the switch [20:56] *** robink has joined #archiveteam-bs [20:56] I doubt anything changed since a few hours ago, see the discussion around 13:00 UTC. [20:59] *** BartoCH has joined #archiveteam-bs [20:59] *** dashcloud has quit IRC (Remote host closed the connection) [20:59] {"message": "Uh oh, something went wrong! Sorry about that. We've been alerted and will be on it shortly", "errors": []} [20:59] that's what I'm seeing [20:59] Kalroth: If anything is happening, it's weffey restarting things to bring them *back* to life. :-) [21:00] The site going read-only was, as far as I know, the only preparation for site-death. [21:00] I've got all the hardware I can to give it one good push [21:01] Looks like things are slowly coming back to life [21:01] My understanding is that the main limit right now is the SQL connection or the DB engine or something close to that. [21:01] *** kristian_ has quit IRC (Quit: Leaving) [21:02] timmc: alright, sounds good [21:02] (not that you're having issues, but that it isn't shutting down just yet :) [21:08] Haha, I just learned that Zork made a series of scheduled posts before the site went read-only, so they've been popping up periodically since then. :-D [21:09] they're all about ghosts and goodbyes and whatnot [21:11] i think my new favorite number [21:11] is 500 [21:12] i like 16384 [21:12] are >4GB zip files ok nowadays? [21:13] xmc: Ah, see ya like it large mate\ [21:13] lol [21:19] I prefer 2520 [21:20] schbirid: Generally yeah, unless you need to deal with Windows XP-era systems [21:20] yay [21:21] LOL, apparently Android didn't add native Zip64 support to their Java runtime until 6.0. WTF Google [21:22] GGG [21:22] (Granted, many android phones don't have more than 4 GB of free space on their internal storage...) [21:33] And according to Wikipedia (but without specifying a source), OS X Yosemite can create ZIP64 archives but not extract them, at least through the GUI. [21:34] *** BartoCH has quit IRC (Read error: Connection reset by peer) [21:35] *** BartoCH has joined #archiveteam-bs [21:35] *** Petri152 has quit IRC (Read error: Operation timed out) [21:38] 20:59 < timmc> Kal.roth: If anything is happening, it's weffey restarting things to bring them *back* to life. :-) <- weffey left on her roadtrip about 3 & 1/2 hours ago, I'm not sure will be at a computer for another 4 hours or so. I think those alert emails are actually turned off now [21:41] and yeah the site is super slow right now. if they werent in the process of shutting down, would be measures to take but yeah, are fairly limited options right now. [21:52] greenie: Ah, so it's just the site recovering from hiccups on its own? [21:52] I don't know, I'm sorry. I don't have access anymore to the stuff that alerts me and tells me why-ish its fussy [21:54] *nod* [21:54] sounds like AT is on their own to figure out appropriate load, though [21:56] likely, yeah. [21:57] So I'm about to archive the Tilt API data. Some of that data is also used when you access the website and click on certain things or scroll to the bottom, and it involves pagination. In some cases, it just appends ?page=2 etc. to the URL, in others it also includes the number of results per page (?page=2&rows=50). The former is much easier to set up, obviously. Should I try to go for maximum playback c [21:57] ompatibility or take the easy route? [21:59] *** pie__ has joined #archiveteam-bs [22:00] BS! :D [22:00] anywhay [22:00] * pie__ pokes joepie91 [22:00] pie__: "I think Gog stores the old versions for compatibility reasons but they don't keep the old installers public. You could try asking support if they can provide you a dowload link for whatever version you want to play." [22:00] ref https://www.gog.com/forum/terraria/i_want_to_get_the_old_version_of_terraria [22:01] * pie__ scratches head [22:01] you think theyd give me all 20 previous versions? :P [22:01] lol [22:01] you could ask [22:01] I could, I don't think they would though...wonder if the game devs would oblige? xD [22:03] oh hey https://www.reddit.com/r/CrossCode/comments/36d1fq/anyone_interested_in_the_old_demo_versions/ [22:03] now how to i talk to this guy [22:04] thats two years old, i hope he still has them [22:06] can we archive MC servers? I'd love to find beta 1.7_03 server and stuff [22:06] bukkit or vanilla, but preferably vanilla [22:07] you mean the server program? [22:07] the minecraft_server.jar [22:07] isnt bukkit open source? so you can just get whichever of that [22:07] bukkit is dead, all the files are gone [22:07] no beta 1.7_03 [22:07] on the other hand iirc you can directly download old mc versions from whatever the name of the company was that does minecraft? [22:07] oh shit, really? [22:07] I had it. once. [22:07] but then my HDD died [22:07] damn. [22:08] not server versions, only client versions [22:08] oh. that doesnt make sense though [22:08] because you need the client and server to have compatible versions? [22:09] you can play singleplayer [22:09] im just guessing but the server cores should be the same since the client version can also start servers? not that that helps you. [22:09] not pre-1.3 [22:10] release 1.3 [22:11] http://minecraft.gamepedia.com/1.3.1 [22:11] look for "Made singleplayer internally use a multiplayer server." [22:11] before that, client and server were separate beasts [22:12] wow is it just me or did crosscode get more expensive? good game tho >.> [22:14] *** SHODAN_UI has quit IRC (Remote host closed the connection) [22:14] https://www.theregister.co.uk/2017/06/23/windows_10_leak/ <- i forsee a very very short future for betaarchive now :( [22:14] do we have a mirror of that? [22:14] i mean, not this stuff [22:14] but betaarchive in general [22:16] "not *just* this stuff" FTFY [22:16] *** Petri152 has joined #archiveteam-bs [22:17] But yes, we should definitely grab a copy. [22:17] archiveteam hug of death [22:18] "The leaked code is Microsoft's Shared Source Kit: according to people who have seen its contents, it includes the source to the base Windows 10 hardware drivers plus Redmond's PnP code, its USB and Wi-Fi stacks, its storage drivers, and ARM-specific OneCore kernel code." welllllll thats not what id have thought most interesting but what do i know [22:18] https://www.theregister.co.uk/2017/06/22/two_men_arrested_probe_microsoft_networks_hack/ might be related [22:19] There are a few things on IA, but no "full" mirror as far as I can see, unless it's disguised somehow. [22:19] it might be blacked out [22:19] True [22:20] i would be very interested in that leak but sheesh who has 8tb lying around? >.> [22:24] i know someone named "kazblox"? (not sure of nick) maybe was involved at betaarchive, so if we can get in contact with him maybe we can arrange a mirror [22:24] Lord_Nigh: please do [22:25] wow, whoever it was deleted their github user profile completely [22:25] kazblox at least [22:25] apparently this is a thing: [22:26] https://www.reddit.com/r/Steam/comments/611h5e/guide_how_to_download_older_versions_of_a_game_on/?st=j4af9vw3&sh=8e9d5630 https://steamdb.info/app/368340/history/ [22:27] wait, WHAT? you can DO THAT? [22:29] in which case i may ask a favor of someone trying to do that for me for crosscode because i dont have a steam or anything....its a good game, worth the money :PPPP [22:29] of course that depends on the person [22:29] wow, and you can even get the versions of the gta games before they had to remove the copyrighted music because licenses expired [22:29] i gather this hole will be patched very quickly [22:29] lol [22:29] if it hasn't already [22:32] Anyone here has the windows 10 leaks? [22:32] iirc the first version of owlboy at least on gog had a debugger .pdb file in it (its an xna .net game anyway so you can decompile it without that, but that makes it way easier) [22:32] not sure if the first steam version had it or not [22:32] im not actually sure how to interpret this steamdb stuff [22:33] cant tell how many versions of CC it has [22:33] pie__: it lets you download all old versions of a game assuming you have the valid release id for that version [22:33] kinda like how with picroma's cube world you can grab old versions off the update server if you have a valid GUID for it [22:34] i fed ALL that stuff into archivebot over a year ago though [22:34] at least the guids i could find [22:34] oh thats cute re cube world [22:34] * re: [22:35] assuming valve doesn't patch this this means i can probably delete 400GB+ of 'old versions of steam games' i have archievd locally [22:35] > assuming < [22:35] >> archiveteam << [22:35] >>> delete <<< [22:35] :P [22:36] *** godane has joined #archiveteam-bs [22:36] if you want me to upload all that shit somewhere, gimme a server... actually its probably easier to send a harddrive via snail mail [22:36] or a usb flash drive or 5 [22:37] * timmc attaches a 2 TB spinning rust drive to a pigeon [22:37] yeah if i had $$ lol [22:37] im just joking, im not equipped for any serious librarian stuff :P [22:38] timmc: that won't work, but if you put it in a basket and tie it to about 3 dozen pigeons... [22:38] maybe timmc has russian pigeons [22:38] problem is packet loss from pigeon predation [22:38] I'd build another NAS just to store that juicy windows leak [22:39] maybe a tape drive would be cheaper at that point, not sure [22:39] https://steamdb.info/blog/steam-music-download-soundtracks/ is more exploiting of download_depot steam console command [22:40] this exploit btw doens't let you download games you don't own [22:40] i don't think [22:40] it only lets you download old versions with known ids of games [22:40] I wish mainstream software publishers cared more about providing historic versions [22:40] s/mainstream/consumer/ [22:40] hmm. i wonder if i can get the no man's sky steam prototype, the one from july 2016... [22:41] ... i would try it, but if theres one thing i definitely do not want to do is get my steam account with like $2000 in games banned... [22:41] *** odemg has joined #archiveteam-bs [22:43] *** Crusher_ has joined #archiveteam-bs [22:44] Hope I didn't miss anything good in the last couple hours [22:44] Just a juicy Microsoft leak. [22:44] Which we haven't obtained (yet). [22:44] https://www.theregister.co.uk/2017/06/23/windows_10_leak/ [22:46] Assuming we are grabbing an archive of that, you want another connection on that? [22:47] It's limited access, we can't just yet [22:47] https://steamdb.info/app/275850/ and https://steamdb.info/depot/275851/manifests/ <- the 20170317 version has about 300mb of unknown data that was dropped in the next version [22:48] also the earliest versions of starbound on steam have full debugging symbol information compiled in [22:48] so theres a ton of "cool stuff" to be found using this download_depot trick [22:48] Interestingly enough, the relevant forum post on their forums doesn't list the Shared Source Kit anymore: https://www.betaarchive.com/forum/viewtopic.php?p=422843#p422843 [22:48] damage control [22:49] oh, and the VERY FIRST VERSION of skyrim on steam doesn't even have steam integration DRM compiled in AT ALL [22:49] its drm free completely by accident [22:49] ooooooh [22:49] Lord_Nigh, please gentleman do your stuff [22:49] quietly. [22:50] i'm actually not touching this stuff personally [22:50] since i don't want my personal steam account to get banned [22:50] just pointing stuff out [22:50] oh right. :I [22:50] is there a policy against downloading older things? [22:50] I'm not touching games with a 10 ft pole [22:50] i don't think so? [22:51] soooo is this like "hacking" or something? [22:51] why would you get banned for this [22:51] i think steam actually does it automatically if you have updates disabled, have an old version installed and do the 'verify local files' command [22:51] it probably downloads the depot [22:51] or at least a set of hashes for all the files [22:52] Lord_Nigh, any idea what this dev package thing means?: https://steamdb.info/sub/66910/ (from https://steamdb.info/depot/368341/subs/) [22:52] Interesting discussion thread about that leak: https://www.betaarchive.com/forum/viewtopic.php?f=72&t=37282 [22:52] no idea about dev package [22:52] what i mean to say is i dont really see through the architecture of this at all [22:53] pie__: it probably won't install unless you have a "license" to it on your account, it probably won't download the depot [22:53] no idea how depos and packages and manifests relate [22:53] Lord_Nigh, yeah thats what i figured [22:53] i think when you do a download_depot appid depotid manifestid command it checks on valves servers to see if you've "bought" that product first, and if you did then it lets you download it [22:53] it wouldnt really make sense otherwise [22:54] its possible it will download it anyway and then try to decrypt it and give a "license error" [22:54] is download_depot part of the steam binaries? [22:54] yes [22:54] oh >steam console [22:54] i just cant read [22:54] go to the url field in steam [22:54] and you can launch the console from there [22:55] looping back to the MS leak - it's 32TB or so [22:55] * Kaz pokes odemg [22:56] Kaz, didnt the article say its 8tb compresseD? [22:56] If you can even confirm it exists. :-) [22:56] ah yeah.. [22:57] well, I don't even have 8tb free atm :( [22:58] Kaz: See the links to the Beta Archive forums I posted above. [22:58] Lord_Nigh, to be fair it doesnt look to me like this method is going to get blocked [22:58] er, you can't normally get to the steam url field, but in the system browser open steam://open/console and steam will use that [22:58] pie__: i agree [22:58] Lord_Nigh, also just noticed i didnt rad half of what you said, might have been bad internet lag [22:59] *erad [22:59] *read :I [22:59] this probably won't be blocked, though companies could in theory decide to force-take-down certain manifests [22:59] looks disappointing [23:00] steam has a way of revoking manifests, i.e. i think that's what happened here: https://www.windowscentral.com/forza-horizon-3-developer-pc-build-accidentally-released-public [23:00] now the question is if theres a properly automated download_depot script somewhere that doesnt require screwing around in guis all over the place [23:00] Well, it's still very interesting. But yeah, not "32 TB" interesting. [23:00] oh. this is a thing. http://crosscode.gamepedia.com/Steam_depots [23:01] er, wait, that was an xbox one leak, not steam related at all [23:02] yeah this method seems to have been around for at least two years [23:04] wonder if theres any way to get those protected manifest IDs :/ but obviously there shouldnt be. [23:09] starting to look like el reg dun been trolled? [23:09] yeah, dunno [23:09] the no man's sky experimental branch has a known password: "3xperimental" iirc [23:09] a bunch of other games have known passwords for beta/other branches [23:09] what do you mean? [23:11] "this branch requires a password" [23:11] whatever that means. (shown in steamdb) [23:12] https://boards.4chan.org/g/thread/61048732#p61049660 [23:13] cc Lord_Nigh [23:13] this looks like a panic baletion? [23:13] baleetion* [23:13] I'm guessing that it drew too much attention [23:15] *** bmcginty has quit IRC (Read error: Operation timed out) [23:19] *** bmcginty has joined #archiveteam-bs [23:20] sometimes private beta passwords for steam stuff leak or people figure them out by trial and error: https://steamcommunity.com/app/40800/discussions/0/133261124636030329/ [23:21] where does it ask you for the password? [23:21] right click on super meat boy in your library and go to properties [23:21] go to the 'betas' tab [23:21] oh. [23:22] i am literally just now installing steam (but ive seen it before a bit) [23:28] *** bmcginty has quit IRC (Ping timeout: 268 seconds) [23:29] *** bmcginty has joined #archiveteam-bs [23:30] Lord_Nigh, is it feasible that i write a python script that watches the download dir, copies the stuff out when dl-ing is finished and then i manually enter the next manifest, ad infinitum? :/ [23:31] yes [23:31] it might be possible to do wit with an autohotkey script too [23:31] im on linux [23:31] for moving cursor and selecting fields [23:31] ah [23:32] this is going to be annoying [23:32] well linux has xdotool [23:32] man i dont want to code this right now lol [23:33] alternatively, this appears to be a thing >.> https://github.com/SteamRE/DepotDownloader [23:33] oh except its for windows only :I [23:33] *** schbirid2 has joined #archiveteam-bs [23:33] might run under wine... [23:35] *** schbirid has quit IRC (Read error: Operation timed out) [23:39] inb4 account stolen [23:40] (with nothing on it) [23:41] and of course i hit a bug https://github.com/SteamRE/DepotDownloader/issues/16 [23:42] want a killing floor key? [23:42] humble was giving them out [23:42] and it expires july 10, i already own it [23:42] oh hey it's on linux too [23:42] too bad I don't really play multiplayer games [23:43] pie__: want it? [23:44] i probably dont have the specs lmao [23:44] * pie__ checks YT vid [23:45] ok fine i might have specs fro half-life level stuff [23:45] *** octarine has joined #archiveteam-bs [23:45] is it Source Engine? [23:45] no clue [23:45] im sooo bad at FPSs xD [23:45] better to have someone use the free key than have it expire by itself in 2 weeks [23:46] this seems like a mildly terrible game haha [23:46] *** odemg has quit IRC (Read error: Operation timed out) [23:50] eh, thanks but i think ill pass. [23:54] if you didn't use the key, can i give it to someone else? [23:55] yeah [23:55] appreciate the gesture though :P [23:56] Frogging: do you want it? [23:56] Nah, not my kind of game. but thans [23:56] thanks [23:56] any idea where steam saves this stuff [23:57] someone in #steamlug on freenode might but most probably have it already :p [23:57] on windows its in program files x86/steam/steamapps/etc [23:59] oh hey steam accepts bitcoin