[00:01] *** Stiletto has quit IRC (Read error: Operation timed out) [00:04] *** Stilett0 has quit IRC (Ping timeout: 246 seconds) [00:08] *** Stilett0 has joined #archiveteam-bs [00:22] *** Mateon1 has quit IRC (Remote host closed the connection) [00:22] *** Mateon1 has joined #archiveteam-bs [00:23] *** drumstick has quit IRC (Read error: Operation timed out) [00:29] *** refeed has joined #archiveteam-bs [00:32] so i found out there was a guy that was scanning videomaker magazine [00:32] turned out he was not scanning all packages [00:32] *all pages [00:33] https://archive.org/details/@mortar [00:42] *** refeed has quit IRC (Ping timeout: 260 seconds) [00:54] *** BlueMaxim has joined #archiveteam-bs [01:23] *** Stilett0 has quit IRC (Ping timeout: 245 seconds) [01:35] *** drumstick has joined #archiveteam-bs [01:49] *** Dimtree has joined #archiveteam-bs [01:50] so looks like i uploaded 501 items today [01:51] nevermind that was 2017-09-06 [01:51] for 2017-09-07 i uploaded 534 items [01:51] :P [02:21] *** drumstick has quit IRC (Read error: Operation timed out) [02:31] *** drumstick has joined #archiveteam-bs [03:01] *** Stilett0 has joined #archiveteam-bs [03:18] *** antomatic has quit IRC (Read error: Operation timed out) [03:18] *** Stilett0 is now known as Stiletto [03:21] *** antomatic has joined #archiveteam-bs [03:21] *** swebb sets mode: +o antomatic [04:55] *** Sk1d has quit IRC (Ping timeout: 194 seconds) [05:01] *** Sk1d has joined #archiveteam-bs [05:11] If someone has time for a technical question [05:11] yipdw set up warctozip.archive.org, after underscor left. [05:12] It's giving me 503s, seems not to respond to port 8083 [05:12] I don't even know where to begin [05:16] Solved it [05:54] gj SketchCow [05:55] I was going to suggest "history | grep start" and then check status for services previously started [06:32] *** Mateon1 has quit IRC (Ping timeout: 255 seconds) [06:32] *** Mateon1 has joined #archiveteam-bs [06:40] *** what_the_ has joined #archiveteam-bs [07:33] *** pikhq has quit IRC (Read error: Operation timed out) [08:04] Good morning, [08:04] Does anyone here run the warrior on proxmox? [09:00] *** pikhq has joined #archiveteam-bs [10:04] I do want to involve me more in this project, I have some storage that can be utilized and also some hardware for it. [10:05] I also works for a hosting company / ISP so I have access to a datacenter, and hopefully I can get some NexSAN's this winter when we replace them with new ones. They have 25TB each. [10:11] *** RichardG has quit IRC (Read error: Connection reset by peer) [10:11] *** RichardG has joined #archiveteam-bs [10:11] *** K4k has quit IRC (Read error: Operation timed out) [10:14] *** K4k has joined #archiveteam-bs [10:31] *** drumstick has quit IRC (Ping timeout: 255 seconds) [10:41] *** drumstick has joined #archiveteam-bs [10:52] *** BlueMaxim has quit IRC (Read error: Connection reset by peer) [11:20] *** Kim___ has joined #archiveteam-bs [11:29] Hi all.I have an issue with the warrior. I have 2 laptops,and I would like to run the "Newsgrabbing" project on them both, but one of the laptops simply wont start that project. They are based on the same internetcnnection, for now anyway. Both laptops run Windows with Virtualbox and with the warrior VM - evrything is standard settings. I have tried to remove the VM several times on the laptop where I am having the problem, but the status is just "The [11:29] warrior is beginning work on a project" and it just hangs there. [11:30] Is there a guide on how to setup a fresh installed mashine with the git? I would like to run the scripts without the warrior VM.. I'm off for the next couple of hours, but I look forward for a reply :D [11:41] *** drumstick has quit IRC (Ping timeout: 370 seconds) [11:45] hook54321, arkiver: Did the owner of imgh.us reply to any of your messages at all? [11:45] Kim___: I don't think you'll be able to run the scripts directly on Windows. I could be wrong though. [11:49] *** TheLovina has joined #archiveteam-bs [12:16] *** odemg has quit IRC (Read error: Operation timed out) [12:31] *** TheLovina has quit IRC (Read error: Operation timed out) [12:40] looks like biography.com has tons of full episodes [13:36] *** TheLovina has joined #archiveteam-bs [14:57] I've got a Japanese proxy now, am crawling a copy of kcna.co.jp [15:20] JAA, I would setup a mashine with debian or ubuntu,and let it run directly on the metal.... sometimes the webinterface hangs, and I haft to reset the warrior.. No responce from webinterface, no traffic and no CPU load for some time = a reset.. Then it starts to work again. [15:21] Is there a guide somewhere on howto run the scripts directly in linux? eg when just having the terminal [15:23] Kim___: Yes, there are instructions in each project repository on GitHub, and sometimes also on the wiki. For example, URLTeam is described at https://github.com/ArchiveTeam/terroroftinytown-client-grab#running-without-a-warrior [15:23] Thankyou JAA I will look into it :D Thx. [15:33] *** odemg has joined #archiveteam-bs [15:54] *** Odd0002 has quit IRC (ZNC - http://znc.in) [16:02] *** klg_ has joined #archiveteam-bs [16:02] *** klg has quit IRC (Read error: Connection reset by peer) [16:36] *** vitzli has joined #archiveteam-bs [16:44] *** vitzli has quit IRC (Quit: Leaving) [16:47] so uh, http://archiveteam.org/index.php?title=Raptr is shutting down in 3 weeks. [16:48] oh shit [16:48] "On September 30, we will start the process of shutting off access to your Raptr account and disabling features." [17:01] *** kristian_ has joined #archiveteam-bs [17:16] *** Mateon1 has quit IRC (Remote host closed the connection) [17:16] *** Mateon1 has joined #archiveteam-bs [17:25] *** BartoCH has joined #archiveteam-bs [17:34] *** klg_ is now known as klg [18:41] *** Mayonaise has quit IRC (Read error: Operation timed out) [18:41] *** atrocity has quit IRC (Ping timeout: 250 seconds) [18:43] *** atrocity has joined #archiveteam-bs [18:44] *** JoshuaDoe has joined #archiveteam-bs [18:46] Hi JoshuaDoe, we need a project/irc channel name for the flipbook thing. For some reason people around here are keen on puns and wordplay. I can take it or leave it. :) [18:46] Lemme get thejsa in here real quick [18:46] slipbook [18:46] flipspook [18:46] He does a lot of the professional talking lol [18:46] flipbookend [18:46] *** thejsa has joined #archiveteam-bs [18:46] And the official English name for the app is "Flipnote Studio 3D" [18:46] I would have guessed "#flipoff" based on earlier names. [18:47] oh nice yeah [18:47] Just got your message, I think the webchat derped [18:47] lol [18:47] i still suggest flipbookend :) [18:47] (or Chrome's flash blocking stopped the notif sound ;-;) [18:47] aw [18:48] Writing up the Wiki page now [18:48] At the moment zino's wanting a project/IRC channel name for this as well [18:48] Metadata doesn't seem to be available afaik unless you manually scrape the web UI [18:48] Question is if we need the warrior for this. Any idea how many files it is? [18:49] File count and exact file size is unknown, I'm currently scraping the keys of the buckets [18:49] and by 'web' I mean extremely limited subset of HTML which is designed for a custom HTML renderer [18:49] File count is definitely in the tens or even hundreds of thousands, I should think [18:49] that sounds reasonably doable [18:49] instead of straight up scraping the html, should instead capture it to .warc and then scrape from there [18:49] scraping will require a little voodoo [18:50] ^ [18:50] Looking forward to hearing the details. [18:50] Houndreds of thousands shouldn't be much of a problem. One machine can do that, so no need for warrior if it's on S3. [18:51] Will go run some packet captures once I'm done with the wiki page as my memory's failing me as to the precise voodoo required [18:51] There's 4 different S3 buckets, and I believe I've already finished dumping the keys for one of them [18:51] Nice. [18:52] jkz-static-tokyo is relatively small [18:53] @JoshuaDoe wasn't there a docs bucket also [18:55] @thejsa I don't recall, I'd have to check message history [18:58] I don't think there is anyways so [18:58] it was jkz-static-tokyo/jkz-docs/* [18:59] yep [18:59] jkz-static-tokyo/jkzadm_docs is interesting [19:01] These psudo-HTML pages, are they available on a public URL we can check? [19:01] They require some voodoo to access as the server requires headers and maybe auth tokens [19:01] One moment, going to grab my 3DS [19:02] No hurry really. I'm guessing we have a few months to fix this? [19:03] April 2, 2018 [19:03] *** Mayonaise has joined #archiveteam-bs [19:03] https://www.nintendo.co.jp/support/information/2017/0908_flipnotestudio3d.html (Japanese language) [19:04] "Service end date and time April 2, 2018 (Monday) AM 10: 00" (presumably Japanese time) [19:05] Good. Lets do this according to the book then, no paniced over-the-night dump. :) [19:05] This is when I realise that I don't actually have the Japanese application installed [19:06] except I do apparently [19:06] \o/ [19:06] 3DS is derping [19:07] just reinstalling it ig [19:16] *** Mayonaise has quit IRC (Read error: Operation timed out) [19:17] *** Mayonaise has joined #archiveteam-bs [19:18] Okay I have a packet dump now [19:18] seems to route requests through CloudFront [19:19] d3o4uj0u31uj5l.cloudfront.net is jkz-static-tokyo [19:20] Right. That host without extra arguments gives access denied. [19:20] jkz-static-tokyo.s3.amazonaws.com [19:21] That's just the bucket listing. Wherent there some psudo-HTML pages? Or was it S3's XML listing you meant with that? [19:21] That's the S3 bucket for static UI data [19:21] One moment [19:22] I'll upload my packet dump now, is in Charles Proxy format but I can export [19:22] pcap would be nice. [19:22] pcap is problematic as the application uses SSL [19:22] Unless Ethereal takes Charles Proxy [19:23] Ah [19:23] Yea, that complicates things [19:23] Would requore local key capture and feeding that to the Ethereal plugin [19:24] I can export as HTTP Archive (.har) [19:24] alternatively perhaps I could try mitmproxy [19:24] would that be better? [19:24] Moment. BRB [19:25] har looks readable. [19:26] okay, one moment while I upload it [19:27] https://muffinti.me/f/FlipnoteGalleryWorld.chls https://muffinti.me/f/FlipnoteGalleryWorld.har [19:27] brb [19:38] http://www.softwareishard.com/har/viewer/ doesn't seem to happy about it. Just a bunch of "log.entries[0].response.redirectURL object value found, but a string is required". Decoding it manually is beyond what I'm going to allocate for this tonight. [19:38] I'll have a look at that tomorrow. We can download the S3 buckets straight off if needed, but I don't have a free machine with 20T up. Would have to be steap between several, so aws sync is not an option until maybe next week when I can start up a server with more disk. [19:39] Would be ashame if we don't download enough to preserve running with the original app if someone wants to fix that in the future. [19:48] we have reverse engineered the pseudo-HTML and created our own server already at https://kaeru.world/ [19:50] *** Odd0002 has joined #archiveteam-bs [19:51] Neat. I don't really need to understand it, but I need to figure out how to get them into a warc. Is all that psudo-HTML also stored in one of the buckets? [19:52] No, it is on a web server (seems to be powered by Apache Tomcat) at https://web.jkz.ctr.app.nintendo.net/ [19:53] it's a dynamic site [19:53] however to access it you need to auth with it [19:54] *** Odd0002 has quit IRC (Client Quit) [19:56] Ah. So there is where the dump comes in. Best would be to figure out to copy whatever it does so we can feel the site to wpull. I'll have a look at that tomorrow, but will be very happy if someone figures it out before I get to it. [19:56] I got Charles to output to an XML file [19:58] can't upload to my server as the disk is full, one moment [19:58] I have downloaded the previous dumps, so you can remove them. [19:59] @JoshuaDoe was dumping the keys of the S3 buckets, seem to have nearly 1GB just in keys [20:00] format is 0/000/001/2a8/a07/6d2/9da5f6947525dca0e2a01422d070aadbe9bc326f/00b696141ac7a892e905e6831b05e6831b0.kwz 217464 [20:00] (key, two spaces, size in bytes) [20:00] deleted for now [20:00] zino: https://muffinti.me/f/FlipnoteGalleryWorld.chlsx [20:01] seems quite easy to parse [20:01] POST requests to nasc.nintendowifi.net/ac are authing with Nintendo, don't think this is required though [20:03] Looks pretty clean. I need to step away for today, but I'll be around tomorrow afternoon EU time. [20:03] Sure - I'm in the UK myself so should probably also take a break [20:04] See you around tomorrow then. Think of a good project name so we can move the detail discussion of of -bs. I'm sure some of the others will appriciate it. :) [20:17] *** Odd0002 has joined #archiveteam-bs [20:21] *** jsa has joined #archiveteam-bs [20:21] Just setup a bouncer on my VPS, am @thejsa [20:21] indeed, @jsa is I [20:22] was using webchat before [20:22] *** thejsa has left [20:24] *** JoshuaDoe has quit IRC (Quit: Page closed) [20:27] *** kristian_ has quit IRC (Ping timeout: 370 seconds) [21:01] JAA: He did not reply to me unfortunately. [21:47] okay so a friend is dumping all of he keys [21:47] okay so a friend is dumping all of the keys for the Flipnote Gallery [21:47] (whoops, forgot I wasn't using Discord there) [21:48] but as far as file count goes it's in the millions [21:48] I definitely underestimated when I said tens / 100s of thousands [22:03] *** kristian_ has joined #archiveteam-bs [22:10] *** drumstick has joined #archiveteam-bs [22:11] *** kristian_ has quit IRC (Quit: Leaving) [23:28] *** BartoCH has quit IRC (Quit: WeeChat 1.9)