#archiveteam-bs 2017-09-08,Fri

↑back Search

Time Nickname Message
00:01 🔗 Stiletto has quit IRC (Read error: Operation timed out)
00:04 🔗 Stilett0 has quit IRC (Ping timeout: 246 seconds)
00:08 🔗 Stilett0 has joined #archiveteam-bs
00:22 🔗 Mateon1 has quit IRC (Remote host closed the connection)
00:22 🔗 Mateon1 has joined #archiveteam-bs
00:23 🔗 drumstick has quit IRC (Read error: Operation timed out)
00:29 🔗 refeed has joined #archiveteam-bs
00:32 🔗 godane so i found out there was a guy that was scanning videomaker magazine
00:32 🔗 godane turned out he was not scanning all packages
00:32 🔗 godane *all pages
00:33 🔗 godane https://archive.org/details/@mortar
00:42 🔗 refeed has quit IRC (Ping timeout: 260 seconds)
00:54 🔗 BlueMaxim has joined #archiveteam-bs
01:23 🔗 Stilett0 has quit IRC (Ping timeout: 245 seconds)
01:35 🔗 drumstick has joined #archiveteam-bs
01:49 🔗 Dimtree has joined #archiveteam-bs
01:50 🔗 godane so looks like i uploaded 501 items today
01:51 🔗 godane nevermind that was 2017-09-06
01:51 🔗 godane for 2017-09-07 i uploaded 534 items
01:51 🔗 godane :P
02:21 🔗 drumstick has quit IRC (Read error: Operation timed out)
02:31 🔗 drumstick has joined #archiveteam-bs
03:01 🔗 Stilett0 has joined #archiveteam-bs
03:18 🔗 antomatic has quit IRC (Read error: Operation timed out)
03:18 🔗 Stilett0 is now known as Stiletto
03:21 🔗 antomatic has joined #archiveteam-bs
03:21 🔗 swebb sets mode: +o antomatic
04:55 🔗 Sk1d has quit IRC (Ping timeout: 194 seconds)
05:01 🔗 Sk1d has joined #archiveteam-bs
05:11 🔗 SketchCow If someone has time for a technical question
05:11 🔗 SketchCow yipdw set up warctozip.archive.org, after underscor left.
05:12 🔗 SketchCow It's giving me 503s, seems not to respond to port 8083
05:12 🔗 SketchCow I don't even know where to begin
05:16 🔗 SketchCow Solved it
05:54 🔗 atluxity gj SketchCow
05:55 🔗 atluxity I was going to suggest "history | grep start" and then check status for services previously started
06:32 🔗 Mateon1 has quit IRC (Ping timeout: 255 seconds)
06:32 🔗 Mateon1 has joined #archiveteam-bs
06:40 🔗 what_the_ has joined #archiveteam-bs
07:33 🔗 pikhq has quit IRC (Read error: Operation timed out)
08:04 🔗 what_the_ Good morning,
08:04 🔗 what_the_ Does anyone here run the warrior on proxmox?
09:00 🔗 pikhq has joined #archiveteam-bs
10:04 🔗 what_the_ I do want to involve me more in this project, I have some storage that can be utilized and also some hardware for it.
10:05 🔗 what_the_ I also works for a hosting company / ISP so I have access to a datacenter, and hopefully I can get some NexSAN's this winter when we replace them with new ones. They have 25TB each.
10:11 🔗 RichardG has quit IRC (Read error: Connection reset by peer)
10:11 🔗 RichardG has joined #archiveteam-bs
10:11 🔗 K4k has quit IRC (Read error: Operation timed out)
10:14 🔗 K4k has joined #archiveteam-bs
10:31 🔗 drumstick has quit IRC (Ping timeout: 255 seconds)
10:41 🔗 drumstick has joined #archiveteam-bs
10:52 🔗 BlueMaxim has quit IRC (Read error: Connection reset by peer)
11:20 🔗 Kim___ has joined #archiveteam-bs
11:29 🔗 Kim___ Hi all.I have an issue with the warrior. I have 2 laptops,and I would like to run the "Newsgrabbing" project on them both, but one of the laptops simply wont start that project. They are based on the same internetcnnection, for now anyway. Both laptops run Windows with Virtualbox and with the warrior VM - evrything is standard settings. I have tried to remove the VM several times on the laptop where I am having the problem, but the status is just "The
11:29 🔗 Kim___ warrior is beginning work on a project" and it just hangs there.
11:30 🔗 Kim___ Is there a guide on how to setup a fresh installed mashine with the git? I would like to run the scripts without the warrior VM.. I'm off for the next couple of hours, but I look forward for a reply :D
11:41 🔗 drumstick has quit IRC (Ping timeout: 370 seconds)
11:45 🔗 JAA hook54321, arkiver: Did the owner of imgh.us reply to any of your messages at all?
11:45 🔗 JAA Kim___: I don't think you'll be able to run the scripts directly on Windows. I could be wrong though.
11:49 🔗 TheLovina has joined #archiveteam-bs
12:16 🔗 odemg has quit IRC (Read error: Operation timed out)
12:31 🔗 TheLovina has quit IRC (Read error: Operation timed out)
12:40 🔗 godane looks like biography.com has tons of full episodes
13:36 🔗 TheLovina has joined #archiveteam-bs
14:57 🔗 HCross2 I've got a Japanese proxy now, am crawling a copy of kcna.co.jp
15:20 🔗 Kim___ JAA, I would setup a mashine with debian or ubuntu,and let it run directly on the metal.... sometimes the webinterface hangs, and I haft to reset the warrior.. No responce from webinterface, no traffic and no CPU load for some time = a reset.. Then it starts to work again.
15:21 🔗 Kim___ Is there a guide somewhere on howto run the scripts directly in linux? eg when just having the terminal
15:23 🔗 JAA Kim___: Yes, there are instructions in each project repository on GitHub, and sometimes also on the wiki. For example, URLTeam is described at https://github.com/ArchiveTeam/terroroftinytown-client-grab#running-without-a-warrior
15:23 🔗 Kim___ Thankyou JAA I will look into it :D Thx.
15:33 🔗 odemg has joined #archiveteam-bs
15:54 🔗 Odd0002 has quit IRC (ZNC - http://znc.in)
16:02 🔗 klg_ has joined #archiveteam-bs
16:02 🔗 klg has quit IRC (Read error: Connection reset by peer)
16:36 🔗 vitzli has joined #archiveteam-bs
16:44 🔗 vitzli has quit IRC (Quit: Leaving)
16:47 🔗 t2t2 so uh, http://archiveteam.org/index.php?title=Raptr is shutting down in 3 weeks.
16:48 🔗 Frogging oh shit
16:48 🔗 t2t2 "On September 30, we will start the process of shutting off access to your Raptr account and disabling features."
17:01 🔗 kristian_ has joined #archiveteam-bs
17:16 🔗 Mateon1 has quit IRC (Remote host closed the connection)
17:16 🔗 Mateon1 has joined #archiveteam-bs
17:25 🔗 BartoCH has joined #archiveteam-bs
17:34 🔗 klg_ is now known as klg
18:41 🔗 Mayonaise has quit IRC (Read error: Operation timed out)
18:41 🔗 atrocity has quit IRC (Ping timeout: 250 seconds)
18:43 🔗 atrocity has joined #archiveteam-bs
18:44 🔗 JoshuaDoe has joined #archiveteam-bs
18:46 🔗 zino Hi JoshuaDoe, we need a project/irc channel name for the flipbook thing. For some reason people around here are keen on puns and wordplay. I can take it or leave it. :)
18:46 🔗 JoshuaDoe Lemme get thejsa in here real quick
18:46 🔗 astrid slipbook
18:46 🔗 astrid flipspook
18:46 🔗 JoshuaDoe He does a lot of the professional talking lol
18:46 🔗 astrid flipbookend
18:46 🔗 thejsa has joined #archiveteam-bs
18:46 🔗 JoshuaDoe And the official English name for the app is "Flipnote Studio 3D"
18:46 🔗 zino I would have guessed "#flipoff" based on earlier names.
18:47 🔗 astrid oh nice yeah
18:47 🔗 thejsa Just got your message, I think the webchat derped
18:47 🔗 JoshuaDoe lol
18:47 🔗 astrid i still suggest flipbookend :)
18:47 🔗 thejsa (or Chrome's flash blocking stopped the notif sound ;-;)
18:47 🔗 astrid aw
18:48 🔗 thejsa Writing up the Wiki page now
18:48 🔗 JoshuaDoe At the moment zino's wanting a project/IRC channel name for this as well
18:48 🔗 thejsa Metadata doesn't seem to be available afaik unless you manually scrape the web UI
18:48 🔗 zino Question is if we need the warrior for this. Any idea how many files it is?
18:49 🔗 JoshuaDoe File count and exact file size is unknown, I'm currently scraping the keys of the buckets
18:49 🔗 thejsa and by 'web' I mean extremely limited subset of HTML which is designed for a custom HTML renderer
18:49 🔗 thejsa File count is definitely in the tens or even hundreds of thousands, I should think
18:49 🔗 astrid that sounds reasonably doable
18:49 🔗 astrid instead of straight up scraping the html, should instead capture it to .warc and then scrape from there
18:49 🔗 thejsa scraping will require a little voodoo
18:50 🔗 JoshuaDoe ^
18:50 🔗 zino Looking forward to hearing the details.
18:50 🔗 zino Houndreds of thousands shouldn't be much of a problem. One machine can do that, so no need for warrior if it's on S3.
18:51 🔗 thejsa Will go run some packet captures once I'm done with the wiki page as my memory's failing me as to the precise voodoo required
18:51 🔗 JoshuaDoe There's 4 different S3 buckets, and I believe I've already finished dumping the keys for one of them
18:51 🔗 zino Nice.
18:52 🔗 thejsa jkz-static-tokyo is relatively small
18:53 🔗 thejsa @JoshuaDoe wasn't there a docs bucket also
18:55 🔗 JoshuaDoe @thejsa I don't recall, I'd have to check message history
18:58 🔗 thejsa I don't think there is anyways so
18:58 🔗 thejsa it was jkz-static-tokyo/jkz-docs/*
18:59 🔗 thejsa yep
18:59 🔗 thejsa jkz-static-tokyo/jkzadm_docs is interesting
19:01 🔗 zino These psudo-HTML pages, are they available on a public URL we can check?
19:01 🔗 thejsa They require some voodoo to access as the server requires headers and maybe auth tokens
19:01 🔗 thejsa One moment, going to grab my 3DS
19:02 🔗 zino No hurry really. I'm guessing we have a few months to fix this?
19:03 🔗 thejsa April 2, 2018
19:03 🔗 Mayonaise has joined #archiveteam-bs
19:03 🔗 thejsa https://www.nintendo.co.jp/support/information/2017/0908_flipnotestudio3d.html (Japanese language)
19:04 🔗 thejsa "Service end date and time April 2, 2018 (Monday) AM 10: 00" (presumably Japanese time)
19:05 🔗 zino Good. Lets do this according to the book then, no paniced over-the-night dump. :)
19:05 🔗 thejsa This is when I realise that I don't actually have the Japanese application installed
19:06 🔗 thejsa except I do apparently
19:06 🔗 zino \o/
19:06 🔗 thejsa 3DS is derping
19:07 🔗 thejsa just reinstalling it ig
19:16 🔗 Mayonaise has quit IRC (Read error: Operation timed out)
19:17 🔗 Mayonaise has joined #archiveteam-bs
19:18 🔗 thejsa Okay I have a packet dump now
19:18 🔗 thejsa seems to route requests through CloudFront
19:19 🔗 thejsa d3o4uj0u31uj5l.cloudfront.net is jkz-static-tokyo
19:20 🔗 zino Right. That host without extra arguments gives access denied.
19:20 🔗 thejsa jkz-static-tokyo.s3.amazonaws.com
19:21 🔗 zino That's just the bucket listing. Wherent there some psudo-HTML pages? Or was it S3's XML listing you meant with that?
19:21 🔗 thejsa That's the S3 bucket for static UI data
19:21 🔗 thejsa One moment
19:22 🔗 thejsa I'll upload my packet dump now, is in Charles Proxy format but I can export
19:22 🔗 zino pcap would be nice.
19:22 🔗 thejsa pcap is problematic as the application uses SSL
19:22 🔗 zino Unless Ethereal takes Charles Proxy
19:23 🔗 zino Ah
19:23 🔗 zino Yea, that complicates things
19:23 🔗 zino Would requore local key capture and feeding that to the Ethereal plugin
19:24 🔗 thejsa I can export as HTTP Archive (.har)
19:24 🔗 thejsa alternatively perhaps I could try mitmproxy
19:24 🔗 thejsa would that be better?
19:24 🔗 zino Moment. BRB
19:25 🔗 zino har looks readable.
19:26 🔗 thejsa okay, one moment while I upload it
19:27 🔗 thejsa https://muffinti.me/f/FlipnoteGalleryWorld.chls https://muffinti.me/f/FlipnoteGalleryWorld.har
19:27 🔗 thejsa brb
19:38 🔗 zino http://www.softwareishard.com/har/viewer/ doesn't seem to happy about it. Just a bunch of "log.entries[0].response.redirectURL object value found, but a string is required". Decoding it manually is beyond what I'm going to allocate for this tonight.
19:38 🔗 zino I'll have a look at that tomorrow. We can download the S3 buckets straight off if needed, but I don't have a free machine with 20T up. Would have to be steap between several, so aws sync is not an option until maybe next week when I can start up a server with more disk.
19:39 🔗 zino Would be ashame if we don't download enough to preserve running with the original app if someone wants to fix that in the future.
19:48 🔗 thejsa we have reverse engineered the pseudo-HTML and created our own server already at https://kaeru.world/
19:50 🔗 Odd0002 has joined #archiveteam-bs
19:51 🔗 zino Neat. I don't really need to understand it, but I need to figure out how to get them into a warc. Is all that psudo-HTML also stored in one of the buckets?
19:52 🔗 thejsa No, it is on a web server (seems to be powered by Apache Tomcat) at https://web.jkz.ctr.app.nintendo.net/
19:53 🔗 thejsa it's a dynamic site
19:53 🔗 thejsa however to access it you need to auth with it
19:54 🔗 Odd0002 has quit IRC (Client Quit)
19:56 🔗 zino Ah. So there is where the dump comes in. Best would be to figure out to copy whatever it does so we can feel the site to wpull. I'll have a look at that tomorrow, but will be very happy if someone figures it out before I get to it.
19:56 🔗 thejsa I got Charles to output to an XML file
19:58 🔗 thejsa can't upload to my server as the disk is full, one moment
19:58 🔗 zino I have downloaded the previous dumps, so you can remove them.
19:59 🔗 thejsa @JoshuaDoe was dumping the keys of the S3 buckets, seem to have nearly 1GB just in keys
20:00 🔗 thejsa format is 0/000/001/2a8/a07/6d2/9da5f6947525dca0e2a01422d070aadbe9bc326f/00b696141ac7a892e905e6831b05e6831b0.kwz 217464
20:00 🔗 thejsa (key, two spaces, size in bytes)
20:00 🔗 thejsa deleted for now
20:00 🔗 thejsa zino: https://muffinti.me/f/FlipnoteGalleryWorld.chlsx
20:01 🔗 thejsa seems quite easy to parse
20:01 🔗 thejsa POST requests to nasc.nintendowifi.net/ac are authing with Nintendo, don't think this is required though
20:03 🔗 zino Looks pretty clean. I need to step away for today, but I'll be around tomorrow afternoon EU time.
20:03 🔗 thejsa Sure - I'm in the UK myself so should probably also take a break
20:04 🔗 zino See you around tomorrow then. Think of a good project name so we can move the detail discussion of of -bs. I'm sure some of the others will appriciate it. :)
20:17 🔗 Odd0002 has joined #archiveteam-bs
20:21 🔗 jsa has joined #archiveteam-bs
20:21 🔗 jsa Just setup a bouncer on my VPS, am @thejsa
20:21 🔗 thejsa indeed, @jsa is I
20:22 🔗 jsa was using webchat before
20:22 🔗 thejsa has left
20:24 🔗 JoshuaDoe has quit IRC (Quit: Page closed)
20:27 🔗 kristian_ has quit IRC (Ping timeout: 370 seconds)
21:01 🔗 hook54321 JAA: He did not reply to me unfortunately.
21:47 🔗 jsa okay so a friend is dumping all of he keys
21:47 🔗 jsa okay so a friend is dumping all of the keys for the Flipnote Gallery
21:47 🔗 jsa (whoops, forgot I wasn't using Discord there)
21:48 🔗 jsa but as far as file count goes it's in the millions
21:48 🔗 jsa I definitely underestimated when I said tens / 100s of thousands
22:03 🔗 kristian_ has joined #archiveteam-bs
22:10 🔗 drumstick has joined #archiveteam-bs
22:11 🔗 kristian_ has quit IRC (Quit: Leaving)
23:28 🔗 BartoCH has quit IRC (Quit: WeeChat 1.9)

irclogger-viewer