#archiveteam-bs 2017-09-08,Fri

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)

WhoWhatWhen
***Stiletto has quit IRC (Read error: Operation timed out)
Stilett0 has quit IRC (Ping timeout: 246 seconds)
Stilett0 has joined #archiveteam-bs
[00:01]
Mateon1 has quit IRC (Remote host closed the connection)
Mateon1 has joined #archiveteam-bs
drumstick has quit IRC (Read error: Operation timed out)
[00:22]
refeed has joined #archiveteam-bs [00:29]
godaneso i found out there was a guy that was scanning videomaker magazine
turned out he was not scanning all packages
*all pages
https://archive.org/details/@mortar
[00:32]
***refeed has quit IRC (Ping timeout: 260 seconds) [00:42]
BlueMaxim has joined #archiveteam-bs [00:54]
...... (idle for 29mn)
Stilett0 has quit IRC (Ping timeout: 245 seconds) [01:23]
drumstick has joined #archiveteam-bs [01:35]
Dimtree has joined #archiveteam-bs [01:49]
godaneso looks like i uploaded 501 items today
nevermind that was 2017-09-06
for 2017-09-07 i uploaded 534 items
:P
[01:50]
....... (idle for 30mn)
***drumstick has quit IRC (Read error: Operation timed out) [02:21]
drumstick has joined #archiveteam-bs [02:31]
....... (idle for 30mn)
Stilett0 has joined #archiveteam-bs [03:01]
.... (idle for 17mn)
antomatic has quit IRC (Read error: Operation timed out)
Stilett0 is now known as Stiletto
antomatic has joined #archiveteam-bs
swebb sets mode: +o antomatic
[03:18]
................... (idle for 1h34mn)
Sk1d has quit IRC (Ping timeout: 194 seconds) [04:55]
Sk1d has joined #archiveteam-bs [05:01]
SketchCowIf someone has time for a technical question
yipdw set up warctozip.archive.org, after underscor left.
It's giving me 503s, seems not to respond to port 8083
I don't even know where to begin
Solved it
[05:11]
........ (idle for 38mn)
atluxitygj SketchCow
I was going to suggest "history | grep start" and then check status for services previously started
[05:54]
........ (idle for 37mn)
***Mateon1 has quit IRC (Ping timeout: 255 seconds)
Mateon1 has joined #archiveteam-bs
[06:32]
what_the_ has joined #archiveteam-bs [06:40]
........... (idle for 53mn)
pikhq has quit IRC (Read error: Operation timed out) [07:33]
....... (idle for 31mn)
what_the_Good morning,
Does anyone here run the warrior on proxmox?
[08:04]
............ (idle for 56mn)
***pikhq has joined #archiveteam-bs [09:00]
............. (idle for 1h4mn)
what_the_I do want to involve me more in this project, I have some storage that can be utilized and also some hardware for it.
I also works for a hosting company / ISP so I have access to a datacenter, and hopefully I can get some NexSAN's this winter when we replace them with new ones. They have 25TB each.
[10:04]
***RichardG has quit IRC (Read error: Connection reset by peer)
RichardG has joined #archiveteam-bs
K4k has quit IRC (Read error: Operation timed out)
K4k has joined #archiveteam-bs
[10:11]
.... (idle for 17mn)
drumstick has quit IRC (Ping timeout: 255 seconds) [10:31]
drumstick has joined #archiveteam-bs [10:41]
BlueMaxim has quit IRC (Read error: Connection reset by peer) [10:52]
...... (idle for 28mn)
Kim___ has joined #archiveteam-bs [11:20]
Kim___Hi all.I have an issue with the warrior. I have 2 laptops,and I would like to run the "Newsgrabbing" project on them both, but one of the laptops simply wont start that project. They are based on the same internetcnnection, for now anyway. Both laptops run Windows with Virtualbox and with the warrior VM - evrything is standard settings. I have tried to remove the VM several times on the laptop where I am having the problem, but the status is just "The
warrior is beginning work on a project" and it just hangs there.
Is there a guide on how to setup a fresh installed mashine with the git? I would like to run the scripts without the warrior VM.. I'm off for the next couple of hours, but I look forward for a reply :D
[11:29]
***drumstick has quit IRC (Ping timeout: 370 seconds) [11:41]
JAAhook54321, arkiver: Did the owner of imgh.us reply to any of your messages at all?
Kim___: I don't think you'll be able to run the scripts directly on Windows. I could be wrong though.
[11:45]
***TheLovina has joined #archiveteam-bs [11:49]
...... (idle for 27mn)
odemg has quit IRC (Read error: Operation timed out) [12:16]
.... (idle for 15mn)
TheLovina has quit IRC (Read error: Operation timed out) [12:31]
godanelooks like biography.com has tons of full episodes [12:40]
............ (idle for 56mn)
***TheLovina has joined #archiveteam-bs [13:36]
................. (idle for 1h21mn)
HCross2I've got a Japanese proxy now, am crawling a copy of kcna.co.jp [14:57]
..... (idle for 23mn)
Kim___JAA, I would setup a mashine with debian or ubuntu,and let it run directly on the metal.... sometimes the webinterface hangs, and I haft to reset the warrior.. No responce from webinterface, no traffic and no CPU load for some time = a reset.. Then it starts to work again.
Is there a guide somewhere on howto run the scripts directly in linux? eg when just having the terminal
[15:20]
JAAKim___: Yes, there are instructions in each project repository on GitHub, and sometimes also on the wiki. For example, URLTeam is described at https://github.com/ArchiveTeam/terroroftinytown-client-grab#running-without-a-warrior [15:23]
Kim___Thankyou JAA I will look into it :D Thx. [15:23]
***odemg has joined #archiveteam-bs [15:33]
..... (idle for 21mn)
Odd0002 has quit IRC (ZNC - http://znc.in) [15:54]
klg_ has joined #archiveteam-bs
klg has quit IRC (Read error: Connection reset by peer)
[16:02]
....... (idle for 34mn)
vitzli has joined #archiveteam-bs [16:36]
vitzli has quit IRC (Quit: Leaving) [16:44]
t2t2so uh, http://archiveteam.org/index.php?title=Raptr is shutting down in 3 weeks. [16:47]
Froggingoh shit [16:48]
t2t2"On September 30, we will start the process of shutting off access to your Raptr account and disabling features." [16:48]
***kristian_ has joined #archiveteam-bs [17:01]
.... (idle for 15mn)
Mateon1 has quit IRC (Remote host closed the connection)
Mateon1 has joined #archiveteam-bs
[17:16]
BartoCH has joined #archiveteam-bs [17:25]
klg_ is now known as klg [17:34]
.............. (idle for 1h7mn)
Mayonaise has quit IRC (Read error: Operation timed out)
atrocity has quit IRC (Ping timeout: 250 seconds)
atrocity has joined #archiveteam-bs
JoshuaDoe has joined #archiveteam-bs
[18:41]
zinoHi JoshuaDoe, we need a project/irc channel name for the flipbook thing. For some reason people around here are keen on puns and wordplay. I can take it or leave it. :) [18:46]
JoshuaDoeLemme get thejsa in here real quick [18:46]
astridslipbook
flipspook
[18:46]
JoshuaDoeHe does a lot of the professional talking lol [18:46]
astridflipbookend [18:46]
***thejsa has joined #archiveteam-bs [18:46]
JoshuaDoeAnd the official English name for the app is "Flipnote Studio 3D" [18:46]
zinoI would have guessed "#flipoff" based on earlier names. [18:46]
astridoh nice yeah [18:47]
thejsaJust got your message, I think the webchat derped [18:47]
JoshuaDoelol [18:47]
astridi still suggest flipbookend :) [18:47]
thejsa(or Chrome's flash blocking stopped the notif sound ;-;) [18:47]
astridaw [18:47]
thejsaWriting up the Wiki page now [18:48]
JoshuaDoeAt the moment zino's wanting a project/IRC channel name for this as well [18:48]
thejsaMetadata doesn't seem to be available afaik unless you manually scrape the web UI [18:48]
zinoQuestion is if we need the warrior for this. Any idea how many files it is? [18:48]
JoshuaDoeFile count and exact file size is unknown, I'm currently scraping the keys of the buckets [18:49]
thejsaand by 'web' I mean extremely limited subset of HTML which is designed for a custom HTML renderer
File count is definitely in the tens or even hundreds of thousands, I should think
[18:49]
astridthat sounds reasonably doable
instead of straight up scraping the html, should instead capture it to .warc and then scrape from there
[18:49]
thejsascraping will require a little voodoo [18:49]
JoshuaDoe^ [18:50]
zinoLooking forward to hearing the details.
Houndreds of thousands shouldn't be much of a problem. One machine can do that, so no need for warrior if it's on S3.
[18:50]
thejsaWill go run some packet captures once I'm done with the wiki page as my memory's failing me as to the precise voodoo required [18:51]
JoshuaDoeThere's 4 different S3 buckets, and I believe I've already finished dumping the keys for one of them [18:51]
zinoNice. [18:51]
thejsajkz-static-tokyo is relatively small
@JoshuaDoe wasn't there a docs bucket also
[18:52]
JoshuaDoe@thejsa I don't recall, I'd have to check message history [18:55]
thejsaI don't think there is anyways so
it was jkz-static-tokyo/jkz-docs/*
yep
jkz-static-tokyo/jkzadm_docs is interesting
[18:58]
zinoThese psudo-HTML pages, are they available on a public URL we can check? [19:01]
thejsaThey require some voodoo to access as the server requires headers and maybe auth tokens
One moment, going to grab my 3DS
[19:01]
zinoNo hurry really. I'm guessing we have a few months to fix this? [19:02]
thejsaApril 2, 2018 [19:03]
***Mayonaise has joined #archiveteam-bs [19:03]
thejsahttps://www.nintendo.co.jp/support/information/2017/0908_flipnotestudio3d.html (Japanese language)
"Service end date and time April 2, 2018 (Monday) AM 10: 00" (presumably Japanese time)
[19:03]
zinoGood. Lets do this according to the book then, no paniced over-the-night dump. :) [19:05]
thejsaThis is when I realise that I don't actually have the Japanese application installed
except I do apparently
[19:05]
zino\o/ [19:06]
thejsa3DS is derping
just reinstalling it ig
[19:06]
***Mayonaise has quit IRC (Read error: Operation timed out)
Mayonaise has joined #archiveteam-bs
[19:16]
thejsaOkay I have a packet dump now
seems to route requests through CloudFront
d3o4uj0u31uj5l.cloudfront.net is jkz-static-tokyo
[19:18]
zinoRight. That host without extra arguments gives access denied. [19:20]
thejsajkz-static-tokyo.s3.amazonaws.com [19:20]
zinoThat's just the bucket listing. Wherent there some psudo-HTML pages? Or was it S3's XML listing you meant with that? [19:21]
thejsaThat's the S3 bucket for static UI data
One moment
I'll upload my packet dump now, is in Charles Proxy format but I can export
[19:21]
zinopcap would be nice. [19:22]
thejsapcap is problematic as the application uses SSL [19:22]
zinoUnless Ethereal takes Charles Proxy
Ah
Yea, that complicates things
Would requore local key capture and feeding that to the Ethereal plugin
[19:22]
thejsaI can export as HTTP Archive (.har)
alternatively perhaps I could try mitmproxy
would that be better?
[19:24]
zinoMoment. BRB
har looks readable.
[19:24]
thejsaokay, one moment while I upload it
https://muffinti.me/f/FlipnoteGalleryWorld.chls https://muffinti.me/f/FlipnoteGalleryWorld.har
brb
[19:26]
zinohttp://www.softwareishard.com/har/viewer/ doesn't seem to happy about it. Just a bunch of "log.entries[0].response.redirectURL object value found, but a string is required". Decoding it manually is beyond what I'm going to allocate for this tonight.
I'll have a look at that tomorrow. We can download the S3 buckets straight off if needed, but I don't have a free machine with 20T up. Would have to be steap between several, so aws sync is not an option until maybe next week when I can start up a server with more disk.
Would be ashame if we don't download enough to preserve running with the original app if someone wants to fix that in the future.
[19:38]
thejsawe have reverse engineered the pseudo-HTML and created our own server already at https://kaeru.world/ [19:48]
***Odd0002 has joined #archiveteam-bs [19:50]
zinoNeat. I don't really need to understand it, but I need to figure out how to get them into a warc. Is all that psudo-HTML also stored in one of the buckets? [19:51]
thejsaNo, it is on a web server (seems to be powered by Apache Tomcat) at https://web.jkz.ctr.app.nintendo.net/
it's a dynamic site
however to access it you need to auth with it
[19:52]
***Odd0002 has quit IRC (Client Quit) [19:54]
zinoAh. So there is where the dump comes in. Best would be to figure out to copy whatever it does so we can feel the site to wpull. I'll have a look at that tomorrow, but will be very happy if someone figures it out before I get to it. [19:56]
thejsaI got Charles to output to an XML file
can't upload to my server as the disk is full, one moment
[19:56]
zinoI have downloaded the previous dumps, so you can remove them. [19:58]
thejsa@JoshuaDoe was dumping the keys of the S3 buckets, seem to have nearly 1GB just in keys
format is 0/000/001/2a8/a07/6d2/9da5f6947525dca0e2a01422d070aadbe9bc326f/00b696141ac7a892e905e6831b05e6831b0.kwz 217464
(key, two spaces, size in bytes)
deleted for now
zino: https://muffinti.me/f/FlipnoteGalleryWorld.chlsx
seems quite easy to parse
POST requests to nasc.nintendowifi.net/ac are authing with Nintendo, don't think this is required though
[19:59]
zinoLooks pretty clean. I need to step away for today, but I'll be around tomorrow afternoon EU time. [20:03]
thejsaSure - I'm in the UK myself so should probably also take a break [20:03]
zinoSee you around tomorrow then. Think of a good project name so we can move the detail discussion of of -bs. I'm sure some of the others will appriciate it. :) [20:04]
***Odd0002 has joined #archiveteam-bs
jsa has joined #archiveteam-bs
[20:17]
jsaJust setup a bouncer on my VPS, am @thejsa [20:21]
thejsaindeed, @jsa is I [20:21]
jsawas using webchat before [20:22]
***thejsa has left
JoshuaDoe has quit IRC (Quit: Page closed)
kristian_ has quit IRC (Ping timeout: 370 seconds)
[20:22]
....... (idle for 34mn)
hook54321JAA: He did not reply to me unfortunately. [21:01]
.......... (idle for 46mn)
jsaokay so a friend is dumping all of he keys
okay so a friend is dumping all of the keys for the Flipnote Gallery
(whoops, forgot I wasn't using Discord there)
but as far as file count goes it's in the millions
I definitely underestimated when I said tens / 100s of thousands
[21:47]
.... (idle for 15mn)
***kristian_ has joined #archiveteam-bs [22:03]
drumstick has joined #archiveteam-bs
kristian_ has quit IRC (Quit: Leaving)
[22:10]
................ (idle for 1h17mn)
BartoCH has quit IRC (Quit: WeeChat 1.9) [23:28]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)