#archiveteam-bs 2017-06-22,Thu

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)

WhoWhatWhen
joepie91https://twitter.com/joepie91/status/877676538019409926 -- "Gathering weak npm credentials -- how poor passwords and auth mechanisms put 52% of the Node.js ecosystem at risk https://github.com/ChALkeR/notes/blob/master/Gathering-weak-npm-credentials.md"
(since we don't have a titlebot here)
[00:02]
Froggingamazing
:p
[00:04]
MrRadarFrom the community that brought you left-pad... [00:04]
joepie91MrRadar: nope, from the developer community [00:05]
Froggingyes, them
:p
[00:05]
joepie91MrRadar: I'm almost certain that you could run these numbers for basically every package registry that doesn't either outsource auth or centralize packaging, and end up with the same result
because the other registries *also* don't seem to have any measures against this
if it weren't for github being reasonably okay at finding leaked credentials and preventing bruteforcing etc., you could probably do this on github too
but nobody seems to have run the numbers for any of the other registries yet :)
[00:05]
MrRadarYeah, I was just taking a cheap shot :P [00:06]
joepie91actually, I think somebody might have done this for CPAN at some point
but I'm not sure
[00:06]
***BlueMaxim has joined #archiveteam-bs
antomatic has joined #archiveteam-bs
swebb sets mode: +o antomatic
[00:10]
joepie91MrRadar: anyway, be wary with the cheap shots like that, because it can very easily be interpreted seriously by people, and lead to people going "oh it's just Node being dumb" (ie. missing the point and leaving all the non-Node shit insecure)
I regularly speak to people who would interpret your statement totally seriously :)
[00:24]
MrRadarYeah, understood [00:26]
lucysunthank you - i was poking around there [00:28]
***dan- has quit IRC (Ping timeout: 260 seconds) [00:32]
crusher2arkiver: i paused all the other warriors, and the imzy ones are just happily exchanging http requests
i can't see why the script isn't doing anything
[00:37]
.... (idle for 16mn)
ah ha...
any idea why i'd be getting: "I give up... process wgetdownload returned exit code -6
[00:53]
***luckcolor has joined #archiveteam-bs [00:54]
..... (idle for 21mn)
joepie91FYI: imgur has started serving up HTML + JS pages for direct image links
which for some reason requires JS and loads React, to show a fucking image
this means that any kind of referer-less archiving is very likely to fail if imgur does not recognize your client as a headless client
(also, what the fuck imgur?)
[01:15]
alembicit's my opinion that theres an emerging evil-pattern to purposely have sites crap the bed w/o javascript to force you to enable it so they can get their analytics [01:27]
***dan- has joined #archiveteam-bs [01:27]
alembicw/o javascript, shopify will literally load the content of the website (as in, you can read the website in view-sources://) but the page renders as a blank white page
someone should write a plugin to decrapify this bullshit
[01:28]
ndiddyjoepie91: : direct links work for me
ex http://i.imgur.com/Nj0znc8.png
[01:31]
MrRadarSame here, though I was seeing some very strange redirects when I was doing a bunch of imgur archiving last eek
It was redirecting from i.imgur.com/whatever.png to URLs along the lines of i.imgur.com/original/w/h/a/whatever ... which redirected to themselves
[01:32]
ndiddyare you sure they were direct image links
i just downloaded that image using wget and it worked fine
http://i.imgur.com/aR6mdR1.png
[01:34]
MrRadarYes, or at least they should have been
And it wasn't consistent for me
Sometimes the direct links would work as expected, sometimes it would redirect
It only affected < 1 % of the grab, so I just re-ran the ones that got the redirect
[01:34]
ndiddyso do you know a better free imagehost
most of the other ones i know of delete images after so long
[01:38]
alembichonestly, they all turn to crap in time...
seems like the best thing atm is to use S3/B2 or spin up a VPS
and I get why those are not perfect replacements
[01:45]
***bitBaron has quit IRC (Quit: My computer has gone to sleep. ZZZzzz…) [01:50]
joepie91ndiddy: some people are reporting HTML pages, others are reporting direct links still work
ndiddy: but imgur is detecting headless clients and referers, so you can't reproduce with wget
[01:51]
DFJustinyeah I started noticing it last week I think [01:51]
joepie91they're definitely throwing up HTML+JS for *some* clients though [01:51]
DFJustineven right click, save as on an image from firefox produces an html file sometimes [01:51]
ndiddyjoepie91: just set your referer to some random site [01:51]
joepie91ndiddy: overriding referer globally will break a bunch of stuff :)
and that doesn't change that creators of archival tools need to be aware of this
[01:52]
Froggingthere is no good free imagehost. it's a hopeless venture and they all fall eventually
I've long since switched to self-hosting images that I share
all free image hosts to date have either shut down, become unusable, or are in the process of doing either of those
[02:03]
crusher2https://i.imgflip.com/1rasb4.jpg [02:10]
.... (idle for 16mn)
joepie91ha
where's that from? :P
[02:26]
Crusher_I was slightly bored and had a few minutes [02:33]
joepie91nice [02:36]
***thuban2 has joined #archiveteam-bs [02:37]
.... (idle for 18mn)
Crusher has joined #archiveteam-bs
Crusher_ has quit IRC (Read error: Connection reset by peer)
[02:55]
Somebody2Has anyone else noticed that twitter recently blocked access by non-logged-in users to the /with_replies pane; e.g. https://twitter.com/textfiles/with_replies
That's quite a big change, and I was surprised to not be able to find *any* mention of it anywhere.
[02:55]
CrusherIt does?
I clicked on it an... Oh wait, I'm logged in...
[02:56]
Somebody2kisspunch: regarding #internetarchive.bak -- probably good to discuss your idea of a Windows client in there, too. [02:58]
dashcloudSomebody2: I thought I saw that today, but I wasn't sure, so I checked, and it's definitely happening now [03:09]
crusher2i've just realized it's faster for me to install ubuntu using the network installer than by disk... [03:11]
***crusher2 is now known as Crusher2 [03:12]
..... (idle for 22mn)
th1x has quit IRC (Ping timeout: 633 seconds)
th1x has joined #archiveteam-bs
[03:34]
.......... (idle for 48mn)
Crusher2 has quit IRC (Ping timeout: 268 seconds) [04:24]
.... (idle for 19mn)
Sk1d has quit IRC (Ping timeout: 250 seconds) [04:43]
Sk1d has joined #archiveteam-bs [04:50]
........................... (idle for 2h14mn)
schbirid has joined #archiveteam-bs [07:04]
..... (idle for 23mn)
Jonison has joined #archiveteam-bs [07:27]
....... (idle for 32mn)
SHODAN_UI has joined #archiveteam-bs [07:59]
.............. (idle for 1h9mn)
j08nY has joined #archiveteam-bs [09:08]
............................... (idle for 2h33mn)
lucysun has quit IRC (Quit: Page closed) [11:41]
..... (idle for 22mn)
icedice has joined #archiveteam-bs
BlueMaxim has quit IRC (Read error: Operation timed out)
BlueMaxim has joined #archiveteam-bs
BlueMaxim has quit IRC (Client Quit)
[12:03]
...... (idle for 27mn)
Crusher has quit IRC (Ping timeout: 492 seconds) [12:38]
timmcarkiver: weffey had to kill API access because the DB was getting overloaded [12:42]
***user0815 has joined #archiveteam-bs [12:47]
qw3rty has joined #archiveteam-bs [12:57]
Hiccup has joined #archiveteam-bs [13:09]
pizzaiolo has joined #archiveteam-bs [13:15]
.... (idle for 16mn)
HiccupHi. How would I go about adding downloads to a WARC file? I used https://webrecorder.io to record the non-download stuff, but there doesn't seem to be a way to add downloads. [13:31]
***SilSte has quit IRC (Quit: http://quassel-irc.org - Chat comfortably. Anywhere.) [13:35]
SilSte has joined #archiveteam-bs [13:40]
timmcarkiver: Maybe try halving the rate on Imyz?
The current rate is hundreds of concurrent sessions and the DB connection is getting overloaded.
[13:44]
.... (idle for 17mn)
***Odd0002 has joined #archiveteam-bs
Hiccup has left
icedice has quit IRC (Read error: Connection reset by peer)
[14:03]
MrRadarKaz: ^^^
(In case you're around)
[14:08]
***thuban3 has joined #archiveteam-bs
thuban2 has quit IRC (Read error: Operation timed out)
[14:22]
...... (idle for 26mn)
qw3rty has left [14:51]
KazHello
tracker limit 200 -> 50
[14:55]
...... (idle for 26mn)
***RichardG has quit IRC (Read error: Connection reset by peer)
qw3rty has joined #archiveteam-bs
RichardG has joined #archiveteam-bs
[15:22]
JAATen days and 1.6 million URLs later, my Tilt grab is finally done. :-)
The website grab, that is. I'll set up the API grab later.
[15:28]
***Crusher has joined #archiveteam-bs [15:42]
..... (idle for 23mn)
Odd0002 has quit IRC (Remote host closed the connection) [16:05]
kristian_ has joined #archiveteam-bs [16:17]
SketchCo1 is now known as SketchCow [16:23]
CrusherIs that who I think it is? [16:24]
xmcdepends on who you think who they are [16:25]
Crusher*Jason Scott's name echoes through the irc* [16:25]
Ahem.. silliness aside, that is him if I'm not mistaken, right? [16:30]
MrRadarYes [16:31]
Froggingmhmm [16:32]
***pizzaiolo has quit IRC (Remote host closed the connection) [16:33]
SketchCow? [16:36]
CrusherI must thank you for your excellent talks, I found them online and that's what inspired me to help in whatever way I can
So thank you.
[16:40]
jrwrSketchCow: is my hero
I hope to get to see one of his talks, or even better meet him at a vendor booth or something :)
[16:45]
***nyany has quit IRC (Ping timeout: 506 seconds) [16:46]
jrwrI love talking to bombastic people, its great fun. I am working on my old BBS Archives, most of them are Japanese [16:47]
joepie91lol: https://twitter.com/mike_hasarms/status/877779717935333377 [16:47]
***greycat has joined #archiveteam-bs [16:49]
greycatHi there! I wanted to ask if anyone from Archive Team knows of Kaitai Struct project and if it's possible to discuss potential collaboration between our projects?
We've launched http://formats.kaitai.io/ recently, which is a site for formal specifications of various file formats
I guess it should be of interest to archival teams, as it provides a machine-readable, non-ambiguous specs which can be visualized as a diagram,
or interpreted with a given file to highlight file structure as a tree,
[16:51]
***kristian_ has quit IRC (Read error: Operation timed out) [16:53]
greycator compiled into a ready-made parser for that file format in many programming languages [16:53]
xmchm, neato [16:54]
greycatI thought of whether we can interlink our projects i.e. http://fileformats.archiveteam.org/wiki/ICO <=> http://formats.kaitai.io/ico/ [16:55]
xmci'm 100% in favor of this [16:57]
***pizzaiolo has joined #archiveteam-bs [16:58]
greycatI thought of adding `meta/xref` key into Kaitai Struct format
that could be used to add links to other projects into any .ksy file
i.e. ico.ksy would include something like
meta:
xref:
archiveteam: http://fileformats.archiveteam.org/wiki/ICO
would "archiveteam" be OK name to reference archiveteam wiki with, or would you prefer something else?
[16:58]
DFJustinit's not really the archive team wiki it's just hosted on the same domain [17:02]
***brayden has quit IRC (Read error: Connection reset by peer)
brayden has joined #archiveteam-bs
swebb sets mode: +o brayden
[17:02]
xmcyeah, it's a separate wiki [17:04]
greycatDFJustin: Ok, then how should I address it and who can I contact about it? [17:04]
***SHODAN_UI has quit IRC (Remote host closed the connection) [17:04]
xmcusing the url is probably the best way to talk about it [17:04]
***bitBaron has joined #archiveteam-bs [17:06]
greycatfor example, I wonder if it would be acceptable to modify FormatInfo template there to add a link to .ksy spec at formats.kaitai.io, if it exists?
It already includes other interlinks, such as `pronom={{PRONOM|fmt/978}}`
[17:06]
xmci approve [17:06]
greycatShall we wait for some quorum / more opinions, or could I try to add it right away? [17:07]
xmci wouldn't expect to see dissent about this. go ahead and add it if you feel comfortable :) [17:08]
***kristian_ has joined #archiveteam-bs [17:11]
joepie91greycat: ohai, somebody pointed me at kaitai a while ago, quite an interesting project :)
greycat: one remark that's not really archiveteam-related: the format documentation is not very googleable
[17:18]
greycatOk, here we go: http://fileformats.archiveteam.org/wiki/ICO [17:19]
joepie91googling for eg. datatypes (u1 and whatnot) yielded no useful results [17:19]
greycatWhat do you think about such a link?
joepie91: What could be done to improve that?
joepie91: Techincally, user guide fully describes that - http://doc.kaitai.io/user_guide.html#_basic_data_types
[17:19]
joepie91greycat: hmm. I think I'd call it "Kaitai Spec" instead of "Formal Spec" -- a lot of people will probably interpret "Formal" as "Official" (even if that's not what it means)
greycat: well, I'm not sure why it's not googleable in the first place
haven't looked into it too deeply
just registered it in the back of my mind :)
greycat: and yeah, I'm aware that the documentation exists, it just wasn't easy to find :p
[17:20]
greycatoh bummer...
I wonder how many people encounter these problems...
[17:22]
joepie91I really quite like the project though
considering giving it a shot in terms of specifying the OpenTTD wire protocol
I have a hand-written implementation in JS right now, but it'd be nicer to have a proper declarative spec
[17:22]
greycatabout "Kaitai Spec" - technically, binary parsing project is called "Kaitai Struct", "Kaitai" is the name of team [17:23]
joepie91right, "Kaitai Struct Spec" then :p [17:23]
greycatwe have other projects beside "Kaitai Struct" ;)
"Kaitai Struct Spec" would be probably too long and it will stretch the box... Would be it alright? May be "Declarative Spec" then?
[17:23]
joepie91greycat: kaitai.io yields kaitai struct though :) [17:24]
greycatOr it will be misleading as well? [17:24]
joepie91I wouldn't worry about stretching the box too much
and you can leave out the .ksy if you call it 'kaitai struct'
main goal is to have accurate information on the site, aesthetics comes second :P
[17:24]
greycatOk, then I'll make it "Kaitai Struct Spec" [17:26]
joepie91greycat: also, fwiw, archiveteam is very much structured as "if you want to get something done, do it" so I wouldn't worry too much about doing something that somebody doesn't like - if they don't like it, they'll tell you and/or fix it, but there's no formal structure on who decides what [17:27]
greycathttp://fileformats.archiveteam.org/wiki/ICO <= ok, here we go
well, at the very least, "Kaitai Struct" is a very new project compared to massive amount of data already gathered by archiveteam
so I thought it would be polite to ask :)
[17:27]
joepie91looks good :)
and yeah, no worries
[17:28]
greycatok, next step: link from formats.kaitai.io to Just Solve Format Problem wiki %)
how shall I name it? I've already got it that just "archiveteam" is a bad idea...
[17:31]
arkiverI saw imzy was at 50 items/min
not sure if I set that
did someone set it to that here?
[17:32]
joepie91greycat: "File Format wiki" is a fairly commonly used shorthand description, I think
arkiver: thoughts on a short name for the fileformats wiki?
[17:32]
MrRadararkiver: Kaz reduced it down from 200 earlier [17:32]
arkiverwhy? [17:32]
MrRadarThey said we were overloading their DB server at 200 [17:33]
arkiverjoepie91: ffwiki ? :P what do we need it for? [17:33]
MrRadar08:44 < timmc> arkiver: Maybe try halving the rate on Imyz?
08:46 < timmc> The current rate is hundreds of concurrent sessions and the DB connection is getting overloaded.
[17:33]
arkiverI can skip just a few more URLs, but not much more
for imzy
I'll do that
[17:33]
greycatjoepie91: "File Format wiki" it's *very* ambiguous :( [17:33]
Crusherarkiver : any idea why an imzy worker would spit a wget -6 error? [17:34]
arkiverno
do you have a log?
[17:34]
xmcgreycat: "just solve the problem" "justsolve" ? [17:34]
greycattechnically, it's called "Just Solve the File Format Problem"... [17:34]
CrusherNot on me, what's the best way to get one from a worker VM? [17:34]
greycatmay be "jstffp"? or "justsolve" might be just fine and non-ambiguous... [17:35]
joepie91arkiver: a reference from another file format spec site to the wiki [17:35]
Crusherwget -6 is an authentication problem btw [17:35]
joepie91arkiver: per format
greycat: idk if it's ambiguous, are there any other file format wikis? :P
[17:35]
xmci feel like there probably are ... [17:41]
Lord_NighSketchCow: is there an "archival standard" for ripping vhs tapes? [17:44]
***bitBaron has quit IRC (Read error: Connection reset by peer) [17:52]
Crusherarkiver : what's the best way to get a log out of the VM?
I remember what the error said
[17:52]
***bitBaron has joined #archiveteam-bs
kristian_ has quit IRC (Read error: Operation timed out)
[17:53]
greycatjoepie91: There is http://fileformats.wikia.com, http://wiki.osdev.org/, http://www.fileformat.info/format/, etc, etc
joepie91: http://wiki.xentax.com/ is file format wiki mostly on game-related file formats
There are many of them
[18:02]
SketchCowArchival Standard is either MPEG2, or a lossless AVI if you're an idiot [18:08]
***Crusher_ has joined #archiveteam-bs
Crusher has quit IRC (Bye)
[18:09]
..... (idle for 20mn)
pikhqThere's better codecs, but MPEG-2 is "good enough" and you can be pretty sure it'll still be supported in future decades.
Makes sense.
[18:29]
Crusher_Hmm. Something made my processor decide to clock down to 190 MHz, and I have no idea why...
>_>
Hold that thought, I just got a bluescreen printing line by line
[18:33]
JAAWhat kind of a machine do you have? I don't think I even *could* clock down my processor to 190 MHz if I tried. [18:35]
MrRadar♫ Tonight we're gonna party like it's 1999! ♫ [18:35]
pikhqSlightly old mobile chips can do that, IIRC. [18:36]
Crusher_It's a Pentium G3258
It *was* going happily at 4.2 GHz
[18:37]
JAAWhat's "slightly old", pikhq? [18:38]
Crusher_LOL I think the Intel fan just HCF'd
It posted, so I probably didn't bake the poor chip
[18:38]
MrRadarChips these days are pretty good at thermal throttling before they release the magic smoke
IIRC the last mainstream desktop CPU that didn't have thermal protection was the AthlonXP
As seen in this classic video: https://www.youtube.com/watch?v=YYQSHXNFvUk
[18:39]
pikhqJAA: Haswell, apparently. :) [18:40]
SketchCowI know everyone was hanging on the edge of their seats on this one, but
https://archive.org/details/sega_sms_library?&sort=publicdate&page=1
[18:41]
Crusher_Well, I just found out Haswell will do whatever it takes to not die. [18:42]
SketchCowI have fixed all the screenshots so they're actually screenshots, and I'm now changing the right items to the Japanese model and re-rendering screenshots because not only does the SMS not boot, sometimes it WILL boot in the wrong region and show a default game. [18:42]
Crusher_Turns out a molex got tangled in the blade [18:42]
pikhqCrusher_: Seems a good feature for a CPU to have. [18:43]
SketchCowOt
It's only down to the Gs with the re-render, so there's a pile of the default game shots, now being redone.
[18:43]
Crusher_pikhq : but then Intel doesn't get to charge me for a new one :P [18:44]
***SHODAN_UI has joined #archiveteam-bs [18:45]
Crusher_Well, it's back up to 4.2 GHz with no complaints, though I don't even want to think about what that frying electronics smell was
42°C idle, Good I didn't bake the thermal paste out of existence
[18:52]
timmcthat was the smell of the "inject random errors" fuse melting open [18:58]
Crusher_Lol
I'm surprised how long it operated without the fan
It was down to convection cooling, there's no case fan circulating where the CPU is
Such is the downside of squeezing a full ATX motherboard in a cardboard box
http://imgur.com/a/DWjPu
[18:59]
timmcthis answers the question of the molex flopping around randomly near the fan [19:06]
JAA:-| [19:07]
timmc1 star - shoebox contained GPU, not shoes
This is like the grown-up version of my home server, which is a raspberry pi in a cardboard box.
[19:07]
Crusher_Lol. My pi-webserver is actually sitting about a foot away [19:08]
***nyany has joined #archiveteam-bs [19:09]
Crusher_It was acting as a webserver for pushing files to people and for grade 11 compsci
It was like the 90s again, a whole class of people with no stylistic taste making really cringy HTML pages
[19:09]
timmctakes me back [19:10]
JAAtimmc: Still better than that shoebox from Reddit though. 2/5 from my side. [19:11]
timmcLink? [19:11]
***th1x has quit IRC (Quit: Leaving) [19:11]
JAAI'd rather not. [19:11]
Kalrothhaha [19:11]
timmcoh
*that* one
no link thank you
[19:12]
JAAYep, that one :-D [19:12]
Crusher_When it's closed, it has an additional fan taking in air to the 570 through that hole in the front [19:12]
Kalrothyeah, no link needed, that image is burned in [19:12]
Crusher_It's not pretty, but it works*
*so long as the molex doesn't flop into the fan
[19:12]
timmcOne of these days I'm going to take apart an old laptop and nail all the parts to a plywood board while keeping them connected. Maybe I can even make it look nice. [19:13]
Crusher_Lol [19:13]
timmcvivisection, but for Acers [19:14]
Crusher_In case anyone was worried about stability, this is NOT the machine I'm using for scraping :P [19:14]
Kalrothi had an bitcoin (well scrypt coin) mining setup with a motherboard and psu in the open and 4 GPUs hanging from a grill-like setup .. :) [19:14]
timmc"no, this is the one that hosts example.com" [19:14]
Kalroth"that one? yeah, thats just 8.8.8.8" [19:15]
JAA"The cloud." [19:17]
Lord_NighSketchCow: i meant more time base correction and what sort of vcr should be used to play/digitize it [19:22]
***Crusher has joined #archiveteam-bs
Crusher_ has quit IRC (Read error: Connection reset by peer)
Crusher_ has joined #archiveteam-bs
Crusher has quit IRC (Read error: Connection reset by peer)
Crusher has joined #archiveteam-bs
Crusher_ has quit IRC (Read error: Connection reset by peer)
[19:30]
JAA has quit IRC (leaving)
JAA has joined #archiveteam-bs
[19:40]
.......... (idle for 45mn)
greeniehey sorry I havent been around, but wanted to remind folks that Imzy will be going entirely offline tonight, probably around midnight EST. Not sure if folks here got stuff they wanted to off the site. [20:26]
CrusherWe're only a third of the way [20:28]
arkiverI'm updating the scripts to skip more URLs, hopefully we'll go faster then [20:29]
CrusherIs there a non warrior script for imzy?
Seeing as the warrior hates me
[20:29]
MrRadarYou just run the script directly
There are instructions in the github repository's readme
[20:30]
CrusherIt's on git? Ok. [20:31]
MrRadarhttps://github.com/archiveteam/imzy-grab [20:31]
arkivergreenie: for me posts are returning 404
or well, 301 and then redirected to the not found page
timmc ^
[20:38]
greeniehm yeah I'm not getting loads to happen on web (too lazy to open my console to see whats up). I'm no longer staff but will ping them and see whats up. There might not be much they can do at this point, I'm not sure. [20:40]
CrusherI'm getting an error during get-wget-lua of Lua not found
I installed Lua 5.1 from repo but it's still getting stuck on it
[20:41]
JAACrusher: liblua5.1-dev [20:45]
CrusherThanks, installing [20:46]
kisspunchthis is basically in the same boat as pop-unders (that is, load the real page in the popup, switch the main page to an add) in terms of filth, except that I'm less sure it's genuinely on purpose
(re: JS needed to load)
[20:48]
Crusher#$&$ Now it won't even tell me why it can't compile [20:58]
***C4K3_ has quit IRC (Read error: Operation timed out)
C4K3 has joined #archiveteam-bs
[21:00]
CrusherAh ha
It's failing on all
all-am*
Specifically: "cat: css.c: no such file or directory"
[21:05]
JAACrusher: Based on my notes, here's how I installed it: packages needed: python3 gcc libssl-dev lua5.1 liblua5.1-dev make autoconf m4 flex
python3 get-pip.py --user
.local/bin/pip install --upgrade --user seesaw
I edited get-wget-lua.sh to force it to use OpenSSL.
This was on Debian Jessie.
My get-wget-lua.sh edit was CONFIGURE_SSL_OPT="--with-ssl=openssl" above the WGET_DOWNLOAD_URL line.
[21:10]
CrusherKk
YESSS! Got it. Thanks JAA
[21:12]
JAA:-) [21:17]
***thuban3 has quit IRC (Read error: Operation timed out) [21:18]
CrusherHmm. Any chance we could raise the rate limiter?
If I understood the message earlier correctly, it's now or never
[21:19]
timmcCrusher: Last night the DB got overloaded, with hundreds of concurrent sessions. [21:20]
***thuban3 has joined #archiveteam-bs [21:20]
timmcSo... running out of time, but there's only so much headroom in the app too. [21:21]
CrusherRight now it looks like nobody is getting anything through...
The tracker is getting dusty at this point xD
[21:21]
timmcWait, does that mean nobody is requesting stuff right now?
or that people are, but are getting errors
[21:23]
CrusherI'm just getting placarded with rate limiting messages
And I haven't seen the tracker update in awhile now
Their site is still functioning (quickly) if anyone's Wondering
I don't have a single thread active
[21:23]
timmcMaybe therate limiter is at 0. :-) [21:30]
JAAarkiver: ^ [21:30]
arkiverhi
yes
posts were dead last time I checked
all 404
ah
not dead anymore
[21:30]
CrusherI was about to say [21:31]
arkiverrestarted
I'll make an update to the scripts soon
[21:31]
CrusherI see a webpage full of cat posts xD [21:31]
arkiverso be prepared to upgrade [21:31]
CrusherDefine upgrade? [21:31]
***SHODAN_UI has quit IRC (Remote host closed the connection) [21:31]
CrusherNo wait, it's working now [21:31]
***Crusher_ has joined #archiveteam-bs [21:34]
JAAYeah, but he's adding more ignore patterns, I think. [21:35]
arkiveryeah [21:35]
***Crusher has quit IRC (Read error: Connection reset by peer) [21:36]
timmcWhat's the error rate like at this point? [21:36]
***Crusher has joined #archiveteam-bs
Crusher_ has quit IRC (Read error: Connection reset by peer)
[21:36]
arkiverhttp://www.bbc.com/news/technology-40326544
:)
[21:38]
CrusherWoot woot! [21:39]
***Crusher_ has joined #archiveteam-bs [21:47]
JAANice. [21:49]
***Crusher has quit IRC (Ping timeout: 492 seconds) [21:50]
Crusher_ has quit IRC (Bye)
Crusher has joined #archiveteam-bs
[21:58]
.... (idle for 18mn)
kristian_ has joined #archiveteam-bs [22:16]
SanquiDoes anybody know of https://anarchivism.org/ ? [22:23]
***Jonison has quit IRC (Read error: Connection reset by peer) [22:30]
schbirid has quit IRC (Quit: Leaving) [22:36]
timmcarkiver: Do you think imzy will finish in time at this rate? [22:37]
arkiverno
but I'm trying to get the rate up
[22:37]
timmcWhat do you think the rate would need to be? [22:38]
arkiverhard to say [22:38]
timmc(assuming you don't filter out any more URLs) [22:38]
arkivera big problem is that many items seem to fail
so those need to be requeued again
and we still have to do communities
we're now only doing posts and users
but we're getting as much as possible
and might maybe just make it
arkiver is afk for an hour
[22:38]
timmcok [22:39]
***Stiletto has quit IRC (Ping timeout: 260 seconds) [22:39]
timmcI wonder if just focusing on posts or on users would help, since otherwise every comment and post is covered twice. [22:41]
..... (idle for 20mn)
***Stilett0 has joined #archiveteam-bs [23:01]
....... (idle for 31mn)
CrusherI don't suppose we can just ask for their hard drives :P [23:32]
***Crusher has quit IRC (Bye) [23:32]
SketchCowHELLO I FEEL BETTER [23:33]
jrwrHELLO SketchCow
What happened?
Also SketchCow, if IA complains about 163.172.128.219 pegging the crap out of their Wayback API, its just the newsgrabber project
[23:34]
***Crusher has joined #archiveteam-bs [23:36]
MrRadararkiver: Imzy's servers appear overwhelmed again. Most of my jobs are failing due to connection timeouts of 5xx errors [23:44]
***bitBaron has quit IRC (Read error: Connection reset by peer) [23:50]
MrRadarAnd now they seem better [23:50]
***bitBaron has joined #archiveteam-bs [23:52]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)