[00:06] JAA: does the mktemp write anything to disk? [00:06] JAA: https://github.com/xrgtn/nullfs [00:07] JAA: or is it simply gone whatever that's save to it? [00:08] Well, depends on where $TMPDIR is located. [00:08] tmp? [00:08] I don't think you can use a nullfs. It seems that wget has to save the file to disk and then read it again to extract the images etc. [00:08] (For whatever reason...) [00:08] ok [00:08] Yeah, /tmp is a ramdisk on my machine, but I don't know what it is on yours. [00:09] my /tmp is on SSD :/ [00:09] :o [00:09] lol [00:10] I specifically moved /tmp to a ramfs when I migrated to SSDs to avoid the wear. :-D [00:10] aye [00:11] i have a 256mb ramdisk mounted, any way to make that vary in size as needed? [00:12] I believe tmpfs does that. [00:12] You specify the maximum size (default is some fraction of the physical RAM you have), and it adapts as necessary. [00:12] It will always look like a FS having that size, but it won't occupy RAM if you don't use it. [00:13] but, with mktemp is the stuff written (and then removed) ? [00:13] to disk i mean [00:13] or does it _vanish_ [00:14] It writes to disk (or RAM or whatever file system you're using). [00:14] You could do the same thing with mkdir and chmod, I believe. [00:15] i don't mind it writing to the ramdisk, as long as it's gone the moment after [00:16] i've been a fool not having tmp and swamp in ram [00:16] swap* [00:16] You'll have to delete it yourself. [00:19] damnit. And here IA deletes 1300 files of CoolCanuk's stuff..That's not going to fly, there must be a better way [00:19] LOL [00:20] Haha [00:20] CoolCanuk: Have you figured out yet why those files disappeared? [00:20] they say I deleted them [00:20] I told them there was no way I accidentally deleted 1300+ files [00:22] *** wp494_ has joined #archiveteam-bs [00:25] *** wp494 has quit IRC (Ping timeout: 492 seconds) [00:34] JAA: --delete-after [00:34] This option tells Wget to delete every single file it downloads, after having done so. It is useful for pre-fetching popular pages through a proxy, e.g.: [00:35] i think that is it [00:36] Yeah, maybe. I didn't try that when I saw that it didn't work with -O /dev/null. But maybe it does the processing before deleting the file. [00:37] Still use a tmpfs or whatever though to avoid the useless writes to disk. [00:37] aye [00:46] Lastet ned: 54 filer, 1,7M på 2,0s (875 KB/s) downloaded .. that was just on the web.archive.org/save/ request [00:47] so no doubt it's working somewhat, not a single file came from elsewhere [00:47] and the folder is 'clean as a whistle' after with --delete-after [00:49] Sweet! [00:49] wget --delete-after --page-requisites -e robots=off 'https://web.archive.org/save/https://twitter.com/hashtag/bogus?f=tweets' [00:49] gold stuff, and thanks for the help :D [00:53] Anytime [00:54] though...is this how waybackmachine eventually turns to captcha? :d [00:54] i hope not hehe [00:55] They won't. [00:56] would it make a difference on their end if i limited the download rate? [00:56] or would it just cause shit to take longer [00:57] That won't make any difference. [00:57] k [00:57] (I think.) [00:59] i wouldn't mind having it run as multiple slow-as-hell threads on my end. I'm deleteing it, after all [01:02] the requests is what makes it save the stuff [01:03] Amazon.fr is discontinuing unlimited cloud storage as well. I guess .co.uk is the last one offering it now? I now accept bets for how long it will take until they announce their changes... [01:04] the internet is over [01:04] (as we know it) [01:06] JAA: wasnt it you who posted this https://youtu.be/1VD_pJOFnZ0 :d [01:07] Yeah, when you posted the vid.me link. I didn't actually watch it. [01:07] aye :d [01:08] the internet is d0000000med :d [01:09] *** ZexaronS has quit IRC (Read error: Connection reset by peer) [01:10] JAA: it's good stuff though ;) [01:11] Yeah, I should watch it probably. [01:11] aye. "Technological Normalcy"... [01:12] JAA: i'm getting to sauced to be in chats. Have a good one, and thanks so much again for the help! Skål! [01:13] *** ola_norsk has quit IRC (øl øl og meira øl) [01:14] *** ZexaronS has joined #archiveteam-bs [01:25] *** pizzaiolo has quit IRC (Remote host closed the connection) [01:34] *** wp494_ is now known as wp494 [01:53] *** superkuh has quit IRC (Quit: the neuronal action potential is an electrical manipulation of reversible abrupt phase changes in the lipid bilaye) [02:05] *** ola_norsk has joined #archiveteam-bs [02:08] JAA: that wget is capturing images like a mofo!! https://web.archive.org/web/20171203012756/https:/twitter.com/hashtag/bogus?f=tweets [02:12] i feel sorry the one having to run pngquant on the shit :d [02:14] for* [02:18] *** ola_norsk has quit IRC (Leaving) [02:38] JAA, did we archivebot http://forums.ncix.com/ after there bankruptcy announcement? It wont be up much longer. [02:45] *** ola_norsk has joined #archiveteam-bs [02:46] JAA: would it be naughty to set --read-timeout intentionally low you think? to prevent even downloading any stuff at the requests? [02:47] basically it would cause IA to go 'saved it! here you go!..and then 'nah, too late!' :d [02:50] the request could run quicker then, i THINK, but save outgoing bandwith for IA.. [02:52] i'm hoping the latter is more beneficial [02:57] a read timeout has the potential to not reach archive.org at all, which is risky. Your request might time out before the request is made. [02:58] hmm [02:58] that makes sense [03:00] i feel it's kind of rotten of me to first request shit, and the moment i get it it's deleted :d [03:00] thats all [03:02] i feel a little like i'm setting fire to stuff that's handed to me. Not sure how else to word it. [03:11] *** Mateon1 has quit IRC (Read error: Operation timed out) [03:11] *** Mateon1 has joined #archiveteam-bs [03:14] the reason i thought it might be an idea is it being an option in 'Downloading' https://www.gnu.org/software/wget/manual/wget.html#Download-Options [03:16] CoolCanuk: could it apply to 'connection timing' ? [03:19] JAA, CoolCanuk: damnit. I suck as this shit. But doesn't this wget stuff warrant a wiki page? :/ [03:23] damn saturdays! It's the hardest day to be drunk at! Illuminati, canunks, ramdisks and damn GET REQUESTS... https://youtu.be/Fm-y9UsJ2L4 [03:24] CoolCanuk: you know tragically hip right? [03:25] *** ola_norsk has quit IRC (sudo killall hexchat) [03:34] yes it needs a wiki page [03:35] if you can actually get it to work, pls start one [03:35] yes i know tragically hip [03:58] could someone make sure I'm doing this right? https://github.com/ArchiveTeam/NewsGrabber-Services/pulls newsgrabber IRC is more dead than ask.com [04:21] hopefully it's right. Because im going HAM on it.. and want to do it right the first time [04:28] *** qw3rty116 has joined #archiveteam-bs [04:32] *** qw3rty115 has quit IRC (Read error: Operation timed out) [05:05] *** Zalgo has joined #archiveteam-bs [05:05] welcome :) [05:08] :p [05:10] if anyone knows of news sites, and doesn't feel comfortable editing github, just pm me it and ill add it to a pull request for #newsgrabber I have a fairly automated process now. [05:12] i believe i read something about dnainfo shutting down if we didnt archive that one already [05:13] dnateam appears to be providing us with an archive, but I haven't heard much about it recently http://archiveteam.org/index.php?title=DNAinfo [05:14] oki [05:15] omg. dnainfo's 404 page is hilarious :P https://www.dnainfo.com/new-york/blah [05:16] lol [05:27] *** Zalgo has quit IRC () [05:31] *** Stilett0 has quit IRC (Ping timeout: 250 seconds) [05:33] notice anything about this article? http://www.elliotlakestandard.ca/2017/12/02/elliot-lake-santa-claus-parade [05:34] i'm uploaing the GDC Keynote Nintendo 1999 Tape [05:34] *uploading [05:34] OMG [05:34] hurry [05:34] jk jk [05:35] *** zalgo has joined #archiveteam-bs [05:35] I mean, it's on YouTube, but your version is better [05:35] its also on youtube [05:36] i archived the whole of markipliers channel to test out a new 2TB drive [05:36] kek [05:36] excellent [05:36] what brand of drive? :o [05:36] seagate [05:36] the opening will make's it longer then that one [05:36] nice [05:37] although i bet if the videos were just a little bit more compressed they could fit on a 1tb [05:37] but quality is more important ;/ [05:37] webm it ;p [05:37] https://www.seagate.com/ca/en/consumer/backup/duet-amazon-drive/ is pretty cool. Too bad my upload speed sucks [05:39] id love to archive some twitch channels but that would take an absolutely absurd amount of storage, more than i have [05:46] *** Stilett0 has joined #archiveteam-bs [05:46] *** zalgo has quit IRC (Remote host closed the connection) [05:51] *** zalgo has joined #archiveteam-bs [05:58] http://www.dtic.mil is giving me 'connection has time out ' error [05:58] so i will have to wait to upload more dtic docs [05:59] name resolution failed for me [05:59] you broke it, dane.. [06:00] http://apacs.dtic.mil/ doesnt work either [06:02] godane: do ANY mil sites work for you? [06:03] army.mil , navy.mil don't work for me [06:03] https://health.mil/ does [06:04] https://github.com/esonderegger/dotmil-domains/blob/master/dotmil-domains.csv [06:04] af.mil is not working either [06:04] oh.. I had to use www.army.mil [06:05] www.af.mil works [06:05] what a terrible set up. they should redirect.. [06:10] you def broke it dane :P :P [06:15] ive reach a new level of stupidity, archiving /r/DataHoarder on reddit [06:15] kek [06:16] godane: my friend is getting the default apache page. weird [06:19] nameservers dont respond. no A record found either [06:28] heh godane look what I found https://archive.fart.website/bin/irclogger_log/archiveteam-bs?date=2017-01-21,Sat&raw=on [06:32] im gonna be going to sleep soon, any ideas for an overnight backup or no [06:34] hm [06:34] dunno [06:40] alright, cya tomorrow, hopefully we can start on vidme then [06:41] hopefully :D [08:54] *** schbirid has joined #archiveteam-bs [09:10] *** dashcloud has quit IRC (Read error: Connection reset by peer) [09:28] *** CoolCanuk has quit IRC (Quit: Connection closed for inactivity) [09:30] *** dashcloud has joined #archiveteam-bs [10:10] *** jschwart has joined #archiveteam-bs [10:55] *** pizzaiolo has joined #archiveteam-bs [11:08] *** Jusque_ has joined #archiveteam-bs [11:12] *** Jusque has quit IRC (Read error: Operation timed out) [11:12] *** Jusque_ is now known as Jusque [11:19] *** BlueMaxim has quit IRC (Read error: Connection reset by peer) [11:28] *** Jusque has quit IRC (Ping timeout: 260 seconds) [11:29] *** Jusque has joined #archiveteam-bs [11:42] odemg: Doesn't look like we did. [11:52] *** zalgo has quit IRC (Read error: Operation timed out) [13:18] *** ZexaronS has quit IRC (Read error: Connection reset by peer) [13:19] *** ZexaronS has joined #archiveteam-bs [13:44] *** alfie has left Textual IRC Client: www.textualapp.com [14:43] *** superkuh has joined #archiveteam-bs [14:45] *** nottom has joined #archiveteam-bs [15:02] *** alex___ has joined #archiveteam-bs [15:32] well, shit [15:32] *** jspiros_ is now known as jspiros [15:33] I had forgotten about A Prairie Home Companion until a moment ago when I was reading some old post that referenced it, so I go to see about archiving old episodes of it [15:34] turns out just a few days ago there was yet another misconduct scandal that led to the distribution contract being severed [15:34] so almost the entire back history of the show is no longer available from the source [15:34] hopefully in a few months the rightsholders will find a new way to distribute them... [15:39] *** dashcloud has quit IRC (Remote host closed the connection) [15:52] Yes [15:55] *** CoolCanuk has joined #archiveteam-bs [15:59] completely unrelated BS: I recently bought a legitimate copy of a TV show from the distributor, and received DVD-Rs (burned with the correct content) in the package [16:00] turns out the distributor felt it was wiser for them to bring disc "production" in-house [16:01] nice violation of the DVD-Video trademark (which was on the packaging) terms [16:01] * jspiros returned it and found an older properly-pressed copy on eBay [16:28] *** TheLovina has quit IRC (Read error: Connection reset by peer) [16:41] *** zalgo has joined #archiveteam-bs [16:51] *** dashcloud has joined #archiveteam-bs [16:54] interesting [17:31] *** schbirid has quit IRC (Read error: Operation timed out) [17:35] anyone alive :P ? [17:37] *** tklk has joined #archiveteam-bs [17:47] *** alex___ has left [18:10] *** Ing3b0rg has quit IRC (Ping timeout: 260 seconds) [18:29] *** ZexaronS has quit IRC (Read error: Connection reset by peer) [18:29] *** Ing3b0rg has joined #archiveteam-bs [18:29] *** ZexaronS has joined #archiveteam-bs [18:37] *** schbirid has joined #archiveteam-bs [18:57] *** zalgo has quit IRC (Remote host closed the connection) [18:58] *** ZexaronS has quit IRC (Read error: Connection reset by peer) [18:59] *** ZexaronS has joined #archiveteam-bs [19:13] *** ZexaronS has quit IRC (Read error: Connection reset by peer) [19:15] *** ZexaronS has joined #archiveteam-bs [19:15] *** Pixi has quit IRC (Quit: Pixi) [19:48] *** Pixi has joined #archiveteam-bs [20:02] *** zalgo has joined #archiveteam-bs [20:23] *** Lord_Nigh has quit IRC (Read error: Operation timed out) [20:23] *** Lord_Nigh has joined #archiveteam-bs [20:27] *** Jusque has quit IRC (ZNC - http://znc.in) [20:28] *** Jusque has joined #archiveteam-bs [20:48] *** nottom has quit IRC (Quit: Page closed) [20:50] hi there [20:52] CoolCanuk: I know you wanted to be able to do item uploads efficiently through the commandline- here's a good way to do that, complete with a way to pre-fill the metadata using a csv: https://github.com/kngenie/ias3upload [20:52] omg thank you !! :O [20:52] so much [20:53] I am so happy now :) [20:53] heya [20:54] actually- here's a more recent one, in case that doesn't work: https://github.com/vmbrasseur/iaupload with a metadata example here: https://github.com/vmbrasseur/iaupload/blob/master/md.yaml.example [21:02] *** zalgo has quit IRC (Read error: Operation timed out) [21:07] *** zalgo has joined #archiveteam-bs [21:37] *** schbirid has quit IRC (Quit: Leaving) [21:41] *** Odd0002 has quit IRC (Quit: ZNC - http://znc.in) [22:02] *** Odd0002 has joined #archiveteam-bs [22:14] *** BlueMaxim has joined #archiveteam-bs [22:18] Breach of the day: PayPal Says 1.6 Million Customer Details Stolen in Breach at Canadian Subsidiary [22:43] *** fie has quit IRC (Ping timeout: 246 seconds) [22:51] that's nice [22:52] "A review of TIO’s network has identified a potential compromise of personally identifiable information for approximately 1.6 million customers. The PayPal platform is not impacted in any way, as the TIO systems are completely separate from the PayPal network, and PayPal’s customers’ data remains secure." [22:52] >a potential compromise of personally identifiable information for approximately 1.6 million customers [22:52] >PayPal’s customers’ data remains secure. [22:52] so whose customer data was compromised [22:53] *** SN4T14_ has joined #archiveteam-bs [22:53] *** Polylith_ has joined #archiveteam-bs [22:55] okay, it was TIO customers' data. I've never heard of that company. Hopefully it's not one of those shadow-behemoths that everyone is involved with and nobody knows about [22:56] *** fie has joined #archiveteam-bs [22:57] *** ppsym has joined #archiveteam-bs [22:59] *** i0npulse has quit IRC (Ping timeout: 248 seconds) [22:59] *** purplebot has quit IRC (Ping timeout: 248 seconds) [22:59] *** Polylith has quit IRC (Ping timeout: 248 seconds) [22:59] *** dboard2 has quit IRC (Ping timeout: 248 seconds) [22:59] *** medowar has quit IRC (Ping timeout: 248 seconds) [22:59] *** SN4T14 has quit IRC (Ping timeout: 248 seconds) [22:59] *** Rai-chan has quit IRC (Ping timeout: 248 seconds) [22:59] *** PurpleSym has quit IRC (Ping timeout: 248 seconds) [22:59] *** ppsym is now known as PurpleSym [23:00] *** medowar has joined #archiveteam-bs [23:00] *** purplebot has joined #archiveteam-bs [23:00] *** i0npulse has joined #archiveteam-bs [23:02] *** Rai-chan has joined #archiveteam-bs [23:02] *** ndiddy has quit IRC (Read error: Operation timed out) [23:06] *** dboard2 has joined #archiveteam-bs [23:14] *** ZexaronS has quit IRC (Read error: Connection reset by peer) [23:15] *** ZexaronS has joined #archiveteam-bs [23:35] lol: https://investor.paypal-corp.com/releasedetail.cfm?releaseid=1048334 [23:35] "PayPal Holdings, Inc. (Nasdaq: PYPL) announced that TIO Networks (TIO), a publicly traded company PayPal acquired in July 2017, has suspended operations to protect TIO's customers. This suspension of services is a result of PayPal's discovery of security vulnerabilities on the TIO platform and issues with TIO's data security program that do not adhere to PayPal's information security standards. TIO is not integrated into PayPal's platform. " [23:41] TIO is a leading multi-channel bill payment processor in North America and processed more than $7 billion USD in consumer bill payments in fiscal 2016. TIO serves 16 million consumer bill pay accounts* and offers convenient solutions for expedited bill payment services to financially underserved consumers. [23:42] sounds like one of those companies that handles pay-your-bills-at-the-supermarket type bills, Frogging [23:42] "TIO integrates with the back office of billing systems to accept, validate, and collect payments via self-service kiosk, retail walk-in, mobile, and web solutions." [23:42] yep [23:43] separate company, acquired by paypal in july of this year, kept operating independently, vulnerabilities found, a month later a likely compromise was found - going to guess that it predates paypal's acquisition and somebody didn't do their due diligence during acquisition talks [23:49] *** Stilett0 has quit IRC () [23:58] fml