#archiveteam-bs 2017-12-03,Sun

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)

WhoWhatWhen
ola_norskJAA: does the mktemp write anything to disk?
JAA: https://github.com/xrgtn/nullfs
JAA: or is it simply gone whatever that's save to it?
[00:06]
JAAWell, depends on where $TMPDIR is located. [00:08]
ola_norsktmp? [00:08]
JAAI don't think you can use a nullfs. It seems that wget has to save the file to disk and then read it again to extract the images etc.
(For whatever reason...)
[00:08]
ola_norskok [00:08]
JAAYeah, /tmp is a ramdisk on my machine, but I don't know what it is on yours. [00:08]
ola_norskmy /tmp is on SSD :/ [00:09]
CoolCanuk:o [00:09]
ola_norsklol [00:09]
JAAI specifically moved /tmp to a ramfs when I migrated to SSDs to avoid the wear. :-D [00:10]
ola_norskaye
i have a 256mb ramdisk mounted, any way to make that vary in size as needed?
[00:10]
JAAI believe tmpfs does that.
You specify the maximum size (default is some fraction of the physical RAM you have), and it adapts as necessary.
It will always look like a FS having that size, but it won't occupy RAM if you don't use it.
[00:12]
ola_norskbut, with mktemp is the stuff written (and then removed) ?
to disk i mean
or does it _vanish_
[00:13]
JAAIt writes to disk (or RAM or whatever file system you're using).
You could do the same thing with mkdir and chmod, I believe.
[00:14]
ola_norski don't mind it writing to the ramdisk, as long as it's gone the moment after
i've been a fool not having tmp and swamp in ram
swap*
[00:15]
JAAYou'll have to delete it yourself. [00:16]
ola_norskdamnit. And here IA deletes 1300 files of CoolCanuk's stuff..That's not going to fly, there must be a better way [00:19]
CoolCanukLOL [00:19]
JAAHaha
CoolCanuk: Have you figured out yet why those files disappeared?
[00:20]
CoolCanukthey say I deleted them
I told them there was no way I accidentally deleted 1300+ files
[00:20]
***wp494_ has joined #archiveteam-bs
wp494 has quit IRC (Ping timeout: 492 seconds)
[00:22]
ola_norskJAA: --delete-after
This option tells Wget to delete every single file it downloads, after having done so. It is useful for pre-fetching popular pages through a proxy, e.g.:
i think that is it
[00:34]
JAAYeah, maybe. I didn't try that when I saw that it didn't work with -O /dev/null. But maybe it does the processing before deleting the file.
Still use a tmpfs or whatever though to avoid the useless writes to disk.
[00:36]
ola_norskaye [00:37]
Lastet ned: 54 filer, 1,7M på 2,0s (875 KB/s) downloaded .. that was just on the web.archive.org/save/ request
so no doubt it's working somewhat, not a single file came from elsewhere
and the folder is 'clean as a whistle' after with --delete-after
[00:46]
JAASweet! [00:49]
ola_norskwget --delete-after --page-requisites -e robots=off 'https://web.archive.org/save/https://twitter.com/hashtag/bogus?f=tweets'
gold stuff, and thanks for the help :D
[00:49]
JAAAnytime [00:53]
ola_norskthough...is this how waybackmachine eventually turns to captcha? :d
i hope not hehe
[00:54]
JAAThey won't. [00:55]
ola_norskwould it make a difference on their end if i limited the download rate?
or would it just cause shit to take longer
[00:56]
JAAThat won't make any difference. [00:57]
ola_norskk [00:57]
JAA(I think.) [00:57]
ola_norski wouldn't mind having it run as multiple slow-as-hell threads on my end. I'm deleteing it, after all
the requests is what makes it save the stuff
[00:59]
JAAAmazon.fr is discontinuing unlimited cloud storage as well. I guess .co.uk is the last one offering it now? I now accept bets for how long it will take until they announce their changes... [01:03]
ola_norskthe internet is over
(as we know it)
JAA: wasnt it you who posted this https://youtu.be/1VD_pJOFnZ0 :d
[01:04]
JAAYeah, when you posted the vid.me link. I didn't actually watch it. [01:07]
ola_norskaye :d
the internet is d0000000med :d
[01:07]
***ZexaronS has quit IRC (Read error: Connection reset by peer) [01:09]
ola_norskJAA: it's good stuff though ;) [01:10]
JAAYeah, I should watch it probably. [01:11]
ola_norskaye. "Technological Normalcy"...
JAA: i'm getting to sauced to be in chats. Have a good one, and thanks so much again for the help! Skål!
[01:11]
***ola_norsk has quit IRC (øl øl og meira øl)
ZexaronS has joined #archiveteam-bs
[01:13]
pizzaiolo has quit IRC (Remote host closed the connection) [01:25]
wp494_ is now known as wp494 [01:34]
.... (idle for 19mn)
superkuh has quit IRC (Quit: the neuronal action potential is an electrical manipulation of reversible abrupt phase changes in the lipid bilaye) [01:53]
ola_norsk has joined #archiveteam-bs [02:05]
ola_norskJAA: that wget is capturing images like a mofo!! https://web.archive.org/web/20171203012756/https:/twitter.com/hashtag/bogus?f=tweets
i feel sorry the one having to run pngquant on the shit :d
for*
[02:08]
***ola_norsk has quit IRC (Leaving) [02:18]
..... (idle for 20mn)
odemgJAA, did we archivebot http://forums.ncix.com/ after there bankruptcy announcement? It wont be up much longer. [02:38]
***ola_norsk has joined #archiveteam-bs [02:45]
ola_norskJAA: would it be naughty to set --read-timeout intentionally low you think? to prevent even downloading any stuff at the requests?
basically it would cause IA to go 'saved it! here you go!..and then 'nah, too late!' :d
the request could run quicker then, i THINK, but save outgoing bandwith for IA..
i'm hoping the latter is more beneficial
[02:46]
CoolCanuka read timeout has the potential to not reach archive.org at all, which is risky. Your request might time out before the request is made. [02:57]
ola_norskhmm
that makes sense
i feel it's kind of rotten of me to first request shit, and the moment i get it it's deleted :d
thats all
i feel a little like i'm setting fire to stuff that's handed to me. Not sure how else to word it.
[02:58]
***Mateon1 has quit IRC (Read error: Operation timed out)
Mateon1 has joined #archiveteam-bs
[03:11]
ola_norskthe reason i thought it might be an idea is it being an option in 'Downloading' https://www.gnu.org/software/wget/manual/wget.html#Download-Options
CoolCanuk: could it apply to 'connection timing' ?
JAA, CoolCanuk: damnit. I suck as this shit. But doesn't this wget stuff warrant a wiki page? :/
damn saturdays! It's the hardest day to be drunk at! Illuminati, canunks, ramdisks and damn GET REQUESTS... https://youtu.be/Fm-y9UsJ2L4
CoolCanuk: you know tragically hip right?
[03:14]
***ola_norsk has quit IRC (sudo killall hexchat) [03:25]
CoolCanukyes it needs a wiki page
if you can actually get it to work, pls start one
yes i know tragically hip
[03:34]
..... (idle for 23mn)
could someone make sure I'm doing this right? https://github.com/ArchiveTeam/NewsGrabber-Services/pulls newsgrabber IRC is more dead than ask.com [03:58]
..... (idle for 23mn)
hopefully it's right. Because im going HAM on it.. and want to do it right the first time [04:21]
***qw3rty116 has joined #archiveteam-bs
qw3rty115 has quit IRC (Read error: Operation timed out)
[04:28]
....... (idle for 33mn)
Zalgo has joined #archiveteam-bs [05:05]
CoolCanukwelcome :) [05:05]
Zalgo:p [05:08]
CoolCanukif anyone knows of news sites, and doesn't feel comfortable editing github, just pm me it and ill add it to a pull request for #newsgrabber I have a fairly automated process now. [05:10]
Zalgoi believe i read something about dnainfo shutting down if we didnt archive that one already [05:12]
CoolCanukdnateam appears to be providing us with an archive, but I haven't heard much about it recently http://archiveteam.org/index.php?title=DNAinfo [05:13]
Zalgooki [05:14]
CoolCanukomg. dnainfo's 404 page is hilarious :P https://www.dnainfo.com/new-york/blah [05:15]
Zalgolol [05:16]
***Zalgo has quit IRC ()
Stilett0 has quit IRC (Ping timeout: 250 seconds)
[05:27]
CoolCanuknotice anything about this article? http://www.elliotlakestandard.ca/2017/12/02/elliot-lake-santa-claus-parade [05:33]
godanei'm uploaing the GDC Keynote Nintendo 1999 Tape
*uploading
[05:34]
CoolCanukOMG
hurry
jk jk
[05:34]
***zalgo has joined #archiveteam-bs [05:35]
CoolCanukI mean, it's on YouTube, but your version is better [05:35]
godaneits also on youtube [05:35]
zalgoi archived the whole of markipliers channel to test out a new 2TB drive
kek
[05:36]
CoolCanukexcellent
what brand of drive? :o
[05:36]
zalgoseagate [05:36]
godanethe opening will make's it longer then that one [05:36]
CoolCanuknice [05:36]
zalgoalthough i bet if the videos were just a little bit more compressed they could fit on a 1tb
but quality is more important ;/
[05:37]
CoolCanukwebm it ;p
https://www.seagate.com/ca/en/consumer/backup/duet-amazon-drive/ is pretty cool. Too bad my upload speed sucks
[05:37]
zalgoid love to archive some twitch channels but that would take an absolutely absurd amount of storage, more than i have [05:39]
***Stilett0 has joined #archiveteam-bs
zalgo has quit IRC (Remote host closed the connection)
[05:46]
zalgo has joined #archiveteam-bs [05:51]
godanehttp://www.dtic.mil is giving me 'connection has time out ' error
so i will have to wait to upload more dtic docs
[05:58]
CoolCanukname resolution failed for me
you broke it, dane..
http://apacs.dtic.mil/ doesnt work either
godane: do ANY mil sites work for you?
army.mil , navy.mil don't work for me
https://health.mil/ does
[05:59]
godanehttps://github.com/esonderegger/dotmil-domains/blob/master/dotmil-domains.csv
af.mil is not working either
[06:04]
CoolCanukoh.. I had to use www.army.mil
www.af.mil works
what a terrible set up. they should redirect..
[06:04]
you def broke it dane :P :P [06:10]
zalgoive reach a new level of stupidity, archiving /r/DataHoarder on reddit
kek
[06:15]
CoolCanukgodane: my friend is getting the default apache page. weird
nameservers dont respond. no A record found either
[06:16]
heh godane look what I found https://archive.fart.website/bin/irclogger_log/archiveteam-bs?date=2017-01-21,Sat&raw=on [06:28]
zalgoim gonna be going to sleep soon, any ideas for an overnight backup or no [06:32]
CoolCanukhm
dunno
[06:34]
zalgoalright, cya tomorrow, hopefully we can start on vidme then [06:40]
CoolCanukhopefully :D [06:41]
........................... (idle for 2h13mn)
***schbirid has joined #archiveteam-bs [08:54]
.... (idle for 16mn)
dashcloud has quit IRC (Read error: Connection reset by peer) [09:10]
.... (idle for 18mn)
CoolCanuk has quit IRC (Quit: Connection closed for inactivity)
dashcloud has joined #archiveteam-bs
[09:28]
......... (idle for 40mn)
jschwart has joined #archiveteam-bs [10:10]
.......... (idle for 45mn)
pizzaiolo has joined #archiveteam-bs [10:55]
Jusque_ has joined #archiveteam-bs
Jusque has quit IRC (Read error: Operation timed out)
Jusque_ is now known as Jusque
[11:08]
BlueMaxim has quit IRC (Read error: Connection reset by peer) [11:19]
Jusque has quit IRC (Ping timeout: 260 seconds)
Jusque has joined #archiveteam-bs
[11:28]
JAAodemg: Doesn't look like we did. [11:42]
***zalgo has quit IRC (Read error: Operation timed out) [11:52]
.................. (idle for 1h26mn)
ZexaronS has quit IRC (Read error: Connection reset by peer)
ZexaronS has joined #archiveteam-bs
[13:18]
...... (idle for 25mn)
alfie has left Textual IRC Client: www.textualapp.com [13:44]
............ (idle for 59mn)
superkuh has joined #archiveteam-bs
nottom has joined #archiveteam-bs
[14:43]
.... (idle for 17mn)
alex___ has joined #archiveteam-bs [15:02]
....... (idle for 30mn)
jspiros_well, shit [15:32]
***jspiros_ is now known as jspiros [15:32]
jspirosI had forgotten about A Prairie Home Companion until a moment ago when I was reading some old post that referenced it, so I go to see about archiving old episodes of it
turns out just a few days ago there was yet another misconduct scandal that led to the distribution contract being severed
so almost the entire back history of the show is no longer available from the source
hopefully in a few months the rightsholders will find a new way to distribute them...
[15:33]
***dashcloud has quit IRC (Remote host closed the connection) [15:39]
SketchCowYes [15:52]
***CoolCanuk has joined #archiveteam-bs [15:55]
jspiroscompletely unrelated BS: I recently bought a legitimate copy of a TV show from the distributor, and received DVD-Rs (burned with the correct content) in the package
turns out the distributor felt it was wiser for them to bring disc "production" in-house
nice violation of the DVD-Video trademark (which was on the packaging) terms
jspiros returned it and found an older properly-pressed copy on eBay
[15:59]
...... (idle for 27mn)
***TheLovina has quit IRC (Read error: Connection reset by peer) [16:28]
zalgo has joined #archiveteam-bs [16:41]
dashcloud has joined #archiveteam-bs [16:51]
CoolCanukinteresting [16:54]
........ (idle for 37mn)
***schbirid has quit IRC (Read error: Operation timed out) [17:31]
CoolCanukanyone alive :P ? [17:35]
***tklk has joined #archiveteam-bs [17:37]
alex___ has left [17:47]
..... (idle for 23mn)
Ing3b0rg has quit IRC (Ping timeout: 260 seconds) [18:10]
.... (idle for 19mn)
ZexaronS has quit IRC (Read error: Connection reset by peer)
Ing3b0rg has joined #archiveteam-bs
ZexaronS has joined #archiveteam-bs
[18:29]
schbirid has joined #archiveteam-bs [18:37]
..... (idle for 20mn)
zalgo has quit IRC (Remote host closed the connection)
ZexaronS has quit IRC (Read error: Connection reset by peer)
ZexaronS has joined #archiveteam-bs
[18:57]
ZexaronS has quit IRC (Read error: Connection reset by peer)
ZexaronS has joined #archiveteam-bs
Pixi has quit IRC (Quit: Pixi)
[19:13]
....... (idle for 33mn)
Pixi has joined #archiveteam-bs [19:48]
zalgo has joined #archiveteam-bs [20:02]
..... (idle for 21mn)
Lord_Nigh has quit IRC (Read error: Operation timed out)
Lord_Nigh has joined #archiveteam-bs
Jusque has quit IRC (ZNC - http://znc.in)
Jusque has joined #archiveteam-bs
[20:23]
..... (idle for 20mn)
nottom has quit IRC (Quit: Page closed) [20:48]
dashcloudhi there
CoolCanuk: I know you wanted to be able to do item uploads efficiently through the commandline- here's a good way to do that, complete with a way to pre-fill the metadata using a csv: https://github.com/kngenie/ias3upload
[20:50]
CoolCanukomg thank you !! :O
so much
I am so happy now :)
[20:52]
zalgoheya [20:53]
dashcloudactually- here's a more recent one, in case that doesn't work: https://github.com/vmbrasseur/iaupload with a metadata example here: https://github.com/vmbrasseur/iaupload/blob/master/md.yaml.example [20:54]
***zalgo has quit IRC (Read error: Operation timed out) [21:02]
zalgo has joined #archiveteam-bs [21:07]
....... (idle for 30mn)
schbirid has quit IRC (Quit: Leaving)
Odd0002 has quit IRC (Quit: ZNC - http://znc.in)
[21:37]
..... (idle for 21mn)
Odd0002 has joined #archiveteam-bs [22:02]
BlueMaxim has joined #archiveteam-bs [22:14]
CoolCanukBreach of the day: PayPal Says 1.6 Million Customer Details Stolen in Breach at Canadian Subsidiary [22:18]
...... (idle for 25mn)
***fie has quit IRC (Ping timeout: 246 seconds) [22:43]
Froggingthat's nice
"A review of TIO’s network has identified a potential compromise of personally identifiable information for approximately 1.6 million customers. The PayPal platform is not impacted in any way, as the TIO systems are completely separate from the PayPal network, and PayPal’s customers’ data remains secure."
>a potential compromise of personally identifiable information for approximately 1.6 million customers
>PayPal’s customers’ data remains secure.
so whose customer data was compromised
[22:51]
***SN4T14_ has joined #archiveteam-bs
Polylith_ has joined #archiveteam-bs
[22:53]
Froggingokay, it was TIO customers' data. I've never heard of that company. Hopefully it's not one of those shadow-behemoths that everyone is involved with and nobody knows about [22:55]
***fie has joined #archiveteam-bs
ppsym has joined #archiveteam-bs
i0npulse has quit IRC (Ping timeout: 248 seconds)
purplebot has quit IRC (Ping timeout: 248 seconds)
Polylith has quit IRC (Ping timeout: 248 seconds)
dboard2 has quit IRC (Ping timeout: 248 seconds)
medowar has quit IRC (Ping timeout: 248 seconds)
SN4T14 has quit IRC (Ping timeout: 248 seconds)
Rai-chan has quit IRC (Ping timeout: 248 seconds)
PurpleSym has quit IRC (Ping timeout: 248 seconds)
ppsym is now known as PurpleSym
medowar has joined #archiveteam-bs
purplebot has joined #archiveteam-bs
i0npulse has joined #archiveteam-bs
Rai-chan has joined #archiveteam-bs
ndiddy has quit IRC (Read error: Operation timed out)
dboard2 has joined #archiveteam-bs
[22:56]
ZexaronS has quit IRC (Read error: Connection reset by peer)
ZexaronS has joined #archiveteam-bs
[23:14]
..... (idle for 20mn)
joepie91lol: https://investor.paypal-corp.com/releasedetail.cfm?releaseid=1048334
"PayPal Holdings, Inc. (Nasdaq: PYPL) announced that TIO Networks (TIO), a publicly traded company PayPal acquired in July 2017, has suspended operations to protect TIO's customers. This suspension of services is a result of PayPal's discovery of security vulnerabilities on the TIO platform and issues with TIO's data security program that do not adhere to PayPal's information security standards. TIO is not integrated into PayPal's platform. "
[23:35]
TIO is a leading multi-channel bill payment processor in North America and processed more than $7 billion USD in consumer bill payments in fiscal 2016. TIO serves 16 million consumer bill pay accounts* and offers convenient solutions for expedited bill payment services to financially underserved consumers.
sounds like one of those companies that handles pay-your-bills-at-the-supermarket type bills, Frogging
"TIO integrates with the back office of billing systems to accept, validate, and collect payments via self-service kiosk, retail walk-in, mobile, and web solutions."
yep
separate company, acquired by paypal in july of this year, kept operating independently, vulnerabilities found, a month later a likely compromise was found - going to guess that it predates paypal's acquisition and somebody didn't do their due diligence during acquisition talks
[23:41]
***Stilett0 has quit IRC () [23:49]
CoolCanukfml [23:58]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)