| Time |
Nickname |
Message |
|
00:01
🔗
|
|
closure has joined #archiveteam-bs |
|
00:35
🔗
|
|
closure has quit IRC (Read error: Connection reset by peer) |
|
00:54
🔗
|
|
closure has joined #archiveteam-bs |
|
01:00
🔗
|
|
closure has quit IRC (Read error: Connection reset by peer) |
|
01:01
🔗
|
|
closure has joined #archiveteam-bs |
|
01:08
🔗
|
|
Stilett0 has quit IRC (Read error: Connection reset by peer) |
|
01:08
🔗
|
|
Stilett0 has joined #archiveteam-bs |
|
01:25
🔗
|
godane |
SketchCow: 623 pdfs ED files are saved doing the audit for ERIC archive |
|
01:29
🔗
|
godane |
hear are all the pdfs added so far by doing this audit : https://archive.org/details/@chris85?and[]=addeddate%3A2018-09+eric&and[]=subject%3A%22ERIC+Archive%22 |
|
01:35
🔗
|
|
closure has quit IRC (Read error: Connection reset by peer) |
|
01:35
🔗
|
|
closure has joined #archiveteam-bs |
|
02:00
🔗
|
|
closure has quit IRC (Read error: Connection reset by peer) |
|
02:01
🔗
|
|
closure has joined #archiveteam-bs |
|
02:09
🔗
|
|
yoyo11 has joined #archiveteam-bs |
|
02:10
🔗
|
|
yoyo11 has quit IRC (Quit: Page closed) |
|
02:11
🔗
|
moufu |
is there a reason warrior projects use wget-lua 1.14.* instead of 1.17.1? I've noticed 1.14.* doesn't extract urls from html5 media tags |
|
02:21
🔗
|
ivan |
probably because no one updated it |
|
02:22
🔗
|
ivan |
wpull might be a better option though |
|
02:34
🔗
|
|
closure has quit IRC (Read error: Connection reset by peer) |
|
02:35
🔗
|
|
closure has joined #archiveteam-bs |
|
02:59
🔗
|
|
closure has quit IRC (Read error: Connection reset by peer) |
|
03:05
🔗
|
|
closure has joined #archiveteam-bs |
|
03:30
🔗
|
Flashfire |
Can someone scrape https://mobile.twitter.com/BJTMarts |
|
03:35
🔗
|
|
closure has quit IRC (Read error: Connection reset by peer) |
|
03:35
🔗
|
ivan |
Flashfire: https://gist.githubusercontent.com/ivan/2e6a0431bb41745060ea8b5ddf218852/raw/e1a9cf3e0f9a4132ed467d92d8c12f9d16955684/gistfile1.txt |
|
03:36
🔗
|
ivan |
Flashfire: you can run snscrape on your computer, no? |
|
03:39
🔗
|
|
archodg__ has joined #archiveteam-bs |
|
03:39
🔗
|
Flashfire |
Yeah but don’t have access to it right now |
|
03:41
🔗
|
|
archodg_ has quit IRC (Ping timeout: 252 seconds) |
|
03:41
🔗
|
|
odemg has quit IRC (Ping timeout: 260 seconds) |
|
03:54
🔗
|
|
odemg has joined #archiveteam-bs |
|
04:27
🔗
|
|
closure has joined #archiveteam-bs |
|
04:32
🔗
|
|
closure has quit IRC (Read error: Connection reset by peer) |
|
04:47
🔗
|
|
closure has joined #archiveteam-bs |
|
04:59
🔗
|
|
closure has quit IRC (Read error: Connection reset by peer) |
|
05:00
🔗
|
|
closure_ has joined #archiveteam-bs |
|
05:37
🔗
|
|
closure has joined #archiveteam-bs |
|
05:37
🔗
|
|
closure_ has quit IRC (Read error: Connection reset by peer) |
|
06:05
🔗
|
|
m007a83 has quit IRC (Quit: Fuck you Comcast) |
|
06:06
🔗
|
|
closure has quit IRC (Read error: Operation timed out) |
|
06:08
🔗
|
|
Mateon1 has quit IRC (Ping timeout: 268 seconds) |
|
06:09
🔗
|
|
Mateon1 has joined #archiveteam-bs |
|
06:10
🔗
|
|
Petri152 has quit IRC (Read error: Operation timed out) |
|
06:10
🔗
|
|
JAA has quit IRC (Read error: Operation timed out) |
|
06:10
🔗
|
|
zyphlar has quit IRC (Read error: Operation timed out) |
|
06:11
🔗
|
|
jspiros has quit IRC (Read error: Operation timed out) |
|
06:13
🔗
|
|
bithippo has quit IRC (Ping timeout: 246 seconds) |
|
06:15
🔗
|
|
c4rc4s has quit IRC (Read error: Operation timed out) |
|
06:17
🔗
|
|
m007a83 has joined #archiveteam-bs |
|
06:23
🔗
|
|
tuluu has quit IRC (irc.Prison.NET west.us.hub) |
|
06:23
🔗
|
|
Coderjo has quit IRC (irc.Prison.NET west.us.hub) |
|
06:23
🔗
|
|
Dimtree has quit IRC (irc.Prison.NET west.us.hub) |
|
06:23
🔗
|
|
jut_ has quit IRC (irc.Prison.NET west.us.hub) |
|
06:23
🔗
|
|
moufu has quit IRC (irc.Prison.NET west.us.hub) |
|
06:23
🔗
|
|
TC01 has quit IRC (irc.Prison.NET west.us.hub) |
|
06:23
🔗
|
|
nyaomi has quit IRC (irc.Prison.NET west.us.hub) |
|
06:23
🔗
|
|
Sanqui has quit IRC (irc.Prison.NET west.us.hub) |
|
06:23
🔗
|
|
joepie91_ has quit IRC (irc.Prison.NET west.us.hub) |
|
06:24
🔗
|
|
logchfoo1 starts logging #archiveteam-bs at Sat Sep 29 06:24:52 2018 |
|
06:24
🔗
|
|
logchfoo1 has joined #archiveteam-bs |
|
06:25
🔗
|
|
zino_ has joined #archiveteam-bs |
|
06:25
🔗
|
|
phirephl- has joined #archiveteam-bs |
|
06:25
🔗
|
|
Polylith_ has joined #archiveteam-bs |
|
06:33
🔗
|
|
Atom-- has joined #archiveteam-bs |
|
06:33
🔗
|
|
achip has joined #archiveteam-bs |
|
06:36
🔗
|
|
Atom__ has quit IRC (Ping timeout: 252 seconds) |
|
06:54
🔗
|
|
achip has quit IRC (west.us.hub irc.Prison.NET) |
|
07:11
🔗
|
|
c4rc4s has joined #archiveteam-bs |
|
07:11
🔗
|
|
zyphlar has joined #archiveteam-bs |
|
07:11
🔗
|
|
Petri152 has joined #archiveteam-bs |
|
07:12
🔗
|
|
JAA has joined #archiveteam-bs |
|
07:12
🔗
|
|
swebb sets mode: +o JAA |
|
07:12
🔗
|
|
bakJAA sets mode: +o JAA |
|
07:15
🔗
|
|
jspiros has joined #archiveteam-bs |
|
07:22
🔗
|
|
achip has joined #archiveteam-bs |
|
08:33
🔗
|
|
C4K3_ has joined #archiveteam-bs |
|
08:35
🔗
|
|
C4K3 has quit IRC (Read error: Operation timed out) |
|
09:00
🔗
|
HCross |
JAA: good news. I got Brozzler working |
|
09:18
🔗
|
godane |
SketchCow: i'm up to 2018-06-27 with kpfa collection |
|
09:19
🔗
|
godane |
trying to get that up to date |
|
09:20
🔗
|
HCross |
or not.. xrdp just decided it didnt want to do anything |
|
09:39
🔗
|
|
Guestiiii has joined #archiveteam-bs |
|
09:40
🔗
|
Guestiiii |
hello |
|
09:40
🔗
|
Flashfire |
hi |
|
09:41
🔗
|
Guestiiii |
I'm trying to get "backups" of pages I bookmark. Is wpull the/a right method for that? |
|
09:41
🔗
|
Guestiiii |
I've seen it has little activity recently, and I'm having trouble actually making it work, so I'm not sure I'm doing it right. |
|
09:42
🔗
|
Guestiiii |
(And sorry if this is the wrong place to ask. I've come here since wpull's docs link to this place) |
|
09:42
🔗
|
Flashfire |
try grabsite made by our own ivan |
|
09:42
🔗
|
Flashfire |
IVAN WE HAVE SOMEONE TO SEE YOU |
|
09:43
🔗
|
Guestiiii |
I've seen that too, it's essentially a nice wrapper around wpull? |
|
09:43
🔗
|
ivan |
yes |
|
09:44
🔗
|
Guestiiii |
I've also seen something acting as a proxy, but losing the original ssl certs is not so good for me |
|
09:44
🔗
|
ivan |
you can for example dump your bookmarks to a list of URLs and use grab-site --1 -i FILE |
|
09:45
🔗
|
Guestiiii |
oh, that's neat |
|
09:45
🔗
|
ivan |
here is a bookmarklet for saving pages to wayback that you may or may not be able to access in the future javascript:(function(){window.open('https://web.archive.org/save/'+(''+window.location));})(); |
|
09:46
🔗
|
ivan |
I usually combine that with compulsive ctrl-s'ing to .mhtml files in Chrome |
|
09:46
🔗
|
Guestiiii |
can you configure firefox so that it calls grabsite whenever you bookmark stuff? |
|
09:46
🔗
|
Guestiiii |
and what about wpull not seeming to be maintained anymore? is that any trouble for you? |
|
09:47
🔗
|
ivan |
you can dump your history from Firefox https://www.gwern.net/Archiving-URLs#batch-job-downloads |
|
09:47
🔗
|
ivan |
wpull 1.2.3 works pretty well for me other than slowness |
|
09:47
🔗
|
Guestiiii |
ok, good |
|
09:47
🔗
|
ivan |
also it not being a headless browser but that's a whole other deal |
|
09:48
🔗
|
Guestiiii |
which means pages don't really look the same? |
|
09:48
🔗
|
ivan |
it doesn't run javascript and scroll pages and click on ajax things |
|
09:48
🔗
|
Guestiiii |
anyway, thanks for grab-site, I'll give it a run! |
|
09:49
🔗
|
Guestiiii |
yeah |
|
09:49
🔗
|
Guestiiii |
but I saw this "phantom-js" thing |
|
09:49
🔗
|
ivan |
if you want to save pages post-javascript-execution both Firefox and Chrome have a ctrl-s that can do that |
|
09:49
🔗
|
ivan |
chromebot in #archivebot can do it too |
|
09:49
🔗
|
ivan |
also webrecorder.io |
|
09:50
🔗
|
Guestiiii |
mmh, so pages that are blank pre-js don't yield good archives with wpull? |
|
09:50
🔗
|
ivan |
usually not |
|
09:50
🔗
|
ivan |
sometimes they are blank but have content blanked with css |
|
09:50
🔗
|
Guestiiii |
ah, shit |
|
09:50
🔗
|
Guestiiii |
and what does the wayback machine work? |
|
09:50
🔗
|
Guestiiii |
*how |
|
09:51
🔗
|
ivan |
wayback is totally insane, it copies javascript to their domain and tries to execute it there |
|
09:51
🔗
|
Guestiiii |
mmh, it's all very hacky it seems :) |
|
09:52
🔗
|
ivan |
I mean your browser lands on web.archive.org and tries to execute archived javascript files (sometimes coming a newer snapshot or from liveweb) |
|
09:53
🔗
|
ivan |
sometimes it even works |
|
09:54
🔗
|
Guestiiii |
but then, why couldn't you do that with wpull ? I mean wpull downloads html+js+everything, and when you open the archive in browser, it runs js ? |
|
09:54
🔗
|
Flashfire |
Hold Ivan grabsite is a GUI wrapper for Wpull yes? Have you considered making a GUI for snscrape |
|
09:54
🔗
|
ivan |
Flashfire: I recommend getting good at the command line |
|
09:55
🔗
|
ivan |
command line operations can be composed and automated |
|
09:55
🔗
|
ivan |
Guestiiii: that will sometimes work but often not because javascript makes assumptions about paths and headers and origin policy |
|
09:55
🔗
|
Guestiiii |
I see |
|
09:56
🔗
|
ivan |
Flashfire: for example if you want something to happen after the current command finishes you can use ; or && or just use the typeahead buffer to type the next command |
|
09:57
🔗
|
Flashfire |
ivan I find the command line interface confusing to be honest not to mention High Sierra fucks over everything |
|
09:57
🔗
|
ivan |
there's no getting around getting good at the command line |
|
09:57
🔗
|
kiska |
You have access to that machine thats in LA |
|
09:57
🔗
|
kiska |
Try and get familiar with ubuntu |
|
09:59
🔗
|
Guestiiii |
(and maybe use fish instead of bash ) |
|
09:59
🔗
|
Flashfire |
Alright I might give dual booting a shot once I clear out my 100GB of error files from my desktop |
|
10:00
🔗
|
kiska |
Instead of doing ssh -d etc etc, for using that as a netflix proxy. Just do ssh flashfire@server... and mess around with shell |
|
10:00
🔗
|
ivan |
man don't tell a shell newbie to use fish |
|
10:00
🔗
|
Flashfire |
Dont ask how I got 100GB of error files if someone wants to find out for me be my guest |
|
10:01
🔗
|
Guestiiii |
ok :( |
|
10:01
🔗
|
ivan |
it's not a POSIX sh and you get screwed if you make it your default on your servers (which shell code do you send over?) |
|
10:01
🔗
|
Flashfire |
Wait .... |
|
10:01
🔗
|
Guestiiii |
why is that, though? |
|
10:01
🔗
|
Guestiiii |
yeah, ok |
|
10:01
🔗
|
Guestiiii |
but it's more *friendly* ! |
|
10:01
🔗
|
ivan |
you can get a similar-ish experience with zsh-autosuggestions |
|
10:02
🔗
|
Guestiiii |
yeah, but I meant the syntax is cleaner, the constructs, etc |
|
10:02
🔗
|
Guestiiii |
it's a more reasonable language imo |
|
10:02
🔗
|
ivan |
give it time |
|
10:07
🔗
|
kiska |
Flashfire: Here is a nice and concise tutorial on Linux and the shell environment http://docs.linuxtone.org/ebooks/Shell/Linux%20Shell%20Scripting%20Tutorial%20v2.0.pdf |
|
10:09
🔗
|
Flashfire |
Alright bookmarked that and what kiska sent me will look at it tommorow |
|
10:10
🔗
|
kiska |
Have fun! |
|
10:11
🔗
|
kiska |
And as I said before, you can't do any damage to the instance since you don't have sudo permissions |
|
10:11
🔗
|
Flashfire |
In the mean time I cant get snscrape to run lol |
|
10:11
🔗
|
kiska |
xD |
|
10:12
🔗
|
Guestiiii |
what's snscrape? |
|
10:13
🔗
|
Guestiiii |
ok I see |
|
10:13
🔗
|
kiska |
Flashfire on my server just do "pip3 install --user git+https://github.com/JustAnotherArchivist/snscrape.git" |
|
10:13
🔗
|
Flashfire |
A Social media scraper made by our own JAA |
|
10:13
🔗
|
kiska |
And it should install, then you can use it |
|
10:14
🔗
|
Flashfire |
Alright how on that server will I find my successful scrapes? |
|
10:15
🔗
|
ivan |
haha you can damage a linux without root |
|
10:15
🔗
|
Flashfire |
If Ivan says there is a way through dumb luck I will find it |
|
10:15
🔗
|
ivan |
Flashfire: snscrape writes to stdout |
|
10:15
🔗
|
ivan |
you can redirect stdout with > file |
|
10:16
🔗
|
Flashfire |
Ok so I screwed up with an error already xd |
|
10:16
🔗
|
Flashfire |
xD |
|
10:16
🔗
|
HCross |
JAA: Brozzler worked.. for about 5 minutes |
|
10:17
🔗
|
Flashfire |
See I screwed that up as well |
|
10:17
🔗
|
HCross |
until the Chrome process walked off somewhere, and bought the whole thing down |
|
10:32
🔗
|
HCross |
yeah, theres something on lolking that Brozzler doesnt like |
|
10:40
🔗
|
godane |
latest scan : reference-series-v5i4 |
|
10:40
🔗
|
godane |
latest scan : https://archive.org/details/reference-series-v5i4 |
|
10:41
🔗
|
Guestiiii |
gtg, thanks for the help! |
|
10:41
🔗
|
|
Guestiiii has quit IRC (Remote host closed the connection) |
|
10:44
🔗
|
godane |
all of my scans in the last few days : https://www.patreon.com/posts/digitize-scans-21717034 |
|
11:00
🔗
|
Flashfire |
Someone scrape https://mobile.twitter.com/marty_balin?lang=en |
|
11:00
🔗
|
Flashfire |
he died |
|
11:20
🔗
|
eientei95 |
JAA: Can you run your SNScraper on https://twitter.com/marty_balin |
|
11:25
🔗
|
eientei95 |
dxrt: When did JAA release their scraper? |
|
11:25
🔗
|
kiska |
https://transfer.sh/o7AMJ/@mart_balin_twitter |
|
11:25
🔗
|
dxrt |
2-3 weeks ago |
|
11:26
🔗
|
eientei95 |
Mind linking it? |
|
11:27
🔗
|
kiska |
https://github.com/JustAnotherArchivist/snscrape |
|
11:27
🔗
|
eientei95 |
thx |
|
11:27
🔗
|
eientei95 |
TIL kiska == dxrt |
|
11:27
🔗
|
kiska |
xD |
|
11:27
🔗
|
kiska |
In bed so my typing speed is trash |
|
11:27
🔗
|
Flashfire |
http://destyy.com A URL shortener |
|
11:29
🔗
|
kiska |
Damn my slow typing speed... |
|
12:14
🔗
|
|
schbirid has joined #archiveteam-bs |
|
12:32
🔗
|
|
BartoCH has quit IRC (Ping timeout: 615 seconds) |
|
12:35
🔗
|
godane |
i found out i was missing episode 269 of simply kpop |
|
12:35
🔗
|
godane |
i have downloaded it and uploading it now |
|
12:51
🔗
|
JAA |
HCross: :-| Well, at least you're making some progress. |
|
12:51
🔗
|
HCross |
I ran it twice, both times it hit a URL and then chrome went "nope, fuck off" and died |
|
12:56
🔗
|
JAA |
Interesting. Do you know which URL, and can you reproduce it in a normal Chromium instance? |
|
12:56
🔗
|
HCross |
ill restart it and see if I can catch it |
|
12:57
🔗
|
HCross |
interesting.. restarted it and it seems to have picked up and carried on |
|
13:10
🔗
|
|
BlueMax has quit IRC (Read error: Connection reset by peer) |
|
13:13
🔗
|
HCross |
JAA: urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7fd744572c50>, 'Connection to ucs02.engageya.com timed out. (connect timeout=60)') |
|
13:23
🔗
|
|
RichardG has joined #archiveteam-bs |
|
13:30
🔗
|
JAA |
HCross: Huh, and that crashes Chromium? |
|
13:30
🔗
|
HCross |
yes |
|
13:30
🔗
|
JAA |
Wow |
|
13:38
🔗
|
HCross |
switched to actual proper google chrome |
|
13:43
🔗
|
|
BartoCH has joined #archiveteam-bs |
|
13:48
🔗
|
|
closure has joined #archiveteam-bs |
|
14:00
🔗
|
|
closure has quit IRC (Read error: Connection reset by peer) |
|
14:00
🔗
|
|
closure_ has joined #archiveteam-bs |
|
14:32
🔗
|
|
closure_ has quit IRC (Read error: Connection reset by peer) |
|
14:33
🔗
|
|
closure has joined #archiveteam-bs |
|
14:58
🔗
|
|
closure has quit IRC (Read error: Connection reset by peer) |
|
14:59
🔗
|
JAA |
So I just looked a bit into tian.yam.com, and it doesn't look good. |
|
15:00
🔗
|
JAA |
There's a sitemap at https://tian.yam.com/site/sitemapindex, but that's definitely incomplete. |
|
15:00
🔗
|
|
closure has joined #archiveteam-bs |
|
15:00
🔗
|
JAA |
The site relies on JS and doesn't work at all without it. |
|
15:00
🔗
|
kiska |
Their "sitemap" contains... a lot of subdomains |
|
15:00
🔗
|
JAA |
The posts themselves do though. |
|
15:01
🔗
|
JAA |
It seems that each user (?) has its own subdomain. |
|
15:01
🔗
|
JAA |
And we only have about 37 hours until the shutdown. |
|
15:01
🔗
|
kiska |
"An internal server error occurred." At the end of the sitemap... |
|
15:02
🔗
|
kiska |
Not good, the server is already having some issues |
|
15:03
🔗
|
JAA |
Oh right, issed that. |
|
15:03
🔗
|
JAA |
missed* |
|
15:03
🔗
|
kiska |
Here is another sitemap http://member.yam.com/SiteMap/?Type=1 |
|
15:03
🔗
|
kiska |
However its in Chinese... |
|
15:04
🔗
|
JAA |
Also, if those numbers in the post URLs are post IDs, there are over 200 million posts... |
|
15:05
🔗
|
jut_ |
Could a warrior project get through that in a day and a half? |
|
15:08
🔗
|
JAA |
If we already had everything set up and their servers can handle our DPoS (and don't ban us), maybe. It's over 1500 requests per second though, so that's quite likely to cause issues (and requires a fair amount of warriors). |
|
15:11
🔗
|
kiska |
Another sitemap: https://service.tian.yam.com/sitemaps_posts.xml |
|
15:11
🔗
|
kiska |
One more: https://service.tian.yam.com/sitemaps_albums.xml |
|
15:12
🔗
|
|
kevinYang has joined #archiveteam-bs |
|
15:13
🔗
|
JAA |
Yeah, it seems that there are two sitemaps per subdomain. |
|
15:14
🔗
|
kiska |
Let me try a baidu search to discover more sitemaps/subdomains |
|
15:26
🔗
|
kiska |
Using site:tian.yam.com on Google yields: "About 1,120,000 results (0.24 seconds) "... |
|
15:32
🔗
|
|
closure has quit IRC (Read error: Connection reset by peer) |
|
15:32
🔗
|
|
closure_ has joined #archiveteam-bs |
|
15:34
🔗
|
kiska |
Looks like tian.yam.com is using cloudflare... |
|
15:44
🔗
|
|
LogicalDa has joined #archiveteam-bs |
|
15:46
🔗
|
LogicalDa |
I'm trying to find out if the Warrior has the arbitrary code execution vulnerabilities that were discovered in Alpine Linux a couple weeks ago https://www.securityweek.com/code-execution-alpine-linux-impacts-containers |
|
15:51
🔗
|
kiska |
JAA: Discovered some subdomains using SecurityTrails |
|
16:01
🔗
|
|
closure_ has quit IRC (Read error: Connection reset by peer) |
|
16:04
🔗
|
|
closure has joined #archiveteam-bs |
|
16:24
🔗
|
kiska |
JAA: There seems to be a search feature that we can abuse... |
|
16:33
🔗
|
|
closure_ has joined #archiveteam-bs |
|
16:33
🔗
|
|
closure has quit IRC (Read error: Connection reset by peer) |
|
16:48
🔗
|
kiska |
JAA: On kiskaAus I have a FDNS archive, its named 2018-08-24-1535112288-fdns_a.json.gz it may have some tian.yam.com subdomains that we can discover |
|
16:49
🔗
|
JAA |
kiska: What's FDNS? |
|
16:50
🔗
|
kiska |
JAA: https://opendata.rapid7.com/sonar.fdns_v2/ |
|
16:50
🔗
|
kiska |
"This dataset contains the responses to DNS requests for all forward DNS names known by Rapid7's Project Sonar. " |
|
16:52
🔗
|
JAA |
Ah, nice. |
|
16:53
🔗
|
kiska |
So its on kiskaAus, however I don't know how to access this dataset |
|
16:54
🔗
|
kiska |
Ah data is structured like this "{"timestamp":"1535156791","name":"\tusa-stemorganics-com.myshopify.com","type":"cname","value":"shops.myshopify.com"}" |
|
16:55
🔗
|
kiska |
I did "zcat 2018-08-24-1535112288-fdns_a.json.gz | grep tian.yam.com" I may have caused the hdd to have a seizure xD |
|
16:56
🔗
|
JAA |
Hmm, so not the raw DNS data, I see. |
|
16:56
🔗
|
kiska |
Here is the first result of that search "{"timestamp":"1535113806","name":"1204697.tian.yam.com","type":"cname","value":"tian-web-1993807926.ap-northeast-1.elb.amazonaws.com"}" |
|
16:56
🔗
|
kiska |
This will take a while... |
|
16:57
🔗
|
kiska |
If you ssh into the pipeline, its running on Window 9 |
|
16:57
🔗
|
|
closure_ has quit IRC (Read error: Operation timed out) |
|
16:59
🔗
|
JAA |
kiska: Hmm, might be a good idea to redirect output to a file... |
|
16:59
🔗
|
kiska |
I've ^C it, can you do the redirection? |
|
17:00
🔗
|
kiska |
And now we wait for it to read 15GB of data at 15MB/s |
|
17:00
🔗
|
JAA |
\o/ |
|
17:01
🔗
|
|
closure has joined #archiveteam-bs |
|
17:01
🔗
|
kiska |
Note: This data is about 1 month old, but it should still be valid |
|
17:06
🔗
|
JAA |
kiska: Should I run it on my server instead? The download would be done in about 12 minutes, and I can probably grep it much faster since the HDD isn't used for much else at the moment. |
|
17:06
🔗
|
JAA |
grepping 1 GB takes ~50 seconds here. |
|
17:06
🔗
|
JAA |
decompressing and grepping* |
|
17:07
🔗
|
kiska |
Its not the hdd, its the CPU |
|
17:07
🔗
|
JAA |
Yeah, my CPU isn't used much either at the moment. |
|
17:07
🔗
|
kiska |
Neither is mine |
|
17:07
🔗
|
kiska |
The process is single threaded, so my X5570 IPC isn't that good |
|
17:08
🔗
|
kiska |
But I say let it run, and see what happens |
|
17:08
🔗
|
JAA |
Ah right, yeah nevermind then. |
|
17:09
🔗
|
JAA |
Yeah, not much slower than on my machine. 1 minute to decompress and grep 1 GB. |
|
17:11
🔗
|
kiska |
I am going to say, even if the entire data set is in memory, it wouldn't be faster |
|
17:13
🔗
|
JAA |
Yeah, decompression seems to be the bottleneck. |
|
17:15
🔗
|
kiska |
So I would say, let it keep doing what its doing, come back in about 15 minutes and see what happens |
|
17:15
🔗
|
kiska |
We probably can get a ton of urls to process this way |
|
17:17
🔗
|
JAA |
Yeah :-) |
|
17:17
🔗
|
kiska |
And for future projects as well |
|
17:18
🔗
|
JAA |
kiska: Uh, what does "tail: tiam.yam.com-fdns: file truncated" mean? |
|
17:18
🔗
|
JAA |
Window 10 |
|
17:18
🔗
|
JAA |
File's empty now. |
|
17:18
🔗
|
kiska |
... |
|
17:19
🔗
|
kiska |
Oh... I am running multiple ssh sessions... |
|
17:33
🔗
|
|
closure_ has joined #archiveteam-bs |
|
17:33
🔗
|
|
closure has quit IRC (Read error: Connection reset by peer) |
|
17:35
🔗
|
kiska |
JAA: It finished, and grabbed less urls than I expected |
|
17:40
🔗
|
|
ndiddy has joined #archiveteam-bs |
|
17:43
🔗
|
kiska |
Right... I am going to grab a beefier server and see how it processes that gzip |
|
17:56
🔗
|
|
Odd0002_ has joined #archiveteam-bs |
|
17:56
🔗
|
|
Jens has quit IRC (Read error: Operation timed out) |
|
17:57
🔗
|
|
actually_ has quit IRC (Read error: Operation timed out) |
|
17:57
🔗
|
|
faolingfa has quit IRC (Read error: Connection reset by peer) |
|
17:57
🔗
|
|
Odd0002 has quit IRC (Read error: Connection reset by peer) |
|
17:57
🔗
|
|
Odd0002_ is now known as Odd0002 |
|
17:57
🔗
|
|
fenn_ has joined #archiveteam-bs |
|
17:57
🔗
|
|
robogoat has quit IRC (Read error: Operation timed out) |
|
17:57
🔗
|
|
robogoat has joined #archiveteam-bs |
|
17:58
🔗
|
|
Muad-Dib has quit IRC (Ping timeout: 260 seconds) |
|
17:58
🔗
|
|
pikhq has joined #archiveteam-bs |
|
17:58
🔗
|
|
pikhq_ has quit IRC (Read error: Connection reset by peer) |
|
17:58
🔗
|
|
closure_ has quit IRC (Read error: Operation timed out) |
|
17:58
🔗
|
|
fenn has quit IRC (Read error: Connection reset by peer) |
|
17:59
🔗
|
|
obskyr has joined #archiveteam-bs |
|
17:59
🔗
|
|
atomicthu has quit IRC (Read error: Connection reset by peer) |
|
18:00
🔗
|
|
atomicthu has joined #archiveteam-bs |
|
18:00
🔗
|
|
faolingf_ has joined #archiveteam-bs |
|
18:01
🔗
|
|
closure has joined #archiveteam-bs |
|
18:02
🔗
|
|
omglolbah has quit IRC (Read error: Operation timed out) |
|
18:02
🔗
|
|
omglolbah has joined #archiveteam-bs |
|
18:03
🔗
|
|
LordNigh2 has joined #archiveteam-bs |
|
18:04
🔗
|
|
Lord_Nigh has quit IRC (Ping timeout: 633 seconds) |
|
18:04
🔗
|
|
LordNigh2 is now known as Lord_Nigh |
|
18:05
🔗
|
|
phuzion has quit IRC (Ping timeout: 633 seconds) |
|
18:07
🔗
|
|
thejsa_ has quit IRC (Ping timeout: 633 seconds) |
|
18:09
🔗
|
|
jrwr has quit IRC (Ping timeout: 633 seconds) |
|
18:10
🔗
|
|
jrwr has joined #archiveteam-bs |
|
18:10
🔗
|
|
thejsa has joined #archiveteam-bs |
|
18:10
🔗
|
|
faolingfa has joined #archiveteam-bs |
|
18:11
🔗
|
|
mundus20- has quit IRC (Ping timeout: 633 seconds) |
|
18:11
🔗
|
|
Muad-Dib has joined #archiveteam-bs |
|
18:11
🔗
|
|
mundus201 has joined #archiveteam-bs |
|
18:12
🔗
|
|
faolingf_ has quit IRC (Ping timeout: 633 seconds) |
|
18:12
🔗
|
|
ReimuHaku has quit IRC (Ping timeout: 633 seconds) |
|
18:17
🔗
|
|
Darkstar has quit IRC (Read error: Connection reset by peer) |
|
18:17
🔗
|
kiska |
JAA: Do you think I can request this data set? https://ant.isi.edu/datasets/requests.html |
|
18:18
🔗
|
kiska |
I believe I can get access to their dns dataset |
|
18:18
🔗
|
|
Darkstar has joined #archiveteam-bs |
|
18:20
🔗
|
|
bztoot has quit IRC (Ping timeout: 633 seconds) |
|
18:20
🔗
|
|
t2t2 has joined #archiveteam-bs |
|
18:21
🔗
|
|
wp494 has quit IRC (Read error: Operation timed out) |
|
18:21
🔗
|
|
wp494 has joined #archiveteam-bs |
|
18:22
🔗
|
|
phuzion has joined #archiveteam-bs |
|
18:26
🔗
|
|
Jens has joined #archiveteam-bs |
|
18:27
🔗
|
|
ReimuHaku has joined #archiveteam-bs |
|
18:29
🔗
|
|
ndiddy has quit IRC (Read error: Operation timed out) |
|
18:33
🔗
|
|
schbirid2 has joined #archiveteam-bs |
|
18:34
🔗
|
|
closure has quit IRC (Read error: Operation timed out) |
|
18:37
🔗
|
|
squires has quit IRC (Read error: Operation timed out) |
|
18:37
🔗
|
|
djsundog has quit IRC (Read error: Operation timed out) |
|
18:37
🔗
|
|
dxrt_ has quit IRC (Read error: Operation timed out) |
|
18:37
🔗
|
|
closure_ has joined #archiveteam-bs |
|
18:37
🔗
|
|
nightpool has quit IRC (Read error: Operation timed out) |
|
18:38
🔗
|
|
nightpool has joined #archiveteam-bs |
|
18:38
🔗
|
|
sep332 has quit IRC (Read error: Operation timed out) |
|
18:38
🔗
|
|
LogicalDa has quit IRC (Read error: Operation timed out) |
|
18:38
🔗
|
|
schbirid has quit IRC (Read error: Operation timed out) |
|
18:38
🔗
|
|
Albardin has quit IRC (Read error: Operation timed out) |
|
18:38
🔗
|
|
RedType has quit IRC (Read error: Operation timed out) |
|
18:39
🔗
|
|
beardicus has quit IRC (Read error: Operation timed out) |
|
18:39
🔗
|
|
ivan has quit IRC (Read error: Operation timed out) |
|
18:40
🔗
|
|
ivan has joined #archiveteam-bs |
|
18:43
🔗
|
|
kiska1 has quit IRC (Read error: Operation timed out) |
|
18:43
🔗
|
|
logchfoo1 has quit IRC (Ping timeout: 601 seconds) |
|
18:44
🔗
|
|
logchfoo2 starts logging #archiveteam-bs at Sat Sep 29 18:44:54 2018 |
|
18:44
🔗
|
|
logchfoo2 has joined #archiveteam-bs |
|
18:45
🔗
|
|
LogicalDa has joined #archiveteam-bs |
|
18:51
🔗
|
|
Mayonaise has quit IRC (Ping timeout: 600 seconds) |
|
18:53
🔗
|
|
REiN^ has quit IRC (Ping timeout: 600 seconds) |
|
18:54
🔗
|
|
unlobito has quit IRC (Ping timeout: 600 seconds) |
|
18:54
🔗
|
|
unlobito has joined #archiveteam-bs |
|
18:54
🔗
|
|
Mayonaise has joined #archiveteam-bs |
|
18:58
🔗
|
|
Albardin has joined #archiveteam-bs |
|
18:59
🔗
|
|
closure_ has quit IRC (Read error: Connection reset by peer) |
|
18:59
🔗
|
|
closure has joined #archiveteam-bs |
|
19:03
🔗
|
|
TigerbotH has quit IRC (Ping timeout: 600 seconds) |
|
19:04
🔗
|
|
PotcFdk has quit IRC (Ping timeout: 600 seconds) |
|
19:04
🔗
|
|
kiska1 has joined #archiveteam-bs |
|
19:05
🔗
|
|
beardicus has joined #archiveteam-bs |
|
19:05
🔗
|
|
closure has quit IRC (Read error: Operation timed out) |
|
19:06
🔗
|
|
zino_ is now known as zino |
|
19:12
🔗
|
JAA |
kiska: That looks interesting. I'll need to take a closer look at it later. |
|
19:13
🔗
|
|
ndiddy has joined #archiveteam-bs |
|
19:14
🔗
|
|
PhrackD has quit IRC (Ping timeout: 600 seconds) |
|
19:23
🔗
|
|
schbirid2 has quit IRC (Remote host closed the connection) |
|
19:24
🔗
|
|
REiN^ has joined #archiveteam-bs |
|
19:24
🔗
|
|
TigerbotH has joined #archiveteam-bs |
|
19:24
🔗
|
|
sep332 has joined #archiveteam-bs |
|
19:25
🔗
|
|
C4K3 has joined #archiveteam-bs |
|
19:26
🔗
|
|
closure has joined #archiveteam-bs |
|
19:27
🔗
|
|
squires has joined #archiveteam-bs |
|
19:28
🔗
|
|
PhrackD has joined #archiveteam-bs |
|
19:28
🔗
|
|
PotcFdk has joined #archiveteam-bs |
|
19:29
🔗
|
|
djsundog has joined #archiveteam-bs |
|
19:29
🔗
|
|
dxrt_ has joined #archiveteam-bs |
|
19:30
🔗
|
|
RedType has joined #archiveteam-bs |
|
19:31
🔗
|
|
closure has quit IRC (Read error: Operation timed out) |
|
19:33
🔗
|
|
closure has joined #archiveteam-bs |
|
20:00
🔗
|
|
closure has quit IRC (Read error: Connection reset by peer) |
|
20:00
🔗
|
|
closure_ has joined #archiveteam-bs |
|
20:10
🔗
|
godane |
so i won that ebay auction |
|
20:33
🔗
|
|
closure_ has quit IRC (Ping timeout: 268 seconds) |
|
21:03
🔗
|
|
closure has joined #archiveteam-bs |
|
21:32
🔗
|
|
closure has quit IRC (Read error: Operation timed out) |
|
21:36
🔗
|
|
closure_ has joined #archiveteam-bs |
|
21:37
🔗
|
|
n00b227 has joined #archiveteam-bs |
|
21:37
🔗
|
|
n00b227 has quit IRC (Client Quit) |
|
21:39
🔗
|
|
n00b985 has joined #archiveteam-bs |
|
21:39
🔗
|
n00b985 |
Not specifically for archiveteam, but how would I add files to an existing archive.org item with the cli? |
|
21:44
🔗
|
JAA |
n00b985: Simpy 'ia upload identifier files' as on the first upload/item creation. Only works if you have access to that item, obviously. |
|
21:52
🔗
|
n00b985 |
I tried that, however after uploading the item didn't display on the website (I waited a few hours), nor with ia list. Only the originally uploaded file existed. |
|
21:59
🔗
|
JAA |
n00b985: Hmm, it can take a while if there's a derive currently running, but usually that should notice that you've uploaded new files and abort, I think. What's the item? (Feel free to PM if you don't want to share it publicly.) |
|
22:00
🔗
|
|
closure_ has quit IRC (Read error: Operation timed out) |
|
22:00
🔗
|
n00b985 |
There is a derive already running since it's a video. Will the additional files appear after the first derive is done? |
|
22:01
🔗
|
|
closure has joined #archiveteam-bs |
|
22:02
🔗
|
JAA |
Yeah, I think so. |
|
22:07
🔗
|
n00b985 |
Ok, thanks |
|
22:23
🔗
|
|
VerifiedJ has quit IRC (Read error: Operation timed out) |
|
22:33
🔗
|
|
closure has quit IRC (Read error: Connection reset by peer) |
|
22:35
🔗
|
|
closure has joined #archiveteam-bs |
|
22:59
🔗
|
|
closure has quit IRC (Read error: Connection reset by peer) |
|
23:06
🔗
|
|
closure has joined #archiveteam-bs |
|
23:23
🔗
|
|
sknebel_ has joined #archiveteam-bs |
|
23:25
🔗
|
|
bakJAA_ has joined #archiveteam-bs |
|
23:25
🔗
|
|
swebb sets mode: +o bakJAA_ |
|
23:25
🔗
|
|
JAA sets mode: +o bakJAA_ |
|
23:26
🔗
|
|
sknebel has quit IRC (Ping timeout: 492 seconds) |
|
23:28
🔗
|
|
bakJAA has quit IRC (Ping timeout: 492 seconds) |
|
23:31
🔗
|
|
closure has quit IRC (Read error: Operation timed out) |
|
23:32
🔗
|
|
BlueMax has joined #archiveteam-bs |
|
23:33
🔗
|
|
n00b985 has quit IRC (Quit: Page closed) |
|
23:33
🔗
|
|
closure has joined #archiveteam-bs |
|
23:39
🔗
|
|
mgrytbak has quit IRC (Ping timeout: 492 seconds) |
|
23:40
🔗
|
|
mgrytbak_ has joined #archiveteam-bs |
|
23:48
🔗
|
|
mgrytbak_ has quit IRC (Ping timeout: 492 seconds) |
|
23:54
🔗
|
|
mgrytbak^ has joined #archiveteam-bs |
|
23:55
🔗
|
|
Atom-- has quit IRC (Read error: Connection reset by peer) |
|
23:56
🔗
|
|
faolingfa has quit IRC (Read error: Connection reset by peer) |
|
23:56
🔗
|
|
faolingfa has joined #archiveteam-bs |
|
23:56
🔗
|
|
Atom-- has joined #archiveteam-bs |
|
23:57
🔗
|
|
dxrt has quit IRC (Quit: ZNC - http://znc.sourceforge.net) |
|
23:57
🔗
|
|
dxrt has joined #archiveteam-bs |
|
23:57
🔗
|
|
hook54321 has quit IRC (Ping timeout: 252 seconds) |
|
23:58
🔗
|
|
closure has quit IRC (Read error: Operation timed out) |
|
23:59
🔗
|
|
Frogging has quit IRC (Ping timeout: 252 seconds) |
|
23:59
🔗
|
|
i0npulse has quit IRC (Ping timeout: 252 seconds) |