#archiveteam-bs 2017-06-11,Sun

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)

WhoWhatWhen
JAAWell, that reduced the number of pending URLs from 390k to 80k. Nice. :D [00:00]
..... (idle for 21mn)
***tfgbd_znc has quit IRC (Ping timeout: 600 seconds) [00:21]
....... (idle for 33mn)
BlueMaxim has joined #archiveteam-bs [00:54]
timmc has left
wp494 has joined #archiveteam-bs
[01:04]
JensRexJAA: I kind of forgot about it. [01:10]
jrwrarkiver: sweet baby jesus thats a ton of data arkiver [01:16]
arkiveryep
so the future of the project is a little unclear
[01:16]
jrwrYa, more then 200TB you need to start getting IA Tech team on it [01:17]
arkiverwell, we can just pump the data in
but they'll notice
especially with this size
[01:17]
jrwrIt looks like a amazing datastore [01:17]
arkiverand might not be happy with this much data
yeah
we coordinated with IA
but was paused after the estimate turned out to be 800 TB
anyway
[01:17]
jrwrIt would be best if they rsynced it right into their own search tool for it, so it had the right metadata [01:18]
arkivernot sure
but I'm off
good day
[01:19]
jrwrjrwr grabs a stick to hold the fort [01:19]
arkiver:P [01:19]
***geronimo has joined #archiveteam-bs
username1 has joined #archiveteam-bs
schbirid2 has quit IRC (Read error: Operation timed out)
geronimo has left
[01:26]
j08nY has quit IRC (Quit: Leaving)
fie has joined #archiveteam-bs
[01:41]
......... (idle for 44mn)
antonizoo has quit IRC () [02:27]
antonizoo has joined #archiveteam-bs
ZexaronS has quit IRC (Leaving)
[02:33]
antonizoo has quit IRC ()
antonizoo has joined #archiveteam-bs
[02:49]
........... (idle for 54mn)
Ravenloft has quit IRC (Ping timeout: 245 seconds) [03:47]
pizzaiolo has quit IRC (Quit: pizzaiolo) [03:59]
..... (idle for 24mn)
GilfoyleSo just curious, anybody have any scripts for downloading say wikiProjects?
Trying to download one in particular and not the entirety of Wikipedia.
[04:23]
Otherwise, looks like I'll need to writer a spider and a basic implementation of PageRank to sort each topic into the correct subgrouping... [04:29]
MrRadarGilfoyle: We have a set of scripts for scraping the API of Mediawiki installations: https://raw.githubusercontent.com/WikiTeam/wikiteam/master/dumpgenerator.py
A bit more info on the WikiTeam page: http://archiveteam.org/index.php?title=WikiTeam
[04:33]
***divingkat has joined #archiveteam-bs [04:36]
divingkatHas anyone recorded things on Video 8 cassettes? [04:41]
***Sk1d has quit IRC (Ping timeout: 250 seconds) [04:55]
Sk1d has joined #archiveteam-bs [05:02]
...... (idle for 29mn)
klg has joined #archiveteam-bs [05:31]
.... (idle for 18mn)
phuzion has quit IRC (Remote host closed the connection)
phuzion has joined #archiveteam-bs
[05:49]
.... (idle for 15mn)
phuzion has quit IRC (Read error: Connection reset by peer)
phuzion has joined #archiveteam-bs
[06:04]
...... (idle for 28mn)
divingkathttps://www.youtube.com/watch?v=8t5TYw2bkOk [06:33]
...... (idle for 27mn)
***tsp_ has joined #archiveteam-bs [07:00]
Simpbrain has quit IRC (Read error: Operation timed out) [07:08]
Simpbrain has joined #archiveteam-bs [07:19]
............. (idle for 1h1mn)
vitzli has joined #archiveteam-bs [08:20]
.... (idle for 18mn)
Honno has joined #archiveteam-bs [08:38]
..... (idle for 22mn)
SHODAN_UI has joined #archiveteam-bs [09:00]
...... (idle for 29mn)
underscor has joined #archiveteam-bs
swebb sets mode: +o underscor
[09:29]
.... (idle for 19mn)
gui7 has joined #archiveteam-bs [09:48]
vitzli has quit IRC (Quit: Leaving)
divingkat has quit IRC (Quit: ChatZilla 0.9.93 [Firefox 53.0.3/20170518000419])
[09:59]
Nazcaokay, NOW pixiv is done, right?
or have we still not done the +18 sweep?
[10:02]
JensRexDoesn't look even slightly done. Still 200k items out. [10:06]
***SHODAN_UI has quit IRC (Remote host closed the connection)
j08nY has joined #archiveteam-bs
[10:14]
JAAYep, we still need to do the 18+ rooms. [10:20]
***icedice has joined #archiveteam-bs [10:33]
............. (idle for 1h3mn)
pizzaiolo has joined #archiveteam-bs [11:36]
....... (idle for 32mn)
SHODAN_UI has joined #archiveteam-bs
icedice has quit IRC (efnet.portlane.se se.hub)
tsp_ has quit IRC (efnet.portlane.se se.hub)
robinak has quit IRC (efnet.portlane.se se.hub)
ItsYoda has quit IRC (efnet.portlane.se se.hub)
BartoCH has quit IRC (efnet.portlane.se se.hub)
DFJustin has quit IRC (efnet.portlane.se se.hub)
jiphex has quit IRC (efnet.portlane.se se.hub)
r3c0d3x has quit IRC (efnet.portlane.se se.hub)
Whopper has quit IRC (efnet.portlane.se se.hub)
Jon has quit IRC (efnet.portlane.se se.hub)
Kenshin has quit IRC (efnet.portlane.se se.hub)
Sanqui has quit IRC (efnet.portlane.se se.hub)
LastNinja has quit IRC (efnet.portlane.se se.hub)
kittymeow has quit IRC (efnet.portlane.se se.hub)
zhongfu has quit IRC (efnet.portlane.se se.hub)
HCross2 has quit IRC (efnet.portlane.se se.hub)
dan- has quit IRC (efnet.portlane.se se.hub)
voltagex has quit IRC (efnet.portlane.se se.hub)
davidar has quit IRC (efnet.portlane.se se.hub)
Meroje has quit IRC (efnet.portlane.se se.hub)
wm_ has quit IRC (efnet.portlane.se se.hub)
kevinr has quit IRC (efnet.portlane.se se.hub)
tephra_ has quit IRC (efnet.portlane.se se.hub)
ThisAsYou has quit IRC (efnet.portlane.se se.hub)
Ctrl-S___ has quit IRC (efnet.portlane.se se.hub)
deathy has quit IRC (efnet.portlane.se se.hub)
alembic has quit IRC (efnet.portlane.se se.hub)
hook54321 has quit IRC (efnet.portlane.se se.hub)
Famicoman has quit IRC (efnet.portlane.se se.hub)
JSharp___ has quit IRC (efnet.portlane.se se.hub)
tklk has quit IRC (efnet.portlane.se se.hub)
floogulin has quit IRC (efnet.portlane.se se.hub)
FalconK has quit IRC (efnet.portlane.se se.hub)
t2t2 has quit IRC (efnet.portlane.se se.hub)
Muad-Dib has quit IRC (efnet.portlane.se se.hub)
raphidae has quit IRC (efnet.portlane.se se.hub)
antonizoo has quit IRC (efnet.portlane.se se.hub)
JensRex has quit IRC (efnet.portlane.se se.hub)
alfie has quit IRC (efnet.portlane.se se.hub)
Rai-chan has quit IRC (efnet.portlane.se se.hub)
purplebot has quit IRC (efnet.portlane.se se.hub)
Aoede has quit IRC (efnet.portlane.se se.hub)
jtn2 has quit IRC (efnet.portlane.se se.hub)
Fletcher has quit IRC (efnet.portlane.se se.hub)
Hecatz has quit IRC (efnet.portlane.se se.hub)
bwn has quit IRC (efnet.portlane.se se.hub)
medowar has quit IRC (efnet.portlane.se se.hub)
voidsta has quit IRC (efnet.portlane.se se.hub)
Riviera has quit IRC (efnet.portlane.se se.hub)
i0npulse has quit IRC (efnet.portlane.se se.hub)
GLaDOS has quit IRC (efnet.portlane.se se.hub)
nyany has quit IRC (efnet.portlane.se se.hub)
PurpleSym has quit IRC (efnet.portlane.se se.hub)
altlabel has quit IRC (efnet.portlane.se se.hub)
yuitimoth has quit IRC (efnet.portlane.se se.hub)
bsmith093 has quit IRC (efnet.portlane.se se.hub)
antomatic has quit IRC (efnet.portlane.se se.hub)
Xibalba has quit IRC (efnet.portlane.se se.hub)
Boppen has quit IRC (efnet.portlane.se se.hub)
pizzaiolo has quit IRC (efnet.portlane.se se.hub)
Simpbrain has quit IRC (efnet.portlane.se se.hub)
brayden has quit IRC (efnet.portlane.se se.hub)
kurt has quit IRC (efnet.portlane.se se.hub)
wacky_ has quit IRC (efnet.portlane.se se.hub)
HUBI has quit IRC (efnet.portlane.se se.hub)
SpaffGarg has quit IRC (efnet.portlane.se se.hub)
balrog has quit IRC (efnet.portlane.se se.hub)
kisspunch has quit IRC (efnet.portlane.se se.hub)
sep332_ has quit IRC (efnet.portlane.se se.hub)
kanzure has quit IRC (efnet.portlane.se se.hub)
c4rc4s has quit IRC (efnet.portlane.se se.hub)
ranma has quit IRC (efnet.portlane.se se.hub)
whydomain has quit IRC (efnet.portlane.se se.hub)
chazchaz_ has quit IRC (efnet.portlane.se se.hub)
winr4r has quit IRC (efnet.portlane.se se.hub)
closure has quit IRC (efnet.portlane.se se.hub)
godane has quit IRC (efnet.portlane.se se.hub)
robogoat has quit IRC (efnet.portlane.se se.hub)
wabu has quit IRC (efnet.portlane.se se.hub)
trs80 has quit IRC (efnet.portlane.se se.hub)
SadDM has quit IRC (efnet.portlane.se se.hub)
jspiros has quit IRC (efnet.portlane.se se.hub)
j08nY has quit IRC (efnet.portlane.se se.hub)
Odd0002 has quit IRC (efnet.portlane.se se.hub)
SilSte has quit IRC (efnet.portlane.se se.hub)
zerkalo has quit IRC (efnet.portlane.se se.hub)
RedType has quit IRC (efnet.portlane.se se.hub)
Nazca has quit IRC (efnet.portlane.se se.hub)
pikhq has quit IRC (efnet.portlane.se se.hub)
Xamayon has quit IRC (efnet.portlane.se se.hub)
mgrytbak has quit IRC (efnet.portlane.se se.hub)
acridAxid has quit IRC (efnet.portlane.se se.hub)
joepie91 has quit IRC (efnet.portlane.se se.hub)
cf has quit IRC (efnet.portlane.se se.hub)
chfoo has quit IRC (efnet.portlane.se se.hub)
eprillios has quit IRC (efnet.portlane.se se.hub)
tapedrive has quit IRC (efnet.portlane.se se.hub)
underscor has quit IRC (efnet.portlane.se se.hub)
espes__ has quit IRC (efnet.portlane.se se.hub)
kvieta has quit IRC (efnet.portlane.se se.hub)
klg has quit IRC (efnet.portlane.se se.hub)
Honno has quit IRC (efnet.portlane.se se.hub)
fie has quit IRC (efnet.portlane.se se.hub)
username1 has quit IRC (efnet.portlane.se se.hub)
BlueMaxim has quit IRC (efnet.portlane.se se.hub)
TheLovina has quit IRC (efnet.portlane.se se.hub)
dashcloud has quit IRC (efnet.portlane.se se.hub)
odemg has quit IRC (efnet.portlane.se se.hub)
zenguy has quit IRC (efnet.portlane.se se.hub)
superkuh has quit IRC (efnet.portlane.se se.hub)
xmc has quit IRC (efnet.portlane.se se.hub)
slyphic has quit IRC (efnet.portlane.se se.hub)
w0rp has quit IRC (efnet.portlane.se se.hub)
zino has quit IRC (efnet.portlane.se se.hub)
ranma_ has quit IRC (efnet.portlane.se se.hub)
Selavi has quit IRC (efnet.portlane.se se.hub)
Frogging has quit IRC (efnet.portlane.se se.hub)
htw has quit IRC (efnet.portlane.se se.hub)
dxrt has quit IRC (efnet.portlane.se se.hub)
twigfoot has quit IRC (efnet.portlane.se se.hub)
MrRadar has quit IRC (efnet.portlane.se se.hub)
atlogbot has quit IRC (efnet.portlane.se se.hub)
chazchaz has quit IRC (efnet.portlane.se se.hub)
atomicthu has quit IRC (efnet.portlane.se se.hub)
TC01 has quit IRC (efnet.portlane.se se.hub)
swebb has quit IRC (efnet.portlane.se se.hub)
Darkstar has quit IRC (efnet.portlane.se se.hub)
dcmorton has quit IRC (efnet.portlane.se se.hub)
Somebody2 has quit IRC (efnet.portlane.se se.hub)
Cameron_D has quit IRC (efnet.portlane.se se.hub)
arkiver has quit IRC (efnet.portlane.se se.hub)
Coderjo has quit IRC (efnet.portlane.se se.hub)
Jonimoose has quit IRC (efnet.portlane.se se.hub)
midas has quit IRC (efnet.portlane.se se.hub)
yipdw has quit IRC (efnet.portlane.se se.hub)
Baljem has quit IRC (efnet.portlane.se se.hub)
lainu has quit IRC (efnet.portlane.se se.hub)
gui7 has quit IRC (efnet.portlane.se se.hub)
phuzion has quit IRC (efnet.portlane.se se.hub)
wp494 has quit IRC (efnet.portlane.se se.hub)
decay has quit IRC (efnet.portlane.se se.hub)
jrwr has quit IRC (efnet.portlane.se se.hub)
C4K3 has quit IRC (efnet.portlane.se se.hub)
Stilett0 has quit IRC (efnet.portlane.se se.hub)
REiN^ has quit IRC (efnet.portlane.se se.hub)
ndiddy has quit IRC (efnet.portlane.se se.hub)
ivan has quit IRC (efnet.portlane.se se.hub)
Smiley has quit IRC (efnet.portlane.se se.hub)
Mayonaise has quit IRC (efnet.portlane.se se.hub)
pipt has quit IRC (efnet.portlane.se se.hub)
dboard has quit IRC (efnet.portlane.se se.hub)
rocode has quit IRC (efnet.portlane.se se.hub)
K4k has quit IRC (efnet.portlane.se se.hub)
will has quit IRC (efnet.portlane.se se.hub)
Petri152 has quit IRC (efnet.portlane.se se.hub)
JAA has quit IRC (efnet.portlane.se se.hub)
mhazinsk has quit IRC (efnet.portlane.se se.hub)
Lord_Nigh has quit IRC (efnet.portlane.se se.hub)
luckcolor has quit IRC (efnet.portlane.se se.hub)
aschmitz has quit IRC (efnet.portlane.se se.hub)
mundus20- has quit IRC (efnet.portlane.se se.hub)
PotcFdk has quit IRC (efnet.portlane.se se.hub)
johnny5 has quit IRC (efnet.portlane.se se.hub)
SHODAN_UI has quit IRC (efnet.portlane.se se.hub)
RichardG has quit IRC (efnet.portlane.se se.hub)
Gilfoyle has quit IRC (efnet.portlane.se se.hub)
greenie has quit IRC (efnet.portlane.se se.hub)
Xibalba has joined #archiveteam-bs
antomatic has joined #archiveteam-bs
bsmith093 has joined #archiveteam-bs
yuitimoth has joined #archiveteam-bs
altlabel has joined #archiveteam-bs
PurpleSym has joined #archiveteam-bs
nyany has joined #archiveteam-bs
GLaDOS has joined #archiveteam-bs
Boppen has joined #archiveteam-bs
i0npulse has joined #archiveteam-bs
Riviera has joined #archiveteam-bs
medowar has joined #archiveteam-bs
voidsta has joined #archiveteam-bs
bwn has joined #archiveteam-bs
Hecatz has joined #archiveteam-bs
Fletcher has joined #archiveteam-bs
jtn2 has joined #archiveteam-bs
Aoede has joined #archiveteam-bs
purplebot has joined #archiveteam-bs
Rai-chan has joined #archiveteam-bs
alfie has joined #archiveteam-bs
JensRex has joined #archiveteam-bs
antonizoo has joined #archiveteam-bs
ranma has joined #archiveteam-bs
whydomain has joined #archiveteam-bs
Baljem has joined #archiveteam-bs
lainu has joined #archiveteam-bs
chazchaz_ has joined #archiveteam-bs
winr4r has joined #archiveteam-bs
closure has joined #archiveteam-bs
c4rc4s has joined #archiveteam-bs
kanzure has joined #archiveteam-bs
yipdw has joined #archiveteam-bs
se.hub sets mode: +oooo antomatic PurpleSym closure yipdw
midas has joined #archiveteam-bs
Jonimoose has joined #archiveteam-bs
Coderjo has joined #archiveteam-bs
arkiver has joined #archiveteam-bs
Cameron_D has joined #archiveteam-bs
Somebody2 has joined #archiveteam-bs
dcmorton has joined #archiveteam-bs
sep332_ has joined #archiveteam-bs
Darkstar has joined #archiveteam-bs
kisspunch has joined #archiveteam-bs
johnny5 has joined #archiveteam-bs
swebb has joined #archiveteam-bs
se.hub sets mode: +ooo Jonimoose arkiver swebb
TC01 has joined #archiveteam-bs
atomicthu has joined #archiveteam-bs
PotcFdk has joined #archiveteam-bs
mundus20- has joined #archiveteam-bs
balrog has joined #archiveteam-bs
chazchaz has joined #archiveteam-bs
atlogbot has joined #archiveteam-bs
MrRadar has joined #archiveteam-bs
kvieta has joined #archiveteam-bs
espes__ has joined #archiveteam-bs
tapedrive has joined #archiveteam-bs
eprillios has joined #archiveteam-bs
chfoo has joined #archiveteam-bs
cf has joined #archiveteam-bs
joepie91 has joined #archiveteam-bs
dxrt has joined #archiveteam-bs
acridAxid has joined #archiveteam-bs
SpaffGarg has joined #archiveteam-bs
HUBI has joined #archiveteam-bs
wacky_ has joined #archiveteam-bs
aschmitz has joined #archiveteam-bs
luckcolor has joined #archiveteam-bs
Lord_Nigh has joined #archiveteam-bs
Xamayon has joined #archiveteam-bs
htw has joined #archiveteam-bs
mhazinsk has joined #archiveteam-bs
JAA has joined #archiveteam-bs
Petri152 has joined #archiveteam-bs
will has joined #archiveteam-bs
mgrytbak has joined #archiveteam-bs
kurt has joined #archiveteam-bs
K4k has joined #archiveteam-bs
Frogging has joined #archiveteam-bs
Selavi has joined #archiveteam-bs
rocode has joined #archiveteam-bs
ranma_ has joined #archiveteam-bs
dboard has joined #archiveteam-bs
greenie has joined #archiveteam-bs
trs80 has joined #archiveteam-bs
SadDM has joined #archiveteam-bs
jspiros has joined #archiveteam-bs
brayden has joined #archiveteam-bs
pipt has joined #archiveteam-bs
zino has joined #archiveteam-bs
Mayonaise has joined #archiveteam-bs
pikhq has joined #archiveteam-bs
se.hub sets mode: +ooo balrog SadDM brayden
w0rp has joined #archiveteam-bs
slyphic has joined #archiveteam-bs
Nazca has joined #archiveteam-bs
xmc has joined #archiveteam-bs
twigfoot has joined #archiveteam-bs
wabu has joined #archiveteam-bs
robogoat has joined #archiveteam-bs
RedType has joined #archiveteam-bs
superkuh has joined #archiveteam-bs
zerkalo has joined #archiveteam-bs
zenguy has joined #archiveteam-bs
Smiley has joined #archiveteam-bs
ivan has joined #archiveteam-bs
odemg has joined #archiveteam-bs
Gilfoyle has joined #archiveteam-bs
ndiddy has joined #archiveteam-bs
REiN^ has joined #archiveteam-bs
Stilett0 has joined #archiveteam-bs
C4K3 has joined #archiveteam-bs
dashcloud has joined #archiveteam-bs
TheLovina has joined #archiveteam-bs
RichardG has joined #archiveteam-bs
godane has joined #archiveteam-bs
jrwr has joined #archiveteam-bs
SilSte has joined #archiveteam-bs
Odd0002 has joined #archiveteam-bs
decay has joined #archiveteam-bs
BlueMaxim has joined #archiveteam-bs
wp494 has joined #archiveteam-bs
username1 has joined #archiveteam-bs
fie has joined #archiveteam-bs
klg has joined #archiveteam-bs
phuzion has joined #archiveteam-bs
Simpbrain has joined #archiveteam-bs
Honno has joined #archiveteam-bs
underscor has joined #archiveteam-bs
gui7 has joined #archiveteam-bs
j08nY has joined #archiveteam-bs
pizzaiolo has joined #archiveteam-bs
SHODAN_UI has joined #archiveteam-bs
se.hub sets mode: +oo xmc underscor
swebb sets mode: +o SketchCow
icedice has joined #archiveteam-bs
tsp_ has joined #archiveteam-bs
robinak has joined #archiveteam-bs
ItsYoda has joined #archiveteam-bs
BartoCH has joined #archiveteam-bs
DFJustin has joined #archiveteam-bs
jiphex has joined #archiveteam-bs
r3c0d3x has joined #archiveteam-bs
Whopper has joined #archiveteam-bs
Jon has joined #archiveteam-bs
Kenshin has joined #archiveteam-bs
Sanqui has joined #archiveteam-bs
LastNinja has joined #archiveteam-bs
kittymeow has joined #archiveteam-bs
zhongfu has joined #archiveteam-bs
HCross2 has joined #archiveteam-bs
dan- has joined #archiveteam-bs
voltagex has joined #archiveteam-bs
davidar has joined #archiveteam-bs
Meroje has joined #archiveteam-bs
wm_ has joined #archiveteam-bs
kevinr has joined #archiveteam-bs
tephra_ has joined #archiveteam-bs
Ctrl-S___ has joined #archiveteam-bs
ThisAsYou has joined #archiveteam-bs
alembic has joined #archiveteam-bs
deathy has joined #archiveteam-bs
hook54321 has joined #archiveteam-bs
Famicoman has joined #archiveteam-bs
JSharp___ has joined #archiveteam-bs
tklk has joined #archiveteam-bs
floogulin has joined #archiveteam-bs
FalconK has joined #archiveteam-bs
t2t2 has joined #archiveteam-bs
raphidae has joined #archiveteam-bs
Muad-Dib has joined #archiveteam-bs
efnet.port80.se sets mode: +o DFJustin
swebb sets mode: +o DFJustin
[12:08]
BlueMaxim has quit IRC (Quit: Leaving) [12:28]
JAATanobb grab complete. That was way quicker than I expected. I grabbed all 13 languages and the mobile versions as well. I skipped some redundant pages though, e.g. those listing all posts by one user in a thread. [12:28]
.................... (idle for 1h37mn)
***ZexaronS has joined #archiveteam-bs
icedice has quit IRC (Quit: Leaving)
[14:05]
.... (idle for 16mn)
pizzaiolo has quit IRC (Read error: Operation timed out) [14:23]
pizzaiolo has joined #archiveteam-bs
pizzaiolo has quit IRC (Read error: Connection reset by peer)
pizzaiolo has joined #archiveteam-bs
[14:32]
MrRadar_ has joined #archiveteam-bs [14:41]
MrRadar has quit IRC (Read error: Operation timed out)
MrRadar_ is now known as MrRadar
[14:49]
.............. (idle for 1h7mn)
odemg has quit IRC (Read error: Operation timed out)
odemg has joined #archiveteam-bs
[15:56]
....................................... (idle for 3h10mn)
tapedrivearkiver: No, I'm using the same format bsmith093 used (https://github.com/JimmXinu/FanFicFare) which extracts the story text into markdown.
Any other way and my limited disk space would be completly used up.
[19:10]
arkiverI see
What are you currently using?
maybe the WARCs could be uploaded to IA for the wayback machine
if there's any copyright issues they'll probably block the pages or website in the wayback machine
[19:11]
tapedrivearkiver: Example format: http://storage.savefanfiction.tk/Prince%20Consort-ffnet_8902231.txt (re-archived today)
All fanfiction.net URLs are robots.txt blocked anyway, actually.
[19:13]
MrRadarYet another good reason to grab them all as WARCs... so when the site goes down in the future and someone removes the robots.txt block there will be content to show
Also, I wonder how well git would handle if you put all these files into it...
[19:14]
***SmileyG has joined #archiveteam-bs [19:15]
username1you could point FFF at warcproxy maybe? [19:17]
tapedriveTotal of 7,382,393 plaintext files, all in the same folder? Few hundred gigabytes? No idea if git would cope with that. [19:20]
***SmileyG has quit IRC (Read error: Connection reset by peer)
SmileyG has joined #archiveteam-bs
[19:20]
jrwryou could move based off the first few letters of its sha1 of the name of the file
into their own folders, or even the first few letters of the title of the file
[19:21]
tapedriveI've thought about doing WARCs and stuff like that, but I'm running everything on a couple of Raspberry Pis and a 2tb HD, so... [19:21]
***Smiley has quit IRC (Read error: Operation timed out) [19:22]
arkiverthing is that once it's not in WARCs, it will probably never go into WARCs and therefor will never get into an archive like the wayback machine [19:22]
username1tapedrive: you will probably get better performance in any case if you split that into subdirectories :D [19:23]
tapedriveusername1: Yeah, I can't realy do much in that directory any more
But it makes merging newly updates stories much easier.
arkiver: I'll think about WARCs for the future. To get them into wayback, do I just upload them to IA with type web?
[19:23]
arkiverand let us know about them
we'll first have to make sure the WARCs are valid of course
and not edited
[19:26]
tapedriveAnd if I upload more WARCs into that item, will they be auto-added to wayback? [19:26]
arkiverI think so
but multiple items might be better in that case
also again depending on the number of WARCs
you could do like 1 or 10 GB per item
[19:26]
tapedriveOkay, I'll see if I can add that in. [19:29]
***medowar has quit IRC (Read error: Connection reset by peer) [19:31]
arkivernice :) [19:32]
..... (idle for 22mn)
***tfgbd_znc has joined #archiveteam-bs
Ravenloft has joined #archiveteam-bs
[19:54]
ndiddyquick question: do you guys know how to download from veoh.com?
there's this rare video that i can only find there
[20:03]
arkiverlink? [20:03]
ndiddyhttp://www.veoh.com/watch/v1313996wddKMNqf?h1=Super+Spacefortress+Macross
it's the only copy i can find of the uncut, hilariously bad dub of the first macross movie
[20:03]
JAAUgh, Flash. [20:05]
username1ndiddy: uh
ndiddy: youtube-dl "http://www.veoh.com/watch/v1313996wddKMNqf?h1=Super+Spacefortress+Macross"
done
[20:06]
ndiddyi tried jdownloader but it only downloads the first 1/4 of it [20:06]
arkivergo to source
search for fullPreviewHashLowPath
and download that URL
[20:07]
JAALow vs. High quality/resolution? [20:08]
arkiverah sorry [20:11]
username1[download] 10.0% of 881.42MiB at 321.50KiB/s ETA 42:07
i can up it elsewhere afterwards if needed
[20:12]
ndiddytell me if it downloads all the way [20:12]
username1:) [20:12]
ndiddyalso, youtube-dl gives me an "unsupported url" error [20:12]
username1update [20:12]
arkiversearch for fullPreviewHashHighPath
that's the same resolution as when HQ is selected in the player
uh
[20:13]
ndiddyi assumed so [20:13]
arkiversame size [20:13]
JAAWhat does "Preview" mean in there though? [20:14]
arkiverno idea [20:14]
ndiddyit looks to be the same size though [20:14]
arkiverbut it's the version that's loaded in browse
yeah
[20:14]
ndiddythe way they have the site set up is kinda strange
it downloads the first couple megs at full speed than throttles you to 300 kbps
[20:14]
JAAYeah, the whole page is really cancerous. [20:15]
arkivermany streaming sites do that [20:15]
JAAThe amount of third-party JavaScript code being pulled there is ridiculous. [20:15]
MrRadarThat kind of throttling is pretty common because most people don't watch most of the video anyways
So they give you just enough to ensure there's a decent buffer, then feed you the rest at approx. the playback rate
So they don't waste too much bandwidth if you close the tab after a minute
[20:16]
username1have you tried watching the network tab while watching via flash? [20:16]
ndiddyno, why? [20:17]
arkiveryeah, it's the same [20:17]
username1to figure out how it works? [20:17]
JAACould it be that the Preview is only the first 25% (what JDownloader grabs)? [20:17]
ndiddylooks like jdownloader was using a different url
http://fcache.veoh.com/file/f/h1313996.mp4?e=1497209817&ri=6000&rs=300&h=fa032ed6a31a75daab4b4503f60e35f9
vs page editing, which gives you http://content.veoh.com/flash/p/2/v1313996wddKMNqf/h1313996.mp4?ct=bdd4eff4404837d31915158fe9f6a3fe3c9aaf63da409283
[20:17]
JAAI see. [20:18]
ndiddydamn it, same thing
arkiver: can you download the whole video?
[20:19]
arkiverwell, yeah, why? [20:20]
ndiddyi just got 200 mb again from that link [20:20]
arkiverin source search for fullPreviewHashHighPath
and download that
[20:21]
ndiddythat's what i did
i'll see if watching the video in a muted tab while downloading fixes anything
i'm assuming that that url is the one the player buffers from
and it won't buffer more than 25% into the video
[20:21]
arkiverbefore you can see fullPreviewHashHighPath you need to have 18+ cookie of course
sure you didn't use fullPreviewHashLowPath?
that gives you a 200 MB one
fullPreviewHashHighPath gives 881 or so
881 MB*
[20:23]
username1[download] 100% of 881.42MiB in 12:40
ERROR: content too short (expected 924240150 bytes and served 239785474)
[20:24]
ndiddysee what i mean [20:24]
username1well, can you actually skip to later via flash? [20:26]
ndiddyyep
it's not like nnd or something
[20:26]
JAADoes anything happen in the Network tab? [20:26]
ndiddyyou mean, like the windows one? [20:26]
username1no, the browser one [20:26]
JAAThe browser's. [20:26]
username1:O [20:26]
ndiddyis that a thing [20:26]
username1zomg :P
you have a lot to learn
[20:26]
MrRadarPress F12 in the tab [20:27]
JAAEssential tool #1 for web devs. :-P [20:27]
username1and people trying to download special things [20:27]
ndiddylooks like there's a ?start parameter
sorry, &start
ex: http://content.veoh.com/flash/f/2/v1313996wddKMNqf/l1313996.mp4?ct=466b8df98e1762caf22b228f86934f1623d189f7d4c429bb&start=4659.76
[20:28]
arkiverit will only send you the video starting from that time [20:33]
ndiddyi guess what i have to do is download the video from the first link, count how many seconds are in it, then redownload and splice all the clips together [20:34]
***SmileyG has quit IRC (Remote host closed the connection)
BlueMaxim has joined #archiveteam-bs
[20:40]
username1 has quit IRC (Quit: Leaving) [20:56]
SHODAN_UI has quit IRC (Remote host closed the connection) [21:01]
........ (idle for 36mn)
BartoCH has quit IRC (Ping timeout: 260 seconds) [21:37]
dashcloud has quit IRC (Remote host closed the connection) [21:48]
dashcloud has joined #archiveteam-bs [21:55]
.......... (idle for 48mn)
Somebody2tapedrive: thank you for grabbing the text of that fanfiction, in any case. It's useful, even though WARCs would be very nice too. [22:43]
tapedriveI'm seeing how easy it would be to add in WARC to my system now.
Although I think it would mean an entire recrawl, which has taken several (about 10) months due to their rate limiting.
[22:45]
ndiddyhttps://archive.org/details/SuperSpacefortressMacross [22:52]
***wp494 has quit IRC (Read error: Connection reset by peer) [22:52]
Ravenloft has quit IRC (Ping timeout: 268 seconds) [23:04]
.... (idle for 17mn)
antomati_ has joined #archiveteam-bs
swebb sets mode: +o antomati_
antomatic has quit IRC (Read error: Operation timed out)
[23:21]
BartoCH has joined #archiveteam-bs [23:33]
.... (idle for 19mn)
jrwrSketchCow: Fix it please - Could not store file "/tmp/phphIFGty" at "mwstore://local-backend/local-public/d/da/Chatpixivicon.gif".
its still broken :(
[23:52]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)