| Time |
Nickname |
Message |
|
00:06
🔗
|
|
jmad980 has quit IRC (Ping timeout: 369 seconds) |
|
00:08
🔗
|
Sketchcow |
Still uploading. |
|
00:08
🔗
|
Sketchcow |
We're past 675 |
|
00:15
🔗
|
|
RichardG has joined #archiveteam |
|
00:25
🔗
|
|
megaminxw has joined #archiveteam |
|
00:28
🔗
|
|
jmad980 has joined #archiveteam |
|
00:42
🔗
|
|
JetBalsa has joined #archiveteam |
|
01:09
🔗
|
|
MMovie2 has joined #archiveteam |
|
01:11
🔗
|
|
MMovie has quit IRC (Read error: Operation timed out) |
|
01:15
🔗
|
|
Ravenloft has quit IRC (Read error: Connection reset by peer) |
|
01:47
🔗
|
|
JesseW has quit IRC (Leaving.) |
|
01:49
🔗
|
|
JesseW has joined #archiveteam |
|
01:54
🔗
|
Start |
so i gathered a list of all the items in jux's s3 bucket (http://user-zip-files.s3.amazonaws.com) and put it into archivebot |
|
01:54
🔗
|
Start |
jux is being saved over a year after it died |
|
01:55
🔗
|
JesseW |
what is jux? |
|
01:56
🔗
|
Start |
http://www.archiveteam.org/index.php?title=Jux |
|
01:56
🔗
|
Start |
it was a blogging website |
|
01:58
🔗
|
JesseW |
nice. What's the format of the zip files? |
|
01:59
🔗
|
Start |
it appears that each zip file contains all the images on a users blog |
|
02:00
🔗
|
Start |
most of the posts were grabbed before it shut down: https://archive.org/details/jux_posts_to_nov_24 |
|
02:01
🔗
|
JesseW |
nice of them to keep paying for AWS for another year. :-) |
|
02:05
🔗
|
|
JesseW has quit IRC (Leaving.) |
|
02:12
🔗
|
|
rctbeast has joined #archiveteam |
|
02:16
🔗
|
|
schbirid2 has joined #archiveteam |
|
02:19
🔗
|
|
schbirid has quit IRC (Read error: Operation timed out) |
|
02:34
🔗
|
|
JesseW has joined #archiveteam |
|
02:37
🔗
|
|
philpem has quit IRC (Ping timeout: 260 seconds) |
|
02:41
🔗
|
|
Ravenloft has joined #archiveteam |
|
02:49
🔗
|
|
Emcy has quit IRC (Ping timeout: 252 seconds) |
|
03:06
🔗
|
|
ohhdemgir has quit IRC (Remote host closed the connection) |
|
03:11
🔗
|
|
ohhdemgir has joined #archiveteam |
|
03:29
🔗
|
|
megaminxw has quit IRC (Quit: Leaving.) |
|
03:57
🔗
|
|
acridAxid has joined #archiveteam |
|
04:04
🔗
|
|
dashcloud has quit IRC (Remote host closed the connection) |
|
04:06
🔗
|
|
dashcloud has joined #archiveteam |
|
04:07
🔗
|
|
dashcloud has quit IRC (Remote host closed the connection) |
|
04:09
🔗
|
|
dashcloud has joined #archiveteam |
|
04:59
🔗
|
|
JetBalsa has quit IRC (Read error: Connection reset by peer) |
|
05:11
🔗
|
xmc |
Start: nice! |
|
05:22
🔗
|
|
rctbeast has quit IRC (Ping timeout: 240 seconds) |
|
05:26
🔗
|
|
acridAxid has quit IRC (marauder) |
|
05:29
🔗
|
|
acridAxid has joined #archiveteam |
|
05:34
🔗
|
|
VADemon has joined #archiveteam |
|
05:42
🔗
|
Sketchcow |
Great work, Start. |
|
05:59
🔗
|
|
megaminxw has joined #archiveteam |
|
07:01
🔗
|
|
WinterFox has joined #archiveteam |
|
07:30
🔗
|
|
FAMAS has joined #archiveteam |
|
07:32
🔗
|
FAMAS |
greetings to all, as this group is dedicated for purposes of data archival, this user is posting requests for volunteers who will participate in actions of video screenshotting contents displayed via digital devices |
|
08:16
🔗
|
|
FAMAS has quit IRC (Quit: http://chat.efnet.org (EOF)) |
|
08:32
🔗
|
|
REiN^ has joined #archiveteam |
|
08:33
🔗
|
|
JesseW has quit IRC (Read error: Operation timed out) |
|
08:39
🔗
|
xmc |
maybe, if you can interest someone in your project |
|
08:40
🔗
|
|
Ghost_of_ has joined #archiveteam |
|
09:12
🔗
|
|
BlueMaxim has quit IRC (Quit: Leaving) |
|
09:13
🔗
|
|
philpem has joined #archiveteam |
|
09:22
🔗
|
|
vOYtEC_ has quit IRC (Read error: Connection reset by peer) |
|
09:36
🔗
|
|
scyther has joined #archiveteam |
|
10:13
🔗
|
|
vOYtEC has joined #archiveteam |
|
11:53
🔗
|
|
Emcy has joined #archiveteam |
|
12:02
🔗
|
|
VADemon_ has joined #archiveteam |
|
12:07
🔗
|
|
VADemon has quit IRC (hub.se irc.efnet.pl) |
|
12:07
🔗
|
|
dashcloud has quit IRC (hub.se irc.efnet.pl) |
|
12:07
🔗
|
|
godane has quit IRC (hub.se irc.efnet.pl) |
|
12:18
🔗
|
|
lytv has quit IRC (Read error: Connection reset by peer) |
|
12:21
🔗
|
|
lytv has joined #archiveteam |
|
12:24
🔗
|
|
SimpBrain has quit IRC (Read error: Operation timed out) |
|
12:27
🔗
|
|
dashcloud has joined #archiveteam |
|
12:32
🔗
|
|
SimpBrain has joined #archiveteam |
|
12:39
🔗
|
|
godane has joined #archiveteam |
|
12:47
🔗
|
|
godane has quit IRC (Excess Flood) |
|
12:49
🔗
|
|
godane has joined #archiveteam |
|
13:04
🔗
|
|
zino_ has joined #archiveteam |
|
13:08
🔗
|
zino_ |
Hmm. Just realized the combined potential disk sizes of the warrior images for Virtualbox is 68GiB. That seems excessive. Mine has ballooned to 62G so far. |
|
13:12
🔗
|
|
nertzy2 has joined #archiveteam |
|
13:30
🔗
|
phuzion |
zino_: what seems excessive about it? You need 8GB for the system parttion, and 60GB for the data partition. |
|
13:30
🔗
|
phuzion |
ps, warrior questions generally go in #warrior |
|
13:34
🔗
|
|
nertzy2 has quit IRC (Quit: This computer has gone to sleep) |
|
13:36
🔗
|
|
megaminxw has quit IRC (Quit: Leaving.) |
|
14:19
🔗
|
|
HarryCros has joined #archiveteam |
|
14:19
🔗
|
|
wp494_ has joined #archiveteam |
|
14:20
🔗
|
|
Emcy_ has joined #archiveteam |
|
14:20
🔗
|
|
RichardG_ has joined #archiveteam |
|
14:20
🔗
|
|
WinterFox has quit IRC (Remote host closed the connection) |
|
14:21
🔗
|
|
Microguru has quit IRC (Ping timeout: 250 seconds) |
|
14:21
🔗
|
|
wp494 has quit IRC (Ping timeout: 250 seconds) |
|
14:21
🔗
|
|
lytv has quit IRC (Ping timeout: 250 seconds) |
|
14:21
🔗
|
|
RichardG has quit IRC (Ping timeout: 250 seconds) |
|
14:21
🔗
|
|
Gfy has quit IRC (Ping timeout: 250 seconds) |
|
14:21
🔗
|
|
alard has quit IRC (Ping timeout: 250 seconds) |
|
14:21
🔗
|
|
diacope has quit IRC (Ping timeout: 250 seconds) |
|
14:22
🔗
|
|
Emcy has quit IRC (Ping timeout: 250 seconds) |
|
14:22
🔗
|
|
HCross has quit IRC (Ping timeout: 250 seconds) |
|
14:22
🔗
|
|
superkuh_ has quit IRC (Ping timeout: 250 seconds) |
|
14:22
🔗
|
|
lytv has joined #archiveteam |
|
14:28
🔗
|
|
Gfy has joined #archiveteam |
|
14:32
🔗
|
|
alard has joined #archiveteam |
|
14:32
🔗
|
|
swebb sets mode: +o alard |
|
14:35
🔗
|
|
superkuh_ has joined #archiveteam |
|
14:36
🔗
|
|
Microguru has joined #archiveteam |
|
14:49
🔗
|
|
Ghost_of_ has quit IRC (Quit: Leaving) |
|
15:29
🔗
|
zino_ |
phuzion: Excessive as in there is no need for it. In what situation should a casual user cache 60GiB of data before uploading? |
|
15:31
🔗
|
|
RichardG_ is now known as RichardG |
|
15:34
🔗
|
dashcloud |
zino_: probably because 60 GB is enough to cover any kind of project, rather than needing to have separate images for different kinds of projects |
|
15:35
🔗
|
dashcloud |
for text or image projects, 60 GB is very likely overkill, but you'll never get there, where as for video projects, 60 GB is large enough to have you not constantly running out of space |
|
15:36
🔗
|
HarryCros |
the issue with large sizes like that, is that people with slow uploads like me will eventually have a large queue of files to upload |
|
15:38
🔗
|
zino_ |
I consider it a bigger problem that a casual contributor suddenly finds his 240G C: SSD maxed out and deletes the whole thing. That will eventually happend even if he never caches more than a gig at any time. |
|
15:39
🔗
|
zino_ |
(This is not an actual problem for me. I'm mostly bike-shedding.) |
|
15:39
🔗
|
SmileyG |
I don't think anyone is doing this casually |
|
15:39
🔗
|
SmileyG |
it was discussed previously |
|
15:39
🔗
|
SmileyG |
tho i wonder why it's expanding to use the entire 60Gb from the off |
|
15:39
🔗
|
SmileyG |
I thought it expanded as needed. |
|
15:41
🔗
|
zino_ |
SmileyG: In a perfect world the same filestructure/blocks would be used every time, but in practise you eventually write to most blocks. Deleting data from blocks will not shrink the image. |
|
15:41
🔗
|
SmileyG |
yah |
|
15:42
🔗
|
SmileyG |
oh right, you've been running it awhile then? |
|
15:42
🔗
|
SmileyG |
there is an option to 'shrink' it again btw |
|
15:42
🔗
|
zino_ |
It's probably been running for 2 years or so. |
|
15:42
🔗
|
zino_ |
VirtualBoxs CLI interface has something for that I think. Will check when I get some time. |
|
15:52
🔗
|
zino_ |
Meh. Waiting for a compile now anyway. I'll do it and pester #warrior with the result. |
|
15:54
🔗
|
|
Atom__ has joined #archiveteam |
|
16:16
🔗
|
zino_ |
Nope. Giving up that. the warrior is a vmdk, not vdi. So VirtualBox's tools can't shrink it. Would involve converting it or installing VMware tools. |
|
16:17
🔗
|
|
Ravenloft has quit IRC (Ping timeout: 606 seconds) |
|
17:28
🔗
|
|
JesseW has joined #archiveteam |
|
17:56
🔗
|
DFJustin |
if you shut the warrior down properly such that no tasks are running, you can just delete the data partition file and recreate an empty one |
|
18:14
🔗
|
|
atomotic has joined #archiveteam |
|
18:23
🔗
|
|
atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com) |
|
18:39
🔗
|
|
scyther has quit IRC (Read error: Connection reset by peer) |
|
19:04
🔗
|
|
diacope has joined #archiveteam |
|
19:29
🔗
|
|
dserodio has joined #archiveteam |
|
19:46
🔗
|
|
Microguru has quit IRC (Read error: Connection reset by peer) |
|
19:59
🔗
|
Sketchcow |
Hey, News grabbing people - I was informed about this project that Archive interacts with: http://gdeltproject.org/ |
|
20:03
🔗
|
Sketchcow |
Please look at it, make sure we're doing something different. |
|
20:07
🔗
|
arkiver |
Yes, we're doing somethinig different |
|
20:08
🔗
|
arkiver |
The GDELT projec misses a lot of non-English websites |
|
20:08
🔗
|
arkiver |
as well as more local newssites |
|
20:09
🔗
|
arkiver |
The GDELT grab also get's videos if a regex for them is provided |
|
20:09
🔗
|
arkiver |
We can also control the newsgrabs better then GDELT |
|
20:10
🔗
|
arkiver |
As far as I know GDELT does the frontpage and the subpages (not sure if they always do subpages) and srape new articles they find |
|
20:10
🔗
|
arkiver |
We can discover items through better manual settings and URLs, like RSS feeds, etc. |
|
20:10
🔗
|
arkiver |
This makes our crawls probably a bit more complete |
|
20:11
🔗
|
arkiver |
I'm not sure about this but I think GDELT covers 2 afghanistan newswebsites |
|
20:11
🔗
|
arkiver |
NewsGrabber currently covers more then 20 |
|
20:11
🔗
|
arkiver |
<arkiver>The GDELT grab also get's videos if a regex for them is provided |
|
20:11
🔗
|
|
dserodio has quit IRC (Read error: Operation timed out) |
|
20:12
🔗
|
arkiver |
^ for that I meant to say NewsGrabber grab also get's videos if a regex for them is provided |
|
20:12
🔗
|
arkiver |
But we're currently only grabbing videos if the website is supported by youtube-dl |
|
20:12
🔗
|
arkiver |
So that's why NewsGrabber is different from the GDELT grab |
|
20:21
🔗
|
JesseW |
Somebody, please add any of the above that isn't already on http://archiveteam.org/index.php?title=NewsGrabber |
|
20:41
🔗
|
|
dserodio has joined #archiveteam |
|
20:56
🔗
|
HarryCros |
and if youtube-dl has decided to behave |
|
21:09
🔗
|
|
WinterFox has joined #archiveteam |
|
21:10
🔗
|
|
WinterFox has quit IRC (Client Quit) |
|
21:10
🔗
|
|
WinterFox has joined #archiveteam |
|
21:19
🔗
|
|
SilSte has quit IRC (Remote host closed the connection) |
|
21:21
🔗
|
|
schbirid2 has quit IRC (Quit: Leaving) |
|
21:47
🔗
|
|
HarryCros is now known as HCross |
|
21:52
🔗
|
|
JesseW has quit IRC (Read error: Operation timed out) |
|
22:25
🔗
|
|
acridAxid has quit IRC (marauder) |
|
22:28
🔗
|
|
acridAxid has joined #archiveteam |
|
23:00
🔗
|
|
megaminxw has joined #archiveteam |
|
23:08
🔗
|
|
nertzy2 has joined #archiveteam |
|
23:09
🔗
|
|
Emcy_ has quit IRC (Ping timeout: 252 seconds) |
|
23:10
🔗
|
|
nertzy2 has quit IRC (Client Quit) |
|
23:12
🔗
|
|
Emcy has joined #archiveteam |
|
23:22
🔗
|
|
JesseW has joined #archiveteam |
|
23:26
🔗
|
arkiver |
The NewsGrabber project now covers more then 30 sites from the UAE! |
|
23:26
🔗
|
arkiver |
We now fully cover the UAE |
|
23:27
🔗
|
JesseW |
awesome! |
|
23:27
🔗
|
arkiver |
Join NewsGrabber: #newsgrabber |
|
23:27
🔗
|
JesseW |
Is there a nicely formatted, automatically updated, list of sites included in newsgrabber written yet? |
|
23:28
🔗
|
arkiver |
https://github.com/ArchiveTeam/NewsGrabber/tree/master/services |
|
23:28
🔗
|
arkiver |
that looks nice I think |
|
23:28
🔗
|
arkiver |
The bot of NewsGrabber, newsbuddy, gives updates on new found links and uploads and can be followed here: #newsgrabberbot |
|
23:34
🔗
|
JesseW |
Hm. I think I'll write something more like what I was thinking of, then. |
|
23:35
🔗
|
arkiver |
what are you thinking of? |
|
23:36
🔗
|
JesseW |
I think it'll be easier to write it than explain it. :-) |
|
23:36
🔗
|
ersi |
making an outdated list |
|
23:36
🔗
|
ersi |
:-) |
|
23:36
🔗
|
JesseW |
ersi: :-P |
|
23:36
🔗
|
HCross |
thing with github is that it updates all the time, and is the list the server feeds off |
|
23:40
🔗
|
JesseW |
I think what I was thinking of is the writehtmllist function in main.py |
|
23:41
🔗
|
HCross |
So http://newsgrabber.harrycross.me |
|
23:42
🔗
|
JesseW |
yes, but apparently with one row per file in services/ and different columns. |