| Time |
Nickname |
Message |
|
00:04
🔗
|
omf_ |
Adding gzip support to wget I think is a huge waste of time. I believe this to be true because wget is a mismatch of terrible code and partial tests. For what Archive Team needs wget is missing far more than just gzip compression on file transfers |
|
00:05
🔗
|
omf_ |
How about not checking head on all files when resuming a grab. How about having a continue option that works more like httrack |
|
00:06
🔗
|
omf_ |
Or eating up all your ram the longer and larger a process is |
|
00:08
🔗
|
omf_ |
There is also the duplicate ID problem wget currently has |
|
00:12
🔗
|
omf_ |
There is also problems with wget's url encoding that make page deduplication harder |
|
00:21
🔗
|
omf_ |
Also wget has no cleanup code for being interrupted which usually means the last warc record gets truncated |
|
00:22
🔗
|
balrog |
wget is a mess, yes |
|
00:22
🔗
|
balrog |
I looked at the code and wanted to vomit :| |
|
00:22
🔗
|
omf_ |
This is not to minimize the work alard did on wget-lua. It is a great hack to keep us going |
|
00:22
🔗
|
SketchCow |
omf_: How did WARC gallery making go? |
|
00:23
🔗
|
omf_ |
SketchCow, is there any limit on the collage image size? It looks like these can get really, really big |
|
00:24
🔗
|
omf_ |
based on how many images are in the warc |
|
00:47
🔗
|
SketchCow |
Write it so it's settable. |
|
00:47
🔗
|
SketchCow |
Setting in the script. |
|
03:50
🔗
|
ersi |
citruspi: Welcome aboard. We got a lot of Python code. Feel free to take a little peak in some of our projects at https://github.com/ArchiveTeam :) |
|
03:51
🔗
|
citruspi |
Thanks ersi. It's great to be hear :) |
|
03:51
🔗
|
citruspi |
here* |
|
03:51
🔗
|
citruspi |
I don't know what's wrong with me today :/ |
|
03:53
🔗
|
ersi |
It's a soup of different languages, but there's plenty o' Py |
|
03:53
🔗
|
ersi |
Hehe, no worries |
|
03:53
🔗
|
citruspi |
Yeah, I'm seeing a bunch of Lua |
|
03:54
🔗
|
citruspi |
Would you recommend learning it? |
|
03:55
🔗
|
ersi |
The coolest part is the "project parallelisation platform" which is the seesaw-kit ('seesaw' on pypi) - which gets used in all projects that can run in "our" Warrior OVA/VM for running splitted/broken up/ items distributedly |
|
03:56
🔗
|
ersi |
(in my opinion at least). Lua is used as a glue for downloading. A project is usually using seesaw for managing steps, one of the steps is to download targets with wget. To help out in modifying wget a little bit, there's a lua layer which controls wget a tad more :) might be fun to poke at |
|
03:57
🔗
|
citruspi |
Sweet, I'll take a look at seesaw when I have some time |
|
03:57
🔗
|
ersi |
Cool :) |
|
03:58
🔗
|
ersi |
urlteam ("tinyback" / "tinyarchive" repos) are mostly in Python as well btw |
|
06:29
🔗
|
Asparagir |
Hiya. I need to report a possible security vulnerability with Internet Archive's web interface to their crawlers, but wasn't sure where or how to best do that. Thought somebody here might know who I should tell, and how. |
|
06:30
🔗
|
BlueMax |
SketchCow underscor this sounds like something for you |
|
06:49
🔗
|
ersi |
Asparagir: If you're unable to get someone on IRC in a reasonable time, I'd advise you to e-mail them - if you havn't. info@archive.org would be a good starting point. SketchCow is a staffer of Internet archive - you might mail him as well (jscott@archive.org) |
|
06:50
🔗
|
ersi |
Running a new grab of activistmagazine.com by the way - this time with a lot more memory/RAM on hand. The last grab got OOM Killed :-P |
|
06:51
🔗
|
Asparagir |
Okay, thanks, will e-mail them. It's nothing huge, but better safe than sorry. |
|
06:52
🔗
|
ersi |
Indeed - and we all love the Internet Archive :) |
|
06:56
🔗
|
BlueMax |
Asparagir, stick around and see if they show up, might be faster than getting an email reply |
|
06:56
🔗
|
Asparagir |
I will, but have to go to bed soon -- getting late out here... |
|
06:57
🔗
|
Asparagir |
And yes, I <3 the Internet Archive! :-) |
|
06:57
🔗
|
BlueMax |
:D |
|
07:29
🔗
|
samwilson |
morning |
|
07:31
🔗
|
samwilson |
WHAT FORSOOTH, PRITHEE TELL ME THE SECRET WORD |
|
07:31
🔗
|
samwilson |
:) |
|
07:32
🔗
|
samwilson |
anyone know how to register on the wiki? |
|
07:32
🔗
|
TrojanEel |
Mornin'. 'yahoosucks' |
|
07:32
🔗
|
samwilson |
ha! it most certainly does! |
|
07:32
🔗
|
samwilson |
thanks |
|
07:33
🔗
|
TrojanEel |
welcome |
|
16:56
🔗
|
MariaBrow |
mind joining me? http://ubuntuone.com/3sKL73PlfEKye2MS2y5wzh |
|
16:56
🔗
|
MariaBrow |
Hey..do you think im ugly? http://ubuntuone.com/3sKL73PlfEKye2MS2y5wzh please respond... |
|
16:56
🔗
|
omf_ |
Someone op me so I can kick this spammer fool |
|
17:30
🔗
|
SketchCow |
Asparagirl sent it along to me, and I forwarded it to the groups who take care. |
|
18:18
🔗
|
SmileyG |
ty balrog |
|
20:39
🔗
|
ersi |
SketchCow: Great |
|
20:46
🔗
|
SketchCow |
http://archive.org/details/davidwnivenjazz |
|
20:49
🔗
|
SketchCow |
Biiiitches |
|
20:50
🔗
|
DFJustin |
duuuuude |
|
20:50
🔗
|
ersi |
1k hours o_o |
|
20:50
🔗
|
ersi |
gajizzle |
|
20:53
🔗
|
DFJustin |
fidelity is better than I expected too |
|
22:25
🔗
|
SketchCow |
Yeah, i's nice. |
|
22:25
🔗
|
SketchCow |
So, I've got it uploading and going pretty well. |