| Time |
Nickname |
Message |
|
07:38
🔗
|
Nemo_bis |
I'm at 95 wikis downloaded with the new script |
|
07:39
🔗
|
Nemo_bis |
And no way to get it consume a good amount of bandwitdh, perhaps I should go with 50 threads. |
|
07:39
🔗
|
Nemo_bis |
underscor, you should run at least 100 for 1000 wikis. ;) |
|
08:29
🔗
|
mutoso |
Um... How do I claim a list? |
|
08:59
🔗
|
Nemo_bis |
mutoso, I suppose you don't have access to our wiki, so just tell me |
|
08:59
🔗
|
mutoso |
Yeah, I don't. |
|
08:59
🔗
|
mutoso |
I started list11. |
|
09:00
🔗
|
Nemo_bis |
ok |
|
09:00
🔗
|
Nemo_bis |
mutoso, did you split the list? |
|
09:00
🔗
|
Nemo_bis |
jobs are very long but not resource-intensive |
|
09:00
🔗
|
Nemo_bis |
I mean split -l 10 list011 list011 |
|
09:00
🔗
|
mutoso |
Oh. I see. No. I didn't. |
|
09:01
🔗
|
Nemo_bis |
I suggest you to to do so. |
|
09:01
🔗
|
Nemo_bis |
Also, how do we contact you for updates? Are you on our mailing list? |
|
09:02
🔗
|
Nemo_bis |
mutoso, I have to go now, join http://groups.google.com/group/wikiteam-discuss if you didn't yet. Thank you! |
|
09:09
🔗
|
mutoso |
Alright. I split the list and joined the mailing list. I'm heading to bed, but I'll be idleing in here. Feel free to ping me. |
|
14:48
🔗
|
underscor |
Nemo_bis: I've been doing 10 threads of 10 |
|
14:48
🔗
|
underscor |
:P |
|
16:25
🔗
|
Nemo_bis |
underscor, slacker |
|
16:55
🔗
|
underscor |
hahahah |
|
17:05
🔗
|
Nemo_bis |
underscor, I'm running 60+ instances on my desktop and I have only an AMD E-350 ! |
|
17:06
🔗
|
Nemo_bis |
208 wikis downloaded so far |
|
17:18
🔗
|
Nemo_bis |
come on, underscor, launch some dozens of instances :) |
|
20:29
🔗
|
Nemo_bis |
emijrp, I'm at 223 wikis archived. |
|
20:30
🔗
|
Nemo_bis |
And I've consumed half of my free disk space, so either you create a script for the upload soon or I'll have to do some aggressive disk cleanup. |
|
20:33
🔗
|
emijrp |
or you get a life |
|
20:35
🔗
|
emijrp |
I can't create a script to upload. That is the usual s3 or form upload at Internet Archive. |
|
20:37
🔗
|
Nemo_bis |
get a life? scripts are doing everything |
|
20:37
🔗
|
Nemo_bis |
Well, you wrote to wait for instructions about upload... |
|
20:38
🔗
|
emijrp |
yes, we have to decide a prodecure |
|
20:38
🔗
|
emijrp |
procedure |
|
20:38
🔗
|
Nemo_bis |
The script is to check what dumps have been completed, fetch the URL from the config file, take the name and license of the wiki from its API, upload with metadata and delete files. |
|
20:38
🔗
|
Nemo_bis |
It's quite a lot of paperwork otherwise. |
|
20:39
🔗
|
emijrp |
ok, that script is easy |
|
20:39
🔗
|
Nemo_bis |
yeah |
|
20:39
🔗
|
emijrp |
i hope other guy step forward, and code some lines |
|
20:39
🔗
|
Nemo_bis |
what about you, underscor :-D |
|
20:51
🔗
|
emijrp |
Nemo_bis: run: wc -l */*titles.txt |
|
20:51
🔗
|
emijrp |
post the sum |
|
20:52
🔗
|
Nemo_bis |
2582906 |
|
20:52
🔗
|
Nemo_bis |
biggest wikis still running though |
|
20:52
🔗
|
underscor |
Nemo_bis: I will! |
|
20:52
🔗
|
Nemo_bis |
underscor, \o/ |
|
20:52
🔗
|
underscor |
I'm working on an automated interface for it |
|
20:52
🔗
|
underscor |
because keeping track of 60 shells is a pain in the ass |
|
20:52
🔗
|
Nemo_bis |
heh |
|
20:53
🔗
|
Nemo_bis |
well, that's another issue; now that launcher.py resumes all uncompleted jobs, I just run them in a detached screen and forget about them, then rerun |
|
20:53
🔗
|
Nemo_bis |
for the upload, remember to follow the correct format :p https://code.google.com/p/wikiteam/wiki/NewTutorial#Publishing_the_dump |
|
20:56
🔗
|
emijrp |
you can work in a launcher for the launcher which launch launcher.py |
|
20:56
🔗
|
emijrp |
3 fucking levels of downloading and hoarding wikis. Inception. |
|
20:57
🔗
|
Nemo_bis |
:D |
|
21:01
🔗
|
emijrp |
60 shells? you have claimed just a list underscor |
|
21:15
🔗
|
Nemo_bis |
underscor, even better, put also the api.php URL in the metadata, invent something or abuse an existing field :p |
|
21:15
🔗
|
Nemo_bis |
emijrp, did you see the new issues I filed? :) |
|
21:24
🔗
|
emijrp |
more? |
|
21:24
🔗
|
emijrp |
lol no, thank |
|
21:24
🔗
|
emijrp |
what a nightmare |
|
21:32
🔗
|
Nemo_bis |
:D |
|
21:33
🔗
|
Nemo_bis |
the issue about redirects should be easy to solve, probably just some flag in urllib or something |
|
21:36
🔗
|
emijrp |
seeya |
|
22:13
🔗
|
underscor |
<emijrp> 60 shells? you have claimed just a list underscor |
|
22:14
🔗
|
underscor |
I was just commenting on Nemo_bis's thing about 60 simultaneous |
|
22:14
🔗
|
underscor |
oh, he left |
|
22:14
🔗
|
underscor |
dammit |