| Time |
Nickname |
Message |
|
00:11
🔗
|
|
katocala has quit IRC (Ping timeout: 255 seconds) |
|
00:11
🔗
|
|
katocala has joined #archiveteam-bs |
|
00:26
🔗
|
|
BlueMax has joined #archiveteam-bs |
|
00:31
🔗
|
|
Arcorann has joined #archiveteam-bs |
|
00:44
🔗
|
arkiver |
We now have a new customized wget with full ZSTD compression support |
|
00:45
🔗
|
arkiver |
with ZSTD we can use trained custom dictionaries, which overall decrease the size of certain WARC records enormously |
|
00:45
🔗
|
arkiver |
a trainer is in place which will train a dictionary on the latest WARC record uploaded to IA for a certain project. |
|
00:46
🔗
|
arkiver |
the new dictionary is then used by warrior and workers to compress data with |
|
00:47
🔗
|
arkiver |
this can decrease the size of data with tens of percentages overall, which means less outgoing bandwidth, and less disk usage on IA |
|
00:57
🔗
|
OrIdow6 |
Does this work in wget-lua, or just plain wget? |
|
00:58
🔗
|
arkiver |
it's wget-at, currently at https://github.com/ArchiveTeam/wget-lua/tree/v1.20.3-at and will be merged with master later |
|
00:59
🔗
|
arkiver |
it's renamed to wget-at (from wget-lua) since this now includes more than just the Lua scripting option |
|
00:59
🔗
|
arkiver |
it also includes the deduplication, URL agnostic deduplication option, and ZSTD compression with dictionaries |
|
01:10
🔗
|
|
DLoader_ has joined #archiveteam-bs |
|
01:20
🔗
|
|
DLoader has quit IRC (Ping timeout: 745 seconds) |
|
01:20
🔗
|
|
DLoader_ is now known as DLoader |
|
01:36
🔗
|
|
lennier2 has joined #archiveteam-bs |
|
01:48
🔗
|
|
lennier1 has quit IRC (Ping timeout: 745 seconds) |
|
01:48
🔗
|
|
lennier2 is now known as lennier1 |
|
01:53
🔗
|
|
lennier1 has quit IRC (Quit: Going offline, see ya! (www.adiirc.com)) |
|
01:54
🔗
|
|
Meli has quit IRC (Read error: Operation timed out) |
|
01:55
🔗
|
|
Frogging has quit IRC (Read error: Operation timed out) |
|
01:55
🔗
|
|
Frogging has joined #archiveteam-bs |
|
01:55
🔗
|
|
legoktm has quit IRC (Read error: Operation timed out) |
|
01:56
🔗
|
|
legoktm has joined #archiveteam-bs |
|
01:56
🔗
|
|
Ctrl has quit IRC (Read error: Operation timed out) |
|
01:57
🔗
|
|
Raccoon` has joined #archiveteam-bs |
|
01:58
🔗
|
|
wessel152 has quit IRC (Read error: Operation timed out) |
|
02:00
🔗
|
|
Raccoon has quit IRC (Read error: Operation timed out) |
|
02:00
🔗
|
|
Raccoon` is now known as Raccoon |
|
02:03
🔗
|
|
Pixi has quit IRC (Read error: Connection reset by peer) |
|
02:03
🔗
|
|
Xibalba has quit IRC (Read error: Operation timed out) |
|
02:03
🔗
|
|
Xibalba has joined #archiveteam-bs |
|
02:03
🔗
|
|
Pixi has joined #archiveteam-bs |
|
02:03
🔗
|
|
Xibalba has quit IRC (Handshake flooding) |
|
02:03
🔗
|
|
Xibalba has joined #archiveteam-bs |
|
02:04
🔗
|
|
lennier1 has joined #archiveteam-bs |
|
02:04
🔗
|
|
phuzion_ has joined #archiveteam-bs |
|
02:11
🔗
|
|
jodizzle has quit IRC (Read error: Connection reset by peer) |
|
02:11
🔗
|
|
jodizzle_ has joined #archiveteam-bs |
|
02:13
🔗
|
|
phuzion has quit IRC (Read error: Operation timed out) |
|
02:14
🔗
|
|
jodizzle_ is now known as jodizzle |
|
02:42
🔗
|
|
lindalap has joined #archiveteam-bs |
|
02:43
🔗
|
lindalap |
https://recording.tk.informatik.tu-darmstadt.de/ |
|
02:43
🔗
|
lindalap |
found randomly |
|
02:43
🔗
|
lindalap |
not saved at Wayback Machine either |
|
02:43
🔗
|
|
katocala has quit IRC (Ping timeout: 496 seconds) |
|
02:44
🔗
|
|
katocala has joined #archiveteam-bs |
|
02:52
🔗
|
|
Meli has joined #archiveteam-bs |
|
03:08
🔗
|
|
qw3rty_ has joined #archiveteam-bs |
|
03:15
🔗
|
|
qw3rty__ has quit IRC (Read error: Operation timed out) |
|
03:39
🔗
|
|
katocala has quit IRC (Ping timeout: 272 seconds) |
|
03:40
🔗
|
|
katocala has joined #archiveteam-bs |
|
03:57
🔗
|
|
Ajay1 has quit IRC (Quit: Goodbye) |
|
04:08
🔗
|
|
Yurume has quit IRC (Quit: No Ping reply in 180 seconds.) |
|
04:09
🔗
|
|
Yurume has joined #archiveteam-bs |
|
04:24
🔗
|
|
DogsRNice has quit IRC (Read error: Connection reset by peer) |
|
04:25
🔗
|
|
lindalap has quit IRC (Quit: lindalap) |
|
05:11
🔗
|
|
OrIdow6 has quit IRC (Remote host closed the connection) |
|
05:12
🔗
|
|
OrIdow6 has joined #archiveteam-bs |
|
06:33
🔗
|
|
kiska has quit IRC (Read error: Connection reset by peer) |
|
06:35
🔗
|
|
kiska has joined #archiveteam-bs |
|
06:39
🔗
|
|
atphoenix has quit IRC (Read error: Connection reset by peer) |
|
06:40
🔗
|
|
atphoenix has joined #archiveteam-bs |
|
07:44
🔗
|
|
britmob_ has joined #archiveteam-bs |
|
07:46
🔗
|
|
benjinsmi has joined #archiveteam-bs |
|
07:47
🔗
|
|
atphoenix has quit IRC (Ping timeout: 265 seconds) |
|
07:47
🔗
|
|
britmob has quit IRC (Ping timeout: 265 seconds) |
|
07:48
🔗
|
|
benjinss has quit IRC (Ping timeout: 265 seconds) |
|
07:48
🔗
|
|
atphoenix has joined #archiveteam-bs |
|
08:11
🔗
|
|
BlueMax has quit IRC (Quit: Leaving) |
|
08:15
🔗
|
|
godane has quit IRC (Read error: Operation timed out) |
|
08:17
🔗
|
|
godane has joined #archiveteam-bs |
|
09:13
🔗
|
|
schbirid has joined #archiveteam-bs |
|
09:13
🔗
|
|
katocala has quit IRC (Read error: Operation timed out) |
|
09:13
🔗
|
|
katocala has joined #archiveteam-bs |
|
09:27
🔗
|
|
brayden has quit IRC (Ping timeout: 272 seconds) |
|
09:32
🔗
|
|
brayden has joined #archiveteam-bs |
|
09:38
🔗
|
|
BlueMax has joined #archiveteam-bs |
|
09:40
🔗
|
|
brayden has quit IRC (Ping timeout: 272 seconds) |
|
09:45
🔗
|
|
brayden has joined #archiveteam-bs |
|
09:51
🔗
|
|
Arcorann has quit IRC (Read error: Connection reset by peer) |
|
10:00
🔗
|
|
Ryz has quit IRC (Quit: Ping timeout (120 seconds)) |
|
10:00
🔗
|
|
Ryz has joined #archiveteam-bs |
|
10:20
🔗
|
schbirid |
is there a way to set a referrer when using http://web.archive.org/save |
|
10:20
🔗
|
schbirid |
? |
|
10:30
🔗
|
|
Arcorann has joined #archiveteam-bs |
|
10:51
🔗
|
arkiver |
no |
|
11:14
🔗
|
|
brayden has quit IRC (Ping timeout: 272 seconds) |
|
11:18
🔗
|
|
brayden has joined #archiveteam-bs |
|
12:02
🔗
|
|
Jens has quit IRC (Remote host closed the connection) |
|
12:03
🔗
|
|
Jens has joined #archiveteam-bs |
|
12:09
🔗
|
|
BlueMax has quit IRC (Quit: Leaving) |
|
12:17
🔗
|
|
atphoenix has quit IRC (Ping timeout: 265 seconds) |
|
12:17
🔗
|
|
atphoenix has joined #archiveteam-bs |
|
13:00
🔗
|
Arcorann |
I recently stumbled on a link to a 4chan archive https://yuki.la and noticed it wasn't mentioned at the wiki, probably worth looking into info for that site |
|
13:03
🔗
|
|
HP_Archiv has joined #archiveteam-bs |
|
13:38
🔗
|
|
HP_Archiv has quit IRC (Quit: Leaving) |
|
13:58
🔗
|
|
acridAxid has quit IRC (Quit: marauder) |
|
14:09
🔗
|
arkiver |
BartoCH: hi! did everything go well with setting up wget-at? |
|
14:11
🔗
|
BartoCH |
git clone && docker build -t pastebin-grab . && docker run --detach --name pastebin-grab -v /home/barto/warrior/pastebin-data/:/grab/data/ pastebin-grab:latest Barto --concurrent=20 |
|
14:12
🔗
|
BartoCH |
that is all i had to do |
|
14:12
🔗
|
BartoCH |
ofc i had an impromptu git pull to do ;) |
|
14:13
🔗
|
arkiver |
ah yes |
|
14:13
🔗
|
kiska |
Come to hackint? |
|
14:13
🔗
|
arkiver |
well good! |
|
14:13
🔗
|
kiska |
Also are your machines getting connection limit reached? |
|
14:13
🔗
|
kiska |
I may have to increase the alloc |
|
14:14
🔗
|
BartoCH |
i have so403 me retfinished stuffs yeah |
|
14:14
🔗
|
BartoCH |
damn, internet is laggy now :D |
|
14:14
🔗
|
kiska |
No rsync issues :D |
|
14:14
🔗
|
arkiver |
oof me too |
|
14:14
🔗
|
BartoCH |
aaand, i was banned by pastebin |
|
14:14
🔗
|
kiska |
403's or conn -1 or conn 100? |
|
14:14
🔗
|
BartoCH |
Pastebin.com has blocked your IP |
|
14:14
🔗
|
arkiver |
yep same |
|
14:14
🔗
|
BartoCH |
403 |
|
14:15
🔗
|
kiska |
Ah so not my target :D |
|
14:15
🔗
|
arkiver |
kiska: no |
|
14:15
🔗
|
BartoCH |
arkiver: we kind of got stuck at the same 300 items point |
|
14:15
🔗
|
kiska |
https://server8.kiska.pw/uploads/feb324249491be79/image.png |
|
14:15
🔗
|
kiska |
Circled is NVMe drives flushing cache :D |
|
14:16
🔗
|
kiska |
Time for some hetzner? :P |
|
14:16
🔗
|
kiska |
Does the 403 tell us how long they ban for? |
|
14:17
🔗
|
arkiver |
no |
|
14:17
🔗
|
arkiver |
but it's temporary |
|
14:18
🔗
|
kiska |
Well thats good at least :D |
|
14:29
🔗
|
|
Meli has quit IRC (Remote host closed the connection) |
|
14:30
🔗
|
|
Meli has joined #archiveteam-bs |
|
14:58
🔗
|
|
Meli has quit IRC (Remote host closed the connection) |
|
14:58
🔗
|
|
Meli has joined #archiveteam-bs |
|
15:01
🔗
|
|
Meli has quit IRC (Remote host closed the connection) |
|
15:02
🔗
|
|
Meli has joined #archiveteam-bs |
|
15:19
🔗
|
|
godane has quit IRC (Read error: Connection reset by peer) |
|
15:21
🔗
|
|
Meli has quit IRC (Remote host closed the connection) |
|
15:23
🔗
|
|
brayden has quit IRC (Ping timeout: 272 seconds) |
|
15:26
🔗
|
|
Meli has joined #archiveteam-bs |
|
15:27
🔗
|
|
brayden has joined #archiveteam-bs |
|
15:59
🔗
|
|
Arcorann has quit IRC (Read error: Connection reset by peer) |
|
16:08
🔗
|
|
DogsRNice has joined #archiveteam-bs |
|
16:14
🔗
|
|
DogsRNice has quit IRC (Ping timeout: 265 seconds) |
|
16:16
🔗
|
|
Gfy_ is now known as Gfy |
|
16:18
🔗
|
|
DogsRNice has joined #archiveteam-bs |
|
17:12
🔗
|
|
godane has joined #archiveteam-bs |
|
17:41
🔗
|
|
wyatt8740 has quit IRC (Read error: Operation timed out) |
|
17:42
🔗
|
|
Ryz has quit IRC (Remote host closed the connection) |
|
17:42
🔗
|
|
kiska1825 has quit IRC (Remote host closed the connection) |
|
17:42
🔗
|
|
Ryz has joined #archiveteam-bs |
|
17:42
🔗
|
|
kiska1825 has joined #archiveteam-bs |
|
17:43
🔗
|
|
wyatt8740 has joined #archiveteam-bs |
|
17:53
🔗
|
|
Meli has quit IRC (Remote host closed the connection) |
|
17:55
🔗
|
|
Meli has joined #archiveteam-bs |
|
18:50
🔗
|
|
exoire has joined #archiveteam-bs |
|
18:55
🔗
|
|
wyatt8740 has quit IRC (Read error: Operation timed out) |
|
18:57
🔗
|
|
wyatt8740 has joined #archiveteam-bs |
|
19:16
🔗
|
|
zyphlar_ has joined #archiveteam-bs |
|
19:58
🔗
|
|
Stiletto has quit IRC (Read error: Operation timed out) |
|
20:00
🔗
|
|
Stiletto has joined #archiveteam-bs |
|
20:07
🔗
|
|
asdf0101 has quit IRC (The Lounge - https://thelounge.chat) |
|
20:08
🔗
|
|
asdf0101 has joined #archiveteam-bs |
|
20:09
🔗
|
|
exoire has quit IRC (Client terminated!) |
|
20:22
🔗
|
|
asdf0101 has quit IRC (Remote host closed the connection) |
|
20:25
🔗
|
|
asdf0101 has joined #archiveteam-bs |
|
20:27
🔗
|
|
Dj-Wawa_ has quit IRC (Quit: Dj-Wawa_) |
|
20:28
🔗
|
|
Dj-Wawa has joined #archiveteam-bs |
|
20:37
🔗
|
|
Dj-Wawa has quit IRC (Quit: Dj-Wawa) |
|
20:38
🔗
|
|
Dj-Wawa has joined #archiveteam-bs |
|
20:41
🔗
|
godane |
SketchCow: looks like manualsbase.com has less dups at 11k ids |
|
20:42
🔗
|
godane |
first 16 of that range had zero file sizes that was the same |
|
20:44
🔗
|
godane |
nevermind |
|
20:45
🔗
|
godane |
either way every id is getting uploaded cause metadata title could be better |
|
21:26
🔗
|
|
zyphlar_ has quit IRC (Quit: Connection closed for inactivity) |
|
21:28
🔗
|
|
katocala has quit IRC (Ping timeout: 745 seconds) |
|
21:28
🔗
|
|
katocala has joined #archiveteam-bs |
|
21:35
🔗
|
|
katocala has quit IRC (Ping timeout: 255 seconds) |
|
21:35
🔗
|
|
katocala has joined #archiveteam-bs |
|
22:14
🔗
|
|
acridAxid has joined #archiveteam-bs |
|
22:45
🔗
|
|
britmob_ has quit IRC (Ping timeout: 265 seconds) |
|
22:47
🔗
|
|
britmob has joined #archiveteam-bs |
|
23:47
🔗
|
|
Stiletto has quit IRC (Read error: Connection reset by peer) |
|
23:49
🔗
|
|
Stiletto has joined #archiveteam-bs |
|
23:52
🔗
|
|
Stiletto has quit IRC (Read error: Connection reset by peer) |
|
23:52
🔗
|
|
Stiletto has joined #archiveteam-bs |
|
23:54
🔗
|
|
wyatt8740 has quit IRC (Read error: Operation timed out) |
|
23:55
🔗
|
|
wyatt8740 has joined #archiveteam-bs |
|
23:57
🔗
|
|
Mateon1 has quit IRC (Read error: Operation timed out) |
|
23:57
🔗
|
|
Mateon1 has joined #archiveteam-bs |