| Time |
Nickname |
Message |
|
00:00
🔗
|
godane |
if anyone wants to copy the roms here go ahead : https://docs.google.com/spreadsheets/d/e/2PACX-1vRnrFguca-cXWAcMr0jn4VU50avjGaLIJIF7oN6gzuoG43zjh_mUTLXBszwrSH-b6NGRsxcfgpGJaJw/pubhtml# |
|
00:00
🔗
|
godane |
https://old.reddit.com/r/Roms/comments/bhrkib/byalvro_retiring/ |
|
00:02
🔗
|
ATrescue_ |
godane: I sent the spreadsheet to chromebot. |
|
00:02
🔗
|
godane |
thanks |
|
00:40
🔗
|
|
bitBaron has joined #archiveteam-bs |
|
00:42
🔗
|
|
bitBaron has quit IRC (Read error: Operation timed out) |
|
01:01
🔗
|
|
omarroth has joined #archiveteam-bs |
|
01:12
🔗
|
|
ATrescue_ has quit IRC (Ping timeout: 260 seconds) |
|
01:13
🔗
|
|
ATrescue has joined #archiveteam-bs |
|
01:15
🔗
|
ATrescue |
Welcome, omarroth! Thank you for visiting us again! |
|
01:18
🔗
|
ATrescue |
YouTuber who bought views: |
|
01:18
🔗
|
ATrescue |
chromebot: a https://www.youtube.com/watch?v=bTokyVe8E3E |
|
01:18
🔗
|
ATrescue |
Oops, wrong channel. |
|
01:19
🔗
|
|
omarroth has quit IRC (Quit: Konversation terminated!) |
|
01:19
🔗
|
|
BlueMax has quit IRC (Remote host closed the connection) |
|
01:22
🔗
|
|
ATrescue_ has joined #archiveteam-bs |
|
01:24
🔗
|
|
enowaldo has joined #archiveteam-bs |
|
01:25
🔗
|
|
ATrescue has quit IRC (Ping timeout: 260 seconds) |
|
01:31
🔗
|
|
enowaldo has quit IRC (Read error: Operation timed out) |
|
01:45
🔗
|
|
Rome has quit IRC (Read error: Connection reset by peer) |
|
01:47
🔗
|
|
Rome has joined #archiveteam-bs |
|
01:51
🔗
|
ATrescue_ |
https://transfer.notkiska.pw/ has HTTP 502 error. |
|
01:56
🔗
|
JAA |
kiska: ^ |
|
01:57
🔗
|
|
icedice2 has joined #archiveteam-bs |
|
02:03
🔗
|
|
icedice has quit IRC (Read error: Operation timed out) |
|
02:16
🔗
|
ATrescue_ |
I guess the YouTube producer “Simplicissimus” is soon going to expose Yo Oli. |
|
02:27
🔗
|
|
ATrescue has joined #archiveteam-bs |
|
02:28
🔗
|
|
icedice2 has quit IRC (Quit: Leaving) |
|
02:28
🔗
|
ATrescue |
Had a browser crash. |
|
02:29
🔗
|
|
ATrescue_ has quit IRC (Ping timeout: 260 seconds) |
|
03:18
🔗
|
|
odemgi has joined #archiveteam-bs |
|
03:20
🔗
|
|
BlueMax has joined #archiveteam-bs |
|
03:20
🔗
|
|
odemgi_ has quit IRC (Ping timeout: 252 seconds) |
|
03:21
🔗
|
|
w00dsman has joined #archiveteam-bs |
|
03:22
🔗
|
|
Dj-Wawa has quit IRC (Quit: Connection closed for inactivity) |
|
03:23
🔗
|
|
BlueMax has quit IRC (Read error: Connection reset by peer) |
|
03:27
🔗
|
|
odemg has quit IRC (Ping timeout: 615 seconds) |
|
03:28
🔗
|
|
Zerote has quit IRC (Read error: Operation timed out) |
|
03:31
🔗
|
|
w00dsman has quit IRC (Leaving) |
|
03:31
🔗
|
arkiver |
PurpleSym: nice! can you please ping me again in like 12 hours in case I forget? |
|
03:31
🔗
|
arkiver |
We can probably do a project |
|
03:33
🔗
|
|
odemg has joined #archiveteam-bs |
|
03:34
🔗
|
godane |
SketchCow: maybe another guy to put under vhsvault collection : https://archive.org/details/@pannoni4 |
|
04:18
🔗
|
|
Odd0002_ has joined #archiveteam-bs |
|
04:18
🔗
|
|
Odd0002 has quit IRC (Read error: Operation timed out) |
|
04:18
🔗
|
|
Odd0002_ is now known as Odd0002 |
|
04:35
🔗
|
|
BlueMax has joined #archiveteam-bs |
|
04:56
🔗
|
|
BlueMax has quit IRC (Read error: Connection reset by peer) |
|
05:24
🔗
|
|
Despatche has quit IRC (Quit: Read error: Connection reset by deer) |
|
05:25
🔗
|
kiska |
uhhh..... OOM killed transfersh... |
|
05:31
🔗
|
|
killsushi has quit IRC (Quit: Leaving) |
|
05:34
🔗
|
kiska |
JAA: I am getting a very weird traceback on transfersh, it was killed because it ran out of memory, but grafana tells me it still had 35% memory remaining on the host |
|
05:38
🔗
|
kiska |
Please don't upload mp4's... it crashes transfersh |
|
05:43
🔗
|
marked |
per process memory quota? |
|
05:45
🔗
|
Flashfire |
ATrescue |
|
05:45
🔗
|
Flashfire |
hear that? |
|
05:51
🔗
|
kiska |
Here is the crash report: https://pastebin.com/YxMCnXQd |
|
05:56
🔗
|
|
ugh has joined #archiveteam-bs |
|
05:57
🔗
|
|
Atom-- has quit IRC (irc.efnet.nl efnet.deic.eu) |
|
05:57
🔗
|
|
atomicthu has quit IRC (irc.efnet.nl efnet.deic.eu) |
|
05:57
🔗
|
|
katocala has quit IRC (irc.efnet.nl efnet.deic.eu) |
|
05:57
🔗
|
|
thejsa has quit IRC (irc.efnet.nl efnet.deic.eu) |
|
05:57
🔗
|
|
legoktm has quit IRC (irc.efnet.nl efnet.deic.eu) |
|
05:57
🔗
|
|
chfoo has quit IRC (irc.efnet.nl efnet.deic.eu) |
|
05:57
🔗
|
|
halt_ has quit IRC (irc.efnet.nl efnet.deic.eu) |
|
06:00
🔗
|
|
thejsa_ has joined #archiveteam-bs |
|
06:03
🔗
|
|
chfoo-_ has joined #archiveteam-bs |
|
06:11
🔗
|
|
BlueMax has joined #archiveteam-bs |
|
06:16
🔗
|
marked |
I don't pretend to understand this, seems to be steps for memory profiling a go app https://groups.google.com/forum/#!msg/golang-nuts/0l8k1z1owVk/O3DRhxZGAgAJ |
|
06:20
🔗
|
|
atomicthu has joined #archiveteam-bs |
|
06:56
🔗
|
|
tomaspark has quit IRC (Remote host closed the connection) |
|
07:00
🔗
|
|
morgan_ has joined #archiveteam-bs |
|
07:21
🔗
|
|
bitBaron has joined #archiveteam-bs |
|
07:24
🔗
|
|
bitBaron has quit IRC (Read error: Operation timed out) |
|
08:03
🔗
|
|
underscor has quit IRC (Ping timeout: 615 seconds) |
|
08:17
🔗
|
|
Dimtree has joined #archiveteam-bs |
|
08:46
🔗
|
|
underscor has joined #archiveteam-bs |
|
09:25
🔗
|
|
SmileyG has joined #archiveteam-bs |
|
09:25
🔗
|
|
Smiley has quit IRC (Read error: Operation timed out) |
|
09:57
🔗
|
JAA |
kiska: Wait wat, that's on a download request, not on the upload? |
|
09:57
🔗
|
kiska |
Yes it was on a GET request |
|
09:57
🔗
|
JAA |
You'd think that the server would simply stream the file from disk with chunked transfer encoding. |
|
09:59
🔗
|
kiska |
Do not try and download that file |
|
09:59
🔗
|
kiska |
Well it won't work anyway since that file doesn't exist since I deleted it |
|
10:00
🔗
|
JAA |
Was just about to suggest that. |
|
10:02
🔗
|
kiska |
I am not even sure whats happening, all I know is that as soon as someone issues a GET request to the server I get that trace report and the server doesn't work. I fixed the server not restarting by using a while true statement xD |
|
10:07
🔗
|
|
GuysFree has quit IRC (Quit: Connection closed for inactivity) |
|
10:09
🔗
|
JAA |
Probably this line here: https://github.com/dutchcoders/transfer.sh/blob/cd602b9256fb77e705da3cf6e09323e2eab233f6/server/handlers.go#L905 |
|
10:10
🔗
|
JAA |
ioutil.ReadAll: "reads from r until an error or EOF and returns the data it read" https://golang.org/pkg/io/ioutil/#ReadAll |
|
10:11
🔗
|
JAA |
Why they do that is beyond me. |
|
10:11
🔗
|
JAA |
Looks like they don't even use the return value, only the error code... |
|
10:11
🔗
|
kiska |
Well I opened the issue at https://github.com/dutchcoders/transfer.sh/issues/217 |
|
10:12
🔗
|
JAA |
Yup just saw, will add that there. |
|
10:13
🔗
|
kiska |
Yep thanks, I didn't add too many details so xD |
|
10:13
🔗
|
JAA |
That code was added to support Range requests apparently. |
|
10:15
🔗
|
kiska |
Eh.... |
|
10:15
🔗
|
JAA |
https://github.com/dutchcoders/transfer.sh/commit/af80d889aff6a70fc69ce1947b302fdb8b557b82 |
|
10:17
🔗
|
kiska |
I am not going to question it, not going to |
|
10:24
🔗
|
|
Zerote has joined #archiveteam-bs |
|
10:43
🔗
|
Kaz |
kiska: right at the top of your error: fatal error: runtime: out of memory |
|
10:43
🔗
|
Kaz |
what does memory usage look like? |
|
10:43
🔗
|
kiska |
Yeah except according to grafana there was still 300MiB free |
|
10:44
🔗
|
kiska |
Kaxz: Here: https://grafana.kiska.pw/d/7NfIlyRWz/telegraf-host-metrics?var-host=ip-172-26-9-138&var-interval=2s&from=now-24h&to=now |
|
10:44
🔗
|
kiska |
Uhh... fingers pls |
|
10:44
🔗
|
Kaz |
don't trust it, that won't see spikes |
|
10:45
🔗
|
kiska |
I was also monitoring via top as well, when the server crashes the minimum free memory was 250MiB |
|
10:49
🔗
|
kiska |
The video file was ~200MiB even if it was reading it all into memory, it shouldn't have crashed with OOM |
|
10:51
🔗
|
JAA |
How fast is your storage? |
|
10:51
🔗
|
JAA |
It's probably reading that into memory very quickly. |
|
10:52
🔗
|
kiska |
xvda aka amazon ebs storage. During the uhohl project I was reading data at 2GB/s sustained and writing >3GB/s |
|
10:53
🔗
|
kiska |
s/3GB/2GB |
|
10:55
🔗
|
JAA |
Right, so it probably fills up that memory before your tools can even notice. |
|
10:55
🔗
|
JAA |
You'd need measurements every 0.1 second or even more to see it. |
|
10:59
🔗
|
kiska |
Can you upload a 200-300MiB file and I'll try 0.1 second ps aux measurement |
|
10:59
🔗
|
JAA |
Sure. |
|
11:01
🔗
|
kiska |
Currently its: root 20760 0.0 2.5 311216 26088 pts/1 Sl+ 05:40 0:00 ./transfersh --provider=local --listener :8080 --temp-path=/transfersh/ --basedir=/transfersh/ |
|
11:01
🔗
|
kiska |
And Mem: 1014492 438452 95332 11144 480708 337760 |
|
11:02
🔗
|
JAA |
Upload should be fine I think. |
|
11:02
🔗
|
JAA |
https://transfer.notkiska.pw/x5bcR/test |
|
11:02
🔗
|
kiska |
... crashed... |
|
11:03
🔗
|
JAA |
On GET? I did a HEAD request, but I'd think that shouldn't trigger the crash. |
|
11:03
🔗
|
|
jut has quit IRC (Ping timeout: 252 seconds) |
|
11:03
🔗
|
kiska |
Someone's lounge client GET requested it |
|
11:04
🔗
|
JAA |
Ah lol |
|
11:05
🔗
|
|
jut has joined #archiveteam-bs |
|
11:05
🔗
|
kiska |
Really?! Its not crashing on demand... |
|
11:06
🔗
|
kiska |
Fine... I'll upload a mp4 file |
|
11:19
🔗
|
|
Verified_ has joined #archiveteam-bs |
|
11:23
🔗
|
kiska |
Finally got it to crash, and indeed it is running out of memory: https://transfer.notkiska.pw/IvPnt/ouput.txt |
|
11:26
🔗
|
kiska |
Hrm... interesting its not running out of memory according to free: http://transfer.notkiska.pw/i1rTs/ouput.txt |
|
11:27
🔗
|
kiska |
My command to run ps aux and free: while true; do date; ps aux | grep transfersh; free; sleep 0.1; done > output |
|
11:28
🔗
|
kiska |
I reduced sleep down to 0.05: http://transfer.notkiska.pw/pjqBq/ouput.txt |
|
11:53
🔗
|
|
BlueMax has quit IRC (Quit: Leaving) |
|
12:16
🔗
|
|
VerifiedJ has joined #archiveteam-bs |
|
12:17
🔗
|
|
katocala has joined #archiveteam-bs |
|
12:23
🔗
|
|
schbirid has joined #archiveteam-bs |
|
12:47
🔗
|
|
enowaldo has joined #archiveteam-bs |
|
12:52
🔗
|
|
bitBaron has joined #archiveteam-bs |
|
12:53
🔗
|
|
t2t2 has quit IRC (Quit: t2t2) |
|
12:57
🔗
|
|
t2t2 has joined #archiveteam-bs |
|
13:55
🔗
|
|
enowaldo has quit IRC (Read error: Operation timed out) |
|
14:00
🔗
|
ATrescue |
Flashfire: kiska: Yes, I heard. I will not upload MP4's. I did not know it was going to crash. |
|
14:03
🔗
|
PurpleSym |
arkiver: This is your reminder for Sketch. The site is very JS-heavy, so it probably won’t play back well in Wayback. |
|
14:22
🔗
|
|
Dimtree has quit IRC () |
|
14:23
🔗
|
|
Dj-Wawa has joined #archiveteam-bs |
|
14:23
🔗
|
|
enowaldo has joined #archiveteam-bs |
|
14:33
🔗
|
|
enowaldo has quit IRC (Read error: Operation timed out) |
|
14:55
🔗
|
|
enowaldo has joined #archiveteam-bs |
|
15:13
🔗
|
|
Dimtree has joined #archiveteam-bs |
|
15:39
🔗
|
|
enowaldo has quit IRC (Ping timeout: 252 seconds) |
|
15:43
🔗
|
|
Despatche has joined #archiveteam-bs |
|
15:59
🔗
|
|
enowaldo has joined #archiveteam-bs |
|
17:16
🔗
|
|
enowaldo has quit IRC (Read error: Operation timed out) |
|
17:17
🔗
|
|
enowaldo has joined #archiveteam-bs |
|
17:38
🔗
|
|
enowaldo has quit IRC (Ping timeout: 252 seconds) |
|
17:55
🔗
|
|
godane has quit IRC (Quit: Leaving.) |
|
17:56
🔗
|
|
tech234a has joined #archiveteam-bs |
|
18:12
🔗
|
|
enowaldo has joined #archiveteam-bs |
|
18:17
🔗
|
|
enowaldo has quit IRC (Ping timeout: 268 seconds) |
|
18:27
🔗
|
|
asdf0101 has joined #archiveteam-bs |
|
18:46
🔗
|
|
enowaldo has joined #archiveteam-bs |
|
18:49
🔗
|
|
bitBaron has quit IRC (Quit: My computer has gone to sleep. 😴😪ZZZzzz…) |
|
18:53
🔗
|
|
icedice has joined #archiveteam-bs |
|
18:55
🔗
|
|
enowaldo has quit IRC (Ping timeout: 265 seconds) |
|
19:16
🔗
|
|
VerifiedJ has quit IRC (Quit: Leaving) |
|
19:23
🔗
|
|
godane has joined #archiveteam-bs |
|
19:24
🔗
|
godane |
so i found a tape about Bobby Orr yesterday so i'm digitizing that |
|
19:25
🔗
|
godane |
i think my computer needs to be unplugged from the wall to fix those usb problems |
|
19:25
🔗
|
godane |
so i maybe able to digitize tapes and stuff again |
|
19:27
🔗
|
godane |
the usb ports are faster then before the unplug and hard drives that was not showing up before show up |
|
19:27
🔗
|
godane |
so there is something too it i think |
|
19:27
🔗
|
godane |
anyways i'm going to test this by digitizing the tapes i currently have |
|
19:28
🔗
|
godane |
and also do a proper digitzing of Dashcloud's christmas tape he sent |
|
19:28
🔗
|
godane |
it was out of sync last time |
|
20:00
🔗
|
|
bitBaron has joined #archiveteam-bs |
|
20:03
🔗
|
godane |
the bobby orr tape came out fine |
|
20:03
🔗
|
godane |
i'm capturing dashcloud christmas tape again |
|
20:08
🔗
|
ATrescue |
purplebot: Sandbox reinstated. |
|
20:31
🔗
|
|
bitBaron has quit IRC (Quit: My computer has gone to sleep. 😴😪ZZZzzz…) |
|
20:39
🔗
|
|
enowaldo has joined #archiveteam-bs |
|
20:44
🔗
|
JAA |
Has anyone here ever worked with warcio and payload digests? Frogging or PurpleSym maybe? It looks to me like warcio is not stripping transfer encoding before calculating the payload hash, leading to invalid hashes in the WARC. |
|
20:46
🔗
|
JAA |
Specifically, the same data is passed to the block and payload digester here: https://github.com/webrecorder/warcio/blob/c64c4394805e13256695f51af072c95389397ee9/warcio/warcwriter.py#L99-L107 |
|
20:49
🔗
|
|
TigerbotH has quit IRC (ZNC - https://znc.in) |
|
20:49
🔗
|
JAA |
wpull's doing the same thing actually: https://github.com/ArchiveTeam/wpull/blob/ddf051aa3322479325ba20aa778cb2cb97606bf5/wpull/warc/format.py#L106-L118 |
|
20:49
🔗
|
|
TigerbotH has joined #archiveteam-bs |
|
20:54
🔗
|
JAA |
Not sure what wget's doing exactly since its code is barely readable, but it is handling blocks and payloads differently at least: https://git.savannah.gnu.org/cgit/wget.git/tree/src/warc.c?h=v1.20.3#n512 (They did check their compliance with the WARC spec recently and change a bunch of things, e.g. the unintended angled brackets around URLs, so that would not be surprising.) |
|
20:55
🔗
|
JAA |
(Side note: https://github.com/iipc/warc-specifications/issues/22 ) |
|
20:56
🔗
|
JAA |
In other words, it seems like all WARCs ever created with wpull or warcio could be invalid. |
|
21:00
🔗
|
JAA |
The wget code does look like it deals with chunked transfer encoding but not with gzip compression, so it's possible that some wget WARCs are also invalid. |
|
21:14
🔗
|
|
chferfa has joined #archiveteam-bs |
|
21:22
🔗
|
|
killsushi has joined #archiveteam-bs |
|
21:34
🔗
|
|
schbirid has quit IRC (Remote host closed the connection) |
|
21:35
🔗
|
|
enowaldo has quit IRC (Read error: Operation timed out) |
|
21:40
🔗
|
JAA |
Confirmed that both wpull and warcio are producing invalid WARC-Payload-Digest values. |
|
21:41
🔗
|
JAA |
For chunked responses, that is. |
|
21:58
🔗
|
JAA |
Filed as https://github.com/ArchiveTeam/wpull/issues/424 and https://github.com/webrecorder/warcio/issues/74 |
|
22:05
🔗
|
|
killsushi has quit IRC (Quit: Leaving) |
|
22:05
🔗
|
|
killsushi has joined #archiveteam-bs |
|
22:07
🔗
|
|
enowaldo has joined #archiveteam-bs |
|
22:25
🔗
|
|
enowaldo has quit IRC (Read error: Operation timed out) |
|
22:51
🔗
|
|
RichardG has quit IRC (Ping timeout: 360 seconds) |
|
22:56
🔗
|
|
chfoo-_ is now known as chfoo |
|
22:58
🔗
|
|
enowaldo has joined #archiveteam-bs |
|
23:09
🔗
|
|
BlueMax has joined #archiveteam-bs |
|
23:10
🔗
|
t3 |
I want to know if https://github.com/webrecorder/webrecorder can be used to automate archiving for stuff with JavaScript for ArchiveBot. |
|
23:12
🔗
|
|
enowaldo has quit IRC (Read error: Operation timed out) |
|
23:37
🔗
|
|
Dj-Wawa has quit IRC (Quit: Connection closed for inactivity) |