#internetarchive.bak 2016-11-14,Mon

↑back Search

Time Nickname Message
00:25 🔗 arkiver HCross: you made just one :P
00:25 🔗 arkiver or I did
01:30 🔗 cmaldonad has joined #internetarchive.bak
01:52 🔗 cmaldonad has quit IRC (Quit: This computer has gone to sleep)
01:53 🔗 cmaldonad has joined #internetarchive.bak
02:16 🔗 cmaldonad has quit IRC (Quit: This computer has gone to sleep)
02:20 🔗 cmaldonad has joined #internetarchive.bak
02:24 🔗 cmaldonad has quit IRC (Client Quit)
02:42 🔗 cmaldonad has joined #internetarchive.bak
02:44 🔗 cmaldonad has quit IRC (Client Quit)
02:47 🔗 cmaldonad has joined #internetarchive.bak
02:56 🔗 cmaldonad has quit IRC (Quit: This computer has gone to sleep)
03:00 🔗 cmaldonad has joined #internetarchive.bak
03:10 🔗 cmaldonad has quit IRC (Quit: This computer has gone to sleep)
03:20 🔗 cmaldonad has joined #internetarchive.bak
03:30 🔗 cmaldonad has quit IRC (Quit: This computer has gone to sleep)
03:39 🔗 VADemon has quit IRC (Quit: left4dead)
04:04 🔗 cmaldonad has joined #internetarchive.bak
04:22 🔗 cmaldonad has quit IRC (Quit: This computer has gone to sleep)
04:35 🔗 cmaldonad has joined #internetarchive.bak
04:54 🔗 cmaldonad has quit IRC (Quit: This computer has gone to sleep)
05:05 🔗 Frogging lol this 60+ GB .MOV file
05:05 🔗 Frogging 2015_02_26_VICTORY_FOR_THE_NET_FULL_EVENT.MOV
05:38 🔗 kyan has quit IRC (Remote host closed the connection)
06:52 🔗 Start has quit IRC (Quit: Disconnected.)
06:52 🔗 Start has joined #internetarchive.bak
08:04 🔗 atomotic has joined #internetarchive.bak
08:29 🔗 atomotic_ has joined #internetarchive.bak
08:29 🔗 atomotic has quit IRC (Ping timeout: 260 seconds)
09:46 🔗 Jon mornings. 444G ~50% of my first chunk of iabak'ing done now. yay.
11:53 🔗 atomotic_ has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
12:44 🔗 atomotic has joined #internetarchive.bak
14:03 🔗 atomotic has quit IRC (Ping timeout: 260 seconds)
14:15 🔗 atomotic has joined #internetarchive.bak
14:43 🔗 atomotic has quit IRC (Ping timeout: 260 seconds)
14:51 🔗 atomotic has joined #internetarchive.bak
15:20 🔗 cmaldonad has joined #internetarchive.bak
15:28 🔗 kyan has joined #internetarchive.bak
15:43 🔗 atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
15:43 🔗 kyan has quit IRC (Remote host closed the connection)
16:11 🔗 kyan has joined #internetarchive.bak
17:30 🔗 Start has quit IRC (Remote host closed the connection)
17:31 🔗 Start has joined #internetarchive.bak
17:36 🔗 Start has quit IRC (Read error: Connection reset by peer)
17:39 🔗 Start has joined #internetarchive.bak
17:52 🔗 atomotic has joined #internetarchive.bak
18:28 🔗 cmaldonad has quit IRC (Quit: This computer has gone to sleep)
18:37 🔗 SketchCow HEY THERE
18:37 🔗 SketchCow So, I'm about to go to DataHoarder
18:37 🔗 SketchCow So expect that influx.
18:43 🔗 yipdw oh great
18:43 🔗 yipdw i can't wait for the new season of Grand Designs
18:44 🔗 SketchCow Hence the warning
18:44 🔗 SketchCow Plus we need to be concocting the FAQs anyway
18:44 🔗 yipdw yep
18:44 🔗 SketchCow Once we go past 1,000 participants
18:44 🔗 SketchCow We're dealing with scale issues anyway, that will be one (massive variant opinions)
18:45 🔗 yipdw scale issues in which part of the system?
18:48 🔗 atomotic has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
18:57 🔗 yipdw there are definitely rough spots w/r/t concurrent creation and on-lining of shards, but I think we have all the primitives to solve that
18:57 🔗 yipdw (for example, one problem is the lack of any sort of locking on the shard counter, which is one reason SHARD13 got skipped)
18:57 🔗 yipdw but there's solutions to explore there
18:58 🔗 db48x has joined #internetarchive.bak
18:59 🔗 yipdw actually, I kinda like that there is no SHARD13
18:59 🔗 yipdw much like I like buildings that have no floor 13
19:00 🔗 yipdw one solution is to not fix that :P
19:13 🔗 joepie91 lol
19:18 🔗 db48x heh
19:19 🔗 db48x what's the actual problem?
19:23 🔗 cmaldonad has joined #internetarchive.bak
19:25 🔗 cmaldonad has quit IRC (Client Quit)
19:28 🔗 SketchCow https://www.reddit.com/r/DataHoarder/comments/5cxtfl/a_plea_to_datahoarders_to_lend_space_for_the/
19:38 🔗 Lord_Nigh SketchCow: see pm
19:57 🔗 VADemon has joined #internetarchive.bak
20:05 🔗 yipdw db48x: oh I was reading over the backlog and mulling over ways we can use machines to help us keep shard numbers straight
20:06 🔗 yipdw it seems to me that this is a problem that can be solved with optimistic locking
20:06 🔗 yipdw aka "oh, the git commit failed, I guess I'd better fetch/merge"
20:06 🔗 yipdw no big changes, just formalizing shard onlining workflow. i'll have to set up a shard myself before actually saying more
20:11 🔗 db48x :)
20:11 🔗 db48x could just be as simple as a text file in the repository
20:12 🔗 yipdw ye
20:12 🔗 yipdw s
20:12 🔗 db48x ever use org-mode?
20:13 🔗 yipdw I have not used emacs much at all, but every day I get more evidence I should atone of my evil ways
20:13 🔗 db48x | SHARD12 | db48x | active | all archiveteam_fire items from 2011-2015 |
20:13 🔗 yipdw and do so
20:13 🔗 db48x :)
20:13 🔗 yipdw but seriously that's like getting me to learn Dvorak
20:13 🔗 yipdw the keyboard layout not the composer
20:14 🔗 db48x you don't even need emacs to edit an org-mode file
20:14 🔗 yipdw true
20:14 🔗 yipdw oh, there's a vim plugin for orgmode
20:14 🔗 db48x nice
20:14 🔗 yipdw never mind, I can keep on my decadent path
20:15 🔗 db48x heh
20:15 🔗 db48x I'm getting ready to disembark
20:15 🔗 db48x bbiab
20:15 🔗 yipdw ok
20:19 🔗 db48x has quit IRC (Ping timeout: 255 seconds)
20:46 🔗 db48x has joined #internetarchive.bak
21:42 🔗 db48x is the stats page really slow for everyone, or is that my internet connection?
21:43 🔗 db48x uh, also ALL.html was last regenerated in May
21:44 🔗 db48x oh, maybe the script was changed to generate index.html instead of ALL.html?
21:44 🔗 db48x that would explain it
21:47 🔗 Kaz very slow in general for me
21:48 🔗 db48x hmm
22:11 🔗 SketchCow It appears that one of the stats page is not working for some images.
22:11 🔗 SketchCow I can't make changes
22:18 🔗 Rye has quit IRC (Remote host closed the connection)
22:25 🔗 db48x SketchCow: which one?
22:27 🔗 Xibalba has joined #internetarchive.bak
22:30 🔗 SketchCow The "graph over time" one.
22:30 🔗 SketchCow I see a broken one
22:32 🔗 db48x well, that's something we can control
22:33 🔗 Xibalba what kind of hardware do you recommend running this?
22:33 🔗 db48x is this on a shard page, or an individual's stat page?
22:33 🔗 db48x Xibalba: it doesn't need much cpu or memory at all
22:34 🔗 db48x ah, http://iabak.archiveteam.org:8080/ give me an error 500
22:34 🔗 db48x same when trying to render a grap
22:34 🔗 db48x h
22:37 🔗 Xibalba db48x: I was wonder, how well would this work say on a raspberry pi 3?
22:39 🔗 HCross2 Xibalba: ask Medowar
22:39 🔗 db48x has quit IRC (Remote host closed the connection)
22:40 🔗 db48x has joined #internetarchive.bak
22:50 🔗 db48x has quit IRC (Ping timeout: 255 seconds)
22:54 🔗 db48x has joined #internetarchive.bak
22:54 🔗 SketchCow Someone offered a service with 1pb of space towards this
22:54 🔗 SketchCow Hence my discussion of shardmasters being at the ready
22:55 🔗 SketchCow I would assign someone to work with this group so it happens quickly
22:55 🔗 yipdw we'll fill a substantial amount of that with the archivebot shards
22:56 🔗 yipdw well, if you consider (128 / 1000) TB a substantial amount, anyway
22:57 🔗 yipdw er ((128 / 1000) * 100)% etc etc units feh feh
22:58 🔗 cmaldonad has joined #internetarchive.bak
22:58 🔗 SketchCow http://iabak.archiveteam.org:8080/render/?width=900&height=500&_salt=1428621391.124&target=legendValue(alias(color(scale(divideSeries(diffSeries(sumSeries(keepLastValue(iabak.shardstats.filecount.*))%2CsumSeries(keepLastValue(iabak.shardstats.numcopies.0.*)%2CkeepLastValue(iabak.shardstats.numcopies.1.*)%2CkeepLastValue(iabak.shardstats.numcopies.2.*)))%2CsumSeries(keepLastValue(iabak.shards
22:58 🔗 SketchCow tats.filecount.*)))%2C100)%2C%27%2300dd00%27)%2C%27%3E%3D3%20backups%27)%2C%27last%27)&target=legendValue(alias(color(scale(divideSeries(sumSeries(keepLastValue(iabak.shardstats.numcopies.2.*))%2CsumSeries(keepLastValue(iabak.shardstats.filecount.*)))%2C100)%2C%27%2393dd93%27)%2C%272%20backups%27)%2C%27last%27)&target=legendValue(alias(color(scale(divideSeries(sumSeries(keepLastValue(iabak.s
22:58 🔗 SketchCow hardstats.numcopies.1.*))%2CsumSeries(keepLastValue(iabak.shardstats.filecount.*)))%2C100)%2C%27%23e89393%27)%2C%271%20backup%27)%2C%27last%27)&target=legendValue(alias(color(scale(divideSeries(sumSeries(keepLastValue(iabak.shardstats.numcopies.0.*))%2CsumSeries(keepLastValue(iabak.shardstats.filecount.*)))%2C100)%2C%27red%27)%2C%27IA%20only%27)%2C%27last%27)&areaMode=stacked&from=-1weeks&vt
22:59 🔗 SketchCow itle=%25&yMax=100&yMin=0&title=Overall%20progress%2C%20%25
22:59 🔗 SketchCow Sorry, that's the URL that's dying when I connect to it
23:01 🔗 db48x yea, all requests to iabak.archiveteam.org:8080 fail
23:01 🔗 db48x and I can't ssh into the server for whatever reason
23:02 🔗 yipdw i can look into it in a bit
23:03 🔗 SketchCow It'll help as we add more people, especially big hitters
23:12 🔗 yipdw hmm
23:12 🔗 yipdw root@iabak:/etc/apache2/sites-enabled# systemctl restart apache2
23:12 🔗 yipdw Failed to restart apache2.service: Activation of org.freedesktop.systemd1 timed out
23:12 🔗 yipdw See system logs and 'systemctl status apache2.service' for details.
23:12 🔗 yipdw that's fun
23:13 🔗 yipdw huh, there are also an asston of zombie git processes
23:14 🔗 yipdw where the precise definition of 'asston' is
23:14 🔗 yipdw $ ps waxu | grep 'Z.*\[git' | wc -l
23:14 🔗 yipdw 6059
23:14 🔗 yipdw should I just kick iabak in the head and see if we can diagnose this when it's not in such a wonky state
23:15 🔗 db48x heh
23:15 🔗 db48x well
23:19 🔗 db48x those shouldn't affect apache
23:20 🔗 db48x I don't think we have a graceful way to shut down the server, but doing it ungracefully won't be any worse than any other kind of transient network error, as far as the clients are concerned
23:20 🔗 yipdw they shouldn't affect apache, it just occurred to me that something looks off
23:20 🔗 yipdw interestingly, there's nothing in the graphite error log
23:21 🔗 yipdw oh wait, no, there is
23:21 🔗 yipdw I was looking at the wrong one
23:22 🔗 db48x how is there a wrong one?
23:22 🔗 yipdw the most recent log file has a .1 suffix, as if there's something funky with log rotation
23:22 🔗 yipdw I was expecting it to be graphite_error.log
23:23 🔗 db48x :P
23:23 🔗 db48x we should find a way to send it all to journald and let it sort it out
23:23 🔗 db48x at some point
23:23 🔗 yipdw in any case, https://gist.githubusercontent.com/yipdw/930b00f4d2bdb1fec86e475e52aea050/raw/39be1045a9a367759494b6a4b51b00fe8283e5d3/gistfile1.txt
23:25 🔗 db48x [Mon Nov 14 17:23:13.406498 2016] [wsgi:error] [pid 13397] [remote 50.172.238.89:49760] File "/usr/lib/python2.7/dist-packages/django/apps/registry.py", line 124, in check_apps_ready
23:25 🔗 db48x [Mon Nov 14 17:23:13.406507 2016] [wsgi:error] [pid 13397] [remote 50.172.238.89:49760] raise AppRegistryNotReady("Apps aren't loaded yet.")
23:26 🔗 db48x apparently apps aren't loaded yet
23:26 🔗 yipdw yeah
23:26 🔗 yipdw I'm not yet familiar enough with Django to know what that means
23:27 🔗 yipdw but maaybe the internet knows
23:27 🔗 db48x well, the front fell off, for one
23:29 🔗 VADemon has quit IRC (Quit: left4dead)
23:43 🔗 dingo has joined #internetarchive.bak
23:44 🔗 dingo hi there
23:45 🔗 db48x dingo: howdy
23:46 🔗 dingo Came here from the post to /r/datahoarder. having a few issues getting a node up. here is the errors I'm getting https://imgur.com/a/InUyj
23:48 🔗 db48x interesting
23:48 🔗 db48x you're running it as root, which is not normally advisable
23:48 🔗 db48x is this a special case of some type?
23:49 🔗 dingo I was just playing around tbh, spun up a vm on my server
23:50 🔗 dingo Once I got a feel for how it worked I was going to put it on an actual VM and give it 2TB
23:50 🔗 db48x ah, ok
23:50 🔗 sevs has joined #internetarchive.bak
23:50 🔗 db48x so the ssl cert error could be a misconfiguration
23:51 🔗 db48x try just running "git clone https://github.com/ArchiveTeam/IA.BAK/" and see if you get the same error
23:51 🔗 yipdw what OS is this? it looks really stripped down
23:52 🔗 Meroje dingo the last error needs `cpan install CGI` to solve
23:52 🔗 yipdw like, CGI.pm missing is pretty interesting
23:52 🔗 dingo Cloning that works fine, and the OS is just CentOS minimal
23:52 🔗 yipdw oh
23:52 🔗 yipdw ok
23:53 🔗 Meroje on ubuntu 16.04 I had to install CGI, not what I call a stripped OS
23:54 🔗 Meroje (but then it fails with "fatal: refusing to merge unrelated histories" which may be caused by an up to date git with different defaults)
23:55 🔗 sevs hi, currently trying to set this up and wondering where to set the name that appears in the stats - would prefer if it wasn't my full fqdn etc.

irclogger-viewer