spider blog



SPIDER's blog

Since Monday 03 October 2005 04:30 pm

Public blogs:

This Week's Tweaks

Thursday 27 May 2010 04:34 pm

Another 'day early' post of the week's tweaks as I'm taking Friday off to enjoy a long weekend with the bank holiday Monday.

mysql status/sysinfo
minor changes to include 'tables' readout and modified so it works on DEV server (mySQL v4.1)

SNG2010/ template site
added 'register interest' form as backup in case the central Finance's online registration/booking tools are not completed in time for SNG event.  Added abstract submission tool (hidden until booking tool goes live).

Research admin
minor changes to pfactsubmissionsviewnew

started to play around with moodle course formats to see how they work - potential of creating a "SPIDER" format? Delved in to moodle/lib. Seems that there are around 35Mb of library files! Main includes are setup.php, dmlib.php, datalib.php, setuplib.php and moodlelib.php. The moodlelib.php file is over 8000 lines of code alone! Combined, these are around 512kb and seem to be core to everything. Interesting to note that the entire SPIDER include (lib equivalent) folder is 528k  (60 files) but only around 80k (about 12 files) are core for every page. Probably explains why Moodle runs so slowly on the DEV server (typically requesting 150+ includes for each page), while SPIDER zips along as its 'thinner' on include requests.

Started looking at the Moodle development wiki, but found its very thin to non-existant in the areas i was intersted in, and most of what is there just suggests you go 'read the code'. Easier said than done, as trawling through nearly two thousand files to find the one that contains the function you are interested is not easy, and thats just the lib folder... Its not very inspiring, and puts you off quite a lot, but I'll persevere next week and see if I can make any more progress.  I'm starting to get a little uneasy at the decision to use Moodle as core VLE, as the code base is starting to look extremely bloated, particularly when compared with SPIDER which does a whole lot more than just 'VLE' but has a much smaller code footprint. However, I guess we won't have to actually do that much with the code itself, and the speed issues can be overcome with hardware.

radiation database v2
completed delivery tool. Minor changes to view order to allow +/- delivered items

SPIDER "history"
while reading around Moodle, VLEs and various other bits n bobs, came across the a site I'd mostly forgotten about - the  'wayback machine' - a huge archive of the web "as it was" at various timepoints.  A quick look at the old SPIDER from back in 2002 delivered up the old "Development diary" we used to use to keep track of SPIDER updates and changes. I'd pretty much forgotten about it, and think the pages were lost during various machine failures (though I've a feeling I've got a copy on an old HDD somewhere), so I added them to the help wiki for a bit of nostalgia.

backup modifications
I've changed the way the smb share on the 37TB SIPBS server in the RC is hooked up to BRUCe (mirror). It was using the admin user account to do things. Thinking it would be smarter to use a dedicated account for this instead, changed the fstab and the credentials file in /etc to use a 'spiderback' user. Created account on the win2008 server controlling the disk array and all seemed ok. Until I ran the rsync script, at which point in choked on various permissions. After much head scratching and google-ing, came up with a solution - tweaking the fstab to change the mount to:

/IPofserver/spider_backup /mnt/sipbssrv2 cifs exec,credentials=/etc/secret_pwd,iocharset=utf8,nobrl,gid=502,uid=501,nounix,file_mode=0777,dir_mode=0777 0 0

This seems to do the trick, though had to add the spiderbackup user to the web and nobody groups to allow it to read all the folders on the mirror. There seems to be a slightly tenous link between the NTFS and SMB share permissions on the remote disk array,  as it seems to pick up some set on the actual files on the mirror itself. A bit of find/chmod'ing sorted that out and now it all seems to work ok.

  >  comment

This Week's Tweaks

Friday 21 May 2010 03:27 pm

Radiation database
started 'delivery' tool. Minor fixes/changes to ordering tool.

Research admin
added pFact submission editing tool for admin/ user own submissions

bug fix in staff view of student progress - was passing incorrect month to reports

mysql status
created admin tool that reports on MySQL server/ slave status - linked from "sys info" tool

template sites
created RICAS site. Tweaks to SNG2010 site. Meeting with finance to arrange for SNG2010 to have e-payment via the university 'shop' system (pilot)

had meeting with central IT to discuss impact of changes as a result of the faculty merger to form HASS. Shouldn't be too much work, as various 'views' giving class code mappings should be made available to allow automation of a lot of the processes to create new instances of classes with new codes. Current thinking that can create a tool that loops through the class code mappings, checks for matching class, creates copy with new code.

// pseudo-code
foreach ($HASSclass as $old=>$new)
     if (is_class($old) )
        // create copy of $old as $new  
        //    - new entry in main table  
        //    - grab n copy staff/tutors related to class  
        // archive old code  
        // maybe link $new -> $old in note field?

should be easy enough...

help wiki
went through whole wiki site and updated all pages to belong to relevant categories. Switched all tables to use MyISAM instead of Innodb so could turn off innodb support in mySQL (should make it run quicker).

  >  comment

mysqld and ulimit

Friday 21 May 2010 10:43 am

I've been doing some mySQL optimisation of late, and after a bit of reading around decided to up the table cache (the number of tables mySQL holds open at any one time) from its current value of 512 up to 768 to allow for a few more tables.

This was how things were in /etc/my.cnf :


so I changed it using

mysql> SET GLOBAL table_cache=768

job done. Or so I thought. Trying to login to SPIDER this morning was taking an age. ssh to server, all seemed ok, but mysql was barely responding.

root> service mysqld restart

all seemed to be ok again. Checking /var/log/mysqld.log showed a number of lines similar to this:

4:06:08 [ERROR] Error in accept: Too many open files

So upping the table cache was causing too many open files... Google the error, seems that open_file_limit is a factor. However, none of the reading about table_cache I had done mentioned that this could cause a problem if the open_file_limit variable was not also increased to account for this. It seems that open_file_limit needs tweaking. 

mysql> show GLOBAL VARIABLES LIKE "open%";
| Variable_name     | Value    |
| open_files_limit  | 1134     |

1134 > 768, so what gives? It turns out that each open table uses 2 files:

1134 !> (768*2) => contention in mysqld = slow to crawl.

Ok, easy enough to fix, just increase the open_files_limit:

mysql> SET GLOBAL open_files_limit=2048;
ERROR 1193 (HY000): Unknown system variable 'open_files_limit'

hmm, not what we wanted to see. Off to the web, see what the MySQL docs say about this:

--open-files-limit and ulimit can increase the number of file descriptors, but only up to the limit imposed by the operating system.

ulimit? new to me. Google to the rescue. ulimit is used to "limit the use of system-wide resources".

> ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 71680
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Aha, seems there is a system limit of 1024 for open files. Need to change that.

> ulimit -n 2048
> ulimit -a
 open files     2048

looks good, except setting values via ulimit this way only works for the current shell, no use... you need to change


in order for the changes to 'stick':

mysql            soft     nofile          2048
mysql            hard    nofile          2048

nofile = open file it seems

> service mysqld restart
mysql> show GLOBAL VARIABLES LIKE "open%";
| Variable_name     | Value    |
| open_files_limit  | 2048     |

success. Now I should be able to up the limits to get the table_cache value I want.  Set the open files to 2048 on the mirror, intend to put it up to 4096 on the live system so there is a bit more room for growth.




  >  comment

This Week's Tweaks

Friday 14 May 2010 04:30 pm

table optimiser
created a script (admin tools, not linked as only sys-admin needs to use it) to run through any database on the SPIDER server and optimise all the tables it contains. Automatically skips the log tables in spiderbase.

add exam results_multi
change the mult-exam results script so that it checks and gives error message if a user tries to put in more result sets than they enter for exam names.

added function to create submit buttons (only on DEV just now)

inc_database library
created a series of db functions to streamline the in-built PHP mySQL functions:
db_query, db_query_row, db_query_update, db_query_add
testing in the radiation database v2 to see how well they work (DEV only)

radiation database v2
completed order form, AJAX view/search of orders

carried out some optimisation of the mySQL database server (/etc/my.cnf) - tweaked key_buffer_size, table_cache, thread_cache_size, tmp_table_size/ max_heap_table_size, query_cache. Installed MySQLWorkbench to monitor these changes. Thread cache and query cache were both off (!), turning these on and optimising them should have had a significant impact on the database performance, though the system was running well already, its running a lot smoother now - e.g. the query cache hitrate is about 62.5% (instead of 0%) so thats a lot more queries being done from the cache instead of hitting the CPU to do them. Also grabbed a copy of mysqlreport.pl which outputs the system variables/status as a useful report to monitor mySQL from the server CLI.

change to news to allow for dynamic select of targets/ email lists completed - should be quicker/easier for users to target news correctly; pulls in maiing lists and auto-creates mailing lists for year groups. Undergoing final testing on DEV server.

optimised javascripts and styles to improve performance and reduce external calls.

WYSIWYG editor
turned on for Safari browser in quizzes & blogs

added fix for Safari bug that allowed assignment submission without file loading correctly. Generates error warning to user and emails devs if assignment submission fails.


  >  comment

This Week's Tweaks

Friday 07 May 2010 04:22 pm

private messaging
updated the tool to give suitable warning messages when attempting to send messages to non-SPIDER users (e.g. if you copy/ paste registration numbers or tried to message a user in a group auto-generated from a class list where the student in the class list is not yet registered).

inc_functions/ inc_functions_status
consolidated the user_in_x style functions into inc_functions_status - saves duplication in various scripts (discussions and groups). Functions include: is_user, is_group, is_in_group, is_class, is_class_editor, is_class_editor_group, is_class_tutor_class, is_class_tutor_group, is_on_class_list, is_teacher_on_class, is_class_coordinator
consequently bugged the groups tool as a result (missed it when abstracting the functions out), which was fixed as soon as a user let us know it was broken.

radiation database v2
started to create new home page and base functions for limiting access, activity conversions. Toying with creating a php library to handle database calls. Started to create order tool.

PEGAUS/Oracle interface
tweaked the oracle_get_programme_users script (pulls all the users on each degree from the central systems) so that it joined users + programmes using the reg number from the programme table - this fixed the problem of users with multiple registration numbers having the wrong registration applied to their degree programmes.

Moodle/SPIDER widgety stuff
messed about on Moodle to see how SPIDER widgets can be used - got an iframe of class content working - sans authentication but in principal should work with Shibboleth. Started to work out how to build custom Moodle blocks to see if that is preferable to iframe option - not sure if blocks can appear anywhere other than on the 'right' of the page though which is not ideal for pulling in 'wide' information/ tools.

Tweaked the main includes to allow for 'widgetOff' session variable to be set so that top/bottom navigation will be hiddden when a tool is in widget mode (on DEV just now, still testing)

assignment - markers
updated assignments so that class editors & class co-ordinators are able to view assignment submissions without having to be set as markers

class reps
updated class home page to only show current session class reps. Updated manage class reps to show session above list too.

refining code to improve performance under Safari

updating the news - add form to make use of AJAX when selecting targets (class, year etc) via string search. WIP on DEV

file manager in template site
updated to allow multipe file/ document stores with advanced sharing options (groups/ individuals, read/write).



  >  comment

1 comment

Blocks can go on the right or left hand side, they default to where the "Add Block" block is located!

Posted by Michael Hughes Saturday 08 May 2010 12:24 pm
You need to log onto Spider to add entries
You need to log in to Spider to post

<<May 2010>>
2 3 4 5 6 7 8
9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30 31          


April 2016 [1]
November 2015 [1]
January 2015 [1]
July 2014 [1]
June 2014 [3]
May 2014 [4]
April 2014 [2]
March 2014 [3]
February 2014 [3]
January 2014 [2]
December 2013 [2]
November 2013 [2]
October 2013 [3]
September 2013 [2]
August 2013 [6]
July 2013 [3]
June 2013 [2]
May 2013 [4]
April 2013 [2]
March 2013 [3]
February 2013 [3]
January 2013 [2]
December 2012 [2]
November 2012 [4]
October 2012 [3]
September 2012 [4]
August 2012 [3]
July 2012 [2]
June 2012 [4]
May 2012 [2]
April 2012 [1]
March 2012 [4]
February 2012 [5]
January 2012 [3]
December 2011 [2]
November 2011 [3]
August 2011 [3]
July 2011 [5]
June 2011 [5]
May 2011 [5]
April 2011 [6]
March 2011 [6]
February 2011 [5]
January 2011 [4]
December 2010 [5]
November 2010 [11]
October 2010 [6]
September 2010 [6]
August 2010 [4]
July 2010 [7]
June 2010 [5]
May 2010 [5]
April 2010 [10]
March 2010 [4]
February 2010 [7]
January 2010 [5]
December 2009 [4]
November 2009 [12]
October 2009 [8]
September 2009 [2]
August 2009 [1]
July 2009 [3]
June 2009 [5]
May 2009 [2]
April 2009 [3]
March 2009 [5]
February 2009 [5]
January 2009 [2]
November 2008 [1]
August 2008 [1]
July 2008 [1]
June 2008 [1]
May 2008 [5]
April 2008 [5]
February 2008 [2]
November 2007 [2]
September 2007 [1]
August 2007 [3]
May 2007 [5]
March 2007 [2]
February 2007 [1]
November 2006 [2]
July 2006 [1]
May 2006 [1]
April 2006 [1]
March 2006 [4]
February 2006 [3]
January 2006 [3]
November 2005 [1]
October 2005 [4]

Favourite blogs