maria yancheva

Demo server + screencast + poster

2009-08-21T00:23:00.008-04:00

Demo:
http://www.cs.toronto.edu:40154/mediawiki

Screencast:
http://www.cs.toronto.edu/~yancheva/myelink/myelink-screencast.htm

Poster:

Mediawiki: Importing large batches of pages from Wikipedia

2009-08-12T10:03:00.003-04:00

How to import/export to Mediawiki

After a few days of trying out various methods of importing (a large number of) pages into my local Mediawiki (including writing my own import script, before I realized Mediawiki had export/import capabilities), I've come up with a procedure that works. Below are steps + solutions to common problems that occur when importing.

Go to the Special:Export page of the source wiki, e.g.: http://en.wikipedia.org/wiki/Special:Export, and select a bunch of articles you want to export. DO NOT select "Include Templates" and click Export. If you select "include templates", the xml file will simply include the template syntax in the pages which use templates, but _not_ the actual source templates. The result would be a lot of ugly syntax in the pages you import making you wonder whether the import worked. (By the way it's interesting that most of the pages in wikipedia seem to be using templates extensively - I realized this when I imported using the "include templates" option).
Now, go to the Special:Import page of the target wiki, choose the XML file you just downloaded, press Import.

Most likely, you will get one or more of these errors. Here are solutions to the ones I encountered:

Session timeout - the file is too big. Try to import smaller batches. (Similarly if you get a maximum file upload size type error).
Fatal error: Allowed memory size of nnnnnnn bytes exhausted (tried to allocate nnnnnnnn bytes) - this is a PHP error - here's how to resolve it.
Error in fetchObject()": Illegal mix of collations for operation ' IN ' (localhost) - this is a mediawiki problem, luckily there's a quick fix here.

For completeness, here's the Python import script I wrote. It seems to work but no guarantees. It's always better to try the mediawiki import first.

Input validation, search sort and my word cloud algorithm

2009-07-15T14:43:00.001-04:00

I've added client-side user input validation and made the search result headings as links, so that when clicked the results are re-sorted based on that criterion. For example, clicking on "Title" sorts the results in alphabetical order, and if clicked again toggles ascending/descending sort. I'd like to optimize how this happens though, because every time the results are re-sorted they are also re-calculated which is not needed. One idea is to 'index' all the wiki pages before the extension runs, so that tags are generated for each page and stored in a database, which can then be searched much faster. That also means that the script which indexes the wiki pages should be run periodically, as content is not static.

I've also spent some time playing around with creating word clouds via PHP's GD library for image creation and my implementation of this algorithm, but I realized that algorithm is too slow to be useful. The reason I'm writing my own implementation is that I previously used the Google WordCloud API but the results it produces are not exactly what I wanted. The google "word clouds" are simply lines of text (plain text, not an image), with a font resize for words which occur more frequently. I wanted something more like wordle.net (but unfortunately that's closed-source). While I couldn't find an open-source project which does the same thing (maybe I didn't search very extensively), today I wrote another implementation using the GD image library, and this time I created my own algorithm. Here's what it looks like with some random words:

And here's how it works using PHP and GD:

Get an array of (word => frequency) pairs.
Randomize the array to produce different-looking word clouds on refresh. (And keep track of the relation between each word and its frequency).
Initialize the image object, available colours, image background and font path via GD.
Slice the image into imaginary lines. The line height is the maximum font size used (which I calculate by getting the highest word frequency and multiplying it by a resize factor).
Keep track of current line number, starting at 0. Place the first word on the current line (line 0). I vary the x-coordinate randomly between 1 and 8 pixels from the left image border (so that consecutive lines don't appear to line up). The y-coordinate is equal to the word height.
Continue placing words on the current line, until the border is reached. Notice that the words stick to the top of the line, and they vary in size (proportional to their frequency), so it doesn't look like they're lined up. I also vary the colour randomly.
Once the right border is reached, increment the variable storing the current line number, and repeat the process until all words are displayed.
Finally, don't display the lines.

Things that need to be improved: the sparsity of the word cloud is due to the fact that the code which determines how far "left" to place a word along a line is a bit of a hack - because I'm not sure how to calculate the varying font width.

Adding features

2009-07-03T12:29:00.004-04:00

Yesterday I finished implementing Ajax using Mediawiki's interface for Ajax calls. I'm happy to say it's working quite nicely now. I also rearranged some of the menus/links for (hopefully) easier navigation.

I also looked into Google's Word Cloud API and added word clouds of the articles' content as another easy way to give the user visual representation of the similarities between articles. Google's documentation is fantastic, so this only took about 5 min to get it working :) (By the way, to avoid being dependent on Google's server availability, I added the API JS scripts and stylesheets to the server locally, so no connection to google takes place when the user decides to display a word cloud).

Next, Lucene.

After that, there are several things I need to do before the extension will be ready for testing:

Add user input validation
Write the project documentation
Tidy up the source code and comment it
Fix any bugs/issues that come up during testing

AJAX and MW

2009-07-02T10:02:00.001-04:00

I spent the last few days refactoring code and incorporating an XMLHttpRequest object to make the rendering of user-specified search options asynchronous. Since changing the options has to re-generate the search results, and generating the results was being handled by the main extension function, I had to either pass selected bits of the results to the external script called by XMLHttpRequest which would then render the search options (for example, weighting of textual and structural search), or move all the processing of results in the external script. I decided to use the second approach because that would make for more readable code and would be a better design decision in the long run. Of course, that led to another problem, as the Mediawiki API is not available for use by the external script (and I was using the API to make calls to the MW database in order to fetch article content for making comparisons). After a few days of searching for a solution, it turns out that MW comes with support/interface for AJAX extensions.

On a different note, I asked some friends for name suggestions, and I got Myelink - (pronounced as my link, it's a play on myelin and link) and here's the explanation I got: "the Myelin Sheath is a layer that increases the speed of neurotransmission by... a lot.. 1000x at times depending on the animal and location. Myelinated neurons are also famous for being in the white matter of brains, which serves as an information pathway for the brain and connects different parts and facilitates communication. Which is what the extensions is set out to do.." I like the connotation and since what I am currently focusing on is generating related pages with the aim to facilitate communication between different contributors of scientific experiments/articles, I think it fits nicely :)

Pages' previews and some technical details

2009-06-18T17:15:00.046-04:00

I finished implementing the 'preview' feature which allows a live embedded view of the pages listed by the extension. It took a while because of several parsing/markup rendering issues I ran into:

Dynamically fetching and displaying the articles' content - this was a bit of a problem because the 'fetching' part is done in PHP, whereas the displaying is done using JS - one is server-side and the other client-side, and I had to coordinate the two. I ended up pre-fetching the content of the articles concurrently with generating the results (i.e. as a page is found to be related, its content is obtained), and then passing that content as an argument to a JS function which can display it on the client-side whenever the user clicks 'preview'. I need to look into AJAX in the future.
Displaying the generated page content - first, it had to be parsed from wiki markup to HTML, then escape the HTML special chars and entities and pass it to the JS function, and from there unescape it so it can be displayed. For obtaining HTML from wiki markup I used a parser function:
```
$html = $parser->parse($content, $pageTitle, new ParserOptions(), true, true)->getText();
```
Escaping the HTML was the easiest part because of PHP's built-in functions:
```
$html = addslashes(htmlspecialchars(htmlentities($html, ENT_QUOTES)));
```
Unescaping the HTML from JavaScript proved a little more difficult. I ended up making use of this great resource: php.js (a package porting PHP functions to JavaScript), so after including the file ($out->addScriptFile("php.js"), and editing the setup script to place it in /skins/common), all I had to do was use html_entity_decode and htmlspecialchars_decode.
Editing the displayed content - show only the section headings with a little snippet of text under each heading. Ideally, the snippet would be the part that is similar to the current page. Getting the content of particular sections is not hard to do using MediaWiki's API, but only when you know what the sections are. The problem is figuring out how many sections a page has. Someone on the mediawiki IRC suggested using the _NUMBEROFHEADING_ magic word, but after looking into the documentation that doesn't seem to exist... or for some reason I can't find it. So I just decided to manually parse through the content and count the sections, which is a hack but it works.

After figuring out how to do this, it seems pretty straight-forward but figuring it out took a bit of time - mostly because you can never be sure how the mediawiki parser will decide to parse and output the text (wiki markup, html, or rendered html). Next I want to work on improving the accuracy of my actual algorithm (finish implementing all the criteria and test with some real pages), and after that there are several things to work on: Sarah's excellent suggestion during Tuesday's talk with Anita Sarma when she pointed out that the algorithm for determining structural similarity can be used to detect emerging patterns and turn them into templates (thanks Sarah), and Anita's suggestion for a feature that allows cloning of entire pages (thanks Anita). I have an idea of how to do the latter, and I think I'll need some sort of persistent data store to implement the first and intellegently detect recurring structural patterns - that will be a lot more fun to code than the silly details I had to work out today.

Updates

2009-06-15T12:03:00.000-04:00

My basic algorithm for page-relatedness is now working and displaying a list of the top 5 most similar articles, with links to them (I'm currently working on the in-page preview). Currently I've implemented this as a tag extension so that users can choose where in the page to place the feature (if at all). I considered the pros and cons of adding this feature as an automatic script which runs whenever a page is loaded and displays the results at the top of the page - on one hand this is good because the user does not have to type in anything in order to invoke this function, and it automatically shows up on every page, by parsing the page content and comparing it for relatedness with the other pages on the wiki; but on the other hand, this could get annoying if you don't want it to show up all the time, and so I settled for the tag extension. This requires the user to remember the tag name ("<relatedpages>") and to insert it, but if the feature itself is useful, then this shouldn't be a problem.

The algorithm I used for determining relatedness is different from the one I was considering to use a few days ago. I decided to customize the search for related pages to the subject of the wikis - since this would be used to compare notes on computational science or biology experiments, I decided to think of ways to make the algorithm work better than a standard content search and comparison algorithm, by customizing it to the subject of the pages being compared. At the same time, 'relatedness' for notes can be characterized not only as similar subject matter but also as similar experiment type (i.e. related page layout and template). So the customization of the search which I wrote consists of the following: 1) Structural similarity and 2) Textual content similarity. For the first part, structural similarity depends on both the types of markup tags used and how often they're used, but also on the relative position of those tags. To reflect that, I do several comparisons: count of number of occurences of structural markup tags (these include both wiki markup like '==' for titles as well as allowed html tags like '<div>' and ''); comparison of relative position of structural tags; and comparison of actual headings (within the '==' tags). More criteria could be added in the future, to improve the search further. For the second part of the similarity score, I thought about how to compare textual similarity and I came up with these two criteria (although again, more criteria almost certainly need to be added to fine-tune the search): first I compare the tokens of length greater than or equal to 8 characters (after trial-and-error and observing comparison results I settled on 8 characters as the sufficient token length which would ignore meaningless results such as "because","through" etc. and filter the most significant terms, like "nucleotide", "ligation", "restriction" etc.); then I also compare the capitalized tokens which occur on the page - the reason for this: a lot of methods, protocols, terms in biology (and science in general) are usually referred to by their abbreviations, eg: FRAP, GFP, FACS, DNA, etc. For this reason, it makes sense to pay special attential to those tokens when comparing two experimental pages.

To summarize the results, I assigned a weight to both scores - the first based on structure and the second based on textual similarity - and displayed both the cumulative and the individual breakdown scores on the wiki page. Here's a screenshot of what it looks like on the page.

Note: As I'm yet not completely familiar with the Hadley wiki pages, I focused on testing the comparison algorithm with the biology-related experiments because they are available at OWW and because I have a vague understanding thereof having done some myself :-) To improve the algorithm in the future however, I could add comparisons which are specific to climate change experiments (e.g. comparing the five-letter names of the experiment runs).

Next: I'll add controls so the user can change the weights assigned to different aspects of the relatedness comparison - whether structural or textual (currently it's 50/50), and can drill down to see the particular criteria used in the comparison and the tokens which were found to be similar (so that he/she can judge whether to trust the comparison). I'll also implement the 'preview' feature for the related pages displayed, so that the user can see a preview of the sections of the page he/she is interested in without navigating away from the current page. The preview will also show the snippets of text which were found to be related to the current page, allowing the user to easily cut and paste information as necessary. To that end, I'm also working on implementing 'copy', 'paste', 'delete' links onto the section headings (where 'edit' is currently located) - I added the links yesterday but I need to figure out how to edit/delete the sections when the user clicks 'delete' or 'paste' (the way it currently works there's no framework for deleting and pasting content into individual sections so the UI has to go through the Edit page, which is what I'm trying to avoid because it's time-consuming. I think I'll have to directly query the db and change the page content from there).

On a more technical note, here's the mediawiki code I've used so far for implementing different features (this was probably the most time-consuming part, as there are very, very, few examples of using mediawiki Hooks):

To create a tag extension, such that it displays content when the user enters the tag name in the wiki text, I registered a parserhook extension and used the setHook('extensiontag', functionToRender) function; example:


/* Note: common abbreviations added to variable and function names:
* wf = wiki function, wg = wiki globals,
* ef = extension function */

/* Place this within the extension setup function. */

global $wgHooks;

/* Register the tag name . efExtensionNameRender is the function called
* when the user uses the  tag. */
$wgParser->setHook( 'relatedpages', 'efExtensionNameRender' );

To load content to every page when it is viewed, without having the user input or type anything, use one of the page rendering hooks. Depending on the exact time when you want the content to be displayed, you could use ParserBeforeStrip, ParserAfterStrip, etc. I found that in order for the html tags (like <a href> which is not one of the recognized tags when wiki text is parsed), to be rendered as html, I had to use the InternalParseBeforeLinks hook.

To add a link on section headers (where 'edit' is located), I used the DoEditSectionLink hook. (The other two similar hooks - EditSectionLink and EditSectionOtherLinks - are deprecated in newer versions. However DoEditSectionLink is available only after version 1.14, and the mediawiki version which can be downloaded from the ubuntu repositories is 1.11, so I had to install the latest version from source to have it work. Though running several different version of mediawiki is quite easy and fast to setup - I set them up to use the same database backend so sharing of page content between them is instantaneous.) When creating a new link I also had to add the tooltip message/hint to the system messages displayed by mediawiki. These are located in mediawiki_dir/languages/messages/MessagesEn.php (or Messages**.php). They are changed not by editing that file, but by using the addMessages() function on the $wgMessageCache global object. Example:


/* Place this in the extension setup function */
global $wgHooks;
$wgHooks['DoEditSectionLink'][] = 'efExtensionNameDoEditSectionLink';
/* end of extension setup function */

function efExtensionNameDoEditSectionLink(($skin, $title, $section, $tooltip, &$result) {
 global $wgMessageCache;
 $wgMessageCache->addMessages( array("copysection" => "copy") );

 // Add tooltips (hint shown on hover over link).
 $wgMessageCache->addMessages( array("copysectionhint" => "Copy section: $1") );

 // This only opens an edit page...I need to figure out how to modify this to edit
 // the page without going through the edit page.
 $copyQuery = array('action' => 'edit', 'section' => $section);

 $copyAttribs = array();
 if ($tooltip) {
    $copyAttribs['title'] = wfMsg('copysectionhint',$tooltip);
 }
 $options = array('known', 'noclasses');

 // this actually adds the link:
 $copy_url = $skin->link($title, wfMsg('copysection'), $copyAttribs,
                          $copyQuery, $options);
 $result .= "[".$copy_url."] ";
 return true;
}

To add CSS/JS scripts (using the BeforePageDisplay hook); example:


/* Place this in the extension setup function to register the  hook */
global $wgHooks;
$wgHooks['BeforePageDisplay'][] = 'efExtensionNameBeforePageDisplay';
/* end of extension setup function */

function efExtensionNameBeforePageDisplay($out) {
   // NB: put 'styles.css' in the default styles directory (usually mediawiki_dir/skins/)
   $out->addStyle("styles.css");
   $out->addScript(
<<
EOD
   );
  return true;
}

Other notes:
System messages, can be accessed from Special:Allmessages. File located in: /languages/messages/MessagesEn.php.

PHP bug #32671 (have to convert float keys to string for associative arrays).

XMLHttpRequest object - DOM API - to send an HTTP request query string to a python file from a JS script. The python script uses the handler() method which expects a req parameter representing the request.

Similarity Algorithms: A measure of relatedness

2009-06-08T16:02:00.003-04:00

As I'm building a tag extension for displaying a list of related wiki pages on mediawiki, I started thinking about how to quantify the relatedness between two pages, and I looked at several algorithms and research papers on the subject of text similarity and comparison algorithms. This is not as trivial as it seems, and there are a number of researchers who are interested in the area, though most applications are related to testing for copied work in student submissions in CS courses, or the other popular application to search engines. As such, the first category of text comparison algorithms I read about focus on comparing code rather than natural language, but there are several key principles which could be adapted to the latter context. The latter (used in search engines) uses a number of strategies such as word-based, keyword-based, n-gram and LSA (latent semantic) analysis. N-gram focuses on categorizing text based - it calculates a text's profile and compares its distance from each category profile. LSA stores word-context data matrices, and calculates similarity based on resulting word and context vectors based on the cosine of their contained angle (as unintuitive as this may sound, the results obtained from this method measure up quite well with human-made comparisons).

Text is parsed into essential tokens which are then compared. An important point is to choose a suitable minimum length for common substrings - if this value is too small then the reported similarity may be too high and vice versa. Thus, the similarity S_N(A,B) between two texts A and B can be defined as the percentage of text in A which can be constructed by combining essential tokens from B of length greater than or equal to N. As N is increased, the similarity S decreases, however a distinction between non-related and related texts can be made since S for related texts decreases more slowly than S for non-related ones. (eg: for N=24, S_related = 75-80% whereas S_non-related = 20-50%). The similarity tester can be made flexible by finding the set of longest common substrings (order-insensitive), unlike the unix diff utility which finds the longest common subsequence (order-sensitive).

Comparing wiki pages - algorithm:

Since processing of page content to determine relatedness may be a slow process, I'll first filter the pages with similar titles (admittedly this method is not perfect, but it works as a prototype). I considered using something like levenshtein() or similar_text() to compare title strings, but these functions compare the char-by-char match which is not too meaningful and fails to notice global similarities in titles. So I've decided to explode() the $pagetitle into an array of title words (say, $titleArray), then go through every other wiki page title, and determine similarity by counting the number of elements of $titleArray contained in a page title, then scale the result by dividing the length of the common tokens by the length of the page title (in chars). Store the results in $simTitles array, with (similarity_score => page_title) key-value pairs. Once this is accomplished, krsort() the $simTitles array, and proceed with examining the page content of the first 10 (or maybe fewer) results.
For the current page (A) and for each page in the array returned from step 1 (B), read page_content from the mediawiki db for both A and B and parse into essential tokens based on this SIM algorithm; In addition to comparing substrings, also compare the highest-frequency words of length greater than 5 (or more) chars (This comparison of lengthier terms would be characteristic of the application of the algorithm to scientific texts, as there are usually terms which may appear often even if the substring analysis does not indicate relatedness -- the reason the SIM algorithm worked well is because its application was primarily to C code, where code in copied assignments would indeed share common substrings where the core algorithm of the program is involved. The difference in application therefore warrants a difference in analysis because while computer languages provide a limited set of allowed expressions, natural language allows greater variation even if the subject of the page is closely related).
Display pages with S >= 80%, in descending order of similarity. (This percentage value was found to provide consistently accurate results according to research). Format output to the wiki page (similar in style to the Recent Changes display on OWW).

Reading:

Lack of documentation

2009-06-04T15:19:00.003-04:00

Looking through the mediawiki API documentation (where it exists) is frustrating (mostly because it is in the form of a reference, rather than a manual -- it's useful only once you've figured it out on your own) so I've been reading through the source code and figuring out which functions can be used for what. Examples of using the API in actual code are scarce. Here's what I found so far:

On one hand, you could access the database (and do other things, like edit pages, authentication, etc) by sending a query string in a URL through an HTTP request to the api.php file (which is one of the main access points). Here's how this method could be implemented:
```
$api_dir = '__wiki_url__/api.php'; // __wiki_url__ is the wikimedia install dir
$query_options = array('action' => 'query',
 'titles' => 'Main Page',
 'prop' => 'info',
 'list' => 'allpages',
 'meta' => 'userinfo',
 'format' => 'xmlfm');
$query_string = http_build_query($query_options);
$query_url = $api_dir . '?' . $query_string;
$results = get_url($query_url); // get_url is a function I wrote which uses cURL
```
What this does: returns information about Main_Page (a page on the wiki), along with a list of all pages on the wiki, and metadata about the current user; the output is in the XMLfm format (fm being the pretty-HTML indent format), and it looks like this:
```
<api>
<query>
 <pages>
 <page pageid="1" ns="0" title="Main Page" touched="2009-06-01T18:16:21Z" lastrevid="14" counter="15" length="537">
 </page>
 <allpages>
 
 
 
 <userinfo name="127.0.0.1" anon="">
 </userinfo>
 </allpages>
 </pages>
</query>
</api>
```
You could also specify 'format' to be 'php', and then use php's unserialize() function to access the data. More about the query options can be found here: MediaWiki API Query. While this method works, it requires quite a bit of code just to set up the query url, options etc, and it seems there must exist a higher level of abstraction, so I continued looking for another method of accessing the API - see below.

The second method: instantiating the classes defined in the source code docs. (So because there is no manual), read the source code here: http://svn.wikimedia.org/doc/files.html (particularly, I found useful reading index.php, Article.php, api.php, User.php, Database.php, PageHistory.php; it's also useful to look at the Classes listing separately). Then implement them in your code, like such:
```
// Accessing the database to get a list of all the user names.
$db_read = wfGetDB(DB_SLAVE); // DB_SLAVE for read access, DB_MASTER for write access
$res = $db_read->query("SELECT user_name FROM user");
$string = "";
while ( $row = $db_read->fetchObject($res) ) {
$string .= "User: ".$row->user_name . ";; ";
}
```
This prints out something like:
```
User: WikiAdmin;;
```
For a quick intro to querying the database, and a few examples, see this blog post (note that the examples from that blog use the built-in insert(), update() etc functions ~~whereas I find it easier to use the query() function and pass it an actual SQL query, as in the snippet above~~. It's better to use those wrapper functions because they make the code more flexible and independent of the underlying database backend. That said, using query() is much simpler to use because you don't have to worry about new syntax.); there's also a hard-to-find mediawiki page on Database Access. And here's the database schema.

The most useful site so far has been MediaWiki Code Snippets which gives many examples of using the API (something that's completely lacking on all of the mediawiki Manual pages).

And something else: querying the database is not necessary when the information you want is available through the global variables: wg variables (also a complete listing of configuration variables here) and global object variables. It took a while to find a mention to this on the mediawiki pages, but these links provide more information. Typically, there are a number of wg (wiki global) variables defined in DefaultSettings.php and LocalSettings.php. Recently however, the concept of Global Object Variables has become more used (for good reason) because it allows instantiations of objects which could be used with various methods without changing the global wg variables.

Making progress

2009-06-03T14:15:00.001-04:00

Today I continued work on the code for the mediawiki extension - I wrote the python setup script which builds the extension directory by copying the correct scripts and modules to the correct subdirectories and applies the user config settings specific to the local mediawiki installation. The build script works nicely. What became a minor problem was the way the HTML is rendered in the wiki page. Here's what happens: I build my extension, add it to the mediawiki extensions, and use the <extensiontag> within a wiki page; the result: the parser seems to apply pre-formatting to the HTML returned -- the formatted (human-readable) HTML and JS code in my python script are displayed in a similar formatted fashion when rendered (eg: having two links on new lines in the HTML code will render the links on new lines in the wiki page itself, even if there's no explicit new line tag). This is a tad annoying; seems related to Mediawiki Bug #8997 on Bugzilla (which was mentioned in the statsjam source code...I'll take a look at the work-around used there). On a different note, it's quite easy to add dynamic content by embedding some JS into the HTML template returned by the CGI script (which makes me think I should look into visualization APIs for creating a nicer UI). The first feature I've started working on is displaying a list of related users/related wiki pages. For that, I need to figure out how to query the mediawiki database for obtaining data about the collection of wiki pages, and to come up with criteria for what defines 'relatedness' between two users.

I'm also making a list of use cases and sketching little hand-drawn snapshots of what the interface would look like (more on that tomorrow when I go over it with Steve).

Interesting collaboration tool

2009-06-02T20:26:00.039-04:00

While taking a break from writing PHP today, I came across the Google Wave Developer Preview from Google I/O 2009. Google Wave is an interesting new tool for integrating email, chat, blogging, wikis and other forms of web communication into a seamless live conversation stream. Features include threading of the message waves (can't really call them email threads, as they can be emails, blog comments, 'wiki'-style notes - all handled as 'waves' sent to the list of recipients), live view of new messages as they are being typed (thus practically a chat as well), editing of own or others' messages (essentially a live wiki thread), and ability to integrate with external websites and mobile devices with embedded 'waves' (such that users can comment on a dozen blogs and see all external comments and discussions integrated as conversations in their Wave client, which is synchronized in real-time). The Wave client acts as a central collection of all of the user's communication on the web. Sounds cool. For the integration alone, it's useful. And it's all open source with a nice API.

The beginnings of a tag extension

2009-06-01T15:55:00.000-04:00

Today I started writing the php files for the mediawiki extension I'll be making. The most important thing right now is to make sure the interfacing of the PHP and Python files works, so for now I focused on writing the PHP 'render' function (part of the main php setup file) which just sends an HTTP request to an interface.py file and that Python file then renders the html and writes it to the http connection. There were some standard things to take care of in the php setup file, like registering the extension features in mediawiki (I used 'parserhook' as the extension class, as I'm currently test-implementing an XML tag; registering it makes sure it shows up on Special:Version), and setting up the parserhook to 'hook' the tag name to the rendering function. I tested using my tag on LocalWiki (my local mediawiki installation) and it works as expected (right now it just displays back the $input and $args). Next I'll start coding the actual python modules, so that the tag will start doing something more interesting (and more useful) than just dumping back the user input.

I also spent some time today trying to install statsjam locally to see how the user interface works but it seems like there's an import error in one of the python files. (I decided it's not worthwhile to spend more time trying to fix it since I still have the source files to look at).

Meanwhile, I'm also installing some of the extensions which are already being used at OWW (Special:Version), with the idea of test using them to figure out first-hand what's missing (keeping in mind the snapshots of the hadley centre workflow tasks and the interviews which I've been reading through).

Exploring the options

2009-05-29T17:01:00.010-04:00

Today I've been going back and forth between writing several simple PHP scripts and test running them through my apache server, and on the other hand, exploring the possibility of creating a MediaWiki extension in Python with a minimal amount of PHP coding required.

The MediaWiki manual which talks about extension writing explains they all consist of three conceptual parts - the setup file, the execution file, and a file with internationalization information about the extension. Of course, that manual suggests all of these parts are to be written in PHP; however: by looking at the statsjam source code (and getting a helpful response from its main developer, thanks Jon!) I think it will be possible to write the main execution code in Python and then interface that with the setup.php file by way of an HTTP request to the Python CGI script. It seems like a fairly straight-forward method so I'll give it a test try with a simple python script later today.

Meanwhile, I've been discussing ideas about features with OWW's developer. Still looking into ways to collaborate more effectively. We'll probably take the discussion over to an OWW page, where users could contribute ideas as well (one already exists: here).

A case study of the Hadley centre + more reading

2009-05-28T16:30:00.004-04:00

Today's meeting with Steve cleared up at least part of the questions we had about the daily (software-related) challenges facing the scientists at the Hadley centre. He showed us a case study of a scientist accomplishing a particular task, and the difficulties he met along the way, with screenshots of the content he was dealing with. While I won't be directly involved with writing software tools specific to the Hadley centre (see previous posts), I found today's discussion really interesting in that it gave me a more accurate idea of the tasks they need to accomplish, and the tools they are missing. One of the things I found quite striking is their method of keeping track of climate simulations. Given a general climate model, say the HadGEM3, they run countless simulations (either for a particular subsystem - eg atmosphere, or a coupled model - eg atmosphere and oceans) by using different variables and parametrizing various subsystems. The complex relationship between the simulations, and the reasons for running simulation 'adghk' after simulation 'admnl' (their computer-generated names are completely arbitrary), as well as the particular changes made from one run to another, are documented in.. well, a hand-written list on a plain wiki page. Just thinking what better things these scientists could be using their time for, as opposed to painstakingly writing out lists of simulation metadata (like simulation names and short descriptions - which could be automatically generated) is frustrating.

Another issue is the lack of documentation of ancillary files (ie files needed for running the simulations, which include supplemental data). A real problem occurs when one person commits an ancillary file to the repository, say a new orography model, and forgets to include essential information about configurational changes which must be made in order for that code to work correctly. Another person tries to use said ancillary orographic file and his simulation crashes because it was actually necessary to change the value of the gravity wave constant... which the first person who committed the orographic file should have documented in the first place! Such poor documentation can lead to sidetracks of a few days to a few weeks, and in general, comments about a file are documented only after the lack of documentation has already caused a problem.

These are interesting challenges to tackle, especially the latter, since the knowledge of which parameters need to be changed with which file is (it seems) something that is difficult to generate automatically. At the same time, you can't rely on scientists to always add the appropriate comments to all their files because humans suffer from a general dislike of documentation.

I spent the rest of the day doing some reading, and I also got around to installing mediawiki.

Some books I'm reading today:
Beginning PHP5
Professional LAMP: Linux, Apache, MySQL, and PHP5 Web Development
Apache Server Unleashed

For reference, here's the output of the mediawiki installation:

Generating configuration file...Database type: MySQL
Loading class: DatabaseMysql
Attempting to connect to database server as root...success.
Connected to 5.0.51a-3ubuntu5.4
Attempting to create database...
Created database wikidb
Creating tables... done.
Initializing data...
Created sysop account WikiAdmin.
Creating LocalSettings.php...

Config file: /etc/mediawiki/LocalSettings.php
Setting permissions: chmod go-rw LocalSettings.php
Path to mediawiki: http://localhost/mediawiki/index.php

MediaWiki and PHP

2009-05-27T20:57:00.014-04:00

During the past two days I have been doing lots of reading to get familiar with the mediawiki engine and PHP in general. This is hardly noteworthy of a post, but here's what I've been looking at for the sake of completeness:

MediaWiki Development Policy
MediaWiki API documentation
MediaWiki source code
The official PHP Manual
A book on PHP5/MySQL Programming

While I was at it, I also set up an apache server - I figured it would be handy to have it configured (so I can install the mediawiki engine locally; it turned out more time-consuming than I expected -- I just spent a few hours installing various dependencies).

A note about the LAMP-server installation: if installed from source, apache creates a nice server root directory where everything is located (however you have to manually add the modules for php, python etc.) If installed from apt-get, the modules are easy to install and to get apache to work with php, mysql but the apache directories are in miscellaneous locations.

For quick reference, the following are the equivalents of the source installation directories:
/etc/apache2/apache2.conf = rootdir/conf/httpd.conf and other config files
/etc/apache2/mods-available and enabled = rootdir/modules
/var/www = roodir/htdocs
/var/log/apache2 = rootdir/log (error and access logs)
/usr/lib/cgi-bin = rootdir/cgi-bin
/etc/init.d/apache2* = rootdir/bin/apachectl*

Other config files:
php config - /etc/php5/apache2/php.ini
mysql config - /etc/mysql/my.cnf
mediawiki - /etc/mediawiki/apache.conf

A new direction

2009-05-25T14:56:00.075-04:00

After today's debrief with Steve and the other undergrads, I've decided to continue my project by exploring the existing set of tools offered through OpenWetWare, rather than start from scratch and create a trac-specific tool which would be used by one particular research centre. A contribution to OWW would be worthwhile because of the potential to adapt it to the needs of scientists of any field - and exploring the idea of OWW as a collaborative (and why not inter-disciplinary?) platform. It would be interesting to later interface this platform with various research centres through their respective software management portals - which is more reasonable than creating a separate plugin for each research centre.

On a related note, here is a summary of the five projects we discussed today (and will be developing this summer):

Social network tool for Trac - analysis of codebase and issue tracking system, as well as mailing lists and a set of wiki pages for extracting information about users working on related projects;
Wiki markup for data visualization - GUI tool for embedding tables and dynamic graphs (possibly interfaced with external tools, like manyeyes) into wiki pages;
Green buildings: what can residential owners do to make their homes more energy-efficient and sustainable - more than a simple calculator of carbon footprint, this tool will use open-access government datasets (like Data.gov) to determine which changes to an existing building are most efficient (prioritizing) and the savings generated in the long-term;
Graph of browsing history (potentially as a Firefox extension) - generating a dynamic visualization of the pages visited by a user, with snapshots of each page for easy recognition - which could be then saved for future reference, or annotated;
Electronic lab notebooks for scientists - what this blog is about.

Currently reading:
List of MediaWiki Extensions (source code)
MediaWiki Extensions on Meta (descriptions)
MediaWiki Gadgets
OpenWetWare Developers - Bill Flanagan's Blog
OpenWetWare Useful Extensions

Prototyping

2009-05-22T16:18:00.002-04:00

At this point I started writing a bit of code - but instead of diving straight into the plugin modules, I decided to create a few macros first. Creating macros is a good starting point because it allows me to test extracting data from trac.db, and at the same time use these macros within my local trac environment to determine if these features are useful, without the complication of creating a GUI. Today I wrote two macros - SearchMacro and RelatedMacro - which was a helpful exercise in searching the database and extracting and manipulating wiki meta data.

A note about the development environment: environment setup (or at the very least --- tracdeveloperplugin from trac-hacks, provides a very nice plugin registry with all interfaces and components, as well as several handy debug tools)

Here's an overview of what my macros do: (Note these are only initial prototypes to test using these features within the context of a wiki page)

SearchMacro - parameters: author, topic, title, groupby and order. Searches through the project's wiki pages based on the specified criteria. The parameters are optional; if no parameters are specified it displays a list of all wiki pages within the current project grouped by author and ordered in ascending order.
--- author, topic and title take any string as an argument;
--- groupby takes one of the following: ('author', 'date');
--- order takes one of the following: ('incr', 'decr').
More parameters could be added easily (e.g. "num_results" to specify the number of results returned and "num_days" to choose results created within the last 5 days for example).
RelatedMacro - takes no parameters and displays a list of "author - title" pairs with links to related wiki pages. The idea is that this macro returns relevant results based on meta data about the user, without the user having to explicitly specify search terms (in contrast to SearchMacro).

Quick Reference:
Trac Data Models - WikiPage and Ticket models
The Trac environment & TracIni
Trac Plugin Development - list of extension points
Database API & /trunk/trac/db
Genshi API - generating markup streams from python code

Next:
0) evaluate prototypes (+more design decisions)
1) learn pyGTK (library for creating GUIs)
2) start coding the actual modules, with graphical interface

The Trac Plugin API

2009-05-21T14:04:00.000-04:00

More preliminary reading.

TracDev - Component Architecture
The plugin API for Trac; trac.core defines four public classes: Component, ExtensionPoint, Interface and ComponentManager. The different tokens of functionality, i.e. services, are modelled by the Component class, and each provides one or more ExtensionPoints, to which other components can plug in to via an Interface. A plugin is itself a Component, and can both use interfaces to connect to other components by way of their extension points, as well as provide extension points of its own. Here's an example, where WikiPage is the core component, IWikiPageManipulator is its interface, and WikiPagePlugin is another component which extends it:

class IWikiPageManipulator(Interface):
   def some_method(some_value):
   ...

class WikiPage(Component):
   plugins = ExtensionPoint(IWikiPageManipulator)
   ...
   def some_action(self, some_value):
       for plugin in plugins:
           plugin.some_method(some_value)
       ...

class WikiPagePlugin(Component):
   implements(IWikiPageManipulator)
   def some_method(self, some_value):
       print some_value
   ...

Some useful interfaces/points of entry:
trac.core - definitions of the four core classes (read as reference);
trac.wiki.api - provides IWikiChangeListener (for keeping track of changes/history), IWikiMacroProvider and IWikiSyntaxProvider (extending the core macros and syntax - could write macros for embedding functionality such as data visualization);
trac.env - components are usually intialized with env (as a component manager); used to store project information (configuration file, sqlite db, templates, plugins, wiki, tickets - in a directory structure);
trac.env.Environment.get_db_cnx() - the most common entry point for a database connection; also useful: get_repository(), get_known_users() - this is probably more useful for the social network graph project;
trac.search.api - ISearchSource interface (search filters) and handy search_to_sql() function;
trac.versioncontrol.api - could be used for exploring the repository; get_history() and get_annotations() for annotated revision history of a particular Node (directory or file); this relates to the idea of tracking relationships between files which were changed in a short period of time;
trac.perm - permissions.

Even more importantly: the database schema - access to all sorts of useful information. The wiki table holds data about all wiki pages and could be searched based on 'name', 'text' or 'comment' for implementing the search tool widget I mentioned in an earlier post; the 'comment' attribute serves the purpose of metadata for wiki entries; could select all wiki pages edited by specific author and order by time to provide a chronological history of wiki edits for a particular user (think about how to extend this to record local changes to a particular wiki page, as opposed to global changes such as adding new pages).

Next: some design decisions to be made - which specific features would be supported by the plugin, and which extension point interfaces could I make use of? Having looked at the plugin API and related documentation, I'm starting to think that I can classify the features of my potential plugin as:

Those which extract information from the project database and provide it to the user. Example: the search feature. Implementation: could consist of a set of well-written queries into trac.db. User interface: custom macro.
Those which make configurational changes and extend the wiki functionality. Example: contextual commenting system. Implementation & user interface: still need to investigate these (meaning I'll be spending lots of time reading source code on trac-hacks :-)

Developing software tools for existing configuration management system

2009-05-20T10:32:00.010-04:00

As a starting point for development of a set of tools constituting an electronic lab notebook, I'll focus on its implementation to the current FCM (flexible configuration management system) in use at the Hadley Centre. The FCM uses svn for version control and trac for issue tracking. Since trac is a software management portal which consists of a wiki, an issue tracker and a subversion repository browser, it would make sense to focus on building a plugin for the Trac wiki (which is already being used by scientists for recording information about their climate simulations...but I need Steve's notes on that).

Specifics of implementation:
trac-hacks.org is a site which provides svn hosting of community-created Trac plugins, macros, scripts, themes, patches. Trac plugins are packaged as python eggs. Python modules and packages are distributed using distutils (setuptools, an extension to distutils, is used to build eggs).

Currently:
I created a local svn repository (using some statsjam source files) & a trac project for testing purposes (it's useful to modify the default trac permissions). I also created several python eggs from existing projects to test using setuptools. Everything's working OK so far. Next I'll examine an existing trac plugin's source code as a useful example and starting point for writing my python modules.

Electronic Lab Notebooks

2009-05-19T03:49:00.135-04:00

It seems that the majority of scientists who record experimental data electronically use one of a small set of tools: a wiki page (e.g. the climate change scientists at Hadley centre; OpenWetWare at MIT), a blog (e.g. a chemical experiment, part of the Open Notebook Science initiative [1]), or spreadsheet software (e.g. the scientists from a medical imaging group at UHN).

All of these tools are not sufficiently suited to scientists' needs for two major reasons: firstly, all of the information needs to be manually inputted (and since a large part of experimental writeups is similar, a lot of the information needs to be copy-pasted from previous pages -- this process is not optimized); secondly, the relationship between related experiments/simulations/papers/authors is not immediately obvious because the linking between related pages is ineffective (if it exists).

Here's a list of preliminary ideas for some features and functionality (some, but not all, of these already implemented, see OpenWetWare [2]):

Modularity - after considering the possibility of building wiki page templates, I think that a template for an entire page, being suited to the particular requirements of one experiment, would be too specific to the particular research group for which it was developed. Instead, dividing up the structure of the page into separate independent modules (i.e. thematic sections), which could be independently added, modified and deleted, would allow for quick & easy customization as well as adaptability to different contexts and experiments (flexibility is important).
Extended functionality - a suite of widgets embedded into a side bar for example, could provide useful features such as searching through existing wiki experiment pages, which would allow a user to find and preview related experiments (as well as re-use content). Example: User A is writing a wiki page about climate simulation A; User B already wrote a very similar wiki page about climate simulation B; User A could use the search widget to find wiki page B and import one or more modules, so that user A can avoiding the need to re-enter repetitive information and can only focus on editting the different bits.
Cross-linking - using the search widget, finding related experiments as well as external resources would be easy;
History of Recent Changes - as the user creates a wiki page, adds modules and searches through related papers, it could be useful to record every change made to the wiki page, chronologically;
Meta data - tags (categorical and descriptive? [4]) to allow for easy browsing & searching through wiki pages;
Permissions - since experiment wiki pages would ideally be searchable and cross-linked, there could be a feature marking certain modules as 'private' (e.g. for personal annotations);
Import/Export of Data - embed data sets into wiki page (e.g. Statsjam - an extension to MediaWiki which allows users to embed database queries and dynamic visualizations into their wiki pages); ability to "export" page summary in a portable format (e.g. reference-card format with abstract and conclusions);
Contextual Comments - allow users browsing a lab notebook to make contextual comments/questions related to a particular section (science as a collaborative effort - [1] - see example of comment system below).

Project starting points

2009-05-19T00:29:00.044-04:00

Out of the topics discussed over the course of the past few days, several seem to have emerged as both significant in their impact on climate modelling research, and viable to attempt development and implementation over the summer.

The climate research effort falls in the domain of computational science, where scientific models are typically written and maintained by climate scientists who are usually unaware or uninterested in software engineering practices and tools. The extent to which software engineering processes and tools could improve the working environment of climate modelling centres has only recently become a topic of interest [1]. A study of the Hadley Centre [2] reveals that careful selection of a software configuration management system for managing the centre's code base has had a largely positive impact on productivity and reduce development release cycles.

Annotated reading:

Advances in software development practices as described from a software engineering standpoint are not met by corresponding practices in most of the large climate change research centres. The specific software needs of science researchers include the use of older programming languages (e.g. Fortran) for modelling and simulation due to the long lifespan of projects; the ability to keep track of code versioning used in individual experiments; as well as the need to arrive at reproducible code results.; Further, scientific code verification and validation is difficult to define since the "correct" results are not known prior to experimentation. Techniques used include bit comparison and integration testing.; Interesting to note is that advances in computing power do not lead to faster modelling techniques - instead, the complexity of the models is increased to improve accuracy and reduce the number of parameters, while maintaining an average run time of about several weeks to several months for large coupled GCMs (general circulation models).; A case study of the Hadley Centre reveals that open-source tools such as Subversion and Trac serve as excellent building blocks for scientific software development practices, as the role of scientific researchers overlaps with that of some members of the open-source community. Specifically, scientific computation requires that scientists are both users and developers of their code, which may have interesting effects on code reliability (Would code tend to be less defect-prone since its developers are most familiar with its application? Or, on the other hand, would code tend to be less reliable since the main developers are largely unqualified in the field of software development?)

[2] D. Matthews, G. Wilson and S. Easterbrook, "Configuration Management for Large-Scale Scientific Computing at the UK Met Office", online paper: Given that the scientific software development process depends on two things: the amount of time it takes scientists to develop the code and the amount of time it takes the supercomputers to run the GCM simulation, and the latter is constantly being improved as per Moore's Law, the inevitable conclusion is that the real problem causing inefficiency in scientific computing is the lack of an effective code development process.
: Characteristics of a successful configuration management system:

Code maintenability (as a way of improving development practices) is often more important than improvements in computing power
The combination of open-source tools with custom-made FCM interface allows much-needed independence of external sources while the FCM system lowers the adoption costs and customizes the system to simplify common tasks
Cooperation between IT specialists (familiar with the scientists' needs) and management (through initiative and support)
A service, not a tool (ongoing software support) - plenty of documentation and live support
Thoughtful migration process - old code + history was imported to svn and tickets were imported into Trac, i.e. minimal disruption to ongoing code development

Introduction.

2009-05-19T00:02:00.006-04:00

I'll be using this blog to record thoughts and impressions related to my current research interests, with the hope that ideas of interest may come out of my daily musings - and to prevent such elusive occurences from getting lost in the mass of the inconsequential.

I have just completed my first year in the Engineering Science program at the University of Toronto, with an interest in computer science. My current summer position is at the Department of Computer Science, working on a new research project aiming to define the role of software engineering tools in enabling climate change research. The project is not limited to a particular application, but if any worthwhile software tools end up being developed this summer they will be tested at the Hadley Centre (MetOffice UK, see also: the unified model and the FCM).

My supervisor is Steve Easterbrook (see also Serendipity).