Python and SublimeText

Function hints and autofill of function parameters - notes to help me out in the future.

Install the SublimeText Anaconda Package (not the Python distribution Anaconda).

The Python Improved package and the Neon Color Scheme provide nice Python typing support

When using Docker, Anaconda doesn't necessarily know where the python package your are editing is so you need to help it by adding the following into the SublimeText Project settings (Project -> Edit Project). Note you can't open up the .sublime-project file using SublimeText as it will just keep opening up the Project and not the file. Add in the settings with the top folder of the python package:

"settings": {
        "extra_paths":
        [
            ""
        ]
    }

Also, make sure you add the python modules using pip that you are using in the Docker python system to the python interpreter (you can also use a virtual env for the project and that should be added to the SublimeText project. You should now have function and method hints from your project and any additional python modules you are using. My Anaconda user settings:

{
    "python_interpreter": "/opt/local/bin/python",
    /*
        If complete_parameters is true, anaconda will add function and class
        parameters to its completions.

        If complete_all_parameters is true, it will add all the possible
        parameters, if it's false, it will add only required parameters
    */
    "complete_parameters": true,
    "complete_all_parameters": false,

    // If true, anaconda draws gutter marks on line with errors
    "anaconda_gutter_marks": true,

    // Inline error messages by inserting extra lines as needed
    "anaconda_linter_phantoms": false,

    /*

        If anaconda_gutter_marks is true, this determines what theme is used.
        Theme 'basic' only adds dots and circles to gutter.

        Other available themes are 'alpha', 'bright', 'dark', 'hard' and
        'simple'. To see icons that will be used for each theme check
        gutter_icon_themes folder in Anaconda package.
    */
    "anaconda_gutter_theme": "bright",

    /*
        If 'outline' (default) anaconda will outline error lines
        If 'fill' anconda will fill the lines
        If 'none' anaconda will not draw anything on error lines
    */
    "anaconda_linter_mark_style": "outline",

    /*
        A list of pep8 error numbers to ignore. By default "line too long" errors are ignored.
        The list of error codes is in this file: https://github.com/jcrocholl/pep8/blob/master/pep8.py.
        Search for "Ennn:", where nnn is a 3-digit number.
    */
    "pep8_ignore": [
        "E127", "E128", "E501", "E402"
    ],

    "anaconda_linter_mark_style": "outline",

    /*
        Set the following option to true if you want anaconda to check
        the validity of your imports when the linting process is fired.

        WARNING: take into account that anaconda compiles and import the
        modules in the JsonServer memory segment in order to check this
    */
    "validate_imports": false,

    /*
        MyPy

        Set the following option to true to enable MyPy checker.
    */
    "mypy": false,

    /*
        Command to execute tests with. nosetests by default
    */
    "test_command": "py.test",

    // Maximum line length for pep8
    "pep8_max_line_length": 79

}

Aurelia View Engine Hook for importing enums, static variables, etc

Thanks to a query from Github user @TheNavigateur asking "Allow[ing] import of exported module items into aurelia template, to allow access to static methods, static variables, enums, type definitions, etc. "

and then Rob Eisenberg's example (issue/comments) of how to do this:

Aurelia custom element async life cycle event

I saw a really good conversation in the Aurelia/framework Github issue queue that I wanted to save for later.  https://github.com/aurelia/framework/issues/367#issuecomment-198104416

From Rob Eisenberg:

We can't provide this (async promise life-cycle events) across all components. It would be a disaster for performance and would no longer map in any way to web components. 

If you don't care about web components, you can use the new CompositionTransaction: http://aurelia.io/docs.html#/aurelia/templating/1.0.0-beta.1.1.2/doc/api/class/CompositionTransaction 

Simply have that injected into your component constructor and then call enlist() this will return you a CompositionTransactionNotifier: http://aurelia.io/docs.html#/aurelia/templating/1.0.0-beta.1.1.2/doc/api/interface/CompositionTransactionNotifier 

You can call done on that when your async operation is complete. The global composition will wait to attach until after you are done.

How do I wait for async data for an Aurelia custom element?

JSON Web Token payloads

I really like the JSON Web Token (JWT) technology for Single Page Application (SPA) user authentication.  I started off quite excited about storing things like permissions and other user profile data in the JWT.  My initial thoughts around JWT is that it was a good ‘state’ variable to hold user profile information on the client side.

The problem with that idea is that it is hard to expire and update a JWT if the user changes their display name or gets new permissions/claims.  If your SPA client is using the JWT payload to hold this mutable (server side mutable) information, your client won’t be updated in a timely manner - it will have to wait for a new JWT to be issued.  It is a little mean to require the user to logout and back in again to be able to use their new permissions or see their new display name show up on the web application.

Don't get me wrong, I'm not talking about concerns of the client side manipulation of the JWT payload.  I don’t trust anything from the client side.  Anything coming from the client has to be checked/validated on the server.  I know the new permissions really matter on the server side, but I would generally only allow users to see functionality that they are permitted (not that they can’t hack the front end code and do whatever they want on the front end - they just would not be able to complete the transaction on the server).

Aurelia delegate vs trigger

Great StackOverflow answer: Aurelia delegate vs trigger question

Aurelia change event Firefox woes

In the continuing adventures of Firefox vs other browsers (see Aurelia Firefox and Input fields), I was watching a checkbox using a change.trigger() to update the checkbox selection state for a search facet.  I noticed in Firefox that the change event checkbox selection value was pre-change vs Chrome and other browsers where the selection value was post-change.

Gitter Aurelia conversation discussed the issue and approach to fix it.

Using change.delegate gives the change event time to finish processing before it's handled.  I'm a little concerned about timing issues still as this doesn't seem deterministic, but it's working pretty well now.

We all make our mountains and we make them as tall as we need them

Putting things into perspective doesn't seem to work well as a coping mechanism.  If we have a problem, it doesn't matter that the problem isn't a big deal to anyone else.  It only matters how large a problem it is to us.  When other people try to minimize a problem (mountain) or put it in perspective, that doesn't help to make our mountain smaller to us, it just minimizes us compared to our mountain.

Aurelia, Firefox and Input fields

I had a mis-behaving checkbox input field on Firefox.  The checkbox was working fine on Chrome and Safari - likely IE as well.  But it wouldn't stay checked on Firefox - kept resetting itself to unchecked.  For the short version, @jsobell on Aurelia's Gitter channel told me that Firefox triggers input attributes based on alphabetical order.  I had a model.bind(), checked.bind(), and change.trigger() on the checkbox in question in that order.  The change.trigger() was running last with the other browsers and first in Firefox.

It was sloppy code on my part, I should have processed everything in the change.trigger function instead of a kludge of model.bind and checked.bind with a change.trigger that was really carrying the main functionality.

Dynamic input list in Aurelia

I struggled with getting this dynamic form pattern figured out in Aurelia so I'm sharing it here for myself in the future and others that might find it useful.

The features needed are:

• Automatically add new blank input fields at the end of the list as needed
• Allow the removal of any items from the list at any point
• Allow changing any items in the list of inputs

Given lots of help from the Aurelia Gitter channel especially by Jeremy Danyow (@jdanyow) and Io Sulfur (@iosulfur) who created several Plunkrs zeroing in on the right solution, we came up with this approach as demonstrated in the Plunkr below.

Anytime you change the blank item field at the end, it will add another empty input field at the end. If you click on the 'X' button, it will remove that item.

Demo Plunkr

The key things to note in the app.html listing below are that the 'items' in line 6 must be your actual array.  If this is in a nested repeat.for loop and items is made available from the parent repeat.for, this approach will not update the original array only the one that is local to the repeat.for context.

The change.delegate on line 8 adds the blank line using the current index value of the repeat and the change event.  You'll see how they are used in the app.js file below.

The click.delegate very simply removes the input field by splicing out that item from the items array.

app.html

The first thing to do is to add a blank item at the end of the items array before we even present it to the View in line 7 of app.js

The addBlank function first sets the current item (based on the $index value passed from the change.delegate function) to the event.target.value and then blanks out the event.target.value.  We blank out the event.target.value as otherwise that value gets added to the end of the list of input fields.

We then check to see if the last input field is empty or not, and if it returns true, we push an empty string onto items.  (Note: if anyone can explain why you get the event.target.value instead of the empty '' string unless we reset the event.target.value, please post it in the comments - because I've not been able to figure it out.

The removeItem is pretty self-explanatory - so that's all.  Thanks for reading, hope it is useful.

Added (2015-08-30) :  When I tried this with an object as the item, I didn't need lines 12 and 13 in app.js.

app.js

Logging in Aurelia

Just some quick notes on setting up logging in Aurelia

Add customer log appender

Create a file in ./resources/custom-log-appender.js and add the following class (or any other file as long as you change the import in the main.js file).

export class CustomLogAppender {
  constructor(){}
  debug(logger, message, ...rest){
    console.debug(`DEBUG [${logger.id}] ${message}`, ...rest);
  }
  info(logger, message, ...rest){
    console.info(`INFO [${logger.id}] ${message}`, ...rest);
  }
  warn(logger, message, ...rest){
    console.warn(`WARN [${logger.id}] ${message}`, ...rest); 
  }
  error(logger, message, ...rest){
    console.error(`ERROR [${logger.id}] ${message}`, ...rest);
  }
}

Update your main.js

import {LogManager} from 'aurelia-framework';
import {CustomLogAppender} from './resources/custom-log-appender';

LogManager.addAppender(new CustomLogAppender());
LogManager.setLevel(LogManager.logLevel.debug);

export function configure(aurelia) {
  aurelia.use
    .standardConfiguration()
    // .developmentLogging()
    .plugin('aurelia-animator-css');

  aurelia.start().then(a => a.setRoot());
}

and in your View Model

# ViewModel
import {LogManager} from 'aurelia-framework';
let logger = LogManager.getLogger('viewmodulename');
logger.debug('me');

export class MyViewModel() {
  logger.info(“Hah”);
}

Enabling ES2016(ES7) Async functions in Aurelia

I was having a lot of problems getting the Javascript ES2016 (ES7) async functions working in Aurelia. I knew I had to update the config.js code in Aurelia by adding es7.asyncFunctions:

System.config({
  defaultJSExtensions: true,
  transpiler: "babel",
  babelOptions: {
    "optional": [
      "es7.decorators",
      "es7.classProperties",
      "es7.asyncFunctions",
      "runtime"
    ]
  },

I then tried to add my async method function to an Aurelia ViewModel:

async activate () {
  try {
    this.results = await this.api.search();
    console.log(`Search results: ${this.results.evidence}`);
  }
  catch (err) {
    console.log(err);
  }
}

and promptly got this error in my javascript console after the page loaded:

ERROR [app-router] ReferenceError: regeneratorRuntime is not defined

Jeff Bellsey (@jbellsey) on Gitter -> Aurelia/Discuss was kind enough to figure out the issue and share it with me.  I was not however smart enough to understand the answer at the time. He told me I had to add the runtime to the babel-options file (<AureliaProjectRoot>/build/babel-options.js):

module.exports = {
  modules: 'system',
  moduleIds: false,
  comments: false,
  compact: false,
  stage:2,
  optional: [
    "es7.decorators",
    "es7.classProperties",
    "es7.asyncFunctions",
    "runtime"
  ]
};

The “runtime” and “es7.asyncFunctions” lines need to be added to the file.

This was a LOT of fun to sort out.  However, I now have, for me, much more understandable code using the async, try, await, catch format.  Thank you Jeff and the other fantastic Aurelians!

 

 

Python SSH Tunnel Example

Notes for myself and hopefully others.

def createTunnel(localport, remoteport, identityfile, user, server):
    """Create SSH Tunnels for Database connections"""

    import shlex
    import subprocess
    import time

    sshTunnelCmd = "ssh -N -L %s:127.0.0.1:%s -i %s %s@%s" % (
		localport, remoteport, identityfile, user, server
	)

    args = shlex.split(sshTunnelCmd)
    tunnel = subprocess.Popen(args)

    time.sleep(2)  # Give it a couple seconds to finish setting up

    return tunnel  # return the tunnel so you can kill it before you stop
				   # the program - else the connection will persist 
				   # after the script ends


def closeSSHTunnel(tunnels):
    """Close SSH tunnels - given the process handles"""

    for tunnel in tunnels:
        tunnel.kill()

localport = 27018  # local port for MongoDB
remoteport = 27017  # remote server port for MongoDB
identityfile = '/home//.ssh/id_rsa.pem'
user = 'ubuntu'
server = 'example.com'

# Start tunnel
tunnel = createTunnel(localport, remoteport, identityfile, user, server)

try:
	# Example usage
	db_server = 'localhost'
	db_port = 27018
	client = pymongo.MongoClient('mongodb://{}:{}'.format(db_server, db_port))
	pydb = client.agencies

finally:
	closeSSHTunnel([tunnel])

Installation Notes for Flask App on Ubuntu 14.04 LTS using gUnicorn

Followed these excellent directions from Real Python and modified for python3 and Ubuntu 14.04.
Start with updating ubuntu and loading additional packages

sudo apt-get updatesudo apt-get install -y python3 python3-pip nginx mongodb supervisorsudo pip3 install virtualenv



sudo mkdir /var/wwwsudo chown ubuntu:ubuntu /var/wwwmkdir /var/www/flask-appmkdir /var/www/flask-app/logscd /var/www/flask-app

Setup virtualenv

virtualenv flask_env
source flask_env/bin/activate
pip install -r requirements.txt
pip install gunicorn

Setup nginx

sudo /etc/init.d/nginx start
sudo rm /etc/nginx/sites-enabled/default
sudo touch /etc/nginx/sites-available/flask-app
sudo ln -s /etc/nginx/sites-available/flask-app /etc/nginx/sites-enabled/flask-app
sudo vim /etc/nginx/sites-enabled/flask-app

Add the following to the nginx flask-app conf file being edited

server {
    location / {
        proxy_pass http://127.0.0.1:8000;
    }
    location /static {
        alias  /var/www/flask-app/flAsk-app/static;
    }
}

sudo service nginx reload

Setup gunicorn start file

Setup bash script to run gunicorn

cd /var/www/flask-app
touch gunicorn_start
chmod a+x gunicorn_start
vim gunicorn_start

Insert the following into the gunicorn_start bash script

#!/bin/bash

NAME="flAsk-app"
FLASKDIR=/var/www/flask-app
VENVDIR=/var/www/flask-app/flask_env
SOCKFILE=/var/www/flask-app/sock
USER=ubuntu

GROUP=ubuntu
NUM_WORKERS=3

echo "Starting $NAME"

# activate the virtualenv
cd $VENVDIR
source bin/activate

export PYTHONPATH=$FLASKDIR:$PYTHONPATH

# Create the run directory if it doesn't exist
RUNDIR=$(dirname $SOCKFILE)
test -d $RUNDIR || mkdir -p $RUNDIR

# Start your unicorn
exec gunicorn runserver:app -b 127.0.0.1:8000 \
  --name $NAME \
  --workers $NUM_WORKERS \
  --user=$USER --group=$GROUP \
  --log-level=debug \

Setup Supervisor

This will run and restart the Flask application when the application files are changed.

cd /etc/supervisor/conf.d
sudo vim flask-app.conf

Insert the following into the flask-app.conf file:

[program:flask-app]
command = /var/www/flask-app/gunicorn_start
user = ubuntu
stdout_logfile = /var/www/flask-app/logs/gunicorn_supervisor.log
redirect_stderr = true

Start flask-app gunicorn:

sudo supervisorctl update
sudo supervisorctl status

You can use the following commands as well:

sudo supervisorctl start flask-app
sudo supervisorctl start all
sudo supervisorctl help|avail|stop|restart

Test that the application is running

Insomnia due to back pain

I've noticed a number of people are posting on medical advice websites with the following symptoms: Sleep for 2-5 hours and wake up with a lot of back pain which goes away after getting up. Even napping during the day lying down will cause back pain in a couple of hours especially after a meal.

If this is happening to you, you may be suffering from acid-reflux erosion of the esophagus which can cause enervation (activation) of the nerves in the esophagus and stomach. This leads to your back pain via a process called Referred Pain. Referred pain is the process where pain induced in one part of your body is 'felt' in another part. As you can see from the chart in the Wikipedia article, stomach-based referred pain shows up in the middle of the back along the spine.

Now these pain symptoms are not going to be an exact match to the chart. Personally, I would feel like an electric charge was running along one of my ribs from my spine to my front during the earlier stages of my symptoms. Everybody's anatomy is slightly different and you'll get variations on the theme. However, it was primarily back pain that I was feeling. Now my pain almost exactly matches the chart.

After way too many specialists including a neurologist who should have had a clue about this, my GP at the time, Dr Karkalis in King of Prussia (fantastic doctor IMHO) who likes to review challenging patient charts at night for fun, thought my symptoms might be due to acid reflux. He was right, but in order to finalize the diagnosis, he prescribed a proton-pump inhibitor. After a few weeks my back pain from sleep was reduced significantly.

Another way to diagnose this and manage it longer term is to tilt the head of your bed up by 15-20 degrees to let gravity keep the acid in your stomach instead allowing it into your esophagus. If your back pain abates (not necessarily overnight though it often does) fairly quickly, then your back pain is likely due to acid reflux.

Sadly, you will need to sleep elevated for the rest of your life if you have acid-reflux back pain. The proton-pump inhibitors are pretty safe drugs, but they are not always be able to control your acid-reflux enough to control the pain completely. As you can see from my hammock blog post, hammocks work really well to set the right angle for sleeping and are much less expensive than adjustable beds.

Your acid reflux will be more or less severe at different times due to exercise, stress, eating habits, body weight so your management of it will need to accommodate the severity of your acid reflux. In other words, you may need to increase the angle of your bed when your acid reflux is more severe.

Also, check with your doctor and get tested for Barret's Esophagus. Basically, serious acid reflux sufferers are at higher risk for esophageal cancer.

Acid Reflux, Camping, Hammocks and Holes

This is a personal blog article.  I generally try to keep this to professional matters, but I would like to record this for those of us suffering from acid reflux.  I've been dealing with it for over a decade now and mostly control it via mechanical means - sleeping on an incline.  After a couple of hours sleeping flat, I get referred pain from my esophagus which feels like back pain.  I try really hard to figure out how to sleep on an incline so I can get 6-8 hours of sleep instead of 2-3 hours of sleep and then 2-3 hours sitting up until the pain subsides, rinse, repeat.  

I was camping last weekend, first time since I've started noticing my acid reflux.  I generally suffer when away from home and try all sorts of tricks with setting the bed on blocks to create an incline, build a mound of pillows, or sleep in a recliner.  I wasn't looking forward to sleeping on the ground for a couple of nights.  

I tried to find a spot on a bit of a slope, but that had to be balanced with the desire to not slide down on a slippery tent floor in a slippery sleeping bag.  The first night was absolutely miserable.  I did manage 5 hours or so I think.  I woke up somewhere between 3-4am in pain and left the tent so I wouldn't disturb my campmates.  Being in pain, cold, and really tired doesn't make for a lovely camping trip.  

I was absolutely dreading the coming night and trying to find a nice spot of ground (shaped like a recliner) to set up my sleeping bag and pad and praying that it wouldn't rain.  My wife fortunately found the right spot and not something I would have thought of.  It was a hole about 2 feet deep, 2 feet wide and 3 feet long in a U shape.  It turned out to be a perfect, recliner-like, shape.  I tried it out and immediately felt comfortable.  Long story short, I had a great night sleeping in the hole.  I heartily recommend digging a hole to sleep in if you have trouble sleeping on your back or sleeping flat due to acid reflux issues.

This experience started me thinking about using a hammock when away from home as it would provide the same shape..  Looking on Google, there are a LOT of articles on using baby hammocks for babies with acid reflux and very few about adults using hammocks to manage acid reflux.  I took the plunge anyway and bought a hammock to try out.  I slept in it last night, and it worked great.  I was able to sleep for a very long time (about 10 hours - trying to catch up on missed sleep).  It was very comfortable all night, and I had no pain indicative of acid reflux this morning. 

It seems to be a successful experiment - though I do need a few more data points to fully confirm it of course.  Assuming that additional data points confirm this approach, I need to figure out how to take the hammock with me when traveling.  Hotels don't generally provide hammock hooks in the wall.  Travel-wise, hammocks don't take up much room which is good as I hate checking bags when I fly.  The hammock I tried out was the ENO Double Nest Hammock which doesn't take up much space or weight (about the size of a grapefruit and less than two pounds).

ENO Double Nest Hammock (Tomato/Khaki)

ENO Double Nest Hammock (Navy/Olive)

These straps are handy to hang the hammock with:  ENO Slap Straps

I'd recommend getting two carabiners to replace the ones that come with the hammock based on reviews I saw on Amazon:  Black Diamond Neutrino Carabiner - Grey

 

 

String Similarity

This post is my attempt at recording a very nice thread of posts on BioNLP.org's mailing list on string similarity measures. Harsha G at Molecular Connections asked about string similarity measures which prompted 

Tools recommended:

• Simmetrics
• Simstring
• secondstring
• automaton
• Python NGRAM
• AGREP
• RE2
• TRE
• FREJ

Papers:

• http://dx.doi.org/10.1093/bioinformatics/btm393 - papers by Okazaki and others on string similarity metrics (shared by Sampo)
• http://dc-pubs.dbs.uni-leipzig.de/files/Cohen2003Acomparisonofstringdistance.pdf
• Pubmed Distance paper - Lu 2009
• Névéol - 2010 - Author Keywords in Biomedical Journal Articles 
• http://aclweb.org/anthology-new/D/D08/D08-1113.pdf
• http://lingpipe-blog.com/2010/04/27/mccallum-bellare-and-pereira-2005-a-conditional-random-field-for-discriminatively-traied-finite-state-string-edit-distance/

Responses:

From Tudor Groza:

Dear Harsha,

I would suggest you have a look at Simmetrics [1] - it is a comprehensive
package for string similarities ranging from basic ones, like Levenshtein
distance to more advanced one, like Smith-Waterman or Needleman-Wunch. You
can find the Java API at [2] - for some reasons the original page is
missing, hence the only way to get to it is via the Web archive.

Hope that this helps.

[1] http://sourceforge.net/projects/simmetrics/
[2]
http://web.archive.org/web/20081225104938/http://www.dcs.shef.ac.uk/~sam/simmetrics/index.html

Kind regards,
Tudor

 

From Sampo Pyysalo:

Dear Harsha, all,

Not sure what your exact needs are, but I've found that in
approximate-matching lookup against many larger biomedical resources it's
good to do a fast, comparatively simple first-pass lookup before running
more advanced string comparison algorithms to avoid the computational costs
of full comparison for a large number of string pairs. I've found Naoaki
Okazaki's simstring (http://www.chokkan.org/software/simstring/) to be
excellent for this first task. The way I'd recommend to use this is to
first filter a large string collection to a reasonably-sized set of best
matches (in terms of a comparatively coarse similarity function like char
n-gram cosine) with simstring and then run more advanced stuff like
custom-cost edit distance for this smaller set.

There are a number of studies by Okazaki as well as Yoshimasa Tsuruoka and
others on the topic of string similarity metrics for domain tasks that may
also be of interest to you, e.g.
http://dx.doi.org/10.1093/bioinformatics/btm393

Cheers,

Sampo

From Florian Leitner:

Dear Harsha,

A good overview is the 2003 W. Cohen paper "promoting" the SoftTFIDF measure and with a very good overview of available similarity measures:

http://dc-pubs.dbs.uni-leipzig.de/files/Cohen2003Acomparisonofstringdistance.pdf

As for libraries to do string similarity matching, there are many, many options available. As they have not been mentioned so far, most prominently, there are the Regular Expression libraries.

-- REGEX
In terms of pure speed, some of Google's own searches are powered by re2 (developed by a Google search engineer), a deterministic RegEx ("DFA")  engine that is significantly faster than the "default" engines available in most other programming languages (because they are all are at least in parts non-deterministic, i.e., "NFAs"). However, due to the pure deterministic nature there is quite some default functionality missing (e.g., lookaheads and -behinds, etc.), so you have to define all variants you wish to match in your patterns (no approximate matches!), while it is blazingly fast:

http://code.google.com/p/re2/

In terms of pure approximate matching speed, don't forget that *nix offers a pretty powerful approximate string matching implementation right at your "fingertips":

http://www.tgries.de/agrep/

Last, another C implementation of a POSIX compliant approximate (DFA-based) regex matcher is TRE, although this is library is therefore somewhat slower than the RE2 engine, too:

http://laurikari.net/tre/

These three regex libraries are probably the most noteworthy if you need raw speed. Then there are a few Java regex libraries that seem noteworthy, too:

First, there is a non-determinisitc RegEx engine (FREJ) to do approximate matching, also in Java:

http://frej.sourceforge.net/

And yet another Java regex implementation, partially DFA and partially NFA, is the Brics Automaton:

http://www.brics.dk/automaton/

(there are much more Java regex libraries, but let Google be your best friend if you need even more pointers...)

-- DISTANCE
Apart from the regex/D- or NFA based implementations, there are distance-based measures to do approx. string matching. A very fast similarity search tool is SimString, an approximate matcher based on distance measures, and already mentioned by Sampo in his post, in C++:

http://www.chokkan.org/software/simstring/

Probably the most well-known package in this domain is the SecondString package from the CMU (from W. Cohen, the author cited above) for approx. string matching in Java, also based on edit distance measures:

http://secondstring.sourceforge.net/

Last I'd mention there is a simple Python module to calculate n-gram-based similarities; while I do love Python very much, alone due to that fact that this is Python-based, it will most likely be the slowest option listed here:

http://packages.python.org/ngram/index.html

Hope this helps to get you up to [matching] speed!

Cheers,
Florian

 From Aurélie Névéol:

Harsha,

Another measure to look into is the "PubMed distance" described in this paper:

Lu Z, Wilbur WJ. Improving accuracy for identifying related PubMed queries by an integrated approach. J Biomed Inform. 2009 Oct;42(5):831-8.

An example of use and evaluation can be found in this other paper:

Névéol A, Islamaj-Doğan R, Lu Z. Author Keywords in Biomedical Journal Articles. Proc AMIA Annu Symp. 2010:537-41.

Best regards,

Aurelie

From Bob Carpenter:

I'd suggest looking further than Jaccard distance in
the LingPipe matchers.  We have TF/IDF matchers based
on character n-grams that are widely used in practice (not
just by us or with our implementation;  note
that this is NOT the same as Cohen et al.'s soft TF/IDF,
which I've never fully understood).

There's also the Jaro-Winkler matchers, which are
tuned for matching single-word names.

LingPipe also has a dictionary-based matcher that will
spot approximate matches (by weighted edit distance) in
text using the Aho-Corasick algorithm for deterministic
matching and suffix arrays for speeding approximate matching.

And you can also use something like an HMM- or CRF-based
chunker to find matches in texts.  It basically then looks
like a named-entity problem.

If you want something fancier that should outperform any of
these methods, check out this paper by McCallum, Bellare and Pereira:


http://lingpipe-blog.com/2010/04/27/mccallum-bellare-and-pereira-2005-a-conditional-random-field-for-discriminatively-traied-finite-state-string-edit-distance/

I'm also quite keen on this method for string comparison
by Dreyer, Eisner and Smith, though I haven't tried it, either:

  http://aclweb.org/anthology-new/D/D08/D08-1113.pdf

And in the end, you may be wanting to do something like cluster
similar terms rather than just provide pairwise similarities.
Andrew McCallum and crew have done some great work on this problem,
and there's a huge swath of "deduplication" and "record linkage"
literature that's related.

CAPEX, OPEX and Cloud?

Here is a really good overview of CAPEX and OPEX as well as how it impacts Cloud Computing initiatives:

http://itprojectfinancials.com/insights/2011/06/05/capitalizing-software-development-costs-from-sdcl-to-agile/

I think we all find the CAPEX/OPEX financial issues to be troubling on multiple levels.  Thanks to Matthew Dube for letting me know about this.

Fable of the porcupine

Fable of the Porcupine:

It was the coldest winter ever, and many animals died because of the cold. The porcupines, realizing the situation, decided to group together to keep warm, but the quills of each one wounded their closest companions. After a while, they decided to distance themselves from one another, but then they began to die, alone and frozen.

They had to make a choice: Either accept the quills of their companions or die. Wisely, they learned to live with the little wounds caused by their close relationships, in order to receive the heat that came from the others. This way they were able to survive.

Moral of the story:

The best relationship is not the one that brings together perfect people; it's when each individual learns to live with the others' imperfections and can admire their good qualities.

Background:

A friend of mine, Nils Onsager, Master Hapkido Instructor, shared this with me.  I found it an excellent parable for diversity and the challenges inherent in diversity.

Lab Notebook Digitization

I've got a few posts planned about our Lab Notebook Digitization project.  We recently had to change our lab notebook archiving process.  We had been using a vendor (Carmel Business Systems - CBS) in San Diego, but when we closed our site there, it didn't make much sense to ship them across the country to be digitized and then have to ship them back.  It is too bad as CBS had done a great job.  They worked with us about about 6 years ago to build a digitization process using digital cameras which was much faster and cheaper than the old system of microfilm with scanners filling in for color lab notebook pages.

When reviewing the options, we looked at the state of the state for outsourcing digitization which six years later is much more mature.  Since we are digitizing our archived notebooks (ones archived with the older process), we were also looking for a less expensive option for several thousand lab notebooks.  We were a bit surprised at the cost of outsourcing digitization in our local market.  So surprised in fact (and due to the rather budget-breaking costs), that we looked into more of a DIY approach.

I was very happy to see how far the DIY digitization marketplace has come.  There are now at least two very good digitization vendors in the marketplace with included software for digitizing books (thank you Google Books and Microsoft and Internet Archive and the EU for creating this market).  I looked at several vendors, but the best ones for my needs were Kirtas and Atiz.  As much as I love the concept of an army of digitization robots stoking my dreams of megalomaniacal world domination, I realized the Kirtas digitization robot would not be suitable for lab notebooks which have foldouts, pockets for photographs and all sorts of other creative ways of communicating intellectual property that are impossible for the Kirtas robot to deal with.  Kirtas does have manual stations, but I wasn't as impressed with those compared to the BookDrive series from Atiz.  The LED lights, V cradle and the V glass for holding the books down were features I really liked along with the price.  Price was definitely a selling feature.

Based on the cost of a fully loaded BookdrivePro with excellent 22 megapixel cameras and the staff required to run it, we estimated our digitization costs per notebook to be about 3-4X cheaper than the best estimate we could get from an external digitization vendor.  We expect to increase that ratio to about 5-6X  after Atiz improves their software based on some suggestions a colleague of mine (James McIninch) made.

Google's Birthday Wishes

I was quite pleased today to see Google's Birthday wishes to me.  It's impressive how pervasive personalization can be and how nice it is (though still a bit creepy - I've taken to turning off the monitor now when I'm talking about something sensitive --  just kidding :)