Code Reuse is Overrated

As an engineer, I have often been conflicted about the interplay of code reuse and dependency management.  It is innate behavior for me to not want to repeat myself and it is through hard and painful experiences that I have learned to be hyper-critical of all new dependencies.

Why re-invent the wheel? Because the existing wheel comes with lots of strings attached.   This article does a good job of explaining the pitfalls in depth:

Redundancy vs dependencies: which is worse?

If it is my wheel and it breaks, I know how to and am able to fix it quickly and cheaply (measured in time).  If I depend on someone else's wheel, now I have lots of problems.  Will they fix it and when?  Will their new version introduce new things I am not expecting? Maybe they too have their own dependencies. That's to name just a few.

I was on a team where a large third-party Java library was added to the project for the sole purposes of using the "isBlank()" function.  That was not a good trade-off of re-use and dependencies. I hear the Javascript/npm world has this same sort of problem in spades.

Too often the Don't Repeat Yourself (DRY) mantra is used as a gospel, devoid of the dependency cost. I've seen development cultures where adding dependencies is done often and effortlessly: it does not even register that there should be a decision process around this. As a group, we need to be more thoughtful about the trade-offs we are making when introducing a dependency and to assign it the proper cost.

Here is another good article related to this theme:

Small Functions considered Harmful

Fallacies of Distributed Computing

 

I came across this Wikipedia page:

Fallacies of Distributed Computing

It lists the fallacies as:

• The network is reliable;
  
• Latency is zero;
  
• Bandwidth is infinite;
  
• The network is secure;
  
• Topology doesn't change;
  
• There is one administrator;
  
• Transport cost is zero;
  
• The network is homogeneous.
  
• We all trust each other.
  

These are all good things to keep in mind while you design a distributed system, but I think the use of the word "fallacy" is a bit overstated. I've seen a lot of designs (and existing systems) where some of those items have been neglected, but the reason is not because the author had "mistaken beliefs". 

Even for someone new to distributed systems, if you asked them "Is the network reliable?", they will rightly know that it is not.  If their first designs do not properly account for this, it is not because they had mistaken beliefs, but more due to their inexperience or oversight.

The same is true for the remaining items: if you asked someone the specific question, you will likely get the right answer, though their designs may still be lax in that area. 

If you forgot to pay your electric bill, I would not conclude that you have the false belief that electricity is free.

Scalable Search Engines

I have now worked at three different companies where we needed to provide a search feature that could scale into the millions of documents.  Two of these involved consumer products and the other consisted of documents for legal review.  I saw a very interesting contrast in the requirements between these two domains that I thought might be worth sharing.

The Use Cases

In the consumer product space, we had millions of users a day searching across hundreds of millions of documents, while in the legal technology space, we only had a few thousand users a day who were only searching across tens of millions of products.  

At this level, it would seem like the legal domain is a much easier system design problem to solve.  However, there are a couple of key difference between the two use cases which can make the legal search problem much, much harder.

Data Shapes

Consumer products result in fairly small documents: there is a title, some attributes and maybe a description that need to be indexed. For legal review, this could involve any type of document: if it is on someone hard drive or in their cloud accounts, it may be subject to legal review.  Out in the wild, there are Excel data files in excess of 120MB and PDF files with tens of thousands of pages.  They all need to be in the search index.  (Processing files that exhibit these extremes is a nightmare in itself, but here we'll stick with the search-related problems.)

Besides the potential for these large documents, the big issues that makes the system design hard is the variance of the sizes. There are plenty of small documents to go with these these larger documents in a legal review.  In contrast, consumer products are relatively uniform in the size of what needs to be indexed.

Query Shapes

With consumer products, the search expressions that are typical will consist of a handful of words at most (on average, about 2.3 words). Users mostly express simple concepts like "Sony 55 in. TV". In the legal space, queries are not only much longer, but a considerable amount of time is spent crafting the searches to hone in on specific concepts. Search expression are serious business for lawyers. Competing parties will negotiate search terms and argue about them before judges. It is not uncommon for a single search to have hundreds of terms.

The size of the search expression is only a small part of the story though.  The legal domain requires all the special search operators might can think of: wildcard searches, proximity searches, range searches, etc.  Not only do they require these, these are used extensively.  

Consider search for a specific person of interest in a legal case whose full legal name is "Larry Thomas Johnson".  How do you search for documents that mention this name so as to not miss any?  "Larry Johnson", "L. Johnson", "L. T. Johnson", "Larry T. Johnson" are some of the more obvious variations that might be out there. In the legal world, they will do something like this: "L? \2 Johnson", where "?" is a wildcard matching any suffix and "\2" is a proximity search looking for occurrences within 2 words.

Requiring these sophisticated queries makes perfect sense in the legal context, but they are also the types of queries that stress a search engine (in both CPU and memory resources).  The contrast of these two domains is between a high volume of simple searches and a lower volume of more complex searches. 

As with the data shapes, in the legal space the query shapes tend to have a very high variance.  There are plenty of simple queries to mix in with the more sophisticated ones.  A handful of sophisticated queries (sometimes just one) can consume enough resources to add latency to the simpler queries.  In contrast, the consumer product search query complexity is much more regularized and predictable.

Precision vs. Recall

Suppose you have a product catalog containing 10,543 televisions and a user searches for the term "TV". Let's say your search engine happens to only match 10,327 for some reason. Is the user going to notice?  Probably not. They have no idea how many products are in your catalog, or even the specific products in your catalog. The recall requirements here are a bit loose.

In contrast, in the legal domain, there is often a single document that is the "smoking gun" that can make or break a case.  If an attorney searches for it and your search engine misses it in the search results, you will soon be out of business. If the document exists, your search engine *must* match it. 

The recall requirements are strict in the legal domain, but the precision requires are not. An attorney will not mind too much to sift through a few non-relevant documents.  They are good at at pattern matching and filtering and would much prefer to get a few extra than to miss any.

Consumers search for products are a bit different.  Showing a bunch of microwave ovens in your search results for a "TV" search is going to be a bad user experience if it happens to frequently.  For product catalogs, trading off some amount of recall to improve precision is a good choice as it leads to "cleaner" search results for the user and they are none the wiser about what they might be missing. But in the legal domain, flawless recall is a requirement that is non-negotiable.

Data Normalization

 It is common for search engines to use "stop words" to filter out words with low semantic content, e.g., "the", "a", "an". Often this filtering is the default behavior.  What the consumer product and legal domains share in common is that stopwords are a bad idea.  There was a consumer brand name "THE" and an important legal case involving "Project A".  Feed that data though a stopword filter and you lose some very important semantic information.

Technology

All of the search technology I have used were based on Lucene. This includes Elasticsearch, SOLR and even a home-grown distributed version of Lucene back in the days before Elasticsearch existed. Lucene is an amazing piece of software and so is Elasticsearch.  There is mostly no reason to use anything else. The only downside I found was in the legal domain where the data shapes and query shapes are atypical of most other domains. 

Specifically, we ran into issues of limits and resource usage.  Earlier versions of Lucene/Elasticsearch lack limits in some important code paths that resulted in run-away CPU or memory usage. As they have address these "holes" they have put limits in place that are too low to support some of the legitimate legal queries that need to be supported.  Some thresholds are adjustable, but not all are, and there is always some amount of peril in playing with too many of the default parameters.

The Lucene and Elasticsearch teams focus on the most common cases for query and data shapes, which is the right thing for them to do. Unfortunately, the legal domain use cases fall outside of the norm.  Consumer product searches, on the other hand, are more inline with their priorities.

Conclusion

There are critically important differences in the search requirements for the two domains I have work in.  It feels like these represent two extremes of a problem space, I wonder how many other domains match one of these two or if there are additional "classes" of search requirements with their own unique characteristics..

Product Normalization

Are the following two products the same?

Do you think this is an easy question to answer or a hard one?  The correct answer is: "it depends" and the bigger part of that is "it depends on how you define the word 'product'".

I have worked on this problem, called "product normalization", for more than 5 years and at two different companies.  Not only was I required to be able to answer questions like these, but I needed to encode the "rules" for doing this and build software systems that could make these decisions automatically (and at scale).

Before diving any deeper, if you believe the answer to the question is either "yes" or "no", let me try to convince you that the answer is not so cut-and-dry. 

The first search-based company I worked for was a comparison shopping company.  It was a site where you could find the best price for a product by comparing all the store prices in one place.  In this context, those two products above are definitely *not* the same.  One of them is going to cost twice as much, so if we showed the price for the "2-pack" from one store next to the price for a "4-pack" from another store, we are not providing the user with a meaningful comparison experience.

The next company I worked for dealt with managing customer reviews. Suppose there are some meaningful reviews on the 2-pack and none for the 4-pack, but the user is looking at the 4-pack product page.  Do we show the 2-pack reviews on the 4-pack page?  In this case, the answer is "yes", we would not want the user to miss out on relevant reviews about the quality of the product.

Therefore, when we are comparing price, the quantity in the package is very important, and these should not be considered the same.  But when trying to make buying decision based on the quality and features of the product, the quantity is not important.

The Fallacy of UPC matching

The most basic way to compare products for "sameness" is to look at their UPC codes (EAN codes for the Europeans).  As illustrated above, this is not going to help if you are matching for the purpose of consolidating product reviews. The 2-pack and 4-pack surely have different UPC codes. 

But suppose we only need to solve the problem in the price comparison domain. Comparing UPC codes will tell us immediately these are not the same.  This starts to make the price comparison case seem trivial to solve: just match on UPC codes.

This would seem to work great, though you have to assume the following:

• you have all the UPC data for all the products; and
  
• the UPC data is accurate for all your products.

However, there are some harsh realities about UPCs that really complicate things:

• assumption of completeness and accuracy of data are always highly dubious;
  
• not all products are assigned UPCs (e..g, clothes);
• some products are assigned more than one UPC;
• UPC values can be re-used for completely different products;
• trivial changes to a product can result in a new UPC (e.g., sometimes just a different packaging causes a new UPC to be used).

 We can illustrate further with a final example. Are these two products the same?

The quantity is the same, they are now both 4-packs, but the packaging is slightly different. 

The answer would seem to be "yes, they are the same" for both the price comparison and the product review cases. However, flip the packages over and you will notice they have completely different UPC values: 

Hopefully this gives you some introductory flavor for the complications of product normalization.  I've written a much more in-depth and technical document on this subject. It's a whopping 60 pages, so not something to read unless you really need to solve this problem and need some ideas of approaches.  You can read it here:

Product Normalization White Paper

QUnit Inside Firefox Extensions

I have a fairly large amount of code inside one of my Firefox extensions (Shopping Helper) .  I also have always manually tested this code, which made changes hard to do. I recently did a major refactoring of the code, and have plans to add more features, so it was time to get serious about automating the testing, with the first step of adding a comprehensive unit test suite.  Now, just about every language has a useful framework,  usually called something like "FooUnit" (e.g., JUnit, PyUnit), so that's the sort of thing I wanted for my extension.


Here's the problem: Firefox extensions are JavaScript, but it runs in a very different context that the normal JavaScript inside an HTML page situation.  This is typically called "chrome code", to differentiate it from the code that runs in the HTML document itself.  This alternate run-time environment has always made debugging and development tricky for Firefox extensions.  It gets further complicated by security issues where the chrome code is trusted more than code inserted into HTML, so there is a "wall" between the two worlds.

Looking for JavaScript unit test frameworks, QUnit seemed to be exactly what I was looking for: same basic framework as the FooUnits, relatively mature and well regarded.  It is nicely done, but it is also not done for the purposes of being used as chrome code inside a Firefox extension.  The question is whether it would run as-is inside chrome code (very doubtful) or could be made to run inside chrome code (more hopeful) without too much effort (a bit worrisome).

Short answer is that QUnit does not run as-is in the chrome context, but can be made to do so with a moderate amount of work (it took me about 5 hours). Since I have now done this work, no one else needs to do this, so for you, it will be relatively easy with all the details I give below.

First off, here are the files you need, which are based on version 1.11.0 of QUnit.

• qunit-chrome-1.11.0.js
• qunit-1.11.0.css

Below are the details of the changes I had to make and how to go about using this inside Firefox chrome code.

 

QUnit Javascript Changes

 

Chrome vs. Non-chrome Control

To not have to fork the QUnit code for the modifications, I defined a global variable QU_ISCHROME whose setting can be changed to easily switch to being used inside a Firefox extension in a chrome window instead of a browser window.  This will default to 'false' to give the previously existing behavior and can be changed to 'true' when including in the XUL/chrome Firefox extension context. It should be a simple one value change to get the effect, though I did not check if it ran outside a XUL window.

 

Altering createElement() calls

Since QUnit creates HTML DOM nodes, and since it will live inside an XML/XUL window, when creating these nodes, we have to explicitly define the namespace when creating them because the XUL namespace is not the standard HTML tags.  We do this by replacing all calls like this:

  document.createElement( "tag" );

with something like this:

  document.createElementNS( QU_HTMLNS, "html:tag" );

where 'QU_HTMLNS' is a conveniently defined global variable we put at the top of the Javascript file which is defined as:

  var QU_HTMLNS = "http://www.w3.org/1999/xhtml";

We add a convenience function QU_createElement() and wrap all node creation in it to keep the code cleaner.

 

 Altering innerHTML Content Assignments

While doing an assignment of an HTML fragment to a node via the innerHTML property works fine in a browser window, this does not work well in a chrome window. Thus, we wrap this assignment in a helper function QU_setInnerHTML() so that when QU_ISCHROME is true, it uses a proper parser directly.

 

Altering innerHtml Content Fetching

Similar to the assignment problem of innerHTML, using it as a value does not work that will in the chrome context either, so we also wrap these in a helper function QU_getInnerHTML().

 

Pedantic QUnit Javascript (optional)

In a more pedantic mode, a number of JavaScript warning were generated around functions that did not always return a value and properties that were used but not defined.

 

Extension Development Environment Notes

I package up my extension with a bash script that conditionally includes testing and debug code based upon a command line parameter.  So all the QUnit code is not in the production build.  This requires a bit of a customized extension environment though one could just manually add and remove the QUnit code if you did not want to have to replicate conditional building. When included by the build script, the code also insert a menu option to allow invoking the unit test window.

 

XUL Window Notes

 

CSS Include

Need to have the CSS includes after the XML declaration, but before the window itself.

  <?xml-stylesheet href="chrome://myextension/skin/qunit-1.11.0.css"
          type="text/css"?>

 

Namespace include

The window XML object needs the attribute to ensure that HTML elements can be put inside it:

  xmlns:html="http://www.w3.org/1999/xhtml"

 

JavaScript include

Within the window XML, include the QUnit JavaScript.  Here I have renamed it to reflect that it is the version with the QU_ISCHROME=true setting:

  <script type="application/x-javascript"
          src="chrome://shophelper/content/qunit-chrome-1.11.0.js"/>

 

Body Elements

As with the normal QUnit usage, you need the special two "div" elements. The twist here is that the HTML tag names need to be preceded with "html:" to explicitly define their namespace.

   <html:div id="qunit"></html:div>
   <html:div id="qunit-fixture"></html:div>

 

Full Example XUL Window

<?xml version="1.0"?>

<?xml-stylesheet href="chrome://global/skin/" type="text/css"?>
<?xml-stylesheet href="chrome://myextension/skin/qunit-1.11.0.css" type="text/css"?>

<window id="MyExtensionUnitTestDialog"
        xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul"
        xmlns:html="http://www.w3.org/1999/xhtml"
       onload="SHUTD_onLoad(event);"
        ondialogcancel="return SHUTD_onDialogCancel();"
         title="My Extension Unit Test Console"
       buttons="cancel">

  <script type="application/x-javascript"
          src="chrome://myextension/content/qunit-chrome-1.11.0.js"/>
  <script type="application/x-javascript"
          src="chrome://myextension/content/unit-test-dialog.js"/>
 
  <vbox flex="1">
    <box>
     <html:div id="qunit"></html:div>
    </box>
    <box>
     <html:div id="qunit-fixture"></html:div>
    </box>
  </vbox>
</window>
 

XUL (extension) Javascript Notes

To test functions in the main extension within the QUnit XUl windo, we need access to the main window where the extension chrome code runs.  I am not sure if there are better ways to do this, but this works by iterating over windows.

    var parentWindow = null
    var wm = Components.classes["@mozilla.org/appshell/window-mediator;1"].getService(Components.interfaces.nsIWindowMediator);
    var enumerator = wm.getEnumerator("navigator:browser");
    while(enumerator.hasMoreElements())
    {
       var win = enumerator.getNext();
       if ( ! win.document )
          continue;
       parentWindow = win;
       break;
    }

You then can invoke functions from the extension with:

  parentWindow.someFunctionName( someArg1, someArg2 );

 

QUnit CSS Changes (optional)

Inside the chrome windows, the CSS property "-moz-border-radius" does not apply, so it gives a warning.

Ignorant Juries - By Design

I was reading this article about a judge prepping a jury for a patent infringement case between Apple and Samsung.  Given the horrendous state of the U.S. patent system, it seems to me that the more you understood about software patents, the more biased you would be and thus much less likely to ever be selected for this jury.  Additionally, the more you understood of the creative process, which really starts by copying (and then tweaking), the more biased you would be against design patents, Thus, it seems that the people most ignorant, and least qualified on the topic of patents are the ones deciding the verdict.

This made me realize that this phenomenon is probably not limited to software patents. In any field, the more informed you are about the topic, the more likely you would be labeled as biased toward a case.  Therefore, our judicial system would seem to be driven by those least qualified to make judgements.


To keep it in perspective though, what is likely a far worse problem is when attorneys choose jurors they think are the most malleable, rather than most qualified.  The judicial process then boils down to which attorney is the better salesperson and not whether there is actual guilt or innocence.

License Plate Sanity Restored

Stupid Design

About 3 years ago, Texas changed their license plate design to something that made them unreadable from more than 10 feet away.  I am sure they looked beautiful in a high-resolution, brightly lit setting to a committee of clueless overseers, but they failed to deliver on what should have been the number one requirement: readable at a distance in a variety of lighting conditions.

How does a group of people get put in charge of redesigning license plates and fail to deliver on the readability requirement?  How is is that at every step of the design and approval process no one ever raised this question? Did they ever even think to consult the number one user of the plates: law enforcement?  Would any police officer fail to spot this fatal design issue immediately?  This was an epic failure of government, committees and common sense.

It may not sound like it, but this is meant to be a positive, uplifting story. We come to that end by noting that Texas has corrected the problem and is (yet again) going to put out a new design, precisely to address the current problem of readability. And it took the Texas government only 3 years to recognize and fix the problem: maybe a new record.

This new design harken back to much older designs, and I like them a lot.  Simple, effective and functional.  Now I have to figure out how to trade in my plate for a new style one.

Sensible Design