The Loop

I was recently asked how to stop Lift’s Mapper from flattening the naming of columns during its “schemification” process. Well, there is a little known configuration object in the net.liftweb.mapper package called MapperRules.

MapperRules contains a whole bunch of stuff that you can use to control the way that Mapper names stuff that it creates and how it will try and inspect your database. If you prefer (like me) to have columns and tables like first_name rather than firstname, then add the following to Boot.scala:

class Boot {
  def boot {
    ...
    MapperRules.tableName =  (_,name) => StringHelpers.snakify(name)
    MapperRules.columnName = (_,name) => StringHelpers.snakify(name)
    ...  
  } 
}

Thats it!

So I have been pretty lax of late with keeping this blog up to date. There is a very good reason for that, and that is I am currently working on a book about Lift – If you haven’t heard, Lift In Action will be available on Manning Publications in Q1 of 2011 and is currently available as an eBook via the Manning website. Its generally eating a lot of my time right now so I haven’t had a chance to post about all the cool new stuff that is making its way into Lift every week – hopefully I will soon get time to write about the awesome new Record persistence system that we are all very excited about as we are currently integrating the extremely awesome Squeryl as a backend for dealing with RDBMS.

By all means, go and checkout the book and let me know what you think as it develops in the author online forums and I will get back to you ASAP. Enjoy!

Continuing with my series of articles about Lift, here’s another hidden gem that not many people know about. Lift can automatically validate your markup so that any validation errors or warnings are high lighted to you as part of the development process. Pretty neat. Here’s what you need to do to enable it:

Configure Boot

  class Boot {
    def boot {
    ......   
      LiftRules.xhtmlValidator = Full(StrictXHTML1_0Validator)
    }
  }

Where of course you can replace StrictXHTML1_0Validator with any one of:

  TransitionalXHTML1_0Validator
  StrictXHTML1_0Validator

  // or subclass:
  GenericValidtor
  // to provide your own custom implementation 

Find this and more great tips in my forthcoming book, Lift in Action

Enjoy!

So im at Scala Days 2010 right now, and during dinner last night their seemed to be some misconceptions about what Lift’s Box[T] monad actually does and how it differs from Option[T]. Now, im not computer scientist, so im not going to be discussing the algebraic validity of Box[T], not shall I be calling it a “tri state option”...

Why bother boxing values?

Have you ever thrown a NullPointerException? Frankly, I don’t know any developers that haven’t had this happen to them at some point because of something they hadn’t considered during development… in short, a NPE is something that was caused by an unexpected series of events causing your application to explode in a variety of ways. This is not good.

Consider a scenario where I want to do something with a value returned from a database; its not uncommon to see programs where one assumes the database query got the correct result, then do some operation on it. Well, news flash, its highly plausible that the database query might not go as you had expected and return some other result….

...Boxes to the rescue! As the name might suggest, a Box is something that you can put stuff in, take stuff out of, and also find empty. Much as you would do with a real life box. Lets illustrate with a simple example:

import net.liftweb.common.{Box,Full,Empty,Failure,ParamFailure}

scala> val x: Box[String] = Empty
x: net.liftweb.common.Box[String] = Empty

scala> val y = Full("Thing")
y: net.liftweb.common.Full[java.lang.String] = Full(Thing)

We have two simple examples here that assign new boxes to vals, and as you can see, x is assigned as a Box[String], yet its actual value is Empty which is a sub type of Box. The second assignment to y has the same type signature, but this time it is “Full” with a value; in this instance “Thing”.

What about exceptions?

Lift has a helper method in net.liftweb.util.Helpers called “tryo” that is a specilized control structure so that if you can execute code in a block that returns a Box’ed value irrespective of what you did in the block and how its execution went. If your using Lift webkit then you already have access to these utility methods, however if you just want to stay light and are not using webkit then the definition of tryo looks like:

  def tryo[T](
      ignore: List[Class[_]], 
      onError: Box[Throwable => Unit])
      (f: => T): Box[T] = {
    try {
      Full(f)
    } catch {
      case c if ignore.exists(_.isAssignableFrom(c.getClass)) => 
        onError.foreach(_(c)); Empty
      case c if (ignore == null || ignore.isEmpty) => 
        onError.foreach(_(c)); Failure(c.getMessage, Full(c), Empty)
    }
  }

There are some other overloads for syntax sugar, but essentially it lets you do:

  tryo {
    // your code here
  }

So lets assume we had some remote API to invoke, and it could not connect, or blow up in some way you hadn’t planned, what would happy given a tryo block / wrap? Consider:

scala> tryo("www".toInt) 
res4: net.liftweb.common.Box[Int] = Failure(
  For input string: "www",
  Full(java.lang.NumberFormatException: For input string: "www"),
  Empty)

As “www” isnt an Int, it unsurprisingly cannot be converted to one, so it blows up with java.lang.NumberFormatException. However, using tryo (boxed values) we are left with a special Box subtype called Failure. This lets us handle the error in a concise way rather than needing to check all the possible outcomes or worry about some awful try-catch block.

scala> tryo("www".toInt) openOr 1234                       
res7: Int = 1234

scala> tryo("www".toInt).map(_.toString).openOr("Invalid Strings")
res8: java.lang.String = Invalid Strings

scala> for(x <- tryo("www".toInt)) yield x
res11: net.liftweb.common.Box[Int] = Failure(
  For input string: "www",
  Full(java.lang.NumberFormatException: For input string: "www"),
  Empty)

So you can see there several ways to interact with Box, Full and Failure subtypes, providing defaults inline and mapping the results etc.

Handling empty values?

But what if we need more information or the value we are looking for isnt a failure, its just the value is Empty (or None for scala Option)? For example, lets assume we were looking for a request parameter to a REST API or similar… rather than throwing a HTTP 500 error with no reason, it would be nice to give the user a much more granular reason. Consider getting a request paramater using Lift’s S.param method.

scala> S.param("id") ?~ "You must supply an ID parameter" 
res17: net.liftweb.common.Failure = 
  Failure(You must supply an ID parameter,Empty,Empty)

The ?~ method allows you to supply an error message and convert the Empty into a Failure. This is an incredibly helpful paradigm when used with for comprehensions.

Chaining boxes

Lets assume that we have a situation where by we could recive a value from several places, or we need to “fall through” to a specific result based on trying several values in a sequence… Box supports this by way of the “or” operand.

scala> val x = Empty
x: net.liftweb.common.Empty.type = Empty

scala> val y = tryo("qqq".toInt) 
y: net.liftweb.common.Box[Int] = 
  Failure(For input string: "qqq",
  Full(java.lang.NumberFormatException: For input string: "qqq"),
  Empty)

scala> x or y or Full("Default")
res18: net.liftweb.common.Box[Any] = Full(Default)

Box has a bunch of features I haven’t covered here, but I hope this helps you understand the rational of this specialised option-esq type.

Note: REST and many other topics discussed within my upcomming book Lift in Action, more information will shortly be published on manning.com

This one is hot off the press – Lift 2.0-SNAPSHOT now has the ability to do notification style service workflows – essentially comet for REST.

So how does this work? Well, following on from my other post on REST services with Lift we are still using a DispatchPF to setup the service, but we now use the statefull dispatcher LiftRules.dispatch rather than statelessDispatchTable and from inside our resource definition we call S.respondAsync. Here’s a complete example:

  LiftRules.dispatch.append {
    case Req("example" :: rest, _, _) => {
      S.respondAsync {
        Thread.sleep(20000) // some computation here
        Full(XmlResponse(<async>response</async>))
      }
    }
  }

The call to S.respondAsync executes the function you pass to it on another thread and if the container you are using supports suspend / resume idiom Lift will pass control to the container and no threads are blocked during its execution; just return a Box[LiftResponse] like normal and Lift will handle notification to listening clients. Sweet.

Note: This subject is discussed in much greater detail within my upcomming book Lift in Action, more information will shortly be published on manning.com

The Goal

So what are we actually trying to achieve here? We’ll use the twitter.com API as an example as its a good illustration in this instance. Consider the following URLs:

http://api.twitter.com/users/timperrett.xml

and

http://api.twitter.com/users/timperrett.json

Its the exact same content, but just presented in different formats. This is very nice, and also very helpful as often you might be in a situation where you need to syndicate a single API from multiple devices or platforms where toolkits may be limited. For example, working with JSON is a lot easier from Java script than it is XML (the latter is of course possible its just more verbose). In the examples above, Twitter are using the format component of the URL following the ”.” to determine the content type the user wants – this is of course one route, and some services may want to add the additional stipulation of a specified accepts and content-type headers in the requests so the server is sure its serving the right content.

The interesting problem here is that if you have the same content, you really only want one point of access for the logic. You simply dont want to be maintaining two set of logic – only the actual code that handles differences in presentation should alter between calls right?

Implementation Pattern

Typically, when I want to create web services in my Lift application I break out my dispatchers into separate objects related to their concerns. This way it keeps things nice and tidy. So, in our example, i’ll be wiring up the following services:

  LiftRules.statelessDispatchTable.append {
    case Req("apps" :: Nil, "json", GetRequest) => 
      ApplicationServices.json.list
    case Req("apps" :: Nil, "xml", GetRequest) => 
      ApplicationServices.xml.list
  }

For those not familiar with Lift’s dispatching mechanism this will expose the following URLs:

• /apps.json
• /apps.xml

All pretty simple so far. As you can see from the code listing above, I have broken my application services out into a singleton object called ApplicationServices which has fields for each content type im going to be presenting. Without further ado lets take a look at the definition of the apps listing service:

object ApplicationServices extends Loggable {
  .....
  protected def _list(mediaAction: List[String] => LiftResponse) =
    () => tryo(mediaAction(Applications.list.map(_.name)))
  .....
}

Ok so this is pretty cool. Im using function passing to let each subsequent definition define how it builds the LiftResponse sub-type; in this instance that’ll be JsonResponse for JSON and XmlResponse for XML.

JSON Listing

def list = _list((apps) => 
  JsonResponse(compact(JsonAST.render(("applications" -> apps)))))

XML Listing

def list = _list((apps) => XmlResponse({
  <applications>
    {apps.map(a => <application name={a} />)}
  </applications>
}))

So you can see that each presentation version is only dealing with how it needs to create the response… its not re-computing the list of applications or such.

Syntax Sugar

Within the overall definition I define the formats as inner-objects and assign them to public fields in order to achieve the nice ApplicationServices.xml.list notation:

object ApplicationServices extends Loggable {
  val json = Json
  val xml = Xml

  protected def _list(mediaAction: List[String] => LiftResponse) =
    () => tryo(mediaAction(Applications.list.map(_.name)))

  object Json {
    // definitions here
  }

  object Xml {
    // definitions here
  }
}

Round Up

Thats pretty much all there is too it. You can leverage Scala’s extremely powerful features to create your web services and keep code LOC to a minimum. Over and out.

In my continuing effort to improve the wider knowledge of Lift, here’s another great nugget that shows you how to provide a 404 handler powered from a Lift template

class Boot {
  def boot {
    // ...other stuff...

    LiftRules.uriNotFound.prepend(NamedPF("404handler"){
      case (req,failure) => 
       NotFoundAsTemplate(ParsePath(List("404"),"html",false,false))
    })

  }
}

And I have a 404.html file in the ./webapp/404.html with the contents:

<lift:surround with="default" at="content">
  <h2>Ooops!</h2>
  <p>Wow, we totally couldn't find your page. Sorry about that.</p>
</lift:surround>

Then whenever a user browses to a URL that is not supported by your application, you can give them a friendly (and nicely designed) webpage rather than some technical message about 404s.

Note: The only cavet is not to use the <lift:menu /> snippet within your 404 template because Lift is unable to calculate the SiteMap when its a URL that is not accounted for within that scheme.

UPDATE The above note is now redundant because of commit by Marius on issue 376

Recently, Marius added an abstraction layer within Lift so that we could support different comet implementations over different servlet containers. Out of the box, we provide two both Jetty 6 and Jetty 7 APIs. This API is however not very well documented so I thought I would just write a short post on how to use it. In your Boot.scala file you need to put something like:

import net.liftweb.http._
import provider.servlet.containers.Jetty7AsyncProvider

class Boot {
  def boot {
    // ...other stuff here...

    LiftRules.servletAsyncProvider = (req) => 
        new Jetty7AsyncProvider(req)

  }
}

Thats pretty much all you need to do in order to swap comet implementations! If you want to use something else, like Resin etc, all you need to is write the correct provider and away you go.

Its quite often that I need to pass tokens between Scala (Java also) and .NET – this instantly provides a whole raft of issues because of how the algorithms are built under the hood in the respective platforms. This short post details how you can generate consistent base64 hashes over both platforms (this can be helpful for security purposes etc)

The Scala way:

  val input = "sample" 
  val encoding = "UTF-8" 

  base64Encode(
    hexDigest(
      input.getBytes(encoding)
    ).getBytes(encoding)
  )

The .NET way

  using System.Security.Cryptography;

  string input = "sample";

  SHA1CryptoServiceProvider shaHasher = 
       new SHA1CryptoServiceProvider();
  byte[] data = shaHasher.ComputeHash(
    Encoding.UTF8.GetBytes(input)
  );

  StringBuilder sBuilder = new StringBuilder();
  for (int i = 0; i < data.Length; i++){
    sBuilder.Append(data[i].ToString("x2"));
  }

  string hash = sBuilder.ToString();

  Convert.ToBase64String(
    Encoding.UTF8.GetBytes(hash)
  );

UPDATE After I posted this code origionally, it was pointed out that it would be good / helpful to show encryption as well as hashing; so here goes:

The Scala way:

  val key: Array[Byte] = "qwehfjgtufkgurifghfkdjfg".getBytes("UTF-8")
  tripleDESEncrypt("sample", key)

The .NET way:

  using System.Security.Cryptography;

  string key = "qwehfjgtufkgurifghfkdjfg";
  string toEnc = "sample";

  TripleDESCryptoServiceProvider DES = 
       new TripleDESCryptoServiceProvider();

  byte[] bytKey = Encoding.UTF8.GetBytes(key);

  DES.Mode = CipherMode.ECB;
  DES.Key = bytKey;
  DES.Padding = PaddingMode.PKCS7;

  ICryptoTransform DESEncrypt = DES.CreateEncryptor();

  byte[] buffer = Encoding.UTF8.GetBytes(toEnc);

  string base64string = Convert.ToBase64String(
    DESEncrypt.TransformFinalBlock(buffer, 0, buffer.Length)
  );

  Response.Write(base64string);

Maybe this will help someone avoid the pain I have endured with this! lol.

Picture the scene: Im sat in my chair, working away and then I get a fairly crazed phone call from a co-worker in sales – he apparently has this stunning new idea about how to leverage twitter as a marketing platform…

As most of my readers are technical (yes, thats you there) I wont labour the marketing peice, but if your interested, David wrote a really nice article explaining his concept about what happens when social media meets cross-media; find it over here

A cunning scheme indeed

So, from the off, this was going to be an interesting campaign to put together because there were numerous issues, including:

• Lots of concurrent (and asynchronous) read / write ops to Twitter API
• Backend communication to XMPie cluster
• Push production status back down to the browser
• Downloading of avatar images automatically from twitter and insert into XMPie asset source
• Doing any of this in the given time frame!?

This project was going to have a sizeable set of works attached no doubt to manage the interplay between XMPie systems and the Twitter API.

Technology Sandwich

To implement this campaign, I ended up using a whole bunch of different technologies… The implementing stack eventually ended up utilising the following:

• Lift
• Dispatch
• Scala Actors
• XMPie uProduce
• XMPie ICP
• Microsoft SQL Server

In earlier drafts of this article I tried to explain the interplay between components, but simply could not find a good way to articulate the meaning. To that end, I hope diagram gets the structure across sufficiently!:

As you can see – quite a number of software components here! I’ve been working with Scala and XMPie uProduce for quite a while now and have built up some very nice abstractions that completely hide calling of services under the hood.

This entire campaign was implemented based on top of ICP, using my Scala abstraction along with the awesome power of Scala Actors, scheduling and some generally very cool libraries like Dispatch

Questions

A few people asked me to write this post, but as im not sure what specifically you might have been interested in, if you want to know more about how its put together, what does what etc then I would gladly take questions! Just leave a comment at the bottom and i’ll get back to you…