Tag Archives: Scala

Leveraging annotation macros to generate caching boilerplate in Scala

There are only two hard things in Computer Science: cache invalidation and naming things.

— Phil Karlton

Since functional programming has reached the world of Serious Real Projects, a bunch of new buzzwords started to creep into the language of software engineers. Some of them are: monads, tail recursion, immutability, purity or currying. One of them is also memoization. This term defines the ability to cache results of calls for pure functions which don’t have side effects and always return the same result for given arguments. It is not something special, but we must be aware that with functional paradigms affecting our codebases with bigger and bigger impact, pure functions are becoming a very common thing. Programming languages and platforms themselves don’t offer too many elegant ways to apply memoization to such functions, so let’s see what options do we have with external libraries.

Memoization is just a kind of caching and there are a lot of ways to cache stuff. I’m not going to dive into aspect-oriented caching with magic annotations offered by some frameworks. Such approach is of course powerful and useful in some contextes, but let us focus on some simpler cases, where we don’t want to pull a framework.

Manual approach

One pretty straightforward method is to use the decorator pattern and wrap our function with a decorator object which stores values and manages the hideous process of cache invalidation. This painful job can be delegated to Guava with its slick API. First, let’s take a look at an example of some expensive function that might need caching:

class GraphBuilder {

  def creatGraph(elementCount: Int): Graph = {
    someExpensiveCode()
  }

}

To do our “manual AOP” and “weave in” the aspect of caching, let’s extract a trait and create a decorator:


trait GraphBuilder {
 def createGraph(elementCount: Int): Graph
}

class ExpensiveGraphBuilder extends GraphBuilder {

  override def creatGraph(elementCount: Int): Graph = {
   someExpensiveCode()
  }
}

class CachedGraphBuilder(inner: GraphBuilder) extends GraphBuilder {

  val graphs = CacheBuilder.newBuilder()
    .maximumSize(10000)
    .expireAfter(10, TimeUnit.MINUTES)
    .build(
       new CacheLoader[Int, Graph]() {
          def load(elementCount: Int) {
            inner.createGraph(elementCount);
          }
       })

  override def createGraph(elementCount: Int): Graph = {
    // hits the cache or calls the inner builder
    graphs.get(elementCount)
   }
}

Then, when we wire our dependencies, we can instantiate the builder as


val graphBuilder = new CachedGraphBuilder(new ExpensiveGraphBuilder())

It’s a decent approach which keeps our code compilant with the open-closed principle (if you don’t count trait extraction). However, some functions are part of traits or objects and cannot be decorated in such way, also it’s annoying to write the same boilerprate over and over again, so if you find yourself tired of it, it’s time to use a framework try something leaner.

Generating boilerplate in compilation time with macro annotations

MacMemo is a simple library that I implemented as an experiment. It introduces an annotation which can be placed over a function definition. When the compiler runs, it parses the annotation and generates boilerplate around function body to instantiate Guava cache and use it. Long story short, the whole above code becomes:


import com.softwaremill.macmemo.memoize

class GraphBuilder {

  @memoize(maxSize = 20000, expiresAfter = 2 hours)
  def creatGraph(elementCount: Int): Graph = {
    someExpensiveCode()
  }
}

We get a very brief solution to ensure that our function results get cached with desired details of invalidation. The macro will generate all necessary boilerplate code and insert it directly into the createGraph() method, wrapping its real code with cache calls.

If you like MacMemo, please star it on GitHub 🙂

Custom cache providers

What if you don’t want Guava? A clever recent contribution of Marcin Kubala extends MacMemo with possibility to define custom providers, so you can easily write your own extension and use memcached, EhCache or whatever you like.

You can achieve this by bringing appropriate  implicit MemoCacheBuilder instance into memoized class scope (the scope of the method definition, not it’s usage, e.g: companion object, implicit val within class or an explicit imports available in scope of class definition).

The MemoCacheBuilder trait has following definition:

case class MemoizeParams(maxSize: Long, expiresAfterMillis: Long, concurrencyLevel: Option[Int])

trait MemoCacheBuilder {
  def build[V <: Object](bucketId: String, params: MemoizeParams): Cache[V]
}

trait Cache[V] {
  def get(key: List[Any], computeValue: => V): V
}

MemoizeParams is a config class instantiated basing on parameters passed to the annotation. The buckedId argument can be used to identify unique key for your annotated method name + enclosing type path (the Guava-based implementation doesn’t use this key but you may find it handy in your implementation). Your MemoCacheBuilder should be responsible for creating an instance of Cache which can take a list of method arguments and return a cached result. This cache instance will be instantiated exactly once per each object of class with annotated method.

How exactly can you bring your builder into the right scope? Here’s an example of one simple way:

class ClassWithMemo {

  implicit val cacheProvider = new MyCustomCacheBuilder

  @memoize(2, 5 days)
  def someMethod(param: Int) = {
    someExpensiveCode()
  }
}

The builder can be as well represented as an object, so we won’t need the ‘new’ keyword here. For more, check Marcin’s examples on GitHub.

Last, but not the least, MacMemo allows to disable its caching globally with system property, so you can easily switch it off for test purposes.

Advertisements

Easy suite tagging with ScalaTest 2.0

If you are using ScalaTest 1.x and you need to tag some tests to make them easily skippable, you have to tag each method separately:

class MySpec extends FlatSpec with ShouldMatchers {

  it should "pass this exercise" taggedAs SlowTest in {
    // ...
  }

  it should "pass another exercise" taggedAs SlowTest in {
    // ...
  }
}

This approach has two major flaws:

  • It’s easy to forget about your tag.
  • If you use BeforeAndAfterAll, the code in beforeAll() will execute anyway. It’s possible that in slow tests this code will initialize some infrastructure (for example bring up the database) which is exactly the thing we want to avoid when we tag our tests.

One solution is to use nested suites and keep the beforeAll() initialization in the master suite. This requires some additional plumbing if we want to execute suites individually. However, with ScalaTest 2.0, we have a better option, allowing us to keep things simple and flexible: suite tagging.

In short, you can now annotate a whole suite, and then run tests with your annotation in exclusion rules. This will eliminate the whole suite with its beforeAll(), afterAll() and any other blocks surrounding method invocations.

How to prepare a tag

Just create a simple Java annotation like following.

package tags;

@org.scalatest.TagAnnotation
@Retention(RUNTIME)
@Target({METHOD, TYPE})
public @interface RequiresDb {
}

Remember – it has to be plain Java annotation. If you try to use Scala techniques like extending StaticAnnotation, then this will probably not work.
Now you can annotate your suite:

@RequiresDb
class MySpec extends FlatSpec with Matchers {
  // ...
}

That’s it, execute tests using this SBT command:

test-only * -- -l tags.RequiresDb

and it should do the trick. Note: use full class name, here it was tags.RequiresDb, where “tags” is just the Java package name.
Moreover, you can use a cool technique from this blog post and replace this pretty ugly incantation with simple

local:test

If you want a full working example, check my GitHub for one.

Dynamic queries in Rogue

I’ve spent some time googling for information about how to build Rogue queries dynamically but surprisingly I couldn’t find any straightforward answer. This is why I created following short post describing how to do it.

An usual query may look like this:

UserRecord where (_.surname eqs "Gates") and (_.age eqs 20) limit(5) fetch()

What if we want to add the our criteria conditionally? My first attempt looked naively like this:

It turns out you can’t compile this code, because type of queryByName is lost. The map().getOrElse() expression returns one of two different types. What you need here is to add ‘query‘ invocation to your record:

That’s pretty much it, somehow it is not easy to find in any examples. The queryByName object can now be used as a base for elegant, dynamic and typesafe Rogue query.  Also, note that orderDesc() and limit() are as well pulled to the first line. Special thanks to Piotr Buda, who showed me his queries in Slick which inspired me to go this way.

Will Node and Scala really dry up?

There’s a quite significant rise of buzz around Google’s Go programming language. Some may think that it’s just another peak of excitement coming after a period of calm because the Scala + Akka / Node plateau is over. Derek Collison says:  “The management layers and infrastructure layers of the newer technologies that provide this cloud delivery model? Within two years, a majority will be written in Go.”. Finally, we have this very interesting talk by Paul Dix, entitled bravely “Why Node and Scala will dry up: Go will drink their milkshake”. I strongly recommend watching this video, which was actually the reason why I decided to write my blog post. As a big enthusiast of Scala I want to address some of Paul’s concerns, so let’s take a look at his “allegations” (but remember to watch the talk first!):

Node.js

I don’t have experience with writing backend using Node, but I code in JS and Paul’s arguments look pretty convincing to me. The performance vs coding in javascript tradeoff just doesn’t appeal to me for most cases. If I was about to write a super-performant mission-critical system then maybe I’d take a chance to investigate it deeper. Since then, EOT.

Dependencies (language / library versions)

Paul mentions that this is the thing that hurts him most in Scala. Well, I can agree that it is a bit painful and dealing with compatibility takes time and effort, sometimes we just got lead into a dead end. On the other side we have another extreme of Java where high focus on backward compatibility results in really crappy debt and yet it still has lots of quirks when it comes to deal with versions. Maybe it’s wishful thinking but I suppose that aggressive strategy of version incompatibility in Scala will pay off in long term. Early adopters have to pay the price but I hope that reward will be worthy.

No centralized home for libraries

That was never a big deal for me, neither with Scala + SBT nor with Java + Maven. The notion of repository is a decent standard and looking up the right repo to get our libraries is usually very easy. I had a few struggles with finding location of some exotic libs in the Internet but it never took much time.

Option sucks

I guess that this part of video was the moment when I decided to write my blog post 😉 Paul states that using Option requires vague and verbose handling which is pretty much comparable to handling nulls. Well, the main difference is that Option makes implicit explicit and brings null handling to a much safer level. Of course, you can always call .get() and hit a NPE but the difference is that you explicitly ignore a warning. It should be obvious since Tony Hoare explained why he calls null reference “the billion dollar mistake”. As for using map to safely work on wrapped values: it’s not Scala’s idiomatic way. Take Maybe in Haskell. However, I can agree that it may be more elegant to deal with in terms of style. Cedric shows a cool approach used in Fantom in this blog post.

Method invocation without dots or parens needs to die in a fire

Okay, I can agree with that 🙂 There are some exceptions though, like test frameworks that give a neat DSLs using this capability, but in typical codebase it’s just confusing, especially if you browse code written by different people.

Pattern matching

As for using pattern matching with Option, I already pointed out that it may look a bit clumsy. However, using pattern matching with extractors and case classes has much more power than using if/else. Some neat examples can be found on Martin Odersky’s course at Coursera.

IO

A very good point. I was also disappointed that such a mature language as Scala requires to browse many resources over the Internet to find a decent third-party library for filesystem manipulation or network operations.

Too many concurrency options

My experience here is still weak but as all the community buzz around new Akka that I observe (especially on Twitter) always made me think that there are no other options for most cases. However, feel welcome to comment if I’m wrong 🙂

Tuples are an abomination

I agree. I always feel uncomfortable when working with tuples, the whole _.1, _.2 stuff is just hard to read in many cases. It’s often a feeling of hitting a wall when I read concise, elegant Scala code and run into a _.1. What was that? Have to go back and check again. And then bang! _.2. What was that? The whole flow of pleasant code reading gets disturbed.

Language footprint is massive

Unfortunately yes. I often get a feeling that Scala is overwhelming with too much stuff to learn, know and follow. Martin Odersky even proposed to split Scala into different language levels to make it easier but it started a controversial discussion (Sigh… I can’t find the source. Was it on Stack Overflow? Please post a comment if you know it).

Go

I won’t comment too much on this language, because all I know now is stuff I learned from Paul’s talk and another cool introductory talk by Konrad given recently in SoftwareMill (in Polish). Just a few first impressions:

  • Controversial approach to versioning and dependencies.
  • Statements are not expressions. I got really used to expressions, imperative code looks a little stiff to me now 🙂
  • Goroutines and channels: +1, quite interesting concepts
  • Garbage collection: may be risky if it’s really too simple?
  • Compilation to native code: +1 for deployment simplicity. Cool for micro services which are actually a pretty strong trend recently.
  • No inheritance: very good!
  • Syntax: a bit too C-ish

Nature abhors a vacuum

We may observe an interesting progress of adaptation of Go language. Let’s see what future brings and never stop learning.

Using Swagger with Scalatra

If you’re considering Scalatra for your web services, you probably should check out Swagger as a support library. One of its coolest features is possibility to automatically create interactive API docs which you can open in your browser and exercise. The official documentation shows a way to describe your services but it seems that new version offers better way to do that, with safe type references. Let’s take a look at an example to make it clearer (full working app is available on GitHub). This short tutorial does not focus on Scalatra itself and I assume that you have some knowledge on how to configure and use it. We are going to explore the subject of integration with Swagger. Okay, fine, here we go.

Project setup

Our example project is prepared for Scala 2.10 with Scalatra 2.2. Assuming we have a working web app build configuration, we now need to add new dependencies:

“org.scalatra” %% “scalatra-swagger” % “2.2.0”
“com.wordnik” % “swagger-core_2.10.0” % “1.2.0”

Now we have to define a main servlet for Swagger. This servlet will provide our documentation as a service. All other servlet that need to be exposed are going to use this one:

Exposing the browser app

To make your documentation available for exploration and execution via browser, add api-docs web resources to the application. They contain some javascript calling the Swagger servlet and styles, which you can plasy with to customize look and feel of your documentation.

Defining API

Now we can document a simple service. Let’s say we have a servlet consuming simple GET requests with optional query parameter:

It parses a query paramter named “type” and returns an object defined by simple case class ExampleItemList which can be easily transformed to JSON. Now what is this additional parameter defined as operation()? It’s a special definition of API operation defined in additional trait (extending SwaggerSupport):

It’s cool that we can use strong Scala typing to define both parameter and response types. Swagger will manage to expand our custom type (ExampleItemList) and prepare a nice documentation (with an example!).

If you browse http://localhost:8080/api-docs/ now, you can see example endpoint definition:

swagger-api-01

Below, you can fill out a form and send a test request. If you describe a POST and your request body is also going to be represented as a case class, you are given a template to fill, neat!

swagger-api-02

Here’s all the code that you need to make Swagger generate above form for you:

Shortcomings

Swagger has some problems with non-simple class serialization. For example, it cannot handle org.joda.time.DateTime. If you get some crazy exceptions during deployment with vague message, this may be the case.

I hope this short post clarified a bit how can you use Swagger to generate your ‘living documentation’ and utility forms to test services. Hopefully the official Scalatra / Swagger docs get more organized and up-to-date soon.
Special thanks to Michał Ostruszka for discovering all this stuff and sharing it with me!

Functional Decorator

Recently I’ve been experimenting with functional programming and Scala. As a developer used to Object Oriented paradigm I was wondering what are the functional equivalents of popular design patterns. One of the best articles summarizing this subject is Mark Seeman’s “Patterns Across Paradigms”. I am currently working on a small project where I had a chance to implement the Decorator Pattern using functional constructs in Scala.

The Object-Oriented approach

My example is be based on “DDD CQRS Leaven”, a project presenting some system and domain modeling concepts. This application was originally created by Sławek Sobótka and Rafał Jamróz, you can browse the codebase on Github. Here we are going to focus only on a small part of the domain, the  Rebate Policy. It’s a simple representation of Strategy Pattern, responsible for calculating eventual rebates for products in an online store. The model of policies can be described in few boxes:

Dead simple so far, right? We have our RebatePolicy contract with a couple of implementations. Now let’s see how it looks like when we add a Decorator:

Here’s the implementation of these components:

Such design allows combining different Domain Policies (Strategies) in a flexible way to obtain object which still matches the RebatePolicy interface and represents the composition. Decorator pattern allows adding new policies and creating various combinations in runtime without modifying existing ones or the “client code”, which keeps using the original abstraction. Neat.

Functional implementation

Trying to achieve similar goals using functional code requires reminding that the GoF Decorator Pattern is, in fact, a supplementary construct required to compensate the shortcomings of typical OO languages. In functional world we can leverage currying and functions as first-class citizens to get same desired effect. Before we explore the functional implementation in Scala, take a look at the RebateDecorator class. It represents an abstract base for all rebates which can wrap other rebates. A RebateDecorator forces our rebate to pass some inner rebate object in the constructor and provides it as a protected member for further use by the inheriting class. Then, the VipRebate class allows creating an instance in two ways: either with some decorated member or without it. Let’s do something similar with functions in Scala:

As you can see, the RebatePolicy type is now a functional type, which means that we speak more directly of our contract: A RebatePolicy is a function which takes a Product, quantity and minimumPrice and returns rebate value of type Money. Standard policy produces a function fulfilling this contract by calculating the rebate with some simple algorithm. What about VipRebate? It’s also a function, but a bit more complex 🙂 In fact, the VipRebate represents kind of a Factory (yes! Another pattern that we get for free!) which allows creating new function of type RebatePolicy with additional parameters: two Money values and innerPolicy. The Option type in Scala gives a way to initialize the rebate with “none” inner policy which is much more elegant than null manipulation that we saw before. Our goal has been achieved with some additional bonuses:

  1. Much more concise. The baroque entourage of Java has been strongly reduced to what’s essential. No more unnecessary ceremony, while the readability and comprehensibility are still high (or even higher). The superfluous RebateDecorator class is no longer needed and it doesn’t “pollute” our real domain logic anymore.
  2. Flexibility. Thanks to currying we have free “functional dependency injection” capabilities. The VipRebate signature allows creating final policy in three steps: first with initial parameters (minimalThreshold, rebateValue). Such call would produce another function which we can pass around and eventually call with optional inner policy argument. This second call will finally produce a RebatePolicy ready to use whenever it is needed. In previous, objective approach we were forced to build our object with all the dependencies right away (with constructor). To achieve more flexibility we would require some setters which breaks immutability and stinks 😉

Consequences

Exploring the world of functional programming is addictive and changes your mindset forever. If you are interested in further learning then you should definitely check the free “Functional programming principles” course on Coursera. First edition just ended but next one will launch probably around spring and you will be more than satisfied 🙂

Scala in Depth: Review

First of all, I would like to thank Manning Publications for providing me a copy of “Scala in Depth”. To keep the promise I present you my personal review which is, at the same time, an opening post for my blog.

Although Scala has been present in the world of programming for quite a long time (~8 years), it is last two years when its popularity has grown extremely rapidly. Recently most Java conferences offered  great talks about it, lots of developers got interested (including me) but there was one major inconvenience: insufficiency of fresh books describing the deep details of language features. There are, of  course, lots of blogs and tutorials presenting Scala with examples. There are also a few decent books (listed on scala-lang official website), but most of them intend only to introduce the language and present its standard appliances. What I have been waiting for impatiently was a descriptive book about advanced features of Scala with additional explanations of mechanism that “just worked” so far.

I must admit that “Scala in Depth” definitely exceeded my expectations. Joshua D. Suereth took the word “Depth” really seriously and prepared a very comprehensive reference of different advanced  aspects of Scala. The book is written in a consistent and expressive form. All the chapters are well structured, describing exactly what they should. The set of selected subjects fits well, nothing is missing or neglected, nothing is forced or unnecessary. Everything right where it belongs. Additionally, there are many examples of bytecode output emitted by the compiler, so even the most inquisitive developers should be fully satisfied.

First four chapters are a strong reference of best patterns and rules for developers who think of building some production code with Scala. It is a decent set of hints and practices to avoid habits from other languages and grip a better understanding of language principles. The most important concepts including functional style (functions, expressions, immutability), Option type, coding conventions and many others are well described to to warm you up before pretty heavy content in the following chapters.

Next three chapters (5, 6, 7) explore the subjects of Scala type system and implicits with surgical precision. I wouldn’t call it shame when I confess that it was very difficult to grab all these details and understand them without confusion. However, I cannot blame the book for being vague or chaotic, the explanations are very well structured and descriptive, the only problem was that I never expected the world of implicits and Scala type system to be so rich. If I should point any flaw, it might be probably the selection of examples. The low-level nature of all advanced appliances suggest that there are not so many “day to day” problems which can be solved with these techniques. They seem to be very well tailored for building frameworks or DSLs, but this subject was clearly covered in “DSLs in Action”, which I recommend as a must for any Scala fan (or, actually, any software developer who doesn’t want to be  a subject of future version of COBOL jokes). I believe that I will go back to check chapters 5, 6 and 7 many more times before I get really comfortable with these subjects, it’s good to have such a precise reference though.

Chapter eight brings a moment’s grace, as the matter of collections is a pretty standard and familiar one after reading any previous Scala book. Fortunately, the author  found many interesting additional aspects, like mixin synchronization, views or paralell collections, which keeps reader’s focus sharp in order to stay on track. I find this particular chapter a little bit too short, maybe it could be enriched with some more examples in future editions.

Twenty pages about Actors may also seem a little disappointing, but in my opinion chapter 9 shouldn’t be any longer. It extends all the introductory information which can be found in other books. Handling failures and scheduling  are sufficiently advanced for most cases, and the chapter ends exactly in a point where the reader realizes that actors and Akka deserve their whole separate book.

Chapter ten settles with the myth that integrating Java and Scala is a piece of cake. There are various nuances that need to be discussed and the author did a perfect job bringing them all out. This chapter explains well what are the best ways to create Java code for integration with Scala, providing a portion of bytecode compiler output for anyone who would doubt in given suggestions. There are also pretty good descriptions of appliances of Scala implicits to integrate with Java with maximum comfort. Chapter ten also deals with some cumbersome topics like serialization or annotations which complete a very decent list of Java/Scala integration issues.

A yummy cake wouldn’t be complete without a cherry on its top, which is the last chapter: “Patterns in functional programming”. As chapters 5, 6 and 7, this one requires maximum focus, let “Monads” be the word to prove that. By the way, Manning announces a new title: “Functional Programming in Scala” planned for release around spring 2013. I think that chapter 11 may serve as an introduction to this whole new purely functional world, but currently I could not get fascinated by it. This  is the point where I want to remind the main flaw of this book, which is, again, lack of strong and palpable examples. Or maybe my spirit is just not ready for  a new religion. Anyway, this chapter could be seen as pretty “hardcore” for Plain Old Object-Oriented Developers (like me) but it certainly does a decent job of discussing the fundamentals of functional patterns.

To sum up, “Scala in Depth” is currently absolutely the most detailed and comprehensive book about this exciting and promising language. Its appearance on the market suggests that Scala is becoming a serious player and that movement towards new paradigms is real.  If you need to get convinced that Scala is worth learning, spare it for later and check out some other titles first (for example, the somehow-still-in-MEAP “Scala in Action”). If you already know the basics of Scala and need answers for all the intriguing questions you have been gathering so far, you will be more than positively surprised.

Oh, and if you’re not convinced you might want to check out this other review: http://myelucubrations.blogspot.co.uk/2012/06/scala-in-depth.html.Brief and substantial.