RubyNetz

Let me start this post by quoting something smart.

“REST [is an] architectural style for distributed hypermedia systems”

That actually is the first line, chapter 5 of Roy Fielding’s thesis about REST. I gave a couple of talks last year about hypermedia and the misunderstandings in REST in the Rails world and this post tries to sum up with that by focusing on the hypermedia aspect of this architectural concept.

Challenge: Make Fruit Salad!

In the last posts we discussed using the Roar gem for designing and implementing REST systems in Ruby. We were using fruits and bowls as an exemplary domain – let’s keep up with that and fix a fruit salad today.

As a first step it makes sense getting a bowl for all the ingredients. In our REST system, we just POST to the bowls URL.

POST http://bowls
Content-type: application/json
------------------------------
{"location": "desk"}

The request body contains a minimal document specifying the location for the created bowl. Here’s what we get back.

{"location":"desk",
 "fruits":[],
 "links":[
  {"rel":"self",  "href":"http://bowls/desk"},
  {"rel":"fruits","href":"http://bowls/desk/fruits"}
]}

By reading this I figure out that the bowl on my desk was created, but doesn’t contain any fruits, yet. Also, these “links” seem understandable, while self might point to the bowl resource itself the fruits link probably directs to a collection resource listing all the ingredients in that very bowl.

Looks as if a URL gets a “meaning” by specifying a rel attribute with it.

I infered all that by intuition, as JSON is a pretty simple representation format. And this is the first problem when trying to understand hypermedia. JSON as it doesn’t include any semantics, the links are, well, useless as they’re not understandable to machines. What we need is a hyper-media type. Get it on!

What Is A Media Type, Sir?

A concept called media type gives both syntax and semantics to a document. It is important to understand that a simple media type like JSON doesn’t define anything. It just provides a medium (or format) to encode information.

Why not put it that way: JSON and XML are like primitive communication mechanics. Where XML uses smoke signals for exchanging messages, JSON is human voice. A poor example.

However, listening to my voice doesn’t mean you understand what I’m saying. Reading JSON is like listening to my sounds, but the noize might not make sense to you. Ok?

The media type is like a language. You understanding the media type “english” and me speaking (JSON) in english (media type) makes us happy chatters. And suddenly, a series of sounds turn into “Would you like another beer?”. You understand me since you know the syntax and the semantics of my sounds. That is a media type!

Now That We Understand Hypermedia, What To Do With It?

Instead of using plain, dumb JSON I’d like to introduce a real hypermedia format called HAL which is “specifically for exposing RESTful hypermedia APIs”.

POST http://bowls
Content-type: application/hal+json
------------------------------
{"location": "desk"}

Note that we used the application/hal+json content type desparately hoping the web service understands this media format. Check the response.

{"location":"desk",
 "_embedded":{"fruits":[]},
 "_links":{
  "self":  {"href":"http://bowls/desk"},
  "fruits":{"href":"http://bowls/desk/fruits"}
}}

It obviously does, and now the links are encoded a little bit different, following the HAL standard. While this doesn’t make a real difference for you human being, it changes a lot in machine context. By using this media format, we added semantics: Now, the HAL-aware client knows that links can be found at the _links key, that they’re keyed by rel and so on.

Think about it briefly. Growing up with human voice you understood JSON before. However, now that you learned HAL, you understand all the bits and pieces in the document just by applying the specification rules.

I’m Still Hungry!

Sorry for that detailed discussion about hypermedia types, I’m really getting hungry as well, so let’s add some fruits to the bowl.

POST http://bowls/desk/fruits
Content-type: application/hal+json
------------------------------
{"title": "Apple"}

an apple a day keeps the doctor away. Assuming it is sufficient to post this mini JSON document to the fruits URL I add an apple to my salad. Did that work?

GET http://bowls/desk
Content-type: application/hal+json
------------------------------
{"location":"desk",
 "_embedded":{
  "fruits":[
   {"title":"Apple",
    "colors":[],
    "_links":{"self":{"href":"http://fruits/apple"}}
  }]  
 "_links":{
  "self":  {"href":"http://bowls/desk"},
  "fruits":{"href":"http://bowls/desk/fruits"}}
}

Yeah! I can see an apple inside the _embedded key, which is HAL’s way to nest resources in representations. Also, the apple contains a link again pointing to its resource.

It is needless to note that I could go on like this and POST more fruits to the fruits collection resource in order to enrich my delicious salad.

What Did I Just Do?

This brief example was brief. Nevertheless, it demonstrated the use of hypermedia. Here’s what I did.

1. First, I POSTed to http://bowls to create a bowl.
2. The returned document points me to its fruits resource.
3. To add fruits, I POST to http://bowls/desk/fruits as defined in the representation.
4. I retrieve the updated bowl by following the self URL of the bowl.

Well, what’s so special about that?

The key is, I didn’t compute any URL except the initial URL in (1.) – this is what we call the single entry point. No knowledge about follow-up URLs was needed to operate my fruit salad API. Everything I need to know is included in the representations.

Actions (aka URLs) that are meaningful in the current application state can be found in the document without any need to build URLs myself. This concept is called HATEOAS – a beautiful name… for a cat.

Thou Shall Not Make Any Fruit Salad Without Hypermedia!

Roy Fielding, the inventor of REST, states that your API is RESTFUL if and only if it is hypermedia-driven! In my words, that’ll mean no URL code should be hard-wired into your REST client – except for the single entry point URL. Any additional resource must be extracted from returned documents. And, hey, this is the very reason why REST is defined as a style for “distributed hypermedia systems”.

How that works with the Roar gem is subject to discussion in my upcoming next post! Stay tuned and enjoy life!

Make your rspec output look more interesting!
(yes we had too much spare time at our hands… )

# spec/spec_helper.rb
# encoding: UTF-8

if defined? RSpec::Core::Formatters::ProgressFormatter
  RSpec::Core::Formatters::ProgressFormatter.class_eval do
    DOTS = ['☘','⚘']
    def example_passed(example)
      output.print green(DOTS[rand(DOTS.size)])
    end
  end
end

Usage

withTimeFrozenAt("2012-01-01", function(){
  it("is frozen", function(){
    // do something useful with time 
  })
})

Code

var withTimeFrozenAt = function(time, fn){
  describe('with time frozen at ' + time, function() {
    var oldDate = Date;

    beforeEach( function() {
      Date = function() {
        return new oldDate(time);
      };
    });

    afterEach(function() {
      Date = oldDate;
    });

    fn();
  });
};

We had some weird issue where CKEDITOR always failed when you tried to open the link diaog, we traced it back to the filebrowser which for some reason always exploded…

CKEDITOR.config.removePlugins = 'filebrowser'
CKEDITOR._.events.dialogDefinition.listeners.shift();

# in your controller ...
BLANK_GIF = Base64.decode64("R0lGODlhAQABAPAAAAAAAAAAACH5BAEAAAAALAAAAAABAAEAAAICRAEAOw==")

render text: BLANK_GIF, type: 'image/gif'

A small snippet to interact with commandline applications that ask questions on stdin, particularly useful for testing.

Usage

interact "rails new foo -m template.rb", /RSpec/ => 'y', /MySql/ => 'n'

Code

 # https://gist.github.com/2166521
 # capture commandline output and send responses if nothing was read
  def interact(command, answers)
    puts command
    line = ""
    write = false

    PTY.spawn(command) do |output, input, pid|
      loop do
        rs, ws, = IO.select([output], [input])

        # read into the buffer
        if r = rs[0]
          write = false
          ret = r.read(1)
          print ret

          return unless ret

          line 

       

In so many ways, Bundler is a godsend. Yet, like all gems (forgive the wordplay), it is not without a few rough edges.

While at the (absolutely fantastic) Wroclaw.rb last weekend, I finally took a moment to round up some of these issues into a lightning talk. Unfortunately, not having paced myself correctly I got cut off just before the final couple of slides. So I resolved to reformulate the content into a blog post. A further impetus was a recent gist by Sven Fuchs (der Travis CI- und Ruby I18n-Meister).

The aim of this article is to highlight a handful of — what I consider to be — the rough edges, and to proffer potential solutions/workarounds.

Perhaps surprisingly, one thing I am not going to talk about is the issue of performance (in other words, that infamous Fetching source index… wait). With the recent launch of Bundler 1.1 you should notice a big speedup. If you have not upgraded, please do. Bundler uses semantic versioning and this release should be fully API-compatible with Bundler 1.0.x.

Here is my run down of the top 3 Bundler pain points:

1. Lack of optional groups
2. Lack of overridable version dependencies
3. Not being able to declare a dependency more than once

1. Lack of optional groups

Use case

You are developing an application (as opposed to a library) and you want it to be usable and deployable 'out of the box.' You do not want your users to have to manually edit (or uncomment lines in) their Gemfile. You will find that this problem arises nearly always with database drivers.

Workaround

The current best workaround is something along these lines:

bundle install --without=postgres,sqlite3

In other words, rather than specifying groups to include, you have to provide a list of what you want to exclude.

gem 'pg',       groups: 'postgres'
gem 'sqlite3',  groups: 'sqlite3'
gem 'mysql',    groups: 'mysql'

A better fix

There is a proposal from @FooBarWidget and @mislav, which I find to be very pragmatic. In summary, this is what it would look like:

Installing optional groups,

bundle install --with=postgres

and in the Gemfile,

group :postgres, optional: true do
  gem 'pg'
end

I would encourage you to +1 the proposal!

2. Lack of overridable version dependencies

Use case

When a new version of Rails appears, it is frustrating to upgrade, only to discover that you cannot proceed until the plugins and libraries you depend upon are updated. In some cases, this is down to definite API incompatibilities, but in other cases, it might have been that the developer of the particular library was just too cautious in specifying the versions the library supports.

Workaround

The current, recommended workflow is to fork the particular library or plugin and specify it as a git dependency in your Gemfile.

A better fix

There is an issue open for this.

Thus, overriding dependencies when a particular gem gemspec is out of date:

gem 'sass-rails' do
  override 'railties', '4.0.0'
end

Alas, this is not something the Bundler team are currently considering adding in, and as such, there is no code. The Bundler team assert that the fork and git dependency workflow works well, but for my development team, it's turned into a maintenance nightmare.

My main argument is that a gemspec is not core application code, but instead metadata. As such, there should be mechanism for easily overriding this metadata – and that this mechanism should not be considered to be in the same light as monkey-patching.

I would also encourage you to comment on (or perhaps even +1) this issue!

3. Not being able to declare a dependency more than once

Use case

Bundler will not currently let you declare a dependency more than once. This means you cannot have different versions or sources for a dependency, even if you scope your dependencies using groups.

Not being able to declare a dependency more than once makes the following things painful:

• testing against multiple versions of a resource (for example, in CI) (problem A)
• development of componentised systems: (perhaps working against a git repository of a dependency, or an Engine in git) (problem B)

In fact, problem B provided my initial motivation for investigating the whole question. At Payango, we are currently building a new platform for prepaid VISA cards; our previous platform was built as a monolithic Rails application, but the new platform is built on the foundations of a Service-oriented Architecture. For each given product on our platform, we have multiple applications running. To share code, we are employing Rails engines and internal libraries extensively.

Of course, in a perfect world — and because we aim to keep concerns separate — our layers would be perfectly tested and respect their contracts. In this perfect world, we would never need to break down, debug or change views in our upstream library or Engine. Unfortunately — despite the best efforts and good design — we do sometimes need to work on a library and the host application simultaneously.

Workaround …

The concept is easy: the solution for both problems is to employ multiple Gemfiles.

… for problem A

Create more than one Gemfile, where you want to use different versions or a specify a tighter version than that defined in the .gemspec.

I have seen this solution used by Ripple, for example:

# Taken from:
# https://github.com/seancribbs/ripple/blob/master/Gemfile.rails31

gem 'activemodel', '~> 3.1.0'

instance_eval File.read(File.expand_path('../Gemfile', __FILE__))

where the .gemspec may contain,

gem.add_dependency "activemodel", [">= 3.0.0", "< 3.3.0"]

such an approach is useful for automated testing, particularly Continuous Integration. Travis CI allows you to specify multiple Gemfiles in your .travis.yml,

gemfile:
  - Gemfile.rails30
  - Gemfile.rails31
  - Gemfile.rails32

… for problem B

The solution lies in the environment variable BUNDLE_GEMFILE. First create a copy of your Gemfile as Gemfile.local and specify your path dependencies in your Gemfile.local, keeping your git or gem dependencies in your original Gemfile.

When it comes to working with the host application or library, just make sure BUNDLE_GEMFILE is set:

BUNDLE_GEMFILE=Gemfile.local bundle install
BUNDLE_GEMFILE=Gemfile.local rake spec
BUNDLE_GEMFILE=Gemfile.local rails server

of course you can wrap this into a shell function,

local_gemfile_on
local_gemfile_off

(See the gist for example functions you can add to your .bash_profile. You might also consider adding your current BUNDLE_GEMFILE to PS1)

Keeping more than one Gemfile is, of course, a big pain. So, use Rake to keep the copy in sync:

task :local_gemfile do
  gemfile = File.read('Gemfile')
  gemfile.gsub!(/git: ["']git@github\.com:payango\/([\w-]+)\.git["']/, 'path: \'../\1\'')
  File.open('Gemfile.local', 'w') { |file| file.write(gemfile) }
end

See the gist for the full tasks.

Two points to bear in mind when using this technique with Rails:

• This will not handle code reloading: just because you do not have to bundle update each time you want to pull in the latest version of your library/Engine, it does not mean you will see all changes straight away. I have not yet worked out a solution for reloading Engines code on change.
• If you are building on top of an older point release of Rails 3.x, you may need to update your config/boot.rb file. Because, older generated boot files overwrote the BUNDLE_GEMFILE environment variable, make sure your file looks like this.

A better fix

A better fix is on its way. Just as I was finishing this article, the following pull request from @josevalim was merged. I have yet to dig through the code, and it will take some time until the feature is released, but what is there already looks promising.

As a conclusion

Let us get this clear (and to end on a positive note):

The Ruby world before Bundler was a much more painful place. Manually installing Gems — with the ensuing Gem activation conflicts — was not fun. The Rails 2 way of dependency management, config.gem was buggy, at best.

Bundler has collectively spared us thousands of hours, and I hope to see it improved further.

What are your pain points?

Wherein I write about Rails' current implementation of logging and ActiveSupport's greatest feature that was added in 3.0.

I've been thinking a lot about logging lately. I'm a big fan of logging. I also spent some quality time with a Rails app again recently. I'm not a big fan of Rails' logging. Something needs to change, either me or Rails' logging. I opted for the latter.

After some digging in I found that Rails 3 has improved quite significantly on the logging front. The output is still the same unparsable mess it used to be, but the way logging is implemented has changed quite drastically, and much for the better, I'd argue.

With Rails 3, one important thing was added, that drives parts of the both logging and benchmarking (e.g. for ActiveRecord's query measurements). Everything is now built around notifications and instrumentation. When ActiveRecord fires a query, it measure the time required and then triggers an event with the recorded time. Whether someone picks up the event is not of its concern. Which is exactly what the new notifications are about: separation of concern.

I was surprised to find that not a lot of people seemed to know about them. Let's have a short look at what it allows you to do.

ActiveSupport::Notifications

The idea is far from new. The Pragmatic Programmer talked about blackboards, a similar mechanism, Cocoa has had something similar for ages, and now Rails has something like it too.

The basic idea is that you have a centralized repository where you can subscribe to events, for example to the event ActionController triggers when it process an action. Here's the code to extract the path that was requested by a user.

ActionSupport::Notifications.subscribe('process_action.action_controller') do |event|
  puts event.payload[:path]
end

Any number of subscribers can be attached to a message, they'll get notified. The default implementation is synchronous, but nothing could and should keep you from adding an implementation that uses a message queue instead.

This is pretty cool. I started using it to track metrics and not clutter the code with the specifics. When something of interest happens, an event is triggered. If someone listens, cool, if they don't we'll keep going. I could have one subscriber that collects metrics and another one for tracer logging.

Rails uses these notifications all over the place, in particular for logging. To avoid having lots of manual subscriptions to specific events, Rails also added a mechanism to subscribe to events for the sole purpose of logging them and without adding subscriptions manually for all of them. The LogSubscriber was born.

ActiveSupport::LogSubscriber

LogSubscribers are exactly that: easy ways to subscribe to events whose purpose is logging. Of course what you do with the events is up to you, but that's their main purpose. Every Rails component uses them, and every component has its own implementation of a LogSubscriber. Here's an excerpt of the one used by ActionController.

Every public method defined, except for logger, will be attached to an event of the same name. So the three relevant events for this subscriber are start_processing and process_action.

To make the improvement of notifications and log subscribers more visible, here's the code that did the same in Rails 2.3. You decide which you like better. I certainly enjoy the decoupled-ness of the log subscriber a lot more.

When you defined your LogSubscriber, you can attach it to a namespace:

ActionSupport::LogSubscriber.attach_to :action_view

This creates a new instance of the LogSubscriber and attaches all methods to their corresponding events. The event names are along the same lines as the example with process_action.action_controller. First the event's name, then the component's name.

Rails' Logging

The example above nicely brings me to Rails' logging. First up: I like Rails' idea of logging everything, for development purposes it's awesome and pretty helpful. When things are moved into production the fun stops for me though. The logging out put is hard to parse, and it's hard to make sense of because it's usually multiple lines per request.

Rails 3.2 recently added tagged logging and a request identifier to work around that. But that still doesn't solve the problem of the output in general being too noisy and hard to parse for a centralized logging service, or any logging service. If you don't care about your logs then I'm sure you're fine, but I care a great deal about my logs. When things break, they're the sole source of truth, and I like to make sure they're valuable enough to fulfill that premise. Rails' request logging gets in the way of that for the reasons outlined above.

To remind you of what we're talking about, here's an example log output for a single request:

Started GET "/" for 127.0.0.1 at 2012-03-10 14:28:14 +0100
Processing by HomeController#index as HTML
  Rendered text template within layouts/application (0.0ms)
  Rendered layouts/_assets.html.erb (2.0ms)
  Rendered layouts/_top.html.erb (2.6ms)
  Rendered layouts/_about.html.erb (0.3ms)
  Rendered layouts/_google_analytics.html.erb (0.4ms)
Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord: 0.0ms)

It reads pretty nicely for sure, but as soon as you add more processes that dump their output in the same log, things get mingled and some of the information is, in my opinion just not necessary in production.

So we have a pretty centralized approach to logging, and me wanting to do something to improve the logging. Clearly the two could be made to work together.

Towards a Better Logging (in Production)

My ideal request logging is a single line per request, nothing more. That's clearly at odds with the output above, but thanks to the fact that (almost) everything is wrapped in log subscribers. Here's a line of something that would fit my purpose pretty nicely:

GET / format=html action=home#index status=200 duration=58.33 view=40.43 db=15.26

It's one line, it contains all the relevant information, it's pretty easy to parse for a machine, and it's easy to read for the human eye. Some told me the latter shouldn't be necessary, and I'm certainly not hung up on it, but I like to be able to skim logs.

The message is pretty clear: the HTTP method, the URL and an slew of optional parameters. If there's an exception that too should end up as single line. You can log the stacktrace of course, but it's much harder to make sense of for the same reason multiple lines of log output are hard to make sense of. They drown in a river of log output from multiple sources. I thought about only storing relations to an exception in the logs instead, e.g. an identifier or an error code. Throw in a request identifier as metadata stored with the exception, and you've got yourself a nice way to correlate exceptions and log lines.

How can we get to the output above? It turns out, it's actually pretty simple. We need to unhook the log subscribers for ActionView and ActionController events and hook in our own. The result is Lograge, a logging experiment I extracted into a library from the Travis CI source code, where I first started playing with the ideas I had around logging.

It adds its own log subscriber, discarding all irrelevant events, only accepting two events instead of the whole bunch included by default. The result is a single log line. Easy on the eyes, easy on the machine.

An Experiment in Logging

My main goal is to eventually have a saner way of logging requests in Rails (or any web framework, for that matter). Lograge is the beginning of that. I already got great feedback on the general idea and on specifics like the log output.

I have also yet to solve how to properly log request parameters, so this is only the beginning. I care a great deal about logging and I'd like to see that eventually improve in Rails so that other people start caring for their logs too, if only in production mode or as an optional feature. If not, Lograge will be here for you. I also have Rails 2.3 on my radar if you're still using that. It's a lot messier to implement, but not impossible.

If you want to know a bit more about the internals, the README is a good place to start. If you have any input on the ideas implemented in Lograge, the log output or anything else, feel free to open an issue. Let's talk about logging, and let's make it better.

Do you remember when we were writing a reusable sidebar element some time ago? The sidebar used in many controllers was implemented using a cell which encapsulated both assets and code in one place. Those were good times.

When needed, we could render the box with a render_cell call anywhere in our app.

#sidebar
  = render_cell :posts, :recent

Reusability With Engines

Now, imagine this sidebar box was so universally usable that you wanna use it in another project. Code is usually distributed with gems in a Ruby environment – but how does that work with partials, helpers, view code?

We got engines in Rails. We got Cells. We got gems. So let’s pack the cell in an engine gem!

Engines were designed to distribute controllers, views and models between Rails projects. As an example, lots of the authorization gems around use engines for reusable login pages or password reset forms that can be used right in your project. There was no easy way for reusing partials and behaviour, though, unless you’re using cells.

The Cell, Gemified.

In order to ship the sidebar posts box I first need to create a Rails engine.

$ rails plugin new sidebar --mountable

This creates a distributable directory with all the files needed to mount it into a Rails app.

As a second step we need to move the PostsCell into that gem.

$ mv app/cells/posts sidebar/app/cells/

You can now include the sidebar gem in any application using the Gemfile.

gem 'sidebar', :path => 'vendor/plugins/sidebar'

To render your reusable component, just use the render_cell call as you did in the originating application.

Conclusion

Damn, this conclusion came fast, but there is really not more to say. Oh, did I mention that you should update to Cells 3.8.3 for full engines support in Rails 3.0, 3.1 and even 3.2?!

Cells + Engines bring real reusability to Rails. It’s da shit.

It has been a while since I blogged about Roar and it’s representers – beautiful South Africa kept me from doing my homework. Nevertheless, in the meantime, the new roar-rails gem emerged bringing together Rails and Roar in a very easy going way. I’d like to describe how this gem works by implementing a fruit and fruit bowl backend service following the examples from the last posts.

Oh, and before I forget mentioning, the Rails example app is available on github where I will keep adding examples from the Ruby On REST blog series.

Setting It Up…

To make things happen it is necessary adding the roar-rails gem to your Gemfile.

gem 'roar-rails'

As we wanna use Rails’ URL helpers in our representers we have to add one line to the environment.

  config.representer.default_url_options = 
    {:host => "yum.my"}

Rendering Fruits, Done Right.

For simplicity let’s assume our Fruit model has the following underlying, highly complex database scheme.

  create_table "fruits" do |t|
    t.string   "title"
  end

Tough stuff, isn’t it?

Clients might want to retrieve fruit representations, that’s why we start with a functional test for handling GET requests on the FruitsController.

class FruitsControllerTest  ActionController::TestCase
  include Roar::Rails::TestCase
 
  test "GET returns fruit document" do
    get :show, :id => 1, :format => :json
    assert_body "{\"title\":\"Apple\",\"links\":[
      {\"rel\":\"self\",
       \"href\":\"http://yum.my/fruits/Apple\"}]}"
  end

Including the Roar::Rails::TestCase module into the test gives us a couple of new or changed methods. One of those is assert_body and all it does it testing equality of the passed string and the last returned document. This test verifies our REST representations are rendered correctly.

Here’s the handling code in the FruitsController.

class FruitsController  ApplicationController
  include Roar::Rails::ControllerAdditions
  respond_to :json
 
  def show
    fruit = Fruit.find_by_id(params[:id])
    respond_with fruit
  end

This is all very familiar code! But, wait… what happens here behind the scenes? The inclusion of the ControllerAdditions module activates a new responder which will take care of the JSON-rendering that is invoked inside respond_with. So, behind the curtain, the following is executed.

fruit.
  extend(FruitRepresenter).
  to_json

The roar-rails gem infers the representer name used for rendering.

The Fruit Representer

Now, where is that representer? It lives in the app/representers directory and its implementation is quite simple.

module FruitRepresenter
  include Roar::Representer::JSON
 
  property :title
 
  link :self do
    fruit_url(title)
  end
end

All it does is defining the title property and the self link. Please note that we can simply use URL helpers here, making it pretty easy to include hypermedia in our representations.

Amazing! This is all the code needed for rendering fruit documents, including hot hypermedia and handling GET requests. Now, what do we need to consume incoming documents, in, let’s say a POST request?

Eat More Fruits!

To create new fruits in our backend service we’d need to accept POST requests. Test it.

class FruitsControllerTest  ActionController::TestCase
  # ...
 
  test "POST creates fruit" do
    post :create, "{\"title\":\"Orange\"}", 
      :format => :json
 
    assert_body "{\"title\":\"Orange\",\"links\":[
      {\"rel\":\"self\",
      \"href\":\"http://yum.my/fruits/Orange\"}]}"
  end

Again, we use assert_body to assure the resource returns the freshly created representation of the orange. Now, check the post: The second argument is the actual JSON document sent by the client. This is an API change. We no longer pass hashes to the “POST”, but use real documents to test our services.

The consuming controller action is hilariously simple as well.

class FruitsController  ApplicationController
  # ...
  def create
    fruit = Fruit.new.
      extend(FruitRepresenter).
      from_json(request.body.string).
      save
 
    respond_with fruit
  end

It creates a new Fruit instance, mixes in the respective representer, parses the incoming JSON document and updates the fruit’s attributes in the #from_json method and then renders the created model.

In the upcoming version of roar-rails, this code can also be written like:

class FruitsController  ApplicationController
  # ...
  def create
    fruit = consume(Fruit).
      save
 
    respond_with fruit
  end

The consume method provided by roar-rails is still under discussion – feel free to add comments to this post if you have a groundbreaking idea how to simplify the consumption steps!

Fruits On This Planet – Unite!

If that was to easy for you, and I bet it was, why not implement the fruit bowl as well. Consider this model class.

class Bowl  ActiveRecord::Base
  has_and_belongs_to_many :fruits
end

Instead of wasting time with the rendering test and parsing code – which is identical to the fruit code – we jump directly into the parsing test.

class BowlsControllerTest  ActionController::TestCase
  include Roar::Rails::TestCase
 
  test "POST creates bowl with fruits" do
    post :create, 
      "{\"fruits\":[{\"title\":\"Orange\"}]}", 
      :format => :json
 
    bowl = Bowl.find(:last)
    assert_body "{\"fruits\":[
      {\"title\":\"Orange\",\"links\":[
        {\"rel\":\"self\",
        \"href\":\"http://yum.my/fruits/Orange\"}]}],
      \"links\":[
        {\"rel\":\"self\",
        \"href\":\"http://bowls/#{bowl.id}\"}]}"
  end

Sometimes I wish my blog was less narrow.

In the test, an initial bowl document is POSTed to the resource. We expect a fresh bowl representation containing fruits as the response.

Let’s look at the controller action to understand how representers work in a nested setup. Speaking of nesting, here’s the representer responsible for rendering and parsing bowls.

module BowlRepresenter
  include Roar::Representer::JSON
 
  property :location
 
  collection :fruits, 
    :class => Fruit, 
    :extend => FruitRepresenter
 
  link :self do
    bowl_url(id)
  end
end

In addition to the location property the collection defines a nesting fruits. The BowlRepresenter now knows that the collection contains Fruit instances that must be represented with the according FruitRepresenter. We already discussed that in one of the last posts.

Here’s the controller action using that representer.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

class BowlsController  ApplicationController
  # ...
 
  def create
    bowl = consume(Bowl)
 
    # puts bowl.fruits.inspect
    bowl.fruits = bowl.fruits.collect do |f|  
      Fruit.find_by_title(f.title)
    end
 
    bowl.save
 
    respond_with bowl
  end

I simply assume the consume call already works. Then, what is that loop? Well, when parsing the incoming document, the bowl representer creates a new Fruit instance for every fruit in the collection. If line 7 would be uncommented, it would output the following.

[#]

That is, before we reset the collection manually. The representer doesn’t know anything about the database – it just maps a fruit representation in the document to a fresh Fruit instance. This is why there is no id set, yet. We have to do that ourselves (lines 8-10). What might look clumsy to you is simplicity – it’s up to you how to map objects to database records.

Conclusion – For Now?!

Come on, it is easy to use representers in your Rails backends using the roar-rails gem. It helps you rendering documents and makes it quite easy to have object-oriented access to a parsed incoming representation – under Rails-conditions.

And, as always, this is just a start. Open source lives from discussion and criticism, so feel free to use the comments form below. In the next post I’ll discuss writing paginated collections using representers. Have a nice day!

If your environments/*.rb look like a repetitive mess, its time to get a config.yml!

• overview of your configuration
• dry code
• (optional) not check in all the passwords/keys of your app (only check in config.example.yml), great for open-source apps
• Can be loaded without loading the environment (e.g. small rake tasks)

# lib/cfg.rb
CFG = YAML.load(ERB.new(
  File.read("config/config.yml")
).result)[Rails.env].with_indifferent_access

# config/application.rb
require "cfg"

# config/config.yml
common: &common
  api:
    airbrake_key: xxx
    google_analytics_key: yyy
  admin_email: admin@example.com

test:
   CFG[:host]}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

(598) [22:45:01 quintus@hades] ~
$ use joke-ruby
ruby 1.9.3p125 (2012-02-16 revision 34643) [x86_64-linux]
(599) [22:45:05 quintus@hades] ~
$ ruby
x = 0
__label__ :foo
puts "In loop here"
x += 1
__goto__ :foo unless x > 5
puts the_answer_to_life_the_universe_and_everything^D
In loop here
In loop here
In loop here
In loop here
In loop here
In loop here
42
(600) [22:46:05 quintus@hades] ~
$

Es zeigt sich, dass weder goto ausgestorben ist, sowie dass es zumindest einen Ruby-Entwickler gibt, der ein Fan von Per Anhalter durch die Galaxis ist.

Ich verbürge mich dafür, dass das obige Beispiel self-contained ist. Keine fiesen Tricks. Ich habe auch nicht den Ruby-Quellcode verändert. Nur eine kleine Option beim Kompilieren gesetzt.

1
2
3
4
5

(602) [22:48:33 quintus@hades] ~/Downloads/ruby-1.9.3-p125
$ tail -n3 vm_opts.h
#define SUPPORT_JOKE                 1

#endif /* RUBY_VM_OPTS_H */

Quelle: http://patshaughnessy.net/2012/2/29/the-joke-is-on-us-how-ruby-1-9-supports-the-goto-statement, gefunden über RubyInside.

Valete,
Quintus

PS: In IRB gibt das einen Segfault, daher habe ich hier Rubys Lesemodus vom Standardinput verwendet.

Tags: C | de | Ruby

With February almost over, it's time to give you news things to read, or at least to make a list of things I've been reading lately.

Continuous Delivery

A rather wordy and very repetitive excursion into the ideas behind continuous delivery, which involves continuous integration, continuous deployment and lots of other things. While I'm all on board with the ideas in the book, it's simply too long. Every chapter repeats a lot of the things from other chapters, mostly with the purpose of being easily accessible on their own. I like the book in general, but 500 pages for a book like this is simply too long.

The Architecture of Open Source Applications

This is an incredible resource, and best of all, it's free. Several open source projects are outlined from how their architecture evolved over time and how they came about to begin with.

The chapters I've very much enjoyed so far are Graphite, HDFS, Erlang and Riak, Sendmail. But the best chapter by far is the one on BerkeleyDB, the key-value store you didn't know you'd find everywhere. It's an exceptional read and should be mandatory for software developers. It's a great story on how to evolve architecture of what started out as a simple library over the course of 20 years.

While the book is free to read online, please consider buying if you like it. It's for a good cause too. There's a second volume in the works, due in March.

Things Caches Do

A short and simple read on what web caches do. If you build web apps, read this. Also has links to more in-depth articles on HTTP and caching.

Probabilistically Bounded Staleness for Practical Partial Quorums

An analysis of how eventual and consistent eventual consistency is. Relevant if you're dealing with Dynamo-style databases, but an interesting read any way you look at it. Accompanied by a website that allows you to calculate the probability that data in a quorum-replicated cluster is consistent over time.

Design of Apache Kafka, a high-throughput distributed messaging system

If you're into messaging, you should read this, especially if all you know is RabbitMQ and Redis for queueing. Both don't scale well and they're not easy to make fault-tolerant. Kafka, built at LinkedIn, follows a very different design to allow being run fully distributed.

Notes on Varnish, from the Architect

The ideas behind Varnish, why Squid's way is outdated, and a perspective on great uses of memory-mapped files. Short and self-congratulatory read.

The LMAX Architecture

After reading this, you'll have a different perspective on things when it comes to building high-throughput systems. LMAX is a real-time trading site, and this article describes how they built the service that manages millions of trades per second. Other than your typical architecture description on highscalability.com, this is one has a lot of great information.

ZooKeeper

A paper on ZooKeeper, a system for coordinating process in distributed systems. This is a surprisingly good read, and it also outlines several use cases for ZooKeeper.

The Lean Startup

People raved about this book, and it was recommended to me from several folks. After reading it, I'm rather meh on it. It felt like reading an over-glorified handbook for a process that startups must adopt to be successful. Overuse of the words "disruptive", "pivot", "startup", and "entrepreneur" all but added to the slightly weird taste the book left me with.

Still, I don't think of reading something as a waste. If anything, this book helped me clarify my thoughts on the matter and gave me more perspective. That's what reading to me is all about.

Why We Get Fat

I got curiously interested in how the body works and, in particular, how carbohydrates affect it. While this book should be taken with a spoon full of salt, I learned quite a bit from it.

Read, Read!

If you want to read something really good like right now. Make it the aforementioned chapter on BerkeleyDB and the ZooKeeper paper.

Over the last months and weeks and days I contributed some tiny gems (I call them microgems) for the really awesome mongoid ORM.

With the release of the third microgem I also began my meta gem MongoidMidgets to easily include all my stuff with a single line in my Gemfile.

1
2

gem "mongoid_midgets",
  :require => ["mongoid_bitfield", "mongoid_touch", "mongoid_upk"]

gem "mongoid_midgets",
  :require => ["mongoid_bitfield", "mongoid_touch", "mongoid_upk"]

The mongoid_touch gem was already introduced in another blog post (Ruby Gem Contributions).

Side fact: Durran, the maintainer of mongoid, asked me, if I want to merge the mongoid_touch into the core. Unfortunatelly I didn't invest the time to do so, because this needs a lot of work to have a really competitable functionality like in ActiveRecord and mongoid is also a huge code base where you can get lost like Alice in Wonderland … ;o)

A short overview of the other microgems:

Mongoid::Bitfield

This tiny gem provides a bitfield or a collection of boolean bits/flags.

This bitfield will be stored as an integer value in the mongodb database.

Motivation

In the project of my current employer I found this tiny bitfields but for ActiveRecord. I also want to have them in Mongoid, too.

Currently these bits in a bitfield aren't searchable, so this gem is just for fun now. Maybe later it will extended to have more usage of it.

Main benefit should be to save a little bit space in the database, because an integer value should always be smaller than an array or a hash of multiple booleans. For models with huge sets of boolean fields it could really helpful despite the fact of missing searchability.

Example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

class User
  include Mongoid::Document
  include Mongoid::Bitfield

  bitfield :flags, :is_admin, :adult, :drinker
end

### no save


user = User.new


user.is_admin #=> false


user.is_admin = true


user.is_admin #=> true

### atomic save


user = User.create!


user.drinker_enable!


user.reload


user.drinker #=> true

class User
  include Mongoid::Document
  include Mongoid::Bitfield

  bitfield :flags, :is_admin, :adult, :drinker
end

### no save

user = User.new

user.is_admin #=> false

user.is_admin = true

user.is_admin #=> true

### atomic save

user = User.create!

user.drinker_enable!

user.reload

user.drinker #=> true

Links

• More infos on github: asaaki/mongoid_bitfield
• Current build status:

Mongoid::UPK

Mongoid::UPK generates more unique _id by using UUID.

UPK stands for "unique primary key" or "uuid primary key".

Motivation

While writing very fast to MongoDB it is possible to take a BSON::ObjectId twice. This will lead to big troubles because you never know which document will be returned.

This gem utilizes the key method and overwrites the automatically given default id with the choosen generator on save. So the real id can be read first after a successful save (I don't want to hack deeper here, because saves are more important than getting a unique id on instance initialization).

UUID is in most cases more reliable, especially if the app instances are set up correctly (and/or running on different machines).

Example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

### base pk => c1b457b03214012f316d10e6baa29f2c


class MyModel
  include Mongoid::Document
  include Mongoid::UPK
  uuid_pk # you can leave this out, because it's the default generator
end

### pretty pk (with dashes) => c1b457b0-3214-012f-316d-10e6baa29f2c


class MyModel
  include Mongoid::Document
  include Mongoid::UPK
  pretty_uuid_pk
end

### self-defined pk gen


class MyModel
  include Mongoid::Document
  include Mongoid::UPK
  unique_pk do
    # code goes here
    # result must be a string
    # example:
    Digest::MD5.hexdigest(Time.now.to_f.to_s)
  end
end

### base pk => c1b457b03214012f316d10e6baa29f2c

class MyModel
  include Mongoid::Document
  include Mongoid::UPK
  uuid_pk # you can leave this out, because it's the default generator
end

### pretty pk (with dashes) => c1b457b0-3214-012f-316d-10e6baa29f2c

class MyModel
  include Mongoid::Document
  include Mongoid::UPK
  pretty_uuid_pk
end

### self-defined pk gen

class MyModel
  include Mongoid::Document
  include Mongoid::UPK
  unique_pk do
    # code goes here
    # result must be a string
    # example:
    Digest::MD5.hexdigest(Time.now.to_f.to_s)
  end
end

Links

• More infos on github: asaaki/mongoid_upk
• Current build status:

And now?

Feel free to contribute!

Or write your own microgems to mongoid.

You can also ask me to include them into MongoidMidgets. If I like them and they are very useful, I will put them into my collection.

I always forget what kinds of crazy things you can do with Ruby's blocks and their parameters, so here's a little write-up on them. I regularly forget things I've learned (must be an age thing), and I found that not even books on the Ruby language fully cover all the gory details on block (and method) parameters. So consider this my personal reference of crazy Ruby block syntax features for future use.

The Basics

In its simplest form, a block parameter is a list of names, to which values passed into the block are assigned. The following iterates over all elements in the hash, emitting key and the corresponding value, printing both key and value.

blk = ->(key, value) {puts key, value}
{username: "roidrage", fullname: "Mathias Meyer"}.each &blk

Yes, that's the boring bit, bear with me.

Splat Parameters

You can use splat parameters too, catching an array of all arguments.

blk = ->(*args) {puts args}
blk.(1, 2, 3) 
# => [1, 2, 3]

Notice that crazy syntax for calling the block too, pretty wild. The fun starts when you combine a splat parameter with one that's fixed.

blk = ->(first, *tail) {puts first}
blk.(1, 2, 3)
# => 1

Why not put another fixed parameter at the end? That'll assign the first element of the arguments to the variable first, the last element to last, and everything in between to middle

blk = ->(first, *middle, last) {puts last}
blk.(1, 2, 3)
# => 3

This can grow to an arbitrary complexity, adding more fixed parameters before and after a splat. In this example, middle will just be an empty array, as the fixed parameters are greedy and steal all the values they can match.

blk = ->(first, *middle, second_last, last) {puts second_last}
blk.(1, 2, 3)
# => 2

Fortunately you can have only one splat parameter.

Just for fun, you can also just specify the splat operator and nothing else.

blk = ->(*) {puts "lolwut?"}

Default parameters

If you want to save some time, you can assign default values to block parameters.

blk = ->(list = [1, 2, 3]) {list.sample}
blk.()

Again, you knew that already. Here's some craziness though, courtesy of Avdi.

blk = ->list = (default = true; [1, 2, 3]) {puts default}
blk.()
# => true
blk.([4, 5, 6])
# => nil

Referencing Other Parameters

You can reference parameters previously defined in the list. Want to do an impromptu mapping on the list above before even entering the block?

blk = ->(list = [1, 2, 3], sum=list.inject(1) {|acc, value| acc*value})) {
  list.sample
}

Don't do that, though. But good to know you can. Note that it only works for parameters listed before the one you're assigning to. You can also shorten the example above by quite a bit.

blk = ->(list = [1, 2, 3], sum=list.inject(:*)) {
  list.sample
}

Block-local parameters

To add more character variety, you can declare variables local to the block by adding a semicolon and another list of parameters. Helps when you want to make sure variables used in the block don't accidentally overwrite or reference variables outside the block's scope. Blocks are closures, so they reference their environment, including variables declared outside the block.

username = "roidrage"
blk = ->(tags; username) {username = "mathias"}
blk.(["nosql", "cloud"])
puts username
# => "roidrage"

You'll be pleased to hear that there's no craziness you can do with block local parameters, like assigning defaults.

Ignoring arguments

This one may look familiar to folks knowledgeable in Erlang. You can ignore specific arguments using _. Combine that with the splat parameter and you can extract the tail of a list while ignoring the first element. Then you can recursively iterate through the tail, ignoring the first element.

blk = ->(_, *tail) {blk.(tail) if tail.size > 0}

When is this useful? Ruby is not a pattern-matching language after all. For instance, imagine an API that expects blocks handed to a method call to expect a certain number of arguments. Ruby gives you warning if the block's arity doesn't match the number it was called with. This way you can silently dump parameters you're not interested in while still conforming to the API.

Okay, I lied to you, this is actually not an operator of sorts, this is a simple variable assignment to a variable called _. It's a neat little trick though to make it obvious that you're not interested in a certain parameter. Also note that _ in irb references the value returned by the last expression executed.

You'll find it used in several places in the Rack source code.

Tuple arguments

This one blew my mind when I found it somewhere in the depths of Rack's source (or somewhere else I don't remember). Think of a hash where each key points to an array of things. Wouldn't it be nice if you could extract them all in one go while iterating over them without having to first iterate over the hash and then over the embedded arrays?

Turns out, the tuple operator is just what we need for this. This is an example from a Chef cookbook I built a while back, specifying some thresholds for an Apach configuration for Monit.

apache_server_status = {
  waitlimit: ["<", "10%"],
  dnslimit: [">", "25%"]
}

apache_server_status.each do |name, (operator, limit)|
  puts "protocol apache-status #{name} #{operator} #{limit}" 
end

Notice the definition of (operator, limt). That little bead nicely extracts the array with operator and a percentage in it into two parameters. Here's another thing that blew my mind, chaining enumerators, collecting values and index from a hash, for example. Note that hashes are sorted in Ruby 1.9, so this is a perfectly valid thing to do.

names = {username: "roidrage", fullname: "Mathias Meyer"}
names.each.with_index do |(key, value), index|
  puts "##{index}: #{key}:#{value}"
end

Hat tip to Sam Elliott for blowing my mind with this.

The end

Know what the best part is? All this works with method calls too. The tuple assignment, using _ to ignore values, the splats, referencing previous parameters.

Some of them even work for multiple variable assigmnents.

_, *tail = [1, 2, 3]
name, (operator, limit) = [:waitlimit, ["<", "10%"]]

Crazy.

When I ask people about how they deploy their Rails apps, most of them answer that they are using good old Capistrano. It is the most popular solution because it has been around since almost when people started deploying Rails applications and it works well in most cases. However, if you want to do a lot of custom tasks and need to look under the hood to extend it, Capistrano seems somewhat complicated. At least it did at the time I left it in favor of Vlad - an imho more intuitive deployment tool.

Though one might not like the offense against Capistrano that was shown when Vlad was released in 2007, it addresses some valid points when arguing against needless complexity. But enough of the history, this was almost five years ago, both of the tools have improved, made their way and all I wanted to share is some snippets and insights I got from using Vlad over the last years.

The main thing I like abot Vlad is that it is based on Rake, which makes it so easy to learn, understand and extend. If you want to do something custom, all you have to do is to write a Rake task. And as you will want to use them across different projects, you can even bundle them up and distribute them as a gem, like I’ve done with vlad-extras, a set of extensions for tools like DelayedJob, ThinkingSphinx, Monit, Whenever and so on.

Here is what a simple config/deploy.rb file might look like:

# Custom variables used in the following config
            set :application, "appname"
            
            # Required variables
            set :domain, "#{application}.com"
            set :deploy_to, "/var/www/#{application}"
            set :repository, "git@github.com:myaccount/#{application}.git"
            
            # Optional variables
            set :user, "deploy" # if different from your current login
            set :bundle_cmd, "/usr/local/bin/bundle"
            set :rake_cmd, "#{bundle_cmd} exec rake"
            set :revision, "origin/master"
            
            # vlad-extras config
            set :copy_shared, {
              'config/maintenance.html' => 'config/maintenance.html',
              'config/database.yml'     => 'config/database.yml' }
            set :symlinks, {
              'assets'              => 'public/assets',
              'config/database.yml' => 'config/database.yml' }
            set :deploy_tasks, %w(
              vlad:maintenance:on
              vlad:update
              vlad:symlink
              vlad:bundle:install
              vlad:assets:precompile
              vlad:migrate
              vlad:start_app
              vlad:maintenance:off
              vlad:cleanup)
            
            # Bundler ships with vlad integration you'll have to require
            # in order to use the vlad:bundle tasks
            require 'bundler/vlad'
            
            # Require custom vlad tasks or recipes from vlad-extras
            require 'vlad/maintenance'
            
            # Some custom tasks, can be included directly in the deploy file
            namespace :vlad do
              namespace :custom_script do
                %w(start stop restart).each do |task|
                  desc "#{task.capitalize} the custom script"
                  remote_task task, :roles => :app do
                    run "cd #{current_release}; RAILS_ENV=#{rails_env}
                      #{bundle_cmd} exec ruby script/custom #{task}"
                  end
                end
              end
            end
            

Here is an example of how easy it is for instance to add staging support:

task :staging do
              set :rails_env, "staging"
              set :deploy_to, "/var/www/#{application}-#{rails_env}"
            end
            
            task :production do
              set :rails_env, "production"
              set :deploy_to, "/var/www/#{application}-#{rails_env}"
            end
            

All you need to do is to change some variables based on the environment that is set with a simple Rake task. You’d invoke it by running

$ bundle exec rake staging vlad:deploy
            

Gemfile

You can include Vlad and it’s requirements in the development group, because it you do not need it in any other environment. Requiring is done in the Rakefile like shown below.

group :development do
              # Deployment
              gem 'vlad', :require => false
              gem 'vlad-git', :require => false
              gem 'vlad-extras', :require => false
            end
            

Rakefile

Add this snippet to your Rakefile, just before MyApp::Application.load_tasks

if Rails.env.development?
              begin
                require 'vlad'
                require 'vlad-extras'
                Vlad.load(scm: :git, web: :nginx, app: :passenger, type: :rails)
              rescue LoadError
                puts 'Could not load Vlad'
              end
            end
            

You might have to configure the options given to Vlad.load.

If you want to get started, take a look at the documentation to find out about the parts that this post leaves out.

Troubleshooting

This part is about some trouble I’ve had and it is not really Vlad specific. You might encounter these problems when trying to deploy to your production server and maybe these tips and links will help.

The deployment shell environment

If Vlad does not find some commands, try the following:

To enable per user PATH environments for ssh logins you need to add to your sshd_config:

PermitUserEnvironment yes
            

After that, restart sshd!

Then in your users ssh home directory (~/.ssh/environment), add something to this effect (your mileage will vary):

PATH=/opt/ruby-1.9.3/bin:/usr/local/bin:/bin:/usr/bin
            

For details on that, see this article on setting the deployment shell environment

SSH Agent Forwarding

Maybe you also need to configure SSH Agent Forwarding:

$ ssh-add ~/.ssh/

Edit your ~/.ssh/config file and add something like this:

Host 
              HostName 
              User 
              IdentityFile ~/.ssh/
              ForwardAgent yes
            

For details on that see this article on SSH Agent Forwarding

Lets say you are running something over VPN,
and VPN suddenly fails, then this will kill the app you dont want to connect via the normal interface.

Its just a small ruby script you can put into e.g. /usr/bin/

vpn-fuse VpnName 'killall secret-app'
vpn-fuse BankNetwork 'killall bank-client'
vpn-fuse iPredator 'killall transmission-gtk'
...

so far only tested on Ubuntu…

vpn-fuse

Need your bookmarks on the go !?

https://mymarks.heroku.com

the source is at https://github.com/grosser/mymarks, happy forking

Problem:
Lots of stuff does not need any translation, so we simply leave it untranslated (e.g. firstname shows as Firstname). But this leaves ugly “missing translation” spans in out html.
Instead of entering senseless translations for everything thats missing, we simply deactivate the tooltips in production, while also avoiding raising/rescuing theses MissingTranslation errors to improve performance(rescue/raise is not cheap).

Code:

# http://grosser.it/2012/02/04/do-not-show-i18n-missing-translation-tooltips-in-production/
# config/initializers/disable_i18n_tooltips.rb
# missing translations
# - Do not show tooltips in production/test
# - Do not raise ( speedup) for every missing translations
I18n::Backend::Base.class_eval do
  def translate_with_default(locale, key, options = {})
    if options[:rescue_format] == :html && ['test','production'].include?(Rails.env)
      default = key.to_s.gsub('_', ' ').gsub(/\b('?[a-z])/) { $1.capitalize }
      options.reverse_merge!(default: default)
    end
    translate_without_default(locale, key, options)
  end

  alias_method_chain :translate, :default
end

If you want to run 1 big/slow test file, use Parallel-Split-Test, it splits it into multiple chunks and then runs them in parallel.

PS: If you have lots of tests you should take a look at Parallel-Tests, which runs multiple files in parallel.

This became necessary because in our current project: spec runtime=3h (without integration) / slowest file=5 minutes. Now this time is only 1/3rd.

Systematisch

Eine der großen Hürden für Dokumentation in Ruby sind, so denke ich, die mangelnden Dokumentationsrichtlinien. Es gibt de facto keine Anleitungen zum Thema wie schreibe ich gute Docs in Ruby? und ich denke, dass dies einer der Hauptgründe war, warum YARD enstanden ist. Rubys Standarddokumentationstool RDoc reicht jedoch zur Dokumentation völlig aus und ich persönlich ärgere mich jedes mal über YARD-Dokumentation, weil sie sich im Quelltext einfach nicht so schön liest wie RDoc und mich außerdem zwingt, neben dem RDoc-Markup auch noch das YARD-Markup zu lernen, wenn ich nicht die Dokumentation als HTML generieren will.

Doch genug davon. Das Fehlen dieser Richtlinien jedenfalls dürfte dafür verantwortlich sein, dass so viele Ruby-Projekte keine oder oder nur schlechte Dokumentation besitzen. Nicht, dass die Entwickler sich ohne Richtlinien nicht zutrauen würden, Dokumentation zu schreiben. Es ist vielmehr so, dass Entwickler ohne gegebene best practice einfach zu faul sind und so einfach nur uninformative Dokumentation entsteht. YARD erzwingt diese best practice beim Dokumentieren, RDoc hingegen nicht (YARD forciert mit @-Direktiven die Strukturierung der Doku, RDoc erlaubt freie Form), und so kommt es, dass YARD-Dokus so oft besser aussehen als RDoc-Dokus. Mal ganz abgesehen davon, dass ich RDocs Standardausgabetemplate (Darkfish) bemerkenswert hässlich finde, aber etwa ruby-doc.org zeigt ja, dass zumindest das kein Problem darstellt.

Also: Legt euch ein System für eure Dokumentation zurecht! Es muss nicht das im folgenden Vorgestellte sein, aber nehmt euch einmal eine halbe Stunde Zeit und überlegt euch, welche Informationen eine Methoden- oder Klassendokumentation enthalten muss, um einen der Materie völlig fremden Nutzer komplett zu informieren. Überlegt euch, wie einer denkt, der keine Ahnung von eurem Code hat und der einfach nur eure Library benutzen will. Pappt euch ein Schema neben den Monitor und haltet euch beim Schreiben einer Methode daran.

Chronologisch

Bei der zeitlichen Erstellung von Dokumentation gibt es zwei Möglichkeiten: Man kann sie vor oder nach dem Code schreiben. Es gibt Leute, die sind der festen Überzeugung, dass man erstmal seine gesamte Dokumentation schreiben sollte und erst dann den Code, aber wenn man während der Entwicklungsphase plötzlich was am API ändern muss, wird die Dokumentation hinfällig und man muss sie neu schreiben, sodass man die Dokumentation dann effektiv zweimal schreibt. Um das zu vermeiden, wähle ich den anderen Ansatz und schreibe erst den Code, und dann die Dokumentation, jedoch nicht im ganzheitlichen Sinne, dass ich erst eine Library vollständig fertigstelle und danach die Dokumentation schreibe, sondern eher in einem progressiven Verfahren. Konkret bedeutet das: Wann immer ich eine (öffentliche, zum API gehörende) Methode fertiggestellt habe, schreibe ich ihre Dokumentation. Oder zumindest versuche ich es, nicht immer gelingt mir diese strikte Abfolge ;-)

Technisch

Dokumentation sollte Grundkenntnisse des Nutzers in der jeweiligen Programmiersprache, das ist in unserem Falle Ruby, voraussetzen. Sprachtutorials haben in einer Dokumentation nichts verloren und machen nur unnötig Arbeit. Stattdessen sollte Dokumentation auf vorhandene Kenntnisse der Nutzer aufsetzen und sich entsprechenden englischen Fachvokabulars bedienen und nicht dem Nutzer beispielhaft erklären, was denn nun Klassen und Instanzen sind. Dafür gibt es bereits genug (Online-)Tutorials.

Ebenfalls technischer Aspekt der Dokumentation ist Präzision. Ich will keine Romane lesen, bevor ich weiß, wie ich eine Programmbibliothek zu bedienen habe. Sicherlich ist es interessant zu erfahren, wieso ein Methode so zu bedienen ist wie es ist, doch gehören solche Aspekte in meinen Augen eher auf die Website eines Projektes denn in ihre Dokumentation. Wenn sie unbedingt in die Dokumentation müssen, dann bitte in einer extra Datei »history.rdoc« oder so ähnlich.

Es gilt demnach: »In der Kürze liegt die Würze«. Jedoch sollten die Beschreibungen nicht derart kurz ausfallen, dass man dem roten Faden einer Erklärung nicht folgen kann. Generell gilt: Klassen- und Moduldokumentation können gern etwas länger sein, sie sollten dann aber auch alle allgemein die Klasse/das Modul betreffenden Aspekte abdecken, sodass keine Notwendigkeit besteht, in jeder Methode eine oder gar mehrere Informationen zu wiederholen. Allenfalls ein »See the class’ introductory text« kann angebracht sein.

Linguistisch

Die Hauptdokumentation eines Projekts sollte auf Englisch abgefasst sein. Es spricht jedoch nichts dagegen, Übersetzungen der Dokumentation zu erstellen, die jedoch immer darauf hinweisen sollten, dass die primäre Informationsquelle die englischsprachige Dokumentation ist. Hierbei gilt: Wird eine Dokumentation übersetzt, dann idiomatisch. Es gibt tatsächlich im Englischen Redewendungen, die es im Deutschen nicht gibt¹, auch wenn viele Leute das scheinbar nicht begreifen wollen. Ein Ausnahme hiervon stellen Fachbegriffe da: Da in der IT-Branche der größte Teil des Vokabulars nun einmal aus der englischen Sprache stammt, sollten stehende Begriffe beibehalten werden. Mir drehen sich jedesmal die Fußnägel hoch, wenn die Sprache von Feldern ist. Wer kommt schon darauf, dass es sich hierbei um Arrays handeln soll?

Inhaltlich

So, jetzt aber zum interessanten Punkt. Wie gestalte ich meine Dokumentation?

Struktur

Generell folgt eine von mir geschriebene Klasse stets dieser Struktur:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

class Classname
  # Extends
  # Includes

  # Konstanten

  # Attribute

  # Klassenmethoden

  # initialize

  # Instanzmethoden
end

Module werden enstprechend mit module anstatt mit class als Schlüsselwort eröffnet.

Klassen und Module

Nun, Klassen- und Moduldokumentationen sind Freitexte, die jedoch den Nutzen und die generelle Idee der jeweiligen Klasse erläutern und anhand von Beispielen verdeutlichen sollten. Auch ein Wort über die generelle Idee der Klasse/des Moduls sollte genausowenig fehlen wie eine Beschreibung des Zusammenspiels mit den anderen Klassen der Programmbibliothek oder sogar, falls sinnvoll und relevant, ihr Zusammenspiel mit Core- und Stdlib-Klassen.

Methoden

Methoden werden anhand ihres Zwecks dokumentiert. Nach einer kurzen Beschreibung der Aufgabe der Methode werden ihre Parameter gelistet, ihr Rückgabewert angegeben und eventuell vom Standard abweichende Exceptions benannt (so ist es etwa unnötig, ArgumentError für falsche Argumentanzahl zu dokumentieren, weil das ohnehin normal ist). Anschließend sollte ein kurzes Anwendungsbeispiel folgen, sofern der Methodengebrauch nicht eindeutig ist (so braucht die Methode #<=> als Basismethode für Comparable wohl kein Beispiel). In summa ergibt sich das folgende Gerüst für eine Methode:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

#Ein paar allgemeine Sätze.
#== Parameters
#[bar]    Beschreibung für Parameter 1.
#[foobar] (7) Beschreibung für Parameter 2.
#== Raises
#[FooBarBazException] Beschreibung, wann diese Exception geworfen
#                     wird.
#== Return value
#Beschreibung des Rückgabewerts in allen Fällen, wenn nötig,
#abhängig von Parametern.
#== Examples
#  foo(1, 2)
#  foo("a", "b")
def foo(bar, foobar = 7)
  # Code...
end

Jeder der Abschnitte außer dem allgemeinen Abschnitt kann weggelassen werden, wenn er für die Funktionsweise der Methode unwichtig ist (beim Examples-Abschnitt sollte man sich das jedoch genau überlegen).

Allgemein benutze ich zur Dokumentation nur das Gatterzeichen # direkt gefolgt vom Text, weil es für lange Texte weniger Tipparbeit darstellt. Es ist jedoch ebenenso möglich, jede Zeile mit "# ", also Gatterzeichen gefolgt von einem Abstand, zu beginnen, RDoc versteht üblicherweise beides. Von der Verwendung mehrzeiliger Kommentare mittels =begin rdoc und =end rate ich ab, weil sie sich nicht vernünftig einrücken lassen.

Die einzelnen Abschnitte der Methodendokumentation werden durch Level-2-Überschriften voneinander getrennt und Volltexte sind nur für die Allgemein- und Rückgabewertbeschreibung erforderlich. Parameter und Exceptions lassen sich wesentlich übersichtlicher in Form von RDocs Definitionslisten dokumentieren — hierbei sollte man auf eine korrekte Einrückung der einzelnen Beschreibungstexte achten, andernfalls ist die Lesbarkeit im Quelltext nicht mehr gewährleistet und man ist wieder gezwungen, die HTML-Dokumentation zu erzeugen, was ja gerade vermieden werden soll. Bei langen Parameternamen ist jedoch die zweite Form von RDocs Definitionslisten vorzuziehen, da bei einer empfohlenen Zeilenlänge von maximal 80 Zeichen (länger wird für das Auge anstrengend zu lesen, da man in den Zeilen verrutscht) sonst zu wenig Platz für den Textkörper hat:

1
2
3
4
5
6

#== Parameters
#[thisisaquitelongparametername]
#  Beschreibung dieses Parameters, auch gerne und problemlos
#  mehrzeilig.
#[thisisanotherlongparametername]
#  Beschreibung dieses Parameters

Die beiden Listentypen sollten nicht innerhalb der Dokumentation einer einzelnen Methode vermischt werden.

Hinsichtlich optionalen Parametern mit Standardwerten sollte man den Wert in Klammern zu Beginn der Beschreibung des jeweiligen Parameters angeben. Typschrift für den Standardwert ist nur erforderlich, wenn Missverständnisse entstehen könnten oder ein längerer Ausdruck folgt.

1
2
3
4

#== Parameters
#[foobar] (3) Beschreibung für den Parameter.
#[barbaz] (<tt>sprintf("%30s", MyModule.foobar)</tt>) Beschreibung
#         für den Parameter[/code]

:yields: und :call-seq:

Gern vernachlässigt werden auch Blöcke—dabei gehört es zu den essentiellen Nutzungsinformationen, welche Parameter ein Block erhält. Ganz zu schweigen davon, dass man überhaupt dokumentieren muss, dass eine Methode einen Block erhalten kann, was ja auch der Fall sein kann, wenn der Methodenkörper kein yield enthält und RDoc die Methode nicht automatisch als Blockmethode kennzeichnet. In solchen Fällen ist die RDoc-Direktive :yields: oder gar ein komplettes :call-seq: anzubringen. Ist ein Block gar optional, ist :call-seq: nicht mehr zu vermeiden, weil es mehr als eine Aufrufmöglichkeit gibt.

1
2
3
4
5
6
7
8
9
10
11
12
13

#call-seq:
#  foo(par){|blockpar|...} → an_object
#  foo(par)                → nil
#
#==Parameters
#[par]      ...
#[blockpar] (Blockargument) Beschreibung des Blockparameters.
#.
#.
#.
def foo(par)
  yield(rand(10)) if block_given?
end

Im Bezug auf die Syntax von :call-seq: sei jedem der Blick in RDocs Markup-Referenz empfohlen. Dort sind auch all die anderen Direktiven aufgelistet, die Dokumentation viel verständlicher machen. Darum: Benutzt sie!

:call-seq: erlaubt es, die Aufrufszeile der HTML-Dokumentation beliebig zu verändern. Um das Gesamtbild der Dokumentation jedoch nicht zu beschädigen, sollte man sich weitestgehend an RDocs Standardaussehen halten, Beispielaufrufe etwa gehören hier nicht hinein (dafür gibt es den Examples-Abschnitt). Im Laufe meiner Programmierung sind mir viele Arten von call-seqs begegnet, aber die folgend beschriebene erscheint mir als die gängiste und auch übersichtlichste:

1. Immer Klammern angeben. Auch dann, wenn die Methode keine Parameter entgegennimmt. Grund: RDoc benutzt die Klammern als Begrenzer des Methodennamens, d.h. alles vom Anfang der ersten Zeile bis zur öffnenden Klammer wird in die Methodenübersicht aufgenommen!
2. Optionale Parameter in eckigen Klammern einschließen, Standardwerte dabei mit Gleichheitszeichen angeben (außer, die call-seq würde dadurch übermäßig lang): foo(x [, y = 7 [, z = 9 ] ]). Das Komma wird ebenfalls in die eckigen Klammern eingeschlossen, da es ja ebenfalls optional ist. Eckige Klammern dürfen daher beliebig tief geschachtelt werden.
3. Restargumente als solche kennzeichnen, falls notwendig, ebenfalls in eckige Klammern einschließen. foo(x, *args); foo(x [, *args]).
4. Blockargumente als Block dokumentieren, außer es gäbe einen definitiven Grund dafür, sie als &amp;block zu dokumentieren. foo(){...}, selten foo(&amp;block).
5. Ein als Liste benannter Parameter getarntes Hash als hsh oder opts dokumentieren. Anonsten gelten die Bestimmungen zu eckigen Klammern und Restargumenten. Jedoch sollten die möglichen Parameter in einer Unterliste der Beschreibung des Hashparameters benannt werden.
6. Den Rückgabewert mit einem Unicode-Pfeil → oder einem ASCII-Pfeil ==> angeben (außer der Rückgabewert der Methode ist unwichtig). Dies hat keine Bedeutung für RDoc, erscheint mir aber sinnvoll.
7. Jede Aufrufmöglichkeit eine Zeile. Nicht mehr, und nicht weniger. Methoden mit mehr als vier Parametern gehören ohnehin verbannt.

Konstanten

Konstanten werden kurz und prägnant in einem einzelnen Satz dokumentiert. Der Wert der Konstanten hat nichts in der Dokumentation verloren, er kann vom RDoc-Template angezeigt werden (Hanna etwa tut dies).

1
2

#My cool constant.
COOL = 4

Attribute

Attribute werden im Quelltext getrennt voneinander mithilfe einzelner attr-Anweisungen definiert und dort einzeln dokumentiert.

1
2
3
4
5

#Attributbeschreibung hier.
attr_reader :foo

#Noch eine Attributbeschreibung.
attr_accessor :bar

Ob Readers, Writers oder Accessors zuerst und in welcher Reihenfolge kommen, ist ziemlich egal, nur sollten sie gruppiert werden, also nicht abwechselnd Readers und Accessors oder noch seltsamere Kombinationen. Attributbeschreibungen sollten genau wie Konstantenbeschreibungen kurz gehalten werden.

Pseudo-Attribute und -methoden

Rubys dynamische Fertigkeiten erlauben die Definition von Methoden on-the-fly. Da in Ruby auch Attribute nur Methoden sind, behandle ich beide in einem Abschnitt, es gelten ähnliche Prinzipien.

Fall 1: Einfaches define_method

Damit kommt RDoc inzwischen selbst ganz gut klar, man muss es nur anweisen, das Konstrukt zu dokumentieren:

1
2
3
4
5

##
# Methodendokumentation
define_method(:foo) do
  # ...
end

Man beachte das doppelte Gatter am Beginn der Dokumentation. Dieses weist RDoc an, den nächsten erkannten Codeblock zu dokumentieren. Und obwohl die Zuordnung des Quelltexts problemlos läuft, sollte man eine solche Dokumentation immer mit einem :call-seq: einleiten, da das mit dem Parametern nicht immer ganz so gut läuft.

Fall 2: Mehrfaches define_method

Hier versagen RDocs Fähigkeiten auf eine ganz üble Art und Weise. Im folgenden Code wird nur eine einzige der vielen definierten Methoden dokumentiert und auch nur einen einzigen Dokumentationsblock von allen dreien erhalten:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

##
# Methode 1

##
# Methode 2

##
# Methode 3

3.times do |i|
  define_method(:"foo#{i}") do
    #...
  end
end

Diesem Problem lässt sich nur durch zwei kleine Tricks Abhilfe schaffen. Der erste ist, die logische Verknüpfung zwischen dem Dokumentationsblock und define_method aufzuheben, indem man mindestens eine andere reguläre (und dokumentierte!) Methode dazwischenschiebt. Ich löse das in der Regel so, dass ich meine Pseudo-Methoden nach den Attributen am Klassenanfang dokumentiere.

Der zweite Trick ist die Verwendung der RDoc-Direktive :method: (oder :singleton-method: für Klassenmethoden, :attr_reader:, :attr_writer: und :attr_accessor: für Attribute), die RDoc anweist, den folgenden Dokumentationsblock als Methodenbeschreibung aufzufassen. Im Gegensatz zu den »handelsüblichen« Direktiven sind diese nicht zusammen mit den üblichen Formatieranweisungen dokumentiert, sondern sind Teil der Dokumentation des Ruby-Parsers. Etwas versteckt, nichtsdestotrotz überaus nützlich.

Die Direktiven erwarten als Parameter einen Methoden/Attributsnamen, welchen sie dann in die HTML-Dokumentation einpflegen, als wäre es eine reguläre Methode/ein reguläres Attribut.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

##
# :method: foo0
#First foo.

##
# :method: foo1
#Second foo.

##
# :method: foo2
#Third foo.

#X method.
def x
  #...
end

3.times do |i|
  define_method(:"foo#{i}") do
    # ...
  end
end

In manchen Fällen, insbesondere in denen, in denen man eine Attributmethode wie z.B. foo= händisch definiert, ist eine Kombination mit der :nodoc:-Direktive sinnvoll.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

##
# :attr_accessor: foo
#Foo foo foo.

#See attribute docs.
def foo # :nodoc:
  @foo
end

#See attribute docs.
def foo=(val) # :nodoc:
  raise(ArgumentError, "val < 0!") if val < 0
  @foo = val
end

Ein paar nützliche Links

• RDoc-Markup-Referenz
• Nur im Ruby-Modus verwendbare Direktiven
• Nur im C-Modus (C-Extensions) verwendbare Direktiven

Damit sind denke ich alle wichtigen Themen abgehakt. Also, liebe Leute: Schreibt brauchbare Dokumentationen!

Valete,
Quintus

¹ »Ich hab’ gebellt und gebellt, aber niemand machte auf! Und das, obwohl es Katzen und Hunde regnete!«

Tags: de | Ruby

Amazon's recent release of DynamoDB, a database whose name is inspired by Dynamo, the key-value database the distributed datastore they've been running in production for a good five to six years now. I think it's great they've finally done it, though from my obverservations, there's little resemblance of what the original Dynamo paper describes, but I'm getting ahead of myself. Traditionally Amazon hasn't been very open about how they implement their services, so some of what I'm stating here may be nothing more than an educated guess. Either way, the result is pretty neat.

Time to take a good look at what it has to offer, how that works out in code, and to make some wild guesses as to what's happening under the covers. I'm using fog's master to talk to DynamoDB in the examples below, but the official Ruby SDK also works. Fog is closer to the bare API metal though, which works out well for our purposes.

My goal is not to outline the entire API and its full set of options, but to dig into the bits most interesting to me and to show some examples. A lot of the focus in other posts is on performance, scalability and operational ease. I think that's a great feature of DynamoDB, but it's pretty much the same with all of their web services. So instead I'm focusing on the effects DynamoDB has on you, the user. We'll look at API, general usage, data model and what DynamoDB's feature generally entails.

The Basics

DynamoDB is a distributed database in the cloud, no surprises, it's not the first such thing Amazon has in its portfolio. S3, SimpleDB, RDS and DynamoDB all provide redudant ways to store different types of data in Amazon's datacenters. Currently, DynamoDB only supports the us-east region.

An important difference is that data stored in DynamoDB is officially stored on SSDs, which has (or at least should have) the benefit of offering predictable performance and greatly reduced latency across the board. The question that remains is, of course: when can I hook up SSDs to my EC2 instances?

The other big deal is that the read and write capacity available to you is configurable. You can tell Amazon how much capacity, in read and write units per second, you expect for your application, and they make sure that capacity is available to you as long as you need it (and pay for it).

Data Model

Data in DynamoDB is stored in tables, a logical separation of data if you will. Table names are only unique on a per-user basis, not globally like S3 buckets.

Tables store items, think rows of data or documents. Items have attributes and values, similar to SimpleDB. Think of it as a JSON document where you can read and write attributes independent of the entire document. Every row can have different attributes and values, but every item needs to have a uniquely identifying key whose name you decide on upfront, when you create the table.

Let's create a table first, but make sure you wear protection, as this is yet another of Amazon's gross-looking APIs. The relevant API action is CreateTable.

dynamo = Fog::AWS::DynamoDB.new(aws_access_key_id: "YOUR KEY",
  aws_secret_access_key: "YOUR SECRET")

dynamo.create_table("people",
  {HashKeyElement: {AttributeName: "username", AttributeType: "S"}},
  {ReadCapacityUnits: 5, WriteCapacityUnits: 5})

This creates a table called "people" with an attribute "username" as the indexing key. This is the key you're using to reference a row, sorry, an item, in a table. The key is of type string, hence the S, you can alternatively specify an N for numeric values. This pattern will come up with every attribute and value pair you store, you always need to tell DynamoDB if it's a string or a number. You also specify initial values for expected read and write capacity, where 5 is the minimum.

You have to provide a proper value for a key, DynamoDB doesn't offer any automatic generation of keys for you, so choose them wisely to ensure a good distribution. A key, that has a lot of data behind it, and that's accessed much more frequently than others will not benefit you. Keep your data simple and store it with multiple keys if you can.

Anyhoo, to insert an item, you can use the PutItem API action, to which you issue a HTTP POST (go figure). The DynamoDB API already gives me headache, but more on that later. Thankfully, a good client library hides the terrible interface from us, only leaving us with the hash structures sent to DynamoDB.

dynamo.put_item("people", {username: {"S" => "roidrage"}})

Specify multiple attributes as separate hash entries, each pointing to a separate hash specifying data type and value. I'll leave it to you to think about how backwards this is, though it must be said that JavaScript is also to blame here, not handling 64 bit integers very well.

You can store lists (or rather sets of data) too, using SS as the datatype.

dynamo.put_item("people", {username: {S: "roidrage"},
                               tags: {SS: ["nosql", "cloud"]}})

Note that you can use PutItem on existing items, but you'll always replace all existing attributes and values.

Conflicts and Conditional Writes

All writes can include optional conditions to check before updating an item. That way you can ensure the item you're writing is still in the same state as your local copy of the data. Think: read, update some attribute, then write, expecting and ensuring to not have any conflicting updates from other clients in between.

This is a pretty neat feature, you can base updates on one attributes based on whether another attribute exists or has a certain value.

dynamo.put_item("people",
    {username: {S: "roidrage"}, fullname: {S: "Mathias Meyer"}},
    {Expected: {fullname: {Exists: false}}})

This operation is only executed if the item as currently stored in DynamoDB doesn't have the attribute the attribute fullname yet. On a subsequent PutItem call, you could also check if the item's fullname attribute still has the same value, e.g. when updating the full name of user "roidrage".

dynamo.put_item("people",
    {username: {S: "roidrage"}, fullname: {S: "Mathias Meyer"}},
    {Expected: {fullname: {Value: {S: "Mathias Meyer"}}}})

It's not the greatest syntax for sure, but it does the job. Which brings me to some musings about consistency. If the condition fails, Amazon returns a 400 response, just like with every other possible error you could cause.

To make proper use of conditional updates in your application and to actually prevent conflicting writes and lost updates, you should use the UpdateItem action instead and only update single attributes, as PutItem always replaces the entire item in the database. But even then, make sure to always reference the right attributes in an update. You could even implement some sort of versioning scheme on top of this, for instance to emulate multi-version concurrency control.

Updating single or multiple attributes is done with the UpdateItem action. You can update attributes without affecting others and add new attributes as you see fit. To add a new attribute street, here's the slightly more verbose code.

dynamo.update_item("people", {HashKeyElement: {S: "roidrage"}},
    {street: {Value: {S: "Gabriel-Max-Str. 3"}}})

There are more options involved than with PutItem, and there are several more available for UpdateItem. But they'll have to wait for just a couple of sentences.

Consistency on Writes

Conditional updates are the first hint that DynamoDB is not like Dynamo at all, as they assume some sort of semi-transactional semantics. Wherein a set of nodes agree on a state (the conditional expression) and then all apply the update. The same is true for atomic counters, which we'll look at in just a minute.

From the documentation it's not fully clear how writes without a condition or without an atomic counter increment are handled, or what happens when two clients update the same attribute at the same time, and who wins based on which condition. Facing the user, there is no mechanism to detect conflicting writes. So I can only assume DynamoDB either lets the last write win or has a scheme similar to BigTable, using timestamps for each attribute.

Writes don't allow you to specify something like a quorum, telling DynamoDB how consistent you'd like the write to be, it seems to be up to the system to decide when and how quickly replication to other datacenters is done. Alex Popescu's summary on DynamoDB and Werner Vogels' introduction suggest that writes are replicated across data centers synchronously, but doesn't say to how many. On a wild guess, two data centers would make the write durable enough, leaving the others to asynchronous replication.

Consistency on Reads

For reads on the other hand, you can tell DynamoDB if stale data is acceptable to you, resulting in an eventually consistent read. If you prefer strongly consistent reads, you can specify that on a per-operation basis. What works out well for Amazon is the fact that strongly consistent reads cost twice as much as eventually consistent reads, as more coordination and more I/O are involved. From a strictly calculating view, strongly consistent write take up twice as much capacity as eventually consistent writes.

From this I can only assume that writes, unless conditional will usually be at least partially eventually consistent or at least not immediately spread out to all replicas. On reads on the other hand, you can tell DynamoDB to involve more than just one or two nodes and reconcile outdated replicas before returning it to you.

Reading Data

There's not much to reading single items. The action GetItem allows you to read entire items or select single attributes to read.

dynamo.get_item("people", {HashKeyElement: {S: "roidrage"}})

Optionally, add the consistency option to get strong consistency, with eventual consistency being the default.

dynamo.get_item("people", {HashKeyElement: {S: "roidrage"}},
                          {ConsistentRead: true})

A read without any more options always returns all data, but you can select specific attributes.

dynamo.get_item("people", {HashKeyElement: {S: "roidrage"}},
                          {AttributesToGet: ["street"]})

Atomic Counters

Atomic counters are a pretty big deal. This is what users want out of pretty much every distributed database. Some say they're using it wrong, but I think they're right to ask for it. Distributed counters are hard, but not impossible.

Counters need to be numerical fields, and you can increment and decrement them using the UpdateItem action. To increment a field by a value, specify a numerical attribute, the value to be incremented by, and use the action ADD. Note that the increment value needs to be a string.

dynamo.update_item("people", {HashKeyElement: {S: "roidrage"}},
                             {logins: {Value: {N: "1"}, Action: "ADD"}})

When the counter attribute doesn't exist yet, it's created for you and set to 1. You can also decrement values by specifying a negative value.

dynamo.update_item("people", {HashKeyElement: {S: "roidrage"}},
                             {logins: {Value: {N: "-1"}, Action: "ADD"}})

Can't tell you how cool I think this feature is. Even though we keep telling people that atomic counters in distributed system are hard, as they involve coordination and increase vulnerability to failure of a node involved in an operation, systems like Cassandra and HBase show that it's possible to do.

Storing Data in Sets

Other than numerical (which need to be specified as strings nonetheless) and string data types, you can store sets of both too. One member can only exist once in an attribute set. The neat part is that you can atomically add new members to sets using the UpdateItem action. In the JSON format sent to the server, you always reference even just single members to add as a list of items. Here's an example:

dynamo.update_item("people", {HashKeyElement: {S: "roidrage"}},
                             {tags: {Value: {SS: ["nosql"]}}})

That always replaces the existing attribute though. To add a member you need to specify an action. If the member already exists, nothing happens, but the request still succeeds.

dynamo.update_item("people", {HashKeyElement: {S: "roidrage"}},
                             {tags: {Value: {SS: ["cloud"]}, Action: "ADD"}})

You can delete elements in a similar fashion, by using the action DELETE.

dynamo.update_item("people", {HashKeyElement: {S: "roidrage"}},
                             {tags: {Value: {SS: ["cloud"]}, Action: "DELETE"}})

The default action is PUT, which replaces the listed attributes with the values specified. If you run PUT or ADD on a key that doesn't exist yet, the item will be automatically created. This little feature and handling of sets and counters in general sounds a lot like things that made MongoDB so popular, as you can atomically push items onto lists and increment counters.

This is another feature that's got me thinking about whether DynamoDB even includes the tiniest bit of the original Dynamo. You could model counters and sets based on something like Dynamo for sure, based on the ideas behind Commutative Replicated Data Types. But I do wonder if Amazon actually did go through all the trouble building that system on top of the traditional Dynamo, or if they implemented something entirely new for this purpose. There is no doubt that operations like these is what a lot of users want even from distributed databases, so either way, they've clearly hit a nerve.

Column Store in Disguise?

The fact that you can fetch and update single attributes with consistency guarantees makes me think that DynamoDB is actually more like a wide column store like Cassandra, HBase or, gasp, Google's BigTable. There doesn't seem to be anything left from the original, content-agnostic idea of the Dynamo data store, whose name DynamoDB so nicely piggybacks on.

The bottom line is there's always a schema of attributes and values for a particular item you store. What you store in an attribute is up to you, but there's a limit of 64KB per item.

DynamoDB assumes there's always some structure in the data you're storing. If you need something content-agnostic to store data but with similar benefits (replication, redundancy, fault-tolerance), use S3 and CloudFront. Nothing's keeping you from using several services obviously. If I used DynamoDB for something it'd probably not be my main datastore, but an add-on for data I need to have highly available and stored in a fault-tolerant fashion, but that's a matter of taste.

A Word on Throughput Capacity

Whereas you had to dynamically add capacity in self-hosting database systems, always keeping an eye on current capacity limits, you can add more capacity to handle more DynamoDB request per second simply by issuing an API call.

Higher capacity means paying more. To save money, You could even adjust the capacity for a specific time of day basis, growing up and down with your traffic. If you go beyond your configured throughput capacity, operations may be throttled.

Throughput capacity is based on size of items you read and the number of reads and writes per second. Reading one item with a size of 1 KB corresponds to one read capacity unit. Add more required capacity units with increased size and number of operations per second.

This is the metric you always need to keep an eye on and constantly measure from your app, not just to validate invoice Amazon sends you at the end of the month, but also to track your capacity needs at all times. Luckily you can track this using CloudWatch and trigger alerts. Amazon can trigger predefined alerts when capacity crosses a certain threshold. Plus, every response to a read includes the consumed read capacity, and the same is true for writes.

Throughput capacity pricing is pretty ingenious on Amazon's end. You pay for what you reserve, not for what you actually use. As you always have to have more capacity available as you currently need, you always need to reserve more in DynamoDB. But if you think about it, this is exactly how you'd work with your own hosted, distributed database. You'll never want to work very close to capacity, unless you're some crazy person.

Of course you can only scale down throughput capacity once per day, but scale up as much as you like, and increases need to be done at least 10%. I applaud you for exploiting every possible opportunity to make money, Amazon!

Data Partitioning

Amazon's documentation suggests that Amazon doesn't use random partitioning to spread data across partitions, partitioning is instead done per table. Partitions are created as you add data and as you increase capacity, suggesting either some sort of composite key scheme or a tablet like partitioning scheme, again, similar to what HBase or BigTable do. This is more fiction than fact, it's an assumption on my part. A tablet-like approach certainly would make distributing and splitting up partitions easier than having a fixed partitioning scheme upfront like in Dynamo.

The odd part is that Amazon actually makes you worry about partitions but doesn't seem to offer any way of telling you about them or how your data is partitioned. Amazon seems to handle partitioning mostly automatically and increases capacity by spreading out your data across more partitions as you scale capacity demand up.

Range Keys

Keys can be single values or based on a key-attribute combination. This is a pretty big deal as it effectively gives you composite keys in a distributed database, think Cassandra's data model. This effectively gives you a time series database in the cloud, allowing you to store sorted data.

You can specify a secondary key on which you can query by a range, and which Amazon automatically indexes for you. This is yet another feature that makes DynamoDB closer to a wide column store than the traditional Dynamo data store.

The value of the range key could be an increasing counter (though you'd have to take care of this yourself), a timestamp, or a time based UUID. Of course it could be anything else and unique entirely, but time series data is just a nice example for range keys. The neat part is that this way you can extract data for specific time ranges, e.g. for logging or activity feeds.

We already looked at how you define a normal hash key, let's look at an example with a more complex key combining a hash key and a range key, using a numerical type to denote a timestamp.

dynamo.create_table("activities", {
      HashKeyElement: {AttributeName: "username", AttributeType: "S"},
      RangeKeyElement: {AttributeName: "created_at", AttributeType: "N"}},
    {ReadCapacityUnits: 5, WriteCapacityUnits: 5})

Now you can insert new items based on both the hash key and a range key. Just specify at least both attributes in a request.

dynamo.put_item("activities", {
    username: {S: "roidrage"}, 
    created_at: {N: Time.now.tv_sec.to_s},
    activity: {S: "Logged in"}})

The idea is simple, it's a timestamp-based activity feed per user, indexed by the time of the activity. Using the Query action, we can fetch a range of activities, e.g. for the last 24 hours. Just using get_item, you always have to specify a specific combination of hash and range key.

To fetch a range, I'll have to resort to using Amazon's Ruby SDK, as fog hasn't implemented the Query action yet. That way you won't see the dirty API stuff for now, but maybe that's a good thing.

dynamo = AWS::DynamoDB.new(access_key_id: "YOUR KEY",
                           secret_access_key: "YOUR SECRET")
activites = dynamo["activities"]

items = activities.items.query(
    hash_key: "roidrage",
    range_greater_than: (Time.now - 85600).tv_sec)

This fetches all activity items for the last 24 hours. You can also fetch more historic items by specifying ranges. This example fetches all items for the last seven days.

items = activities.items.query(
    hash_key: "roidrage",
    range_greater_than: (Time.now - 7.days.ago).tv_sec,
    range_less_than: Time.now.tv_sec)

Note that the Query action is only available for tables with composite keys. If you don't specify a range key, DynamoDB will return all items matching the hash key.

Queries using Filters

The only things that's missing now is a way to do richer queries, which DynamoDB offers by way of the Scan action. fog doesn't have an implementation for this yet, so we once again turn to the AWS Ruby SDK.

Scanning allows you to specify a filter, which can be an arbitrary number of attribute and value matches. Once again the Ruby SDK abstracts the ugliness of the API underneath into something more readable.

activities.items.where(:activity).begins_with("Logged").each do |item|
  p item.attributes.to_h
end

You can include more than one attribute and build arbitrarily complex queries, in this example to fetch only items related to the user "roidrage" and him logging in.

activities.items.where(:activity).equals("Logged in").
          and(:username).equals("roidrage").each do |item|
  p item.attributes.to_h
end

You can query for ranges as well, combining the above with getting only items for the last seven days.

activities.items.where(:activity).equals("Logged in").
    and(:username).equals("mathias").
    and(:created_at).between(7.days.ago.tv_sec, Time.now.tv_sec).each {|item|
  p item.attributes.to_h
end

Filters fetch a maximum of 1 MB of data. As soon as that's accumulated, the scan returns, but also includes the last item evaluated, so you can continue where the query left off. Interestingly, you also get the number of items remaining. Something like paginating results dynamically comes to mind. Running a filter is very similar to running a full table scan, though it's rather efficient thanks to SSDs. But don't expect single digit millisecond responses here. As scans heavily affect your capacity throughput, you're better off resorting to their use only for ad-hoc queries, not as a general part of your application's workflow.

Unlike Query and GetItem, the Scan action doesn't guarantee strong consistency, and there's no flag to explicitly request it.

The API

DynamoDB's HTTP API has got to be the worst ever built at Amazon, you could even think it's the first API every designed at Amazon. You do POST requests even to GET data. The request and response can include conditions and validations and their respective errors, the proper Java class name of an error and other crap. Not to mention that every error caused by any kind of input on your end always causes a 400.

Here's an example of a simplified request header:

POST / HTTP/1.1
Content-Type: application/x-amz-json-1.0
x-amz-target: DynamoDB_20111205.GetItem

And here's a response body from an erroneous request:

{"__type":"com.amazon.coral.validate#ValidationException",
"message":"One or more parameter values were invalid:
The provided key size does not match with that of the schema"}

Lovely! At least there's a proper error message, though it's not always telling you what the real error is. Given that Amazon's documentation is still filled with syntactical errors, this is a bit inconvenient.

The API is some bastard child of SOAP, but with JSON instead of XML. Even using a pretty low level library like fog doesn't hide all the crap from you. Which worked out well in this case, as you see enough of the API to get an idea about its basic workings.

The code examples above don't read very Ruby like as I'm sure you'll agree. Though I gotta say, the Ruby SDK provided by AWS feels a lot more like Ruby in its interface.

I don't have very high hopes to see improvements on Amazon's end, but who knows. S3, for example, got a pretty decent REST API eventually.

Pricing

Pricing is done (aside from throughput capacity) per GB stored. If you store millions of items, and they all exceed size of just a few KB, expect to pay Amazon a ton of money, storage pricing trumps throughput pricing by a lot. Keep data stored in an item small. If you only store a few large items, it works too, but you may end up being better off choosing one of Amazon's other storage options. You do the math. Pricing for storage and the maximum size for a single item always includes attribute names, just like with SimpleDB.

To give you an idea how pricing works out, here's a simple calculation. 100 GB of data, 1000 reads per second, 200 writes per second, item size is 4 KB on average. That's $1253.15 every month, not including traffic. Add 10 GB of data the second month and you're at $1263.15. You get the idea. Pricing is much more affected by read and write operations vs. item size. Make your items 6 KB in size, and you're already at $1827.68.

Bottom Line

Though Amazon is doing a pretty good job at squeezing the last drop of money out of their customers using DynamoDB, think about what you're getting in return. No operations, no hosting facilities, let alone in three different datacenters, conditional writes and atomic counters, and a database that (I assume) has years of experience in production forged into it.

As usually the case with Amazon's Web Services, using something like DynamoDB is a financial tradeoff. You get a data store that scales up and down with your demand without you having to worry about the details of hardware, operations, replication and data performance. In turn you pay for every aspect of the system. The price for storage is likely to go down eventually, making it a more attractive alternative to hosting an open source NoSQL database system yourself. Whether this is an option for your specific use case, only you're able to make that decision.

If you store terrabytes of data, and that data is worth tens of thousands of dollars per month in return for not having to care about hosting, by all means, go for DynamoDB. But at that size, just one or two months of hosting on Amazon pays off buying servers and SSDs for several data centers. That obviously doesn't cover operational costs and personell, but it's just something to think about.

Closing Thoughts

Sorted range keys, conditional updates, atomic counters, structured data and multi-valued data types, fetching and updating single attributes, strong consistency, and no explicit way to handle and resolve conflicts other than conditions. A lot of features DynamoDB has to offer remind me of everything that's great about wide column stores like Cassandra, but even more so of HBase. This is great in my opinion, as Dynamo would probably not be well-suited for a customer-facing system. And indeed, Werner Vogel's post on DynamoDB seems to suggest DynamoDB is a bastard child of Dynamo and SimpleDB, though with lots of sugar sprinkled on top.

Note that it's certainly possible and may actually be the case that Amazon has built all of the above on top of the basic Dynamo ingredients, Cassandra living proof that it's possible. But if Amazon did reuse a lot of the existing Dynamo code base, they hid it really well. All the evidence points to at least heavy usage of a sorted storage system under the covers, which works very well with SSDs, as they make sequential writes and reads nice and fast.

No matter what it is, Amazon has done something pretty great here. They hide most of the complexity of a distributed system from the user. The only option you as a user worry about is whether or not you prefer strong consistency. No quorum, no thinking about just how consistent you want a specific write or read operation to be.

I'm looking forward to seeing how DynamoDB evolves. Only time will tell how big of an impact Amazon's entering the NoSQL market is going to have. Give it a whirl to find out more about it.

Want to know how the original Dynamo system works? Have a look at the Riak Handbook, a comprehensive guide to Riak a distributed database that implements the ideas of Dynamo and adds lots of sugar on top.

I'm happy to be proven wrong or told otherwise about some of my assumptions here, so feel free to get in touch!

Resources

Be sure to read Werner Vogels' announcement of DynamoDB, and Adrian Cockcroft's comments have some good insights on the evolution of data storage at Netflix and how Cassandra, SimpleDB and DynamoDB compare.

• DynamoDB
• DynamoDB FAQs
• Werner Vogels: Amazon DynamoDB - A Fast and Scalable NoSQL Database
• Amazon DynamoDB - Internet-Scale Data Storage the NoSQL Way
• Adrian Cockcroft: Thoughts on SimpleDB, DynamoDB and Cassandra
• AWS SDK for Ruby
• Alex Popescu: Notes about Amazon DynamoDB
• DynamoDB Developer Guide
• Dynamo, a Node.js library for DynamoDB
• dinerl, an Erlang client for DynamoDB

Here’s a little Rails tip for splitting up seed data for various environments: Create the folder db/seeds/. For each environment you want to seed put a file in there named after the env. Your general seeds are put into db/seeds/all.rb. Here’s what it might look like:

|___seeds
              | |___all.rb
              | |___development.rb
              | |___staging.rb
              | |___production.rb
              |___seeds.rb
              

Change the content of your db/seeds.rb to something like this:

['all', Rails.env].each do |seed|
                seed_file = "#{Rails.root}/db/seeds/#{seed}.rb"
                if File.exists?(seed_file)
                  puts "*** Loading #{seed} seed data"
                  require seed_file
                end
              end
              

Running rake db:seed will now load up the seed data defined in db/seeds/all.rb and the current environment.

Man nehme folgendes, vereinfachtes Model:

1
2
3

class Blog::Post < ActiveRecord::Base
  validates :title, :presence => true
end

Ich habe gestern den ganzen Tag versucht, dieses Model zu lokalisieren, leider verweigerte mir I18n stur die Zusammenarbeit. Folgender YAML-Code gibt eine Deprecation-Warning:

1
2
3
4
5
6
7
8
9
10

de:
  #...
  activerecord:
    models:
      blog:
        post: Eintrag
    attributes:
      blog:
        post:
          title: Titel

[DEPRECATION WARNING] Nested I18n namespace lookup under “activerecord.attributes.blog” is no longer supported

Stimmt auch ganz offensichtlich.

1
2
3
4
5
6

irb(main):001:0> I18n.locale = :de
=> :de
irb(main):002:0> Blog::Post.model_name.human
=> "Post"
irb(main):003:0> Blog::Post.human_attribute_name "title"
=> "Title"

Leider führt folgendes YAML-Code zum selben Ergebnis:

1
2
3
4
5
6
7
8

de:
  #...
  activerecord:
    models:
      post: Eintrag
    attributes:
      post:
        title: Titel

1
2
3
4
5
6

irb(main):001:0> I18n.locale = :de
=> :de
irb(main):002:0> Blog::Post.model_name.human
=> "Post"
irb(main):003:0> Blog::Post.human_attribute_name "title"
=> "Title"

Nach einiger Sucherei habe ich heute endlich die richtige Schlagwortkombination gefunden, die mich nach Stackoverflow verbachte. Demnach ist die einzig richtige und funktionierende Art und Weise, Übersetzungen für Models in Namespaces zu erstellen, Model und Namespace durch einen Slash / zu trennen (hehe, da ist wohl ein Bug in CodeRay):

1
2
3
4
5
6
7
8

de:
  #...
  activerecord:
    models:
      blog/post: Eintrag
    attributes:
      blog/post:
        title: Titel

1
2
3
4
5
6
7

irb(main):001:0> I18n.locale = :de
=> :de
irb(main):002:0> Blog::Post.model_name.human
=> "Eintrag"
irb(main):003:0> Blog::Post.human_attribute_name "title"
=> "Titel"
irb(main):004:0>

Jaaa, so ist es richtig.

Valete,
Quintus

Tags: de | Ruby

• stores your passwords in a file on your disk
• encrypts the file with a master password
• is designed for every-day-use
• is written in 234 lines of understandable Ruby code… Read it!
• is tested with 222 Cucumber steps

This announcement is actually a follow-up of my previous tutorial on how to build a password safe in ruby. Unfortunately, I was not using it and did not update it for quite a while, sorry for that.. But I finally managed to refactor it, so now it comes with specs, colors, new features and thousands of other tweaks and improvements :D

Impression / Usage

Install

Using a Ruby 1.9, do

Github: github.com/janlelis/pws

Usage tips

I’ve got the following lines in my ~./bashrc

export PWS="$HOME/.safe/pws"
alias pw='pws'

The first line changes the password file location. In that ~/.safe folder, I actually use git to backup some of my pws files.

The second line allows a handy syntax for getting a password:

Update: Added paragraph on the e-commerce platform I'm using.

Several people have asked me about the publishing tool chain I'm using for the Riak Handbook and the NoSQL Handbook. As Avdi just published his, now's as good a good time as any to throw in mine too. After all, it's always nice to know all the options you have.

All text is written in Markdown. I started out with Textile, but just didn't like Textile's way of breaking down single lines into separate paragraphs, so I switched to Markdown later on. Not a painless process, but sufficiently easy thanks to Pandoc.

The Markdown files, one per chapter, are bundled together into a big file and then fed into GitHub's Redcarpet library. I use Albino and Pygments for syntax highlighting, embedded into a custom renderer for Redcarpet. See the Redcarpet README for an example. Redcarpet also has some MultiMarkdown extensions for tables and other things, so that comes in pretty handy.

To generate the PDF I'm using Prince, a neat little tool that takes HTML and a bunch of CSS and generates a nice PDF. It's a commercial tool, and costs some $500 for a single license. That was what threw me off a tiny bit, but luckily there's a handy service called DocRaptor which is basically a pretty API around Prince, but much much cheaper.

You can use Prince locally to generate infinite test PDFs, and then use DocRaptor to generate the final result. Which is what I do. Before generating the PDF through DocRaptor I upload the generated HTML, which includes all the CSS, to S3, to avoid some issues I've had with Unicode characters in the HTML, and tell DocRaptor to fetch it from a URL instead.

You can use web fonts easily, I went for fonts from Google Web Fonts, which are easily embeddable with Prince and DocRaptor. Prince also allows you to do some customizations specific, stuff like page titles, page numbers, custom settings for left and right pages, footnotes, and the like. See the documentation for a full list.

For ePub generation I generate a stripped-down version of the HTML, not including syntax highlighting, because iBooks doesn't like custom-styled pre or code elements, if you style them iBooks chooses to ignore any font setting and use the default font, which is usually not monospaced.

The HTML then goes through Calibre to generate the ePub file, and then I use kindlegen to generate a Kindle file from that. Calibre has a pretty ugly user interface, but it comes with a set of command line tools you can use to automatically convert ebooks from one format to another.

All of the above is wrapped into a Rakefile and some small Ruby classes. I'll be sure to put it up for your perusal soon, but this list should get you started.

To make the book available for purchase I'm using FastSpring. They're not my favorite choice, as they make things like customizing your shop fairly hard, but I chose them because their way of payout (monthly or bi-monthly), and the fact that they take care of VAT, were both reasons important to me, especially being in Europe, where tax laws throw a lot of stones at you for wanting to try and sell things internationally. Unfortunately FastSpring doesn't have any means of updating products and allowing and notifying users of the updates being available, so I built a small Sinatra app to capture orders and take care of sending out the updates. Not pretty, but it's a small sacrifice compared to the tax hassle I'd have with any other fulfillment provider.

Curious about the result? You should buy the book!

Happy New Year to all my fellow readers! I hope you had a good party!

In this post I wanna clarify why representers in Roar are much more versatile than one might think at first sight.

Model != Resource

I know, Rails teaches us that every resource is a model with exactly one representation. This is wrong. Not only can a resource be just anything, ranging from simple 1-to-1 mappings of database rows to virtual models like a workflow but also can your business models be represented in completely different contexts.

Let’s get back to our fruity example to learn more about this.

orange = Fruit.find_by_title("Orange")
orange.extend(FruitRepresenter).to_json
#=> {"title":"Orange","colors":["orange"]}

In the last discussion we had a Fruit and a Bowl model which were represented in their respective resources. While the fruit just contains some properties the bowl was a composition thereof. This is pretty “rails-conform” (I use this word with a negative undertone) and can simply be abstracted to hundreds of projects where people expose users, blog posts or events in their resources. This is what we call a 1-to-1 mapping of model to resource.

Let’s have some Booze!

We wrote a simple FruitRepresenter for mapping apples, lemons or oranges to documents, and we had the BowlRepresenter to provide documents for fruit collections. That was nice. But you can do more with fruits. Why not open a distillery and make some good booze from pears or grapes?

What that means for our REST API is that we have to represent fruits as ingredients. We also want to document what fruits are contained in a bottle of schnaps.

module IngredientRepresenter
  include Roar::Representer::JSON
 
  property :title
  collection :tastes_like
end

The new IngredientRepresenter looks a bit similar to our FruitRepresenter, however, it represents a fruit in another context.

pear = Fruit.find_by_title("Pear")
pear.extend(IngredientRepresenter).to_json
#=> undefined method `tastes_like'

An exception. While the pear exposes a #title method it does not have a #tastes_like method. Why did I do that? I wanted to emphasize that the representer doesn’t guess anything from the represented object (the pear). It is up to the fruit to provide decent accessors like the #tastes_like reader. No magic or implicit semantics in Roar. And this is a good thing.

Another Context, Another Module.

We could easily push the missing methods into the IngredientRepresenter itself, but we will learn in an upcoming post that it is better to provide additional accessors in a separate module.

module Ingredient
  def tastes_like
    TastesLike.query(title) # returns array.
  end
 
  def tastes_like=(*)
  end
end

In the reader, we query the world-famous TastesLike™ web service. Since we don’t need a writer, the second method ain’t doing nothing.

pear.extend(Ingredient, IngredientRepresenter).to_json
#=> {"title":"Pear","tastes_like":["pear","mango"]}

Cool, this is a fruit represented in a completely different context. What about the parsing, does that work, too?

apple = Fruit.new
apple.extend(Ingredient, IngredientRepresenter)
apple.from_json("{\"title\":\"Apple\",
  \"tastes_like\":[\"Pear\",\"Apple\",\"Mango\"]}")
#=> #

Yepp, parsing an ingredient and mapping it to a fruit also works fine.

Fruits in a Bottle.

Let’s see how collections can be represented in different contexts.

module BottleRepresenter
  include Roar::Representer::JSON
 
  collection :fruits, :class => Fruit, 
    :extend => [Ingredient, IngredientRepresenter]
end

The bottle representer knows that it’s composed of Fruit instances. It does use the IngredientRepresenter for parsing and rendering, though. BTW, if you don’t like the long :extend assignment you are free to “hard-include” the modules directly in your models.

Summary

So what we’re doing here is having one model used in two completely different contexts: the conventional “fruit” context and the “ingredient” context. Imagine you’d be doing this with #as_json and different hash-parsing algorithms in your controllers – good luck.

This is my first jruby code (written for the clipboard gem) and I was quite surprised that you only need one-liners ;)

Listing 1

/59/jruby_clipboard_oneliners.rb

def copy(text)
  Java::JavaAwt::Toolkit.default_toolkit.system_clipboard.set_contents(Java::JavaAwtDatatransfer::StringSelection.new(text), nil)
end

def paste
  Java::JavaAwt::Toolkit.default_toolkit.system_clipboard.get_data(Java::JavaAwtDatatransfer::DataFlavor.stringFlavor)
end

There’s nothing like giving a quick update about Cells on a rainy sunday evening. Almost a year ago we were happy to introduce state-args into Cells. State-args help to prevent saving state in instance variables and instead pass variables from #render_cell as arguments, making your code more explicit.

One big problem was that filters couldn’t take advantage of that feature, thanks to a design flaw in Rails’ callback implementation.

Example?

Consider the following call to render a freaky bassist cell.

render_cell :bassist, :play, "C-sharp", "4/4"

The cell class wants to process the incoming data in a before_filter.

class BassistCell  Cell::Rails
  include Cell::Filters
 
  before_filter :prepare
 
  def prepare(state, tone, timing)
    @before = "In #{tone} and #{timing}"
  end

What now works with the new cells-filters gem is that filters (here, a before filter) can receive the state-args. The implementation is very simple using the hooks gem for easy peasy hooks and callbacks in Ruby.

However, we introduce a dependency to another gem, that’s why cells-filters was released as a separate gem. Go and try it! We might need to add some more behaviour to it, but please don’t ask me to completely bloat it.

You can still use the old-school filters from the Rails Callbacks module but you won’t be able to retrieve state-args. I don’t feel like fixing this in the Rails core itself as the implementation is a bit too complex for me.

Cells Development and a Roadmap.

Dudes, Cells is getting more and more of a standard in Rails development. In the last year, 2011, we had almost 90.000 downloads – this is awesome! Thank you so much.

We have more plans with Cells for 2012, here’s a short summary what we’re planning.

• Get rid of the Rails dependency at all. We already got Cell::Base to use cells without a controller. Now we want to go one step further and make cells completely decoupled from Rails. Imagine cells (and, numerous “helper” gems like formtastic) being renderable in Sinatra or without a framework at all.
• Use tilt as an alternative rendering engine. The Rails rendering layer is outdated and should use an abstracting rendering gem like tilt – until we can achieve this in the core, we will provide this in cells. That’ll make huge parts of cells completely independent from Rails.

• Screencasts! Yeah, I’m working on a screencast series about cells, best practices and interesting solutions users sent in over the recent years.

Beside that, we’re permanently trying to incorporate the cells architecture into the Rails core. Let’s see where this will lead to in Rails 4.0. Happy new year, and… Cheers!

This blogpost had a loooong, long way to go, and I’m immensely glad I am finally able to write it, but also quite anxious about it - Not about the post as such, but about the implications of my acts that lead up to this post.

I am relatively risk-averse. I think many of us are, and I think we germans are especially good at it. Change is frowned upon.

That being said, sometimes change is just inevitable. For me, it has been since at least the before-last end-of-year, where I almost broke up with my girlfriend and, by the time the year was done, both of us lived, after 7 years of living together, in our own flats again. We’re happy together, with the usual highs and lows, but the separate living spaces give us shelter when things are rough, which, you know, also inevitably happens.

So, that’s change on a totally personal level and that’s totally not what I was going to write about - Nevertheless, with someone who doesn’t draw very distinct lines between personal and professional life, it plays an important part.

From that time on, a lot of things started to happen at once. First of all, although I loved (and still do) working at mindmatters, I started to look for alternatives - I wanted to, so to speak, level-up my work life. Which is actually not that easy, especially at an awesome place like mindmatters. So, I only have had two valid ideas back then: Founding a startup (and maybe applying for something like HACKFWD) or going freelance. Now, I’ve been doing freelance work before (during my studies) and I always hated the overhead of writing invoices, making sure that people pay me, etc, etc. Also, what we do at mindmatters is mostly client work, and going freelance would basically mean doing the same thing, with less security, less awesome colleagues and bosses, less project management support and slightly higher pay.

Then Florian, my colleague and foodie inspiration number one, convinced our founders (at mindmatters, that is) to open an office in london - I won’t go into the details why we think that this is a brilliant idea - that’s a whole different blog post and not one I should write, but of course I was keen to look at it and in the beginning I was quite sure that I would end up in London at the beginning of 2012.

Then I spent my summer vacation in San Francisco, 3 weeks of awesomeness in the Haight and in the Mission (Home and Work) and of course I fell in love with it. I was hoping to be able to write a blog post about SF as well, but somehow I never managed. Well, I hope to be back this year for a few weeks - We’ll see how that goes. What SF did to me, though, was giving me thoughts on a pretty much existential scale. What I learned there was that there are a lot of awesome places to live all around the world. But what I also learned there is how much I love my little, totally behind and totally mismanaged, not-having-a-super-awesome-tech-scene, Hamburg. My hometown. I’ve been born here, in a blue-collar neighbourhood called Barmbek before my parents and me moved to what people probably would call the outskirts of northeastern Hamburg. I basically lived there until 2010, when I moved much closer to the city centre, to lovely Eimsbüttel. Ironically, as a side note, my girlfriend now lives a few blocks away from the place my parents and my grandmother (who died in 2011, may Jutta Krutisch rest in peace) and I lived until the late 70s.

This feeling of home was, unfortunately, only amplified when I spent three weeks in London after coming back from SF, working on a project there for mindmatters, so-to-speak paving the way for mindmatters.co.uk and getting a good feeling for how it is to live there. Don’t get me wrong, I love London. I love the food options as much as Florian (although not as fiercely, maybe), I love the vibrant tech scene, I love the openness of an english pub, I love that I instantly found a place that sells me american IPA’s, an addiction I formed in SF, I love all of that.

But I also very much love the quietness of Hamburg. And I started to hate everything that’s connected to Air traffic. The easyJet connection to London is okay-ish and bearable if everything’s fine. One little thing and it turns into this nightmarish hour long wait at the gate which the converts into a situation where the BorisBike from London Bridge to Kings Cross (where I was staying at the time) is actually the sane way to go home an 1:30 on a monday morning.

So even if I am able to tuck away my environmentalist heart (which bled quite heavily in 2011 due to all the flights I took) for a second, moving to london would have meant either giving up my relationship (which, as I wrote, just recovered from what I would call an existential crisis) or spending way too much time and nerves on my carbondioxide budget.

Additionally to that, having a long-distance relationship may be fine, as long as you are already rooted in the place you live, but having one from a city where you actually just moved to also probably prevents you from actually growing any roots in that new city.

In the end, I wasn’t able to make any clear commitments to Florian and the mindmatters peeps that I would go over with him in the beginning of 2012.

What I had sworn to myself, though, somewhere in the process, was that I would not simply go back to business as usual when not going to London, but finally, finally, make sure that I also change course professionally.

And that’s what’s going to happen with me, starting Feb 2012. I’ll reduce my work for mindmatters to 20% of a normal contract (meaning about a day a week on average) and I will start freelancing for the rest of the time.

So why stay on the 20%? First of all, I love mindmatters. I am one of their first full time employees, starting in 2007 (which also means that I will be able to celebrate my 5 years of mindmatters in April, something I very much look forward to). I think the way Frank and Wolfgang steer their little company sets an example on transparency, accountability and humanity. This sounds huge, but I mean it. Most companies I know of don’t openly discuss their business goals, even their business model, their finance status and various other things openly in a company meeting every four weeks. Most companies I know would not work in a way where I as an employee can come up to my CEO and say “BTW, I’m planning to leave over the course of the next half a year, let’s talk how we can make this as smooth as possible”. Don’t get me wrong, it’s not all flowers and fluffy pink clouds in mindmatters-land - There are disagreements, there are arguments, there are misunderstandings, heck, after all, this is not a democratic parliament, this is a privately held company with business- and other goals. But I have never, ever before seen owners try so hard to keep up their vision (even through hard times and tough winds) on how to run a company and realign this vision with the help of every employee over and over again, iterating and iterating, and thus becoming, if the word would not be almost meaningless by 2012, a truly agile company.

So, I want to be part of that for as long as possible. I want to play my own part in keeping up that spirit, helping with my experience, my wit and my bad puns.

Regarding my work as a freelancer, although I have not signed any contracts by now, I am most probably already booked out for 2-3 days per week in the first quarter, which should at least give me a base on which to ponder what to do next.

It is actually not my goal to spend the rest of 2012 doing freelance work, but to try out various ideas, of some of which you hopefully will hear about “very soon now”, although, if you have something interesting for me, hit me with it, because I can’t look much further than a few months right now and actually everything beyond the start of February is still quite cloudy right now.

Which gives me the creeps, but also makes me quite happy. I am, for once, completely on my own, without being alone in it.

Happy new year, everyone

No need to check in normal code, just do everything as usual and this interceptor will prevent you from spamming people who do not want any emails.
Its not perfect(missing bc/bcc filters but if should be fine for 90% of cases)

#config/initializers/blacklisted_emails.rb
class MailInterceptor
  def self.delivering_email(message)
    if User.where(:email => message.to, :receive_emails => false).any?
      message.perform_deliveries = false
    end
  end
end

Mail.register_interceptor(MailInterceptor)

Here's a list of things I've been reading lately or that I'm about to read, and that I found to be worth sharing. If you're looking for something to read over the holidays, I'm happy to give you some suggestions. Books, papers, articles, and videos, something for everyone.

Scalability Rules

A list of 50 rules related to scalability, in an easy to read recipe style. They leave some stuff to the imagination, and I don't agree with every single rule, especially not with the one that demands software should always be easy to rollback, but they give you good food for thought for your own applications.

Time, Clocks, and the Ordering of Events in a Distributed System

The earliest paper (1978!) to mention the notion of clocks as a means to track ordering of events in distributed systems, the predecessor to vector clocks, if you will. A must read.

Harvest, Yield, and Scalable Tolerant Systems

A recap of CAP, making the whole notion of it a bit more flexible by adding tuning knobs for graceful degradation. Hat tip to Coda Hale and his article "You can't sacrifice partition tolerance" for pointing me to this.

Problems with CAP, and Yahoo's little known NoSQL system

Also related to CAP, this article introduces the notion of PACELC, which basically adds latency to the CAP equation. CAP has been criticized quite a few times for being too strict in this regard, and while the name PACELC is a bit odd, the added notion of latency makes a lot of sense.

Replication and the latency-consistency tradeoff

Another one from Daniel Abadi, another one related to CAP, this time talking about replication, consistency, and latency.

It's the latency, stupid!

Going further back in time, this paper talks about latency in all its glory. Sure, it talks about modem speed connections, but extrapolate that into today's network bandwidth and you still have latency. Or you can read the next posts too.

It's Still The Latency, Stupid...pt. 1 and It's Still The Latency, Stupid...pt. 2

A more recent update on latency, because it still matters more than bandwidth.

Crash-only Software

I've been pondering fault-tolerant and cloud-ready systems for a while now, here's one related to the topic, software that crashes as a means to make it more fault-tolerant.

Systems that Never Stop (video)

Great talk by Joe Armstrong, inventor of Erlang, laying down six laws for fault-tolerant systems. All laws lead to Erlang obviously, but it all makes a lot of sense.

Why Do Computers Stop and What Can Be Done About It?

Related to Joe's talk, this paper discusses hardware reduncancy and reliable storage by means of process pairs, modularity and transactions. Have yet to read this one, but going to be interesting thinking about how these ideas, stemming from hardware, apply to software and have been implemented by Erlang.

Working With Unix Processes

A little indie-published ebook on handling Unix processes. Code is focused on Ruby, but most if not all of the book is easily applicable to any other language or a basic Unix environment.

SEDA: An Architecture for Well-Conditioned, Scalable Internet Services

SEDA was an idea for web and application server concurrency based on using queues to condition and handle requests. While the idea has not exactly made it through, I found the model to be strikingly similar to the actor model, in a different way, but still very similar.

A Retrospective on SEDA

SEDA, ten years later, by the author of the original paper. I gotta say, he talks a lot what they got wrong, but I for one think SEDA had a pretty big impact on the bigger picture of web application architecture. Probably something worth discussing in a separate post.

Why Events Are A Bad Idea

A paper comparing threads and events for highly concurrent servers. I'd recommend taking this with a grain of salt. A lot has changed since this paper was written, but what I like about reading papers like this is that it gives you a historic perspective, same for SEDA.

Understanding Virtual Memory

Nice summary of how virtual memory works on Linux.

The Declarative Imperative: Experiences and Conjectures in Distributed Logic

To be honest, this is a slightly confusing paper. It starts out modeling things in an oscure language called Datalog, but then dives into making some conjectures about distributed logic, which was to me the more interesting part.

A brief history of Consensus, 2PC and Transaction Commit

This article is full of gold. An extraordinarily compact view on the topic, but with an abundance of links to papers to dive deeper.

Going to keep posting reading lists like this in the future. So much good stuff to read out there. Lots of great knowledge collected in papers.

Last but not least, why not add the Riak Handbook to your reading list as well?

Those of you having asked for mountable cells will be happy with the recent 3.8.0 release! We entirely removed the ActionController dependency. That essentially means you can have cells outside of the standard Rails stack. Cool, heh?

Cells 3.8.0 Released

Two little things in 3.8 changed. Cell::Base and Cell::Rails are now different classes with different semantics. Don’t panic – if you still derive your cells from Cell::Rails and use the #render_cell interface, only, nothing will change for you.

However, we had to tweak the API of some semi-public methods, here’s the new signatures.

Cell::Rails.create_cell_for(name, controller)
Cell::Rails.render_cell_for(name, state, controller, *args)

Well, nothing serious, it’s just the controller sliding to the end. If you use this anywhere, be sure to change it.

What’s new?

What is new is that Cell::Base is now decoupled from the ActionController. There simply is no AC-reference in Base anymore. However, this reference is still managed in Cell::Rails on purpose:

• Calls to #url_for and friends automatically query the parent controller for host and action name. This makes it easy to embedd cells into a controller view and still having correct links in the cell.
• Having the AC reference, people can still use request, params and session – if they need it. We strongly discourage accessing HTTP properties from a view component, though.

Let’s see what the new decoupled Base brings us.

Cells on Routes.

Deriving your cell from Base makes it routeable.

class PostsCell  Cell::Base
  def show
    @posts = Post.find(:all)
    render
  end

Routeable? Mountable? Well, let’s look at config/routes.rb to understand this.

match "/posts" => proc { |env|
  content = 
    Cell::Base.render_cell_for(:posts, :show)
  [ 200, {}, [ content ]]
}

How cool is that? You can mount a cell to a route without the ActionController overhead and still use the rendering, the caching and testing you all love from Cells. And, hey, it’s super super fast now.

The lack of the AC dependency makes cells even more versatile.

More on Routing

Since we got rid of the AC instance we have to take care of our URLs ourself. The cell still has access to all the URL helpers. Nevertheless, those helpers usually query the controller for host name portions, etc. That’s actually very simple with cells, too.

class PostsCell  Cell::Base
  def default_url_options
    {:host => "apotomo.de"}
  end

Just override the #default_url_options method as used from AC. This allows using the URL helpers as usual.

PostsCell.new.posts_url
#=> "http://apotomo.de/posts"

Let me know how these changes work for you! I’m eager to see new cases of applications in the next weeks.

We talked about Representers earlier last week. Today, I wanna describe DCI and how it fits perfectly together with representers in Roar.

What is DCI?

For years we’ve been taught that “Fat Models, Skinny Controllers” is the way to organize code. Push all your domain code into your ActiveRecord classes. Many people, including me, agree that this is bullshit. This leads to bloated classes, usually combined with one or two object instances taking care of an entire workflow (aka request).

DCI is an architectural pattern that makes the model “fat-on-demand”: Data objects are enhanced with roles at runtime, but only when needed.

The DCI pattern consists of three concepts.

1. Data objects are supposed to contain application data. Nothing else. This would be your ActiveRecord row instance, but without any predefined behaviour. Just data.
2. Context objects define what is done in the current workflow. In Rails, a controller running an action could be a context object.
3. Roles are behaviour for data objects in a particular context. In Ruby, a Rails controller could mixin a module (a “role”) into an empty model row and thus let it process something.

Hooray for DCI – But Why?

Now the thing is, I absolutely hate to hype something (except stuff I did, of course). However, DCI makes code better maintainable, easier to follow and way easier to test. Period.

BTW, did you already check out my new Roar gem? It’s hip!

Think of a classical model found in many Rails applications with maybe one hundred methods. Or fifty. Fifty methods to interfere with each other. In a DCI setup, an object has a limited behaviour scope. The limited knowledge makes it easier to debug and predictable.

One Model, Multiple Faces

Since representers are modules they fit perfectly into the DCI approach, making them roles. Let’s consider the fruit representer from last week, again.

require 'roar/representer/json'
 
module FruitRepresenter
  include Roar::Representer::JSON
 
  property :title
  collection :colors
end

You still can include this representer into its “host” class directly.

class Fruit
  include Roar::Representer
  include FruitRepresenter
end

The problem now is that the Fruit class is limited to one representation, only. Forever and a day you may only render one particular representation. The key about representations, however, is that one model may be represented by multiple, yeah, representations.

Now, back to the start, let’s assume the Fruit class is some empty ActiveRecord class without any prior knowledge of representations at all.

class Fruit  ActiveRecord::Base
end
 
lemon = Fruit.from_json("{\"title\":\"Lemon\"}")
#=> undefined method `from_json' for Fruit:Class

Yeah, there is no #from_json class method.

Representers On DCI

In order to use the DCI approach, we first need an instance. Here’s how the lemon instance, and only this instance, can be extended with the fruit representer.

lemon = Fruit.new.extend(FruitRepresenter)
lemon.from_json("{\"title\":\"Lemon\"}")
lemon.title #=> "Lemon"

Awesome! The built-in Ruby #extend method makes the lemon instance aware of its representation and mixes in #from_json and friends.

Naturally, this works in both ways.

orange = Fruit.find_by_title("Orange")
orange.extend(FruitRepresenter).to_json
#=> {"title":"Apple","colors":["green"]}

The mixed-in representer also lets us render the JSON document.

Gimme Complexity!

On a flat level with single objects, this works out-of-the-box with recent Roar/representable versions. Problems arise when we have nested setups.

module BowlRepresenter
  include Roar::Representer::JSON
  include Roar::Representer::Feature::Hypermedia
 
  property :location
  collection :items, :as => Fruit,
    :extend => FruitRepresenter
 
  link :self do
    "http://bowl/#{location}"
  end
end

Using the :extend option we can give representable a hint about the representer to use when rendering/parsing contained objects.

And, hey, this works.

empty_bowl = Bowl.new
empty_bowl.extend(BowlRepresenter)
emtpy_bowl.from_json("{\"location\":\"kitchen\", \
  \"items\":[{\"title\":\"Lemon\",\"colors\":[]}]}")
 
puts empty_bowl.items.first.title #=> "Lemon"

Discussion Needed!

Having representers on an object level makes the whole thing really clean and versatile. You can test document rendering and parsing without polluting classes. And you can have a FruitRepresenter, a RottenVegetableRepresenter and what else you like for your orange instance. BTW, oranges ain’t no vegetables.

We still discuss about the API and need your opinion. Right now, we got this

collection :items, :as => Fruit,
    :extend => FruitRepresenter

Here, :as describes the class that encapsulates the contained data. The :extend option helps representable to find the correct module for the nested item. Now, what about this?

collection :items, :class => Fruit,
    :module => FruitRepresenter

Does that make sense? These options really just describe what’s on the right side of the hash rocket. Thanks to Scott for discussion here. Now, tell us your story!

The idea of building and storing user timelines (think Twitter) in Riak confused me at first. It sounds like such a spot-on case for time series databases. Yet Yammer managed to make the idea pretty popular. The whole thing lacked implementation though, because they kept their to themselves, which I don't blame them for at all.

So let's have a look at how simple it is to build one. You can see a timeline of all Apple Macintosh products above, but that's not what we want to build.

Instead we want to build something like the Twitter timeline. A user follows many other users, and wants to look at a feed built from their activities, so something like the timeline shown below.

How do you model a timeline in Riak?

For every user we store one object in Riak. Every timeline contains a list of tweet ids, or whatever activity you're referencing, or it can contain the whole tweets. Something like this should work:

If you want to store more data, turn the list into an array of hashes containing whatever information is necessary to rebuild the timeline later.

Adding entries to the timeline

To add new entries to a timeline, prepend them to the existing list of entries, here's some Ruby code to show how it's done.

The code assumes you take care of followership somewhere else. You can store that data in Riak too, but the code is oblivious to its whereabouts.

Conflicts, siblings, oh my!

The fun starts when two clients update the same timeline, you get a conflict and siblings. The strength of a simple data structure like the example above is that they're easy to merge together while still keeping ordering based on the ids. The ids are ordered only in this example, Twitter somewhat makes sure they are.

When you get a conflict, a smart Riak library like Ripple helps you find out about it. To add on the earlier example, here's a version of add that detects conflicts.

Suddenly you have two or more objects instead of one, each containing a different timeline. To turn them into a single list, you merge all of them together, discard the duplicates, and restore order based on the id. Here's some Ruby to do that.

You iterate over all timeline objects and keep adding unique activities to a new list, returning that when done.

Sort, and done!

All that's left to do is sort the result.

There's the whole code. To spare you the pain of having to write your own library, all this is bundled into a gem called riaktivity. If you're doing Python, the Brett Hoerner has got you covered with timak. Be sure to watch the original talk by Yammer, it's well worth it.

There's ups and downs to this approach, and things that need to be taken care of. More on that and modeling data for Riak in general is covered in the Riak Handbook, the definitive guide on Riak.

One of the most common questions to ask about Riak is: how do I get a list of all the keys in my bucket, in the cluster, or that have an attribute that matches my query using MapReduce?

The motivation behind it is simple: you want to delete all the keys in a bucket, count the amount of keys stored in your cluster entirely, you want to clear out your cluster or you want to run ad-hoc queries on the data stored in a bucket.

All valid in their own right.

But things are not so simple with Riak. To understand why, let's take a quick look under the covers.

What's in a bucket?

A bucket is a namespace in Riak. It's not a physically distinctive entity like a table in a relational database. You can set some properties on it, things like replication levels, commit hooks, quorum, but that's it. Those are stored in the cluster's configuration which is gossiped around the cluster just like the data that identifies what partition goes on which machine.

In fact, when you specify a bucket and a key to fetch or write some data, they're stuck together to find the location in the cluster. Consider a bucket-key combination like users/roidrage. To find the location, Riak hashes both, not just the key. Both bucket and key uniquely identify a piece of data, allowing you to have multiple object with the same key, but in different buckets.

When an object is stored in Riak's storage backends, it uses both bucket and key name to identify it. What you get as a result are files that contain an abundance of different bucket-key combinations and their respective objects, sometimes not even in any order. The only physical distinction Riak has for data on disk is the partition they belong to. Everything else is up to the storage backend. There's no distinction between buckets on disk.

One reason for this is consistent hashing. If you remember the last installment of this series, I mentioned that consistent hashing's downside is that you lose key ordering. Keys are randomly spread out through the cluster. Some ordering still exists depending on the backend, but in general, ordering is lost.

Listing all of the keys

So to list keys in a bucket, Riak has to go through all of the keys in every partition, and I mean ALL OF THEM. Here's a picture of keys and an impersonation of Riak, having to take care of all of them.

No big deal, right? Unless of course, you store millions and millions of them, and want to find about all the keys from say, the bucket users, which may not even have to be more than 1000. To do that, Riak goes through every partition, every partition loads the keys either from memory (Bitcask) or disk (LevelDB) and sifts through them, finding the ones belonging to the users bucket.

All that said, it's certainly not impossible to do, if you have some time to wait, depending on the amount of data stored.

$ curl 'localhost:8098/buckets/users/keys?keys=true'

But wait, don't do that. Do this instead, streaming the keys instead of waiting for them all to arrive and then having them dumped at once.

$ curl 'localhost:8098/buckets/users/keys?keys=stream'

That's much better.

Listing keys has an impact on your Riak nodes, so if you can avoid it, don't do it!

So how do I really get all of the keys?

If select * from riak is not a great option, then what is?

Instead of relying on Riak, build an index on the keys. Thanks to Riak 2i (Secondary Indexes), this is easy. In fact, you get indexing of keys for free when using the LevelDB backend, just use the index $key. This takes advantage of LevelDB's sorted file structure. Neat!

But, and here's the kicker, you can only fetch ranges of keys. So instead of asking for all the keys, you ask for a range large enough to fit all the keys.

$ curl 'localhost:8098/buckets/users/index/$key/0/zzzz'

This finds all the keys that start with something lexicographically larger than 0 and less than zzzz and returns them to you in a list. Now there's a slim chance you'll get users with names like that, but I'll leave that exercise, or proper validations, up to you.

Using that list, you can count the number of keys in that bucket, or you can delete them one by one.

Ideally...

In an ideal world, listing keys in a bucket would be possible and not an expensive operation. Riak could for example allow users to store buckets in separate files. The downside is that with a lot of buckets, you'll hit the limits of open file descriptors in no time, a bit of a bummer. But until something better comes along, secondary indexes are a nice tool to at least avoid resorting to listing all of the keys.

Curious about other ways to index and query data in Riak? You'll like the Riak Handbook, which will be published later this week. Covers Riak's secondary indexes and other strategies to query, inspect and analyze data.

Check back in tomorrow for an introduction on storing timelines in Riak.

One of the most common questions to ask about Riak is: how do I get a list of all the keys in my bucket, in the cluster, or that have an attribute that matches my query using MapReduce?

The motivation behind it is simple: you want to delete all the keys in a bucket, count the amount of keys stored in your cluster entirely, you want to clear out your cluster or you want to run ad-hoc queries on the data stored in a bucket.

All valid in their own right.

But things are not so simple with Riak. To understand why, let's take a quick look under the covers.

What's in a bucket?

A bucket is a namespace in Riak. It's not a physically distinctive entity like a table in a relational database. You can set some properties on it, things like replication levels, commit hooks, quorum, but that's it. Those are stored in the cluster's configuration which is gossiped around the cluster just like the data that identifies what partition goes on which machine.

In fact, when you specify a bucket and a key to fetch or write some data, they're stuck together to find the location in the cluster. Consider a bucket-key combination like users/roidrage. To find the location, Riak hashes both, not just the key. Both bucket and key uniquely identify a piece of data, allowing you to have multiple object with the same key, but in different buckets.

When an object is stored in Riak's storage backends, it uses both bucket and key name to identify it. What you get as a result are files that contain an abundance of different bucket-key combinations and their respective objects, sometimes not even in any order. The only physical distinction Riak has for data on disk is the partition they belong to. Everything else is up to the storage backend. There's no distinction between buckets on disk.

One reason for this is consistent hashing. If you remember the last installment of this series, I mentioned that consistent hashing's downside is that you lose key ordering. Keys are randomly spread out through the cluster. Some ordering still exists depending on the backend, but in general, ordering is lost.

Listing all of the keys

So to list keys in a bucket, Riak has to go through all of the keys in every partition, and I mean ALL OF THEM. Here's a picture of keys and an impersonation of Riak, having to take care of all of them.

No big deal, right? Unless of course, you store millions and millions of them, and want to find about all the keys from say, the bucket users, which may not even have to be more than 1000. To do that, Riak goes through every partition, every partition loads the keys either from memory (Bitcask) or disk (LevelDB) and sifts through them, finding the ones belonging to the users bucket.

All that said, it's certainly not impossible to do, if you have some time to wait, depending on the amount of data stored.

$ curl 'localhost:8098/buckets/users/keys?keys=true'

But wait, don't do that. Do this instead, streaming the keys instead of waiting for them all to arrive and then having them dumped at once.

$ curl 'localhost:8098/buckets/users/keys?keys=stream'

That's much better.

Listing keys has an impact on your Riak nodes, so if you can avoid it, don't do it!

So how do I really get all of the keys?

If select * from riak is not a great option, then what is?

Instead of relying on Riak, build an index on the keys. Thanks to Riak 2i (Secondary Indexes), this is easy. In fact, you get indexing of keys for free when using the LevelDB backend, just use the index $key. This takes advantage of LevelDB's sorted file structure. Neat!

But, and here's the kicker, you can only fetch ranges of keys. So instead of asking for all the keys, you ask for a range large enough to fit all the keys.

$ curl 'localhost:8098/buckets/users/index/$key/0/zzzz'

This finds all the keys that start with something lexicographically larger than 0 and less than zzzz and returns them to you in a list. Now there's a slim chance you'll get users with names like that, but I'll leave that exercise, or proper validations, up to you.

Using that list, you can count the number of keys in that bucket, or you can delete them one by one.

Ideally...

In an ideal world, listing keys in a bucket would be possible and not an expensive operation. Riak could for example allow users to store buckets in separate files. The downside is that with a lot of buckets, you'll hit the limits of open file descriptors in no time, a bit of a bummer. But until something better comes along, secondary indexes are a nice tool to at least avoid resorting to listing all of the keys.

Curious about other ways to index and query data in Riak? You'll like the Riak Handbook, which will be published later this week. Covers Riak's secondary indexes and other strategies to query, inspect and analyze data.

Check back in tomorrow for an introduction on storing timelines in Riak.

You can speed up development load time and get rid of rspec in development if you simply not require it in :development and just load the tasks in the Rakefile where they are needed.

# Gemfile
group :development, :test do
  gem 'rspec-rails', :require => false
end

# Rakefile
require File.expand_path('../config/application', __FILE__)
require 'rspec-rails' if ['test', 'development'].include?(Rails.env)
...

Repeat for any other testing lib for even more speedup

The simplicity of consistent hashing is pretty mind-blowing. Here you have a number of nodes in a cluster of databases, or in a cluster of web caches. How do you figure out where the data for a particular key goes in that cluster?

You apply a hash function to the key. That's it? Yeah, that's the whole deal of consistent hashing. It's in the name, isn't it?

The same key will always return the same hash code (hopefully), so once you've figured out how you spread out a range of keys across the nodes available, you can always find the right node by looking at the hash code for a key.

It's pretty ingenious, if you ask me. It was cooked up in the lab chambers at Akamai, back in the late nineties. You should go and read the original paper right after we're done here.

Consistent hashing solves the problem people desperately tried to apply sharding to pretty nicely and elegantly. I'm not going to bore you with the details on how exactly consistent hashing works. Mike Perham does a pretty good job at that already, and there are many more blog posts explaining implementations and theory behind it. Also, that little upcoming book of mine has a full-length explanation too. Here's a graphic showing the basic idea of consistent hashing, courtesy of Basho.

Instead I want to look at the practical implications of consistent hashing in distributed databases and cache farms.

Easier to Avoid Hotspots

When you put data on nodes based on a random result, which is what the hash function calculates, a value that's a lot more random than the key it's based on, it's easier to avoid hotspots. Why?

Assume a key based on an increasing value, or a simple range of keys, based on the hour of the day, like 2011-12-11-13. You add new hours and therefore new data as time passes, and keys are stored based on the range they fall in. For example, the keys 2011-12-11-18 until 2011-12-11-23 are stored on the same node, with the rest of the keys stored on other nodes, just because the ranges or the partitioning scheme happen to be set up this way.

For a consumer-facing site, the evening hours are usually the busiest time of the day. They create more data, more writes, and possibly more reads too. For the hours between 18:00 and 23:00, all the load goes to the single node that carries all the relevant data.

But when you determine the location in the cluster based solely on the hash of the key, chances are much higher that two keys lexicographically close to each other end up on different nodes. Thus, the load is shared more evenly. The disadvantage is that you lose the order of keys.

There are partitioning schemes that can work around this, even with a range-based key location. HBase (and Google's BigTable, for that matter) stores ranges of data in separate tablets. As tablets grow beyond their maximum size, they're split up and the remaining parts re-distributed. The advantage of this is that the original range is kept, even as you scale up.

Consistent Hashing Enables Partitioning

When you have a consistent hash, everything looks like a partition. The idea is simple. Consistent hashing forms a keyspace, which is also called continuum, as presented in the illustration. As a node joins the cluster, it picks a random number, and that number determines the data it's going to be responsible for. Everything between this number and one that's next in the ring and that has been picked by a different node previously, is now belong to this node. The resulting partition could be of any size theoretically. It could be a tiny slice, or a large one.

First implementations of consistent hashing still had the problem that a node picking a random range of keys resulted in one node potentially carrying a larger keyspace than others, therefore still creating hotspots.

But the improvement was as simple as it was ingenious. A hash function has a maximum result set, a SHA-1 function has a bit space of 2^160. You do the math. Instead of picking a random key, a node could choose from a fixed set of partitions, like equally size pizza slices. But instead of picking the one with the most cheese on, everyone gets an equally large slice. The number of partitions is picked up front, and practically never changes over the lifetime of the cluster.

For good measure, here's a picture of a sliced pizza.

Partitioning Makes Scaling Up and Down More Predictable

With a fixed number of partitions of the same size, adding new nodes becomes even less of a burden than with just consistent hashing. With the former, it was still unpredictable how much data had to be moved around to transfer ownership of all the data in the range of the new node. One thing's for sure, it already involved a lot less work than previous methods of sharding data.

With partitioning, a node simply claims partitions, and either explicitly or implicitly asks the current owners to hand off the data to them. As a partition can only contain so many keys, and randomness ensures a somewhat even spread of data, there's a lot less unpredictability about the data that needs to be transferred.

If that partitions just so happens to carry the largest object by far in you whole cluster, that's something even consistent hashing can't solve. It only cares for keys.

Going back to HBase, it cares for keys and the size of the tablet the data is stored in, as it breaks up tablets once they reach a threshold. Breaking up and reassigning a tablet requires coordination, which is not an easy thing to do in a distributed system.

Consistent Hashing and Partitioning Enable Replication

Consistent hashing made one thing a lot easier: replicating data across several nodes. The primary means for replication is to ensure data survives single or multiple machine failures. The more replicas you have, the more likely is your data to survive one or more hardware crashes. With three replicas, you can afford to lose two nodes and still serve the data.

With a fixed set of partitions, a new node can just pick the ones it's responsible for, and another stack of partitions it's going to be a replica for. When you really think about it, both processes are actually the same. The beauty of consistent hashing is that there doesn't need to be master for any piece of data. Every node is simply a replica of a number of partitions.

But replication has another purpose besides ensuring data availability.

Replication Reduces Hotspots (Even More!!!)

Having more than one replica of a single piece of data means you can spread out the request load even more. With three replicas of that data, residing on three different nodes, you can now load-balance between them. Neat!

With that, consistent hashing enables a pretty linear increase in capacity as you add more nodes to a cluster.

Consistent Hashing Enables Scalability and Availability

Consistent hashing allows you to scale up and down easier, and makes ensuring availability easier. Easier ways to replicate data allows for better availability and fault-tolerance. Easier ways to reshuffle data when nodes come and go means simpler ways to scale up and down.

It's an ingenious invention, one that has had a great impact. Look at the likes of Memcached, Amazon's Dynamo, Cassandra, or Riak. They all adopted consistent hashing in one way or the other to ensure scalability and availability.

Want to know more about distributed databases in general and Riak in particular? You'll like the Riak Handbook, a hands-on guide full of practical examples and advice on how to use Riak to ensure scalability and availability for your data.

In the next installment we're looking at the consequences and implications of losing key ordering in a Riak cluster.

The simplicity of consistent hashing is pretty mind-blowing. Here you have a number of nodes in a cluster of databases, or in a cluster of web caches. How do you figure out where the data for a particular key goes in that cluster?

You apply a hash function to the key. That's it? Yeah, that's the whole deal of consistent hashing. It's in the name, isn't it?

The same key will always return the same hash code (hopefully), so once you've figured out how you spread out a range of keys across the nodes available, you can always find the right node by looking at the hash code for a key.

It's pretty ingenious, if you ask me. It was cooked up in the lab chambers at Akamai, back in the late nineties. You should go and read the original paper right after we're done here.

Consistent hashing solves the problem people desperately tried to apply sharding to pretty nicely and elegantly. I'm not going to bore you with the details on how exactly consistent hashing works. Mike Perham does a pretty good job at that already, and there are many more blog posts explaining implementations and theory behind it. Also, that little upcoming book of mine has a full-length explanation too. Here's a graphic showing the basic idea of consistent hashing, courtesy of Basho.

Instead I want to look at the practical implications of consistent hashing in distributed databases and cache farms.

Easier to Avoid Hotspots

When you put data on nodes based on a random result, which is what the hash function calculates, a value that's a lot more random than the key it's based on, it's easier to avoid hotspots. Why?

Assume a key based on an increasing value, or a simple range of keys, based on the hour of the day, like 2011-12-11-13. You add new hours and therefore new data as time passes, and keys are stored based on the range they fall in. For example, the keys 2011-12-11-18 until 2011-12-11-23 are stored on the same node, with the rest of the keys stored on other nodes, just because the ranges or the partitioning scheme happen to be set up this way.

For a consumer-facing site, the evening hours are usually the busiest time of the day. They create more data, more writes, and possibly more reads too. For the hours between 18:00 and 23:00, all the load goes to the single node that carries all the relevant data.

But when you determine the location in the cluster based solely on the hash of the key, chances are much higher that two keys lexicographically close to each other end up on different nodes. Thus, the load is shared more evenly. The disadvantage is that you lose the order of keys.

There are partitioning schemes that can work around this, even with a range-based key location. HBase (and Google's BigTable, for that matter) stores ranges of data in separate tablets. As tablets grow beyond their maximum size, they're split up and the remaining parts re-distributed. The advantage of this is that the original range is kept, even as you scale up.

Consistent Hashing Enables Partitioning

When you have a consistent hash, everything looks like a partition. The idea is simple. Consistent hashing forms a keyspace, which is also called continuum, as presented in the illustration. As a node joins the cluster, it picks a random number, and that number determines the data it's going to be responsible for. Everything between this number and one that's next in the ring and that has been picked by a different node previously, is now belong to this node. The resulting partition could be of any size theoretically. It could be a tiny slice, or a large one.

First implementations of consistent hashing still had the problem that a node picking a random range of keys resulted in one node potentially carrying a larger keyspace than others, therefore still creating hotspots.

But the improvement was as simple as it was ingenious. A hash function has a maximum result set, a SHA-1 function has a bit space of 2^160. You do the math. Instead of picking a random key, a node could choose from a fixed set of partitions, like equally size pizza slices. But instead of picking the one with the most cheese on, everyone gets an equally large slice. The number of partitions is picked up front, and practically never changes over the lifetime of the cluster.

For good measure, here's a picture of a sliced pizza.

Partitioning Makes Scaling Up and Down More Predictable

With a fixed number of partitions of the same size, adding new nodes becomes even less of a burden than with just consistent hashing. With the former, it was still unpredictable how much data had to be moved around to transfer ownership of all the data in the range of the new node. One thing's for sure, it already involved a lot less work than previous methods of sharding data.

With partitioning, a node simply claims partitions, and either explicitly or implicitly asks the current owners to hand off the data to them. As a partition can only contain so many keys, and randomness ensures a somewhat even spread of data, there's a lot less unpredictability about the data that needs to be transferred.

If that partitions just so happens to carry the largest object by far in you whole cluster, that's something even consistent hashing can't solve. It only cares for keys.

Going back to HBase, it cares for keys and the size of the tablet the data is stored in, as it breaks up tablets once they reach a threshold. Breaking up and reassigning a tablet requires coordination, which is not an easy thing to do in a distributed system.

Consistent Hashing and Partitioning Enable Replication

Consistent hashing made one thing a lot easier: replicating data across several nodes. The primary means for replication is to ensure data survives single or multiple machine failures. The more replicas you have, the more likely is your data to survive one or more hardware crashes. With three replicas, you can afford to lose two nodes and still serve the data.

With a fixed set of partitions, a new node can just pick the ones it's responsible for, and another stack of partitions it's going to be a replica for. When you really think about it, both processes are actually the same. The beauty of consistent hashing is that there doesn't need to be master for any piece of data. Every node is simply a replica of a number of partitions.

But replication has another purpose besides ensuring data availability.

Replication Reduces Hotspots (Even More!!!)

Having more than one replica of a single piece of data means you can spread out the request load even more. With three replicas of that data, residing on three different nodes, you can now load-balance between them. Neat!

With that, consistent hashing enables a pretty linear increase in capacity as you add more nodes to a cluster.

Consistent Hashing Enables Scalability and Availability

Consistent hashing allows you to scale up and down easier, and makes ensuring availability easier. Easier ways to replicate data allows for better availability and fault-tolerance. Easier ways to reshuffle data when nodes come and go means simpler ways to scale up and down.

It's an ingenious invention, one that has had a great impact. Look at the likes of Memcached, Amazon's Dynamo, Cassandra, or Riak. They all adopted consistent hashing in one way or the other to ensure scalability and availability.

Want to know more about distributed databases in general and Riak in particular? You'll like the Riak Handbook, a hands-on guide full of practical examples and advice on how to use Riak to ensure scalability and availability for your data.

In the next installment we're looking at the consequences and implications of losing key ordering in a Riak cluster.

This post introduces the new Roar gem, a framework-agnostic REST framework for Ruby that puts focus on object-oriented REST documents.

More object-orientation?

The calls for more object-orientation in Ruby frameworks are growing ever louder. Most web frameworks, like Rails, come with a nice database abstraction, a Rack-like HTTP endpoint, routing, and a template engine. It’s the lack of additional abstraction layers that make many programmers ask for “more OOP”.

Already yawning? Yeah, I wrote about that earlier this year. Let’s talk about working solutions now. Let’s talk about REST.

REST is about Documents!

REST is not only about pretty URLs. The problem ain’t URLs at all. Most frameworks have a decent routing layer to connect resource logic (aka controllers) to URLs. However, the key of REST is pushing documents between clients and servers, so why don’t we finally focus on documents?

It’s the document parsing, processing and rendering that makes our heads spinning.

Note: I here use the terms representation and document interchangable just because I can!

Parsing and Processing

Both clients and services have to parse incoming documents. Consider this JSON document.

{"title":  "Lemon",
 "colors": ["green"]}

This could be sent to a resource to create a new fruit, or be retrieved by some client which issued a GET request.

In Rails, and many HTTP gems, this is automatically parsed using a JSON gem.

params = JSON['{"title":  "Lemon",
                "colors": ["green"]}']
 
params["title"] #=> "Lemon"
params["colors"][0] #=> "green"

We’re using a hash access to read properties from that document. This is handy! Now, assume the sender forgot to include the colors.

params = JSON['{"title":  "Lemon"}']
 
params["colors"][0]
#=>NoMethodError: undefined method `[]' for nil:NilClass

An exception, shit! While processing the parsed document (aka hash), we have to take care of missing keys, wrong types etc. We as the consumer have to know about all the pitfalls with this particular document.

Rendering

Now that our parsing and processing code is finished, how do we render outgoing representations? We could use #to_json.

@lemon.to_json(:only => [:title, :colors])

Being a quick solution for rendering small models this approach hits the wall when we have nested models but still want a plain JSON representation, or when we want to embed hypermedia. Also, we put “REST” logic into our database layer. It’s up to you to judge.

What about template languages? Let’s try jbuilder for JSON rendering.

Jbuilder.encode do |json|
  json.title  @lemon.title
  json.colors @lemon.colors
end

Nice! But, wait. How do I test this? Can I test this separately? What if I want different representations for the same model? Do I need different templates? And… is it cool to have format knowledge both in our parsing layer and in our rendering layer? I mean, both need to know about the title property and the colors array.

Representers: OOP to our Documents!

I won’t further discuss the shown concepts here. Let me just criticize that they are not object-oriented and encourage the distribution of document internals about the entire framework.

Here’s how representers can handle all that and make you a happy man at the same time!.

require 'roar/representer/json'
 
module FruitRepresenter
  include Roar::Representer::Base
 
  property :title
  collection :colors
end

This is all we need to declare the basic syntax of the fruit document. Note that it resides in a module which can be used anywhere in your app.

Rendering a Lemon

Now let’s say our database layer provides us a class Fruit with ActiveRecord-like behaviour.

lemon = Fruit.new(:title => "Lemon")
puts lemon.title #=> "Lemon"

Since the representer knows everything about the document format it can be used for rendering JSON. We just have to include the FruitRepresenter module into the class and new methods will be available.

Fruit.class_eval do # DB layer.
  include Roar::Representer::JSON
  include FruitRepresenter
end
 
lemon.to_json
#=> "{\"title\":\"Lemon\",\"colors\":[]}"

Awesome, the FruitRepresenter module gives us a #to_json method. Internally, this method compiles the JSON document by asking the represented object (aka lemon) for its properties and values. Remember, the representer knows about those properties, since we declared them. It does not know anything about ActiveRecord and the like.

Now, what about parsing?

Parsing a Lemon

The great thing about representers is: they work bidirectional. That alone is one of the main advantages to many other gems out there. Bidirectional? Let’s just see what that is.

orange = Fruit.from_json("{\"title\":\"Orange\"}")
puts orange.inspect
#=> #<0x9df73a8>

We can parse representations and create objects from it! That’s bidirectional: rendering and parsing is combined in one representer.

Again, the representer does not know anything about the underlying database – it just creates an object it was mixed into, reads from the document and assigns properties using polite setters. Nothing more, bro.

Don’t like the class method? If you already have an instance, you can use #from_json on the instance.

orange.from_json("{\"title\":\"Yummy Orange\"}")
orange.title #=> "Yummy Orange"

Nesting Representers

Now that we have fruits, lets put ‘em in a bowl. A fruit bowl contains a list of fruits.

bowl = Bowl.new
bowl.items  lemon
bowl.items  orange

In order to bind the fruit bowl to a REST resource we write another representer module.

module BowlRepresenter
  include Roar::Representer::Base
 
  property :location
  collection :items, :as => Fruit
end
 
Bowl.class_eval do # DB layer.
  include Roar::Representer::JSON
  include BowlRepresenter
end

Properties can be typed using the :as option. The target class must be a representer, too. That’s the only requirement.

bowl.to_json
#=> {"items":[
  {"title":"Lemon","colors":[]},
  {"title":"Orange","colors":[]}
]}

Typed properties allow us to build more complex representations without breaking with OOP. The collection assignment makes building lists as easy as possible. And if you still need to tweak things, there are some helpful configuration options around – more on that in one of the next posts.

Representers love HATEOAS

HATEOAS? Check out this talk I gave if you want to learn about Hypermedia and HATEOAS.

Hypermedia helps clients to operate the API without having to compute URLs. We want hypermedia in our documents.

module BowlRepresenter
  include Roar::Representer::Feature::Hypermedia
 
  link :self do
    "http://bowls/#{location}"
  end
end

Including the Hypermedia feature dynamically adds a new method #link to our representer module. What does it do?

bowl.location = :desk
bowl.to_json
#=> {
"items":[
  {"title":"Lemon","colors":[]},
  {"title":"Orange","colors":[]}],
"links":[{"rel":"self","href":"http://bowls/desk"}]
}

Seems like an easy way to embed links in our documents. In roar-rails the Rails URL helpers work like a charm within the link blocks – making it super simple to provide hypermedia to consumers and still being railsy.

Representers on the Client Side

Ok, representers basically are plain modules. Why not use ‘em on the client side as well? The client doesn’t have access to the model layer – if it does, something’s wrong! However, it’s absolutely ok to share the representer modules.

class Bowl
  include Roar::Representer::JSON
  include Roar::Representer::Feature::HttpVerbs
  include BowlRepresenter # from a gem.
end

See how we simply use an empty class with the mixed-in representer on the client side? Remember, we don’t want to have access to the database here.

The next step is creating a new fruit bowl.

bowl = Bowl.new
bowl.location = "kitchen"
bowl.post!("http://bowls")

This code is enough to send a POST request to the specified URL. The bowl instance will serialize a JSON document, send it to the URL using the HttpVerbs feature, receive the new bowl, deserialize it and update its properties accordingly. You may know this behaviour from REST client gems like ActiveResource – here, with less code and manageable complexity.

Consume Hypermedia, it’s gooood.

The representer also makes it easy to extract hypermedia links from documents.

bowl.links[:self]
#=> "http://bowls/kitchen"

No parsing needed, the representer knows where to find your hypermedia. How cool is that?

XML is not a Crime!

Roar comes with both JSON and XML representers.

module BowlRepresenter
  include Roar::Representer::XML
end
 
bowl.to_xml
#=> '
#  
#'

Now choose if you want JSON or XML – it’s up to you! I still don’t understand the XML hating – did I miss something or is it just “too java”?

Roar – and more?

The Roar project is still evolving. However, the current API is almost stable. New features, new concepts, testing helpers, etc will be added over time just like in every framework. Roar is framework-agnostic and can be used anywhere, as long as it’s Ruby.

Its modulare feature architecture makes it easy to extend your documents. Currently, we have some exciting on-going brainstorming and hacking on features like client-side caching in the representer instance, generic error handling, validations, DSLs, … We need you to try it out, complain, propose, help, and spread the word. Thank you for reading until here, you rock!

Following up on the series of posts about the Rails Asset Pipeline, here are some things to keep in mind when you are using Nginx as your webserver.

Installation via Passenger

When you are installing Nginx automatically with the script that comes with Passenger, you will not be able to serve the gzipped file variants that get precompiled. That’s because Passenger currently does not configure Nginx with the http_gzip_static_module that would be necessary for including the directive gzip_static on. I’ve opened a pull request for that, but for now you’ll have to choose the manual installation, if that’s what you are after - and why wouldn’t you?

# download and untar nginx
                wget -O /tmp/nginx.tar.gz http://www.nginx.org/download/nginx-1.0.10.tar.gz
                tar xzvf /tmp/nginx.tar.gz
                rm /tmp/nginx.tar.gz
                
                # install nginx via passenger install script, already includes
                # --with-http_ssl_module
                # --add-module for passenger
                passenger-install-nginx-module --auto --prefix='/opt/nginx' --nginx-source-dir='/tmp/nginx-1.0.10' --extra-configure-flags="--with-http_gzip_static_module"
                

You might want to change the version and prefix to fit your needs. Like mentioned in the comments you don’t have to take care of including the passenger or ssl module, because the installation script adds them automatically.

Nginx configuration

This is already part of the Asset Pipeline Guide, but I’ll mention it here again, so that this article is complete - here are the directives for the assets location config:

location ~ ^/(assets)/  {
                  root /path/to/site;
                  gzip_static on;
                  expires max;
                  add_header Cache-Control public;
                  # access_log /dev/null;
                }
                

I’ve also disabled access logging, so that asset requests do not clutter up the log.

Mathias brought up a good point, why one might not want to do that.

Sendfile Header

In your app’s production.rb you should uncomment to following line, to let Nginx send static files:

config.action_dispatch.x_sendfile_header = 'X-Accel-Redirect'
                

Verify that it’s working

After deploying the changes, you can check if Nginx is now serving the gzipped compressed versions of your asset files. For that you can use tools like this HTTP Compression Tester and enter the address of your application.js and application.css

So you’ve installed (or upgraded to) ubuntu 11.10 and everything looks great… Except – uh!, lots of gedit plugins are only compatible with gedit 2! But don’t be sad.. or angry.. This guide points out, how to, nevertheless, create a solid foundation that allows you getting prodcutive with gedit!

Setting up gedit, the slim and powerful, pretty and fast, flexible and stable code editor ;)

This guide assumes an ubuntu 11.10(gnome shell) system, but will also work for similar variants (Btw, what’s the first thing you should adjust in a new ubuntu system? Change the gnome theme to radiance!).

Aside from that, you should already have installed ruby and git (sudo apt-get install git-core). If you still use GNOME 2/gedit 2, please use this previous article instead.

Scaffold

Firstly, ensure that you’ve got the official gedit-plugins package:

sudo apt-get install gedit-plugins

Besides this official package, I usually recommend installing the gedit-gmate package. However, one disadvantage is that plugins get not stored in the default user plugin directory. Moreover, it comes with an extra Ruby language specification for Ruby files in Rails, so editing Ruby files can lead to different colorizations, which is quite confusing. Nevertheless, gmate comes with lots of useful additions that will be used below.

On the other hand, I’ve started to improve the Ruby (and related) gedit configurations and called this rubybuntu-gedit. It’s not finished, yet, but already includes better Ruby specifications. Still, it does not include other web-related language definitions, but gmate has them! In some folder, clone the gmate repo:

git clone https://github.com/gmate/gmate

Copy the files from gmate/lang-specs to /usr/share/gtksourceview-3.0/language-specs:

sudo cp gmate/lang-specs/*.lang /usr/share/gtksourceview-3.0/language-specs

Next up, install the rubybuntu-gedit gem. It will copy more language files to the above folder, delete the rails and rhtml files and set some gnome mime types (install as sudo when prompted):

gem install rubybuntu-gedit
rubybuntu-gedit install -3

After these steps, gedit can speak languages like Ruby 1.9, Sass, Haml, Cucumber, Gemfile.lock etc. ;)

Settings

Now, start gedit!

Here are some recommendations for the basic settings at the Edit/Preferences dialog:

Under the Font & Colors tab, you have to either select the RubyBuntuOne or the RubyBuntuTwo theme to use the enhanced language specifications from above.

The redundant toolbar can be deactivated in the View menu. The default shortcut for toggling the bottom- and sidebars is (<Ctrl>+)<F9>.

Hot hot keys

Before getting into the plugins, let me highlight that good shortcuts for good features are very important. What is the use of a plugin that you do not use, because you always forget how to use it?

ubuntu/gnome offers a nice way to change menu shortcuts: The global Editable menu shortcuts setting. It allows you to change a shortcut of a menu item by moving the mouse over the item and pressing the desired shortcut.

Unfortunately, it has disappeared in ubuntu 10.4 from the appearance settings. But it is not lost, it is just hidden. You can get it back in GNOME 3 with the gsettings command:

gsettings set org.gnome.desktop.interface can-change-accels true

And while we’re on it, let’s also improve the way the home key works in gedit (should jump to the first char, instead of the absolute beginning of a line):

gsettings set org.gnome.gedit.preferences.editor smart-home-end before

Plugins

Installing gedit plugins is quite easy. Just place the downloaded files of a plugin at: ~/.local/share/gedit/plugins/. Pay attention that the .plugin file must lay exactly in the directory. In most cases, it comes with an additional .py file and/or a folder named after the plugin. Plugins can be activated at Edit/Preferences/Plugins.

Below is a list of plugins I use, ordered by subjective importance. Every plugin without a download link is available in the official plugin package or in gmate(gedit3). I’ve alse added a shortcut suggestion for some of them.

Snap open

Code comment

Advanced Find/Replace

Snippets

Word completion

Draw Spaces

External Tools

#!/bin/sh
ruby

Smart Highlight

Pair Character Completion

Color Picker

Open URI Context Menu

Embedded Terminal

Regex Search and Replace

I miss my gedit 2 plugins!!!11

This list is definitely shorter than the one for gedit 2, many plugins are not ported, yet. For example, I haven’t found a replacement for these ones:

• Smart Indent
• TODO List
• Rails Extract Partials

In addition, I miss an indent plugin that allows you to (un)indent using <Alt> and the arrow keys. Although gtksourceview (which is a part of gedit) offers indenting using (<Shift>)<Tab>, it’s absolutely not satisfying when trying to indent single lines.

I will update this post with new gedit 3 compatible plugins that I’ll use (or maybe write). New plugins can also be found at:

• the official wiki
• the gmate repository

Rails Footnotes

Last but not least, some extra sugar for Rails-developers: The rails-footnotes gem! It adds useful information about the current Rails request to the bottom of each rendered browser page. It also provides helpful links to associated files – which open in gedit!

But that is still not everything… When you get an error page, you can click on the referenced line and gedit will open the accordant document and jump to the line!

Total convenience! I gained much productivity using this tool. Because it’s written with textmate in mind, some tweaking is needed. Let’s start with adding this line to the development section of your Gemfile:

Then run the generator:

The next step is to create a textmate handler that the browser uses to open txmt:// links. Create it somewhere on your disk, for example in ~/.txmt_handler:

#!/bin/bash
# simple txmt: to gedit handler (simple because: needs this order and completeness)
# see http://rbjl.net/58
if [[ $1 =~ ^txmt://open\?url=file://(.*?)\&line=([0-9]*)\&column=([0-9]*)$ ]]
then
  exec /usr/bin/gedit ${BASH_REMATCH[1]} +${BASH_REMATCH[2]}:${BASH_REMATCH[3]}
fi

After making this file executable (chmod +x), you need to add the txmt: protocoll handler to your browser. In firefox, just go to about:config and create a boolean key named network.protocol-handler.expose.txmt that you set to false. When you now click on a txmt: link for the first time, firefox will open a popup, in which you can select the script from above.

Very fast test execution from inside Rubymine.
Cmd+F8 + 2 seconds == test results

1 Activate DRB in Rubymine
Run > Edit configurations > Defaults > RSpec > Use DRB Server

2 a) Start spork from Rubymine
Tools > Run Spork DRB Server

2 b) (alternatively) start spork from the commandline
Code

  # spec/spec_helper.rb
Spork.prefork do
  if ENV["RUBYMINE_HOME"]
    puts "Rubymine support"
    $:.unshift(File.expand_path("rb/testing/patch/common", ENV["RUBYMINE_HOME"]))
    $:.unshift(File.expand_path("rb/testing/patch/bdd", ENV["RUBYMINE_HOME"]))
  end
end

and run it via:

RUBYMINE_HOME=/Applications/Rubymine\ 3.2.4.app/ spork

Sometimes you’ll want to refer to your image assets from inside of your CoffeeScript (or JavaScript). While we have nice helpers for that in SASS (namely image-path and image-url), we would have to resort to appending .erb to every .js or .coffee file, we want to reference images in, so that we could use something like this in there:

icon = 
                  

I didn’t like it that way, because ERB inside of CoffeeScript looks odd and having the file end with .erb messes up syntax highlighting. But we can circumvent this easily by adding this (admittedly not the most beautiful) piece of code to one single file that ends on .erb (i.e. images.js.coffee.erb) and let it provide the helper method we need:

                  imgs = {}
                  Dir.chdir("#{Rails.root}/app/assets/images/") do
                    imgs = Dir["**"].inject({}) {|h,f| h.merge! f => image_path(f)}
                  end
                  %>
                  
                  window.image_path = (name) ->
                     imgs.to_json %>[name]
                  

This adds up all images inside of app/assets/images and makes them available via the global helper method image_path - your milage may vary, but you get the idea. From there on it’s simply

icon = image_path('icon.png')
                  

inside of your CoffeeScript (or JavaScript) files and it also works in production as the returned filename contains the digest.

I am using the Jekyll blog generator for some projects (still not sure if I really like it..) and came across the following requirement the other day:

• The blog should only have an index page, no (visible) sub-posts
• This index page should use pagination
• There should be the ability to link to a specific blog post

The problem was: I wanted to use anchor links (http://some.url/#post-identifier), but – because of the pagination – blog posts “change” their index page (e.g. in jekyll, they move from /#example to /page2/#example).

I decided to fix this using this small piece of JavaScript code. If the blog post is not found on the current page, it will redirect the user to the correct page. Nothing special, but why not share? ;)

See this permalink for an example in action (you will be redirected to page 2). The code might easily be adopted for other blogging engines.

Im Rahmen des OpenData-Wettbewerbs Apps4Deutschland fand in Bremen am Freitag, den 11. November 2011 die Auftaktveranstaltung zu Apps4Bremen statt. Nach dem Vorbild Berlin macht nun auch das Land Bremen Teile der Daten staatlicher Stellen öffentlich und läd dazu ein, diese zu verwerten und sinnvoll aufzubereiten - auf der Apps4Deutschland-Website heißt es dazu:

Behörden in Deutschland erheben, speichern und verarbeiten viele interessante Daten. Dazu gehören Wetterdaten ebenso wie Informationen über die Luftqualität oder über die Verwendung der Steuergelder. Diese Daten sind von öffentlichem Interesse und bergen ein großes Potenzial für unsere Gesellschaft, weil sie Transparenz, Beteiligung, Innovationen und neue Dienstleistungen fördern.

Mit dem Wettbewerb „Apps für Deutschland“ laden wir Designer, Entwickler, Journalisten, Forscher und die breite Öffentlichkeit ein, Anwendungen zu schaffen, um diese Daten nutzbar zu machen.

Da ich das Thema Open Government spannend finde und für sehr wichtig halte, war der Wettberwerb ein guter Anlass, um mir eine neue Spielwiese für die offenen Daten Bremens zu schaffen. Das Projekt besteht aktuell nur aus einer Übersicht der Bremer Kindertagesstätten, lässt sich aber auf dem bisherigen Stand gut erweitern, da es bereits Beispiele für Datenimporter, Geokodierung und Darstellung von Orten auf einer Google Map enthält. Basis dafür sind Ruby on Rails und Backbone.js und der Quelltext ist auf GitHub verfügbar, so dass das Projekt gerne erweitert werden kann. Gerne lasse ich mich auch auf Ideen, Verbesserungswünsche oder Fehler hinweisen.

Wer sich inspirieren lassen möchte, kann auch mal einen Blick in den OpenData Showroom werfen, der Projekte aus aller Welt enthält, die sich offener Daten bedienen. Würde mich freuen, wenn sich viele weitere Leute finden, die mit den Daten was nettes auf die Beine stellen.

Das zumindest behauptet Peter Cooper von RubyInside mit Berufung auf diese Nachricht von Matz. Und die sieht ziemlich authentisch aus. Damit ist der Wahn, alles mögliche auf #autoload umzustellen, wohl erst einmal am Ende angekommen. Ich denke, dass das eine gute Nachricht für die Windows-Nutzer unter den Rubyisten ist, denn für Programme wie OCRA stellt #autoload eine ziemlich harte Nuss da. Ich persönlich halte den enstehenden Geschwindigkeitsvorteil von #autoload gegenüber einem ordinären #require für eher marginal; der Sinn sollte ja angeblich sein, später benötigte Komponenten später zu laden, aber das kann man auch mit #require tun, was dann auch noch expliziter ist. Rubys #require-Methode ist nun einmal nicht dasselbe wie ein #include in C(++); das kann man jederzeit auch innerhalb von Methoden aufrufen und ist nicht zwangsweise daran gebunden, es an den Dateianfang zu schreiben, weil der Interpreter/Compiler sich sonst daran verschlucken würde¹. Nichtsdestotrotz sollte man diesen Gebrauch von #require natürlich gut überlegen und nicht grundlos machen. Es ist wesentlich übersichtlicher, wenn alles einzubindende am Anfang der betroffenen Datei zu finden ist.
Ich persönlich ziehe ein explizites #require jedenfalls dem schwer zu überblickenden #autoload vor (von der von Matz angesprochenen Thread-Problematik mal ganz abgeshen).

Freuen wir uns also auf:

Warning: Use of #autoload is deprecated. Use an explicit #require instead.

Valete,
Quintus

¹ OK, man kann zwar an prinzipiell jeder Stelle in C(++)-Code ein #include machen, aber effektiv wird dann ja trotzdem alles auf einmal geladen. Und im Übrigen ist das schlechter Stil.

Tags: de | Ruby

Ruby 1.8 dies in June 2012. This post shows some small examples, why this is good ;)

Version used for testing: ruby-1.8.7-p352. Everything works well in 1.9.

Bug: Inconsistent Float representation

>> a=1215449.9999999993
=> 1215450.0 
>> puts "a as Float:   #{a}\na as Integer: #{a.to_i}"
a as Float:   1215450.0
a as Integer: 1215449

Bug: Encoding issues

>> a = "text with ─ unicode"
=> "text with ─ unicode"
>> a.tr "─", '-'
=> "text with --- unicode"

Bug: RUBY_ENGINE is not defined

Not really a bug, but still pretty annoying. You always have to check if the constants exists before using it.

X Virtual Framebuffer

http://de.wikipedia.org/wiki/Xvfb

e.g. use it to create screenshots.

For a basic implementation, see this:https://github.com/leonid-shevtsov/headless/blob/master/lib/headless.rb

Original article writen by suung and published on praktikantenblog | direct link to this article | If you are reading this article elsewhere than praktikantenblog, it has been illegally reproduced and without proper authorization.

First off: I love Rails’ asset pipeline from the bottom of my heart. It’s a great thing and makes frontend development such a lovely task. There are two little drawbacks though when using it in development: It slows down page reloads and clutters up the log output. Here I’ll show you how to get rid of that.

When you are mainly doing backend development or most of your apps frontend code doesn’t change too much do yourself a favor and precompile your assets for the development environment:

$ RAILS_ENV=development bundle exec rake assets:precompile
                      

This really speeds up page reloads as Rails does not need to look up and precompile the asset files every time a request hits the app anymore.

Another change you might want to make is to set config.assets.debug to false in your development environment. This makes Sprocket concatenate all your files (users.css, search.css, …) into one javascript and stylesheet file (application.css):

config.assets.debug = false
                      

This also cleans up the noisy logging info concerning the asset files that were previously loaded one by one.

When you need to precompile additional assets that aren’t already referenced in you application.js or application.css you’ll want to move the config.assets.precompile directive from production.rb to application.rb:

config.assets.precompile += %w( search.js )
                      

This isn’t directly related but while you are at speeding up our development experience you might also want to have a look at the rails-dev-tweaks gem. This gem defines rules for managing what files get reloaded on page requests in development.

Capybara has current_url, which returns an url that is unmatcheable since it includes a randomized port, and current_path with contains neither query nor fragment(aka hash/anchor) and is therefore rather useless.

Behold the perfect current_path_info, which returns the full path and query + fragment.

Code

def current_path_info
  current_url.sub(%r{.*?://},'')[%r{[/\?\#].*}] || '/'
end

Today I had an interesting experience which I’d like to share with my friends. No, I didn’t discover my dark side, I knew that before. What I found out is that mocking as a test method comes in handy once you get it.

Why Don’t You Use RSpec?

I’ve never been an RSpec fan. Not that I don’t like the BDD-approach, which I do appreciate, I was always scared about its internal complexity. I happened to stay with MiniTest, and this has nothing – and i mean nothing – to do with the opinion of Mr. Rails.

Now, I don’t say that RSpec sucks and is too complex. This would be bullshit and I might even soon switch to using this powerful framework (just to make my friend Justin happy) but right now, this test framework still is too powerful for me.

1
2
3
4
5
6
7
8
9
10

class StringTest  MiniTest::Spec
  describe "String" do
    before do
      @string = "Be nice."
    end
 
    describe "#to_xml" do
      it "returns a string" do
        assert_kind_of String, @string.to_xml
      end

That’s pure MiniTest code, included in Ruby 1.9 and working out-of-the-box.

Testing with Assertions Is Not A Crime!

What MiniTest::Spec gives me is basically a BDD interace for classical testers. I say classic as I learned that testers not using expectations are called classic. Old school. Yo.

I can use the keyword methods describe and it to phrase my requirements to the object (line 2 and 8). Before and after hooks provide convenient setup and teardown blocks (line 3).

However, I still use assertions rather than matchers as found in RSpec. I could use matchers, though. MiniTest provides these.

  it "returns a string" do
    @string.to_xml.must_be_kind_of String
  end

Wow, this is “better readable” in many people’s opinion. I still prefer assertions. But that’s not the point of this essay. What I was going to talk about is mocking. Right, mocking.

Who needs Mocking?

Now let’s say my #to_xml method accepts an additional parameter to configure the behaviour. We’d have to test that, too.

  it "accepts the uppercase option" do
    assert_equal @string.to_xml(uppercase: true),
      "BE NICE."
  end

This could go on forever, and we test all the features that the method provides in numerous test cases – or its – uppercase, lowercase, reverse, whatever.

Until now, I’ve been testing only using state verification. I run the method and test the state of my string instance afterwards using assertions. This is called state verification.

Purposeless, I now decide that my app needs another method. Here’s the implementation.

class String
  def to_exemel(options={})
    log "stupid phonetic alias called"
    to_xml(options)
  end

I told you, it doesn’t make any sense. What I want to point out is, the method #to_exemel delegates its arguments to the #to_xml method, which we already tested thoroughly.

Expect the Unexpected!

Testing the logging makes sense, I already use mocking here, as I mock the #log method. More on that in a minute.

1
2
3
4
5
6

describe "#to_exemel" do
  it "delegates to #to_xml" do
    String.any_instance.expects(:log)
 
    @string.to_exemel
  end

However, testing all the behaviour of #to_xml again doesn’t. Here, I found out that mocking is cool, too.

1
2
3
4
5
6
7
8

describe "#to_exemel" do
  it "delegates to #to_xml" do
    @string.
      expects(:to_xml).
      with(:uppercase => false)
 
    @string.to_exemel(:uppercase => false)
  end

First, we define expectations: In the ongoing test case, the #to_xml method must be called on the @string instance along with the specified options hash (line 3-5).

I then call the actual method which is subject of our test. The method will be run – and the mocking framework internally checks, if the #to_xml method is really invoked the way we want it (line 7). If it doesn’t, the test will fail!

I Never Knew It’s Called Behaviour Verification!

What we do here is behaviour verification – we no longer have an assertion checking the final string state (we could do that additionally) but set up an expectation, which changes the workflow of our test slightly.

• I saved a lot of lines of test code since I do not assert all of #to_xml’s features in the #to_exemel method, again. This is cool.

• Nevertheless, after mocking my test case includes a lot of knowledge about internal implementation. The test now’s aware that #to_exemel forwards to #to_xml. Changing this implementation detail within the class will break the expectation tests, although the second method might still work from the public perspective.

It seems to be a matter of careful choice when to use mocking, and when not. I for myself still lack the experience to make smart assumptions here. However, I can call now myself a Mocker according to Martin Fowler’s great post, which you should definitely read if you haven’t, yet!

Mocking frameworks?

I found mocha to be a very simple to use mocking framework integrating seamless into MiniTest. Never peeked at the source, though. RSpec comes with a powerful mocking functionality as well, which most people might know, use and appreciate.

Outsights?

The example was stupid, I hope I still could point out the difference between status verification and behaviour verification in this short post. Let me know what you think about it. Good nite, and Mock’n‘Roll!

I got tired of clicking through the github menus, so after some toying around with the octopi github client (which does not work as expected…), I wrote this small script using pure rest-client.

Creates a new travis hook based on the travis info in your .gitconfig
and also triggers the initial test-run.

add to ~/.gitconfig

[travis]
  user = xxx
  token = yyy

download & run the Script
wget https://raw.github.com/grosser/dotfiles/master/bin/github-add-travis

./github-add-travis github-password project-name

Problem:

  expected there to be content "\n2\n" in "\n2\n" (RSpec::Expectations::ExpectationNotMetError)

Research:

https://github.com/jnicklas/capybara/issues/56

Answer: WTF!

Solution:

  page.find(".#{count_name}-count").text.strip.should == "\n#{count}\n"
 

Original article writen by suung and published on praktikantenblog | direct link to this article | If you are reading this article elsewhere than praktikantenblog, it has been illegally reproduced and without proper authorization.

Some time ago I came across Teleport which pitches itself as a lightweight way to set up Ubuntu machines. Having failed at mastering the steep learning curve that full-blown tools like Chef or Puppet provide, I was willing to give Teleport a shot. Teleport stays simple by focusing on Ubuntu and providing just the right set of tools to set up the whole Ruby/Rails eco system. This makes it less powerful than the other aforemention tools, but in my case it’s just what I was looking for.

Some facts: Teleport …

• consists of a configuration (Telfile) plus files you want to be present on the target
• copies itself onto the target via ssh and then runs itself there (hence the name)
• works idempotently - run it over and over again without breaking the target

My setup

For each project I store everything that belongs to Teleport in a specific deployment folder. This folder might look something like this:

|____Telfile
                        |____recipes
                        | |____install_passenger_nginx.rb
                        |____files
                        | |____etc
                        | | |____gemrc
                        | | |____init.d
                        | | | |____nginx
                        | |____Gemfile
                        | |____Gemfile.lock
                        | |____opt
                        | | |____nginx
                        | | | |____conf
                        | | | | |____nginx.conf
                        

The configuration is done inside the Telfile:

user "deploy"
                        ruby "1.9.3"
                        server "my-own-vps"
                        
                        apt "deb http://ppa.launchpad.net/chris-lea/node.js/ubuntu lucid main", key: "C7917B12"
                        
                        packages %w(openssl libreadline5 libreadline5-dev curl
                          git-core libopenssl-ruby libcurl4-openssl-dev imagemagick
                          libsqlite3-0 libsqlite3-dev sqlite3 libxml2-dev libxslt1-dev
                          libmysqlclient-dev mysql-client mysql-server nodejs)
                        
                        recipes %w(
                          install_passenger_nginx.rb)
                        

user defines the name of the account you will later use to deploy your app. This user gets added to the sudoers and your ssh key is uploaded for password-less login. ruby tells Teleport which version to install and server is the name you gave to the machine in your ~/.ssh/config.

As you can see you can also define customs apts and packages that should get installed. For performing special setup routines or invoking more complex tasks you can utilize the recipes feature. Recipes are standalone files that can be written in Ruby and allow for sharing and reuse. Here’s an example of how to install nginx via the passenger gem:

version_gem = gem_version("passenger")
                        version_installed = `passenger-config --version`.strip
                        
                        if !version_gem
                          warn "Passenger is not installed. Please add it to your Gemfile."
                        elsif version_installed == version_gem.to_s
                          banner "Passenger #{version_installed} is already installed"
                        else
                          banner "Installing Nginx via Passenger (installed: #{version_installed}, gem version: #{version_gem})"
                          run "passenger-install-nginx-module", %w(--auto --auto-download --prefix=/opt/nginx)  # install without prompting
                          run "update-rc.d nginx defaults"                                                      # activate autostart
                          run "service nginx restart"                                                           # restart nginx
                        end
                        

The files directory contains files and directories that get copied over to the target machine - not much magic about that. To install system-wide gems you can place a Gemfile and Gemfile.lock inside the files directory.

Finally you can run the setup with

$ teleport my-own-vps
                        

Keep in mind that you can re-run this every time you make a change to the Telfile or add new files. Teleport strives to be idempotent and can be run repeatedly without breaking the target machine(unless you explicitely messed up something ;)).

Let’s go…

There is much more to Teleport than what I could cover in this short post. For example you can define multiple servers with different roles or reverse engineering existing servers which makes it very easy to generate a Telfile from your current setup. In case you want to get started head over to the project wiki which contains a lot of information and examples.

The cucumber-rails gem recently deprecated the tableish method which could be used to verify data in tables using Table#diff!. Tableish was easy to use but also had some limitations so that from now on you are advised to use Capybara’s finder API which is much more flexible.

In case you are now searching for a replacement for tableish, look no further:

rows = find("table#selector").all('tr')
                          table = rows.map { |r| r.all('th,td').map { |c| c.text.strip } }
                          expected_table.diff!(table)
                          

This maps the table into a two dimensional array containing all cell contents. The last line diffs this array against the expected_table which is the table that’s passed into the step definition like explained on this Cucumber wiki page.

You’ll have to replace table#selector with the actual table selector you are looking for.

When using capybara-webkit for testing elements that get inserted into the page via JavaScript I had to change this a little bit, because the finder did not seem to work with capybara-webkit. One can circumvent this by parsing the page body and using the finder on the new result:

html = Capybara::Node::Simple.new(body)
                          rows = html.find("table#selector").all('tr')
                          table = rows.map { |r| r.all('th,td').map { |c| c.text.strip } }
                          expected_table.diff!(table)
                          

Spaß beiseite, keiner sprach davon, einen “offiziellen Editor” für Ruby zu bewerben. Nur der Eintrag ganz unten in der NEWS-Datei lässt doch schmunzeln:

1
2

Turning on/off indentation warnings with directives.
  ("# -*- warn-indent: true -*-" / "# -*- warn-indent: false -*-")

Da fühlt man sich als Emacs-Nutzer auf jeden Fall bestätigt. :-p

Valete,
Quintus

Tags: de | Linux | Ruby

…und für alle, die wie ich dem Hanna-Template für RDoc anhängen, habe ich die Ruby-Dokumentation mal mit Hanna erzeugt. Hier geht’s zum Download: ruby-1.9.3-p0-doc.tar.xz (XZ-komprimierter Tarball).

Tags: de | Ruby

just stumpled upon this in the redmine gemfile

https://github.com/jberkel/interactive_editor

Original article writen by suung and published on praktikantenblog | direct link to this article | If you are reading this article elsewhere than praktikantenblog, it has been illegally reproduced and without proper authorization.

Was ist der GIL eigentlich?

Dafür muss man ein wenig ausholen. Wir schreiben das Jahr 1992, alle Welt (oder zumindest ein kleines Dorf in Japan) feiert das Release von Ruby 1.0. Leistungsfähige Hardware ist etwas für Universitäten und große Firmen, Leute, die erschwingliche Rechner mit mehreren Prozessorkernen prophezeihen, werden als Phantasten verschrieen oder gleich als Hexer verbrannt. Langsam erobert Ruby die Welt (oder zumindest Japan) und Hardware wird günstiger. Das neu gebildete »Core-Team« um den Erfinder der Programmiersprache Ruby Yukihiro »Matz« Matsumoto passt Ruby an diese Gegebenheiten an. Ruby 1.2 erscheint. 1.4. Die damals noch als Abtrünnige verfolgten Advokaten der Mehrkernmaschinen finden sich bestätigt, und die Welt (oder zumindest Japan) muss anerkennen, dass sie doch recht gehabt haben. Die Hexer werden rehabilitiert und Threads halten in die Programmiesprache Einzug — doch ist die Codebasis von Ruby schon groß und eigenwillig geworden, und niemand fühlt sich in der Lage, gleich den gesamten Code den neuen Gegebenheiten anzupassen. Die Idee: Wir brauchen gar keine echte Parallelität! Wir tun einfach nur so!
Die Idee wird schleunigst eingebaut. Ruby 1.6 erscheint, und wir haben: Threads! Alle Rubyisten verfallen plötzlich dem einen Gedanken: Wir können die neuen Zweikernprozessoren auslasten! Doch, wie so oft, es sollte wohl nicht sein. Es gab nur Green Threads, d.h. Ruby kümmerte sich selbst um das Scheduling von Threads anstatt es das Betriebssystem tun zu lassen. Und es gab noch etwas: Den GIL, Global Interpreter Lock. Um nämlich die alte Codebasis vor allzuvielen Modifikationen zu schützen, hat sich das Core-Team folgendes überlegt: Es darf nur ein VALUE (C-Typ eines Ruby-Objekts) zu einem Zeitpunkt verändert werden, andernfalls entstehen im Ruby-Kern Race-Conditions. Um das umzusetzen, wurde der GIL eingebaut: Egal, wie viele Threads laufen, nur eine Ruby-Methode kann zeitgleich ausgeführt werden. Keine Race-Conditions im Ruby-Kern, und alles ist in Butter.
Dann veröffentlichen die Pragmatic Programmers »Programming Ruby«, das erste Buch über Ruby in nicht-japanischer Sprache und plötzlich will die ganze Welt (diesmal wirklich) die junge Sprache aus Japan kennenlernen. Und dann geschah, was selbst die Propheten aus den ersten Jahren nicht vorhersehen konnten: Moore’s Law neigt sich dem Ende zu. Anstatt immer mehr Transistoren in einen Kern zu packen, fangen die Computerhersteller an, immer mehr Kerne in einen Prozessor zu packen! Plötzlich wird der als Schutzmechanismus gedachte GIL implizit zur Bremse, weil er durch die Blockade aller Ruby-Anweisungen bis auf eine den Ruby-Interpreter auf einem einzelnen Prozessorkern einsperrt. Auch das Release von Ruby 1.8 ändert an dieser Situation nichts.
Unter Zugzwang, beginnt das Core-Team damit, Überlegungen über die weitere Zukunft von Ruby anzustellen. Große Umwälzungen werden geplant, syntaktische Ungereimtheiten sollen entfernt und ein konsistentes API soll erstellt werden. Und: Der GIL soll weg. Große Mengen an Quellcodeänderungen stehen an, und man bereitet sich darauf vor, die Major-Versionsnummer zu erhöhen: Ruby 2.0 soll die neue Inkarnation heißen!
Doch die Entwicklung verzögert sich. Immer neue Bugs und neue Features strömen in den Interpreter hinein und wieder hinaus. Das geplante Release-Datum verstreicht. Ein Jahr später spricht Matz ein Machtwort: Wir brauchen eine neue Version!
Die unfertige Entwicklerversion 1.9, zu erkennen an der ungeraden Versionsnummer, wird in aller Eile einem Feature Freeze unterzogen und ein weiteres Jahr später erscheint Ruby 1.9.0. In der Ruby-Community bricht ein Sturm los. Die Änderungen zur Vorversion sind abgrundtief und ozeanweit, nicht zu rechtfertigen, dass dies keinen Sprung der Major-Versionsnummer gab! Die Ruby-Community teilt sich in 1.8-Fanatiker und 1.9-Phantasten. Das deutschsprachige Ruby-Forum führt zwei verschiedene Rubriken für 1.8 und 1.9. Neben zahlreichen anderen Neuerungen wurde in Ruby 1.9 der GIL angegriffen: Ruby überlässt das Thread-Scheduling nun wie es sich gehört dem Betriebssystem, bleibt aber bei der Blockade aller Ruby-Anweisungen bis auf einer. Jedoch haben die Entwickler einen Kniff eingebaut, der es C-Extensions ermöglicht, dem Interpreter mitzuteilen: »Eh du, ich brauche die Ruby-Runtime gerade nicht, ich mache externe Berechnungen. Lass’ mal einen anderen Thread laufen, während ich arbeite, ich sag’ dir schon bescheid, wenn ich wieder einen VALUE brauche.«
Ruby 1.9.1 und 1.9.2 erscheinen, dann dieser Blogpost, der sich mit dem oben beschriebenen Kniff namens rb_thread_blocking_region() befasst.

Umgehen des GIL

Wie man sieht, handelt es sich beim GIL um eine historisch gewachsene Struktur, die Ruby auch nicht in aller Eile wird abwerfen können. Ich hege nach wie vor die Hoffnung, dass mit Ruby 2 auch der MRI endlich Parallelität ermöglich wird, aber es ist glaube ich wahrscheinlicher, dass Rubinius vorher 1.9-Kompatibilität erreicht. Andererseits, Duke Nukem forever ist erschienen und damit auch GNU Hurd! :-D
Wie oben beschrieben, kann man den Ruby-Interpreter aber dazu bewegen, Nicht-Ruby-Code nebenher laufen zu lassen, während anderer Ruby-Code läuft. Das geht nur innerhalb von C-Extensions und auch dort nur mithilfe einer Funktion namens rb_thread_blocking_region. Das liest man am besten als (rb_thread)(blocking_region), denn das hat nichts damit zu tun, dass hier ein Thread blockiert wird, sondern damit, dass hier im Namespace rb_thread eine Funktion blocking_region() existiert.

Man kann sich den GIL wie eine gigantische Mutex vorstellen. Wer schonmal nebenläufig in Ruby mit Threads programmiert hat (soweit es eben ging), kennt folgendes Muster:

1
2
3
4

mutex = Mutex.new
mutex.synchronize
  #Hier darf nur ein Thread drin werkeln...
end

Wäre der GIL in Ruby implementiert, sähe das wohl etwa so aus:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

class RubyVM

  def initialize(argv)
    #...
    @GIL = Mutex.new
  end

  # ...

  def execute_expr(expr)
    @GIL.synchronize do
      evaluate(expr)
    end
  end

end

rb_thread_blocking_region() nun kann man sich so vorstellen:

1
2
3
4
5
6
7
8
9
10
11
12

class RubyVM

  # Man beachte, dies wird irgendwo innerhalb von
  # #execute_expr aufgerufen.
  def rb_thread_blocking_region(blocking_function, arg, unblock_function, arg)
    @GIL.unlock
    blocking_function(arg)
    #...
    @GIL.lock
  end

end

Sinn und zweck von rb_tbr() (ich kürze diesen langen Namen jetzt einfach mal ab, das ist aber kein gültiger C-Code) ist es, blocking_function aufzurufen, ohne der Funktion aber den GIL zuzugestehen, den sich somit ein anderer Thread besorgen kann. Selbstverständlich darf man innerhalb von blocking_function dann nicht Gebrauch von der Ruby-API machen, da wir dann bei den von mir weiter oben beschriebenen Race-Conditions im Ruby-Kern ankämen.

Anwendung im C-Code

Die Funktionssignatur von rb_tbr() findet sich in intern.h und sieht so aus:

1
2

VALUE rb_thread_blocking_region(rb_blocking_function_t *func, void *data1,
                                rb_unblock_function_t *ubf, void *data2);

Wer einen Blick auf meinen Pseudo-Ruby-Code weiter oben wirft, erkennt, dass ich die Signatur in etwa übernommen hatte und errät vielleicht auch schon den Sinn der ganzen Sache. Nun, eins nach dem anderen. Das erste Argument, rb_blocking_function_t *func, ist ein Funktionspointer zu der Funktion, die ohne GIL auskommen soll. Ruby erwartet folgende Signatur von dieser Funktion:

VALUE funcname(void* ptr);

Der Rückgabewert dieser Funktion wird automatisch zum Rückgabewert von rb_tbr() — egal, ob die Operation unterbrochen wird oder nicht, doch dazu später mehr. Zunächst einmal haben wir da noch das zweite Argument, *data1. Dies ist ein Argument, das einfach (wie man oben an meinem Pseudo-Ruby-Code auch sehen kann) an die blockierende Funktion weitergereicht wird. Es handelt sich hierbei um einen Pointer vom Typ void, weil die blockierende Funktion auch mehrere Argumente haben könnte und deren Typen nicht bekannt sind — auf diese Art und Weise kann man jedoch mit unorthodoxem Casting alles zurecht biegen. Wer sich schonmal intensiver mit Rubys C-API befasst hat, kennt das Problem von rb_rescue(), das sich da genauso verhält. In meinen Augen sollte man das anders lösen.
Nun aber zum dritten Argument, rb_unblock_function_t *ubf. Auch dies ist ein Funktionspointer, und die Funktion, auf die er zeigt, soll wie folgt aussehen:

void funcname(void* ptr);

Diese Funktion ist dazu gedacht, die blockierende Funktion zu unterbrechen. Der Grund für die Existenz dieses Parameters ist recht einfach: Es gibt Situationen, in denen muss der Interpreter den Thread, in dem die blockierende Funktion ausgeführt wird, abrrechen. Zwei fallen mir spontan ein: Der explizite Aufruf von Thread#terminate und das Beenden des Interpreters, entweder regulär durch Erreichen des Dateiendes des Hauptskripts oder explizit durch Aufruf von #exit. Das vierte Argument von rb_tbr() funktioniert analog zum Zweiten, nur dass der Pointer diesmal an die Abbruchfunktion weitergereicht wird.

So, nun wird es aber Zeit für ein konkretes Beispiel. Wir haben folgende externe Library, gegen die wir unsere C-Extension linken wollen (ist nicht die schönste Lib, und sie wird auch mit mehr als zwei Threads bedenklich, aber hier geht’s ums Beispiel):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

/* mylib.c */
#include <unistd.h>
#include "mylib.h"

/* Checked by long_calculation() and set by abort_calculation().
 * If this is 1, long_calculation() aborts it's exection.
 */
static int abort = 0;

/**
 * Needs a long time to calculate something very important.
 * Call abort_calculation() if you want to abort the calculation.
 * If you do so, this function immediately returns 0.
 *
 * You can set the calculation time with +seconds+.
 */
int long_calculation()
{
  int i;

  /* Use for loop to allow aborting the calculation */
  for(i = 0; i < SLEEP_SECONDS; i++){
    if(abort){
      abort = 0; /* For next time */
      return 0;
    }
    else
      sleep(1); /* Simulate computation */
  }
  return 42;
}

/**
 * Aborts the calculation taken out by long_calculation().
 * If no calculation is taken out, prevents the next call to
 * long_calculation() from computing anything.
 */
void abort_calculation()
{
  abort = 1;
}

1
2
3
4
5
6
7
8
9

/* mylib.h */
#ifndef MYLIB_H
#define MYLIB_H
#define SLEEP_SECONDS 10

int  long_calculation();
void abort_calculation();

#endif

Wie man erkennt, bietet die Lib eine Funktion long_calculation() an, die irgendeine lange Berechnung durchführt und das Ergebnis zurückgibt. Dauert einem das zu lange, so kann man in einem anderen Thread abort_calculation() aufrufen, was wiederum long_calculation() zum sofortigen Zurückgeben von 0 zwingt.

Im folgenden eine C-Extension, die eine Klasse Foo erstellt, die zwei Instanzmethoden enthält: Foo#blocking_foo ohne den Aufruf von rb_tbr() und Foo#nonblocking_foo mit dem Aufruf von rb_tbr(). Beide bedienen sich einer Funktion do_long_calculation(), die der von rb_tbr() erwarteten Funktionssignatur entspricht und das Ergebnis von long_calculation() in einen Ruby-Numeric umwandelt.

Hier zunächst do_long_calculation():

1
2
3
4
5
6
7
8

/*
 * Calls the external long_calculation() and returns
 * the result as a VALUE. 
 */
static VALUE do_long_calculation(void* ptr)
{
  return INT2NUM(long_calculation());
}

Der Parameter ptr existiert nur, um es rb_tbr() recht zu machen. Ich benutze ihn ganz offensichtlich nicht, ihr könnt das in euren C-Extensions natürlich anders handhaben.

Nun die blockende Funktion:

1
2
3
4
5
6
7
8
9
10
11

/*
 * call-seq:
 *   blocking_foo() ==> an_integer
 *
 * Calls some long-running external function and blocks
 * the whole MRI. 
 */
static VALUE blocking_foo(VALUE self)
{
  return do_long_calculation(NULL);
}

Da do_long_calculation() ohnehin sein Argument ignoriert, kann ich einen beliebigen Wert hineingeben, hier etwa den NULL-Pointer.

Wie der Aufruf einer solchen Funktion aus Ruby heraus abläuft, sollte jedem, der schonmal eine C-Extension programmiert hat, klar sein.

1
2
3

require_relative "./foo"
f = Foo.new
f.blocking_foo

Die dritte Zeile wird 10 Sekunden blockieren und dann 42 zurückgeben. Interessant wird es hierbei:

1
2
3
4
5
6
7
8
9

require_relative "./foo"

Thread.new do
  f = Foo.new
  f.blocking_foo
end

sleep 1
#Interpreter shutdown here

Normalerweise würde das Beenden des Interpreters jeden Thread, auf den nicht explizit mit Thread#join gewartet wird, abschießen. Da Ruby aber keine Möglichkeit kennt, die blocking_foo()-Funktion zu unterbrechen, zögert sich das Beenden des Interpreters so lange heraus, bis blocking_foo() zurückgibt.

Ein anderes Beispiel ähnlich dem aus meinem vorigen Posting:

1
2
3
4
5
6
7
8

require_relative "./foo"

Thread.new do
  f = Foo.new
  f.blocking_foo
end

loop{puts "!"}

Weil f.blocking_foo den GIL hält und nicht wieder hergibt, können die Anweisungen im Hauptthread nicht ausgeführt werden, bis f.blocking_foo zurückgibt. Keine Ausrufungszeichen erscheinen.

Nun probieren wir beide Beispiele im Zusammenspiel mit rb_tbr(). Zunächst brauchen wir eine Abbruchfunktion, die der vom dritten Parameter rb_tbr()s Genüge tut:

1
2
3
4
5

/* Calls the external abort_calculation(). */
static void do_abort_calculation(void* ptr)
{
  abort_calculation();
}

Auch hier ist der Parameter ptr nur wieder Dekor, damit die Funktion zur gewünschten Signatur passt. Nun der interessante Teil: Wie macht man das mit rb_tbr()?

1
2
3
4
5
6
7
8
9
10
11
12
13
14

/*
 * call-seq:
 *   nonblocking_foo() ==> an_integer
 *
 * Same as #blocking_foo, but doesn't block MRI while
 * executing.
 */
static VALUE nonblocking_foo(VALUE self)
{
  return rb_thread_blocking_region(do_long_calculation,
                                   NULL,
                                   do_abort_calculation,
                                   NULL);
}

Es ist einfach nur die Anwendung von dem, was ich weiter oben beschrieben habe. Wir teilen rb_tbr() mit, dass do_long_calculation() den GIL nicht braucht und geben der Funktion einen Parameter mit (wie schon zuvor den Nullpointer). Hinzu kommt, dass wir Ruby noch mitteilen, wie es die Funktion unterbrechen kann: Indem es do_abort_calculation() aufruft, welchem wir ebenfalls einen Nullpointer als Parameter mitgeben. Da der Rückgabewert von rb_tbr() dem Rückgabewert von do_long_calculation() entspricht und ich den nicht weiterverarbeiten möchte, gebe ich ihn einfach als Rückgabewert von nonblocking_foo() zurück.
Wer sich wundert, warum ich innerhalb von do_long_calculation() INT2NUM() aufrufen kann: INT2NUM() ist keine Funktion des Ruby-APIs, sondern ein Makro (Code aus ruby.h):

1
2
3

#define INT2FIX(i) ((VALUE)(((SIGNED_VALUE)(i))<<1 | FIXNUM_FLAG))
/* ... */
# define INT2NUM(v) INT2FIX((int)(v))

Hier können keine Race-Conditions entstehen, weil der VALUE erst von uns »erstellt« wird und ein anderer Thread überhaupt keinen Zugriff darauf hat.

Nun dieselben Tests wie oben noch einmal:

1
2
3
4
5
6
7
8
9

require_relative "./foo"

Thread.new do
  f = Foo.new
  f.nonblocking_foo
end

sleep 1
# Interpreter shutdown here

Und siehe da, der Interpreter fährt plötzlich sehr viel schneller herunter als mit f.blocking_foo. Der Grund dafür ist, dass Ruby beim Herunterfahren des Interpreters Gebrauch von der Abbruchfunktion gemacht hat.

Und wie sieht es mit der Nebenläufigkeit aus?

1
2
3
4
5
6
7
8

require_relative "./foo"

Thread.new do
  f = Foo.new
  f.nonblocking_foo
end

loop{puts "!"}

Auch das funktioniert! f.nonblocking_foo hält nicht den GIL, also kann der Hauptthread, oder genauer gesagt, dessen einzelne Anweisungen, ihn sich schnappen und muss nicht auf das Ende von f.nonblocking_foo warten.

Soweit der kleine Ausflug in Rubys C-Interna in Sachen Multithreading. Der gesamte Code für diesen Blogpost kann hier heruntergeladen werden, ihr könnt

$ rake compile

nutzen, um sowohl mylib als auch die C-Extension foo zu kompilieren. Nutzt anschließend

$ rake test

um die Datei test.rb auszuführen. Grund hierfür ist, dass sich mylib nicht an einem Ort befindet, der vom Betriebssystem durchsucht wird, und das Einbinden von foo würde mit einem LoadError enden, der euch sagt, dass mylib.so nicht gefunden werden konnte.

Valete,
Quintus

Tags: C | de | Ruby

All my new projects will be released under MIT / those I work on will be switched, which will still means the same as before: Do wtf you want with this code, no need for any naming/license copying.

After reading some articles I realized that public domain does not always work since some countries do not allow it and its also unclear what happens when the code causes problems lawsuit.

Ill also try to add a s.license = ‘MIT’ to all gemspecs, to make license management easier.

/\/((?:[\w\-\_]+\.\w{3,4}))(?:\?.+)?$/

Original article writen by suung and published on praktikantenblog | direct link to this article | If you are reading this article elsewhere than praktikantenblog, it has been illegally reproduced and without proper authorization.

Und ich muss sagen, es gefällt mir. Rubinius ist zwar offiziell noch nur 1.8-kompatibel (undenkbar für mich, dahin zurückzugehen!), aber der momentane Trunk kann auch schon 1.9, wenn auch nicht perfekt (im heutigen Schnelltest sind mir allerdings keine Probleme aufgefallen, außer, dass die Startzeit fast 20 Sekunden betragen hat).

Installation

Man kann Rubinius momentan nur mithilfe eines 1.8er-Rubies installieren (außerdem müssen Rake und RubyGems installiert dafür sein). Hat man also ein solches fertig und—das ist wichtig—als erstes im PATH eingetragen, so klont man einfach das Rubinius-Repository und geht auch ansonsten den folgenden Kommandos nach:

1
2
3
4

$ git clone https://github.com/rubinius/rubinius.git
$ cd rubinius
$ ./configure --prefix=/pfad/wos/hin/soll
# rake install

Wichtig dabei, es ist nicht möglich, ein Ruby mit einem Program-Suffix zu verwenden. Mein 1.8er ist ja bekanntlicherweise als /opt/rubies/ruby-1.8.7-p352/bin/ruby18 installiert, aber das hat nicht hingehauen. Ich musste ein wenig mit Symlinks spielen, um das geradezurücken.

Mein Rubinius habe ich nach /opt/rubies/rubinius innstalliert, daher heißt die Rubinius-Executable /opt/rubies/rubinius/rbx. Startet man diese ohne jegliche Argumente, landet man in einer 1.8er-IRB. Alternativ kann man Rubinius auch den Pfad zu irgendeinem Ruby-Skript geben, das es dann ausführt.

Da ich ja besonders an der Nebenläufigkeit von Threads interessiert bin (MRI trägt ja bekanntlicherweise den GIL mit sich herum, der verhindert, dass mehr als eine Ruby-Anweisung gleichzeitig ausgeführt werden kann), habe ich folgendes Skript geschrieben:

1
2
3
4
5
6
7

Thread.new do
  loop do
    puts "!"
  end
end

loop{}

Die Qualität mag jeder anzweifeln wie er will, aber die Endlosschleife im Hauptthread blockiert MRI absolut. Kein einziges Ausrufungszeichen erscheint auf der Kommandozeile. Rubinius hingegen meistert das souverän und weicht auf einen anderen Prozessorkern aus, um den zweiten Thread durchzuführen. Ich bin restlos begeistert. Endlich wahre Concurrency mit Ruby! :-D

Wie steht’s nun aber mit Ruby 1.9? Da ich wie gesagt unter keinen Umständen zu 1.8 zurückkehren will (dafür bin ich viel zu sehr ein Fan des neuen Encoding-Systems!), wollte ich natürlich Rubinius im 1.9er-Modus testen. Wie sich herausstellt, ist das so experimentell, dass es nicht mal irgendwo offiziell dokumentiert ist, wie man den Modus startet, aber ein Blogpost brachte mich auf die richtige Spur. So geht’s:

$ rbx -X19

Die Startzeit ist wie gesagt sehr beträchtlich und belastet einen meiner vier Kerne auch die ganze Zeit lang mit 100%. Schließlich aber startet eine IRB im 1.9er-Modus und zumindest in IRB konnte ich keine Geschwindigkeitsunterschiede zu 1.8 feststellen. Ich gehe aber davon aus, dass “normaler” Code in Dateiform schon einen signifikaten Geschwindigkeitsunterschied aufweisen wird.

Da Rubinius auch noch C-API-kompatibel zum MRI ist, gibt es nur noch wenig, was mich davon abhält, überzuspringen. Sobald die 1.9-Kompatibilität offiziell wird (das ist mit dem nächsten Mayor Release 2.0.0 geplant), werde ich wohl schnellstens wechseln. Die echte Parallelität von Threads hat mich begeistert!

Valete,
Quintus

PS: Für die Windows-Nutzer unter euch: Laut Roadmap wird Windows noch nicht offiziell unterstützt. Aber vielleicht mag es mal jemand trotzdem probieren?

Tags: C++ | de | Ruby

My codebrawl entry (gist) got the 5th place (out of 20). Want to use it in real life :)

http://www.imaginek12.com

Original article writen by antje and published on praktikantenblog | direct link to this article | If you are reading this article elsewhere than praktikantenblog, it has been illegally reproduced and without proper authorization.

i never use 'or' so i found this interesting.

Original article writen by suung and published on praktikantenblog | direct link to this article | If you are reading this article elsewhere than praktikantenblog, it has been illegally reproduced and without proper authorization.

As Ruby programmer you are used to write code like this, if you want to have a default value or rather make sure, that something is an Array or Hash, in order to rely on that for later treatment

  def filters

    @filters ||= { }

  end

In Active Record that does not work, cuz:

@query.instance_variable_get :@filters
=> nil

@query.attributes

=> {"name"=>"_", "column_names"=>nil, "group_by"=>nil, "project_id"=>1, "sort_criteria"=>nil, "user_id"=>0, "search_mode"=>"all", "filters"=>{:subproject_id=>{:value=>[], :operator=>"*"}}, "is_public"=>false}

there is also obie's great post about this: http://www.jroller.com/obie/entry/default_values_for_activerecord_attributes

Original article writen by suung and published on praktikantenblog | direct link to this article | If you are reading this article elsewhere than praktikantenblog, it has been illegally reproduced and without proper authorization.

A fresh install for my laptop, with some Desktop tools and all the rails/ruby stuff I need.

1. Choose a short name for your computer, it will later show in the prompt. Only enter your firstname/alias, it will always show in the toolbar.
2. Enable canonical partners in “Software Sources” > Other Software
3. Install RBEnv

   sudo apt-get install curl zlib1g-dev libssl-dev libreadline6-dev libncursesw5-de libxml2 libxml2-dev libxslt1-dev libsqlite3-dev -y
   cd
   git clone git://github.com/sstephenson/rbenv.git .rbenv
   echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc
   echo 'eval "$(rbenv init -)"' >> ~/.bashrc
   git clone git://github.com/sstephenson/ruby-build.git
   cd ruby-build
   sudo ./install.sh
   rbenv install # choose which rubies you want (atm ree is broken...)
   
   # rbenvsudo
   function rbenvsudo(){
     executable=$1
     shift 1
     sudo $(rbenv which $executable) $*
   }
   

4. Rubymine to /opt/rubymine

   Command key:
   Choose “meta is mapped to left win” in advanced keyboard layout options

   Executable:
   /opt/rubymine/bin/rubymine.sh and setup mine as executable then

   alias m="mine . > /dev/null 2>&1 &"
   

   Fix multi-clipboard issues:

   echo 'idea.use.alt.clipboard.sync=true' >> /opt/rubymine/bin/idea.properties
   

   Sun-java: It is no longer in ubuntu repos.

   sudo add-apt-repository ppa:ferramroberto/java
   sudo apt-get update
   sudo apt-get install sun-java6-jre
   export RUBYMINE_JDK="/usr/lib/jvm/java-6-sun/jre"
   

5. Map unity menu from super to Super+z, so you can use it with shortcuts (see below) and with rubymine commands
   sudo apt-get install compizconfig-settings-manager -y
   open compiz application and rebind unity laucher @ “ubuntu unity plugin”, dont mess with anything else, I managed to make ubuntu no longer boot into a GUI with just a few clicks
6. Shortcuts
   Super+D = desktop
   Super+E = open home
   Super+Up = maximize
   Super+Down = minimize
   Super+Q = quit application (Alt+F4)
7. Chrome sudo apt-get install chromium-browser -y
8. Git sudo apt-get install git-core
9. Ruco commandline editor gem install ruco
10. copy important dotfiles + firefox history/forms/… database from .mozilla
11. dotfiles
12. Multi-clipboard sudo apt-get install clipit
    (set history shortcut: Ctrl+Alt+V)
13. Skype + add skype to startup
    (video fix in dotfiles –> add add /home/xxxx/dotfiles/bin/video-skype to “Startup Spplications” )
14. Mysql:
    sudo apt-get install mysql-server mysql-client libmysql-ruby libmysqlclient-dev -y
    I`d recommend no root password
15. Redis sudo apt-get install redis-server -y
16. Memcached sudo apt-get install memcached
17. ImageMagic sudo apt-get install imagemagick libmagick9-dev -y
18. Arial/Verdana etc fonts: sudo apt-get install msttcorefonts
19. VirtualBox
20. Nginx + Passenger
21. Restrcited codes sudo apt-get install ubuntu-restricted-extras -y
22. Terminal
    Set Ctrl+Shift+L for clear+reset in terminal, so the history gets removed on keypress
    Set Ctrl+t for new tab
23. Postgres
    sudo apt-get install postgresql postgresql-devel
    gem install pg

    # change /etc/postgresql/9.1/main/pg_hba.conf from peer to trust
    local all postgres trust

    sudo /etc/init.d/postgresql restart

    use username: postgres + no password in database.yml

24. Reverse scrolling
    If you often work on macs this can reduce your confusion.
    sudo add-apt-repository ppa:zedtux/naturalscrolling
    sudo apt-get update
    sudo apt-get install naturalscrolling -y

The 4th episode discusses partial updates of widgets.

• Using the event instance to transport payload data between widgets.
• Updating the page by sending back arbitrary JavaScript and markup from trigger states.
• Extracting partials to separate widget views.

• Using #render in both views and states for DRY code.

BTW, please feel free to send me feedback or requests for future episodes – I appreciate!

The 4th episode discusses partial updates of widgets.

• Using the event instance to transport payload data between widgets.
• Updating the page by sending back arbitrary JavaScript and markup from trigger states.
• Extracting partials to separate widget views.

• Using #render in both views and states for DRY code.

BTW, please feel free to send me feedback or requests for future episodes – I appreciate!

Is it a good sign when a minor release doesn’t yield any stunning new features? When there’s nothing breathtaking to talk about? I guess with Cells it is – the gem finally seems to get into what it should have been 6 years ago: a mature view components framework for Rails.

The 3.7 release, again, lost code – it’s goal was getting rid of the #options hang-over which succeeded. No more state in your cell, except if you choose to do so.

Here’s what changed.

Rails 3.x

Well, needless to say that 3.7 still runs with Rails 3.0 and 3.1. We were able to incorporate a couple of changes into Rails 3.1 to make Cells’ life even easier. Thanks, José for being such a patient maintainer and happy birthday to you

Caching Inheritance and Conditionals

Arthur Gunn brought us the :if option for caching. So, if you ever needed conditional caching in your cell, here it is.

class CommentsCell  Cell::Base
  cache :show, 
    :if => proc { |cell, opts| opts[:enable_cache] }

To pass in the required options, I’d use #render_cell.

render_cell(:comments, :show, :enable_cache => false)

See how easy it is to bypass (or enable) caching using :if? Also, don’t forget that the second block parameter is the options passed to #render_cell, which brings us to the next core change.

BTW, as a nice extension and in good OOP manners cpb made the cache configurations inheritable. A cell derived from CommentsCell will automatically inherit the cache setting for #show – no need for redundancy.

No more Options!

Luckily, we were able to remove the #options behaviour from Cells – it added unnecessary state, and we try to avoid internal state wherever possible (unlike the Rails core). Where you used to access the magic options hash in a state method you now use explicit state-args.

class CommentsCell  Cell::Base
  def show(user, comments)
    @user, @comments = user, comments
    render
  end

Since the method expects arguments (“state-args”), here’s how you’d pass them into.

render_cell(:comments, :show, user, @comments)

Any additional argument after the state name (2nd argument) is passed to the state. The rest is up to you.

If you want the old behaviour with the options hash that was “simply there”, just include the Deprecations module.

class CommentsCell  Cell::Base
  include Deprecations
 
  def show
    @user = options.first

Test your View Assigns

Another small addition is the #view_assigns method in Cell::TestCase. This works just like in conventional controller test (both Test::Unit and RSpec).

it "should assign correct values" do
  render_cell(:comments, :show)
  assert_equal @user_1, view_assigns[:user]

The view_assigns hash captures all instance variables that were assigned in the rendering cycle. Some people may like that – I don’t. However, here it is.

Have fun with Cells!

Problem
When using composed_of with Money you get this nice error when trying to get the money object while the cents column is nil.
Additionally Money.new(nil) does not work, neither do strings like Money.new(params[:cents]).

Solution
Overwrite the initializer to also accept nil and strings

# config/initializers/money.rb
require 'money'
class Money
  def initialize_with_default_currency(*args)
    args[0] = args[0].to_i if args[0].is_a?(String) or args[0].blank?
    args[1] ||= Money.default_currency
    initialize_without_default_currency(*args)
  end
  alias_method_chain :initialize, :default_currency
end

I just released Apotomo 1.2 today. Yeah. People screaming, clapping, hooting, going nuts, I can hear it. Ok, this release is nothing special, but it fixes a severe bug. No, it wasn’t just a bug, it was more, it was a design flaw!

What’s new?

Nothing. I already said that. However, in the last release we added some option that allowed the following.

  responds_to_event :click, :passing => :root

The :passing option makes it easier to attach event handlers to other widgets, for instance, :root. This worked fine on a 2-level setup, but having deeper widget trees, this failed silently.

The problem – in short – was that the parent widget sometimes wasn’t available yet when the child set up its event handlers. The fix: The constructor Widget.new now requires the parent widget as the first argument. Makes things way simpler and explicit.

What can I do with that?

Well, you now can safely use the :passing option anywhere. You can also use root and parent in your has_widgets blocks now. They will be what you expect.

class CommentsWidget  Apotomo::Widget
  has_widgets do |me|
    me.root # => returns the real root
    me.parent # => well, yeah...

All in all, the new constructor makes the Apotomo core a lot cleaner and more understandable.

What changed?

Be warned that widget doesn’t return a real widget instance anymore (line 3). Don’t be worried, you have plenty of ways do to what you want.

1
2
3
4
5
6
7
8

has_widgets do |root|
  root  widget(:comments) # This still works.
  widget(:comments).markup! # This doesn't.
 
  root[:comments].markup! # This works, of course.
 
  comments = CommentsWidget(root, :comments) # works.
  comments.markup! # yepp, works.

Remember, you’re free to use the real constructor in a has_widgets block (line 7) if you need to grab the created instance for whatever reason. You could also use the block for that.

  has_widgets do |root|
    root  widget(:comments) do |comments|
      comments.markup!
    end

In short, the syntax now is a DSL construction, what happens internally is a call to the actual constructor. Easy.

The second big change is that we got rid of the after_add hook. Since widgets receive their parent explicitely in the constructor, there’s no need for a hook after adding a widget. Use the after_initialize hook for that.

Check the CHANGELOG for further notes and let me know if the changes work for you.

Kevin- thanks for donating LOVE to Austin!

Starten wir einfach mit der Anwendung, die wir beim letzten mal programmiert haben:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

# -*- coding: utf-8 -*-

require "wx"

class MainFrame < Wx::Frame
  include Wx

  def initialize(parent = nil)
    super(parent, title: "Hello, world!", size: [400, 400])
    create_widgets
    setup_event_handlers
  end

  private

  def create_widgets
    @text_ctrl   = TextCtrl.new(self, pos: [20, 20], size: [200, -1])
    @save_button = Button.new(self, pos: [240, 20], label: "Save")
    @show_button = Button.new(self, pos: [20, 50], label: "Show")
  end

  def setup_event_handlers
    evt_button(@save_button, :on_save_button_clicked)
    evt_button(@show_button, :on_show_button_clicked)
  end

  # Event handling

  def on_save_button_clicked(event)
    @name = @text_ctrl.value
  end

  def on_show_button_clicked(event)
    md = MessageDialog.new(self, 
                           caption: "Your name", 
                           message: "Your name is: #{@name}",
                           style: ICON_INFORMATION | OK)
    md.show_modal
  end

end

class MyApp < Wx::App

  def on_init
    @mainwindow = MainFrame.new
    @mainwindow.show
  end

end

MyApp.new.main_loop

Diesmal liegt unser Augenmerk darauf, die lästigen Positionsberechnungen loszuwerden und außerdem ein dynamisches Layout zu schaffen. Wer das Fenster etwa mal vergrößert und verkleinert hat, dürfte festgestellt haben, dass das Layout darauf in keinster Weise reagiert, was für Anwendungen, die maximiert werden sollen, denkbar schlecht ist. wxRuby bietet für solche Dinge Objekte namens Sizers an.

Was sind Sizers?

Ein Sizer lässt sich vom Prinzip her mit einem Spanngurt vergleichen: Man macht ihn an zwei Seiten fest, und er nimmt den gesamten Raum zwischen diesen beiden Punkten ein, etwa so:

|(================================)|

Wenn man nun z.B. auf der rechten Seite etwas zwischen Wand und Gurt packt, zieht der Gurt sich zusammen und nimmt nur noch den verbliebenen Raum ein.

|(==============================)##|

Ich sehe ein, das Beispiel hinkt etwas, aber mir fiel nichts besseres ein. Die korrekte Darstellung des zusammengezogenen Sizers sähe so aus:

|(==============================##)|

Denn in wxRuby “befüllt” man Sizers und packt sie nicht irgendwo zwischen. Neben den sich verkleinernen Sizers gibt es auch noch welche, die sich genau umgekehrt verhalten: Sie versuchen, soviel Raum wie möglich zu bekommen.

Vorüberlegungen

Die in wxRuby am häufigsten gebrauchte Sizer-Klasse ist Wx::BoxSizer, genau genommen Wx::HBoxSizer für horizontale “Spanngurte” und Wx::VBoxSizer für vertikale (beides Subklassen von Wx::BoxSizer). Baut man eine GUI, ist es also sinnvoll, sich zunächst zu überlegen, ob die einzelnen Widgets horizontal oder vertikal voneinander abhängen. Die oberste Ebene der GUI (Sizer werden fürgewöhnlich verschachtelt, um den gewünschten Effekt zu erreichen) ist in der Regel vertikal orientiert, d.h. die Widgets befinden sich untereinander. Hat man aber ein Eingabefeld mit einem Button rechts daneben (so wie wir oben), ist dies eine horizontale Beziehung innerhalb einer Zeile des vertikalen Sizers. Skizziert man die vorhandene GUI mithilfe von Sizers, so könnte etwa das folgende Layout entstehen:

1
2
3
4
5

↑
| (Textfeld) ←--→ (Button)
| (Leer)
| (Button)
↓

Umsetzung

Diese Skizze gilt es nun in Sizer-Code umzusetzen. Der erste Schritt dabei ist es, die expliziten Positionsangaben aus den new-Aufrufen zu entfernen (wir lassen ja schließlich wxRuby rechnen):

1
2
3
4
5

def create_widgets
  @text_ctrl   = TextCtrl.new(self)
  @save_button = Button.new(self, label: "Save")
  @show_button = Button.new(self, label: "Show")
end

Um das Layout von der Logik zu trennen, führen wir eine neue Methode ein, die ebenfalls aus initialize heraus aufgerufen wird:

1
2
3
4
5
6
7
8
9
10
11
12

def initialize(parent = nil)
  super(parent, title: "Hello, world!", size: [400, 400])
  create_widgets
  make_layout
  setup_event_handlers
end

#...

def make_layout

end

Achtet bitte darauf, die Methode nicht layout zu nennen, denn es gibt schon eine Methode Wx::Window#layout, die schon von wxRuby belegt ist und nicht überschrieben werden sollte.

Sizers!

Fangen wir mit dem obersten, dem vertikal orientierten, Sizer an.

1
2
3
4

def make_layout
  @top_sizer = VBoxSizer.new
  self.sizer = @top_sizer
end

Es ist wohl nicht zwangsweise notwendig, den Sizer in einer Instanzvariablen zu speichern, aber ich persönlich halte das der Übersicht wegen für besser. Wer mag, kann den Sizer auch direkt bei self.sizer setzen oder in einer lokalen Variable speichern (wir brauchen ihn nämlich gleich wieder). Die schon angesprochene zweite Zeile teilt wxRuby mit, dass dieses Fenster nicht manuell, sondern mithilfe eines Sizers verwaltet werden soll und welcher Sizer das denn sein soll. wxRuby benutzt diesen “obersten” Sizer als Grundlage für alle folgenden Layout-Berechnungen.

Jetzt haben wir also einen Sizer, der so klein wie möglich zu sein versucht und neue Elemente vertikal ausrichtet, d.h. würden wir Widgets an den Sizer anfügen, käme soetwas dabei heraus:

1
2
3
4
5

↑
(Widget1)
(Widget2)
(Widget3)
↓

Genau das machen wir jetzt auch, indem wir den show_button hinzufügen:

1
2
3
4
5
6

def make_layout
    @top_sizer = VBoxSizer.new
    self.sizer = @top_sizer

    @top_sizer.add_item(@show_button)
  end

Man erkennt, neue Elemente werden mithilfe von Sizer#add_item an einen Sizer angefügt. Das ist aber noch längst nicht alles, es gilt, auch das Textfeld und rechts neben ihm den zweiten Button hinzuzufügen — das ist jedoch nicht ganz so einfach. Schaut man sich die weiter oben verfertigte Skizze an, so stellt man fest, dass sich Speicherbutton und Textfeld in einer horizontalen Beziehung befinden. Wir brauchen demnach einen horizontalen Sizer, arbeiten aber momentan mit einem Vertikalen. Was also tun? Ganz einfach: Einfach einen horizontalen Sizer zum Vertikalen hinzufügen!

1
2
3
4
5
6
7
8
9
10
11

def make_layout
    @top_sizer = VBoxSizer.new
    self.sizer = @top_sizer

    sizer = HBoxSizer.new
    sizer.add_item(@text_ctrl)
    sizer.add_item(@save_button)
    @top_sizer.add_item(sizer)

    @top_sizer.add_item(@show_button)
  end

Sizers zu verschachteln ist nichts unübliches in wxRuby, man muss lediglich darauf achten, nicht den Überblick zu verlieren¹. Wer diese GUI startet, wird ein ziemlich eng gedrängtes Layout sehen — wie schon gesagt, versuchen Sizers standardmäßig sich spanngurtartig so klein wie irgend möglich zusammenzuziehen.

Um die Sache aufzulockern, benötigen wir etwas: Eine Möglichkeit, aus einem Schrumpf- einen Wachs-Sizer zu machen. In wxRuby schaut das so aus:

1
2
3
4
5
6
7
8
9
10
11

def make_layout
    @top_sizer = VBoxSizer.new
    self.sizer = @top_sizer

    sizer = HBoxSizer.new
    sizer.add_item(@text_ctrl, proportion: 1)
    sizer.add_item(@save_button)
    @top_sizer.add_item(sizer, flag: EXPAND)

    @top_sizer.add_item(@show_button)
  end

Fügt man ein Widget oder einen Untersizer zu einem Sizer hinzu, so kann man Flags übergeben, von denen eine Wx::EXPAND (gleichbedeutend mit Wx::GROW) ist. Sie weist das hinzugefügte Element an, sich nicht nach der so-klein-wie-möglich-Variante zu verhalten, sondern eben genau umgekehrt; das Element versucht, soviel Platz wie möglich einzunehmen.
Das gilt aber nicht rekursiv, d.h. die Elemente, die dem HBox-Sizer hinzugefügt werden, zeigen wieder das Minimal-Verhalten. Damit der neue Platz auch genutzt wird, bedienen wir uns eines weiteren Parameters namens proportion. Er gibt an, in welchem Verhältnis der zur Verfügung stehende Platz verteilt wird (standardmäßig haben alle Elemente eine proportion von 0). Hat man etwa zwei Widgets, von denen das erste eine Proportion von 1 hat, das zweite aber eine von 2, erhält man ein Verhältnis von 1:2, bei dem das zweite Widget doppelt soviel Platz einnehmen wird wie das erste. Man kann jedoch auch wesentlich kompliziertere Verhältnisse wählen, z.B. 23:11 und ist natürlich nicht nur auf zwei Widgets beschränkt, auch Verhältnisse wie 2:1:1 sind möglich, bei dem das erste Widget doppelt so groß würde wie das jeweils zweite und dritte².

Um schlussendlich noch den Show-Button ans untere Ende der GUI zu verschieben, benötigen wir ein Füllelement, dass wir zwischen den Untersizer und den Button packen. Idealerweise sollte dieses Füllelement sich so groß wie möglich machen und sich den Umstängen anpassen — in wxRuby gibt es für solche zwecke Spacers, genauer genommen sogenannte Stretch Spacers (normale Spacers kann man nur auf eine fixe Pixelbreite/-höhe einstellen). Indem wir einen solchen zwischen den Show-Button und den Kindsizer einfügen, erhalten wir das gewünschte Layout:

1
2
3
4
5
6
7
8
9
10
11
12
13

def make_layout
    @top_sizer = VBoxSizer.new
    self.sizer = @top_sizer

    sizer = HBoxSizer.new
    sizer.add_item(@text_ctrl, proportion: 1)
    sizer.add_item(@save_button)
    @top_sizer.add_item(sizer, flag: EXPAND)

    @top_sizer.add_stretch_spacer

    @top_sizer.add_item(@show_button)
  end

Sizers können aber noch mehr. Es handelt sich bei ihnen um ziemlich vielseitig verwendbare Layout-Elemente und nach dieser Einführung empfiehlt es sich, mal einen Blick auf ihre Dokumentation zu werfen. Es gibt auch noch mehr als nur die hier vorgestellten HBox- und VBox-Sizers, etwa einen Grid-Sizer für ein Tabellenlayout, aber die hier vorgestellten Sizers und Layout-Techniken sind die am häufigsten angewandten. Mit ihnen als Startpunkt sollte es nicht mehr allzu schwer sein, die Dokumentation zu verstehen.

Zum Schluss noch den vollständigen Code des Testprogramms, das nun vollständig auf Größenveränderungen reagiert; probiert es ruhig einmal aus, vergrößert, verkleinert, maximiert und minimiert — die GUI denkt mit!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

# -*- coding: utf-8 -*-

require "wx"

class MainFrame < Wx::Frame
  include Wx

  def initialize(parent = nil)
    super(parent, title: "Hello, world!", size: [400, 400])
    create_widgets
    make_layout
    setup_event_handlers
  end

  private

  def create_widgets
    @text_ctrl   = TextCtrl.new(self)
    @save_button = Button.new(self, label: "Save")
    @show_button = Button.new(self, label: "Show")
  end

  def make_layout
    @top_sizer = VBoxSizer.new
    self.sizer = @top_sizer

    sizer = HBoxSizer.new
    sizer.add_item(@text_ctrl, proportion: 1)
    sizer.add_item(@save_button)
    @top_sizer.add_item(sizer, flag: EXPAND)

    @top_sizer.add_stretch_spacer

    @top_sizer.add_item(@show_button)
  end

  def setup_event_handlers
    evt_button(@save_button, :on_save_button_clicked)
    evt_button(@show_button, :on_show_button_clicked)
  end

  # Event handling

  def on_save_button_clicked(event)
    @name = @text_ctrl.value
  end

  def on_show_button_clicked(event)
    md = MessageDialog.new(self, 
                           caption: "Your name", 
                           message: "Your name is: #{@name}",
                           style: ICON_INFORMATION | OK)
    md.show_modal
  end

end

class MyApp < Wx::App

  def on_init
    @mainwindow = MainFrame.new
    @mainwindow.show
  end

end

MyApp.new.main_loop

Valete,
Quintus

¹ Das Gem wx_sugar, welches Erweiterungen für wxRuby bereitstellt, erlaubt es, Sizer-Verschachtelungen mithilfe von Blöcken zu realisieren. Ich habe das aber bislang noch nicht gebraucht. Außerdem ist wx_sugar ziemlich fett, und wenn man nur dieses eine Feature will…

² Es ist übrigens meines Wissens nach nicht möglich, für Widgets einfach die Wx::EXPAND-Flag zu übergeben, man muss mit proportion arbeiten. Wieso ist mir noch nicht ganz klar. Wx::EXPAND scheint nur für Sizers zu funktionieren.