As for hghub, the project stalled due to other commitments, and the fact that git has grown on me, and I’ve switched to it and GitHub. For those looking for the same kinda thing for mercurial, please check out bitbucket.org (Thanks Dennis!). I’ve had a quick look at it, and it looks impressive.

Lastly, the project I’ve been working on lately is now coming to fruition - please go check out stripe.com and tell me what you think!

1

history|awk '{a[$2]++} END{for(i in a){printf "%5d\t%s\n ",a[i],i}}'|sort -rn|head

and got this:

1
2
3
4
5
6
7
8
9
10

100	cd
62	ls
60	git
28	ssh
26	hg
19	mate
17	psql
15	sudo
15	mv
14	rake

Looks like I’ve spent too much time looking for things, and not enough time working on them.

Which leads me on to one of my favorite topics - information management. I seem to go through so much information and generate so many notes it’s not funny, and trying to look after it is difficult. I’ve settled on Evernote and EagleFiler for now. Evernote is great for note taking and jotting things down wherever I am, and EagleFiler works well for storing larger documents, manuals etc. I also use a bunch of other tools around this - I plan on writing up some more of my processes here real soon, so keep an eye out. Also, if you have any tools or methods you find indispensable please let me know - I’d love to hear.

I think this takes EC2 away from being a cool idea, and turns it into a serious hosting platform. Bring it on!

The email from Amazon:

Dear Amazon EC2 Developer,
Many Amazon EC2 customers have been requesting that we let them know ahead of time about features that are currently under development so that they can better plan for how that functionality might integrate with their applications. To that end, we would like to share some details about a major upcoming feature that many of you have requested - persistent storage for EC2.

This new feature provides reliable, persistent storage volumes, for use with Amazon EC2 instances. These volumes exist independently from any Amazon EC2 instances, and will behave like raw, unformatted hard drives or block devices, which may then be formatted and configured based on the needs of your application. The volumes will be significantly more durable than the local disks within an Amazon EC2 instance. Additionally, our persistent storage feature will enable you to automatically create snapshots of your volumes and back them up to Amazon S3 for even greater reliability.

You will be able to create volumes ranging in size from 1 GB to 1 TB, and will be able to attach multiple volumes to a single instance. Volumes are designed for high throughput, low latency access from Amazon EC2, and can be attached to any running EC2 instance where they will show up as a device inside of the instance. This feature will make it even easier to run everything from relational databases to distributed file systems to Hadoop processing clusters using Amazon EC2.

When persistent storage is launched, Amazon EC2 will be adding several new APIs to support the persistent storage feature. Included will be calls to manage your volume (CreateVolume, DeleteVolume), mount your volume to your instance (AttachVolume, DetachVolume) and save snapshots to Amazon S3 (CreateSnapshot, DeleteSnapshot).

This new functionality is already being used privately by a handful of EC2 customers, and will be publicly available later this year. We will be expanding the private offering as we get closer to launch. Please sign up if you are interested in participating.

We hope this information is useful to you as you plan, design and deploy your applications in Amazon EC2.

Sincerely,
The Amazon EC2 Team

First, unlock according to this guide: “Mac”:http://www.iclarified.com/entry/index.php?enid=893 “Windows”:http://www.iclarified.com/entry/index.php?enid=849
Then, tap on installer on the home screen. Tap on sources at the bottom right. Tap edit, then add. Enter http://wpool.com/iphone , then tap OK. Wait for it to refresh the sources, then tap on Install on the bottom of the screen. Scroll down to the localisation (1.1.4) menu, then for each item in there tap on it, then tap install. Once they are all down hold down the power button to turn the phone off, then turn it back on - and enjoy!

Sorry if this is a little short - kinda busy right now!

1
2
3

sudo apt-get install apache2-prefork-dev libapr1-dev
sudo gem install passenger
passenger-install-apache2-module

I then added the apache config lines to the end of the apache config, and set my site up like:

1
2
3
4
5
6
7

<VirtualHost *:80>
ServerName lstoll.net
DocumentRoot /home/xxx/sites/lstoll.net/current/public
ServerAdmin xxx@xxx.net
ErrorLog /home/xxx/logs/lstoll.net_error.log
CustomLog /home/xxx/logs/lstoll.net_access.log combined
</VirtualHost>

Then, when apache started up it automatically started the rails app, and it just worked. It’s simple to use with capistrano, here’s my deploy tasks from deploy.rb:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

namespace :deploy do
  desc "Custom restart task for passenger"
  task :restart, :roles => :app, :except => { :no_release => true } do
    run "touch #{deploy_to}/current/tmp/restart.txt"
  end
 
  desc "Custom start task for passenger"
  task :start, :roles => :app do
    # Don't need to do anything, it's automatic
  end
 
  desc "Custom stop task for passenger"
  task :stop, :roles => :app do
    # currently no way I know of to 'stop' the app
  end
 
end

Once that is done, capistrano can flag to passenger to restart the app.

A bit of casual benchmarking shows it’s fast too. I’m very happy this is out there, it makes running rails apps as easy as PHP - saving all the messing around with mongrel/thin/monit. It also works with rails caching, so so more messing around with rewrite rules to get cached files to be read right off disk. This should hopefully make rails better on shared hosts, and lower the barrier of entry. Only downside is that this is currently rails only - so for merb/camping/anything else the old way is still needed. I’m hoping that soon passenger will support rack, or mod_rack/mod_rubinius will come out, so this ease is brought to all ruby frameworks.

First, I created a folder for the migrations to live in:

1

mkdir -p db/ar_migrations

I then appended a few tasks ripped out of Rails and slightly tweaked to the Rakefile:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

APP_BASE = File.dirname(File.expand_path(__FILE__))
 
namespace :db do
  task :ar_init do
    # Load the database config
    require 'active_record'
    database_yml = YAML::load(File.open(APP_BASE + "/config/database.yml"))
    init_env == "rake" ? current_env = "development".to_sym : current_env = init_env.to_sym # Default to dev env, symbolize
    ActiveRecord::Base.establish_connection(database_yml[current_env])
    # set a logger for STDOUT
    ActiveRecord::Base.logger = Logger.new(STDOUT)
  end
 
  desc "Migrate the database through scripts in db/migrate. Target specific version with VERSION=x. Turn off output with VERBOSE=false."
  task :migrate => :ar_init  do
    ActiveRecord::Migration.verbose = ENV["VERBOSE"] ? ENV["VERBOSE"] == "true" : true
    ActiveRecord::Migrator.migrate(APP_BASE + "/db/ar_migrations/", ENV["VERSION"] ? ENV["VERSION"].to_i : nil)
    Rake::Task["db:schema:dump"].invoke if ActiveRecord::Base.schema_format == :ruby
  end
 
  namespace :schema do
    desc "Create a db/ar_schema.rb file that can be portably used against any DB supported by AR"
    task :dump => :ar_init do
      require 'active_record/schema_dumper'
      File.open(ENV['SCHEMA'] || APP_BASE + "/db/ar_schema.rb", "w") do |file|
        ActiveRecord::SchemaDumper.dump(ActiveRecord::Base.connection, file)
      end
    end
 
    desc "Load a ar_schema.rb file into the database"
    task :load => :ar_init do
      file = ENV['SCHEMA'] || APP_BASE + "/db/ar_schema.rb"
      load(file)
    end
  end
end

These will read from database.yml in config/ , so make sure you’ve got datamapper set up.

Lastly, Let’s create an example migration to create the datamapper sessions table in db/ar_migrations/001_create_dm_session_table.rb:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

class CreateDmSessionTable < ActiveRecord::Migration
  def self.up
    create_table(:sessions, :id => false) do |t|
      t.string :session_id, :null => false
      t.text :data
      t.timestamp :updated_at
    end
    # Need this because ar in unhappy with a varchar primary key
    sql = "ALTER TABLE sessions ADD CONSTRAINT sessions_pkey PRIMARY KEY  ( session_id )"
    execute(sql)
 
    add_index :sessions, :session_id
    add_index :sessions, :updated_at
  end
 
  def self.down
    drop_table :sessions
  end
end

Once this is done, you can run rake db:migrate , and the DB will get the sessions table added. The db:schema:load task is also there, so you should use this when you first deploy your app to a new server. The naming is also consistent with rails, so capistrano should interact with it out of the box, with the exception that the environment to load/migrate needs to be called MERB_ENV instead of RAILS_ENV

When creating additional migrations you will need to follow the ‘proper’ naming scheme: the filename should start with an incrementing three digit number, and the rest should be a ‘flattened’ version of the class name. (i.e migration class AddPostTable becomes add_post_table).

I did have a few problems getting the postgres gem working with it - here’s what I had to do

*Mac OS X:*

1
2

PATH=$PATH:/Library/PostgresPlus/8.3/bin/
sudo gem install postgres -- --with-pg-include=/Library/PostgresPlus/8.3/include --with-pg-lib=/Library/PostgresPlus/8.3/lib/ ARCHFLAGS='-arch i386'

*Linux:*

1
2
3
4

PATH=$PATH:/Library/PostgresPlus/8.3/bin/
sudo ln -s /usr/lib/libssl.so.0.9.8 /usr/lib/libssl.so.4
sudo ln -s /usr/lib/libcrypto.so.0.9.8 /usr/lib/libcrypto.so.4
sudo gem install postgres -- --with-pg-include=/Library/PostgresPlus/8.3/include --with-pg-lib=/Library/PostgresPlus/8.3/lib

You’ll probably want to make the PATH change permanent for access to psql.

And that should get you going. Those of you who use postgres should definately check it out at “http://www.enterprisedb.com/”:http://www.enterprisedb.com/ - and for those using any other DB it’s worth checking out too.

1

abort: Python support for SSL and HTTPS is not installed

The solution was simple:

1

sudo port install py25-socket-ssl

After that, HTTPS operations worked fine.

So this is a little document I put together on how I got it up and running on my Ubuntu 7.04 server, with trac support and web access. This document extends my previous setup doc which as of yet unpublished - to it is assumed you already have trac and subversion installed, with a htpasswd file for authentication against them located at /etc/apache2/projects.htpasswd - please adjust the references to this below to suit your environment. You will also need to make changes to the apache config file to suit your naming etc.

Getting the basic system up and running

Start a root shell & install install the software:

1
2

sudo -s
apt-get install mercurial

Set up the repository storage

1
2
3

mkdir -p /var/lib/hg/htdocs
mkdir /var/lib/hg/repos
cp /usr/share/doc/mercurial/examples/hgwebdir.cgi /var/lib/hg/htdocs/

Create the apache config file /etc/apache2/sites-available/hg.lstoll.net and make it look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

<VirtualHost *:80>
ServerName hg.lstoll.net
RewriteEngine On
RewriteRule ^/(.*)$ https://hg.lstoll.net/$1 [R]
</VirtualHost>
 
<VirtualHost *:443>
ServerName hg.lstoll.net
DocumentRoot /var/lib/hg/htdocs
ServerAdmin lstoll@lstoll.net
ErrorLog /var/log/apache2/hg.lstoll.net_ssl_error.log
CustomLog /var/log/apache2/hg.lstoll.net_ssl_access.log combined
 
RewriteEngine on
RewriteRule ^/(.*) /hgwebdir.cgi/$1
 
<Directory /var/lib/hg/htdocs>
DirectoryIndex hgwebdir.cgi
AddHandler cgi-script .cgi
Options ExecCGI
AllowOverride None
</Directory>
 
<Location />
AuthUserFile /etc/apache2/projects.htpasswd
AuthType Basic
AuthName "hg.lstoll.net"
Require valid-user
Order allow,deny
allow from all
</Location>
</VirtualHost>

Create the hg web directory config file

1
2
3
4
5
6

cat > /var/lib/hg/htdocs/hgweb.config << EOF
[collections]
/var/lib/hg/repos/ = /var/lib/hg/repos
[web]
style = gitweb
EOF

Correct permissions, and enable the site in apache

1
2
3
4

chown -R www-data:www-data /var/lib/hg
chmod +x /var/lib/hg/htdocs/hgwebdir.cgi
a2ensite hg.lstoll.net
apache2ctl graceful

You should now be able to browse to the site, authenticate, and see that their is no repositories available. So lets create one:

1

su www-data -c "hg init /var/lib/hg/repos/hgtest"

You should now have a basic mercurial environment up and running.

If like me you used user@host for the htpasswd usernames previously (so you can do things like have trac send email notifications), you will need to also add usernames without the @ symbol - mercurial doesn’t seem to handle these well when trying to save a username and password in a clone operation. I now maintain both a user@host and user accounts for all users.

Migrating a SVN repository into your new setup

Install hgsvn

1
2

apt-get install python-setuptools
easy_install hgsvn

change to the repo dir, and import the repository

1
2
3
4
5

cd /var/lib/hg/repos
su www-data -c "hgimportsvn file:///var/lib/svn/test"
cd test
su www-data -c "hgpullsvn"
su www-data -c "hg update"

Clean out the svn remnants

1
2

find . -name .svn | xargs rm -fr
find . -name .hgignore | xargs rm -fr

Setting up the trac integration

Install the plugin

1
2
3
4
5
6

cd /usr/share/trac/plugins
svn co http://svn.edgewall.com/repos/trac/sandbox/mercurial-plugin
cd mercurial-plugin
python setup.py bdist_egg
cd dist
easy_install TracMercurial-0.10.0.2-py2.5.egg

(note - the egg filename may differ for you)
Edit the config file for the project you are moving to mercurial

1

nano /var/lib/trac/test/conf/trac.ini

enable the plugin by adding something like this:

1
2

[components]
tracvc.hg.* = enabled

You will also need to change your repository config to something like this:

1
2
3

[trac]
repository_type = hg
repository_dir = /var/lib/hg/repos/test

Your trac project is now set up to read the mercurial repository

Installing the client on Mac OS X, and checking it out

This assumes you have macports Installed.

Install mercurial

1

sudo port install mercurial

Create a local copy to work with:

1

hg clone http://username:password@hg.lstoll.net/test

Replace username and password with the username and password you set up. Quick note - your password will be stored in an insecure format, and will be echoed on the screen when performing an operation against the central repository - be aware of this
(Optional) Set up the Mercurial TextMate bundle

1
2
3

mkdir -p ~/Library/Application Support/TextMate/Bundles
cd ~/Library/Application Support/TextMate/Bundles
svn co http://macromates.com/svn/Bundles/trunk/Bundles/Mercurial.tmbundle

Sources: http://ww2.samhart.com/book/export/html/49
http://www.selenic.com/mercurial/wiki/index.cgi/HgWebDirStepByStep