Adding log purging job using Chef

Previous post talked about the linux command to purge old log files. Here is the way of doing the same thing, but with chef

cookbooks/housekeep/recipes/default.rb

# we need cron for scheduler
package cron do
  action :install
end

include_recipe "housekeep::clear_log"

cookbooks/housekeep/recipes/clear_log.rb

cron "housekeep application log files" do
  user "housekeeper"
  # mailto "your@example.com"
  
  # 00:01 every night
  hour "0"
  minute "1"

  # clean up logs older than 30 days
  command "/find ~/log/application-*.log.tar.gz -mtime +30 -exec rm {} \;"
end

Snippet available in gist:https://gist.github.com/798157

Linux searching for files order than X days

one of the frequently used commands for system maintenance is purging old files, e.g. log files, database backup, temporary files.

find ... -exec Command

In Linux, to search files older than 30 days:

find ~/log/application-*.log.tar.gz -mtime +30

to remove the files returned by find command, use option exec:

find ~/log/application-*.log.tar.gz -mtime +30 -exec rm {} \;

Adding a cron job

Edit the cron job list

crontab -e

Add the following lines:

# m h  dom mon dow   command
# MAILTO=your@example.com
0 1 * * * find ~/log/application-*.log.tar.gz -mtime +30 -exec rm {} \;

Note that the job is added to the cron job list of current login user. The find and rm command will also be executed as login user.

RabbitMQ Nameless Exchange

Not that sweet!

---
Nameless exchange

In previous parts of the tutorial we knew nothing about exchanges, but still were able to send messages to queues. That was possible because we were using a default "" empty string (nameless) exchange. Remember how publishing worked:

channel.basic_publish(exchange='',
routing_key='hello',
body=message)

The empty string exchange is special: messages are routed to the queue with the name specified by routing_key, if it exists.

Hello RabbitMQ

Conceptually, RabbitMQ is a message post office.

Casting
Broker is the physical post office.
Exchange is a named message box.
Queue is a queue.
Binding uses Routing Key to bind a Queue to an Exchange.
Producer delivers message to Exchange with a Routing Key.
Consumer consumes message from Queue.

Scenario
Broker: localhost
Exchange: StockMessageBox
Queue 1: 9OnQueue
Queue 2: 9SingQueue
Queue 3: 9BQueue
Binding 1: StockMessageBox => Price.700(Routing Key) => 9OnQueue
Binding 2: StockMessageBox => Price.716 => 9SingQueue
Binding 3: StockMessageBox => Price.# => 9BQueue
Producer: HKEX, which delivers stock price to StockMessageBox
Consumer: 9On, 9Sing, 9B, who consumes stock price from their queues

--
Now Producer HKEX publishes messages into StockMessageBox

JSON published: {Routing Key: Price.700, Close: 5.3}
Matched queues: 9OnQueue, 9BQueue

JSON published:{Routing Key: Price.716, Close: 3.2}
Matched queues: 9SingQueue, 9BQueue

JSON published:{Routing Key: Price.992, Close: 22.5}
Matched queues: 9BQueue

JSON published:{Routing Key: News.700, Info: "Profits increase 20%!"}
Matched queues: None
--

StockMessageBox(the Exchange) forward messages to queues by matching the Routing Key in the message and the Binding.13:19 27/01/2011

Timing your rake test cases in Rails 3: test_benchmark

i was overriding the setup and teardown of ActiveSupport::TestCase in test_helper to time my rake test cases. but in my current project, there are many test cases already overriding the setup method. So, if i still use this way of time tracking, I would have to change too many files.

so, after googling (oh, this google-ing form passes the checkspell ! ) site:github.com, i found test_benchmark. it is super easy to setup.


Include Gem and Rake

if you are using Rails 3, just

1. add following line to Gemfile (better put it in group :test):
   

   gem "test_benchmark"

2. run:
   

   bundle install

3. run:
   

   bundle install

   (or rake test:units if you want to find slow model test cases)

then you can see the top 15 slowest test cases:


Turn it off

you dont want to see all this figures every time you rake test. to switch it off, provide

rake test BENCHMARK=false

Turn it off by default

this is cool gem and some options available to configure.
but actually, i want the other way round.
I want it off by default and turn it on ad-hoc

so i added one line in my test_helper.rb:

Test::Unit::UI::Console::TestRunner.set_test_benchmark_limits(0, 0) unless ENV["SHOW_TIME"]

so now the usage becomes

rake test SHOW_TIME=true

there are some more options there, check out the details at timocratic github:
https://github.com/timocratic/test_benchmark

Regular Expression Generator

Perl

PHP

Python

Java

Javascript

ColdFusion

C

C++

Ruby

VB

VBScript

J#.net

C#.net

C++.net

VB.net

This is simple and brilliant idea !
http://txt2re.com/

it provides suggestion of string groups and possible data type,
you just need to click and select
it provides the code snippet for every programming language mentioned above too.

Finding Intersection of streets using Google Geocoding API

after releasing the Green Hong Kong to public and open source it,
my next step is to improve the accuracy of Recycle Bins location.

my last release was just blindly submitting the whole address (not an exact address) string to Google Map API and asked for possible locations.
First, i submit the chinese address.
if no response for chinese address, then i submit again with english address.

Using this way i got almost 99% latitude and longitude back.
BUT, most of them are not accurate.


There are variety of recycle bin addresses Government website provided.
e.g. near the MTR exit, outside XXX Building, opposite to the Bank, near the XXX Lamp Pole, etc.
basically the text is clear ... to me, but not the computer.
so i need to enhance the intelligent in determine nearest latitude and longitude of the recycle bins.


First case i want to tackle is the intersection case.
Government website told me that some recycle bins are located at the intersection of 2 streets.
my first thought is using simple maths to draw 2 straight lines and calculate the intersection.
i look at the Google Geocoding API,  see if there is any concrete data about the street.
then i find out that there is a Address Component Type called "intersection".
which is actually what i needed.

alright, I am getting lazy, because Google did everything for me already.

First time on Heroku

saw my colleagues using that for a while.

this is the first time i start using it from scratch.
http://greenhk.heroku.com

Besides the performance gain by this cloud hosting thing, the most surprising thing is the powerful toolset it comes with.

1)

Deployment is just a line of command

git push heroku master

then it will push your code to the git reportsitory, handle all the migrations and restart the rails application.

2)

It's like connecting to an application running on your local machine

Troubleshoot on the remote production server and tail the log is just a line of

heroku logs --tail

then it will send back the next block of log incremental changes.

and ... it is free !!

to me, i don't need big database thingy, so i can apply the free plan and just enjoy the performance and simplicity it provides !!

Pivotal Tracker no longer free ...

my company has been using it for a while.
it's time to re-evaluate the online story management application again ~