Linux Tips 17: Find and replace text in multiple files

|
Multiple file find and replace is a rarely used, but an extremely time saving, ability of Linux that I cannot live without. It can be achieved by chaining a few commands together to get the list of files you want to change (find), make sure the files contain the strings you want to replace (grep), and do the replacements (sed).

Lets say we have a lot of code that uses the function registerUser that was implemented when there was only one class of users, but now another class of users need to access the system and what were "users" are now called "admins" so we need to change all calls to registerUser to registerAdmin. The command needed would be:


find . -type f | xargs grep -l 'registerUser' | xargs sed -i '' -e 's/registerUser/registerAdmin/g'

The first part of the command is find, which finds all files and excludes directories. That result is then piped to grep, which lists all files that contain registerUser. The results of is then sent to sed, which replaces all occurances of registerUser with registerAdmin.

The command is quite long and hard to remember, which is why I normally write it down somewhere. Having it archived on my blog means I can just look here in the future.

CakePHP - Using custom form tags without losing form auto fill magic

|
The CakePHP FormHelper is one of the biggest time savers in the whole framework. It not only takes care of generating common elements as well as performs auto-filling of pretty much all input fields. The only problem I find with it is the difficulty in generating forms using $form->create(...) that submit to the current page, ie a form with action="". This is because when the URL in $form->create(...) is set as '' or null, the default action URL of /model/add takes over.

The solution I end up with is often writing my own <form> and </form> tags instead of using $form->create() and $form->end(). The issue with using custom form tags is that CakePHP will no longer auto fill input fields meaning that a lot of code has to be implemented to detect and specify input values. Overcoming this is quite simple and simply requires the use of $form->end() with a custom form tag, such that your form code looks like:


<form action="" method="POST">
<?= $form->input("User.name") ?>
<?= $form->input("User.email") ?>
<?= $form->end() ?>

Linux Tips 16: Removing crashed processes

|
In my previous tip on monitoring Linux processes I wrote about top as a tool for doing just that. In this post I will extend on that and explain the main commands used to stop processes that are taking over the system with excessive CPU or memory usage. There are two commands.

The kill command will stop a process for a given process id (PID). To use it on a process with the PID 21194 simply execute kill -9 21194 and the process will be killed. If you need to remove more processes belonging to the same name, ie remove all Apache instances, you need to use the command killall -SIGKILL apache

If you need to kill all processes with a common name, ie PostgreSQL, which can create processes based on the database that it is serving you can use a combination of ps, grep, awk, and kill. The command is: kill `ps aux | grep "postgre" | grep -v "grep" | awk '{print $2}'` The main command is the ps aux... that takes the results of ps aux, finds &quto;postgre" using grep, then removes lines with "grep" and uses awk to get all secondary field values (the PID). kill then loops through all matching PIDs effectively killing all processes.

A good summary of web development ...

|


Why is this true? I think because most developers are not that invested in the products they produce and most customers want everything done yesterday, which leads to a lack of thought and consideration as to what is really important. Really, try asking a client what is important on a list of features and most of them will pick at least half, and when everything is important nothing can be. Although when everything is implemented 80% of the client's time will be spent using only 20% of features.

Is there a solution to this problem? I don't know. Contracting companies and clients both want to keep hours to a minimum and most are simply unable to picture perfection so whatever works well enough is often what is delivered and accepted. However I think that it would be beneficial with willing clients to try to reverse the default time allocation and spend the most time on design and testing and far less time on coding and ad-hoc changes. While I do not consider changes to be bad when working in an agile process there must be a clear vision for what the software is intended for so that feature creep does not become an issue.

Personally I do not mind creating a few solutions to give clients a real choice. Although in today's economic circumstances some may say that it is not financially wise to do so, however I believe that good products will keep on delivering value to both the creator and the client and is really more of an investment by both parties instead of just the end result of a simple transaction.

Linux Tips 15: Using top to identify slow processes

|
For a long time I had been using ps - aux as a way of seeing just what processes were slowing down a server or *nix based computer. However top is a far better tool for this task as it shows a real time view of processes. There are very few commands available and they can be found by typing ? once the program has launched. The most useful for my purposes is o, after pressing o, type cpu, to have top order all processes by cpu usage to find any runaway processes.

Top is an interactive command line program and looks like:

Using PHPMailer to create a centralized email system in Cake PHP

|
Prior to Cake PHP 1.2 there were no built-in email components that were readily available and people had to resort to using their own components with PHPMailer. I do still believe it is better to use a custom PHPMailer component under most circumstances as it is easier to configure, supports more services (ie Gmail), and gives people far greater email authoring options. However currently the number one reason for me to continue using a PHPMailer component is that fact that it allows me to centralize all email communication within an application; although my approach may strike some as mixing the controller/model boundaries too much.

The component is a basic wrapper around PHPMailer, which you should download and unpack into a folder called yourapp/app/vendors/mailer such that yourapp/app/vendors/mailer/class.phpmailer.php exist. Then in your components directory create the file mailer.php with the following content:


<?php
/*
Emails:
sendSampleEmail
(I normally list all emails here for quick reference)
*/
class MailerComponent extends Object {
var $phpMailer;
var $testMode = false;
var $from = 'automailer@example.com';
var $fromName = 'Example.com Automailer';
var $sig = "Regards,

Automailer";
var $parent;

// Create test email for use in testing mode
function createTestEmail($email) {
$email = preg_replace('/(.*?)@(.*)/', "testmail+\$1@gmail.com", $email);
return $email;
}

// Startup functions
function startup(&$controller) {
App::import('vendor', 'Mailer', array('file' => 'mailer/class.phpmailer.php'));
$this->parent =& $controller;
}

// Send an email
function send($to = null, $subject = null, $message = null, $attachments = array(), $riders = array(), $replyTo = null, $replyToName = '', $from = null, $fromName = null) {
// Set up mail
$this->phpMailer = new PHPMailer();
$this->phpMailer->IsSendmail();
$this->phpMailer->From = ($from) ? $from : $this->from;
$this->phpMailer->FromName = ($fromName) ? $fromName : $this->fromName;
$this->phpMailer->Subject = $subject;
$this->phpMailer->MsgHTML($message);

// Set reply to
if ($replyTo) $this->phpMailer->AddReplyTo($replyTo, $replyToName);

// Test mode switching, can also do domain based switching
if (true) {
$this->testMode = true;
}

// Add address(s)
if (!is_array($to)) $to = array($to);
foreach ($to as $address) {
if ($this->testMode) $address = $this->createTestEmail($address);
$this->phpMailer->AddAddress($address);
}

// Add rider(s)
if (!$this->testMode) $this->phpMailer->AddBCC('backup@example.com');
if (!empty($riders)) {
foreach ($riders as $r) {
if ($this->testMode) $r = 'tester@example.com';
$this->phpMailer->AddBCC($r);
}
}

// Add attachments
foreach ($attachments as $a) {
$this->phpMailer->AddAttachment($a);
}

// Send email
$success = $this->phpMailer->Send();
$this->phpMailer->ClearAddresses();
$this->phpMailer->ClearAttachments();
return $success;
}

// Send sample email with data from a form
function sendSampleEmail($data) {
$subject = "Sample Email - Contact from {$data['name']} ({$data['ip']})";
$contents = array();
foreach ($data as $field => $value) $contents[] = ucwords($field) . ": {$value}";
$contents = join("\n", $contents);
$message = nl2br("Dear Admin,

The following contact request was received.

---- Begin contact contents ----

{$contents}

---- End contact contents ----

{$this->sig}");
$this->send('admin@example.com', $subject, $message);
}
}
?>


The component has support for sending all emails to a testing account and centers around the send function. The send function parameters are pretty self explanatory with the only weird parameter being $riders, which is my unconventional way of saying bccs. Each call of the send function creates a new PHPMailer object, which may seem wasteful but in my experience prevents a lot of strange issues such as person B receiving email content intended for person A that was sent previously.

The sample email function in the component sends a very simple contact form to the administrator of example.com. The $data variable is provided by the controller and is then used to compose a unique email. The message that is composed is very plain HTML that is simply a string with all new lines converted to HTML line breaks. For those needing to send complex HTML emails I normally have the message data be retrieve by calling something like $this->parent->requestAction("/mycontroller/emailFor/12"); which means the email is rendered by just another function in a controller that can be styled and tested easily.

As the $this->parent refers to the calling controller one can use that to load models, perform finds, access submitted variables, etc. The only thing to keep in mind is the level of dependency between the component and the calling controller. Although I do not find this to be a problem as the component is more of a function-based container that makes managing email content easier. Under extremely simple circumstances one would have the email functions be stored in the relevant controller instead of in a separate component.

To use the component just include it as a component in your controller with:


var $components = array('Mailer');


And allow sending with a function like:


function contact() {
if ($this->data) {
$this->Mailer->sendSampleEmail($this->data['Contact']);
$this->Session->setFlash('Your message has been received. We will get back to you shortly.');
$this->redirect('/contact');
}
}


While the component is set up to use the default mail sender it is also possible to set it up using a hosted provider like Gmail. Just see my post on using Gmail with PHPMailer to view the configuration settings and place them in the "Set up mail" section of the send function.

Linux Tips 14: Changing the default cron editor to vim

|
For some reason a lot of servers I work with have set up the default cron editor, the one that is launched using crontab -e, to nano. As a long time vim user I find nano to be too slow and strange for quick editing. Therefore I found the need to set up the account to use vim instead. To enable vim for use in crontab is easy, simply execute the following line at the command prompt:


export VISUAL=vim

You can check that it worked using the command env The output of env should show VISUAL=vim if the export command was executed successfully, alternatively you can just execute crontab -e to see which editor loaded. If you want the setting to be set all the time you need to add the line to your .bashrc or .bash_profile, depending on your *nix distro.

Please note that you can also use any other editor that can be launched from the command line. Just replace vim with your editor of choice.

Improved Cake PHP debug log messages CSS style

|
As much as I like the CakePHP DebugKit component it sometimes appears to cause PHP fatal errors to not be shown, leaving me scratching my head at a white screen. At which point I need to remove the component inclusion code to see what I have done wrong.

In search of a lighter weight solution I went to Google and found a nice CSS style for the class cake-sql-log that automatically hid the SQL log table thereby removing much of the CakePHP debug mode output.

However the problem with the style is that when working with a lot of queries it is too easy for the SQL log to fill up the whole screen and make it very difficult to view the queries. Therefore I have included my modified version below that creates a zoom effect on cells that makes readable only the cell that you are hovering over. The CSS styling code required are:

.cake-sql-log { font-family:monospace; position:fixed; top:99%; z-index:100000; width:100%; background:#000; color:#FFF; border-collapse:collapse; }
.cake-sql-log caption { background:#900; color:#FFF; }
.cake-sql-log:hover { top:auto; bottom:0; }
.cake-sql-log td { font-size:3px; padding:1px; border:1px solid #999; background:#FFF; color:#000; }
.cake-sql-log td:hover { font-size:10px; background:#FFA; }

When the style is applied the default view of the SQL log will just show a minimal red bar at the bottom of the window:

And when you hover over the red bar the full log will be shown as with only the hovered cell readable:

Working with multiple UTC/GMT timezones in MySQL

|
When working with international timezones it is not always possible to work off UTC/GMT time. At times the webhost may prevent you from changing configuration settings or data may already exist that prevent you from updating the system timezone.

Luckily with MySQL's date functionality it is not too hard to work around this problem. In this MySQL tip I will show how it is possible to work with GMT timezones for any record with a little date manipulation. Please note that I assume the dates you are working with are recognized MySQL date formats of either DATE, DATETIME, or TIMESTAMP.

The first thing that you need to do is find out the time offset between your database and GMT. You can do this by using the query SELECT NOW() and checking the difference using a Google query. As an example the server I am working on shows the time as "2009-03-16 01:35:15" while Google tells me the current GMT time is "6:35am Monday". This means the GMT offset of the server is +5 hours (6 - 1).

The second thing that you need is for your users to tell you what their GMT offset is. For example I am on the Australian east coast so my GMT offset is +10. For demonstration purposes I will assume this value is stored in the "settings" table in the field "gmt_offset". Using this two bits of information we can get the user's local time by using the following query.




SELECT
DATE_ADD(NOW(), INTERVAL 5 HOUR) as gmt_time,
DATE_ADD(NOW(), INTERVAL (5 + gmt_offset) HOUR) as local_time
FROM
settings
WHERE
settings.user = 'Paul'

A more complex example where manual time offsetting is required is for time related user notifications such as in a calendar application. In this example I will show how a notification table where a user set notification time is stored in the server's default time can be used so that we can retrieve notifications that need to be sent out correctly.

SELECT
n.*,
DATE_ADD(NOW(), INTERVAL (5 + s.gmt_offset) HOUR) as local_time_start,
DATE_ADD(DATE_ADD(NOW(), INTERVAL (5 + s.gmt_offset) HOUR), INTERVAL 1 MINUTE) as local_time_end
FROM
notifications AS n
LEFT OUTER JOIN settings AS s ON (n.user = s.user)
HAVING
local_time_start <= n.notify_at
AND n.notify_at < local_time_end

The trick with the above query is we are treating notify_at as the user would expect, in the user's local time, therefore instead of adjusting it to the server time we simply calculate what the user's local time is at the current server time and see if any users need to have their notifications sent within this minute. While it is possible to shift the notification time to the server time and retrive required notification records based on that I find it more intuitive to leave user input as is.

Using the above method means that the system does not actually need to worry about any timezone issues unless the system in actioning a user set timezone as we can expect all users to work within their own. However such a method can mean more complications if your application's users are allowed to share date related information, in which case 3 offsets need to be combined (server offset, sharer offset, sharee offset) to show an accurate local time.

Managing semi-static select options in Cake PHP using table-less models

|
In Cake PHP a model normally represents a database table. However there are times when it is not necessary to create a table to store data, in most cases this data would be a semi-static enumeration, as the size of an application increases the use of such semi-static lists increases.

For example options such as "Yes", "No", "Active", "Inactive" would simply be stored as tinyints in MySQL however they may need to be presented to the user as "Yes" or "No". For Cake PHP to automatically generate a drop down list with "Yes" and "No" a foreign key would need to be used instead, adding an unnecessary join to make select generation simpler.

The solution is to create a model that does not use a table at all. For most of my programs I call this model Staticselect as its main purpose is to give me quick access to commonly used drop down options that I can set in my controllers for the FormHelper to use in views. The model code is:


<?php
class Staticselect extends AppModel {
var $name = 'Staticselect';
var $useTable = false;

// Person titles
function titles() {
$titles = array('Mr.', 'Mrs.', 'Ms.', 'Miss.');
return $this->toOptions($titles);
}

// Convert array to key value options
function toOptions($a) {
$os = array();
foreach ($a as $v) $os[$v] = $v;
return $os;
}

// No/yes options with int values
function noYesInt() {
return array('0' => 'No', '1' => 'Yes');
}
}
?>

The above code shows a few sample functions. The most useful is probably toOptions, which will take an array and turn it into a hash where the keys and values are the same so that when generating selects the value selected by the user is also submitted with the form. The function noYesInt is a simple example of data I commonly store in such a model.

For those less familiar with the way Cake PHP works with options I will also show how the model can be used in a controller and view. The controller function where the static options are to be used should contain a line with:



$this->set('nyOptions', $this->Staticselect->noYesInt());


While the view would use the data in a form input using the code:



<?= $form->input('Settings.hideEmail', array('options' => $nyOptions)) ?>