Amazon S3 is a cost-effective, redundant, and straight-forward solution for off-site data backups. Using the highly recommended aws command line tool by Tim Kay and a simple shell script running nightly, creating an automatic remote database backup is easy.
After installing aws, I created a shell script as shown below.
#!/bin/bash
export AWS_ACCESS_KEY_ID= # your aws access key id
export AWS_SECRET_ACCESS_KEY= # your secret access key
DBUSER= # the database username
DBPASS= # the database password
DBNAME= # the name of the database
BUCKET= # the root bucket to store the backup
FILE=$DBNAME.bkp.$(date +_%b_%d_%y.%s).sql.gz
mysqldump -u $DBUSER -p$DBPASS $DBNAME | gzip -9 > /tmp/$FILE
aws put $BUCKET/bkp/sql/$FILE /tmp/$FILE --private
rm -f /tmp/$FILE # cleanup
A nightly (at 1AM) cron is then setup for the user running the above script
0 1 * * * /home/app/s3dbbkp 2>&1
This is clearly a very simple implementation, but for my current needs (and likely others) it is perfect.
During a normal development cycle, multiple branches of trunk may exist simultaneously with different features/fixes being developed within each. In order to keep trunk relatively stable at all times, it is a good idea to merge any changes to trunk since the branch was created into that branch when the branch is ready to be pushed into trunk. Confusing? The process is as follows
branchbranch (other developers are doing the same in their respective branches)branch
trunk since the branch was created into the branchtrunk changesbranch back into trunkThis is by no means the only way to manage development of a new feature/fix within Subversion, and I make no claim that it is the best. This is simply what works for my team at work presently. The end of the process described above (starting with 3) looks as follows
$ cd /path/to/branch
$ svn log --stop-on-copy # Run this from within the branch root - notice the revision # when the log output stops
$ svn merge -r ###:HEAD /path/to/trunk . # Here ### is the revision number from above
# Resolve any conflicts and test all code again in the branch
# Now to get the changes from the branch merged successfully back into trunk
$ cd /path/to/trunk
$ svn merge /path/to/branch -r ###:HEAD --accept theirs-full
$ svn resolve -R --accept working *
$ svn ci -m "Merging branch ____ into trunk" # Commit the branch merge into trunk
The 2nd and 3rd from last lines above are the ones of interest; they are the ones responsible for getting the work back into trunk. These lines are really only necessary since we had first merged work from trunk into the branch. The first line
$ svn merge /path/to/branch -r ###:HEAD --accept theirs-full
merges all changes from the branch since it was created back into trunk. This includes all code merged from trunk to the branch, so we are guaranteed that the branch has the latest and greatest of all code. As such, we force SVN to resolve all conflicts by accepting the branch copy’s version of every file within our working trunk copy. The next line
$ svn resolve -R --accept working *
Resolves any tree conflicts that have been introduced.
The majority of the file management I do at work is through the terminal, however sometimes it is necessary to view/work with files through Nautilus. I’ve added the following command to my ~/.bashrc, causing openme to launch the current working directory in Nautilus.
alias openme='nautilus --browser . 2> /dev/null'
Recently I needed to parse all unique email addresses from a very large plaintext log file. A simple grep would not suffice because there were many cases where multiple email addresses appeared on the same line.
Realizing there are many solutions to this problem, here is the one I came across1 which I found most suitable for my needs.
perl -wne'while(/[\w\.\-]+@[\w\.\-]+\w+/g){print "$&\n"}' path/to/log_file | sort -u > parsed_unique_sorted_emails.txt
Using cPanel/WHM it is very easy to transfer an existing cPanel account between servers. Coordinating this with a client who manages DNS on their own can make things a bit tricky, especially when their MX records point to the same server for webmail service.
In order to guarantee no downtime on the site, typically I copy a cPanel account over to the new server including all site files and email in advance of notifying a client it is okay to change the DNS. I also tell the site at the newly located cPanel account to continue to read and write to the original database on the old server using the server’s IP address as the host. This gives the client the flexibility to change the DNS at a time convenient for them.
Once the DNS has propagated to the new server, a bit of cleanup is required. Following the example above,
localhost.To accomplish #1 above is simple. Via SSH on the new server run the following commands
mysqladmin drop your_database_name
# Now update your site to point to localhost instead of the original server's IP
Then, from the original server run the following commands
mysqldump -u your_cpanel_user -p your_database_name | ssh your_cpanel_user@the_new_server_ip mysql your_database_name
This will minimize downtime for your users1 while updating the site to read/write from the local copy of the database now containing the newest information.
Item #2 above is equally simple via rsync from the old server
rsync -av --progress /home/your_cpanel_user/mail/ root@the_new_server_ip:/home/your_cpanel_user/mail
This command properly updates the new mail server’s mail folder for all users with the latest changes to their accounts prior to the MX record change.
It goes without saying that this work should be done very late at night. ↩
Soon after updating to PHP5.3 on my local computer I found that some legacy code I maintain which can not be updated currently no longer worked. After a bit of research1, I found the best solution for now was for me to downgrade back to the latest stable version of PHP 5.2, PHP 5.2.10. I needed to reinstall a few PEAR and PECL extensions which much of the code I maintain depends on.
[~] $ sudo pecl install memcache
pecl.php.net is using a unsupported protocal - This should never happen.
install failed
[~] $ sudo pear install Image_Color
pear.php.net is using a unsupported protocal - This should never happen.
install failed
I knew my PHP installs were done within /usr, so I found the PEAR .channels/ directories with
[~] $ cd /usr
[/usr] $ sudo find . -type d -name .channels
./lib/php/.channels
./share/pear/.channels
deleted them
[/usr] $ sudo find . -type d -name .channels -exec rm -rf {} \;
and then updated the channels
[/usr] $ sudo pear update-channels
Now installing extensions from PEAR and PECL is back to normal.
The shell provides multiple shortcuts via the usage of the exclamation point when trying to execute recently performed commands. Err the Blog’s Bash Cheat Sheet provides the following tips and more.
If you need to run the most recently executed command again
!!
If the last command run requires super user privileges
sudo !!
Run the most recently executed command beginning with foo
!foo
Retrieve the last ‘word’ of the most recently executed command. In the example below, !$ would have the value my_database.
mysqldump -u root -p my_database
!$ # Will contain the value 'my_database'
Similar to the example above, this shortcut contains all ‘words’ except the first in the most recently run command. Using the same mysqldump command above, !* would have the value -u root -p my_database.
mysqldump -u root -p my_database
!* # Will contain the value '-u root -p my_database'
Note that the above commands can have :p appended, printing the commands or ‘words’ that the exclamation point replacement would use instead of running them as part of a new command in shell. The example below would print the most recently executed command beginning with foo
!foo:p
On Mac/Linux the shell’s command prompt is typically fairly uninformative, displaying only the top level of the current working directory path. If you are currently at /home/me/sites/mysite, the prompt might look something like
[mysite]$ ...
This isn’t as useful as it could be. LinDesk details multiple ways to make the command prompt more informative. Based on the information there I now have the following PROMPT_COMMAND set in my ~/.bash_profile.
PROMPT_COMMAND='DIR=`pwd|sed -e "s!$HOME!~!"`; if [ ${#DIR} -gt 30 ]; then CurDir=${DIR:0:12}...${DIR:${#DIR}-15}; else CurDir=$DIR; fi'
PS1="[\$CurDir] \$ "
This now makes the same directory path above for /home/me/sites/mysite look like this
[/home/me/s...ite/public_html] $
Using CentOS 5 on my VPS, I recently found that newly created cPanel accounts running cPanel/WHM 11.24 RELEASE seemed to have some trouble properly setting permissions to allow the jailshell cPanel users to access/manage their crontab.
Setting up cron jobs via cPanel’s web interface itself worked just fine, but when trying to manage a jailshell users crontab via shell, the permissions were not set up properly to allow for this.
As d4ly you can see I don’t have permission to view/edit the crontab for d4ly
d4ly@d4ly.com [~]# crontab -e
cron/d4ly: Permission denied
So as root I did the following
root@server [~]# chmod 4775 /usr/bin/crontab
root@server [~]# cd /var/spool/cron
root@server [/var/spool/cron]# chmod 0755 .
root@server [/var/spool/cron]# chown d4ly.d4ly. d4ly
root@server [/var/spool/cron]# ls -l
total 24
drwxr-xr-x 2 root root 4096 Feb 4 17:03 ./
drwxr-xr-x 13 root root 4096 Sep 9 09:51 ../
-rw------- 1 d4ly d4ly 37 Feb 4 14:54 d4ly
-rw------- 1 root root 1555 Jan 19 20:52 mailman
-rw------- 1 root root 32 Jan 11 16:42 munin
-rw------- 1 root root 485 Jan 26 13:17 root
Then I tested things out once more as d4ly
d4ly@d4ly.com [~]# crontab -e
MAILTO="d4ly"
0 0 * * * ping d4ly.com # This cron is for testing jailshell crontab editing
The source code for D4core is maintained in and SVN repository. I have found it helpful to create a .cleanup and .commit Bash script in the root directory of our framework’s repository.
.cleanup has the task of removing cached files, temporary files, and log files which are no longer relevant to the current development cycle. It also use it currently to reformat our XML configuration file with pretty whitespace using xmllint.
#!/bin/bash
echo "Cleaning D4core..."
rm -f D4core/Content/Cache/Smarty/*
rm -f D4core/Content/Compile/Smarty/*
rm -f D4core/Content/Cache/Minify/*
rm -f D4core/Config/Backup_*
rm -f D4core/Log/Error/*
rm -f D4core/Log/Logs/*
xmllint -format D4core/Config/Config.xml -output D4core/Config/Config.xml
echo 'Running `svn status`...'
svn status
echo "Cleaning D4core completed."
.commit (which calls .cleanup initially) is responsible for managing the commit process. This file is run only after all project files have been added/removed/moved within the working copy and we are ready to commit to SVN.
#!/bin/bash
./.cleanup
svn commit
echo "D4core SVN commit complete."
By running ./.cleanup before committing code to the repository it becomes a lot easier to ensure the status and location of all files within the working copy are correct. These simply scripts easily save me 5 minutes a day in shell.
Below is one way to quickly and simply clear all /.svn directories from a SVN working copy; useful in a variety of situations.
find . -type d -name .svn -exec rm -rf '{}' ; -print 2> /dev/null
Restarting httpd while trying to alleviate high load on one of the servers I manage, I was recently presented with the following error
(28)No space left on device: Couldn't create accept lock
After some research I found running the following command1
ipcs -s
showed there were a bunch of semaphores hanging around even though apache had been stopped normally via
service httpd restart
Similar to the code snippet mentioned by Major over at RackerHacker, the following command clears out those semaphores, letting httpd start up without a problem
ipcs -s | grep nobody | perl -e 'while (<STDIN>) { @a=split(/\s+/); print `ipcrm sem $a[1]`}'
Using Handbrake to rip DVDs, I use a slight customization of the default AppleTV encoding settings. I find the quality-to-filesize ratio is great for nearly zero setup time. But when trying to play the encoded .mp4 video files on my PlayStation 3, the video format was unrecognized.
Running this1 over a movie.mp4 AppleTV encoded movie fixes this issue, while still maintaining compatibility with AppleTV.
perl -pi.bak -e "s|(avcC.{3}).(.{7}).|1x292x29|" movie.mp4
In the case where I have ripped one of my DVDs of TV show episodes and am left with multiple .mp4 files requiring this fix, I run this script from within the folder
#!/usr/bin/bash
for file in `find . -type f -name *.mp4`;
do
perl -pi.bak -e "s|(avcC.{3}).(.{7}).|1x292x29|" $file;
done;
Here is the link to the original forum thread where I found this information. Thanks, Paul! ↩
