Subversion and Shell Script : ^M character

The next assignment was to put all shell scripts, perl scripts and data files under source control and obviously subversion was my choice as we are using the same for source control of application code base.

Everything went smooth initially but we ran into issues while executing shell scripts post svn update on UNIX box. It was quite clear that EOL character introduced while editing our files in Windows development environment converts into the infamous ^M character.

Removing these characters using dos2unix or stripping the character via VI editor using find and replace were not functional solutions in the long run; so I started searching again for something which might be of help in this case and eventually got to the Solution

The subversion properties are a neat solution to such issues; just set the repository property svn:eol-style as native and let subversion maintain the EOF characters by itself.

The newlines of Windows will never be ^M of UNIX.

Exclude directories from tar

All I wanted was to exclude a couple of directories while creating a tar file; after extensive googling for almost 3 days I got a number of examples using --exclude option; but none of them resolved the problem and syntax were different everywhere.

Within inches from going for the alternative approach, of listing files and directories to be included, and passing it on to tar as source; My colleague's search came with another approach, and it finally worked.

Lets cut to the chase and see how to resolve this problem:

The best way is to create a list of all the directories to be excluded in a file, one entry per row with no extra spaces towards end, and pass it as an parameter to tar.

Here goes the Syntax:
tar cvfX path_of_the_tar path_of_the_exclude_file path_of_directory_to_tar

Let’s first create a file with all directories we want to exclude:
$ cat excluded-files.txt
test/a
test/b/ba

Now just use the syntax with the file name which created earlier:
$ tar cvfX test.tar excluded-files.txt test
a test/ 0K
a test/a excluded
a test/b/ 0K
a test/b/ba excluded
a test/b/bb/ 0K
a test/b/bb/bba.txt 0K
a test/c/ 0K
a test/c/ca/ 0K
a test/c/ca/caa 0K
a test/c/cc/ 0K
a test/c/cc/cca 0K

If you see excluded in the verbose output; you are done!!!

Environment Friendly Cron Shell Scripts

You created a neat shell script which makes use of some environment variables and all the tests went successful. But as soon as the script is scheduled in crontab; none of the environment variables were recognized and script fails.

Since crontab is executed as a different user in UNIX so assuming that your environment variables defined in $HOME/.profile will be available to any other user is wrong.

A simple solution is to execute the environment variables at the beginning, just after where you define the shell, of the script:

example:
#!bin/bash
. /home/user/.profile

where /home/user is the absolute path of your $HOME variable.

This should get you going.

TIP: Also using following export and assignation in a single line causes issues with shell scripts so make sure that
export MY_VAR=MY_VALUE
is replaced by
MY_VAR=MY_VALUE
export MY_VAR

Otherwise errors in your .profile file may cause issues with your shell script. All the best!!!