The Problem

I recently had a situation where I had a comma separated value (CSV) file that I wanted to easily parse within a shell script. Unfortunately the CSV data contained some double quoted strings with embedded commas, for example:

"Adygeya, Republic",RU-AD,21250,RU,Russian Federation

This made parsing the file quite painful, particularly as only the strings with an embedded comma were double quoted like this.

The Solution

I devised a utility script that can parse the data and replace the delimiter with a new character of the users choice, which makes CSV files far easier to work with if you pick a character you know won’t be in your data.

The Script

This script borrows the core CSV parsing from this rather good post on CSV parsing with awk, but I’ve edited it to allow the substitution of the delimiter and ensure it still outputs a single record per line.

#!/bin/bash

input=$1
delimiter=$2

if [ -z "$input" ];
then
	echo "Input file must be passed as an argument!"
	exit 98
fi

if ! [ -f $input ] || ! [ -e $input ];
then
	echo "Input file '"$input"' doesn't exist!"
	exit 99
fi

if [ -z "$delimiter" ];
then
	echo "Delimiter character must be passed as an argument!"
	exit 98
fi

gawk '{
	c=0
	$0=$0","
	while($0) {
		delimiter=""
		if (c++ > 0) # Evaluate and then increment c
		{
			delimiter="'$delimiter'"
		}

		match($0,/ *"[^"]*" *,|[^,]*,/)
		# save what matched in f
		s=substr($0,RSTART,RLENGTH)
		# remove extra stuff
		gsub(/^ *"?|"? *,$/,"",s)
		printf (delimiter s)
		# "consume" what matched
		$0=substr($0,RLENGTH+1)
	}
	printf ("\n")
}' $input

Sample Input


$ cat testprovinces.csv
Province,ProvinceCode,CriteriaId,CountryCode,Country
Australian Capital Territory,AU-ACT,20034,AU,Australia
Piaui,BR-PI,20100,BR,Brazil
"Adygeya, Republic",RU-AD,21250,RU,Russian Federation
Bío-Bío,CL-BI,20154,CL,Chile


Sample Output


$ ./change-delimiter testprovinces.csv '^'
Province^ProvinceCode^CriteriaId^CountryCode^Country
Australian Capital Territory^AU-ACT^20034^AU^Australia
Piaui^BR-PI^20100^BR^Brazil
Adygeya, Republic^RU-AD^21250^RU^Russian Federation
Bío-Bío^CL-BI^20154^CL^Chile


I was recently doing some housekeeping on my websever: removing archived software downloads that I no longer needed and looking for any problems when I found one that initially had me stumped. Looking at the size of my home directory, I found it was around 600mb with no immediately obvious reason as to why.

I have hardly any files in my home directory, so I quickly located the problem to ~/Mailbox/.Sent/cur, which is, as the path suggests, related to my email server. It looked like copies of my sent mail was being stored and not deleted.

Curiously, the mailbox directory is used by Dovecot, but Dovecot doesn’t handle sending mail so why would it be storing sent mail? Particularly as Dovecot wouldn’t have access to outgoing mail to store it. Some googling turned up nothing, which further suggested Dovecot wasn’t the issue.

Since the problem was with outgoing mail, I then turned my attention to Postfix, my smtp server, which would have access outgoing mail, but again, as with Dovecot, found no information on configuring Postfix to store outgoing mail.

At this point I had no obvious theory to go with, so I started looking at the files themselves for something to go on. There were 374 of these files (which I counted using the technique in this post) and as I started looking at the contents I realised they were copies of my automated backup emails.

These backup emails are sent using Mutt via a bash script running on a cron job that executes once a day. It just so happens that I’ve had this set up running around just over a year, so the number of these stored emails supports the theory that Mutt is the perpetrator

To confirm Mutt was creating these files, I used Mutt to send a mail and then checked to see if it appeared in the directory. It did.

Therefore, it seems I have a copy of every single email sent via Mutt since setting up my email system. This arrangement is good for archiving, but bad for disk usage.

The Solution
So I looked into Mutt’s configuration and found an option in Mutt’s configuration file (.muttrc) that controls the storing of sent mail: set copy.

set copy = yes will enable storing sent mail
set copy = no will disable storing sent mail

However, this isn’t optimal, as I would like to save Mutt’s recently sent email (as a lot of it is auto generated by the server and is my only copy if it never reaches the intended recipient) but delete anything older than a certain period.

You can achieve this using the find command

find . -ctime +90 -exec rm -f -v '{}' \;

This command searches in the current directory for files that are older than 90 days (using -ctime +90) and then executes the command rm -f -v on each result by using -exec rm -f -v '{}' \ (the '{}'\ tells find to run the command on each result).

Now that I had a command that did what I want, I set up a cron job to run nightly to execute the command in the correct directory:

00 03 * * * root cd /home/chris/Maildir/.Sent/cur; find . -ctime +90 -exec rm -f -v '{}' \;

This job runs as root every day at 3am. I added a cd command to enter the correct directory, but I could have just as easily put the path in the argument of the find command. Indeed, if you want to run this command on several directories, it would be better to list each directory as an argument to find, but in my case where I only wanted to run this on a single directory I found using a separate cd command more legible.

With that done, I now have a much smaller archive of sent mail that is automatically cleaned up for me without me having to do anything.

• find
• wc

We’ll use the find command to locate all the files (and exclude directories and other non-files) and then the wc command to count the files.

Find has many options, but we will only be using the type option:
find . -type f

The . indicates the directory to search in and can be replaced with any absolute or relative path. This command will also find files in subdirectories. To exclude subdirectories you can use:

find . ! -name . -prune -type f

The ! -name . -prune will ignore any directories that are not the current one.

The find command will give us a list of all the files we want to count, to count the number of entries in that list we must pass the output to wc to count it.

By default, wc will count words, newlines, and bytes. Since the output from find is a list of files, each separated by a newline, we can tell wc to only count newlines by using the -l argument: wc -l
Combining this into a single command, we get the final command to find count the number of files in a directory excluding sub-directories:
find . ! -name . -prune -type f | wc -l

(Of course, if you want to include sub-directories, simply remove the ! -name . -prune arguments and use:
find . -type f | wc -l instead)