Copy file preserving directory structure in Linux

Use cp --parents /d1/d2/d3/f4 /e1/e2/e3 to get the copy /e1/e2/e3/d1/d2/d3/f4.

[Courtesy: this]

Finding files after excluding some directories

Normally, to find a file recursively under a directory, we do something like

find path/to/directory -name "*filename*"

To exclude finding in some sub-directories under directory, we can do

find path/to/directory -name "*filename*" -not -path "*subdir1*" -not -path "*subdir2*"

[Courtesy: this]

Restarting Nautilus on Ubuntu

Sometimes Nautilus does not work/open. One quick thing to try is to do killall nautilus in a terminal and then try to open it in the usual way. (Courtesy: this)

Check whether the fonts are embedded in a PDF document

To check whether the fonts are embedded in a PDF document, do the following.

On Ubuntu

Run pdffonts mydoc.pdf at the terminal. It will show a table listing all fonts in the PDF document, and the 'emb' field is 'yes' if the font is indeed embedded.

Anywhere

Open the document in Adobe Reader, and select File...Properties. Under the 'Fonts' tab, you can see the list of fonts. Embedded fonts are followed by the phrase 'Embedded' or 'Embedded subset' within brackets.

Removing old linux kernel images in Ubuntu

To see how many linux kernel image versions you have, do ls /lib/modules. You will see a list of directories (e.g. 2.6.32-51-generic, 2.6.32-52-generic, etc.)---one directory per kernel image version.

Note down the versions that you want to remove (usually, you should keep the last two versions---the last one for normal booting, and the previous one as a fallback).

Open Synaptic. Search for the version number (e.g. "2.6.32-51"). Scroll down the entries for the kernel image and kernel headers, e.g.

• linux-image-2.6.32-51-generic
• linux-headers-2.6.32-51-generic
• linux-headers-2.6.32-51

Usually three entries of the kind mentioned above will be shown as installed in Synaptic. Choose each one, and "Mark for Complete Removal". Apply the changes.

After the Synaptic finishes applying the changes, you can see that the corresponding directories have been deleted from /lib/modules. The next time you start Ubuntu, the removed kernel images will not appear in the (GRUB) boot menu.

Using bc in calculations

To use the linux command bc for calculation in shell programs, do the following:

• Add the line scale=2 to the file ~/.bcrc.
• Then use var1=`echo "($var2+$var3)/$var4" | bc` to set var1.

Here

• The scale parameter tells bc to use 2 decimal digits.

Piping stderr

To pipe stderr (instead of stdout) use `|&' instead of `|'. This works from Bash 4 onwards. It is shorthand for `2>&1 |', which redirects bytes sent to stderr (2) to stdout (1), and then pipes it (|).

[Courtesy stackoverflow.]

Random reordering of lines in a file

To generate a random reordering or shuffling of lines in a file, use
shuf inp.txt > out.txt

Shell script for scheduling jobs

Suppose you want to schedule many runs of myscript.py for different parameters, but never more than 8 at a time (e.g. because your machine has only 8 processor cores), then you can do:

 for param in `cat params.txt`; do  
   # wait if more than 7 jobs are running together  
   njobs=`ps aux | grep myscript | wc -l`  
   if [ $njobs -gt 8 ]; then          # 7 jobs + 1 grep  
     while [ $njobs -gt 8 ]; do  
       sleep 10  
     done  
   fi  
   python myscript.py param &     # schedule background job here  
 done  

Grep tips

To get all lines not beginning with a tab, do
grep -vP "^\t" abc.txt
-P turns on Perl-style regex (otherwise tab cannot be specified as \t). "^\t" matches lines beginning with a tab. -v inverts the selection.

For most cases though, using -E (extended regex) instead of -P works, and is better supported.

Using `less'

View text files using less, including very large files. It also opens gzip-ed files. Use less -S to truncate long lines (instead of wrapping around).

Information about your Linux system

Here are some commands that give information about your linux system.

To get  the number of CPU's/cores in the system, use

less /proc/cpuinfo

To see if your processor supports 64-bit OS, you can Google the processor name (got using /proc/cpuinfo). To see if your OS is 64-bit, use

uname -a

If it says x_86_64 somewhere in the output, then it is a 64-bit OS, else it is probably a 32-bit OS.

To see where (in which directories) a package is installed in Ubuntu, use
dpkg -L <packagename>

Displaying Unicode text in the terminal / console

If gnome-terminal has problems displaying Unicode (UTF-8) characters properly, try using konsole (from KDE). It worked for me.

Scheduling a regular job in Linux

To execute the shell script myjob.sh every Friday and Saturday of July at 0,20 and 40 minutes past 10pm, do

crontab -e 

In the editor that opens up, enter

0,20,40 22-23 * 7 fri-sat /absolute/path/to/myjob.sh 

and save. You are done!
The format is for specifying jobs is

Min Hour DoM MoY DoW Cmd.

See Ian's article for more.

Reading ISO files in Linux

Given the file

/path/to/iso/abc.iso

do

mkdir /path/to/mount/dir/abcdir
mount -o loop /path/to/iso/abc.iso /path/to/mount/dir/abcdir
cd /path/to/mount/dir/abcdir

You can now start using the contents of the ISO file.

Find and Replace across multiple files in Linux

For example, do this (Courtesy Rushi):
find mydir -name "*.tex" | xargs sed -i 's/foo/bar/g'
This replaces `foo' by `bar' in all .tex files in the directory mydir and its subdirectories.

This is useful for changes to latex code stored in a directory structure, since
find: gets the correct recursive file listing
-i: edit the files in place
/g: replace all occurrences in the file (globally)
foo: is a regular expression (POSIX.2 BRE)

[search tex]

Using tar and gzip

To tar a list of files/directories, do

tar -cf mytar.tar file1 file2 dir3 file4

To add more files to this, do

tar -rf mytar.tar file5 dir6 dir7

To list the contents of the tar, do
tar -tf mytar.r
To delete files from the tar, list the contents, do

tar --delete -f mytar.tar file1 dir2

using the exact path as given in the `list contents' step.

To compress the tar, do

gzip -c mytar.tar > mytar.tar.gz

To decompress the file, do

gunzip -c mytar.tar.gz > mytar.tar

To extract files from the tar, do

tar -xf mytar.tar file2 dir3

To extract everything, do

tar -xf mytar.tar

To simultaneously tar and compress, do

tar -czf mytar.tar.gz file1 dir2 file3

To simultaneously decompress and extract, do

tar -xzf mytar.tar.gz

(Note: Additions, using tar -rf, are not allowed for compressed tar files)

Using wget

Full Site
To download a full website, do

wget -c -w 15 --mirror -p -k -P /path/to/dir http://my.website.com -a my.log

This will download the entire website, waiting 15 seconds between retrievals. It will copy everything needed to show the page correctly (-p), convert links for local viewing (-k), store the files in /path/to/dir (-P), and write (append) messages to my.log (instead of the console). If stopped in between, run this command again, and it will start where it left off (-c).

Add --spider at the end to do a dry run first.
Add --random-wait to vary the wait time (for sites that block automated downloading).
Add --limit-rate=20k to limit download speed to 20 kilobytes per second (an example politeness limit; use your discretion).

Courtesy: this, this and this.

Small part of site
To download a small part (e.g. a given page and its first-level links) to the current directory, do

wget -c -w 3 -a my.log -r -l 1 -p -k http://somesite.com/interesting/link.html

This will download the page link.html and all links in it (-r), but will not recurse further to links of links (-l 1).

A URL list
To download a list of files specified by URLs, do
wget -c -w 15 -i urls.txt
This will download all URLs listed in the file urls.txt.

Backup and Restore using rsync

I backup my data onto a second hard disk in my PC, like this:

rsync -av --delete /disk1/my/data /disk2/backup > backup.log 2> backup.err

This creates a folder called data in /disk2/backup and copies the contents of the folder /disk1/my/data to it.

If I rerun this command, it will copy only the changed files/directories from /disk1/my/data into /disk2/backup/data.

Restore is simple; /disk2/backup/data is like a mirror. Just browse and copy back whatever is needed.

Note: Using /disk1/my/data/ (with the trailing slash) instead of /disk1/my/data does not cause the directory data to be created. The contents of data are copied directly into backup.

For a one-time copy to an external hard disk, I did this:

rsync -avW --no-compress --delete --modify-window=2 /path/to/src  /path/to/dst > bakup.log 2> bakup.err


Compression is useful for network transfers but can be skipped for local copying. The 'modify-window' option was required since the destination was on an NTFS drive, and there could be time stamp issues when copying from ext3 to NTFS. The 'W' option (copy whole files) might speed things up since partial file transfer need not be supported.

Using xargs to redirect to command line arguments

To delete all .bak files in a directory, do
rm *.bak or ls *.bak | xargs rm
Essentially, rm is executed once with the whole of the piped input as argument.

To move the .bak files to another place, do
mv *.bak diffdir or ls *.bak | xargs -I var1 mv var1 diffdir.
Here, mv is executed separately for each file listed by ls. var1 is a placeholder. It is replaced each time by a file, and mv is executed.

To rename the .bak files to .bakup files, do
ls *.bak | xargs -0 -I {} mv {} {}up.
{} is another way of specifying a placeholder. -0 takes care of white space anomalies in the piped input.