Finding files after excluding some directories

Normally, to find a file recursively under a directory, we do something like

find path/to/directory -name "*filename*"

To exclude finding in some sub-directories under directory, we can do

find path/to/directory -name "*filename*" -not -path "*subdir1*" -not -path "*subdir2*"

[Courtesy: this]

Python regex to match decimal numbers

Here is a Python regex to match regular decimal numbers (without the e or exponent part):
r'([+-]?([0-9]+(\.[0-9]*)?|\.[0-9]+))'

Lenovo Motorola G5 Plus wifi issue after update

After a recent update on my Lenovo Motorola G5 Plus phone, I started facing wireless connectivity issues. It would randomly get disconnected from wifi and connect again. My laptop was connected to the same wifi and did not have this problem, so it was not a problem with the wifi. I tried restarting the phone, the modem and the router. I tried resetting the phone wireless settings, forgetting the network, and restarting (phone, modem, router). None of these worked. Finally, the following action worked for me.

• Open the wifi router's settings page in the browser (for me, this was at http://192.168.0.1). 
• Login and check the Wireless settings. Under "Channel", the value will "auto" or some particular channel (e.g. channel 1, channel 2, etc.).
• Change the value to channel 10.

This fixed the problem for me.

[Courtesy: this]

Taking screenshot on Moto G5 Plus

To take a screenshot on the Moto G5 Plus phone, press the Volume-Down button and the Power button together. To see the screen shot, go to Photos --> Albums --> Screenshots.

Pandas Dataframe Tips

When dropping rows with NaN's in a Pandas Dataframe, this

df = df.dopna()

may be faster than this

df.dropna(inplace=True).

---

Casting converting a column of a dataframe to float (for example) can be very slow if done like this:

df[col1] = pd.to_numericdf(df[col1])

or

df[col1] = df[col1].apply(pd.to_numeric)

or

df[col1] = df[col1].astype(float)

If you know how to do this efficiently, please tell me. As of now, my only work around is to avoid doing this operation if possible.

Passing lists in numba

If you plan to @jit() a Python function using numba, and one of the arguments is a list, it will be treated as an object, and the jit-ted function will probably be slower than the original. Instead, explicitly specify the argument data types (e.g. int32, double, unit64, etc.) within the @jit([data types]) declaration and, importantly, when calling the function, remember to convert the list into a numpy array of the data type specified in the jit declaration. For all your efforts, you should be rewarded with a good speed-up (if you have long-running loops).

Check whether the fonts are embedded in a PDF document

To check whether the fonts are embedded in a PDF document, do the following.

On Ubuntu

Run pdffonts mydoc.pdf at the terminal. It will show a table listing all fonts in the PDF document, and the 'emb' field is 'yes' if the font is indeed embedded.

Anywhere

Open the document in Adobe Reader, and select File...Properties. Under the 'Fonts' tab, you can see the list of fonts. Embedded fonts are followed by the phrase 'Embedded' or 'Embedded subset' within brackets.

Debugging in iPython

To debug a function func in module mod.py, do the following at the iPython prompt.

from IPython.core.debugger import Pdb
ipdb = Pdb()
import mod
ipdb.runcall(mod.func, [args, for, func])

This will start the iPython debugger at the first line of func.

Using LaTeX in Blogger

I found this answer on tex.stackexchange about how to include LaTeX formulas on Blogger. It worked for me! Thanks to MathJax and Matthew!

Code formatting in Blogger

I found this answer on stackoverflow about how to format source code in blogs on blogger, which pointed this tutorial with detailed instructions. It worked for me with Python! Many Thanks to David Craft and Alex Gorbatchev.

Also note the useful tip here about loading only the format styles that you need for your blog, to speed up page load.

How-to (for Python)

Add the following to Blogger template above the </head> tag.

In the blog editor, switch to HTML mode and insert code within <pre> as follows.

You can find more brushes here.

A matplotlib example

The following matplotlib script shows some cases where the matplotlib defaults were not sufficient for the job, and how to customize the relevant properties. The main tweaks were:

• changing the font type from Type 3 to True Type
• scaling the y values
• using latex syntax in the labels
• changing the fontsize for axis labels, axis ticks, and legends
• two legends
• using proxy objects (lines or patches) for the legends
• using axis coordinates for locating the legend
• semi-transparent legends and grid lines

Here is the python code and the resulting figure:

import matplotlib.pyplot as plt
import matplotlib as mpl
import matplotlib.patches as mpatches
import numpy as np
import json
import sys
import os

# change font type to True Type to avoid Type 3 fonts
# (which are not allowed by some conferences)

mpl.rcParams['pdf.fonttype'] = 42


# x = 1-d array with x values
# ys = six 1-d arrays with y values (one for each line we want to plot).
#      Of these 3 are of one kind, and the other 3 are of another kind.
# xlabel, ylabel = labels for the x and y axis

x, ys, xlabel, ylabel = get_plot_info(datafile)


# I want to scale the y values so that the y axis is easier to read.

ys = ys / 10   # ys is a numpy array


# The plot will be shrunk when it is included in the paper, so that the 
# default fontsize becomes too small. Select a larger fontsize globally.

fontsize = 30


# Plot the first three lines in blue with different line styles;
# then plot the other three in green with similar line styles.

plt.plot(x, ys[0], 'b-', lw=3)
plt.plot(x, ys[1], 'b--', lw=3)
plt.plot(x, ys[2], 'b:', lw=3)
plt.plot(x, ys[3], 'g-', lw=3)
plt.plot(x, ys[4], 'g--', lw=3)
plt.plot(x, ys[5], 'g:', lw=3)


# Set axis labels with larger fontsize.
# Mention the scaling done to the y values.

plt.xlabel(xlabel, fontsize=fontsize)
plt.ylabel(ylabel + r' $\div 10$', fontsize=fontsize)  # latex syntax works!


# Increase the fontsize of the axis ticks

ax = plt.gca()
for labx in ax.get_xticklabels():
    labx.set_fontsize(fontsize)
for laby in ax.get_yticklabels():
    laby.set_fontsize(fontsize)


# If you want the lines to reach the left and right extremities of the graph,
# reset the x-limits.
ax.set_xlim( (min(x), max(x)) )

# Instead of showing 6 entries in the legend (one for each of the 6 lines),
# we show one legend for the color code, and another for the line style.
# ('MCTM-...' are methods, and 'en' etc. are languages for which we ran
# the method.)

# first legend (for color code)

blue_patch = mpatches.Patch(color='blue')
green_patch = mpatches.Patch(color='green')

# loc=(x,y) are the coordinates of the lower left corner of the legend,
# where (0,0) if the lower left corner of the axes, and (1,1) is the
# upper right corner.
# framealpha is the transparency level (0=transparent, 1=opaque).

leg1 = plt.legend((blue_patch, green_patch), ('MCTM-DSGNP','MCTM-D'), 
                  loc=(.3,.6), fontsize=fontsize, framealpha=.5)
plt.gca().add_artist(leg1)

# second legend (for line style)

# plot empty arrays in black (to avoid the colors used above).

l_en, = plt.plot([], [], 'k-', lw=3, label='en')
l_hi, = plt.plot([], [], 'k--', lw=3, label='hi')
l_hir, = plt.plot([], [], 'k:', lw=3, label='hir')
plt.legend((l_en, l_hi, l_hir), ('en','hi', 'hir'), 
                  loc='lower right', fontsize=fontsize, framealpha=.5)

# Add grid lines, but make them semi-transparent (otherwise they appear too
# prominent when the figure is shrunk).

plt.grid(alpha=.5)


# The large fontsize pushes axis labels out of the figure; but matplotlib
# offers a function to auto-correct this.

plt.tight_layout()


# The saving format is chosen automatically based on the file extension.
plt.savefig(figname+'.pdf')

Very Quick File Server

Just cd into the desired directory, and run python -m SimpleHTTPServer. You get the message "Serving HTTP on 0.0.0.0 port 8000 ...". Now go the browser and type localhost:8000, and start browsing files/folders in that directory. Share the link http://<your IP addr>:8000 to let your friends browse as well.

[Tested on Ubuntu 10.04 (64-bit) with Python 2.6.5.]

Simple Java development setup

A simple setup for small Java development projects could be:

• Create the source directory ($PROJ_HOME) which will contain all .java files.
• Create the directory $PROJ_HOME/proj_name which will contain all the .class files.
• Add $PROJ_HOME to the CLASSPATH.
• During development, keep a terminal open with working directory $PROJ_HOME, and run "javac *.java -d ." to compile all source files and store the class files in proj_name.
• Run a class from anywhere using java proj_name.ClassName [args].

In addition:

• Store metadata files etc. required by the code in $PROJ_HOME/meta. If they need to be passed by command line arguments, you can pass $PROJ_HOME/meta/datafile.txt as an argument wherever you are running the class.
• Store sample test data files to quickly test the code in $PROJ_HOME/test.

Python tips

To compile a python script without executing it, do
python -m py_compile my_script.py
[Courtesy: stackoverflow]

Downloading a blog on blogspot

This http://blogname.blogspot.com/search?max-results=N lists the latest N posts from the blog. Instead of N, type the number of posts. If your blog has less than 1000 posts, you can save this page: http://blogname.blogspot.com/search?max-results=1000

Unicode tips for Python

• To use non-Latin characters in regular expressions, use u'...' instead of r'...', even if you have to escape every backslash; e.g. the regex u'(?u)[०-९]\\s' matches a Devanagari digit followed by whitespace.
• Remove zero-width joiners/non-joiners from Unicode text to get a normalized representation; otherwise words that are rendered the same in a browser/editor will be stored differently, and will not be equal on comparison; e.g. use the regex u'[\\u200D\\u200C]' and replace all matches with u'' (the empty string).

Shell script for scheduling jobs

Suppose you want to schedule many runs of myscript.py for different parameters, but never more than 8 at a time (e.g. because your machine has only 8 processor cores), then you can do:

 for param in `cat params.txt`; do  
   # wait if more than 7 jobs are running together  
   njobs=`ps aux | grep myscript | wc -l`  
   if [ $njobs -gt 8 ]; then          # 7 jobs + 1 grep  
     while [ $njobs -gt 8 ]; do  
       sleep 10  
     done  
   fi  
   python myscript.py param &     # schedule background job here  
 done  

A `best practice' for writing papers in LaTeX

To handle complexity and versions, I do the following when I start writing a paper in LaTeX.

• Create new directory Conf1.
• Within this, create a new file Conf1.tex, and two new directories: tex and images.
• Conf1.tex contains: document class, title, author, bibliography style, etc., which are specific to the conference/journal. This is followed by \input{tex/main}.
• Create main.tex in directory tex. The tex directory contains all the LaTeX code, and could have further subdirectories if required (e.g., I usually have a directory for Experiments, another for Algorithm, etc.). All tex files are included in the paper by adding \input{tex/subdir/abc.tex} to main.tex in the appropriate place.
• All package inclusions (\usepackage{}), command declarations, etc. required for the paper are included in tex/include.tex.
• The first line of main.tex is \input{tex/include} followed by \begin{document} and \maketitle. Include all your LaTeX code after this. End with \bibliography{tex/biblio} followed by \end{document}, where tex/biblio.bib is the bibliography file.
• All images are stored in the directory images.

Some things to note:

• You can use git/svn to do versioning. Until I get used to them, I am using the following method. When I have a significant frozen version, I copy the directory tex to texN where N is 1,2,3.... The timeline of versions from the earliest to the current is tex1, tex2, ..., texN, tex.
• I put the .bib file into tex because I found the bibliography changes as the paper develops.
• On the other hand, images usually accumulate. So, only the directory tex needs versioning.
• In order to use the material for another conference/journal, follow the steps above to create Conf2 and Conf2.tex. Then copy images and tex from Conf1 and compile. Your first draft of Conf2.pdf is ready!
• I read somewhere that using PDFLaTeX gives more compact files, also avoids some font problems which arise due to DVI to PDF conversion when using LaTeX. I also had problems using Beamer with LaTeX. 
• If you decide to use PDFLaTeX, remember that it does NOT accept images in .eps format. I found that .png files did not render text well on conversion. Creating / converting all images in / to .pdf format (using Inkscape) worked for me.
• I use Inkscape for most drawings, and Gimp for screen capture (and also for format conversion).

Find and Replace across multiple files in Linux

For example, do this (Courtesy Rushi):
find mydir -name "*.tex" | xargs sed -i 's/foo/bar/g'
This replaces `foo' by `bar' in all .tex files in the directory mydir and its subdirectories.

This is useful for changes to latex code stored in a directory structure, since
find: gets the correct recursive file listing
-i: edit the files in place
/g: replace all occurrences in the file (globally)
foo: is a regular expression (POSIX.2 BRE)

[search tex]

Applying for a passport: Tips

Here are some tips for anyone applying for a passport in India. (Note: Some of these tips are specific to Bangalore.)

• Apply for passport online at www.passportindia.gov.in
• I found some bugs in the online form; so I downloaded the editable pdf (called eForm) from the site and filled it up. You need Adobe Acrobat Reader for this (Evince etc. might not work).
• Print out the application receipt; it has all the main details of the application, and has to be shown at the entrance.
• Book an appointment slot online at your preferred Passport Seva Kendra (PSK). Slots become available for online booking at 8 am. Login at 7.56 am and start trying. [Thanks to Prasad, for the update on the new times!]
• Documents required:
• Use the site to find out. But, dont stop there. Call the toll-free number to check if the rules have changed recently. 
• E.g. I was asked for two adress-proofs, instead of one. I had to go back home, just for that. If you are using an annual bank statement as proof, ensure it has the seal and signature from the bank. I used the annual statement that comes by post, which is printed on the bank letter-head, and has no explicit seal or handwritten signature. (They finally accepted it, but only reluctantly. Also, in the spirit of bargaining, they kept the original statement itself, rather than a photocopy.)
• At the PSK
• Try to get a 9 am appointment. Later slots tend to get crowded by slowpokes from earlier slots, and it all snowballs.
• Try to go earlier than the mentioned reporting time. There is a single queue at the entrance, where people are redirected to counters, and the earlier you get to a counter, the earlier you get a token, and the earlier you will come in the rest of the queue.
• Do not carry laptops/USB sticks, or other electronics (cell phones were allowed when I was there), as you might have to leave them outside.
• Steps
• Go to counter (there are separate ones for Tatkaal and Normal; ask at entry, the Tatkaal counter might be less crowded). Get your documents checked. Get the token number. It will be printed on a small sheet. Keep this safe; you need it till the end.
• Go inside, and wait for your token's turn. Large displays will indicate when it is your token's turn, and also the counter to go to.
• First, you go to an A-counter, get the docs checked and pay the money. You get two copies of the receipt, one for you, and one that goes with the file. Keep both safe.
• Then you again wait, looking at the displays, and go to a B-counter, where your documents are checked again. Finally you go to a C-counter, where your documents are checked again. The official stamps your old passport (if any), and tells you to leave.
• Look for a counter where you get the acknowledgement for the passport application. Show your copy (the Applicant's Copy) of the receipt and get the acknowldegement. Go to the exit, return the token to the security guy, and leave.
• I was there from 11am till 4pm. I got hungry, and ate a burger there. I had some stomach problems after that. You might want to take some food and water when you go there.
• It can be a long wait. You could carry a book or something to kill the time, but you have to watch the displays alertly. I dont know what happens if you miss the chance.
• When I was coming out the PSK at 4pm, the police were carting away two-wheelers parked in the 'No Parking' area in front of the PSK. Ensure you park your vehicle in a safe place before you enter the PSK.