Ripping Info from xe.com Mail

I Subscribed to the daily Currency Exchange Table Mail from xe.com , based on the USD. I get the Mail in Maildir Format onto my mailserver. I needed a way to process Files in that Folder, get rid of everything except the actual Rates Table, save it as a textfile in a webfolder named after the associated date, and also generate an extra small file called active_rates.txt, which contained a few selected Rates calculated against each other and the US Dollar. Maybe someone finds this useful or wants to learn something about awk (which is awesome..and fearsome), so here goes:


#!/usr/local/bin/bash

maildir="/path/to/maildir"
workdir="/path/to/destination/dir"
file="/path/to/active.txt"
if ! [[ -f "$file" && -d "$maildir" && -d "$workdir" ]]; then
echo “One or more of your Variables could not be found”
exit 1
fi
cd $workdir || exit
for file in $maildir/*
do
[ -f "$file" ] || continue
DATE=$(awk ‘/Rates as of/{print $4}’ $file)
sed -n -e ‘/Rates as of/,/ZMK Zambia Kwacha/p’ $file | awk ‘!n[$0]++’ > $DATE
newname=$(echo “$DATE” | sed ’s/[^0-9]//g’)
[ "$DATE" = "$newname" ] && continue
[ -f "$newname" ] && continue
mv “$DATE” “$newname.txt”
done
if [ -f "`date +%Y%m%d`.txt" ]; then
cp -v `date +%Y%m%d`.txt active.txt
elif [ -f "`date -v-1d +%Y%m%d`.txt" ]; then
cp -v `date -v-1d +%Y%m%d`.txt active.txt
elif [ -f "`date -v-2d +%Y%m%d`.txt" ]; then
cp -v `date -v-2d +%Y%m%d`.txt active.txt
else
echo “Rates havent been updated, maybe the Mail didnt came in ?”
fi
#STARTING TO CREATE THE SHORT VERSION, ACTIVE_SHORT.TXT
(
echo “============================================”
awk /”Rates as of”/’{print $1 ” ” $2 ” ” $3 ” ” $4 ” ” $5 ” ” $6 ” ” $ 7}’ $file
echo “============================================”
for a in USD EUR CNY HKD TWD JPY; do
echo “RATES for $a”
for i in USD EUR CNY HKD TWD JPY; do
[ "$a" = "$i" ] && continue
awk “/$a|$i/ { if (\$1 == \”$a\”)dN = \$NF; else dD = \$NF; } END { print(\”\t 1 $a equals $i\”, dD/dN);}” $file
done
echo “”
done
echo “============================================”
awk /”Rates as of”/’{print $1 ” ” $2 ” ” $3 ” ” $4 ” ” $5 ” ” $6 ” ” $ 7}’ $file
echo “============================================”
) > $workdir/active_short.txt


This basically places a rates table.txt file and an active_short.txt in the folder you specify. An example of this short version is here .

This could be improved as always; but for me currently it works fine, being called once a day by cron. Its annoying that it reprocesses all the Mails, but currently it only takes like 15 seconds for 2 years of rates mail; so i will leave it as is for now. Feel free to improve it though, and send it to me then

Webcam Image Sorting Script

I have a Panasonic Web cam Model BL-C131E , which transmits Pictures in Jpeg Format to my FTP Server. Basically, even though i configured the Camera so it only shoots and transmits Images when there is activity, the Folder that is used to hold the Images becomes HUGE quickly. Not huge in size, but a huge total file count, which makes it almost impossible to view the Folder from within a Finder Window. Try to view a Folder with 100000 Items in it, it gets annoying soon enough. Long story short, i came up with this Shell Script which i run periodically inside my Picture Folder. This little Program goes trough all the Files, creates a Folder Hierachy by Days and Hour of the Day, and sorts all the Pictures into them. You can easily adjust this for your own needs. Notice that my Web cam saves the Pictures using a timestamps as the Filename. Example: 20080221193828458.jpg . From there i extracted the Date and the Hour of the Day. Your Mileage may greatly vary, so use at your own Risk. I have tested this thing on a Folder with 100000 Files, and it finished in like ten minutes. Surely that can be improved, but in general, i never ever sort 100k of Pictures. I did not test for anything else, because i know the Filenames look always the same. Not sure what happens if there are spaces or other characters in the Filename, so be careful.


#!/usr/local/bin/bash

for filename in *.jpg
do
if [ -d "${filename:0:8}"/"${filename:8:2}" ]; then
mv “$filename” “${filename:0:8}”/”${filename:8:2}”/
else
mkdir -p “${filename:0:8}”/”${filename:8:2}”
mv “$filename” “${filename:0:8}”/”${filename:8:2}”/
fi
done


My Workplace getting wiretapped #Update2

Hello, today is the day. Our ISP came by and issued me the Firewall Unit i am required to install. It looks similar to our own Firewall, sports 256MB of RAM, has an 80 GB Harddrive from Maxtor, but only two ethernet Ports. After it will be installed, i will write more about how it works. Maybe its not such a bad thing after all, just a useless one. Using my own Firewall i was more than happy. Now i have a Firewall after my Firewall. Well, fine.

More to come…

Change root Password on iPhone 1.1.3 and 1.1.4 opened using iPlus or else

Got a new iPhone yesterday, with 1.1.4 Firmware. Works nice, though if you dare to change the SSH root Password from apline to something more secret using the inbuilt passwd command, you are in for a headache. Using the passwd command will send your iPhone into an endless crashing Loop, which i havent managed to get out of or fix except for going trough a restore procedure. If you do want to change you r root password though, simply open up your shell on any computer that has perl installed, type this

perl -e 'print crypt("password", "XX")."\n"'


Where you change your password , and the XX to two random characters. These type of Passwords cannot be longer than 8 characters btw. Anyhow, after you ran above command, use the resulting text to replace the crypted password in your /etc/master.passwd file on your iPhone. Login to your iPhone using SSH, or use the Terminal Application, type vi /etc/master.passwd, and look for this line :

root:YOUROUTPUTHERE:0:0::0:0:System Administrator:/var/root:/bin/sh


Replace the content immediately after the root: with the output from that command we used early on. Done, password changed.
Maybe do the same for the mobile User, which is also a line in the same file.

NOTE:
Someone mailed me and stated that the above method did not work for him at all. Quite the opposite, he had to reinstall his Firmware again. Annoying. According to him, the
following snipped to generate the Password Hash worked fine.

openssl passwd -crypt -salt /s myNewPasswd

Personally right now i don’t have another iPhone to try this on, but i think i should still mention his method here.

Circumvent Wikipedia blocking

Hello, found a good one, at least for german Users. Basically from where i live, i cannot access wikipedia.de . Sometimes the Page comes up, but then it doesn’t process my searches. If you want to lookup something in Wikipedia, try this
http://wissen.spiegel.de/wissen/dokument/dokument.html
enter your search text there, and it will give you the german wikipedia article.

Just like here , where i looked up Informatik:
http://wissen.spiegel.de/wissen/dokument/dokument.html?id=Informatik&suchbegriff=Informatik&top=Wikipedia

As you can see, the content of the URL can be modified to include your search term and bring you wikipedia, even if you cannot get to wikipedia directly.

So what we are doing is, we are getting content trough a proxy. Nothing revolutionary, but while open proxy’s tend to quickly be found and blocked, this Site for example will probably continue to be operational for a long time. I personally am reading the news on it for the last 8 Years, and never had a problem. Now we just have to find a Site that is interfacing with the English Version of Wikipedia in that way, and then we have Wikipedia in English as well.

Making Graphs for FreeBSD & rrdtool

I have setup some Graphs on all Machines i am administering. Nice to look at, can warn you of Problems, and may impress your Superior.

The below will work on any OS with small adaptions to binary and database paths. I used FreeBSD to do this.

1. Install rrdtool. One way of many could be cd /usr/ports/*/rrdtool && make install clean .

2. Think about what data you want to monitor. In this example, we are monitoring CPU Process Count and Bandwidth.

3. Create the Databases

#!/bin/sh

/usr/local/bin/rrdtool create /var/rrd/bw_fxp0.rrd \
$OPTIONS \
DS:in:COUNTER:120:U:U \
DS:out:COUNTER:120:U:U \
$RRAS


4. Get some Data into the Database. However you do it, i did it this way

#!/bin/sh

/usr/local/bin/rrdtool update /var/rrd/bw_fxp0.rrd `/usr/bin/netstat -inb | /usr/bin/grep '^fxp0.*‘ \
| /usr/bin/awk ‘{printf “N:%s:%s”, $7, $10}’`


5. Generate some Graphs. The Script below will generate Graphs using Data from the last Hour.

#!/bin/sh

/usr/local/bin/rrdtool graph /usr/local/www/data/rrd/bw_fxp0_out.png \
$TIME \
--lower-limit=0 \
--title="Bandwidth for bw_fxp0_out 1hour" \
DEF:out=/var/rrd/bw_fxp0.rrd:out:AVERAGE LINE2:out#FF0000:"out" \
GPRINT:out:MIN:"Min\: %lg" \
GPRINT:out:MAX:"Max\: %lg"


6. I basically saved these kind of scripts in my /usr/local/bin/rrd/ Directory, and called it at certain Times from Cron. Cron Entries for this could look like this

*/1 * * * * /bin/sh /usr/local/bin/rrd/rrd_insert_data.sh
*/10 * * * * /bin/sh /usr/local/bin/rrd/rrd_update_graph1h.sh


After setting this all up, you can view Graphs like this

Note: RRDTOOL can be used to graph pretty much ANYTHING. It is totally worth to go to http://oss.oetiker.ch/rrdtool/ and read all the Info you can find. I found rrdtool to be totally useful, and couldnt live without it.

Enjoy,
Overrider

Have had an iPhone

Unfortunately my iPhone fell on its glass Platter, which resulted in a big crack right accross it. Afterwards i realized i probably could have had that fixed for not much money, but at the moment, purely in caveman mode, i threw the iPhone against the wall, and that, it did not survive.

My Workplace getting wiretapped #Update1

Yesterday i sat down with our ISP Customer Service Rep to discuss the Box to be installed. Basically it is a Firewall Router. With making sure that every Business has one, they are trying to fight the Problem of illegal pornography, hacking, spamming and DDOS Attacks coming from unprotected Computers sitting behind inadequate Firewalls. At first it looks like a good thing, at second, i somehow doubt the usefulness of this. My CS-Rep told me that for now only Businesses with a static IP must install this, but in the long run, also normal ADSL connected Business with dynamic IP will have to submit to this Service. This Service, while required, is not free of course, but costs about 300 RMB or 30 Euro a month. It will be installed directly behind our own Firewall, so with that i have now two Firewalls. Fine. Things that bother me already, is the Units ability to record Internet Traffic for up to three Weeks, so if ever “something happens”, someone can come here and looks at the last three weeks Data flowing onto the Internet. Another thing is, the unit is in Chinese Language only, and can be remotely administered. Now what exactly that may mean i will know once i got the Unit.
One big advertisement from China Telecom is the Routers ability to detect Viruses people are downloading. Im not sure how this will work, but to be honest i don’t think it is scanning the Data-stream constantly for Virii, but rather when you download an executable or script file for a malicious signature when attempting to download it. Should that be the case, then obviously it is no use when someone downloads the malicious Program from a SSL protected Site.

Update 2

My Workplace getting wiretapped

The Internet in China leaves lot to be desired. I have mentioned this several times. It is way slow. Really, slow to a point where the Internet is barely useable. In the Company where i work, we have the best of the best line available for the money we could shell out for Internet. A whopping 5500RMB (about 550 Euro) a month, gives us 4 static IP Adresses we are free to use, and share a maximum Bandwidth of 4M . Thats not 4 Megabytes, that is 4M, equal to about 0.5 Megabyte, or 500 Kilobyte up or download speed as a maximum. There is no Guarantee to this however, try to download a Macintosh Update, you get blazing speeds of a few Kilobytes per second if you are lucky. The same goes for downloading Skype, Filemaker Updates, really anything.

Today i was informed by our ISP, the honorable China Telecom, that until May21 we have to let them install a device directly before our Router to the Internet, that controls and records whatever we are doing online for several days. Everything going trough is now directly looked at by them if desired. Sure, for example our Mails are using SSL Encryption, but will they let me keep using it? Or will they require me to use plain text? I am very very concerned about this, and before i will give in (which i will have to eventually), i want to know exactly why and how this will be done. What do they want to look at? We dont do anything nasty here, but our Company Mails, Internet Banking and other sensitive stuff could be at risk as far as i am concerned, if i let them install some “device”, whatever it is in front of my Line. I mean, i suspected anyways that they control and check all the stuff coming from an Internet Connection, but please, at least be discrete about it and not in our faces.

Update 1

Ok this is funny

With all the MS Talk going on there, please allow me to post this here, as it is pretty damn funny. Microsoft Real Good Edition, lol. Just click on the link to start it up and try it.
Boot Windows Real Good Edition
The WinRG flash demo was written by James Cliffe in Macromedia Flash.