Ledscripts.com

Make TechCrunch worth reading again... with greasemonkey!

noreply@blogger.com (Jon Coulter) — Thu, 04 Jun 2009 20:03:00 +0000

For those of you that enjoy tech and startup news, TechCrunch is (or at least used to be) a good source for news.

Over the past year or so it has gone down hill in a few regards:

Full of biased articles
Every other post is about Twitter (many times consecutive posts are about Twitter)
There are petty posts posted by certain authors every once in a while just to stir things up.

So I figured, rather then abandon TechCrunch all together, why not just write a GreaseMonkey script to clean it up when the page loads. That way I can read it without ever seeing a post about Twitter!

My script removes any posts related to twitter as well as has a filter in place to remove articles written by a specific author (though this is only an example, I suggest you edit the script to your liking).

You can download my basic script here. Give it a try, improve upon it, let me know if it is of value to you at all.

Google searching with SQL

noreply@blogger.com (Jon Coulter) — Fri, 13 Mar 2009 22:03:00 +0000

A co-worker over at chacha and I were talking one day and decided it would be "cool" to be able to search google with select statement (note the sarcastic air quotes around "cool"?)

For example, why not write a select statement like so:

SELECT * FROM google g
WHERE g.query = 'barack obama'
and g.title like '%policy%' and g.text like '%economic%'
and g.site <> 'wsj.com'
and g.filetype in ('html') and g.filetype <> 'jpg';

... instead of using the google advanced search format (which becomes '"barack obama" intitle:*policy* intext:*economic* -site:wsj.com filetype:html -filetype:jpg')

Now you can! I took about 3 hours and a few open source sql parsing modules and created sqlsear.ch. You simply type in a select statement (the 'where' clause is all that matters right now) and sqlsear.ch converts it to google's advanced search syntax and runs the search for you!

Special thanks to the sweds for the .ch top-level domain extension and for letting me purchase it!

Go on over and give it a try. Drop me a line if you can think of anything you want improved/changed/fixed or whatever.

- Jon

Twilio Responses made Super Easy

noreply@blogger.com (Jon Coulter) — Fri, 13 Mar 2009 02:01:00 +0000

In case you haven't heard of it, Twilio is a super cool, super easy to use and program voice service started last year. It is basically a full on PBX system that you can program very easily with and can scale to handle any demand you can throw at it, as it is built using Amazon's cloud infrastructure.

(Side node: I'm very jealous of it. This is exactly the kind of cool yet simple thing I wish I could think of!)

While the Twilio folks have provided libraries for their REST API, the "response" side of the equation is left completely up to you to form. This is very easy to do; the easy with which you can do this is, in fact, key to how simple the service is to use in general. However, I don't like writing code that concatenates xml strings together, nor do I want to use the heavy-handed xml libraries that are out there to simply print results like "Doggies are cute!". One key thing on "hand crafting" the xml responses is that you have to remember to encode any special xml characters. If you are using any sort of user-input values to generate this xml, then you have to be constantly aware of this and escape everything.

So, to the point, I wrote a simple PHP class (2 classes, really) and dropped them over on the ledscripts google code page. This basically allows you to general the xml needed in a fashion similar to the "flow" you are outputting yourself. Let's look at an example.

To generate a simple response, you can do something like this:


$response = new TwilioResponse();
$response->Say('Doggies are Cute');
$response->Respond();

(I know it is more lines, but it makes more sense and gets cooler).

Let's look at a better example (easier to read here):


$response = new TwilioResponse();

$response->Say('this is a test, my friend',
array('voice' => 'woman', 'loop' => 2)
);

// Gather, with the alternate method of doing verbs (as a "get" that can be passed as sub-verbs)
$response->Gather(
array(
$response->GetVerb(TwilioResponse::V_SAY, 'Thank you for calling. Please press 1 for more options'),
$response->GetVerb(TwilioResponse::V_PAUSE),
$response->GetVerb(TwilioResponse::V_SAY, 'Go on. Press something')
),
array(
'action' => '/handle-response.php?a=b&c=d', // note that the & here must be encoded... the library takes care of it!
'numDigits' => 1
)
);
$response->Respond();

You can find out more in the examples folder in the repository. This may not be a perfect library, as I put it together in the course of about 1 hour, but it has worked very well for what I've done.

Let me know if any bugs are found, or if you have any requests/suggestions for improvements!

Here is the SVN page on google code. To check out just this library and the examples:
svn checkout http://ledscripts-opensource.googlecode.com/svn/trunk/libraries/php/twilio .

noreply@blogger.com (Jon Coulter) — Fri, 09 Nov 2007 17:36:00 +0000

I tend to get myself involved with projects that have a lot of small files -- on the order of a few million. These files are often only good to me for a short period of time, such as when I'm importing a million xml files into a SQL Database for processing later. I've also worked on projects where millions of spam emails pile up in queues to be filtered by humans, which they obviously never will be, and simply need to be removed.

The annoying thing about this is the time/effort it takes to remove all of these files quickly and with as little system resource consumption as possible.

In this entry I'm going to show you a couple of ways you can set your self up for quick removal of files that you know (or are pretty sure) will be temporary, so you need to quickly remove them.

Traditionally, if you had a big directory structure to remove, you would simply use rm -rf to remove the files:

rm -rf /home/mybigpath

Of course, this will take forever and chew up all sort of valuable system resources. We want our CPUs being used for business logic, not recursive directory searching and unlink operation!

So how do we avoid doing an unlink of all of those files? Well the key here is to set ourselves up knowing that we're going to remove these files in the first place. Basically, we're going to create another file system specifically for these millions of files, and when we're done with them, we'll simply remove/format the file system we were using. There are several ways we can do this, just to name a few:

We can dedicate whole partitions of hard drives (logic or physical) to this temporarly location.
We can use something lvm (or lvm2) to manage these temporary chunks of disk space.
We can use loop-device file mounting to create truly temporary file stores that require one unlink operation to blow away!

I personally like #3 for thinks I know are going to be deleted, and #2 for things that I might want to keep around, so I need to be safe.

I'm going to first show you option #3 (loop-device file), as this is the easiest to make use of. The basic steps for this:

Create an "empty" file on your real file system.
Format that file as a file system block.
Mount the file using the "-o loop" option.
Create millions/billions/trillions of files within the newly mounted file system, and do whatever processing needs to be done on them.
When you're done with the files, unmount the file and simply delete the large fake file system, which is a very simple and efficient operation for the OS to do.

Let's dig into the details



~# cd /mnt
/mnt# mkdir fakepoint
# This creates a 200MB file
/mnt# dd if=/dev/zero of=fake.img bs=1M count=200
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 6.93631 seconds, 30.2 MB/s     

# We're using reiserfs in this example, but any file system should work
/mnt# mkreiserfs -f fake.img
/mnt# mount -o loop /mnt/fake.img /mnt/fakepoint
/mnt# df -h /mnt/fakepoint/
Filesystem            Size  Used Avail Use% Mounted on
/mnt/fake.img         200M   33M  168M  17% /mnt/fakepoint     

/mnt# cd /mnt/fakepoint/     

# create 1,000,000 dummy files
/mnt/fakepoint# perl -e
     'for(1..1_000_000) { open(F,">",$_); print F "hello"; close(F) }'     

# do a bunch of processing on your files     

# what if we need to expand our file
# because we didn't allocate enough?
/mnt# umount /mnt/fakepoint/     

# seek 201mb into the file (so we don't overwrite what we have)
# and add another 200mb
/mnt# dd if=/dev/zero of=fake.img bs=1M count=200 seek=201
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 5.97021 seconds, 35.1 MB/s     

# resize the "file system"
/mnt# resize_reiserfs fake.img      

# Re-mount
/mnt# mount -o loop /mnt/fake.img /mnt/fakepoint
/mnt# df -h /mnt/fakepoint/
Filesystem            Size  Used Avail Use% Mounted on
/mnt/fake.img         401M  132M  270M  33% /mnt/fakepoint     

# all done with these files, we want to blow them away now
# simple as this:
/mnt# umount /mnt/fakepoint/
/mnt# rm fake.img

Notice that we can even re-size this "fake" file system. This allows us nearly total flexibility in how we handle these millions of files.

With this technique, you could even "transfer" these files from one Linux system to another by simply copying the file system-file (fake.img, in this case) from one computer to another.

This works very well for simple projects and things that are temporary. If you want a more permanent solution, you could use lvm2 to create a real logical disk partition that could be managed with enterprise class tools. Most of the concepts stay the same, except you would use lvm2 to do all the dist-size management (the "dd" steps.)

On a side note, here is a nice tutorial on using loop-devices to create encrypted data blocks. Very cool stuff.

The Correct way to Backup PostgreSQL

noreply@blogger.com (Jon Coulter) — Wed, 19 Sep 2007 16:40:00 +0000

Begin a full time DBA can make you realize how important backups are. Enterprise database systems like Oracle and Microsoft's SQL Server provide excellent backup mechanisms for doing "live" backups of the databases that guarantee consistency and allow for point-in-time recovery.

In the open source world, however, we're left with old skoul methods of doing backups such as cold backups, or even worse, text-based "dumps" of the databases. I'll let you waste your time reading about how MySQL "recommends" you backup a database, and how PostgreSQL's pg_dump works. These methods are very resource intensive (creating a bunch of "insert" statements and formatting everything as plain text) and can even lock our real working processes (in the case of MySQL) while it is doing the backup!

Thankfully, the folks as PostgreSQL introduced Online-Backups in version 8.0. In my opinion, this is the single biggest advancement in this release of the database engine. It how handles it's backup is a more Oracle-before-RMAN fashion (which worked great for many many years for Oracle.) It basically works like this:

You setup the database instance to make use of, and keep historical "logs" (WAL for PostgreSQL, "arhive redo logs" for Oracle.) A contiguous chain of these logs, plus a full backup from some point in the past (more in this in a second) can be used to "replay" every transaction/change in your database. MySQL sort of implements this with it's binary logs, but it is not quite the same thing.

You tell the database how to backup these "logs"

You do a "full backup" of the database which is on-line, so no body is blocked from doing any work. In fact, the only work involved in this backup is really the I/O operations required to do a binary copy of the data files to a backup location.

Once you tell the database engine you are done with the backup, it will start (or continue) backing up the logs to a location of your choosing.

In the event of a failure or need for recovery you can take the full backups (step 3) and apply the saved logs (steps 1, 2 and 4) to recover your database. Even cooler then that, you can point-in-time recovery. Which essentially means that if you had somebody "accidentally" delete a bunch of data at 1:34:00 PM, you can restore the database up to 1:33:59 PM and get back the lost data!

I'll let you read more details on the official PostgreSQL backuppage about it, but I would highly recommend this method of backup and recovery to any PostgreSQL DBA that has critical data to take care of.

ServAdmin Open Sourced

noreply@blogger.com (Jon Coulter) — Sat, 15 Sep 2007 01:35:00 +0000

I've decided to release a web-hosting application that I had built from scratch a couple of years ago, ServAdmin. You can access the SVN repository on Google code here: http://servadmin.googlecode.com/svn/trunk/

Some of the features:

Single administrative panel to control any number of servers (unlimited)
Written from scratch -- does not rely on 3rd party packages
Written in about 95% PHP5 code.
Secure communication between servers using stunnel.
- This allows you to have servers spread across the Internet where communications between them will happen on open networks, and still maintain a secure environment.
OS-independent - Right now I've only written the 'Linux' server pieces of code, but this could be easily applied to any operation system. It is designed in such a way that you simply have to write the layer for whatever OS you're concerned with.
Controls MySQL, Apache, Bind and vpopmail to make a complete hosting environment.

If anybody out there is interested in picking up development, just let me know. It is really only a few polishes away from being a complete product.

Nozbe API and .Net

noreply@blogger.com (Jon Coulter) — Tue, 11 Sep 2007 01:37:00 +0000

While browsing this very nice 5000+ Resources to Do Just About Anything Online blog entry, I can across a very nice "Getting things Done" site named Nozbe.

What I particularly liked about it was that it had a very simple API that you can work with. Basically, you make a specially formatted request and it sends results back on a JSON response.

JSON was likely chosen as the response type for a few reasons:

You can make the requests via AJAX so it is very easy to integrate with web sites.
You can easily use it with PHP5's very nice json_encode/decode functions.

I, however, wanted to be able to write a windows application so that I could quickly add tasks to my project lists without having to actually open a web browser. The ideal tool would be a simple icon-tray application that I would rick click on and say "add action" (actions are what Nozbe calls what I would call a task.)

I decided to start writing a .Net application to do just this. In .Net (C#, in my case) you have to have some external library to handle the JSON parsing -- because writing it yourself would be a pointless waste of time.

Thankfully, Json.Net has done just that for us. Thanks to this nice library and about 10 minutes of fiddling around, I was able to piece together a "proof of concept" application to show myself that want I wanted to do, could be done.

For now, I'll throw up my demo application for anybody whom wants to do download and extend. If I actually follow through on completing the application, I'll post the updated app as well.

Side note: I guess I better create an action item in Nozbe to finish the application! If only there were some Windows application that would allow me to do this very easily....

- Jon

Download Nozbe .Net API Integration Proof of Concept