Ledscripts.com Blog

I tend to get myself involved with projects that have a lot of small files — on the order of a few million.  These files are often only good to me for a short period of time, such as when I’m importing a million xml files into a SQL Database for processing later.  I’ve also worked on projects where millions of spam emails pile up in queues to be filtered by humans, which they obviously never will be, and simply need to be removed.

The annoying thing about this is the time/effort it takes to remove all of these files quickly and with as little system resource consumption as possible.

In this entry I’m going to show you a couple of ways you can set your self up for quick removal of files that you know (or are pretty sure) will be temporary, so you need to quickly remove them.

(more…)

Begin a full time DBA can make you realize how important backups are.  Enterprise database systems like Oracle and Microsoft’s SQL Server provide excellent backup mechanisms for doing “live” backups of the databases that guarantee consistency and allow for point-in-time recovery.

 In the open source world, however, we’re left with old skoul methods of doing backups such as cold backups, or even worse, text-based “dumps” of the databases.  I’ll let you waste your time reading about how MySQL “recommends” you backup a database, and how PostgreSQL’s pg_dump works.  These methods are very resource intensive (creating a bunch of “insert” statements and formatting everything as plain text) and can even lock our real working processes (in the case of MySQL) while it is doing the backup!

(more…)

I’ve decided to release a web-hosting application that I had built from scratch a couple of years ago, ServAdmin. You can access the SVN repository on Google code here: http://servadmin.googlecode.com/svn/trunk/

 Some of the features:

• Single administrative panel to control any number of servers (unlimited)
• Written from scratch — does not rely on 3rd party packages
• Written in about 95% PHP5 code. 
• Secure communication between servers using stunnel.
  • This allows you to have servers spread across the Internet where communications between them will happen on open networks, and still maintain a secure environment.
• OS-independent - Right now I’ve only written the ‘Linux’ server pieces of code, but this could be easily applied to any operation system.  It is designed in such a way that you simply have to write the layer for whatever OS you’re concerned with.
• Controls MySQL, Apache, Bind and vpopmail to make a complete hosting environment.

If anybody out there is interested in picking up development, just let me know.  It is really only a few polishes away from being a complete product.