Scripting 101
This post is in response to a query in another thread regarding a comment I made about scripting. I use scripting extensively in my home automation environment and it allows me to quickly create software to control the diverse hardware I have installed.
First, a disclaimer; I am not a computer programmer by training - I'm an E.E. The only formal programming training I had in school was with a language called Fortran. Heck, most of the stuff I use today didn't even exist when I was in school. For the most part, what I've learned first came from comments by colleagues or references I've run across reading newsgroups, web pages, technical journals, etc. I'd then dig in a bit to find out more and see if it was something that might apply to what I was doing at work or home. A small percentage of the time I'd come across a real "jewel" that then became part of my "toolbox" going forward. A perfect example of this is scripting languages, which I've been using both at work and home for a long time. Further disclaimer; everything below is from my perspective as an engineer with
30 years experience designing hardware and occasional software. Any omissions or errors are mine and mine alone.A BIT OF HISTORY:
Scripting is not new; it's been around in various forms almost as long as the computer itself. In a nutshell, it's nothing more than automation - something we all have an interest in or we wouldn't be tracking this newsgroup. In this case the automation target is the computer, with the goal of manipulating it to do things automatically that otherwise we would have to do manually. Every type of computer environment I'm familiar with supports scripting in one form or another. In the original DOS environment they were called "batch files" and, although somewhat primitive compared to scripting languages in use today, they allowed someone who was not strictly a computer programmer to be able to put together a "program" that would accomplish a specific task. The most common example of this was a file called "autoexec.bat" which allowed a user to configure his computer the same way every time it started up. All you needed was a text editor and you could edit this file to tailor your computer to the way you wanted. Many people who did this had limited knowledge of just how computers worked and didn't even realize that they were writing a computer program, albeit a simple one.
At about the same time that the PC and DOS were being introduced to the home computer user, a new operating system called Unix was establishing itself in the industrial world. Like other operating systems, it had an underlying base which controlled the computer, plus a user interface which allowed the user to interact with it. Also, like most other operating systems of the time, the user interface was based on a command line which allowed someone to type in the operation that was to be performed. However, unlike many other operating systems, the Unix interface was a standalone program called a "shell" with the original version simply being called "sh" and normally placed in the /bin directory. People soon began to tinker with the interface, leading to a series of improvements that allowed command line editing, a history buffer of previously executed commands, etc. And, of course, the ability to quickly put together a text file of commands to be executed, called "shell scripts". With subsequent versions of shells (csh, tcsh, korn and many others), shell scripts became more sophisticated and started including concepts that were previously in the domain of programmming languages, such as control constructs (if/then), looping (for, while) and the ability to create and use internal variables. However, unlike most programming languages of the time, these scripts did not need to be compiled (a process which takes the instructions in the script text file and turns it into the low-level ones and zeros that drive the computer hardware). Instead, these programs relied on something else to "interpret" them; in this case the interpreter was the operating system itself.
The concept of interpretation was not new in computer languages; one of the first languages - Basic - was based on this approach. Soon the approch was being applied to a new breed of computer languages. These languages consisted of a program which ran under the operating system of the computer and directly executed the instructions in the user source text file. This offered many advantages over traditional programming:
1) Scripts could be constructed and executed quickly - no lengthly compile/link operations.2) A person did not need to be an expert programmer to use them. In fact, more often than not, script languages were targeted at individuals who had a specific task to get done that required the use of a computer, but they themselves were not necessarily computer experts. Nor did they have computer experts at their bidding to write the program they needed.
3) Scripting languages were usually in the "high level" language category; they dealt with concepts that were common regardless of the particular computer or operating system being used. All you needed was an interpreter that ran on the specific platform and you could run your script. As a bonus, the same script could often run on multiple platforms with little or no changes.4) Most of them are open source and can be downloaded for free from a number of sites.
I first became aware of of this new breed of scripting languages in the late 1980's. Like many other engineers, I had a PC at home and was playing with languages like QuickBasic, which allowed me to create and compile my own programs that ran under DOS on the PC. However, the computer world was undergoing a revolution. At work I was starting to use this thing called "X Windows" and, even at home, I found that the operating system was evolving towards a Windows approach. I looked at what it would take to write programs that used this new graphical approach and quickly became discouraged; hundreds of lines of code just to pop up a simple window?! I had become spoiled in being able to quickly produce a simple program that did what I needed. Now I was looking at something that appeared to have a steep learning curve followed by an increased amount of time to produce the program I needed. When I described my problem to a friend at work, he mentioned that he had just heard about a thing called Tcl/Tk that might be what I was looking for. I dug into it and found that Tcl stood for Tool Control Language and was developed a few years before by John Ousterhout at the University of California at Berkeley. Although initially targeted to ease interfacing with interactive integrated circuit design tools, Tcl evolved into a general purpose scripting language. It was also designed to be extensible, meaning it was relatively easy to create additional packages that could become part of the language. In fact, one of the first extensions was a thing called Toolkit (Tk - also designed by Ousterhout) which targeted the Graphical User Interface. I found that I could write a simple script that would pop up a window which contained a label with the ubiquitous "Hello, World!", along with an exit button that would close the window when clicked. Oh, happy days!! I'm back in business again! What was the effort to write this script? One line of code!!! Not only that, but this script ran both on my workstation at work and my PC at home. Although differences in the native window managers made the program look slightly different between the two environments, it basically did the same thing. Other scripting languages started appearing around this time, most notably Perl, and today scripting languages such as Python and Ruby even incorporate modern concepts such as Object Orientation. Although I've played with or worked with most of them, my favorite still remains Tcl for general purpose work. It is interesting to note that many popular extensions that were originally developed for the Tcl language (such as Tk mentioned above and Expect, an extension which automates many aspects of software testing) have since been ported to these newer scripting languages as well.
TCL IN HOME AUTOMATION:
Nearly six years ago, after years of saving my money and dreaming about home automation, I had a chance to build my "dream house". The automation controller was the OnQ 1050 (HAI OmniPro) and initially allowed me to control security, lighting and HVAC, either through keypads or a program called PC Access running on my PC. The PC was connected to the OnQ controller over an RS-232 link from one of the COM ports. Everything was great - until I got my first electric bill. Over $600!! This was for a home that was supposed to be energy efficient; although around 5500 sq. ft. it had concrete form exterior walls and a host of other characteristics that were supposed to reduce energy. The builder was little help ("What did you expect? It's a big house and this is the middle of winter"), but I felt something was wrong. The HA controller did not log the length of time that a given HVAC unit was on, something that I felt would help me prove to the builder that there was a problem with one of the zones. Somehow, I needed a way to collect this data and do it fast - the builder was already most of the way through the punchlist and time was running out for me. I had already determined that, although the OnQ didn't directly tell me whether an HVAC unit was on or off, I could read the current zone temperature from the thermostat. I could also read the current setpoint and, calculating whether the current temperature was above or below this, I could infer whether the unit was on or off (actually, there is built-in hysteresis, but this was close enough for what I needed). It only took a couple of nights to write and debug the script and a week to collect enough data to go to the builder and prove to him that not only was there a problem, but which specific HVAC was causing it. Once he replaced the unit the energy usage dropped back to where I felt it should be.
So far, so good. Soon I was writing other Tcl scripts for various purposes. One collected and logged "event" information such as when the security alarm was turned on or off, things that would trip an alarm such as a motion detector tripping or a window opening, etc. Another allowed more sophisticated control over my newly installed sprinkler system (ie: don't water if the temperature is below a certain point or if the moisture sensor indicates the ground is already wet). These were fun and easy to write but there was one very major drawback; they all communicated over the single RS-232 link to the OnQ controller, which could only do one thing at a time. This meant that to run any given script the one that was currently running had to be stopped. For a while I considered trying to roll everything into one giant Home Control program but I could see myself constantly tinkering with it over time as I added new features or upgraded old ones. Fortunately, Tcl offered the capability of a better solution through the use of sockets. For those that are not familiar with them, sockets offer a method of communication between programs using the same networking protocols that are used for Internet communication. From the user perspective, sockets operate similar to files on a filesystem; you can open a specific one that you are interested in, read or write data as long as you want and then close it when you are done. Communication over sockets can be between two programs running on the same computer or between separate computers that are connected using the TCP/IP protocol (usually over wired or wireless Ethernet). What this meant was that I could write a single program that would act like a classic server; it would be the only thing that would communicate with the OnQ over the RS-232 link. Various client programs, such as the sprinkler controller and event logger, would connect to it via the socket interface and send requests. These would be queued up by the server and processed one after the other in the order they were received. Associated with each request was the Channel ID (socket) that the request was received from so that after the request had been sent to the OnQ for processing and the response received back, it could then be returned to the correct originating client. This offered two big advantages. First, I could add or modify clients at will without disturbing a running system. Second, once I had developed a client, I could run as many instances as I wanted on any networked platform that could communicate with the server. Note that I said platform; as long as a Tcl/Tk interpreter is available, you should be able to run the script almost anywhere: MS Windows (from '98 through 2003 Server, including WinCE), all flavors of Linux, Solaris, HP-UX, etc. Since there is an interpreter for my wireless Dell Axim I can even run clients there (although sometimes I do customize the GUI since there is a smaller screen area to work with).
HOW TO GET STARTED:
If any of the above sounds interesting and you would like to learn more, then it's easy to get started. The first thing you will need is an interpreter for your platform. While you can download from many places, if you are just getting started I would recommend the Active State distribution
Ready to write your first script? I told you that you could pop up a simple window with a single line of code. Well, here it is. Open up your favorite text editor and type in the following:
tk_messageBox -message "Hello, World!" -type ok
Save the file, making sure that it is saved in TEXT mode; for example, if you are using Wordpad under windows, it will normally try to save the file in Rich Text Format (RTF) so you will need to select TEXT format in the "Save as type:" pulldown. You will also have to make sure that the file extension is set to ".tcl", for example "hello.tcl". Next, open up the file browser for your system and go to the folder where you just saved the file. Tcl is commonly pronounced "tickle" among programmers so the icon is usually shown as a feather. Double click on the file and a small window should pop up that says (oddly enough) Hello, World! At the bottom of the window is a small button that says "OK" in the middle. Click on the button and the window should close. Notice, however, that there is still another (blank) window that popped up on your desktop. This one should have the name you gave the script in the title bar. This is the toplevel window - to close it you will need to click on the "X" in the upper right-hand corner. Next, lets modify our program so that this is done automatically when you click the OK button. Start your editor again and add a second line below the first:
tk_messageBox -message "Hello, World!" -type ok exit
Previously, when you clicked the OK button, the script advanced from the first line (which had just finished execution) to the next line. Well, there was no next line so the interpreter simply waited in the shell, which was that empty toplevel window that you manually closed. Run your script again and this time, when you click on the OK button, the program should completely exit. Congratulations! You've just written your first Tcl script. If you decide to go forward from here, you will learn that the (blank) toplevel window that popped up along side your Hello, World window is where your main program will really go and it is this window that you will resize to whatever size you want and fill with whatever "widgets" you want to contain in your GUI.
For the next step, I would recommend (if you're using Windows) clicking on the Start menu and selecting the newly installed Tcl installation. There should be a section called demos that will show many of the things that you can do with the language. Start with the Tk demo and you will see examples of many of the basic widgets, such as labels, buttons, listboxes, menus, etc. The nice thing is that as you select each one, it will pop up a window showing the example of that widget. Next, click on the button that says "See Code" and it will pop up the entire script that it took to create that window. You will be astonished to find the amount that you can do with a relatively small script.
If you're really hooked at this point, your next thought might be to run out and buy a book on Tcl so that you can really get rolling. Don't. You will, if you're still using Tcl in two months, but right now everything you need is installed right on your computer. There is excellant online help that comes with the interpreter along with a host of examples that will help get you started. There is also a thriving user community; the comp.lang.tcl newsgroup is a great place to ask questions if you get stuck, but often you can answer questions quicker yourself by checking the Tcler's Wiki at
CONCLUSIONS:
Rolling your own home automation software may not be for everyone but if you are interested in customizing your setup, scripting languages provide capabilities that would otherwise be beyond most of us. My most recent home automation addition was whole-house audio distribution. I have three of the Channel Plus MDS-6 amplifiers connected to speakers that were installed when the house was built. The MDS-6 is controlled via keypads installed in the same general area as the speakers. Each keypad communicates over a Cat-5 cable that runs back to the main amplifier. All very good, but there is no other control interface and I wanted a way to tie it into my overall HA framework. I designed some hardware that spoofs the MDS-6 into thinking it's getting commands from a remote keypad, when in reality the commands are originating on an Atmel AVR microcontroller. The code for the AVR is written in "C", but the overall audio interface client running on the PC (and the Dell Axim) is written in Tcl/Tk. Without Tcl to implement the user interface portion, the overall goals of the project would have been difficult at best and impossible at worst. Scripting languages have opened up a whole world of opportunity for me and, hopefully, they can for you too.