UNIX/LINUX TRICKS, TIPS, and EXAMPLES
  • LAST UPDATE: 08 August 2010.

    UNIX/LINUX TRICKS, TIPS, and EXAMPLES

      
    "Some say God speaks to him in UNIX..."
    SUBJECT   SUBJECT   SUBJECT   SUBJECT
    VI Editor Tips   Shell Script (.ksh) Tips   awk & sed Tips   SQL Database Tips

    GENERAL NOTES: The information in this web page is subject to the following disclaimers and caveats:
  • 02 April 2010: Join me in this tribute to the late Dr Henry Edward Roberts, inventor of the Altair 8800 computer.
  • This is an ongoing work, is maybe almost 1% done, so check back often for updates.
  • These script examples are actual, written-for-production, and used in: Farmland Industries, Sprint, USDA, and Qwest.
  • It was rewarding to receive these farewell emails on leaving Farmland, after 7 years as their sole, on-site "IT" guy.
  • As oldest technical person wherever I worked, it was easy to feel related to that cunningest of Dilbert characters, "Wally"
  • None of the script examples are newer than 2005 and most will be from the 1990s.
  • Most have had ".txt" appended to their runtime file names so they will be displayed OK by your web browser.
  • All the examples represent a fertile copy & paste resource of logically and syntactically correct programming dogma.
  • All of the script examples are based on the Bourne Shell (sh) and the Korn Shell (ksh). In 2009 the current default for Linux is the "Bourne Again" Shell (bash).
  • The bash shell may not handle the exampled sh and ksh scripts correctly due to likely changes in both syntax and capability in the intervening period.
  • Important: Many of the script examples begin with a line something like this:   #!/bin/ksh
    This directive seeks to have the script interpreted (i.e., run) using the shell within the path indicated. This may or may not work as expected (or at all) in your environment. If the script fails to run, then edit line #1 to be this:   #!/bin/bash
  • Oracle & Informix sql (aka "Sequel") database scripts may be either standalone or embedded within sh or ksh scripts. Ditto for any isql (Informix) and sqlplus (Oracle) scripts.
  • Various UNIX/Linux ("U/L") brands and release levels may utilize directories, utility programs, and configuration (and log) files that may be of different names and in different locations than was the case in the U/L environment that was in effect at the time and place when the TT&E were commited to text.
  • The various U/L utility programs (e.g, tar  ls  and ftp) in your U/L environment may have more, less, and/or different modifying arguments available than for the same-named utilities in the U/L environment that was in effect at the time and place when the TT&E were commited to text.
  • There may be entirely new (and/or more powerful) utilities available in your environment (e.g., yum) than were available in the U/L environment that was in effect at the time and place when the TT&E were commited to text.
  • The web is an inexhaustible resource for all things U/L. Use it to your advantage. This page is embarassingly feeble.

    BACK TO TOP

  • SHELL TIPS, TRICKS, and EXAMPLES

    Read or download (right-click on link) the text (.txt) files:
    DESCRIPTION FILE NAME   DESCRIPTION FILE NAME   DESCRIPTION FILE NAME
    Essential UNIX Security info unix_sec.txt   Essential Scripting Standards my_way.txt   Array Code & Sample Output araynfo.ksh
    Uses sed to beautify man manit.ksh   Embedding ftp commands in a script reps2nt.ksh   UNIX Encryption Discussion crypto.txt
    Excellent Scripting Tutorial: steve-parker.org   Another great Scripting Tutorial here: grymoire.com   Handy script menu template: Menu_DRM.ksh
    Combining ksh, SQL, and awk into a single shell script: anc_order.ksh   Some cute loops: cuteloops.txt   A simple Bourne shell line-editor for the vi-impaired: shed_1.4.txt  

    On Linux systems, Windows-like GUIs (e.g., KDE & Gnome) often are assigned to application-bound Users who are never expected (or allowed) to see or use a command-line prompt.

    Those programmers who must use MS Windows for all or part of their computer life can find a rich UNIX-like programming environment with either the Mortice Kern Toolkit or the public cygwin (free).

    The UNIX "shell" is a command-line user interface that has several popular versions. The earliest is sh followed by ksh, and finally bash (on most Linux-based systems). Every user may request (or be assigned) any of these three shells. A User is advised to accept and use the "standard" (i.e., default) shell for his system.

    The interested reader is referred to this on-line bash reference manual. And here is the on-line Korn shell reference.

    Effective use of variables and functions within UNIX/Linux shell scripts is an essential skill for U/L Admins and Users alike. A shell script can be thought of as an electronic circuit. The latter is composed of conductors, capacitors, resistors, transistors, etc. The job of the circuit is to do something useful, such as tune and play music from an FM station. Similarly, a shell script has a specific job to do. Here is a link for shell command-line examples: pixelbeat.org (thanks to Zach Lummis for the tip). Be mindful that the examples are for common operations and are based on linux.

    A script variable can be likened to an electronic capacitor in the physical world. The capacitor is first energized to an appropriate voltage by the circuit, and this charge is held until needed later. Likewise, a variable in a script can be "charged" (populated) with a "voltage" (numeric or character content) by the programmer. At some later time, the running script can use or alter the variable content based on runtime events (such as the login id and current directory content).

    Variables are often used within scripts to set up log file targets near the beginning of a script such as in this example:

    lgdir="${HOME}/log"
    tmpfi="/tmp/junk$$"
    outfi="/tmp/`basename ${0} .ksh`.txt" # Output File
    logfi="$lgdir/`basename ${0} .ksh`.log" # Log File

    In the above code segment the script uses $ to evaluate (i.e., utilize the contents of) three variables: {HOME} {0} lgdir
    The four variables lgdir tmpfi outfi logfi are each simultaneously created and populated in a single statement like this: tmpfi="/tmp/junk$$" Here, use of the $$ construct appends the Process Id Number to the word "junk".

    Here is an excellent link to good web based reference for use of variables.

    A script function is the equivalent of an integrated circuit in the electronic world. It is a virtual "black box" with a simple, dedicated purpose. The function has input and output just as its electronic counterpart.

    As an example, below is the "_dang" function which I devised to intercept (with trap) a User kill-command (Control-C on the keyboard, and signal 9 to UNIX) and then perform an orderly, and logged, termination of the script:

    function _dang
    { # Required Arguments:
    # $1 = Name of this Program
    # $2 = Exit Error-Code (non-zero)
    # $3 = User-Abuse string
    cat << BOX | tee -a $log_fi

    #### ABNORMAL TERMINATION ALERT ####

    SCRIPT ABENDED: `which $1`
    AS OF: `date`
    LOG FILE: $log_fi
    EXIT CODE: $2
    REASON: $3

    BOX
    exit $2
    } # EOF _dang
    And here is how the _dang function is accessed later within the same script:
    # Trap and log User-termination:
    trap "_dang \"$0\" 9 \"Terminated by User\"." 2
    Here is an excellent link to good web based reference for use of functions.

    This handy script-menu template is a working example of how and why to use variables, functions, loops, and conditionals in your scripts: Menu_DRM.ksh - I consider it to be the single most important example to study and analyze.
    BACK TO TOP


    VI EDITOR TIPS, TRICKS, and EXAMPLES

    Before one can do much with UNIX/Linux one must become skillful with use of its native text editor, vi (short for "visual"). vi is found on every U/L machine and is used for creating and editing scripts, configuration files, and program source code.

    Most of the U/L world has to co-exist with MS-Windows, sometimes to a very large extent. A good way to maintain vi skills while using a Windows machine is to download and install a vi-workalike text editor, called vim, onto the Windows platform. Thereafter, you can use vim on text (.txt) files and give up the simplistic (and weak) MS-Notepad text editor.

    vim is more colorful and graphics-oriented than is vi, and it is also available for U/L based systems. Load vim on U/L at your own peril, because if you get addicted to vim then you will find yourself disappointed when working on a vi-only system (most of them).

    Read or download this text file of vi commands (all work OK with vim, too).

    A long, long time ago, the UNIX system for which I was Administrator had only the ed and vi editors available. The User production application that I wrote (a combination of shell and SQL scripts) occasionally required the User to input a short, descriptive paragraph that would be stored in the database (Informix). Since neither of the native editors were User-friendly, I had to create my own, simple text editor. I called it shed which was yettanuther Bourne Shell (sh) script. shed should still work OK (your results may vary) if you download it, and make it executable. Here it is: shed_1.4.sh

    BACK TO TOP


    AWK & SED TIPS, TRICKS, and EXAMPLES

    awk is a powerful programming language available on all UNIX systems. Since awk is an interpreted language (i.e., uncompiled), an awk program written on HP/UX will run without modification on AIX, SOLARIS, LINUX, or any other UNIX-like system (provided that said system has the compiled awk-interpreter installed).

    In addition to conditional branching (e.g., if/then/else) and loop processing, awk offers the power of multi-dimensional array processing plus a wealth of built-in string and math functions. It can read from and write to multiple files, and execute system commands. Using variable contents and system utilities, awk can write - on the fly - shell and SQL scripts to control subsequent data processing. The possibilities are endless. You have to understand at least some basic awk commands to be conversant with UNIX/Linux.

    The awk files and examples below show how a programmer can integrate an interactive .ksh (or .sh) shell script and an embedded awk program to produce formatted output (e.g., a report) from an input data file. Much use is made of variables and functions to provide real-time values during processing. The context of these examples is working scripts ("production") for a wireless telephone company. Line-numberd versions of the .ksh example script are included to facilitate discussion as suggested at the bottom of this awk overview file: awk_fyi.txt.

    Read or download (right-click on link) these text files:
    DESCRIPTION FILE NAME   DESCRIPTION FILE NAME   DESCRIPTION FILE NAME
    awk example #1: ckurf.ksh   awk example #2: ckurf_num.txt   awk OVERVIEW: awk_fyi.txt
    awk example #3: reseq.ksh   awk example #4: reseq_num.txt   awk TUTORIAL #1: vectorsite.net
    awk example #5: see_edf.ksh   awk example #6: see_edf_num.txt   awk TUTORIAL #2: grymoire.com
    awk example #7: copy_tables.ksh   awk example #8: see_ct_dat.ksh   awk report example: see_ct_dat.out.txt
    awk example #9 (Bourne): pmpopp.sh.txt   #9 calls this awk file: pop_all.awk.txt"   #9 awk and sh fyi: readme.awk.txt

    The sed files and examples below usually show how a programmer can integrate an interactive .ksh shell script, an embedded awk program, and an embedded sed call to produce formatted output (e.g., a report) from an input data file.

    Read or download (right-click on link) these text files:
    DESCRIPTION FILE NAME   DESCRIPTION FILE NAME   DESCRIPTION FILE NAME
    sed example #1: procit.ksh  


    BACK TO TOP


    SQL TIPS, TRICKS, and EXAMPLES

    Probably all industrial-strength relational databases (e.g., Oracle) are standardized on the Structured Query Language (SQL for short, pronounced sequel). The examples here are all character (non-gui) based that Users ran on Wyse 50 serial terminals (no Windows). Nothing here is point-&-click, click-&-drag, or any other type of effete foofaw. This all comes from back in the day when real men were real programmers who wrote real code by hand (not by mouse). You might say we lived to write printf() statements!

    Read or download (right-click on link) these text files:
    DESCRIPTION FILE NAME   DESCRIPTION FILE NAME   DESCRIPTION FILE NAME
    sh & sql example #1: batA96.sh   #1 calls this awk file: batA96.awk   All-in-one: ksh, sql, & awk: anc_order.ksh
    sh & sql example #2: pmpopp.sh   #2 calls this awk file: pop_all.awk   Some Stored Procedures "readme": README.txt
    Stored Procedures ex #1: rcup_spl.sql   Stored Procedures ex #2: ven_hist.sql  

    I did a lot of work using Informix Standard Engine ("SE"), including its Stored Procedures language. My roots with Informix (originally named Relational Database Systems) go deep, beginning with when I met founders Roger Sippl and Laura King at a Usenix convention in 1985. At that time, they were just introducing their SQL-compliant database. All Informix products were eventually purchased and are now marketed by IBM (eval copies are available free). Informix SE and ESQLC are available for Linux. Highly recommended by almost everyone (but I've never used it - yet) is MySQL. It is available for free download for almost every type of system, including even [ugh] Windows. Give it a whirl.



    BACK TO TOP

    See my wee, tiny web page how-to tutorial.



    Return to Dan & Sheryl's Home Page