Skip to topic | Skip to bottom
Home
Neuralyte

Down with Spammers!

Neuralyte.ProjectJshr1.24 - 08 Feb 2005 - 14:51 - TWikiGuesttopic end

Start of topic | Skip to actions

jsh - a library of scripts for bash and other *nix shells

Free Ice-Cream! See below...

Jsh is a library of shellscripts and also a set of useful config files.

The latest scripts can be seen here.

Documentation for documented scripts can be found here and the big list of all the scripts here.

Note that many of the scripts expect some of the others to be available on the PATH. You are recommended to follow the installation instructions below and run scripts from within the jsh environment. (Alternatively, use compilejshscript or its web-interface below for a single script.)

News

  • Added folding javascript to page. I wonder what it does in IE. Do you hate it? Can you even read this?
    • Works better in Konqueror than Mozilla, which sometimes adds gaps between blocks.
    • Trees beat lists, but graphs are better.
      • We should probably start to split some of these sections into separate Wiki pages.
  • Started competitions to encourage people to try out jsh.
  • 21 Jun 2004: Jsh gets its first script with a GUI:
    • grip_realplayer (using Xdialog, which is wizard-like), and
    • extract_clips_from_video, which uses mplayer as a "plugin"!

Features

  • Jsh gives you:
    • A colourful shell environment with handy configuration,
    • Lots of useful command-line programs and tools,
    • And a library of functions for writing higher-level shellscripts.
  • Suitable for beginner terminal users and experienced console hackers alike.
    • Provides configurations for common applications which by default are simple and boring.
      • Colours for ls, vim, screen, cvs, diff and grep.
      • A colourful and informative shell prompt
      • Simple keybindings for command-line manipulation
      • Enhanced tab-completion.
      • TODO: add links to scripts in this section as proof!
    • Provides handy little scripts to squeeze more power from the command-line.
      • trimempty, takecols, dropcols, foreachdo, withalldo (beats xargs!), ...
  • Introduction

Download and Install

  • You can get jsh for Unix environments (including Cygwin) using this installer: [requires cvs]
  • Alternatively, you can get a minimal install of just one file (and a bunch of symlinks), which will download other jsh scripts when they are needed: [experimental! requires wget or lynx, works best in zsh]
  • Hwi has a browseable CVS repository of the code, where you may grab a tarball.
  • You can also compile a standalone copy of a script using the following form:
    • Note that the compiler (actually an inter-script dependency resolver and shell function builder with dodgy hacks) is experimental at the time of writing, and only a few scripts have defined dependencies.

User Documentation

  • Starting jsh after installation:
    • To start a shell in the jsh environment, run "<install_dir>/jsh".
    • To have your shell always start in the jsh environment, add the following lines to your ~/.bash_profile or just the second to your ~/.zshrc:
      • export JPATH=<install_dir>
      • . $JPATH/startj
      • learn more
        • The use of "." in the last line is an example of "sourcing", which means running the script in the current shell, rather than in its own shell.
    • To start jsh without all the fancy user shell setup:
      • Before running jsh or sourcing startj: export STARTJ_SIMPLE=something
      • Or alternatively, source startj-simple: . $JPATH/startj-simple
      • This will simply add $JPATH/tools to your path, and initialise any variables needed for jsh scripts.
  • For documentation on a script, use "jdoc <scriptname>" or "man", or run the command without arguments, or with "--help".
  • To update your jsh installation, type "updatejsh" from within the environment. This simply updates CVS, and re-links the tools/ directory to add any new scripts.
  • There is a list of documented scripts and also a big list of all the scripts.
  • And now a list of recently updated scripts.

Bug reports / Feature requests / Other feedback

  • Please report any problems you encounter to joey without_this_bit @neuralyte.org .
  • Sometimes we'll need to fix a bit of code.
  • Sometimes the code is fine, in which case documentation is probably needed!

Competitions!

  • In order to clean up Jsh so that it's useful to You, I need a user base who can provide feedback and feature requests.
  • And in light of rumours about the difficulty in the initial learning curve of jsh, I propose:
  • BUG COMPETITION!
    • The senders of the next three bug reports to jsh will earn a FREE ICE-CREAM!
    • Small print: bugs must be reproduceable, worthwhile, hard for the user to fix, but easy for the authour to fix;
    • limited to one ice-cream per author per week, content of ice-cream subject to change, offer ends when oil runs out.
  • DOCUMENTATION COMPETITION!
    • The contributors of the next three pieces of documentation (either for a jsh script or to the Wiki) will earn a FREE ICE-CREAM!
    • Small print: documentation must be > 64 words, and worthwhile.
  • BONUS PRIZES for FEATURE REQUESTS!
    • I would also welcome feedback with regard to needed introductory documentation, or recommendations for handy new scripts.
    • Prizes (ice-creams or otherwise) may be awarded for good feature requests.
  • 5 ice-creams still available.

Developer Documentation: Overview and Policy

  • Status
    • Jsh is at present an unweighted mixture of useful stable scripts, handy template scripts, and prototyped functions.
    • Documented scripts should work just fine (check beforehand with: "jdoc <command>"), but others (e.g. templates) may need a little tweaking to do what you need.
    • BitsOfItAreExcellent?, and ItWorksFineForMe?, but the scripts could do with some n-D categorisation (e.g. command/lib/stable/template metadata) for the end-user.
    • Regarding people's fears given the Turing completeness of shellscript: I try to use common-sense with regard to error-trapping, so most problems should simply result in a non-zero exit code. In my experience of jsh, and to quote the Debian promise: "If you break it, you get to keep both halves." Obviously, you should have some idea of the purpose of a script and its inputs and outputs before you run it. If you wish to voice any specific concerns about security here, I will attempt to address them.
  • What's jsh made of?
    • $JPATH/shellscripts contains a working CVS checkout of the jsh scripts. You can trim this tree, as long as you don't remove any scripts which others depend on! (You can use jdoc or jshdepwiz to check for dependencies.)
    • $JPATH/tools contains symlinks to each of those scripts. This means only that one directory needed to be added to your $PATH. You can trim this too, with less-permanent effects.
      • Why do we need this directory on the $PATH? Because some scripts rely on others being available in order to run (in the interests of divide-and-conquer, and re-usability). Unless you use compilejshscript, or until a jsh-subset-installer exists, this is the simplest way to ensure dependencies are resolved.
      • refreshtoollinks regenerates the $JPATH/tools/ directory, and is called by updatejsh.
    • $JPATH/tmp (or wherever is chosen by "jgettmpdir -top") contains temporary files for use by jsh scripts, which are cleaned up where possible, but may occasionally need cleaning manually. [How do I catch Ctrl+C breakouts in a shscript, so I can cleanup when an intentionally neverending script is stopped.]
    • $JPATH/trash contains files you "deleted" with the "del" command. If you are not running reclaimspace in your crontab, then you will have to purge this directory manually to actually remove files.
  • Some policy (currently an explanation of keywords found in scripts):
    • Look for the following keywords in scripts (I get vim to colour them brightly): BUG, BUGS, TODO, CONSIDER, DONE
    • Dependency meta comments in scripts:
      • Comments in a script of the form "# jsh-(ignore-)(ext-)depends: .*" are meta-data describing the dependency of that script on other jsh scripts or on external Unix commands.
      • This meta-data is generated by jshdepwiz, with a developer checking its suggestions.
      • The dependency data is used by compilejshscript and jshstub. It will be useful for any future jsh-subset-installer.
      • The -ignore option means jshdepwiz has guessed those dependencies, but a developer has confirmed they are not dependencies, and jshdepwiz should ignore them in future.
      • TODO: need an optional-depends tag (akin to "suggested" rpms/debs), to describe situations where the dependency is not vital, but enhances the script if present.
    • Variables:
      • JPATH: should be set, many scripts use it ("jdoc JPATH", then answer yes), but it might not be needed for compiled(standalone) scripts.
      • PATH: should have $JPATH/tools in it, in order to find dependent jsh scripts, unless dependencies are resolved otherwise.
      • TOPTMP: can be set if the user wants jsh to use temp files in a specific location, otherwise generated by jgettmpdir -top, which is called if needed by jgettmp.
      • DEBUG: if this variable exists, some apps will produce pretty debuggin info
  • This could be a FAQ section. Please post Qs 4 me 2 A.
  • Config (rc) files:
    • Some programs / scripts need config files or directories, to persist state between sessions. Traditionally for Unix programs this is the file or directory:
      • ~/.<program_namerc>
    • Originally jsh intended to keep the config for a script in:
      • $JPATH/data/<scriptname>.whatever
    • However the new policy (if endorsed by users), will be to store configs in:
      • $HOME/.jsh-<scriptname>rc , or
      • $HOME/.jsh-<scriptname>/* , or possibly
      • $HOME/.jsh-<scriptname>-<whatever>
    • This way, the config is separated from the installation directory, and in the traditional Unix location, but the "jsh-" keeps all jsh config files separate from others, and the association with the program is also obvious.
    • TODO: refactor scripts to migrate any config files in other locations to the new policy location.
    • Also:
      • Always use $HOME, not ~. It works more often, and standardisation makes things easy to track.
      • CONSIDER: Standardise overriding of config file, eg by: export JSH_<scriptname_in_uppercase>_CFG=<alternative_config_location>
      • "Config" might not be the appropriate word. Some programs just like to keep a cache. At the moment, scripts tend to store this kind of data in $TOPTMP, $TOPTMP/memo, or in /tmp/something. Policy here would be good.
        • Ideally we should move $TOPTMP out of $JPATH/tmp and into /tmp/jsh-tempdir-<username>, again to separate from installation, and also so that Linux can clear it automatically (only change of autocleaning $JPATH/tmp has been reclaimspace).
  • I have documented only partially used standards above, but they are the standards I have used so far, and will be useful for refactoring in future.
    • For example, since we know what to look for, it is fairly easy to use grep (in $JPATH/tools) to find instances of standardised code in jsh...
      • To find implementations of the $JPATH/data standard: higrep "\$JPATH/data" *
      • To find references to /tmp/<something>: higrep "\(^\|[^H]\)/tmp/" *
      • To find implementations of the $TOPTMP standard: higrep "\$TOPTMP/" *
    • For jsh tidiness, any uses of the top two standards should be refactored into the bottom standard.
      • However, the drawback is that the shell must have a defined $TOPTMP when the script is called, or the script must source ". jgettmpdir -top". (Previosuly $JPATH/tmp was known to exist, and $JPATH existed, or /tmp was used.) This might make the desirability of the change desirable. However, it might seem reasonable to have all scripts depend on just one that sets up and checks the standard environment (either called when script is run, or when the user shell starts like it is now).

Todo

  • On the web page:
    • Provide more examples of useful scripts on the web page. (See third todo.)
    • Provide easy ways to search for useful scripts. (Currently "jdoc " then "y" but that's really a developer search.)
    • Make pretty online documentation for scripts which have it. (For auto-extraction, provide a simple way for developers to indicate that there are docs are in the comments in the code.)
    • Can we use some clever javascript (or maybe some serverside method) to create hyperlinks for any jsh scripts mentioned in this webpage, to an online copy of each script. For example: "You can use makebackup to create incremental backups of important data.
  • To the code:
    • Option to start jsh with $JPATH/tools placed at the far end of $PATH, hence overriding nothing (save aliases).
    • Separate user friendliness from library scripts. (Could use aliases for user. Need to disable bells+whistles when being used as library.)
      • This is partially achieved by startj -simple, which only loads the library and no bash/zsh configs. But, eg. tools which use colour to highlight their output not do so if they are being used as library functions.
    • Tidy up categorisation (make n-D).
    • Remove any joey-specific data. (Move to config.)
    • Allow partial install, checking dependencies. (Only really a case of symlinks in tools/ .)
    • Provide centralised configuration. Or provide standardised configuration for each tool.
    • Check dependencies for external programs. (Determine which standin/ tools are needed during install/update.)
    • Flesh out documentation.
    • Finish the debug info warn error scripts.
    • Should I turn all these separate scripts into functions?
      • I believe it would mean turning all 'exit's into 'return's, and localising local variables, which is kind of neat anyway, because...
      • It would make loading jsh scripts as functions (which might make things faster) more likely to work.
        • makeshfunction and compilejshscript currently attempt to convert shellscripts into shell functions, but there are probably unwanted side-affects compared to how jsh works normally, running each script in a separate shell.
    • Policy decisions:
      • If all documentation code in jsh scripts is written in a standardised way, then we can automate stripping of documentation from scripts for runtime / distribution as functions.
      • Should jsh scripts store config files in ~/ or /etc? Should jsh be in the config filenames? Should we standardise that script uses config files /etc/jsh/.conf and ~/.jsh_.conf by default? (We should really automate so that user can specify alternative places to look for configs and/or so that we can change the standards later! 4sure: abstract config handling thru a script!)
    • Another solution (for the latter at least) is that jsh scripts might specify in their dependencies, which version of each dependent script they require, so that the most minimal implementation can be used.
    • Well evolved jsh scripts tend to hold comments noting why various previous/alternative implementations are not ideal. These are useful if a developer wants to alter the script, but should be stripped for runtime/distribution.

History/Origins

  • How did it start? I was editing polyray scripts, running qbasic, and compiling djgpp code under DOS. I needed some batch scripts to help me do that..
  • When I moved to Linux, I decided to port needed batch scripts. One of my earliest shell hacks must have been the d.zsh (go back in directory history) script.
  • Jsh is also an expression of my attempt to learn shellscript and unix. Jsh remembers a lot of the things I can't be bothered to, by reference in my own language. In this sense, it is useless.
  • Who cares?

Related links

-- JoeyTwiddle - 24 May 2004


to top


You are here: Neuralyte > WelcomePage > ProjectsPage > ProjectJsh

to top

Copyright © 1999-2011 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback