     doit 0.9, Sep 21, 2021

     unknown

     Copyright © 2010, mininiss

   Generated by Sphinx 4.2.0.

doit
****

     doit 0.9, Sep 21, 2021

     unknown

     Copyright © 2010, mininiss

Contents:

1 Installing
************

   * Using pip(1) or easy_install(2):

          $ pip install doit
          $ easy_install doit

   * Download(3) the source and:

          $ python setup.py install

   * Get latest development version:

          $ bzr branch lp:doit

   * Ubuntu packages

     Add the following line to your system’s software sources

          deb http://ppa.launchpad.net/doit-team/ppa/ubuntu karmic main

     PPA archive(4).  (help(5))

     Note:
        * ‘doit’ dependends on the packages pyinotify(6) (for linux).
          macfsevents(7) (mac)

        * for python2.4 and python2.5 users ‘doit’ depends on the
          packages ‘simplejson’ and ‘multiprocessing’.  On python 2.6
          the stdlib ‘json’ and ‘multiprocessing’ are used.

     Warning: for Windows users:

        * New releases are not tested on Windows.

        * There is a bug on setup tools.  Check this bug(8) for
          instructions.

   ---------- Footnotes ----------

   (1) http://pip.openplans.org/

   (2) http://peak.telecommunity.com/DevCenter/EasyInstall

   (3) http://pypi.python.org/pypi/doit

   (4) https://launchpad.net/~doit-team/+archive/ppa/

   (5) https://launchpad.net/+help/soyuz/ppa-sources-list.html

   (6) http://trac.dbzteam.org/pyinotify

   (7) http://pypi.python.org/pypi/MacFSEvents

   (8) http://bugs.launchpad.net/doit/+bug/218276

2 Tasks
*******

2.1 Intro
=========

‘doit’ is all about automating the execution of tasks.  Tasks can
execute external shell commands/scripts or python functions (actually
any callable).  So a task can be anything you can code :)

Tasks are defined using python(1), in a plain python file with some
conventions.  A function that starts with the name ‘task_’ defines a
`task generator' recognized by ‘doit’.  These functions must return (or
yield) dictionaries representing a `task'.  A python module/file that
defines `tasks' for ‘doit’ is called `dodo' file (that is something like
a ‘Makefile’ for ‘make’).

     Note: You should be comfortable with python basics.  If you don’t
     know python yet check Python tutorial(2) and Dive Into Python(3).

Take a look at this example (file dodo.py):

     def task_hello():
         """hello from shell & python! """

         def python_hello():
             output = open("hello.txt", "a")
             output.write("Python says Hello World!!!\n")
             output.close()
             return True

         return {'actions': ['echo Hello World!!! > hello.txt',
                             (python_hello,)]
                 }

When ‘doit’ is executed without any parameters it will look for tasks in
a file named ‘dodo.py’ in the current folder and execute its tasks.

     eduardo@eduardo~$ doit
     .  hello

On the output it displays which tasks were executed.  In this case the
‘dodo’ file has only one task, ‘hello’.

   ---------- Footnotes ----------

   (1) http://python.org/

   (2) http://docs.python.org/tut/

   (3) http://www.diveintopython.org/

2.2 Actions
===========

Every `task' must define `actions'.  It can optionally defines other
attributes like ‘targets’, ‘dependencies’, ‘doc’ …

Actions define what the task actually do.  A task can define any number
of actions.  There 2 kinds of ‘actions’: `cmd-action' and
`python-action'.

2.2.1 cmd-action
----------------

If ‘action’ is a string it will be executed by the shell.

The result of the task follows the shell convention.  If the process
exits with the value ‘0’ it is successful.  Any other value means the
task failed.

2.2.2 python-action
-------------------

If ‘action’ is a tuple ‘(callable, *args, **kwargs)’ - only ‘callable’
is required.  ‘args’ is a sequence and ‘kwargs’ is a dictionary that
will be used as positional and keywords arguments for the callable.  see
Keyword Arguments(1).

The result of the task is given by the returned value of the ‘action’
function.  So it must return a `boolean' value ‘True’, ‘None’, a
dictionary or a string to indicate successful completion of the task.
Use ‘False’ to indicate task failed.  If it raises an exception, it will
be considered an error.  If it returns any other type it will also be
considered an error but this behavior might change in future versions.

   ---------- Footnotes ----------

   (1)
http://docs.python.org/tutorial/controlflow.html#keyword-arguments

2.2.3 example
-------------

It is easy to include dynamic (on-the-fly) behavior to your tasks with
python code from the ‘dodo’ file.  Let’s take a look at another example:

     def task_hello():
         """hello from shell & python! """

         def python_hello(times, text):
             output = open("hello.txt", "a")
             output.write(times * text)
             output.close()
             return True

         msg = 3 * "hi! "
         return {'actions': ['echo %s > hello.txt' % msg,
                             (python_hello, [3, "py!\n"])]
                 }

The function ‘task_hello’ is a `task generator' not the task itself.
The body of the task generator function is always executed when the dodo
file is loaded.

     Note: The body of task generators are executed even if the task is
     not going to be executed.  So in this example the line ‘msg = 3 *
     “hi! “’ will always be executed.  The body of task generators
     should be used to create task metadata only, not execute tasks!
     From now on when it said that a `task' is executed, read the task’s
     actions are executed.

2.3 sub-tasks
=============

Most of the time we want to apply the same task several times in
different contexts.

The task function can return a python-generator that yields
dictionaries.  Since each sub-task must be uniquely identified it
requires an additional field ‘name’.

     def task_create_file():
         for i in range(3):
             filename = "file%d.txt" % i
             yield {'name': filename,
                    'actions': ["touch %s" % filename]}

     eduardo@eduardo:~$ doit
     .  create_file:file0.txt
     .  create_file:file1.txt
     .  create_file:file2.txt

2.4 Task selection
==================

By default all tasks are executed in the same order as they were defined
(the order may change to satisfy dependencies).  You can control which
tasks will run in 2 ways.

Another example

     DOIT_CONFIG = {'default_tasks': ['t3']}

     def task_t1():
         return {'actions': ["touch task1"],
                 'targets': ['task1']}

     def task_t2():
         return {'actions': ["echo task2"]}

     def task_t3():
         return {'actions': ["echo task3"],
                 'file_dep': ['task1']}

2.4.1 DOIT_CONFIG -> default_tasks
----------------------------------

`dodo' file defines a dictionary ‘DOIT_CONFIG’ with ‘default_tasks’, a
list of strings where each element is a task name.  In the example above
we don’t want to “install” by default.

     eduardo@eduardo:~$ doit
     .  t1
     .  t3

Note that the only the task `t3' was specified to be executed.  But its
dependencies include a target of another task.  So those tasks were
automatically executed also.

2.4.2 command line selection
----------------------------

From the command line you can control which tasks are going to be
execute by passing its task name.  Any number of tasks can be passed as
positional arguments.

     eduardo@eduardo:~$ doit t2
     .  t2

You can also specify which task to execute by its target:

     eduardo@eduardo:~$ doit task1
     .  t1

2.4.3 sub-task selection
------------------------

You can select sub-tasks from the command line specifying its full name.

     def task_create_file():
         for i in range(3):
             filename = "file%d.txt" % i
             yield {'name': filename,
                    'actions': ["touch %s" % filename]}

     eduardo@eduardo:~$ doit create_file:file2.txt
     .  create_file:file2.txt

2.4.4 wildcard selection
------------------------

You can also select tasks to be executed using a glob(1) like syntax (it
must contais a ‘*’).

     eduardo@eduardo:~$ doit create_file:file*
     .  create_file:file1.txt
     .  create_file:file2.txt
     .  create_file:file3.txt

   ---------- Footnotes ----------

   (1) http://docs.python.org/library/glob.html

3 Dependencies & Targets
************************

‘dependencies’ and ‘targets’ are optional attributes of a task.

Dependency

     A `dependency' indicates an input to the task execution.

Target

     A `target' is the result/output produced by the task execution.

i.e.  In a compilation task the source file is a `dependency', the
object file is a `target'.

     def task_compile():
         return {'actions': ["cc -c main.c"],
                 'file_dep': ["main.c", "defs.h"],
                 'targets': ["main.o"]
                 }

3.1 up-to-date
==============

‘doit’ automatically keeps track of file-dependencies.  It saves the
signature (MD5) of the dependencies every time the task is completed
successfully.  In case there is no modification in the dependencies and
the targets already exist, it skip the task execution to save time, as
it would produce the same output from the previous run.

So if there are no modifications to the dependencies and you run ‘doit’
again.  The execution of the task’s actions is skipped.

     eduardo@eduardo:~$ doit
     .  compile
     eduardo@eduardo:~$ doit
     -- compile

Note the ‘--’ (2 dashes, one space) on the command output on the second
time it is executed.  It means, this task was up-to-date and not
executed.

3.2 example - no targets
========================

Different from most build-tools dependencies are on tasks not on
targets.  So ‘doit’ can take advantage of the “execute only if not
up-to-date” feature even for tasks that not define targets.

Lets say you work with a dynamic language (python in this example).  You
don’t need to compile anything but you probably wants to apply a
lint-like tool (PyChecker(1)) to your source code files.  You can define
the source code as a dependency to the task.

     def task_checker():
         return {'actions': ["pychecker sample.py"],
                 'file_dep': ["sample.py"]}

     eduardo@eduardo:~$ doit
     .  checker
     eduardo@eduardo:~$ doit
     -- checker

Note the ‘--’ again.

   ---------- Footnotes ----------

   (1) http://pychecker.sourceforge.net/

3.3 task-dependency
===================

We have seen only dependencies on files up to now.  On ‘doit’ you can
also define `task-dependency'.  To define a dependency on another task
use the task name (whatever comes after ‘task_’ on the function name) in
the “task_dep” attribute.  It is used to enforce tasks are executed on
the desired order.  By default tasks are executed on the same order as
they were defined in the ‘dodo’ file.

     Note: A `task-dependency' `only' indicates that another task should
     be “executed” before itself.  The task-dependency might not really
     be executed if it is `up-to-date'.

     Note: `task-dependency' s are `not' used to determine if a task is
     up-to-date or not.  If a task defines only `task-dependency' or no
     dependencies at all it will always be executed.

This example we make sure we include a file with the latest revision
number of the bazaar repository on the tar file.

     def task_tar():
         return {'actions': ["tar -cf foo.tar *"],
                 'task_dep':['version'],
                 'targets':['foo.tar']}

     def task_version():
         return {'actions': ["bzr version-info > revision.txt"]}

     eduardo@eduardo:~$ doit
     .  version
     .  tar

3.3.1 groups
------------

You can define group of tasks by adding tasks as dependencies and
setting its ‘actions’ to ‘None’.

     def task_foo():
         return {'actions': ["echo foo"]}

     def task_bar():
         return {'actions': ["echo bar"]}

     def task_mygroup():
         return {'actions': None,
                 'task_dep': ['foo', 'bar']}

Note that tasks are never executed twice in the same “run”.

3.4 targets
===========

Targets can be any file path (a file or folder).  If a target doesn’t
exist the task will be executed.  There is no limitation on the number
of targets a task may define.

Lets take the compilation example again.

     def task_compile():
         return {'actions': ["cc -c main.c"],
                 'file_dep': ["main.c", "defs.h"],
                 'targets': ["main.o"]
                 }

If there are no changes in the dependency the task execution is skipped.
But if the target is removed the task is also executed again.  But only
if does not exist.  If the target is modified but the dependencies do
not change the task is not executed again.

     eduardo@eduardo:~$ doit
     .  compile
     eduardo@eduardo:~$ doit
     -- compile
     eduardo@eduardo:~$ rm main.o
     eduardo@eduardo:~$ doit
     .  compile
     eduardo@eduardo:~$ echo xxx > main.o
     eduardo@eduardo:~$ doit
     -- compile

3.5 result-dependency
=====================

In some cases you can not determine if a task is “up-to-date” only based
on some input files, you might need to run an external process to check
that.  `doit' defines a “result-dependency” to deal with these cases
without need to create an intermediate file with the reulsts of the
process.

i.e.  Suppose you want to send an email everytime you run `doit' on a
bazaar repository that contains a new revision number.

     def task_version():
     	return {'actions': ['bzr version-info --custom --template="{revno}\n"']}

     def task_send_email():
     	return {'actions': ['echo "TODO: send an email"'],
     	        'result_dep': ['version']}

Note the “restul_dep” with the name of the task (version).  ‘doit’ will
keep track of the output of the task `version' and will execute
`send_email' only when the bazaar repository has a new version since
last time `doit' was executed.

The “result” from the dependent task compared between different runs is
given by its last action.  The content for python-action is the value of
the returned string.  For cmd-actions is the output send to stdout plus
stderr.

3.6 calculated-dependencies
===========================

Calculation of dependencies might be an expensive opration, so not
suitable to be done on task-generators.  For this situation is better to
delegate the calculation of depencies to another task.  The task
calcutating dependencies must have a python-action returning a
dictionary with ‘file_dep’, ‘task_dep’, result_dep, or another
‘calc_dep’.

On the example below ‘mod_deps’ prints on the screen all direct
dependencies from a module.  The depencies itself are calculated on task
‘get_dep’ (note: get_dep has a fake implementation where the results are
taken from a dict).

     DOIT_CONFIG = {'verbosity': 2}

     MOD_IMPORTS = {'a': ['b','c'],
                    'b': ['f','g'],
                    'c': [],
                    'f': ['a'],
                    'g': []}

     def print_deps(mod, dependencies):
         print "%s -> %s" % (mod, dependencies)
     def task_mod_deps():
         """task that depends on all direct imports"""
         for mod in MOD_IMPORTS.iterkeys():
             yield {'name': mod,
                    'actions': [(print_deps,(mod,))],
                    # file_dep: False => make sure action is always executed
                    'file_dep': [mod, False],
                    'calc_dep': ["get_dep:%s" % mod],
                    }

     def get_dep(mod):
         # fake implementation
         return {'file_dep': MOD_IMPORTS[mod]}
     def task_get_dep():
         """get direct dependencies for each module"""
         for mod in MOD_IMPORTS.iterkeys():
             yield {'name': mod,
                    'actions':[(get_dep,[mod])],
                    'file_dep': [mod],
                    }

3.7 getargs
===========

‘getargs’ provides a way to use values computed in one task from another
task.

     DOIT_CONFIG = {'default_tasks': ['use']}

     def task_compute():
        def comp():
            return {'x':5,'y':10, 'z': 20}
        return {'actions': [(comp,)]}

     def task_use():
        return {'actions': ['echo %(x)s %(z)s - 5 20'],
                'getargs': {'x':'compute.x',
                            'z':'compute.z'},
                'verbosity': 2,
                }

3.8 run-once
============

Sometimes there is no dependency for a task but you do not want to
execute it all the time.  If you use ‘True’ as dependency (in
“file_dep”) the task will not be executed again after the first
successful run.  This is mostly used together with targets.

Suppose you need to download something from internet.  There is no
dependency though you do not want to download it many times.

     def task_get_pylogo():
         url = "http://python.org/images/python-logo.gif"
         return {'actions': ["wget %s" % url],
                 'targets': ["python-logo.gif"],
                 'file_dep': [True]
                 }

Note that even with `run-once' the file will be downloaded again in case
the target is removed.

     eduardo@eduardo:~$ doit
     .  get_pylogo
     eduardo@eduardo:~$ doit
     -- get_pylogo
     eduardo@eduardo:~$ rm python-logo.gif
     eduardo@eduardo:~$ doit
     .  get_pylogo

3.9 never-upto-date
===================

A dependency (file_dep) value ‘False’ can also be used in a dependency
list to indicate that task should never be considered up-to-date.

3.10 Dependency None
====================

‘None’ values as “file_dep” will just be ignored.  This is used when the
value is dinamically calculated.

3.11 keywords on actions
========================

It is common situation to use task information such as `targets',
`dependencies', or `changed' in its own actions.  Note: Dependencies are
only `file-dependencies'.

For `cmd-action' you can use the python notation for keyword
substitution on strings.  The string will contain all values separated
by a space (” “).

For `python-action' create a parameter in the function ‘doit’ will take
care of passing the value when the function is called.  The values are
passed as list of strings.

     def task_hello():
         """hello from shell & python! """

         def python_hello(targets):
             output = open(targets[0], "a")
             output.write("Python says Hello World!!!\n")
             output.close()

         return {'actions': ['echo Hello World!!! > %(targets)s',
                             (python_hello,)],
                 'targets': ["hello.txt"]
                 }

3.12 Environment setup
======================

Some tasks require some kind of environment setup.  Tasks may have a
list of “setup” task.

   * the setup-task will be executed only if the task is to be executed
     (not up-to-date)

   * setup-tasks are just normal tasks that follow all other task
     behavior

Task may also define a ‘teardown’ actions.  These actions are executed
after all tasks have finished their execution.

Example:

     ### task setup env. good for functional tests!
     DOIT_CONFIG = {'verbosity': 2,
                    'default_tasks': ['withenvX', 'withenvY']}

     def start(name):
         print "start %s" % name
     def stop(name):
         print "stop %s" % name

     def task_setup_sample():
         for name in ('setupX', 'setupY'):
             yield {'name': name,
                    'actions': [(start, (name,))],
                    'teardown': [(stop, (name,))],
                    }

     def task_withenvX():
         for fin in ('a','b','c'):
             yield {'name': fin,
                    'actions':['echo x %s' % fin],
                    'setup': ['setup_sample:setupX'],
                    }

     def task_withenvY():
         return {'actions':['echo y'],
                 'setup': ['setup_sample:setupY'],
                 }

4 Command run
*************

Most of the time you just want to execute your tasks that’s what `run'
does.  Since it is by far the most common operation it is also the
default, so if you don’t specify any sub-command to `doit' it will
`run'.  So ‘$ doit’ and ‘$ doit run’ are the same thing.

The basics of task selection were introduced in Task Selection.

4.1 dodo file
=============

By default all commands are relative to ‘dodo.py’ in the current folder.
You can specify a different `dodo' file containing task with the flag
‘-f’.  (This is valid for all sub-commands)

     eduardo@eduardo:~$ doit -f release.py

4.2 verbosity
=============

By default the stdout from a task is captured and its stderr is sent to
the console.  If the task fails or there is an error the stdout and a
traceback (if any) is displayed.

There are 3 levels of verbosity:

0:

     capture (do not print) stdout/stderr from task.

1 (default):

     capture stdout only.

2:

     do not capture anything (print everything immediately).

You can control the verbosity by:

   * –verbosity/-v command line option.

     change verbosity of all executed tasks.

     eduardo@eduardo:~$ doit --verbosity 2

   * task attribute verbosity

     def task_print():
         return {'actions': ['echo hello'],
                 'verbosity': 2}

     eduardo@eduardo:~$ doit
     .  print
     hello

4.3 parameters
==============

It is possible to pass option parameters to the task through the command
line.

Just a add a ‘params’ fiels to the task dictionary.  ‘params’ must be a
list of dictionaries where every entry is an option parameter.  Each
parameter must define a name, and a default value.  It can optionally
define a “short” and “long” names to be used from the command line (it
follows unix command line conventions).  It may also specify a type the
parameter should be converted to.

See the example:

     def task_py_params():
         def show_params(param1, param2):
             print param1
             print 5 + param2
         return {'actions':[(show_params,)],
                 'params':[{'name':'param1',
                            'short':'p',
                            'default':'default value'},

                           {'name':'param2',
                            'long':'param2',
                            'type': int,
                            'default':0}],
                 'verbosity':2,
                 }

     def task_cmd_params():
         return {'actions':["echo mycmd %(flag)s xxx"],
                 'params':[{'name':'flag',
                            'short':'f',
                            'long': 'flag',
                            'default': ''}],
                 'verbosity': 2
                 }

For python-actions the python function must define arguments with the
same name as a task parameter.

     eduardo@eduardo:~$ doit py_params -p abc --param2 4
     .  py_params
     abc
     9

For cmd-actions use python string substitution notation:

     eduardo@eduardo:~$ doit cmd_params -f "-c --other value"
     .  cmd_params
     mycmd -c --other value xxx

4.4 title
=========

By default when you run ‘doit’ only the task name is printed out on the
output.  You can customize the output passing a “title” function to the
task:

     def show_cmd(task):
         return "executing... %s" % task.name

     def task_custom_display():
         return {'actions':['echo abc efg'],
                 'title': show_cmd}

     eduardo@eduardo:~$ doit
     .  executing... Cmd: echo abc efg

4.5 dir (cwd)
=============

By default relative paths of file used on the ‘dodo’ file and the
“current working directory” used on python execution is the same as the
location of the ‘dodo’ file.  You can specify a different `cwd' with the
–dir/-d option.

     eduardo@eduardo:~$ doit --dir path/to/another/cwd

4.6 continue
============

By default the execution of tasks is halted on the first task failure or
error.  You can force it to continue execution with the option
–continue/-c

     eduardo@eduardo:~$ doit --continue

4.7 parallel execution
======================

‘doit’ supports parallel execution (using multiple processes)
–process/-n.  The multiprocessing(1) module is used.  So the same
restrictions also apply to the use of multiprocessing in ‘doit’.

     eduardo@eduardo:~$ doit -n 3

   ---------- Footnotes ----------

   (1) http://docs.python.org/library/multiprocessing.html

4.8 reporter
============

‘doit’ provides different “reporters” to display running tasks info on
the console.  Use the option –reporter/-r to choose a reporter.  Apart
from the default it also includes:

        * executed-only: Produces zero output unless a task is executed

        * json: Output results in JSON format

     eduardo@eduardo:~$ doit --reporter json

4.9 custom_reporter
===================

It is possible to define your own custom reporter.  Check the code on
doit/reorter.py… It is easy to get started by sub-classing the default
reporter as shown below.  The custom reporter must be configured using
DOIT_CONFIG dict.

     from doit.reporter import ConsoleReporter

     class MyReporter(ConsoleReporter):
         def execute_task(self, task):
             self.outstream.write('MyReporter --> %s\n' % task.title())

     DOIT_CONFIG = {'reporter': MyReporter,
                    'verbosity': 2}

     def task_sample():
         for x in range(3):
             yield {'name': str(x),
                    'actions': ['echo out %d' % x]}

4.10 output-file
================

The option –output-file/-o let you output the result to a file.

     eduardo@eduardo:~$ doit --output-file result.txt

4.11 config
===========

Command line parameters can be set straight on a ‘dodo’ file.  This
example below sets the default tasks to be run, the ‘continue’ option,
and a different reporter.

     DOIT_CONFIG = {'default_tasks': ['my_ytask_1', 'my_task_2'],
                    'continue': True,
                    'reporter': 'json'}

So if you just execute

     eduardo@eduardo:~$ doit

it will have the same effect as executing

     eduardo@eduardo:~$ doit --continue --reporter json my_task_1 my_task_2

You need to check doit_cmd.py(1) to find out how parameter maps to
config names.

     Note: The parameters ‘–file’ and ‘–dir’ can not be used on config
     because they control how the dodo file itself is loaded.

   ---------- Footnotes ----------

   (1)
http://bazaar.launchpad.net/~schettino72/doit/trunk/annotate/head%3A/doit/doit_cmd.py

5 Other Commands
****************

Lets use more complex example to demonstrate the command line features.
The example below is used to manage a very simple C project.

     DOIT_CONFIG = {'default_tasks': ['link']}

     # map source file to dependencies
     SOURCE = {
         'main': ["defs.h"],
         'kbd': ["defs.h", "command.h"],
         'command': ["defs.h", "command.h"],
         }

     def task_link():
         "create binary program"
         OBJECTS = ["%s.o" % module for module in SOURCE.iterkeys()]
         return {'actions': ['cc -o %(targets)s %(dependencies)s'],
                 'file_dep': OBJECTS,
                 'targets': ['edit'],
                 'clean': True
                 }

     def task_compile():
         "compile C files"
         for module, dep in SOURCE.iteritems():
             dependencies = dep + ['%s.c' % module]
             yield {'name': module,
                    'actions': ["cc -c %s.c" % module],
                    'targets': ["%s.o" % module],
                    'file_dep': dependencies,
                    'clean': True
                    }

     def task_install():
         "install"
         return {'actions': ['echo install comes here...'],
                 'task_dep': ['link'],
                 'doc': 'install executable (TODO)'
                 }

5.1 help
========

‘doit’ comes with several commands.  ‘doit help’ will list all available
commands.

You can also get help from each available command.  e.g.  ‘doit help
run’.

‘doit help task’ will deisplay information on all fields/attributes a
task dictionary from a ‘dodo’ file accepts.

5.2 list
========

`list' is used to show all tasks available in a `dodo' file.

     eduardo@eduardo:~$ doit list
     link : create binary program
     compile : compile C files
     install : install executable (TODO)

The task description is taken from the first line of task function
doc-string.  You can also set it using the `doc' attribute on the task
dictionary.

5.3 forget
==========

Suppose you change the compilation parameters in the compile action.  Or
you changed the code frmo a python-action.  `doit' will think your task
is up-to-date based on the dependencies but actually it is not!  In this
case you can use the `forget' command to make sure the given task will
be executed again even with no changes in the dependencies.

If you do not specify any task, the default tasks are “`forget'”.

     eduardo@eduardo:~$ doit forget

     Note: `doit' keeps track of which tasks are successful in the file
     ‘.doit.db’.  This file uses JSON.

5.4 clean
=========

A common scenario is a task that needs to “revert” its actions.  A task
may include a `clean' attribute.  This attribute can be ‘True’ to remove
all of its target files.  If there is a folder as a target it will be
removed if the folder is empty, otherwise it will display a warning
message.

The `clean' attribute can be a list of actions, again, an action could
be a string with a shell command or a tuple with a python callable.

You can specify which task to `clean'.  If no task is specified the
clean operation of default tasks are executed.

     eduardo@eduardo:~$ doit clean

5.5 ignore
==========

It is possible to set a task to be ignored/skipped (that is not
executed).  This is useful for example when you are performing checks in
several files and you want to skip the check in some of them
temporarily.

     def task_create_file():
         for i in range(3):
             filename = "file%d.txt" % i
             yield {'name': filename,
                    'actions': ["touch %s" % filename]}

     eduardo@eduardo:~$ doit
     .  create_file:file0.txt
     .  create_file:file1.txt
     .  create_file:file2.txt
     eduardo@eduardo:~$ doit ignore create_file:file1.txt
     ignoring create_file:file1.txt
     eduardo@eduardo:~$
     .  create_file:file0.txt
     !! create_file:file1.txt
     .  create_file:file2.txt

Note the ‘!!’, it means that task was ignored.  To reverse the ‘ignore’
use ‘forget’ sub-command.

5.6 auto
========

     Warning: Supported on Linux and Mac only.

‘auto’ sub-command is an alternative way of executing your tasks.  It is
a long running process that only terminates when it is interrupted
(Ctrl-C). When started it will execute the given tasks.  After that it
will watch the file system for modifications in the file-dependencies.
When a file is modified the tasks are re-executed.

     eduardo@eduardo:~$ doit auto

6 Tools
*******

Following the `batteries-included' philosophy `doit' includes some
common used actions.  (well not many by now :p)

6.1 create_folder
=================

Creates a folder if it does not exist yet.

     from doit.tools import create_folder

     BUILD_PATH = "_build"

     def task_build():
         return {'actions': [(create_folder, [BUILD_PATH]),
                             'touch %(targets)s'],
                 'targets': ["%s/file.o" % BUILD_PATH]
                 }

6.2 title_with_actions
======================

Return task name task actions from a task.  This function can be used as
‘title’ attribute of a task dictionary to provide more detailed
information of the action being executed.

     from doit.tools import title_with_actions

     def task_with_details():
         return {'actions': ['echo abc 123'],
                 'title': title_with_actions}

6.3 set_trace
=============

‘doit’ by default redirects stdout and stderr.  Because of this when you
try to use the python debugger with ‘pdb.set_trace’, it does not work
propoerly.  To make sure you get a propper PDB shell you should use
doit.tools.set_trace instead of ‘pdb.set_trace’.

     def need_to_debug():
         # some code here
         from doit import tools
         tools.set_trace()
         # more code

     def task_X():
         return {'actions':[(need_to_debug,)]}

     Note: Note that the orignal stdout is not restored, so the behavior
     of the program from the point the breakpoint is set might be
     different.

7 FAQ
*****

7.1 `doit' is too verbose, why don’t you use decorators?
========================================================

With decorators:

     from doit.i_dont_exist import decorators

     @decorators.dependencies('a','b')
     @decorators.task
     def task_mytask():
        x = 1
        y = 2
        z = 3

Without decorators:

     def task_mytask():
        def the_action():
            x = 1
            y = 2
            z = 3

        return {'actions': [the_action],
                'dependencies': ['a','b']}

   * not sure I stack the decorators in the correct order :P

   * using decorators is `one' line shorter ( but i am not counting the
     import line).  not a big saver in terms of verbosity at all.

   * you need to import the decorator

   * decorators were created to change the behavior of functions.  Of
     course it can do other stuff but it is an overkill.  Decorators may
     look `cool' but are much harder understand, code and debug.

   * we can not replace the dictionaries syntax because it is more
     powerful that using decorators.  The implementation is also
     simpler.

   * There should be one – and preferably only one – obvious way to do
     it.  (zen of python)

8 Changes
*********

8.1 0.9.0 (`2010-06-08')
========================

   - support for dynamic calculated dependencies “calc_dep”

   - support for user defined reporters

   - support “auto” command on mac

   - fix installer on mac.  installer aware of different python versions

   - deprecate ‘dependencies’.  use file_dep, task_dep, result_dep.

8.2 0.8.0 (`2010-05-16')
========================

   - parallel execution of tasks (multi-process support)

   - sub-command “list” option “–deps”, show list of file dependencies

   - select task by wildcard (fnmatch) i.e.  test:folderXXX/*

   - task-setup can be another task

   - task property “teardown” substitute of setup-objects cleanup

   - deprecate setup-objects

8.3 0.7.0 (`2010-04-08')
========================

   - configure options on dodo file (deprecate DEFAULT_TASKS)(#524387)

   - clean and forget act only on default tasks (not all tasks)
     (#444243)

   - sub-command “clean” option “clean-dep” to follow dependencies
     (#444247)

   - task dependency “False” means never up-to-date, “None” ignored

   - sub-command “list” by default do not show tasks starting with an
     underscore, added option (-p/–private)

   - new sub-command “auto”

8.4 0.6.0 (`2010-01-25')
========================

   - improve (speed optimization) of check if file modified (#370920)

   - sub-command “clean” dry-run option (-n/–dry-run) (#444246)

   - sub-command “clean” has a more verbose output (#444245)

   - sub-command “list” option to show task status (-s/–status)
     (#497661)

   - sub-command “list” filter tasks passed as positional parameters

   - tools.set_trace, PDB with stdout redirection (#494903)

   - accept command line optional parameters passed before sub-command
     (#494901)

   - give a clear errror message if .doit.db file is corrupted (#500269)

   - added task option “getargs”.  actions can use computed values from
     other tasks (#486569)

   - python-action might return a dictionary on success

8.5 0.5.1 (`2009-12-03')
========================

   - fix.  task-result-dependencies should be also added as
     task-dependency to force its execution.

8.6 0.5.0 (`2009-11-30')
========================

   - task parameter ‘clean’ == True, cleans empty folders, and display
     warning for non-empty folders

   - added command line option –continue.  Execute all tasks even if
     tasks fails

   - added command line option –reporter to select result output
     reporter

   - added executed-only reporter

   - added json reporter

   - support for task-result dependency #438174

   - added sub-command ignore task

   - added command line option –outfile.  write output to specified file
     path

   - added support for passing arguments to tasks on cmd line

   - added command line option –dir (-d) to set current working
     directory

   - removed dodo-sample sub-command

   - added task field ‘verbosity’

   - added task field ‘title’

   - modified default way a task is printed on console (just show “.
     name”), old way added to doit.tools.task_title_with_actions

8.7 0.4.0 (`2009-10-05')
========================

   - deprecate anything other than a boolean values as return of python
     actions

   - sub-cmd clean (#421450)

   - remove support for task generators returning action (not documented
     behavior)

   - setup parameter for a task should be a list - single value
     deprecated (#437225)

   - PythonAction support ‘dependencies’, ‘targets’, ‘changed’
     parameters

   - added tools.create_folder (#421453)

   - deprecate folder-dependency

   - CmdActions reference to dependencies, targets and changed
     dependencies (#434327)

   - print task description when printing through doit list (#425811)

   - action as list of commands/python (#421445)

   - deprecate “action” use “actions”

8.8 0.3.0 (`2009-08-30')
========================

   - added subcommand “forget” to clear successful runs status (#370911)

   - save run results in text file using JSON. (removed dbm)

   - added support for DEFAULT_TASKS in dodo file

   - targets md5 is not checked anymore.  if target exist, task is
     up-to-date.  it also supports folders

   - cmd line sub-commands (#370909)

   - remove hashlib dependency on python 2.4

   - sub-cmd to create dodo template

   - cmd-task supports a list of shell commands

   - setup/cleanup for task (#370905)

8.9 0.2.0 (`2009-04-16')
========================

   - docs generated using sphinx

   - execute once (dependency = True)

   - group task

   - support python 2.4 and 2.6

   - folder dependency

8.10 0.1.0 (`2008-04-14')
=========================

   - initial release

9 Developer’s docs
******************

This document presents a general overview of how ‘doit’ internals works.

API docs(1).

   ---------- Footnotes ----------

   (1) api/index.html

9.1 Execution flow - command ‘run’
==================================

‘doit’ is invoked using the script ‘bin/doit’.  It uses ‘cmdparse’ to
parse the command line arguments.

  1. the dodo module is loaded (get_module(1))

  2. collect all task generator functions.  (load_task_generators(2))

  3. task generator functions are executed and the information of
     generated tasks are saved

  4. the “raw” task information is processed (TaskSetup(3)).  tasks are
     organized (sorted) in an order to fulfill their dependencies.

  5. filter selected tasks

  6. run selected tasks run_tasks(4)

   ---------- Footnotes ----------

   (1) api/doit.main-module.html#get_module

   (2) api/doit.main-module.html#load_task_generators

   (3) api/doit.main.TaskSetup-class.html

   (4) api/doit.runner-module.html#run_tasks

9.1.1 Load “task generators”
----------------------------

First it imports the dodo file.  Than it introspects the module
collecting all functions named with a starting ‘task_’.  These functions
are “task generators”, they are not tasks on its own but they build
tasks.

The task generators are ordered by the line where they were defined.  By
default tasks are executed in this order.

9.1.2 Get tasks from task generators
------------------------------------

Generators return python dictionaries to represent tasks.  The task name
is given by function name of the task generator less the initial string
‘task_’.  For example the task name for “task_doXYZ” would be “doXYZ”.

There are two kinds of tasks generators.  Task generators that define a
single a task, or define multiple sub-tasks.  To define multiple
sub-tasks instead of returning a single dictionary it must return a
generator using python-generator(1)

Sub-tasks must include the field ‘name’ on its dictionary
representation.  A sub-task name will be formed as
“<task_name>:<subTask_name>”.

   ---------- Footnotes ----------

   (1)
http://docs.python.org/tut/node11.html#SECTION00111000000000000000000

9.1.3 Set task’s task-dependencies
----------------------------------

Collect all task-dependencies for every task.  task-dependencies can
come from:

  1. A task generator that return multiple sub-tasks creates a dumb task
     with no action that depends on all sub-tasks.

  2. A task-dependency explicit defined in the ‘task_dep’ as
     “<depends-on-task_name>”.

  3. result_dep are also implicit task_dep.

  4. A file-dependency where the file is also a target from another
     task.  If task-B depends on file-TA and file-TA is a target of
     task-A, task-B depends on task-A. Or task-A is a task-dependency of
     task-B.

9.1.4 Filter selected tasks
---------------------------

From the command line the user can choose which tasks to execute (by
default all tasks are executed).  The dodo file can also define a list
DEFAULT_TASKS. In this case the tasks are executed in the order they
were list in the command line (not in the order they were defined).

Just go over all the defined tasks and select the ones present in the
list filter.  If no list filter is defined all tasks are selected.

9.1.5 Order tasks
-----------------

The selected tasks are sorted in the order they will be executed.  If
task has any task-dependency they are added before the task being
processed.  Tasks are never added twice.  Task can not have cyclic
dependency, it is detected and an error is reported.

9.1.6 Execute tasks
-------------------

The runner executes only tasks that are not up-to-date.  A db file
(using JSON format) is used to keep information on tasks and their
file-dependencies.  there is a dictionary for every task.  each task has
a dictionary where key is dependency (abs file path), and the value is
the dependency’s modified timestamp, size and signature.

The following rules apply to dependencies:

   - if a task does not define any file-dependency or True (run-once) it
     is always executed.

   - if none of the file-dependencies is not modified (changed its
     signature) the task action is not executed

   - if the task is completed successfully all dependency records are
     updated.

If a target does not exist the task is executed.

If a task is a “run-once” (has a dependency True).  It will not be
executed after the first successful run.  As long as its execution is
not forced by a target.

In case any task fails the whole execution is aborted.

10 Related Projects
*******************

These are the main build tools in use today.

        - make(1)

        - ant(2)

        - SCons(3)

        - Rake(4)

There are many(5) more…

In this post(6) I briefly explained my motivation to start another build
tool like project.

   ---------- Footnotes ----------

   (1) http://www.gnu.org/software/make/

   (2) http://ant.apache.org/

   (3) http://www.scons.org/

   (4) http://rake.rubyforge.org/

   (5) http://en.wikipedia.org/wiki/List_of_build_automation_software

   (6)
http://schettino72.wordpress.com/2008/04/14/doit-a-build-tool-tale/