3. Procedures and Algorithms

Procedures and Algorithms

A grammar for a language specifies conditions or rules that strings of word forms must satisfy to be sentences in the language. A grammar may be said to license those strings that meet its rules. A grammar might also license derivation trees for the sentences it licenses. But grammars generally do not include instructions for how the human language processor, or a computer program, might apply the rules to produce or analyse sentences. For example, the rules of a phrase-structure grammar are often called "rewriting rules." This suggests that their application involves some kind of symbol replacement operation. A phrase-structure grammar, however, includes no information about how this operation is performed in the many different strategies or procedures that might be followed to analyse sentences. Consequently, in order to write computer programs that determine whether or not a given string of word forms is a sentence, or that model the cognitive processes of sentence analysis, one must devise procedures that specify how the rules of a grammar can be applied.

This note includes the following topics:

• Procedures
  • Equivalent Procedures
  • Decision Procedures
  • Equivalence of Procedures and Grammars
  • Turing's Thesis
• Algorithms
• Examples

Procedures

A procedure is a recipe or method for accomplishing a result such as solving a problem or performing a task. A procedure is usually considered to have the following characteristics:

• It consists of a finite sequence of discrete steps.
• Each step is an operation or instruction that can be performed by the agent or device expected to carry out the procedure.
• Each operation or instruction is finite in that it is represented by a finite number of symbols.
• The device or agent can carry out the procedure automatically, without additional instructions or resort to outside guidance, by simply performing the sequence of operations comprising the procedure.

Thus, a procedure is a step by step method for performing a task or solving a problem, presented with sufficient precision and detail, and in an appropriate form and language, that it is completely and unambiguously interpretable and executable by the particular agent or device intended to perform the procedure. Furthermore, all possible circumstances that might arise in the course of performing the procedure are taken into account so that the agent or device can carry out the procedure automatically without requiring any additional instructions.

Procedures consist of a finite number of discrete steps each of which is comprised of a rule, or an instruction or operation that can be applied, performed or executed by the device or agent that is expected to carry out the procedure. Although the steps of a procedure are ordered in a sequence, and are performed in this order, some steps can consist of tests such as those required to determine whether or not a problem has been solved, or a task has been completed. Such tests can also determine what particular sequences of steps in the procedure are to be performed when certain conditions are met, rather than others which might be performed under different conditions.

Other steps in a procedure can consist of instructions that cause sequences of operations or instructions to be repeated or iterated a number of times until specified conditions are met. Consequently, although a procedure might consist of a relatively small number of steps, some of these steps can be repeated so that the device or agent performing the procedure may actually execute a relatively large number of operations to carry out some tasks or solve some problems.

Procedures may be devised that build or change things in the real world. These procedures might be performed by machines or by human beings. The procedures of interest here, however, are computational in nature. By "computational" in this context, it is meant that steps of a computational procedure that a machine or a human being performs can consist of operations such as the reading, writing, comparing, erasing, and replacing of symbols. The operations of a computational procedure could include storing symbols in, and retrieving them from memory; but these operations are simply special kinds of writing and reading operations. The symbols of interest to us include graphemes or word forms, and symbols such as syntactic category labels.

Procedures might also be used as models of the operation of physical systems. For example, some procedures are devised as computational models of the sequences of operations that the cognitive system is conjectured to perform as it processes sentences. The operations comprising the steps of such procedures are determined by the psychological evidence that might be available, and by the psycholinguistic theory of sentence processing that is applied in developing the model.

Equivalent Procedures. Several different procedures might be devised to solve the same problem or perform the same task. Thus, there is normally no unique procedure for a given problem or task; however, since they solve the same problem or perform the same task, the different procedures can be described as equivalent. That is,

Two procedures are equivalent just in case they accomplish the same result (that is, they perform the same task(s) or solve the same problem(s)).

The selection of a particular, preferred procedure from among several equivalent procedures is often based upon considerations of relative efficiency. A procedure might be said to be more efficient than another if fewer steps must be performed, and hence less time is required to solve the problem or perform the task.

Decision Procedures. A typical problem to which a procedure is applied in computational linguistics is that of determining whether or not a given string a over a vocabulary V_T is a sentence in the language L(V_T). A procedure of this nature will report "yes" if a Î L(V_T), but will report "no" if a Ï L(V_T); that is, the procedure will display "no" if a Î L^c(V_T), where L^c(V_T) = V*_T - L(V_T), the complement of the language L(V_T) relative to the closure of the vocabulary V_T.

A procedure of this kind is called a decision procedure; that is, if a problem requires only a "yes" or "no" answer, then a procedure designed to solve the problem is called a decision procedure. A decision procedure for a language L(V_T) can be called a recogniser or acceptor for L(V_T).

For any string a Î V*_T, an acceptor A for the language L(V_T) reports "yes," and is said to accept a, just in case a Î L(V_T); otherwise, A reports "no."

Equivalence of Procedures and Grammars. If there is a grammar G that licenses all, and only, the sentences of L(V_T), we can write L(G) = L(V_T), where L(G) denotes the language licensed by G. Then an acceptor for L(V_T) is an acceptor for L(G). An acceptor A for a language L(G) is said to be equivalent to G. In other words, we can say that

A procedure A and a grammar G are equivalent, and A is an acceptor for L(G), just in case A accepts all, and only, the sentences in L(G).

An acceptor A for the language L(G) licensed by a grammar G can be called an acceptor for G. Although the A may be equivalent to G, A might not perform operations that correspond to the rules of G.

Some grammars license syntactic structures or derivation trees for the sentences that they license. These trees are made up of subtrees each of which consists of a mother node and its daughter nodes. Each of these subtrees corresponds to a rule of the grammar.

Consider that G is a grammar that licenses derivation trees for the sentences it licenses, and that A is an acceptor for G. Then, assume that, for each sentence licensed by G, A produces the derivation tree (or trees) licensed by G. The procedure A can be called a parser for G.

Although the operations comprising the procedure A may not be identical to the rules of grammar G, the operations of A at least have the same effect as the rules of G. We can therefore claim that the operations of A correspond to the rules of G, and that A and G are strongly equivalent. In other words, we can say that

A procedure A and a grammar G are strongly equivalent, and A is a parser for G, just in case A produces all, and only, the derivation trees licensed by G.

In some psycholinguistic models, the grammar for a language is assumed to provide evidence for the operations performed by the human language processor as it produces or analyses sentences. This assumption has been formulated by Bresnan and Kaplan as the Strong Competence Hypothesis:

There is a direct correspondence between the rules of a grammar and the operations performed by the human language processor.

Underlying this hypothesis is the further assumption that the language processing components of the cognitive system must perform a procedure to produce and analyse sentences. The Strong Competence Hypothesis is thus a statement of the strong equivalence of this neural procedure and a grammar.

Turing's Thesis. It was stated in the introductory paragraph of this note that grammars do not necessarily include instructions that determine how their rules are applied to analyse sentences. Thus, in computational and psycholinguistic models of sentence analysis, it is often the case that procedures are devised that specify how the rules of a chosen grammar are applied. Different procedures might be developed on the basis of different psychological or computational theories. But it is generally assumed that these procedures can be implemented by writing computer programs that perform the operations of the procedure.

This assumption is well-founded in that no computational procedure has been found that cannot be implemented as a program that runs on a computer with sufficient resources. It is nonetheless an assumption: it has never been proven that a computer with sufficient resources can perform any given computational procedure. This assumption is essentially Turing's Thesis, which may be paraphrased as follows:

A Turing Machine can perform any computational procedure.

Turing never proved his thesis; but no counterexample has yet been identified.

One can, in principle, write a program to be executed by a general-purpose digital computer with sufficient resources that emulates the operation of a Turing Machine. The "sufficient resources" here refers to the availability of a memory device that can emulate the infinite tape of a Turing Machine. It turns out in practice that, for the procedures we normally devise, the infinite memory of a Turing Machine is not required. Thus, by making an assumption comparable to Turing's thesis, we can claim that it is possible to write a computer program that is equivalent to a given computational procedure, where

A computer program C is equivalent to a computational procedure A just in case C achieves the same result as A (that is, C solves the same problem(s) or performs the task(s) as A).

If the operations comprising the procedure A can be emulated by a computer program C, then we can say that C is strongly equivalent to A. In other words

A computer program C and a computational procedure A are strongly equivalent just in case C is equivalent to A and C emulates the operations of A.

If a computer program C is equivalent to a procedure A, and if A is an acceptor for a grammar G, then C is an acceptor for G. We can therefore state that

A computer program C and a grammar G are equivalent just in case C accepts all, and only, the sentences in L(G).

Furthermore, if a computer program C is strongly equivalent to a procedure A, and if A is an parser for a grammar G, then C is a parser for G.

Hence, we can say that A computer program C and a grammar G are strongly equivalent just in case C produces all, and only, the derivation trees licensed by G.

Algorithms

A procedure, by definition, can consist of only a finite number of steps; but, as a consequence of iteration, some sequences of steps may be performed infinitely many times, depending upon the particular problem or task, or perhaps upon procedure itself. Thus, although it might in fact be the case that a string a is a sentence in the language L(V_T), it could require infinitely many steps of the procedure to confirm that a Î L(V_T). Furthermore, if a Ï L(V_T), it might take an infinite number of steps to establish that a Î L^c(V_T). Consequently, because infinitely many steps must be performed to solve some problems, a procedure might not yield a solution, and stop, until infinite time has elapsed.

An algorithm is a procedure for solving a problem or performing a task that will always stop within a finite time. Thus, if the problem has a solution, or the task has an attainable goal, an algorithm will yield the solution or achieve the goal in a finite number of steps, and hence, in finite time. Furthermore, an algorithm will also stop after finitely many steps have been executed if the the problem actually has no solution or the goal of the task is unattainable. Algorithms are therefore sometimes called effective procedures.

A decision procedure that requires at most finitely many steps to yield a "yes" or "no" result is called a decision algorithm. For example, a decision algorithm will require a finite amount of time to determine whether or not an arbitrary string a over a vocabulary V_T is a sentence in the language L(V_T).

If a decision algorithm can be devised for the problem of determining whether or not a Î L(V_T), then this problem is said to be algorithmically decidable. The language L(V_T) itself is said to be recursively decidable or recursive.

It is possible that a decision procedure will report "yes" after at most finitely many steps when a string a Î L(V_T); but if a Ï L(V_T), the procedure may not stop in a finite number of steps. Infinitely many steps might be required in such cases to establish that a Î L^c(V_T). Procedures of this nature are called decision semi-algorithms. A language L(V_T) for which there is at best only a semi-algorithm to determine whether or not a string a Î L(V_T) is not recursively decidable.

It is inherent in the nature of some problems and tasks that, although they may have solutions or attainable goals, any procedure could require infinitely many steps to yield a solution or goal. No algorithm, or semi-algorithm, can be devised to solve such problems. Although it is possible that some particular solutions might be discovered within a finite number of steps, such problems are considered to be intractable in general.

Problems for which algorithms can be devised to solve them are considered to be tractable in principle. The caveat "in principle" is attached because, although algorithms yield solutions in finitely many steps, the actual number of steps performed, and consequently the time required, can be quite large. In such cases, a problem might be intractable in practice. For example, the time needed to determine whether or not a string a Î L(V_T) could increase exponentially with the length of a so that, notwithstanding L(V_T) is recursively decidable, the time required could be extremely large for longer strings.

For further information, see Procedures: Examples and Discussion.