Corpus-driven generation of weather forecasts

Abstract

In traditional natural language generation (NLG), carefully analysing a corpus of example texts and determining the single correct sublanguage behind it is seen as one of the main tasks of the NLG system builder. In practice, this often means elimination of variation in the corpus and specification of conditions for rule application to the point where an NLG system becomes (virtually) deterministic. This approach is time-consuming, does not apply objective criteria for deciding what is correct, and contributes to the lack of robustness and reusable components in NLG. Moreover, with variation regarded as a ‘bug’ to be eliminated, systems run the risk of implementing a subjective or restrictive view of the domain sublanguage. This research note argues that relative frequency provides an objective, easily applicable tool for dealing with corpus variation. The probabilistic approach to NLG can also help cut down on manual corpus analysis, make systems more robust and components more reusable. A methodology is described that combines use of a base generator with a separate, automatically adaptable, probabilistic decision-making component. Three different decision-making techniques are compared and evaluated with a focus on their ability to model idiolectal variation. 1