English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We report here initial experiments with a method for generating text from functional brain images. The technique is based on recent work in which patterns of brain activation during picture-viewing were used to generate detailed images closely resembling the original stimuli. We apply the same general approach to generate descriptive text from fMRI data collected during single-word reading. In adapting the technique to this purpose, we make use of recent Bayesian methods for text representation, comparing results against benchmark text from the online encyclopedia Wikipedia. Introduction Over the last decade, functional magnetic resonance imaging (fMRI) has become a primary tool for identifying the neural correlates of mental activity. Traditionally, the aim of fMRI experiments has been to identify discrete, coherent neuroanatomic regions engaged during specific forms of information processing. More recently, it has become clear that important information can be extracted from fMRI by attending instead to broadly distributed patterns of activation. The application of machine learning techniques for pattern classification [20] has enabled impressive feats of “brain reading,” making it possible to infer the class of object viewed by an experimental participant, to track the process of memory retrieval, to predict decisions or mistakes, or even (controversially) to detect lies [17] [13] [8] [5]. The key step in “brain-reading” applications of fMRI involves classifying brain images into a set of discrete categories. For example, given a brain image collected during single-word reading, the task might be to decide which among a set of candidate words triggered the image [13]. This approach, which continues to be highly fruitful, has recently benefitted from the application of sophisticated models that allow prediction of brain activation patterns induced by stimuli from outside the initial training set [14] [9]. In a dramatic departure from the standard approach, a small set of recent studies has demonstrated the feasibility of a generative approach to fMRI decoding. Beginning with fMRI data collected as participants viewed complex images, Naselaris [16] and colleagues constructed entire images that strikingly resembled the original stimuli (see also [15] and [21]). The crucial ingredient in this approach is a generative model, which captures the way in which specific aspects of the stimulus give rise to particular sub-patterns of distributed brain activity. Once established, this generative model can then be inverted, in order to synthesize a complex artifact (e.g., a picture) from a single pattern of brain activity, as diagrammed in Figure 1. To date, the generative approach to fMRI decoding has been applied only in the visual/pictorial domain, as just described. In the present work, we extend it to the generation of written text. The long-range aspiration is to begin with a brain image encoding some mental content, and to generate from it a verbal description of that content. In the present work, we focused on a simplified version of this challenge: We began with brain images collected during viewing of single words naming concrete concepts (e.g., house or dog, together with a line drawing of the item named), and from these attempted to generate text describing the relevant concept, in the spirit of an encyclopedia entry. The online encyclopedia Wikipedia served as a gold-standard reference, against which our text-generation results could be compared. As a further simplification of the problem, we followed a step common in machine learning work on text representation (see, for instance, [12]) by ignoring syntax and word order, treating texts as simple collections of words.

Generating descriptive text from functional brain images