Tweet
Brain imaging reveals the
movies in our mind
Imagine tapping into the mind of a coma patient, or
watching one's own dream on YouTube. With a cutting-edge
blend of brain imaging and computer simulation, scientists
at the University of California, Berkeley, are bringing these
futuristic scenarios within reach.
This set of paired images provided by Shinji Nishimoto of the University of California,
Berkeley on Wednesday, Sept. 21, 2011 shows original video images, upper row, and
those images reconstructed by computer from brain scans. While volunteers watched
movie clips, a scanner watched their brains. And from their brain activity, a computer
made rough reconstructions of what they viewed. Scientists reported that result Thursday,
Sept. 22, 2011 and speculated such an approach might be able to reveal dreams and
hallucinations someday. In the future, it might help stroke victims or others who have no
other way to communicate, said Jack Gallant, a neuroscientist at the University of
California, Berkeley, and co-author of the paper.
Using functional Magnetic Resonance Imaging (fMRI) and computational
models, UC Berkeley researchers have succeeded in decoding and
reconstructing people's dynamic visual experiences – in this case, watching
Hollywood movie trailers.
As yet, the technology can only reconstruct movie clips people have
already viewed. However, the breakthrough paves the way for reproducing
the movies inside our heads that no one else sees, such as dreams and
memories, according to researchers.
"This is a major leap toward reconstructing internal imagery," said Professor
Jack Gallant, a UC Berkeley neuroscientist and coauthor of the study to be
published online Sept. 22 in the journal Current Biology. "We are opening a
window into the movies in our minds."
Eventually, practical applications of the technology could include a better
understanding of what goes on in the minds of people who cannot
communicate verbally, such as stroke victims, coma patients and people
with neurodegenerative diseases.
It may also lay the groundwork for brain-machine interface so that people
with cerebral palsy or paralysis, for example, can guide computers with
their minds.
However, researchers point out that the technology is decades from
allowing users to read others' thoughts and intentions, as portrayed in
such sci-fi classics as "Brainstorm," in which scientists recorded a person's
sensations so that others could experience them.
Previously, Gallant and fellow researchers recorded brain activity in the
visual cortex while a subject viewed black-and-white photographs. They
then built a computational model that enabled them to predict with
overwhelming accuracy which picture the subject was looking at.
In their latest experiment, researchers say they have solved a much more
difficult problem by actually decoding brain signals generated by moving
pictures.
"Our natural visual experience is like watching a movie," said Shinji
Nishimoto, lead author of the study and a post-doctoral researcher in
Gallant's lab. "In order for this technology to have wide applicability, we
must understand how the brain processes these dynamic visual
experiences."
Nishimoto and two other research team members served as subjects for
the experiment, because the procedure requires volunteers to remain still
inside the MRI scanner for hours at a time.
They watched two separate sets of Hollywood movie trailers, while fMRI
was used to measure blood flow through the visual cortex, the part of the
brain that processes visual information. On the computer, the brain was
divided into small, three-dimensional cubes known as volumetric pixels, or
"voxels."
"We built a model for each voxel that describes how shape and motion
information in the movie is mapped into brain activity," Nishimoto said.
The brain activity recorded while subjects viewed the first set of clips was
fed into a computer program that learned, second by second, to associate
visual patterns in the movie with the corresponding brain activity.
Brain activity evoked by the second set of clips was used to test the
movie reconstruction algorithm. This was done by feeding 18 million
seconds of random YouTube videos into the computer program so that it
could predict the brain activity that each film clip would most likely evoke
in each subject.
Finally, the 100 clips that the computer program decided were most similar
to the clip that the subject had probably seen were merged to produce a
blurry yet continuous reconstruction of the original movie.
Reconstructing movies using brain scans has been challenging because the
blood flow signals measured using fMRI change much more slowly than the
neural signals that encode dynamic information in movies, researchers said.
For this reason, most previous attempts to decode brain activity have
focused on static images.
"We addressed this problem by developing a two-stage model that
separately describes the underlying neural population and blood flow
signals," Nishimoto said.
Ultimately, Nishimoto said, scientists need to understand how the brain
processes dynamic visual events that we experience in everyday life.
"We need to know how the brain works in naturalistic conditions," he said.
"For that, we need to first understand how the brain works while we are
watching movies."
Brain imaging reveals the movies in our mind
--------
Very interesting.
Minority Report + Inception?
ha.
movies in our mind
Imagine tapping into the mind of a coma patient, or
watching one's own dream on YouTube. With a cutting-edge
blend of brain imaging and computer simulation, scientists
at the University of California, Berkeley, are bringing these
futuristic scenarios within reach.
This set of paired images provided by Shinji Nishimoto of the University of California,
Berkeley on Wednesday, Sept. 21, 2011 shows original video images, upper row, and
those images reconstructed by computer from brain scans. While volunteers watched
movie clips, a scanner watched their brains. And from their brain activity, a computer
made rough reconstructions of what they viewed. Scientists reported that result Thursday,
Sept. 22, 2011 and speculated such an approach might be able to reveal dreams and
hallucinations someday. In the future, it might help stroke victims or others who have no
other way to communicate, said Jack Gallant, a neuroscientist at the University of
California, Berkeley, and co-author of the paper.
Using functional Magnetic Resonance Imaging (fMRI) and computational
models, UC Berkeley researchers have succeeded in decoding and
reconstructing people's dynamic visual experiences – in this case, watching
Hollywood movie trailers.
As yet, the technology can only reconstruct movie clips people have
already viewed. However, the breakthrough paves the way for reproducing
the movies inside our heads that no one else sees, such as dreams and
memories, according to researchers.
"This is a major leap toward reconstructing internal imagery," said Professor
Jack Gallant, a UC Berkeley neuroscientist and coauthor of the study to be
published online Sept. 22 in the journal Current Biology. "We are opening a
window into the movies in our minds."
Eventually, practical applications of the technology could include a better
understanding of what goes on in the minds of people who cannot
communicate verbally, such as stroke victims, coma patients and people
with neurodegenerative diseases.
It may also lay the groundwork for brain-machine interface so that people
with cerebral palsy or paralysis, for example, can guide computers with
their minds.
However, researchers point out that the technology is decades from
allowing users to read others' thoughts and intentions, as portrayed in
such sci-fi classics as "Brainstorm," in which scientists recorded a person's
sensations so that others could experience them.
Previously, Gallant and fellow researchers recorded brain activity in the
visual cortex while a subject viewed black-and-white photographs. They
then built a computational model that enabled them to predict with
overwhelming accuracy which picture the subject was looking at.
In their latest experiment, researchers say they have solved a much more
difficult problem by actually decoding brain signals generated by moving
pictures.
"Our natural visual experience is like watching a movie," said Shinji
Nishimoto, lead author of the study and a post-doctoral researcher in
Gallant's lab. "In order for this technology to have wide applicability, we
must understand how the brain processes these dynamic visual
experiences."
Nishimoto and two other research team members served as subjects for
the experiment, because the procedure requires volunteers to remain still
inside the MRI scanner for hours at a time.
They watched two separate sets of Hollywood movie trailers, while fMRI
was used to measure blood flow through the visual cortex, the part of the
brain that processes visual information. On the computer, the brain was
divided into small, three-dimensional cubes known as volumetric pixels, or
"voxels."
"We built a model for each voxel that describes how shape and motion
information in the movie is mapped into brain activity," Nishimoto said.
The brain activity recorded while subjects viewed the first set of clips was
fed into a computer program that learned, second by second, to associate
visual patterns in the movie with the corresponding brain activity.
Brain activity evoked by the second set of clips was used to test the
movie reconstruction algorithm. This was done by feeding 18 million
seconds of random YouTube videos into the computer program so that it
could predict the brain activity that each film clip would most likely evoke
in each subject.
Finally, the 100 clips that the computer program decided were most similar
to the clip that the subject had probably seen were merged to produce a
blurry yet continuous reconstruction of the original movie.
Reconstructing movies using brain scans has been challenging because the
blood flow signals measured using fMRI change much more slowly than the
neural signals that encode dynamic information in movies, researchers said.
For this reason, most previous attempts to decode brain activity have
focused on static images.
"We addressed this problem by developing a two-stage model that
separately describes the underlying neural population and blood flow
signals," Nishimoto said.
Ultimately, Nishimoto said, scientists need to understand how the brain
processes dynamic visual events that we experience in everyday life.
"We need to know how the brain works in naturalistic conditions," he said.
"For that, we need to first understand how the brain works while we are
watching movies."
Brain imaging reveals the movies in our mind
--------
Very interesting.
Minority Report + Inception?
ha.