Fenestration

fenestration is a python library for generating foveated pooling windows. These pooling windows tile the input image space such that windows are small at the center and grow in size towards the outer edge. This effect is meant to sample the input in a similar fashion to the human visual system, in which information is sampled more finely near the fovea and more coarsely in the periphery. These windows can be used in a variety of contexts, particularly for generating more biologically-inspired image metamers and processing images at multiple scales, like those created by the steerable pyramid implementation.

The original implementation of these pooling windows was published in Freeman, J., & Simoncelli, E. P. (2011). Metamers of the ventral stream. Nature Neuroscience. Matlab code for these windows exist as part of Jeremy Freeman’s repo for this paper. This repo is not a direct port of that code, but a conceptual reimplementation, following the math outlined in the supplemental materials, and also includes a version using Gaussian windows. More recently, these windows were also used to generate stimuli in Broderick, W. F., Rufo, G., Winawer, J. & Simoncelli, E. P. (2023). Foveated metamers of the early visual system. eLife.

For information on how to use the package, view the Tutorials and API pages linked at the top!

Installation

To use fenestration yourself, you can install it directly from GitHub:

pip install git+https://github.com/plenoptic-org/fenestration.git

Alternatively, if you want use a local copy, you can clone the GitHub repository with the lines below, using SSH:

git clone git@github.com:plenoptic-org/fenestration.git

or HTTPS:

git clone https://github.com/plenoptic-org/fenestration.git

If you cloned the repository, navigate to the directory and run the following to create a virtual environment and download all dependencies:

cd fenestration
# create virtual environment
python -m venv .venv
# activate environment
.venv/Scripts/activate
# install all dependencies
pip install -e .

This code works with the python versions currently supported by PyTorch.

Related packages

• plenoptic: This package is largely built to be compatible with plenoptic, particularly for metamer generation and for steerable pyramids. We recommend you check out plenoptic for many other stimulus processing tools!

• pytorch: Optimized tensor library for deep learning using GPUs and CPUs. We require stimulus inputs to fenestration to be pytorch tensors in order to support GPU and CPU processing.

Support

This package is supported by the Simons Foundation Flatiron Institute’s Center for Computational Neuroscience.