Jabberwocky Ecology

Data science competition for converting remote sensing to ecological data

Scaling-up ecological patterns and processes is crucial to understanding the effects of environmental change on natural systems and human society. We are piloting a Data Science Challenge where multiple groups attempt to use the same remote sensing data from low flying airplanes to infer the location and type of trees in forests. This will allow forests to be studied in detail at much larger scales than is currently possible. This kind of collaborative data analysis challenge has proven highly effective in other fields for quickly improving methods for converting image data to useful information.

There are three sets of tasks: 1) identifying individual trees in remote sensing images; 2) aligning ground data with remote sensing data; and 3) classifying trees into species.

Screen Shot 2017-11-06 at 10.33.22 AM

Teams (or individuals) can participate in all of them or just pick the tasks they are most interested in. Tasks 2 and 3 can be accomplished using just tabular data. Task 1 requires working directly with spatial data. Details of the different tasks and links to the data are available at the challenge website:


We plan to write a general paper about the competition, the data, and the performance of the different methods used. Individual participants will be invited to write and publish associated short papers on the methods they used and results they produced. We already have a journal that has agreed to publishing all of these related contributions together into a collection (pending review of course).

The challenge is already open and the deadline for submissions is December 15th. Once you sign up on the website you will receive an email with some additional details. If you have any questions feel free to respond to that email or checkout the FAQ to see if they have already been answered.

This challenge is sponsored by the National Institute of Standards Technology as part of it’s Data Science Evaluation series and is also partially supported by he Gordon and Betty Moore Foundation’s Data-Driven Discovery Initiative through grant GBMF4563.It uses data from the National Ecological Observatory Network in addition to data collected by the organizers. It is being organized by the Data Science Research lab, the Weecology lab, and Stephanie Bohlman’s lab all at the University of Florida.

Data Retriever 2.1: Python Interface, Autocomplete & More

EcoData Retriever logo

We are exited to announce a new release of the Data Retriever, our software for making it quick and easy to get clean, ready to analyze, data.

The Data Retriever, automates the downloading, cleaning, and installing of data into your choice of databases and flat file formats. Instead of hours tracking down the data on the web, downloading it, trying to import it, running into issues, fixing one problem, and then encountering the next, all you need to do is run a single command from the command line, R, or (now!!) Python:

$ retriever install csv iris
> portal_data <- rdataretriever::fetch('portal')
In [1]: import retriever as rt
In [2]: rt.install_postgres('breed-bird-survey')

Major changes

  • Python interface: While the retriever is written in Python the package previously only had a command line interface. Now you can access the full power of the retriever from directly inside Python. See the full tutorial for more details.
  • Conda packaging: The conda package manager has become one of the two main ways to install Python packages. You can now install the retriever using
conda install retriever -c conda-forge
  • Command line autocomplete: As the number of datasets and backends supported by the retriever goes it can be difficult to remember specific names. Using Tab will now autocomplete retriever commands, backends, and dataset names. (Currently only available of OSX and Linux)


  • We also made some changes to the metadata script system so if you’ve previously installed the retriever you should update your scripts using:
retriever reset scripts
retriever update

Find out more

To find out more about the Data Retriever checkout the:


Ongoing work on the Data Retriever lead by Henry Senyondo is made possible by the generous support of the Gordon and Betty Moore Foundation’s Data Driven Discovery Initiative. This kind of active support for the development and maintenance of research oriented software makes sustainable software development at universities possible. Shivam Negi developed the Python interface as part of his Google Summer of Code project. You can read more about his time in GSOC at his blog.

Twelve different folks contributed code to this release. A big thanks to Henry Senyondo, Ethan White, Shivam Negi, Andrew Zhang, Kapil Kumar, Kunal Pal, Amritanshu Jain, Kevin Amipara, David LeBauer, Amritanshu Jain, Goel Akash, and
Parth-25m for making the retriever better.

The Portal Weather Station

How has our collection of weather data at the Portal Project changed through time? Weecology Project Manager Glenda Yenni recounts the saga of the weather stations.

The Portal Project

For the history of the project, weather monitoring has always accompanied the collection of rodent, plant, and ant data. At first, this was done manually. Portalites from 1980 to 1989 measured rain in a rain gauge, and used something called a hygrothermograph to measure temperature and humidity.

hygrothermograph Hygrothermograph

Then things started to get fancy. In 1989, an automated weather station was installed. This is the desert though, and leaving expensive toys out in the rain, dust and lightning takes it’s toll.

Sunset in the desert jungle At least the lightning storms leave us with some nice scenery after they try to blow up our weather station.

All things considered, our weather stations have stood up pretty well. The first lasted from 1989 until 2002. And the station from 2002 is still limping along, although it’s had its moments (it tends to have a bit of a tantrum after being struck by lightning). We connected to…

View original post 385 more words

A Vegetation History

Today’s reblog from the Portal Project is a piece by Tom Valone how the site inspired him to ponder vegetation change.

The Portal Project

My Portal story begins in 1991 when Jim Brown offered me the opportunity to be the Portal postdoc. Among other things, this meant I organized the yearly ant census in which we spent about 2 weeks counting the abundance of ant colonies on the experimental plots. In one of my first summers, I hired Don Sias to help with the census. Don was a non-traditional student and had previously traveled extensively throughout the southwest. One day he mentioned to me that the vegetation in the San Simon valley looked pretty “beat up”. By that he meant the vegetation, dominated by shrubs, had the look of a grassland that had become desertified.

August 2015

I was intrigued by Don’s comment for two reasons. First, Brown and Heske (1990) had recently described a significant increase in grass cover on Portal plots that removed kangaroo rats and mentioned that the site was “near the zone…

View original post 357 more words

Portal: Then and Now

Love to see paired photos of places in the past and now? My PhD student Erica Christensen has a treat for you. Here’s a post about her attempts to reshoot old photos from the site.

The Portal Project

A lot can change in 40 years.  This is perhaps never more apparent than when you find a box of old photographs, and start comparing then to now.  When the Weecology lab immigrated from Utah to Florida in 2015, just such a box surfaced: a glimpse back in time to the beginning of the portal project.  I did my best to re-create some of these photographs—trying to line up horizons and mountains—to show how the site has changed over 40 years.

One thing is immediately apparent: the shrubs have grown up.  The left side of these photographs were taken in 1977 (photographer unknown), and the right side in 2015 (photographer Erica Christensen).  What happened to the rows upon rows of aluminum flashing, indicating the location of the rodent fences?  I assure you the fences are still there, they’re just obscured by the jungle.

The fact that you used to be…

View original post 339 more words

2017 Summer Plant Census

Someone (**cough** **cough** Morgan) fell down on their job rebloging the Portal Project 40th anniversary posts to Jabberwocky. But that means this week is the week o’ Portal as we reblog the various posts from the past few weeks. First up, what happens when we go out in the summer to count plants?

The Portal Project

Twice a year the Portal crew gets a little larger, and spends a few extra days, and we count plants on all 384 quadrats. Despite some of us being in our second decade of visiting the site, and everyone on the plant crew being intimately familiar with most of the species at the site, and that the rodent RA has been watching the plants grow and giving us monthly updates, we still never really know what we’re going to find once we get out there. The desert does what it wants.

The uncertainty seems especially high for the summer plant community. Some years we arrive to an ocean of grass, waving in the breeze. Those are the years we spend a lot of ‘quality time’ with each quadrat. Other years we arrive to a dustbowl. We walk around the site laying our PVC quadrat down and picking it back up…

View original post 205 more words

Ode to six-legged wonder

While the Portal Project focuses primarily on rodents and plants, there is more to our ecosystem than just those taxa. This week’s Portal Project post focuses on some insect work at the site.

The Portal Project

If four legs are great, six legs are better. Right? For forty years now, the Portal Project has primarily focused on two-legged creatures trapping, studying, and sometimes cuddling small, furry four-legged creatures. But we haven’t ignored the six-legged inhabitants of our long-term research site, and I am going to tell you more about them now.

roachantmouse Which one of these is least like the other?

I am the Ernest lab entomologist, who doesn’t consider herself an entomologist. I am interested in biodiversity, community, and macroecological patterns like those studied by generations of Portal rodent researchers. I just happen to study them using bees. There are over twenty thousand species of bees in the world, and about four thousand in North America. My research so far has focused on the community ecology of native bees in a global hotspot of bee diversity in California. But as fate or luck would have it…

View original post 932 more words