Data Retriever 2.1: Python Interface, Autocomplete & More
We are exited to announce a new release of the Data Retriever, our software for making it quick and easy to get clean, ready to analyze, data.
The Data Retriever, automates the downloading, cleaning, and installing of data into your choice of databases and flat file formats. Instead of hours tracking down the data on the web, downloading it, trying to import it, running into issues, fixing one problem, and then encountering the next, all you need to do is run a single command from the command line, R, or (now!!) Python:
$ retriever install csv iris
> portal_data <- rdataretriever::fetch('portal')
In [1]: import retriever as rt In [2]: rt.install_postgres('breed-bird-survey')
Major changes
- Python interface: While the retriever is written in Python the package previously only had a command line interface. Now you can access the full power of the retriever from directly inside Python. See the full tutorial for more details.
- Conda packaging: The conda package manager has become one of the two main ways to install Python packages. You can now install the retriever using
conda install retriever -c conda-forge
- Command line autocomplete: As the number of datasets and backends supported by the retriever goes it can be difficult to remember specific names. Using Tab will now autocomplete retriever commands, backends, and dataset names. (Currently only available of OSX and Linux)
- We also made some changes to the metadata script system so if you’ve previously installed the retriever you should update your scripts using:
retriever reset scripts retriever update
Find out more
To find out more about the Data Retriever checkout the:
- Website
- Documentation
- GitHub repositories for the Data Retriever and the rdataretriever R package.
Acknowledgments
Ongoing work on the Data Retriever lead by Henry Senyondo is made possible by the generous support of the Gordon and Betty Moore Foundation’s Data Driven Discovery Initiative. This kind of active support for the development and maintenance of research oriented software makes sustainable software development at universities possible. Shivam Negi developed the Python interface as part of his Google Summer of Code project. You can read more about his time in GSOC at his blog.
Twelve different folks contributed code to this release. A big thanks to Henry Senyondo, Ethan White, Shivam Negi, Andrew Zhang, Kapil Kumar, Kunal Pal, Amritanshu Jain, Kevin Amipara, David LeBauer, Amritanshu Jain, Goel Akash, and
Parth-25m for making the retriever better.
The Portal Weather Station
How has our collection of weather data at the Portal Project changed through time? Weecology Project Manager Glenda Yenni recounts the saga of the weather stations.
For the history of the project, weather monitoring has always accompanied the collection of rodent, plant, and ant data. At first, this was done manually. Portalites from 1980 to 1989 measured rain in a rain gauge, and used something called a hygrothermograph to measure temperature and humidity.
Hygrothermograph
Then things started to get fancy. In 1989, an automated weather station was installed. This is the desert though, and leaving expensive toys out in the rain, dust and lightning takes it’s toll.
At least the lightning storms leave us with some nice scenery after they try to blow up our weather station.
All things considered, our weather stations have stood up pretty well. The first lasted from 1989 until 2002. And the station from 2002 is still limping along, although it’s had its moments (it tends to have a bit of a tantrum after being struck by lightning). We connected to…
View original post 385 more words
A Vegetation History
Today’s reblog from the Portal Project is a piece by Tom Valone how the site inspired him to ponder vegetation change.
My Portal story begins in 1991 when Jim Brown offered me the opportunity to be the Portal postdoc. Among other things, this meant I organized the yearly ant census in which we spent about 2 weeks counting the abundance of ant colonies on the experimental plots. In one of my first summers, I hired Don Sias to help with the census. Don was a non-traditional student and had previously traveled extensively throughout the southwest. One day he mentioned to me that the vegetation in the San Simon valley looked pretty “beat up”. By that he meant the vegetation, dominated by shrubs, had the look of a grassland that had become desertified.
I was intrigued by Don’s comment for two reasons. First, Brown and Heske (1990) had recently described a significant increase in grass cover on Portal plots that removed kangaroo rats and mentioned that the site was “near the zone…
View original post 357 more words
Portal: Then and Now
Love to see paired photos of places in the past and now? My PhD student Erica Christensen has a treat for you. Here’s a post about her attempts to reshoot old photos from the site.
A lot can change in 40 years. This is perhaps never more apparent than when you find a box of old photographs, and start comparing then to now. When the Weecology lab immigrated from Utah to Florida in 2015, just such a box surfaced: a glimpse back in time to the beginning of the portal project. I did my best to re-create some of these photographs—trying to line up horizons and mountains—to show how the site has changed over 40 years.
One thing is immediately apparent: the shrubs have grown up. The left side of these photographs were taken in 1977 (photographer unknown), and the right side in 2015 (photographer Erica Christensen). What happened to the rows upon rows of aluminum flashing, indicating the location of the rodent fences? I assure you the fences are still there, they’re just obscured by the jungle.
The fact that you used to be…
View original post 339 more words
2017 Summer Plant Census
Someone (**cough** **cough** Morgan) fell down on their job rebloging the Portal Project 40th anniversary posts to Jabberwocky. But that means this week is the week o’ Portal as we reblog the various posts from the past few weeks. First up, what happens when we go out in the summer to count plants?
Twice a year the Portal crew gets a little larger, and spends a few extra days, and we count plants on all 384 quadrats. Despite some of us being in our second decade of visiting the site, and everyone on the plant crew being intimately familiar with most of the species at the site, and that the rodent RA has been watching the plants grow and giving us monthly updates, we still never really know what we’re going to find once we get out there. The desert does what it wants.
The uncertainty seems especially high for the summer plant community. Some years we arrive to an ocean of grass, waving in the breeze. Those are the years we spend a lot of ‘quality time’ with each quadrat. Other years we arrive to a dustbowl. We walk around the site laying our PVC quadrat down and picking it back up…
View original post 205 more words
Ode to six-legged wonder
While the Portal Project focuses primarily on rodents and plants, there is more to our ecosystem than just those taxa. This week’s Portal Project post focuses on some insect work at the site.
If four legs are great, six legs are better. Right? For forty years now, the Portal Project has primarily focused on two-legged creatures trapping, studying, and sometimes cuddling small, furry four-legged creatures. But we haven’t ignored the six-legged inhabitants of our long-term research site, and I am going to tell you more about them now.
Which one of these is least like the other?
I am the Ernest lab entomologist, who doesn’t consider herself an entomologist. I am interested in biodiversity, community, and macroecological patterns like those studied by generations of Portal rodent researchers. I just happen to study them using bees. There are over twenty thousand species of bees in the world, and about four thousand in North America. My research so far has focused on the community ecology of native bees in a global hotspot of bee diversity in California. But as fate or luck would have it…
View original post 932 more words
The Spectabulous Spectabs of Portal
The latest installment from the Portal Project Blog on the watch for Banner-tailed Kangaroo Rats
Much beloved by those who have worked at the Portal Project, the banner-tailed kangaroo rat (Dipodomys spectabilis) is one of the most charismatic rodents at the site (for us smammal lovers who think rodents can be charismatic, anyway). The fact that they have a nickname—spectabs—attests to this fondness. Look at that mighty tufted tail! Those giant, majestic furred feet! Weighing in at over 100 grams as adults, they are twice the size of our other kangaroo rat species (D. ordii and D. merriami). What’s not to love?
Dipodomys spectabilis
As avid readers of the Portal blog might recall, the site used to be much grassier back in the day. At the start of the project in 1977, spectabs were running the show at Portal; we even had some plots that excluded only D. spectabilis because they were so dominant! For the spectabs, this was a desert…
View original post 665 more words
Portal Phenocam
How did we get those daily pics of the desert turning green in a week? Meet Portal’s new toy: the Phenocam.
You may have noticed the super-cool daily images featured in last week’s post. They’re from our new network camera.
For starters, it allows us to do things like watch our desert field site turn from brown to green in no time flat (and back to brown again this winter).
But even cooler, our camera is part of the PhenoCam Network. They’re organizing a network of near-surface remote sensing images from sites all over the world. This creates a time series of images, in RGB and infrared, that can be used for phenology monitoring by the PhenoCam folks, us, or anyone who’s interested.
The PhenoCam folks make all the imagery freely available to download. From installation and configuration to image analysis, they provide awesome support. And their R package phenopix provides a quickstart to using phenocam imagery.
How fast can a desert turn green?
Ever wondered how fast the desert can turn green when the rains get going? There’s a Portal blog post on that (with pics)!
In the desert, water is life. Without it, the desert is brown and dusty. At our site, the rains come twice a year – once during the ‘winter’ (I put that in quotes for our readers where winter means snow and/or extended periods below freezing) and once during the summer. Water in the summer and water in the winter don’t have the same effect on the desert, though. Plants need both warmth and water to grow. When rain falls in the desert in the winter, growth is slow and typically waits until the warmer temperatures of spring. In the summer, though, the high temperatures and the rain from Arizona’s monsoons make for an explosive combination. How fast can the desert turn green? Here’s a series of photos from our site – one per day for a week that we think conveys this better than words. Enjoy the slide show:
You…
View original post 65 more words
Pregnancy in Kangaroo rats
A guest post from last week on the Portal Blog about studying Kangaroo rat placentas!
~While everyone’s busy at ESA this week, we’d like to keep the 40th anniversary ball rolling with a guest post from a visiting researcher at Portal. Jess Dudley has been using the Portal area to compare pregnancy in kangaroo rats and Australian marsupials. We’ll be featuring other guest posts through the rest of the year. (If you’d like to do something similar, please send us your info!)~
In July 2015 I travelled the 24+ hours from Sydney, Australia to the beautiful town of Portal to research pregnancy in Kangaroo rats. To everyone’s astonishment we do not have Kangaroo rats in Australia! I am sure I don’t need to explain my fascination with Kangaroo rats with this audience but in terms of pregnancy they have some unique features which differ from most rodents. This finding by King and Tibbitts in the 1960’s led me to wonder how the placenta forms during pregnancy…
View original post 318 more words
Weecology at ESA
We have a modest sized group of current folks at ESA this week presenting on all the cool things they’ve been doing. We’re also around and always happy to try to find time to grab a coffee or just a few minutes to chat science.
Our schedule for the week is:
Monday
Get a double dose of rapid change in ecological communities from the Portal Project with Morgan Ernest and Erica Christensen.
02:50 PM – 03:10 PM in C120-121. Erica Christensen (w/Dave Harris & Morgan Ernest). Novel approach for the analysis of community dynamics: Separating rapid reorganizations from gradual trends.
03:20 PM – 03:40 PM in C120-121. Morgan Ernest (w/Erica Christensen). Do existing communities slow community reorganization in response to changes in assembly processes?
Tuesday
Find out what we can learn about how natural systems may change in response to climate from looking at large datasets with Ethan White and Kristina Riemer.
01:50 PM – 02:10 PM in D139. Kristina Riemer (w/Rob Guralnick & Ethan White). No general relationship between mass and temperature in endotherm species.
02:30 PM – 02:50 PM in Portland Blrm 256. Ethan White (w/Dave Harris & Shawn Taylor). Data-intensive approaches to forecasting biodiversity.
Thursday
Check out a new project with a new and exciting research tool for us (metabarcoding) at the poster session.
04:30 PM – 06:30 PM in the Exhibit Hall. Ellen Bledsoe (w/Sam Wisely & Morgan Ernest). DNA metabarcoding of fecal samples provides insight into desert rodent diet partitioning.
Collaborations
There are also plenty of weecology collaborations being presented this week:
- Transient species are common: Implications for ecological inference (Thursday at 9:20 am in C120-121)
- Modeling community assembly and the functioning of ecosystems (Thursday at 4:20 pm in E143-144)
- Advancing biodiversity-ecosystem function research by integrating community assembly: The CAFE approach (Friday at 8 am in Portland Blrm 257)
- Diversity alone is not enough: Nitrogen enrichment and community assembly determine ecosystem response to drought (Friday at 9:40 am in Portland Blrm 257)
We’re really looking forward to catching up with old friends and meeting new people this week.
The Portal Project 40th Anniversary
The Portal Project turns 40 this year! In celebration, we will be regularly posting about the history of the site, new things going on, natural history of the desert, and other fun things over at the Portal Blog.
Funded by the National Science Foundation to study the importance of competition and granivory in desert ecosystems, the Portal Project first started collecting data in the summer of 1977. The initial grant was just for 5 years, yet 40 years later the site is still collecting data on plants, rodents, and weather.
To our friends who study paleoecology, 40 years is an eyeblink but in the span of a human life, 40 years is a long time. As you might expect, much has changed on the project. For one thing, after 40 years, the team running the site has changed. The original team of scientists, Jim Brown, Dinah Davidson, and Jim Reichman have all retired from the daily challenges of training students and writing grants, though some are still doing science. In their place, Tom Valone and I do our best to keep things running, studying the mysteries of the…
View original post 578 more words