Upgrading OpenWrt on an Arduino Yun Without a MicroSD Card

An entire fantasy football season has passed since finishing the Trophy of the Future, my “world’s first internet-connected fantasy football trophy,” and my league has a new champion. That means it’s time to revisit the project to tie up some loose ends! More on that will come in time, but first I want to share the solution to the initial problem I encountered.

I wanted to upgrade the Yun to the latest version of OpenWrt because opkg was throwing weird “failed to download” errors. From scanning the Arduino forums, it seemed like updating OpenWrt was the recommended first step (and as it turns out, the update also fixes Heartbleed, so it’s worth doing anyway). The instructions assume you have a MicroSD mounted in your Yun, which I don’t, but it is still possible to do the upgrade without one.

The OpenWrt image clocks in at 16 MB, and by running df -h I saw that the Yun’s onboard memory had around 20 MB free in the /tmp directory. I downloaded the latest Yun image from the Arduino website and unzipped it on my Mac, then used scp to move the file to the Yun:

1
scp openwrt-ar71xx-generic-yun-16M-squashfs-sysupgrade.bin root@arduino.local:/tmp/

According to Arduino’s upgrade guide, the next step would be to run run-sysupgrade on the OpenWrt image, but a second problem surfaces here. It turns out that the first thing run-sysupgrade does is copy the image to /tmp, which errors because the file already exists. I could move the file to a subdirectory so run-sysupgrade could copy it back, but the Yun’s onboard memory doesn’t have enough space for that.

The solution here, noted on OpenWrt’s system upgrade page, is to use sysupgrade – note the lack of run- in that command’s name. Like this:

1
sysupgrade /tmp/openwrt-ar71xx-generic-yun-16M-squashfs-sysupgrade.bin

Three minutes later, you have upgraded OpenWrt on your Arduino Yun! All the configurations should be the same, but you will have to reinstall any opkg packages.

Sharing our videos, forever

This post was originally published on the Cooper Hewitt Labs blog.

Our galleries and Pen experience are driven by the idea that everything a visitor can see or do in the museum itself should be accessible later on.

Part of getting the collections site and API (which drives all the interfaces in the galleries designed by Local Projects) ready for reopening has involved the gathering and, in some cases, generation of data to display with our exhibits and on our new interactive tables. In the coming weeks, I’ll be playing blogger catch-up and will write about these new features. Today, I’ll start with videos.

Screen-Shot-2014-12-10-at-3.09.22-PM

Besides the dozens videos produced in-house by Katie – such as the amazing Design Dictionary series – we have other videos relating to people, objects and exhibitions in the museum. Currently, these are all streamed on our YouTube channel. While this made hosting much easier, it meant that videos were not easily related to the rest of our collection and therefore much harder to find. In the past, there were also many videos that we simply didn’t have the rights to show after their related exhibition had ended, and all the research and work that went into producing the video was lost to anyone who missed it in the gallery. A large part of this effort was ensuring that we have the rights to keep these videos public, and so we are immensely grateful to Matthew Kennedy, who handles all our image rights, for doing that hard work.

A few months ago, we began the process of adding videos and their metadata in to our collections website and API. As a result, when you take a look at our page for Tokujin Yoshioka’s Honey-Pop chair , below the object metadata, you can see its related video in which our curators and conservators discuss its unique qualities. Similarly, when you visit our page for our former director, the late Bill Moggridge, you can see two videos featuring him, which in turn link to their own exhibitions and objects. Or, if you’d prefer, you can just see all of our videos here.

In addition to its inclusion in the website, video data is also now available over our API. When calling an API method for an object, person or exhibition from our collection, paths to the various video sizes, formats and subtitle files are returned. Here’s an example response for one of Bill’s two videos:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{
  "id": "68764297",
  "youtube_url": "www.youtube.com/watch?v=DAHHSS_WgfI",
  "title": "Bill Moggridge on Interaction Design",
  "description": "Bill Moggridge, industrial designer and co-founder of IDEO, talks about the advent of interaction design.",
  "formats": {
    "mp4": {
      "1080": "https://s3.amazonaws.com/videos.collection.cooperhewitt.org/DIGVID0059_1080.mp4",
      "1080_subtitled": "https://s3.amazonaws.com/videos.collection.cooperhewitt.org/DIGVID0059_1080_s.mp4",
      "720": "https://s3.amazonaws.com/videos.collection.cooperhewitt.org/DIGVID0059_720.mp4",
      "720_subtitled": "https://s3.amazonaws.com/videos.collection.cooperhewitt.org/DIGVID0059_720_s.mp4"
    }
  },
  "srt": "https://s3.amazonaws.com/videos.collection.cooperhewitt.org/DIGVID0059.srt"
}

The first step in accomplishing this was to process the videos into all the formats we would need. To facilitate this task, I built VidSmanger, which processes source videos of multiple sizes and formats into consistent, predictable derivative versions. At its core, VidSmanger is a wrapper around ffmpeg, an open-source multimedia encoding program. As its input, VidSmanger takes a folder of source videos and, optionally, a folder of SRT subtitle files. It outputs various sizes (currently 1280×720 and 1920×1080), various formats (currently only mp4, though any ffmpeg-supported codec will work), and will bake-in subtitles for in-gallery display. It gives all of these derivative versions predictable names that we will use when constructing the API response.

VidsmangCartoon

Because VidSmanger is a shell script composed mostly of simple command line commands, it is easily augmented. We hope to add animated gif generation for our thumbnail images and automatic S3 uploading into the process soon. Here’s a proof-of-concept gif generated over the command line using these instructions. We could easily add the appropriate commands into VidSmanger so these get made for every video.

anim

For now, VidSmanger is open-source and available on our GitHub page! To use it, first clone the repo and the run:

1
./bin/init.sh

This will initialize the folder structure and install any dependencies (homebrew and ffmpeg). Then add all your videos to the source-to-encode folder and run:

1
./bin/encode.sh

Now you’re smanging!

Rethinking Search on the Collections Site

This post was originally published on the Cooper Hewitt Labs blog.

One of my longer-term projects since joining the museum has been rethinking how the search feature functions on the collections website. As we get closer to re-opening the museum with a suite of new technologies, our work in collaboration with Local Projects has prompted us to take a close look at the moving pieces that comprise the backend of our collections site and API. Search, naturally, forms a large piece of that. Last week, after a few weeks of research and experimentation, I pushed the first iteration live. In this post, I’ll share some of the thoughts and challenges that informed our changes.

First, a glossary of terms for readers who (like me, a month ago) have little-to-no experience with the inner-workings of a search engine:

  • Platform: The software that actually does the searching. The general process is that we feed data to the platform (see “index”), and then we ask it for results matching a certain set of parameters (see “query”). Everything else is handled by the platform itself. Part of what I’ll get into below involves our migration from one platform, Apache Solr, to another, Elasticsearch.
  • Index: An index is the database that the search platform uses to perform searches on. The search index is a lot like the primary database (it probably could fill that role if it had to) but it adds extra functionality to facilitate quick and accurate retrieval of search results.
  • Query: The rules to follow in selecting things that are appropriate to provide as search results. For users, the query could be something like “red concert poster,” but we have to translate that into something that the search provider will understand before results can be retrieved. Search providers give us a lot of different ways we can query things (ranges of a number, geographic distance or word matching to name a few), and a challenge for us as interface designers is to decide how transparent we want to make that translation. Queries also allow us to define how results should be sorted and how to facet results.
  • Faceting/Aggregation: A way of grouping results based on traits they posses. For example, faceting on “location” when you search our collection for “cat” reveals that 80 of our cat-related things are from the USA, 16 are from France, and so on.
  • Analysis (Tokenization/Stemming etc): A process that helps a computer work with sentences. Tokenization, for example, would split a search for “white porcelain vase” into the individual tokens: “white,” “porcelain” and “vase,” and then perform a search for any number of those tokens. Another example is stemming, which would allow the platform to understand that if a user searches for “running,” then items containing other words like “run” or “runner” are also valid search results. Analysis also gives us the opportunity to define custom rules that might include “marathon” and “track” as valid results in a search for “running.”

The State of Search

Our old search functionality showed its symptoms of under-performance in a few ways. For example, basic searches — phrases like “red concert poster” — turned up no results despite the presence of such objects in our collection, and searching for people would not return the person themselves, only their objects. These symptoms led me to identify what I considered the two big flaws in our search implementation.

On the backend, we were only indexing objects. This meant that if you searched for “Ray Eames,” you would see all of the objects we have associated with her, but to get to her individual person page, you would have to first click on an object and then click on her name. Considering that we have a lot of non-objects1, it makes sense to index them all and include them, where relevant, in the results. This made my first objective to find a way to facilitate the indexing and querying of different types of things.

On the frontend, we previously gave users two different ways to search our collection. The default method, accessible through the header of every page, performed a full text search on our Solr index and returned results sorted by image complexity. Users could also choose the “fancy search” option, which allows for searches on one or more of the individual fields we index, like “medium,” “title,” or “decade.” We all agreed here that “fancy search” was confusing, and all of its extra functionality — faceting, searching across many fields — shouldn’t be seen as “advanced” features. My second objective in rethinking how search works, then, was to unify “fancy” and “regular” search into just “search.”

Objective 1: Update the Backend

Our search provider, Solr, requires that a schema be present for every type of thing being indexed. The schema (an XML file) tells Solr what kind of value to expect for a certain field and what sort of analysis to perform on the field. This means I’d have to write a schema file — anticipating how I’d like to form all the indexed data — for each new type of thing we want to search on.

One of the features of Elasticsearch is that it is “schemaless,” meaning I can throw whatever kind of data I want at the index and it figures out how to treat it. This doesn’t mean Elasticsearch is always correct in its guesses — for example, it started treating our accession numbers as dates, which made them impossible to search on — so it also gives you the ability to define mappings, which has the same effect as Solr’s schema. But if I want to add “people” to the index, or add a new “location” field to an object, using Elasticsearch means I don’t have to fiddle with any schemas. This trait of Elasticsearch alone made worth the switch (see Larry Wall’s first great virtue of programmers, laziness: “the quality that makes you go to great effort to reduce overall energy expenditure”) because it’s important to us that we have the ability to make quick changes to any part of our website.

Before building anything in to our web framework, I spent a few days getting familiar with Elasticsearch on my own computer. I wrote a python script that loops through all of the CSVs from our public collections repository and indexed them in a local Elasticsearch server. From there, I started writing queries just to see what was possible. I was quickly able to come up with a lot of the functionality we already have on our site (full-text search, date range search) and get started with some complex queries as well (“most common medium in objects between 1990-2000,” for example, which is “paper”). This code is up on Github, so you can get started with your own Cooper Hewitt search engine at home!

Once I felt that I had a handle on how to index and query Elasticsearch, I got started building it into our site. I created a modified version of our Solr indexing script (in PHP) that copied objects, people, roles and media from MySQL and added them to Elasticsearch. Then I got started on the endpoint, which would take search parameters from a user and generate the appropriate query. The code for this would change a great deal as I worked on the frontend and occasionally refactored and abstracted pieces of functionality, but all the pieces of the pipeline were complete and I could begin rethinking the frontend.

Objective 2: Update the Frontend

Updating the frontend involved a few changes. Since we were now indexing multiple categories of things, there was still a case for keeping a per-category search view that gave users access to each field we have indexed. To accommodate these views, I added a tab bar across the top of the search forms, which defaults to the full-collection search. This also eliminates confusion as to what “fancy search” did as the search categories are now clearly labeled.

tabs

The next challenge was how to display sorting. Previously, the drop-down menu containing sort options was hidden in a “filter these results” collapsible menu. I wanted to lay out all of the sorting options for the user to see at a glance and easily switch between sorting modes. Instead of placing them across the top in a container that would push the search results further down the page, I moved them to a sidebar which would also house search result facets (more on that soon). While it does cut in to our ability to display the pictures as big as we’d like, it’s the only way we can avoid hiding information from the user. Placing these options in a collapsible menu creates two problems: if the menu is collapsed by default, we’re basically ensuring that nobody will ever use them. If the menu is expanded by default, then it means that the actual results are no longer the most important thing on the page (which, on a search results page, they clearly are). The sidebar gives us room to lay out a lot of options in an unobtrusive but easily-accessible way2.

sorting

The final challenge on the frontend was how to handle faceting. Faceting is a great way for users who know what they’re looking for to narrow down options, and a great way for users who don’t know what they’re looking for to be exposed to the various buckets we’re able to place objects in to.

Previously on our frontend, faceting was only available on fancy search. We displayed a few of the faceted fields across the top of the results page, and if you wanted further control, users could select individual fields to facet on using a drop-down menu at the bottom of the fancy search form. When they used this, though, the results page displayed only the facets, not the objects. In my updates, I’ve turned faceting on for all searches. They appear alongside the search results in the sidebar.

facets

Doing it Live

We initially rolled these changes out about 10 days ago, though they were hidden from users who didn’t know the URL. This was to prove to ourselves that we could run Elasticsearch and Solr alongside each other without the whole site blowing up. We’re still using Solr for a bit more than just the search (for example, to show which people have worked with a given person), so until we migrate completely to Elasticsearch, we need to have both running in parallel.

A few days later, I flipped the switch to make Elasticsearch the default search provider and passed the link around internally to get some feedback from the rest of the museum. The feedback I got was important not just for working out the initial bugs and kinks, but also (and especially for myself as a relative newbie to the museum world) to help me get the language right and consider all the different expectations users might have when searching our collection. This resulted in some tweaks to the layout and copy, and some added functionality, but mostly it will inform my bigger-picture design decisions going forward.

A Few Numbers…

Improving performance wasn’t a primary objective in our changes to search, but we got some speed boosts nonetheless.

Query Before (Solr) After (Elasticsearch)
query=cat, facets on 162 results in 1240-1350ms 167 results in 450-500ms
year_acquired=gt1990, facets on 13,850 results in 1430-1560ms 14,369 results in 870-880ms
department_id=35347493&period_id=35417101, facets on 1,094 results in 1530-1580ms 1,150 results in 960-990ms

There are also cases where queries that turned up nothing before now produce relevant results, like “red concert poster,” (0 -> 11 results) “German drawings” (0 -> 101 results) and “checkered Girard samples” (0 -> 10 results).

Next Steps

Getting the improved search in front of users is the top priority now – that means you! We’re very interested in hearing about any issues, suggestions or general feedback that you might have — leave them in the comments or tweet us @cooperhewittlab.

I’m also excited about integrating some more exiting search features — things like type-ahead search and related search suggestion — on to the site in the future. Additionally, figuring out how to let users make super-specific queries (like the aforementioned “most common medium in objects between 1990-2000”) is a challenge that will require a lot of experimentation and testing, but it’s definitely an ability we want to put in the hands of our users in the future.

New Search is live on our site right now – go check it out!

1 We’ve been struggling to find a word to use for things that are “first-class” in our collection (objects, people, countries, media etc.) that makes sense to both museum-folk and the laypeople. We can’t use “objects” because those already refer to a thing that might go on display in the museum. We’ve also tried “items,” “types” and “isas” (as in, “what is this? it is a person”). But nothing seems to fit the bill.

2 We’re not in complete agreement here at the labs over the use of a sidebar to solve this design problem, but we’re going to leave it on for a while and see how it fares with time. Feedback is requested!

Open Source Soundboard for Live Performance

Screen Shot 2014-06-15 at 10.40.02 PM

Last semester, I needed a bare-bones soundboard to cue up and play sound clips for AARPlane, the midterm performance of my Puppets class at ITP. Not knowing my way around pro audio software (which I’m sure makes this a simple task) and seeing that the current landscape of online soundboards consists of awfully-designed holdovers from the days of Flash, I decided to build my own. It’s far from full-featured – currently it will just play, pause and replay sound clips – but if you need to need a “world’s-dumbest”-style soundboard, I think you should give this a try.

Getting it up and running is straightforward-ish (and definitely needs to be straightforward-er):

  1. Install NodeJS
  2. Download the code from GitHub
  3. Create a sounds folder inside of public and place all of your web-encoded audio files in it (mp3s work fine)
  4. Open the project folder in Terminal and run npm install followed by node app
  5. Point your browser to 127.0.0.1:3000
  6. Click a sound to play it! Click again to pause!

All of the source code is available on GitHub, along with my list of issues (I’m accepting pull requests!).

The Adventures of Teen Bloggers

teenbloggers_01

For my thesis project at ITP, I built The Adventures of Teen Bloggers, a creative-nonfiction graphic adventure that lets you become a real teen blogger from the heyday of LiveJournal.

With the project, I want to highlight the tension between the value our online data has as historical documents and the embarrassment we feel about the things we shared in a prior phase of our lives.

Upon starting the game, players are asked to pick a LiveJournal account. Any extant LiveJournal is a playable character in the game, though I encourage players to play with their own account if they had one. Players are then placed in a high school hallway, where they must navigate the world only saying things that their selected LiveJournal user wrote on their blog.

The game is a work in progress as I continue to research online archiving. Specific next steps are outlined on Github; the source code for the project is there as well. You can play the game here.

Below is the presentation I gave on the project during ITP’s Thesis Week.

Announcing SkyDesigner! Sam Brenner joins the Labs

This post was originally published on the Cooper Hewitt Labs blog.

As part of my application for my position at the Cooper Hewitt museum, I built SkyDesigner, a web application that lets users replace the color of the sky with a picture of a similarly-colored object from the Cooper Hewitt’s collection. The “sky” idea comes from the original assignment, which was to create an application using both a weather API and the Cooper Hewitt API, but you can use SkyDesigner to swap out colors from anything you can take a picture of (meaning, it’s great for selfies). Give it a try now!

687474703a2f2f7777772e73616d6a6272656e6e65722e636f6d2f70726f6a656374732f736b792f6c69622f696d672f30312e6a7067 687474703a2f2f7777772e73616d6a6272656e6e65722e636f6d2f70726f6a656374732f736b792f6c69622f696d672f30322e6a7067

Here’s how it works: first, users take a picture. If they’re on a computer, they can use their webcam. If they’re on a smartphone, they can use the built-in camera. Android users get (in my opinion) the better experience, because Android supports getUserMedia – this means that users can start their camera and take a picture without ever having to leave the application. iOS doesn’t support getUserMedia yet, so they are sent off to the native iOS camera app to take their picture, which then gets passed back to the browser. Once I receive the picture, I load it into a canvas.

In the next step, users tap on their picture to select a color. The color’s hex code is sent straight to the Cooper-Hewitt API’s search method, where I search for similarly-colored objects that have an associated image. While waiting for a response from the API, I also tell the canvas to make every pixel within range of the selected color become transparent. When I get the image back from the API, I load it in behind the canvas and presto! It shows through where the selected color used to be. Finally, the image is titled based on the object’s creator and your current weather information.

It’s built using HTML, CSS and JavaScript. The original application had PHP to talk to the API but that’s since been ported to JavaScript since I now have the luxury of running the site on the Collections website itself where we have our own built-in API hooks.

Being a weekend project, there are some missing features – sharing is a big one – but I think it demonstrates the API’s ability to provide fresh, novel ways into a museum’s vast collection. Here’s the link again, and you can also find the source on GitHub.

Soil as Medium Final Project Proposal: Compost Pile Composter

compostcompost

My first idea for a final project was to take the “fake compost fake rock” from my midterm project and make it real. After a little more thought, I decided that I wanted to deal with the action of composting more directly. Getting people to cover up unsightly pipes in their yards with an equally unsightly fake pile of compost is funny, but I’d like to work on something that is ultimately productive.

As a variation on the theme of things-that-look-other-versions-of-themselves (I associate such items with museum gift stores), I’m going to modify an indoor compost bin to look like an outdoor compost pile. In ReNatured we frequently talked about the frustrating idea that “nature” is something separate from the streets, sidewalks, parks and empty lots that form our urban environment. I hope that this project will help to break down that divide. (And I’ve also been meaning to start composting in my apartment, so this solves that problem too!)

My intended audience for this project are those whose vision of good design means clean, expensive and high-polish. By playing around functionally (but not formally) in the museum-gift-store-product vernacular I want to show first that the appearance of a compost pile is nothing to be afraid of and second that it’s mostly a non-issue with a worm bin anyway.

My materials will be the Worm Factory composting kit and custom-printed vinyl stickers that I will cut and apply to the Worm Factory. I’ve ordered the worm kit, so the remaining weeks will involve designing the stickers and then printing, cutting and applying them to the worm bin. Tangentially related will be my own use of the worm bin, which I hope to get set up once it arrives.

Thesis Project Progress

It’s been a while since I’ve posted anything about my thesis project here, and as ITP theses are wont to do, it has evolved quite a bit. I’ll post more details in due course but I wanted to post some screenshots of what I’ve been busy with.

Screen Shot 2014-04-18 at 10.26.16 AM Screen Shot 2014-04-18 at 10.26.04 AM Screen Shot 2014-04-18 at 10.26.10 AM

Soil as Medium: Midterm

fakecompost

We had a quick midterm for Soil as Medium, for which the assignment was to make a speculative design piece dealing with some of the issues we’ve covered so far in the class. Between this class and the Fungus Among Us, I’ve been thinking about how I can acknowledge the complexities of the many interacting systems that we humans would prefer to simplify.

I am reminded of the perennial favorite SkyMall, purveyors of everything from PajamaJeans to a combo smartphone charger/UV-sanitizer. SkyMall commoditizes our dreams of the Good Life in most high-end fashion, sparing no modern problem from the simplifying power of technology. Lawncare is, of course, one of those modern problems, but one in which oversimplification of the problem has proved harmful enough to disturb a refined yet delicate system.

For my midterm assignment, I’ve redesigned four SkyMall home and garden offerings to be a little more aware of the complexities of soil science. All images can be enlarged by clicking.

Continue reading