My First (R) Shiny App: An Annotated Tutorial

Image Credit: Doug Buckley of http://hyperactive.to

Image Credit: Doug Buckley of http://hyperactive.to

I’ve been meaning to learn Shiny for 2 years now… and thanks to a fortuitous email from @ImADataGuy this morning and a burst of wild coding energy about 5 hours ago, I am happy to report that I have completely fallen in love again. The purpose of this post is to share how I got my first Shiny app up and running tonight on localhost, how I deployed it to the http://shinyapps.io service, and how you can create a “Hello World” style program of your own that actually works on data that’s meaningful to you.

If you want to create a “Hello World!” app with Shiny (and your own data!) just follow these steps:

0. Install R 3.2.0+ first! This will save you time.
1. I signed up for an account at http://shinyapps.io.
2. Then I clicked the link in the email they sent me.
3. That allowed me to set up my https://radziwill.shinyapps.io location.
4. Then I followed the instructions at https://www.shinyapps.io/admin/#/dashboard
(This page has SPECIAL SECRET INFO CUSTOMIZED JUST FOR YOU ON IT!!) I had lots 
of problems with devtools::install_github('rstudio/shinyapps') - Had to go 
into my R directory, manually delete RCurl and digest, then 
reinstall both RCurl and digest... then installing shinyapps worked.
Note: this last command they tell you to do WILL NOT WORK because you do not have an app yet! 
If you try it, this is what you'll see:
> shinyapps::deployApp('path/to/your/app')
Error in shinyapps::deployApp("path/to/your/app") : 
C:\Users\Nicole\Documents\path\to\your\app does not exist
5. Then I went to http://shiny.rstudio.com/articles/shinyapps.html and installed rsconnect.
6. I clicked on my name and gravatar in the upper right hand corner of the 
https://www.shinyapps.io/admin/#/dashboard window I had opened, and then clicked 
"tokens". I realized I'd already done this part, so I skipped down to read 
"A Demo App" on http://shiny.rstudio.com/articles/shinyapps.html
7. Then, I re-installed ggplot2 and shiny using this command:
install.packages(c('ggplot2', 'shiny'))
8. I created a new directory (C:/Users/Nicole/Documents/shinyapps) and used
setwd to get to it.
9. I pasted the code at http://shiny.rstudio.com/articles/shinyapps.html to create two files, 
server.R and ui.R, which I put into my new shinyapps directory 
under a subdirectory called demo. The subdirectory name IS your app name.
10. I typed runApp("demo") into my R console, and voila! The GUI appeared in 
my browser window on my localhost.
-- Don't just try to close the browser window to get the Shiny app 
to stop. R will hang. To get out of this, I had to use Task Manager and kill R.
--- Use the main menu, and do Misc -> Stop Current Computation
11. I did the same with the "Hello Shiny" code at http://shiny.rstudio.com/articles/shinyapps.html. 
But what I REALLY want is to deploy a hello world app with MY OWN data. You know, something that's 
meaningful to me. You probably want to do a test app with data that is meaningful to you... here's 
how you can do that.
12. A quick search shows that I need jennybc's (Github) googlesheets package to get 
data from Google Drive viewable in my new Shiny app.
13. So I tried to get the googlesheets package with this command:
devtools::install_github('jennybc/googlesheets')
but then found out it requires R version 3.2.0. I you already have 3.2.0 you can skip 
to step 16 now.
14. So I reinstalled R using the installr package (highly advised if you want to 
overcome the agony of upgrading on windows). 
See http://www.r-statistics.com/2013/03/updating-r-from-r-on-windows-using-the-installr-package/
for info -- all it requires is that you type installR() -- really!
15. After installing R I restarted my machine. This is probably the first time in a month that 
I've shut all my browser windows, documents, spreadsheets, PDFs, and R sessions. I got the feeling 
that this made my computer happy.
16. Then, I created a Google Sheet with my data. While viewing that document, I went to 
File -> "Publish to the Web". I also discovered that my DOCUMENT KEY is that 
looooong string in the middle of the address, so I copied it for later:
1Bs0OH6F-Pdw5BG8yVo2t_VS9Wq1F7vb_VovOmnDSNf4
17. Then I created a new directory in C:/Users/Nicole/Documents/shinyapps to test out 
jennybc's googlesheets package, and called it jennybc
18. I copied and pasted the code in her server.R file and ui.R file
from https://github.com/jennybc/googlesheets/tree/master/inst/shiny-examples/01_read-public-sheet 
into files with the same names in my jennybc directory
19. I went into my R console, used getwd() to make sure I was in the
C:/Users/Nicole/Documents/shinyapps directory, and then typed
 runApp("jennybc")
20. A browser window popped up on localhost with her test Shiny app! I played with it, and then 
closed that browser tab.
21. When I went back into the R console, it was still hanging, so I went to the menu bar 
to Misc -> Stop Current Computation. This brought my R prompt back.
22. Now it was time to write my own app. I went to http://shiny.rstudio.com/gallery/ and
found a layout I liked (http://shiny.rstudio.com/gallery/tabsets.html), then copied the 
server.R and ui.R code into C:/Users/Nicole/Documents/shinyapps/my-hello -- 
and finally, tweaked the code and engaged in about 100 iterations of: 1) edit the two files, 
2) type runApp("my-hello") in the R console, 3) test my Shiny app in the 
browser window, 4) kill browser window, 5) do Misc -> Stop Current Computation 
in R. ALL of the computation happens in server.R, and all the display happens in ui.R:

server.R:

library(shiny)
library(googlesheets)
library(DT)

my_key <- "1Bs0OH6F-Pdw5BG8yVo2t_VS9Wq1F7vb_VovOmnDSNf4"
my_ss <- gs_key(my_key)
my_data <- gs_read(my_ss)

shinyServer(function(input, output, session) {
 output$plot <- renderPlot({
 my_data$type <- ordered(my_data$type,levels=c("PRE","POST"))
 boxplot(my_data$score~my_data$type,ylim=c(0,100),boxwex=0.6)
 })
 output$summary <- renderPrint({
 aggregate(score~type,data=my_data, summary)
 })
 output$the_data <- renderDataTable({
 datatable(my_data)
 })

})

ui.R:

library(shiny)
library(shinythemes)
library(googlesheets)

shinyUI(fluidPage(
 
 # Application title
 titlePanel("Nicole's First Shiny App"),
 
 # Sidebar with controls to select the random distribution type
 # and number of observations to generate. Note the use of the
 # br() element to introduce extra vertical spacing
 sidebarLayout(
 sidebarPanel(
     helpText("This is my first Shiny app!! It grabs some of my data 
from a Google Spreadsheet, and displays it here. I      
also used lots of examples from"),
     h6(a("http://shiny.rstudio.com/gallery/", 
href="http://shiny.rstudio.com/gallery/", target="_blank")),
     br(),
     h6(a("Click Here for a Tutorial on How It Was Made", 
href="https://qualityandinnovation.com/2015/12/08/my-first-shin     
y-app-an-annotated-tutorial/",
      target="_blank")),
      br()
 ),
 
 # Show a tabset that includes a plot, summary, and table view
 # of the generated distribution
 mainPanel(
    tabsetPanel(type = "tabs", 
    tabPanel("Plot", plotOutput("plot")), 
    tabPanel("Summary", verbatimTextOutput("summary")), 
    tabPanel("Table", DT::dataTableOutput("the_data"))
 )
 )
 )
))


23. Once I decided my app was good enough for my practice round, it was time to 
deploy it to the cloud.
24. This part of the process requires the shinyapps and dplyr 
packages, so be sure to install them:

devtools::install_github('hadley/dplyr')
library(dplyr)
devtools::install_github('rstudio/shinyapps')
library(shinyapps)
25. To deploy, all I did was this: setwd("C:/Users/Nicole/Documents/shinyapps/my-hello/")
deployApp()

CHECK OUT MY SHINY APP!!

If Japan Can, Why Can’t We? A Retrospective

if-japan-canJune 24, 1980 is kind of like July 4, 1776 for quality management… that’s the pivotal day that NBC News aired its one hour and 16 minute documentary called “If Japan Can, Why Can’t We?” introducing W. Edwards Deming and his methods to the American public. The video has been unavailable for years, but as of just last week, it’s been posted on YouTube. So my sophomore undergrads in Production & Operations Management took a step back in time to get a taste of the environment in the manufacturing industry in the late 1970’s, and watched it during class this week.

The last time I watched it was in 1997, in a graduate industrial engineering class. It didn’t feel quite as dated as it does now, nor did I have the extensive experience in industry as a lens to view the interviews through. But what did surprise me is that the core of the challenges they were facing aren’t that much different than the ones we face today — and the groundbreaking good advice from Deming is still good advice today.

  • Before 1980, it was common practice to produce a whole bunch of stuff and then check and see which ones were bad, and throw them out. The video provides a clear and consistent story around the need to design quality in to products and processes, which then reduces (or eliminates) the need to inspect bad quality out.
  • It was also common to tamper with a process that was just exhibiting random variation. As one of the line workers in the documentary said, “We didn’t know. If we felt like there might be a problem with the process, we would just go fix it.” Deming’s applications of Shewhart’s methods made it clear that there is no need to tamper with a process that’s exhibiting only random variation.
  • Both workers and managers seemed frustrated with the sheer volume of regulations they had to address, and noted that it served to increase costs, decrease the rate of innovation, and disproportionately hurt small businesses. They noted that there was a great need for government and industry to partner to resolve these issues, and that Japan was a model for making these interactions successful.
  • Narrator Lloyd Dobyns remarked that “the Japanese operate by consensus… we, by competition.” He made the point that one reason Japanese industrial reforms were so powerful and positive was that their culture naturally supported working together towards shared goals. He cautioned managers that they couldn’t just drop in statistical quality control and expect a rosy outcome: improving quality is a cultural commitment, and the methods are not as useful in the absence of buy-in and engagement.

The video also sheds light on ASQ’s November question to the Influential Voices, which is: “What’s the key to talking quality with the C-Suite?” Typical responses include: think at the strategic level; create compelling arguments using the language of money; learn the art of storytelling and connect your case with what it important to the executives.

But I think the answer is much more subtle. In the 1980 video, workers comment on how amazed their managers were when Deming proclaimed that management was responsible for improving productivity. How could that be??!? Many managers at that time were convinced that if a productivity problem existed, it was because the workers didn’t work fast enough, or with enough skill — or maybe they had attitude problems! Certainly not because the managers were not managing well. Implementing simple techniques like improving training programs and establishing quality circles (which demonstrated values like increased transparency, considering all ideas, putting executives on the factory floor so they could learn and appreciate the work being done, increasing worker participation and engagement, encouraging work/life balance, and treating workers with respect and integrity) were already demonstrating benefits in some U.S. companies. But surprisingly, these simple techniques were not widespread, and not common sense.

Just like Deming advocated, quality belongs to everyone. You can’t go to a CEO and suggest that there are quality issues that he or she does not care about. More likely, the CEO believes that he or she is paying a lot of attention to quality. They won’t like it if you accuse them of not caring, or not having the technical background to improve quality. The C-Suite is in a powerful position where they can, through policies and governance, influence not only the actions and operating procedures of the system, but also its values and core competencies — through business model selection and implementation. 

What you can do, as a quality professional, is acknowledge and affirm their commitment to quality. Communicate quickly, clearly, and concisely when you do. Executives have to find the quickest ways to decompose and understand complex problems in rapidly changing external environments, and then make decisions that affect thousands (and sometimes, millions!) of people. Find examples and stories from other organizations who have created huge ripples of impact using quality tools and technologies, and relate them concretely to your company.

Let the C-Suite know that you can help them leverage their organization’s talent to achieve their goals, then continually build their trust.

The key to talking quality with the C-suite is empathy.

 

You may also be interested in “Are Deming’s 14 Points Still Valid?” from Nov 19, 2012.

Control Charts in R: A Guide to X-Bar/R Charts in the qcc Package

xbar-chartStatistical process control provides a mechanism for measuring, managing, and controlling processes. There are many different flavors of control charts, but if data are readily available, the X-Bar/R approach is often used. The following PDF describes X-Bar/R charts and shows you how to create them in R and interpret the results, and uses the fantastic qcc package that was developed by Luca Scrucca. Please let me know if you find it helpful!

Creating and Interpreting X-Bar/R Charts in R

What if Your Job Was Focused on Play?

james-siegal

James Siegal (picture from his Twitter profile, @jsiegal at http://twitter.com/jsiegal)

Last weekend, I had the opportunity to talk to James Siegal, the President of KaBOOM! – a non-profit whose mission is lighthearted, but certainly not frivolous: to bring balanced and active play into the daily lives of all kids! James is another new Business Innovation Factory (BIF) storyteller for 2015… and I wanted to find out how I could learn from his experiences to bring a sense of play into the work environment. (For me, that’s at a university, interacting with students on a daily basis.)

Over the past 20 years, KaBOOM! has built thousands of playgrounds, focusing on children growing up in poverty. By enlisting the help of over a million volunteers, James and his organization have mobilized communities using a model that starts with kids designing their dream playgrounds. It’s a form of crowdsourced placemaking.

Now, KaBOOM! is thinking about a vision that’s a little broader: driving social change at the city level. Doing this, they’ve found, requires answering one key question: How can you integrate play into the daily routine for kids and families? If play is a destination, there are “hassle factors” that must be overcome: safety, travel time, good lighting, and restroom facilities, for starters. So, in addition to building playgrounds, KaBOOM! is challenging cities to think about integrating play everywhere — on the sidewalk, at the bus stop, and beyond.

How can this same logic apply to organizations integrating play into their cultures? Although KaBOOM! focuses on kids, he had some more generalizable advice:

  • The desire for play has to be authentic, not forced. “We truly value kids, and we truly value families. Our policies and our culture strive to reflect that.” What does your organization value at its core? Seek to amplify the enjoyment of that.
  • We take our work really seriously,” he said. “We don’t take ourselves too seriously. You have to leave your ego at the door.” Can your organization engage in more playful collaboration?
  • We drive creativity out of kids as they grow older, he noted. “Kids expect to play everywhere,” and so even ordinary elements like sidewalks can turn into experiences. (This reminded me of how people decorate the Porta-Potties at Burning Man with lights and music… although I wouldn’t necessarily do the same thing to the restrooms at my university, it did make me think about how we might make ordinary places or situations more fun for our students.)

KaBOOM! is such a unique organization that I had to ask James: what’s the most amazing thing you’ve ever observed in your role as President? He says it’s something that hasn’t just happened once… but happens every time KaBOOM! organizes a new playground build. When people from diverse backgrounds come together with a strong shared mission, vision, and purpose, you foster intense community engagement that yields powerful, tangible results — and this is something that so many organizations strive to achieve.

If you haven’t made plans already to hear James and the other storytellers at BIF, there may be a few tickets left — but this event always sells out! Check the BIF registration page and share a memorable experience with the BIF community this year: http://www.businessinnovationfactory.com/summit/register

3 Steps to Creating an Innovative Performance Culture

Image Credit: Doug Buckley of http://hyperactive.to

Image Credit: Doug Buckley of http://hyperactive.to

Want to leapfrog over your competitors by designing an extremely high-performance culture for your organization? If so, I have the secret formula.

It starts here: in his August post to ASQ’s View From the Q blog, guest blogger James Lawther asks:

What are your DOs and DON’Ts of creating a performance culture?

Citing Deming and Drucker, and noting how so many organizations rely on a “carrots and sticks” approach to performance management, he converges on the following recommendation: “The way to create a high performance culture is to seek out poor performance, embrace it and fix it, not punish it.” I think, though, that this is not a new approach… rather than improving upon poor performance, why don’t we seek out truly amazing performance and then just make more of it? These three steps will help you do it:

  • Eliminate power relationships. Power is poison! It creates and cultivates fear (which, according to Deming, we need to drive out). Unfortunately, our educational system and our economy are firmly steeped in power relationships… so we’re not accustomed to truly cooperative relationships. (In fact, being reliant on the income from our jobs shoehorns us into power relationships before we even begin working.) Holacracy is one approach that some organizations are trying out, but there are many possibilities for shifting from organizational structures that are designed around power and control, versus those that are designed to stimulate interest, creativity, and true collaboration.
  • Create systems to help everyone find (and share) their unique skills, talents, and gifts. This is the key to both engagement and high performance — and this isn’t a one-shot deal. These skills, talents, and gifts are extremely dependent on the organizational context, the external environment, and a person’s current interests… and all of these change over time!
  • Create systems to help people become stewards of their own performance. Accenture and Google have both recently given up performance reviews… and Deming has always warned about them! Unless we’re managing our own performance, and the process and outcomes are meaningful to us individually, we’ll just be dragged down by another power relationship.

Quality professionals are great at designing and setting up systems to achieve performance goals! Now, we have an innovation challenge: adopt the new philosophy, design quality systems that substitute community in place of power and control, and use our sophisticated and capable information systems to give people agency over their own performance.

“Creative teamwork utterly depends on true communication and is thus very seriously hindered by the presence of power relationships. The open-source community, effectively free of such power relationships, is teaching us by contrast how dreadfully much they cost in bugs, in lowered productivity, and in lost opportunities.” — E. S. Raymond in The Cathedral and the Bazaar

A Chat with Jaime Casap, Google’s Chief Education Evangelist

jaime-casap-head

“The classroom of the future does not exist!”

That’s the word from Jaime Casap (@jcasap), Google’s Chief Education Evangelist — and a highly anticipated new Business Innovation Factory (BIF) storyteller for 2015.  In advance of the summit which takes place on September 16 and 17, Morgan and I had the opportunity to chat with Jaime about a form of business model innovation that’s close to our hearts – improving education. He’s a native New Yorker, so he’s naturally outspoken and direct. But his caring and considerate tone makes it clear he’s got everyone’s best interests at heart.

At Google, he’s the connector and boundary spanner… the guy the organization trusts to “predict the future” where education is concerned. He makes sure that the channels of communication are open between everyone working on education-related projects. Outside of Google, he advocates smart and innovative applications of technology in education that will open up educational opportunities for everyone.  Most recently, he visited the White House on this mission.

jaime-quote-image

The current system educational system is not broken, he says. It’s doing exactly what it was designed to do: prepare workers for a hierarchical, industrialized production economy. The problem is that the system cannot be high-performing because it’s not doing what we need it to for the upcoming decades, which requires leveraging the skills and capabilities of everyone.

He points out that low-income minorities now have a 9% chance of graduating from college… whereas a couple decades ago, they had a 6% chance. This startling statistic reflects an underlying deficiency in how education is designed and delivered in this country today.

So how do we fix it?

“Technology gives us the ability to question everything,” he says.  As we shift to performance-based assessments, we can create educational experiences that are practical, iterative, and focused on continuous improvement — where we measure iteration, innovation, and sustained incremental progress.

Measuring these, he says, will be a lot more interesting than what we tend to measure now: whether a learner gets something right the first time — or how long it took for a competency to emerge. From this new perspective, we’ll finally be able to answer questions like: What is an excellent school? What does a high-performing educational system look (and feel) like?

Jaime’s opportunity-driven vision for inclusiveness  is an integral part of Google’s future. And you can hear more about his personal story and how it shaped this vision next month at BIF.

If you haven’t made plans already to hear Jaime and the other storytellers at BIF, there may be a few tickets left — but this event always sells out! Check the BIF registration page and share a memorable experience with the BIF community this year: http://www.businessinnovationfactory.com/summit/register

Data Quality is Key for Asset Management in Data Science

This post was motivated by two recent tweets by Dr. Diego Kuonen, Principal of Statoo Consulting in Switzerland (who you should definitely follow if you don’t already – he’s one of the only other people in the world who thinks about data science and quality). First, he shared a slide show from CIO Insight with this clickbaity title, bound to capture the attention of any manager who cares about their bottom line (yeah, they’re unicorns):

“The Best Way to Use Data to Cut Costs? Delete It.”

I’m so happy this message is starting to enter corporate consciousness, because I lived it throughout the decade of the 2000’s — working on data management for the National Radio Astronomy Observatory (NRAO). I published several papers during that time that present the following position on this theme (links to the full text articles are at the bottom of this post):

  • First, storing data means you’ve saved it to physical media; archiving data implies that you are storing data over a longer (and possibly very long) time horizon.
  • Even though storage is cheap, don’t store (or archive) everything. Inventories have holding costs, and data warehouses are no different (even though those electrons are so, so tiny).
  • Archiving data that is of dubious quality is never advised. (It’s like piling your garage full of all those early drafts of every paper you’ve ever written… and having done this, I strongly recommend against it.)
  • Sometimes it can be hard to tell whether the raw data we’re collecting is fundamentally good or bad — but we have to try.
  • Data science provides fantastic techniques for learning what is meant by data quality, and then automating the classification process.
  • The intent of whoever collects the data is bound to be different than whoever uses the data in the future.
  • If we do not capture intent, we are significantly suppressing the potential that the data asset will have in the future.

Although I hadn’t seen this when I was deeply enmeshed in the problem long ago, it totally warmed my heart when Diego followed up with this quote from Deming in 1942:

dont-archive-it

 

In my opinion, the need for a dedicated focus on understanding what we mean by data quality (for our particular contexts) and then working to make sure we don’t load up our Big Data opportunities with Bad Data liabilities will be the difference between competitive and combustible in the future. Mind your data quality before your data science. It will also positively impact the sustainability of your data archive.

Papers where I talked about why NOT to archive all your data are here:

  1. Radziwill, N. M., 2006: Foundations for Quality Management of Scientific Data Products. Quality Management Journal, v13 Issue 2 (April), p. 7-21.
  2. Radziwill, N. M., 2006: Valuation, Policy and Software Strategy. SPIE, Orlando FL, May 25-31.
  3. Radziwill, N.M. and R. DuPlain, 2005: A Framework for Telescope Data Quality Management. Proc. SPIE, Madrid, Spain, October 2-5, 2005.
  4. DuPlain, R. F. and N.M. Radziwill, 2006: Autonomous Quality Assurance and Troubleshooting. SPIE, Orlando FL, May 25-31.
  5. DuPlain, R., Radziwill, N.M., & Shelton, A., 2007: A Rule-Based Data Quality Startup Using PyCLIPS. ADASS XVII, London UK, September 2007.

 

« Older Entries Recent Entries »