Podcast: Machine Learning’s Role in Workplace Inclusion

Intro: [00:00:01] Welcome to the Workology podcast a podcast for the disruptive workplace leader. Join host Jessica Miller-Merrell, founder of Workology.com as she sits down and gets to the bottom of trends tools and case studies for the business leader HR and recruiting professional who is tired of the status quo. Now here’s Jessica with this episode of Workology.

Jessica: [00:00:26] Welcome to a new series on the Workology podcast that we’re kicking off that focuses on the future of work. This series is in collaboration with the Partnership on Employment and Accessible Technology or PEAT. You can learn more about PEAT at peatworks.org.

Jessica: [00:00:43] The economy is strong and there is an assumption that everyone who wants to find work can be gainfully employed quickly and easily. However a 2015 study by the annual disability statistics compendium reported that while employment for people without disabilities is 75.4 percent the employment rate for people with disabilities is 34.4 percent. Crowd work, Also known the gig economy could be the answer for providing people with disabilities an opportunity to generate income while also providing them flexibility. Welcome to the Workology podcast. We are continuing a series on the Future of Work. This series is in collaboration with the Partnership. on Employment and Accessible Technology or PEAT. Today I’m joined with Jeff Bigham. He is an associate professor at Carnegie Mellon University. Jeff welcome to the Workology podcast.

Jeff: [00:01:37] Great to be here. Thanks for having me.

Jessica: [00:01:39] Can you talk with us a little bit about your background?

Jeff: [00:01:42] Sure. I’m a computer scientist by training and so I got my undergraduate and graduate degrees in computer science. I started out my Ph.D. work thinking that I wanted to work on our artificial intelligence. So trying to make computers law eventually as intelligent as people are able to help people with very difficult task. And I started that. But a bit of the way through I kind of got frustrated with what we were able to do with artificial intelligence with automation.

And I wanted to start you know helping real people and I happened to connect with a professor at the University of Washington where I who ended up being my advisor on projects around accessibility so trying to make computers that work well for people with disabilities. This turns out was a really great way for me to bring in the work that I was really excited about technically the artificial intelligence work into real tools that helped real people in this case people with disabilities. The same way I got connected with this is that I also kind of eventually got frustrated with what the artificial intelligence technology would allow us to do when we were building tools.

And so I started augmenting what we were doing with the computer with the intelligence of real people so people we would find out on the web who we could pay a little bit of money to kind of make our computers smarter. And so you know algorithms wouldn’t have to just be what the computer could do that could also be what the computer with people on the web could do to support people with disabilities. And so this ended up eventually being called human computation or crowdsourcing. And in the last few years it’s really connected with this idea of the gig economy where you hire people for small amounts of time to do bits of work for you. And so I guess that’s kind of my background the short abbreviated version of how I got here.

Jessica: [00:03:29] Great. And it’s thank you for setting the stage because we are talking about the gig economy and its accessibility. Can you talk a little bit more about your research and some some of the work that you’ve done because I’ll have some links in the transcript of the podcast? It’s very interesting around the hearing impaired and the sight impaired.

Jeff: [00:03:49] Yeah sure. So some of the first projects that I did at the intersection of computer science and the gig economy were around supporting people who are blind. And so we had an iPhone application that a blind person could use to take a picture of something they’d like to know about. Ask a question of what they’d like to know and then we would get an answer back to them within about 30 seconds. So this was an application called the ISS and it was available online the App Store. We had about ten or fifteen thousand users and they were 100,000 different questions. And what was interesting about it as you might have guessed is that we weren’t doing this with with computers alone because even though computer vision which is kind of the field that tries to do that sort of thing automatically has gotten a lot better.

The technology can’t yet answer arbitrary questions about anything in the world. So we were recruiting people from the web paying you know a small amount of money for a small amount of their time to answer these questions. And so that’s one of the first projects that got me started is relatively simple and since then we’ve done all sorts of different things so we had similar projects where we’ve been converting speech and say a lecture two words on a screen. So captioning the lecture deaf or hard of hearing people and doing that not with automatic technology which still doesn’t work well enough to do that to do it reliably enough but using people again we were crowded out on the web and in this case we had to get a group of them because it turns out it’s really hard to type that natural speaking rates.

We had to get a bunch of them together all kind of contributing to this task and then stitch those things back together into one final captioning stream that the person could look at to follow the lecture. So those are kind of two different projects and we’ve got a whole bunch of other things but that’s what kind of got me excited about it is because you know I kind of grew up with this notion that of all this great science fiction technology is going to be coming out. But you know it’s kind of slow to get here and it even though I’m really impressed by what we’ve been able to do in artificial intelligence and computer science in achieving some of that. It turns out that by bringing in people you’re able to get there sooner and kind of understand like what we’d actually want to build and get it out to people to benefit to benefit them in their lives.

Jessica: [00:05:58] I think sometimes we forget about the people aspect we just assume we make an assumption that the technology is just going to do exactly what we want it to do like close captioning. Live stream captioning I think is particularly important with the popularity of livestream video that you see all over with social media and the web, but those things the technology hasn’t caught up yet. So you’re saying that there are a lot of human sort of computation elements to those areas.

Jeff: [00:06:26] So it’s still it’s still true that for say closed captioning if you see closed captioning on a television it’s still done by people. There have obviously been lots of work that’s tried to make it done ought to be done automatically and they’re kind of getting better and there’s ways you can kind of use both people and the technology to make it on to get it done a little cheaper. But still it’s the people behind the scenes and I think this is true of a lot of the technology that we see all of this great stuff that’s out there and we assume that it must be just that the computers are very smart but in lots of different ways it’s often people that are that are powering it behind the scenes in a way that makes it seem like it’s just the computer. But almost all of this is some interesting mix of people and computers working together.

Jessica: [00:07:11] Well, let’s go back and talk a little bit about Crowd work and maybe the benefits of crowd work for individuals with disabilities.

Jeff: [00:07:19] Yes, so I mean one of the big differences between what we call it crowd work or gig gig work is the flexible nature of it. And so the way these these marketplaces usually work is that you know you can choose to sign up to be part of the pool of available workers and then when work comes in you can in different ways either choose to accept it or not. And so this is maybe people would be more familiar with how this works and say like an Uber case right. So you can sign up to be an Uber driver and you know you can decide when you’re going to work and you can decide whether or not you’re going to take a ride. When and when a request comes in and there’s a little bit of a you know Uber’s case in particular you know they do various things to encourage you to work at certain times and you know discourage you from not accepting rides but generally that’s that’s the idea. You can sign up to be a worker take work when it’s available if you want.

This is true with lots of different types of work and so online you can sign up to be a worker on various platforms. One of the ones that we work with a lot is called Mechanical Turk which is run by Amazon so the same Amazon that you you might buy a lot of products with and have them shipped to your home. With Mechanical Turk you can sign up to be a worker. And in this case you’re not driving a car for them you’re doing a little job. So you’re taking a bit of audio and you’re converting it to text or you’re looking at an image and you’re saying like what’s in that image or you’re looking at an image of a receipt when you’re kind of transcribing that kind of start to imagine what these are being used for. So maybe some of this is being used to keep track of expenses or something anyway so there’s all these little jobs you can sign up for.

The parts of this that might be beneficial to some people with disability is the idea that it’s flexible that you can kind of work when you want you know you don’t necessarily have to  put in a straight eight hour standard workday. Maybe you can kind of work and as you’re able over time you know you don’t. You don’t have to travel so the big promise of digital labor was we wouldn’t have to spend as much time commuting which can be much more difficult for some people with disabilities. So for all these reasons it might be beneficial. This idea of kind of more flexible work. Of course there’s also problems which I imagine we’ll get into but those are kind of the benefits that we see that could be could be beneficial.

Jessica : [00:09:43] Let’s talk about maybe some of the downsides. Which one of them that I see is that some of these tasks are really entry entry level and sometimes really low paying or what are some of the other downsides that you see.

Jeff: [00:09:59] So this has been a this has been a problem with these platforms the marketplaces for this sort of work. And part of it’s due to the fact that because anyone can sign up there’s not really a long history that each employer would have with each worker and so they don’t really have a sense of what expertise that worker brings.

You know they really can’t as easily put up jobs that require certain expertise or knowledge about the employer or the work that’s being done. And so in general it’s been kind of the lower level work. These various efforts to try to improve this so to push more toward the expert work so to enable employers to have a better sense of who they’re hiring to even enable workers to kind of have a longer longer history or build a longer history with with employers and certainly the wage issue across all of these platforms is a big concern. And it’s not really clear. It’s essentially before networked analogy you know there were still contractors.

Jeff: [00:10:57] It’s just that it tended to not be worth being a contractor for you know a minute long job right. That wouldn’t have really made sense. But now with network technology it’s actually pretty easy to connect to a worker who’s willing to do a minute long job for you. And unfortunately, our our legal framework has not quite caught up to that. So one difference between a true contract worker and a traditional worker is you know there’s not the same types of guarantee of pay and benefits and things like this.

And we’re already seeing this kind of shakeout where maybe certain types of gig labor is not is not actually a contract and know the courts are kind of figuring that out. I hope that we figure out some of that because I think there’s a lot of promise in that. We talked about the benefits. These are these are problems for everyone who would be a crowd worker. I think that people with disabilities face a couple of other interesting challenges.

So in particular it turns out that one nice thing about traditional work at least in the United States is that if you have a disability or you can request or are guaranteed certain types of accommodations from your employer. This is not as easily true with with the gig work where if you’re working with somebody for just a minute or two it’s not really clear who’s responsible for providing accommodations. We’re also seeing because kind of one can just post work and you know goes up for a little while and come back down. We’re seeing that a lot of that work is created and kind of presented in a way that is not accessible so that people with certain disabilities cannot do it even though they’d be perfectly capable of doing the work.

Jeff: [00:12:46] And so there’s these different challenges and it’s kind of difficult to even get a handle on it because as opposed to you know a large company where you could go if you saw over time a pattern of that large company not accommodating people with disabilities as they are required to do. You could actually go talk to that company. You could potentially even go through legal means to try to force them to do what they’re supposed to do with these short jobs. People post things and take them down. It’s really difficult to even know who to talk to or to have. The chance to find the times. So that is up there for a little while. And so it presents these new problems and I think this is something we’re still getting a handle on. But I think it’s really important to get ahead of because we we believe and it seems as though this sort of work is and will continue to become more and more popular. Maybe not everyone will be on mechanical turk worker but it seems like the idea of flexible labor is really attractive to employers.

And so we imagine that they’re going to try to keep pushing this to be more and more of their workforce. And so we need to get ahead of it from lots of different means both the kind of how do we bring in some of the nice qualities of traditional work into this like you know building expertise over time guarantees of pay other things like that that would apply to everyone. And then also things like how do we make sure that people with disabilities in particular are not shut out of this increasingly big part of what work is.

Jessica: [00:14:16] I think this is a interesting dilemma. I do really hope that the public policy right will have some law or some court decisions that might help influence us to move in that direction.

I would also like to see companies say if I’m maybe using a Mechanical Turk or some of these other web based platforms that are focused on the gig economy maybe really taking some time and putting some effort into thinking about how they can make their gig’s or their projects or tasks attractive to people with disabilities so that they’re kind of helping to further this this area.

Jeff: [00:14:58] Yeah yeah I think I think that it’s one of these things where I guess over the years that I’ve worked in accessibility which has not been that long but after 15 or 20 years you know I’ve been I’ve become a little bit pessimistic about that sort of approach working at scale so you always have a handful of companies that for a variety of reasons either personal connection or you know maybe even a prior lawsuit will go above and beyond their peers.

Jeff: [00:15:25] And I think that’s awesome and that should be supported and we should be celebrating people like that. But in general I think that there are still lots of reasons why it doesn’t happen like lack of awareness and we can try to build awareness but lack of awareness or you know it’s always seen as even though I think there’s arguments against this that you know it’s this extra cost that a company claims they cannot afford or it’s something that they intend to do but they’ll get around to it later. And so I think that it’s hard. And even even the you know changing policy doesn’t necessarily change this I mean you know there’s policy but then there’s enforcing that policy there’s kind of interpreting what it means to be accessible and at what point it needs to be accessible and it needs to be accessible to me with lots of difficulties in this.

I mean as researchers one of the things that we have done is try to build tools that make it easier for people without expending extra effort or resources to make things that are just out of the box accessible or to have people who come to a task have them be able to trigger something that then reformat that task so that it’s more accessible to them. And so I think there’s lots of different ways that this can be pushed pushed on. But I do think it’s something that you know it used to be we were concerned I think very legitimately concerned that the Web was not as accessible as it could be. And now the Web is taking on a greater and greater role in everything in our lives including now employment. I think that it’s becoming more and more important that we figure out how to do something to solve this.

Jessica: [00:16:58] Let’s take a little bit of a reset here. This is Jessica Miller-Merrell and you’re listening to the Workology podcast in partnership with PEAT. Today and we are talking about machine learning and inclusion with Jeff Bigham. You can connect with Jeff on Twitter at Jeff Bigham @ jeffbigham.

Announcer: [00:17:15] The Workology Podcast Future of Work series is supported by PEAT. The partnership on Employment and Accessible Technology. PEAT’s initiative is to foster collaboration and action around accessible technology in the workplace. Is funded by the U.S. Department of Labor’s Office of Disability Employment Policy ODEP. Learn more about PEAT @peatworks.org. that’s p e a t w o r k s dot org.

Jessica: [00:17:45] Jeff, in one of your recent research articles which I’ll include in the transcript of the podcast you mention the importance of these crowd workers to skill up. Can you talk about why this is important and maybe what they can do?

Jeff: [00:18:00] Yeah so this is this is in general connecting back to that theme that we talked about of you know how do we how do we make it so that crowd work which does have this kind of tendency to be short kind of independent you know jobs where where workers are assumed not to have any particular skill because that skill might not be known. How do we gradually make it so that more of the qualities of traditional work are are in crowd work.

And so one of the things that is nice about at least good employment is that you can over time build expertise in that employment and that over time you can kind of skill up to maybe new jobs new roles maybe greater responsibility. And so as a first step at this we were trying to see if we could build in two tasks that workers are already doing trainings so that they that workers were be able to graduate into doing higher skilled work that might be better paid that might have better prospects long term for them.

And so it’s a really hard task but one of the things we were trying to do was in the context of just audio transcription so converting an audio file into text. It turns out you can do this as a non expert worker so who is someone who doesn’t have a lot of training in this. But it’s really slow so if you know there’s an hour of audio or there’s a half hour of this podcast and a non expert worker who can type and and listen to the podcast but doesn’t have special skills in doing this. If they try to do it it takes them about four to five times as long as the as the podcast to caption it.

So that that ends up being a long time and they end up not being paid very well to do it. But there’s all these different methods you could use to type faster. And so one of the expert skills there is called stenography. So this is using corded keyboards so typing kind of multiple keys at once typing not just letter by letter but typing what is you can think of as kind of phonemes so like a word would only have a word that might have many letters maybe could be typed with only a couple of chords. So we are teaching workers who didn’t have this special skill.

The chords that they might need to eventually graduate into being a stenographer as they were earning money which seems to be important especially for people on these platforms who for whom money is often the primary motivator. And so you can imagine though extending this to all kinds of things. So you know could you as a crowd worker train to do almost anything that you want or could you earn a college degree while you’re making money as opposed to you know our university system now.

It works very differently where you put out a bunch of money you know take out a lot of loans in many cases and then that over many years you graduate to two high skilled work maybe you could actually earn money while you did it. And so that’s the ideas that we were exploring with that.

Jessica: [00:21:09] And I would think that as as an employer or a company who is working with with good workers if you found someone that was reliable completed their tasks on time and did high quality work that you would want to encourage them to upscale their skills to be able to do more work for you because of your track record for success.

Jeff: [00:21:30] Yeah and I think some of it’s just building in the tools that allow work that will allow employers to even tracks things like that. So the way that many of the platforms are set up right now it’s difficult to tell who’s really doing a good job.

You can figure it out if you put in effort. But one of the reasons why put people put work up on crowdsourcing platforms now is because there’s a lot of work to do. I mean a very standard type of task is you know I might have a million images that I need labeled and I put them up on a crowd platform so that I can get them all label reasonably quickly. The problem once you get to that scale is not only don’t you have you know the internal resources to do the work yourself.

You also may not even have the internal resources to evaluate all of that work at least  in a kind of manual way in which you might easily recognized the best workers. Certainly people build up models of what workers seem to be doing the best job over time and they try to recruit those workers again. But then they may not even have a good idea and this is part of I think where the research is important. Well what’s the next step like. What’s the ladder that will take this worker who’s doing maybe a relatively low level image labeling task. What’s the ladder that gets them to a more interesting you know expert skill over time. And so I think that’s kind of where we’re trying to still figure figure thing about.

Jessica: [00:22:56] Do you think that’s the responsibility of the web based platform to be able to help make suggestions on how to scale up or kind of for like the next sort of level of expertise or is it the responsibility of the employers?

Jeff: [00:23:10] Oh it’s a good question. I mean I think that many of us are thinking about kind of the sort of the ethics of this sort of employment and it seems like in practice the platforms will have a much better opportunity to be able to do something here because you know an employer depending on you know how they use the platform may not even have work at the different levels of skill. They may only have a job you know here and there and so they may not even get the kind of experience with a worker over time that you’d need to make such judgments.

And so I think in practice that might need to be the platform. You know it’s a different question to ask. You know should they. And you know who’s responsible. And these sorts of things. And I do think the platforms should should be both for you know it being the right thing and also just for their long term interests. Think a lot more about workers and how they can better support workers make sure that they’re treated fairly. And all of these things. And so I guess from that perspective it seems like it always has to be the platforms and it probably should be the platforms as well.

Jessica: [00:24:16] I think as competition among platforms grow right there more and more every day trying to get the interest level out of the gig workers to come on over submit their profiles. It would behoove them to help those individuals scale up in those areas so that they can keep having the best highest quality workers with the best skill to offer to to these employers.

Jeff: [00:24:42] Yeah. And I think part of it is you know I think I think there’s there’s two sorts of people that the platforms are trying to attract. One is they’re trying to attract the workers and one of the other is they’re trying to attract the employers and you can probably see in the near term interests of the company that’s trying to do this. They probably see him attracting the employers who were paying the money as being the best for their best use of their time.

But I agree that over the longer term it seems like attracting good workers keeping workers happy making sure that they are improving and and feel satisfied with the platform it seems like that’s the long term when what are some ways art that you’re seeing in terms of these web based platforms that are making their systems and software and technology more accessible to individuals with disabilities.

Jessica: [00:25:34] Well you know I haven’t seen a lot.

Jeff: [00:25:36] I think well you know there is there was one fun example where I think I think there was an early story about how there was Uber driver who was deaf and it got a lot of popular press and I think it was one of especially the most more positive press that that Uber has gotten. And so you know Uber has now put into their app various support for drivers who are deaf. So if you get a driver who is deaf you know there’s an alert you get on your phone and it says Just a heads up your drivers deaf and there’s support for you to communicate back and forth with with the driver.

So I think that’s a really interesting thing. I mean I think it happened my my opinion is that it happened because they just happened kind of accidentally maybe Uber didn’t even expect it but then it did happen and people saw that as a really positive thing. And then when Uber Uber saw that and they built it into their app because of that. And so I think you probably there’s probably more examples of this sort of kind of ad hoc kind of accidental you know stories developing into support. But I would definitely like to see a lot more. I mean I think that most of the platforms really haven’t done much of anything to support people with disabilities in particular.

And especially you know where and when the platform allows requesters to submit jobs in whatever form they want. There’s very little feedback that those and those requesters or employers get about what’s accessible what’s not what’s the expectations for the platforms. And so. So I think there’s these point examples but but really we need to be doing a lot better.

Jessica: [00:27:11] You know hopefully maybe I mean I’m an optimistic that podcasts or conversations like we’re having right now and with some of the other folks is part of this series can help change that because my hope is that these platforms people who are helping build these web based platforms will take a listen and say, “Hey, Oh I hadn’t thought about this particular point of view.”  Maybe we can kind of move the needle a little bit more forward and drive some awareness.

Jeff: [00:27:40] Yeah I think that would be that would be wonderful.

Jessica: [00:27:42] I do want to ask you about some of your research because I thought this was a really interesting work that you’re doing and one of them one of your research was focused on study participants who were blind and they had different responsibilities and tasks that they were required to complete.

Jeff: [00:28:01] But if they didn’t complete the tasks on time or at a particular level they placed blame on themselves for failing to complete the task instead of looking at the access the tech the lack of accessible technology that was for the individuals to be able to use. What suggestions do you have in terms of maybe technology or ways to help increase the performance and productivity for people with disabilities. From your work.

Sure yeah. So there’s been a lot of work that has explored how to make the Web more accessible and more usable for for everyone including people with disabilities. One of the first steps is there are these great guidelines that are put out by the E3 see the world wide web consortium.

The SCAG WCA. So these are basic things that you can do if you’re creating a web page that can make sure that at least in theory people with disabilities will be able to access the web page and use it where it gets tricky and where we’re probing with our research is when the page maybe is technically accessible or maybe it’s not but there’s a usability problem that prevents people with disabilities from being able to not only accomplish their. Task but actually figure out what’s going on. So in the example you mentioned what we’re exploring was this concept of not knowing what you don’t know and so if you go to a web page it’s often kind of tricky to know if if you’re unable to complete a task you expect to be able to do on that Web page then.

Is that because of the information maybe not being there on being there at all? Is it because you can’t figure out how to do it and if you happen to be a person with a disability or in our case then that study a person who’s blind? Is it because the information that you would need to complete the task is actually hidden from you or is inaccessible so maybe you a blind person very often does not use a mouse because you can’t see the cursor?

And so maybe you have to interact with the web page using the mouse. And so it’s there, but you wouldn’t know it’s there because you’d have to use a mouse to figure out that it’s there. And so we were kind of probing at that issue. I think I think it pushes this understanding of what accessibility is to being as much about often about usability as it is accessibility. And so unfortunately on the Web we’re still at the stage where we can’t get people to do even the basic things reliably to make their web pages accessible. But you know we’re still pushing to understand well what are in practice are the problems that people run into how we might resolve those that sort of thing.

Jessica: [00:30:56] Well this has definitely been enlightening for me and I’m thinking about my own Web site and the different things that I might be able to do to make it more accessible to all different kinds of people and and their background. So I appreciate your time today. Where can people go to learn more about you and what you do.

Jeff: [00:31:16] So my web page is my first and last name JefferyBigham.com. So that’s a great place to start. We have all of our research papers up open so that you can just read them if you want and a bunch of fun videos too about the projects we’ve worked on so I’d say people should start there.

Jessica: [00:31:32] Awesome. Well thank you so much for taking the time to talk with us. Appreciate it. Great. Thank you so much.

Jessica: [00:31:38] Inclusion comes in all forms and I’m inspired by Jeff’s research as well as others and the technology that supports these efforts. As the gig economy continues to grow and the war for talent becomes even more competitive companies will need to look beyond the traditional way of hiring recruiting and employment for all types of individuals including people with disabilities.

Exit: [00:31:59] Thank you for joining the work Algy podcast a podcast for the disruptive workplace leader who is tired of the status quo. This is Jessica Miller-Merrell. Until next time you can visit Workology.com to listen to all our previous podcast episodes.