{"id":2731,"date":"2021-03-12T13:47:01","date_gmt":"2021-03-12T13:47:01","guid":{"rendered":"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/?page_id=2731"},"modified":"2021-03-13T21:58:38","modified_gmt":"2021-03-13T21:58:38","slug":"ar2021-humanizing-ai-filling-the-gaps-with-multi-faceted-research-joel-branch-chief-product-officer-lucd","status":"publish","type":"page","link":"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/ar2021-humanizing-ai-filling-the-gaps-with-multi-faceted-research-joel-branch-chief-product-officer-lucd\/","title":{"rendered":"AR2021-Humanizing AI: Filling the Gaps with Multi-Faceted Research (Joel Branch, Lucd)"},"content":{"rendered":"<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-2732\" src=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck701-300x168.png\" alt=\"\" width=\"464\" height=\"260\" srcset=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck701-300x168.png 300w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck701-1024x573.png 1024w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck701-768x430.png 768w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck701-600x336.png 600w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck701.png 1158w\" sizes=\"(max-width: 464px) 100vw, 464px\" \/><\/p>\n<p>\u2014&gt; Good afternoon! I work at a Lucd, a company that helps other enterprises get their AI roadmaps planned out<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-2733\" src=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck702-300x172.png\" alt=\"\" width=\"526\" height=\"301\" srcset=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck702-300x172.png 300w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck702-1024x588.png 1024w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck702-768x441.png 768w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck702-600x344.png 600w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck702.png 1234w\" sizes=\"(max-width: 526px) 100vw, 526px\" \/><\/p>\n<div>\u2014&gt; There is a lot to be excited about regarding AI<\/div>\n<ul>\n<li>\n<div>Examples like automation in cars that help you stay in your lane, and automated driving<\/div>\n<\/li>\n<li>\n<div>Deep Fakes are an example as well<\/div>\n<\/li>\n<\/ul>\n<div><\/div>\n<div>\u2014&gt; My observation is that AI is uninformed and hurried, resulting in deployments that don\u2019t operate well in real world<\/div>\n<ul>\n<li>\n<div>Reasons for this include infrastructure limits, and a lack of talent<\/div>\n<\/li>\n<\/ul>\n<div><\/div>\n<div>\u2014&gt; But a major reason is a lack of humanized AI<\/div>\n<ul>\n<li>\n<div>Our goal should be creating governance frameworks that democratize the development of AI solutions<\/div>\n<ul>\n<li>\n<div>Not creating interfaces like Google Home or Alexa<\/div>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-2734\" src=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck703-300x168.png\" alt=\"\" width=\"550\" height=\"308\" srcset=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck703-300x168.png 300w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck703-1024x575.png 1024w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck703-768x431.png 768w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck703-600x337.png 600w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck703.png 1236w\" sizes=\"(max-width: 550px) 100vw, 550px\" \/><\/p>\n<div>\u2014&gt; AI is not new, and is as old as computing itself, but lot of breakthroughs in past 25 years have caused advancement and overall popularity of the discipline<\/div>\n<div><\/div>\n<div>\u2014&gt; Popular milestones include :<\/div>\n<ul>\n<li>\n<div>1997, Deep Blue beating Gary Kasparov<\/div>\n<\/li>\n<li>\n<div>2000, ASIMO robot displayed human-like actions<\/div>\n<\/li>\n<li>\n<div>2011, IBM Watson beating former Jeopardy champions<\/div>\n<\/li>\n<li>\n<div>2016, DeepMind beating the world Go champion<\/div>\n<\/li>\n<li>\n<div>2020, GPT-3 program exhibited language capabilities that could mimic human writing<\/div>\n<\/li>\n<\/ul>\n<div><\/div>\n<div>\u2014&gt; AI commoditization has increased as we have moved through time<\/div>\n<ul>\n<li>\n<div>Lot of key AI technology is just an API call away, with minimal code required<\/div>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-2735\" src=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck704-300x169.png\" alt=\"\" width=\"529\" height=\"300\" \/><\/p>\n<div>\u2014&gt; Next, I want to cover next trends in AI value and applications<\/div>\n<div><\/div>\n<div>\u2014&gt; Trends for AI are quite attractive<\/div>\n<ul>\n<li>\n<div>Gartner expects AI augmentation will creation trillions of dollars in business value<\/div>\n<\/li>\n<li>\n<div>What do I mean by AI augmentation?<\/div>\n<ul>\n<li>\n<div>Not chatbot through customer service flow, but rather how augmentation helps business work better, from analyzing data, to writing reports<\/div>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<div><\/div>\n<div><\/div>\n<div>\u2014&gt; AI engineering is also a top tech strategy<\/div>\n<ul>\n<li>\n<div>Developing AI through low to no-code frameworks, and hiring people with deepest AI backgrounds<\/div>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-2736\" src=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck705-300x170.png\" alt=\"\" width=\"501\" height=\"284\" srcset=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck705-300x170.png 300w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck705-1024x579.png 1024w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck705-768x434.png 768w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck705-600x339.png 600w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck705.png 1242w\" sizes=\"(max-width: 501px) 100vw, 501px\" \/><\/p>\n<div>\u2014&gt; Next I want to talk about the state of AI Deployments, from the enterprise perspective<\/div>\n<ul>\n<li>\n<div>Even with hype and capabilities, AI efforts are largely failing<\/div>\n<\/li>\n<\/ul>\n<div><\/div>\n<div>\u2014&gt; Companies report little to no impact from AI, and close to 90% of data science projects and don\u2019t make into production<\/div>\n<p>&nbsp;<\/p>\n<div><\/div>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-2737\" src=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck706-300x147.png\" alt=\"\" width=\"444\" height=\"219\" srcset=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck706-300x147.png 300w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck706-1024x502.png 1024w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck706-768x377.png 768w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck706-600x294.png 600w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck706.png 1170w\" sizes=\"(max-width: 444px) 100vw, 444px\" \/><\/p>\n<div>\u2014&gt; So why is enterprise AI failing?<\/div>\n<div><\/div>\n<div>\u2014&gt; For many reasons, and not just infrastructure problems<\/div>\n<ul>\n<li>\n<div>Differentiating factor and problems<\/div>\n<\/li>\n<\/ul>\n<div><\/div>\n<div>\u2014&gt; Starting point is a lack of organizational support<\/div>\n<ul>\n<li>\n<div>AI limited to certain projects<\/div>\n<\/li>\n<li>\n<div>There is a limited supply of AI talent<\/div>\n<\/li>\n<li>\n<div>AI Development teams are composed of near exclusively AI-related talent<\/div>\n<ul>\n<li>\n<div>In my prior work at ESPN, my own AI work was isolated from the rest of ESPN\u2019s organization<\/div>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-2750\" src=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/orgsupport-300x158.png\" alt=\"\" width=\"465\" height=\"245\" srcset=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/orgsupport-300x158.png 300w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/orgsupport-1024x540.png 1024w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/orgsupport-768x405.png 768w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/orgsupport-600x316.png 600w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/orgsupport.png 1112w\" sizes=\"(max-width: 465px) 100vw, 465px\" \/><\/p>\n<div>\u2014&gt; Next I want to talk about the state of AI Deployments, from the enterprise perspective<\/div>\n<ul>\n<li>\n<div>Even with hype and capabilities, AI efforts are largely failing<\/div>\n<\/li>\n<\/ul>\n<div><\/div>\n<div>\u2014&gt; Companies report little to no impact from AI, and close to 90% of data science projects and don\u2019t make into production<\/div>\n<p>&nbsp;<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-2739\" src=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck709-300x170.png\" alt=\"\" width=\"462\" height=\"262\" srcset=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck709-300x170.png 300w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck709-1024x581.png 1024w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck709-768x436.png 768w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck709-600x340.png 600w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck709.png 1220w\" sizes=\"(max-width: 462px) 100vw, 462px\" \/><\/p>\n<div>\u2014&gt; There are also adoption challenges,<\/div>\n<ul>\n<li>\n<div>Close to 50% of effort is spent on data access and cleaning<\/div>\n<\/li>\n<\/ul>\n<div><\/div>\n<div>\u2014&gt; There is black-box decision-making<\/div>\n<ul>\n<li>\n<div>Lack of transparency and rationality in AI output produced<\/div>\n<\/li>\n<li>\n<div>AI, for context, works as probabilistic learning, with no hard or fast decisions<\/div>\n<ul>\n<li>Done with network of nodes and layers<\/li>\n<\/ul>\n<\/li>\n<li>\n<div>Understanding why AI made a decision is hard to understand<\/div>\n<\/li>\n<li>\n<div>Information goes in, something comes out, but no clear idea why<\/div>\n<\/li>\n<\/ul>\n<div><\/div>\n<div>\u2014&gt; AI Bias<\/div>\n<ul>\n<li>\n<div>Examples of race bias in online ads, job postings, etc.<\/div>\n<\/li>\n<li>\n<div>Fueled by overly technically driven solutions, and having a lot of engineering staff at the table<\/div>\n<ul>\n<li>But this is missing non-technical rules that can handle bias<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-2740\" src=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck710-300x171.png\" alt=\"\" width=\"494\" height=\"283\" srcset=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck710-300x171.png 300w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck710-600x342.png 600w\" sizes=\"(max-width: 494px) 100vw, 494px\" \/><\/p>\n<div>\u2014&gt; Example of Gender Shades project which analyzed accuracy of facial recognition companies like Microsoft, Face ++, IBM<\/div>\n<ul>\n<li>\n<div>Evaluated accuracy of algorithms, and data-set to balance among gender and skin types in training data used by AI<\/div>\n<\/li>\n<\/ul>\n<div><\/div>\n<div>\u2014&gt; Exposure of biased performance resulted in internal reviews at Microsoft and IBM<\/div>\n<div><\/div>\n<div>\u2014&gt; The results?<\/div>\n<ul>\n<li>\n<div>Some facial detection algorithms do better with men rather than women<\/div>\n<\/li>\n<li>\n<div>Unfairness in lighter\/darker color subject is significant going from 97%, to 75% accuracy<\/div>\n<\/li>\n<li>\n<div>Bias problem gets worse, the more features are interested<\/div>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-2741\" src=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck711-300x170.png\" alt=\"\" width=\"471\" height=\"267\" srcset=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck711-300x170.png 300w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck711-1024x580.png 1024w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck711-768x435.png 768w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck711-600x340.png 600w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck711.png 1228w\" sizes=\"(max-width: 471px) 100vw, 471px\" \/><\/p>\n<div>\u2014&gt; AI efforts are also failing through an immature development process<\/div>\n<ul>\n<li>\n<div>Short process with immature orgs\/deparments that follow on process<\/div>\n<\/li>\n<\/ul>\n<div><\/div>\n<div>\u2014&gt; There\u2019s straightforward data collection, but with biased or insufficient data<\/div>\n<div><\/div>\n<div>\u2014&gt; People then directly create a model, without explainability (on their laptop, with little expert oversight)<\/div>\n<ul>\n<li>\n<div>Lot of work is not tracked, unversioned one<\/div>\n<\/li>\n<li>\n<div>Little to no audit trail<\/div>\n<\/li>\n<\/ul>\n<div><\/div>\n<div>\u2014&gt; Goes right into deployed system and \u00a0begins causing a mess<\/div>\n<ul>\n<li>\n<div>AI is inaccurate, unfair, or non-trustworthy<\/div>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-2742\" src=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck712-300x171.png\" alt=\"\" width=\"518\" height=\"297\" \/><\/p>\n<div>\u2014&gt; To recap, we talked about why you should care about AI, even with just a financial point of view<\/div>\n<div><\/div>\n<div>\u2014&gt; Then discussed why AI is failing<\/div>\n<ul>\n<li>\n<div>Largely caused by technical talent being left to their own devices<\/div>\n<\/li>\n<\/ul>\n<div><\/div>\n<div>\u2014&gt; So, what do we do?<\/div>\n<p>&nbsp;<\/p>\n<div><\/div>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-2743\" src=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck713-300x169.png\" alt=\"\" width=\"492\" height=\"279\" \/><\/p>\n<div>\u2014&gt; First order, address development workflow shown earlier<\/div>\n<ul>\n<li>\n<div>Overall, benefits of MLOps (borrowed from DevOps) offers governance, accountability, clear stakeholder responsibilities, to manage AI deployment<\/div>\n<\/li>\n<li>\n<div>Distinctly have phase for data analysis<\/div>\n<ul>\n<li>\n<div>Have phase for validating data, if data is invalid or inaccurate can go backwards<\/div>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<div>Also work to prepare data and make sure the data that goes into model is good for production<\/div>\n<\/li>\n<\/ul>\n<div><\/div>\n<div>\u2014&gt; Then you finally have ML system deployment<\/div>\n<ul>\n<li>\n<div>Still this is only part of the solution, as guidelines emphasize roles of engineers<\/div>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-2751\" src=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/humanization-300x168.png\" alt=\"\" width=\"620\" height=\"347\" srcset=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/humanization-300x168.png 300w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/humanization-1024x575.png 1024w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/humanization-768x431.png 768w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/humanization-600x337.png 600w, https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/humanization.png 1168w\" sizes=\"(max-width: 620px) 100vw, 620px\" \/><\/p>\n<div>\u2014&gt; We expand number of people brought into ML Ops process, and add people to mix, including a product researcher and AI ethicist to make sure it\u2019s all developed responsibility with target customer in mind<\/div>\n<div><\/div>\n<div>\u2014&gt; How do we get to include these folks?<\/div>\n<ul>\n<li>\n<div>Enter product researcher and HCI researcher<\/div>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-2745\" src=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck715-300x173.png\" alt=\"\" width=\"503\" height=\"292\" \/><\/p>\n<div>\u2014&gt; \u00a0Relevant challenges<\/div>\n<ul>\n<li>\n<div>Exploding amount of data, and dimensions associated with data<\/div>\n<ul>\n<li>\n<div>Data is getting bigger and wider<\/div>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<ul>\n<li>\n<div>Intersectional bias (hard to identify)<\/div>\n<\/li>\n<\/ul>\n<div><\/div>\n<div>\u2014&gt; Two solutions<\/div>\n<ul>\n<li>\n<div>Latent Space Data Exploration: New ways to visualize data in multi-dimensional data, and seeing relationships between data, so people can catch bad data<\/div>\n<\/li>\n<li>\n<div>Data Sub-Group Analysis: Identifying sub-groups for comparative analysis through AI<\/div>\n<ul>\n<li>\n<div>Need visualization experts to take audit specialists to essentially look at different groups in AI training data, and compare the results of different data sets as close as possible<\/div>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-2746\" src=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck716-300x161.png\" alt=\"\" width=\"557\" height=\"301\" \/><\/p>\n<div>\u2014&gt; Example of latent space data- Visualizing how digits are grouped together in a latent space, where 12*12 features are grouped<\/div>\n<ul>\n<li>\n<div>It\u2019s then a matter of figuring out if data can be separated<\/div>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-2747\" src=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck717-300x153.png\" alt=\"\" width=\"592\" height=\"301\" \/><\/p>\n<div>\u2014&gt; Challenges in HCI<\/div>\n<ul>\n<li>\n<div>Explainability: Lack of explainability of AI models, limits capacity of deployment and evaluation<\/div>\n<\/li>\n<li>\n<div>MLOps requires a lot of automation, but this process is still premature, and requires people to have AI domain knowledge<\/div>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-2748\" src=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck718-300x155.png\" alt=\"\" width=\"488\" height=\"254\" \/><\/p>\n<div>\u2014&gt; Next, I&#8217;ll show a slide on AI explainability<\/div>\n<ul>\n<li>\n<div>Left-hand side shows features and values. Visual of AI model that listed out features key to making its decisions<\/div>\n<\/li>\n<\/ul>\n<div><\/div>\n<div>\u2014&gt; We just don\u2019t want explainability for ML engineers, but want to get causality as well<\/div>\n<div><\/div>\n<ul>\n<li>\n<div>HCI experts need explainability framework to create something cognitively understandable for humans<\/div>\n<ul>\n<li>\n<div>Includes metrics of explainability (i.e. \u00a0goodness of explanations)<\/div>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<div><\/div>\n<div>\u2014&gt; Can integrate AI explanations with knowledge graphs or ontologies, and interpret AI into something humans can understand<\/div>\n<ul>\n<li>\n<div>Or augmenting training data used for AI models with features that can be expressed by a human<\/div>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-2749\" src=\"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-content\/uploads\/sites\/6\/2021\/03\/d2deck719-300x145.png\" alt=\"\" width=\"501\" height=\"244\" \/><\/p>\n<div>\u2014&gt; Humane AI is necessary, important, as it\u2019 s maturing, and many open problems exist and require a cross disciplinary focus<\/div>\n<div><\/div>\n<div>\u2014&gt; Thank you!<\/div>\n<div><\/div>\n<p>&nbsp;<\/p>\n<div><\/div>\n<div><span style=\"text-decoration: underline\">Q&amp;A<\/span><\/div>\n<div><\/div>\n<ol>\n<li><span style=\"font-weight: 400\">Is there a meta-pattern or algorithm for QA-ing AI systems for other features (other than biological features)?<\/span><\/li>\n<\/ol>\n<div><\/div>\n<div>A: There are explainability techniques that are specifically for medical\/biological arena, and for Natural Language processing<\/div>\n<div><\/div>\n<div>\u2014&gt; There might be other QA techniques, but can\u2019t name any right now<\/div>\n<div><\/div>\n<div><\/div>\n<p>2. <span style=\"font-weight: 400\">Is there a heuristic that you use to help you look for problems that only reveal at scale?<\/span><\/p>\n<div><\/div>\n<div>A: What he\u2019s seen is that discipline of monitoring of models in production.<\/div>\n<ul>\n<li>\n<div>If you have model in production, do a warm-start of a new model<\/div>\n<ul>\n<li>\n<div>Don\u2019t just start new recommendation engine on January 1st<\/div>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<div>Also do synthetic data analysis (when you don\u2019t have enough training data for particular domain)<\/div>\n<ul>\n<li>\n<div>You can use as much data as you can from Internet as a proxy<\/div>\n<\/li>\n<li>\n<div>Be willing to give models time, and a distribution of daily data<\/div>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<div><\/div>\n<div>3. <span style=\"font-weight: 400\">Not sure if I\u2019m following this correctly, is latent space visualization effectively just visualization of parameters identified by exploratory factor analysis<\/span><\/div>\n<div><\/div>\n<div>\u2014&gt; No, latent space focuses on compression \u00a0of data such as text data or image data<\/div>\n<ul>\n<li>\n<div>Looks at how you can compress multi-dimensional data into a simpler-dimension data (i.e analyzing three dimensions instead of thirty dimensions )<\/div>\n<\/li>\n<li>\n<div>Look at PCA to see what it means to go from high to low dimensional space for latent sapce<\/div>\n<\/li>\n<\/ul>\n<div><\/div>\n<div>\u2014&gt; You are visualizing features, not parameters<\/div>\n<ul>\n<li>\n<div>Technique isn\u2019t perfect, in that you have to work with parameters in order to see if there are ways data can be clustered in any way<\/div>\n<ul>\n<li>\n<div>Techniques exist to help automate that<\/div>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<div><\/div>\n<div>4. <span style=\"font-weight: 400\">What\u2019s our current knowledge about AI trust? What do we know and the gap?<\/span><\/div>\n<div><\/div>\n<div>A: It\u2019s evolving. Hard to say what we know about AI trust, since it\u2019s evolving<\/div>\n<ul>\n<li>\n<div>Tools we have now fall into combination of explainability analysis and visualizing explainability<\/div>\n<\/li>\n<\/ul>\n<div><\/div>\n<div>\u2014&gt; Ultimate goal of AI trust is for human being to ask question to the AI, and the AI to explain to the human convincingly why it gave a specific answer to them over other answers.<\/div>\n<ul>\n<li>\n<div>Needs to have back and forth between person and AI<\/div>\n<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>\u2014&gt; Good afternoon! I work at a Lucd, a company that helps other enterprises get their AI roadmaps planned out &nbsp; &nbsp; \u2014&gt; There is a lot to be excited about regarding AI Examples like automation in cars that help you stay in your lane, and automated driving Deep Fakes are an example as well [&hellip;]<\/p>\n","protected":false},"author":334,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_seopress_robots_primary_cat":"","footnotes":""},"acf":[],"_links":{"self":[{"href":"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-json\/wp\/v2\/pages\/2731"}],"collection":[{"href":"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-json\/wp\/v2\/users\/334"}],"replies":[{"embeddable":true,"href":"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-json\/wp\/v2\/comments?post=2731"}],"version-history":[{"count":4,"href":"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-json\/wp\/v2\/pages\/2731\/revisions"}],"predecessor-version":[{"id":3044,"href":"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-json\/wp\/v2\/pages\/2731\/revisions\/3044"}],"wp:attachment":[{"href":"https:\/\/rosenfeldmedia.com\/advancing-research-2021\/wp-json\/wp\/v2\/media?parent=2731"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}