Representative Data Challenges

  • Comments posted to this topic are about the item Representative Data Challenges

  • I am currently going through "o'reilly data science from scratch python".

    During the Naive Bayes explanation and implemention they mention exactly this.

    In the book its called "smoothening" to prevent such things from happening due to incomplete training Data.

  • Well, I'm very disappointed this morning in the slant of this.  Just a couple days ago we had an article that proposed that we probably need to allow more 'failure' to promote learning and improving our product.  Today, suddenly we have to throw up the 'diversity' challenge when there is a technology or software problem.

    Oh, how I wish we could move past having to find the same old complaint of lack of diversity and get on with making things work better.  I don't believe I EVER saw the development of a system that was totally right the first release.

    Steve, not to pick on you, but to make my point, in the years I've been involved with this community and seen your photograph,  I don't think I EVER had a thought regarding the tone of anyone's skin or their ethnicity.  Now suddenly I will forever be aware of your skin tone.

    Perfect example of the concept of 'self-fulfilling prophecy'.  Should have left out the links to the diversity articles. Truth is that I don't give a sh!t what color your skin is.  Please don't you all get mad at me for this.

     

    • This reply was modified 3 years, 10 months ago by  skeleton567.
    • This reply was modified 3 years, 10 months ago by  skeleton567.

    Rick
    Disaster Recovery = Backup ( Backup ( Your Backup ) )

  • I have to second what Rick said. We all have two eyes, two ears, and a brain, all of which work similarly. No ethnicity is different enough that we cannot reproduce with each other. We are much more alike than different, and I am uncomfortable with unsupported assertions to the contrary.

  • GeorgeCopeland wrote:

    I have to second what Rick said. We all have two eyes, two ears, and a brain, all of which work similarly. No ethnicity is different enough that we cannot reproduce with each other. We are much more alike than different, and I am uncomfortable with unsupported assertions to the contrary.

    George, thanks for the support.  Years ago my dear first wife commented 'I don't care what's between your legs.  Show me what's between your ears".

    Rick
    Disaster Recovery = Backup ( Backup ( Your Backup ) )

  • At the end of the day it's an algorithm that fails to find the right answer, is it really a surprise that you can find common features in the failures? I think it's important to remember that machine learning algorithms are designed with an expected failure rate in mind, so we need to account for that when we design a process around them.

  • Here's another thought on our common approach to creating algorithms.    Error reporting has always been one of my pet peeves because so many times we are told that 'something happened', but not 'what happened'.

    Just yesterday afternoon I was setting up a new phone, which is replacing my antique iPhone 4!  The process was failing but the only information provided was "403 forbidden".

    I think a great example of this is often portrayed in the dialog in scenes in surgical stories where the surgeons are always commenting 'Now I'm going to...'.  For decades we have created software that attempts to tell us after something goes wrong.  Why don't we create a rolling self-cleaning log of WHAT WE ARE GOING TO DO instead of WHAT WE DID.  Then at least know the last thing that was attempted instead of having to guess what the algorithm tried to do that failed.

    Rick
    Disaster Recovery = Backup ( Backup ( Your Backup ) )

  • We probably shouldn't discount Steve's experience. Much of the populations of the world where we need to provide uniformly working solutions is of different colors and shades, though designed and built to work on a singular group. I recall the joke about how the automatic towel dispenser didn't work for the guys from India because it only worked for Americans, aka the white guys coding at the bank. Turns out it was a design defect and offensive humor that harmed the team.

  • jschmidt 17654 wrote:

    We probably shouldn't discount Steve's experience. Much of the populations of the world where we need to provide uniformly working solutions is of different colors and shades, though designed and built to work on a singular group. I recall the joke about how the automatic towel dispenser didn't work for the guys from India because it only worked for Americans, aka the white guys coding at the bank. Turns out it was a design defect and offensive humor that harmed the team.

    I'm not in any way discounting Steve's experience.  In your example, the largest failure is in the joke, not the failure itself.

     

    Rick
    Disaster Recovery = Backup ( Backup ( Your Backup ) )

  • You should reread your post, but if you don't see it now, you might never see it.

  • Some interesting observations related to this discussion:

    https://hsm.stackexchange.com/questions/12684/how-did-people-make-things-perfectly-straight

    Rick
    Disaster Recovery = Backup ( Backup ( Your Backup ) )

  • jschmidt 17654 wrote:

    You should reread your post, but if you don't see it now, you might never see it.

    Some people see what they believe.  Others believe what they see.

    Rick
    Disaster Recovery = Backup ( Backup ( Your Backup ) )

  • skeleton567 wrote:

    I'm not in any way discounting Steve's experience.  In your example, the largest failure is in the joke, not the failure itself.

    I have to second Rick again. In your example, I blame bank management. If you cause a workplace disruption around me, I fix it immediately. If they had fired the loudmouth on the spot, that would have set things right very smartly. I sure as heck don't blame the towel dispenser engineers, engineers being notorious for designing bad user interfaces.

  • I am intrigued by your choice of issue to discount. The topic was about design and implementation leaving out vast swaths of the population, which most will agree requires diverse input, and it was met with a stereotype of "I don't see color." I recounted an old joke about poor design and the damage it caused in a diverse environment. You assigned blame to an unknown "management" entity and excused poor design. I now regret contributing here, but the buzz on this morning's slack channel was intriguing.

  • Nevermind, thanks for your comment.

Viewing 15 posts - 1 through 15 (of 22 total)

You must be logged in to reply to this topic. Login to reply