Automatic Redaction of PII

Data privacy and protection has become a hot topic. Across the last few years, as I've worked with customers that deal with the GDPR or other laws, they've become more concerned and careful with their use of data. I find less and less resistance from developers about using sensitive production data in development environments, but still too much.

Using data that's stored in databases or other text files is one thing. What about data in less structured forms? I've dealt with a few customers over the years that recorded customer interactions for various purposes. Call center or financial organizations commonly do this, and sometimes deal with sensitive information. I know I've given my date of birth, credit card number, bank account number, and other data to representatives at various times.

Those calls are often recorded, and the IT staffs have often had to ensure extra security is applied to these files. Not everyone has access to listen for various reasons, but certainly when there is sensitive information inside the audio (or video), this data needs the same protection we'd apply to data in other forms. Providing protection, or redacting the information, isn't an easy task.

I saw recently that the Amazon Transcribe service will now redact some PII information automatically. This service can be configured to automatically remove the information in text, which is fantastic. This is a great way to start to use technology in a safe way to ensure that we have less data  leakage when we re-use data. Certainly people might look at this data to better train reps, but it's also likely someone wold look through transcripts to determine why customers are calling in and use that information to better design applications. In either case, there isn't any need to expose PII data to them.

This doesn't protect against the data inside the audio, but perhaps companies can delete and remove those recordings sooner with transcripts available and more quickly reduce their potential attack surfaces. We'll always have some liability, but reducing that and not unnecessarily creating issues is part of what we want to do when protecting data.

Steve Jones - SSC Editor

Join the debate, and respond to the editorial on the forums

The fourth annual State of Database DevOps report is now available for download. Kendra Little announces the launch in her latest blog. Check out the first high level findings of the report here.

Google presented its system of using deep-learning...

The video recording of my RStudio::conf talk, MLOPS for R with Azure Machine Learning, is now available for streaming thanks to the fine folks at RStudio. The talk begins...

In this post, we will discuss the strengths and capabilities of each service and why you might choose one over the other for all or part of your machine learning workflow.

SQL Server Machine Learning allows you to run R and Python scripts from SQL Server. When SQL Server 2019 was released, Microsoft enhanced the security for this functionality, but it caused some existing code writing to the file system to to break. In this article, Dennes Torres explains the security enhancement and describes three ways to work with it.

Yes, it is true that from SQL Server 2016, Trace Flag 1117 and 1118 are not required. Recently I was hired by a large banking firm to help. First appeared...

In this blog, I would share my knowledge about how to fix error 15580 – Cannot drop the master key because dialog "GUID" is encrypted by it. First appeared on...

Having used Azure CLI to query Azure DevOps to obtain a resultset containing useful information on builds, releases, and so on, it is likely that you will want to consume these results in PowerShell so that you can build upon them. I thought this would be an easy thing to do, but..no.

Tim Radney talks about how to decide whether a move to Azure SQL Database Managed Instance makes sense and, if so, which tier. The post Azure SQL Managed Instance Performance...

According to this post, I rate 2019 as a Failure. ...

As part of its move toward 'role-based training,' ...

In celebration of Women’s History Month starting next week, I was asked to write this blog about my experiences as a woman in the SQL Community, and it really...

My team built a process to load from a couple of b...

Caching data is one way to improve application performance. In this article, Chandra Kudumla explains a service for caching data found in Azure called Azure Cache for Redis. … The...

Click to learn more about author Jack Mardack. Digital transformation is not a one-size-fits-all undertaking. Each organization begins its journey from a unique starting point and a 100% migration...

Knowing what’s happening on your servers is vital to ensuring business continuity. In this session Grant Fritchey and Jeremiah Peschka show you the best way to set up monitoring on your environment, and how Redgate SQL Monitor provides a powerful and fast way of achieving this. Thursday March 12, 3-4pm GMT / 10-11am Central. Can't join us live? register to receive the recording.

The role of the DBA is evolving, with automated builds, cloud and DevOps being the new A,B,C,Ds in the day to day management of databases, you need to up-skill and learn about Database Reliability Engineering. Date and time: Tuesday March 31, 4-5pm BST / 10-11am Central Can't join us live? register to receive the recording.

Join Microsoft MVP Grant Fritchey as he discusses database DevOps in Financial Services with our panel of industry experts. They will explore the similarities and differences in their experiences of DevOps and how that looks for Financial Services in today’s landscape. Thursday March 24, 3-4pm GMT / 10-11am Central. Can't join us live? register to receive the recording.

The BI edition comes to Baton Rouge

SQL Saturday comes to Belgium

I had the need to pull specific revisions of files out of a git repository, save those files, and then execute the contents. This all worked fine until it didn't. I received some complaints that unicode characters in the files we getting mangled, and sure enough they were. But why? In this post I'll explain what happened to me, and ways you can avoid it yourself.

Whether we’re talking nHibernate, Linq, or Entity Framework, the degree of loathing for these tools by most DBAs is really hard to measure. Yet, I’ve come to believe that code generation tools are a very important part of what we do. Further, that they are not evil, or wrong, or bad.

One of the worst things we can do as data analysts is to interpret a regression line as the most important thing on a visual. The important thing here is the per-state set of data points, but our eyes are drawn to the line. The line mentally replaces the data, but in doing so, we lose the noise. And boy, is there a lot of noise.

There really are technology stacks and business use cases that should never be moved off of big iron. Then there’s the rest of us. Chances are very high if...

A thorough examination of the bar chart with tips on how to design nice ones.

Jim Harris explains the relevance of DevOps, DataOps and ModelOps for data analytics practitioners. The post Why Ops-ing is in (DevOps, ModelOps, DataOps) appeared first on SAS Blogs.

DevOps is now a part of the volubility of the DBA. Let us talk about today how Efficient Monitoring is one of the important drivers for DevOps. First appeared on...

It's not only for DevOps-related best practices.

In SQL Server 2019 there have been some big licensing changes about what you can and can’t do on a passive fail-over instance. Especially if you have Software Assurance.

The various ways to do it, using either AlwaysOn Monitoring dashboard in SQL Server Management Studio or Performance Monitor counters.

Intel's security plans sound a lot like "we're going to catch up to AMD."

In this article in the series, Robert Sheldon provides a deep dive into HDDs.

Okay, the title of this blog post could also have been “SUMX returns unexpected results with duplicates”. The results only seem incorrect because an incorrect assumption might have been... The...

Early last year, we first caught wind that Microsoft was working on a new cloud-based printing feature. The idea is simple, bring printing into the modern age with a...

I've mentioned before that I really don't mind working with large tables. They're only slow to use if you're looking through all the rows when trying to find the...

The Land Of Do-Overs Of the new things in SQL Server 2019 that I plan on presenting about, the Batch…

I decided to write this off the back of a conversation I was having the other day around the SOS_SCHEDULER_YIELD wait type. The conversation went something along the lines... The...

Max degree of parallelism (MAXDOP) is a setting in SQL Server that controls how many processors may be used for parallel plan execution. Parallel plan execution is good—it lets SQL Server make the best use of all those processors in modern servers. However, MAXDOP can be configured incorrectly. Too much or too little parallelism can negatively impact server performance.

Jeff Iannucci shares a little something for those of you who don’t want Query Store drowning in all your ad hoc nonsense.

In some cases locks are serializable, and if you don’t maintain your indexes you may find run-of-the-mill modifications taking quite a long time.

Erik Darling presents a series of videos tackling each of the signs that your execution plan isn't revealing the whole truth.

I recently was updating my PBIX files to use the Incremental refresh. I ran into a timeout error as shown below.

A friend called me recently and started our conversation with: “I know you dislike pie charts, but…can you help me create one?” 

If you have worked with larger tables in either Po...

Want to use a custom font within Power BI Desktop,...

A quick point: while the Power Query Query Diagnostics functionality is relatively new, it’s based on Power BI/Power Query trace logging that has been around for a while. I’ve...

Lots of folks ask if you can use groups with Power...

Scenario is you want to use export to excel when v...

This article describes the types of relationships available in Power BI and Analysis Services, clarifying the differences in cardinality and filter propagation of physical relationships.

This post demonstrates how the order of steps added to a query can make a big performance difference and drastically effect the number of steps generated by the designer. I’ll demonstrate how to use the new query Diagnostics tools to compare and understand query performance.

Do you have a Power BI report that has a row of card visuals on it? It’s a very common thing to do, but if your report is too slow to render you may find that a design like this is part of the problem.

As someone who enjoys creating unique content in Power BI that’s always “one step away from useful” and not what Power BI was intended for, I now feel like I’ve missed out on a few years’ worth of opportunities. I should have started learning Power Apps long ago.

Management has requested an easy way to view a subset of strategic reports located in different Power BI workspaces. You can ask the users to mark reports and dashboards as favorites so they can access pertinent content in the Favorites menu, but you’re looking for an easier configuration, such as to create a book of reports with a built-in navigation that organizes reports in groups.

The Power Apps visual is a powerful tool for adding writeback and other functionality from Power Apps into Power BI reports. It allows users to take action directly from Power BI reports and update the data in real time.

I love SQL Prompt, and I’m regularly impressed b...

Phil Factor provides a powerful DOS batch script which, when coupled with SQL Compare CLI, allows you to build databases from source, during development, and fill them with the...

Thanks for watching this week's Power BI news roun...

Launched at AWS re:Invent 2017, Amazon Transcribe is an automatic speech recognition (ASR) service that makes it easy for AWS customers to add speech-to-text capabilities to their applications. At...

Tomaz Kastrun reports on a worrying trends in the IT community, including a decline in awareness of how much community is worth, and a decline in respect for community contributions.

Wayne Sheffield explains a couple of potential issues when using Google Chrome with SSRS reports: constant log in prompts to the SSRS server, and the infamous Kerberos Double-Hop.

You are asked to provide information about logins and database users, but the login name does not match the name of the database user account.

Extended Events can do things that simply are not possible with Profiler and another example comes from the stack of audit events that exist only in Extended Events. One...

Cypress and Broadcom chip bug bit iPhones, Macs, Android devices, Echoes, and more.

Losing data to an intruder is not a great look for a law enforcement partner.

YAGNI, KISS, DDD and unit testing. Wait, that's four!

When and why to choose monolithic architecture is usually a matter of what works best for each business. Whatever the reason for using monolithic services, you still have to support them. They do, however, bring their own reliability and scaling challenges, and that’s what we’ll tackle in this post.

Overview If you are looking for major performance ...

Overview In short, no.  However, for many years t...

Overview If you are looking for major performance ...

Overview In short, no.  However, for many years t...

Pivoting data in SQL Server is something that comes up infrequently for me. Whenever the need arises, I have to pause and ask myself "What is it I'm trying...

Kenneth Fisher discovers a weird option for the UPDATE command in a stack overflow answer

A procedure populates a temp table with the key values, which are then used to delete from the related tables, then delete from the primary table. However, the query populating our temp table was taking far too long, 84 seconds.

Jess Pomfret rounds up all the suggested SSMS hacks, time management skills, useful tools and more.

SQL Server 2016 gave us the STRING_SPLIT() function, but what about the reverse – compiling a set of values into one delimited string? We only had to wait 15 months for the release of SQL Server 2017, and the STRING_AGG() function.

If you’re the kind of person who likes to read books, Brent suggests a few that he thinks would expand your SQL horizons.

If you frequently need to pull all of the memberships that were active on a specific date, that’s where performance tuning gets hard. You don’t know the UserId, and even worse, you can’t predict the date/time you’re looking up, or if it’s always Right Now.

Occasionally you might need to know the physical location of a row stored in SQL Server. The code in this post uses the undocumented feature, %%PHYSLOC%%, which returns a binary representation in hexadecimal of the location of each row returned in a SELECT statement.

Sorting data requires memory, and Sort operators particularly may ask for quite a bit of memory. Why? Because all the columns you’re selecting, need to to be sorted by the column in the ORDER BY.

In SQL Server 2017, there is still a value for password in sys.syslogins, but in SQL Server 2019 it is now null. Taryn Pratt explains how this 'gotcha' caused problems during a SQL Server 2019 upgrade.

The sys.dm_exec_describe_first_result_set_for_object() DMF takes as its parameter the object_id of a procedure or trigger and describes the first result metadata for the module with that ID. Phil Factor explains why a developer might find this handy.

sp_describe_undeclared_parameters() promises a great deal, but delivers less. It was added mostly for executing application queries via sp_prepare using a TDS link, but has several quirks and restrictions that rather spoils its more general use.

Phil Factor explains why an ORDER BY clause should always specify the sort columns using their names, or aliases, rather than using an integer to specify the position of a column in the SELECT list.

Northrop says this is a first step toward a fleet of satellite-servicing vehicles.

Recently I had an interesting conversation with my client about Emoji in SQL Server. It is possible to create your database with any emoji. First appeared on SQL SERVER –...

I've written about WSL2 and it's glorious wonders many times. As its release (presumably) grows closer - as of this writing it's on Windows Insiders Slow and Fast -...

Sometimes you need to know if you’re inside a docker container from the shell. Here’s how you can do that.. First spin up a container.. docker run -ti --rm...

Azure Kubernetes Services makes Deploying a SQL Server Big Data Cluster is a very simple process but you need some understanding of the the various Kubernetes Pods and Docker Containers created as a result.