Sometimes, duplicate rows can cause problems with an analysis or a specific data science technique.įor example, imagine you’re working with sales data. When you’re working with some types of data, it might be normal – even expected – to have duplicate rows of data.īut there are some instances where having duplicate rows of data is bad. There’s also a different type of duplicate, where two rows have the same value for one or more important columns (but not necessarily every column). So for example, if two rows had the same value for every column, we’d consider those to be duplicate rows. Importantly, it’s possible for a dataframe to have duplicate rows of data. It’s Possible to Have Rows with the Exact Same Data If you’ve ever used Excel, a Pandas dataframe is really a lot like an Excel spreadsheet, in the sense that they both have this row-and-column structure. For example, if you had a dataframe with sales data, the individual rows might record the sales information for individual people.

Typically, the columns of a dataframe are variables, and the rows typically record individual data records. Pandas dataframes store data in a row-and-column format. Let’s look more carefully at dataframe structure. We use dataframes to store certain types of data, and we use Pandas techniques to manipulate dataframe data. (Remember: the drop duplicates method operates on Pandas dataframes.)Ī dataframe is a data structure in Python that’s available in the Pandas package. Dataframes Store Python Dataįirst, let’s quickly review what a dataframe is. This will give you some context, and help you understand exactly what this technique does, and why we might use it. I’ll show you some examples of this in the examples section, but first, I want to quickly review some fundamentals about Pandas and Pandas dataframes. Stated simply, the Pandas drop duplicates method removes duplicate rows from a Pandas dataframe. Having said that, if you really want to know how this technique works, you should probably read the whole tutorial.Īn Introduction to Pandas Drop Duplicates If you need something specific, you can click on one of the links above.

Examples: How to drop duplicate rows from a dataframe.

An Introduction to Pandas Drop Duplicates.

You can click on any of the following links, and it will take you to the appropriate section in the tutorial. The tutorial will explain what the technique does, explain the syntax, and it will also show you clear examples.

In this way, we can easily identify and remove the duplicate data from our data frame in pandas.This tutorial will show you how to use Pandas drop duplicates to remove duplicate rows from a dataframe. We can verify this by looking at the number of rows before and after removing the duplicates. This will remove all of the duplicate rows from the data frame and only return the unique rows. In line 23, we use the function drop_duplicates() on the entire data frame. Now that we have identified the duplicate data in our data frame, it is time to remove the duplicates. In line 20, we print the number of rows that are duplicates using the sum() function. In the output, we can see that the last five rows are not duplicates. Note that, here, we have not used any column name before using the duplicated() function. In line 17, we print whether there is an entire duplicate row in the data frame. In the sum, True represents 1 and False represents 0. In line 14, we print the number of duplicate values in the zip_code column by using the sum() function. Here, we can see that, of those last five entries, there is one zip_code that is a duplicate. We then print the last five entries in our data frame. In line 11, we print whether there are any duplicates ( True indicating duplicate, False indicating unique) in the zip_code column. At this point, we have our data loaded as a data frame in df. In line 6, we read the data as a data frame and pass the column names and index. In line 3, we create a list of column names that are present in our data. In line 1, we import the required package.

Dropping duplicate rows in pandas