Скачать или смотреть How to Find Duplicates in a DataFrame with Compound Criteria in Python Pandas

Learn how to efficiently find duplicates in a DataFrame by using compound criteria in Python Pandas. This guide provides step-by-step instructions for working with duplicate values, ensuring you keep only the necessary data.
---
This video is based on the question https://stackoverflow.com/q/63678603/ asked by the user 'tlre0952b' ( https://stackoverflow.com/u/808946/ ) and on the answer https://stackoverflow.com/a/63678703/ provided by the user 'wwnde' ( https://stackoverflow.com/u/8986975/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: Find duplicates in dataframe by compound criteria?

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Finding Duplicates in a DataFrame by Compound Criteria

When working with data, it's common to encounter duplicate entries. However, there are times when you want to filter duplicates based on specific criteria. If you've ever grappled with such a situation in Python using Pandas, you’re not alone! In this post, we will discuss how to identify and keep only the desired duplicates from a DataFrame based on compound criteria.

The Problem

Consider the following DataFrame structure:

[[See Video to Reveal this Text or Code Snippet]]

Here’s the goal:
We want to keep rows where col2 (in this case, the value '3') has duplicate instances but only retain those with unique values for col3.

For our example:

Keep:

The first instance of Bob (1 3 bob)

The instance of Rose (4 3 rose)

Discard:

The second instance of Bob (3 3 bob) because it duplicates an existing col3 value of 'bob'.

The row with Alice (2 1 alice) since '1' in col2 does not have duplicates.

The expected output would be:

[[See Video to Reveal this Text or Code Snippet]]

The Solution

To achieve this, we can utilize a combination of duplicated(), drop_duplicates(), and the loc accessor. Let’s break down the solution into manageable steps.

Step 1: Identify Duplicated Values in col2

First, we will filter our DataFrame to find duplicates in col2:

[[See Video to Reveal this Text or Code Snippet]]

This operation identifies all the rows where col2 has duplicates, allowing us to focus only on relevant entries for our criteria.

Step 2: Retain Only Unique Values in col3

Next, we will drop duplicates from col3 but retain the first occurrence:

[[See Video to Reveal this Text or Code Snippet]]

This filters our results further by ensuring we only retain the first instance of each unique col3 value associated with the duplicated col2.

Step 3: Extract Indices of the Filtered Data

Now, we need to extract the indices of the rows that meet our criteria:

[[See Video to Reveal this Text or Code Snippet]]

Step 4: Final Filtering with loc

Finally, we can filter our original DataFrame using the indices we just retrieved:

[[See Video to Reveal this Text or Code Snippet]]

Final Result

When we run through this process, the output will give us:

[[See Video to Reveal this Text or Code Snippet]]

Conclusion

This method of filtering duplicates in a DataFrame based on compound criteria is powerful for data analysis tasks. By using the combination of methods such as duplicated(), drop_duplicates(), and loc, you can effectively manage duplicate data, ensuring you only keep the entries that matter.

Feel free to adapt this solution to your specific use-case, and happy coding with Pandas!