Скачать или смотреть Python Tutorial : Clean and Validate

Want to learn more? Take the full course at https://learn.datacamp.com/courses/ex... at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work.

---

In the previous lesson, we read data from the National Survey of Family Growth and selected a column from a DataFrame. In this lesson, we'll check for errors and prepare the data for analysis.

We'll use the same DataFrame we used in the previous lesson, nsfg, which contains one row for each pregnancy in the survey.

I'll select the variable birthwgt_lb1, which contains the pound part of birth weight, and assign it to pounds.

And birthwgt_oz1 contains the ounce part of birth weight, so I'll assign that to ounces.

Before we do anything with this data, we have to validate it. One part of validation is confirming that we are interpreting the data correctly.

We can use value_counts() to see what values appear in pounds and how many times each value appears.

By default, the results are sorted with the most frequent value first, so I use sort_index() to sort them by value instead, with the lightest babies first and heaviest babies last.

As we'd expect, the most frequent values are 6-8 pounds, but there are some very light babies, a few very heavy babies, and two values, 98, and 99, that indicate missing data.

We can validate the results by comparing them to the codebook, which lists the values and their frequencies. The results here agree with the codebook, so we have some confidence that we are reading and interpreting the data correctly.

Another way to validate the data is with describe(), which computes summary statistics like the mean, standard deviation, min, and max.

Here are the results for pounds. count is the number of values. The minimum and maximum values are 0 and 99, and the 50th percentile, which is the median, is 7.

The mean is about 8.05, but that doesn't mean much because it includes the special values 98 and 99. Before we can really compute the mean, we have to replace those values with NaN to represent missing data.

The replace() method does what we want; it takes a list of values we want to replace and the value we want to replace them with. np dot nan means we are getting the special value NaN from the NumPy library, which is imported as np.

The result from replace() is a new Series, which I assign back to pounds.

Remember that the mean of the original series was about 8 point 05 pounds. The mean of the new series is about 6 point 7 pounds. It makes a big difference when you remove a few 99-pound babies!

Instead of making a new Series, you can call replace() with inplace=True, which modifies the existing Series "in place", that is, without making a copy. Here's what that looks like for ounces. Since we didn't make a new series, we don't have to assign it back to ounces.

Now we want to combine pounds and ounces into a single Series that contains total birth weight.

Arithmetic operators work with Series objects; so, to convert from ounces to pounds, we can divide by 16 (there are 16 ounces in a pound). Then we can add the two Series objects to get the total.

Here are the results. The mean is about 7 point 1, which is a little more than what we got before we added in the ounces part.

Now we're close to answering our original question, the average birth weight for babies in the U.S., but as we'll see in the next lesson, we're not there yet.

Let's first practice what we learned in this video.

#DataCamp #PythonTutorial #ExploratoryDataAnalysisinPython