Pandas Series

[rubydamodar]

0. Introduction to Pandas and Pandas Series

• Pandas Overview

  • Installation (pip install pandas)
  • Integration with other libraries: NumPy, Matplotlib, Seaborn
  • Real-world applications of Pandas in data analysis and machine learning

• Pandas Series Basics

  • Underlying data structure (1D ndarray with axis labels)
  • Homogeneous vs heterogeneous data handling
  • Series as a foundation for Pandas DataFrame (each DataFrame column is a Series)

---

1. Basics of Pandas Series

• Creating a Pandas Series in More Detail

  • Creating Series with explicit dtype (e.g., dtype=int64, dtype=float32)
  • Handling type conversions (astype() method)
  • Creating Series from:
    • Lists: pd.Series([1, 2, 3, 4])
    • Dictionaries: pd.Series({"a": 1, "b": 2})
    • NumPy Arrays: pd.Series(np.array([1, 2, 3]))
    • Scalars: pd.Series(5, index=[0, 1, 2, 3])

• Index Customization

  • Customizing index labels with strings, timestamps, etc.
  • Multi-indexing (hierarchical indexes) with Series
  • pd.Series([10, 20], index=[(2023, 'Q1'), (2023, 'Q2')])

• Series Metadata

  • dtype: Handling categorical, object, and mixed data types
  • Handling time series data with DatetimeIndex

---

2. Data Operations in Pandas Series

• Element-wise Operations and Broadcasting

  • Vectorized operations between Series and scalars
    • Adding a scalar to every element: series + 5
    • Operations on multiple Series: series1 + series2 (alignment by index)
  • Handling misaligned data (NaN insertion on index misalignment)

• Advanced Mathematical Functions

  • Handling NaNs during arithmetic operations (skipna=True)
  • Standard deviation, variance, and other statistical metrics (std(), var())
  • Applying universal NumPy functions: np.log(series), np.exp(series)

• Function Application to Series

  • Using applymap() for element-wise custom functions
  • Applying complex functions with lambda expressions
  • Using .apply() for row-wise and element-wise operations, like:

    series.apply(lambda x: x if x > 0 else 0)

---

3. Indexing and Selection in Pandas Series

• Position-based Indexing with .iloc[]

  • Accessing elements by integer-location indexing:
    • Single element: series.iloc[2]
    • Multiple elements: series.iloc[[0, 2, 4]]
    • Slicing: series.iloc[2:5]

• Label-based Indexing with .loc[]

  • Selecting data by index labels:
    • Single label: series.loc['a']
    • List of labels: series.loc[['a', 'b', 'c']]
    • Slicing with labels: series.loc['a':'c']

• Advanced Boolean Indexing

  • Boolean masking: series[series > 10]
  • Combining conditions: (series > 10) & (series < 20)
  • Using .where() and .mask() for conditional replacements
    • series.where(series > 10, other=0)

---

4. Handling Missing Data in Series

• Detecting Missing Data with More Detail

  • Detecting NaN (isnull(), notnull())
  • Counting missing values in a Series: series.isnull().sum()
  • Working with pd.NaT for missing datetime data

• Advanced Techniques for Handling Missing Data

  • Filling missing values with specific values: fillna(0)
  • Forward-fill/backward-fill missing values: fillna(method='ffill'), fillna(method='bfill')
  • Limiting the number of replacements while filling: fillna(method='ffill', limit=1)
  • Interpolation techniques (interpolate()):
    • Linear interpolation: series.interpolate(method='linear')
    • Time-based interpolation for time series data

---

5. Series Indexing and Reindexing

• Understanding the Index Object

  • Immutable nature of index: Attempting to modify results in error
  • Changing the index without changing the data: series.index = new_index
  • Resetting the index: series.reset_index()

• Reindexing with Fine Control

  • Changing the Series to conform to a new index: series.reindex(['a', 'b', 'c'])
  • Filling missing values during reindexing: series.reindex(new_index, fill_value=0)
  • Using methods during reindexing (ffill, bfill):

    series.reindex(new_index, method='ffill')

• Set Operations on Index

  • Index union: series1.index.union(series2.index)
  • Index intersection: series1.index.intersection(series2.index)
  • Symmetric difference: series1.index.symmetric_difference(series2.index)

---

6. Series Alignment and Broadcasting

• Series Alignment in Depth

  • How Series align during operations:
    • series1 + series2 aligns data by index labels
    • NaN appears where indices do not match
  • Filling missing values in alignments:

    series1.add(series2, fill_value=0)

• Broadcasting Operations

  • Broadcasting operations with scalar values:

    series * 10  # Multiplies each element by 10

---

7. Data Transformation in Series

• Using Map for Value Mapping

  • Mapping values in a Series:

    series.map({'a': 1, 'b': 2})

  • Mapping complex functions with map():

    series.map(lambda x: x ** 2)

• Apply for More Advanced Operations

  • Applying custom functions to transform data:

    series.apply(np.sqrt)

• Sorting Operations

  • Sorting by index or values:
    • Sorting by index: series.sort_index()
    • Sorting by values: series.sort_values(ascending=False)
  • Sorting with NaNs:
    • NaNs appear at the end by default; customize with na_position='first'

---

8. Working with Time Series Data

• Time Series Creation and Manipulation

  • Creating Series with DatetimeIndex:

    pd.Series([1, 2, 3], index=pd.date_range('2023-01-01', periods=3))

  • Resampling and frequency conversion:
    • Resampling daily to weekly: series.resample('W').mean()

• Handling Dates and Timestamps

  • Accessing year, month, and day from datetime index:

    series.index.year, series.index.month

---

9. Combining and Merging Series

• Appending and Concatenation in Detail

  • Appending two Series:

    series1.append(series2)

  • Concatenating multiple Series with axis alignment:

    pd.concat([series1, series2])

• Combining Series Data with Alignment

  • Combining Series using indexes:

    series1.combine_first(series2)

---

10. Advanced Techniques for Series

• Memory Optimization

  • Reducing memory usage with data types:

    series.astype('int32')

• Working with Categorical Data

  • Converting a Series to categorical:

    series.astype('category')

  • Benefits of categorical data (memory efficiency, faster comparisons)

---

11. Series Visualization

• Advanced Plotting with Series
  • Visualizing Series with various plot types:

    series.plot(kind='line')  # Line plot
    series.plot(kind='bar')   # Bar plot
    series.plot(kind='hist')  # Histogram

[jiwooverse]

Done