Standardizing timeseries in Pandas using interpolation

» python » Standardizing timeseries in Pandas using interpolation

By : Khan Saba

Date : November 23 2020, 03:01 PM

this will help First timer, be nice! , You're looking for:

code :

data['Raw'] = pd.to_datetime(data['Raw'])

data = data.set_index('Raw').resample('1min').mean()

data = data.assign(RawDt=pd.to_datetime(data.Raw))\
       .groupby(pd.Grouper(key='RawDt', freq='1min'))\
       .agg({'DataValue' : 'mean', 'Raw' : 'first'}).reset_index(drop=True)

Share :

Resampling timeseries with a given timedelta and binning or interpolation

By : Payal Maji

Date : March 29 2020, 07:55 AM

I hope this helps . I have a simple time-series, driven by datetime values (that is, it records data points at regular intervals), Series1:

code :

import io
import pandas as pd

data = io.StringIO('''\
datetime,window   
2015-05-28 17:00:00,0.0
2015-05-28 17:55:28,1.0
2015-06-08 07:35:31,0.0
2015-06-08 08:04:30,1.0
2015-06-18 17:11:55,0.0
2015-06-18 18:11:52,1.0
2015-06-19 18:14:09,0.0
''')

s = pd.read_csv(data).set_index('datetime').squeeze()
s.index = pd.to_datetime(s.index)

upsampled = s.resample('min').ffill()
upsampled['2015-06-08 07:30':'2015-06-08 08:10']

# datetime
# 2015-06-08 07:30:00    1.0
# 2015-06-08 07:31:00    1.0
# 2015-06-08 07:32:00    1.0
# 2015-06-08 07:33:00    1.0
# 2015-06-08 07:34:00    1.0
# 2015-06-08 07:35:00    1.0
# 2015-06-08 07:36:00    0.0
# 2015-06-08 07:37:00    0.0
# 2015-06-08 07:38:00    0.0
# 2015-06-08 07:39:00    0.0
# 2015-06-08 07:40:00    0.0
# 2015-06-08 07:41:00    0.0
# 2015-06-08 07:42:00    0.0
# 2015-06-08 07:43:00    0.0
# 2015-06-08 07:44:00    0.0
# 2015-06-08 07:45:00    0.0
# 2015-06-08 07:46:00    0.0
# 2015-06-08 07:47:00    0.0
# 2015-06-08 07:48:00    0.0
# 2015-06-08 07:49:00    0.0
# 2015-06-08 07:50:00    0.0
# 2015-06-08 07:51:00    0.0
# 2015-06-08 07:52:00    0.0
# 2015-06-08 07:53:00    0.0
# 2015-06-08 07:54:00    0.0
# 2015-06-08 07:55:00    0.0
# 2015-06-08 07:56:00    0.0
# 2015-06-08 07:57:00    0.0
# 2015-06-08 07:58:00    0.0
# 2015-06-08 07:59:00    0.0
# 2015-06-08 08:00:00    0.0
# 2015-06-08 08:01:00    0.0
# 2015-06-08 08:02:00    0.0
# 2015-06-08 08:03:00    0.0
# 2015-06-08 08:04:00    0.0
# 2015-06-08 08:05:00    1.0
# 2015-06-08 08:06:00    1.0
# 2015-06-08 08:07:00    1.0
# 2015-06-08 08:08:00    1.0
# 2015-06-08 08:09:00    1.0
# 2015-06-08 08:10:00    1.0
# Freq: T, Name: window   , dtype: float64

result = upsampled.resample('H').mean()
result['2015-06-08 06:00':'2015-06-08 09:00']

# datetime
# 2015-06-08 06:00:00    1.000000
# 2015-06-08 07:00:00    0.600000
# 2015-06-08 08:00:00    0.916667
# 2015-06-08 09:00:00    1.000000
# Freq: H, Name: window   , dtype: float64

upsampled = s.resample('s').ffill()
result = upsampled.resample('H').mean()
result['2015-06-08 06:00':'2015-06-08 09:00']

# datetime
# 2015-06-08 06:00:00    1.000000
# 2015-06-08 07:00:00    0.591944
# 2015-06-08 08:00:00    0.925000
# 2015-06-08 09:00:00    1.000000
# Freq: H, Name: window   , dtype: float64

Upsample timeseries in pandas with interpolation

By : Ping Woo

Date : March 29 2020, 07:55 AM

This might help you A way of getting this at least partially right (for real data, the results are not great, I had better success with scipy's interp1d) is to use mean() in between the methods:

code :

>>> series.resample(rule='0.5S').mean().interpolate(method='linear')
2000-01-01 00:00:00.000    0.0
2000-01-01 00:00:00.500    1.0
2000-01-01 00:00:01.000    1.5
2000-01-01 00:00:01.500    2.0
2000-01-01 00:00:02.000    2.5
2000-01-01 00:00:02.500    3.0
2000-01-01 00:00:03.000    3.5
2000-01-01 00:00:03.500    4.0
2000-01-01 00:00:04.000    4.5
2000-01-01 00:00:04.500    5.0
2000-01-01 00:00:05.000    6.0
2000-01-01 00:00:05.500    6.5
2000-01-01 00:00:06.000    7.0
2000-01-01 00:00:06.500    7.5
2000-01-01 00:00:07.000    8.0
Freq: 500L, dtype: float64

Filling NA in timeseries data with different interpolation techniques

By : ruby

Date : March 29 2020, 07:55 AM

may help you . If you want to try and compare several interpolation methods as stated, you can use the na.interpolation() function from the imputeTS package.
For linear interpolation:

code :

library("imputeTS")
na.interpolation(df, option = "linear")

library("imputeTS")
na.interpolation(df, option = "spline")

library("imputeTS")
na.interpolation(df, option = "stine")

Seperate timeseries in pandas and give ID to each timeseries

By : user3612050

Date : March 29 2020, 07:55 AM

it fixes the issue I think this is what you want. I have reconstructed the data frame as one was not provided.

code :

import pandas as pd
import numpy as np
times = pd.date_range(start ='1/1/2020',end='1/20/2020',periods = 100)
df = pd.DataFrame(list(zip(times, np.random.uniform(size = 100), np.random.uniform(size = 100))), 
                  columns = ['datetime', 'sensor_1_value', 'sensor_2_value'])
df['date_only'] = df.datetime.dt.date
df['ID'] = df.groupby(['date_only']).ngroup() + 1