logo
down
shadow

How to iterate multilevel dataframe in python


How to iterate multilevel dataframe in python

By : user2172626
Date : October 23 2020, 08:10 AM
I wish did fix the issue. OK, your json is not valid because {('1144986','1'): 'whatever'} isn't a valid json, instead it's an extremely convoluted python dictionary definition containing lists and other dictionaries which, luckily, pandas can handle but not in a graceful manner I'm afraid.
Now watch the magic (sorry for the images, textual representation is not informative):
code :
js_dict = {('1144986','1'):{'batsmen'  ... # this line won't execute, just for completeness

# Get the initial dataframe
df = pd.DataFrame.from_dict(js_dict, orient='index')
teams = pd.concat(df['team'].apply(pd.DataFrame.from_dict, orient='index')
                  .apply(lambda x: x.T)
                  .to_dict()).reset_index(level=2,drop=True)
# Join back to the initial dataframe
df = df.join(teams).drop('team', axis=1)
batsmen = pd.concat(df['batsmen']
                .apply(pd.DataFrame)
                .to_dict()).reset_index(level=2,drop=True)
# Join with the initial dataframe renaming some columns in the process
batsmen = batsmen.join(df).drop(['batsmen', 'bowlers'], axis=1)
batsmen = batsmen.rename(columns={'name': 'batsman_name',
                              'stats': 'batsman_stats'})
batsman_stats = pd.concat(batsmen['batsman_stats']
                          .apply(pd.DataFrame)
                          .apply(lambda x: x.T).to_dict()).reset_index(level=2, drop=True)
# Join back with some renamings
batsmen = batsman_stats.join(batsmen).drop('batsman_stats', axis=1)
batsmen = batsmen.rename(columns={'name': 'batsman_stats_name',
                                  'text': 'batsman_stats_text',
                                  'value': 'batsman_stats_value'})
bowlers = pd.concat(df['bowlers']
                .apply(pd.DataFrame)
                .to_dict()).reset_index(level=2,drop=True)
# Join with the initial dataframe renaming some columns in the process
bowlers = bowlers.join(df).drop(['batsmen', 'bowlers'], axis=1)
bowlers = bowlers.rename(columns={'name': 'bowler_name',
                                  'stats': 'bowler_stats'})
bowler_stats = pd.concat(df['bowler_stats']
                         .apply(pd.DataFrame)
                         .apply(lambda x: x.T)
                         .to_dict()).reset_index(level=2, drop=True)

# Merge back and rename
bowlers = bowler_stats.join(bowlers).drop('bowler_stats', axis=1)
bowlers = bowlers.rename(columns={'name': 'bowler_stats_name',
                        'text': 'bowler_stats_text',
                        'value': 'bowler_stats_value'})


Share : facebook icon twitter icon
How do I convert a wide dataframe to a long dataframe for a multilevel structure with 'quadruple nesting'?

How do I convert a wide dataframe to a long dataframe for a multilevel structure with 'quadruple nesting'?


By : user4031284
Date : March 29 2020, 07:55 AM
To fix the issue you can do I think your problem can be solved with a two step approach:
melt your data into a long data.frame (or as I did, in a long data.table) split the variable column with all the labels into separate columns for each required grouping variable.
code :
library(data.table)
longData <- melt(setDT(wideData), id.vars="id")
longData[, c("moment", "intervention", "number", "behavior") := 
                tstrsplit(variable, "_", type.convert = TRUE)
       ][, variable:=NULL]
> head(longData,15)
    id       value moment intervention number behavior
 1:  1 -0.07747254     t0         fear      1     diet
 2:  2 -0.76207379     t0         fear      1     diet
 3:  3  1.15501244     t0         fear      1     diet
 4:  4  1.24792369     t0         fear      1     diet
 5:  5 -0.28226121     t0         fear      1     diet
 6:  1 -1.04875354     t1         fear      1     diet
 7:  2 -0.91436882     t1         fear      1     diet
 8:  3  0.72863487     t1         fear      1     diet
 9:  4  0.10934261     t1         fear      1     diet
10:  5 -0.06093002     t1         fear      1     diet
11:  1 -0.70725760     t0         know      1     diet
12:  2  1.06309003     t0         know      1     diet
13:  3  0.89501164     t0         know      1     diet
14:  4  1.48148316     t0         know      1     diet
15:  5  0.22086835     t0         know      1     diet
library(splitstackshape)
longData <- cSplit(longData, sep="_", "variable", "wide", type.convert=TRUE)
names(longData) <- c("id","value","moment","intervention","number","behavior")
library(tidyr)
separate(longData, variable, c("moment", "intervention", "number", "behavior"), sep="_", remove=TRUE)
How to iterate a multilevel dict in python

How to iterate a multilevel dict in python


By : seyrankal
Date : March 29 2020, 07:55 AM
I wish this helpful for you It's unclear exactly what you want to "iterate", especially since you trimmed down the data and didn't really specify what the output you wanted.
So here's a guess based on adding a little more sample data:
code :
data = {u'level1':
            {u'level2': [{u'Data':
                            {u'id': u'220',
                             u'byte': u'13',
                             u'time': u'1444',
                             u'ch': u'38'}},
                         {u'Data':
                            {u'id': u'440',
                             u'byte': u'26',
                             u'time': u'2888',
                             u'ch': u'76'}}]}}

ids = []
for item in data['level1']['level2']:
    ids.append(item['Data']['id'])

print(ids)  # -> [u'220', u'440']
ids = [item['Data']['id'] for item in data['level1']['level2']]
Elegant way to iterate into a multilevel pandas DataFrame

Elegant way to iterate into a multilevel pandas DataFrame


By : Papori
Date : March 29 2020, 07:55 AM
around this issue I have a DataFrame: , I suggest:
code :
beta = [np.polyfit(a[col]['AA'], a[col]['BB'], deg= 1) for col in a.stack().columns]
How to access a multilevel Pandas Dataframe in Python - Storing Bloomberg data in dataframe

How to access a multilevel Pandas Dataframe in Python - Storing Bloomberg data in dataframe


By : Stefan Radenkovic
Date : March 29 2020, 07:55 AM
With these it helps I have installed Bloomberg API and pdblp library. I am able to get historical data and store it in Dataframe. But i am not sure how to access data from multilevel dataframe. , The 'date' is in your index. Pull it out with reset_index.
code :
df = df.reset_index()
df['date']
Is possible concat one level and other multilevel dataframe and have in axis 1 one level and multilevel at the same time

Is possible concat one level and other multilevel dataframe and have in axis 1 one level and multilevel at the same time


By : samdroid
Date : March 29 2020, 07:55 AM
I wish did fix the issue. I solve without concatenate, because is not possible have in axis 1 diferent levels. I decide to use the data in dataframe x as a index in dataframe y.
So, follow this steps:
code :
d = {'A': np.random.randint(0, 10, 5)
   , 'B': np.random.randint(0, 10, 5)
   , 'C': np.random.randint(0, 10, 5)
   , 'D': np.random.randint(0, 10, 5)}
x = pd.DataFrame(d)

   A  B  C  D
0  7  1  6  8
1  4  0  5  6
2  7  5  0  7
3  8  4  3  8
4  9  1  4  0
index = [x[col] for col in x.columns]
from functools import reduce

v = ['u','v','z']
l = ['300','350','400','450','500'] * len(v)
d = ['1','2','3','4'] * len(l)

size = len(v) * len(l) * len(d)

der_v = reduce(lambda x,y: x+y, [[i] * 20 for i in v])
der_l = reduce(lambda x,y: x+y, [[i] * 4 for i in l])
der_d = reduce(lambda x,y: x+y, [[i] for i in d])

arrays =[der_v,der_l,der_d]
y = pd.DataFrame(np.random.randint(0, 1, (5,60)), columns=arrays, index=index)
y.columns = y.columns.rename(['variables', 'level','days'], level=[0,1,2])
y.index.names = ['A','B','C','D']
print(y)

variables   u                              ...   z                            \
level     300          350          400    ... 400    450          500         
days        1  2  3  4   1  2  3  4   1  2 ...   3  4   1  2  3  4   1  2  3   
A B C D                                    ...                                 
7 1 6 8     0  0  0  0   0  0  0  0   0  0 ...   0  0   0  0  0  0   0  0  0   
4 0 5 6     0  0  0  0   0  0  0  0   0  0 ...   0  0   0  0  0  0   0  0  0   
7 5 0 7     0  0  0  0   0  0  0  0   0  0 ...   0  0   0  0  0  0   0  0  0   
8 4 3 8     0  0  0  0   0  0  0  0   0  0 ...   0  0   0  0  0  0   0  0  0   
9 1 4 0     0  0  0  0   0  0  0  0   0  0 ...   0  0   0  0  0  0   0  0  0   

variables     
level         
days       4  
A B C D       
7 1 6 8    0  
4 0 5 6    0  
7 5 0 7    0  
8 4 3 8    0  
9 1 4 0    0  

[5 rows x 60 columns]
Related Posts Related Posts :
  • Submitting login form with scrapy
  • How do i edit the favicon in the Browsable API in Django REST framework?
  • multiprocessing.Pool.map_async doesn't seem to... do anything at all?
  • Python Selenium: Stale Element Reference Exception Error
  • Datetime conversion - How to extract the inferred format?
  • Import YAML variables automatically?
  • How to create a powershell shortcut for my python file
  • Python's 'set' operator doesn't work with numpy.nan
  • Pass object fields and one2many fields on same method - Odoo v8
  • Select columns based on column name and location in Pandas
  • Standardizing timeseries in Pandas using interpolation
  • How many tweets can be collected?
  • how format specifier taking value while tuple list is passed
  • How to print a numpy array with data type?
  • Timeout child thread for python3
  • How can I regroup a dataframe and accumulate a colume's values?
  • Bulk Insert into SQL Server with Python not working
  • Removing last rows of each group based on condition in a pandas dataframe
  • Why the css file can not be found in Django template?
  • targeting center of mass - scipy / numpy
  • Foursquare - get tips from VENUE_ID
  • Unpack a dictionary to format
  • encoding special characters in python2
  • Replacing integers with NaN results in the entire column becoming float dtype
  • Python 3.6 - BeautifulSoup4, parse table AttributeError: ResultSet object has no attribute 'findAll'
  • Convert panda date list to python list of date strings
  • escape response from Scrapy to parse json
  • How to create a same dropdown menu for different labels?
  • Why are some python variables uppercase whereas others are lowercase?
  • Machine Learning, What are the common techniques for feature engineering and presenting the model?
  • Modify value of a Django form field during clean() and validate again
  • Heroku Django app can't start up -- 'No module named site'
  • Getting list of dates (excluding weekends)
  • Im trying to create the regular expression to include the text and not the href
  • Python file.readline(2) reads first 2 charectars
  • Groupby with handling empty bin in python
  • Modifying Gcode
  • calling a value in a dictionary within a dictionary (reading a json file)
  • Bouncing ball invalid syntax why is that?
  • Python making a counter
  • Python rstrip and split
  • What does the String mean in numpy.r_?
  • How to correctly extend variable __all__ in a __init__.py?
  • Python behaves weird with piped input
  • Python 3 two dimensional list comprehension
  • How to slice image by broadcasting slices? Error: 'only integer scalar arrays can be converted to a scalar index' in pyt
  • (Python Beginner) Need a start on classes
  • IndexError: At least one sheet must be visible
  • How to solve a system of linear equations over the nonnegative integers?
  • Pandas keep the most complete rows
  • "List index out of range" error in Python Memory Match game
  • Numpy: how to use argmax results to get the actual max?
  • Google Cloud Dataflow can't import 'google.cloud.datastore'
  • Calculate pandas DataFrame column by custom routine which accepts dictionary as input
  • Connect to a Class Method by it's method name holded into a var in a for loop in python
  • PyQt5 signals and threading.Timer
  • Replace 2 characters in a string in python
  • Passing command line arguments from a folder script to a file script
  • Understand the syntaxe X[Y == c] in Numpy
  • Optimize beginner python script about substring replacement
  • shadow
    Privacy Policy - Terms - Contact Us © voile276.org