logo
Tags down

shadow

Looping through pixels with Cython still slow


By : user2174574
Date : October 17 2020, 08:10 PM
wish of those help The looping isn't any problem, but appending lists to lists is very slow. To avoid this you can either allocate an array large enough for the data and shrink it afterwards (or copy the data in an array which has the exact size you need) or you can implement your function using std:vector.
In this answer I use Numba because I'm not that experienced in performant Cython coding, but a Cython implementation should be straight forward. Numba has also a limited internal representation of list and tuples, but I don't know if the same is available within Cython.
code :
import numpy as np
import numba as nb

@nb.njit()
def get_image_data_arr(image_data):
  array_text = np.empty((image_data.shape[0]*image_data.shape[1],2),dtype=np.int64)
  ii=0
  for y in range(image_data.shape[0]):
    for x in range(image_data.shape[1]):
      if image_data[y, x] < 210:
        array_text[ii,0]=x
        array_text[ii,1]=y
        ii+=1
  return array_text[:ii,:]

@nb.njit()
def get_image_data(image_data):
  list_text = []
  for y in range(image_data.shape[0]):
    for x in range(image_data.shape[1]):
      if image_data[y, x] < 210:
         #appending lists
         list_text.append([x, y])
         #appending tuples
         #list_text.append((x, y))
  return list_text
#Create some data
image_data=np.random.rand(1683*1240).reshape(1683,1240)*255
image_data=image_data.astype(np.uint8)


get_image_data (Pure Python)                   : 3.4s
get_image_data (naive Numba, appending lists)  : 1.1s
get_image_data (naive Numba, appending tuples) : 0.3s
get_image_data_arr:                            : 0.012s
np.argwhere(image_data<210)                    : 0.035s


Share : facebook icon twitter icon

Slow division in cython


By : Selvin
Date : March 29 2020, 07:55 AM
it fixes the issue Firstly, you need to call the functions many (>1000) times, and take an average of the time spent in each, to get an accurate idea of how different they are. Calling each function once will not be accurate enough.
Secondly, the time spent in the function will be affected by other things, not just the loop with divisions. Calling a def i.e. Python function like this involves some overhead in passing and returning the arguments. Also, creating a numpy array in the function will take time, so any differences in the loops in the two functions will be less obvious.
code :
@cython.boundscheck(False)
@cython.wraparound(False)
@cython.nonecheck(False)
@cython.cdivision(True) 
@cython.profile(True)
cdef double example1(double[:] xi, double[:] a, double[:] b, int D):

    cdef int k
    cdef double theSum = 0.0

    for k in range(D):
        theSum += (xi[k] - a[k]) / (b[k] - a[k])

    return theSum

@cython.boundscheck(False)
@cython.wraparound(False)
@cython.nonecheck(False)
@cython.profile(True)
@cython.cdivision(False)
cdef double example2(double[:] xi, double[:] a, double[:] b, int D):

    cdef int k
    cdef double theSum = 0.0

    for k in range(D):
        theSum += (xi[k] - a[k]) / (b[k] - a[k])

    return theSum


def testExamples():
    D = 100000
    x = np.random.rand(D)
    a = np.zeros(D)
    b = np.random.rand(D) + 1

    for i in xrange(10000):
        example1(x, a, b, D)
        example2(x, a, b,D) 
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
10000    1.546    0.000    1.546    0.000 test.pyx:26(example2)
10000    0.002    0.000    0.002    0.000 test.pyx:11(example1)

Slow multiprocessing with cython


By : Marina Lozovanu
Date : March 29 2020, 07:55 AM
wish helps you Cython can have translation costs if you go between C and Python types too much, which could contribute. There's also the fact that the speedup in Python will be higher, which hides overhead.
One suggestion is to use nogil functions and see whether threading has a lower overhead.

vectorization of looping on an array from cython


By : Alessandro Magnolo
Date : March 29 2020, 07:55 AM
help you fix your problem The only difference for the generated C code is that in inplace_addlocal the end variable for the loop is an int, while in inplace_add it's a Py_ssize_t.
Since your loop counter is an int, in the inplace_add version, there would be an aditional overhead due to casting between the two types when the comparison is performed.
code :
Py_ssize_t __pyx_t_1;
int __pyx_t_2;
int __pyx_t_3;
int __pyx_t_4;

__pyx_t_1 = (__pyx_v_a.shape[0]);
for (__pyx_t_2 = 0; __pyx_t_2 < __pyx_t_1; __pyx_t_2+=1) {
  __pyx_v_i = __pyx_t_2;
int __pyx_t_1;
int __pyx_t_2;
int __pyx_t_3;
int __pyx_t_4;

__pyx_v_n = (__pyx_v_a.shape[0]);
__pyx_t_1 = __pyx_v_n;
for (__pyx_t_2 = 0; __pyx_t_2 < __pyx_t_1; __pyx_t_2+=1) {
  __pyx_v_i = __pyx_t_2;

Why is this loop so slow in Cython?


By : JotaCê
Date : March 29 2020, 07:55 AM
Hope this helps You're appending to a bytes object with +=. That's really slow, since it has to copy the whole existing bytes object every time.
Don't do that. One better option would be to use a bytearray, and only build a bytes object out of the bytearray at the end.

Alternative to looping? Vectorisation, cython?


By : Satyavara
Date : March 29 2020, 07:55 AM
Does that help Here's another option, which separates the calculation of the rates/years matrix and appends it to the input df later on. Still does looping in the script itself (not "externalized" to some numpy / pandas function). Should be fine for 5k rows I'd guesstimate.
code :
import pandas as pd
import numpy as np

# def gen_df1():

# create the inital df without years/rates
df = pd.DataFrame({'Total': [100, 20, 30, 40, 10], 
                   'Yr_to_Use': [2020, 2021, 2021, 2019, 2020], 
                   'First_Year_Del': [5, 2, 7, 9, 10],
                   'Del_rate': [10, 5, 16, 18, 30]})

# get number of rates + remainder
n, r = np.divmod((df['Total']-df['First_Year_Del']), df['Del_rate'])

# get the year of the last rate considering all rows
max_year = np.max(n + r.astype(np.bool) + df['Yr_to_Use'])

# get the offsets for the start of delivery, year zero is 2019
offset = df['Yr_to_Use'] - 2019
# subtracting the year zero lets you use this as an index...

# get a year index; this determines the the columns that will be created
yrs = np.arange(2019, max_year+1)

# prepare a n*m array to hold the rates for all years, initalize with all zero
out = np.zeros((df['Total'].shape[0], yrs.shape[0]))
# n: number of rows of the df, m: number of years where rates will have to be payed

# calculate the rates for each year and insert them into the output array
for i in range(df['Total'].shape[0]):
    # concatenate: year of the first rate, all yearly rates, a final rate if there was a remainder
    if r[i]: # if rest is not zero, append it as well
        rates = np.concatenate([[df['First_Year_Del'][i]], n[i]*[df['Del_rate'][i]], [r[i]]])
    else: # rest is zero, skip it
        rates = np.concatenate([[df['First_Year_Del'][i]], n[i]*[df['Del_rate'][i]]])
    # insert the rates at the apropriate location of the output array:
    out[i, offset[i]:offset[i]+rates.shape[0]] = rates

# add the years/rates matrix to the original df    
df = pd.concat([df, pd.DataFrame(out, columns=yrs.astype(str))], axis=1, sort=False)
Related Posts Related Posts :
  • String Manipulation Recursive Function
  • Filter after Groupby and Sum in pandas?
  • writing a custom function Multiply the average of x,y
  • Spotify API fetch authorization code from redirect_uri
  • sklearn use RandomizedSearchCV with custom metrics and catch Exceptions
  • IndexOutOfRange error when filling a List Python
  • sns stripplot with just top n number of categories
  • Python classes keep calling eachother
  • How do I create a Dataframe_new in python from an existing Dataframe_old.
  • calculating an intercept point between a straight line and an ellipse - python
  • Integrating Tensorflow object detection with keras cnn classifier
  • How to skip comma while reading CSV file in python?
  • Stop Integrating when Output Reaches 0 in scipy.integrate.odeint
  • Changing the current graph of tf.placeholder objects in Tensorflow: Is it possible?
  • Logical error in while statement when used with or operator
  • django-rest-framework: int() argument must be a string, a bytes-like object or a number, not Deferred Attribute
  • how to remove a whitespace in a list in python?
  • How to reduce the number of row repetitions in a numpy array
  • Python: Dividing values of nested list with values with values of dictionary
  • Printing empty Pyramid
  • Python: How to save log file toSharePoint
  • Python Pandas count most frequent occurrences
  • How can I store / cache values from methods in a class for later use in other methods of the same class?
  • Sklearn: Pass class names to make_scorer
  • PyTorch - applying attention efficiently
  • How do I capitalize each parameter in a function definition using Python?
  • Regex matching of a bytes pattern gives unusual results - '.' not equivalent to [\x00-\xff]
  • I need help converting this REST API Curl command to Python requests
  • How do you make a variable comparison to decide a better score in a dice game?
  • How do I run sumo-gui on instant-veins-4.7.1-i1.ova
  • Deal with NAN values when creating models with python
  • Python requests: having a space in header for posting
  • Adding a column to a pandas dataframe based on cell values
  • Get mongod rs.status() results from a python script
  • ImportError: C extension: No module named 'parsing' not built
  • python pandas update column values related to previous updated row during iteration over it
  • 3 nested loops: Optimizing a simple simulation for speed
  • Assign subset of values to pandas dataframe with MultiIndex
  • How to group two sets of buttons on each top corner of the screen using Tkinter?
  • django login using class based for custom user
  • MRJob sort reducer output
  • Python Pandas Counts using rolling time window
  • Getting or editing a string from a column in a csv file with pandas
  • Python - Delete row in matrix/array if row contains
  • Using dicom Images with OpenCV in Python
  • Odoo ghost record
  • Creating and assigning multiple variables in a tkinter application
  • Graph dictionary
  • No changes to original dataframe after applying loop
  • AUC of Random forest model is lower after tuning parameters using hypergrid search and CV with 10 folds
  • Python: Reading multiple CSV files, and assigning each to a different variable
  • How to identify empty rectangle using OpenCV
  • How to iterate multilevel dataframe in python
  • How to limit the contour plot with a line plot?
  • Why subclassing a str or int behaves differently from subclising a list or dict?
  • Python decode with translation table
  • i need to click unordered links in the below URL using selenium, python
  • How to join pandas dataframe with itself?
  • How to apply a color cast to a video frame in OpenCV Python?
  • Is there any existing library for median filtering with kernel size greater then 5 using OpenCL acceleration in python?
  • shadow
    Privacy Policy - Terms - Contact Us © voile276.org