logo
down
shadow

Scipy fmin_tnc not optimizing cost function


Scipy fmin_tnc not optimizing cost function

By : Alan Campos
Date : November 19 2020, 03:01 PM
Any of those help Your gradient is broken! (somehow: math vs. np.shapes?)
I'm not gonna analyze your gradient (looks like logistic regression; read this then), but here a demo which supports my claim:
code :
from scipy.optimize import check_grad
from sklearn.datasets import make_classification
X_examples, Y_labels = make_classification()

...

print(check_grad(cost, gradient, np.random.random(size=20), X_examples, Y_labels))
# 61.666912359 (example value; indeterministic because of PRNG-usage above)
# result: The square root of the sum of squares (i.e. the 2-norm) of the difference between
# grad(x0, *args) and the finite difference approximation of grad using func
# at the points x0.
result = opt.fmin_tnc(func=cost, x0=x0, fprime=None, approx_grad=True, args=
(X_examples, Y_labels))
NIT   NF   F                       GTG
  0    1  6.931471805599453E+01   3.69939729E+03
tnc: fscale = 0.0164412
  1   14  5.488792098836968E+01   1.91876351E+03
  2   26  4.598767699927674E+01   5.90835511E+02
  3   39  4.255784649333560E+01   3.42105829E+02
  4   44  3.153577598035866E+01   3.09160832E+02
  5   50  2.224511577357391E+01   4.36685983E+01
  6   54  2.157944424362721E+01   3.39632081E+01
  7   59  2.136340974081865E+01   2.97596794E+01
  8   73  1.997400905570375E+01   1.08022452E+01
  9   75  1.984787529493228E+01   1.05379689E+01
 10   79  1.979578396181381E+01   1.16542972E+01
 11   88  1.939906531954665E+01   9.19521103E+00
tnc: fscale = 0.329776
 12   91  1.867469105176042E+01   4.61533306E+00
 13   94  1.834698220306902E+01   1.37837652E+00
 14   98  1.818150860822102E+01   1.39090344E+00
 15  102  1.817553527302105E+01   1.22472879E+00
 16  118  1.810790027768505E+01   6.81994565E-01
 17  134  1.807103645037105E+01   1.11197844E+00
 18  148  1.805232746344993E+01   1.31428979E+00
 19  154  1.804213064407819E+01   7.10363935E-01
 20  168  1.803844265918305E+01   5.98840568E-01
 21  170  1.803426849431969E+01   7.44231905E-01
 22  175  1.803266497733540E+01   8.68201237E-01
 23  189  1.799550032713761E+01   1.13982866E+00
 24  191  1.799048283812780E+01   7.41273919E-01
 24  200  1.799048283812780E+01   7.41273919E-01
tnc: Maximum number of function evaluations reached


Share : facebook icon twitter icon
How to return cost, grad as tuple for scipy's fmin_cg function

How to return cost, grad as tuple for scipy's fmin_cg function


By : Kiran S
Date : March 29 2020, 07:55 AM
will help you You can use scipy.optimize.minimize with jac=True. If that's not an option for some reason, then you can look at how it handles this situation:
code :
class MemoizeJac(object):
    """ Decorator that caches the value gradient of function each time it
    is called. """
    def __init__(self, fun):
        self.fun = fun
        self.jac = None
        self.x = None

    def __call__(self, x, *args):
        self.x = numpy.asarray(x).copy()
        fg = self.fun(x, *args)
        self.jac = fg[1]
        return fg[0]

    def derivative(self, x, *args):
        if self.jac is not None and numpy.alltrue(x == self.x):
            return self.jac
        else:
            self(x, *args)
            return self.jac
fmemo = MemoizeJac(f, fprime)
xopt = fmin_cg(fmemo, x0, fmemo.derivative)
class MemoizeJac(object):
    def __init__(self, fun):
        self.fun = fun
        self.value, self.jac = None, None
        self.x = None

    def _compute(self, x, *args):
        self.x = numpy.asarray(x).copy()
        self.value, self.jac = self.fun(x, *args)

    def __call__(self, x, *args):
        if self.value is not None and numpy.alltrue(x == self.x):
            return self.value
        else:
            self._compute(x, *args)
            return self.value

    def derivative(self, x, *args):
        if self.jac is not None and numpy.alltrue(x == self.x):
            return self.jac
        else:
            self._compute(x, *args)
            return self.jac
Cost Function and Gradient Seem to be Working, but scipy.optimize functions are not

Cost Function and Gradient Seem to be Working, but scipy.optimize functions are not


By : Adrian Roberts
Date : March 29 2020, 07:55 AM
should help you out This was a very difficult problem to debug, and illustrates a poorly documented aspect of the scipy.optimize interface. The documentation vaguely indicates that theta will be passed around as a vector:
code :
minimize f(x) subject to

g_i(x) >= 0,  i = 1,...,m
h_j(x)  = 0,  j = 1,...,p 
theta = theta.reshape(-1, 1)                                           
scipy.minimize -- get cost function vs iteration?

scipy.minimize -- get cost function vs iteration?


By : Mohammed Abd Alaziz
Date : March 29 2020, 07:55 AM
this one helps. I figured out a sort of hack using stdlib features, it uses a "deep" redirect of sys.stdout. Note that this does not work with jupyter since IPython hijacks sys.stdout, which removes the .fileno attribute.
It may be possible to patch Jupyter using a tempfile.SpooledTemporaryFile in this way, removing this issue. I don't know.
code :
import os
import sys
import tempfile

class forcefully_redirect_stdout(object):
    ''' Forces stdout to be redirected, for both python code and C/C++/Fortran
        or other linked libraries.  Useful for scraping values from e.g. the
        disp option for scipy.optimize.minimize.
    '''
    def __init__(self, to=None):
        ''' Creates a new forcefully_redirect_stdout context manager.

        Args:
            to (`None` or `str`): what to redirect to.  If type(to) is None,
                internally uses a tempfile.SpooledTemporaryFile and returns a UTF-8
                string containing the captured output.  If type(to) is str, opens a
                file at that path and pipes output into it, erasing prior contents.

        Returns:
            `str` if type(to) is None, else returns `None`.

        '''

        # initialize where we will redirect to and a file descriptor for python
        # stdout -- sys.stdout is used by python, while os.fd(1) is used by
        # C/C++/Fortran/etc
        self.to = to
        self.fd = sys.stdout.fileno()
        if self.to is None:
            self.to = tempfile.SpooledTemporaryFile(mode='w+b')
        else:
            self.to = open(to, 'w+b')

        self.old_stdout = os.fdopen(os.dup(self.fd), 'w')
        self.captured = ''

    def __enter__(self):
        self._redirect_stdout(to=self.to)
        return self

    def __exit__(self, *args):
        self._redirect_stdout(to=self.old_stdout)
        self.to.seek(0)
        self.captured = self.to.read().decode('utf-8')
        self.to.close()

    def _redirect_stdout(self, to):
        sys.stdout.close() # implicit flush()
        os.dup2(to.fileno(), self.fd) # fd writes to 'to' file
        sys.stdout = os.fdopen(self.fd, 'w') # Python writes to fd

if __name__ == '__main__':
    import re
    from scipy.optimize import minimize

    def foo(x):
        return 1/(x+0.001)**2 + x

    with forcefully_redirect_stdout() as txt:
        result = minimize(foo, [100], method='L-BFGS-B', options={'disp': True})

    print('this appears before `disp` output')
    print('here''s the output from disp:')
    print(txt.captured)
    lines_with_cost_function_values = \
        re.findall(r'At iterate\s*\d\s*f=\s*-*?\d*.\d*D[+-]\d*', txt.captured)

    fortran_values = [s.split()[-1] for s in lines_with_cost_function_values]
    # fortran uses "D" to denote double and "raw" exp notation,
    # fortran value 3.0000000D+02 is equivalent to
    # python value  3.0000000E+02 with double precision
    python_vals = [float(s.replace('D', 'E')) for s in fortran_values]
    print(python_vals)
Linear regression: Cost function working independently, but not within a scipy.optimize function

Linear regression: Cost function working independently, but not within a scipy.optimize function


By : SHENGXU ZHAO
Date : March 29 2020, 07:55 AM
will help you I am implementing regularised linear regression. The data can be found here: https://onedrive.live.com/?cid=506A31CDF6E4A865&id=506A31CDF6E4A865%21107&parId=root&o=OneUp , The issue is in defining the parameters array as
code :
theta = np.ones((2, 1))
ValueError: shapes (12,2) and (1,2) not aligned: 2 (dim 1) != 1 (dim 0)
theta = np.ones(2)
import numpy as np 
import matplotlib.pyplot as plt 
import scipy.optimize as optimize 
from scipy.io import loadmat

data = loadmat('ex5data1.mat')
x = data['X']
X = data['X']
X = np.insert(X, 0, 1, axis=1)
y = data['y']
theta = np.ones(2)

def cost_function(theta, X, y, reg_param):
    theta = np.matrix(theta)
    X = np.matrix(X)
    y = np.matrix(y)
    m = float(y.shape[0])
    h = X * theta.T
    error = np.power((h - y), 2)
    error = np.sum(error)
    term = error / (2*m)
    reg = (reg_param * np.sum(np.power(theta[1:, :], 2))) / (2*m)

return term + reg

print "Cost function: \n %s" % (cost_function(theta, X, y, 1))

def cost_function_gradient(theta, X, y, reg_param):
    theta = np.matrix(theta)
    X = np.matrix(X)
    y = np.matrix(y)
    m = float(y.shape[0])

    grad = np.zeros((len(X[0]) + 1, 1))
    reg = np.multiply(theta.T[1:, :], reg_param/m)

    for j in xrange(len(X[0])):
        term = np.multiply((X * theta.T) - y, X[:, j + 1])
        term = np.sum(term) / m
        grad[j + 1, 0] = term + reg

    grad[0, 0] = np.sum(np.multiply((X*theta.T - y), X[:, 0])) / m

    return grad

print "Cost function gradient: \n %s" % (cost_function_gradient(theta, X, y, 1))

reg_param = 0
opt = optimize.fmin_cg(cost_function, theta, args=(X, y, reg_param), maxiter=200)
optimize.fmin_tnc is not giving right answer in scipy.optimize?

optimize.fmin_tnc is not giving right answer in scipy.optimize?


By : Althea Schutte
Date : March 29 2020, 07:55 AM
Does that help I did some tests by changing shapes of arrays, flattening it, reshaping it but nothing worked.
Since we were inputting a one dimension theta in fmin_tnc by flattening the theta, so i thought of changing the gradient function assuming that it will receive a single dimension theta instead of 3*1.
code :
def grad( theta, X, Y):
    temp = (1/m) * X.T @ (sigmoid(X @ theta) - Y)  
    return temp
def grad( theta, X, Y):
    temp = (1/m) * (X.T @ (sigmoid(X @ theta[:,np.newaxis]) - Y))  
    return temp
Related Posts Related Posts :
  • Element disappears when I add an {% include %} tag inside my for loop
  • Django Rest Framework with either a slug or a pk lookup field for the DetailAPIView
  • Flask doesn't stream on Lambda
  • Generate all permutations of fixed length where the elements come from two different sets
  • Making function for calculating distance
  • How to handle multiprocessing based on the limit of CPU's
  • Django - static files is not working
  • Remove x axis and y axis black lines with matplotlib
  • tkinter: assigning multiple functions to one button
  • flask-jwt-extended: Fake Authorization Header during testing (pytest)
  • Setting pandas dataframe value based on row and column conditions
  • swig char ** as a pointer to a char *
  • Confusion over `a` and `b` attributes from scipy.stats.uniform
  • How can I do groupy.apply() without sort my index?
  • Querying Google Cloud datastore with ancestor not returning anything
  • Read value from one thread in Python: queue or global variable?
  • Django - context process query being repeated 102 times
  • Convert a list of images and labels to np array to train tensorflow
  • Lambda not supporting NLTK file size
  • Numpy ndarray image pixel mean for pixel values greater than zero: Normalizing image
  • Understanding output of np.corrcoef for two matrices of different sizes
  • Finding longest perfect match between two strings
  • what is wrong with my cosine similarity? Tensorflow
  • How to manage user content in django?
  • Receiving unsupported operand error while comparing random number and user input.
  • How to wrap the process of creating start_urls in scrapy?
  • How to mark 'duplicated sequence' in pandas?
  • Boolean indexing on multidimensionnal array
  • Unmodified column name index in patsy
  • Cleaner way to unpack nested dictionaries
  • Importing a python module to enable a script to be run from command line
  • Maya Python read and set optionMenu value via variable
  • How can I bind a property to another property in Kivy?
  • Python extracting specific line in text file
  • How to implement n-body simulation with pymunk?
  • Python / matplotlib: print to resolution and without white space / borders / margins
  • Sum up the second value from one dictionary with all values from another dictionary
  • Robot Framework: Open a chrome browser without launching URL
  • Generate inline Bokeh scatterplots in Jupyter using a for loop
  • Group list of dictionaries python
  • Efficient way to apply multiple Boolean mask to set values in a column using pandas
  • Lazy evaluation of a Python dictionary
  • id of xpath is getting changed every time in selenium python 2.7 chrome
  • Matplotlib RuntimeWarning displaying a 3D plot
  • Cannot install pyqt5 for python3.4 on windows 10
  • Gravity Problems
  • Where to position `import` modules inside an class?
  • Python OpenCV: Cannot resize image
  • Print on the same spot in IPython console
  • Disable logging except in tests
  • Writing json to file in s3 bucket
  • Sorting numpy array created by laspy
  • Open an XML file through URL and save it
  • How to build a 2-level dictionary?
  • error installing scipy using pip on windows 10
  • __str__ from my own matrix, python
  • python re how to Extract fields use findall()?
  • how to read a value from text HI file using python?
  • How to use horizontal scrolling in treeview,here i use tree view to make a table
  • Dependant widgets in tkinter
  • shadow
    Privacy Policy - Terms - Contact Us © voile276.org