Submitting login form with scrapy

» python » Submitting login form with scrapy

By : Sachith Dassanayake

Date : November 23 2020, 03:01 PM

I think the issue was by ths following , Try to log in with browser and check which data should be sent.

code :

Share :

Login form not submitting

By : jasperbeckham

Date : March 29 2020, 07:55 AM

this will help I am creating a login form for my java app. But I can't get it to submit. I can't find any errors in my form. , Submit button type should be submit.

Scrapy and submitting a javascript form

By : Konstantin Drygin

Date : March 29 2020, 07:55 AM

I hope this helps you . Obviously the page consists of two frames, a short glance at the source reveals their names 'contents' and 'main'. So your script above nearly does the job, merely missing a single line pointing to the right frame called 'main' with driver.switch_to_frame('main'). Also the form does not react to ENTER-key, we indeed have to select the button and press it :-).
This code is working:

code :

import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

driver = webdriver.Firefox()
driver.get("http://agmarknet.nic.in/")
time.sleep(4)

driver.switch_to_frame('main')
textinput = driver.find_element_by_name('cmm')
textinput.send_keys("banana")
time.sleep(1)

button = driver.find_element_by_name("Go3")
button.click()
driver.close()

Scrapy - Submitting form with scraped results of other website

By : niexianghan

Date : March 29 2020, 07:55 AM

Does that help first Selector xpaths return a list, so what you have in item['title'] is a list, which you are passing into formdata={'query': item['title']} is that.
Now, if you still need to parse unhandled status (like still being able to parse a respone with 400 status), you should use the errback parameter in the request, something like this:

code :

...
    yield scrapy.FormRequest(url="https://www.themoviedb.org/", 
                             formdata={'query': item['title']},
                             callback=self.parse_tmdb,
                             errback=self.parse_error,
                             meta = {'item': item})
def parse_error(self, response):
    # do your magic
    ...
...

In my spring login form url getting changed after submitting the form

By : Bala Kumar

Date : March 29 2020, 07:55 AM

I think the issue was by ths following , You are missing your context path,
Into your login button add the next:

code :

<a href="${pageContext.request.contextPath}/login">Login</a>

<form action="${pageContext.request.contextPath}/login" method="POST">
    <button type="submit" class="button button-block"/>Get Started</button>
</form>

How can using scrapy to login scrapy without form element

By : raclefeast

Date : March 29 2020, 07:55 AM

fixed the issue. Will look into that further Web scraping is about requests and responses, so all you need is simulate all user requests. FormRequest just helps us to avoid extra work with forms. In this case you need make a proper login Request.
Go to needed page and open developer tools in your browser (e.g. Chrome) Check a preserve log option in Network tab. Fill credentials at the page and push login button. Find out the login request (after button was pressed) Check Headers tab in the request and find out request type and parameters (it can be GET with some querystring parameters or POST with some Form Data In your code try to reproduce the login request using a simple scrapy Request instead of FormRequest