Connecting to a website that does not have API (with requests) - python

I'm trying to connect to the following website with requests (in Python) - link.
I'm really new to dealing with web from Python so I have no idea what to do. As I understand (although I'm not sure) we need to create a certificate that contains the username and the password. I got the content of the page as follows:
r = requests.get(url,verify=False)
But I don't understand how to connect. Furthermore, how to keep the session alive so I could enter another page on the website and it won't ask for the username and password again?
EDIT: from previous thread I learn that I need to do something like:
import requests
LOGIN = 'https://www.hackthis.co.uk/index.php?login'
PROTECTED_PAGE = 'https://www.hackthis.co.uk/news'
payload = {
'username': 'VALIDUSERNAME',
'password': 'VALIDPASSWORD'
}
with requests.session() as s:
s.post(LOGIN, data=payload)
response = s.get(PROTECTED_PAGE)
print(response.text)
So i did:
payload = {
'TapuzLogin1$ctl01$UserName': 'USER',
'TapuzLogin1$ctl01$Password': 'PASS'
}
with requests.Session() as s:
p = s.post("https://www.tapuz.co.il/Common/SignInPage.aspx", verify=False,data=payload)
print(p.text)
But it doesn't connect. Should the username field be TapuzLogin1$ctl01$UserName and the password field TapuzLogin1$ctl01$Password? Also, Should I press connect afterwards or that should to the trict?

Related

Web scraping with python request - how to persist authentication after post?

Using python requests to persist authentication once I have posted credentials.
import requests
s = requests.session()
posturl= 'https://item-api-prod.liveauctioneers.com/auth/spalogin?c=20170802'
form = {'password': 'password', 'username': 'username'}
p=s.post(url=posturl, data = form)
So far authenticates successfully, but when try to get using the same session, the request hangs:
geturl = 'https://www.liveauctioneers.com/item/40018826_1930s-1950s-italian-german-hp-porcelain-children-items'
g=s.get(geturl)
...Hangs...
However if I get outside of the session, returns successfully but is not authenticated:
g2=requests.get(geturl)

Cookies must be enabled to use GitHub in requests module python

I want to check some usernames & password for logging in to GitHub.
I Inspected & got that the first input name of github login is "login" & second one is "password" & "Form" uses "POST" method. I have written this code but when I want to check this code, It tells me (Cookies must be enabled to use GitHub)
allData = {
'login': x,
'password': y
}
siteRequest = requests.post('https://github.com/login', data=allData)
print(siteRequest.content, end='\n\n')
How should I enable cookies ?
Note : I know github has API but some sites don't, So I want to increase my ability & knowledge about this method ... Thanks!
GitHub's login page sets a cookie with a session ID on load, this session is used to keep you logged in.
To solve your problem simply create a requests session object, this will keep track of the cookies, then use the same session to post your login.
allData = {
'login': x,
'password': y
}
site_session = requests.Session()
_ = site_session.get('https://github.com/login') # This is just to get the initial
# cookie in the session
siteRequest = site_session.post('https://github.com/login', data=allData)
print(siteRequest.content, end='\n\n')

Odoo 8 controller authentication cross site

I have odoo 8 controller which returns certain data that is only accessible to logged users. Web site is located on different server then odoo itself. How can i request authentication from my page (as mentioned on different server)?
p.s. On my development environment it all works since im calling from localhost/index.html requesting to a localhost/getData (through a reverse proxy to localhost:8069/getData for cors). But when i put it on our real server it responds with some missing session error
Thank you
You need to authenticate from your remote server and obtain a session_id. Once you have this, pass it as a cookie with subsequent requests.
import requests
from requests import Request,Session
import json
b_url = "http://yourodooserver.com"
# or "http://201.100.100.12:8069" (or whatever the ip is)
url = "{}/web/session/authenticate".format(b_url)
db = "<YOURDBNAME>"
user = "<YOURUSERNAME>"
passwd = "<YOURPASSWORD>"
s = Session()
data = {
'jsonrpc':'2.0',
'params': {
'context': {},
'db': db,
'login': user,
'password': passwd,
},
}
headers = {
'Content-type': 'application/json'
}
req = Request('POST',url,data=json.dumps(data),headers=headers)
prepped = req.prepare()
resp = s.send(prepped)
session_id = json.loads(resp.text)['result']['session_id']
# NOW MAKE REQUESTS AND PASS YOUR SESSION ID
res = requests.get(b_url + "/your/controller/path",cookies={'session_id':str(session_id)})
print(res.text)

Posting Request Data

I am trying to post requests with Python to register an account.
It is not creating the account.
Any help would be great!
It has to accept the user's email and password and confirmation of their password.
import requests
with requests.Session() as c:
url="http://icebithosting.com/register.php"
EMAIL="charliep1551#gmail.com"
PASSWORD = "test"
c.get(url)
login_data= dict(username=EMAIL,password=PASSWORD,confirm=PASSWORD)
c.post(url, data=login_data,)
page=c.get("http://icebithosting.com/")
print (page.content)
Your form file names are incorrect, they should be:
email:'foo#bar.com'
password:'bar'
confirm_password:'bar' # confirm_password
Which you can see if you monitor the request in chrome tools:

using mattermost api via gitlab oauth as an end-user with username and password (no client_secret)

In our team we use gitlab (https://git.example) and the bundled mattermost chat (https://chat.example).
For mattermost we'd like to have a dedicated bot-user (the web hooks have limitations wrt. private channels etc.), which actually logs in exactly like a normal user.
We created that user in gitlab, and can log into our chat with it via chrome (chat login redir --> gitlab oauth, type in username and pw --> redir back to chat --> authed).
Now i searched for python libraries that can actually do this, but i can only find some which require a client_id and client_secret... From my understanding (please correct me if i'm wrong) this is not what we're searching for, as we don't want to create another application to auth via gitlab, but login to our chat (which already has an id (known) and a secret (unknown)) as a user via gitlab.
As we couldn't find such a lib, we also inspected the network requests in chrome and tried to re-create it in python via requests, but can't get it to work (needless to say it involves parsing html and csrf tokens)...
Taking yet another attempt and a lot of guesswork we tried to acquire an access_token manually via
client_id = 'the one of mattermost in our gitlab'
user = 'username'
pw = 'password'
r = requests.post(
'https://git.example/oauth/token',
data={
"grant_type": "password",
"username": user,
"password": pw,
"client_id": client_id,
}
)
access_token = r.json()['access_token']
This seems to work (and the token looks good), but using it in the mattermost API only results in a 401:
ri = requests.get(
'https://chat.example/api/v1/users/me',
headers={'Authorization': 'Bearer ' + access_token}
)
ri.status_code, ri.json()
(401,
{u'detailed_error': u'token=...xxx...',
u'id': u'api.context.session_expired.app_error',
u'is_oauth': False,
u'message': u'Invalid or expired session, please login again.',
u'request_id': u'...yyy...',
u'status_code': 401})
Sadly http://docs.mattermost.com/developer/web-service.html#oauth2 currently doesn't shed more light into this, which is why I'm asking here. Did i maybe miss something obvious to "activate" that access_token in mattermost?
Actually i finally got this running by imitating the browser's behavior as follows, but i'd still be interested in a more generic solution that doesn't involve parsing any of the gitlab server's html...:
import requests
from pyquery import PyQuery as pq
client_id = '...your_mattermost_client_id...'
user = '...username...'
pw = '...userpass...'
gitlab = 'https://git.example'
chat = 'https://chat.example'
team = '/your_team'
r = requests.get(
chat + team + '/login/gitlab'
)
q = pq(r.content)
csrf_token = q('#new_ldap_user input[name="authenticity_token"]')[0].value # watch out, several tokens for ldap vs. normal login, inspect the page to find correct one
r2 = requests.post(
gitlab + '/users/auth/ldapmain/callback', # or whatever the browser callback for your auth-method was
cookies=r.cookies,
data={
'authenticity_token': csrf_token,
'username': user,
'password': pw,
}
)
# print [h.url for h in r2.history] + [r2.url] # to check the redirects
me = requests.get(
chat + '/api/v1/users/me',
cookies=r2.cookies,
)
print me.json() # if everything went well you're now authorized
# remember to set cookies in the follow-up requests

Resources