Concatenate SQL string with args - python

I am trying to use a default query and then append some conditions. I would like to know how can i concatenate a string while unpacking args.
filters = []
if industry:
filters.append('industry_id')
filters.append('role_id')
...
As far as I can see this doesn't work (invalid syntax).
print "SELECT city, " + *filters + " FROM histogram"

Use join to do what you need:
print "SELECT city, " + ', '.join(filters) + " FROM histogram"
which prints:
SELECT city, industry_id, role_id FROM histogram

Related

What to look for when assembling user generated SQL Sever query in Python from excel?

I am using Excel as a form for users to create, delete and update entries in a SQL Server Table. I am then taking this input into Python via a Data Frame, and creating a SQL string. I then execute it via pyodbc cursor. For instance, below is how I can get a valid and functional Update query.
ParamstoPass=len(ClassCheckMark.columns)
L_Cols=list()
L_Vals=list()
tableName=ClassCheckMark[ClassCheckMark.columns[1]][0]
SQL_Query='update ' + tableName + ' set '
for i in range(2, ParamstoPass):
L_Cols.append(ClassCheckMark[ClassCheckMark.columns[i]].name)
L_Vals.append(ClassCheckMark[ClassCheckMark.columns[i]][0])
for i in range(1, len(L_Cols)):
SQL_Query=SQL_Query+'[' + L_Cols[i] +']=' +"'" + str(L_Vals[i]) +"', "
SQL_Query=SQL_Query[:-2]+' where ID=' + "'" + str(L_Vals[0]) +"'"
cursor.execute(SQL_Query)
cnn.commit()
cnn.close()
But I know there are some undesirable characters that a user can may enter in Excel that will then make it into the query.
So what is the best way to validate the SQL String in python? Should I look for specific characters like '\', "\0", "\n", "\r", "'", '"', "\x1a"? Or what is the best industry method for this objective?
And I realize that in general this is not the best way to accomplish the goal of user interaction with a DB, but due to various constrains am going with this approach.
Thank you.
After building your L_Cols and L_Vals lists I would suggest validating the column names against the table metadata, constructing a parameterized SQL command, and then executing it. For example:
# test data
L_Cols = ['ID', 'FirstName', 'Photo']
L_Vals = [123, 'bob', None]
tablename = "People"
# validate list of column names
valid_column_names = [x.column_name for x in cursor.columns(tablename).fetchall()]
for col_name in L_Cols:
if col_name not in valid_column_names:
raise ValueError("[{0}] is not a valid column name for table [{1}]".format(col_name, tablename))
# build SQL command text
SQL_Query = "UPDATE [" + tablename + "] SET "
SQL_Query += ", ".join("[" + x + "]=?" for x in L_Cols[1:])
SQL_Query += " WHERE [" + L_Cols[0] + "]=?"
print(SQL_Query) # UPDATE [People] SET [FirstName]=?, [Photo]=? WHERE [ID]=?
# move ID value to the end of the list of parameters
params = L_Vals[1:] + L_Vals[0:1]
print(params) # ['bob', None, 123]
# (edit by OP)
# as in my case, some elements were of unicode markup, which threw
# ProgrammingError: ('Invalid parameter type. param-index=0
# param-type=numpy.int64', 'HY105').
# May need to add params=[str(x) for x in params]
cursor.execute(SQL_Query, params)

How to make dynamic updates to the database using a list in python

if year in Year:
#print 'executing'
for rows in range(1,sheet.nrows):
records = []
FIP = str(sheet.cell(rows, 1).value)
for cols in range(9,sheet.ncols):
records.append(str(sheet.cell(rows,cols).value))
cur.execute("UPDATE " + str(table_name) + " SET " + (str(variables[0]) + "= \'{0}\', ".format(records[0])
+ str(variables[1]) + " = \'{0}\', ".format(records[1])
+ str(variables[2]) + " = \'{0}\', ".format(records[2])
+ str(variables[3]) + " = \'{0}\', ".format(records[3])
+ str(variables[4]) + " = \'{0}\',".format(records[4])
+ str(variables[5]) + " = \'{0}\', ".format(records[5])
+ str(variables[6]) + " = \'{0}\' ".format(records[6])+
"WHERE DATA_Year='2010'AND FIPS='{0}'".format(FIP)))
The above code is updating 7 columns whose names are stored in the list 'variables'.
I want to make it dynamic so that if number of elements(columns) in the list 'variables' is increased, it should update all the columns and not just 7.
I tried doing that using this code:
if year in Year:
#print 'executing'
for rows in range(1,sheet.nrows):
records = []
FIP = str(sheet.cell(rows, 1).value)
for cols in range(9,sheet.ncols):
records.append(str(sheet.cell(rows,cols).value))
for x in range(0,len(variables)):
#print x
cur.execute("UPDATE " + str(table_name) + " SET " + (str(variables[x])) + "= \'{0}\', ".format(records[x])
+ "WHERE DATA_Year='2010' AND FIPS='{0}'".format(FIP))
But I am getting the error:
pypyodbc.ProgrammingError: (u'42000', u"[42000] [Microsoft][ODBC SQL Server Driver][SQL Server]Incorrect syntax near the keyword 'WHERE'.")
It would be great if someone can help me figure out what's wrong with my code and whether there is a alternate way of doing what I am trying to do.
You will find it easier to use parameter-substitution. See params, and note that execute takes a sequence argument. Then that line starts to look something like,
cur.execute(sql, records)
If memory permits (it probably does), you may find executemany performs better: you call it once with an array of records.
With that in mind, the dynamic part of your question comes into focus. Construct the parameterized query string as you iterate over cols. When that's done, you should have a matched set (in effect) of parameter placeholders and elements in records. Then tack on the WHERE clause, append FIP to records, and execute.
HTH.

TypeError: not all arguments converted during string formatting. What am I not doing right

I have tried:
sql = "INSERT INTO "+ tablenane + " ("+ ",".join(headers) +") VALUES (" + ",".join(["%s"] * len(headers)) + ")"
and:
sql = "INSERT INTO "+ tablenane + " ("+ ",".join(headers) +") VALUES (" + ",".join(["?"] * len(headers)) + ")"
but still get the same error when I run cursor.executemany(sql, dataset)
dataset = [(a,b,c),(d,e,f)] - A list of tuples
Both columns and values are dynamic, because there 40 columns.
All table columns are of type VARCHAR and I ensure the data is converted to string before insertion
Using python 2.7 and mysql
I read somewhere that MySQL-python does not allow "?".
All I was missing was the db.commit()
Final query was
sql = "INSERT INTO "+ tablenane + " ("+ ",".join(headers) +") VALUES (" + ",".join(['%s'] * len(headers)) + ")"
cursor.executemany(sql, dataset)
db.commit(). Though not sure why insertion needs a commit yet creation does not.
Thanks

Is it possible to query by list in Hive?

I have a list of ids
[id1, id2, id3.......]
I would like to query all the rows such that ids match one of the id in the list
Right now I am doing it in a hacky way
# This is a python script
id_list_str = "`_id` = '" + str(_id[0]) + "' "
for m_id in _id[1:]:
id_list_str += " OR `_id` = '" + str(m_txn_id) + "' "
hive_query = "SELECT" \
" `_id`, " \
" time, " \
" state " \
"FROM " \
" transaction " \
"WHERE " \
" %s " \
% (id_list_str)
I don't think this scales as the number of id in my list increase, is there another way of doing this?
Thanks
In hive try a lateral view explode to expand the list. This will work if you have many lists as well.
select a.id
, a.time
, a.state
from transaction a
left semi join
(SELECT distinct id
from list_of_ids LATERAL VIEW explode(id_list_Str) idTable as id
)
t on a.id = t.id
Which Hive version are you using?
Hive 0.13 now do support IN/EXISTS in the WHERE-clause .. The issue https://issues.apache.org/jira/browse/HIVE-784

Django Raw Query

lists = []
lists = cursor.execute(
"select a.employee_id,a.first_name,"
"c.breakfast_count, b.lunch_count,"
"d.dinner_count from qpscsmas_employee_details a,"
"qpscsmas_emp_lunch b,"
"qpscsmas_emp_breakfast c,"
"qpscsmas_emp_dinner d "
"where a.rfidcardno = %s "
"and b.rfidcardno = %s "
"and c.rfidcardno = %s "
"and d.rfidcardno = %s,"
"['s','s','s','s']"
)
print lists
The above query is giving me an empty list.
The problem with this is that it is not showing any error also.
It is working fine in MySQL when i am giving the exact values in place of the parameters
First you need to remove the double quotes from your parameters list and separate it with comma from your query string, e.g.:
cursor.execute("SELECT foo FROM bar WHERE baz = %s", [self.baz])
After you executed the query you need to fetch the rows, see Executing custom SQL directly. You may fetch the next one (cursor.fetchone()) or fetch all rows (cursor.fetchall()).
Examples:
row = cursor.fetchone()
rows = cursor.fetchall()
This cleaned up query should work:
cursor.execute(
"select a.employee_id,a.first_name,"
"c.breakfast_count, b.lunch_count,"
"d.dinner_count from qpscsmas_employee_details a,"
"qpscsmas_emp_lunch b,"
"qpscsmas_emp_breakfast c,"
"qpscsmas_emp_dinner d "
"where a.rfidcardno = %s "
"and b.rfidcardno = %s "
"and c.rfidcardno = %s "
"and d.rfidcardno = %s",
['s','s','s','s']
)

Resources