Iterating through a nested list to produce min and max values ignoring NoneType - python

I need to find the min and max values in a transposed nested list, ignoring any none types.
This the nested list I have:
x = [[1, 20, 50],
[5, 6, 7],
[11, 42, 2],
[7, 32, None]]
I am wanting to ignore the None in the third column and would expect to have the following output:
min
[1, 6, 2]
max
[11,42,50]
I need to do this using the standard python library

Pure python solution:
In [16]: x = [[1, 20, 50],
...: [5, 6, 7],
...: [11, 42, 2],
...: [7, 32, None]]
...:
In [17]: [min((y for y in x if y is not None), default=None) for x in zip(*x)]
Out[17]: [1, 6, 2]
In [18]: [max((y for y in x if y is not None), default=None) for x in zip(*x)]
Out[18]: [11, 42, 50]
Note that for [[None]] the code above returns [None] as there are neither no min nor max elements. If you want this code to raise an exception just remove default=None. If you want to exclude None from the resulting list just wrap with a list comprehension like [z for z in (...) if z is not None]
Numpy solution with casting to float to automatically convert None to nan:
In [12]: import numpy as np
In [13]: a = np.array(
...: [[1, 20, 50],
...: [5, 6, 7],
...: [11, 42, 2],
...: [7, 32, None]],
...: dtype=np.float)
...:
In [14]: np.nanmin(a, axis=0).astype(np.int)
Out[14]: array([1, 6, 2])
In [15]: np.nanmax(a, axis=0).astype(np.int)
Out[15]: array([11, 42, 50])

Related

How to “zip” several N-D arrays in Numpy?

The conditions are following:
1) we have a list of N-D arrays and this list is of unknown length M
2) dimensions each arrays are equal, but unknown
3) each array should be splitted along 0-th dimension and resulting elements should be grouped along 1-st dimension of length M and then stacked back along 0-th dimension of the same length it was
4) resulting rank should be N+1 and the lenght of 1-st dimension should be M
Above is the same as zip, but in the world of N-D arrays.
Currently I do the following way:
xs = [list of numpy arrays]
grs = []
for i in range(len(xs[0])):
gr = [x[i] for x in xs]
gr = np.stack(gr)
grs.append(gr)
grs = np.stack(grs)
Can I write shorter with bulk operations?
UPDATE
Here is what I want
import numpy as np
sz = 2
sh = (30, 10, 10, 3)
xs = []
for i in range(sz):
xs.append(np.zeros(sh, dtype=np.int))
value = 0
for i in range(sz):
for index, _ in np.ndenumerate(xs[i]):
xs[i][index] = value
value += 1
grs = []
for i in range(len(xs[0])):
gr = [x[i] for x in xs]
gr = np.stack(gr)
grs.append(gr)
grs = np.stack(grs)
print(np.shape(grs))
This code apparantly works correctly, producing arrays of shape (30, 2, 10, 10, 3). Is it possible to avoid loop?
Seems you need to transpose the array with respect to its 1st and 2nd dimension; You can use swapaxes for this:
np.asarray(xs).swapaxes(1,0)
Example:
xs = [np.array([[1,2],[3,4]]), np.array([[5,6],[7,8]])]
grs = []
for i in range(len(xs[0])):
gr = [x[i] for x in xs]
gr = np.stack(gr)
grs.append(gr)
grs = np.stack(grs)
grs
#array([[[1, 2],
# [5, 6]],
# [[3, 4],
# [7, 8]]])
np.asarray(xs).swapaxes(1,0)
#array([[[1, 2],
# [5, 6]],
# [[3, 4],
# [7, 8]]])
np.stack takes an axis parameter; looking at the shape of grs, I guessed that np.stack(xs, 1) does the same thing.
In [490]: x
Out[490]:
array([[[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]],
[[12, 13, 14, 15],
[16, 17, 18, 19],
[20, 21, 22, 23]]])
In [491]: x.shape
Out[491]: (2, 3, 4)
In [494]: xs = [x, x+10, x+100]
In [495]: grs = []
...: for i in range(len(xs[0])):
...: gr = [x[i] for x in xs]
...: gr = np.stack(gr)
...: grs.append(gr)
...: grs = np.stack(grs)
...:
In [496]: grs
Out[496]:
array([[[[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]],
[[ 10, 11, 12, 13],
[ 14, 15, 16, 17],
[ 18, 19, 20, 21]],
[[100, 101, 102, 103],
[104, 105, 106, 107],
...
[116, 117, 118, 119],
[120, 121, 122, 123]]]])
In [497]: grs.shape
Out[497]: (2, 3, 3, 4)
testing np.stack:
In [499]: np.allclose(np.stack(xs, 1),grs)
Out[499]: True

Python Numpy generate coordinates for X and Y values in a certain range

I am trying to generate an array of coordinates that will lie on an image.
The image being used is a 640x480 pixel frame. I have been able to make an array of all the x and y points that I wish to use. I am trying to plot small circles at each of these points just for visualization and for later use so I am trying to get the coordinates into a form that can be entered into the OpenCV circle function. Below is what I have got so far:
Ypts = np.arange(5, 480, 5)
Xpts = np.arange(5, 640, 5)
I have tried using
[pts]= np.vstack([Xpts, Ypts]).T
and
coordinate = []
for x in range(Xpts.size):
for y in range(Ypts.size):
coordinate.append((x, y))
With no success, the output I get for coordinate is [0, 0], [0, 1], [0, 2], ... instead of points relating to the values of Xpts and Ypts.
On a smaller scale this is an example of what the x and y arrays are:
Xpts = [5, 10, 15, 20, 25, 30, 35]
Ypts = [5, 10, 15]
and what i am trying to get to as an answer is:
Points = [[5, 5],
[5, 10],
[5, 15],
[10, 5],
[10, 10],
[10, 15],
[15, 5],
[15, 10],
......,
[35, 15]]
You can use itertools.product which will basically return all possible combinations of the provided lists. See this code for an example:
import numpy as np
import itertools
Xpts=[5, 10, 15, 20, 25, 30, 35]
Ypts=[5, 10, 15]
Points = np.array(list(itertools.product(Xpts, Ypts)))
This will return the following:
array([[ 5, 5],
[ 5, 10],
[ 5, 15],
[10, 5],
[10, 10],
[10, 15],
[15, 5],
[15, 10],
[15, 15],
[20, 5],
[20, 10],
[20, 15],
[25, 5],
[25, 10],
[25, 15],
[30, 5],
[30, 10],
[30, 15],
[35, 5],
[35, 10],
[35, 15]])
You can generate all the possible combinations with numpy's mgrid:
>>> py, px = np.mgrid[5:480:5, 5:640:5]
>>> points = np.c_[py.ravel(), px.ravel()]
>>> points
array([[ 5, 5],
[ 5, 10],
[ 5, 15],
...,
[475, 625],
[475, 630],
[475, 635]])
People have already suggested numpy.mgrid and itertools. Use them.
However, for the sake of education, Python provides us with a powerful concept known as "list comprehension". You can obtain the Cartesian product by running:
[(a,b) for a in range(5, 485, 5) for b in range(5, 645, 5)]
Change (a,b) to [a,b] if you don't want a list of tuples.
Inspired by this solution and with focus on performance, you can use np.meshgrid as well -
X2D,Y2D = np.meshgrid(Ypts,Xpts)
out = np.column_stack((Y2D.ravel(),X2D.ravel()))
Sample run -
In [39]: Xpts=np.array([5, 10, 15, 20, 25, 30, 35])
...: Ypts=np.array([3, 6, 9])
...:
In [40]: X2D,Y2D = np.meshgrid(Ypts,Xpts)
In [41]: np.column_stack((Y2D.ravel(),X2D.ravel()))
Out[41]:
array([[ 5, 3],
[ 5, 6],
[ 5, 9],
......
[35, 6],
[35, 9]])
Runtime test
As usual I am listing timings for the vectorized approaches listed thus far to solve the problem. So, the credit for the approaches listed here goes to the respective authors.
Approaches listed as functions :
def itertools_based():
Ypts = np.arange(5, 480, 5)
Xpts = np.arange(5, 640, 5)
return np.array(list(itertools.product(Xpts, Ypts)))
def c__based():
py, px = np.mgrid[5:640:5,5:480:5]
return np.c_[py.ravel(), px.ravel()]
def meshgrid_based():
Ypts = np.arange(5, 480, 5)
Xpts = np.arange(5, 640, 5)
X2D,Y2D = np.meshgrid(Ypts,Xpts)
return np.column_stack((Y2D.ravel(),X2D.ravel()))
Finally verify and time them :
In [111]: %timeit itertools_based()
...: %timeit c__based()
...: %timeit meshgrid_based()
...:
100 loops, best of 3: 9.16 ms per loop
1000 loops, best of 3: 380 µs per loop
10000 loops, best of 3: 198 µs per loop
In [112]: np.allclose(itertools_based(),c__based())
Out[112]: True
In [113]: np.allclose(itertools_based(),meshgrid_based())
Out[113]: True

Elementwise multiplication ignoring certain rows of a matrix

Let's say I have a matrix like this:
import numpy as np
a = np.array([[1, 2, 3], [89, 43, 2], [12, -3, 4], [-2, 4, 7]])
array([[ 1, 2, 3],
[89, 43, 2],
[12, -3, 4],
[-2, 4, 7]])
and a vector that looks like this:
b = np.array([1, 2, 3])
If I now want to do a elementwise multiplication I can simply do
c = a * b
and obtain
array([[ 1, 4, 9],
[89, 86, 6],
[12, -6, 12],
[-2, 8, 21]])
My question is: How can I do this kind of multiplication only for certain rows in my matrix? I currently do it like this:
E = a.copy()
# ignore these rows
ignInd = [1, 3]
for ind in xrange(a.shape[0]):
if ind not in ignInd:
E[ind, :] = a[ind, :] * b
The matrix E looks as desired (the rows 1 and 3 are the same as in a):
array([[ 1, 4, 9],
[89, 43, 2],
[12, -6, 12],
[-2, 4, 7]])
Can someone come up with a smarter solution than this?
It seems like you could just do the multiplication and then put back the original data where you want to ignore ...
>>> import numpy as np
>>> a = np.array([[1,2,3],[89,43,2],[12, -3, 4], [-2, 4, 7]])
>>> b = np.array([1,2,3])
>>> c = a * b
>>> ignInd = [1,3]
>>> c[ignInd, :]
array([[89, 86, 6],
[-2, 8, 21]])
>>> c[ignInd, :] = a[ignInd, :]
>>> c
array([[ 1, 4, 9],
[89, 43, 2],
[12, -6, 12],
[-2, 4, 7]])
You can index an NumPy array directly with another NumPy array. In your case, you have the indexes of rows you want to ignore, so you can build an array of indexes to include from this:
In [21]: ignInd = [1,3] #ignore these rows
In [22]: ind = np.array([i for i in range(a.shape[0]) if i not in ignInd])
In [23]: E2 = a.copy()
In [24]: E2[ind,:] = a[ind,:]*b
In [25]: E2
Out[25]:
array([[ 1, 4, 9],
[89, 43, 2],
[12, -6, 12],
[-2, 4, 7]])
EDIT: as #DSM comments, for large arrays it would be more efficient to build the index array using NumPy's vectorized methods, viz. ind = np.setdiff1d(np.arange(len(a)), ignInd) instead of the list comprehension used above.
You can use boolean indexing with np.in1d to select the rows excluded in the give indices list. The implementation would look something like this -
E = a.copy()
mask = ~np.in1d(np.arange(a.shape[0]),ignInd,)
E[mask] = a[mask]*b
For the 1st row in a
a[0] = a[0] * b
..and so on for the other rows.

Python: How do I multiply and sum elements across two lists (without using a library)?

I've two lists (m1 and m2) containing lists of numbers. I'm trying to do element-wise multiplication and sum including the cross product to obtaining a final list (result) as follow:
m1 = [[1, 2, 3], [4, 5, 6]]
m2 = [[7, 9, 2], [8, 1, 3]]
[[1*7+2*9+3*2,1*8+2*1+3*3],[4*7+5*9+6*2,4*8+5*1+6*3]]
result = [[31,19],[85,55]]
You can play with python built-in functions and a nested list comprehension :
>>> [[sum(t*k for t,k in zip(i,j)) for j in m2] for i in m1]
[[31, 19], [85, 55]]
You can also use itertools.product to find the products between sub-lists :
>>> from itertools import product
>>> [sum(t*k for t,k in zip(i,j)) for i,j in product(m1,m2)]
[31, 19, 85, 55]
Let's break the problem into the smaller pieces. At the lowest level, we have two small lists: [1, 2, 3] and [7, 9, 2] and want to multiply them, item by item:
item1 = [1, 2, 3]
item2 = [7, 9, 2]
zip(item1, item2) # ==> [(1, 7), (2, 9), (3, 2)]
[x * y for x, y in zip(item1, item2)] # ==> [7, 18, 6]
sum(x * y for x, y in zip(item1, item2)) # ==> 31
Now, we can put this to work inside a double loop:
[[sum(x * y for x, y in zip(item1, item2)) for item2 in m2] for item1 in m1]
# ==> [[31, 19], [85, 55]]
If you want it without importing any modules, you can do it this way:
>>> m1 = [[1, 2, 3], [4, 5, 6]]
>>> m2 = [[7, 9, 2], [8, 1, 3]]
>>> [[sum(map(lambda (s,t):s*t, zip(x,y))) for y in m2] for x in m1]
[[31, 19], [85, 55]]

how do i add an arbitrary number of lists in python

I'm trying to add lists together like a function i found on here but the problem is that i cant figure out how to add more than a specific number of lists. I want it to add an arbitrary number of lists so m+n+o should be [12,15,18] Thanks!
m = [1, 2, 3]
n = [4, 5, 6]
o = [7, 8, 9]
def zipper(a,b):
x = [a[i] + b[i] for i in range(len(a))]
print x
print zipper(m,n)
You can use zip:
In [1]: m = [1, 2, 3]
...: n = [4, 5, 6]
...: o = [7, 8, 9]
...:
In [2]: map(sum, zip(m,n,o))
Out[2]: [12, 15, 18]
or numpy.sum:
In [4]: import numpy as np
In [5]: np.sum([m,n,o],axis=0)
Out[5]: array([12, 15, 18])

Resources